Solved

AS/400 with BPCS jobs crashing with MCH3402 error - missing object - How do I find what is the missing object?

Posted on 2007-04-11
16
4,438 Views
Last Modified: 2013-12-06
Hi ,
We are running AS/400 V4R5 with BPCS 6.1. We had a system crash last thursday due to storage issue. All seemed well on Tuesday but a couple of BPCS programs do not complete correctly with error messages , the first of which is MCH3402 - trying to find object that is not there.
This occurred at the same time as a clean up of non used libraries (see storage issue above).
When I run the BPCS Program ORD570 from my client access session, the program completes like it has done everything is should , but it does not submit the batch job to complete the pick confirm process as requested. Looking at the job log for my session shows the MCH error and then get the following:


 Tried to refer to all or part of an object that no longer exists
 Internal failure occurred in query processor.                  
 SQL system error.                                              
 Cursor C not open.                                              
 Cursor C not open.                                              
 Tried to refer to all or part of an object that no longer exists
 Internal failure occurred in query processor.                  
 SQL system error.                                              
 Cursor C9 not open.                                            
 Cursor C9 not open.                                            


Question: How can I find out what the object is that it is looking for?

I would be grateful for any suggestion as to how to fix or diagnose.

I look forward to hearing from you

Andrew.
0
Comment
Question by:Cedarsrus
  • 7
  • 4
  • 3
  • +1
16 Comments
 
LVL 14

Accepted Solution

by:
daveslater earned 400 total points
ID: 18890844
Tried to refer to all or part of an object that no longer exists

This means that the onject was there but has been deleted, usually a system  users space etc but it could be a work file or a service program instance.

OS/400 creates pointer to the object, then if the object is deleted then removed from qrplobj this message is issued.

I have seen it issues on client access session,  but have never found what causes it, i'm guessing it is internal sockets.

If possible you should try a rclstg and an IPL to see if that can fix the issue.
0
 

Author Comment

by:Cedarsrus
ID: 18890895
Thanks Dave - we are planning to run the RCLSTG tonight to see if that shows anything or corrects it.
0
 
LVL 27

Assisted Solution

by:tliotta
tliotta earned 100 total points
ID: 18892205
Minor note... a MCH3402 would normally indicate that a pointer was resolved for an object but the object was not found when the pointer was used. This usually happens inside of a job -- for example, step #1 in a job locates a program and calls it, step #2 deletes the program, then step #3 tries the call again using the original resolved pointer : MCH3402.

Since your case seems not to match that kind of scenario, what with the standard programs running into problems, it almost certainly indicates that some internal pointer that's used inside of DB2 objects or library objects or something along those lines, is in trouble. Dave's suggestion of RCLSTG is the best next step.

Perhaps a logical file member got damaged, but the logical file object header has a pointer to the member and needs to be re-resolved. The BPCS program opens the file, DB2 pulls the member pointer from the LF object, then can't find the member itself. Or maybe a BPCS user profile had been creating objects in QTEMP; the crash destroyed all of the various QTEMPs that were in use, but the "owned objects" list in the *USRPRF object didn't get cleaned up.

Overall, too many possibilities to guess in a system that's object-based throughout. Quite possibly, the object is one that's below the MI boundary anyway and you'll never know what it is. Run the cleanup and review anything that gets recovered. Probably nothing but QTEMP objects, but file members might get indicated.

Tom
0
 

Author Comment

by:Cedarsrus
ID: 18894008
The RCLSTG put some journal receivers into QRCL but nothing significant. We still have the problem. Anyone with any thoughts?
Our thinking is that if we don;t , we restore back to Monday nights backup ( the clean one before the problem) and work from there?
Do either of you guys have any other thoughts?
0
 
LVL 14

Expert Comment

by:daveslater
ID: 18896289
Hi
my best guess is that the SQL objects associated with the program is damaged, missing or corrupt.
If you have the source you could try a re-compile to re-create the SQL object, failing that the quickest I would be getting the back up tapes ready.
Initially I would re-store the program library as I don’t think it is a data issue otherwise other programs would be crashing.

Dave
0
 

Author Comment

by:Cedarsrus
ID: 18896609
Unfortunately I don't have source -

More detailed messages behind the job log shown at the top:
                              Display All Messages                              
                                                             System:   UNIDARE  
 Job . . :   CDF12         User . . :   AGUY          Number . . . :   618300  
                                                                               
     Job 618300/AGUY/CDF12 started on 12/04/07 at 10:19:17 in subsystem QINTER  
       in QSYS. Job entered system on 12/04/07 at 10:19:17.                    
     Library list changed.                                                      
     Library BPCSENG not found.                                                
     Library BPCSENG not found.                                                
 2 > sql                                                                        
 2 > dspdta sf99105                                                            
     DFU could not find program SF99105 in library *LIBL.                      
 2 > dspdtaara sf99105                                                          
     Open of member EWRL03 was changed to SEQONLY(*NO).                        
     Tried to refer to all or part of an object that no longer exists.          
     Internal failure occurred in query processor.                              
     SQL system error.                                                          
     Tried to refer to all or part of an object that no longer exists.          
                                                                        More...
 Press Enter to continue.                                                      
                                                                               
 F3=Exit   F5=Refresh   F12=Cancel   F17=Top   F18=Bottom                      
                                                                               
                                                                               
                         Additional Message Information                        
                                                                               
 Message ID . . . . . . :   MCH3402       Severity . . . . . . . :   40        
 Message type . . . . . :   Escape                                              
 Date sent  . . . . . . :   12/04/07      Time sent  . . . . . . :   10:26:06  
                                                                               
 Message . . . . :   Tried to refer to all or part of an object that no longer  
   exists.                                                                      
 Cause . . . . . :   The most common cause is that a stored address to an      
   object is no longer correct because that object was deleted or part of the  
   object was deleted.                                                          
                                                                               
                                                                               
                                                                               
                                                                               
                                                                               
                                                                               
                                                                               
                                                                         Bottom
 Press Enter to continue.                                                      
                                                                               
 F3=Exit   F6=Print   F9=Display message details   F12=Cancel                  
 F21=Select assistance level                            

                            Display Message Details                            
                                                                               
 Message ID . . . . . . :   MCH3402       Severity . . . . . . . :   40        
 Date sent  . . . . . . :   12/04/07      Time sent  . . . . . . :   10:26:06  
 Message type . . . . . :   Escape                                              
 CCSID  . . . . . . . . :   65535                                              
                                                                               
 From program . . . . . . . . . :   QQQIMPLE                                    
   From library . . . . . . . . :     QSYS                                      
   From module  . . . . . . . . :     QQQIMPLE                                  
   From procedure . . . . . . . :     QQQIMPLE                                  
   From statement . . . . . . . :     21971                                    
                                                                               
 To program . . . . . . . . . . :   QQQIMPLE                                    
   To library . . . . . . . . . :     QSYS                                      
   To module  . . . . . . . . . :     QQQIMPLE                                  
   To procedure . . . . . . . . :     QQQIMPLE                                  
   To statement . . . . . . . . :     21971                                    
                                                                               
                                                                         Bottom
 Press Enter to continue.                                                      
                                                                               
 F1=Help   F3=Exit   F12=Cancel                                                
                                                                                                       
                                                                               
                         Additional Message Information                        
                                                                               
 Message ID . . . . . . :   CPF4204       Severity . . . . . . . :   50        
 Message type . . . . . :   Escape                                              
 Date sent  . . . . . . :   12/04/07      Time sent  . . . . . . :   10:26:07  
                                                                               
 Message . . . . :   Internal failure occurred in query processor.              
 Cause . . . . . :   A system failure has occurred in the query processor      
   program.  The query definition template number is 0.                        
 Recovery  . . . :   See the low-level messages.  Correct any errors and try    
   your request again.  If the problem continues, report the problem (ANZPRB    
   command).                                                                    
                                                                               
                                                                               
                                                                               
                                                                               
                                                                               
                                                                               
                                                                         Bottom
 Press Enter to continue.                                                      
                                                                               
 F3=Exit   F6=Print   F9=Display message details   F12=Cancel                  
 F21=Select assistance level                                                    
                                                                               
                            Display Message Details                            
                                                                               
 Message ID . . . . . . :   CPF4204       Severity . . . . . . . :   50        
 Date sent  . . . . . . :   12/04/07      Time sent  . . . . . . :   10:26:07  
 Message type . . . . . :   Escape                                              
 CCSID  . . . . . . . . :   65535                                              
                                                                               
 From program . . . . . . . . . :   QQQQUERY                                    
   From library . . . . . . . . :     QSYS                                      
   From module  . . . . . . . . :     QQQQUERY                                  
   From procedure . . . . . . . :     STARFC                                    
   From statement . . . . . . . :     17990                                    
                                                                               
 To program . . . . . . . . . . :   QQQQUERY                                    
   To library . . . . . . . . . :     QSYS                                      
   To module  . . . . . . . . . :     QQQQUERY                                  
   To procedure . . . . . . . . :     QQQQUERY                                  
   To statement . . . . . . . . :     9031                                      
                                                                               
                                                                         Bottom
 Press Enter to continue.                                                      
                                                                               
 F1=Help   F3=Exit   F12=Cancel                                                
                                                                               
                         Additional Message Information                        
                                                                               
 Message ID . . . . . . :   SQL0901       Severity . . . . . . . :   50        
 Message type . . . . . :   Diagnostic                                          
 Date sent  . . . . . . :   12/04/07      Time sent  . . . . . . :   10:26:07  
                                                                               
 Message . . . . :   SQL system error.                                          
 Cause . . . . . :   An SQL system error has occurred.  The current SQL        
   statement cannot be completed successfully.  The error will not prevent      
   other SQL statements from being processed. Previous messages may indicate    
   that there is a problem with the SQL statement and SQL did not correctly    
   diagnose the error. If precompiling, processing will not continue beyond    
   this statement.                                                              
 Recovery  . . . :   See the previous messages to determine if there is a      
   problem with the SQL statement. To view the messages, use the DSPJOBLOG      
   command if running interactively, or the WRKJOB command to view the output  
   of a precompile.  An application program receiving this return code may      
   attempt further SQL statements.  Correct any errors and try the request      
                                                                        More...
 Press Enter to continue.                                                      
                                                                               
 F3=Exit   F6=Print   F9=Display message details   F12=Cancel                  
 F21=Select assistance level                                                    
                                                                               

                             Display Message Details                            
                                                                               
 Message ID . . . . . . :   SQL0901       Severity . . . . . . . :   50        
 Date sent  . . . . . . :   12/04/07      Time sent  . . . . . . :   10:26:07  
 Message type . . . . . :   Diagnostic                                          
 CCSID  . . . . . . . . :   65535                                              
                                                                               
 From program . . . . . . . . . :   QSQROUTE                                    
   From library . . . . . . . . :     QSYS                                      
   From module  . . . . . . . . :     QSQFETCH                                  
   From procedure . . . . . . . :     CK_DEBUG                                  
   From statement . . . . . . . :     14551                                    
                                                                               
 To program . . . . . . . . . . :   QSQROUTE                                    
   To library . . . . . . . . . :     QSYS                                      
   To module  . . . . . . . . . :     QSQFETCH                                  
   To procedure . . . . . . . . :     CK_DEBUG                                  
   To statement . . . . . . . . :     14551                                    
                                                                               
                                                                         Bottom
 Press Enter to continue.                                                      
                                                                               
 F1=Help   F3=Exit   F12=Cancel                                                
0
 
LVL 14

Expert Comment

by:daveslater
ID: 18896781
Hi
I would re-store the program library to a different library.
Then try the program over the restored library.

My gut tells me that it is the associated SQL object, but I have not proof.

Dave
0
 

Author Comment

by:Cedarsrus
ID: 18900962
Infor / SSA have recommeded that the problem will be remedied by installing the latest Group PTF's for V4R5- Aaaggggh!.

Now where can I get a set of those ?
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 27

Expert Comment

by:tliotta
ID: 18902201
Ouch! Do you mean OS/400 V4R5? Is that the version of operating system you're running?

I should've added more info earlier, but didn't think. RCLSTG generally only applies to objects _marked_ as "damaged". In at least some cases, objects only get marked during a save operation.

I.e., you might have to run the related libraries through a SAVLIB command before RCLSTG works to its fullest extent.

Also, note that RCLSTG can run in different modes. By default, it _should_ run appropriately; but you'll want to ensure that it selects *ALL rather than only *DBXREF. I.e., RCLSTG SELECT(*ALL).

Tom
0
 

Author Comment

by:Cedarsrus
ID: 18904931
Anyone out there got SF99105 handy - the PTF for the database group?
0
 
LVL 13

Expert Comment

by:_b_h
ID: 18908048
Did they identify specific PTF(s)?
Barry
0
 
LVL 27

Expert Comment

by:tliotta
ID: 18908867
Also, verify first whether or not you already have the "latest" DB2 group installed. It's been too long since I've seen V4R5 -- is the level held in a data area such as QGPL/QSF99105 or QGPL/SF99105?

Tom
0
 
LVL 13

Expert Comment

by:_b_h
ID: 18909633
Use DSPDTAARA SF99105 to see the level.
Barry
0
 

Author Comment

by:Cedarsrus
ID: 18911678
Hmmmm Weird. A programmer gets in touch on my way home on friday night and say "you don't need the PTF. It's likely to be data."

Blow me down - she was right - after an hour on the phone late friday night we had the billing run going again just by saving, deleting and restoring 2 program objects.

Hopefully the other program will be up and running by Monday.
0
 
LVL 14

Expert Comment

by:daveslater
ID: 18916636
Cedarsrus
I asked you to try that in
ID:18896781 Author:daveslater Date:04.12.2007 at 11:48AM BST
0
 

Author Comment

by:Cedarsrus
ID: 18916767
Ok,

Heres the result: Deleting and restoring the program objects took the problem away. I had tried to isolate an individual program and ran that from a library that was higher in the library list with little success - There were other elements and aother programs that gave the same errors. My limited understanding is that the pointers in the SQL objects got "damaged" and deleting and restoring them put it right. These objects did not appear as damaged by SAVLIB or RCLSTG. I'm grateful to those who answered as it was of some support.
Thanks for listening!
Andrew
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
AS400 application emulation 7 48
Windows 10 PRo Upgrade 21 62
Thin secure Windows 10 5 48
AS400 - SBMJOB with command prompter 3 27
I don't know if many of you have made the great mistake of using the Cisco Thin Client model with the management software VXC. If you have then you are probably more then familiar with the incredibly clunky interface, the numerous work arounds, and …
Sometimes a user will call me frantically, explaining that something has gone wrong and they have tried everything (read - they have messed it up more and now need someone to clean up) and it still does no good, can I help them?!  Usually the standa…
This video discusses moving either the default database or any database to a new volume.
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now