Solved

Question with few Qshell commands runs with IFS directory

Posted on 2007-03-27
17
1,766 Views
Last Modified: 2013-12-06
Hi,

I have one question about few command used with IFS directory. We receive few files from different system via NDM ( Connectivity Services). That files directly comes into IFS directory (lets say DirA) . One job ( lets say JobA) keep polling that file every 10 min. If he gets any file then it zips the file into different IFS directory (Lets say DirB) and deletes the file from  original location.
When JobA finds any file in DirA then it zips all the file which are available with *.* parameter. So all the file will be ziped into one file.Then it deletes all the file again by *.* paramater.
Now problem is when JobA start running and zips all the file with *.* lets say it has got 3 files.And the same time transmission of 4th file is running.That means 4th file has not fully arrived. So it zip (*.*) that file wont get include. But when next command deletd *.* runs the transmisson of 4th file has completed and so that file gets deleted with out getting zipped.
As a soltuion of this what I am planning is I will list all the files with 'LS' command. I will take that output in to one variable and fetch all files one by one. After fetching the file from variable I will copy the file into new directory and then delete from original directory. Once I copy and delete all the file I will zip those files from new directory.This will insure that I will copy only those files which have been included into 'LS' command. So if transmission of any file is runnign while I run the 'LS' command that file wont get included into that and that file wont get copied and deleted.
Is this a right solution and if yes then will the file which transmisson is running while I run the LS command be included into the output of 'LS' command ?

My apologies for the length of this question but I wanted to make sure that every one understand my question.

Regards
Prashant
0
Comment
Question by:inform_prashant
  • 7
  • 7
17 Comments
 
LVL 27

Expert Comment

by:tliotta
Comment Utility
Prashant:

If you go to a PC and access a mapped drive to your IFS, you can create a new text file and open it with Notepad. While you have the file open, you can type lines into it and periodically use the "Save" menu item to save the content that you've typed so far.

Note that the file can stay open until you exit Notepad.

In the meantime, you can start an interactive QShell session in a workstation window and see the progress. Set the mapped directory as your current directory and enter the ls utility. It should list your text file even though you also have it open in your Notepad window. If you enter the cat utility and reference your text file, it should list any lines that you've already saved.

That should give you enough info to test a few things out.

Tom
0
 
LVL 3

Expert Comment

by:RQnone
Comment Utility
I think you have the correct solution but I would suggest you change the order in which it runs.
The moment you start writing a file to the IFS it will show up with 'LS' even though its not complete.

First use your 'LS' to list the files. I assume you are sending this to a DB2 file. I prefer to use the 'FIND" command because it includes the full path from the root directory
QSH CMD('find /DirA/* >/qsys.lib/mylib.lib/ifsfiles.file/ifsfiles.mbr')

Second delay your job for the 10 minutes.

Third read from the file MYLIB/IFSFILES and use command MOV to move the objects from DirA to DirB and zip from DirB. This will just save you the extra step of deleting the files in DirA.
 
I should admit, the only way this will work is if the files take less then 10 minutes to write to the directory. If there is a posibility they take more then that long you may want to consider increasing the delay or using an API to check for locks on the IFS object but that gets very complicated.

Dan
0
 
LVL 27

Expert Comment

by:tliotta
Comment Utility
Prashant:

AFAIK, the only real solution is not to use shell utilities, but to write against the IFS APIs instead. While the APIs are more difficult, they allow the fine control that you need.

For example, the open() API allows you to specify O_SHARE_NONE in the oflag parameter to set an exclusive lock on a file. It will also return a [EBUSY] error if some other process is using the file.

By combining opendir() to get a list of files and open() to process the list while testing locks, you should be able to handle all possibilities.

Tom
0
 

Author Comment

by:inform_prashant
Comment Utility
Dan,
As you adviced me the delay of 10 min but the problem is this jobA runs every 10 min and polls for the file.So 10:00, 10:10, 10:15....this job will run and look for newfiles. So I think that will be tough thing to do for me. Ans second thing is even if I put 5 min delay then there could be an issue as below.
Let's say JobA ran at 10:00 finds 3 completed files and 4th beaing written. So after 5 min I will start  moving all the file but mean time I might get 5th file as well which is being written so we might have issue with 5th file then.

Tom,
Can you give me example about how to use the API to list out all the complated files and move those files into new directory.

Thanks to both for your valuable hlep...

Regards
Prashant Patel
0
 
LVL 27

Expert Comment

by:tliotta
Comment Utility
Prashant:

You might want to visit here:

http://www.scottklement.com/rpg/ifs_ebook/opendir.html

That's in the middle of Scott Klement's e-book right where he discusses opendir() in ILE RPG. You can take the [Up] link at the bottom of the page to jump to the contents of the chapter or navigate elsewhere. Almost everything you'll want to know is there, and he does an excellent job of keeping things simple enough.

Tom
0
 

Author Comment

by:inform_prashant
Comment Utility
Hi Tom,
I saw the link you provided and that is very usefule. Scott is doing excellent job and thanks to you for providing this information.

Now I think I should incorporate the below logic to solve my issue.(Please correct me if  am wrong !!!)

First I will list all the files with 'LS' command. Then for all files I will check the permission to list out all the files for which we have write access with Access() API. (Here I am asumming that for any file which transmission is still running , I won't get write access) Once I gets list of the files for which I have write access I will move all these files again via QShell command. (Or can you advice mw that whcih API is avaialbe to move the files from one directory to another). Then I will process al the files from new directory and delete those files from that.

Is this correct or you have still have some suggestion on this ?

Thanks again....

Regards
Prashant Patel
0
 
LVL 27

Expert Comment

by:tliotta
Comment Utility
Prashant:

That sounds like a very good plan. I haven't needed to do what you're doing, so I can't be certain. However, it should be easy to test by a similar technique to my first comment. Access a mapped drive from your PC. Create two text files with Notepad, but leave the 2nd text file open. If you test your program against that directory, you should be able to capture the first file and not the 2nd file because you won't get proper access.

If nothing else, you'll get some direct evidence of what works in your environment. Keep in mind that authorities can affect results.

You might also be able to use the MOV command in CL. That's one less point to have to jump into Qshell.

Tom
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 

Author Comment

by:inform_prashant
Comment Utility
Hi Tom,

OK, Let me try then see what happens...

Thanks a lot for your help...

Regards
Prashant Patel
0
 

Author Comment

by:inform_prashant
Comment Utility
Hi,

Is ther eany way to check the access of file in IFS directory from CL program ? ( may be asking a meaningless question.....but just thought)....

Thanks
Prashant Patel
0
 

Author Comment

by:inform_prashant
Comment Utility
Hey Tom,

I did one test as below.

I start sending a hugh file to AS/400 system from my local machine via FTP. Same time I started Checking access of all file in IFS. The new file got included into LS coomand and when I checked the access it gives me exclusive rights even if the file is still not fully tansferred. Can you help me now ?

Regards
Prashant
0
 

Author Comment

by:inform_prashant
Comment Utility
Any answer please Tom ??

Regards
Prashant Patel
0
 
LVL 27

Expert Comment

by:tliotta
Comment Utility
Yes and no. I have a trivial ILE CL proc that's calling a couple APIs that I _hope_ will provide an answer but the results are not good yet. The result always returns errno 3021 (EINVAL) and I can't find out yet what's invalid about the values. On top of that, our in-house Exchange server is throwing fits today, so I can't access some resource info I'd really like to grab.

It's likely to be another day before I can hit the next step.

Tom
0
 
LVL 27

Accepted Solution

by:
tliotta earned 250 total points
Comment Utility
Prashant:

Assuming you have at least V5R3 available, here's an ILE CL proc that demonstrates a possibility:

------------ Begin
pgm

   dcl   &fname       *char  512
   dcl   &oflag       *int     4

/* See QSYSINC/H.FCNTL for more values... */
/* Leading zeroes indicate constants are OCTAL... */
/* #define O_SHARE_RDONLY 000000200000 Share with readers only */
/* #define O_SHARE_WRONLY 000000400000 Share with writers only */
/* #define O_SHARE_RDWR   000001000000 Share with readers and writers */
/* #define O_SHARE_NONE   000002000000 Share with neither */

   dcl   &oshrnone    *int     4     value( 524288 )
   dcl   &oshrrdwr    *int     4     value( 262144 )
   dcl   &oshrrd      *int     4     value( 65536 )
   dcl   &ordwr       *int     4     value( 4 )
   dcl   &ord         *int     4     value( 1 )
   dcl   &owronly     *int     4     value( 2 )
   dcl   &otext       *int     4     value( 16777216 )

   dcl   &x00         *char    1     value( x'00' )
   dcl   &x0000       *char    4     value( x'00000000' )

   dcl   &rc          *int     4     value( 0 )
   dcl   &curerrno    *char    4     value( ' ' )

   dcl   &fildes      *int     4     value( 0 )

   dcl   &rc_fail     *int     4     value( -1 )

/* Experiment with write-only, read-write, whatever...*/
/* Experiment with share-none, share-read, whatever... */

   chgvar      &oflag             ( &oshrnone + &owronly  + &otext )

/* Use a streamfile name that works for you... */

   chgvar      &fname             ( +
                                    '/home/TOML/big.savf' *cat  +
                                    &x00                       +
                                  )

   callprc     'open'             ( +
                                    &fname       +
                                  ( &oflag   *byval ) +
                                  ( &x0000   *byval ) +
                                  ( &x0000   *byval ) +
                                  ) +
                            rtnval( &rc )

/* Note: &rc *eq &fildes on success; x'FFFFFFFF' on fail... */

   if ( &rc *eq &rc_fail )  do
      callprc  'PLURERRNO'        ( +
                                    &curerrno         +
                                  )
      if ( &curerrno *eq '3029' )  +
         sndpgmmsg   msg( 'File reports EBUSY...' ) topgmq( *EXT )
   enddo
   else  do

      chgvar   &fildes              &rc
      callprc  'close'            ( +
                                  ( &fildes  *byval ) +
                                  ) +
                            rtnval( &rc )
      if ( &rc *eq &rc_fail )  do
         callprc  'PLURERRNO'     ( +
                                    &curerrno         +
                                  )
         dmpclpgm
      enddo
   enddo

dmpclpgm

   return

endpgm
------------ End

You could change it to accept &fname (file name) as a parm, but I just wanted to be sure that various flag bits did what they're documented as doing.

The proc sets bits to attempt to open a streamfile with some specific sharing allowed. The sharing here is O_SHARE_NONE as documented for the fcntl() API. Also, it opens the file for O_WRONLY, but you could possibly use less restrictive modes, possibly even just O_RDONLY. (I didn't test O_RDONLY.)

The open() is either successful or it fails. If it fails, the return-code is x'FFFFFFFF' which is integer -1. And if it fails, I retrieve the latest errno by calling an ILE RPG proc. The RPG calls the __errno builtin API, converts the value into a 4-digit character representation and passes that value back out. If that value is '3029', then the system is saying that the requested sharing mode cannot be be set at this time. The value '3029' means EBUSY and can be seen in QSYSINC/H.ERRNO.

The RPG is needed because ILE CL can't call the __errno builtin at V5R3. At V5R4, pointer variables can be declared in CL and RPG wouldn't be needed. Almost all of the CL can be done prior to V5R3 except for passing the &oflag parameter to open() by value; pre-V5R3 can only pass by reference. (However, the program can be compiled on a V5R4 system back to V5R2 because IBM put that capability into the previous-release capabilities.)

The RPG looks like:

------- Begin
     H Debug
     H Nomain
     H*dftactgrp( *NO )
     H*    actgrp( *CALLER )
     H*    bnddir( 'QC2LE' )
     H indent( '  ' )
      /eject
      *--------------------------------------------------
      *  Procedure definitions
      *--------------------------------------------------

     D PLURERRNO       pr                  extproc( 'PLURERRNO' )
     d  CurErrNo                      4
      /eject
      ****************************************************************
      *      D E F I N I T I O N     S P E C I F I C A T I O N       *
      ****************************************************************
      *
      * Program Status Data Structure
      *
     D PGMDS          sds
     d  Pgmq##           *PROC
     d  ErrorSts         *STATUS
     d  PrvStatus             16     20s 0
     d  SrcLinNum             21     28
     d  Routine          *ROUTINE
     d  NumParms         *PARMS
     d  ExcpType              40     42
     d  ExcpNum               43     46
     d  MsgId                 40     46
      *
     d  PgmLib                81     90
     d  ExcpData              91    170
     d  ExcpId               171    174
     d  LastFile             175    184
     d  FileErr              209    243
     d  JobName              244    253
     d  User                 254    263
     d  JobNumA              264    269
     d  JobNum               264    269s 0
     d  JobDate              270    275s 0
     d  RunDate              276    281s 0
     d  RunTime              282    287s 0
     d  PgmCrtDt             288    293
     d  PgmCrtTm             294    299
     d  CmplrLvl             300    303
     d  SrcFile              304    313
     d  SrcLib               314    323
     d  SrcMbr               324    333
     d  ProcPgm              334    343
     d  ProcMod              344    353
      /eject
     D p_errno         s               *
     D errno           s             10i 0 based( p_errno )
     D cerrno          s              4s 0

      * Error number
     D errnof          pr              *   extproc( '__errno' )

     ** PSSR fields
     D ReturnPt        s              6
      /eject

      *--------------------------------------------------

      *  For messages that should be sent back to the calling program.

     DMsgfloc          c                   'QCPFMSG   *LIBL'
     DMsgAccObj        c                   'IFS access error'

     DProgramVar       ds
     d msglen                  1      4b 0
     d stkctr                  5      8b 0
     d rtvlen                  9     12b 0
     d msgqln                 13     16b 0
     d pgmwtt                 17     20b 0

     Dmsgerr           ds
     d provid                  1      4b 0 inz( 56 )
     d avail                   5      8b 0
     d rtnmsg                  9     15
     d rsvr                   16     16
     d rtndta                 17     56
      /eject
      *=====================================================================
      *   S U B P R O C E D U R E   D E F I N I T I O N S
      *=====================================================================


     p PLURERRNO       B                   export
      *--------------------------------------------------
      *  Procedure Interface
      *--------------------------------------------------

     D PLURERRNO       PI

     d  CurErrNo                      4

      * ================ Begin Calculation Specifications ============

     C                   eval      p_errno    = errnof
     C                   eval      cerrno     = errno
     C                   move      cerrno        CurErrNo

     C                   return


      /eject
C*--- *--------------------------------------------------------------------
     C     Sendmsg       begsr
C*--- *--------------------------------------------------------------------

     C                   call      'QMHSNDPM'
     c                   parm      'CPF9898'     msgid             7
     c                   parm      Msgfloc       msgf             20
     c                   parm                    msgdta           40
     c                   parm      40            msglen
     c                   parm      '*INFO     '  msgtype          10
     c                   parm      '*         '  stkent           10
     c                   parm      2             stkctr
     c                   parm                    msgkey            4
     c                   parm                    msgerr

     C                   endsr

      *==============================================================*
      *                                                              *
      * Program Error Subroutine                                     *
      *                                                              *
     C     *PSSR         begsr
      *==============================================================*
      *
      *  End the program
      *

     C                   move      '*CANCL'      ReturnPt
      *
     C                   endsr
     p PLURERRNO       E
------- End

It could be smaller, but I left in bits from the module that I cloned it from. The actual real working parts are in the procedure prototypes and the three simple C-specs.

Compile the two modules. Then CRTPGM with BNDDIR(QC2LE). Experiment as you wish.

The CL shows the two APIs -- open() and close() -- as well as the flags that are needed. The RPG shows __errno. That's all that it took for me.

One added item...

Once it worked, I realized that the test with Notepad was flawed. Apparently, Notepad doesn't put any locks nor any sharing modes on the file. It was perfectly possible to have the same file open with Notepad and with EDTF at the same time. (It looks like EDTF does things about the same way as Notepad.)

But, I had a 400MB streamfile that I could FTP and test with. As long as FTP was handling the file, I got EBUSY reported. As soon as FTP finished, the open() would succeed.

I wouldn't recommend actually using CL for this purpose. You'd be better off converting the CL to RPG, C or even COBOL. But the APIs will be the same and the binary values will be the same.

Hope that helps you move forward.

Tom
0
 

Author Comment

by:inform_prashant
Comment Utility
Hey Tom,

I was experimetning the same thing in CL/RPG. I cretaed one small RPG just to check with flag O_SHARE_NONE. I called that proc from CL to open the IFS file which was abc.TXT file. But what I found is though there was no other job who has opened that file still my procedure was getting -1 in resonse from Open() API. Do you know why ?
0
 
LVL 27

Expert Comment

by:tliotta
Comment Utility
Prashant:

Without knowing the errno, there's no way to guess.

When the RC is -1, you must retrieve errno to see what error condition exists. And you must retrieve errno before any other procedure might a different errno.

At first, I kept getting RC=-1. Then I had it pointed out to me that I had taken some of the _octal_ constants and converted them as hexadecimal. After smacking my forehead a couple times and replacing incorrect constants, everything worked as expected.

Bind in the proc that I supplied or simply bind in the __errno builtin into your RPG and see what error is happening whenever you run into RC=-1.

Tom
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Many people tend to confuse the function of a virus with the one of adware, this misunderstanding of the basic of what each software is and how it operates causes users and organizations to take the wrong security measures that would protect them ag…
Sometimes a user will call me frantically, explaining that something has gone wrong and they have tried everything (read - they have messed it up more and now need someone to clean up) and it still does no good, can I help them?!  Usually the standa…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now