inform_prashant
asked on
Question with few Qshell commands runs with IFS directory
Hi,
I have one question about few command used with IFS directory. We receive few files from different system via NDM ( Connectivity Services). That files directly comes into IFS directory (lets say DirA) . One job ( lets say JobA) keep polling that file every 10 min. If he gets any file then it zips the file into different IFS directory (Lets say DirB) and deletes the file from original location.
When JobA finds any file in DirA then it zips all the file which are available with *.* parameter. So all the file will be ziped into one file.Then it deletes all the file again by *.* paramater.
Now problem is when JobA start running and zips all the file with *.* lets say it has got 3 files.And the same time transmission of 4th file is running.That means 4th file has not fully arrived. So it zip (*.*) that file wont get include. But when next command deletd *.* runs the transmisson of 4th file has completed and so that file gets deleted with out getting zipped.
As a soltuion of this what I am planning is I will list all the files with 'LS' command. I will take that output in to one variable and fetch all files one by one. After fetching the file from variable I will copy the file into new directory and then delete from original directory. Once I copy and delete all the file I will zip those files from new directory.This will insure that I will copy only those files which have been included into 'LS' command. So if transmission of any file is runnign while I run the 'LS' command that file wont get included into that and that file wont get copied and deleted.
Is this a right solution and if yes then will the file which transmisson is running while I run the LS command be included into the output of 'LS' command ?
My apologies for the length of this question but I wanted to make sure that every one understand my question.
Regards
Prashant
I have one question about few command used with IFS directory. We receive few files from different system via NDM ( Connectivity Services). That files directly comes into IFS directory (lets say DirA) . One job ( lets say JobA) keep polling that file every 10 min. If he gets any file then it zips the file into different IFS directory (Lets say DirB) and deletes the file from original location.
When JobA finds any file in DirA then it zips all the file which are available with *.* parameter. So all the file will be ziped into one file.Then it deletes all the file again by *.* paramater.
Now problem is when JobA start running and zips all the file with *.* lets say it has got 3 files.And the same time transmission of 4th file is running.That means 4th file has not fully arrived. So it zip (*.*) that file wont get include. But when next command deletd *.* runs the transmisson of 4th file has completed and so that file gets deleted with out getting zipped.
As a soltuion of this what I am planning is I will list all the files with 'LS' command. I will take that output in to one variable and fetch all files one by one. After fetching the file from variable I will copy the file into new directory and then delete from original directory. Once I copy and delete all the file I will zip those files from new directory.This will insure that I will copy only those files which have been included into 'LS' command. So if transmission of any file is runnign while I run the 'LS' command that file wont get included into that and that file wont get copied and deleted.
Is this a right solution and if yes then will the file which transmisson is running while I run the LS command be included into the output of 'LS' command ?
My apologies for the length of this question but I wanted to make sure that every one understand my question.
Regards
Prashant
I think you have the correct solution but I would suggest you change the order in which it runs.
The moment you start writing a file to the IFS it will show up with 'LS' even though its not complete.
First use your 'LS' to list the files. I assume you are sending this to a DB2 file. I prefer to use the 'FIND" command because it includes the full path from the root directory
QSH CMD('find /DirA/* >/qsys.lib/mylib.lib/ifsfi
Second delay your job for the 10 minutes.
Third read from the file MYLIB/IFSFILES and use command MOV to move the objects from DirA to DirB and zip from DirB. This will just save you the extra step of deleting the files in DirA.
I should admit, the only way this will work is if the files take less then 10 minutes to write to the directory. If there is a posibility they take more then that long you may want to consider increasing the delay or using an API to check for locks on the IFS object but that gets very complicated.
Dan
The moment you start writing a file to the IFS it will show up with 'LS' even though its not complete.
First use your 'LS' to list the files. I assume you are sending this to a DB2 file. I prefer to use the 'FIND" command because it includes the full path from the root directory
QSH CMD('find /DirA/* >/qsys.lib/mylib.lib/ifsfi
Second delay your job for the 10 minutes.
Third read from the file MYLIB/IFSFILES and use command MOV to move the objects from DirA to DirB and zip from DirB. This will just save you the extra step of deleting the files in DirA.
I should admit, the only way this will work is if the files take less then 10 minutes to write to the directory. If there is a posibility they take more then that long you may want to consider increasing the delay or using an API to check for locks on the IFS object but that gets very complicated.
Dan
Prashant:
AFAIK, the only real solution is not to use shell utilities, but to write against the IFS APIs instead. While the APIs are more difficult, they allow the fine control that you need.
For example, the open() API allows you to specify O_SHARE_NONE in the oflag parameter to set an exclusive lock on a file. It will also return a [EBUSY] error if some other process is using the file.
By combining opendir() to get a list of files and open() to process the list while testing locks, you should be able to handle all possibilities.
Tom
AFAIK, the only real solution is not to use shell utilities, but to write against the IFS APIs instead. While the APIs are more difficult, they allow the fine control that you need.
For example, the open() API allows you to specify O_SHARE_NONE in the oflag parameter to set an exclusive lock on a file. It will also return a [EBUSY] error if some other process is using the file.
By combining opendir() to get a list of files and open() to process the list while testing locks, you should be able to handle all possibilities.
Tom
ASKER
Dan,
As you adviced me the delay of 10 min but the problem is this jobA runs every 10 min and polls for the file.So 10:00, 10:10, 10:15....this job will run and look for newfiles. So I think that will be tough thing to do for me. Ans second thing is even if I put 5 min delay then there could be an issue as below.
Let's say JobA ran at 10:00 finds 3 completed files and 4th beaing written. So after 5 min I will start moving all the file but mean time I might get 5th file as well which is being written so we might have issue with 5th file then.
Tom,
Can you give me example about how to use the API to list out all the complated files and move those files into new directory.
Thanks to both for your valuable hlep...
Regards
Prashant Patel
As you adviced me the delay of 10 min but the problem is this jobA runs every 10 min and polls for the file.So 10:00, 10:10, 10:15....this job will run and look for newfiles. So I think that will be tough thing to do for me. Ans second thing is even if I put 5 min delay then there could be an issue as below.
Let's say JobA ran at 10:00 finds 3 completed files and 4th beaing written. So after 5 min I will start moving all the file but mean time I might get 5th file as well which is being written so we might have issue with 5th file then.
Tom,
Can you give me example about how to use the API to list out all the complated files and move those files into new directory.
Thanks to both for your valuable hlep...
Regards
Prashant Patel
Prashant:
You might want to visit here:
http://www.scottklement.com/rpg/ifs_ebook/opendir.html
That's in the middle of Scott Klement's e-book right where he discusses opendir() in ILE RPG. You can take the [Up] link at the bottom of the page to jump to the contents of the chapter or navigate elsewhere. Almost everything you'll want to know is there, and he does an excellent job of keeping things simple enough.
Tom
You might want to visit here:
http://www.scottklement.com/rpg/ifs_ebook/opendir.html
That's in the middle of Scott Klement's e-book right where he discusses opendir() in ILE RPG. You can take the [Up] link at the bottom of the page to jump to the contents of the chapter or navigate elsewhere. Almost everything you'll want to know is there, and he does an excellent job of keeping things simple enough.
Tom
ASKER
Hi Tom,
I saw the link you provided and that is very usefule. Scott is doing excellent job and thanks to you for providing this information.
Now I think I should incorporate the below logic to solve my issue.(Please correct me if am wrong !!!)
First I will list all the files with 'LS' command. Then for all files I will check the permission to list out all the files for which we have write access with Access() API. (Here I am asumming that for any file which transmission is still running , I won't get write access) Once I gets list of the files for which I have write access I will move all these files again via QShell command. (Or can you advice mw that whcih API is avaialbe to move the files from one directory to another). Then I will process al the files from new directory and delete those files from that.
Is this correct or you have still have some suggestion on this ?
Thanks again....
Regards
Prashant Patel
I saw the link you provided and that is very usefule. Scott is doing excellent job and thanks to you for providing this information.
Now I think I should incorporate the below logic to solve my issue.(Please correct me if am wrong !!!)
First I will list all the files with 'LS' command. Then for all files I will check the permission to list out all the files for which we have write access with Access() API. (Here I am asumming that for any file which transmission is still running , I won't get write access) Once I gets list of the files for which I have write access I will move all these files again via QShell command. (Or can you advice mw that whcih API is avaialbe to move the files from one directory to another). Then I will process al the files from new directory and delete those files from that.
Is this correct or you have still have some suggestion on this ?
Thanks again....
Regards
Prashant Patel
Prashant:
That sounds like a very good plan. I haven't needed to do what you're doing, so I can't be certain. However, it should be easy to test by a similar technique to my first comment. Access a mapped drive from your PC. Create two text files with Notepad, but leave the 2nd text file open. If you test your program against that directory, you should be able to capture the first file and not the 2nd file because you won't get proper access.
If nothing else, you'll get some direct evidence of what works in your environment. Keep in mind that authorities can affect results.
You might also be able to use the MOV command in CL. That's one less point to have to jump into Qshell.
Tom
That sounds like a very good plan. I haven't needed to do what you're doing, so I can't be certain. However, it should be easy to test by a similar technique to my first comment. Access a mapped drive from your PC. Create two text files with Notepad, but leave the 2nd text file open. If you test your program against that directory, you should be able to capture the first file and not the 2nd file because you won't get proper access.
If nothing else, you'll get some direct evidence of what works in your environment. Keep in mind that authorities can affect results.
You might also be able to use the MOV command in CL. That's one less point to have to jump into Qshell.
Tom
ASKER
Hi Tom,
OK, Let me try then see what happens...
Thanks a lot for your help...
Regards
Prashant Patel
OK, Let me try then see what happens...
Thanks a lot for your help...
Regards
Prashant Patel
ASKER
Hi,
Is ther eany way to check the access of file in IFS directory from CL program ? ( may be asking a meaningless question.....but just thought)....
Thanks
Prashant Patel
Is ther eany way to check the access of file in IFS directory from CL program ? ( may be asking a meaningless question.....but just thought)....
Thanks
Prashant Patel
ASKER
Hey Tom,
I did one test as below.
I start sending a hugh file to AS/400 system from my local machine via FTP. Same time I started Checking access of all file in IFS. The new file got included into LS coomand and when I checked the access it gives me exclusive rights even if the file is still not fully tansferred. Can you help me now ?
Regards
Prashant
I did one test as below.
I start sending a hugh file to AS/400 system from my local machine via FTP. Same time I started Checking access of all file in IFS. The new file got included into LS coomand and when I checked the access it gives me exclusive rights even if the file is still not fully tansferred. Can you help me now ?
Regards
Prashant
ASKER
Any answer please Tom ??
Regards
Prashant Patel
Regards
Prashant Patel
Yes and no. I have a trivial ILE CL proc that's calling a couple APIs that I _hope_ will provide an answer but the results are not good yet. The result always returns errno 3021 (EINVAL) and I can't find out yet what's invalid about the values. On top of that, our in-house Exchange server is throwing fits today, so I can't access some resource info I'd really like to grab.
It's likely to be another day before I can hit the next step.
Tom
It's likely to be another day before I can hit the next step.
Tom
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hey Tom,
I was experimetning the same thing in CL/RPG. I cretaed one small RPG just to check with flag O_SHARE_NONE. I called that proc from CL to open the IFS file which was abc.TXT file. But what I found is though there was no other job who has opened that file still my procedure was getting -1 in resonse from Open() API. Do you know why ?
I was experimetning the same thing in CL/RPG. I cretaed one small RPG just to check with flag O_SHARE_NONE. I called that proc from CL to open the IFS file which was abc.TXT file. But what I found is though there was no other job who has opened that file still my procedure was getting -1 in resonse from Open() API. Do you know why ?
Prashant:
Without knowing the errno, there's no way to guess.
When the RC is -1, you must retrieve errno to see what error condition exists. And you must retrieve errno before any other procedure might a different errno.
At first, I kept getting RC=-1. Then I had it pointed out to me that I had taken some of the _octal_ constants and converted them as hexadecimal. After smacking my forehead a couple times and replacing incorrect constants, everything worked as expected.
Bind in the proc that I supplied or simply bind in the __errno builtin into your RPG and see what error is happening whenever you run into RC=-1.
Tom
Without knowing the errno, there's no way to guess.
When the RC is -1, you must retrieve errno to see what error condition exists. And you must retrieve errno before any other procedure might a different errno.
At first, I kept getting RC=-1. Then I had it pointed out to me that I had taken some of the _octal_ constants and converted them as hexadecimal. After smacking my forehead a couple times and replacing incorrect constants, everything worked as expected.
Bind in the proc that I supplied or simply bind in the __errno builtin into your RPG and see what error is happening whenever you run into RC=-1.
Tom
If you go to a PC and access a mapped drive to your IFS, you can create a new text file and open it with Notepad. While you have the file open, you can type lines into it and periodically use the "Save" menu item to save the content that you've typed so far.
Note that the file can stay open until you exit Notepad.
In the meantime, you can start an interactive QShell session in a workstation window and see the progress. Set the mapped directory as your current directory and enter the ls utility. It should list your text file even though you also have it open in your Notepad window. If you enter the cat utility and reference your text file, it should list any lines that you've already saved.
That should give you enough info to test a few things out.
Tom