why is my file size changed after ftp PUT
Hi experts,
I have to write some function that will check if the FTPed file is complete.
I was thinking about using the transfered bytes info in the FTPLOG,
this is the same as the [ (recordlength + 2) * number of records ],
but because I'm not sure if what will happen to this value in case of resending bytes
in case of a bad line or whatever, I looked to get this info from my files,
but I see a lot of diferent sizes and even more confusing
the file that is FTPed has a diferent size then the original one
example:
I copied file Myfile1 to MyFile2 and did a CLRPFM to the copy.
Now I FTPed the data to MyFile2 with:
PUT MYLIB/MYFILE1 MYLIB/MYFILE2
and..... surprise .... a load of diferent sizes....
MyFile1___________________
Total number of formats . . . . . . . . . . : 1
Total number of fields . . . . . . . . . . . : 9
Total record length . . . . . . . . . . . . : 43
Total number of members . . . . . . . . . : 1
Total number of members not available . . : 0
Total records . . . . . . . . . . . . . . : 525
Total deleted records . . . . . . . . . . : 0
Total of member sizes . . . . . . . . . . : 32768
MyFile2___________________
Total number of formats . . . . . . . . . . : 1
Total number of fields . . . . . . . . . . . : 9
Total record length . . . . . . . . . . . . : 43
Total number of members . . . . . . . . . : 1
Total number of members not available . . : 0
Total records . . . . . . . . . . . . . . : 525
Total deleted records . . . . . . . . . . : 0
Total of member sizes . . . . . . . . . . : 28672
To make it more confusing !!!!!
The number of FTPed bytes in the FTPlog = 23625 (what is (43+2) * 525)
if I do a DIR in FTP : I get the following:
DIR MYLIB/MYFILE*
227 Entering Passive Mode (127,0,0,1,70,98).
125 List started.
USR 53248 05/06/15 16:55:22 *FILE MYFILE1
USR *MEM MYFILE1.MYFILE1
USR 40960 06/04/21 10:25:19 *FILE MYFILE2
USR *MEM MYFILE2.MYFILE2
Please tell me the size of my file !!!!!!!!!!!!!!!!!
I have to write some function that will check if the FTPed file is complete.
I was thinking about using the transfered bytes info in the FTPLOG,
this is the same as the [ (recordlength + 2) * number of records ],
but because I'm not sure if what will happen to this value in case of resending bytes
in case of a bad line or whatever, I looked to get this info from my files,
but I see a lot of diferent sizes and even more confusing
the file that is FTPed has a diferent size then the original one
example:
I copied file Myfile1 to MyFile2 and did a CLRPFM to the copy.
Now I FTPed the data to MyFile2 with:
PUT MYLIB/MYFILE1 MYLIB/MYFILE2
and..... surprise .... a load of diferent sizes....
MyFile1___________________
Total number of formats . . . . . . . . . . : 1
Total number of fields . . . . . . . . . . . : 9
Total record length . . . . . . . . . . . . : 43
Total number of members . . . . . . . . . : 1
Total number of members not available . . : 0
Total records . . . . . . . . . . . . . . : 525
Total deleted records . . . . . . . . . . : 0
Total of member sizes . . . . . . . . . . : 32768
MyFile2___________________
Total number of formats . . . . . . . . . . : 1
Total number of fields . . . . . . . . . . . : 9
Total record length . . . . . . . . . . . . : 43
Total number of members . . . . . . . . . : 1
Total number of members not available . . : 0
Total records . . . . . . . . . . . . . . : 525
Total deleted records . . . . . . . . . . : 0
Total of member sizes . . . . . . . . . . : 28672
To make it more confusing !!!!!
The number of FTPed bytes in the FTPlog = 23625 (what is (43+2) * 525)
if I do a DIR in FTP : I get the following:
DIR MYLIB/MYFILE*
227 Entering Passive Mode (127,0,0,1,70,98).
125 List started.
USR 53248 05/06/15 16:55:22 *FILE MYFILE1
USR *MEM MYFILE1.MYFILE1
USR 40960 06/04/21 10:25:19 *FILE MYFILE2
USR *MEM MYFILE2.MYFILE2
Please tell me the size of my file !!!!!!!!!!!!!!!!!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Murph:
If both sender and receiver are AS/400s and FTP did not complete a transfer _and_ there were no errors logged, you need to bring it up with IBM support. Business cannot afford an FTP client (or server) that simply skips/drop records without issuing error codes.
Because different platforms measure in different sizes and even different file systems have different size definitions, you can't rely on sizes after the transfer. AFAIK, FTP will add CR and/or LF between records; that by itself will throw off the number of "transferred bytes" when working in /QSYS.LIB file system.
Again, the only reasonable number in /QSYS.LIB would be (record length times # records). I suspect that there is a logged message somewhere that indicates an interrupted transfer. This may be on either side.
If you have no control over the sending side, you'll need to get them to send some kind of indication -- e.g., a control record that includes counts that you can verify.
Tom
If both sender and receiver are AS/400s and FTP did not complete a transfer _and_ there were no errors logged, you need to bring it up with IBM support. Business cannot afford an FTP client (or server) that simply skips/drop records without issuing error codes.
Because different platforms measure in different sizes and even different file systems have different size definitions, you can't rely on sizes after the transfer. AFAIK, FTP will add CR and/or LF between records; that by itself will throw off the number of "transferred bytes" when working in /QSYS.LIB file system.
Again, the only reasonable number in /QSYS.LIB would be (record length times # records). I suspect that there is a logged message somewhere that indicates an interrupted transfer. This may be on either side.
If you have no control over the sending side, you'll need to get them to send some kind of indication -- e.g., a control record that includes counts that you can verify.
Tom
ASKER
Yeah, a control record....
That is what I sugested, but the receiving partner says "hey it's your FTP file, so not our resposibility"
So an EOF record or someting like that is out of scope.
I will use the records * length to do the checking....
Thanks for your comment...
That is what I sugested, but the receiving partner says "hey it's your FTP file, so not our resposibility"
So an EOF record or someting like that is out of scope.
I will use the records * length to do the checking....
Thanks for your comment...
Hmmm... "receiving partner"... so, you're sending and they're receiving?
And you can't send a control record??
I suppose you could send a remote command to do a "SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)" or something similar. Something like:
==> quote rcmd SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)
That would need to be updated in your script before execution, but would at least log info on the receiving AS/400. Of course, it only works if the target _is_ an AS/400. You'd retrieve member info prior to running FTP.
I still can't grasp that there is no error message(s) logged on one side or the other. That should _not_ happen and needs to be resolved.
Tom
And you can't send a control record??
I suppose you could send a remote command to do a "SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)" or something similar. Something like:
==> quote rcmd SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)
That would need to be updated in your script before execution, but would at least log info on the receiving AS/400. Of course, it only works if the target _is_ an AS/400. You'd retrieve member info prior to running FTP.
I still can't grasp that there is no error message(s) logged on one side or the other. That should _not_ happen and needs to be resolved.
Tom
ASKER
Hi Tom
I will check again, but the other site is head office and so not very cooperative....
For other sites we developed a foolproof solution so....... onlu OH gives us a hard time (as usual)
I will check again, but the other site is head office and so not very cooperative....
For other sites we developed a foolproof solution so....... onlu OH gives us a hard time (as usual)
Ahhh... Head office. That does explain a few things. I've dealt with "them" before.
"There can't be a problem on _our_ side. If there was, our guys would have found it."
There is one potential solution that can help you react to errors better than by relying on FTP log messages. Review Scott Klement's FTPAPI stuff at:
http://www.scottklement.com/ftpapi/
This can let you talk more directly with a remote FTP server and then detect errors when they happen rather than trying to track them down after the fact. It isn't trivial, but it does provide far better recovery.
Tom
"There can't be a problem on _our_ side. If there was, our guys would have found it."
There is one potential solution that can help you react to errors better than by relying on FTP log messages. Review Scott Klement's FTPAPI stuff at:
http://www.scottklement.com/ftpapi/
This can let you talk more directly with a remote FTP server and then detect errors when they happen rather than trying to track them down after the fact. It isn't trivial, but it does provide far better recovery.
Tom
ASKER
Thanks Tom I will look to this tomorrow
ASKER
"If FTP says it's done, then it's done." I wish this was tru, We discovered last week
a file with over 5000 records and only 3300 are received, without any error.
the result was a 2 Mln $ gap between the sales and financial system.
SO I like to check the the file on the receiving site (what is available on disk)
with the original source.
The receiving site is als an AS/400 and the receicving file dous exist with tha same layout.
But the exceptans that both DIR commands wil give the same result is not tru.
What do the info "transfered bytes" in the FTPLOG mean, the total bytes transfered?
from the sourcefile or the total bytes transfered including the possible resends?
Regards,
Murph