why is my file size changed after ftp PUT

Hi experts,

I have to write some function that will check if the FTPed file is complete.
I was thinking about using the transfered bytes info in the FTPLOG,
this is the same as the [ (recordlength + 2) * number of records ],
but because I'm not sure if what will happen to this value in case of resending bytes
in case of a bad line or whatever, I looked to get this info from my files,
but I see a lot of diferent sizes and even more confusing
the file that is FTPed has a diferent size then the original one

example:

I copied file Myfile1 to MyFile2 and did a CLRPFM to the copy.
Now I FTPed the data to MyFile2 with:
  PUT MYLIB/MYFILE1 MYLIB/MYFILE2

and..... surprise .... a load of diferent sizes....

MyFile1__________________________________________
Total number of formats  . . . . . . . . . . :           1
Total number of fields . . . . . . . . . . . :           9
Total record length  . . . . . . . . . . . . :          43
Total number of members  . . . . . . . . . :                 1
Total number of members not available  . . :                 0
Total records  . . . . . . . . . . . . . . :               525
Total deleted records  . . . . . . . . . . :                 0
Total of member sizes  . . . . . . . . . . :             32768

MyFile2__________________________________________
Total number of formats  . . . . . . . . . . :           1
Total number of fields . . . . . . . . . . . :           9
Total record length  . . . . . . . . . . . . :          43
Total number of members  . . . . . . . . . :                 1
Total number of members not available  . . :                 0
Total records  . . . . . . . . . . . . . . :               525
Total deleted records  . . . . . . . . . . :                 0
Total of member sizes  . . . . . . . . . . :             28672

To make it more confusing !!!!!
The number of FTPed bytes in the FTPlog = 23625 (what is (43+2) * 525)

if I do a DIR in FTP : I get the following:
DIR MYLIB/MYFILE*                          
227 Entering Passive Mode (127,0,0,1,70,98).                          
125 List started.                                                    
USR             53248 05/06/15 16:55:22 *FILE      MYFILE1
USR                                     *MEM       MYFILE1.MYFILE1
USR             40960 06/04/21 10:25:19 *FILE      MYFILE2          
USR                                     *MEM       MYFILE2.MYFILE2

Please tell me the size of my file !!!!!!!!!!!!!!!!!

LVL 17
MurpheyApplication ConsultantAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

tliottaCommented:
Murph:

"Size" has a number of meanings depending on what's being measured. Data-space will have a size for example, but the only physical requirement that connects that to member size is that data space will fit within the member (and therefore never be larger). Then, the data within a data space will have a size (essentially record length times number of records), but there might be numerous linked data spaces since none can be larger than 16MB.

Objects get an allocated size that relates to the number of sectors/segments/pages required to hold the objects and requests for space allocation. You can create a PF, for example with:

 ==>  crtpf  mylib/myfile rcdlen(80)

...and get Total of member sizes: 8192.

But then see:

 ==>  crtpf  mylib/myfile rcdlen(80)  ALLOCATE(*YES)

...and get Total of member sizes: 823296.

The "allocated" space is reserved for use even though there is no data.

Now, create a PF and add 20,000 records to it. By default, that will take up the initial allowed 10,000 records, add three extents of 1,000 records and cause requests for permission to add more records until 20,000 is achieved. Whenever extents are added, the size of the extents is not a multiple of record length, but a multiple of something else. I'll have to do some research to remember if it's sector size or page size or whatever.

IIRC, there are also circumstances where the added blocks of space can change depending on how big the data space already is. I.e., you might cause a data space to grow by 4096 bytes today; but next week it'll grow by 16384 because the file has grown in the meantime.

When you use CPYF CRTFILE(*YES) to copy one file to another, the system can essentially create a duplicate object -- the objects are on the same system, so it's trivial to copy all attributes. But when you FTP between systems, the receiving system has no idea what the file is like on the sending system. It might be a Windows file or who knows? The way the remote site measures size might be physical bytes or number of clusters or any unit of measure. The receiving system just adds extents as needed, never knowing if there's only one more byte to come or if another couple gigabytes are on the way.

In short, what size do you need to know? It seems to me that the only reliable size for you is (record length times # records). But it seems a waste of time. If FTP says it's done, then it's done. Error correction is built in. If the error correction is failing, then that needs to be fixed by the vendor. A PTF in one piece of system code is just as bad as another. How can you choose which to trust?

Tom


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
MurpheyApplication ConsultantAuthor Commented:
Hello Tom,

"If FTP says it's done, then it's done." I wish this was tru, We discovered last week
a file with over 5000 records and only 3300 are received, without any error.
the result was a 2 Mln $ gap between the sales and financial system.
SO I like to check the the file on the receiving site (what is available on disk)
with the original source.
The receiving site is als an AS/400 and the receicving file dous exist with tha same layout.
But the exceptans that both DIR commands wil give the same result is not tru.

What do the info "transfered bytes" in the FTPLOG mean, the total bytes transfered?
from the sourcefile or the total bytes transfered including the possible resends?

Regards,
Murph

tliottaCommented:
Murph:

If both sender and receiver are AS/400s and FTP did not complete a transfer _and_ there were no errors logged, you need to bring it up with IBM support. Business cannot afford an FTP client (or server) that simply skips/drop records without issuing error codes.

Because different platforms measure in different sizes and even different file systems have different size definitions, you can't rely on sizes after the transfer. AFAIK, FTP will add CR and/or LF between records; that by itself will throw off the number of "transferred bytes" when working in /QSYS.LIB file system.

Again, the only reasonable number in /QSYS.LIB would be (record length times # records). I suspect that there is a logged message somewhere that indicates an interrupted transfer. This may be on either side.

If you have no control over the sending side, you'll need to get them to send some kind of indication -- e.g., a control record that includes counts that you can verify.

Tom
JavaScript Best Practices

Save hours in development time and avoid common mistakes by learning the best practices to use for JavaScript.

MurpheyApplication ConsultantAuthor Commented:
Yeah, a control record....
That is what I sugested, but the receiving partner says "hey it's your FTP file, so not our resposibility"
So an EOF record or someting like that is out of scope.
I will use the records * length to do the checking....

Thanks for your comment...
tliottaCommented:
Hmmm... "receiving partner"... so, you're sending and they're receiving?

And you can't send a control record??

I suppose you could send a remote command  to do a "SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)" or something similar. Something like:

 ==>  quote rcmd SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)

That would need to be updated in your script before execution, but would at least log info on the receiving AS/400. Of course, it only works if the target _is_ an AS/400. You'd retrieve member info prior to running FTP.

I still can't grasp that there is no error message(s) logged on one side or the other. That should _not_ happen and needs to be resolved.

Tom
MurpheyApplication ConsultantAuthor Commented:
Hi Tom

I will check again, but the other site is head office and so not very cooperative....

For other sites we developed a foolproof solution so....... onlu OH gives us a hard time (as usual)
tliottaCommented:
Ahhh... Head office. That does explain a few things. I've dealt with "them" before.

"There can't be a problem on _our_ side. If there was, our guys would have found it."

There is one potential solution that can help you react to errors better than by relying on FTP log messages. Review Scott Klement's FTPAPI stuff at:

http://www.scottklement.com/ftpapi/

This can let you talk more directly with a remote FTP server and then detect errors when they happen rather than trying to track them down after the fact. It isn't trivial, but it does provide far better recovery.

Tom
MurpheyApplication ConsultantAuthor Commented:
Thanks Tom I will look to this tomorrow
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
IBM System i

From novice to tech pro — start learning today.