?
Solved

why is my file size changed after ftp PUT

Posted on 2006-04-21
8
Medium Priority
?
610 Views
Last Modified: 2008-02-01
Hi experts,

I have to write some function that will check if the FTPed file is complete.
I was thinking about using the transfered bytes info in the FTPLOG,
this is the same as the [ (recordlength + 2) * number of records ],
but because I'm not sure if what will happen to this value in case of resending bytes
in case of a bad line or whatever, I looked to get this info from my files,
but I see a lot of diferent sizes and even more confusing
the file that is FTPed has a diferent size then the original one

example:

I copied file Myfile1 to MyFile2 and did a CLRPFM to the copy.
Now I FTPed the data to MyFile2 with:
  PUT MYLIB/MYFILE1 MYLIB/MYFILE2

and..... surprise .... a load of diferent sizes....

MyFile1__________________________________________
Total number of formats  . . . . . . . . . . :           1
Total number of fields . . . . . . . . . . . :           9
Total record length  . . . . . . . . . . . . :          43
Total number of members  . . . . . . . . . :                 1
Total number of members not available  . . :                 0
Total records  . . . . . . . . . . . . . . :               525
Total deleted records  . . . . . . . . . . :                 0
Total of member sizes  . . . . . . . . . . :             32768

MyFile2__________________________________________
Total number of formats  . . . . . . . . . . :           1
Total number of fields . . . . . . . . . . . :           9
Total record length  . . . . . . . . . . . . :          43
Total number of members  . . . . . . . . . :                 1
Total number of members not available  . . :                 0
Total records  . . . . . . . . . . . . . . :               525
Total deleted records  . . . . . . . . . . :                 0
Total of member sizes  . . . . . . . . . . :             28672

To make it more confusing !!!!!
The number of FTPed bytes in the FTPlog = 23625 (what is (43+2) * 525)

if I do a DIR in FTP : I get the following:
DIR MYLIB/MYFILE*                          
227 Entering Passive Mode (127,0,0,1,70,98).                          
125 List started.                                                    
USR             53248 05/06/15 16:55:22 *FILE      MYFILE1
USR                                     *MEM       MYFILE1.MYFILE1
USR             40960 06/04/21 10:25:19 *FILE      MYFILE2          
USR                                     *MEM       MYFILE2.MYFILE2

Please tell me the size of my file !!!!!!!!!!!!!!!!!

0
Comment
Question by:theo kouwenhoven
  • 4
  • 4
8 Comments
 
LVL 27

Accepted Solution

by:
tliotta earned 200 total points
ID: 16513130
Murph:

"Size" has a number of meanings depending on what's being measured. Data-space will have a size for example, but the only physical requirement that connects that to member size is that data space will fit within the member (and therefore never be larger). Then, the data within a data space will have a size (essentially record length times number of records), but there might be numerous linked data spaces since none can be larger than 16MB.

Objects get an allocated size that relates to the number of sectors/segments/pages required to hold the objects and requests for space allocation. You can create a PF, for example with:

 ==>  crtpf  mylib/myfile rcdlen(80)

...and get Total of member sizes: 8192.

But then see:

 ==>  crtpf  mylib/myfile rcdlen(80)  ALLOCATE(*YES)

...and get Total of member sizes: 823296.

The "allocated" space is reserved for use even though there is no data.

Now, create a PF and add 20,000 records to it. By default, that will take up the initial allowed 10,000 records, add three extents of 1,000 records and cause requests for permission to add more records until 20,000 is achieved. Whenever extents are added, the size of the extents is not a multiple of record length, but a multiple of something else. I'll have to do some research to remember if it's sector size or page size or whatever.

IIRC, there are also circumstances where the added blocks of space can change depending on how big the data space already is. I.e., you might cause a data space to grow by 4096 bytes today; but next week it'll grow by 16384 because the file has grown in the meantime.

When you use CPYF CRTFILE(*YES) to copy one file to another, the system can essentially create a duplicate object -- the objects are on the same system, so it's trivial to copy all attributes. But when you FTP between systems, the receiving system has no idea what the file is like on the sending system. It might be a Windows file or who knows? The way the remote site measures size might be physical bytes or number of clusters or any unit of measure. The receiving system just adds extents as needed, never knowing if there's only one more byte to come or if another couple gigabytes are on the way.

In short, what size do you need to know? It seems to me that the only reliable size for you is (record length times # records). But it seems a waste of time. If FTP says it's done, then it's done. Error correction is built in. If the error correction is failing, then that needs to be fixed by the vendor. A PTF in one piece of system code is just as bad as another. How can you choose which to trust?

Tom


0
 
LVL 16

Author Comment

by:theo kouwenhoven
ID: 16516941
Hello Tom,

"If FTP says it's done, then it's done." I wish this was tru, We discovered last week
a file with over 5000 records and only 3300 are received, without any error.
the result was a 2 Mln $ gap between the sales and financial system.
SO I like to check the the file on the receiving site (what is available on disk)
with the original source.
The receiving site is als an AS/400 and the receicving file dous exist with tha same layout.
But the exceptans that both DIR commands wil give the same result is not tru.

What do the info "transfered bytes" in the FTPLOG mean, the total bytes transfered?
from the sourcefile or the total bytes transfered including the possible resends?

Regards,
Murph

0
 
LVL 27

Expert Comment

by:tliotta
ID: 16528126
Murph:

If both sender and receiver are AS/400s and FTP did not complete a transfer _and_ there were no errors logged, you need to bring it up with IBM support. Business cannot afford an FTP client (or server) that simply skips/drop records without issuing error codes.

Because different platforms measure in different sizes and even different file systems have different size definitions, you can't rely on sizes after the transfer. AFAIK, FTP will add CR and/or LF between records; that by itself will throw off the number of "transferred bytes" when working in /QSYS.LIB file system.

Again, the only reasonable number in /QSYS.LIB would be (record length times # records). I suspect that there is a logged message somewhere that indicates an interrupted transfer. This may be on either side.

If you have no control over the sending side, you'll need to get them to send some kind of indication -- e.g., a control record that includes counts that you can verify.

Tom
0
Veeam and MySQL: How to Perform Backup & Recovery

MySQL and the MariaDB variant are among the most used databases in Linux environments, and many critical applications support their data on them. Watch this recorded webinar to find out how Veeam Backup & Replication allows you to get consistent backups of MySQL databases.

 
LVL 16

Author Comment

by:theo kouwenhoven
ID: 16528197
Yeah, a control record....
That is what I sugested, but the receiving partner says "hey it's your FTP file, so not our resposibility"
So an EOF record or someting like that is out of scope.
I will use the records * length to do the checking....

Thanks for your comment...
0
 
LVL 27

Expert Comment

by:tliotta
ID: 16529594
Hmmm... "receiving partner"... so, you're sending and they're receiving?

And you can't send a control record??

I suppose you could send a remote command  to do a "SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)" or something similar. Something like:

 ==>  quote rcmd SNDMSG 'Records: nnnn' TOUSR(QSYSOPR)

That would need to be updated in your script before execution, but would at least log info on the receiving AS/400. Of course, it only works if the target _is_ an AS/400. You'd retrieve member info prior to running FTP.

I still can't grasp that there is no error message(s) logged on one side or the other. That should _not_ happen and needs to be resolved.

Tom
0
 
LVL 16

Author Comment

by:theo kouwenhoven
ID: 16530015
Hi Tom

I will check again, but the other site is head office and so not very cooperative....

For other sites we developed a foolproof solution so....... onlu OH gives us a hard time (as usual)
0
 
LVL 27

Expert Comment

by:tliotta
ID: 16536529
Ahhh... Head office. That does explain a few things. I've dealt with "them" before.

"There can't be a problem on _our_ side. If there was, our guys would have found it."

There is one potential solution that can help you react to errors better than by relying on FTP log messages. Review Scott Klement's FTPAPI stuff at:

http://www.scottklement.com/ftpapi/

This can let you talk more directly with a remote FTP server and then detect errors when they happen rather than trying to track them down after the fact. It isn't trivial, but it does provide far better recovery.

Tom
0
 
LVL 16

Author Comment

by:theo kouwenhoven
ID: 16536996
Thanks Tom I will look to this tomorrow
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This shares a stored procedure to retrieve permissions for a given user on the current database or across all databases on a server.
Exchange administrators are always vigilant about Exchange crashes and disasters that are possible any time. It is quite essential to identify the symptoms of a possible Exchange issue and be prepared with a proper recovery plan. There are multiple…
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
This lesson discusses how to use a Mainform + Subforms in Microsoft Access to find and enter data for payments on orders. The sample data comes from a custom shop that builds and sells movable storage structures that are delivered to your property. …
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question