File transfer protocols: how to verify data integrity

andy7789
andy7789 used Ask the Experts™
on
Hi Xperts,

I am having troubles in transferring large files from windows desktop to a centos server. I've never had that problem before with small files using FTP, but large files (300MB) make a trouble. I upload a file, then download it back. When comparing hex codes, it shows me a few different bits.

I tried CuteFTP and Dreamweaver.

are there any FTP clients that verify transfer blocks? Is there a better way of achieving some better data integrity?

Please advise

thanks
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Commented:
Data integrity is an optional for FTP sever, not all of them support it. You have 2 options. First - use FTP server and client with data integrity support (for example WS_FTP). Second - transfer files in archive with CRC32 checks (.zip and .rar).
Commented:
A time-proven solution would be to put larger files into a distributed .rar (not .zip) archive split up in chunks of 50-100 mb size. Use no compression, and add a .sfv checksum file (http://www.quicksfv.org/).

Commented:
addendum:
The .sfv is for the partial rar files. Check their integrity before unpacking.

Author

Commented:
What would be a better transfer option than FTP?

I am trying SFTP via secure shell. Is it better? As I remember, it checks transfer blocks during transmission.

It is easy to check MD5, but the question remains - how to improve transfer quality to avoid re-transmissions?

Commented:
The transfer quality in FTP should be as good or bad as any other transfers. It's not the transfer that I suspect here, it's the storage.

For quite some time it has been known among FTP practitioners that files originating on a Windows system, uploaded via FTP to a Linux file system and redownloaded to a Windows system again may (but need not) result in slight checksum corruption. This seems to apply to certain filetypes rather than others. Placing files inside distributed rar archives is a proof way to avoid this.

Unfortunately, I never came around to inquiring about the theoretical implications of this phenomenon. I know it sounds a bit like an urban myth. But if this happened to me, I'd try to practically counter it the way I described above.

Commented:
>> What would be a better transfer option than FTP?

Have you tried Scp?
http://winscp.net/eng/index.php

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial