?
Solved

MD5 checksum needs to be ftp in Ascii mode & types of text files

Posted on 2011-09-08
30
Medium Priority
?
1,911 Views
Last Modified: 2013-12-09
I'm doing files transfer between my ftp client (which is a Linux box)
to a remote ftp server (on Win 2003).

The datafiles transferred in both directions come with checksum
( .md5 ) file : these checksum files are generated on the Linux
as well as on the Windows end.

From my Linux box, when I do ftp put or get of these md5 files, they
have to be in Ascii mode, else the files transferred will be unusable.

Q1:
What type of text file are these md5 files?  Are they Unix text file
at UNIX end & Windows/DOS text file at Windows end?

Q2:
Are these md5 files just flat text files or they're indexed text files?
Are they variable length record text files or fixed record length
text files (from IBM Mainframe, HP MPE & OpenVMS days I learnt
these are the various text files available)

Q3:
As I'm phasing out ftp & replacing it with sftp, can I just use
"unix2dos" on the md5 files generated at Linux end before
sftp'ing the md5 file over to Windows & vice-versa (ie after
getting an md5 file over from Windows, I do dos2unix on the
md5 file that was pulled over)?
0
Comment
Question by:sunhux
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 6
  • 4
  • +3
30 Comments
 
LVL 16

Accepted Solution

by:
AlexPace earned 504 total points
ID: 36504336
Plain FTP automatically converts between Windows CRLF and Unix LF line endings in ASCII mode.  

Also, SFTP text file transfers using the Robo-FTP client for Windows do automatic conversion of line endings if you specify ASCII mode.  I don't know about your Linux client but it is possible.

The MD5 checksums are probably just there to allow you to verify the file was sucessfully downloaded and has not been altered.
0
 

Author Comment

by:sunhux
ID: 36511901

>  I don't know about your Linux client but it is possible.
There are ways to use Perl & unix2dos or dos2unix with sftp to do
the conversion but is there an sftp/scp client running on Linux that
can do auto-conversion that you can suggest?  Preferably an sftp
client that can be run at Linux command line & not from GUI
0
 
LVL 29

Assisted Solution

by:Jan Springer
Jan Springer earned 500 total points
ID: 36523662
Q1:  flat file
Q2:  ascii text, length doesn't matter since md5sums generated on a file should be the same
Q3:  using sftp, you should be able to just "diff" the files on windows to verify that they are identical
0
Secure Your WordPress Site: 5 Essential Approaches

WordPress is the web's most popular CMS, but its dominance also makes it a target for attackers. Our eBook will show you how to:

Prevent costly exploits of core and plugin vulnerabilities
Repel automated attacks
Lock down your dashboard, secure your code, and protect your users

 
LVL 12

Expert Comment

by:mccracky
ID: 36524520
The md5 files are just flat textfiles with the md5 checksums next to the filename.  It should "just work" between the systems.  I've done it several times, but not lately (I don't have a Windows box right now to check things).
0
 
LVL 35

Expert Comment

by:Duncan Roe
ID: 36526276
Q3. Yes you can do that. Be sure to transfer in binary mode if you do
0
 
LVL 9

Expert Comment

by:parparov
ID: 36527185
sftp is safe platform-wise, there is no binary or ascii in it, since every piece of data is encoded (encrypted) while in transit. So if you sftp md5 files you only need to verify that they arrive in their completeness (exit status).
0
 
LVL 35

Expert Comment

by:Duncan Roe
ID: 36528247
In other words, everything goes binary. Or am I missing something?
0
 
LVL 16

Assisted Solution

by:AlexPace
AlexPace earned 504 total points
ID: 36530860
Under the hood everything is transmitted in binary.  Some FTP clients that are also SFTP clients will change between unix-style and windows-style line endings on text files if you specify "ascii" mode for your SFTP transfer.  These clients present the same interface for FTP and SFTP but in reality they are very different protocols that have a lot of overlaping functionality.
0
 

Author Comment

by:sunhux
ID: 36536148

The last reply from AlexPace was what I look forward the most but
I'm wishing if there's such an sftp client that runs on Linux RHES, is there any? :
> Some FTP clients that are also SFTP clients will change between unix-style and windows-style
> line endings on text files if you specify "ascii" mode for your SFTP transfer
0
 

Author Comment

by:sunhux
ID: 36536151

Preferably this Linux sftp client has the option of running at Linux command
prompt, not at GUI
0
 
LVL 29

Expert Comment

by:Jan Springer
ID: 36536172
sftp should be installed in /usr/bin/
0
 

Author Comment

by:sunhux
ID: 36536844

Yes, understood there's sftp in /usr/bin/  but what I need is an sftp / scp
client running on Linux (RHES 4.x) that can autodetect (or let me specify
to use Ascii / Binary) like Windows winscp.  

Let me know the link / rpm for me to download if you know of one such
sftp / scp client for Linux?
0
 
LVL 29

Assisted Solution

by:Jan Springer
Jan Springer earned 500 total points
ID: 36536935
You don't need that with sftp/scp.

I use sftp to transfer files and their resepctive md5sums via a cron job from Linux to Windows and it's a non-issue.
0
 
LVL 12

Assisted Solution

by:mccracky
mccracky earned 248 total points
ID: 36537646
IIRC, SFTP just uses ssh/scp "under the hood" (FTPS uses ftp over TLS/SSL).  Just go ahead and script with sftp or scp and keys and you should be good to go.  I think you are stressing too much over the ascii/binary thing when you go sftp.

For a GUI, I find Filezilla a good choice for a client, but cli, I usually just use scp.  

scp should be in your openssh package.
0
 

Author Comment

by:sunhux
ID: 36544181

One ex-colleague told me (before he left the organization) that when he
ftp over md5 checksum files in binary mode to the remote Windows ftp
server, the md5 file would be unusable at the remote end & thus md5 files
had to be ftp'ed over in Ascii mode.  I'm clueless though as to what exactly
is the nature of the unusable md5 but are you certain md5 files scp'ed over
(ie equiv of ftp in binary) to remote Windows platform won't result in
unusable md5 file?  I don't have access to the remote Windows end &
have difficulty testing this out with the remote Windows end as don't
really know the parties there
0
 
LVL 9

Expert Comment

by:parparov
ID: 36544258
scp is not equivalent of ftp in binary
scp is a different method of transfer via encryption that assuredly preserves the file in its entirety.
0
 
LVL 12

Expert Comment

by:mccracky
ID: 36545534
the md5sum file will be exactly the same after the transfer.
0
 

Author Comment

by:sunhux
ID: 36556905

What about md5 checksum files that I "get in binary" from the remote Windows
server?  In my Linux end, it will have end of line character <CR><LF> while
most apps in Linux/Unix expects <LF> end-of-line character.  Though I've
not verified if such md5 would be usable on my Linux apps, I can see that
these md5 files are currently being transferred in ASCII mode (in the ftp scripts
my predecessor wrote)
0
 
LVL 35

Assisted Solution

by:Duncan Roe
Duncan Roe earned 496 total points
ID: 36557444
I just tried one of those, and they don't work. The Cr is treated as the last character of the file name, so the file is not then found. Session below
07:34:54$ md5sum -c slackware64-13.37-install-dvd.iso.md5
: No such file or directorynstall-dvd.iso
: FAILED open or readtall-dvd.iso
md5sum: WARNING: 1 listed file could not be read
07:35:10$ md5sum -c slackware64-13.37-install-dvd.iso.md5.bu
slackware64-13.37-install-dvd.iso: OK
07:37:13$ diff slackware64-13.37-install-dvd.iso.md5.bu slackware64-13.37-install-dvd.iso.md5|cat -A
1c1$
< 96352381b9233c11c6c735ee7f79934b  slackware64-13.37-install-dvd.iso$
---$
> 96352381b9233c11c6c735ee7f79934b  slackware64-13.37-install-dvd.iso^M$

Open in new window

0
 

Author Comment

by:sunhux
ID: 36558126

As duncan reported in his test, I'll something to ensure files that I get over from the
remote Windows server via sftp/scp is properly converted.  Any possibility of coming
up with a Linux Shell script that would immediately do 'dos2unix' after each file is
pulled over from Windows (in binary mode that is)
0
 
LVL 9

Expert Comment

by:parparov
ID: 36558176
How were the .md5 files used by duncan:
1) generated? (tool, OS)
2) transferred? (up to precise command line as detailed as possible)
0
 
LVL 35

Assisted Solution

by:Duncan Roe
Duncan Roe earned 496 total points
ID: 36559096
All I did was save a Unix file in dos mode (with CrLf). I do this all the time at work, when saving files to the shared Windows file system from Linux under VMware. So, yes it was generated, but I can assure you that is what a binary xfer of a dos .txt file would look like.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36560470
but we're going in circles. If you transfer with sftp/scp the whole issue of 'binary' xfer is not relevant.
0
 

Author Comment

by:sunhux
ID: 36560755

I'm pulling md5 files (which have CRLF end-of-line) over from a remote Windows ssh / sftp
server to my LInux, is this not relevant?  

 I'll also need to push md5 files from Linux (which have LF end-of-line) to the remote
Windows server
0
 
LVL 9

Expert Comment

by:parparov
ID: 36560783
how are md5 files produced?
0
 
LVL 12

Expert Comment

by:mccracky
ID: 36563480
What is relevant, is what I said earlier.  

If you transfer the files through sftp (scp) the file will end up at the other end as EXACTLY the same file.

If you need to do other processing to change CR/LF or not is another matter.

@parparov - on linux, at least, with the md5sum program.  There are a couple of programs to do it on Windows, but I can't recall their names at the moment.
0
 
LVL 9

Assisted Solution

by:parparov
parparov earned 252 total points
ID: 36563624
mccracky,

What I was pointing out is that the md5 files produced on Linux and on Windows may vary, so I asked how they were doing it.

A few people already incl. myself have been reiterating the binary/ascii issue irrelevancy when using sftp/scp.
0
 

Author Closing Comment

by:sunhux
ID: 36587397
thanks
0

Featured Post

Turn your laptop into a mobile console!

The CV211 Laptop USB Console Adapter provides a direct Laptop-to-Computer connection for fast and easy remote desktop access with no software to install.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Fine Tune your automatic Updates for Ubuntu / Debian
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
Suggested Courses
Course of the Month12 days, 13 hours left to enroll

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question