Solved

MD5 checksum needs to be ftp in Ascii mode & types of text files

Posted on 2011-09-08
30
1,831 Views
Last Modified: 2013-12-09
I'm doing files transfer between my ftp client (which is a Linux box)
to a remote ftp server (on Win 2003).

The datafiles transferred in both directions come with checksum
( .md5 ) file : these checksum files are generated on the Linux
as well as on the Windows end.

From my Linux box, when I do ftp put or get of these md5 files, they
have to be in Ascii mode, else the files transferred will be unusable.

Q1:
What type of text file are these md5 files?  Are they Unix text file
at UNIX end & Windows/DOS text file at Windows end?

Q2:
Are these md5 files just flat text files or they're indexed text files?
Are they variable length record text files or fixed record length
text files (from IBM Mainframe, HP MPE & OpenVMS days I learnt
these are the various text files available)

Q3:
As I'm phasing out ftp & replacing it with sftp, can I just use
"unix2dos" on the md5 files generated at Linux end before
sftp'ing the md5 file over to Windows & vice-versa (ie after
getting an md5 file over from Windows, I do dos2unix on the
md5 file that was pulled over)?
0
Comment
Question by:sunhux
  • 9
  • 6
  • 4
  • +3
30 Comments
 
LVL 16

Accepted Solution

by:
AlexPace earned 126 total points
ID: 36504336
Plain FTP automatically converts between Windows CRLF and Unix LF line endings in ASCII mode.  

Also, SFTP text file transfers using the Robo-FTP client for Windows do automatic conversion of line endings if you specify ASCII mode.  I don't know about your Linux client but it is possible.

The MD5 checksums are probably just there to allow you to verify the file was sucessfully downloaded and has not been altered.
0
 

Author Comment

by:sunhux
ID: 36511901

>  I don't know about your Linux client but it is possible.
There are ways to use Perl & unix2dos or dos2unix with sftp to do
the conversion but is there an sftp/scp client running on Linux that
can do auto-conversion that you can suggest?  Preferably an sftp
client that can be run at Linux command line & not from GUI
0
 
LVL 28

Assisted Solution

by:Jan Springer
Jan Springer earned 125 total points
ID: 36523662
Q1:  flat file
Q2:  ascii text, length doesn't matter since md5sums generated on a file should be the same
Q3:  using sftp, you should be able to just "diff" the files on windows to verify that they are identical
0
 
LVL 12

Expert Comment

by:mccracky
ID: 36524520
The md5 files are just flat textfiles with the md5 checksums next to the filename.  It should "just work" between the systems.  I've done it several times, but not lately (I don't have a Windows box right now to check things).
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 36526276
Q3. Yes you can do that. Be sure to transfer in binary mode if you do
0
 
LVL 9

Expert Comment

by:parparov
ID: 36527185
sftp is safe platform-wise, there is no binary or ascii in it, since every piece of data is encoded (encrypted) while in transit. So if you sftp md5 files you only need to verify that they arrive in their completeness (exit status).
0
 
LVL 34

Expert Comment

by:Duncan Roe
ID: 36528247
In other words, everything goes binary. Or am I missing something?
0
 
LVL 16

Assisted Solution

by:AlexPace
AlexPace earned 126 total points
ID: 36530860
Under the hood everything is transmitted in binary.  Some FTP clients that are also SFTP clients will change between unix-style and windows-style line endings on text files if you specify "ascii" mode for your SFTP transfer.  These clients present the same interface for FTP and SFTP but in reality they are very different protocols that have a lot of overlaping functionality.
0
 

Author Comment

by:sunhux
ID: 36536148

The last reply from AlexPace was what I look forward the most but
I'm wishing if there's such an sftp client that runs on Linux RHES, is there any? :
> Some FTP clients that are also SFTP clients will change between unix-style and windows-style
> line endings on text files if you specify "ascii" mode for your SFTP transfer
0
 

Author Comment

by:sunhux
ID: 36536151

Preferably this Linux sftp client has the option of running at Linux command
prompt, not at GUI
0
 
LVL 28

Expert Comment

by:Jan Springer
ID: 36536172
sftp should be installed in /usr/bin/
0
 

Author Comment

by:sunhux
ID: 36536844

Yes, understood there's sftp in /usr/bin/  but what I need is an sftp / scp
client running on Linux (RHES 4.x) that can autodetect (or let me specify
to use Ascii / Binary) like Windows winscp.  

Let me know the link / rpm for me to download if you know of one such
sftp / scp client for Linux?
0
 
LVL 28

Assisted Solution

by:Jan Springer
Jan Springer earned 125 total points
ID: 36536935
You don't need that with sftp/scp.

I use sftp to transfer files and their resepctive md5sums via a cron job from Linux to Windows and it's a non-issue.
0
 
LVL 12

Assisted Solution

by:mccracky
mccracky earned 62 total points
ID: 36537646
IIRC, SFTP just uses ssh/scp "under the hood" (FTPS uses ftp over TLS/SSL).  Just go ahead and script with sftp or scp and keys and you should be good to go.  I think you are stressing too much over the ascii/binary thing when you go sftp.

For a GUI, I find Filezilla a good choice for a client, but cli, I usually just use scp.  

scp should be in your openssh package.
0
Scale it in WD Gold

With up to ten times the workload capacity of desktop drives, WD Gold hard drives employ advanced technology to deliver among the best in reliability, capacity, power efficiency and performance.

 

Author Comment

by:sunhux
ID: 36544181

One ex-colleague told me (before he left the organization) that when he
ftp over md5 checksum files in binary mode to the remote Windows ftp
server, the md5 file would be unusable at the remote end & thus md5 files
had to be ftp'ed over in Ascii mode.  I'm clueless though as to what exactly
is the nature of the unusable md5 but are you certain md5 files scp'ed over
(ie equiv of ftp in binary) to remote Windows platform won't result in
unusable md5 file?  I don't have access to the remote Windows end &
have difficulty testing this out with the remote Windows end as don't
really know the parties there
0
 
LVL 9

Expert Comment

by:parparov
ID: 36544258
scp is not equivalent of ftp in binary
scp is a different method of transfer via encryption that assuredly preserves the file in its entirety.
0
 
LVL 12

Expert Comment

by:mccracky
ID: 36545534
the md5sum file will be exactly the same after the transfer.
0
 

Author Comment

by:sunhux
ID: 36556905

What about md5 checksum files that I "get in binary" from the remote Windows
server?  In my Linux end, it will have end of line character <CR><LF> while
most apps in Linux/Unix expects <LF> end-of-line character.  Though I've
not verified if such md5 would be usable on my Linux apps, I can see that
these md5 files are currently being transferred in ASCII mode (in the ftp scripts
my predecessor wrote)
0
 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 124 total points
ID: 36557444
I just tried one of those, and they don't work. The Cr is treated as the last character of the file name, so the file is not then found. Session below
07:34:54$ md5sum -c slackware64-13.37-install-dvd.iso.md5
: No such file or directorynstall-dvd.iso
: FAILED open or readtall-dvd.iso
md5sum: WARNING: 1 listed file could not be read
07:35:10$ md5sum -c slackware64-13.37-install-dvd.iso.md5.bu
slackware64-13.37-install-dvd.iso: OK
07:37:13$ diff slackware64-13.37-install-dvd.iso.md5.bu slackware64-13.37-install-dvd.iso.md5|cat -A
1c1$
< 96352381b9233c11c6c735ee7f79934b  slackware64-13.37-install-dvd.iso$
---$
> 96352381b9233c11c6c735ee7f79934b  slackware64-13.37-install-dvd.iso^M$

Open in new window

0
 

Author Comment

by:sunhux
ID: 36558126

As duncan reported in his test, I'll something to ensure files that I get over from the
remote Windows server via sftp/scp is properly converted.  Any possibility of coming
up with a Linux Shell script that would immediately do 'dos2unix' after each file is
pulled over from Windows (in binary mode that is)
0
 
LVL 9

Expert Comment

by:parparov
ID: 36558176
How were the .md5 files used by duncan:
1) generated? (tool, OS)
2) transferred? (up to precise command line as detailed as possible)
0
 
LVL 34

Assisted Solution

by:Duncan Roe
Duncan Roe earned 124 total points
ID: 36559096
All I did was save a Unix file in dos mode (with CrLf). I do this all the time at work, when saving files to the shared Windows file system from Linux under VMware. So, yes it was generated, but I can assure you that is what a binary xfer of a dos .txt file would look like.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36560470
but we're going in circles. If you transfer with sftp/scp the whole issue of 'binary' xfer is not relevant.
0
 

Author Comment

by:sunhux
ID: 36560755

I'm pulling md5 files (which have CRLF end-of-line) over from a remote Windows ssh / sftp
server to my LInux, is this not relevant?  

 I'll also need to push md5 files from Linux (which have LF end-of-line) to the remote
Windows server
0
 
LVL 9

Expert Comment

by:parparov
ID: 36560783
how are md5 files produced?
0
 
LVL 12

Expert Comment

by:mccracky
ID: 36563480
What is relevant, is what I said earlier.  

If you transfer the files through sftp (scp) the file will end up at the other end as EXACTLY the same file.

If you need to do other processing to change CR/LF or not is another matter.

@parparov - on linux, at least, with the md5sum program.  There are a couple of programs to do it on Windows, but I can't recall their names at the moment.
0
 
LVL 9

Assisted Solution

by:parparov
parparov earned 63 total points
ID: 36563624
mccracky,

What I was pointing out is that the md5 files produced on Linux and on Windows may vary, so I asked how they were doing it.

A few people already incl. myself have been reiterating the binary/ascii issue irrelevancy when using sftp/scp.
0
 

Author Closing Comment

by:sunhux
ID: 36587397
thanks
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

The term "Bad USB" is a buzz word that is usually used when talking about attacks on computer systems that involve USB devices. In this article, I will show what possibilities modern windows systems (win8.x and win10) offer to fight these attacks wi…
Security measures require Windows be logged in using Standard User login (not Administrator).  Yet, sometimes an application has to be run “As Administrator” from a Standard User login.  This paper describes how to create a shortcut icon to launch a…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now