sunhux
asked on
MD5 checksum needs to be ftp in Ascii mode & types of text files
I'm doing files transfer between my ftp client (which is a Linux box)
to a remote ftp server (on Win 2003).
The datafiles transferred in both directions come with checksum
( .md5 ) file : these checksum files are generated on the Linux
as well as on the Windows end.
From my Linux box, when I do ftp put or get of these md5 files, they
have to be in Ascii mode, else the files transferred will be unusable.
Q1:
What type of text file are these md5 files? Are they Unix text file
at UNIX end & Windows/DOS text file at Windows end?
Q2:
Are these md5 files just flat text files or they're indexed text files?
Are they variable length record text files or fixed record length
text files (from IBM Mainframe, HP MPE & OpenVMS days I learnt
these are the various text files available)
Q3:
As I'm phasing out ftp & replacing it with sftp, can I just use
"unix2dos" on the md5 files generated at Linux end before
sftp'ing the md5 file over to Windows & vice-versa (ie after
getting an md5 file over from Windows, I do dos2unix on the
md5 file that was pulled over)?
to a remote ftp server (on Win 2003).
The datafiles transferred in both directions come with checksum
( .md5 ) file : these checksum files are generated on the Linux
as well as on the Windows end.
From my Linux box, when I do ftp put or get of these md5 files, they
have to be in Ascii mode, else the files transferred will be unusable.
Q1:
What type of text file are these md5 files? Are they Unix text file
at UNIX end & Windows/DOS text file at Windows end?
Q2:
Are these md5 files just flat text files or they're indexed text files?
Are they variable length record text files or fixed record length
text files (from IBM Mainframe, HP MPE & OpenVMS days I learnt
these are the various text files available)
Q3:
As I'm phasing out ftp & replacing it with sftp, can I just use
"unix2dos" on the md5 files generated at Linux end before
sftp'ing the md5 file over to Windows & vice-versa (ie after
getting an md5 file over from Windows, I do dos2unix on the
md5 file that was pulled over)?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
The md5 files are just flat textfiles with the md5 checksums next to the filename. It should "just work" between the systems. I've done it several times, but not lately (I don't have a Windows box right now to check things).
Q3. Yes you can do that. Be sure to transfer in binary mode if you do
sftp is safe platform-wise, there is no binary or ascii in it, since every piece of data is encoded (encrypted) while in transit. So if you sftp md5 files you only need to verify that they arrive in their completeness (exit status).
In other words, everything goes binary. Or am I missing something?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
The last reply from AlexPace was what I look forward the most but
I'm wishing if there's such an sftp client that runs on Linux RHES, is there any? :
> Some FTP clients that are also SFTP clients will change between unix-style and windows-style
> line endings on text files if you specify "ascii" mode for your SFTP transfer
ASKER
Preferably this Linux sftp client has the option of running at Linux command
prompt, not at GUI
sftp should be installed in /usr/bin/
ASKER
Yes, understood there's sftp in /usr/bin/ but what I need is an sftp / scp
client running on Linux (RHES 4.x) that can autodetect (or let me specify
to use Ascii / Binary) like Windows winscp.
Let me know the link / rpm for me to download if you know of one such
sftp / scp client for Linux?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
One ex-colleague told me (before he left the organization) that when he
ftp over md5 checksum files in binary mode to the remote Windows ftp
server, the md5 file would be unusable at the remote end & thus md5 files
had to be ftp'ed over in Ascii mode. I'm clueless though as to what exactly
is the nature of the unusable md5 but are you certain md5 files scp'ed over
(ie equiv of ftp in binary) to remote Windows platform won't result in
unusable md5 file? I don't have access to the remote Windows end &
have difficulty testing this out with the remote Windows end as don't
really know the parties there
scp is not equivalent of ftp in binary
scp is a different method of transfer via encryption that assuredly preserves the file in its entirety.
scp is a different method of transfer via encryption that assuredly preserves the file in its entirety.
the md5sum file will be exactly the same after the transfer.
ASKER
What about md5 checksum files that I "get in binary" from the remote Windows
server? In my Linux end, it will have end of line character <CR><LF> while
most apps in Linux/Unix expects <LF> end-of-line character. Though I've
not verified if such md5 would be usable on my Linux apps, I can see that
these md5 files are currently being transferred in ASCII mode (in the ftp scripts
my predecessor wrote)
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
As duncan reported in his test, I'll something to ensure files that I get over from the
remote Windows server via sftp/scp is properly converted. Any possibility of coming
up with a Linux Shell script that would immediately do 'dos2unix' after each file is
pulled over from Windows (in binary mode that is)
How were the .md5 files used by duncan:
1) generated? (tool, OS)
2) transferred? (up to precise command line as detailed as possible)
1) generated? (tool, OS)
2) transferred? (up to precise command line as detailed as possible)
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
but we're going in circles. If you transfer with sftp/scp the whole issue of 'binary' xfer is not relevant.
ASKER
I'm pulling md5 files (which have CRLF end-of-line) over from a remote Windows ssh / sftp
server to my LInux, is this not relevant?
I'll also need to push md5 files from Linux (which have LF end-of-line) to the remote
Windows server
how are md5 files produced?
What is relevant, is what I said earlier.
If you transfer the files through sftp (scp) the file will end up at the other end as EXACTLY the same file.
If you need to do other processing to change CR/LF or not is another matter.
@parparov - on linux, at least, with the md5sum program. There are a couple of programs to do it on Windows, but I can't recall their names at the moment.
If you transfer the files through sftp (scp) the file will end up at the other end as EXACTLY the same file.
If you need to do other processing to change CR/LF or not is another matter.
@parparov - on linux, at least, with the md5sum program. There are a couple of programs to do it on Windows, but I can't recall their names at the moment.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
thanks
ASKER
> I don't know about your Linux client but it is possible.
There are ways to use Perl & unix2dos or dos2unix with sftp to do
the conversion but is there an sftp/scp client running on Linux that
can do auto-conversion that you can suggest? Preferably an sftp
client that can be run at Linux command line & not from GUI