Solved

How FTP pulls files

Posted on 2013-11-04
6
460 Views
Last Modified: 2013-11-05
I am pulling files from a customer's ftp sever on a daily basis.  Up until recently, we had no problems but suddenly we are getting files with today's date and the file has some other day's data.  It turns out that we rename the file on our side and the customer is trying to say we are messing up the data when we rename it.

Can someone please explain how ftp works when there are multiple files with virtually the same file name with a different date.

For example if the customer had three files on the ftp server
M700111.1234   10/31/2013
M700111.4567    11/01/2013
M700111.8999     11/02/2013

Given the following script what would be pulled from the customer's ftp server.    

1.  open 999.999.999.999
2.  user MICKEYMOUSE
3.  PASSWORD
4.  prompt
5.  lcd c:\TEMP\
6.  bin
7.  mget *.*
8.  quit
0
Comment
Question by:sherbug1015
6 Comments
 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 39622301
That should get all of the files in the current remote directory.
0
 
LVL 52

Expert Comment

by:Bill Prew
ID: 39622387
It sounds like you are assuming that there will only be one file named M700111.<anything> ?  If that's the case, then you are likely to run into problems given that there can now be multiple files with that same base name, but different extensions.

I suspect that you are renaming the file based on the system clock on the receiving computer, and so multiple files could get the same time stamp when they are pulled, which may be a problem.

I guess the root question is what do you want to happen when there are more than one of these files?  Pull the latest one? Pull all and rename with the modified date from the remote system?  Etc...

~bp
0
 

Author Comment

by:sherbug1015
ID: 39622535
billprew -  that is what I would have thought as well.  I just went back to the customers ftp site and they had dropped another file out there since this morning when I first picked up.

There is a file called

BCL.SDOA.1442238  -  11/04/13  2:20 p.m.
BCL.SDOA.1442078  -  11/03/13  2:20 p.m.

I picked up the 11/03 file this morning and when I ran my script, it only picked up the 11/4 file.  Same file name, different extension.

How does it know that the 11/03 file has already been picked up is my question.  Is there something I am missing here.  It must be using some kind of system time to determine what has been picked up.  Can you confirm anything like that.

thanks
0
DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 39622590
If you have not renamed the file, then you probably won't see the 'over-write warning' dialog in a script like you would in a desktop client.  For a test, rename both files on your computer and run your script again.  I suspect it will then re-download both of them.
0
 
LVL 52

Assisted Solution

by:Bill Prew
Bill Prew earned 300 total points
ID: 39622597
The "mget *.*" should pull ALL files in that folder down to the client computer.  That being said, it could depend a bit on the FTP client software you are using on your computer.  If the file is still in the destination folder on the client PC, it may be "smart" enough to see that it is already there and validate it hasn't changed, and therefore not pull it down again.  But I don't think the Windows built in FTP command line client is smart enough to do that.

I expect thought, even if that is true, if you deleted any existing files from the destination folder on the client computer, it will then pull all files down again.  I very much doubt the server end is doing anything to determine what has already been sent down, that would not be typical at all.

~bp
0
 
LVL 16

Accepted Solution

by:
AlexPace earned 200 total points
ID: 39623042
The FTP protocol does not support wildcards.
MGET is a client command, not an FTP protocol command.
It works like this:
  1. Client requests a directory LIST
  2. Client parses list for files that match *.* or whatever pattern you used.
  3. Client downloads the matching files one at a time.

The sad thing about the FTP protocol is that the format of the directory list is not standard.
There is significant variation in the listing formats returned by different FTP servers.
This means that sometimes step #2 above fails because the list parsing routine failed.
It happens but it is rare.
Most likely the file just wasn't there when you ran the script.
Add a line to save a remote listing to a text file.
That file will help you see what the FTP server returned as a listing so you can determine if it is a parsing problem or if the file just wasn't there.
ftp.exe has no built-in means to skip a file because a local copy already exists.


edit: I've seen FTP servers scripted to delete a file after it is downloaded but those are almost always hosted by banks.
0

Featured Post

DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Being a system administrator some time we require to do things remotely, one of them is installing software. Here I am going to tell you how to install software through wmic (Windows management instrument console). I am not at all saying that this i…
You may have already been in the need to update a whole folder stucture using a script. Robocopy does it well and even provides a list of non-updated files in a log (if asked to). Generally those files that were locked by a user or a process by the …
In this video I am going to show you how to back up and restore Office 365 mailboxes using CodeTwo Backup for Office 365. Learn more about the tool used in this video here: http://www.codetwo.com/backup-for-office-365/ (http://www.codetwo.com/ba…
As a trusted technology advisor to your customers you are likely getting the daily question of, ‘should I put this in the cloud?’ As customer demands for cloud services increases, companies will see a shift from traditional buying patterns to new…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

24 Experts available now in Live!

Get 1:1 Help Now