Solved

How FTP pulls files

Posted on 2013-11-04
6
462 Views
Last Modified: 2013-11-05
I am pulling files from a customer's ftp sever on a daily basis.  Up until recently, we had no problems but suddenly we are getting files with today's date and the file has some other day's data.  It turns out that we rename the file on our side and the customer is trying to say we are messing up the data when we rename it.

Can someone please explain how ftp works when there are multiple files with virtually the same file name with a different date.

For example if the customer had three files on the ftp server
M700111.1234   10/31/2013
M700111.4567    11/01/2013
M700111.8999     11/02/2013

Given the following script what would be pulled from the customer's ftp server.    

1.  open 999.999.999.999
2.  user MICKEYMOUSE
3.  PASSWORD
4.  prompt
5.  lcd c:\TEMP\
6.  bin
7.  mget *.*
8.  quit
0
Comment
Question by:sherbug1015
6 Comments
 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 39622301
That should get all of the files in the current remote directory.
0
 
LVL 53

Expert Comment

by:Bill Prew
ID: 39622387
It sounds like you are assuming that there will only be one file named M700111.<anything> ?  If that's the case, then you are likely to run into problems given that there can now be multiple files with that same base name, but different extensions.

I suspect that you are renaming the file based on the system clock on the receiving computer, and so multiple files could get the same time stamp when they are pulled, which may be a problem.

I guess the root question is what do you want to happen when there are more than one of these files?  Pull the latest one? Pull all and rename with the modified date from the remote system?  Etc...

~bp
0
 

Author Comment

by:sherbug1015
ID: 39622535
billprew -  that is what I would have thought as well.  I just went back to the customers ftp site and they had dropped another file out there since this morning when I first picked up.

There is a file called

BCL.SDOA.1442238  -  11/04/13  2:20 p.m.
BCL.SDOA.1442078  -  11/03/13  2:20 p.m.

I picked up the 11/03 file this morning and when I ran my script, it only picked up the 11/4 file.  Same file name, different extension.

How does it know that the 11/03 file has already been picked up is my question.  Is there something I am missing here.  It must be using some kind of system time to determine what has been picked up.  Can you confirm anything like that.

thanks
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 39622590
If you have not renamed the file, then you probably won't see the 'over-write warning' dialog in a script like you would in a desktop client.  For a test, rename both files on your computer and run your script again.  I suspect it will then re-download both of them.
0
 
LVL 53

Assisted Solution

by:Bill Prew
Bill Prew earned 300 total points
ID: 39622597
The "mget *.*" should pull ALL files in that folder down to the client computer.  That being said, it could depend a bit on the FTP client software you are using on your computer.  If the file is still in the destination folder on the client PC, it may be "smart" enough to see that it is already there and validate it hasn't changed, and therefore not pull it down again.  But I don't think the Windows built in FTP command line client is smart enough to do that.

I expect thought, even if that is true, if you deleted any existing files from the destination folder on the client computer, it will then pull all files down again.  I very much doubt the server end is doing anything to determine what has already been sent down, that would not be typical at all.

~bp
0
 
LVL 16

Accepted Solution

by:
AlexPace earned 200 total points
ID: 39623042
The FTP protocol does not support wildcards.
MGET is a client command, not an FTP protocol command.
It works like this:
  1. Client requests a directory LIST
  2. Client parses list for files that match *.* or whatever pattern you used.
  3. Client downloads the matching files one at a time.

The sad thing about the FTP protocol is that the format of the directory list is not standard.
There is significant variation in the listing formats returned by different FTP servers.
This means that sometimes step #2 above fails because the list parsing routine failed.
It happens but it is rare.
Most likely the file just wasn't there when you ran the script.
Add a line to save a remote listing to a text file.
That file will help you see what the FTP server returned as a listing so you can determine if it is a parsing problem or if the file just wasn't there.
ftp.exe has no built-in means to skip a file because a local copy already exists.


edit: I've seen FTP servers scripted to delete a file after it is downloaded but those are almost always hosted by banks.
0

Featured Post

3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Being a system administrator some time we require to do things remotely, one of them is installing software. Here I am going to tell you how to install software through wmic (Windows management instrument console). I am not at all saying that this i…
This article was inspired by a question here at Experts Exchange (http://www.experts-exchange.com/Software/Photos_Graphics/Images_and_Photos/Q_28629170.html). The requirements stated in that question are (1) reduce the file size of a large number of…
In a recent question (https://www.experts-exchange.com/questions/28997919/Pagination-in-Adobe-Acrobat.html) here at Experts Exchange, a member asked how to add page numbers to a PDF file using Adobe Acrobat XI Pro. This short video Micro Tutorial sh…

831 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question