Solved

How FTP pulls files

Posted on 2013-11-04
6
484 Views
Last Modified: 2013-11-05
I am pulling files from a customer's ftp sever on a daily basis.  Up until recently, we had no problems but suddenly we are getting files with today's date and the file has some other day's data.  It turns out that we rename the file on our side and the customer is trying to say we are messing up the data when we rename it.

Can someone please explain how ftp works when there are multiple files with virtually the same file name with a different date.

For example if the customer had three files on the ftp server
M700111.1234   10/31/2013
M700111.4567    11/01/2013
M700111.8999     11/02/2013

Given the following script what would be pulled from the customer's ftp server.    

1.  open 999.999.999.999
2.  user MICKEYMOUSE
3.  PASSWORD
4.  prompt
5.  lcd c:\TEMP\
6.  bin
7.  mget *.*
8.  quit
0
Comment
Question by:sherbug1015
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 39622301
That should get all of the files in the current remote directory.
0
 
LVL 56

Expert Comment

by:Bill Prew
ID: 39622387
It sounds like you are assuming that there will only be one file named M700111.<anything> ?  If that's the case, then you are likely to run into problems given that there can now be multiple files with that same base name, but different extensions.

I suspect that you are renaming the file based on the system clock on the receiving computer, and so multiple files could get the same time stamp when they are pulled, which may be a problem.

I guess the root question is what do you want to happen when there are more than one of these files?  Pull the latest one? Pull all and rename with the modified date from the remote system?  Etc...

~bp
0
 

Author Comment

by:sherbug1015
ID: 39622535
billprew -  that is what I would have thought as well.  I just went back to the customers ftp site and they had dropped another file out there since this morning when I first picked up.

There is a file called

BCL.SDOA.1442238  -  11/04/13  2:20 p.m.
BCL.SDOA.1442078  -  11/03/13  2:20 p.m.

I picked up the 11/03 file this morning and when I ran my script, it only picked up the 11/4 file.  Same file name, different extension.

How does it know that the 11/03 file has already been picked up is my question.  Is there something I am missing here.  It must be using some kind of system time to determine what has been picked up.  Can you confirm anything like that.

thanks
0
Get your Conversational Ransomware Defense e‑book

This e-book gives you an insight into the ransomware threat and reviews the fundamentals of top-notch ransomware preparedness and recovery. To help you protect yourself and your organization. The initial infection may be inevitable, so the best protection is to be fully prepared.

 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 39622590
If you have not renamed the file, then you probably won't see the 'over-write warning' dialog in a script like you would in a desktop client.  For a test, rename both files on your computer and run your script again.  I suspect it will then re-download both of them.
0
 
LVL 56

Assisted Solution

by:Bill Prew
Bill Prew earned 300 total points
ID: 39622597
The "mget *.*" should pull ALL files in that folder down to the client computer.  That being said, it could depend a bit on the FTP client software you are using on your computer.  If the file is still in the destination folder on the client PC, it may be "smart" enough to see that it is already there and validate it hasn't changed, and therefore not pull it down again.  But I don't think the Windows built in FTP command line client is smart enough to do that.

I expect thought, even if that is true, if you deleted any existing files from the destination folder on the client computer, it will then pull all files down again.  I very much doubt the server end is doing anything to determine what has already been sent down, that would not be typical at all.

~bp
0
 
LVL 16

Accepted Solution

by:
AlexPace earned 200 total points
ID: 39623042
The FTP protocol does not support wildcards.
MGET is a client command, not an FTP protocol command.
It works like this:
  1. Client requests a directory LIST
  2. Client parses list for files that match *.* or whatever pattern you used.
  3. Client downloads the matching files one at a time.

The sad thing about the FTP protocol is that the format of the directory list is not standard.
There is significant variation in the listing formats returned by different FTP servers.
This means that sometimes step #2 above fails because the list parsing routine failed.
It happens but it is rare.
Most likely the file just wasn't there when you ran the script.
Add a line to save a remote listing to a text file.
That file will help you see what the FTP server returned as a listing so you can determine if it is a parsing problem or if the file just wasn't there.
ftp.exe has no built-in means to skip a file because a local copy already exists.


edit: I've seen FTP servers scripted to delete a file after it is downloaded but those are almost always hosted by banks.
0

Featured Post

[Webinar] How Hackers Steal Your Credentials

Do You Know How Hackers Steal Your Credentials? Join us and Skyport Systems to learn how hackers steal your credentials and why Active Directory must be secure to stop them. Thursday, July 13, 2017 10:00 A.M. PDT

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Using dates in 'DOS' batch files has always been tricky as it has no built in ways of extracting date information.  There are many tricks using string manipulation to pull out parts of the %date% variable or output of the date /t command but these r…
AutoHotkey is an excellent, free, open source programming/scripting language for Windows. It started out as a keyboard/mouse macros product, but has expanded into a robust language. This article provides an introduction to it, with links to addition…
In this video we outline the Physical Segments view of NetCrunch network monitor. By following this brief how-to video, you will be able to learn how NetCrunch visualizes your network, how granular is the information collected, as well as where to f…
If you’ve ever visited a web page and noticed a cool font that you really liked the look of, but couldn’t figure out which font it was so that you could use it for your own work, then this video is for you! In this Micro Tutorial, you'll learn yo…

623 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question