Solved

Downloading from Web in batch mode

Posted on 1998-09-11
5
282 Views
Last Modified: 2010-04-21
I have a C program which processes some data. The data file is located in an anon ftp site. What I want to do is to download this file from inside my program using something like the "system" function or the like. The quesion is what to use for downloading. Does Netscape work in batch mode? I tried to use Lynx, it connected OK and started downloading but then gave me some strange error message. Do you know if there is a workaround?

Thanks,

IZ
0
Comment
Question by:ilyaz
  • 3
5 Comments
 

Accepted Solution

by:
bknowles earned 50 total points
Comment Utility
If the data is on an ftp site, why not use plain old ftp?  If you need to do recursive ftp, then get ncftp and use it instead.

If you want to mirror the entire site to your local hard drive and then access it directly from your application, there's the Perl package "mirror" which will do that for you (this is the same package that most sites use to set up "official" mirrors of popular sites like winsite, simtel, tucows, etc...).
0
 
LVL 1

Expert Comment

by:jreese
Comment Utility
You can download the data file directly from your C program using sockets. You can check any good C reference book/web page, and use the man pages on gethostbyname, gethostbyaddr, gethostent, sethostent, endhostent, herror, etc. Feel free to ask me for any specifics. Briefly, you'll want to connect to port 21 (ftp), then execute the command to download the data file. It's pretty straightforward.

Hope that points you in the right direction,

Jim
0
 

Expert Comment

by:bknowles
Comment Utility
I get the impression that he doesn't want to have to implement the entire ftp protocol himself, he instead wants to use another program to do that for him.

Under those circumstances, the answer I gave above should be helpful.
0
 

Author Comment

by:ilyaz
Comment Utility
Sorry for late reaction. I ended up using ftp.expect for ftp-located files and lynx -source for http-located ones.
0
 

Expert Comment

by:bknowles
Comment Utility
Watch the use of HTTP -- with server-side includes, what was a dynamic way to put in a "This page was last changed on..." or "You are the Xth visitor to this page" or any number of other things, will now become hard-coded text.

IMO, I would recommend using ftp for everything possible, and fall back to http only when you have no choice.  Also note that Lynx will skip all graphics, Java, JavaScript, etc....
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Introduction Regular patching is part of a system administrator's tasks. However, many patches require that the system be in single-user mode before they can be installed. A cluster patch in particular can take quite a while to apply if the machine…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now