Solved

Downloading from Web in batch mode

Posted on 1998-09-11
5
314 Views
Last Modified: 2010-04-21
I have a C program which processes some data. The data file is located in an anon ftp site. What I want to do is to download this file from inside my program using something like the "system" function or the like. The quesion is what to use for downloading. Does Netscape work in batch mode? I tried to use Lynx, it connected OK and started downloading but then gave me some strange error message. Do you know if there is a workaround?

Thanks,

IZ
0
Comment
Question by:ilyaz
  • 3
5 Comments
 

Accepted Solution

by:
bknowles earned 50 total points
ID: 2006927
If the data is on an ftp site, why not use plain old ftp?  If you need to do recursive ftp, then get ncftp and use it instead.

If you want to mirror the entire site to your local hard drive and then access it directly from your application, there's the Perl package "mirror" which will do that for you (this is the same package that most sites use to set up "official" mirrors of popular sites like winsite, simtel, tucows, etc...).
0
 
LVL 1

Expert Comment

by:jreese
ID: 2006928
You can download the data file directly from your C program using sockets. You can check any good C reference book/web page, and use the man pages on gethostbyname, gethostbyaddr, gethostent, sethostent, endhostent, herror, etc. Feel free to ask me for any specifics. Briefly, you'll want to connect to port 21 (ftp), then execute the command to download the data file. It's pretty straightforward.

Hope that points you in the right direction,

Jim
0
 

Expert Comment

by:bknowles
ID: 2006929
I get the impression that he doesn't want to have to implement the entire ftp protocol himself, he instead wants to use another program to do that for him.

Under those circumstances, the answer I gave above should be helpful.
0
 

Author Comment

by:ilyaz
ID: 2006930
Sorry for late reaction. I ended up using ftp.expect for ftp-located files and lynx -source for http-located ones.
0
 

Expert Comment

by:bknowles
ID: 2006931
Watch the use of HTTP -- with server-side includes, what was a dynamic way to put in a "This page was last changed on..." or "You are the Xth visitor to this page" or any number of other things, will now become hard-coded text.

IMO, I would recommend using ftp for everything possible, and fall back to http only when you have no choice.  Also note that Lynx will skip all graphics, Java, JavaScript, etc....
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question