Solved

Downloading from Web in batch mode

Posted on 1998-09-11
5
308 Views
Last Modified: 2010-04-21
I have a C program which processes some data. The data file is located in an anon ftp site. What I want to do is to download this file from inside my program using something like the "system" function or the like. The quesion is what to use for downloading. Does Netscape work in batch mode? I tried to use Lynx, it connected OK and started downloading but then gave me some strange error message. Do you know if there is a workaround?

Thanks,

IZ
0
Comment
Question by:ilyaz
  • 3
5 Comments
 

Accepted Solution

by:
bknowles earned 50 total points
ID: 2006927
If the data is on an ftp site, why not use plain old ftp?  If you need to do recursive ftp, then get ncftp and use it instead.

If you want to mirror the entire site to your local hard drive and then access it directly from your application, there's the Perl package "mirror" which will do that for you (this is the same package that most sites use to set up "official" mirrors of popular sites like winsite, simtel, tucows, etc...).
0
 
LVL 1

Expert Comment

by:jreese
ID: 2006928
You can download the data file directly from your C program using sockets. You can check any good C reference book/web page, and use the man pages on gethostbyname, gethostbyaddr, gethostent, sethostent, endhostent, herror, etc. Feel free to ask me for any specifics. Briefly, you'll want to connect to port 21 (ftp), then execute the command to download the data file. It's pretty straightforward.

Hope that points you in the right direction,

Jim
0
 

Expert Comment

by:bknowles
ID: 2006929
I get the impression that he doesn't want to have to implement the entire ftp protocol himself, he instead wants to use another program to do that for him.

Under those circumstances, the answer I gave above should be helpful.
0
 

Author Comment

by:ilyaz
ID: 2006930
Sorry for late reaction. I ended up using ftp.expect for ftp-located files and lynx -source for http-located ones.
0
 

Expert Comment

by:bknowles
ID: 2006931
Watch the use of HTTP -- with server-side includes, what was a dynamic way to put in a "This page was last changed on..." or "You are the Xth visitor to this page" or any number of other things, will now become hard-coded text.

IMO, I would recommend using ftp for everything possible, and fall back to http only when you have no choice.  Also note that Lynx will skip all graphics, Java, JavaScript, etc....
0

Featured Post

Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…

773 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question