Solved

Downloading from Web in batch mode

Posted on 1998-09-11
5
321 Views
Last Modified: 2010-04-21
I have a C program which processes some data. The data file is located in an anon ftp site. What I want to do is to download this file from inside my program using something like the "system" function or the like. The quesion is what to use for downloading. Does Netscape work in batch mode? I tried to use Lynx, it connected OK and started downloading but then gave me some strange error message. Do you know if there is a workaround?

Thanks,

IZ
0
Comment
Question by:ilyaz
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
5 Comments
 

Accepted Solution

by:
bknowles earned 50 total points
ID: 2006927
If the data is on an ftp site, why not use plain old ftp?  If you need to do recursive ftp, then get ncftp and use it instead.

If you want to mirror the entire site to your local hard drive and then access it directly from your application, there's the Perl package "mirror" which will do that for you (this is the same package that most sites use to set up "official" mirrors of popular sites like winsite, simtel, tucows, etc...).
0
 
LVL 1

Expert Comment

by:jreese
ID: 2006928
You can download the data file directly from your C program using sockets. You can check any good C reference book/web page, and use the man pages on gethostbyname, gethostbyaddr, gethostent, sethostent, endhostent, herror, etc. Feel free to ask me for any specifics. Briefly, you'll want to connect to port 21 (ftp), then execute the command to download the data file. It's pretty straightforward.

Hope that points you in the right direction,

Jim
0
 

Expert Comment

by:bknowles
ID: 2006929
I get the impression that he doesn't want to have to implement the entire ftp protocol himself, he instead wants to use another program to do that for him.

Under those circumstances, the answer I gave above should be helpful.
0
 

Author Comment

by:ilyaz
ID: 2006930
Sorry for late reaction. I ended up using ftp.expect for ftp-located files and lynx -source for http-located ones.
0
 

Expert Comment

by:bknowles
ID: 2006931
Watch the use of HTTP -- with server-side includes, what was a dynamic way to put in a "This page was last changed on..." or "You are the Xth visitor to this page" or any number of other things, will now become hard-coded text.

IMO, I would recommend using ftp for everything possible, and fall back to http only when you have no choice.  Also note that Lynx will skip all graphics, Java, JavaScript, etc....
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question