Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Downloading from Web in batch mode

Posted on 1998-09-11
5
Medium Priority
?
338 Views
Last Modified: 2010-04-21
I have a C program which processes some data. The data file is located in an anon ftp site. What I want to do is to download this file from inside my program using something like the "system" function or the like. The quesion is what to use for downloading. Does Netscape work in batch mode? I tried to use Lynx, it connected OK and started downloading but then gave me some strange error message. Do you know if there is a workaround?

Thanks,

IZ
0
Comment
Question by:ilyaz
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
5 Comments
 

Accepted Solution

by:
bknowles earned 150 total points
ID: 2006927
If the data is on an ftp site, why not use plain old ftp?  If you need to do recursive ftp, then get ncftp and use it instead.

If you want to mirror the entire site to your local hard drive and then access it directly from your application, there's the Perl package "mirror" which will do that for you (this is the same package that most sites use to set up "official" mirrors of popular sites like winsite, simtel, tucows, etc...).
0
 
LVL 1

Expert Comment

by:jreese
ID: 2006928
You can download the data file directly from your C program using sockets. You can check any good C reference book/web page, and use the man pages on gethostbyname, gethostbyaddr, gethostent, sethostent, endhostent, herror, etc. Feel free to ask me for any specifics. Briefly, you'll want to connect to port 21 (ftp), then execute the command to download the data file. It's pretty straightforward.

Hope that points you in the right direction,

Jim
0
 

Expert Comment

by:bknowles
ID: 2006929
I get the impression that he doesn't want to have to implement the entire ftp protocol himself, he instead wants to use another program to do that for him.

Under those circumstances, the answer I gave above should be helpful.
0
 

Author Comment

by:ilyaz
ID: 2006930
Sorry for late reaction. I ended up using ftp.expect for ftp-located files and lynx -source for http-located ones.
0
 

Expert Comment

by:bknowles
ID: 2006931
Watch the use of HTTP -- with server-side includes, what was a dynamic way to put in a "This page was last changed on..." or "You are the Xth visitor to this page" or any number of other things, will now become hard-coded text.

IMO, I would recommend using ftp for everything possible, and fall back to http only when you have no choice.  Also note that Lynx will skip all graphics, Java, JavaScript, etc....
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

My previous tech tip, Installing the Solaris OS From the Flash Archive On a Tape (http://www.experts-exchange.com/articles/OS/Unix/Solaris/Installing-the-Solaris-OS-From-the-Flash-Archive-on-a-Tape.html), discussed installing the Solaris Operating S…
FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question