----- GRABBING WEB PAGE TO DG/UX ------

Forgive my ignorance of anything Unix related, but I'm trying to work out if/how I can grab a page from the web and download to my DG/UX server. Everything is set up correctly - i.e, I can e-mail, and ftp to various sites etc, but haven't got a clue how to write a little script to download a web page. I'm sure it must be possible - this is assuming I can get into the site on port 80. If anyone can enlighten me then I'd be very grateful - and if they have a script which I can cut and paste then even better. Sorry this isn't many points, but hope you can help anyway.
Thanks,
Dave
LVL 1
dgb001Asked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
mliberiConnect With a Mentor Commented:
/*************************************************************/
/* wwwget.c                                   M.Liberi, 1999 */
/*************************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <time.h>
#include <unistd.h>

#define true    1
#define false   0
typedef char               boolean;
typedef unsigned char      byte;
typedef unsigned short     word;
typedef unsigned long      longword;

#include <netdb.h>
#include <sys/socket.h>
#include <netinet/in.h>

byte *pname;

void helpscreen(void)
{
  if (isatty(1))
    printf("\33[2J\33[H");
  printf("%s, (c) M.Liberi, ver. 1.00, Feb 1999\n"
"\n"
"get a www page\n"
"\n"
"syntax: %s [option]... pathname\n"
"  -h           this help screen\n"
"  -v           verbose, info msg to stdout\n"
"  -s           silent, no error msg to stderr\n"
"  --           no more options\n"
"  -t           Header only\n"
"  -Yhostname   connect trough a proxy\n"
"  -Hhostname   hostname to connect with                (default localhost)\n"
"  -p<n>        use portnumber n                        (default www/tcp)\n"
"\n", pname, pname);
}

int main(int argc, byte **argv)
{
  boolean               verbose=false, silent=false;
  int                   sd, c, st, PORT=80;
  char                  *hostname="localhost",
                        *proxy=NULL,
                        *method="GET";
  FILE                  *r, *w;
  struct servent        *se;
  struct hostent        *he;
  struct sockaddr_in    sa;

  pname=*argv;        /* program name */
  if (se=getservbyname("www","tcp"))
    PORT=se->s_port;
  while (++argv, --argc)         /* option parsing cycle */
  {
    if (**argv!='-')
      break;
    switch(argv[0][1])
    {
      case 'h':
        helpscreen();
        return 1;
      case 'v':
        verbose=true;
        break;
      case 's':
        silent=true;
        break;
      case '-':
        ++argv, --argc;
        goto fineopzioni;
      case 'Y':
        proxy=*argv+2;
        break;
      case 'H':
        hostname=*argv+2;
        break;
      case 'p':
        PORT=atoi(*argv+2);
        break;
      case 't':
        verbose=true;
        method="HEAD";
        break;
      default:
        if (!silent)
          fprintf(stderr,"Warning! invalid option -%c\n", argv[0][1]);
    }
  }
  fineopzioni:
  if (!argc){
    helpscreen();
    return 1;
  }
  if (!proxy) proxy=hostname;
  if (!(he=gethostbyname(proxy))){
    if (!silent)
      fprintf(stderr,"%s: gethostbyname(%s)\n", pname, proxy);
    return 255;
  }
  if ((sd=socket(AF_INET,SOCK_STREAM,0)) == -1) {
    if (!silent)
      fprintf(stderr,"%s: socket()\n", pname);
    return 254;
  }
  sa.sin_family = AF_INET;
  sa.sin_port   = htons(PORT);
  sa.sin_addr   = *((struct in_addr *)he->h_addr);
  if (connect(sd,(struct sockaddr *)&sa,sizeof(struct sockaddr))==-1){
    if (!silent)
      fprintf(stderr,"%s: connect()\n", pname);
    return 253;
  }

  if ( (!(r=fdopen(sd,"rb"))) || (!(w=fdopen(sd,"wb"))) ){
    if (!silent)
      fprintf(stderr,"%s: fdopen()\n", pname);
    return 252;
  }

  /* send get request */
  fprintf(w,"%s ", method);
  if (strcmp(proxy,hostname))
    fprintf(w,"http://%s", hostname);
  fprintf(w,"/%s HTTP/1.1\r\n", *argv);
  fprintf(w,"Host: %s\r\n", hostname);
  fprintf(w,"Connection: close\r\n");
  fprintf(w,"Cache-Control: no-cache\r\n");
  fprintf(w,"\r\n");
  fflush(w);

  /* get page */
  st=0;
  while ((c=fgetc(r))!=EOF)
    if (st>1 || verbose) fputc(c,stdout); else
    switch(c){
      case '\n':
        st+=1;
      case '\r':
        break;
      default:
        st=0;
    }

  fclose(r);
  fclose(w);
  return 0;
}

You can also find a compiled version (AIX 4) of this program (among other stuff) at the following address:

http://www.geocities.com/SiliconValley/Horizon/5745
0
 
bluepetCommented:
You want to grab a webpage from somewhere to your unix right?

I suggest you search for this software

websnake

use win95.... grab the webpages and ftp them over to your unix...

websnake is a cool... tool to download and even mirror a whole site...

bluepet
0
 
dgb001Author Commented:
Thanks for the comment, but that's not really what I'm trying to do - I can write a VB program to do that easily enough to run on Win95 and FTP it across.

My problem is that I have a super-douper Unix server sat around not doing very much, and a Win95 PC with about 150MB hard drive, and 16MB of RAM, which is so unbelievably slow. Well, there are some web pages with real-time data that I want to have the latest data from continuously, so running a proggie on the PC is a bit out of the question (because it's too slow and would hog everything else). So hence why I want the Unix server to grab the data - I can then easily write a program which sits on the Unix server and parses the data and sticks it into a format which I want, and continuously displays the fresh data.

But thanks anyway, I appreciate your comments.
Dave

0
Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
dgb001Author Commented:
Hmm, yes, that looks like the sort of thing I'm looking for mliberi. If you'll excuse my delay then could you give me a couple of days to try it out (kinda busy at the moment). But that looks good, so thanks in advance. Hopefully throw the points your way in a couple of days.

Regards,
Dave



0
 
dgb001Author Commented:
Hello again, just wondering (given my zero knowledge of Unix and C etc) if you can help out with with compiling the above.

I 've tried to compile it using the following syntax :
#  cc grabdata.c -o

but get the following error message, and it quits:
Fatal error in /usr/sde/default/usr/lib/gcc/cc1
Exit status 041

Any ideas ?
Many thx,
Dave
0
 
mliberiCommented:
I compile with the command "cc -o wwwget wwwget.c"

the problem in your command is that nothing follows -o option
0
 
dgb001Author Commented:
Thx for the suggestion  - I tried it the same way as you suggested (even renamed the file) but still get the same error.

If you have any other ideas then that would be good, but if not then don't worry - I'll still give you the points 'cos I think it's probably more down to my lack of Unix/C knowledge than anything else.
Regards
Dave
0
 
dgb001Author Commented:
Thx

0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.