Solved

----- GRABBING WEB PAGE TO DG/UX ------

Posted on 1999-01-29
8
360 Views
Last Modified: 2013-12-06
Forgive my ignorance of anything Unix related, but I'm trying to work out if/how I can grab a page from the web and download to my DG/UX server. Everything is set up correctly - i.e, I can e-mail, and ftp to various sites etc, but haven't got a clue how to write a little script to download a web page. I'm sure it must be possible - this is assuming I can get into the site on port 80. If anyone can enlighten me then I'd be very grateful - and if they have a script which I can cut and paste then even better. Sorry this isn't many points, but hope you can help anyway.
Thanks,
Dave
0
Comment
Question by:dgb001
  • 5
  • 2
8 Comments
 
LVL 3

Expert Comment

by:bluepet
ID: 2009446
You want to grab a webpage from somewhere to your unix right?

I suggest you search for this software

websnake

use win95.... grab the webpages and ftp them over to your unix...

websnake is a cool... tool to download and even mirror a whole site...

bluepet
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009447
Thanks for the comment, but that's not really what I'm trying to do - I can write a VB program to do that easily enough to run on Win95 and FTP it across.

My problem is that I have a super-douper Unix server sat around not doing very much, and a Win95 PC with about 150MB hard drive, and 16MB of RAM, which is so unbelievably slow. Well, there are some web pages with real-time data that I want to have the latest data from continuously, so running a proggie on the PC is a bit out of the question (because it's too slow and would hog everything else). So hence why I want the Unix server to grab the data - I can then easily write a program which sits on the Unix server and parses the data and sticks it into a format which I want, and continuously displays the fresh data.

But thanks anyway, I appreciate your comments.
Dave

0
 
LVL 3

Accepted Solution

by:
mliberi earned 20 total points
ID: 2009448
/*************************************************************/
/* wwwget.c                                   M.Liberi, 1999 */
/*************************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <time.h>
#include <unistd.h>

#define true    1
#define false   0
typedef char               boolean;
typedef unsigned char      byte;
typedef unsigned short     word;
typedef unsigned long      longword;

#include <netdb.h>
#include <sys/socket.h>
#include <netinet/in.h>

byte *pname;

void helpscreen(void)
{
  if (isatty(1))
    printf("\33[2J\33[H");
  printf("%s, (c) M.Liberi, ver. 1.00, Feb 1999\n"
"\n"
"get a www page\n"
"\n"
"syntax: %s [option]... pathname\n"
"  -h           this help screen\n"
"  -v           verbose, info msg to stdout\n"
"  -s           silent, no error msg to stderr\n"
"  --           no more options\n"
"  -t           Header only\n"
"  -Yhostname   connect trough a proxy\n"
"  -Hhostname   hostname to connect with                (default localhost)\n"
"  -p<n>        use portnumber n                        (default www/tcp)\n"
"\n", pname, pname);
}

int main(int argc, byte **argv)
{
  boolean               verbose=false, silent=false;
  int                   sd, c, st, PORT=80;
  char                  *hostname="localhost",
                        *proxy=NULL,
                        *method="GET";
  FILE                  *r, *w;
  struct servent        *se;
  struct hostent        *he;
  struct sockaddr_in    sa;

  pname=*argv;        /* program name */
  if (se=getservbyname("www","tcp"))
    PORT=se->s_port;
  while (++argv, --argc)         /* option parsing cycle */
  {
    if (**argv!='-')
      break;
    switch(argv[0][1])
    {
      case 'h':
        helpscreen();
        return 1;
      case 'v':
        verbose=true;
        break;
      case 's':
        silent=true;
        break;
      case '-':
        ++argv, --argc;
        goto fineopzioni;
      case 'Y':
        proxy=*argv+2;
        break;
      case 'H':
        hostname=*argv+2;
        break;
      case 'p':
        PORT=atoi(*argv+2);
        break;
      case 't':
        verbose=true;
        method="HEAD";
        break;
      default:
        if (!silent)
          fprintf(stderr,"Warning! invalid option -%c\n", argv[0][1]);
    }
  }
  fineopzioni:
  if (!argc){
    helpscreen();
    return 1;
  }
  if (!proxy) proxy=hostname;
  if (!(he=gethostbyname(proxy))){
    if (!silent)
      fprintf(stderr,"%s: gethostbyname(%s)\n", pname, proxy);
    return 255;
  }
  if ((sd=socket(AF_INET,SOCK_STREAM,0)) == -1) {
    if (!silent)
      fprintf(stderr,"%s: socket()\n", pname);
    return 254;
  }
  sa.sin_family = AF_INET;
  sa.sin_port   = htons(PORT);
  sa.sin_addr   = *((struct in_addr *)he->h_addr);
  if (connect(sd,(struct sockaddr *)&sa,sizeof(struct sockaddr))==-1){
    if (!silent)
      fprintf(stderr,"%s: connect()\n", pname);
    return 253;
  }

  if ( (!(r=fdopen(sd,"rb"))) || (!(w=fdopen(sd,"wb"))) ){
    if (!silent)
      fprintf(stderr,"%s: fdopen()\n", pname);
    return 252;
  }

  /* send get request */
  fprintf(w,"%s ", method);
  if (strcmp(proxy,hostname))
    fprintf(w,"http://%s", hostname);
  fprintf(w,"/%s HTTP/1.1\r\n", *argv);
  fprintf(w,"Host: %s\r\n", hostname);
  fprintf(w,"Connection: close\r\n");
  fprintf(w,"Cache-Control: no-cache\r\n");
  fprintf(w,"\r\n");
  fflush(w);

  /* get page */
  st=0;
  while ((c=fgetc(r))!=EOF)
    if (st>1 || verbose) fputc(c,stdout); else
    switch(c){
      case '\n':
        st+=1;
      case '\r':
        break;
      default:
        st=0;
    }

  fclose(r);
  fclose(w);
  return 0;
}

You can also find a compiled version (AIX 4) of this program (among other stuff) at the following address:

http://www.geocities.com/SiliconValley/Horizon/5745
0
Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 1

Author Comment

by:dgb001
ID: 2009449
Hmm, yes, that looks like the sort of thing I'm looking for mliberi. If you'll excuse my delay then could you give me a couple of days to try it out (kinda busy at the moment). But that looks good, so thanks in advance. Hopefully throw the points your way in a couple of days.

Regards,
Dave



0
 
LVL 1

Author Comment

by:dgb001
ID: 2009450
Hello again, just wondering (given my zero knowledge of Unix and C etc) if you can help out with with compiling the above.

I 've tried to compile it using the following syntax :
#  cc grabdata.c -o

but get the following error message, and it quits:
Fatal error in /usr/sde/default/usr/lib/gcc/cc1
Exit status 041

Any ideas ?
Many thx,
Dave
0
 
LVL 3

Expert Comment

by:mliberi
ID: 2009451
I compile with the command "cc -o wwwget wwwget.c"

the problem in your command is that nothing follows -o option
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009452
Thx for the suggestion  - I tried it the same way as you suggested (even renamed the file) but still get the same error.

If you have any other ideas then that would be good, but if not then don't worry - I'll still give you the points 'cos I think it's probably more down to my lack of Unix/C knowledge than anything else.
Regards
Dave
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009453
Thx

0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
How to backup and restore the DHCP configuration on Solaris 9 7 55
CPU monthly average? 12 131
OpenLDAP Proxy to Active Directy 6 363
Problem logging tar errors 11 63
When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question