Solved

----- GRABBING WEB PAGE TO DG/UX ------

Posted on 1999-01-29
8
357 Views
Last Modified: 2013-12-06
Forgive my ignorance of anything Unix related, but I'm trying to work out if/how I can grab a page from the web and download to my DG/UX server. Everything is set up correctly - i.e, I can e-mail, and ftp to various sites etc, but haven't got a clue how to write a little script to download a web page. I'm sure it must be possible - this is assuming I can get into the site on port 80. If anyone can enlighten me then I'd be very grateful - and if they have a script which I can cut and paste then even better. Sorry this isn't many points, but hope you can help anyway.
Thanks,
Dave
0
Comment
Question by:dgb001
  • 5
  • 2
8 Comments
 
LVL 3

Expert Comment

by:bluepet
ID: 2009446
You want to grab a webpage from somewhere to your unix right?

I suggest you search for this software

websnake

use win95.... grab the webpages and ftp them over to your unix...

websnake is a cool... tool to download and even mirror a whole site...

bluepet
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009447
Thanks for the comment, but that's not really what I'm trying to do - I can write a VB program to do that easily enough to run on Win95 and FTP it across.

My problem is that I have a super-douper Unix server sat around not doing very much, and a Win95 PC with about 150MB hard drive, and 16MB of RAM, which is so unbelievably slow. Well, there are some web pages with real-time data that I want to have the latest data from continuously, so running a proggie on the PC is a bit out of the question (because it's too slow and would hog everything else). So hence why I want the Unix server to grab the data - I can then easily write a program which sits on the Unix server and parses the data and sticks it into a format which I want, and continuously displays the fresh data.

But thanks anyway, I appreciate your comments.
Dave

0
 
LVL 3

Accepted Solution

by:
mliberi earned 20 total points
ID: 2009448
/*************************************************************/
/* wwwget.c                                   M.Liberi, 1999 */
/*************************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <time.h>
#include <unistd.h>

#define true    1
#define false   0
typedef char               boolean;
typedef unsigned char      byte;
typedef unsigned short     word;
typedef unsigned long      longword;

#include <netdb.h>
#include <sys/socket.h>
#include <netinet/in.h>

byte *pname;

void helpscreen(void)
{
  if (isatty(1))
    printf("\33[2J\33[H");
  printf("%s, (c) M.Liberi, ver. 1.00, Feb 1999\n"
"\n"
"get a www page\n"
"\n"
"syntax: %s [option]... pathname\n"
"  -h           this help screen\n"
"  -v           verbose, info msg to stdout\n"
"  -s           silent, no error msg to stderr\n"
"  --           no more options\n"
"  -t           Header only\n"
"  -Yhostname   connect trough a proxy\n"
"  -Hhostname   hostname to connect with                (default localhost)\n"
"  -p<n>        use portnumber n                        (default www/tcp)\n"
"\n", pname, pname);
}

int main(int argc, byte **argv)
{
  boolean               verbose=false, silent=false;
  int                   sd, c, st, PORT=80;
  char                  *hostname="localhost",
                        *proxy=NULL,
                        *method="GET";
  FILE                  *r, *w;
  struct servent        *se;
  struct hostent        *he;
  struct sockaddr_in    sa;

  pname=*argv;        /* program name */
  if (se=getservbyname("www","tcp"))
    PORT=se->s_port;
  while (++argv, --argc)         /* option parsing cycle */
  {
    if (**argv!='-')
      break;
    switch(argv[0][1])
    {
      case 'h':
        helpscreen();
        return 1;
      case 'v':
        verbose=true;
        break;
      case 's':
        silent=true;
        break;
      case '-':
        ++argv, --argc;
        goto fineopzioni;
      case 'Y':
        proxy=*argv+2;
        break;
      case 'H':
        hostname=*argv+2;
        break;
      case 'p':
        PORT=atoi(*argv+2);
        break;
      case 't':
        verbose=true;
        method="HEAD";
        break;
      default:
        if (!silent)
          fprintf(stderr,"Warning! invalid option -%c\n", argv[0][1]);
    }
  }
  fineopzioni:
  if (!argc){
    helpscreen();
    return 1;
  }
  if (!proxy) proxy=hostname;
  if (!(he=gethostbyname(proxy))){
    if (!silent)
      fprintf(stderr,"%s: gethostbyname(%s)\n", pname, proxy);
    return 255;
  }
  if ((sd=socket(AF_INET,SOCK_STREAM,0)) == -1) {
    if (!silent)
      fprintf(stderr,"%s: socket()\n", pname);
    return 254;
  }
  sa.sin_family = AF_INET;
  sa.sin_port   = htons(PORT);
  sa.sin_addr   = *((struct in_addr *)he->h_addr);
  if (connect(sd,(struct sockaddr *)&sa,sizeof(struct sockaddr))==-1){
    if (!silent)
      fprintf(stderr,"%s: connect()\n", pname);
    return 253;
  }

  if ( (!(r=fdopen(sd,"rb"))) || (!(w=fdopen(sd,"wb"))) ){
    if (!silent)
      fprintf(stderr,"%s: fdopen()\n", pname);
    return 252;
  }

  /* send get request */
  fprintf(w,"%s ", method);
  if (strcmp(proxy,hostname))
    fprintf(w,"http://%s", hostname);
  fprintf(w,"/%s HTTP/1.1\r\n", *argv);
  fprintf(w,"Host: %s\r\n", hostname);
  fprintf(w,"Connection: close\r\n");
  fprintf(w,"Cache-Control: no-cache\r\n");
  fprintf(w,"\r\n");
  fflush(w);

  /* get page */
  st=0;
  while ((c=fgetc(r))!=EOF)
    if (st>1 || verbose) fputc(c,stdout); else
    switch(c){
      case '\n':
        st+=1;
      case '\r':
        break;
      default:
        st=0;
    }

  fclose(r);
  fclose(w);
  return 0;
}

You can also find a compiled version (AIX 4) of this program (among other stuff) at the following address:

http://www.geocities.com/SiliconValley/Horizon/5745
0
Does Powershell have you tied up in knots?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

 
LVL 1

Author Comment

by:dgb001
ID: 2009449
Hmm, yes, that looks like the sort of thing I'm looking for mliberi. If you'll excuse my delay then could you give me a couple of days to try it out (kinda busy at the moment). But that looks good, so thanks in advance. Hopefully throw the points your way in a couple of days.

Regards,
Dave



0
 
LVL 1

Author Comment

by:dgb001
ID: 2009450
Hello again, just wondering (given my zero knowledge of Unix and C etc) if you can help out with with compiling the above.

I 've tried to compile it using the following syntax :
#  cc grabdata.c -o

but get the following error message, and it quits:
Fatal error in /usr/sde/default/usr/lib/gcc/cc1
Exit status 041

Any ideas ?
Many thx,
Dave
0
 
LVL 3

Expert Comment

by:mliberi
ID: 2009451
I compile with the command "cc -o wwwget wwwget.c"

the problem in your command is that nothing follows -o option
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009452
Thx for the suggestion  - I tried it the same way as you suggested (even renamed the file) but still get the same error.

If you have any other ideas then that would be good, but if not then don't worry - I'll still give you the points 'cos I think it's probably more down to my lack of Unix/C knowledge than anything else.
Regards
Dave
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009453
Thx

0

Featured Post

VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Hello fellow BSD lovers, I've created a patch process for patching openjdk6 for BSD (FreeBSD specifically), although I tried to keep all BSD versions in mind when creating my patch. Welcome to OpenJDK6 on BSD First let me start with a little …
Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

803 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question