Solved

----- GRABBING WEB PAGE TO DG/UX ------

Posted on 1999-01-29
8
348 Views
Last Modified: 2013-12-06
Forgive my ignorance of anything Unix related, but I'm trying to work out if/how I can grab a page from the web and download to my DG/UX server. Everything is set up correctly - i.e, I can e-mail, and ftp to various sites etc, but haven't got a clue how to write a little script to download a web page. I'm sure it must be possible - this is assuming I can get into the site on port 80. If anyone can enlighten me then I'd be very grateful - and if they have a script which I can cut and paste then even better. Sorry this isn't many points, but hope you can help anyway.
Thanks,
Dave
0
Comment
Question by:dgb001
  • 5
  • 2
8 Comments
 
LVL 3

Expert Comment

by:bluepet
ID: 2009446
You want to grab a webpage from somewhere to your unix right?

I suggest you search for this software

websnake

use win95.... grab the webpages and ftp them over to your unix...

websnake is a cool... tool to download and even mirror a whole site...

bluepet
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009447
Thanks for the comment, but that's not really what I'm trying to do - I can write a VB program to do that easily enough to run on Win95 and FTP it across.

My problem is that I have a super-douper Unix server sat around not doing very much, and a Win95 PC with about 150MB hard drive, and 16MB of RAM, which is so unbelievably slow. Well, there are some web pages with real-time data that I want to have the latest data from continuously, so running a proggie on the PC is a bit out of the question (because it's too slow and would hog everything else). So hence why I want the Unix server to grab the data - I can then easily write a program which sits on the Unix server and parses the data and sticks it into a format which I want, and continuously displays the fresh data.

But thanks anyway, I appreciate your comments.
Dave

0
 
LVL 3

Accepted Solution

by:
mliberi earned 20 total points
ID: 2009448
/*************************************************************/
/* wwwget.c                                   M.Liberi, 1999 */
/*************************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <time.h>
#include <unistd.h>

#define true    1
#define false   0
typedef char               boolean;
typedef unsigned char      byte;
typedef unsigned short     word;
typedef unsigned long      longword;

#include <netdb.h>
#include <sys/socket.h>
#include <netinet/in.h>

byte *pname;

void helpscreen(void)
{
  if (isatty(1))
    printf("\33[2J\33[H");
  printf("%s, (c) M.Liberi, ver. 1.00, Feb 1999\n"
"\n"
"get a www page\n"
"\n"
"syntax: %s [option]... pathname\n"
"  -h           this help screen\n"
"  -v           verbose, info msg to stdout\n"
"  -s           silent, no error msg to stderr\n"
"  --           no more options\n"
"  -t           Header only\n"
"  -Yhostname   connect trough a proxy\n"
"  -Hhostname   hostname to connect with                (default localhost)\n"
"  -p<n>        use portnumber n                        (default www/tcp)\n"
"\n", pname, pname);
}

int main(int argc, byte **argv)
{
  boolean               verbose=false, silent=false;
  int                   sd, c, st, PORT=80;
  char                  *hostname="localhost",
                        *proxy=NULL,
                        *method="GET";
  FILE                  *r, *w;
  struct servent        *se;
  struct hostent        *he;
  struct sockaddr_in    sa;

  pname=*argv;        /* program name */
  if (se=getservbyname("www","tcp"))
    PORT=se->s_port;
  while (++argv, --argc)         /* option parsing cycle */
  {
    if (**argv!='-')
      break;
    switch(argv[0][1])
    {
      case 'h':
        helpscreen();
        return 1;
      case 'v':
        verbose=true;
        break;
      case 's':
        silent=true;
        break;
      case '-':
        ++argv, --argc;
        goto fineopzioni;
      case 'Y':
        proxy=*argv+2;
        break;
      case 'H':
        hostname=*argv+2;
        break;
      case 'p':
        PORT=atoi(*argv+2);
        break;
      case 't':
        verbose=true;
        method="HEAD";
        break;
      default:
        if (!silent)
          fprintf(stderr,"Warning! invalid option -%c\n", argv[0][1]);
    }
  }
  fineopzioni:
  if (!argc){
    helpscreen();
    return 1;
  }
  if (!proxy) proxy=hostname;
  if (!(he=gethostbyname(proxy))){
    if (!silent)
      fprintf(stderr,"%s: gethostbyname(%s)\n", pname, proxy);
    return 255;
  }
  if ((sd=socket(AF_INET,SOCK_STREAM,0)) == -1) {
    if (!silent)
      fprintf(stderr,"%s: socket()\n", pname);
    return 254;
  }
  sa.sin_family = AF_INET;
  sa.sin_port   = htons(PORT);
  sa.sin_addr   = *((struct in_addr *)he->h_addr);
  if (connect(sd,(struct sockaddr *)&sa,sizeof(struct sockaddr))==-1){
    if (!silent)
      fprintf(stderr,"%s: connect()\n", pname);
    return 253;
  }

  if ( (!(r=fdopen(sd,"rb"))) || (!(w=fdopen(sd,"wb"))) ){
    if (!silent)
      fprintf(stderr,"%s: fdopen()\n", pname);
    return 252;
  }

  /* send get request */
  fprintf(w,"%s ", method);
  if (strcmp(proxy,hostname))
    fprintf(w,"http://%s", hostname);
  fprintf(w,"/%s HTTP/1.1\r\n", *argv);
  fprintf(w,"Host: %s\r\n", hostname);
  fprintf(w,"Connection: close\r\n");
  fprintf(w,"Cache-Control: no-cache\r\n");
  fprintf(w,"\r\n");
  fflush(w);

  /* get page */
  st=0;
  while ((c=fgetc(r))!=EOF)
    if (st>1 || verbose) fputc(c,stdout); else
    switch(c){
      case '\n':
        st+=1;
      case '\r':
        break;
      default:
        st=0;
    }

  fclose(r);
  fclose(w);
  return 0;
}

You can also find a compiled version (AIX 4) of this program (among other stuff) at the following address:

http://www.geocities.com/SiliconValley/Horizon/5745
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009449
Hmm, yes, that looks like the sort of thing I'm looking for mliberi. If you'll excuse my delay then could you give me a couple of days to try it out (kinda busy at the moment). But that looks good, so thanks in advance. Hopefully throw the points your way in a couple of days.

Regards,
Dave



0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 1

Author Comment

by:dgb001
ID: 2009450
Hello again, just wondering (given my zero knowledge of Unix and C etc) if you can help out with with compiling the above.

I 've tried to compile it using the following syntax :
#  cc grabdata.c -o

but get the following error message, and it quits:
Fatal error in /usr/sde/default/usr/lib/gcc/cc1
Exit status 041

Any ideas ?
Many thx,
Dave
0
 
LVL 3

Expert Comment

by:mliberi
ID: 2009451
I compile with the command "cc -o wwwget wwwget.c"

the problem in your command is that nothing follows -o option
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009452
Thx for the suggestion  - I tried it the same way as you suggested (even renamed the file) but still get the same error.

If you have any other ideas then that would be good, but if not then don't worry - I'll still give you the points 'cos I think it's probably more down to my lack of Unix/C knowledge than anything else.
Regards
Dave
0
 
LVL 1

Author Comment

by:dgb001
ID: 2009453
Thx

0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

In tuning file systems on the Solaris Operating System, changing some parameters of a file system usually destroys the data on it. For instance, changing the cache segment block size in the volume of a T3 requires that you delete the existing volu…
Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now