Solved

C Language question

Posted on 1997-06-05
4
244 Views
Last Modified: 2010-04-15
I am in need of source code (or fragments thereof) for writting a small 'webcrawler' style utility.

The program needs to open a session with a (specified) Webserver on port 80, send a URL and then write the resultant page to a local flat file.

My C skills are largely self taught but once I have the basic functions I should be able to work out the rest.


Thanks in advance.

Marty
0
Comment
Question by:mravell
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 
LVL 5

Accepted Solution

by:
julio011597 earned 0 total points
ID: 1250948
I'll assume you are on Unix box.

Doing the job is quite simple, just involves a bit of socket programming, since you are actually coding a simple client application.

Following is a sample code for the get_url function:
please check function return types and include files on your OS, since there may be slight differences.

--//--
/*
#include <stdlib.h>
#include <stdio.h>
#include <string.h>     /* bzero() and bcopy() */
#include <netdb.h>      /* gethostbyname() */
#include <sys/socket.h> /* socket() and connect() */
#include <arpa/inet.h>  /* htons() */

/* get_url()
 * answer: file to write to (must be already open for writing)
 * host: e.g. "www.experts-exchange.com"
 * url : e.g. "/topics/comp/lang/c"
 * err_str: to eventually store an error message
 *
 * returns 0 if succesful, errno otherwise
 */

int get_url(FILE *answer, const char *host, const char *url, char err_str[])
{
  int sock;
  struct hostent *hep;
  struct sockaddr_in sai;
  FILE *infile, *outfile;
  char buff[BUFSIZ];

  if(!(hep = gethostbyname(host))) {
    strcpy(err_str, "gethostbyname error");
    return errno;
  }
  if((sock = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
    strcpy(err_str, "socket error");
    return errno;
  }

  bzero((char *)&sai, sizeof(struct sockaddr_in));
  sai.sin_family = AF_INET;
  sai.sin_port = htons((unsigned short)80);
  bcopy(hep->h_addr, (char *)&(sai.sin_addr.s_addr), hep->h_length); /* note: could just be 'sai.sin_addr' */

  if(connect(sock, (struct sockaddr *)&sai, sizeof(sai)) == -1) {
    strcpy(err_str, "connect error");
    return errno;
  }

  /* fine: now we are connected */
  /* let's work with stdio */
  if(!(infile = fdopen(sock, "r"))) {
    strcpy(err_str, "fdopen(in) error");
    return errno;
  }
  if(!(outfile = fdopen(sock, "w"))) {
    strcpy(err_str, "fdopen(out) error");
    return errno;
  }

  /* now, the query: i'll leave error checking now on! */
  fprintf(outfile, "GET %s\n", url);
  fflush(outfile); /* this is mandatory: outfile would buffer */

  while(fgets(buff, BUFSIZ, infile)) fputs(buff, answer);
  fflush(answer); /* this is not mandatory:) */

  /* cleanup */
  /* note: the two fclose's are also mandatory: */
  /* the file tables are not automagically deallocated */
  /* if you just close the socket (at least on my system), */
  /* and you could go out of resources if you loop */
  /* to connect to more than one site */
  fclose(infile); fclose(outfile);
  close(sock);

  return 0;
}
--//--

Ok, hope i didn't make too many mistakes in this small window...

Let me know, julio
0
 
LVL 5

Expert Comment

by:julio011597
ID: 1250949
Hello, i tried it and works! <g>

Joke apart, i had to add:
#include <sys/errno.h>
for errno to be defined.

Also, forget the '/*' just after the first cut-here.

Cheers, julio
0
 
LVL 5

Expert Comment

by:julio011597
ID: 1250950
Anybody in??
-julio
0
 

Author Comment

by:mravell
ID: 1250951
Thanks Julio, this is exactly what I was after!

Cheers
Marty
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

An Outlet in Cocoa is a persistent reference to a GUI control; it connects a property (a variable) to a control.  For example, it is common to create an Outlet for the text field GUI control and change the text that appears in this field via that Ou…
Preface I don't like visual development tools that are supposed to write a program for me. Even if it is Xcode and I can use Interface Builder. Yes, it is a perfect tool and has helped me a lot, mainly, in the beginning, when my programs were small…
The goal of this video is to provide viewers with basic examples to understand and use pointers in the C programming language.
The goal of this video is to provide viewers with basic examples to understand how to create, access, and change arrays in the C programming language.
Suggested Courses

710 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question