Link to home
Start Free TrialLog in
Avatar of neilmcaliece
neilmcaliece

asked on

Get remote file via http from within c program on linux

Hi,

I'm relatively new to C, that is I used to program in it about 15 years ago but most of it is now forgotten and I've lost / given away my reference books over the last few house moves.

I need to download the contents of an external URL into a variable from within a standard C program which can be compiled to run on a Linux system.

Can anyone tell me either how to do this or give me information which will lead me to the solution.

Basicly I need an easy to use and free library which will sllow me to do the above along with sample code.

Thanks

Neil

Avatar of Tommy Braas
Tommy Braas
Flag of Australia image

I don't know about sample code, but here are the steps to solve your problem:

   1. Create a "struct hostent" with information about the server you're trying to connect to.
   2. Create a "struct sockaddr_in" with the information needed to create a socket (port etc.)
   3. Create and connect a socket to the server
   4. Issue a "GET" command for the URL you're trying to access, to the socket created in 3
   5. Read from the socket created in 3, and handle the result

Please find some code below (from an ftp server/client) that shows how to set things up. You will have to change some of this code!

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/param.h>
#include <sys/stat.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdio.h>
#include <signal.h>
#include <syslog.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <fcntl.h>

#define ERROR_CREATING_SOCKET -3
#define UNKNOWN_HOST -1

int socket_basic_setup( int tcp_port, int *socket_fd, struct sockaddr_in *socket_data )
{
      /* Create the socket, will be using internet addressing, will be connection oriented */
      (*socket_fd) = socket( AF_INET, SOCK_STREAM, 0 );

      if( *socket_fd < 0 )
      {
            perror("socket_basic_setup");
            return ERROR_CREATING_SOCKET;
      }
      
      /* Set defaults */
      memset(socket_data, 0, sizeof(socket_data));
      socket_data->sin_family      = AF_INET;              /* setting communication domain to internet adress */
      socket_data->sin_addr.s_addr = htonl(INADDR_ANY);    /* set socket to local host adress */
      socket_data->sin_port        = htons(tcp_port);   /* setting port number to in_tcp_port */

      return 0;
}

/*
 Function: client_socket_setup

 Purpose:  to handle setting up of a socket pointing to some host on the internet

 Return:   0 if successfull, -1 otherwise
 */
int client_socket_setup(int tcp_port, int *socket_fd, struct sockaddr_in *socket_data, struct hostent *host_info)
{
      if( socket_data )
      {
            if( socket_basic_setup(tcp_port, socket_fd, socket_data) )
            {
                  perror("client_socket_setup");
                  return ERROR_CREATING_SOCKET;
            }
            
            socket_data->sin_family = host_info->h_addrtype;

            memcpy( (char *)&socket_data->sin_addr, (char *)host_info->h_addr, host_info->h_length );

            return 0;
      }
      else
      {
            return ERROR_CREATING_SOCKET;
      }
}


struct hostent *get_host_info(short array[6])
{
      char *ip_address = NULL;
      struct hostent *host_info = NULL;
      
      ip_address = get_ip_address_as_string(array);
      if( ip_address )
      {
            host_info = gethostbyname(ip_address);
            free(ip_address);
            return host_info;
      }
      return NULL;
}


int send_command(const char *command, const char *parameter)
{
      char *temp;

      if( parameter )
      {
            temp = (char*)calloc(strlen(command) + strlen(parameter) + 4, sizeof(char));
            sprintf(temp, "%s %s\r\n", command, parameter);
      }
      else
      {
            temp = (char*)calloc(strlen(command) + 3, sizeof(char));
            sprintf(temp, "%s\r\n", command);
      }
      if( send(control_socket, temp, strlen(temp), 0) < 0 )
      {
            printf("421 %s\n", RS_SERVICE_NOT_AVAILABLE);
            return RC_SERVICE_NOT_AVAILABLE;
      }
      free(temp);
      return read_response();
}

int start_client(const char *host_name, int port)
{
      int control_socket;
      struct sockaddr_in server;
      struct hostent *host_info = gethostbyname(host_name);

      if( host_info )
      {
            if( client_socket_setup(port, &control_socket, &server, host_info ) )
            {
                  fprintf(stderr, "%s", strerror(errno));
                  return errno;
            }

            //printf("Connecting to server %s...\n", host_info->h_name);

            /* attempt to connect to server */
            if (connect(control_socket, (struct sockaddr *)&server, sizeof(server)) < 0)
            {
                  fprintf(stderr, "%s", strerror(errno));
                  return errno;
            }

            /* We are connected to the server. */
            printf("Connected to server %s\n", host_info->h_name);

            return request_processor();
      }
      else
      {
            printf("Unknown host: %s\n", host_name);
      }
}


ASKER CERTIFIED SOLUTION
Avatar of sunnycoder
sunnycoder
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Hi neilmcaliece (et al),

There's a 'standard Windows' API called URLDownloadToFile() that seems to be exactly what you want.

http://msdn.microsoft.com/library/default.asp?url=/workshop/networking/moniker/reference/functions/urldownloadtofile.asp

The API is available to most C++ implementations, VB, and Delphi.  But I'm unable to find any mention of it for use with C and/or linux.

Do you have Borland C++ Builder or Delphi on your linux system?

Kent
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Sorry for the FU to my own mail. The path to store the file is Windows at the moment
you have to replace it with a more sensible value....

I tested it here on
Windows XP and lcc-win32
Debian Linux gcc (3.3.x)

Regards
Friedrich
Don't re-invent the wheel. Use wget & popen:

#include <stdio.h>
#include <unistd.h>

char* url="http://www.somewhere.com";

main(){
     FILE           *pfp;
    char            s1[80];

    sprintf(s1,"wget -O - '%s'",url);

    if ((pfp = popen(s1, "r")) == NULL) {
        perror("popen: ");
        return;
    }
    fgets(s1, sizeof(s1), pfp);
    printf("First HTML line: %s\n",s1);
    pclose(pfp);
}

This will display the first line of the HTML output. wget also knows proxies, and more.