ftp protocol replacement

Hi s_federici,

curl.haxx.se/

Cheers!
Sunnycoder

ok, sunnycoder. It seems to me that curl is the answer to all the questions I post :-)
Well, how do I use it to perform HTTP transfer of local file (and viceversa?). I need clear examples.

Hi s_federici,

http://curl.haxx.se/libcurl/
ibcurl is a free and easy-to-use client-side URL transfer library, supporting FTP, FTPS, TFTP, HTTP, HTTPS, TELNET, DICT, FILE and LDAP.
...
ibcurl is free, thread-safe, IPv6 compatible, feature rich, well supported, fast, thoroughly documented...

Cheers!
Sunnycoder

Hi s_federici,

> Well, how do I use it to perform HTTP transfer of local file (and
> viceversa?). I need clear examples.
http://curl.haxx.se/libcurl/using/apps.html

Cheers!
Sunnycoder

Yes, I have red it. But where do I find useful examples to implement my http file transfer?

What OS are you targetting? Using Windows, a simple

URLDownloadToFile(NULL,"http://www.server.com/file.bin","C:\\temp\\file.bin",0,NULL);

should work also.

Ok, we are closer. I'm using windows. Could you please post a simple but complete C program that I can test to perform file transfer? I meas with all include and stuff at the right please. Otherwise I won't know for a long time if you suggestion will work for me. In reality I would like to use curl's tcl bindings, but C is ok.

Sure, e.g.

#include <windows.h>

#pragma comment(lib,"urlmon.lib")

int main () {

    URLDownloadToFile(NULL,"http://www.server.com/file.bin","C:\\temp\\file.bin",0,NULL);
}

That's it.

BTW, if you need progress information, see e.g. http://support.microsoft.com/default.aspx?scid=kb;en-us;234913 ("How To Provide Download/Upload Progress Information when Using WinInet"). This article demonstrates an alternative method for HTTP downloads also.

Ok, it worked very well. One last thing: how do I perform the upload operation? A similar example would be perfect!

http://www.linuxdevcenter.com/pub/a/linux/2005/05/05/libcurl.html
http://curl.haxx.se/libcurl/c/libcurl-tutorial.html
http://curl.haxx.se/libcurl/c/visual_studio.pdf
http://curl.haxx.se/libcurl/c/example.html

From the last link

#include <stdio.h>
#include <curl/curl.h>
 
int main(void)
{
  CURL *curl;
  CURLcode res;
 
  curl = curl_easy_init();
  if(curl) {
     curl_easy_setopt(curl, CURLOPT_URL, "curl.haxx.se");
     res = curl_easy_perform(curl);
 
      /* always cleanup */
      curl_easy_cleanup(curl);
    }
    return 0;
  }

Code to fetch a page

Sorry, I was too quick. I'm testing your code at home. I will be able to test it behind the office firewall of my customer. I'll let you know if it does work. Please, both of you, submit also a "put" example. Thanks.

'Upload' isn't that easy. Assuming that you're thinking of a HTTP POST, see http://support.microsoft.com/kb/177188/en-us ("SAMPLE: Using HttpSendRequestEx for Large POST Requests")

http://curl.haxx.se/libcurl/c/example.html

has code for all simple examples including get and put

To jrk: I would like to try your suggestion but I don't have any ASP ready server to use that will swollow the ASP script that is supposed to be able to receive the document via HTTP:

   Server.ScriptTimeout = 36000

   'Get size of POST data
   PostSize = Request.TotalBytes

   'Read POST data in 1K chunks
   BytesRead = 0      
   For i = 1 to (PostSize/1024)
     ReadSize=1024
     PostData = Request.BinaryRead(ReadSize)
     BytesRead = BytesRead + ReadSize      
   Next
   
   'Read remaining fraction of 1K
   ReadSize=TotalBytes - BytesRead
   If ReadSize <> 0 Then
     PostData = Request.BinaryRead(ReadSize)
     BytesRead = BytesRead + ReadSize
   End If

   ' Send results back to client
   Response.Write BytesRead
   Response.Write " bytes were read."

Can you help me to translate it to PHP? Or, at least, once the ASP script is run, how do I get the transferred document?

To sunnycoder: I would really like to test libcurl, but I wasn't able to find the libcurl.dll and libcurl.lib in the last binary distribution (7.15.3) of libcurl. So I tried compiling it with VC6 and I got 141 error messages. If you have those two files ready please let me know.

To sunnycoder: I realized that I had downloaded a wrong win32 version. I found the two missing files. I'm going to test it

Ok, the test proposed in the visual studio tutorial I downloaded from the libcurl site went well. It performs a get operation. I then tried the httpput.c example to perform a put (I also found a CGI script solving the problem of how to retrieve the uploaded document). Unfortunately the httpput.c example doesn't compile in my Visual C environment. I'm sure I'm doing something wrong, but I don't have any clue. The errors are the following:

C:\Project\libcurl\VisualStudio\MyApplication2\MyApplication.cpp(35) : error C2664: 'fread' : cannot convert parameter 4 from 'void *' to 'struct _iobuf *'
        Conversion from 'void*' to pointer to non-'void' requires an explicit cast
C:\Project\libcurl\VisualStudio\MyApplication2\MyApplication.cpp(60) : error C2065: 'open' : undeclared identifier
C:\Project\libcurl\VisualStudio\MyApplication2\MyApplication.cpp(62) : error C2065: 'close' : undeclared identifier

The whole code is pasted below

#include "stdafx.h"
#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>

#include <curl/curl.h>

/*
 * This example shows a HTTP PUT operation. PUTs a file given as a command
 * line argument to the URL also given on the command line.
 *
 * This example also uses its own read callback.
 *
 * Here's an article on how to setup a PUT handler for Apache:
 * http://www.apacheweek.com/features/put
 */

size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
{
  size_t retcode;

  /* in real-world cases, this would probably get this data differently
     as this fread() stuff is exactly what the library already would do
     by default internally */
  retcode = fread(ptr, size, nmemb, stream);

  fprintf(stderr, "*** We read %d bytes from file\n", retcode);

  return retcode;
}

int main(int argc, char **argv)
{
  CURL *curl;
  CURLcode res;
  FILE * hd_src ;
  int hd ;
  struct stat file_info;

  char *file;
  char *url;

  if(argc < 3)
    return 1;

  file= argv[1];
  url = argv[2];

  /* get the file size of the local file */
  hd = open(file, O_RDONLY) ;
  fstat(hd, &file_info);
  close(hd) ;

  /* get a FILE * of the same file, could also be made with
     fdopen() from the previous descriptor, but hey this is just
     an example! */
  hd_src = fopen(file, "rb");

  /* In windows, this will init the winsock stuff */
  curl_global_init(CURL_GLOBAL_ALL);

  /* get a curl handle */
  curl = curl_easy_init();
  if(curl) {
    /* we want to use our own read function */
    curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);

    /* enable uploading */
    curl_easy_setopt(curl, CURLOPT_UPLOAD, TRUE) ;

    /* HTTP PUT please */
    curl_easy_setopt(curl, CURLOPT_PUT, TRUE);

    /* specify target URL, and note that this URL should include a file
       name, not only a directory */
    curl_easy_setopt(curl,CURLOPT_URL, url);

    /* now specify which file to upload */
    curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);

    /* and give the size of the upload */
    curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, file_info.st_size);

    /* Now run off and do what you've been told! */
    res = curl_easy_perform(curl);

    /* always cleanup */
    curl_easy_cleanup(curl);
  }
  fclose(hd_src); /* close the local file */

  curl_global_cleanup();
  return 0;
}

C:\Project\libcurl\VisualStudio\MyApplication2\MyApplication.cpp(35) : error C2664: 'fread' : cannot convert parameter 4 from 'void *' to 'struct _iobuf *'
        Conversion from 'void*' to pointer to non-'void' requires an explicit cast

Explicitly cast stream parameter to a FILE *
size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream)
{
  size_t retcode;

  FILE * fptr = (FILE *) stream;
  retcode = fread(ptr, size, nmemb, fptr);
...
OR

retcode = fread(ptr, size, nmemb, (FILE *)stream);

C:\Project\libcurl\VisualStudio\MyApplication2\MyApplication.cpp(60) : error C2065: 'open' : undeclared identifier
C:\Project\libcurl\VisualStudio\MyApplication2\MyApplication.cpp(62) : error C2065: 'close' : undeclared identifier

This is strange ... these are standard library functions ... Seems like we missed out some header files ... Try adding
#include <sys/types.h>
If that does not work, then pls refer to help page for open and close for your platform ... that should list the header files to be included.

Cheers!
sunnycoder

Ok, I could replace the "offending" lines by adding an underscore in front of all problematic functions. But I still need a clear explanation of how to perform a complete FTP (with get and put). And directions for put given at http://www.apacheweek.com/features/put are a bit too complex for me (I usually use PHP on an apache server. So, in theory, they should give me the exact way to have it working).

Just a quick thought (maybe too late) : why not just run your FTP server on port 80 ? If HTTP is allowed, then port 80 should be open ...

I guess the problem is the client, not the server. My problem is moving some documents from my customer's PC to a web server (via FTP) and viceversa. Or I am wrong and you are right?

My apologies ... I wasn't aware that there was already an HTTP server running on the machine running the FTP server.

regarding file uploads through HTTP, using PHP, check this page :

http://be2.php.net/features.file-upload

It's the way I've done it a few times, and it's working fine :)

How can I attach such a procedure to my software?

That is PHP, so it will be accessible from a web page, and will be uploaded to the server. It won't pass through your application, nor do you need to write an application for this. All you have to do is write a PHP page following the information on that page, and that's it.

If however you need to attach it to your application, you can always either use libcurl (as has been mentioned earlier), or just do the (simple) HTTP POST request yourself.

I suggest first getting it to work without your application though ...

I guess that I would love to "just do the (simple) HTTP POST request myself" and having the site to store the data in the right place. How could I do that? Using libcurl? Do I need some PHP code to receive the post? If this is simple I just need a working example.

The page I gave has a working example. It consists of a HTML frontend (form) to submit the file, and a PHP page that will receive the file and store it somewhere.

libcurl would indeed be the easiest way to implement this in your application ... check the tutorials about that that were given earlier by someone else.

I already checked the tutorial(s). But they are or too far from what I need or too complex for me to understand. So I need a working example of how I can replace ftp commands in my application with something that will reach the same goal (having my documents moved to a server when ftp is blocked).

Sorry, the examples are clear, the problem is that you compiled them as C++ code whereas they are written in C.

Another point is that you are using Unix stuff unchanged, which usually won't work.
And this is the only place sunnycode get it slightly wrong open and close are not standard function as isn't fstat that's unix POSIX stuff.


Downloading is as easy as:
#include <stdio.h>
#include <curl/curl.h>



static int fetch_simple_page (CURL *curl){
      CURLcode res;
    FILE *my_file = fopen("LCC_README", "w");
      char simple_text_url [] = "ftp://ftp:anonymous@ftp.cs.virginia.edu/pub/lcc-win32/README";
      res = curl_easy_setopt(curl, CURLOPT_URL, simple_text_url);
    res = curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
    // res = curl_easy_setopt(curl, CURLOPT_VERBOSE);
      res = curl_easy_setopt(curl, CURLOPT_WRITEDATA, my_file);
      res = curl_easy_perform(curl);
    fclose(my_file);
    return 1;
}


int main(void)
{
  CURL *curl;
  int irval = 0;


  curl = curl_easy_init();
  if(curl) {
    irval = fetch_simple_page(curl);
      if (irval){
      puts("Got the page");
       }


    /* always cleanup */
    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
  return 0;
}

This works on Unices and Windows without any trouble.

So you have to change the things as sunnycoder has pointed out. And the tips with using libcurl are quite ok. Changing that to the http protocol is just changing the name of the URL so that is hardly "rocket" science.


Uploading is as simple you just have to modify the example to be runnable under Windows (and I would not compile it as C++).

so here we go with your troubles:
hd = open(file, O_RDONLY) ;
  fstat(hd, &file_info);
  close(hd) ;

This is not "Standard" C but Windows has some POSIX compatiblity libraries so it may be that _open _fstat and _close work.

However there is no counterpart to
struct stat file_info;  AFAIK.

so we have to replace it all with Windows functions:
The three lines above are just there to find out the size of the to be
transmitted file:
the windows counterpart is:
#include <stdio.h>
#include <windows.h>


int main(void){
      HANDLE fh;
      DWORD file_size;
      char * file_name = "file_size_windows.c";

      fh = CreateFile (file_name,
                        GENERIC_READ,
                              FILE_SHARE_READ,
                              NULL,
                              OPEN_EXISTING,
                              0, NULL);


      if (NULL != fh) {
            /* we can get the size of the file */
            file_size = GetFileSize(fh, NULL);
      }
      CloseHandle(fh);
      printf("The file %s is %d bytes large\n", file_name, file_size);

      return 0;
}

So you have to replace the lines with the here given code ane use file_size here:
url_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, file_info.st_size);


Now if you compile the code with a C++ compiler you have to see that the proper casts are in place.

So the code will work on Windows that way.

Regards
Friedrich

That was a long cooment. I'm going to test it. In the meantime, thanks :-)

Finally I tested fridom's suggestion. But it didn't work. The C file compiled fine (as it did after sunnycode's suggestion), but when I run the example I keep getting the usual error message:

Request Entity Too Large
The requested resource /branches/robertofederici/pages/sections/section_2_2_upload.txt does not allow request data with PUT requests, or the amount of data provided in the request exceeds the capacity limit.

So I guess I need a ready-to-run cgi or php script to put on my server that will be able to get my posted data and save them in the final location. But I need help on how to do that. Otherwise my FTP replacement will not work.

The script can't be the problem, if you can do an ftp upload you can do it with libcurl also.
The question is what kind of access do you have?

Regards
Friedrich

So, it means that I have to put username and password somewhere, I guess. How do I specify them with libcurl?

put it in the URL string like this
ftp://username:password@remote_host/path_to_the_file

Regards
Friedrich

This method perfectly works at my place (as normal ftp does work). But it doesn't work at my customer's office as their networks is set so that FTP is NOT allowed. I resume my request:

1. I need to transfer a document from a PC to a server (that runs an http server and an ftp server) running Apache and PHP (and allows for CGIs)
2. the client PC does not allow using FTP

The GET part of ftp has been solved in several different ways, but to have an FTP replacement I do need also to be able to perform the POST operation. Thanks.

You did not tell us if http uploads are allowed. If yes then you just have to replace the URL with it's http counterpart. And now I'm not in the mood to spend extra time on it.
If you do not understand that libcurl is just a programmatic tool for your job, no-one can help you with the libcurl comes with examples and I bet a post is part of it also.

Over and out
Friedrich

From my request:

"On the other hand he is perfectly able to do HTTP file transfer"

And from one of my previous comments:

"Finally I tested fridom's suggestion. But it didn't work. The C file compiled fine (as it did after sunnycode's suggestion), but when I run the example I keep getting the usual error message:

Request Entity Too Large
The requested resource /branches/robertofederici/pages/sections/section_2_2_upload.txt does not allow request data with PUT requests, or the amount of data provided in the request exceeds the capacity limit.

So I guess I need a ready-to-run cgi or php script to put on my server that will be able to get my posted data and save them in the final location. But I need help on how to do that. Otherwise my FTP replacement will not work."

That is: replacing the URL with it's http counterpart doesn't work.

This said, I'm going to test the last fridom's suggestion.

I went to http://curl.haxx.se/libcurl/c/curl_formadd.html. And it is another very complex page. If using libcurl is simple, please just post a clear example in whatever language/library that I can easily install of how to have whatever document moved from my PC (that doesn't allow for FTP) to my web Apache/PHP server. That is what I need. If I would be able to understand the content of the links you proposed I wouldn't ask to EE.

Thanks for any help you can give me to solve my problem

This does not answer the question is it possible to upload data? And in which way is it possible. The error message are with PUT I can not see what PUT an POST have in common. So you did not provide the information that is needed to help you better as we did.

So it's up to you to let us know what works and how you can send message to you server and at least show what you have tried to get that data to the server. So either you can get the data to your server then you can get them there with libcurl also. If it's not the final place it as simple as loging to the remote machine (hopefully a Unix) and have a crontab which moves the file in the final location.

Of use e.g expect to steer that remote computer from your local machine.
If you need a f... PHP script for moving the file. Here's the code for that without any error checking:
(initial page)

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE html
     PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">


<html>
 <head>
  <title>Moving file with PHPs</title>
 </head>
 
 <body>
  <form action="move-it.php" method="post">
   <p>The data please</p>
   <table>
    <tr>
     <td>File to move</td>
     <td> <input name="file-to-move" type="text" size="30" />
            </td>
    </tr>
    <tr>
     <td>Move to where?</td>
     <td>
              <input name="move-to" type="text" size="30" />
            </td>
    </tr>
   </table>
         <input type="submit" value="Move it">
 
  </form>
   
   
</body>
</html>

work is done in this php file:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <title>Data submitted</title>
  </head>

  <body>

<?php
$file_name = $_POST["file-to-move"];
$move_to = $_POST["move-to"];

# <!-- use move_uploaded_file if the upload was done with  a
# Post request -->
  rename($file_name, $move_to);

echo "<p> Please check t1 for the moved file for the written values </p>"
?>


  </body>
</html>

I do warn you this code works directly on the file system it would be really dumb to use this code unchanged. But I'm not in the mood to work on something which seems to be an endless story.


How should we know what works and what doesn't?
You do not post the used code and you do not tell us what you have done in the past to get the pages to that server.

Friedrich

Friedriech, your way of stating things is really disturbing to me. If you are "not in the mood to spend extra time on it", please, allow other people to help me with my problem if you don't know how to solve it. If you need further explanations about what my problem is, please do start by reading my question. After all it is plain english. Thanks.

Now a few comments to your last comment:

1. I do not need to move staff from one location of my server to another location. I do need to transfer a file from my customer's machine to a web server.
2. The web server also supports FTP, but my customer firewall doesn't.
3. My customer's firewall does allow him to POST data via HTTP.
4. the used code is completely listed in my 8th comment.
5. The data posted via the code listed in my 8th comment, does reach my server but, if I'm correct, it seems that a cgi (or similar) is needed so that the server can put it in the right location
6. (I guess) I need clear and detailed instructions on how to set up my web server (Apache/PHP/GCI) so that the POSTED data will be stored in the final correct location.

Thanks in advance for any help

I've already posted this page :

http://be2.php.net/features.file-upload

but here it is again ... I'll be more specific this time ... example 38-2 shows you what PHP code you need to receive a POST containing a file, and put that file in a certain location. It's a basic example, but once you get that to work, you can add all functionality you want.

Example 38-1 is an HTML form you can use to test the above code ... ie. you select a file from your hard disk, and click submit. That file will then be POSTed to the above script.

It's really disturbing to me that you do not give answers on questions. We gave you the code for uploading a file, I posted the PHP code needed to rename the file on the server. So what's the point.

So go figuring.
- how the post of you file works
- where this stuff is saved
- set up a crontab job or a page to move the file to the folder.

We can not know that, and you do not tells us.

Friedrich

Well I think the page from infinity even shows how to upload this file programmatically:
Ist Beispiel 38-1. And this form has to be build with libcurl for that one can use the formadd sutff. But it seems we are supposed to do that.

Friedrich

Friederich, your statement "We gave you the code for uploading a file" is wrong. As I clearly replied in my comments, the code to perform upload DOESN'T work on my customer's PC as (I'm a bit tired of saying it again and again) he CANNOT use ftp and, when using the http, the code DOESN'T work as it is. I think it needs a cgi or similar on the server to receive the file, but this is what I'm asking, I'm not supposed to guess about the info I ask. So, PLEASE, allow other people to answer my question and next time, before answering, DO READ the question. Even simply apologizing if you didn't understand the question at start it is not a crime. Thanks.

I had to do a lot of work myself to devise how to properly set everything. But in the end it worked. Sorry if I didn't award very general suggestions about libcurl. Thanks to all.

Just writing things do not work is not helpful. And I don't have chrystal balls in my tool box. So we all pointed out what might work. Now you do not tell us in any of your messages how "upload" works. I wrote that if upload is working it can be automated with libcurl. So I was fully up to the point. It can't be my task to figuring out why things might not work for you. If you feel bad served you can simply ask for a refund of your points.

Friedrich

Well. In reality... it didn't work.

It worked on my PC (that has no blocks of any sort) but the compiled exe with libcurl and formadd is unable to make the file transfer from my customer's PC to the server. I cannot understand why.

My customer cannot do FTP, but he is perfectly able to use HTML forms to send files (i.e. attachments via webmail). I thought that http post via libcurl could be used as an exact replacement of HTML forms. But, apparently, I was wrong.

Who would like to give it a try can download my experimental exe from http://www.sitibs.com/files/stefano/260606.zip.

It will move the included postit2.c file to http://www.sitibs.com/branches/robertofederici/pages/sections/postit2.c.

Before running the exe, you can remove the postit2.c file from

http://www.sitibs.com/branches/robertofederici/pages/sections/ 

by running

http://www.sitibs.com/branches/robertofederici/pages/sections/remove.php