libcurl - taking long time on curl_easy_perform

Hi Experts,

I've got some standard libcurl code that uploads a file to a server with a post.  The file gets uploaded correctly, however, the curl_easy_perform function blocks for about 30 seconds - meanwhile, the file is already sitting on the server, entirely uploaded.

I thought I might be setting the wrong content length of the file I'm uploading, but the file on the server (that has the correct hash after re-downloading it), has exactly the number of bytes that I uploaded.

Any clue how I can get this function to un-block faster?

Thanks,
Mike

[snip]
        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
        curl_easy_setopt(curl, CURLOPT_USERAGENT, USERAGENT);
        curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
        curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&chunk);
	curl_easy_setopt(curl, CURLOPT_READFUNCTION, simple_read_callback);
	res = curl_easy_perform(curl);

Open in new window

LVL 1
threadyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

sarabandeCommented:
do you know whether the read callback was called after upload? if yes, you may check why it was called and whether you could signal end of upload by this.

if that doesn't help but you know in write callback when the last tranche was written, you could try to invoke a deferred thread that cancels the current  curl.

Sara
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
threadyAuthor Commented:
Yes the read callback was called numerous times to completion- the entire file quickly gets uploaded.

I'm pretty sure the problem is with my content length- the length I'm specifying is the length of the file- not of the entire post.  It should be of the entire post (including headers I believe).

How does one properly calculate the content length with libcurl?

Thanks!
Mike
0
sarabandeCommented:
i was mixing-up read and write. sorry.

for an upload the read callback was called. it has arguments 'size' and 'nitems' where normally size was 1 and nitems the number of bytes you could pass with that call. generally, you should pass size*nitems bytes as long as the end of upload was not reached. if you have less than size*nitems bytes left, you have to pass the rest. in any case you would return the amount of bytes you copied into the read buffer. if the read callback was called again after you already have passed all the data you would return 0 to signal end of upload.

i read in the docs:

If you set this callback pointer to NULL, or don't set it at all, the default internal read function will be used. It is doing an fread() on the FILE * userdata set with CURLOPT_READDATA.

did you try that?

How does one properly calculate the content length with libcurl?
i don't see where you could specifiy the total length of the upload before being called back? didn't you have to manage it in the read callback solely?

Sara
0
Cloud Class® Course: Microsoft Exchange Server

The MCTS: Microsoft Exchange Server 2010 certification validates your skills in supporting the maintenance and administration of the Exchange servers in an enterprise environment. Learn everything you need to know with this course.

threadyAuthor Commented:
I am not using the CURLOPT_READDATA - I need the callback because I'm decrypting the file as I'm uploading it.  

If I have less than the number of bytes (num * size), I do give less.  The entire file is uploaded every time.  The only issue I have is with the pause in my call to curl_easy_perform.  This function triggers the upload and sends the entire file through my callback.  The call blocks every time for about a minute though, probably because the content length is incorrect.

I should mention that I'm doing a form upload, and that I have set the following as well, beforehand:
    curl_formadd(
                 &formpost, &lastptr,
                        CURLFORM_COPYNAME, "file",
                        CURLFORM_FILENAME, strDestinationFileName,
                        CURLFORM_STREAM, &transferItem,
                        CURLFORM_CONTENTSLENGTH, lFileSize,
                        CURLFORM_CONTENTTYPE, GetContentTypeForMultipartUpload(strDestinationFileName),
                        CURLFORM_END);

That CURLFORM_CONTENTSLENGTH is of the filesize - which is probably less than what I should be sending.  Maybe this is a red herring.

Thanks for your help!
Mike
0
sarabandeCommented:
curl_easy_setopt(curl, CURLOPT_READFUNCTION, simple_read_callback);
can you post the code of simple_read_callback?

i would assume this is the pulprit since curl_easy_perform would no longer care for the length given with CURLFORM_CONTENTSLENGTH if the read callback would return 0. if i am right you would need to replace it by your own function. with the current settings it could be that the read function tries to get more data though the form entirely was passed.

That CURLFORM_CONTENTSLENGTH is of the filesize - which is probably less than what I should be sending.

that depends on whether the encrypted data have same length as the decrypted data or not. on windows os, file size is not a reliable value if the files are text files since each pair of CRLF will be turned to a single linefeed. then the number of bytes read from file is less than the file size. which function does the decrypting? is that also a callback function?

Sara
0
threadyAuthor Commented:
There's 1 line of code in the callback- it's just a call to our ReadFile (which decrypts), and gives back the length requested.  I can't post that code though.. Not allowed, sorry.

The length I'm getting is of the decrypted data - the filesize indeed would not work.  The value is from a database and is correct.  Good guesses though as usual Sara!

I'll have to verify if my last read is indeed zero size.  I will get back as soon as I can- currently finishing up something else.

Thanks again,
Mike
0
sarabandeCommented:
it's just a call to our ReadFile (which decrypts), and gives back the length requested.  
it shouldn't give back the length requested but the length ReadFile provides.

the point is that if ReadFile don't has data anymore the callback should return 0 and so signal end.

if that alone doesn't prevent from hanging you may check whether the ReadFile is blocking somehow. ifReadFile is correct but curl_easy_perform still is waiting, you may try to abort the transaction by returning CURL_READFUNC_ABORT from read callback.

note, you also may check your write callback whether it gets called. perhaps the curl wants to perform some kind of ackknowledge at end and the hanging is because the write callback is not answering.

Sara
0
threadyAuthor Commented:
Of course you're right, I give back the length the readfile provides - I just worded that wrong.

There is in fact a zero returned.  And now I'm really stumped.
-  The curl_easy_perform takes about 35 seconds to upload a 200 MB file (to localhost server).
-  The simple_read_callback then returns a 0 properly
-  Then there is about 1-2 minute wait time for the curl_easy_perform to unblock
-  The WriteMemoryCallback is indeed called , after that long wait, with the result of the Post (the response from the server).

It could also be something waiting in the server, but I cannot check that easily.

I'd be weary of returning CURL_READFUNC_ABORT - it's already returning a zero and not being called back after that point...
0
threadyAuthor Commented:
Other times I timed this, the 35 seconds was just about 2-4 seconds....  and the wait time was less than 30-40 seconds...  So it's not constant...  But the wait is always there.
1
sarabandeCommented:
you could try to call

     
curl_easy_cleanup(curl);

Open in new window


in own timeout handler - say 1 second after read callback has returned 0.

however, the curl may crash after unblocking if you do so.

the only thing left seems to be to move to curl 'multi' which allows asynchronous transfer without blocking calls.

Sara
0
threadyAuthor Commented:
Thank you for your continued awesome help Sara!  I'm thinking the problem is with our particular server code.  I noticed the same issue from a browser (it's a proprietary implementation, nothing I can look into further for now).

Best,
Mike
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.