Downloading large files > 2GB

Hi,

Im using curl function ( using C ) to download files from remote machine which is accessable via https. I used fopen() to create files and given the FILE* pointer to fwrite(). The problem is im able to download files upto 2 GB. I think this is the limitaion of fopen() in 32 bit windows machines. Is there any way to download files larger than 2 GB. because my files are huge like 50-100 GB. nevermind about the speed and transfer rate...

The code snippet is attached how im using curl to download those files..


I need answers for the following questions.

1. Is there way to use curl to download large files > 2GB. i.e any curl option need to be set?
2.How to overcome the fopen() limitations in 32 bit machines or how to use fopen64() in 32 bit machines. As i also tried #define _FILE_OFFSET_BITS 64
3. Is there anyway to use the handle given by CreateFile() to fwrite() ?

Thanks in advance
curl_easy_setopt( curl, CURLOPT_URL, "https://myhostname/myfile" );
curl_easy_setopt(curl, CURLOPT_USERPWD, "login:pwd");

curl_easy_setopt(curl,CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_easy_setopt( curl, CURLOPT_FILE, FILE*);
curl_easy_setopt( curl, CURLOPT_HEADER ,0);
curl_easy_setopt( curl,CURLOPT_CONNECTTIMEOUT,60);

Open in new window

LVL 3
ilavaAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

evilrixSenior Software Engineer (Avast)Commented:
>> I think this is the limitaion of fopen() in 32 bit windows machines.
Correct. Standard  functions are limited to 2GB.
http://en.wikipedia.org/wiki/Large_file_support

You'll probably have to use system functions like CreateFile
http://msdn.microsoft.com/en-us/library/aa363858(VS.85).aspx 
evilrixSenior Software Engineer (Avast)Commented:
Some functions have been given Large File Support.

eg. http://msdn.microsoft.com/en-US/library/75yw9bf3(v=VS.80).aspx
josepvallsCommented:
Have you considered using an external application to handle the download?
I used to run into problems like this all the time and then started using wget (there is a windows port).
It supports, many protocols, compression, you can launch parallel downloads, resume them, etc.

http://gnuwin32.sourceforge.net/packages/wget.htm
Exploring ASP.NET Core: Fundamentals

Learn to build web apps and services, IoT apps, and mobile backends by covering the fundamentals of ASP.NET Core and  exploring the core foundations for app libraries.

itsmeandnobodyelseCommented:
>>>> You'll probably have to use system functions like CreateFile

If you consider that you also might think of using CopyFile which spares the open and close.

>>>> Have you considered using an external application to handle the download?

For a first approach you also could launch xcopy.exe (either with system or CreateProcess).


>>>> 3. Is there anyway to use the handle given by CreateFile() to fwrite() ?
No, the handle you get from CreateFile can be used for ReadFile/WriteFile/CloseHandle .


pkadianCommented:
Yes , Definitely the problem is because of 32 bit system , however there is an undocumented solution to resolve the issue on 32 bit system itself.
Please use the followings to resolve the issue
_fseeki64 & _ftelli64 to resolve the issue.

What you have to do is first import these 2 methods in your source file, like..
extern "C" int __cdecl _fseeki64(FILE *, __int64, int);
extern "C" __int64 __cdecl _ftelli64(FILE *);

Now you have to use these methods instead of fseek and ftell on all the places in your code. Make sure you should use now __in64  instead of Long variable while using these methods.
pkadianCommented:
You might need to something like following...


      _fseeki64(m_File, 0,SEEK_CUR)
      __int64 pos = _ftelli64(m_File);

      _fseeki64(m_File, CurrentPos,SEEK_SET)
      _ftelli64(m_File);
      ...
      fWrite(...)
      ...      
      _fseeki64(m_File,pos,SEEK_SET)
      _ftelli64(m_File);

Please read little bit about seeking and fTell to understand.
evilrixSenior Software Engineer (Avast)Commented:
>>  there is an undocumented solution to resolve the issue on 32 bit system itself.

Undocumented? http:#32143115
pkadianCommented:
Undocumented to say if these can work on 32 bit System also.
Solution to work with 32 bit system itself by importing them is undocumented.
evilrixSenior Software Engineer (Avast)Commented:
>> Undocumented to say if these can work on 32 bit System also.
But it is documented! The MSDN tells you what platforms are supported. The list includes 32 bit platforms.
itsmeandnobodyelseCommented:
Today I ported a logfile check program (which supports huge logfiles greater than 2 GB) from UNIX to Win32 and the fopen has no problems with the 4GB text file. I used stat64 and fpos64_t where the latter was defined as __int64 signed (!!!) integer on WIN32 platform.
evilrixSenior Software Engineer (Avast)Commented:
Askers reason for deletion is, "none of the answers helpful" yet there was never any response from the asker to the suggested solutions. I'd be interested to know why the asker feels none of the suggestions would assist.
evilrixSenior Software Engineer (Avast)Commented:
I have no specific recommendation. I think all the expert comments were, potentially, helpful and would have lead to a solution... if they didn't I'd like to know why (a) to see if we can offer other ideas and (b) to allow me to review what advice I did offer as, clearly, it was not as helpful as I'd hoped,

ilava,

It would be wonderful if you could give us some feedback. I have no objection to a delete if we really didn't solve your issue but please do us the courtesy of letting us know why.

Thanks.
ilavaAuthor Commented:
Hi,

Thanks for all the comments... here is how i resolved the problem..

created a file with createFile function. then i used a vcallback function to write contents with the help of the created handle. and also i used CURL_SETOPT_IGNORE_CONTENT_LENGTH option which resolves my problem easily.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C

From novice to tech pro — start learning today.