Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


single file size limitation?

Posted on 2002-06-24
Medium Priority
Last Modified: 2012-06-21
Hello All...
   Is anyone aware of a single file size limitation for opening/reading a file in a Delphi app? I'm testing my app on Win 2000 and XP and it works correctly on reading and processing large files of up to around 4 gigs in size. Over and above that, it gives incorrect and wacky results (the results consistemtly show as a much smaller number than expected)... like some internal counter has been exceeded.

  I seem to be able to correctly process several files whose TOTAL exceeds 4 gigs, no problem... as long as each of those files are less than 4 gigs. But as soon as any one single file exceeds 4 gigs, something gets blown. Don't know whether to think this is an operating system limitation, or a Delphi limitation.

I'm using D3 Professional.

Question by:aztec
  • 14
  • 8
  • 6
LVL 11

Expert Comment

ID: 7106576
This is not a Delphi problem at all. The size limit of 4 Gig (2 Gig for FAT file systems) stems from the simple fact that the size is stored in a DWORD. It is a limit of many file systems.

Author Comment

ID: 7109225
I see... but my app can create OUTPUT files of sizes greater than 4 gigs...

How can this be?
LVL 11

Expert Comment

ID: 7109620
Then it simply may be that your program eats up 1000 % CPU. Show your code.
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Author Comment

ID: 7109628
1000% ??

you mean 100%? How could using 100% of CPU result in a limitation of the file sizes I can read in?
LVL 11

Expert Comment

ID: 7109644
Sorry, posted to the wrong question.

NTFS does not have the size limit, but as long as you use the file functions with DWORD for position or size parameters you will be lost in your file. Sequential reading and writing works, but repositioning should be modulo 4 Gigs.

Author Comment

ID: 7109673
ok, no problem!

Well, I am using 'BlockRead' to read in the data file. Maybe this could be the problem? If I switch to using maybe TFileStream instead it might solve this?
  Below is my code when I read in the file:

var buf : array[0..49151] of char;
    fromfile : file;

   blockread(fromfile, buf, 1, numread);
   recsread:=recsread + 1;
   for z:=0 to 49151 do
     if ((ord(buf[z])>= 0) and (ord(buf[z]) <= 31)) or
        ((ord(buf[z])>= 127) and (ord(buf[z]) <= 255))  then
        if bigstring <> '' then find_valid;
       bigstring:=concat(bigstring, buf[z]);
       if length(bigstring) = 255 then find_valid;

   bytes_processed:=bytes_processed + 49152;
   if recsread mod 10 = 0 then
     hperc:=calc_status(totsizeoffile, bytes_processed);

 until EOF(fromfile);

LVL 11

Expert Comment

ID: 7109758
You should revert to Win32 API. The function family CreateFile, ReadFiel, WriteFile also contains functions for file size and seeking (GetFileSizeEx and SetFilePointerEx) which use Int64 to overcome the DWORD limit.

Author Comment

ID: 7109774
So TFileStream wouldn't help?

OK, how do I use these Win32 API file commands? Would you have some samples?

Are they faster/more efficient for reading/writing than the regular Read/Write statements for text files?

Author Comment

ID: 7109776
I see nothing for these commands in the Delphi 3 Help file...
LVL 11

Expert Comment

ID: 7109886
Get the Platform SDK from Microsoft. It contains all the help you need. The Delphi help is outdated because they have to license it from MS.

CreateFile ReadFile etc are the core functions for files. All Delphi functions have to base on them.

Author Comment

ID: 7109958
can I download this somewhere? Is it free?
LVL 11

Expert Comment

ID: 7110014
Yes, go to www.microsoft.com and search for "Platform SDK".

Accepted Solution

Cynna earned 200 total points
ID: 7110539

This free component (with source) enables you
to work with 64-bit file sizes:



Author Comment

ID: 7112294
Robert - does the "Readfile" from the Platform SDK let you do like a "Readln"? Where it will read in one full line (ended by a CR/LF) at a time (without having to specify byte-count)? I am reading in text-files like this and this is essential for me.

Cynna - thanks for the component suggestion. But when I try to install it, it gives a compile error in the "GPHugeF" file - File Not Found "SysConst.dcu". I am using Delphi 3 Pro. Maybe I don't have this file?
  Also, will this component address the issues I mention aboe to Robert?


Author Comment

ID: 7112376
Also Robert - there is like 7 different components of the Platform SDK... do I need them all, or is there only a specific one I need to install?
LVL 11

Expert Comment

ID: 7112708
It is always wise to have all possible help available.
ReadFile does not allow to read lines.

Author Comment

ID: 7112740
So can I do this then? How can I read text files > 4 gigs in a line by line fashion?
LVL 11

Expert Comment

ID: 7112872
1. Rethink your strategy. >4 Gigs line by line will take ages. Having unstructured textual information of that size indicates poor design.
2. Write a simple class with a big buffer where you pick the lines from. That is basic programming technique.

Author Comment

ID: 7113138
(1) But this is the structure of my data. Large files consisting of one email address on each line - which I must read in and process one at a time. I cannot change this.

(2) How do I do this and just pick out 1 address/line at a time (ie. up to the first CR/LF) ?

Expert Comment

ID: 7113202

> File Not Found "SysConst.dcu"

Well, D3 obviously doesn't have this unit. Try this:

1. Copy this to NotePad:

unit SysConst;


  SUnknown = '<unknown>';
  SInvalidInteger = '''%s'' is not a valid integer value';
  SInvalidFloat = '''%s'' is not a valid floating point value';
  SInvalidDate = '''%s'' is not a valid date';
  SInvalidTime = '''%s'' is not a valid time';
  SInvalidDateTime = '''%s'' is not a valid date and time';
  STimeEncodeError = 'Invalid argument to time encode';
  SDateEncodeError = 'Invalid argument to date encode';
  SOutOfMemory = 'Out of memory';
  SInOutError = 'I/O error %d';
  SFileNotFound = 'File not found';
  SInvalidFilename = 'Invalid filename';
  STooManyOpenFiles = 'Too many open files';
  SAccessDenied = 'File access denied';
  SEndOfFile = 'Read beyond end of file';
  SDiskFull = 'Disk full';
  SInvalidInput = 'Invalid numeric input';
  SDivByZero = 'Division by zero';
  SRangeError = 'Range check error';
  SIntOverflow = 'Integer overflow';
  SInvalidOp = 'Invalid floating point operation';
  SZeroDivide = 'Floating point division by zero';
  SOverflow = 'Floating point overflow';
  SUnderflow = 'Floating point underflow';
  SInvalidPointer = 'Invalid pointer operation';
  SInvalidCast = 'Invalid class typecast';
  SAccessViolation = 'Access violation at address %p. %s of address %p';
  SStackOverflow = 'Stack overflow';
  SControlC = 'Control-C hit';
  SPrivilege = 'Privileged instruction';
  SOperationAborted = 'Operation aborted';
  SException = 'Exception %s in module %s at %p.'#$0A'%s%s';
  SExceptTitle = 'Application Error';
  SInvalidFormat = 'Format ''%s'' invalid or incompatible with argument';
  SArgumentMissing = 'No argument for format ''%s''';
  SInvalidVarCast = 'Invalid variant type conversion';
  SInvalidVarOp = 'Invalid variant operation';
  SDispatchError = 'Variant method calls not supported';
  SReadAccess = 'Read';
  SWriteAccess = 'Write';
  SResultTooLong = 'Format result longer than 4096 characters';
  SFormatTooLong = 'Format string too long';
  SVarArrayCreate = 'Error creating variant array';
  SVarNotArray = 'Variant is not an array';
  SVarArrayBounds = 'Variant array index out of bounds';
  SExternalException = 'External exception %x';
  SAssertionFailed = 'Assertion failed';
  SIntfCastError = 'Interface not supported';
  SSafecallException = 'Exception in safecall method';
  SAssertError = '%s (%s, line %d)';
  SAbstractError = 'Abstract Error';
  SModuleAccessViolation = 'Access violation at address %p in module ''%s''. %s of address %p';
  SCannotReadPackageInfo = 'Cannot access package information for package ''%s''';
  sErrorLoadingPackage = 'Can''t load package %s.'#13#10'%s';
  SInvalidPackageFile = 'Invalid package file ''%s''';
  SInvalidPackageHandle = 'Invalid package handle';
  SDuplicatePackageUnit = 'Cannot load package ''%s.''  It contains unit ''%s,''' +
    ';which is also contained in package ''%s''';
  SWin32Error = 'Win32 Error.  Code: %d.'#10'%s';
  SUnkWin32Error = 'A Win32 API function failed';
  SNL = 'Application is not licensed to use this feature';

  SShortMonthNameJan = 'Jan';
  SShortMonthNameFeb = 'Feb';
  SShortMonthNameMar = 'Mar';
  SShortMonthNameApr = 'Apr';
  SShortMonthNameMay = 'May';
  SShortMonthNameJun = 'Jun';
  SShortMonthNameJul = 'Jul';
  SShortMonthNameAug = 'Aug';
  SShortMonthNameSep = 'Sep';
  SShortMonthNameOct = 'Oct';
  SShortMonthNameNov = 'Nov';
  SShortMonthNameDec = 'Dec';

  SLongMonthNameJan = 'January';
  SLongMonthNameFeb = 'February';
  SLongMonthNameMar = 'March';
  SLongMonthNameApr = 'April';
  SLongMonthNameMay = 'May';
  SLongMonthNameJun = 'June';
  SLongMonthNameJul = 'July';
  SLongMonthNameAug = 'August';
  SLongMonthNameSep = 'September';
  SLongMonthNameOct = 'October';
  SLongMonthNameNov = 'November';
  SLongMonthNameDec = 'December';

  SShortDayNameSun = 'Sun';
  SShortDayNameMon = 'Mon';
  SShortDayNameTue = 'Tue';
  SShortDayNameWed = 'Wed';
  SShortDayNameThu = 'Thu';
  SShortDayNameFri = 'Fri';
  SShortDayNameSat = 'Sat';

  SLongDayNameSun = 'Sunday';
  SLongDayNameMon = 'Monday';
  SLongDayNameTue = 'Tuesday';
  SLongDayNameWed = 'Wednesday';
  SLongDayNameThu = 'Thursday';
  SLongDayNameFri = 'Friday';
  SLongDayNameSat = 'Saturday';



2. Save that text as 'sysconst.pas' in the same folder
   you extracted GpHugeF.pas

3. Now it should compile OK.

>Also, will this component address the issues I mention aboe to Robert?

Yes it will.

> (2) How do I do this and just pick out 1 address/line at a time (ie. up to the first CR/LF) ?

Use BlockRead method of TGpHugeFile to read file chunk by chunk, and search for CR/LF inside every chunk.

Author Comment

ID: 7113224
ok Cynna, thanks.

For fastest file reading, what do you recommend as an efficient 'chunk-size'? 2 megs? 4 megs?


Expert Comment

ID: 7114067

Well generally, the more - the better. It depends on
your target system average expected free mem.
For example, I'd use 8Mb as a start point on my system...

Anyway, you could very easily change several values and
measure how it impacts your performance, so you could
make best judgement for yourself.

Author Comment

ID: 7116814
OK Cynna, I did what you suggested for the sysconst.pas, now when installing the HugeFile component, I get a new error on this linein the GPHugeF.pas file:

  start : int64

"Error: Undeclared identifier: 'int64'"

Any suggestions? I am using Delphi 3 Pro.


Expert Comment

ID: 7117326

D3 - well that's a kind of a problem...
Int64 wasn't introduced until D4, I think.

You might try replacing Int64 with Comp. But the
problem might be that Comp is float, while Int64 is
(surprise, surprise...) integer. This can create
problems if GPHugeF.pas uses operations like div
(specific to integers) on Int64 types. You might
try tweeking this code a bit, if you get any other

Other then that I'm afraid I can't think of anything
simple, considering your platform. Additional problem
is I have D5, and can't really test D3 specific code....

If you get stuck again, post it, and I'll try helping you
a bit later.

Author Comment

ID: 7117468
Thanks for the suggestion Cynna. I made the change of int64 to comp (for 2 variables - start and stop), but it gave a new compile error for this line:

stop := stop + $100000000;

"Integer expression too large"

Plus there were other compile errors everywhere "Win32Check" was mentioned. :-(

Am I stuck? Is it possible to email directly the creator of this component?

Expert Comment

ID: 7121602
Well, aztec, you're pretty close to beeing stuck, as far as I can see...

But, try one more final idea:

procedure TForm1.Button1Click(Sender: TObject); // read file
const  cBufSize = 2048;
var Buffer   : Array [0..cBufSize] of Char;
    hRead    : THandle;
    FileName : String;
    read     : DWORD;
   // Ensure our Buffer is always zero-terminated:
   // Place your full file name here:
   hRead := CreateFile( PChar(FileName),
                        FILE_SHARE_READ or FILE_SHARE_WRITE,
                        0 );
   while ReadFile(hRead, Buffer, cBufSize, read, nil) do begin
         // ----------- Process Buffer: ------------------
         // .... your block-parsing code here ...
         // ----------------------------------------------
         Memo1.Text:=StrPas(Buffer); // just a demo
         Application.ProcessMessages; Sleep(1);
         if read<cBufSize then Break;
   // Final block processing:
   // ----------- Process Buffer: ------------------
   // .... your block-parsing code here ...
   // ----------------------------------------------
   Memo1.Text:=StrPas(Buffer); // just a demo

This code uses WinAPI native file handling operations.
It relies on sequential read, so it doesn't touch
your D3 limitation.
I used it just as a demo. You should use your large
file name in var FileName.
If you see the end of your large file in Memo1, when it
finishes, it works OK. If it works OK, write your block
parsing code instead of Memo1.Text:=... part and you're done.
If it doesn't, I'm afraid I'm all out of quick-ideas for this one...

Good luck!


Author Comment

ID: 7122489
Thanks Cynna - I just went ahead and upgraded to the D6 Personal Edition ...everything works fine now!

(thanks for the final D3 suggestion however!)


Expert Comment

ID: 7123280
> ..I just went ahead and upgraded to the D6 Personal Edition...

Smart move.

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A lot of questions regard threads in Delphi.   One of the more specific questions is how to show progress of the thread.   Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to the…
Hello everybody This Article will show you how to validate number with TEdit control, What's the TEdit control? TEdit is a standard Windows edit control on a form, it allows to user to write, read and copy/paste single line of text. Usua…
this video summaries big data hadoop online training demo (http://onlineitguru.com/big-data-hadoop-online-training-placement.html) , and covers basics in big data hadoop .
Whether it be Exchange Server Crash Issues, Dirty Shutdown Errors or Failed to mount error, Stellar Phoenix Mailbox Exchange Recovery has always got your back. With the help of its easy to understand user interface and 3 simple steps recovery proced…
Suggested Courses
Course of the Month15 days, 20 hours left to enroll

581 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question