[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 945
  • Last Modified:

does BlockRead have limitations?

Does blockread have limitations or am i doing something wrong?

Here is my test code:

   == sample of variables used==
F: File;
type
Trecordstructure=packed record  
{...record}
end;
z:integer;
Ttest:Trecordstructure;

=== actual test code begins ===

AssignFile(F, 'filename.dat');
Rewrite(F, sizeOf(Ttest));
CloseFile(F);

   == so far, created the file - no problems

AssignFile(F, 'filename.dat');
reset(F, sizeOf(Ttest));

     === fill file with 2,600,000 copies of the record as a test ===
z:=0;
repeat
    blockwrite(f, Ttest, 1);
until z>= 2600000;
closefile(f);

   == so far all seems to be working ==

AssignFile(F, 'filename.dat');
reset(F, sizeOf(Ttest));

    === This is where the problem is (note i am seeking to 10 less that the amount put into the file ===
seek(f, 2599990);
blockread(f, Ttest, 1);
closefile(f);
{code to show record content}


I keep getting error 131 (An attempt was made to move the file pointer before the beginning of the file. )

Does Seek or Blockread have a limitation?  I checked the file created and divided it by the size of Ttest and it works out to be 2600000 - so what is going on?










0
winuser2000
Asked:
winuser2000
  • 8
  • 3
  • 2
  • +5
4 Solutions
 
winuser2000Author Commented:
Strange - just did some more testing and it seems seek gives up at around the 2Gb file mark yet blockwrite seems to be fine.

Anyone got a way around this?
0
 
BlackTigerXCommented:
what version of Windows are you using?

that could be more of a Windows (the version you have) limitation

and most important, what file system are you using? (FAT, FAT32, NTFS, etc)
0
 
Wim ten BrinkCommented:
In general, Delphi uses the Integer type for file sizes. This is nice for files less than 2 GB (The maximum value of an integer) but flawed when files are bigger. It's a flaw in a big part of the Delphi VCL and actually, Windows itself.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
winuser2000Author Commented:
Windows 2000, FAT 32, Delphi 4 Pro

But, then how come Blockwrite can write to the maximum file size allowed on a FAT32 system (4Gb) yet seek cannot move to it?
0
 
BlackTigerXCommented:
you can try doing 2 Seek calls...
0
 
winuser2000Author Commented:
tried that, didn't work.

Still find it odd that blockwrite will happily write files till it hits the 4Gb file size but seek is unable to move above the 2Gb div sizeof(record).

Do you know if blockread is also affected or is it down to seek needing to be "fixed"
0
 
vadim_tiCommented:
you can work with TFileStream.

its Seek method
function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64;

0
 
bpanaCommented:
check out GpHugeFile (http://swiss.torry.net/filedrvother.htm)

GpHugeFile / TGpHugeFileStream v.3.07a
By Primoz Gabrijelcic. Encapsulation of Windows file-handling routines that allows work with >2GB files. Included is support for non-buffered access (FILE_FLAG_NO_BUFFERING) and buffering for sequentially accessed files. TGpHugeFile is interface to 64-bit file functions with some added functionality. TGpHugeFileStream is descendant of TStream that wraps TGpHugeFile.

Fully functional
Source: Included
Download: D5 D6
0
 
winuser2000Author Commented:
vadim_ti  - the seek method is Int64 in Delphi 4 pro too?

thanks bpana, i'll download that and give it a look over.
0
 
Wim ten BrinkCommented:
> But, then how come Blockwrite can write to the maximum file size allowed on a FAT32 system (4Gb) yet seek cannot move to it?

The Integer type is 32-bits and thus allows 4 GB of values. An unsigned integer would therefore have the maximum value of 4GB. However, integers in Delphi always have a sign, changing their range from 0..4 GB to -2GB..+2GB. Beyond the 2 GB the datatype has an overflow...

Another problem within Delphi is that these integer values are inherited from Delphi's past history. The compiler needs to be backwards compatible so there could not be too many changes to the core of the runtime. Using INT64 for filesizes would solve quite a few of these problems but it would also drastically change the memory structures of the core file system structures. (They would become bigger.)

Then again, not many people do create files over 2 GB in size. Those who do, often use a database instead. And even in a database it would be a huge amount of diskspace. We're talking about two million megabytes here. And a megabyte is already a million bytes. Fortunately, the 64-bits filesystem allows Windows to create even bigger files.

About Delphi being able to write and not to read from big files, well simple... To write to a file, Delphi can just call the Windows API and stuff gets written. To read data, Delphi is probably looking at the filesize or current position in the file and this value will have an overflow. Instead of being e.g. somewhere around the 3 GB it's suddenly minus 1 GB. An invalid value, thus crash...

This limitation just became clear when Delphi 3 and 4 got on the market. And back then not many people were too worried about it because files that huge were considered absurd. Many users didn't even have a harddisk that big anyway. But in a few years, lots of things can change. Suddenly computer have more than 640 KB, and disks more than 2 GB...
0
 
vadim_tiCommented:
I do not know about Delphi 4, i have Delphi 7
But you can see in help or in sources
0
 
joncmoraCommented:
Workshop_Alex seem to be right.  Seek() is expecting a Longint/Integer, a signed 32-bit integer type. Therefore it can only access half the 4G max since the leftmost bit is used for the sign.

The file size actually won't matter (OS dependent), just the number of records, since you used Reset(F, SizeOf(TTest)). This would mean you can have at most (MaxInt / 2) records.
0
 
Wim ten BrinkCommented:
Unless I'm mistaken, all versions prior to Delphi 6 or 7 also had the same flaw in the TStream class. Basically, filesize and other file position based fields are based on the Longint type, which has a value range between –2147483648 and 2147483647. Thus, for huge files even the TStream might not work.

Now, the problem actually lies in the Windows API, although Borland itself also has a flaw. The Windows API has a function called ReadFile() which can read an x amount of bytes from a file. However, x is of type DWORD (range 0..4294967295) and thus doesn't work well. Borland made things slightly worse by typecasting the DWORD to a LongInt type. (Or integer type.) Borland therefore divided the maximum size in half!

Thus, even Windows itself can't handle files this big very well either. Then again, no developer could just write a record of 2 GB to disk anyway since the current systems don't even have that amount of memory available! Even if you've 4 GB of memory, most of it won't be accessible that easily, since the OS and your own application will be eating quite a lot of it. Thus, it is not such a big deal that Windows has this limitation. the 2 GB border is like the 640 KB limit some 20 years ago. No one could really imagine needing more than this amount of memory.

So, how to deal with huge files? Well, either use a professional SQL-based database like SQL Server or Oracle for such a huge amount of data or try to find some component that can handle huge files, like bpana suggested.
Or split the file up in smaller pieces with smaller records...

It doesn't matter if the OS or the filesystem can handle files this big. The problem is that the Windows API is too limited...
0
 
JukiCommented:
the limitation is in the seek function; it has limitation of 2^32 bytes (meaning that it uses int for seeking
rather than int64). The limitation exists even in delphi7. The easiest way to overcome this is to use
custom procedure, ie. below is my bigseek which does that on streams...

procedure BigSeek(isofile: tstream; FromLocation: TSeekOrigin; AmountSeek: int64);
var
  thispos : int64;
  stepsize : integer;
begin
  stepsize := 134217728;
  thispos := isofile.Position;

  if FromLocation = SoBeginning then begin
    isofile.Seek(0,soFromBeginning);
    while isofile.Position < (AmountSeek - Stepsize) do begin
      isofile.Seek(stepsize,soFromCurrent);
    end;
    isofile.Seek(AmountSeek - isofile.Position,soFromCurrent);
    exit;
  end;

  if (FromLocation = SoCurrent) and (AmountSeek > 0) then begin
    while isofile.Position < (thispos + AmountSeek - stepsize) do begin;
      isofile.Seek(stepsize,soFromCurrent);
    end;
    isofile.Seek((thispos + AmountSeek - isofile.Position),soFromCurrent);
    exit;
  end;

  if FromLocation = SoFromEnd then begin
    isofile.Seek(0,soFromEnd);
    while isofile.Position > (AmountSeek + Stepsize) do begin
      isofile.Seek(-stepsize,soFromCurrent);
    end;
    isofile.Seek(AmountSeek + isofile.Position,soFromCurrent);
    exit;
  end;

end;


Juhani Suhonen

0
 
winuser2000Author Commented:
Thanks Juki but Delphi 4 Pro does not have anything like TSeekOrigin - otherwise it may have solved my problem.

Looks like i will have to give up on getting delphi 4 professional to work on 4 GB files.

Will close this question shortly unless someone else can solve it.
0
 
JukiCommented:
nope - tseekorigin is constant type declaring start point for
seek; it's really integer between 0 and 2 with the following meaning :

0 = seek from start (soFromBeginning)
1 = seek from current (soFromCurrent)
2 = seek from end (soEnd or soFromEnd);

so you can replace tseekorigin with cardinal and respect
the values mentioned above OR you can add type to
your project :

type
  TSeekOrigin = (soBeginning, soCurrent, soEnd);


Juhani Suhonen
0
 
Slick812Commented:
hello  winuser2000 , , I don't think that Blockread will get you for 4 Gb files, as you have already figured, but I think that the API File functions are NOT limited by file size (although the WriteFile and ReadFile are limited to DWORD amounts) but you can write to the file as much as you want, and it will keep growing, as in the code below


procedure TForm1.sbut_WriteFileClick(Sender: TObject);
type
  TaRec = record
    I1, I2, I3, I4, I5, I6, I7, I8: Integer;
    end;

var
aRec1: TaRec;
hFile1, BytesWrite, i: Cardinal;
begin
ZeroMemory(@aRec1, SizeOf(aRec1));
aRec1.I3 := 123;

hFile1 := CreateFile('E:\Large.File',GENERIC_WRITE,FILE_SHARE_READ,nil,
       CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL or FILE_FLAG_SEQUENTIAL_SCAN,0);
if hFile1 = INVALID_HANDLE_VALUE then
  begin
  Showmessage('ERROR - Could NOT create file');
  Exit;
  end;
for i := 0 to 80000000 do
if Not WriteFile(hFile1,aRec1, SizeOf(aRec1), BytesWrite, nil) or
    (SizeOf(aRec1) <> BytesWrite) then
    begin
    Showmessage('ERROR - Could NOT Write file');
    Break;
    end;

CloseHandle(hFile1);
end;


= = = = = = = = =  == =  = =

and the  API function for "Seek" in a file  SetFilePointer( )   should be able to do More that 4 Gig

DWORD SetFilePointer(

    HANDLE hFile,      // handle of file
    LONG lDistanceToMove,      // number of bytes to move file pointer
    PLONG lpDistanceToMoveHigh,      // address of high-order word of distance to move  
    DWORD dwMoveMethod       // how to move
   );

it has a  high-order  Distance to move, which shoud give you alot more than 4 Gig access to a file

although, I have only used it in files as large as one Gig
0
 
winuser2000Author Commented:
Slick812  - your idea seems to be the solution to it all - just one question though.

as a test i declare X as an Int64 and pass it to LONG lDistanceToMove - but it will only go to the 2Gb range.

ie: if i create a record such as

type
 TaRec = record
   I1, I2, I3: string[255];
   end;

and then write it 4294967294 div SizeOf(arec1) times it produces the expected 4,294,967,040 byte file but refuses to read back any record above 2796202*sizeof(arec1) which is  2Gb.

according to microsoft it should handle 2^32 bytes which is 4 Gb.

question is - why won't it move beyond the 2Gb barrier when it is supposed to be able to handle 4Gb (and well beyond if PLONG lpDistanceToMoveHigh is used according to microsoft).
0
 
winuser2000Author Commented:
It's ok, i got it worked out and it works perfectly!

Slick 812 - you get the points

but i will increase the points and award some to others for their help too

I'll be back soon to close this question.

Again, thank you VERY much slick812!
0
 
Slick812Commented:
if you do not want to mess with the Hi DWord pointer in the SetFilePointer function, I have used several calls to it, in sequence using a large integer value in the  lDistanceToMove parameter and   FILE_CURRENT  , the following code may work for you -


SetFilePointer(hFile1,MAXLONG,nil, FILE_CURRENT);
SetFilePointer(hFile1,MAXLONG,nil, FILE_CURRENT);  // now file position is moved  4 Gigs foward
0
 
winuser2000Author Commented:
Original 250 points for Slick812 for an excellent solution.

Additional points given to bpana(50), Workshop_Alex(100) and Juki(100) for their help.  (all graded as A)
0

Featured Post

How to Use the Help Bell

Need to boost the visibility of your question for solutions? Use the Experts Exchange Help Bell to confirm priority levels and contact subject-matter experts for question attention.  Check out this how-to article for more information.

  • 8
  • 3
  • 2
  • +5
Tackle projects and never again get stuck behind a technical roadblock.
Join Now