Solved

does BlockRead have limitations?

Posted on 2004-09-02
21
812 Views
Last Modified: 2010-04-05
Does blockread have limitations or am i doing something wrong?

Here is my test code:

   == sample of variables used==
F: File;
type
Trecordstructure=packed record  
{...record}
end;
z:integer;
Ttest:Trecordstructure;

=== actual test code begins ===

AssignFile(F, 'filename.dat');
Rewrite(F, sizeOf(Ttest));
CloseFile(F);

   == so far, created the file - no problems

AssignFile(F, 'filename.dat');
reset(F, sizeOf(Ttest));

     === fill file with 2,600,000 copies of the record as a test ===
z:=0;
repeat
    blockwrite(f, Ttest, 1);
until z>= 2600000;
closefile(f);

   == so far all seems to be working ==

AssignFile(F, 'filename.dat');
reset(F, sizeOf(Ttest));

    === This is where the problem is (note i am seeking to 10 less that the amount put into the file ===
seek(f, 2599990);
blockread(f, Ttest, 1);
closefile(f);
{code to show record content}


I keep getting error 131 (An attempt was made to move the file pointer before the beginning of the file. )

Does Seek or Blockread have a limitation?  I checked the file created and divided it by the size of Ttest and it works out to be 2600000 - so what is going on?










0
Comment
Question by:winuser2000
  • 8
  • 3
  • 2
  • +5
21 Comments
 

Author Comment

by:winuser2000
Comment Utility
Strange - just did some more testing and it seems seek gives up at around the 2Gb file mark yet blockwrite seems to be fine.

Anyone got a way around this?
0
 
LVL 13

Expert Comment

by:BlackTigerX
Comment Utility
what version of Windows are you using?

that could be more of a Windows (the version you have) limitation

and most important, what file system are you using? (FAT, FAT32, NTFS, etc)
0
 
LVL 17

Expert Comment

by:Wim ten Brink
Comment Utility
In general, Delphi uses the Integer type for file sizes. This is nice for files less than 2 GB (The maximum value of an integer) but flawed when files are bigger. It's a flaw in a big part of the Delphi VCL and actually, Windows itself.
0
 

Author Comment

by:winuser2000
Comment Utility
Windows 2000, FAT 32, Delphi 4 Pro

But, then how come Blockwrite can write to the maximum file size allowed on a FAT32 system (4Gb) yet seek cannot move to it?
0
 
LVL 13

Expert Comment

by:BlackTigerX
Comment Utility
you can try doing 2 Seek calls...
0
 

Author Comment

by:winuser2000
Comment Utility
tried that, didn't work.

Still find it odd that blockwrite will happily write files till it hits the 4Gb file size but seek is unable to move above the 2Gb div sizeof(record).

Do you know if blockread is also affected or is it down to seek needing to be "fixed"
0
 
LVL 6

Expert Comment

by:vadim_ti
Comment Utility
you can work with TFileStream.

its Seek method
function Seek(const Offset: Int64; Origin: TSeekOrigin): Int64;

0
 
LVL 6

Assisted Solution

by:bpana
bpana earned 50 total points
Comment Utility
check out GpHugeFile (http://swiss.torry.net/filedrvother.htm)

GpHugeFile / TGpHugeFileStream v.3.07a
By Primoz Gabrijelcic. Encapsulation of Windows file-handling routines that allows work with >2GB files. Included is support for non-buffered access (FILE_FLAG_NO_BUFFERING) and buffering for sequentially accessed files. TGpHugeFile is interface to 64-bit file functions with some added functionality. TGpHugeFileStream is descendant of TStream that wraps TGpHugeFile.

Fully functional
Source: Included
Download: D5 D6
0
 

Author Comment

by:winuser2000
Comment Utility
vadim_ti  - the seek method is Int64 in Delphi 4 pro too?

thanks bpana, i'll download that and give it a look over.
0
 
LVL 17

Assisted Solution

by:Wim ten Brink
Wim ten Brink earned 100 total points
Comment Utility
> But, then how come Blockwrite can write to the maximum file size allowed on a FAT32 system (4Gb) yet seek cannot move to it?

The Integer type is 32-bits and thus allows 4 GB of values. An unsigned integer would therefore have the maximum value of 4GB. However, integers in Delphi always have a sign, changing their range from 0..4 GB to -2GB..+2GB. Beyond the 2 GB the datatype has an overflow...

Another problem within Delphi is that these integer values are inherited from Delphi's past history. The compiler needs to be backwards compatible so there could not be too many changes to the core of the runtime. Using INT64 for filesizes would solve quite a few of these problems but it would also drastically change the memory structures of the core file system structures. (They would become bigger.)

Then again, not many people do create files over 2 GB in size. Those who do, often use a database instead. And even in a database it would be a huge amount of diskspace. We're talking about two million megabytes here. And a megabyte is already a million bytes. Fortunately, the 64-bits filesystem allows Windows to create even bigger files.

About Delphi being able to write and not to read from big files, well simple... To write to a file, Delphi can just call the Windows API and stuff gets written. To read data, Delphi is probably looking at the filesize or current position in the file and this value will have an overflow. Instead of being e.g. somewhere around the 3 GB it's suddenly minus 1 GB. An invalid value, thus crash...

This limitation just became clear when Delphi 3 and 4 got on the market. And back then not many people were too worried about it because files that huge were considered absurd. Many users didn't even have a harddisk that big anyway. But in a few years, lots of things can change. Suddenly computer have more than 640 KB, and disks more than 2 GB...
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 
LVL 6

Expert Comment

by:vadim_ti
Comment Utility
I do not know about Delphi 4, i have Delphi 7
But you can see in help or in sources
0
 

Expert Comment

by:joncmora
Comment Utility
Workshop_Alex seem to be right.  Seek() is expecting a Longint/Integer, a signed 32-bit integer type. Therefore it can only access half the 4G max since the leftmost bit is used for the sign.

The file size actually won't matter (OS dependent), just the number of records, since you used Reset(F, SizeOf(TTest)). This would mean you can have at most (MaxInt / 2) records.
0
 
LVL 17

Expert Comment

by:Wim ten Brink
Comment Utility
Unless I'm mistaken, all versions prior to Delphi 6 or 7 also had the same flaw in the TStream class. Basically, filesize and other file position based fields are based on the Longint type, which has a value range between –2147483648 and 2147483647. Thus, for huge files even the TStream might not work.

Now, the problem actually lies in the Windows API, although Borland itself also has a flaw. The Windows API has a function called ReadFile() which can read an x amount of bytes from a file. However, x is of type DWORD (range 0..4294967295) and thus doesn't work well. Borland made things slightly worse by typecasting the DWORD to a LongInt type. (Or integer type.) Borland therefore divided the maximum size in half!

Thus, even Windows itself can't handle files this big very well either. Then again, no developer could just write a record of 2 GB to disk anyway since the current systems don't even have that amount of memory available! Even if you've 4 GB of memory, most of it won't be accessible that easily, since the OS and your own application will be eating quite a lot of it. Thus, it is not such a big deal that Windows has this limitation. the 2 GB border is like the 640 KB limit some 20 years ago. No one could really imagine needing more than this amount of memory.

So, how to deal with huge files? Well, either use a professional SQL-based database like SQL Server or Oracle for such a huge amount of data or try to find some component that can handle huge files, like bpana suggested.
Or split the file up in smaller pieces with smaller records...

It doesn't matter if the OS or the filesystem can handle files this big. The problem is that the Windows API is too limited...
0
 

Assisted Solution

by:Juki
Juki earned 100 total points
Comment Utility
the limitation is in the seek function; it has limitation of 2^32 bytes (meaning that it uses int for seeking
rather than int64). The limitation exists even in delphi7. The easiest way to overcome this is to use
custom procedure, ie. below is my bigseek which does that on streams...

procedure BigSeek(isofile: tstream; FromLocation: TSeekOrigin; AmountSeek: int64);
var
  thispos : int64;
  stepsize : integer;
begin
  stepsize := 134217728;
  thispos := isofile.Position;

  if FromLocation = SoBeginning then begin
    isofile.Seek(0,soFromBeginning);
    while isofile.Position < (AmountSeek - Stepsize) do begin
      isofile.Seek(stepsize,soFromCurrent);
    end;
    isofile.Seek(AmountSeek - isofile.Position,soFromCurrent);
    exit;
  end;

  if (FromLocation = SoCurrent) and (AmountSeek > 0) then begin
    while isofile.Position < (thispos + AmountSeek - stepsize) do begin;
      isofile.Seek(stepsize,soFromCurrent);
    end;
    isofile.Seek((thispos + AmountSeek - isofile.Position),soFromCurrent);
    exit;
  end;

  if FromLocation = SoFromEnd then begin
    isofile.Seek(0,soFromEnd);
    while isofile.Position > (AmountSeek + Stepsize) do begin
      isofile.Seek(-stepsize,soFromCurrent);
    end;
    isofile.Seek(AmountSeek + isofile.Position,soFromCurrent);
    exit;
  end;

end;


Juhani Suhonen

0
 

Author Comment

by:winuser2000
Comment Utility
Thanks Juki but Delphi 4 Pro does not have anything like TSeekOrigin - otherwise it may have solved my problem.

Looks like i will have to give up on getting delphi 4 professional to work on 4 GB files.

Will close this question shortly unless someone else can solve it.
0
 

Expert Comment

by:Juki
Comment Utility
nope - tseekorigin is constant type declaring start point for
seek; it's really integer between 0 and 2 with the following meaning :

0 = seek from start (soFromBeginning)
1 = seek from current (soFromCurrent)
2 = seek from end (soEnd or soFromEnd);

so you can replace tseekorigin with cardinal and respect
the values mentioned above OR you can add type to
your project :

type
  TSeekOrigin = (soBeginning, soCurrent, soEnd);


Juhani Suhonen
0
 
LVL 33

Accepted Solution

by:
Slick812 earned 250 total points
Comment Utility
hello  winuser2000 , , I don't think that Blockread will get you for 4 Gb files, as you have already figured, but I think that the API File functions are NOT limited by file size (although the WriteFile and ReadFile are limited to DWORD amounts) but you can write to the file as much as you want, and it will keep growing, as in the code below


procedure TForm1.sbut_WriteFileClick(Sender: TObject);
type
  TaRec = record
    I1, I2, I3, I4, I5, I6, I7, I8: Integer;
    end;

var
aRec1: TaRec;
hFile1, BytesWrite, i: Cardinal;
begin
ZeroMemory(@aRec1, SizeOf(aRec1));
aRec1.I3 := 123;

hFile1 := CreateFile('E:\Large.File',GENERIC_WRITE,FILE_SHARE_READ,nil,
       CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL or FILE_FLAG_SEQUENTIAL_SCAN,0);
if hFile1 = INVALID_HANDLE_VALUE then
  begin
  Showmessage('ERROR - Could NOT create file');
  Exit;
  end;
for i := 0 to 80000000 do
if Not WriteFile(hFile1,aRec1, SizeOf(aRec1), BytesWrite, nil) or
    (SizeOf(aRec1) <> BytesWrite) then
    begin
    Showmessage('ERROR - Could NOT Write file');
    Break;
    end;

CloseHandle(hFile1);
end;


= = = = = = = = =  == =  = =

and the  API function for "Seek" in a file  SetFilePointer( )   should be able to do More that 4 Gig

DWORD SetFilePointer(

    HANDLE hFile,      // handle of file
    LONG lDistanceToMove,      // number of bytes to move file pointer
    PLONG lpDistanceToMoveHigh,      // address of high-order word of distance to move  
    DWORD dwMoveMethod       // how to move
   );

it has a  high-order  Distance to move, which shoud give you alot more than 4 Gig access to a file

although, I have only used it in files as large as one Gig
0
 

Author Comment

by:winuser2000
Comment Utility
Slick812  - your idea seems to be the solution to it all - just one question though.

as a test i declare X as an Int64 and pass it to LONG lDistanceToMove - but it will only go to the 2Gb range.

ie: if i create a record such as

type
 TaRec = record
   I1, I2, I3: string[255];
   end;

and then write it 4294967294 div SizeOf(arec1) times it produces the expected 4,294,967,040 byte file but refuses to read back any record above 2796202*sizeof(arec1) which is  2Gb.

according to microsoft it should handle 2^32 bytes which is 4 Gb.

question is - why won't it move beyond the 2Gb barrier when it is supposed to be able to handle 4Gb (and well beyond if PLONG lpDistanceToMoveHigh is used according to microsoft).
0
 

Author Comment

by:winuser2000
Comment Utility
It's ok, i got it worked out and it works perfectly!

Slick 812 - you get the points

but i will increase the points and award some to others for their help too

I'll be back soon to close this question.

Again, thank you VERY much slick812!
0
 
LVL 33

Expert Comment

by:Slick812
Comment Utility
if you do not want to mess with the Hi DWord pointer in the SetFilePointer function, I have used several calls to it, in sequence using a large integer value in the  lDistanceToMove parameter and   FILE_CURRENT  , the following code may work for you -


SetFilePointer(hFile1,MAXLONG,nil, FILE_CURRENT);
SetFilePointer(hFile1,MAXLONG,nil, FILE_CURRENT);  // now file position is moved  4 Gigs foward
0
 

Author Comment

by:winuser2000
Comment Utility
Original 250 points for Slick812 for an excellent solution.

Additional points given to bpana(50), Workshop_Alex(100) and Juki(100) for their help.  (all graded as A)
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Objective: - This article will help user in how to convert their numeric value become words. How to use 1. You can copy this code in your Unit as function 2. than you can perform your function by type this code The Code   (CODE) The Im…
Hello everybody This Article will show you how to validate number with TEdit control, What's the TEdit control? TEdit is a standard Windows edit control on a form, it allows to user to write, read and copy/paste single line of text. Usua…
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now