Solved

Efficient way to split a file into segments ?

Posted on 2011-09-18
1
415 Views
Last Modified: 2016-09-29
I retrieve a file size from hte internet, lets say the file content (size) is 17mb.. (the file size will vary in future usage, so the algorithm must be compatible)

what is the most efficient way to split it into segments ? i have a specific amout of threads that each one downloads from AOffset to BOffset, my current algorithm is to divide it by the amount of connections that the user chose...

this is the code that makes the split:
 
var
  nLoop,
  nEndDiv,
  nConnections      : Integer;

  i64FileSize,
  i64Start,
  i64End,
  i64End2           : Int64;  

  nConnections := StrToInt( SpinBox1.Text );
    nEndDiv := nConnections;  

    bStartZero := True;

    for nLoop := 1 to nConnections do
    begin 

     if ( bStartZero = True ) then i64Start := 0;
      bStartZero := False;

      i64End :=
       i64FileSize div (nEndDiv);

      dec( nEndDiv );

      i64End2 := i64End;

      with TFetchDataThread.Create(
      alabel[nLoop], apbar[nLoop], hOpenFile[nLoop], hInetFile[nLoop], i64Start, i64End ) do
      begin
        Priority := tpNormal;
        Start;
      end;

      i64Start := i64End2 + 1;
    end;

Open in new window

 

this is the code that downloads:
 
procedure TFetchDataThread.Execute;
type
  TypeByteArray = array [1..1024] of Byte;
var
  Buffer         : TypeByteArray;
  BytesToRead    : DWORD;
  BytesToWrite   : DWORD;

  BufferLen,
  BytesWritten   : DWORD;
  EndProgress    : Cardinal;
  i: Integer;
begin
  FProgressBar.Min := Extended( FStartOffset + 0.0 );
  FProgressBar.Max := Extended( FEndOffset   + 0.0 );

  InternetSetFilePointer( FInetFile, FStartOffset, nil, FILE_BEGIN, 0 );

  EndProgress := SetFilePointer( FDestFile, FEndOffset, nil, FILE_BEGIN );

  SetFilePointer( FDestFile, FStartOffset, nil, FILE_BEGIN );

  BytesToRead := SizeOf( Buffer );
  BytesToWrite := SizeOf(Buffer);

  try
    repeat

      InternetReadFile(
       FInetFile, @Buffer, BytesToRead, BufferLen );

      LockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

      if ( FCurrentOffset > EndProgress ) then
      WriteFile(
       FDestFile, Buffer, BytesToWrite, BytesWritten, nil )
      else
      WriteFile(
       FDestFile, Buffer, BytesToWrite, BytesWritten, nil );

      UnlockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

      FCurrentOffset :=
       SetFilePointer( FDestFile, 0, nil, FILE_CURRENT );

      FProgressBar.Value := FCurrentOffset;

      Synchronize( UpdateGUI );
    until FCurrentOffset >= EndProgress;
  finally
    CloseHandle( FDestFile );
    InternetCloseHandle( FInetFile );
  end;
end;

Open in new window

0
Comment
Question by:rotem156
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 
LVL 25

Accepted Solution

by:
epasquier earned 500 total points
ID: 36563416
well, it's pretty obvious that each thread should manage differently the last block. In your current code you read 1024 bytes whatever the position in your file is.

for the same example of 40.528.057 bytes per block, that would mean
39578 loops reading 1024 bytes
and one reading 185

I suppose one quick fix would be :
repeat
//== FIX
      BytesToRead:=EndProgress-FCurrentOffset;  
      if BytesToRead>1024 Then BytesToRead:=1024;
//== END FIX
      InternetReadFile(
       FInetFile, @Buffer, BytesToRead, BufferLen );

      LockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

//== WHAT IS THAT ALL ABOUT ??
//      if ( FCurrentOffset > EndProgress ) then
//      WriteFile(
//       FDestFile, Buffer, BytesToWrite, BytesWritten, nil )
//      else
//== ???
      WriteFile(
       FDestFile, Buffer, BytesToWrite, BytesWritten, nil );

      UnlockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

      FCurrentOffset :=
       SetFilePointer( FDestFile, 0, nil, FILE_CURRENT );

      FProgressBar.Value := FCurrentOffset;

      Synchronize( UpdateGUI );
    until FCurrentOffset >= EndProgress;

Open in new window

0

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Okay. So what exactly is the problem here? How often have we come across situations where we need to know if two strings are 'similar' but not necessarily the same? I have, plenty of times. Until recently, I thought any functionality like that wo…
When there is a disconnect between the intentions of their creator and the recipient, when algorithms go awry, they can have disastrous consequences.
Finds all prime numbers in a range requested and places them in a public primes() array. I've demostrated a template size of 30 (2 * 3 * 5) but larger templates can be built such 210  (2 * 3 * 5 * 7) or 2310  (2 * 3 * 5 * 7 * 11). The larger templa…
I've attached the XLSM Excel spreadsheet I used in the video and also text files containing the macros used below. https://filedb.experts-exchange.com/incoming/2017/03_w12/1151775/Permutations.txt https://filedb.experts-exchange.com/incoming/201…

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question