Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Efficient way to split a file into segments ?

Posted on 2011-09-18
1
Medium Priority
?
420 Views
Last Modified: 2016-09-29
I retrieve a file size from hte internet, lets say the file content (size) is 17mb.. (the file size will vary in future usage, so the algorithm must be compatible)

what is the most efficient way to split it into segments ? i have a specific amout of threads that each one downloads from AOffset to BOffset, my current algorithm is to divide it by the amount of connections that the user chose...

this is the code that makes the split:
 
var
  nLoop,
  nEndDiv,
  nConnections      : Integer;

  i64FileSize,
  i64Start,
  i64End,
  i64End2           : Int64;  

  nConnections := StrToInt( SpinBox1.Text );
    nEndDiv := nConnections;  

    bStartZero := True;

    for nLoop := 1 to nConnections do
    begin 

     if ( bStartZero = True ) then i64Start := 0;
      bStartZero := False;

      i64End :=
       i64FileSize div (nEndDiv);

      dec( nEndDiv );

      i64End2 := i64End;

      with TFetchDataThread.Create(
      alabel[nLoop], apbar[nLoop], hOpenFile[nLoop], hInetFile[nLoop], i64Start, i64End ) do
      begin
        Priority := tpNormal;
        Start;
      end;

      i64Start := i64End2 + 1;
    end;

Open in new window

 

this is the code that downloads:
 
procedure TFetchDataThread.Execute;
type
  TypeByteArray = array [1..1024] of Byte;
var
  Buffer         : TypeByteArray;
  BytesToRead    : DWORD;
  BytesToWrite   : DWORD;

  BufferLen,
  BytesWritten   : DWORD;
  EndProgress    : Cardinal;
  i: Integer;
begin
  FProgressBar.Min := Extended( FStartOffset + 0.0 );
  FProgressBar.Max := Extended( FEndOffset   + 0.0 );

  InternetSetFilePointer( FInetFile, FStartOffset, nil, FILE_BEGIN, 0 );

  EndProgress := SetFilePointer( FDestFile, FEndOffset, nil, FILE_BEGIN );

  SetFilePointer( FDestFile, FStartOffset, nil, FILE_BEGIN );

  BytesToRead := SizeOf( Buffer );
  BytesToWrite := SizeOf(Buffer);

  try
    repeat

      InternetReadFile(
       FInetFile, @Buffer, BytesToRead, BufferLen );

      LockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

      if ( FCurrentOffset > EndProgress ) then
      WriteFile(
       FDestFile, Buffer, BytesToWrite, BytesWritten, nil )
      else
      WriteFile(
       FDestFile, Buffer, BytesToWrite, BytesWritten, nil );

      UnlockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

      FCurrentOffset :=
       SetFilePointer( FDestFile, 0, nil, FILE_CURRENT );

      FProgressBar.Value := FCurrentOffset;

      Synchronize( UpdateGUI );
    until FCurrentOffset >= EndProgress;
  finally
    CloseHandle( FDestFile );
    InternetCloseHandle( FInetFile );
  end;
end;

Open in new window

0
Comment
Question by:rotem156
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
1 Comment
 
LVL 25

Accepted Solution

by:
epasquier earned 2000 total points
ID: 36563416
well, it's pretty obvious that each thread should manage differently the last block. In your current code you read 1024 bytes whatever the position in your file is.

for the same example of 40.528.057 bytes per block, that would mean
39578 loops reading 1024 bytes
and one reading 185

I suppose one quick fix would be :
repeat
//== FIX
      BytesToRead:=EndProgress-FCurrentOffset;  
      if BytesToRead>1024 Then BytesToRead:=1024;
//== END FIX
      InternetReadFile(
       FInetFile, @Buffer, BytesToRead, BufferLen );

      LockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

//== WHAT IS THAT ALL ABOUT ??
//      if ( FCurrentOffset > EndProgress ) then
//      WriteFile(
//       FDestFile, Buffer, BytesToWrite, BytesWritten, nil )
//      else
//== ???
      WriteFile(
       FDestFile, Buffer, BytesToWrite, BytesWritten, nil );

      UnlockFile(
       FDestFile, FStartOffset, 0, BytesToRead, 0 );

      FCurrentOffset :=
       SetFilePointer( FDestFile, 0, nil, FILE_CURRENT );

      FProgressBar.Value := FCurrentOffset;

      Synchronize( UpdateGUI );
    until FCurrentOffset >= EndProgress;

Open in new window

0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Have you ever had your Delphi form/application just hanging while waiting for data to load? This is the article to read if you want to learn some things about adding threads for data loading in the background. First, I'll setup a general applica…
Prime numbers are natural numbers greater than 1 that have only two divisors (the number itself and 1). By “divisible” we mean dividend % divisor = 0 (% indicates MODULAR. It gives the reminder of a division operation). We’ll follow multiple approac…
Video by: ITPro.TV
In this episode Don builds upon the troubleshooting techniques by demonstrating how to properly monitor a vSphere deployment to detect problems before they occur. He begins the show using tools found within the vSphere suite as ends the show demonst…
I've attached the XLSM Excel spreadsheet I used in the video and also text files containing the macros used below. https://filedb.experts-exchange.com/incoming/2017/03_w12/1151775/Permutations.txt https://filedb.experts-exchange.com/incoming/201…
Suggested Courses

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question