• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 193
  • Last Modified:

Faster file processing

Making a simple encryption program for personal use. In fact by the method I'm not sure it is really encryption.. It alters each byte individually, and I think therein lies my problem. It's simply no good for files bigger than 10k because it takes SO long. I started trying to read it into blocks of four bytes but I don't think it'll be much better, if not worse.

I'm looking for algorithms rather than code. possibly you can leave the actual encryption part as an empty block

...
{encryption on ___ }
...

etc

Thanx in advance
Synthetics
0
Synthetics
Asked:
Synthetics
  • 4
  • 3
  • 2
  • +1
5 Solutions
 
DeerBearCommented:
Hi.

Encryption algorithms can be divided into two very big branches: symmetric and asymmetric ones.
Symmetric ones rely on a single password to be used to encode/decode the file, while asymmetric makes use of two different keys, public and private one.

Usually, a ready made encryption algorithm will perform better than a "home made" one, because there'll have been a study behind the former.

Coming to the question, the only way I know of to speed up file read/write is to use custom buffers.
Look at this code:

Type
 
   TMyBuffer = array[0..1023] of byte;

var F : File;

procedure InitFile(AFileName : String);
begin
  AssignFile(F,AFileName);
  Reset(F,SizeOf(TMyBuffer));
end;

procedure ReadData(var ABuf : TMyBuffer);
begin
  read(F,ABuf,SizeOf(ABuf));
end;

procedure DoneFile;
begin
  CloseFile(F);
end;

With this code, you're using a custom buffer of 1KB, but you can enlarge it if you prefer.
This is the safest way to speed things up AFAIK.

Cheers,

Andrew
0
 
ZhaawZSoftware DeveloperCommented:
>> I started trying to read it into blocks of four bytes but I don't think it'll be much better, if not worse.
Does it look like "if a file is 1KB, you read it [1024 / 4 =] 256 times"? It`s better/faster to read it into 512B, 1024B or similar blocks and then work with memory, not with hdd.
0
 
aikimarkCommented:
Synthetics,

Maybe it is your choice or implementation of encryption that is the reason for the slow performance you're observing.  Without using any real keys, what does your code look like?

The cause may lie in several places, most commonly:
1. data structures
2. I/O routines
3. data manipulation methods

==============================
The simplest encryption routines are one-for-one replacement routines, where you replace one byte value with another byte value.  You would use an array of (substitution) byte values arranged in the order association with their (replaced) byte values.
Example:
To replace 'A' with '$', you would put a value of 36 in position 65 of a 0..255 array.

More sophisticated versions of this use multiple replacement tables, either for each position in the input stream or for cyclic replacement of sequential positions in the stream.

The simplest symetric keys use the XOR operation to flip the bits in the input stream's bytes with the byte values of a key.  Applying the same operation with an encrypted string will decrypt the string.  This is analogous to flipping the value of a boolean variable.
Example:
bVar := Not bVar;
bVar := Not bVar;
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
SyntheticsAuthor Commented:
yes I think my main problem is because I'm reading one byte at a time, but I wasn't sure how to implement reading larger numbers because the files might not, and with larger values probably won't, be a multiple of that value. As I mentioned, I started trying to read in blocks of 4 bytes, but it still wasn't very effective.

How can I read in blocks of 1024 bytes etc and still work with files not of an exact multiple of 1024?

My encryption at the moment is a byte-by-byte XOR process. I'm an amateur at best and wanted to start off simple and then develop it and improve it over time. Faster processing is a must though!

Synthetics
0
 
aikimarkCommented:
1. Why read it in 1024 byte chunks?
2. What difference does the length of the file have on the use of your XOR operations?
3. If you are going to do your I/O in chunks, and apply a block-level cypher, then you could just pad the final block with zeroes or random byte sequences.
0
 
SyntheticsAuthor Commented:
1. because, I think, most of the time being taken up by my program is the reading of data from the input file. So reading in bigger chunks means the processor spends less time waiting for the data from RAM, and overall faster file processing.. that's my understanding anyway
2. it doesn't, at the moment each byte is XOR'd on its own
3. me = noob. What's a block cypher? and if I pad the end how do I know what the original file size is when I want to decrypt it?

Synthetics
0
 
aikimarkCommented:
you should do some research on encryption algorithms to improve your understanding of the process and options available to you.  Start with a search for block cypher.
0
 
SyntheticsAuthor Commented:
yup, would do if it was commercial. it's really for my own use though, so wanted to see what I could come up with. Just had the main problem of speed that I wanted to overcome.

Thank you for links, will test asap

Synthetics
0
 
ZhaawZSoftware DeveloperCommented:
Here`s a small, simple example.
File size is 8`269`608 bytes. This took about 0.66 seconds. Block size is 4KB (4096 bytes).

To decrypt a crypted file, assign f1 with crypted file, but f2 - with output file (decrypted).


procedure TForm1.FormCreate(Sender: TObject);
var
  f1 : file; //untyped file variables to work with a file;
  f2 : file; //  f1 : input; f2 : output;
  b : array [0..4095] of byte; //4KB
  n : integer; //loop variable, used for crypting
  size : integer; //count of bytes that are read from a file
  t : cardinal; //variable to calculate needed time for crypting
begin
t := GetTickCount;
AssignFile(f1, 'D:\share\progs\winamp5\winamp507_silvertide.exe');
AssignFile(f2, 'D:\share\progs\winamp5\winamp507_silvertide_crypted.exe');
FileMode := fmOpenRead;
Reset(f1, 1);
FileMode := fmOpenWrite;
Rewrite(f2, 1);
while not eof(f1) do begin
  BlockRead(f1, b[0], 4096, size);
  for n := 0 to size - 1 do begin
    b[n] := b[n] xor (n and $ff);
  end;
  BlockWrite(f2, b[0], size);
end;
CloseFile(f1);
CloseFile(f2);
ShowMessageFmt('It took %.2f seconds', [(GetTickCount - t) / 1000]);
end;
0

Featured Post

Receive 1:1 tech help

Solve your biggest tech problems alongside global tech experts with 1:1 help.

  • 4
  • 3
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now