?
Solved

Faster file processing

Posted on 2005-03-03
10
Medium Priority
?
191 Views
Last Modified: 2010-04-05
Making a simple encryption program for personal use. In fact by the method I'm not sure it is really encryption.. It alters each byte individually, and I think therein lies my problem. It's simply no good for files bigger than 10k because it takes SO long. I started trying to read it into blocks of four bytes but I don't think it'll be much better, if not worse.

I'm looking for algorithms rather than code. possibly you can leave the actual encryption part as an empty block

...
{encryption on ___ }
...

etc

Thanx in advance
Synthetics
0
Comment
Question by:Synthetics
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
  • 2
  • +1
10 Comments
 
LVL 5

Assisted Solution

by:DeerBear
DeerBear earned 160 total points
ID: 13451966
Hi.

Encryption algorithms can be divided into two very big branches: symmetric and asymmetric ones.
Symmetric ones rely on a single password to be used to encode/decode the file, while asymmetric makes use of two different keys, public and private one.

Usually, a ready made encryption algorithm will perform better than a "home made" one, because there'll have been a study behind the former.

Coming to the question, the only way I know of to speed up file read/write is to use custom buffers.
Look at this code:

Type
 
   TMyBuffer = array[0..1023] of byte;

var F : File;

procedure InitFile(AFileName : String);
begin
  AssignFile(F,AFileName);
  Reset(F,SizeOf(TMyBuffer));
end;

procedure ReadData(var ABuf : TMyBuffer);
begin
  read(F,ABuf,SizeOf(ABuf));
end;

procedure DoneFile;
begin
  CloseFile(F);
end;

With this code, you're using a custom buffer of 1KB, but you can enlarge it if you prefer.
This is the safest way to speed things up AFAIK.

Cheers,

Andrew
0
 
LVL 11

Assisted Solution

by:ZhaawZ
ZhaawZ earned 160 total points
ID: 13454474
>> I started trying to read it into blocks of four bytes but I don't think it'll be much better, if not worse.
Does it look like "if a file is 1KB, you read it [1024 / 4 =] 256 times"? It`s better/faster to read it into 512B, 1024B or similar blocks and then work with memory, not with hdd.
0
 
LVL 46

Assisted Solution

by:aikimark
aikimark earned 160 total points
ID: 13470893
Synthetics,

Maybe it is your choice or implementation of encryption that is the reason for the slow performance you're observing.  Without using any real keys, what does your code look like?

The cause may lie in several places, most commonly:
1. data structures
2. I/O routines
3. data manipulation methods

==============================
The simplest encryption routines are one-for-one replacement routines, where you replace one byte value with another byte value.  You would use an array of (substitution) byte values arranged in the order association with their (replaced) byte values.
Example:
To replace 'A' with '$', you would put a value of 36 in position 65 of a 0..255 array.

More sophisticated versions of this use multiple replacement tables, either for each position in the input stream or for cyclic replacement of sequential positions in the stream.

The simplest symetric keys use the XOR operation to flip the bits in the input stream's bytes with the byte values of a key.  Applying the same operation with an encrypted string will decrypt the string.  This is analogous to flipping the value of a boolean variable.
Example:
bVar := Not bVar;
bVar := Not bVar;
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 5

Author Comment

by:Synthetics
ID: 13478589
yes I think my main problem is because I'm reading one byte at a time, but I wasn't sure how to implement reading larger numbers because the files might not, and with larger values probably won't, be a multiple of that value. As I mentioned, I started trying to read in blocks of 4 bytes, but it still wasn't very effective.

How can I read in blocks of 1024 bytes etc and still work with files not of an exact multiple of 1024?

My encryption at the moment is a byte-by-byte XOR process. I'm an amateur at best and wanted to start off simple and then develop it and improve it over time. Faster processing is a must though!

Synthetics
0
 
LVL 46

Expert Comment

by:aikimark
ID: 13482752
1. Why read it in 1024 byte chunks?
2. What difference does the length of the file have on the use of your XOR operations?
3. If you are going to do your I/O in chunks, and apply a block-level cypher, then you could just pad the final block with zeroes or random byte sequences.
0
 
LVL 5

Author Comment

by:Synthetics
ID: 13486763
1. because, I think, most of the time being taken up by my program is the reading of data from the input file. So reading in bigger chunks means the processor spends less time waiting for the data from RAM, and overall faster file processing.. that's my understanding anyway
2. it doesn't, at the moment each byte is XOR'd on its own
3. me = noob. What's a block cypher? and if I pad the end how do I know what the original file size is when I want to decrypt it?

Synthetics
0
 
LVL 46

Accepted Solution

by:
aikimark earned 160 total points
ID: 13487082
0
 
LVL 46

Expert Comment

by:aikimark
ID: 13487101
you should do some research on encryption algorithms to improve your understanding of the process and options available to you.  Start with a search for block cypher.
0
 
LVL 5

Author Comment

by:Synthetics
ID: 13487128
yup, would do if it was commercial. it's really for my own use though, so wanted to see what I could come up with. Just had the main problem of speed that I wanted to overcome.

Thank you for links, will test asap

Synthetics
0
 
LVL 11

Assisted Solution

by:ZhaawZ
ZhaawZ earned 160 total points
ID: 13489319
Here`s a small, simple example.
File size is 8`269`608 bytes. This took about 0.66 seconds. Block size is 4KB (4096 bytes).

To decrypt a crypted file, assign f1 with crypted file, but f2 - with output file (decrypted).


procedure TForm1.FormCreate(Sender: TObject);
var
  f1 : file; //untyped file variables to work with a file;
  f2 : file; //  f1 : input; f2 : output;
  b : array [0..4095] of byte; //4KB
  n : integer; //loop variable, used for crypting
  size : integer; //count of bytes that are read from a file
  t : cardinal; //variable to calculate needed time for crypting
begin
t := GetTickCount;
AssignFile(f1, 'D:\share\progs\winamp5\winamp507_silvertide.exe');
AssignFile(f2, 'D:\share\progs\winamp5\winamp507_silvertide_crypted.exe');
FileMode := fmOpenRead;
Reset(f1, 1);
FileMode := fmOpenWrite;
Rewrite(f2, 1);
while not eof(f1) do begin
  BlockRead(f1, b[0], 4096, size);
  for n := 0 to size - 1 do begin
    b[n] := b[n] xor (n and $ff);
  end;
  BlockWrite(f2, b[0], size);
end;
CloseFile(f1);
CloseFile(f2);
ShowMessageFmt('It took %.2f seconds', [(GetTickCount - t) / 1000]);
end;
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A lot of questions regard threads in Delphi.   One of the more specific questions is how to show progress of the thread.   Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to the…
Creating an auto free TStringList The TStringList is a basic and frequently used object in Delphi. On many occasions, you may want to create a temporary list, process some items in the list and be done with the list. In such cases, you have to…
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor (https://www.adremsoft.com/). Top Charts is a view in which you can set seve…
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…
Suggested Courses
Course of the Month7 days, 20 hours left to enroll

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question