asked on

C#.NET. How Do I save an array of integers VERY Fast

Hi

Im developing an application using Visual Studio 2008 and C#.NET

Im reading images from a camera and I need to save them to hard disk as quickly as possible.
The image array is defined thus. It contains 1.5M shorts

short[] image = new short[1500000];

Im currently using the code below to save each image. However its not fast enough. A bottleneck occurs after a few dozen images.

Ive found some articles on the Internet that say the BinaryFormatter class is inefficient.

Has anyone had a similar problem or have an idea how I can speed things up. Im not really interested in getting the file size small (although if thats a side effect all the better) I just want the array to be saved to disk as quickly as possible.

Thanks.

System.Runtime.Serialization.Formatters.Binary.BinaryFormatter  bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
            int count = 0;

            string fName = Directory.GetCurrentDirectory();
            fName  = fName + @"/" + Convert.ToString(count) + @".txt";

                Stream myStream = new FileStream(fName, FileMode.Create);
                bf.Serialize(myStream, numbers);
                myStream.Close();

Open in new window

Stephan

Do you want to save this from a fileupload or directory?

What kind of extension needs it to be? Just .txt?

w00te

Check this out:
http://www.codeproject.com/KB/files/fastbinaryfileinput.aspx
Near the 2nd half of the article they have an interesting pattern for rapid writing involving converting into byte buffers first. Worth a try :)
-w00te

rangers99

ASKER

Hi stephan

Just assume there is an array with 1.5 million integers and start from there.
As for file extension it doesnt matter. Im thinking a .txt file wont be suitable because the file would have to be in binary format.

ASKER CERTIFIED SOLUTION

kaufmed

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

rangers99

ASKER

Thanks for that Kaufmed. Its something I will definetly try out if I can't get another solution.

I already have a separate thread that grabs the images from the camera and reads them into a buffer in my C# code. So Im thinking another thread could be one too many (I rarely develop multithreading applications).

rangers99

ASKER

woote. Thanks for the link. However I cant see the bit where the array is written to hard disk. The binary data is just written to an array of bytes.

kaufmed

>> So Im thinking another thread could be one too many

Depending on how old your machine is I wouldn't think you'd have to worry about it that much if you (at present), only have two threads. ThreadPool is already allocated for you (you'll notice that QueueUserWorkItem is a static call) and manages the creation and cleanup of the threads it deals with.

rangers99

ASKER

This article states that serialization is very inefficent in C#.NET using the binaryFormatter class.
http://www.codeproject.com/KB/dotnet/FastSerializer.aspx

Its not clear to me whether I can use this code to make saving an array of integers (using serialisation) to a hard disk much faster than using the BinaryFormatter class.

Volox

I'm not sure exactly how it compares speed wise because but you might try converting the shorts over to chars and then writing them out as unicode using a BinaryWriter. Like this...

short[] image = new short[1500000];
            
char[] chars = Array.ConvertAll<short, char>(image, b => (char)b);

string fName = Directory.GetCurrentDirectory();
fName = fName + @"/" + Convert.ToString(count) + @".txt";

Stream myStream = new FileStream(fName, FileMode.Create);
BinaryWriter bw = new BinaryWriter(myStream, Encoding.Unicode);
bw.Write(chars);
bw.Flush();
bw.Close();
myStream.Close();

Open in new window

SOLUTION

Volox

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

Volox

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

rangers99

ASKER

Thanks for that. Bear with me folks while I try out these ideas

rangers99

ASKER

kaufmed, Your solution made a noticeable difference to the speed. Thanks

Volox, You 1st idea slowed things down by 3 times!

Volox

Kind of figures with all the memory allocation and the iteration of the array for conversion; that's why after giving it some thought I posted the other two.