Solved

C#.NET. How Do I save an array of integers VERY Fast

Posted on 2010-08-31
14
486 Views
Last Modified: 2013-12-17
Hi

Im developing an application using Visual Studio 2008 and C#.NET

Im reading images from a camera and I need to save them to hard disk as quickly as possible.
The image array is defined thus. It contains 1.5M shorts

short[] image = new short[1500000];

Im currently using the code below to save each image. However its not fast enough. A bottleneck occurs after a few dozen  images.

Ive found some articles on the Internet that say the BinaryFormatter class is inefficient.

Has anyone had a similar problem or have an idea how I can speed things up. Im not really interested in getting the file size small (although if thats a side effect all the better) I just want the array to be saved to disk as quickly as possible.


Thanks.
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter  bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();

            int count = 0;



            string fName = Directory.GetCurrentDirectory();

            fName  = fName + @"/" + Convert.ToString(count) + @".txt";



                Stream myStream = new FileStream(fName, FileMode.Create);

                bf.Serialize(myStream, numbers);

                myStream.Close();

Open in new window

0
Comment
Question by:rangers99
  • 6
  • 4
  • 2
  • +2
14 Comments
 
LVL 16

Expert Comment

by:Stephan
ID: 33571281
Do you want to save this from a fileupload or directory?

What kind of extension needs it to be? Just .txt?
0
 
LVL 12

Expert Comment

by:w00te
ID: 33571331
Check this out:
http://www.codeproject.com/KB/files/fastbinaryfileinput.aspx
Near the 2nd half of the article they have an interesting pattern for rapid writing involving converting into byte buffers first.  Worth a try :)
-w00te
0
 

Author Comment

by:rangers99
ID: 33571423
Hi stephan

Just assume there is an array with 1.5 million integers and start from there.
As for file extension it doesnt matter. Im thinking a .txt file wont be suitable because the file would have to be in binary format.
0
 
LVL 74

Accepted Solution

by:
käµfm³d   👽 earned 350 total points
ID: 33571480
Have you considered multithreading the app? You could use the ThreadPool class to queue work--in this case, writing the data to disk. If all you are doing is writing the files to disk and there is no chance of other code in your app (or other apps) from touching the files while they are being written, then I believe you could get away with something as simple as the following, where CaptureMethod is simulating your code which receives the image from the camera.
using System;
using System.IO;
using System.Threading;
using System.Windows.Forms;

namespace WindowsFormsApplication3
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void CaptureMethod()
        {
            short[] image = new short[1500000];
            ImageData data;     // Passed to the worker method

            for (int i = 0; i < 15; i++)
            {
                data.count = i + 1;
                data.image = image;
                ThreadPool.QueueUserWorkItem(WriteFile, data);  // Send the writes to the queue
            }
        }

        /// <summary>
        /// Worker method
        /// </summary>
        /// <param name="imageData">Object which holds the image-related data</param>
        private void WriteFile(object imageData)
        {
            ImageData data = (ImageData)imageData;
            System.Runtime.Serialization.Formatters.Binary.BinaryFormatter bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();

            string fName = Directory.GetCurrentDirectory();
            fName = fName + @"/" + Convert.ToString(data.count) + @".txt";

            using (Stream myStream = new FileStream(fName, FileMode.Create))
            {
                bf.Serialize(myStream, data.image);
            }
        }

        // Just simulating the camera here
        private void button1_Click(object sender, EventArgs e)
        {
            CaptureMethod();
        }
    }

    /// <summary>
    /// A struct to hold the actual image data and it's sequence number
    /// </summary>
    struct ImageData
    {
        public int count;
        public short[] image;
    }
}

Open in new window

0
 

Author Comment

by:rangers99
ID: 33571611
Thanks for that Kaufmed. Its something I will definetly try out if I can't get another solution.

I already have a separate thread that grabs the images from the camera and reads them into a buffer in my C# code. So Im thinking another thread could be one too many (I rarely develop multithreading applications).
0
 

Author Comment

by:rangers99
ID: 33571641
woote. Thanks for the link. However I cant see the bit where the array is written to hard disk. The binary data is just written to an array of bytes.
0
 
LVL 74

Expert Comment

by:käµfm³d 👽
ID: 33571818
>>  So Im thinking another thread could be one too many

Depending on how old your machine is I wouldn't think you'd have to worry about it that much if you (at present), only have two threads. ThreadPool is already allocated for you (you'll notice that QueueUserWorkItem is a static call) and manages the creation and cleanup of the threads it deals with.
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 

Author Comment

by:rangers99
ID: 33579812
This article states that serialization is very inefficent in C#.NET using the binaryFormatter class.
http://www.codeproject.com/KB/dotnet/FastSerializer.aspx

Its not clear to me whether I can use this code to make saving an array of integers (using serialisation) to a hard disk much faster than using the BinaryFormatter class.
0
 
LVL 8

Expert Comment

by:Volox
ID: 33582541
I'm not sure exactly how it compares speed wise because but you might try converting the shorts over to chars and then writing them out as unicode using a BinaryWriter.  Like this...
short[] image = new short[1500000];

            

char[] chars = Array.ConvertAll<short, char>(image, b => (char)b);



string fName = Directory.GetCurrentDirectory();

fName = fName + @"/" + Convert.ToString(count) + @".txt";



Stream myStream = new FileStream(fName, FileMode.Create);

BinaryWriter bw = new BinaryWriter(myStream, Encoding.Unicode);

bw.Write(chars);

bw.Flush();

bw.Close();

myStream.Close();

Open in new window

0
 
LVL 8

Assisted Solution

by:Volox
Volox earned 150 total points
ID: 33582631
I would point out that if you can perform the conversion to chars on the input side then you save what is probably an expensive operation since the ConvertAll not only has to convert every value but it also has to allocat another 1500000 worth of array.  If your in-memory object is char rather than short and you convert on the input side, then you save the memory space and the array iterration.

The other performance techinique you might try is breaking up the array so that it is an array of arrays.  Make your sub-arrays sized to match memory / disk allocation sizes - so 512 each or 1024 each etc.  Then you write out the arrays one by one.  You might spin more but sometimes the stream objects and their underlying buffers work more effeciently when handed more reasonable sized chunks.  Think about it like trying to write data to a database (wihtout concern for network overhead)... one transaction with a million rows versus a thousand transactions with a thousand rows.  There is a balance in there where the underworkings start to struggle to keep track of the amount of data that is going to be committed.

The last techinique I might suggest is this... when you read the data, put it into a memory stream instead of into an array.  Then you should be able to just stream it from the memory stream into a file in it's binary form.
MemoryStream ms = new MemoryStream(1500000);

BinaryWriter bw = new BinaryWriter(ms, Encoding.Unicode);



object source;



while ( source.HasMoreData() )

{

    bw.Write( source.getNextShortValue() );

}

bw.Flush();



string fName = Directory.GetCurrentDirectory();

fName = fName + @"/" + Convert.ToString(count) + @".txt";



FileStream myStream = new FileStream(fName, FileMode.Create);



myStream.Write(ms.GetBuffer(), 0, (int)ms.Length);



ms.Close();

bw.Close();

myStream.Close();

Open in new window

0
 
LVL 8

Assisted Solution

by:Volox
Volox earned 150 total points
ID: 33582677
Here is one last example with a loop moving the data from memory into the file stream in chunks.  I'm willing to bet this outperforms trying to do it all in one shot.  This also has the advantage of not blowing up in the event that your image size exceeds an 'int' worth in size (since that is the max count you can specify to write out in one call to the stream).

You might also find the use of a BufferedStream on top of your file stream could potentially improve performance because it is supposed to be intelligent about when the calls go to the OS; but I've had mixed results.

MemoryStream ms = new MemoryStream(1500000);

BinaryWriter bw = new BinaryWriter(ms, Encoding.Unicode);



object source;



while ( source.HasMoreData() )

{

    bw.Write( source.getNextShortValue() );

}

bw.Flush();



string fName = Directory.GetCurrentDirectory();

fName = fName + @"/" + Convert.ToString(count) + @".txt";



FileStream myStream = new FileStream(fName, FileMode.Create);

BinaryReader br = new BinaryReader(ms, Encoding.Unicode);



br.BaseStream.Position = 0;

int pos = 0;

int chunkSize = 1024;

int readLength = 0;

byte[] dataChunk = new byte[chunkSize];

do

{

    readLength = br.Read(dataChunk, 0, chunkSize);

    if (readLength > 0)

    {

        myStream.Write(dataChunk, 0, readLength);

        // It may also may help performance to do a flush here

        myStream.Flush();

    }

} while ( readLength == chunkSize );



ms.Close();

bw.Close();

myStream.Close();

Open in new window

0
 

Author Comment

by:rangers99
ID: 33588151
Thanks for that. Bear with me folks while I try out these ideas
0
 

Author Closing Comment

by:rangers99
ID: 33615882
kaufmed, Your solution made a noticeable difference to the speed. Thanks

Volox, You 1st idea slowed things down by 3 times!
0
 
LVL 8

Expert Comment

by:Volox
ID: 33618924
Kind of figures with all the memory allocation and the iteration of the array for conversion; that's why after giving it some thought I posted the other two.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

This article is for Object-Oriented Programming (OOP) beginners. An Interface contains declarations of events, indexers, methods and/or properties. Any class which implements the Interface should provide the concrete implementation for each Inter…
Today I had a very interesting conundrum that had to get solved quickly. Needless to say, it wasn't resolved quickly because when we needed it we were very rushed, but as soon as the conference call was over and I took a step back I saw the correct …
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.
This video explains how to create simple products associated to Magento configurable product and offers fast way of their generation with Store Manager for Magento tool.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now