• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 522
  • Last Modified:

C#.NET. How Do I save an array of integers VERY Fast

Hi

Im developing an application using Visual Studio 2008 and C#.NET

Im reading images from a camera and I need to save them to hard disk as quickly as possible.
The image array is defined thus. It contains 1.5M shorts

short[] image = new short[1500000];

Im currently using the code below to save each image. However its not fast enough. A bottleneck occurs after a few dozen  images.

Ive found some articles on the Internet that say the BinaryFormatter class is inefficient.

Has anyone had a similar problem or have an idea how I can speed things up. Im not really interested in getting the file size small (although if thats a side effect all the better) I just want the array to be saved to disk as quickly as possible.


Thanks.
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter  bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
            int count = 0;

            string fName = Directory.GetCurrentDirectory();
            fName  = fName + @"/" + Convert.ToString(count) + @".txt";

                Stream myStream = new FileStream(fName, FileMode.Create);
                bf.Serialize(myStream, numbers);
                myStream.Close();

Open in new window

0
rangers99
Asked:
rangers99
  • 6
  • 4
  • 2
  • +2
3 Solutions
 
StephanLead Software EngineerCommented:
Do you want to save this from a fileupload or directory?

What kind of extension needs it to be? Just .txt?
0
 
w00teCommented:
Check this out:
http://www.codeproject.com/KB/files/fastbinaryfileinput.aspx
Near the 2nd half of the article they have an interesting pattern for rapid writing involving converting into byte buffers first.  Worth a try :)
-w00te
0
 
rangers99Author Commented:
Hi stephan

Just assume there is an array with 1.5 million integers and start from there.
As for file extension it doesnt matter. Im thinking a .txt file wont be suitable because the file would have to be in binary format.
0
Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

 
käµfm³d 👽Commented:
Have you considered multithreading the app? You could use the ThreadPool class to queue work--in this case, writing the data to disk. If all you are doing is writing the files to disk and there is no chance of other code in your app (or other apps) from touching the files while they are being written, then I believe you could get away with something as simple as the following, where CaptureMethod is simulating your code which receives the image from the camera.
using System;
using System.IO;
using System.Threading;
using System.Windows.Forms;

namespace WindowsFormsApplication3
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void CaptureMethod()
        {
            short[] image = new short[1500000];
            ImageData data;     // Passed to the worker method

            for (int i = 0; i < 15; i++)
            {
                data.count = i + 1;
                data.image = image;
                ThreadPool.QueueUserWorkItem(WriteFile, data);  // Send the writes to the queue
            }
        }

        /// <summary>
        /// Worker method
        /// </summary>
        /// <param name="imageData">Object which holds the image-related data</param>
        private void WriteFile(object imageData)
        {
            ImageData data = (ImageData)imageData;
            System.Runtime.Serialization.Formatters.Binary.BinaryFormatter bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();

            string fName = Directory.GetCurrentDirectory();
            fName = fName + @"/" + Convert.ToString(data.count) + @".txt";

            using (Stream myStream = new FileStream(fName, FileMode.Create))
            {
                bf.Serialize(myStream, data.image);
            }
        }

        // Just simulating the camera here
        private void button1_Click(object sender, EventArgs e)
        {
            CaptureMethod();
        }
    }

    /// <summary>
    /// A struct to hold the actual image data and it's sequence number
    /// </summary>
    struct ImageData
    {
        public int count;
        public short[] image;
    }
}

Open in new window

0
 
rangers99Author Commented:
Thanks for that Kaufmed. Its something I will definetly try out if I can't get another solution.

I already have a separate thread that grabs the images from the camera and reads them into a buffer in my C# code. So Im thinking another thread could be one too many (I rarely develop multithreading applications).
0
 
rangers99Author Commented:
woote. Thanks for the link. However I cant see the bit where the array is written to hard disk. The binary data is just written to an array of bytes.
0
 
käµfm³d 👽Commented:
>>  So Im thinking another thread could be one too many

Depending on how old your machine is I wouldn't think you'd have to worry about it that much if you (at present), only have two threads. ThreadPool is already allocated for you (you'll notice that QueueUserWorkItem is a static call) and manages the creation and cleanup of the threads it deals with.
0
 
rangers99Author Commented:
This article states that serialization is very inefficent in C#.NET using the binaryFormatter class.
http://www.codeproject.com/KB/dotnet/FastSerializer.aspx

Its not clear to me whether I can use this code to make saving an array of integers (using serialisation) to a hard disk much faster than using the BinaryFormatter class.
0
 
VoloxCommented:
I'm not sure exactly how it compares speed wise because but you might try converting the shorts over to chars and then writing them out as unicode using a BinaryWriter.  Like this...
short[] image = new short[1500000];
            
char[] chars = Array.ConvertAll<short, char>(image, b => (char)b);

string fName = Directory.GetCurrentDirectory();
fName = fName + @"/" + Convert.ToString(count) + @".txt";

Stream myStream = new FileStream(fName, FileMode.Create);
BinaryWriter bw = new BinaryWriter(myStream, Encoding.Unicode);
bw.Write(chars);
bw.Flush();
bw.Close();
myStream.Close();

Open in new window

0
 
VoloxCommented:
I would point out that if you can perform the conversion to chars on the input side then you save what is probably an expensive operation since the ConvertAll not only has to convert every value but it also has to allocat another 1500000 worth of array.  If your in-memory object is char rather than short and you convert on the input side, then you save the memory space and the array iterration.

The other performance techinique you might try is breaking up the array so that it is an array of arrays.  Make your sub-arrays sized to match memory / disk allocation sizes - so 512 each or 1024 each etc.  Then you write out the arrays one by one.  You might spin more but sometimes the stream objects and their underlying buffers work more effeciently when handed more reasonable sized chunks.  Think about it like trying to write data to a database (wihtout concern for network overhead)... one transaction with a million rows versus a thousand transactions with a thousand rows.  There is a balance in there where the underworkings start to struggle to keep track of the amount of data that is going to be committed.

The last techinique I might suggest is this... when you read the data, put it into a memory stream instead of into an array.  Then you should be able to just stream it from the memory stream into a file in it's binary form.
MemoryStream ms = new MemoryStream(1500000);
BinaryWriter bw = new BinaryWriter(ms, Encoding.Unicode);

object source;

while ( source.HasMoreData() )
{
    bw.Write( source.getNextShortValue() );
}
bw.Flush();

string fName = Directory.GetCurrentDirectory();
fName = fName + @"/" + Convert.ToString(count) + @".txt";

FileStream myStream = new FileStream(fName, FileMode.Create);

myStream.Write(ms.GetBuffer(), 0, (int)ms.Length);

ms.Close();
bw.Close();
myStream.Close();

Open in new window

0
 
VoloxCommented:
Here is one last example with a loop moving the data from memory into the file stream in chunks.  I'm willing to bet this outperforms trying to do it all in one shot.  This also has the advantage of not blowing up in the event that your image size exceeds an 'int' worth in size (since that is the max count you can specify to write out in one call to the stream).

You might also find the use of a BufferedStream on top of your file stream could potentially improve performance because it is supposed to be intelligent about when the calls go to the OS; but I've had mixed results.

MemoryStream ms = new MemoryStream(1500000);
BinaryWriter bw = new BinaryWriter(ms, Encoding.Unicode);

object source;

while ( source.HasMoreData() )
{
    bw.Write( source.getNextShortValue() );
}
bw.Flush();

string fName = Directory.GetCurrentDirectory();
fName = fName + @"/" + Convert.ToString(count) + @".txt";

FileStream myStream = new FileStream(fName, FileMode.Create);
BinaryReader br = new BinaryReader(ms, Encoding.Unicode);

br.BaseStream.Position = 0;
int pos = 0;
int chunkSize = 1024;
int readLength = 0;
byte[] dataChunk = new byte[chunkSize];
do
{
    readLength = br.Read(dataChunk, 0, chunkSize);
    if (readLength > 0)
    {
        myStream.Write(dataChunk, 0, readLength);
        // It may also may help performance to do a flush here
        myStream.Flush();
    }
} while ( readLength == chunkSize );

ms.Close();
bw.Close();
myStream.Close();

Open in new window

0
 
rangers99Author Commented:
Thanks for that. Bear with me folks while I try out these ideas
0
 
rangers99Author Commented:
kaufmed, Your solution made a noticeable difference to the speed. Thanks

Volox, You 1st idea slowed things down by 3 times!
0
 
VoloxCommented:
Kind of figures with all the memory allocation and the iteration of the array for conversion; that's why after giving it some thought I posted the other two.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: MCSA MCSE Windows Server 2012

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

  • 6
  • 4
  • 2
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now