Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Writing Random Data C# - Performance.

Posted on 2010-08-17
4
Medium Priority
?
535 Views
Last Modified: 2013-12-17
Hi,

Ok I have some code below that I would like to improve - it works but it runs very slowly.

What I am trying to do is write a program that generates a "random" data file of a
size determined by the user in GB which is chosen as a multiple. so if they
select 5.5 the 5.5 GB of random data is written.

It runs really slowly in the generate data phase and I guess the file write phase.

Obviously I am not doing something correct and it performs poorly. My guess is
List<Byte> file_data = new List<Byte>();

So I am interested to see if the performance can be improved or are my expectations unrealistic ? :)

Thanks,

Ward.

using System;
using System.Collections.Generic;
using System.Collections;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.IO;

namespace FileGen
{
    public partial class Form1 : Form
    {
        private string file_name;
        decimal total_file_size;

        public Form1()
        {
            InitializeComponent();
        }

        private void fnUpdate_Progress(decimal position)
        {

            if (position % 1048576 == 0)
            {
                decimal percentage = (position / total_file_size) * 100;
                percentage = Math.Round(percentage, 2);
                label_Position.Text = position.ToString("#,##0") + " / " + total_file_size.ToString("#,##0") + " (" + percentage.ToString() + "%)";
                progressBar_File.Value = (int)percentage;
            }

            Application.DoEvents();
        }

        private void fnCalculate_File_Size(decimal gb)
        {
            decimal gigabyte = 1073741824;

            decimal temp_file_size = gb * gigabyte;
            total_file_size = temp_file_size;

            string temp_format = temp_file_size.ToString("#,##0");

            label_FileSize.Text = temp_format + " bytes.";
            Application.DoEvents();

        }

        private void button_Exit_Click(object sender, EventArgs e)
        {
            Application.Exit();
        }

        private void button_Path_Click(object sender, EventArgs e)
        {

            SaveFileDialog saveFileDialog1 = new SaveFileDialog();

            saveFileDialog1.Filter = "dat files (*.dat)|*.dat|All files (*.*)|*.*";
            saveFileDialog1.FilterIndex = 2;
            saveFileDialog1.RestoreDirectory = true;

            if (saveFileDialog1.ShowDialog() == DialogResult.OK)
            {               
                file_name = saveFileDialog1.FileName;
                textBox_Path.Text = file_name;
                button_Generate.Enabled = true;
            }

        }

        private void Form1_Load(object sender, EventArgs e)
        {
            comboFileSize.SelectedIndex = 11;
            file_name = "";
            progressBar_File.Minimum = 0;
            progressBar_File.Maximum = 100;
            progressBar_File.Step = 1;
            progressBar_File.Value = 0;
            label_FileSize.Text = "";
            label_Status.Text = "";
            label_Position.Text = "";
            fnCalculate_File_Size(4);
        }

        private Byte RandomNumber(int min, int max)
        {
            Random random = new Random();
            return (Byte)random.Next(min, max);
        }

        private void button_Generate_Click(object sender, EventArgs e)
        {

            List<Byte> file_data = new List<Byte>();
            Byte data_byte;
            decimal file_pos;

            if (file_name == "")
            {
                return;
            }

            // ---------------------------- Very SLOW Here ------------------------ 

            label_Status.Text = "Generating data...";
            Application.DoEvents();

            for (file_pos = 1; file_pos <= total_file_size; file_pos++)
            {
                data_byte = RandomNumber(0, 255);
                file_data.Add(data_byte);
                fnUpdate_Progress(file_pos);
            }                     

            FileStream fs1 = new FileStream(file_name, FileMode.OpenOrCreate,
            FileAccess.Write);
            BinaryWriter bw = new BinaryWriter(fs1);

            label_Status.Text = "Writing data to file...";
            Application.DoEvents();

            file_pos = 1;

            // ---------------------- Probably SLOW Here? -----------------------------

            foreach (Byte b in file_data)
            {
                bw.Write(b);
                fnUpdate_Progress(file_pos);
                file_pos++;
            }
            
            bw.Close();
            fs1.Close();

            Application.Exit();
        }


        private void comboFileSize_SelectedValueChanged(object sender, EventArgs e)
        {
            string gb_size;
            decimal gb;

            gb_size = (string)comboFileSize.SelectedItem;

            gb = Convert.ToDecimal(gb_size);

            fnCalculate_File_Size(gb);
        }
    }
}

Open in new window

0
Comment
Question by:whorsfall
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 64

Accepted Solution

by:
Fernando Soto earned 2000 total points
ID: 33456959
Hi whorsfall;

I have modified your code to get better performance. Each time you make a function call you take a small hit but making the number of function calls to the random function and progress changing give a very big performance hit. Also waiting to the end to write the data to the file is not a good idea because the amount of memory needed to hold it. So getting a chunk of data and writing a chunk of data will increase performance. Also checking progress in the loop that creates the data and writes the data is better then making a call to find out that it is not needed yet will also slow you down a bit.

Fernando
using System;
using System.Collections.Generic;
using System.Collections;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.IO;

namespace FileGen
{
    public partial class Form1 : Form
    {
        private string file_name;
        decimal total_file_size;

        public Form1()
        {
            InitializeComponent();
        }

        private void fnUpdate_Progress(decimal position)
        {

            decimal percentage = (position / total_file_size) * 100;
            percentage = Math.Round(percentage, 2);
            label_Position.Text = position.ToString("#,##0") + " / " + total_file_size.ToString("#,##0") + " (" + percentage.ToString() + "%)";
            progressBar_File.Value = (int)percentage;

            Application.DoEvents();
        }

        private void fnCalculate_File_Size(decimal gb)
        {
            decimal gigabyte = 1073741824;

            decimal temp_file_size = gb * gigabyte;
            total_file_size = temp_file_size;

            string temp_format = temp_file_size.ToString("#,##0");

            label_FileSize.Text = temp_format + " bytes.";
            Application.DoEvents();

        }

        private void button_Exit_Click(object sender, EventArgs e)
        {
            Application.Exit();
        }

        private void button_Path_Click(object sender, EventArgs e)
        {

            SaveFileDialog saveFileDialog1 = new SaveFileDialog();

            saveFileDialog1.Filter = "dat files (*.dat)|*.dat|All files (*.*)|*.*";
            saveFileDialog1.FilterIndex = 2;
            saveFileDialog1.RestoreDirectory = true;

            if (saveFileDialog1.ShowDialog() == DialogResult.OK)
            {
                file_name = saveFileDialog1.FileName;
                textBox_Path.Text = file_name;
                button_Generate.Enabled = true;
            }

        }

        private void Form1_Load(object sender, EventArgs e)
        {
            comboFileSize.SelectedIndex = 11;
            file_name = "";
            progressBar_File.Minimum = 0;
            progressBar_File.Maximum = 100;
            progressBar_File.Step = 1;
            progressBar_File.Value = 0;
            label_FileSize.Text = "";
            label_Status.Text = "";
            label_Position.Text = "";
            fnCalculate_File_Size(4);
        }

        private List<Byte> RandomNumber(int min, int max)
        {
            Random random = new Random();
            List<Byte> randList = new List<byte>();
            for (int r = 0; r < 4097; r++)
            {
                randList.Add((Byte)random.Next(min, max));
            }
            return randList;
        }

        private void button_Generate_Click(object sender, EventArgs e)
        {

            List<Byte> file_data = new List<Byte>();
            List<Byte> data_byte;
            decimal file_pos;

            if (file_name == "")
            {
                return;
            }

            // ---------------------------- Very SLOW Here ------------------------ 

            label_Status.Text = "Generating data and Writing data to file...";
            Application.DoEvents();

            FileStream fs1 = new FileStream(file_name, FileMode.OpenOrCreate,
            FileAccess.Write);
            BinaryWriter bw = new BinaryWriter(fs1);
            file_pos = 0;
            do
            {
                data_byte = RandomNumber(0, 255);
                if (file_pos + data_byte.Count <= total_file_size)
                {
                    file_data.AddRange(data_byte);
                    file_pos += data_byte.Count;
                }
                else
                {
                    if (file_pos < total_file_size)
                    {
                        file_data.AddRange(data_byte.Take((int)(total_file_size - file_pos)));
                        file_pos += total_file_size - file_pos;
                    }
                }
                bw.Write(file_data.ToArray());
                file_data.Clear();
                if( file_pos % 10 == 0 ) fnUpdate_Progress(file_pos);
            } while (file_pos <= total_file_size);

            bw.Close();
            fs1.Close();

            label_Status.Text = "File and data successfully created";

            Application.Exit();
        }


        private void comboFileSize_SelectedValueChanged(object sender, EventArgs e)
        {
            string gb_size;
            decimal gb;

            gb_size = (string)comboFileSize.SelectedItem;

            gb = Convert.ToDecimal(gb_size);

            fnCalculate_File_Size(gb);
        }
    }
}

Open in new window

0
 
LVL 64

Expert Comment

by:Fernando Soto
ID: 33456986
You will need to check the limits on file size and chunk sizes used to make sure that all adds up.
0
 
LVL 6

Expert Comment

by:r3nder
ID: 33457053
try switching loops - use LINQ TakeAny();
//17%  slower for average sized collections using any()
// find implemented via LINQ
    public static bool FindViaLinq(IEnumerable<int> list, int target)
    {
        return list.Any(item => item == target);
    }
 
 
    // find implemented via standard iteration
    public static bool FindViaIteration(IEnumerable<int> list, int target)
    {
        foreach (var i in list)
        {
            if (i == target)
            {
                return true;
            }
        }
 
        return false;
    }


//90% slower using takewhile()
// Linq form
    public static int GetTargetPosition1(IEnumerable<int> list, int target)
    {
        return list.TakeWhile(item => item != target).Count();
    }
 
    // traditionally iterative form
    public static int GetTargetPosition2(IEnumerable<int> list, int target)
    {
        int count = 0;
 
        foreach (var i in list)
        {
            if(i == target)
            {
                break;
            }
 
            ++count;
        }
 
        return count;
    }
//Much faster  
// a newer method that uses linq but evaluates the count in a closure.
    public static int TakeWhileViaLinq2(IEnumerable<int> list, int target)
    {
        int count = 0;
        list.Any(item =>
            {
                if(item == target)
                {
                    return true;
                }
 
                ++count;
                return false;
            });
        return count;
    }

Open in new window

0
 
LVL 10

Expert Comment

by:Najam Uddin
ID: 33462847
if your requirement is just a file of specified size, without any concern of its content, than you can use
"fsutil file createnew C:\1mbfile.txt 1000000

ie fsutil file createnew <name of file> <size in bytes>
0

Featured Post

How to Use the Help Bell

Need to boost the visibility of your question for solutions? Use the Experts Exchange Help Bell to confirm priority levels and contact subject-matter experts for question attention.  Check out this how-to article for more information.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

We all know that functional code is the leg that any good program stands on when it comes right down to it, however, if your program lacks a good user interface your product may not have the appeal needed to keep your customers happy. This issue can…
This article is for Object-Oriented Programming (OOP) beginners. An Interface contains declarations of events, indexers, methods and/or properties. Any class which implements the Interface should provide the concrete implementation for each Inter…
Video by: ITPro.TV
In this episode Don builds upon the troubleshooting techniques by demonstrating how to properly monitor a vSphere deployment to detect problems before they occur. He begins the show using tools found within the vSphere suite as ends the show demonst…
In response to a need for security and privacy, and to continue fostering an environment members can turn to for support, solutions, and education, Experts Exchange has created anonymous question capabilities. This new feature is available to our Pr…

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question