Best way to save and import data in a visual studio 2008 application

Hi all

I'm looking for some advice on the best file format and import methods to use in my VB application. The sheer number of alternatives and the fact that I'm quite new to this type of programming is driving me nuts.

My application is business simulation that requires the administrator to input team decisions from printed forms. There are about 30 fields of data for each team and each team has a VB form to input the data. Once processed, the simulation generates reports and accounts for each team. This process is then repeated for each period in the simulation (up tom 12 periods)

I would like to store all the data for one simulation in one file and add to it when each period is played,b ut also keep the results from previous periods. I am thinking of storing the data from the VB forms in arrays as it is entered into the simulation software and then storing the results for each team into other arrays. These can then be displayed on the screen as tables. Then the whole lot would be written to a CSV file.

The CSV file would also contain summary data about that particular simulation (team names, current quarter, etc.) as well as the results of previous rounds.

My questions are:

Are arrays the best way to store the table variables?
Is a CSV file the best way to save this data to a file?
Does it matter in a CSV file if the rows do not all contain the same number of fields?
Would it be easier to use datagrids/tables instead of arrays?

If this is all too complicated, can anyone point me in the direction of a good article (or book) on how to process and file nlarge amounts of data.

I'd prefer not to use a database type application.

Any help greatly appreciated.


Who is Participating?
MikeTooleConnect With a Mentor Commented:
...Are arrays the best way to store the table variables?
Almost certainly not - if by that you mean to store 30 pieces of data as entries in any array and only know which is which by the value of the index.
Better to define object classes that describe the data more explicitly.  
A Simulation class could describe a simulation and contain a collection of Result class instances to store the data for a team for each period input.
The Result class would have 30 properties, one for each of the fields.
...Is a CSV file the best way to save this data to a file?
   Probably not, good as an interface/transfer mechanism, but too lightweight for this.
...Does it matter in a CSV file if the rows do not all contain the same number of fields?
   Yes, but you can leave fields blank (two consecutive commas)
...Would it be easier to use datagrids/tables instead of arrays?
No need for tables or datasets unless you're connecting to a backend database, and a datagrid is just the way to interact with data and isn't really connected to the storage method.

Using the object approach, the class instances are the in-memory store of data and acts as the source of data for display / analysis. These objects are then stored on disk as serialized versions.

You can use the .Net BinaryFormatter to convert an object to be stored into a file, I've attached a couple of routines that I used to Serialize/Unserialize objects.

 (JSON serialization (as suggested in the other response) would also probably do the job - I'm not familiar with it)

The approach I'd recommend is to define a class to describe the
   Assuming that the collection of fields always the same (with some possibly not filled-in) I would recommend
...There are about 30 fields of data for each team...

''' <summary>
        ''' Restore an object from a file containing the serialized version of it 
        ''' </summary>
        ''' <param name="Type">The data type for the current instance, used to construct the filename to store the serialized object</param>
        Friend Shared Function RestoreObject(ByVal [Type] As DataType) As Object
            With SaveFileInfo(Type)
                If .Exists Then
                    Dim binFmtr = New BinaryFormatter
                        Dim fStream As Stream = New FileStream(.FullName, FileMode.Open)
                        Dim obj As Object = binFmtr.Deserialize(fStream)
                        Return obj
                    Catch ex As Exception
                        Utility.LogError(System.Reflection.MethodBase.GetCurrentMethod(), "Unable to restore saved data: " + ex.Message)
                        Return Nothing
                    End Try
                    Return Nothing
                End If
            End With
        End Function
        ''' <summary>
        ''' Serialize object and save to a file
        ''' </summary>
        ''' <param name="obj">The collection of data items to be saved</param>
        ''' <param name="Type"></param>
        ''' <param name="FileDate">The LastWriteTime of the imported file</param>
        ''' <remarks></remarks>
        Friend Shared Sub SaveObject(ByVal obj As Object, ByVal [Type] As DataType, ByVal FileDate As Date)
            Dim SaveFile = SaveFileInfo(Type)
            Dim fStream As New FileStream(SaveFile.FullName, FileMode.Create)
            'Construct a BinaryFormatter to serialize the data to the stream. 
            Dim binFmtr As New BinaryFormatter
                binFmtr.Serialize(fStream, obj)

            Catch e As SerializationException
                Utility.LogError(System.Reflection.MethodBase.GetCurrentMethod(), e.Message)
                SaveFile.LastWriteTime = FileDate
            End Try
        End Sub

Open in new window

Bob LearnedCommented:
I would take a look at JSON Serialization.  Javascript Object Notation is a compact serialization syntax, that doesn't have the overhead that XML suffers from.  The amount of code to serialize and deserialize makes this a good choice.

JSON Serialization
TerrygordonAuthor Commented:
Thanks Mike. This makes things a lot clearer.

Glad to help.
Sorry about the garbage paragraph at the end - it was work-in-progress and should've been deleted.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.