Best way to save and import data in a visual studio 2008 application

Posted on 2011-02-18
Last Modified: 2012-08-14
Hi all

I'm looking for some advice on the best file format and import methods to use in my VB application. The sheer number of alternatives and the fact that I'm quite new to this type of programming is driving me nuts.

My application is business simulation that requires the administrator to input team decisions from printed forms. There are about 30 fields of data for each team and each team has a VB form to input the data. Once processed, the simulation generates reports and accounts for each team. This process is then repeated for each period in the simulation (up tom 12 periods)

I would like to store all the data for one simulation in one file and add to it when each period is played,b ut also keep the results from previous periods. I am thinking of storing the data from the VB forms in arrays as it is entered into the simulation software and then storing the results for each team into other arrays. These can then be displayed on the screen as tables. Then the whole lot would be written to a CSV file.

The CSV file would also contain summary data about that particular simulation (team names, current quarter, etc.) as well as the results of previous rounds.

My questions are:

Are arrays the best way to store the table variables?
Is a CSV file the best way to save this data to a file?
Does it matter in a CSV file if the rows do not all contain the same number of fields?
Would it be easier to use datagrids/tables instead of arrays?

If this is all too complicated, can anyone point me in the direction of a good article (or book) on how to process and file nlarge amounts of data.

I'd prefer not to use a database type application.

Any help greatly appreciated.


Question by:Terrygordon
  • 2
LVL 96

Expert Comment

by:Bob Learned
ID: 34931480
I would take a look at JSON Serialization.  Javascript Object Notation is a compact serialization syntax, that doesn't have the overhead that XML suffers from.  The amount of code to serialize and deserialize makes this a good choice.

JSON Serialization
LVL 27

Accepted Solution

MikeToole earned 500 total points
ID: 34933463
...Are arrays the best way to store the table variables?
Almost certainly not - if by that you mean to store 30 pieces of data as entries in any array and only know which is which by the value of the index.
Better to define object classes that describe the data more explicitly.  
A Simulation class could describe a simulation and contain a collection of Result class instances to store the data for a team for each period input.
The Result class would have 30 properties, one for each of the fields.
...Is a CSV file the best way to save this data to a file?
   Probably not, good as an interface/transfer mechanism, but too lightweight for this.
...Does it matter in a CSV file if the rows do not all contain the same number of fields?
   Yes, but you can leave fields blank (two consecutive commas)
...Would it be easier to use datagrids/tables instead of arrays?
No need for tables or datasets unless you're connecting to a backend database, and a datagrid is just the way to interact with data and isn't really connected to the storage method.

Using the object approach, the class instances are the in-memory store of data and acts as the source of data for display / analysis. These objects are then stored on disk as serialized versions.

You can use the .Net BinaryFormatter to convert an object to be stored into a file, I've attached a couple of routines that I used to Serialize/Unserialize objects.

 (JSON serialization (as suggested in the other response) would also probably do the job - I'm not familiar with it)

The approach I'd recommend is to define a class to describe the
   Assuming that the collection of fields always the same (with some possibly not filled-in) I would recommend
...There are about 30 fields of data for each team...

''' <summary>
        ''' Restore an object from a file containing the serialized version of it 
        ''' </summary>
        ''' <param name="Type">The data type for the current instance, used to construct the filename to store the serialized object</param>
        Friend Shared Function RestoreObject(ByVal [Type] As DataType) As Object
            With SaveFileInfo(Type)
                If .Exists Then
                    Dim binFmtr = New BinaryFormatter
                        Dim fStream As Stream = New FileStream(.FullName, FileMode.Open)
                        Dim obj As Object = binFmtr.Deserialize(fStream)
                        Return obj
                    Catch ex As Exception
                        Utility.LogError(System.Reflection.MethodBase.GetCurrentMethod(), "Unable to restore saved data: " + ex.Message)
                        Return Nothing
                    End Try
                    Return Nothing
                End If
            End With
        End Function
        ''' <summary>
        ''' Serialize object and save to a file
        ''' </summary>
        ''' <param name="obj">The collection of data items to be saved</param>
        ''' <param name="Type"></param>
        ''' <param name="FileDate">The LastWriteTime of the imported file</param>
        ''' <remarks></remarks>
        Friend Shared Sub SaveObject(ByVal obj As Object, ByVal [Type] As DataType, ByVal FileDate As Date)
            Dim SaveFile = SaveFileInfo(Type)
            Dim fStream As New FileStream(SaveFile.FullName, FileMode.Create)
            'Construct a BinaryFormatter to serialize the data to the stream. 
            Dim binFmtr As New BinaryFormatter
                binFmtr.Serialize(fStream, obj)

            Catch e As SerializationException
                Utility.LogError(System.Reflection.MethodBase.GetCurrentMethod(), e.Message)
                SaveFile.LastWriteTime = FileDate
            End Try
        End Sub

Open in new window


Author Comment

ID: 34934344
Thanks Mike. This makes things a lot clearer.

LVL 27

Expert Comment

ID: 34934459
Glad to help.
Sorry about the garbage paragraph at the end - it was work-in-progress and should've been deleted.

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

1.0 - Introduction Converting Visual Basic 6.0 (VB6) to Visual Basic 2008+ (VB.NET). If ever there was a subject full of murkiness and bad decisions, it is this one!   The first problem seems to be that people considering this task of converting…
If you need to start windows update installation remotely or as a scheduled task you will find this very helpful.
Need to grow your business through quality cloud solutions? With everything required to build a cloud platform and solution, you may feel like the distance between you and the cloud is quite long. Help is here. Spend some time learning about the Con…
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…

930 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now