Solved

database import from csv file

Posted on 2006-06-22
3
259 Views
Last Modified: 2012-05-05
I have a program that grabs information from a comma delimited file and imports it into an sqlite database. Problem I have now is that the same code that I had been using doesn't work correctly when the fields are blank. The last 2 columns of my csv file only has a few records with info so out of 350 records only 5 of the records may have data in them. the data file would look something like the following:

10014,Doe,John,1/4/1990,8,M,H,100,5/2/2005,1,F,,1
43521,Doe,Jane,7/23/1991,8,M,H,150,3/8/2005,1,F,H

How can I make sure that this file gets imported regardless of blank columns or even write n/a into the columns if they are blank? I just need to be ably to import this file in regardless if some fields are blank or not. Also, this file has no header row.
Below is the code I am currently using to do this.

Private Sub ImportFile(ByVal filename As String, ByVal SQLTable As String)
        Dim fso As New System.IO.StreamReader(filename)
        Dim line As String

        Try
            If Not fso Is Nothing Then
                Do
                    line = fso.ReadLine()
                    DoSql(line, SQLTable)
                Loop Until line Is Nothing
            End If
        Catch ex As Exception
            Trace.WriteLine(ex.Message)
        End Try

    End Sub

    Private Sub DoSql(ByVal fields As String, ByVal SQLTable As String)
        Dim sql_command As String
        Dim value_string As String

        value_string = ""
        For Each item As String In fields.Split(CChar(","))
            value_string &= "'" & item & "'" & ","
        Next

        'remove the last comma
        value_string = value_string.Substring(0, value_string.Length - 1)

        sql_command = String.Format("INSERT INTO " & SQLTable & " VALUES(" & value_string & ")")
        Trace.WriteLine(sql_command)

        Try
            'Execute the SQL command here
            '
        Catch ex As Exception
            'trap syntax errors
            Trace.WriteLine(ex.Message)
        End Try
    End Sub

Thank you
0
Comment
Question by:heydude
3 Comments
 
LVL 96

Expert Comment

by:Bob Learned
ID: 16964210
What happens when the field is blank?  What is missing?

Bob
0
 
LVL 34

Accepted Solution

by:
Sancler earned 350 total points
ID: 16964873
The Microsoft.Jet.OLEDB.4.0 driver for delimited text files works (in the absence of a schema.ini) by scanning the first few rows of the source file to determine how many columns there are and what the field types are.  I think the number of rows scanned can be altered, but I'd have to look up how to do that.  But I've just tested on a dummy file and with that (without me remembering ever having altered anything) the break point came at 25 rows.  Your are not bothered about datatypes, but you are bothered about the number of columns.  So provided there is some entry in the last two columns somewhere in the first 25 rows of your csv source file this should work.

    Private Function GetCSVTable(ByVal DirName As String, ByVal FileName As String) As DataTable
        Dim ConStr As String = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & DirName & ";Extended Properties=""text;HDR=No;FMT=Delimited"""
        Dim con As New OleDbConnection(ConStr)
        Dim sql As String = "SELECT * FROM " & FileName
        Dim da As New OleDbDataAdapter(sql, con)
        Dim dt As New DataTable
        da.Fill(dt)
        Return dt
    End Function

I've just tested it and empty columns in any particular source row were represented by "[Null]" in the datagrid I used for my test: it may be DBNull in the datatable, I didn't check.

But this approach would give you discrete rows of data that you could then transfer to your SQL database, checking for nulls and reacting accordingly as you did so.

Roger
0
 

Author Comment

by:heydude
ID: 16965407
I actually fixed it, I'm not sure why, but I added a dummy column to my table and it worked. All records imported in no problem. Strange. I'm giving Sancler the points for his explanation and code example that I can use for other programs.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Copy/Clone an object. 9 30
Visual Studio .Net 2012 and Crystal Reports 9 46
Receiving a string from a WebService Push 21 36
Advice in Xamarin 21 56
This tutorial demonstrates one way to create an application that runs without any Forms but still has a GUI presence via an Icon in the System Tray. The magic lies in Inheriting from the ApplicationContext Class and passing that to Application.Ru…
Microsoft Reports are based on a report definition, which is an XML file that describes data and layout for the report, with a different extension. You can create a client-side report definition language (*.rdlc) file with Visual Studio, and build g…
This Micro Tutorial will give you a basic overview how to record your screen with Microsoft Expression Encoder. This program is still free and open for the public to download. This will be demonstrated using Microsoft Expression Encoder 4.
Windows 10 is mostly good. However the one thing that annoys me is how many clicks you have to do to dial a VPN connection. You have to go to settings from the start menu, (2 clicks), Network and Internet (1 click), Click VPN (another click) then fi…

910 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

25 Experts available now in Live!

Get 1:1 Help Now