Advertisement

06.13.2008 at 08:38AM PDT, ID: 23483099
[x]
Attachment Details

How do I make working SqlBulkCopy in VB.NET?

Asked by mjasic in .NET, Visual Studio

Tags: ,

We have a txt file (size around 2 G) that loads very slowly > 8 hrs. So, we decided to use bulk copy to accelerate the process.

This txt file has some special headers (the second line contains the actual column headers) and one line of footer.

First, I don't know how to get rid of the first line of header and the last one (footer) in StreamReader.

Second, when I cut these lines manually (it is not easy with 2 G so in my test I used only 500 lines) and process,
there is an error message stating as follows:
System.InvalidOperationException was unhandled
  Message="The given ColumnName 'BIN_LOC' does not match up with any column in data source."
  Source="System.Data"

The column BIN_LOC exists in my destination file!

What is wrong with the code and how to fix the problem of header/footer?
Thanks for your time.

MikeStart Free Trial
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
Sub BulkImport()
 
        Const strDestTable As String = "[SCM-OBJ-12_input_im_ftn]"
        Const strFilename As String = "C:\Mydata.txt"
 
        Dim i As Long = 0
        Dim dbConn As SqlConnection = New SqlConnection("server=MyServer;uid=MyName;pwd=MyPass;database=MyDB")
        Dim sr As StreamReader = New StreamReader(strFilename)
        Dim line As String = sr.ReadLine()
        Dim strArray As String() = line.Split("~")
 
 
        Dim dt As DataTable = New DataTable()
        Dim row As DataRow
 
        For Each s As String In strArray
            dt.Columns.Add(New DataColumn())
        Next
 
        Do
            row = dt.NewRow()
            row.ItemArray = line.Split("~")
            dt.Rows.Add(row)
            i = i + 1
            line = sr.ReadLine()
 
        Loop While Not line = String.Empty
 
        Dim bc As SqlBulkCopy = New SqlBulkCopy(dbConn, SqlBulkCopyOptions.TableLock, Nothing)
        bc.DestinationTableName = strDestTable
 
        bc.BatchSize = dt.Rows.Count
        bc.ColumnMappings.Add("BIN_LOC", "BIN_LOC")
        bc.ColumnMappings.Add("MATL_TYPE_IN", "MATL_TYPE_IN")
        bc.ColumnMappings.Add("MFG_PART_NO", "MFG_PART_NO")
        bc.ColumnMappings.Add("LOC_PRM_IN", "LOC_PRM_IN")
        bc.ColumnMappings.Add("LOC_SCND", "LOC_SCND")
        bc.ColumnMappings.Add("CUST_ID_IN", "CUST_ID_IN")
        bc.ColumnMappings.Add("CO_NO", "CO_NO")
        bc.ColumnMappings.Add("MFR_PART_NBR", "MFR_PART_NBR")
 
        dbConn.Open()
        bc.WriteToServer(dt)
        dbConn.Close()
        bc.Close()
End Sub
 
--------------------------------------------------------------------
Here is a sample of the beginning of the file:
 
SCM-OBJ-12~SCM-OBJ-12_IM_FTN_INPUT~2008-06-10~0509
BIN_LOC~MATL_TYPE_IN~MFG_PART_NO~LOC_PRM_IN~LOC_SCND~CUST_ID_IN~CO_NO~MFR_PART_NBR~FTN_NBR~MATL_TYPE_FTN~STORG_LOC_ID~FTN_NBR_LAST~RECD_DT~SVC_DT~MFG_CODE~OH_PRICED_QTY~DEFAULT_UPR~MFR_CD~LOC_PRM_FTN~PME_NBR~CUST_ID_FTN~MTC_EVT_NM~RTN_FOR_INSPEC_DT~SER_NBR~MFR_DT~FTN_STS_CD~SER_NBR_TYP_CD~ITEM_CLASS_CODE~ITT_CUST_ID~ITT_MATL_TYPE~ITT_LOC_PRM~ITT_STORG_LOC_ID
                  ~  ~3555111-4                 ~     ~          ~  ~PP1 ~3555111-7                 ~R8Z0L1~  ~177-22 11 ~      ~20070307~        ~99193~ 000000000.0000~ 00000000.01000~99193~ABC  ~               ~PP~               ~        ~    060122212409~        ~ANS~M~0702~  ~  ~     ~
[+][-]06.14.2008 at 05:57AM PDT, ID: 21785170

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06.16.2008 at 06:36AM PDT, ID: 21793411

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06.16.2008 at 06:42AM PDT, ID: 21793460

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06.16.2008 at 06:56AM PDT, ID: 21793623

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06.16.2008 at 07:18AM PDT, ID: 21793816

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06.16.2008 at 07:30AM PDT, ID: 21793939

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06.16.2008 at 07:40AM PDT, ID: 21794055

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 7-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]06.16.2008 at 07:44AM PDT, ID: 21794086

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 7-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]06.20.2008 at 03:52PM PDT, ID: 21835484

View this solution now by starting your 7-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

 

About this solution

Zones: .NET, Visual Studio
Tags: VB.NET, System.InvalidOperationException was unhandled - Message="The given ColumnName 'BIN_LOC' does not match up with any column in data source."
Sign Up Now!
Solution Provided By: mjasic
Participating Experts: 2
Solution Grade: A
 
 
 
Loading Advertisement...
20080716-EE-VQP-32 / EE_QW_2_20070628