Solved

delimited text file to sql

Posted on 2003-11-03
9
393 Views
Last Modified: 2010-04-16
I have a delimited text file and I'm putting it into sql.
the problem is that the text file is quite large and changes all the time.  I am building a C# service that will read the textfile and put all new records into the DB.  That works fine for the first time when I initially populate but my problem is keeping it light.  I need to compare via date what resides in the db and what resides in the textfile as to avoid duplicate records.  Is there an efficient way to take the 2 datatable's (one from sql and one from the text file) and do a compare and put the difference of the results into a 3rd datatable which will be for a bulk insert via dataAccessApplicationBlock.........
0
Comment
Question by:painlessprod
  • 6
  • 3
9 Comments
 
LVL 22

Accepted Solution

by:
_TAD_ earned 205 total points
ID: 9673691


You mentioned a bulk insert, what about update?

Are you doing inserts only?  no updates, no deletes?



                  DataTable d1 = new DataTable(); // text file
                  DataTable d2 = new DataTable(); // data base
                  DataTable d3 = new DataTable(); // table for bulk insert

                  for(int i=0;i<d1.Rows.Count; i++)
                        if(!d2.Rows.Contains(d1.Rows[i].Table.Columns["PrimaryKey"]))
                              d3.Rows.Add(d1.Rows[i]);



...
..
.

bulk insert data from datatable 3
0
 

Author Comment

by:painlessprod
ID: 9673734
The problem is there's no primary key in the text file.  
There are no delete's, or updates.
there is a primary key for the sql
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 9673853


How do you determine unique-ness?

something must distinguish one row from another in your text file.

0
 

Author Comment

by:painlessprod
ID: 9673886
well, I guess the only thing that I could use would be the line# of the text file because the text file is appended to only at the end and I could store that # in the db
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 

Author Comment

by:painlessprod
ID: 9681621
Err:

An unhandled exception of type 'System.ArgumentException' occurred in system.data.dll
Additional information: This row already belongs to another table.

//dt =file
//dtSQL =SQL DB
//dtComp = output


                  for(int j=0;j<dtSQL.Rows.Count; j++)
                  {
                        if(!TD.dt.Rows.Contains(dtSQL.Rows[j].Table.Columns["ROWNUMBER"]))
                        {
                              TD.dtComp.Rows.Add(TD.dt.Rows[j]);

                        }
                  }
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 9682096
                                                  DataRow r;

for(int i=0;i<d1.Rows.Count; i++)
                        if(!d2.Rows.Contains(d1.Rows[i].Table.Columns["PrimaryKey"]))
                        {
                              r = d3.NewRow();
                              for(int j=0;j<r.Table.Columns.Count;j++)
                                    r[j] = d1.Rows[i].Table.Columns[j];
                              d3.Rows.Add(r);
                        }




eh...


first you need to create a new row object, then you need to give it the schema of the table you want to add it to, finally load the values into the schema and then add that row onto the end of the data table you want
0
 

Author Comment

by:painlessprod
ID: 9682365
got it, thnx
0
 

Author Comment

by:painlessprod
ID: 9683300
_TAD_ I hate to continue bugging you but this one is a new one to me.

In order to do the contains statement I had to create primary keys for all three datatables.  I've done that
and now it says that the speicfied cast in no tvalid on this line
if(!dtSQL.Rows.Contains(TD.dt.Rows[i].Table.Columns["ROWNUMBER"]))


so i did this for all three datatables
MessageBox.Show(dtSQL.Columns["11"].DataType.ToString());
MessageBox.Show(TD.dt.Columns[11].DataType.ToString());
MessageBox.Show(TD.dtComp.Columns[11].DataType.ToString());

11 being the index of "ROWNUMBER"

and it returned system.int32 for all of them so I'm confused why it says that my cast is still not correct.


any help would be a great, thnks
0
 

Author Comment

by:painlessprod
ID: 9688613
Hello all,

I have 3 dataTables
One that comes directly from SQL
One that comes from a text file
and one that does the comparison between the 2 and populates itself with waht's not in sql but resides in the text file.

All three have an initialized Primary Key of System.Int32.

dt //textfile
dtSQL //sql
dtComp //difference

for(int i=0;i<TD.dt.Rows.Count; i++)
if(!TD.dt.Rows.Contains(dtSQL.Rows[i].Table.Columns["ROWNUMBER"])) // this is where I get the error
      {
      r = TD.dtComp.NewRow();
      for(int j=0;j<r.Table.Columns.Count;j++)
      r[j] = TD.dt.Rows[i].Table.Columns[j];
      TD.dtComp.Rows.Add(r);
}

0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
This video discusses moving either the default database or any database to a new volume.
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now