delimited text file to sql

I have a delimited text file and I'm putting it into sql.
the problem is that the text file is quite large and changes all the time.  I am building a C# service that will read the textfile and put all new records into the DB.  That works fine for the first time when I initially populate but my problem is keeping it light.  I need to compare via date what resides in the db and what resides in the textfile as to avoid duplicate records.  Is there an efficient way to take the 2 datatable's (one from sql and one from the text file) and do a compare and put the difference of the results into a 3rd datatable which will be for a bulk insert via dataAccessApplicationBlock.........
painlessprodAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

_TAD_Commented:


You mentioned a bulk insert, what about update?

Are you doing inserts only?  no updates, no deletes?



                  DataTable d1 = new DataTable(); // text file
                  DataTable d2 = new DataTable(); // data base
                  DataTable d3 = new DataTable(); // table for bulk insert

                  for(int i=0;i<d1.Rows.Count; i++)
                        if(!d2.Rows.Contains(d1.Rows[i].Table.Columns["PrimaryKey"]))
                              d3.Rows.Add(d1.Rows[i]);



...
..
.

bulk insert data from datatable 3
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
painlessprodAuthor Commented:
The problem is there's no primary key in the text file.  
There are no delete's, or updates.
there is a primary key for the sql
0
_TAD_Commented:


How do you determine unique-ness?

something must distinguish one row from another in your text file.

0
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

painlessprodAuthor Commented:
well, I guess the only thing that I could use would be the line# of the text file because the text file is appended to only at the end and I could store that # in the db
0
painlessprodAuthor Commented:
Err:

An unhandled exception of type 'System.ArgumentException' occurred in system.data.dll
Additional information: This row already belongs to another table.

//dt =file
//dtSQL =SQL DB
//dtComp = output


                  for(int j=0;j<dtSQL.Rows.Count; j++)
                  {
                        if(!TD.dt.Rows.Contains(dtSQL.Rows[j].Table.Columns["ROWNUMBER"]))
                        {
                              TD.dtComp.Rows.Add(TD.dt.Rows[j]);

                        }
                  }
0
_TAD_Commented:
                                                  DataRow r;

for(int i=0;i<d1.Rows.Count; i++)
                        if(!d2.Rows.Contains(d1.Rows[i].Table.Columns["PrimaryKey"]))
                        {
                              r = d3.NewRow();
                              for(int j=0;j<r.Table.Columns.Count;j++)
                                    r[j] = d1.Rows[i].Table.Columns[j];
                              d3.Rows.Add(r);
                        }




eh...


first you need to create a new row object, then you need to give it the schema of the table you want to add it to, finally load the values into the schema and then add that row onto the end of the data table you want
0
painlessprodAuthor Commented:
got it, thnx
0
painlessprodAuthor Commented:
_TAD_ I hate to continue bugging you but this one is a new one to me.

In order to do the contains statement I had to create primary keys for all three datatables.  I've done that
and now it says that the speicfied cast in no tvalid on this line
if(!dtSQL.Rows.Contains(TD.dt.Rows[i].Table.Columns["ROWNUMBER"]))


so i did this for all three datatables
MessageBox.Show(dtSQL.Columns["11"].DataType.ToString());
MessageBox.Show(TD.dt.Columns[11].DataType.ToString());
MessageBox.Show(TD.dtComp.Columns[11].DataType.ToString());

11 being the index of "ROWNUMBER"

and it returned system.int32 for all of them so I'm confused why it says that my cast is still not correct.


any help would be a great, thnks
0
painlessprodAuthor Commented:
Hello all,

I have 3 dataTables
One that comes directly from SQL
One that comes from a text file
and one that does the comparison between the 2 and populates itself with waht's not in sql but resides in the text file.

All three have an initialized Primary Key of System.Int32.

dt //textfile
dtSQL //sql
dtComp //difference

for(int i=0;i<TD.dt.Rows.Count; i++)
if(!TD.dt.Rows.Contains(dtSQL.Rows[i].Table.Columns["ROWNUMBER"])) // this is where I get the error
      {
      r = TD.dtComp.NewRow();
      for(int j=0;j<r.Table.Columns.Count;j++)
      r[j] = TD.dt.Rows[i].Table.Columns[j];
      TD.dtComp.Rows.Add(r);
}

0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C#

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.