asked on

delimited text file to sql

I have a delimited text file and I'm putting it into sql.
the problem is that the text file is quite large and changes all the time. I am building a C# service that will read the textfile and put all new records into the DB. That works fine for the first time when I initially populate but my problem is keeping it light. I need to compare via date what resides in the db and what resides in the textfile as to avoid duplicate records. Is there an efficient way to take the 2 datatable's (one from sql and one from the text file) and do a compare and put the difference of the results into a 3rd datatable which will be for a bulk insert via dataAccessApplicationBlock.........

ASKER CERTIFIED SOLUTION

_TAD_

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

painlessprod

ASKER

The problem is there's no primary key in the text file.
There are no delete's, or updates.
there is a primary key for the sql

_TAD_

How do you determine unique-ness?

something must distinguish one row from another in your text file.

painlessprod

ASKER

well, I guess the only thing that I could use would be the line# of the text file because the text file is appended to only at the end and I could store that # in the db

painlessprod

ASKER

Err:

An unhandled exception of type 'System.ArgumentException' occurred in system.data.dll
Additional information: This row already belongs to another table.

//dt =file
//dtSQL =SQL DB
//dtComp = output

                  for(int j=0;j<dtSQL.Rows.Count; j++)
                  {
                        if(!TD.dt.Rows.Contains(dtSQL.Rows[j].Table.Columns["ROWNUMBER"]))
                        {
                              TD.dtComp.Rows.Add(TD.dt.Rows[j]);

                        }
                  }

_TAD_

DataRow r;

for(int i=0;i<d1.Rows.Count; i++)
                        if(!d2.Rows.Contains(d1.Rows[i].Table.Columns["PrimaryKey"]))
                        {
                              r = d3.NewRow();
                              for(int j=0;j<r.Table.Columns.Count;j++)
                                    r[j] = d1.Rows[i].Table.Columns[j];
                              d3.Rows.Add(r);
                        }

eh...

first you need to create a new row object, then you need to give it the schema of the table you want to add it to, finally load the values into the schema and then add that row onto the end of the data table you want

painlessprod

ASKER

got it, thnx

painlessprod

ASKER

_TAD_ I hate to continue bugging you but this one is a new one to me.

In order to do the contains statement I had to create primary keys for all three datatables. I've done that
and now it says that the speicfied cast in no tvalid on this line
if(!dtSQL.Rows.Contains(TD.dt.Rows[i].Table.Columns["ROWNUMBER"]))

so i did this for all three datatables
MessageBox.Show(dtSQL.Columns["11"].DataType.ToString());
MessageBox.Show(TD.dt.Columns[11].DataType.ToString());
MessageBox.Show(TD.dtComp.Columns[11].DataType.ToString());

11 being the index of "ROWNUMBER"

and it returned system.int32 for all of them so I'm confused why it says that my cast is still not correct.

any help would be a great, thnks

painlessprod

ASKER

Hello all,

I have 3 dataTables
One that comes directly from SQL
One that comes from a text file
and one that does the comparison between the 2 and populates itself with waht's not in sql but resides in the text file.

All three have an initialized Primary Key of System.Int32.

dt //textfile
dtSQL //sql
dtComp //difference

for(int i=0;i<TD.dt.Rows.Count; i++)
if(!TD.dt.Rows.Contains(dtSQL.Rows[i].Table.Columns["ROWNUMBER"])) // this is where I get the error
      {
      r = TD.dtComp.NewRow();
      for(int j=0;j<r.Table.Columns.Count;j++)
      r[j] = TD.dt.Rows[i].Table.Columns[j];
      TD.dtComp.Rows.Add(r);
}