Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

delimited text file to sql

Posted on 2003-11-03
9
Medium Priority
?
400 Views
Last Modified: 2010-04-16
I have a delimited text file and I'm putting it into sql.
the problem is that the text file is quite large and changes all the time.  I am building a C# service that will read the textfile and put all new records into the DB.  That works fine for the first time when I initially populate but my problem is keeping it light.  I need to compare via date what resides in the db and what resides in the textfile as to avoid duplicate records.  Is there an efficient way to take the 2 datatable's (one from sql and one from the text file) and do a compare and put the difference of the results into a 3rd datatable which will be for a bulk insert via dataAccessApplicationBlock.........
0
Comment
Question by:painlessprod
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 3
9 Comments
 
LVL 22

Accepted Solution

by:
_TAD_ earned 615 total points
ID: 9673691


You mentioned a bulk insert, what about update?

Are you doing inserts only?  no updates, no deletes?



                  DataTable d1 = new DataTable(); // text file
                  DataTable d2 = new DataTable(); // data base
                  DataTable d3 = new DataTable(); // table for bulk insert

                  for(int i=0;i<d1.Rows.Count; i++)
                        if(!d2.Rows.Contains(d1.Rows[i].Table.Columns["PrimaryKey"]))
                              d3.Rows.Add(d1.Rows[i]);



...
..
.

bulk insert data from datatable 3
0
 

Author Comment

by:painlessprod
ID: 9673734
The problem is there's no primary key in the text file.  
There are no delete's, or updates.
there is a primary key for the sql
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 9673853


How do you determine unique-ness?

something must distinguish one row from another in your text file.

0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 

Author Comment

by:painlessprod
ID: 9673886
well, I guess the only thing that I could use would be the line# of the text file because the text file is appended to only at the end and I could store that # in the db
0
 

Author Comment

by:painlessprod
ID: 9681621
Err:

An unhandled exception of type 'System.ArgumentException' occurred in system.data.dll
Additional information: This row already belongs to another table.

//dt =file
//dtSQL =SQL DB
//dtComp = output


                  for(int j=0;j<dtSQL.Rows.Count; j++)
                  {
                        if(!TD.dt.Rows.Contains(dtSQL.Rows[j].Table.Columns["ROWNUMBER"]))
                        {
                              TD.dtComp.Rows.Add(TD.dt.Rows[j]);

                        }
                  }
0
 
LVL 22

Expert Comment

by:_TAD_
ID: 9682096
                                                  DataRow r;

for(int i=0;i<d1.Rows.Count; i++)
                        if(!d2.Rows.Contains(d1.Rows[i].Table.Columns["PrimaryKey"]))
                        {
                              r = d3.NewRow();
                              for(int j=0;j<r.Table.Columns.Count;j++)
                                    r[j] = d1.Rows[i].Table.Columns[j];
                              d3.Rows.Add(r);
                        }




eh...


first you need to create a new row object, then you need to give it the schema of the table you want to add it to, finally load the values into the schema and then add that row onto the end of the data table you want
0
 

Author Comment

by:painlessprod
ID: 9682365
got it, thnx
0
 

Author Comment

by:painlessprod
ID: 9683300
_TAD_ I hate to continue bugging you but this one is a new one to me.

In order to do the contains statement I had to create primary keys for all three datatables.  I've done that
and now it says that the speicfied cast in no tvalid on this line
if(!dtSQL.Rows.Contains(TD.dt.Rows[i].Table.Columns["ROWNUMBER"]))


so i did this for all three datatables
MessageBox.Show(dtSQL.Columns["11"].DataType.ToString());
MessageBox.Show(TD.dt.Columns[11].DataType.ToString());
MessageBox.Show(TD.dtComp.Columns[11].DataType.ToString());

11 being the index of "ROWNUMBER"

and it returned system.int32 for all of them so I'm confused why it says that my cast is still not correct.


any help would be a great, thnks
0
 

Author Comment

by:painlessprod
ID: 9688613
Hello all,

I have 3 dataTables
One that comes directly from SQL
One that comes from a text file
and one that does the comparison between the 2 and populates itself with waht's not in sql but resides in the text file.

All three have an initialized Primary Key of System.Int32.

dt //textfile
dtSQL //sql
dtComp //difference

for(int i=0;i<TD.dt.Rows.Count; i++)
if(!TD.dt.Rows.Contains(dtSQL.Rows[i].Table.Columns["ROWNUMBER"])) // this is where I get the error
      {
      r = TD.dtComp.NewRow();
      for(int j=0;j<r.Table.Columns.Count;j++)
      r[j] = TD.dt.Rows[i].Table.Columns[j];
      TD.dtComp.Rows.Add(r);
}

0

Featured Post

On Demand Webinar - Networking for the Cloud Era

This webinar discusses:
-Common barriers companies experience when moving to the cloud
-How SD-WAN changes the way we look at networks
-Best practices customers should employ moving forward with cloud migration
-What happens behind the scenes of SteelConnect’s one-click button

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: Najam
Having new technologies does not mean they will completely replace old components.  Recently I had to create WCF that will be called by VB6 component.  Here I will describe what steps one should follow while doing so, please feel free to post any qu…
Calculating holidays and working days is a function that is often needed yet it is not one found within the Framework. This article presents one approach to building a working-day calculator for use in .NET.
Monitoring a network: how to monitor network services and why? Michael Kulchisky, MCSE, MCSA, MCP, VTSP, VSP, CCSP outlines the philosophy behind service monitoring and why a handshake validation is critical in network monitoring. Software utilized …
Do you want to know how to make a graph with Microsoft Access? First, create a query with the data for the chart. Then make a blank form and add a chart control. This video also shows how to change what data is displayed on the graph as well as form…

719 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question