SQL insert data into a record set from a datatable where that data does not exist in the recordset


I feel that I may have bitten into  too large a pie, but here goes:
in VB 2005
I have a datatable with new information, I also have a recordset with the same data. The recordset is extracted from a master record keeping xlsx woorkbook, the datatable is from a DBF. The column structures of each is the same. The DBF contains new data as well as some old data. I need to only take the new data and place it into the recordset so that I can update the master record.

The way to identify new data is to compare 2 columns Name and Date if the NEW name & date do not
match any Name & Date in the master file then the record goes in.

The first time this needs to run I will have over 11,000 DBF files to place into the master woorkbook, after that the load is 3-10 a day. I have already got a solution that works using excel opened by VB.net and I loop through the (dbf) datatable using dataview and then search each record individually using the Find command in the excel woorkbok to see if the record exists if not then I place it in and so on. This way takes about 45sec to do each DBF ..... not too bad with 3 - 10 but a disaster with 11,000.

I am not at all experianced with this and have already tried a few different alternatives. However I do need the master to be an excel woorkbook, and the updates do come from the dbf files those are my constants, anything else is up for grabs.

Can anyone help me to put together some kind of strategy to do this fast if my idea of the recordset is not the best.

Thank you

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

It would be better to import all records from the database into your excel file then you can play with the everyday update.
Read Import or connect to data from the Excel help

wbr Janos

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
MichMatAuthor Commented:
Can this be done via VB.net?
I looked at it and this method may be what I can use, providing that it can be done programatocaly.
I tried looking around but so far I found no example to do this.


Why would you like to use VB.net? I don't know which database you use but I'm sure you can easily import this data. In worst case you have to use ODBC-driver but it is not a big play.
Really it is simple :-)
Did you read the help suggested?
MichMatAuthor Commented:
Yes, thank you I read the help file, it seams that this process is usefull however it is a VB.net program that Im writing and so I would need to do this with in the program.

MichMatAuthor Commented:
Not my solution but a solution for someone else perhaps, no help to me.
Thank you for your time

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Applications

From novice to tech pro — start learning today.