Auto Updating SQL Tables Using Data From CSV Files

I have a need where I need to retrieve CSV files from an external source which will contain either updated or new records for tables in my SQL database (one CSV file per table).

This needs to be a job that will open each file (file name will be the table name to be updated or added to) in the folder, update existing records or add new records, depending on what's in the file (could be both, or one or the other).

Is SSIS the best approach for this (this is where I'm leaning)? If so, how would one go about automating this process? Some things to note:

1. The files could be different every day since not all tables may or may be updated or added to every day.
2. There may be no files in the folder on any given day.
3. Obviously, since each CSV is related to a different table, the column names will be different for each CSV (which would probably affect column mappings in the Data Flow of the SSIS package), and I really don't want to create a SSIS package for every single possible table in the database. That'd be 100s of SSIS packages.

Any help would be appreciated, especially on how to determine if a record in the CSV is new or if it's an existing record. If it's an existing record, all columns will be updated in the table based on what's in the CSV.

Please let me know if you need more details from me on this.

I'd also be open to purchasing existing software if this becomes too cumbersome.

Thanks in advance!
infotechelgAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

arnoldCommented:
You can use common table expressions to load the cvs as a table....
0
arnoldCommented:
What is in the data?
Do you know powershell, vbscripting
That willy doe the csv, and create the inserts as needed.

Data types, etc
0
Vitor MontalvãoMSSQL Senior EngineerCommented:
SSIS solution should be more easy to implement as you can add .net code into the packages.
Alternatively you can use bcp utility that is more fast to load data. If you're good with Powershell you might be able to add some automation to control the different files and tables.
And a last alternative is to use full T-SQL code by importing data with the BULK INSERT command.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
infotechelgAuthor Commented:
Thanks, everyone, for the suggestions. I'll see what I can come up with, and then report back. Any other suggestions/tips are welcome.
0
infotechelgAuthor Commented:
I've decided to just handle this all via a .NET Console Application instead of using SSIS. However, all of your comments were helpful and gave me good ideas to get this accomplished, so thanks for your input.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
SQL

From novice to tech pro — start learning today.