• Status: Solved
  • Priority: High
  • Security: Public
  • Views: 42
  • Last Modified:

Auto Updating SQL Tables Using Data From CSV Files

I have a need where I need to retrieve CSV files from an external source which will contain either updated or new records for tables in my SQL database (one CSV file per table).

This needs to be a job that will open each file (file name will be the table name to be updated or added to) in the folder, update existing records or add new records, depending on what's in the file (could be both, or one or the other).

Is SSIS the best approach for this (this is where I'm leaning)? If so, how would one go about automating this process? Some things to note:

1. The files could be different every day since not all tables may or may be updated or added to every day.
2. There may be no files in the folder on any given day.
3. Obviously, since each CSV is related to a different table, the column names will be different for each CSV (which would probably affect column mappings in the Data Flow of the SSIS package), and I really don't want to create a SSIS package for every single possible table in the database. That'd be 100s of SSIS packages.

Any help would be appreciated, especially on how to determine if a record in the CSV is new or if it's an existing record. If it's an existing record, all columns will be updated in the table based on what's in the CSV.

Please let me know if you need more details from me on this.

I'd also be open to purchasing existing software if this becomes too cumbersome.

Thanks in advance!
0
infotechelg
Asked:
infotechelg
  • 2
  • 2
2 Solutions
 
arnoldCommented:
You can use common table expressions to load the cvs as a table....
0
 
arnoldCommented:
What is in the data?
Do you know powershell, vbscripting
That willy doe the csv, and create the inserts as needed.

Data types, etc
0
 
Vitor MontalvãoMSSQL Senior EngineerCommented:
SSIS solution should be more easy to implement as you can add .net code into the packages.
Alternatively you can use bcp utility that is more fast to load data. If you're good with Powershell you might be able to add some automation to control the different files and tables.
And a last alternative is to use full T-SQL code by importing data with the BULK INSERT command.
0
 
infotechelgAuthor Commented:
Thanks, everyone, for the suggestions. I'll see what I can come up with, and then report back. Any other suggestions/tips are welcome.
0
 
infotechelgAuthor Commented:
I've decided to just handle this all via a .NET Console Application instead of using SSIS. However, all of your comments were helpful and gave me good ideas to get this accomplished, so thanks for your input.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now