I have a need where I need to retrieve CSV files from an external source which will contain either updated or new records for tables in my SQL database (one CSV file per table).
This needs to be a job that will open each file (file name will be the table name to be updated or added to) in the folder, update existing records or add new records, depending on what's in the file (could be both, or one or the other).
Is SSIS the best approach for this (this is where I'm leaning)? If so, how would one go about automating this process? Some things to note:
1. The files could be different every day since not all tables may or may be updated or added to every day.
2. There may be no files in the folder on any given day.
3. Obviously, since each CSV is related to a different table, the column names will be different for each CSV (which would probably affect column mappings in the Data Flow of the SSIS package), and I really don't want to create a SSIS package for every single possible table in the database. That'd be 100s of SSIS packages.
Any help would be appreciated, especially on how to determine if a record in the CSV is new or if it's an existing record. If it's an existing record, all columns will be updated in the table based on what's in the CSV.
Please let me know if you need more details from me on this.
I'd also be open to purchasing existing software if this becomes too cumbersome.
Thanks in advance!