Solved

SSIS import multiple CSVs into associated tables

Posted on 2016-11-22
3
69 Views
Last Modified: 2016-11-23
Hi

We currently have 3 files that are uploaded through SFTP to us on a daily bases.  Filename examples:

 - Invoices_20161122.csv
 - Agencies_20161122.csv
-  Sales_20161122.csv

The file names will change the date stamp every day when they are uploaded to the SFTP site

The files need to be uploaded in their associated tables

- Invoices
- Agencies
- Sales

The tables would need to be dropped and recreated when the files are imported.  The first row in the files contains the column names.

How can I create an SSIS package to:

- Drop the tables
- Recreate the tables when importing the CSV file and name them appropriately
- Move the old files an archive folder


Thank you for all of your help
0
Comment
Question by:thomasm1948
3 Comments
 
LVL 28

Expert Comment

by:Pawan Kumar
ID: 41898350
Have you tried anything? With which part you need help ?
0
 
LVL 12

Accepted Solution

by:
Arifhusen Ansari earned 500 total points
ID: 41898541
Hi,

Based on your explanation. I can give you below solution.
You can create a single package to achieve this. But rather i would suggest, to create three packages. So that it would be easy to maintain.

1) Rather than drop a table and recreate the same. you can truncate that table.

For this you can use Execute script task and in that write a script to truncate table.

2) If you only have one file per type per day. You can use Data flow task directly to load the file in the table.

  e.g Invoices_20161122.csv will be only one for that day.

3) You have to use Flat File Connection as your file is of type csv.
4) when you create the package initially use the static connection to csv file. Develop the package if it's loaded perfectly.
5) You need to create the dynamic connection so that from next time it will automatically pick a file and load it.

For this you have to declare 3 variable.
1) FolderPath: path for your folder.
e.g. "E:\\FlatFileLoadOrCSV\\
2) FileName: this will be generated based on expression as we have to dynamically pick a file.
e.g. Expression would be like:  "Invoices_"+(DT_WSTR,4)  year(getdate())+(DT_WSTR,2)  month(getdate())+(DT_WSTR,2)  day(getdate())+".csv"

Evaluated value: Invoices_20161123.csv

3) FilePath: This is exact file path from where package will pick a file.
e.g. Expression is @[User::FolderPath]+@[User::FileName]

Evaluated value: E:\FlatFileLoadOrCSV\Invoices_20161123.csv

4) After this you just need to configure you connection manager to pick a file dynamically.

Click on File Connection go to property. Go to expression option click on ellipsis button and set the connection string property.

Refer below screenshot.

2016-11-23_10-30-05.png

Hope it will help you.

I have attached the ssis package as well. I have changed the extension to .txt. Download the file and just change it to .dtsx.

Hope it will help you.
FlatFileLoadOrCSV.txt
0
 

Author Closing Comment

by:thomasm1948
ID: 41899069
Thank this works out perfectly
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
How to share SSIS Package? 6 37
SQL Replication question 9 42
SQL View nearest date 5 36
Query to capture 5 and 9 digit zip code? 4 20
Why is this different from all of the other step by step guides?  Because I make a living as a DBA and not as a writer and I lived through this experience. Defining the name: When I talk to people they say different names on this subject stuff l…
I have a large data set and a SSIS package. How can I load this file in multi threading?
Via a live example, show how to extract insert data into a SQL Server database table using the Import/Export option and Bulk Insert.
Using examples as well as descriptions, and references to Books Online, show the documentation available for datatypes, explain the available data types and show how data can be passed into and out of variables.

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question