SSIS import multiple CSVs into associated tables

Posted on 2016-11-22
Medium Priority
Last Modified: 2016-11-23

We currently have 3 files that are uploaded through SFTP to us on a daily bases.  Filename examples:

 - Invoices_20161122.csv
 - Agencies_20161122.csv
-  Sales_20161122.csv

The file names will change the date stamp every day when they are uploaded to the SFTP site

The files need to be uploaded in their associated tables

- Invoices
- Agencies
- Sales

The tables would need to be dropped and recreated when the files are imported.  The first row in the files contains the column names.

How can I create an SSIS package to:

- Drop the tables
- Recreate the tables when importing the CSV file and name them appropriately
- Move the old files an archive folder

Thank you for all of your help
Question by:thomasm1948
LVL 38

Expert Comment

by:Pawan Kumar
ID: 41898350
Have you tried anything? With which part you need help ?
LVL 13

Accepted Solution

Arifhusen Ansari earned 2000 total points
ID: 41898541

Based on your explanation. I can give you below solution.
You can create a single package to achieve this. But rather i would suggest, to create three packages. So that it would be easy to maintain.

1) Rather than drop a table and recreate the same. you can truncate that table.

For this you can use Execute script task and in that write a script to truncate table.

2) If you only have one file per type per day. You can use Data flow task directly to load the file in the table.

  e.g Invoices_20161122.csv will be only one for that day.

3) You have to use Flat File Connection as your file is of type csv.
4) when you create the package initially use the static connection to csv file. Develop the package if it's loaded perfectly.
5) You need to create the dynamic connection so that from next time it will automatically pick a file and load it.

For this you have to declare 3 variable.
1) FolderPath: path for your folder.
e.g. "E:\\FlatFileLoadOrCSV\\
2) FileName: this will be generated based on expression as we have to dynamically pick a file.
e.g. Expression would be like:  "Invoices_"+(DT_WSTR,4)  year(getdate())+(DT_WSTR,2)  month(getdate())+(DT_WSTR,2)  day(getdate())+".csv"

Evaluated value: Invoices_20161123.csv

3) FilePath: This is exact file path from where package will pick a file.
e.g. Expression is @[User::FolderPath]+@[User::FileName]

Evaluated value: E:\FlatFileLoadOrCSV\Invoices_20161123.csv

4) After this you just need to configure you connection manager to pick a file dynamically.

Click on File Connection go to property. Go to expression option click on ellipsis button and set the connection string property.

Refer below screenshot.


Hope it will help you.

I have attached the ssis package as well. I have changed the extension to .txt. Download the file and just change it to .dtsx.

Hope it will help you.

Author Closing Comment

ID: 41899069
Thank this works out perfectly

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Windocks is an independent port of Docker's open source to Windows.   This article introduces the use of SQL Server in containers, with integrated support of SQL Server database cloning.
MSSQL DB-maintenance also needs implementation of multiple activities. However, unprecedented errors can hamper the database management. In that case, deploying Stellar SQL Database Toolkit ensures fast and accurate database and backup repair as wel…
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.

600 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question