Solved

Automating using SSIS (DTXS files) to import FLAT FILES

Posted on 2008-06-16
4
779 Views
Last Modified: 2013-11-30
I have 4 flat files which I am receiving updates for on daily basis.  I have currently used SSIS to import these files into SQL Server 2005 and have created dtxs files for each one.  

Questions:

Is it possible after the first run of the configed dtxs files (each import a flat file to a SQL Server 2005 DB) to automate this process so that imports can occur on a daily basis even if columns are removed/added to future flat files?  If so please explain how this is done (In detail, pictures would be nice).

Just to clarify my vendor has told me that they will remove or added different columns to these flat files at any time.  I do not want to recreate my orginal dtxs files, each time this occurs.
0
Comment
Question by:RunnerNwizard
4 Comments
 
LVL 5

Accepted Solution

by:
fesnyng earned 250 total points
ID: 21801487
Besides getting another vendor?  

Data structure reflects the business structure.   If it is changing every data load, then the business is changing.  And, unfortunately, you will need to change your ETL each time the data structure changes.

What I have done in the face of changing data is to make the import as simple as possible -- i.e. use the "Import Data..." task ( in SSMS, right click the data base, select Tasks, Select Import Data ...).  Let SSIS figure out what the columns are.  Then use stored procedures (which will be modified each time the vendor changes the data structure) to extract and transform the data.  This is more along the lines of ELT (Extract Load Transform) but is "the least effort" that I have found in the face of constantly changing data structure.  

Not the answer you were looking for, I know.   Hopefully, I am wrong and there is a better solution among the experts here.
0
 
LVL 30

Assisted Solution

by:nmcdermaid
nmcdermaid earned 250 total points
ID: 21862581
Well you could probably automate adding a new column to a staging table and adding a new column to the connection manager but I agree with fesnyq: its more fundamental than that - where does the data in your staging table go after a new column is added??? You certainly can't automate that.
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Use this article to create a batch file to backup a Microsoft SQL Server database to a Windows folder.  The folder can be on the local hard drive or on a network share.  This batch file will query the SQL server to get the current date & time and wi…
Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.
Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now