Solved

Automating using SSIS (DTXS files) to import FLAT FILES

Posted on 2008-06-16
4
785 Views
Last Modified: 2013-11-30
I have 4 flat files which I am receiving updates for on daily basis.  I have currently used SSIS to import these files into SQL Server 2005 and have created dtxs files for each one.  

Questions:

Is it possible after the first run of the configed dtxs files (each import a flat file to a SQL Server 2005 DB) to automate this process so that imports can occur on a daily basis even if columns are removed/added to future flat files?  If so please explain how this is done (In detail, pictures would be nice).

Just to clarify my vendor has told me that they will remove or added different columns to these flat files at any time.  I do not want to recreate my orginal dtxs files, each time this occurs.
0
Comment
Question by:RunnerNwizard
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
4 Comments
 
LVL 5

Accepted Solution

by:
fesnyng earned 250 total points
ID: 21801487
Besides getting another vendor?  

Data structure reflects the business structure.   If it is changing every data load, then the business is changing.  And, unfortunately, you will need to change your ETL each time the data structure changes.

What I have done in the face of changing data is to make the import as simple as possible -- i.e. use the "Import Data..." task ( in SSMS, right click the data base, select Tasks, Select Import Data ...).  Let SSIS figure out what the columns are.  Then use stored procedures (which will be modified each time the vendor changes the data structure) to extract and transform the data.  This is more along the lines of ELT (Extract Load Transform) but is "the least effort" that I have found in the face of constantly changing data structure.  

Not the answer you were looking for, I know.   Hopefully, I am wrong and there is a better solution among the experts here.
0
 
LVL 30

Assisted Solution

by:nmcdermaid
nmcdermaid earned 250 total points
ID: 21862581
Well you could probably automate adding a new column to a staging table and adding a new column to the connection manager but I agree with fesnyq: its more fundamental than that - where does the data in your staging table go after a new column is added??? You certainly can't automate that.
0

Featured Post

How our DevOps Teams Maximize Uptime

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us. Read the use case whitepaper.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
A Stored Procedure in Microsoft SQL Server is a powerful feature that it can be used to execute the Data Manipulation Language (DML) or Data Definition Language (DDL). Depending on business requirements, a single Stored Procedure can return differe…
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.

734 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question