BI questions

I have few questions data BI development

1. What options are available for ensuring consistent configuration of SSIS packages between development, QA, and production run time environments?

2. The source table for a data flow task has been changed so that one of the varchar(50) columns is now a varchar(100).  I want to know if this will break the data flow task. what steps would I need to take?

3. When designing a high volume data flow pipeline, Is it good to use a sort task? why or why not? What other options are there if I must have sorted data?
Who is Participating?
ValentinoVConnect With a Mentor BI ConsultantCommented:
In addition to Jim's response:

1. The most common method used is called indirect package configuration.  In short: you set up an OS environment variable that contains the full path to the configuration XML file or a connection string to the configuration database.  This allows you to point to the DEV config on the DEV server, to UAT on the UAT server, and so on...

More details: Best Practices for Integration Services Configurations

2. As long as the length of the data in the source table does not exceed 50 chars I don't think it's a problem.  But once the data gets longer, you'll start getting the truncation issue mentioned by Jim.

3. Why would you want to use the Sort component?  If you need incoming data to be sorted for some reason, do it in the source SELECT statement.

If you really, really want to get your data sorted and can't do it in the source query, work with an intermediary table: do whatever manipulation you need to do to the data and write it, unsorted, to the intermediary table.  Then set up a second flow following the first one that reads from the intermediary table, using an ORDER BY clause in the source query.
Jim HornConnect With a Mentor Microsoft SQL Server Developer, Architect, and AuthorCommented:
Just so you know, the style in which this question was written makes it look like homework, which is against EE policy.  

1.  Don't know.

2.  If the destination for the column we're talking about is less than 100 characters, it will throw a truncation warning, which may result in an error depeding on the settings.

3.  No.  Sorts are blocking components, which means the entire set must be ready for the sort before it can process, and then the entire set must be sorted before it moves on.  

Using a blocking transaction prevents the package from streaming data.
wasabi3689Author Commented:
This is not homework. Homework doesn't have QA, TEST and Production environment. I am researching a BI project and some questions brought up by the developers but I cannot answer.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.