Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Private
  • Views: 29
  • Last Modified:

BI questions

I have few questions data BI development

1. What options are available for ensuring consistent configuration of SSIS packages between development, QA, and production run time environments?

2. The source table for a data flow task has been changed so that one of the varchar(50) columns is now a varchar(100).  I want to know if this will break the data flow task. what steps would I need to take?

3. When designing a high volume data flow pipeline, Is it good to use a sort task? why or why not? What other options are there if I must have sorted data?
0
wasabi3689
Asked:
wasabi3689
2 Solutions
 
Jim HornMicrosoft SQL Server Developer, Architect, and AuthorCommented:
Just so you know, the style in which this question was written makes it look like homework, which is against EE policy.  

1.  Don't know.

2.  If the destination for the column we're talking about is less than 100 characters, it will throw a truncation warning, which may result in an error depeding on the settings.

3.  No.  Sorts are blocking components, which means the entire set must be ready for the sort before it can process, and then the entire set must be sorted before it moves on.  

Using a blocking transaction prevents the package from streaming data.
0
 
wasabi3689Author Commented:
This is not homework. Homework doesn't have QA, TEST and Production environment. I am researching a BI project and some questions brought up by the developers but I cannot answer.
0
 
ValentinoVBI ConsultantCommented:
In addition to Jim's response:

1. The most common method used is called indirect package configuration.  In short: you set up an OS environment variable that contains the full path to the configuration XML file or a connection string to the configuration database.  This allows you to point to the DEV config on the DEV server, to UAT on the UAT server, and so on...

More details: Best Practices for Integration Services Configurations

2. As long as the length of the data in the source table does not exceed 50 chars I don't think it's a problem.  But once the data gets longer, you'll start getting the truncation issue mentioned by Jim.

3. Why would you want to use the Sort component?  If you need incoming data to be sorted for some reason, do it in the source SELECT statement.

If you really, really want to get your data sorted and can't do it in the source query, work with an intermediary table: do whatever manipulation you need to do to the data and write it, unsorted, to the intermediary table.  Then set up a second flow following the first one that reads from the intermediary table, using an ORDER BY clause in the source query.
0

Featured Post

Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now