SolvedPrivate

BI questions

Posted on 2014-02-16
3
24 Views
Last Modified: 2016-02-10
I have few questions data BI development

1. What options are available for ensuring consistent configuration of SSIS packages between development, QA, and production run time environments?

2. The source table for a data flow task has been changed so that one of the varchar(50) columns is now a varchar(100).  I want to know if this will break the data flow task. what steps would I need to take?

3. When designing a high volume data flow pipeline, Is it good to use a sort task? why or why not? What other options are there if I must have sorted data?
0
Comment
Question by:wasabi3689
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
3 Comments
 
LVL 65

Assisted Solution

by:Jim Horn
Jim Horn earned 150 total points
ID: 39864669
Just so you know, the style in which this question was written makes it look like homework, which is against EE policy.  

1.  Don't know.

2.  If the destination for the column we're talking about is less than 100 characters, it will throw a truncation warning, which may result in an error depeding on the settings.

3.  No.  Sorts are blocking components, which means the entire set must be ready for the sort before it can process, and then the entire set must be sorted before it moves on.  

Using a blocking transaction prevents the package from streaming data.
0
 

Author Comment

by:wasabi3689
ID: 39865399
This is not homework. Homework doesn't have QA, TEST and Production environment. I am researching a BI project and some questions brought up by the developers but I cannot answer.
0
 
LVL 37

Accepted Solution

by:
ValentinoV earned 150 total points
ID: 39870037
In addition to Jim's response:

1. The most common method used is called indirect package configuration.  In short: you set up an OS environment variable that contains the full path to the configuration XML file or a connection string to the configuration database.  This allows you to point to the DEV config on the DEV server, to UAT on the UAT server, and so on...

More details: Best Practices for Integration Services Configurations

2. As long as the length of the data in the source table does not exceed 50 chars I don't think it's a problem.  But once the data gets longer, you'll start getting the truncation issue mentioned by Jim.

3. Why would you want to use the Sort component?  If you need incoming data to be sorted for some reason, do it in the source SELECT statement.

If you really, really want to get your data sorted and can't do it in the source query, work with an intermediary table: do whatever manipulation you need to do to the data and write it, unsorted, to the intermediary table.  Then set up a second flow following the first one that reads from the intermediary table, using an ORDER BY clause in the source query.
0

Featured Post

Edgartown IT Case Study

Learn about Edgartown's quest to ensure the safety and security of the entire town's employee and citizen data. Read the case study!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Why is this different from all of the other step by step guides?  Because I make a living as a DBA and not as a writer and I lived through this experience. Defining the name: When I talk to people they say different names on this subject stuff l…
For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed
Via a live example, show how to shrink a transaction log file down to a reasonable size.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question