SSIS: Pivot with dynamic column names?

SSIS 2008 SP1 CU6

I have to write a SSIS package.  There are always a fixed set of columns, and a dynamic set of column value pairs controlled by a table.  This dynamic set of column values can change without notice; this is by design.

I have a SP that issues a query that will return something like this:

userid,useremail,attr_name,seq,attr_value
1,moe@stooges.com,name,Moe Howard
1,moe@stooges.com,hair,mop-top
1,mo3@stooges.com,killer move,eye-poke
2,larry@stooges.com,name,Larry Howard
2,larry@stooges.com,hair,finger in light socket
2,larry@stooges.com,killer move,hammer to forehead
3,curly@stooges.com,name,Curly Howard
3,curly@stooges.com,hair,baldie sour
3,curly@stooges.com,killer move,baseball bat

To pivot into:

userid,email,name,hair,killer move
1,moe@stooges.com,Moe Howard,mop-top, eye poke
2,larry@stooges.com,Larry Howard,finger in light socket,hammer to forehead
3,curly@stooges.com,Curly Howard,baldie sour,baseball bat

a classic pivot...

The problem:  Name, hair and killer move could change to name, number of movies, income bracket and some other demographic with the next execution.  The pivot column name values are *never* static.  There are thousands of variations in just the first database I reviewed.

Is it even possible to do this with a SSIS pivot step in a data flow?  I don't want to waste time, and right now I cannot figure out how to do it.
johnftamburoAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

brad2575Commented:
Just create the SQL query dynamically using variables for the 3 field names.

Something like this:

EXEC('
    select ' + DynamicFieldName1 + ', ' + DynamicFieldName2 + '
    FROM table
')

This does not do all the pivot stuff but you would just fill in the blanks with your variable field names.

If you are using SSIS you can populate 3 variables with the field names you are using, and have a variable for teh SQL select statement you want to use and then use an expression to dynamically build the query variable using the 3 variable field names.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Megan BrooksSQL Server ConsultantCommented:
Are you saying that the columns in the input table are dynamic? I don't think SSIS would deal well with that directly, but you still should be able to bring the normalized data into a data flow task pipeline, provided that you do the normalizing in SQL and only present a fixed set of columns to SSIS.
You could do the normalizing, for example, in a stored procedure. It could use the metadata views (INFORMATION_SCHEMA.TABLES, INFORMATION_SCHEMA.COLUMNS, etc.) to identify the dynamic columns, and it could then build the UNION query dynamically and use sp_executesql to execute it.
You could also build the SQL query using a script task and then pass it the query string to the source component using a package variable. In that case you might enclose the script task within a Foreach Loop Container that queries and iterates through the column metadata. The script task would concatenate the query for each column into a query string that could be passed to the script task in a variable. Or the metadata could be read and the query could be built using a Data Flow task, again, leaving the query in a package variable.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server

From novice to tech pro — start learning today.