importing data using SSIS

Putoch
Putoch used Ask the Experts™
on
Hi there,
I am looking to import some files using SSIS 2005 -
the format in which I receive the files is what is throwing me off.
Rather than a normal table schema (rows and columns) the formatting is like follows:

{u'23456789': {u'2014-08-19': {u'date': u'2014-08-19'
                                  u'downloads': 50
                                  u'net_downloads': 49
                                  u'product_id': 39794100070
                                  u'returns': 1
                                  u'revenue': u'49.00'
                                  u'updates': 0}
                  u'2014-08-20': {u'date': u'2014-08-20'
                                  u'downloads': 40
                                  u'net_downloads': 40
                                  u'product_id': 39794100070
                                 u'returns': 0
                                  u'revenue': u'40.00'
                                  u'updates': 0}}}


I can receive this info in csv or txt file. if I open in CSV everything is on the first column A.

Can anyone advise me on how to import this sort of formatting into a normal database.
do I need to use .net to call each of the fields and split the data out?

if you have any reference sites I can look at please let me know,
thanks,p
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Vitor MontalvãoMSSQL Senior Engineer
Distinguished Expert 2017
Commented:
Will be useful if you could find a pattern and fields divisors ( { , u' or : )
Jim HornSQL Server Data Dude
Most Valuable Expert 2013
Author of the Year 2015
Commented:
Agreed.  What is the source of this file?

I don't see any row/column delimeters, and it doesn't appear to be XML, so afaik unless it's in a format that is covered by a third-party tool such as Pragmatic Works Task Factory or CozyRoc, you're going to have to import it all as one column, then parse the hell out of it in T-SQL to extract values.
BI Consultant
Most Valuable Expert 2011
Commented:
I don't think this will be possible using the standard Flat File source but besides the option that Jim mentions there's another one: the .NET Script component.  You can use it as Source in a Data Flow.

But it does indeed mean that you'll need to write some string manipulation code to extract the field names and values out of it.  The Split function should help you there.  It also means that you'll need to set up a data flow that uses the Key/Value pattern, this way you don't have dynamic field names going down the flow (which wouldn't work anyway).

So you'll end up with a table that contains some identifiers (file name? or that '23456789' number?  or both?), a key (date, downloads, ...) and a value ('2014-08-19', '50', ...)

Once you've got that and your end goal is a regular flat table you'll need to pivot the values somehow, should be possible through T-SQL.

Setting this up won't be trivial but it should be possible...

Author

Commented:
thank you all for the comments.
I may get to work!
I will repost my solution if/when I get it done.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial