importing data using SSIS

Posted on 2014-08-28
Last Modified: 2016-02-10
Hi there,
I am looking to import some files using SSIS 2005 -
the format in which I receive the files is what is throwing me off.
Rather than a normal table schema (rows and columns) the formatting is like follows:

{u'23456789': {u'2014-08-19': {u'date': u'2014-08-19'
                                  u'downloads': 50
                                  u'net_downloads': 49
                                  u'product_id': 39794100070
                                  u'returns': 1
                                  u'revenue': u'49.00'
                                  u'updates': 0}
                  u'2014-08-20': {u'date': u'2014-08-20'
                                  u'downloads': 40
                                  u'net_downloads': 40
                                  u'product_id': 39794100070
                                 u'returns': 0
                                  u'revenue': u'40.00'
                                  u'updates': 0}}}

I can receive this info in csv or txt file. if I open in CSV everything is on the first column A.

Can anyone advise me on how to import this sort of formatting into a normal database.
do I need to use .net to call each of the fields and split the data out?

if you have any reference sites I can look at please let me know,
Question by:Putoch
    LVL 44

    Assisted Solution

    by:Vitor Montalvão
    Will be useful if you could find a pattern and fields divisors ( { , u' or : )
    LVL 65

    Assisted Solution

    by:Jim Horn
    Agreed.  What is the source of this file?

    I don't see any row/column delimeters, and it doesn't appear to be XML, so afaik unless it's in a format that is covered by a third-party tool such as Pragmatic Works Task Factory or CozyRoc, you're going to have to import it all as one column, then parse the hell out of it in T-SQL to extract values.
    LVL 37

    Accepted Solution

    I don't think this will be possible using the standard Flat File source but besides the option that Jim mentions there's another one: the .NET Script component.  You can use it as Source in a Data Flow.

    But it does indeed mean that you'll need to write some string manipulation code to extract the field names and values out of it.  The Split function should help you there.  It also means that you'll need to set up a data flow that uses the Key/Value pattern, this way you don't have dynamic field names going down the flow (which wouldn't work anyway).

    So you'll end up with a table that contains some identifiers (file name? or that '23456789' number?  or both?), a key (date, downloads, ...) and a value ('2014-08-19', '50', ...)

    Once you've got that and your end goal is a regular flat table you'll need to pivot the values somehow, should be possible through T-SQL.

    Setting this up won't be trivial but it should be possible...

    Author Comment

    thank you all for the comments.
    I may get to work!
    I will repost my solution if/when I get it done.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Enabling OSINT in Activity Based Intelligence

    Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

    My client sends data in an Excel file to me to load them into Staging database. The file contains many sheets that they have same structure. In this article, I would like to share the simple way to load data of multiple sheets by using SSIS.
    The Delta outage: 650 cancelled flights, more than 1200 delayed flights, thousands of frustrated customers, tens of millions of dollars in damages – plus untold reputational damage to one of the world’s most trusted airlines. All due to a catastroph…
    Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
    Via a live example combined with referencing Books Online, show some of the information that can be extracted from the Catalog Views in SQL Server.

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    18 Experts available now in Live!

    Get 1:1 Help Now