[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now


importing data using SSIS

Posted on 2014-08-28
Medium Priority
Last Modified: 2016-02-10
Hi there,
I am looking to import some files using SSIS 2005 -
the format in which I receive the files is what is throwing me off.
Rather than a normal table schema (rows and columns) the formatting is like follows:

{u'23456789': {u'2014-08-19': {u'date': u'2014-08-19'
                                  u'downloads': 50
                                  u'net_downloads': 49
                                  u'product_id': 39794100070
                                  u'returns': 1
                                  u'revenue': u'49.00'
                                  u'updates': 0}
                  u'2014-08-20': {u'date': u'2014-08-20'
                                  u'downloads': 40
                                  u'net_downloads': 40
                                  u'product_id': 39794100070
                                 u'returns': 0
                                  u'revenue': u'40.00'
                                  u'updates': 0}}}

I can receive this info in csv or txt file. if I open in CSV everything is on the first column A.

Can anyone advise me on how to import this sort of formatting into a normal database.
do I need to use .net to call each of the fields and split the data out?

if you have any reference sites I can look at please let me know,
Question by:Putoch
LVL 53

Assisted Solution

by:Vitor Montalvão
Vitor Montalvão earned 400 total points
ID: 40292351
Will be useful if you could find a pattern and fields divisors ( { , u' or : )
LVL 66

Assisted Solution

by:Jim Horn
Jim Horn earned 800 total points
ID: 40292407
Agreed.  What is the source of this file?

I don't see any row/column delimeters, and it doesn't appear to be XML, so afaik unless it's in a format that is covered by a third-party tool such as Pragmatic Works Task Factory or CozyRoc, you're going to have to import it all as one column, then parse the hell out of it in T-SQL to extract values.
LVL 37

Accepted Solution

ValentinoV earned 800 total points
ID: 40292459
I don't think this will be possible using the standard Flat File source but besides the option that Jim mentions there's another one: the .NET Script component.  You can use it as Source in a Data Flow.

But it does indeed mean that you'll need to write some string manipulation code to extract the field names and values out of it.  The Split function should help you there.  It also means that you'll need to set up a data flow that uses the Key/Value pattern, this way you don't have dynamic field names going down the flow (which wouldn't work anyway).

So you'll end up with a table that contains some identifiers (file name? or that '23456789' number?  or both?), a key (date, downloads, ...) and a value ('2014-08-19', '50', ...)

Once you've got that and your end goal is a regular flat table you'll need to pivot the values somehow, should be possible through T-SQL.

Setting this up won't be trivial but it should be possible...

Author Comment

ID: 40293026
thank you all for the comments.
I may get to work!
I will repost my solution if/when I get it done.

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Why is this different from all of the other step by step guides?  Because I make a living as a DBA and not as a writer and I lived through this experience. Defining the name: When I talk to people they say different names on this subject stuff l…
This article shows gives you an overview on SQL Server 2016 row level security. You will also get to know the usages of row-level-security and how it works
This videos aims to give the viewer a basic demonstration of how a user can query current session information by using the SYS_CONTEXT function
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Suggested Courses

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question