?
Solved

SSIS, Source is text file with matrixed data, Destination needs to be flat

Posted on 2014-01-13
5
Medium Priority
?
362 Views
Last Modified: 2016-02-10
Hi guys

I have a text file with 'matrixed' data like this:
10/11/13      22     420     131             810     813     440
     555  255.21  255.76  257.74                                        *
     559  296.21  296.76  298.74                                        *
     551
     560
     562
     550
     660
     650  300.23  300.78  302.87                                        *
     640  301.13  301.68  303.77                                        *
     620  309.30  309.85  311.94                                        *
     600
     595
     810                                  110.30  110.30                *

Open in new window

I need to write a data flow task that takes this 'matrixed' data and does...
Delete all rows where there are no values in the 2nd and 3rd column
Delete all columns after the 3rd column
Writes what's left to a flat table like this..
555         22       255.21
555        420       255.76
559         22       296.21 
559        420       296.76
650         22       300.23
650        420       300.78
640         22       301.13
640        420       301.68
620         22       309.30
620        420       309.85

Open in new window

Points for any useful design ideas on how to pull this off.
SSIS 2008 R2 + Task Factory, can also write in 2012 if needed.

Thanks in advance.
Jim
0
Comment
Question by:Jim Horn
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 83

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 200 total points
ID: 39777224
Silly question... how many rows?  Is it a repetitive task that needs to be programmed?
0
 
LVL 66

Author Comment

by:Jim Horn
ID: 39777250
>Silly question... how many rows?
Couple hundred.  It's a pricing file for commodities by location.

>Is it a repetitive task that needs to be programmed?
Yes, it will be scheduled to execute daily at oh-dark-thirty.
0
 
LVL 83

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 200 total points
ID: 39777297
What programming language does it need to be?  It looks like it wouldn't be too hard in command line PHP.  Do you have a sample file (other than what you posted because what you posted may be reformatted)?
0
 
LVL 37

Accepted Solution

by:
ValentinoV earned 400 total points
ID: 39785660
Hey Jim,

I assume you'd prefer to get this done using SSIS, right?  Did you try out the Flat File Source with Fixed Width format?  AFAIK that should work okay-ish.

That first line looks like a header, if you don't need that you'll probably want to skip it because the FF source can only handle one line format.  That's done through the Header rows to skip setting.

If you do need it then you might want to investigate into using the Script Component as source. Or use a separate data flow (I guess two FF sources using the same file in the same data flow won't work but haven't tried it out to know for sure).

VV.
0
 
LVL 66

Author Comment

by:Jim Horn
ID: 39912800
The ultimate solution ended up being to import the flat file as a source, and lots of derived columns to slice-dice.  

The gig ended, and I can't elaborate more on that, so I'll lower and split points for helpful advise.
0

Featured Post

Get MySQL database support online, now!

At Percona’s web store you can order your MySQL database support needs in minutes. No hassles, no fuss, just pick and click. Pay online with a credit card.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
Via a live example, show how to extract information from SQL Server on Database, Connection and Server properties
Viewers will learn how to use the INSERT statement to insert data into their tables. It will also introduce the NULL statement, to show them what happens when no value is giving for any given column.
Suggested Courses

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question