Solved

SSIS, Source is text file with matrixed data, Destination needs to be flat

Posted on 2014-01-13
5
338 Views
Last Modified: 2016-02-10
Hi guys

I have a text file with 'matrixed' data like this:
10/11/13      22     420     131             810     813     440
     555  255.21  255.76  257.74                                        *
     559  296.21  296.76  298.74                                        *
     551
     560
     562
     550
     660
     650  300.23  300.78  302.87                                        *
     640  301.13  301.68  303.77                                        *
     620  309.30  309.85  311.94                                        *
     600
     595
     810                                  110.30  110.30                *

Open in new window

I need to write a data flow task that takes this 'matrixed' data and does...
Delete all rows where there are no values in the 2nd and 3rd column
Delete all columns after the 3rd column
Writes what's left to a flat table like this..
555         22       255.21
555        420       255.76
559         22       296.21 
559        420       296.76
650         22       300.23
650        420       300.78
640         22       301.13
640        420       301.68
620         22       309.30
620        420       309.85

Open in new window

Points for any useful design ideas on how to pull this off.
SSIS 2008 R2 + Task Factory, can also write in 2012 if needed.

Thanks in advance.
Jim
0
Comment
Question by:Jim Horn
  • 2
  • 2
5 Comments
 
LVL 82

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 50 total points
ID: 39777224
Silly question... how many rows?  Is it a repetitive task that needs to be programmed?
0
 
LVL 65

Author Comment

by:Jim Horn
ID: 39777250
>Silly question... how many rows?
Couple hundred.  It's a pricing file for commodities by location.

>Is it a repetitive task that needs to be programmed?
Yes, it will be scheduled to execute daily at oh-dark-thirty.
0
 
LVL 82

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 50 total points
ID: 39777297
What programming language does it need to be?  It looks like it wouldn't be too hard in command line PHP.  Do you have a sample file (other than what you posted because what you posted may be reformatted)?
0
 
LVL 37

Accepted Solution

by:
ValentinoV earned 100 total points
ID: 39785660
Hey Jim,

I assume you'd prefer to get this done using SSIS, right?  Did you try out the Flat File Source with Fixed Width format?  AFAIK that should work okay-ish.

That first line looks like a header, if you don't need that you'll probably want to skip it because the FF source can only handle one line format.  That's done through the Header rows to skip setting.

If you do need it then you might want to investigate into using the Script Component as source. Or use a separate data flow (I guess two FF sources using the same file in the same data flow won't work but haven't tried it out to know for sure).

VV.
0
 
LVL 65

Author Comment

by:Jim Horn
ID: 39912800
The ultimate solution ended up being to import the flat file as a source, and lots of derived columns to slice-dice.  

The gig ended, and I can't elaborate more on that, so I'll lower and split points for helpful advise.
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Suggested Solutions

International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
This video shows, step by step, how to configure Oracle Heterogeneous Services via the Generic Gateway Agent in order to make a connection from an Oracle session and access a remote SQL Server database table.
Viewers will learn how to use the SELECT statement in SQL and will be exposed to the many uses the SELECT statement has.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now