Improve company productivity with a Business Account.Sign Up

x
?
Solved

How do I strip out a second CR/LF pair that randomly appears in a flat file

Posted on 2013-05-21
6
Medium Priority
?
929 Views
Last Modified: 2016-02-10
Hi!

I'm using SSIS to import some flat files.  They are fixed length and use a CRLF as a row delimiter.  However, there are some cases (like every 5000 lines) where there is a second CRLF pair.  This messes up my import and rejects a number of records.  I'm trying to figure out a way to remove any occurrences of <CR><LF><CR><LF> to just be <CR><LF>   Any suggestions?
0
Comment
Question by:ITMikeK
  • 3
  • 3
6 Comments
 
LVL 16

Expert Comment

by:Surendra Nath
ID: 39185149
ok, this is what I will do, I will try to use a script component and replace the double crlf charecter with a single one.

the below code might be of some use (note this is not tested, so test it yourself and let us know)

 Dim data As String
 data = file.ReadToEnd()
data = data.replace(ControlChars.CrLf+ControlChars.CrLf,ControlChars.CrLf)

 Dim writer As New System.IO.StreamWriter(Dts.Variables("@[User::str_SourcePath]").Value.ToString(), False)
 writer.Write(data)
 writer.Flush()
 writer.Close()

Open in new window

0
 

Author Comment

by:ITMikeK
ID: 39185188
Shouldn't this be possible using the REPLACE transformation function with a derived column?   I'm wrestling with the syntax, but would be something like this:

COLUMN ALIAS: ENDOFROW

REPLACE(ENDOFROW,"\n\n","\n")   ?
0
 
LVL 16

Expert Comment

by:Surendra Nath
ID: 39185198
No, because SSIS will determine your end of row by the CRLF character.
SO, by the time the your replace function actually gets hit then SSIS might have converted the data from your file into rows and it might be struggling to assign the values to columns, where it might fail ( as there is an empty line where there are two carriage returns).

So, you have to do that before loading the file using the below script task.
0
What Kind of Coding Program is Right for You?

There are many ways to learn to code these days. From coding bootcamps like Flatiron School to online courses to totally free beginner resources. The best way to learn to code depends on many factors, but the most important one is you. See what course is best for you.

 

Author Comment

by:ITMikeK
ID: 39185286
i haven't written a SSIS script before, but I am a proficient C# developer.  How would I add the script?  When I drop the component on my canvas and double click to edit it, it launches a new instance of VS and another solution.  Not sure how to proceed.
0
 
LVL 16

Accepted Solution

by:
Surendra Nath earned 1000 total points
ID: 39185302
Cool, I am a novice in SSIS too, so cannot help you out much here....
If I am with same issue then I will a C# executable which will convert the crlf+crlf charecters into one crlf charecter from the C# console program and add this to the calling script of SSIS as well...
0
 

Author Closing Comment

by:ITMikeK
ID: 39188615
I was able to strip out the offending characters with Notepad++ in about 30 seconds.  When I have more time, I'll look at automating this into a script.  Thanks for the direction.
0

Featured Post

Build your data science skills into a career

Are you ready to take your data science career to the next step, or break into data science? With Springboard’s Data Science Career Track, you’ll master data science topics, have personalized career guidance, weekly calls with a data science expert, and a job guarantee.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

What if you have to shut down the entire Citrix infrastructure for hardware maintenance, software upgrades or "the unknown"? I developed this plan for "the unknown" and hope that it helps you as well. This article explains how to properly shut down …
Microsoft Access has a limit of 255 columns in a single table; SQL Server allows tables with over 255 columns, but reading that data is not necessarily simple.  The final solution for this task involved creating a custom text parser and then reading…
Via a live example, show how to set up a backup for SQL Server using a Maintenance Plan and how to schedule the job into SQL Server Agent.
Viewers will learn how to use the UPDATE and DELETE statements to change or remove existing data from their tables. Make a table: Update a specific column given a specific row using the UPDATE statement: Remove a set of values using the DELETE s…

585 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question