How do I strip out a second CR/LF pair that randomly appears in a flat file

Hi!

I'm using SSIS to import some flat files.  They are fixed length and use a CRLF as a row delimiter.  However, there are some cases (like every 5000 lines) where there is a second CRLF pair.  This messes up my import and rejects a number of records.  I'm trying to figure out a way to remove any occurrences of <CR><LF><CR><LF> to just be <CR><LF>   Any suggestions?
ITMikeKAsked:
Who is Participating?
 
Surendra NathConnect With a Mentor Technology LeadCommented:
Cool, I am a novice in SSIS too, so cannot help you out much here....
If I am with same issue then I will a C# executable which will convert the crlf+crlf charecters into one crlf charecter from the C# console program and add this to the calling script of SSIS as well...
0
 
Surendra NathTechnology LeadCommented:
ok, this is what I will do, I will try to use a script component and replace the double crlf charecter with a single one.

the below code might be of some use (note this is not tested, so test it yourself and let us know)

 Dim data As String
 data = file.ReadToEnd()
data = data.replace(ControlChars.CrLf+ControlChars.CrLf,ControlChars.CrLf)

 Dim writer As New System.IO.StreamWriter(Dts.Variables("@[User::str_SourcePath]").Value.ToString(), False)
 writer.Write(data)
 writer.Flush()
 writer.Close()

Open in new window

0
 
ITMikeKAuthor Commented:
Shouldn't this be possible using the REPLACE transformation function with a derived column?   I'm wrestling with the syntax, but would be something like this:

COLUMN ALIAS: ENDOFROW

REPLACE(ENDOFROW,"\n\n","\n")   ?
0
2018 Annual Membership Survey

Here at Experts Exchange, we strive to give members the best experience. Help us improve the site by taking this survey today! (Bonus: Be entered to win a great tech prize for participating!)

 
Surendra NathTechnology LeadCommented:
No, because SSIS will determine your end of row by the CRLF character.
SO, by the time the your replace function actually gets hit then SSIS might have converted the data from your file into rows and it might be struggling to assign the values to columns, where it might fail ( as there is an empty line where there are two carriage returns).

So, you have to do that before loading the file using the below script task.
0
 
ITMikeKAuthor Commented:
i haven't written a SSIS script before, but I am a proficient C# developer.  How would I add the script?  When I drop the component on my canvas and double click to edit it, it launches a new instance of VS and another solution.  Not sure how to proceed.
0
 
ITMikeKAuthor Commented:
I was able to strip out the offending characters with Notepad++ in about 30 seconds.  When I have more time, I'll look at automating this into a script.  Thanks for the direction.
0
All Courses

From novice to tech pro — start learning today.