Solved

Using SSIS to convert csv to xml

Posted on 2008-06-10
6
3,667 Views
Last Modified: 2013-11-07
I have a sample CSV file containg data in the following format:

CandidateID,CQN,CentreCode,ExamRef,Forename,Surname,Middlename,DOB,Gender
123456,ABC123,A0001CZ,XYZ123,Joe,Blogs,,21/02/1990,M
123456,ABC124,A0001CZ,XYZ987,Joe,Blogs,,21/02/1990,M
123457,ABC125,B0006CZ,XYZ123,Jane,Blogs,,18/05/1990,F
123457,ABC126,B0006CZ,XYZ987,Jane,Blogs,,18/05/1990,F

I need to convert this csv data into xml format which will provide the xml source for an SSIS package I have written. The xml source needs to be in the following format:

<data>
 <user>
   <id>0</id>
   <forename>Dave</forename>
   <surname>Dixon</surname>
   <middleName>Wakefield</middleName>
   <dob>07/10/1982</dob>
   <gender>M</gender>
   <addressLine1>31 Marlborough Road</addressLine1>
   <addressLine2/>
   <town>Shipley</town>
   <county>7</county>
   <country>0</country>
   <postCode>BD18 3NX</postCode>
   <telephone>22222</telephone>
   <email>example@test.com</email>
   <ethnicOrigin>0</ethnicOrigin>
   <accountExpiryDate>07/03/2009</accountExpiryDate>
   <username>user_xyz</username>
   <extraInfo>
     <specialRequirements/>
   </extraInfo>
   <specialRequirements/>
   <candidateRef>xyz</candidateRef>
   <centreRoles>
     <centre id="1">
       <added>
         <i>6</i>
       </added>
       <removed/>
     </centre>
   </centreRoles>
   <centres>
     <added/>
     <removed/>
   </centres>
   <qualifications>
     <added/>
     <removed/>
   </qualifications>
   <retired>0</retired>
 </user>
</data>

Is it possible to transform the CSV data to XML in SSIS? Thanks
0
Comment
Question by:meninga
  • 2
  • 2
6 Comments
 
LVL 8

Expert Comment

by:srnar
ID: 21751707
Yes it is possible. You can import the CSV into a SQL Server table and then use SELECT FOR XML AUTO statement. Try this link:

msdn defintion
http://msdn.microsoft.com/en-us/library/ms173812.aspx

Longer text from magazin:
http://msdn.microsoft.com/en-us/library/aa175790(SQL.80).aspx
0
 

Author Comment

by:meninga
ID: 21752734
Thanks smar. I want to avoid using a staging table so I've worked out how to transform the data using the Script Component. This XML now complies to the format required to execute a stored procedure.
0
 
LVL 8

Expert Comment

by:srnar
ID: 21753813
Congratulations. Do you mean SSIS Script task? In version 2005 it can be coded only in VB.NET - it has nothing to do with C#.
0
 

Accepted Solution

by:
meninga earned 0 total points
ID: 21758000
smar, theres a script component within the data flow task in SSIS 2005 and I have used this for the conversion (an existing stored procedure needs this xml as a parameter).  The script is in VB.Net and I have posted it below in case you find this handy in the future...
I was tempted to write a web method in c# and convert the CSV into xml using xslt but as I will be needing to process large amounts of data, I thought it would be better to use an SSIS based approach. Then we can just place the CSV in the required directory and launch the SSIS package and our DB is updated appropriately.

Imports System
 

Imports System.Data
 

Imports System.Math
 

Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
 

Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
 

Imports System.IO
 

Imports System.Reflection
 

Public Class ScriptMain
 

    Inherits UserComponent
 

    Private targetFile As String
 

    Private xmlWriter As StreamWriter
 

    Private rootElement As String = "Data"
 

    Private rowElement As String = "User"
 

    Private columns As Integer()
 

    Private columnames As String()
 

    Public Overrides Sub AcquireConnections(ByVal Transaction As Object)
 

        targetFile = CType(Me.Connections.TestConnection.AcquireConnection(Nothing), String)
 

    End Sub
 

    Public Overrides Sub PreExecute()
 

        xmlWriter = New StreamWriter(targetFile, False)
 

        xmlWriter.WriteLine(FormatElement(rootElement))
 

        Dim input As IDTSInput90 = ComponentMetaData.InputCollection(0)
 

        ReDim columns(input.InputColumnCollection.Count)
 

        columns = Me.GetColumnIndexes(input.ID)
 

        Dim column As IDTSInputColumn90
 

        ReDim columnames(input.InputColumnCollection.Count)
 

        Dim counter As Integer
 

        counter = 0
 

        For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
 

            columnames(counter) = column.Name
 

            counter = counter + 1
 

        Next
 

    End Sub
 

    Public Overrides Sub PostExecute()
 

        xmlWriter.WriteLine(FormatElement(rootElement, True))
 

        xmlWriter.Close()
 

    End Sub
 

    'Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
 

    '    Dim column As IDTSInputColumn90
 

    '    Dim rowType As Type = Row.GetType()
 

    '    Dim columnValue As PropertyInfo
 

    '    With xmlWriter
 

    '        .Write(FormatElement(rowElement))
 

    '        For Each column In Me.ComponentMetaData.InputCollection(0).InputColumnCollection
 

    '            columnValue = rowType.GetProperty(column.Name)
 

    '            Try
 

    '                .Write(FormatElement(column.Name) + columnValue.GetValue(Row, Nothing).ToString() + FormatElement(column.Name, True))
 

    '            Catch ex As Exception
 

    '                .Write(FormatElement(column.Name) + "" + FormatElement(column.Name, True))
 

    '            End Try
 

    '        Next
 

    '        .WriteLine(FormatElement(rowElement, True))
 

    '    End With
 

    'End Sub
 

    Public Overrides Sub ProcessInput(ByVal InputID As Integer, ByVal Buffer As Microsoft.SqlServer.Dts.Pipeline.PipelineBuffer)
 

        While Buffer.NextRow()
 

            xmlWriter.Write(FormatElement(rowElement))
 

            Dim counter As Integer
 

            counter = 0
 

            For Each index As Integer In columns
 

                Dim value As Object = Buffer(index)
 

                xmlWriter.Write(FormatElement(columnames(counter).ToString()) + value.ToString() + FormatElement(columnames(counter).ToString(), True))
 

                counter = counter + 1
 

            Next
 

            xmlWriter.WriteLine(FormatElement(rowElement, True))
 

        End While
 

    End Sub
 

    Private Function FormatElement(ByVal elementName As String) As String
 

        Return FormatElement(elementName, False)
 

    End Function
 

    Private Function FormatElement(ByVal elementName As String, ByVal closingTag As Boolean) As String
 

        Dim returnValue As String
 

        If closingTag Then
 

            returnValue = "</"
 

        Else
 

            returnValue = "<"
 

        End If
 

        returnValue += elementName + ">"
 

        Return returnValue
 

    End Function
 

End Class

Open in new window

0

Featured Post

Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Android studio getdrawable(int) is deprecated 4 43
Runtime Error 2 28
C#  Radio button search for Date not DateTime 4 35
Reset asp.net login password 4 34
Update (December 2011): Since this article was published, the things have changed for good for Android native developers. The Sequoyah Project (http://www.eclipse.org/sequoyah/) automates most of the tasks discussed in this article. You can even fin…
Jaspersoft Studio is a plugin for Eclipse that lets you create reports from a datasource.  In this article, we'll go over creating a report from a default template and setting up a datasource that connects to your database.
The viewer will learn how to synchronize PHP projects with a remote server in NetBeans IDE 8.0 for Windows.
The viewer will learn how to use and create new code templates in NetBeans IDE 8.0 for Windows.

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now