Question

Extracting data from multiple tables to flat files using SSIS

Asked by: vvkp

Hello,
1. I have lot of different tables data to be extracted to flat files like table1, table2, table3.....tableXXX.
2. I need to run select * from each table and capturing the data in flat files by naming the file as table name.
I am trying to use foreach loop in the case but getting error...I captured table names (table1, table2 etc.,) and Query (select * from table1, select * from table2 etc., ) to a temp table and then using the variables tablename and query as variables in the foreach loop container and ending with error.

Error at Getting Data to Flatfiles [OLE DB Source [1]]: SSIS Error Code DTS_E_OLEDBERROR.  An OLE DB error has occurred. Error code: 0x80040E0C.
An OLE DB record is available.  Source: "Microsoft SQL Native Client"  Hresult: 0x80040E0C  Description: "Command text was not set for the command object.".

So my question is ...I heard some where that each table contains different columns and the each query leads to different columns and hence the columns are changing flatfile extraction is not possible. Is that correct?
Can you kindly suggest some method to my scenario?
thanks in advance.
Krishna.

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2009-10-11 at 08:59:39ID24802845
Tags

Using different queries in SSIS foreach loop container.

Topic

SSIS

Participating Experts
1
Points
200
Comments
35

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. SSIS flat file convert
    I am converting a flat file as fixed width using ssis. I have the package built but instead of running a sql after the package builds the table, I would like to do all the cast and converts in the package. How in the ssis package can I do a convert of varchar(6) to datetime?...
  2. SSIS - how to import a DT_STR data type from a csv flat f…
    Hi I am very new to SSIS so the answer to this may be very easy indeed - dont forget to suggest the obvious! I am importing a csv file in which the first column represents a date, and is in the form: 2007-04-25 00:00:00.000 The output column data type is DT_STR and the des...
  3. SSIS OLE DB error
    Hi All: I am trying to connect a pipe formatted flat file to my SQL server by creating a SSIS package,I am getting the following errors: [OLE DB Destination [1200]] Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E2F. An OLE DB reco...
  4. SSIS
    First let me say t hat all of my experience is with DTS packages in SQL Server 2000. I am trying to create a very simple Bulk Insert Task in SSIS to load a pipe delimieted file to an empty table. I am able to preview the source file perfectly in the Connection Manager and ...
  5. SSIS Import flat file with zone signed values
    Can SSIS flat file connection convert zone signed values?

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: vvkpPosted on 2009-10-12 at 08:40:08ID: 25552157

Interesting....no body answered my question and infact no body tried to answer...is this that much tough? Just wondering...

 

by: stelth240Posted on 2009-10-14 at 20:14:29ID: 25576953

Sometimes experts skim over questions and may not answer them immediately  It has been a few days though at this point.

It is true that if the tables contain different columns or data types, called metadata, the tables have to be explicitly created within a data source.  A Data Flow Task will always contain a source, a destination and may contain transformations, all of which send data through the "pipeline".  The pipeline always has to have a specific set of metadata assigned to it.  It's done this way because of validation, which needs to know what the columns and data types are at design-time and run-time.

A Script Task can be used to create a data table that you can fill with the specific table you're looking for, and then it can write that data table out to a text file.  A loop in the Script Task can loop over the tables needed.  Unfortunately, that really negates the need to use SSIS at that point, unless other processing is going on that SSIS is needed for.

If you need any help with creating a Script Task for it, let me know.

 

by: vvkpPosted on 2009-10-14 at 20:20:06ID: 25576968

Hello Steith,
Thanks a lot for a detailed reply. I am newbe in this area.
I shall be very thankful to you and badly needed too for that scrpipt task. Kindly help me.
Table1, table2, table3 ....table12.... all the queries are select * from table1, select * from table2....like that all the tables. All the tables have different columns,
Thanks in advance. Really waiting for the help with lot of hopes to save my skin at work.

 

by: stelth240Posted on 2009-10-15 at 05:22:13ID: 25579698

This code should do what you're looking for, though there's still a lot to be desired with this setup.  Take a look at it and see if it fits your need.  The only thing you'll probably need to change is the Initial Catalog in the connection string and the path as well.  Also, if you're using a connection manager, you can grab it's connection string by doing:

cn = New SqlConnection(Dts.Connections("ConnectionManagerName").ConnectionString)

    Public Sub Main()
        Dim ShowHeadings As Boolean = True
        Dim TextQualified As Boolean = True
        Dim delimiter As String = ", "
        Dim FilePath As String = "C:\"
        Dim result As Integer = Dts.Results.Success 
        Dim cn As SqlConnection
        Dim cm As SqlCommand
        Dim sw As StreamWriter
        Dim reader As SqlDataReader 
        Try
            cn = New SqlConnection("Data Source=(local);Initial Catalog=Database;Trusted_Connection=True;")
            cn.Open() 
            cm = New SqlCommand()
            cm.Connection = cn
            cm.CommandType = CommandType.Text 
            For TableNum As Integer = 1 To 12
                cm.CommandText = "SELECT * FROM Table" & CStr(TableNum)
                reader = cm.ExecuteReader(CommandBehavior.SingleResult) 
                Dim dt As New DataTable()
                dt.Load(reader) 
                sw = New StreamWriter(FilePath & "Table" & CStr(TableNum) & ".txt", False) 
                If ShowHeadings Then
                    For Each Col As DataColumn In dt.Columns
                        sw.Write(Col.ColumnName)
                        If Col.Ordinal <> dt.Columns.Count - 1 Then sw.Write(delimiter)
                    Next 
                    sw.WriteLine()
                End If 
                For Each row As DataRow In dt.Rows
                    Dim arr As Object() = row.ItemArray
                    Dim bob As Type 
                    For ColNum As Integer = 0 To arr.Length - 1
                        Dim TypeString As String = arr(ColNum).GetType().ToString() 
                        If TypeString = "System.String" Then sw.Write("""")
                        sw.Write(arr(ColNum).ToString())
                        If TypeString = "System.String" Then sw.Write("""")
                        If ColNum <> arr.Length - 1 Then sw.Write(delimiter)
                    Next ColNum 
                    sw.WriteLine()
                Next 
                sw.Close()
            Next TableNum 
            Dts.TaskResult = Dts.Results.Success
        Catch ex As Exception
            Dts.Events.FireError(0, String.Empty, ex.Message, String.Empty, 0)
            result = Dts.Results.Failure
        Finally
            sw.Dispose()
            cm.Dispose()
            If Not cn.State = ConnectionState.Closed Then cn.Close()
            cn.Dispose()
        End Try 
    End Sub

                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:

Select allOpen in new window

 

by: vvkpPosted on 2009-10-15 at 10:37:55ID: 25582795

First of ALL , please accept my SINCERE thanks for all the help you did.

I pasted the code as it is in script task...its throwing some errors like

        Dim cn As SqlConnection - SqlConnection is not defined.
        Dim cm As SqlCommand - SqlCommand is not defined
        Dim sw As StreamWriter - Streamwriter is not defined
        Dim reader As SqlDataReader - SqlDataReader is not defined
        StreamWriter - streamwriter is not defined.

        Another thing is my tables are not defined like table1, table2 etc., they have different names each. Just an example I mentioned. Each table has different name.
How to fix these? By the way this procedure works for different tables also?

 

by: stelth240Posted on 2009-10-15 at 11:34:36ID: 25583397

Wow... I completely forgot to mention a few other pieces that are pretty crucial.  I'll mention them in a second.  As far as the table names, it really sounded like you were using Table1, 2, etc., but I had a feeling that's not what they were exactly.  There are two ways to work with the table names.  You can either define a List or array of table names in the code and iterate through that, or you can create a ForEach Loop in the Control Flow layout.  You would then set the loop to an Item Enumerator, add a column to the collection, and fill it with table names.  Then you would assign that loop to a variable that will hold the current table name.  If you need help with any of that, let me know.

Back to the other pieces... I forgot to mention that you'll have to add a reference to the VSA project, and add a few import lines to the top of the script.  Here's how to add the references:

In the script window, go to the Project menu, and click Add Reference.  Then in the list that appears, find System.Xml (probably the last one) and click the Add button at the bottom.  Then Click OK.

To add the Import lines, just write these three lines at the top where the other Import lines are:

Import System.Data.SqlClient
Import System.IO
Import System.Xml

It doesn't matter what order you write them in.  Once you do that, the code should run fine.

 

by: vvkpPosted on 2009-10-15 at 12:11:52ID: 25583713

I did exactly what you instructed....
Unfortunately I am getting the following error...

SSIS package "Flatfile_Nodata.dtsx" starting.
Error: 0x0 at Script Task: Keyword not supported: 'provider'.
Error: 0x2 at Script Task: The script threw an exception: Object reference not set to an instance of an object.
Task failed: Script Task

 

by: stelth240Posted on 2009-10-15 at 12:16:19ID: 25583757

Is it possible to copy and paste the entire contents of the script into a comment?  The provider piece points to something possible with the connection string for the SqlConnection variable.  Did you try to do the:

cn = New SqlConnection(Dts.Connections("ConnectionManagerName").ConnectionString)

or use the original code?  The Connection Manager above is for the OleDB connection to the SQL Server database, and the error above indicates that you're using a connection manager other than that one.

 

by: vvkpPosted on 2009-10-15 at 12:23:34ID: 25583829

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.

Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
Imports System.Data.SqlClient
Imports System.IO
Imports System.Xml

Public Class ScriptMain

      ' The execution engine calls this method when the task executes.
      ' To access the object model, use the Dts object. Connections, variables, events,
      ' and logging features are available as static members of the Dts class.
      ' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
      '
      ' To open Code and Text Editor Help, press F1.
      ' To open Object Browser, press Ctrl+Alt+J.


    Public Sub Main()
        Dim ShowHeadings As Boolean = True
        Dim TextQualified As Boolean = True
        Dim delimiter As String = "^^"
        Dim FilePath As String = "C:\"
        Dim result As Integer = Dts.Results.Success
        Dim cn As SqlConnection
        Dim cm As SqlCommand
        Dim sw As StreamWriter
        Dim reader As SqlDataReader
        Try
            cn = New SqlConnection(Dts.Connections("INDIA.DBA").ConnectionString)

            cn.Open()
            cm = New SqlCommand()
            cm.Connection = cn
            cm.CommandType = CommandType.Text
            For TableNum As Integer = 1 To 12
                cm.CommandText = "SELECT * FROM DBA.dbo.invoices" & CStr(TableNum)
                reader = cm.ExecuteReader(CommandBehavior.SingleResult)
                Dim dt As New DataTable()
                dt.Load(reader)
                sw = New StreamWriter(FilePath & "invoices" & CStr(TableNum) & ".txt", False)
                If ShowHeadings Then
                    For Each Col As DataColumn In dt.Columns
                        sw.Write(Col.ColumnName)
                        If Col.Ordinal <> dt.Columns.Count - 1 Then sw.Write(delimiter)
                    Next
                    sw.WriteLine()
                End If
                For Each row As DataRow In dt.Rows
                    Dim arr As Object() = row.ItemArray
                    Dim bob As Type
                    For ColNum As Integer = 0 To arr.Length - 1
                        Dim TypeString As String = arr(ColNum).GetType().ToString()
                        If TypeString = "System.String" Then sw.Write("""")
                        sw.Write(arr(ColNum).ToString())
                        If TypeString = "System.String" Then sw.Write("""")
                        If ColNum <> arr.Length - 1 Then sw.Write(delimiter)
                    Next ColNum
                    sw.WriteLine()
                Next
                sw.Close()
            Next TableNum
            Dts.TaskResult = Dts.Results.Success
        Catch ex As Exception
            Dts.Events.FireError(0, String.Empty, ex.Message, String.Empty, 0)
            result = Dts.Results.Failure
        Finally
            sw.Dispose()
            cm.Dispose()
            If Not cn.State = ConnectionState.Closed Then cn.Close()
            cn.Dispose()
        End Try

        Dts.TaskResult = Dts.Results.Success
    End Sub

End Class

 

by: vvkpPosted on 2009-10-15 at 12:25:57ID: 25583853

My connection name is INDIA.DBA
Servername is INDIA
Database name is DBA
tables names are invoices1, invoices2, invoices3, .....

 

by: stelth240Posted on 2009-10-15 at 12:44:07ID: 25584006

It looks like the connection string for an OLEDB Connection Manager is in the OleDbConnection format, which actually makes a lot of sense.  In that case, you can keep the SqlConnection object, and just specify the connection string right in the code like my example originally does.

If you don't want to do that, you'll have to change the SqlConnection object to an OleDbConnection object.  I haven't tested if it works but I can throw it in my code and see if it works.  You can also try it but I don't know if a SqlCommand object can use an OleDbConnection object.  It uses the provider identifier so it should work with the Connection Manager's connection string.

 

by: vvkpPosted on 2009-10-15 at 12:53:01ID: 25584108

For our environment I think Oledbconnection is needed but not sure.
How can I change Sqlconnection object o an OleDbconnection object?

 

by: stelth240Posted on 2009-10-15 at 12:54:18ID: 25584126

This is what I have now, and it tested 100% successfully.  It's the whole thing so do a select all and paste this code in.  It used my connection manager that I set up on my package, so it should work with your connection manager too if it's OLE DB.  Let me know how it works for you.  Also keep in mind that this package is checking 12 tables.  Make sure you have 12 invoices tables before running  it because if you have less it will fail again.

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task. 
Imports System
Imports System.Data
Imports System.Data.OleDb
Imports System.IO
Imports System.Math
Imports System.Xml
Imports Microsoft.SqlServer.Dts.Runtime 
Public Class ScriptMain 
	' The execution engine calls this method when the task executes.
	' To access the object model, use the Dts object. Connections, variables, events,
	' and logging features are available as static members of the Dts class.
	' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
	' 
	' To open Code and Text Editor Help, press F1.
    ' To open Object Browser, press Ctrl+Alt+J. 

    Public Sub Main()
        Dim ShowHeadings As Boolean = True
        Dim TextQualified As Boolean = True
        Dim delimiter As String = "^^"
        Dim FilePath As String = "C:\"
        Dim result As Integer = Dts.Results.Success 
        Dim cn As OleDbConnection
        Dim cm As OleDbCommand
        Dim sw As StreamWriter
        Dim reader As OleDbDataReader 
        Try
            cn = New OleDb.OleDbConnection(Dts.Connections("INDIA.DBA").ConnectionString)
            cn.Open() 
            cm = New OleDbCommand()
            cm.Connection = cn
            cm.CommandType = CommandType.Text 
            For TableNum As Integer = 1 To 12
                cm.CommandText = "SELECT * FROM DBA.dbo.invoices" & CStr(TableNum)
                reader = cm.ExecuteReader(CommandBehavior.SingleResult) 
                Dim dt As New DataTable()
                dt.Load(reader) 
                sw = New StreamWriter(FilePath & "Invoices" & CStr(TableNum) & ".txt", False) 
                If ShowHeadings Then
                    For Each Col As DataColumn In dt.Columns
                        sw.Write(Col.ColumnName)
                        If Col.Ordinal <> dt.Columns.Count - 1 Then sw.Write(delimiter)
                    Next 
                    sw.WriteLine()
                End If 
                For Each row As DataRow In dt.Rows
                    Dim arr As Object() = row.ItemArray
                    Dim bob As Type 
                    For ColNum As Integer = 0 To arr.Length - 1
                        Dim TypeString As String = arr(ColNum).GetType().ToString() 
                        If TypeString = "System.String" Then sw.Write("""")
                        sw.Write(arr(ColNum).ToString())
                        If TypeString = "System.String" Then sw.Write("""")
                        If ColNum <> arr.Length - 1 Then sw.Write(delimiter)
                    Next ColNum 
                    sw.WriteLine()
                Next 
                sw.Close()
            Next TableNum 
            Dts.TaskResult = Dts.Results.Success
        Catch ex As Exception
            Dts.Events.FireError(0, String.Empty, ex.Message, String.Empty, 0)
            result = Dts.Results.Failure
        Finally
            sw.Dispose()
            cm.Dispose()
            If Not cn.State = ConnectionState.Closed Then cn.Close()
            cn.Dispose()
        End Try 
    End Sub 
End Class

                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:

Select allOpen in new window

 

by: vvkpPosted on 2009-10-15 at 13:04:47ID: 25584253

Awesome. You are the man....! Thanks a BUNCH. I owe you a very good lunch..
It basically worked for me. Now I can convert to my environment becuase we have a table that contains tables list those are needed to be run for sql commands. As you I can try foreach loop and get back to you.
Quick question...can we implement all the flat file formats here also like ....

locale:  Unicode
Format: Delimited
Head row delimiter {CR}{LF}
row delimiter  {CR}{LF}
Also I saw every identity columns have {  }  Is it possible to implement all those things in this script?
Please let me know

 

by: stelth240Posted on 2009-10-15 at 13:44:56ID: 25584711

For the table list, you can put a data flow task above the Script Task, and in there grab the table list in the source, and put it into a recordset destination.  Then you can put the Script Task in a ForEach Loop and use the ADO enumerator to grab that list and store it in a variable to use in the Script Task.

Yes, you actually can implement those formats.  I'm not sure about the unicode locale, though the StreamWriter object might have a property for encoding.

For the delimiter, that's already set in the script, though there's nothing set to say delimited or not.  Also, Fixed Width formatting would be more difficult to achieve, but it is possible.  The header and row delimiters can be set like this:

Dim HeadDelimiter As String = Chr(10) & Chr(13)
Dim RowDelimiter As String = Chr(10) & Chr(13)

and then instead of the sw.WriteLine() methods, you would replace those with sw.Write(HeadDelimiter) and sw.Write(RowDelimiter) in the appropriate area.

I don't know what the {  } is in reference to the identity columns.  Is it an array of data?

This script task is definitely starting to sound more and more like a custom task, though they're not simple to create so it would only be beneficial if this script was used over and over in multiple packages.

 

by: stelth240Posted on 2009-10-15 at 13:51:53ID: 25584783

For the encoding type, you can add this line anywhere before the StreamWriter is initialized:

    Dim EncodingType As System.Text.Encoding = System.Text.Encoding.Unicode

and then change the sw variable to this:

    sw = New StreamWriter(FilePath & "Invoices" & CStr(TableNum) & ".txt", False, EncodingType)

You can use any of the properties in the System.Text.Encoding class besides Unicode.  You can also add an Imports line for System.Text so you don't have to keep writing it out.

 

by: vvkpPosted on 2009-10-15 at 16:31:39ID: 25585860

I am almost done with the script except that {  before and } after uniqueidentifier values.
by the way, Is it {CR}{LF} Chr(10) & Chr(13) or Chr(13) & Chr(10)
I am getting all the rows as single row...am I doing something wrong?

 

by: stelth240Posted on 2009-10-15 at 19:52:19ID: 25586745

As far as I'm aware, it should be

Dim HeadDelimiter As String = Chr(10) & Chr(13)
Dim RowDelimiter As String = Chr(10) & Chr(13)

and then you would replace the WriteLine methods with this:

sw.Write(HeadDelimiter)
sw.Write(RowDelimiter)

That would go in place of the WriteLine methods, like in the code below.  There's a property in the StreamWriter that's called NewLine.  It has the {CR} {LF} delimiter in there by default, and you could use that, but then you wouldn't be able to specify a header and a row one independently.

If it still all goes on one line, let me know.

Dim HeadDelimiter As String = Chr(10) & Chr(13)
 
If ShowHeadings Then
     For Each Col As DataColumn In dt.Columns
         sw.Write(Col.ColumnName)
         If Col.Ordinal <> dt.Columns.Count - 1 Then sw.Write(delimiter)
     Next 
sw.Write(HeadDelimiter)
End If 

                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:

Select allOpen in new window

 

by: vvkpPosted on 2009-10-16 at 14:15:58ID: 25593421

Hi Stelth,
I am getting everything as expected except two thins...can you help me for that too please?

1. I am getting uniqueidentifier as it is where as if I do by DTS package for table I am getting { }
     example: with the  script I am getting the value  as ====>   0g0e30c9-8eae-4g79-9882-07cg8bc5c10b
                    with DTS pack I am getting the value as ====>  {0G0E30C9-8EAE-4G79-9882-07CG8BC5C10B}

Why is that difference?

2. Date format is also different...
     example:   with the script I am getting====>   10/7/2009 7:25:39 AM
                       with dts package I am getting as====>   2009-10-07 07:25:39.507000000

Why is that?
Is it possible to fix this? Kindly let me know. I know you worked a lot for me. Really I appreciate. Thanks in advance.

 

by: stelth240Posted on 2009-10-16 at 14:36:18ID: 25593562

This question is definitely getting more complex, especially for only 125 pts.  Is it possible to maybe raise the question points?  I definitely want to keep helping out, but for the amount of work involved and points being the only compensation to show for it, it would definitely be nice.

I'm not too sure what you mean when you say dts.  Are you talking about the SSIS package or an old dts package?  If you're talking about an SSIS package, were you actually sending the table through a data flow task to test the data?  If that's true, then it seems like the script is changing how the formatting is set.  I've never worked with uniqueidentifier types before, only datetime types.  The datetime type never looked like what the "dts package" line shows.  Though with the script the AM doesn't make sense either... it should really be mm/dd/yyyy hh:MM:ss.mmm.

I'm going to test the data in a little bit and get back to you.  It seems like the two data types act differently within SSIS than they do in .NET.

 

by: vvkpPosted on 2009-10-16 at 14:53:35ID: 25593660

Thanks for your  reply.
Yes...DTS means SSIS...sorry about writing DTS. Definitely I can give more points Really you helped me.  I want you make really happy in terms of pointsfor all the help. If you are in texas I love to meet you for giving good party for you.

 

by: vvkpPosted on 2009-10-16 at 14:54:27ID: 25593664

By the way how can I raise the points?

 

by: stelth240Posted on 2009-10-16 at 15:24:56ID: 25593798

I did verify that it's because the data types are slightly different.  Here are the data types for each scenario:

DB Engine = uniqueidentifier, datetime

SSIS = DT_GUID, (DT_DBDATE, DT_DBTIMESTAMP)

.NET = System.Guid, System.DateTime

When SSIS converts a DT_GUID entry to a string, it tacks the braces on the ends automagically.  The other two systems (DB Engine and .NET) don't do that.  I believe you really don't want the braces there so you should leave them out, but if you want them in, read on.

The same goes for the date.  The System.DateTime type automatically converts the entry to a formatted date/time when the .ToString() method is used with no arguments.  On the flip side though, I'm surprised SSIS is outputting that many zeros in the time.  There should only be 3 zeros after it which is what the 507 would represent in your example.

Anyway, in order to format any specific values to whatever you want in your output, you're going to have to put some more type checks in the code.  At the part where it says:

-------------------------------------------------------------------------------------
Dim TypeString As String = arr(ColNum).GetType().ToString()

If TypeString = "System.String" Then sw.Write("""")
sw.Write(arr(ColNum).ToString())
If TypeString = "System.String" Then sw.Write("""")
-------------------------------------------------------------------------------------

You're going to want to add extra If blocks like this:

-------------------------------------------------------------------------------------
If TypeString = "System.Guid" Then sw.Write("{")
...
If TypeString = "System.Guid" Then sw.Write("}")
-------------------------------------------------------------------------------------

and:

-------------------------------------------------------------------------------------
...
If TypeString = "Sytem.DateTime" Then
    sw.Write(CDate(arr(ColNum)).ToString("MM/dd/yyyy HH:mm:ss.fff"))
Else
    sw.Write(arr(ColNum).ToString())
End If
...
-------------------------------------------------------------------------------------

That last part would end up replacing the regular sw.Write that was already there with that whole block.  It's not very elegant, but it works.

Unfortunately, it is definitely starting to look like a very complex setup, especially thinking about all the other formats that are necessary.  I'm curious though, how many tables are needed that have different formatting that's causing this script task to be necessary?  We often have well over 10 - 15 tables that we set up in data flow tasks all the time which are much easier to maintain and work with than a script task of this caliber.  Just curious.

 

by: stelth240Posted on 2009-10-16 at 15:28:51ID: 25593816

You can raise the points with the Increase Points button right near the comment box.  It has the number of points in the question currently and can be changed to any number up to 500.  This question is definitely more of a 500 point question, though the amount is totally up to you, especially if you have limited points to spend.

I have to head out in a bit so I may not get any immediate responses but I'll check back soon.

 

by: vvkpPosted on 2009-10-16 at 16:00:27ID: 25593960

The date format is not working. I am getting the same old format.

 

by: stelth240Posted on 2009-10-17 at 07:46:00ID: 25596069

It worked great for me.  Must have been mistyped within the code.  Here's the full code I used including the text qualifier, the GUID braces, and the datetime check.  It's the whole script so you can replace everything in your script with it... hopefully you haven't changed anything else in it yet.

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.
 
Imports System
Imports System.Data
Imports System.Data.OleDb
Imports System.IO
Imports System.Math
Imports System.Xml
Imports Microsoft.SqlServer.Dts.Runtime
 
Public Class ScriptMain
 
	' The execution engine calls this method when the task executes.
	' To access the object model, use the Dts object. Connections, variables, events,
	' and logging features are available as static members of the Dts class.
	' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
	' 
	' To open Code and Text Editor Help, press F1.
    ' To open Object Browser, press Ctrl+Alt+J.
 
 
    Public Sub Main()
        Dim ShowHeadings As Boolean = True
        Dim ShowGuidBraces As Boolean = True
        Dim TextQualified As Boolean = True
        Dim delimiter As String = "^^"
        Dim FilePath As String = "C:\"
        Dim result As Integer = Dts.Results.Success
        Dim EncodingType As System.Text.Encoding = System.Text.Encoding.Unicode
 
        Dim cn As OleDbConnection
        Dim cm As OleDbCommand
        Dim sw As StreamWriter
        Dim reader As OleDbDataReader
 
        Try
            cn = New OleDb.OleDbConnection(Dts.Connections("INDIA.DBA").ConnectionString)
            cn.Open()
 
            cm = New OleDbCommand()
            cm.Connection = cn
            cm.CommandType = CommandType.Text
 
            For TableNum As Integer = 1 To 12
                cm.CommandText = "SELECT * FROM dbo.invoices" & CStr(TableNum)
                reader = cm.ExecuteReader(CommandBehavior.SingleResult)
 
                Dim dt As New DataTable()
                dt.Load(reader)
 
                sw = New StreamWriter(FilePath & "invoices" & CStr(TableNum) & ".txt", False, EncodingType)
 
                If ShowHeadings Then
                    For Each Col As DataColumn In dt.Columns
                        sw.Write(Col.ColumnName)
                        If Col.Ordinal <> dt.Columns.Count - 1 Then sw.Write(delimiter)
                    Next
 
                    sw.WriteLine()
                End If
 
                For Each row As DataRow In dt.Rows
                    Dim arr As Object() = row.ItemArray
                    Dim bob As Type
 
                    For ColNum As Integer = 0 To arr.Length - 1
                        Dim TypeString As String = arr(ColNum).GetType().ToString()
 
                        If TextQualified AndAlso TypeString = "System.String" Then sw.Write("""")
                        If ShowGuidBraces AndAlso TypeString = "System.Guid" Then sw.Write("{")
 
                        If TypeString = "System.DateTime" Then
                            sw.Write(CDate(arr(ColNum)).ToString("MM/dd/yyyy HH:mm:ss.fff"))
                        Else
                            sw.Write(arr(ColNum).ToString())
                        End If
 
                        If ShowGuidBraces AndAlso TypeString = "System.Guid" Then sw.Write("}")
                        If TextQualified AndAlso TypeString = "System.String" Then sw.Write("""")
 
                        If ColNum <> arr.Length - 1 Then sw.Write(delimiter)
                    Next ColNum
 
                    sw.WriteLine()
                Next
 
                sw.Close()
            Next TableNum
 
            Dts.TaskResult = Dts.Results.Success
        Catch ex As Exception
            Dts.Events.FireError(0, String.Empty, ex.Message, String.Empty, 0)
            result = Dts.Results.Failure
        Finally
            sw.Dispose()
            cm.Dispose()
            If Not cn.State = ConnectionState.Closed Then cn.Close()
            cn.Dispose()
        End Try
 
    End Sub
 
End Class

                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:

Select allOpen in new window

 

by: vvkpPosted on 2009-10-17 at 16:06:32ID: 25597956

Thank you very much. Really I appreciate all the help.
Here is the thing ....
I am not getting data from table which has the column type varbinary (256)
Just getting the word  System.Byte[]  in the place of the data for that column.
Other than that every thing is perfect.

 

by: stelth240Posted on 2009-10-17 at 17:32:14ID: 25598172

I don't know a lot about the varbinary column, only how to convert it.  The code to convert the value is a rathelastly, it r long statement:

sw.Write("0x" & BitConverter.ToString(CType(arr(ColNum), System.Byte())).Replace("-", String.Empty))

What this does is it takes the column data, converts it to a System.Byte array, then displays it as a string using the BitConverter class.  This is necessary because the System.Byte() value would have to be looped over to get the values which would then have to be concatenated together.  BitConverter does all that for you.  Finally, a 0x is added to the beginning, and BitConverter outputs each hex value with a dash, so the replace command removes those.

Semi-complicated, but easier than it could be.  I'm not sure if that's the format you're looking for (0xA12B34C56D78E90F) but that's what will output.  The full code is below.  I replaced the TypeString piece with the Typeof...Is operator so the check is more managed.

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic
' The ScriptMain class is the entry point of the Script Task.
 
Imports System
Imports System.Data
Imports System.Data.OleDb
Imports System.IO
Imports System.Math
Imports System.Xml
Imports Microsoft.SqlServer.Dts.Runtime
 
Public Class ScriptMain
 
	' The execution engine calls this method when the task executes.
	' To access the object model, use the Dts object. Connections, variables, events,
	' and logging features are available as static members of the Dts class.
	' Before returning from this method, set the value of Dts.TaskResult to indicate success or failure.
	' 
	' To open Code and Text Editor Help, press F1.
    ' To open Object Browser, press Ctrl+Alt+J.
 
 
    Public Sub Main()
        Dim ShowHeadings As Boolean = True
        Dim ShowGuidBraces As Boolean = True
        Dim TextQualified As Boolean = True
        Dim delimiter As String = "^^"
        Dim FilePath As String = "C:\"
        Dim result As Integer = Dts.Results.Success
        Dim EncodingType As System.Text.Encoding = System.Text.Encoding.Unicode
 
        Dim cn As OleDbConnection
        Dim cm As OleDbCommand
        Dim sw As StreamWriter
        Dim reader As OleDbDataReader
 
        Try
            cn = New OleDb.OleDbConnection(Dts.Connections("INDIA.DBA").ConnectionString)
            cn.Open()
 
            cm = New OleDbCommand()
            cm.Connection = cn
            cm.CommandType = CommandType.Text
 
            For TableNum As Integer = 1 To 12
                cm.CommandText = "SELECT * FROM dbo.invoices" & CStr(TableNum)
                reader = cm.ExecuteReader(CommandBehavior.SingleResult)
 
                Dim dt As New DataTable()
                dt.Load(reader)
 
                sw = New StreamWriter(FilePath & "invoices" & CStr(TableNum) & ".txt", False, EncodingType)
 
                If ShowHeadings Then
                    For Each Col As DataColumn In dt.Columns
                        sw.Write(Col.ColumnName)
                        If Col.Ordinal <> dt.Columns.Count - 1 Then sw.Write(delimiter)
                    Next
 
                    sw.WriteLine()
                End If
 
                For Each row As DataRow In dt.Rows
                    Dim arr As Object() = row.ItemArray
                    Dim bob As Type
 
                    For ColNum As Integer = 0 To arr.Length - 1
                        Dim Col As Object = arr(ColNum)
 
                        If TextQualified AndAlso TypeOf Col Is System.String Then sw.Write("""")
                        If ShowGuidBraces AndAlso TypeOf Col Is System.Guid Then sw.Write("{")
 
                        If TypeOf Col Is System.DateTime Then
                            sw.Write(CDate(arr(ColNum)).ToString("MM/dd/yyyy HH:mm:ss.fff"))
                        ElseIf TypeOf Col Is System.Byte() Then
                            sw.Write("0x" & BitConverter.ToString(CType(arr(ColNum), System.Byte())).Replace("-", String.Empty))
                        Else
                            sw.Write(arr(ColNum).ToString())
                        End If
 
                        If ShowGuidBraces AndAlso TypeOf Col Is System.Guid Then sw.Write("}")
                        If TextQualified AndAlso TypeOf Col Is System.String Then sw.Write("""")
 
                        If ColNum <> arr.Length - 1 Then sw.Write(delimiter)
                    Next ColNum
 
                    sw.WriteLine()
                Next
 
                sw.Close()
            Next TableNum
 
            Dts.TaskResult = Dts.Results.Success
        Catch ex As Exception
            Dts.Events.FireError(0, String.Empty, ex.Message, String.Empty, 0)
            result = Dts.Results.Failure
        Finally
            sw.Dispose()
            cm.Dispose()
            If Not cn.State = ConnectionState.Closed Then cn.Close()
            cn.Dispose()
        End Try
 
    End Sub
 
End Class

                                              
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:

Select allOpen in new window

 

by: vvkpPosted on 2009-10-19 at 09:12:50ID: 25606518

Thank you very much for the reply.
Now its working fine.

Quick question before closing this thread....

Is this script generalized for a table that may contain any type of columns?

After some time the table is altered with different type of columns then this won't work?

Please let me know about the drawbacks of the script.
Appreciate your time and all the patience and kindness you showed to me.

 

by: vvkpPosted on 2009-10-19 at 09:23:57ID: 25606605

Another question ... Is there anyway I can send personal message to you or anyone who answers the questions posted in this website?

 

by: stelth240Posted on 2009-10-19 at 15:18:18ID: 25609492

As far as I know, not at the moment.  I know they're possibly planning to add personal messages into the site, but right now I've never seen an option to private message anyone.  I'll look around though just to make sure.  They might have something here, like in the profile section.

 

by: stelth240Posted on 2009-10-19 at 15:42:46ID: 25609628

Sorry, I missed your comment before the last one.  This actually will work if columns are changed.  The only issue you may find is that a data type that wasn't checked could output differently than you expect.  For instance, you captured DateTime, Uniqueidentifier, varchar (and subsequently char, nchar, nvarchar as well) and varbinary.  If, though, another data type comes in that you haven't formatted, like bit, it could output differently than you'd want.  You would have to add that as a check in the code.

To check for any of the other data types, you would do the same check I did with TypeOf...Is.  Here is a table showing what SQL types match what .NET types: http://msdn.microsoft.com/en-us/library/4e5xt97a(VS.80).aspx.  It's the first table on the page.

But to go back to the question, yes, as long as all the data types will now output as you expect, changing any data should not affect the output.  This is because it's not based on metadata, it's based on a datatable that's iterated.

Hope this helps.  Let me know if there's anything else.

 

by: vvkpPosted on 2009-10-22 at 12:31:12ID: 31639797

He is an EXCELLENT of EXCELLENT guy I ever seen upto now...really lot  of patience and service minded with good heart in help.

 

by: vvkpPosted on 2009-10-22 at 12:47:38ID: 25638207

Hello,
Erroneously I accepted incompleted solution rather than the correct one. Any one could fix that please? The correct solution is stelth240: 's THIRD comment from bottom.

 

by: vvkpPosted on 2009-10-22 at 12:50:18ID: 25638241

stelth240 ... Thank you very much for all your help ...can you email me if possible to vvkprasad@gmail.com  just to say hello and thanks...!

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...