[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.
05/27/2009 at 12:27PM PDT, ID: 24442726
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!
8.0

transfering large amounts of data sql using c#

Asked by kesea in C# Programming Language, SQL Server 2005, SQL Query Syntax

What is the best way to transfer a large amount of data
Say 1.7 Million rows by 8 columns of data into SQL server.  I have an ODBC connection to a prioritary database and I can read the data out (the database is very dumb and can basically just do a read).  I then am importing this into SQL.  

I was doing this line by line using an insert and doing about 150 rows/second, but that takes hours.  Any other way to do this faster?  A code sample in VB.net or C# would be great of putting a bunch of data into a and writing it to SQL. 
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:
124:
125:
126:
127:
128:
129:
130:
131:
132:
133:
134:
135:
136:
137:
138:
139:
140:
141:
142:
143:
144:
145:
146:
147:
148:
149:
150:
151:

string sqlConnectionString = "Data Source=ABC-cgy10;database=ABC;user id=A;pwd=B";
            string tableName = "ABCWTimesheetCurrentTTTTimesheetDetail";
            currentCount_i++;
            I_callback.StepTo(0);
            I_callback.SetText("ABCWTimesheetCurrentTTTTimesheetDetail(" + currentCount_i.ToString() + "/" + NumberOfDatabase_i.ToString() + ")");
            // SQL Server Connection String
            SqlConnection sqlPayrollConnection = new SqlConnection(sqlConnectionString);
            sqlPayrollConnection.Open();
 
            string myConnString = "Provider=SQLOLEDB;Data Source=ABC-cgy10;Initial Catalog=ABC;User Id=A2;Password=B";
            //create the database connection
            OleDbConnection aConnection = new OleDbConnection(myConnString);
 
            //*Let's check if the entry exists! /
            string sqlString = "drop table " + tableName;
            OleDbCommand aCommand = new OleDbCommand(sqlString, aConnection);
            aConnection.Open();
            OleDbDataReader aReader = aCommand.ExecuteReader();
            //Give the starting point of where to get the data from.
            aConnection.Close();//*/
 
            // Establish the database server
            string connectionString = "Data Source=ABC-cgy10;Initial Catalog=ABC;User Id=A;Password=B";
            SqlConnection connection2 =
                 new SqlConnection(connectionString);
            Server server = new Server(new ServerConnection(connection2));
 
            // Create table in my personal database
            Database db = server.Databases["ABCPAYROLL-ST"];
 
            // Create new table, called TestTable
            Table newTable = new Table(db, "ABCWTimesheetCurrentTTTTimesheetDetail");
 
            // Add "Employee_ID" Column
            Column Employee_ID_Column = new Column(newTable, "Employee_ID");
            Employee_ID_Column.DataType = DataType.VarChar(20);
 
            // Add "Regular_Hours" Column
            Column Regular_Hours_Column = new Column(newTable, "Regular_Hours");
            Regular_Hours_Column.DataType = DataType.VarChar(20);
 
            // Add "Regular_Hours" Column
            Column Overtime_Hours_Column = new Column(newTable, "Overtime_Hours");
            Overtime_Hours_Column.DataType = DataType.VarChar(20);
 
            // Add "Regular_Hours" Column
            Column Other_Premium_Hours_Column = new Column(newTable, "Other_Premium_Hours");
            Other_Premium_Hours_Column.DataType = DataType.VarChar(20);
 
            // Add "Regular_Hours" Column
            Column Total_Hours_Column = new Column(newTable, "Total_Hours");
            Total_Hours_Column.DataType = DataType.VarChar(20);
 
            // Add "Regular_Hours" Column
            Column Date_Column = new Column(newTable, "Date");
            Date_Column.DataType = DataType.VarChar(20);
 
            // Add "Verify_Field_1" Column
            Column Verify_Field_1_Column = new Column(newTable, "Verify_Field_1");
            Verify_Field_1_Column.DataType = DataType.VarChar(20);
 
            // Add "Verify_Field_2" Column
            Column Verify_Field_2_Column = new Column(newTable, "Verify_Field_2");
            Verify_Field_2_Column.DataType = DataType.VarChar(20);
 
            // Add "Regular_Hours" Column
            Column Verify_Field_3_Column = new Column(newTable, "Verify_Field_3");
            Verify_Field_3_Column.DataType = DataType.VarChar(20);
 
            // Add "Verify_Field_4" Column
            Column Verify_Field_4_Column = new Column(newTable, "Verify_Field_4");
            Verify_Field_4_Column.DataType = DataType.VarChar(20);
 
            // Add "Verify_Field_5" Column
            Column Verify_Field_5_Column = new Column(newTable, "Verify_Field_5");
            Verify_Field_5_Column.DataType = DataType.VarChar(20);
 
            // Add "Timesheet_Date" Column
            Column Timesheet_Date_Column = new Column(newTable, "Timesheet_Date");
            Timesheet_Date_Column.DataType = DataType.DateTime;
 
            // Add Columns to Table Object
            newTable.Columns.Add(Employee_ID_Column);
            newTable.Columns.Add(Regular_Hours_Column);
            newTable.Columns.Add(Overtime_Hours_Column);
            newTable.Columns.Add(Other_Premium_Hours_Column);
            newTable.Columns.Add(Total_Hours_Column);
            newTable.Columns.Add(Date_Column);
            newTable.Columns.Add(Verify_Field_1_Column);
            newTable.Columns.Add(Verify_Field_2_Column);
            newTable.Columns.Add(Verify_Field_3_Column);
            newTable.Columns.Add(Verify_Field_4_Column);
            newTable.Columns.Add(Verify_Field_5_Column);
            newTable.Columns.Add(Timesheet_Date_Column);
 
            // Physically create the table in the database
            newTable.Create();
 
            string queryString = "SELECT * FROM CURRENT_TTT_TIMESHEET_DETAIL";
            using (OdbcConnection connection = new OdbcConnection("DSN=ABCWTimesheet;UID=A;PWD=B"))
            {
                OdbcCommand command = new OdbcCommand(queryString, connection);
                connection.Open();
                OdbcDataReader reader = command.ExecuteReader();
 
                int count_i = 0;
                int startCount_i = Environment.TickCount;
                int lastCount_i = Environment.TickCount;
 
                //Give the starting point of where to get the data from.
                while (reader.Read())
                {
                    if (reader.GetValue(reader.GetOrdinal("Date")).ToString().Length > 0)
                    {
                        string sqlstring = "insert into " + tableName + " (" +
                            "Employee_ID, Regular_Hours, Overtime_Hours, Other_Premium_Hours, Total_Hours, Date," +
                            "Verify_Field_1, Verify_Field_2, Verify_Field_3, Verify_Field_4, Verify_Field_5, Timesheet_Date" +
                            ") VALUES (" +
                            "'" + reader.GetValue(reader.GetOrdinal("Employee_ID")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Regular_Hours")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Overtime_Hours")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Other_Premium_Hours")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Total_Hours")).ToString() + "'," +
                            "'" + String.Format("{0:MM/dd/yyyy}", Convert.ToDateTime(reader.GetValue(reader.GetOrdinal("Date")).ToString())) + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Verify_Field_1")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Verify_Field_2")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Verify_Field_3")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Verify_Field_4")).ToString() + "'," +
                            "'" + reader.GetValue(reader.GetOrdinal("Verify_Field_5")).ToString() + "'," +
                            "'" + String.Format("{0:MM/dd/yyyy}", Convert.ToDateTime(reader.GetValue(reader.GetOrdinal("Timesheet_Date")).ToString())) + "'" +
                            ")";
                        SqlCommand updateCommand = new SqlCommand(sqlstring, sqlPayrollConnection);
                        updateCommand.ExecuteNonQuery();
                    }
                    count_i++;
                    if (count_i % 2000 == 0)
                    {
                        int progress_i = count_i / 20000;
                        if (progress_i < 100)
                            I_callback.StepTo(progress_i);
                        lastCount_i = Environment.TickCount;
                        int rowsPerSecond = (count_i * 1000) / (lastCount_i - startCount_i);
                        I_callback.SetProgressText("Reading/Writing at " + rowsPerSecond + " rows per second");
                    }
                }
                // Call Close when done reading.
                reader.Close();
                connection.Close();
                aConnection.Close();
            }
        }
[+][-]05/27/09 01:10 PM, ID: 24486987

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

 

About this solution

Zones: C# Programming Language, SQL Server 2005, SQL Query Syntax
Sign Up Now!
Solution Provided By: kesea
Participating Experts: 1
Solution Grade: A
 
 
[+][-]05/27/09 12:33 PM, ID: 24486635

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091028-EE-VQP-86 - Hierarchy / EE_QW_3_20080625