?
Solved

How do I speed up writing to a MySQL database when using VB 2005

Posted on 2009-12-23
12
Medium Priority
?
392 Views
Last Modified: 2013-12-25
I am writing very large files to a MySql database (5.0) through VB 2005 (using MySQL ODBC 3.51 driver) and the speed is very slow.  It is taking 20 to 25 minutes to write 51,000 records.  I have enclosed the code for the insert statement and the execute command.  Any help to speed this up would be greatly appreciated.

 


myCommand.CommandText = "Insert into cmix " _
      & "values('" & strDate & "', " & seq & ", " & endSeq _
      & ", '" & strWhse & "', " & batchNo & ", '" _
      & mailId & "', " & rank & ", '" _
      & sku & "', '" & desc _
       & "', " & qty & ", " & weight & ")" myCommand.ExecuteNonQuery()

Open in new window

0
Comment
Question by:shyanne
12 Comments
 
LVL 14

Accepted Solution

by:
shahprabal earned 501 total points
ID: 26112562
0
 
LVL 24

Assisted Solution

by:Jeff Certain
Jeff Certain earned 498 total points
ID: 26119671
Can't see enough of your code to see if this is the case, but... if you're creating a new connection for each command, that will be a HUGE bottleneck.
You may also want to consider spinning up BackgroundWorkers to allow multi-threaded updates. Much of what you're waiting on is network latencies, so you can run some pretty heavy parallelism.
0
 
LVL 83

Assisted Solution

by:CodeCruiser
CodeCruiser earned 501 total points
ID: 26120419
ODBC is inherently slow. Can you not use Native .NET drivers? How are you reading the file and populating the variables being used in the command above?
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:shyanne
ID: 26130527
I open the database connection before I read the file.  The file is a simple comma delimited text file in which I use the streamreader object  along with the ReadLine method

myCommand.Connection = conn
conn.open

inputFile = File.OpenText("filename")
do  until inputFile.Peek = -1
      strInput = inputFile.ReadLine
      ***Here I set the variables***
Loop

I was experimenting with the load statement in MySQL.  Since this is a comma delimited file it might be faster but I have to add some fields (like the date and a calculated field) and without creating a new comma delimited file I don't know how I could use the load statement.
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 26131529
Did you explore the option of using the native drivers instead of ODBC?
0
 

Author Comment

by:shyanne
ID: 26131681
I'm trying that now.  I have the connection to the database using a dataset but I can't get an insert to work.
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 26133214
Did you configure the dataset using designer? You may want to try the commandbuilder object.
0
 

Author Comment

by:shyanne
ID: 26134047
I did use the designer.  I wanted to see how long it would take with an access database and it took even longer.  What sort of time is resonable to write 51,000 + records?  
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 26136231
I think the main time consuming task is file reading. Comment out the db code and see how long it takes to just read the file and assign variables. Also try to use the file.readalllines method.
0
 

Author Comment

by:shyanne
ID: 26140400
I commented out the db code and read the file and wrote to a text file and it took seconds.  I was wondering if it was something in vb.net so I wrote and ran the code in VB 6.0 and I still have the same problem.  I am going to test the network next.  Any other ideas?  Does it matter that the table is of the  InnoDB type?  Would I see increased performance if it was a MyISAM table type?
0
 
LVL 83

Expert Comment

by:CodeCruiser
ID: 26141585
To be honest, i have no knowledge of mysql so can not comment about the table type. Do test the network. Try writing to a file on the mysql server computer.
0
 

Author Closing Comment

by:shyanne
ID: 31669399
No solution was found but I was provided with some good feedback on different things to try.
0

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Your data is at risk. Probably more today that at any other time in history. There are simply more people with more access to the Web with bad intentions.
Backups and Disaster RecoveryIn this post, we’ll look at strategies for backups and disaster recovery.
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
This lesson covers basic error handling code in Microsoft Excel using VBA. This is the first lesson in a 3-part series that uses code to loop through an Excel spreadsheet in VBA and then fix errors, taking advantage of error handling code. This l…
Suggested Courses

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question