Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Fastest way to load data from a csv stream into access

Posted on 2012-03-28
6
Medium Priority
?
664 Views
Last Modified: 2012-06-21
I have a stream of CSV data that I need to load into an access table. The table is initially blank and this will be the only process running.  There are a few values in the input CSV data that need to be cleaned (dates need to be formatted and some strings need to be escaped).

Currently I have the data in a CSVReader and read the data one line at a time and create a OleDbCommand and set the CommandText for the insert.  I then .ExecuteNonQuery to add the single line to the database and then read the next line from the reader and repeat.

Is there a faster way to get the data into access?
0
Comment
Question by:hotshottools
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 74

Expert Comment

by:Jeffrey Coachman
ID: 37778325
CSV, typically denotes a text "file",...but you say that your is a csv "stream"?

Can you save this strem as a file...?
If so, then the import becomes easy.
Just use the Import utility in Access:
External Data-->Import-->Text File-->...

JeffCoachman
0
 

Author Comment

by:hotshottools
ID: 37778342
I'm doing this in code to create the access file and import the data.  My current solution works but seems slow with I'm importing 100,000 records.

My goal is to speed up the code.
0
 
LVL 74

Expert Comment

by:Jeffrey Coachman
ID: 37778709
<My goal is to speed up the code.>
"Speed" is a relative term.

Can you state how long this takes in chronological terms...?

Also it seems then that the creation of this stream is what is taking the most time.

Access can only "easily" take in disk files, not streams directly...

Care to post the details of how you are creating the stream?
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 20

Expert Comment

by:clarkscott
ID: 37780403
Can your csv file be saved as a txt file?   If so...
1.  Open Access and LINK to the txt file.  When the 'wizard' appears, check the box for "first row contains headers" (yes or no) and determine the delimiter (tab, comma, etc.).  Now - look for the ADVANCED button.  This will allow you to set up the FILE STRUCTURE for the txt file (dates, text, numbers, etc.).  When finished - SAVE THE SETUP (there will be a button).

Link to the table where this data should go.
Create a APPEND QUERY using the txt link updating to the target table.

You can "adjust" any data formatting, etc. in the update query.

That will run faster.
Scott C
0
 
LVL 10

Accepted Solution

by:
eguilherme earned 2000 total points
ID: 37781282
check the comments in this question:
http://stackoverflow.com/questions/7070011/writing-large-number-of-records-bulk-insert-to-access-in-net-c

i have not tested it yet, but apparently using access interop to write to the table is way faster than the ado.net method..
0
 

Author Closing Comment

by:hotshottools
ID: 37782224
Moved from OLEDB command to using interop.dbo and found an initial speed increase of 30%.  I'm sure with a little modifications I can cut the time in half.

Thanks;
0

Featured Post

Veeam Task Manager for Hyper-V

Task Manager for Hyper-V provides critical information that allows you to monitor Hyper-V performance by displaying real-time views of CPU and memory at the individual VM-level, so you can quickly identify which VMs are using host resources.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In Part II of this series, I will discuss how to identify all open instances of Excel and enumerate the workbooks, spreadsheets, and named ranges within each of those instances.
Explore the ways to Unlock VBA Project Password Excel 2010 & 2013 documents. Go through the article and perform the steps carefully to remove VBA Excel .xls file.
Show developers how to use a criteria form to limit the data that appears on an Access report. It is a common requirement that users can specify the criteria for a report at runtime. The easiest way to accomplish this is using a criteria form that a…
Learn how to number pages in an Access report over each group. Activate two pass printing by referencing the pages property: Add code to the Page Footers OnFormat event to capture the pages as there occur for each group. Use the pages property to …
Suggested Courses

618 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question