• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 747
  • Last Modified:

Fastest way to load data from a csv stream into access

I have a stream of CSV data that I need to load into an access table. The table is initially blank and this will be the only process running.  There are a few values in the input CSV data that need to be cleaned (dates need to be formatted and some strings need to be escaped).

Currently I have the data in a CSVReader and read the data one line at a time and create a OleDbCommand and set the CommandText for the insert.  I then .ExecuteNonQuery to add the single line to the database and then read the next line from the reader and repeat.

Is there a faster way to get the data into access?
1 Solution
Jeffrey CoachmanMIS LiasonCommented:
CSV, typically denotes a text "file",...but you say that your is a csv "stream"?

Can you save this strem as a file...?
If so, then the import becomes easy.
Just use the Import utility in Access:
External Data-->Import-->Text File-->...

hotshottoolsAuthor Commented:
I'm doing this in code to create the access file and import the data.  My current solution works but seems slow with I'm importing 100,000 records.

My goal is to speed up the code.
Jeffrey CoachmanMIS LiasonCommented:
<My goal is to speed up the code.>
"Speed" is a relative term.

Can you state how long this takes in chronological terms...?

Also it seems then that the creation of this stream is what is taking the most time.

Access can only "easily" take in disk files, not streams directly...

Care to post the details of how you are creating the stream?
Get 10% Off Your First Squarespace Website

Ready to showcase your work, publish content or promote your business online? With Squarespace’s award-winning templates and 24/7 customer service, getting started is simple. Head to Squarespace.com and use offer code ‘EXPERTS’ to get 10% off your first purchase.

Can your csv file be saved as a txt file?   If so...
1.  Open Access and LINK to the txt file.  When the 'wizard' appears, check the box for "first row contains headers" (yes or no) and determine the delimiter (tab, comma, etc.).  Now - look for the ADVANCED button.  This will allow you to set up the FILE STRUCTURE for the txt file (dates, text, numbers, etc.).  When finished - SAVE THE SETUP (there will be a button).

Link to the table where this data should go.
Create a APPEND QUERY using the txt link updating to the target table.

You can "adjust" any data formatting, etc. in the update query.

That will run faster.
Scott C
check the comments in this question:

i have not tested it yet, but apparently using access interop to write to the table is way faster than the ado.net method..
hotshottoolsAuthor Commented:
Moved from OLEDB command to using interop.dbo and found an initial speed increase of 30%.  I'm sure with a little modifications I can cut the time in half.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Train for your Pen Testing Engineer Certification

Enroll today in this bundle of courses to gain experience in the logistics of pen testing, Linux fundamentals, vulnerability assessments, detecting live systems, and more! This series, valued at $3,000, is free for Premium members, Team Accounts, and Qualified Experts.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now