?
Solved

Access and a Large Text File:  over 8GB

Posted on 2013-10-24
8
Medium Priority
?
594 Views
Last Modified: 2013-10-30
Hello All,

I was hoping to get suggestions on the best way to handle this situation.   We have a large text file with many fields and many records.  The size of the file is over 8GB.   We need to be able to look at the data to ensure fields contain correct data, and to be able to massage some of the data.   We will be using MS Access 2010 as the front end.

I know there is a 2GB limit for an Access file, so I was thinking that we could import all the data to a SQL Server table.  We could then use Access to view and massage the data using vba code.    

I want to install SQL Server on my desktop and create a database with that.  Then I could link to the table that houses all the imported information.

My question is, can you install SQL Server and create a database all on your desktop and not on a server?
0
Comment
Question by:sgtoverlord
  • 4
  • 3
8 Comments
 
LVL 58
ID: 39598348
<<My question is, can you install SQL Server and create a database all on your desktop and not on a server? >>

 Yes, the Express edition is free and you'll have no problems doing that.

 What you've outlined is doable.

Jim.
0
 

Author Comment

by:sgtoverlord
ID: 39598407
Awesome.  

Now I am fairly astute at Access and VBA code, but I haven't created a SQL Server Database before.   Is this something that I should be able to fumble my way thru, or is it completely different?  If so, do you know of a good website or web tutorial that would assist me with this endeavor (and gives correct information)?

One other question:  Will this method be more efficient (faster) than using access to link to the text file as a table itself?   I am hoping that this is the case.  Our process currently takes hours & hours.   I was hoping that using Access with a local SQL Server db would increase the efficiency.
0
 
LVL 26

Expert Comment

by:jerryb30
ID: 39598568
I believe the express edition has a 4gb limit.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:sgtoverlord
ID: 39598816
poo
0
 
LVL 58

Accepted Solution

by:
Jim Dettman (Microsoft MVP/ EE MVE) earned 2000 total points
ID: 39598863
0
 

Author Comment

by:sgtoverlord
ID: 39598972
Well, I do see that there is a 4GB limit with SQL Server 2005 Express Edition.   And A 10GB limit with SQL Server 2008 R2.  

My company has licenses for SQL 2005 Business Intelligence Development Studio.  Does anyone know the size limit for this?

But is this file size what I am supposed to be worried about.   Let me explain:   Currently, we are using Access 2003 to link to a text file that is 8GB+ in size.   We are able to do some crunching with this arrangement, it just takes forever (7 hours+).    

I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.  

Any thoughts on this???
0
 
LVL 58
ID: 39599974
<<But is this file size what I am supposed to be worried about.   Let me explain:   Currently, we are using Access 2003 to link to a text file that is 8GB+ in size.   >>

 Just because it's 8GB in the file doesn't mean it will be 8GB in the database.   Some may be duplicated for example.  

 When brought into a DB and normalized, it's possible that size would go down.

<<We are able to do some crunching with this arrangement, it just takes forever (7 hours+).  >>

 Depends on what your doing there may be ways to speed that up as it stands.

<<I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.  

Any thoughts on this??? >>

  Too hard to say without a lot more detail (sample of the file, what your looking to get out of it, etc).

Jim.
0
 

Author Closing Comment

by:sgtoverlord
ID: 39612362
Thank you for the information.
0

Featured Post

Prep for the ITIL® Foundation Certification Exam

December’s Course of the Month is now available! Enroll to learn ITIL® Foundation best practices for delivering IT services effectively and efficiently.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

An alternative to the "For XML" way of pivoting and concatenating result sets into strings, and an easy introduction to "common table expressions" (CTEs). Being someone who is always looking for alternatives to "work your data", I came across this …
Windows Explorer let you handle zip folders nearly as any other folder: Copy, move, change, and delete, etc. In VBA you can also handle normal files and folders, but zip folders takes a little more - and that you'll find here.
Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.
Look below the covers at a subform control , and the form that is inside it. Explore properties and see how easy it is to aggregate, get statistics, and synchronize results for your data. A Microsoft Access subform is used to show relevant calcul…
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question