Solved

Access and a Large Text File:  over 8GB

Posted on 2013-10-24
8
500 Views
Last Modified: 2013-10-30
Hello All,

I was hoping to get suggestions on the best way to handle this situation.   We have a large text file with many fields and many records.  The size of the file is over 8GB.   We need to be able to look at the data to ensure fields contain correct data, and to be able to massage some of the data.   We will be using MS Access 2010 as the front end.

I know there is a 2GB limit for an Access file, so I was thinking that we could import all the data to a SQL Server table.  We could then use Access to view and massage the data using vba code.    

I want to install SQL Server on my desktop and create a database with that.  Then I could link to the table that houses all the imported information.

My question is, can you install SQL Server and create a database all on your desktop and not on a server?
0
Comment
Question by:sgtoverlord
  • 4
  • 3
8 Comments
 
LVL 57
ID: 39598348
<<My question is, can you install SQL Server and create a database all on your desktop and not on a server? >>

 Yes, the Express edition is free and you'll have no problems doing that.

 What you've outlined is doable.

Jim.
0
 

Author Comment

by:sgtoverlord
ID: 39598407
Awesome.  

Now I am fairly astute at Access and VBA code, but I haven't created a SQL Server Database before.   Is this something that I should be able to fumble my way thru, or is it completely different?  If so, do you know of a good website or web tutorial that would assist me with this endeavor (and gives correct information)?

One other question:  Will this method be more efficient (faster) than using access to link to the text file as a table itself?   I am hoping that this is the case.  Our process currently takes hours & hours.   I was hoping that using Access with a local SQL Server db would increase the efficiency.
0
 
LVL 26

Expert Comment

by:jerryb30
ID: 39598568
I believe the express edition has a 4gb limit.
0
Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

 

Author Comment

by:sgtoverlord
ID: 39598816
poo
0
 
LVL 57

Accepted Solution

by:
Jim Dettman (Microsoft MVP/ EE MVE) earned 500 total points
ID: 39598863
0
 

Author Comment

by:sgtoverlord
ID: 39598972
Well, I do see that there is a 4GB limit with SQL Server 2005 Express Edition.   And A 10GB limit with SQL Server 2008 R2.  

My company has licenses for SQL 2005 Business Intelligence Development Studio.  Does anyone know the size limit for this?

But is this file size what I am supposed to be worried about.   Let me explain:   Currently, we are using Access 2003 to link to a text file that is 8GB+ in size.   We are able to do some crunching with this arrangement, it just takes forever (7 hours+).    

I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.  

Any thoughts on this???
0
 
LVL 57
ID: 39599974
<<But is this file size what I am supposed to be worried about.   Let me explain:   Currently, we are using Access 2003 to link to a text file that is 8GB+ in size.   >>

 Just because it's 8GB in the file doesn't mean it will be 8GB in the database.   Some may be duplicated for example.  

 When brought into a DB and normalized, it's possible that size would go down.

<<We are able to do some crunching with this arrangement, it just takes forever (7 hours+).  >>

 Depends on what your doing there may be ways to speed that up as it stands.

<<I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.  

Any thoughts on this??? >>

  Too hard to say without a lot more detail (sample of the file, what your looking to get out of it, etc).

Jim.
0
 

Author Closing Comment

by:sgtoverlord
ID: 39612362
Thank you for the information.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

In a multiple monitor setup, if you don't want to use AutoCenter to position your popup forms, you have a problem: where will they appear?  Sometimes you may have an additional problem: where the devil did they go?  If you last had a popup form open…
Why is this different from all of the other step by step guides?  Because I make a living as a DBA and not as a writer and I lived through this experience. Defining the name: When I talk to people they say different names on this subject stuff l…
What’s inside an Access Desktop Database. Will look at the basic interface, Navigation Pane (Database Container), Tables, Queries, Forms, Report, Macro’s, and VBA code.
Viewers will learn how to use the SELECT statement in SQL and will be exposed to the many uses the SELECT statement has.

823 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question