Solved

Access and a Large Text File:  over 8GB

Posted on 2013-10-24
8
494 Views
Last Modified: 2013-10-30
Hello All,

I was hoping to get suggestions on the best way to handle this situation.   We have a large text file with many fields and many records.  The size of the file is over 8GB.   We need to be able to look at the data to ensure fields contain correct data, and to be able to massage some of the data.   We will be using MS Access 2010 as the front end.

I know there is a 2GB limit for an Access file, so I was thinking that we could import all the data to a SQL Server table.  We could then use Access to view and massage the data using vba code.    

I want to install SQL Server on my desktop and create a database with that.  Then I could link to the table that houses all the imported information.

My question is, can you install SQL Server and create a database all on your desktop and not on a server?
0
Comment
Question by:sgtoverlord
  • 4
  • 3
8 Comments
 
LVL 57

Expert Comment

by:Jim Dettman (Microsoft MVP/ EE MVE)
Comment Utility
<<My question is, can you install SQL Server and create a database all on your desktop and not on a server? >>

 Yes, the Express edition is free and you'll have no problems doing that.

 What you've outlined is doable.

Jim.
0
 

Author Comment

by:sgtoverlord
Comment Utility
Awesome.  

Now I am fairly astute at Access and VBA code, but I haven't created a SQL Server Database before.   Is this something that I should be able to fumble my way thru, or is it completely different?  If so, do you know of a good website or web tutorial that would assist me with this endeavor (and gives correct information)?

One other question:  Will this method be more efficient (faster) than using access to link to the text file as a table itself?   I am hoping that this is the case.  Our process currently takes hours & hours.   I was hoping that using Access with a local SQL Server db would increase the efficiency.
0
 
LVL 26

Expert Comment

by:jerryb30
Comment Utility
I believe the express edition has a 4gb limit.
0
 

Author Comment

by:sgtoverlord
Comment Utility
poo
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 57

Accepted Solution

by:
Jim Dettman (Microsoft MVP/ EE MVE) earned 500 total points
Comment Utility
0
 

Author Comment

by:sgtoverlord
Comment Utility
Well, I do see that there is a 4GB limit with SQL Server 2005 Express Edition.   And A 10GB limit with SQL Server 2008 R2.  

My company has licenses for SQL 2005 Business Intelligence Development Studio.  Does anyone know the size limit for this?

But is this file size what I am supposed to be worried about.   Let me explain:   Currently, we are using Access 2003 to link to a text file that is 8GB+ in size.   We are able to do some crunching with this arrangement, it just takes forever (7 hours+).    

I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.  

Any thoughts on this???
0
 
LVL 57

Expert Comment

by:Jim Dettman (Microsoft MVP/ EE MVE)
Comment Utility
<<But is this file size what I am supposed to be worried about.   Let me explain:   Currently, we are using Access 2003 to link to a text file that is 8GB+ in size.   >>

 Just because it's 8GB in the file doesn't mean it will be 8GB in the database.   Some may be duplicated for example.  

 When brought into a DB and normalized, it's possible that size would go down.

<<We are able to do some crunching with this arrangement, it just takes forever (7 hours+).  >>

 Depends on what your doing there may be ways to speed that up as it stands.

<<I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.  

Any thoughts on this??? >>

  Too hard to say without a lot more detail (sample of the file, what your looking to get out of it, etc).

Jim.
0
 

Author Closing Comment

by:sgtoverlord
Comment Utility
Thank you for the information.
0

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Join & Write a Comment

Introduction The Visual Basic for Applications (VBA) language is at the heart of every application that you write. It is your key to taking Access beyond the world of wizards into a world where anything is possible. This article introduces you to…
This article explains how to reset the password of the sa account on a Microsoft SQL Server.  The steps in this article work in SQL 2005, 2008, 2008 R2, 2012, 2014 and 2016.
With Microsoft Access, learn how to start a database in different ways and produce different start-up actions allowing you to use a single database to perform multiple tasks. Specify a start-up form through options: Specify an Autoexec macro: Us…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now