sgtoverlord
asked on
Access and a Large Text File: over 8GB
Hello All,
I was hoping to get suggestions on the best way to handle this situation. We have a large text file with many fields and many records. The size of the file is over 8GB. We need to be able to look at the data to ensure fields contain correct data, and to be able to massage some of the data. We will be using MS Access 2010 as the front end.
I know there is a 2GB limit for an Access file, so I was thinking that we could import all the data to a SQL Server table. We could then use Access to view and massage the data using vba code.
I want to install SQL Server on my desktop and create a database with that. Then I could link to the table that houses all the imported information.
My question is, can you install SQL Server and create a database all on your desktop and not on a server?
I was hoping to get suggestions on the best way to handle this situation. We have a large text file with many fields and many records. The size of the file is over 8GB. We need to be able to look at the data to ensure fields contain correct data, and to be able to massage some of the data. We will be using MS Access 2010 as the front end.
I know there is a 2GB limit for an Access file, so I was thinking that we could import all the data to a SQL Server table. We could then use Access to view and massage the data using vba code.
I want to install SQL Server on my desktop and create a database with that. Then I could link to the table that houses all the imported information.
My question is, can you install SQL Server and create a database all on your desktop and not on a server?
ASKER
Awesome.
Now I am fairly astute at Access and VBA code, but I haven't created a SQL Server Database before. Is this something that I should be able to fumble my way thru, or is it completely different? If so, do you know of a good website or web tutorial that would assist me with this endeavor (and gives correct information)?
One other question: Will this method be more efficient (faster) than using access to link to the text file as a table itself? I am hoping that this is the case. Our process currently takes hours & hours. I was hoping that using Access with a local SQL Server db would increase the efficiency.
Now I am fairly astute at Access and VBA code, but I haven't created a SQL Server Database before. Is this something that I should be able to fumble my way thru, or is it completely different? If so, do you know of a good website or web tutorial that would assist me with this endeavor (and gives correct information)?
One other question: Will this method be more efficient (faster) than using access to link to the text file as a table itself? I am hoping that this is the case. Our process currently takes hours & hours. I was hoping that using Access with a local SQL Server db would increase the efficiency.
I believe the express edition has a 4gb limit.
ASKER
poo
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Well, I do see that there is a 4GB limit with SQL Server 2005 Express Edition. And A 10GB limit with SQL Server 2008 R2.
My company has licenses for SQL 2005 Business Intelligence Development Studio. Does anyone know the size limit for this?
But is this file size what I am supposed to be worried about. Let me explain: Currently, we are using Access 2003 to link to a text file that is 8GB+ in size. We are able to do some crunching with this arrangement, it just takes forever (7 hours+).
I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.
Any thoughts on this???
My company has licenses for SQL 2005 Business Intelligence Development Studio. Does anyone know the size limit for this?
But is this file size what I am supposed to be worried about. Let me explain: Currently, we are using Access 2003 to link to a text file that is 8GB+ in size. We are able to do some crunching with this arrangement, it just takes forever (7 hours+).
I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.
Any thoughts on this???
<<But is this file size what I am supposed to be worried about. Let me explain: Currently, we are using Access 2003 to link to a text file that is 8GB+ in size. >>
Just because it's 8GB in the file doesn't mean it will be 8GB in the database. Some may be duplicated for example.
When brought into a DB and normalized, it's possible that size would go down.
<<We are able to do some crunching with this arrangement, it just takes forever (7 hours+). >>
Depends on what your doing there may be ways to speed that up as it stands.
<<I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.
Any thoughts on this??? >>
Too hard to say without a lot more detail (sample of the file, what your looking to get out of it, etc).
Jim.
Just because it's 8GB in the file doesn't mean it will be 8GB in the database. Some may be duplicated for example.
When brought into a DB and normalized, it's possible that size would go down.
<<We are able to do some crunching with this arrangement, it just takes forever (7 hours+). >>
Depends on what your doing there may be ways to speed that up as it stands.
<<I guess my main goal is to start using Sql Server to increase the speed at which this stuff is processed.
Any thoughts on this??? >>
Too hard to say without a lot more detail (sample of the file, what your looking to get out of it, etc).
Jim.
ASKER
Thank you for the information.
Yes, the Express edition is free and you'll have no problems doing that.
What you've outlined is doable.
Jim.