What is the best practice to partition data from a very large table?

Posted on 2005-04-25
Medium Priority
Last Modified: 2010-03-19

We have a very large table with millions of rows. We would like to partition the data based on the canlendar year. What is the best practice and the most efficient way to partition (to archive) those old data into the new partition? Is creating a new database the way to go? If not, could someone shed some lights on this for us?

Question by:mei_liu
  • 2
  • 2

Expert Comment

ID: 13864304

We have a similar situation where we log all the hits to a database and it grows very big and hence on a monthly basis we export the data to a
delimited file and then zip it and store to conserve space. As we export the traffic data to Webtrends, the only purpose for store the data is for rebuilding the reports. Similarly if you are storing the old data for archiving purposes and do not require the data for operations then I would recommed it to store in a zipped text file.

If you need old data for operations, I would say that creating a new databse or creating separate table in the same database would both be fine. My preference will be a new database for the following reasons:
1. New database will allow you to backup and archive separately.
2. The current databse will be small and easier to manage.
3. In case you are using a full recovery model then, this will also keep a check on the transaction log.

Hope this helps...


Author Comment

ID: 13868405
Hi Kailash,

Thanks for your help. Have you used other methods other than exporting the data as text file, and store it elsewhere?

1. Have you yourself exported the old data to a new database?
2. How was your experience like if any?
3. What is the cause for the newly created database with imported old data to be in such a large size?
4. How could we control the size of the database with imported old data?

As we need the old data in production for qurey purpose, we can't take them offline and store them elsewhere. This is the main reason why we have big concern over its size.


Accepted Solution

ckailash earned 2000 total points
ID: 13868708
Hi Mei Liu,

Yes we do have some old data which we need for operations in in that case we export the old to another database using a dts and it works fine. The only problem you can have is the database you are exporting the data from will be locked if you are doung some sorting while export or doing a delete after export. The time of lock will depend on size of the database.

I prefer a new database for the old data because I can then set the recovery model of the old database as Simple and thus the size of the database will be small. A Simple recovery model for the old data is fine as the data does not change. This way I can keep the Current databse with a Full recovery model.


Author Comment

ID: 13869353
Thanks Kailash.

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This month, Experts Exchange sat down with resident SQL expert, Jim Horn, for an in-depth look into the makings of a successful career in SQL.
MSSQL DB-maintenance also needs implementation of multiple activities. However, unprecedented errors can hamper the database management. In that case, deploying Stellar SQL Database Toolkit ensures fast and accurate database and backup repair as wel…
Using examples as well as descriptions, and references to Books Online, show the documentation available for date manipulation functions and by using a select few of these functions, show how date based data can be manipulated with these functions.
Via a live example, show how to extract insert data into a SQL Server database table using the Import/Export option and Bulk Insert.
Suggested Courses

621 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question