Question

How to keep MSSQL Log File OS space usage in check?

Asked by: ankitmathur

Hi All,

A few questions about Log File usage in MSSQL Server 2000.

Q1. Does providing a table with Clustered Index causes Log File Space to grow ?
Q2. How can I reclaim OS Space from log file ?

Now a lil' insight on the system am working upon.

I have a database which has records in millions in some of its tables. So much so, the DB size has grown to above 80GB and Log file grown to 40GB+. This is after I recreated a new log file only recently. But the speed at which log file is growing is amusing me and leaving me with a lot of issues like unable to restore the database on any other machine as combined the DB is using such a big space that I get into a space crunch issue on my test servers.

My transaction log backups job is being run every 15 mins successfully. Transaction log size is also at a manageable 15MB avg size. But still log file is growing in bounds. Earlier it used to remain aound 20GB. But after I executed a clustered index on one mof the biggets tables of all I'm facing this issue.

Can someone help me decipher my two questions above and steps to manage this DB keeping the size in check ?

Thanks
Ankit Mathur

This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.

Subscribe now for full access to Experts Exchange and get

Instant Access to this Solution

  • Plus...
  • 30 Day FREE access, no risk, no obligation
  • Collaborate with the world's top tech experts
  • Unlimited access to our exclusive solution database
  • Never be left without tech help again

Subscribe Now

Asked On
2008-08-07 at 23:57:26ID23631863
Tags

Microsoft

,

SQL Server

,

2000

,

Log File

Topic

MS SQL Server

Participating Experts
2
Points
125
Comments
14

Trusted by hundreds of thousands everyday for fast, accurate and reliable tech support.

  • "The time we save is the biggest benefit of Experts Exchange to Warner Bros. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange." Mike Kapnisakis, Warner Bros.
  • "Our team likes having a resource that is more secure than just using Google and most experts using this service really know their stuff. It's nice to look here first versus using Google." Dayna Sellner, Lockheed Martin
  • "Anytime that I've been stumped with a problem, 9 out of 10 times Experts Exchange has either the accepted solution or an open discussion of the potential solution to the problem." Kenny Red, eBay Inc.

See what Experts Exchange can do for you.

Got a question?

We've got the answer.

Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.

Screenshot of Experts Exchange Knowledgebase

Need individual assistance?

Our experts are ready to help.

If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.

Screenshot of Experts Exchange Knowledgebase

Want to learn from the best?

Read articles from industry experts.

Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.

Screenshot of an Article

Working on a long term project?

Store your work and research.

Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.

Screenshot of Experts Exchange Knowledgebase

Access the answers to your technology questions today.

Subscribe Now

30-day free trial. Register in 60 seconds.

What Makes Experts Exchange Unique?

Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.

Trusted by the world's most respected brands.

image of each brand's logo

Faithfully serving IT professionals since 1996.

Experts Exchange Logo

Try it out and discover for yourself.

Subscribe Now

30-day free trial. Register in 60 seconds.

Related Solutions

  1. Cluster Gobbling Space. Help!
    Dear Oracle Users, I am having a serious space problem with a table cluster on a 8.0.4.3 database on Solaris. I have two tables A and B in a CLUSTER. table A has 3 million rows table B has 6 million rows I have obtained the average rowsize for each table using select avg(n...
  2. 1.2 million inserts trough odbc
    What is the fastest way to insert 1.2 million records using odbc. dbf -> Change values -> mssql
  3. memory grow rapidly
    I use Visual Studio 2003 (VB.NET) to write a simple program. In this program, the timer will call a funciton to check records of MSSQL via ADODB and activate Crystal Report 11 to print some document. However, I found the memory usage for this program is growing rapidly. What...

Free Tech Articles

  1. WARNING: 5 Reasons why you should NEVER fix a computer for free.
    It is in our nature to love the puzzle. We are obsessed. The lot of us. We love puzzles. We love the challenge. We thrive on finding the answer. We hate disarray. It bothers us deep in our soul. W...
  2. SCCM OSD Basic troubleshooting
    SCCM 2007 OSD is a fantastic way to deploy operating systems, however, like most things SCCM issues can sometimes be difficult to resolve due to the sheer volume of logs to sift through and the dispe...
  3. Migrate Small Business Server 2003 to Exchange 2010 and Windows 2008 R2
    This guide is intended to provide step by step instructions on how to migrate from Small Business Server 2003 to Windows 2008 R2 with Exchange 2010. For this migration to work you will need the fo...
  4. Create a Win7 Gadget
    This article shows you how to create a simple "Gadget" -- a sort of mini-application supported by Windows 7 and Vista. Gadgets can be dropped anywhere on the desktop to provide instant information, ...
  5. Outlook continually prompting for username and password
    There have been a lot of questions recently regarding Outlook prompting for a username and password whilst using Exchange 2007. There are a few reasons why this would happen and I will try to cover t...
  6. Backup Exchange 2010 Information Store using Windows Backup
    There seems to be quite a lot of confusion around the ability to backup Exchange 2010 using the built in Windows Backup feature. This stems from the omission of this feature prior to Exchange 2007 s...

Cloud Class Webinars

  1. Avoiding Bugs in Microsoft Access
    Alison Balter takes and in-depth look at avoiding bugs in Access. In this webinar you will learn about using the immediate window to debug your applications, invoking the debugger, using breakpoints to troubleshoot, stepping through code, setting the next statement to execute, ...
  2. Top 10 Best New Features in Visio 2010
    Scott Helmers gives live demonstrations of the top 10 new features in Visio 2010. This webinar will teach you how to create compelling diagrams by adding shapes to the page with a single click, linking the shapes in a diagram to data in Excel (or SQL Server, or SharePoint), ...
  3. IT Consultant Business Secrets Revealed
    Michael Munger, Experts Exchange tech pro and IT consultant, pulls back the curtain on his very successful businesses and answers question on every IT consultant and business owner should know about. He shares secrets on what he did to solve the 5 most common problems in IT, ...
  4. Disaster Recovery and Business Continuity
    Quest CTO, Mike Billon, gives an overview of the steps involved in building a dunamic disaster recovery plan. Through case studies and an examination of software/hardware tooles for monitoring and testing, you'll gain a better understandin of where you are, where you want ...
  5. Organize Your Visio Diagrams with Containers and Lists
    Scott Helmers uses cross functional flowcharts, wireframe diagrams, data graphic legends and seating charts to teach you: how to ustilize all three new structured diagram components in Visio 2010, the best practices for organizeing shapes in previous version of Visio, how to organize ...
  6. How to Us Objects, Properties, Events and Methods in Microsoft Access
    Alison Dalter gives an in-depbth look at objects, properties, events and methods in Microsoft Access. In this webinar you will learn about using the object browser, referring to objects, working with properties and methods, working with object variables, understanding the ...

Join the Community

Give a Little. Get a Lot.

Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.

Join the Community

Answers

 

by: angelIIIPosted on 2008-08-08 at 00:07:42ID: 22187955

> Transaction log size is also at a manageable 15MB avg size
sorry, but 15MB for the log file is FAR too small. especially with a DB of 80GB, the number of transactions will eventually be much more than just 15MB (or did you mean 15GB? -> that would be too large)

CREATING a clustered index will use some transaction space indeed.
having one should not increase/decrease log space as such...

>My transaction log backups job is being run every 15 mins successfully.
that's fine.

>Q2. How can I reclaim OS Space from log file ?
you can try the DBCC SHRINKFILE command. however, as log files are a circular buffer, in case the "internal write pointer" is at the end of the file, it won't do anything. however, after some more transactions (with the internal write pointer having moved forward, ie to the beginning of the file), + a log backup, then the command will succeed.
note however that you should not try to reduce the log file tooo small, as otherwise the performance of the database drops because it will have to grow the file again.

conclusion:
with regular t-log backups, the log file size should come to a stable size quickly.
if it does not, check out the output of DBCC OPENTRAN to see if you have some open transactions which keep the internal log buffers from being reclaimable.



 

by: brejkPosted on 2008-08-08 at 00:08:43ID: 22187957

Q1. Yes.
Q2. Perform BACKUP LOG statement (this should be scheduled quite often to keep your log in a reasonable size) and then DBCC SHRINKFILE - if not helps at once, perform both (BACKUP LOG + DBCC) many times. If not helps after several executions - check if there is no open transaction blocking log reuse (use DBCC OPENTRAN to detect SPID running the transaction and then DBCC INPUTBUFFER to see the last statement of the SPID). You can go with your database into SINGLE_USER (use ALTER DATABASE and WITH ROLLBACK IMMEDIATE option) to rollback all running transactions (including those long running) and then perform log truncation.

Use DBCC SQLPERF(LOGSPACE) to check the size and usage space of the log.

 

by: ankitmathurPosted on 2008-08-08 at 00:25:16ID: 22188033

Hi angelIII,

>sorry, but 15MB for the log file is FAR too small. especially with a DB of 80GB
No, my TrnLog size remains at a manageable 15MB avg. Approx. number of records entered into my database daily is around 0.5 million to 0.75 million and every minute avg of 300-400 records.
No record is being deleted and 90% of the time its only insertions that's going on. Updations are also not that regular.

>CREATING a clustered index will use some transaction space indeed.
>having one should not increase/decrease log space as such...
I was also expecting an increase in log file size. But the quantum jump its taking is somewhat bothering me. It's been a week since I recreated a new log file. I hope it should have reached the state of stabilization.

I have tried DBCC OPENTRAN several times earlier too and just now too. It came out with "No active open transactions." result.

Is there any way to monitor my DB size growth? I think I'll have to do a major archiving exercise.

Ankit

 

by: angelIIIPosted on 2008-08-08 at 00:26:54ID: 22188041

do you have some database maintenance plans created recently, which do for example a rebuild index?

 

by: ankitmathurPosted on 2008-08-08 at 00:35:33ID: 22188077

Hi brejk,
>Use DBCC SQLPERF(LOGSPACE) to check the size and usage space of the log.
DB_Name      LogSize(MB)       Log Space Used(%)      Status
DB_Name      41785.055       0.31160215                            0

This is my latest result of SQLPERF command. Earlier also when I ran this command I saw an almost similar picture in Log Space used and that's why I went for recreating a fresh log file. But as we all know. Didn't helped much.

You recommended BACKUP LOG command. As of now I'm using Database Maintenance Plan Wizard to take my Transaction Log backups. Are they using different tech. ?? Would using a BACKUP LOG be more benefical in any manner?

Also you suggested DBCC SHRINKFILE. I'm actually trying to avoid it doing by myself and leave it to be done using Maintenance Plans again which is done every weekend while performing OPTIMIZATIONS JOB. I read somewhere that DBCC SHRINKFILE should not be used too frequently as it hampers with DB working so I don't quite prefer it to be done on my LIVE server. Any other suggestions?

Thanks for your inputs.
Ankit

 

by: angelIIIPosted on 2008-08-08 at 00:46:22ID: 22188123

You recommended BACKUP LOG command. As of now I'm using Database Maintenance Plan Wizard to take my Transaction Log backups. Are they using different tech. ?? Would using a BACKUP LOG be more benefical in any manner?

it's the same

also you suggested DBCC SHRINKFILE. I'm actually trying to avoid it doing by myself and leave it to be done using Maintenance Plans again which is done every weekend while performing OPTIMIZATIONS JOB.

also here, the maintenance plan uses DBCC SHRINKFILE, so there is no difference.

note that you should NOT auto-shrink or regularly shrink your database, it will just result in more I/O to first shrink, and later grow the database.
use DBCC SHRINKFILE only for manually shrinking a data/log file when it had got out of hand.

 

by: brejkPosted on 2008-08-08 at 00:54:23ID: 22188144

@ankitmathur:
angelIII is right. Also, do not shrink log to the minimal size possible. It will result in almost automatic autogrow and - further- more virtual log files in your log. You should consider shrinking to the size that fits your database needs (this is something to capture by monitoring your log and its growth).

However, having backup strategy that incudes log backups is the most important.

 

by: ankitmathurPosted on 2008-08-08 at 01:26:01ID: 22188231

My optimizations job is scheduled to run this Sunday. Let's hope that it does help me recover some OS space too alongwith an improved DB performance.

Till then this post remains open.

Also, If anybody can suggest me some good points to optimize DB performance for a DB of such a size I'll be more than grateful.

I'd like to make sure I'm doing all the right things possible to manage this DB and its oversized tables.

Ankit

 

by: ankitmathurPosted on 2008-08-10 at 23:03:04ID: 22202060

Hi Friends,

Yesterday was a very eventful Sunday. The DB I was talking about made all of us sweat.

A lil' insight about what happened yesterday.

Our server stopped responding to any command. We tried restarting the SQL Agent first so that scheduled jobs atleast can run somehow esp. backups but always got Error: 14258. Tried restarting MSSQL Service. Same error again. Even rebooted the whole machine but only to see the same error again. After googling the error found the solutions provided were already applicable so not of much use.

So had to restore to another server and made that LIVE.

Now on further investigation we found that our data files have grown to such an extent that our MSSQL could not actually load it properly. Only after leaving the server with no intervention for a few HOURS it started responding. My MDF file is now a monstrous 125GB and Log file 48GB. Certainly not to amany DBAs delight when we are trying to restrict our size to some where manageable 40-50GB.

Now I need some more help from you all in identifying what caused this to grow to such an extent. Till saturday my MDF was arnd 80GB and on Sunday it went to 125GB. I can't blame OPTIMIZATION JOB much as the backup that happened just before we were reported this issue is still at Saturday levels. So whatever happened, has to be after backups were done.

I hope some of you can give some insights on finding the cause of this problem. We have to understand why it rose to such high size and how to get it back to manageable levels.

Waiting for some inputs.
Ankit

 

by: angelIIIPosted on 2008-08-10 at 23:25:17ID: 22202122

If I had to bet, I would say you got victim of a SQL injection, adding some data to all the varchar column large enough to hold the addition of the data, resulting in the data growth you saw ...
as you cannot open the db, you might want to open if with some "hex editors" (like ultraedit), and look if you "see" somethink like <script .... > ... </script>  in the data.
might not be obvious if the columns are nvarchar ...

 

by: ankitmathurPosted on 2008-08-10 at 23:47:29ID: 22202207

Hi angelIII,

We're not negating any of our options and are in the process of finding out what caused the issue. Since we have more 150 tables it should take some time before I can respond back with something I could find out. Scanning this much tables manually will take some time.

Any suggestions as to what else we should look for ?

Ankit

 

by: angelIIIPosted on 2008-08-10 at 23:55:13ID: 22202242

a growing data file means that tables/indexes got added or data added.
there is nothing else I am aware of ..

 

by: ankitmathurPosted on 2008-08-14 at 07:41:14ID: 22230724

Hi All,

Checkedmy DB thouroughly. Am not a victim of SQL Injection. But why my DB size rose to such levels is still not clear. Now we're trying to somehow reduce space from our MDF and LDF files. While we've been able to get our LDF file to shrink, our real concern right now is MDF file. Which is simply not ready to give up any space. Whatever ways we've tried so far and that includes SHRINKDATABASE and SHRINKFILE too.

I know I may be opening up another question here in this post which doesn't seem right here. So I'll open this question as a  new post.

Ankit

 

by: ankitmathurPosted on 2008-08-14 at 07:45:41ID: 31484668

Answers were partially relevant but not comprehensive as I was expecting perhaps. Since my issue got to some to other stage than my question so am deciding to close the post here as asking new questions in an old post will somehow mislead future information seekers who come to this post.

20120131-EE-VQP-002

3 Ways to Join

30-Day Free Trial

The Experts

98% positive feedback on 31,087 answers since March 2000. angeliii is a Microsoft Most Valuable Professional for his work with MS SQL Server & Develoment.

He has also proven his knowledge of Visual Basic Programming, PHP Scripting and Oracle Databases.

The Experts

97% positive feedback on 10,752 answers since July 2000. lrmoore has more than 18 years experience in the networking industry.

The six-time Mircosoft MVPs specialties include firewalls, virtual private networking, and network management.

Testimonials

"...and excellent source for support... Kind of like having your very own IT dept." Electriciansnet

Testimonials

"I was apprehensive at signing up at first. However... it has already made my life as an IT administrator much easier." JaCrews

Testimonials

"WOW! You guys have great, active, and knowledgeable people on here." moore50

Business Clients

Business Clients

In the Press

"If you’ve got a question... Experts Exchange can supply an answer.”

In the Press

"...an invaluable aid for both IT professionals and those who require tech support."

In the Press

"where IT professionals provide quick answers on just about any topic"

Business Account Plans

Loading Advertisement...