?
Solved

windows server 2008 R2 slow, sluggish due to cbs.log

Posted on 2012-09-19
46
Medium Priority
?
4,395 Views
Last Modified: 2012-09-24
I am running windows server 2008 R2, and for the last month or so, it's been running very slow.  It's my DC, and DNS as well. After looking into the issue today, I believe the problem is due to the cbs.log issue. it sometimes goes up as high as 39MB/sec disk I/O.
I have attached a jpg of the issue.

How do I fix this?  How do I get the system to stop doing what is doing, as my other windows servers are fine, they don't have this issue.
cbslog.jpg
0
Comment
Question by:afacts
  • 24
  • 12
  • 10
46 Comments
 

Author Comment

by:afacts
ID: 38414779
what ever is causing the issue is causing the entire server to crawl.  For example, when opening server manager, under the roles summary and features summary, it just says "collecting data..."  It's been saying that for like 10 minutes now.   Not sure what can cause this?
collecting-data.jpg
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38414820
What you describe is a symptom of a larger problem, and is itself not the cause. the CBS.log file logs activity from the windows component-based-servicing (CBS) activities. Component based servicing is responsible for things like installing windows patches when you run windows update, repairing programs such as when you launch an office app that has been corrupted, or you use a feature that was not yet installed, you get that "installing" pop-up for a few seconds or minutes. During normal operations, the CBS is idle and doesn't do much, therefore there is very little to log.

The level of output you describe indicates to me that something very significant is happening on your server that is keeping CBS busy. Either a patch did not install properly and is continuously trying to re-install in the background, a background service is crashing and SFC is kicking in (SFC logs to CBS as well), or some similar not-normal/not-healthy activity.

I'd start by actually viewing a copy of the log and reviewing the latest entries. See what components are writing to the log file, and then backtrace them to find the source of the issue.

-Cliff
0
 

Author Comment

by:afacts
ID: 38414853
how do I view the file?  It's 2.5 GB and notepad or notepad ++ won't open it.
0
Configuration Guide and Best Practices

Read the guide to learn how to orchestrate Data ONTAP, create application-consistent backups and enable fast recovery from NetApp storage snapshots. Version 9.5 also contains performance and scalability enhancements to meet the needs of the largest enterprise environments.

 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38415010
Ouch. Yeah, it should not be nearly that big. It should be in the single or double-digit megabytes, not gigabytes.

Copy it to your workstation (I would never try to open a file that large on the server itself) and then open it in something like MS Word, which can open very large files better than notepad. It isn't the ideal tool for log files, but log files usually aren't that large....so we have to deal with that...
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38415100
try opening in edit plus freeware and post some output here
0
 

Author Comment

by:afacts
ID: 38415308
MS word 2012 won't open it because it's larger than 512 mb.  
This stinks, how do I open this 2.7 gb file?

Can I just delete the file on the server, will it create another file again by itself?
If it does that, then I can just look at the new file, as after a few minutes, it should write enough data to it?
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38415428
I have never deleted that file. I suspect you probably can't as it *should* be "in use" and hence have a file-lock. Even if you can, I suspect the service would expect the file to be there and balk. I'd consider that strategy very high risk.

Grab gvim and give it a go. It'll take awhile, but it should load.
0
 

Author Comment

by:afacts
ID: 38415487
How do I use the Windows Error Reproting tool?  Can't I use this tool to open the file?  Not sure where to find or how to use te tool?

I will try the vgim as soon as I return from lunch.
0
 

Author Comment

by:afacts
ID: 38415897
it's really slow, as it's been crashing almost every time I try to open the file.  Here's what I was able to get from it.  It looks to me that it's having problems with windows updates or something.   How would i fix this?  This is just from the beginning of the file.  cbs3 is more from the middle of the file.
cbs1.txt
cbs2.txt
cbs3.txt
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38415913
can you stop Windows Update service and see if the utilization goes down
0
 

Author Comment

by:afacts
ID: 38415923
well, but how do I fix the root cause of the issue?  I will still need the machine to receive windows updates.
0
 

Author Comment

by:afacts
ID: 38415932
i just turned off the service, so i guess it can take a few days to see if it still behaves like that.
0
 

Author Comment

by:afacts
ID: 38415934
but what would be the permanent fix?
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38415949
If its an only DC I would recommend to create a new DC-> Transfer the roles :-> demote this :-> format and repromote
0
 

Author Comment

by:afacts
ID: 38415991
you got to be kidding me, so that sounds like a Microsoft Answer.

So are you saying that the corruption is to great and it's not worth fixing?
0
 

Author Comment

by:afacts
ID: 38415996
So I disabled the windows update service, restarted the server and it looks like it's still doing it.
CBS-afterReboot.log
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38415997
Taking risk on single DC environment can never be recommended ...If one goes down you may end up creating entire domain again
0
 

Author Comment

by:afacts
ID: 38416003
actually, I have 3 DCs, this is just one of them, its' my ONLY one that is a physical, so it is very important as since I'm in a clustered environment, I have to have a physical DC running, if for some reason my other VM dc's fail.
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38416015
You have 3 DCs, 2 virtual, 1 physical, yes? And the physical DC is the one failing? Does it have any other roes besides ADDS and DNS?
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38416021
can you read below it seems closure to you
http://www.shdon.com/blog/2010/01/22/windows-update-woes
0
 

Author Comment

by:afacts
ID: 38416042
It only has AD and DNS only, yes, it's a physical DC.
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38416092
Are you facing this issue with this DC or all the 3
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38416095
If only this one is facing the issue then reinstallation is a clean way to workaround and have a healthy one
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38416109
Honestly? I would demote the server, remove it from the domain, wipe it, and re-install.

I'm not saying you *can't* fix the problem. You probably can. However, since the entire design of AD is to be redundant and replicate, this would be faster than digging into the logs and finding the corrupt data/update. Realistically in a 3 DC/1 site domain, you are probably looking at an hour of effort, max, to do what I suggested. You've probably already spent more time trying to open that huge log file.

It is a matter of measuring risk/reward. Is the time it'll take to fix the problem worth it vs the time it'll take to simply rebuild? I'd argue in most cases it isn't.
0
 

Author Comment

by:afacts
ID: 38418523
yeap, i went to demote the server and I get the following.
"Failed to install Active Directory Domain services binaries". The error was : "The referenced assembly is not installed on your system".

In server manager, when I go to roles, it says error.

How in the world do I fix this?
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38418714
You will have to manually clean up active directory. That is covered here:

http://technet.microsoft.com/en-us/library/cc781245(v=WS.10).aspx
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38418725
Follow below three activities in order

1) Seize FSMO role:
http://www.petri.co.il/seizing_fsmo_roles.htm

2)Forcefull removal of DC:
http://support.microsoft.com/kb/332199

3)Metadata cleanup:
http://www.petri.co.il/delete_failed_dcs_from_ad.htm
0
 

Author Comment

by:afacts
ID: 38418883
How can I tell if this DC has the FSMO role or not?  Maybe it doesnt and I can skip this step?
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38418914
run
netdom query fsmo

Open in new window

on any of the working DC
0
 

Author Comment

by:afacts
ID: 38418946
this is terrible, all the roles are on my dc that is failed.  So I guess I need to sieze them, right?
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38418968
Yes
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38418997
If you follow the documentation I provided, the force removal will handle discovering the FSMO roles for you. There is a reason why Microsoft provided official documentation for such events.

-Cliff
0
 

Author Comment

by:afacts
ID: 38419000
Will do it on MOnday, as I have two leave in 2 hours and I don't want to leave it hanging without a physical DC.  I guess I can do the siezing part today, but demoting the server, I will do on MOnday and reformat and reinstall will be on Monday.
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38419016
Realistically you should be able to demote, clean up, reinstall, and promote in 2 hours. I've gotten so used to the process I can do it in about a half-hour (if I install from a fast media source.)

I suspect the way things sound, since you couldn't even demote the server, that you are without a physical DC anyways. It isn't reliable. So if you don't want to leave the system without a physical DC over the weekend, it doesn't matter if you wait to demote or not...that DC is already failing in significant ways. To fulfill your desire to have a physical DC, you really should push forward.
0
 

Author Comment

by:afacts
ID: 38419019
so the forceremoval option actually moves the fsmo roles over to another DC?
how does it know to what other server to move it to?
0
 

Author Comment

by:afacts
ID: 38419039
when I typed dcpromo /forceremoval
I get the same error message:
"failed to install active directory domain services binaries.  the error was : the referenced assembly is not installed on your system"

Great, so what do I do know?
0
 
LVL 60

Accepted Solution

by:
Cliff Galiher earned 750 total points
ID: 38419068
Unplug the domain controller from the network (basically simulating a complete failure) and proceed to the metadata cleanup step. As indicated in the document I sent you:
 
  "If you do not transfer operations master roles before you forcibly remove AD DS, the roles are transferred during the metadata cleanup process automatically."

This is the type of scenario that would most often occur after a hardware failure. It is not entirely uncommon and the cleaup process will handle it.
0
 

Author Comment

by:afacts
ID: 38419072
but the document you sent is for 2003 only, it doesn't say anything to 2008, is there one for 2008 R2?
0
 
LVL 60

Expert Comment

by:Cliff Galiher
ID: 38419087
Not a lot has changed. More options for metadata cleanup. Same basic format, same document just revised:

http://technet.microsoft.com/en-us/library/cc816907(v=WS.10).aspx
0
 

Author Comment

by:afacts
ID: 38419154
since my server is still up, I was able to move the domain naming master to my dc2.
How do I move all the other roles to my other server?
fsmo.jpg
0
 
LVL 18

Expert Comment

by:Sarang Tinguria
ID: 38419174
Hi

Use below link to transfer all other roles in gui

http://kpytko.wordpress.com/2011/08/26/transferring-fsmo-roles-from-gui/
0
 

Author Comment

by:afacts
ID: 38419378
how do I remove the old server from the sites and services section, under servers, as it deleted one of them, but under the otehr DC, it's still listed there.  I tried to right click and delete, but it wont let me.
ADerror.jpg
0
 

Author Comment

by:afacts
ID: 38419383
never mind, it looks like it just updated.
0
 

Author Comment

by:afacts
ID: 38419819
i wiped the box, reinstalled the OS and am trying to dcpromo it, and now I get this error, it's attached.

Could this be because I am using the same name as I did before?  Do I have to change the name to something not used before?
dcpromoerror.jpg
0
 
LVL 18

Assisted Solution

by:Sarang Tinguria
Sarang Tinguria earned 750 total points
ID: 38419838
Make sure your DNS is pointing to correct internal DNS server in NIC card and user ID you are using have sufficient rights i.e Administrator
0
 

Author Closing Comment

by:afacts
ID: 38429324
Thanks for the help!
0

Featured Post

Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

New Windows 7 Installations take days for Windows-Updates to show up and install. This can easily be fixed. I have finally decided to write an article because this seems to get asked several times a day lately. This Article and the Links apply to…
After seeing many questions for JRNL_WRAP_ERROR for replication failure, I thought it would be useful to write this article.
This tutorial will walk an individual through locating and launching the BEUtility application and how to execute it on the appropriate database. Log onto the server running the Backup Exec database. In a larger environment, this would generally be …
There are cases when e.g. an IT administrator wants to have full access and view into selected mailboxes on Exchange server, directly from his own email account in Outlook or Outlook Web Access. This proves useful when for example administrator want…
Suggested Courses

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question