Best Way To Reboot 1000 Servers
I have the need to reboot about 1,000 servers on a staggered schedule. I was wondering about the best way to approach this. Most of the servers are VM s. Most are in active directory but not all. Was wondering if a script via GPO or a scheduled task is the best method?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Powershell would be my choice if I had to choose one, however it seems like the scheduled task to execute a batch script would be your easiest. Are you needing to do this more than 1x, if so then you could always do as Darrell suggested and group the servers in a batch script and schedule the task to run it.
Plenty of options, but really depends on how often it needs run and for what reason probably.
Plenty of options, but really depends on how often it needs run and for what reason probably.
I may have failed to be clear. I would create a scheduled task and 4 different schedules - one for each of the first through fourth Tuesdays of the month. These would be deployed to the servers directly so that, in the event connectivity to the server/site was interrupted, the task would still be capable of rebooting the server in question. I would have the task set a flag file to allow the process to be easily monitored or I would periodically query the Windows even logs of the servers in question to test the EventLog events to determine the duration a server had been operating to ensure the servers were rebooting as expected and receive an exception report indicating the servers which had not properly rebooted and the error code generated by the scheduled tasks.
We currently do this for servers in our environment and, at times, something goes awry with a server or nine and we receive alerts these servers have an uptime exceeding a set threshold.
Again, it would be helpful to know the reasons these servers need a reboot, the services the servers provide, and how often they need to be rebooted.
For the command-and-control systems, we use the Poiwershell job engine to run the monitoring process in parallel with a process queue length of 100-250 servers at once depending upon the resources available and the data we want to gather from each device.
We currently do this for servers in our environment and, at times, something goes awry with a server or nine and we receive alerts these servers have an uptime exceeding a set threshold.
Again, it would be helpful to know the reasons these servers need a reboot, the services the servers provide, and how often they need to be rebooted.
For the command-and-control systems, we use the Poiwershell job engine to run the monitoring process in parallel with a process queue length of 100-250 servers at once depending upon the resources available and the data we want to gather from each device.
Depends on your intentions really. Do you want them to restart on their own, regardless of your interactions or are you looking for a way to reboot them more efficiently?
I generally use shutdown /r /t 10 for my command and command-line options but you may need the server to do something before it reboots and that may take longer than the inferred 10 seconds.
Additionally, I recommend a quarter of the fleet get rebooted at a time and, if there is more than 1 server per location, not rebooting all of the servers at one location at the same time.
I usually pick early Tuesday morning so that, in the event of 3-day weekends, I have a technical resource guaranteed available to support any misfires or servers that fail to come back up because someone left a bootable flash drive in the USB port, etc. (our servers actually will not mount USB drives at all and are disallowed booting from such devices without modification of the BIOS using a password).
What is the purpose of rebooting the servers? Perceived stability gains? Need to reboot after some other event occurs? Someone told you to do so? Major application (POS software, for example) has a component with a known memory leak?
Just curious as you may want to gather additional information during the same window of opportunity.
Do you want to run any kind of logging of the script? If so, have the script write a file that a central process can search for to ensure the reboot happened as expected.