What is an appropriate way to make backups of a production server running CentOS?
The server is essentially a web host - it runs cPanel WHM, it has Apache, MySQL, Exim Mail and a handful of other services running on it. There is some special configuration done in cPanel for some of the user's web spaces (some of them have their permissions tweaked so they can talk to each other).
All we have in place right now is automysqlbackup for the databases... which isn't much.
Ideally we'd like to be able to restore on a file-by-file basis, but also we need a disaster recovery strategy to restore if the entire server goes down (e.g. the RAID array fails, datacenter goes down or something else disasterous happens)
It would be really nice to have the ability to "roll back" the production server a few hours in the event that we accidentally break it somehow (for example, upgrade cPanel and the upgrade fails, now we need to roll back somehow).
Since it is a production server... there obviously can be zero downtime but we can handle a bit of performance degradation in the middle of the night.
What's a good way to accomplish this?