[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 777
  • Last Modified:

Disaster Recover Plan

I need to develop a disaster recover plan.

I plan to have servers off site. I am considering backuping on tape and physically brings the tapes to the other site or having them replicate over the WAN.
What do you think is better?

I am running one DC and an exchange server.

What should I backup on the production servers (system state or just AD files)?
What would the system state do for me?
Do I need the same model servers?

I am very open to ALL IDEAS. Any other approach to our disaster recovery plan will be greatly appreciated.

Thanks...Sorry for all the questions.
Mancoi
0
mancoi
Asked:
mancoi
  • 5
  • 5
  • 4
  • +7
5 Solutions
 
mancoiAuthor Commented:
How should I point the users to that server?
0
 
scrathcyboyCommented:
Remember, mancoi, tapes are yesteryears technology.  The new trend in backup is direct to disk, like a remote hard drive on another server, and yes, it can be at another location.  Tapes are fraught with problems.

Also remember, any time you want to recover the computer setup, you must do a live-state backup, meaning backup all the open system files with snapshot technology.  The latest versions of Ghost Enterprise and Acronis true image for servers both claim to do snapshots of the system setting while the OS is running.  If this is true, then whatever program can save the full system state, that is the one you want to use.

Backup plan -- backup the system state ONCE A WEEK.  This is all open files on the server or WS, so you can recover its setup.  Backup the crucial data from the server, and any WS that is saving crucial data, ONCE A DAY.  This is your most recent data files, and you can do it as a DIFF, or an INCREMENTAL, whichever you like.

Finally as to how to backup, consider a VPN from one site to the next.  You can buy two Linksys BEFVP41 VPN endpoint routers for total $120, and you configure them identical protocols to talk to each other, one at one site, and the other at the other site.  The VPN acts just like part of the network.  You share a drive on that remote computer at other end of VPN tunnel, and after hours, you backup everything you want to the remote drive.

This system has got tapes beat cold.  Tapes always foul up, and they dont restore well or are not reliable.  With the hard disk option, if the server or WS dies, you just bring the backup drive from other location and restore, or better still you can restore to the original system right over the VPN -- this is the best of all worlds.
0
 
Duncan MeyersCommented:
Replication and backup aren't *really* the same thing - for example, if you were replicating your file server in real time to a remote site and you got walloped by a nasty worm, then the damage that the worm did will get replicated to your DR site. Result: corrupt data everywhere!

Using tape to back up your systems is the preferred option - tape has the considerable advantage that it is cheap, it is physically robust, easy to transport and store (with better longevity/magnetic chracteristics fro long-term storage than disc), and performance is far better than that of disc for the characteristic I/O profile of backups.

My suggestion would be to back up to DLT-S (the new name for SDLT) or LTO-2. Have a tape each for Monday - Thursday. It's probably worthwhile doing full backups every night if the volume is not too big. Take a weekly backup every Friday, whcih you retain for 4 weeks. You then take a monthly backup which you retain for as long as your business and local legislation requires. Store the weekly and monthly tapes off-site. You could also consider taking a backup to disc so that yopu can have speedy restores - but that is less necessary if you implement Microsoft's Volume Shadow copy Services. Users can then restore their own deleted files from snapshots.

Finally, you can also consider a product like Symantec's Replication Exec (or rsync if you're prepared to do the work to make it go) for replication between your production and DR site. In deciding whether or not you need a replication technology, take a look at how long you can be without your critical servers, and how much it would cost the business if you lost them. I'd suggest that in most instances, Exchange is *not* a critical service (cue flames!) - after all, it's simple to pick up a phone and, you know, actually talk to someone... Your critical servers are most likely to be financial applications, payroll apps, and your file server itself - but that is a call only your business can make.

So: if you've got big bags of cash, I'd reccommend replication to a remote DR site using a replication package to minimize downtime in the event of a physical disaster, and backups to tape, kept off site, to protect against data loss.


0
Transaction-level recovery for Oracle database

Veeam Explore for Oracle delivers low RTOs and RPOs with agentless transaction log backup and transaction-level recovery of Oracle databases. You can restore the database to a precise point in time, even to a specific transaction.

 
scrathcyboyCommented:
" Replication and backup aren't *really* the same thing"
Nonsense, there are by definition the same thing,
"Using tape to back up your systems is the preferred option"
Archaic, expensive, failure prone, outdated, always cause problem.

But then there are always people who cling to the outdated past and cant see the future.

You decide mancoi, it is your question, the old stolid expensive way of the past that give failed results
Or the new fast efficient backup to instantly and always-online media.  Of course, not everyone can see the future.  Tapes are dead, the trouble is there is so many installations that people cannot see how inefficent they are.  Good luck, and bye./
0
 
BouL82Commented:
The requirements of the DR plan are not really clear. What is the time frame you need to be up and running again after a failure?
How big are your exchange databases. Only a DR plan for exchange and your domain controllers?

Some ideas:

But what about an iSCSI storage controller on a different location and run your backups to that storage controller over iSCSI so you have your data secure on another location without the hassle of tape backups.
Or cluster your exchange server over two datacentres using iSCSI technology.
And make sure you get 2 Domain Controller since this is the cornerstone of your infrastructure especially when the environment gets bigger.
The most important things to backup are your Active Directory and Exchange databases from here you can recover if you have got propper backups of these.

Some words about exchange not being criticial... Nowadays companies rely on email heavily, it doens't look professional when emails got bounced an i don't wanna be in the shoes of the IT manager or be the engineer responsible for a malfunction mail server and having the whole company on my neck complaining and asking when it's working again...

Good luck with your DR plan.
0
 
Froggy_chrisCommented:
Hi,

Working on that as well, I'm orienting my DRP to:
  - Use external service to get External DRP facility + machine (then, I don't immobilize my hardware). Fee are actually reasonnable,
  - Using Virtual Machine (based on Virtual server 2005 R2) to create the restore. Image are created and stored on DVD sent to the facility provider
  - We use tape backup for the daily backup that are sent to outisde storage.


I actually more in fever of tape backup than disk backup for argument developed by meyersd. Disk are still fragile when moved accross building (I'm probably achaic, but well, backup media are tested regularly and everything is well. Simply don;t forgert to replace according to supplier indications (50 passes usually)).

I aslo follow boul82 for other point, the first thing you need is to define your requirements. Loads of things changes depending of what your requirements are.

Chris

0
 
AlexGGGGCommented:
Replication implies that two machines hold two  (to a certain extent) identical copies of data. So once something is deleted from the "master" machine, it is soon deleted from the "shadow" machine as well, and previous versions are not kept.

The backup implies previous versions are kept in case needed later.

The media types and transfer methods does not affect the difference between backup and relplication. If you copy data via VPN/WAN to the remote location and save all the versions of the files from, say, five years, this is a backup (rather than a replica) of data, despite the fact no tape is involved.

Replication is in the essence having a single backup copy of the current state, and, as such, prone to operator-, virus-, and software-malfunction-induced failures.
0
 
BouL82Commented:
If you backup things to disk, depending on what kind of storage but for example a simple SAN/NAS can be FC,iSCSI,CIFS or NFS. You can make use of Snapshot technoloy which keeps previous versions of data as well. The same for Windows Shadow Copy. Then it's possible to replicate backed up data as well.

Cheers,
Roeland
0
 
David_FongCommented:
Got to agree with meyersd, tape shipping is definitely the cheapest and most reliable way to go, disks are far too expensive if there is a lot of data and you want to keep it for more than a couple of days. Data retention life measured in 10's of years rather than years as long as you retension every 6 months, far faster than disk unless you've got a big array, in fact too fast for most servers which is why disk-staging is used in enterprise backups so there's dedicated disks to supply the tape drive fast enough to keep it streaming.

People have been predicting the demise of tape since I joined the industry 20 years ago and will no doubt be predicting its demise for the next 20 years as well.
0
 
ECNSSMTCommented:
In the financial sector, and personally for the banks and financial institutions that I've WORKED for, tape continues to be a strong resource used in archiving and disaster recovery (or more aptly termed Business Continuation).  The points that David Fong, Meyersd and others attest to are points worth considering when you develop your plan to ensure that your organization has valid data and access to that valid data going thru any crisis.  

Looking backwards in time, if your company needs to keep historical data; a good tape backup methodology that preserves the data state of your company per a daily, weekly or yearly snapshot is the common call.

Looking forward; many companies hedge the tape technology investiment into the Business Continuation arena.  There is a savings in cost, time, and competing technologies here.

Having said that, the focus of Disaster Recovery or really Business Continuation should not center on tapes, but rather the scope of the disaster.  

In Business Continuation planning you are looking at the question; what do I do to ensure that my business continues to work if a problem occurs that could potentially shut me down for a day, up to 2 weeks or for more than 2 weeks.  Your concentration will be on the computer aspects, but you will find it beneficial to include other business groups in on the outline of your planning so that they may plan for contingencies themselves.  IT IS VERY INVOLVED.

Your concentration: quick blurbs
One day - (maybe up to 72 hours) this is usually associated with equipment failure.  If you scheme your equipment out just right i.e. ensure that you have UPS' in place for brown outs, properly redundant equipement like RAIDed servers, standby WAN connections, and contracts to cover the quick replacement of defective equipement, you've got most of this genre covered.
Up to 2 weeks - (temporary displacement) when some or all of the onsite facility is deemed unusable.  When you declare this type of disaster, your thought is what is the minimum I need on a offsite facility to continue my business.  Tapes are a great and cheap asset here; especially if you STORE YOUR daily TAPE ARCHIVES at an OFFSITE facility.  Don't want a fire (disaster) to take out the business and the tape archives all at the same time. Overall cost is a factor here.  Tape with a minimally replicated off-site server equipment is great, and the more you can add to this means the less time it takes to get everything back to production.  Other things to consider is temporary staff space, alternate communication contingencies.  You may want to look at a http://www-03.ibm.com/servers/storage/services/featured/essgdps.html, http://www.comdisco.com/, they are full-service providers in this arena.  
$$$You can really go burzerk and do the live update to the active disaster recovery site$$$.  BTW BIG companies have the resources and needs to do this.

Beyond 2 weeks- (permanent displacement) - its time to look for a new site for the business and the question becomes Can I rebuild with the data that I stored offsite.  Tape is great in this scenario also.  Financing and personnell planning is good here also.  

Although I didn't EXCLUDE anything in the disaster scenarios, its your job to include the best affordable technology to ensure that the business continues to operate.

Did I say too much??  And hopefully it made sense...

Regards,
0
 
David_FongCommented:
Didn't Sungard buy Comdisco?
0
 
Expert4XPCommented:
These are all great comments.  

I'd like to especially strongly agree with this statement by ECNSSMT:
>> Having said that, the focus of Disaster Recovery or really Business Continuation should not center on tapes, but rather the scope of the disaster.  

An easy way to test this theory and your company's readiness is to ask each department and person:  What will you do and how will you proceed if this building is destroyed (by fire, flood, hurricane, etc.)?  What is their disaster plan for the documents stored in their desk, in their file cabinets, etc.

It's really NOT a pc or server problem.  It is a company problem that must be addressed within the total context as ECNSSMT stated.

Many times companies think if the computer is recovered, all their problems are solved.  Not in my opinion.
0
 
Froggy_chrisCommented:
I've read something saying that DRP has to be seen from business point of view, not technical at the first place. I guess it summarize everything :)

and DRP must be seen company-wise (buildaing, access, who's doing what... and who takes over is first one "disappear"..) IT/technical solution will come at the very end, as tools to build-up the DRP.

Chris
0
 
ECNSSMTCommented:
David Fong,  If they did, I didn't keep up with the news...

Regards,
0
 
ECNSSMTCommented:
Froggy Chris, (no offense just wanted to use the name)

its a process and the focus is "what do I do to ensure that my business continues to work if a problem occurs..." everything else in one sense falls into place with that idea.  Everyone technically has a role in this process; the IT person has his (or her focus).  This is the same for the department heads, accountants etc.

Its a script to ensure that everything is still producing even when the building isn't there.

Regards,  
0
 
ECNSSMTCommented:
Hi mancoi
>How should I point the users to that server?

Anywhich way you can.  DR sites have office space.  Users can also telecommute if you provide the means for it.  The more you can integrate DR ideas into regular production, the more it means you are effectively using a tool you are paying for.

Regards,

0
 
mancoiAuthor Commented:
I am very pleased with the brilliant responses. I beleive this forum is one of the best.

I think everyone put extreme thought into this forum. I applause everyone!!!

This is a small business 30 users and needs to prepare a Hurricane recovery system. I plan to have a remote DC and an exchange sever in another state. The data is minimal. I use Backup Exec.

What files should I backup? The remote servers on stand by have the same applications and hardware.

scrathcyboy - I am giong to implement a VPN to a remote site and backup to a remote drive. Should this work w/Backup Exec?What should I send across the WAN? Just the backup files ro everything?
meyersd - I will definately backup to tape. I wil take everything you said into consideration.
BouL82 - What is iSCSI? do you mean backing up to a hard drive off site?
Froggy_chris  - I like your DVD idea. I will probably do both DVD and DLT Tape.
AlexGGGG - I will not have 2 identical replicating systems. I plan to only use my remote servers to act as a temporary replcaement.
David_Fong - I will ship my tapes in the event of a Hurricane.
ECNSSMT - Great ideas...you did not say too much!! I am looking backwards, but mainly forward. For now, I need to develope an immediate disaster recovery system. This is meant by backing up the day before the hurricane hits. I would like to replicate accross the WAN and to Tape or DVD and ship ASAP. I will be addressing what will be needed to keep the business running when an office is shut down due to a hurricane, but the rest of the country is still going forward. Employees around the country will still be able to work using the remote servers.
Expert4XP - I will be thinking about physical data as well.

Another thing to think about is the reverse recovery plan. If data is being updated at a remote location, it must be brought back to the main office when the disaster clears.

Thanks, please keep the forum going and add more ideas if anyone likes.
0
 
Expert4XPCommented:
mancoi, one more thought (I can't remember if anyone else mentioned this or not).

It's important to TEST your Business Recovery plan on a regular basis.  Have your CEO/CIO declare an "emergency" some Saturday morning.  Try and bring up your network using the remote site.  Have the other 30 (or key individuals) go to your new hot-site and see if they can conduct business with what they have saved in their department offsite vault. etc etc.

I was part of a large corporation and we did this each year (everyone hated doing it...) and then we had a debriefing the following week and made notes so that the next one was better than the last one.

I remember reading once that many companies go bankrupt after a major disaster such as a fire or natural disaster; not because they couldn't get the computer up and running, but they lost their customer records, couldn't continue ordering or shipping, and as a consequence their customers (can't blame them really) went to a new vendor.

Sometimes the physical computers and restoring the server are the LEAST of the items really needed for business resumption.  It's hard for companies to realize this, but I bet everyone here knows it is the truth.

Great discussion!
0
 
Duncan MeyersCommented:
>Should this work w/Backup Exec?What should I send across the WAN? Just the backup files ro everything?
Take a look at Replication Exec http://www.symantec.com/Products/enterprise?c=prodinfo&refId=50 - the console integrates with Backup Exec. You replicate the data back to your primary site and back it up there. If you need to restore individual files, you can restore them over the wire. If you lose the server, you can rebuild it at the primary site and ship it out to the remote site once it's rebuilt and ready. This all depends on how much you have to spend and how long an outage you can tolerate,
0
 
Froggy_chrisCommented:
ECNMSST,

I, in fact, agree with you. Got problems to express it (as my nickname shows, I'm not native english speaker, so sometimes I miss my points :)).

The idea I tried to point out is that a DRP built by IT only, based on "backup & restore" capacity/knowledge or pure technical consideration is not enough and will be dangerous for the company at the end.

Chris
0
 
BouL82Commented:
iSCSI is SCSI over IP. A cheaper alternative for SCSI over Fiber Channel. A great advantage of iSCSI is that it runs over ip so you can build geographicly dispersed cluster much cheaper then over traditional dark fiber.
An iSCSI disk appaers as a locally mounted physical disk to the OS. Even if it's miles away. It just communicates over your network interface with the iSCSI controller.

http://en.wikipedia.org/wiki/ISCSI

If you plan to have a second site (DC), what you can do possibly is build a geo cluster. What you need is 2 iSCSI arrays which can replicate data directional or bidirectional.
So in case of HW failure or disaster you can swith over to the second site within minutes. If your iSCSI array supports snapshots you can make regular snapshots of your data as an backup. You can restore snapshots within minutes as well. Or you can choose todo traditional backup to disk, which takes longer to restore but works as well.
So now you have two fault tolerant exchange servers over two DC's and your data offsite as well. Either in de first DC or in the second DC, online on disk.
If you need to store your backups for a long time, you may decide to move data to tape as well for archiving purposes.
Disk kind of so-called disk-to-disk-to-tape backups are pretty common nowadays, we use it for a larghe trading environment and a bank.
But maybe overkill for a smaller environement, but like i said before. I don't know what the business requirements are.

- What is the retention time of backups; how long do need to be archived?
- What is the time you need to get back online?
- etc etc

Cheers,
Roeland
0
 
mancoiAuthor Commented:
I agree....Great Forum.

Expert4XP - Great point!! Every company needs to have an "Emergancy Day" This will sharpen skills and surface loop holes in the DRP.

meyersd  - That link is very useful. Great source!!

 BouL82 -  Our organization is retaining tapes for two months of data. Should I broaden my retention time? In an event of a Hurricane ( corporate office in S. Florida)
Everyone expects a mass power outage. This could last for about a week as seen last year. The corporate office in S. Florida will close a day before the hurricane hits shore. If the hurricane hits the shore, we are most sure the S. Florida office will have no power. In the meantime the rest of the offices around the country are moving forward with sales and B2B. So, with that being said, the day before the hurricane hits, I need to get our remote servers (which are located in another office far away from Florida) up and running to host MS Exchange and AD and other Accouting Apps. I have one day to do Full Backups, send the tapes off-site, replicate backups over WAN and configure offices to point to the remote servers. Our company equips everyone with a laptop, so extra batteries are needed ( approx. 3 hr. battery life per battery = 3 hr. of work a day) and WiFi Verison cards. This should hold the company together for a few days.
Any comments?


0
 
Expert4XPCommented:
>> I have one day to do Full Backups, send the tapes off-site, replicate backups over WAN and configure offices to point to the remote servers.

The Full Backps have to be part of a daily/weekly backup plan.  I would never depend upon being able to take some action immediagtely or a day BEFORE a disaster occurs.  What happens if (God forbid) a plane hits the building, a fire erupts, etc.?

While planning specifically for hurricanes is understandable in Florida, a disaster plan (and the testing) should assume that no one is in the office, the office is burned down, gone.  period.  Where do we go from here?

What about all the information on those laptops that half your employees left at the office (now destroyed)?  It's not just the server, but all information in that building that is used for a Business Resumption Plan (not disaster recovery plan).
0
 
mancoiAuthor Commented:
Backups are done on a regular basis, but the last one would be made sure it completes when everone leaves the buiding (last day of work and the start of Hell for me if unprepared).

Backing up all users computers will be done as well.

Employees would be required to bring their laptops and extra batteries with them in the event of a hurricane.( unless they have to evacuate or its a high category hurricane)
This would ensure work from them for about 4 hours per day. or how ever long the batteries will last them.
Generators could be used to recharge the batteries as well.

Hurricane season is near. CEO's are wary. Its all about preparation for the worst.
0
 
David_FongCommented:
Have you got email sorted out if there's a disaster? Fairly simple to do, just have a second MX record pointing to the DR site and stop the SMTP service on the DR site so that it won't accept mail. Then all you have to do if the primary site goes down is start the service after restoring Exchange and the mail will be recieved as long as the primary is dead. Best done now rather than getting DNS changed and waiting for it to propagate around the world later.
0
 
Expert4XPCommented:
Mancoi, it's good that everyone is thinking about and planning for hurricane season.  You're ahead there.

I was trying to point out (without being too dramatic) that it's the *unplanned* disasters that need to be considered--such as an explosion, immediate fire, electrical fire, etc. that destroy EVERYTHING in the buildings.  Start with that as the baseline for disaster planning.  Assume you can't take anything out of the buildings except yourself.  Don't plan in that someone will run a backup job and grab some tapes from the fire.  (that only happens on tv)

I know you're trying to validate and complete your plan now.  I remember that there are Business Resumption Planning seminars that in the long run could help you.  BRP is really a profession in itself, especially for a large company with multiple locations to consider.
0
 
ECNSSMTCommented:
Hello Mancoi,

Tape retension time is per the need of the organization.  Banks have a requirement to be able to provide 3 years of historical data to several regualtory bodies; in general the banks I've been with keep about 7 years of tape backups; this includes email data.   I can also say that this is a general practice in the financial industry.

As you provide more information, it sounds like you are attempting to harden your business infrastructure against an annual occurance.  The question is how to allocate your limited resources to combat this.  If the building and more importantly the network room is structurally capable of surviving the primary and secondary effects of the hurricane, do you want to put in a gas-powered generator to provide electricty, if the building didn't already provision for that.  Do you want your employees to return to the work site immediately after the devastating event? Or do you want everyone to regroup at the DR site.  The DR site, assuming that it didn't get hit by the same event will have a fully functional facility; power, network and network infrastructure, telco lines,  so users with laptops and batteries will be a minor point.

I couldn't figure where you were intending the users to be working from but a lot of the details becomes very important and becomes a contingency that has to be planned for.  If email is the only concern and the users have to be in or around the disaster area (I can only think of insurance people as having this type of a requirement)  then maybe laptop equipped with Verizon wireless cards, with internet and OWA access to the primary or DR site (this may have some value as an active tool, if these users are travelling to many local client sites).  If you can hedge the technology to provide an active service instead of a dormant feature the technology becomes more valuable in many ways.

Still thinking about a lot of the other things said.  There are a lot of variables and a lot of possible solution.  You want a detailed process that can cover the hurricane scenario.

Regards,
0
 
Jennifer1024Commented:
For more information on building a Disaster Recovery Plan try the following website. www.drj.com. As a DRP beginner myself not that log ago this site helped me to get started. If you have the chance to go to there conference as well this is a great way to talk to others in the same boat. Everyone here is correct to some extent. You need to consider the business side of the equation  as well as the technical when creating the plan. The best place to start is going to be doing a business impact analysis. Whether you do this yourself by asking the different departments what programs are essential for them to be up and running and in what time frame or you hire a consultant to do the asking for you this is where you determine what is necessary and budget for those instances. Keep in mind also that you have to plan for the worst but the worst is not always what happens. Plan for not being able to reach your building and all the equipment being fine. Make sure you create your command center point where everyone is going to meet to determine the extent of the disaster. Figure in small disasters such as recovering partial program data or for hard disk failure. DRJ also has a free magazine that you can recieve with many tips and tools to use. We personally use tape as our backup medium and have found it very reliable.
0
 
Expert4XPCommented:
mancoi -- I almost hate to see this closed, because it has been such an interesting and informative discussion.

Thanks and good luck!
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

  • 5
  • 5
  • 4
  • +7
Tackle projects and never again get stuck behind a technical roadblock.
Join Now