Backing up 8.6TB within 12 hours

Hi experts,

Currently in our infrastructure, we have 11 Citrix servers totalling at about 1.5TB and 11 other servers totalling at around 7.2TB.

We use zeeam backup and replication 8.0 to run our backups.

There are two backup jobs set up in veeam, broken down as mentioned above:

11 servers totalling 7.2TB - run daily. (This was set to reverse incremental) but was taking about 12 hours to complete, so I had to change this to 'incremental' which now only takes a couple of hours; the problem is, to complete a full backup weekly, this is looking to take around 42 hours roughly)

And another 11 servers totalling at just 1.4TB currently set to reverse incremental, and is run once a week, and takes around 8 hours.

Currently the repository drive is connected to the backup server via iSCSI, and seems to transfer at a very low speed of about 5 - 10MB/s

Does anyone have any better suggestions. Any particular drive or interface to use that will smash this easily?

I need to be able to backup a total of 8.6TB (and growing) in a 12 hour window.. Is this possible?

I don't want it to be a fine line, I would hope I can have this done in about 6 - 8 hours? Any technology out there that is capable of this?

Look forward to your suggestions
Nathan LindleyIT Support EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

does your backup include/run on a separate dedicated network or is the data flow into the backup server and the backup data to storage flows over the same single network connection?
Scheduling individual server backups at a time would meet your criteria by adding an intermediary local 3TB drive that is not network depended.
Disk to disk to iscsi/tape

What is your iscsi storage of?
Nathan LindleyIT Support EngineerAuthor Commented:
Hi Arnold,

I am new to the organisation and did not set this up, so I can't be 100% on the network connection; but I do know that our backup server is the only physical server we have, which is on the same subnetwork as all the Virtual machines.

So, I think the answer to your first question is, we have dataflow into a backup server, which then stores the backup onto an externally connected iSCSI device. I hope this makes sense.

So, this iSCSI setup is an iSCSI software initiator, and the target is a hardware storage array, which is a NAS consisting of 4 x 2TB disks in RAID5 configuration.

not sure of the spindle speed of the drives though.
Does the server have multiple network interface specifically one solely dedicated to accessing the iscsi device which is network dependent.
The difficulty from this vantage point is that the vms might also require access to the iscsi storage.

Altering the schedule such that individual vms are backed up on separate schedules. I.e you have 11 vms amounting to 7.2TB
If you want to limit to weekdays. Breaking up two vms per day for full while the others are incremental. You may want to check what the per server backup is and the rate of change.  If all are if equal size and of equal rate of change, your daily backup full plus incremental will be in the 1-1.5Tb range and will meet your need to have the backup complete within a shorter time window.

The offset is on the other side the rate of restore in the event you need it?

Not to be, raid 5 ....... Not optimal in such an environment, a rebuild should one fail will take a long time.  Based on the size, these are sata drives with 7200 rm likely 5400 if they are 2.5 inch drives.

From the reading you have one and only one physical server hosting 22 vms?
Or you have other physical servers functioning as hosts to 11vms each with this one server functioning as the sole backup?
Nathan LindleyIT Support EngineerAuthor Commented:

we have 3 ESXi hosts and a HP NAS that manage the 11 VM's - I will look to backing up 3 - 4 servers a night, leaving the hugest server (SQL-01) which is 4.5TB alone, to back up on it's own job at weekend
Make sure you account for he restore time requirements.
Does your setup include transition of vms live should one host fail?
Depending on the age of the setup, he backup might not be the higher priority for you at this time.
I.e. Upgrade cycle renewing the oldest host.
Nathan LindleyIT Support EngineerAuthor Commented:
I have just realised, that taking 3 - 4 FULL backups every night, does not leave room for daily incremental backups for the other servers..if you follow?
It does, what do you mean?  Is there a limit on how many backup plans you can have?
You are effectively reducing the amount of data transferred daily while on a weekly basis the same amount of total data is transferred.

Those servers should be similarly subdivided.

Perhaps I am not clear of what the remaining issue is.

I.e. You have five for each day full backups each includes a couple.
The backup is either initiated from the backup server, or from the client.

Please clarify the basis of your statement.
Nathan LindleyIT Support EngineerAuthor Commented:
Let's say a FULL backup is taken of server-01 on Monday; an incremental backup is taken on Tuesday.

But if I have full backups taking place every day of the week, this leaves no room for an incremental backup of a server fully backed up the day before for example, hope this clarifies
The smaller full backup will be done within a few hours leaving room for the incremental.  I am not understanding the backup plans available to you.
Just to jump in here, first of all your backup depends on a lot of factors - including some things not commonly thought of, for instance:

1.  Speed of the physical disks on the VM servers.  You can only read the data as fast as the physical hardware allows.  7.1TB is A LOT of data to try to copy to begin with if you were just doing a straight file copy.  Add in processing, compression, etc and that time can expand.

2.  Speed of the physical disks on the backup hardware.  Same applies as #1.  If they are not capable of the speed, then you will never achieve this.  Some testing would be in order.  Copy a file of known size to the backup NAS unit and see what kind of throughput you get.  Extrapolate that out to determine what the copy speed is.  FYI - RAID 5 has one of the poorer throughputs.

3.  Speed of the network.  If you are using iSCSI then how fast is the connection between the NAS and the VM servers?  1 TB of data will take about 3 hours to copy over an optimal Gigabit network.  That does not account for other data moving across the network.  Ideally this means you would have a dedicated network for your storage.

4.  Other process on the VM's themselves that might cause increased IO on the host boxes during the backup process.

As others suggested above you may need to rethink you backup job strategy and that may solve it, however you definitely need to look at your overall setup and ensure it is even capable of what you are asking it to do,.

In our case we have a similar setup, but I use port-bonding in our switches and have 4 NICS dedicated to the storage network to increase the throughput to where we need it.  Additionally we have segmented networks so the storage net never sees any other traffic.

Hope this helps a bit.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Brian CTXSupportCitrix ConsultantCommented:
How much is changing on the Citrix servers that require them to be backed up daily?  Seems excessive to me.
It depends on what the line of business is and how much data ...
I.e. An architectural firm/engineering firm/audio video editing/graphical.... documents will be of a significant size.

The backup could reflect snapshot of the VM I.e ............
Brian CTXSupportCitrix ConsultantCommented:
Properly built and configured Citrix servers wouldn't be housing any data that needed to be backed up.  User data should be redirected to a home drive, and the servers should be managed from a master image.
Nathan LindleyIT Support EngineerAuthor Commented:
OK, it appears that I need to suggest we change the backup target device. I will also look into the network throughput etc, but right now, I am having a capacity issue also (running out of room to back everything up)

In total, all our servers use around 8.5TB of storage. This does not include daily incremental backups.

I think it would be safe to say I need at least 12TB of backup storage, and after reading about RAID configurations, I think a RAID10 setup would be good for speed and reliability.

I have been looking at one of these,

If I filled it with 10x 3TB drives and configured it to RAID10, am I correct in saying this would give me 15TB of storage to use, or would this give me only 7.5TB to use?
Brian CTXSupportCitrix ConsultantCommented:
You can get immediate relief if you remove servers that don't need to be backed up.  What version of Citrix are you running?  Of the 11 servers, what are the roles (Controller, WebInterface/StoreFront/Worker servers)  Are you redirecting the user folders and profile to a home drive or profile share?  Are you using a SQL or SQL Express database?  If you have an up-to-date master image, you could turn of backups entirely for the worker servers.  If the database is on SQL, then the vital information for the controller would be there, so the controller wouldn't need to be backed up very often.  Web Interface/StoreFront configuration is generally very static, and wouldn't need to be backed up often.  So your backups that remain would be the file and database servers.
I like the QNAP boxes. One possibility for increasing your backup throughput is to get two NAS boxes for backup targets and arrange your network connections so that there is sufficient capacity to do backups in parallel instead of only serially.
If you are upgrading/increasing capacity, you should not merely go up one level to pass for now, depending on the age of the one you have, you could use one for the full backups of some systems and incremental.
Multiple destination devices for different systems. do you keep more than one full backup at a time?

It depends on the one you have, some provide the option of chaining that could ......

Another option is to use less frequent full backups as Brian pointed out, the system's themselves need only be fully backed up on changes (update installs/application installs/etc.)full backup right before and if it goes well, right after discarding the prior ......

Tony JohncockLead Technical ArchitectCommented:
Brian beat me to it but generally speaking I don't backup Citrix servers (depending on the role of course) at all, let alone daily. For things like XenApp workers, they should be changing so rarely that what's the need?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.