?
Solved

I need to backup 2TB a day of uncompressed apache logs. What are my backup options?

Posted on 2009-12-28
8
Medium Priority
?
343 Views
Last Modified: 2013-12-01
I need to backup 2TB a day of uncompressed apache logs. What are my backup hardware options?
0
Comment
Question by:martin_2110
  • 4
  • 2
  • 2
8 Comments
 
LVL 3

Accepted Solution

by:
HamStab earned 750 total points
ID: 26134882
For that much storage i would recommend a NAS with several 1.5TB hard drives in it. You can find them online, you will probably need a 2u or 4u depending on how many days you want to backup.

Why are your apache logs so large? :O
0
 
LVL 7

Author Comment

by:martin_2110
ID: 26134938
Looking for off site backups. I don't really consider nas a backup. We need to keep 2 years of logs around so backing up to disk is going to cost to much. The apache logs are 300GB a day currently with plans to grow to 1TB. They are that big because we are a large site.
0
 
LVL 3

Expert Comment

by:HamStab
ID: 26134977
To my knowledge, NAS would be the only and most cost effective way to backup the logs. With that much information, having a service for backup would cost 4 livers per hour. Having off-site backups where you would upload the files using bandwidth would cost even more unless you purchased a private 10mbit or 100mbit (which unmetered 10mbit should be fine for just backups).

I would actually check with your datacenter and see what they offer. But your best bet would be a NAS and to colo it to your datacenter and make sure they will allow network traffic being sent to and from your server you're backing up from. With that said, they might want you to get a switch of your own and pay for a tech to run a cable and install.

I used to help run a very large website and we just never backed up apache logs, that might be your best bet :X
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 
LVL 26

Assisted Solution

by:arober11
arober11 earned 750 total points
ID: 26145735
Are you sure you require 2 years of logs with no compression?  As your looking a  1.5+ petabytes of storage uncompressed, which isn't cheap. Note the logs would compress to > 10% or their original size.

What access requirements do you have?

Will multiple processes require simultaneous access to the logs?

If on-line / real time access is required, what flavour of SAN do you have at present?

What operations will you be performing on the logs? Note only 64-bit versions of the standard log analysers are happy with log files over 2GB.

Have you considered directing all SSL traffic to separate virtual hosts / logs and only keeping the SSL logs?

I take it you have multiple load balanced servers and some form of scheduled daily log rotation across the servers.
0
 
LVL 7

Author Comment

by:martin_2110
ID: 26194944
Yes we currently do about 500GB a day I am planning for 1.5TB a day. Yes it is already compressed with about a 1/8 ratio. I realize it is a crazy amount of logs. We process the logs via our own scripts. This is all http we don't do https. I am starting to lean towards LTO4 tape drives. We currently backup to nas and will archive off to TAPE.
0
 
LVL 7

Author Comment

by:martin_2110
ID: 26196413
Sorry Uncompressed not compressed.
0
 
LVL 26

Expert Comment

by:arober11
ID: 26206154
If your going for tape then you'll want a library and a robot or three, else you may end up loosing logs if a single robot fails (log rotate daemons  often truncate logs when there isn't space to rotate), and I guess you may want to recall logs from time to time, so again another solution.

I'd suggest you contact the likes of HP, Omniture and EMC with you requirements, and they'll propose various options, send a few bodies over to go other your requirements, and generally leave a few free pens.

You may need to find a good local Enterprise Architect to review the proposal's, as any solution will cost several pennies.
0
 
LVL 7

Author Closing Comment

by:martin_2110
ID: 31670602
Did not really get what I was looking for I ended up using my solution, Thanks for the help guys.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article shows how to use a free utility called 'Parkdale' to easily test the performance and benchmark any Hard Drive(s) installed in your computer. We also look at RAM Disks and their speed comparisons.
A look at what happened in the Verizon cloud breach.
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …
Suggested Courses

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question