Best Way to keep Huge XML Request and Response Logs.

I am having thousands of XML Requests/Responses Log with each log size can be anywhere 1mb to 10mb. We need to keep them only for investigation if something goes wrong. We need to able to locate and open the log file. As of now we are keeping the same in a folder on the server but since the daily usage of log files are 50-60 GB per day and frequently ran out of space. So my question is what is the best way of managing such data? Ideally i want it to be in a db that we can archive and search for it even after 3 months. Should i zip the same and keep the pointer in db?
Maven 0001Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

I would explore different compression options.  Performance is not an issue. compression % is the prime consideration. (7-zip, winzip, gzip, and rar are good candidates)

Save some meta data in a database, allowing you to search for the log you need and then decompress it.
Gary PattersonVP Technology / Senior Consultant Commented:
Agree with aikimark that compression is the way to go (in or out of the database, depending on your needs).  

Disagree that 7zip, winzip, gzip, or rar are good candidates.  XML has some particular characteristics compared to random data that make it a candidate for compression techniques specific to XML.

Look at EXI.  EXI is a binary XML format that often compresses better than gzip, and is much more efficient.

Next is the "int the database" or "outside the database" question.  This is more complicated, and a lot depends on the nature of the data, the capacity of your db servers, frequency of access, performance needs, data replication needs, existing infrastructure, cost, retention requirements (troubleshooting, audit, compliance reasons), and more.  

Large, infrequently-accessed documents needed just for troubleshooting purposes might be best stored external to the database on an inexpensive storage array, with just a link or a file naming convention to identify them.

Small documents that are frequently accesses, and have compliance-based retention requirements, retention requirement, and backup requirements may be better off stored in the database - especially if you already have a replicated highly-available database infrastructure in place.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
I just did a test of an XML file and 7zip compressed it from 164k down to 10k = 6.556% of its original size.  I got this reduction using both the Ultra and Maximum compression levels, both employing the PPMd compression method.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.