Best Way to keep Huge XML Request and Response Logs.

Posted on 2014-08-09
Medium Priority
Last Modified: 2014-09-11
I am having thousands of XML Requests/Responses Log with each log size can be anywhere 1mb to 10mb. We need to keep them only for investigation if something goes wrong. We need to able to locate and open the log file. As of now we are keeping the same in a folder on the server but since the daily usage of log files are 50-60 GB per day and frequently ran out of space. So my question is what is the best way of managing such data? Ideally i want it to be in a db that we can archive and search for it even after 3 months. Should i zip the same and keep the pointer in db?
Question by:Maven 0001
  • 2
LVL 46

Expert Comment

ID: 40251161
I would explore different compression options.  Performance is not an issue. compression % is the prime consideration. (7-zip, winzip, gzip, and rar are good candidates)

Save some meta data in a database, allowing you to search for the log you need and then decompress it.
LVL 36

Accepted Solution

Gary Patterson earned 200 total points
ID: 40253277
Agree with aikimark that compression is the way to go (in or out of the database, depending on your needs).  

Disagree that 7zip, winzip, gzip, or rar are good candidates.  XML has some particular characteristics compared to random data that make it a candidate for compression techniques specific to XML.

Look at EXI.  EXI is a binary XML format that often compresses better than gzip, and is much more efficient.  


Next is the "int the database" or "outside the database" question.  This is more complicated, and a lot depends on the nature of the data, the capacity of your db servers, frequency of access, performance needs, data replication needs, existing infrastructure, cost, retention requirements (troubleshooting, audit, compliance reasons), and more.  

Large, infrequently-accessed documents needed just for troubleshooting purposes might be best stored external to the database on an inexpensive storage array, with just a link or a file naming convention to identify them.

Small documents that are frequently accesses, and have compliance-based retention requirements, retention requirement, and backup requirements may be better off stored in the database - especially if you already have a replicated highly-available database infrastructure in place.
LVL 46

Expert Comment

ID: 40317000
I just did a test of an XML file and 7zip compressed it from 164k down to 10k = 6.556% of its original size.  I got this reduction using both the Ultra and Maximum compression levels, both employing the PPMd compression method.

Featured Post

The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you see single cell contains number and text, and you have to get any date out of it seems like cracking our heads.
Today, unlike web development, the mobile landscape is complex enough for a software engineer and Android is posing more challenging environment thanks to its fragmentation issues on hardware and software fronts.
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …
Starting up a Project

600 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question