Best Way to keep Huge XML Request and Response Logs.

Posted on 2014-08-09
Last Modified: 2014-09-11
I am having thousands of XML Requests/Responses Log with each log size can be anywhere 1mb to 10mb. We need to keep them only for investigation if something goes wrong. We need to able to locate and open the log file. As of now we are keeping the same in a folder on the server but since the daily usage of log files are 50-60 GB per day and frequently ran out of space. So my question is what is the best way of managing such data? Ideally i want it to be in a db that we can archive and search for it even after 3 months. Should i zip the same and keep the pointer in db?
Question by:Maven 0001
    LVL 44

    Expert Comment

    I would explore different compression options.  Performance is not an issue. compression % is the prime consideration. (7-zip, winzip, gzip, and rar are good candidates)

    Save some meta data in a database, allowing you to search for the log you need and then decompress it.
    LVL 34

    Accepted Solution

    Agree with aikimark that compression is the way to go (in or out of the database, depending on your needs).  

    Disagree that 7zip, winzip, gzip, or rar are good candidates.  XML has some particular characteristics compared to random data that make it a candidate for compression techniques specific to XML.

    Look at EXI.  EXI is a binary XML format that often compresses better than gzip, and is much more efficient.

    Next is the "int the database" or "outside the database" question.  This is more complicated, and a lot depends on the nature of the data, the capacity of your db servers, frequency of access, performance needs, data replication needs, existing infrastructure, cost, retention requirements (troubleshooting, audit, compliance reasons), and more.  

    Large, infrequently-accessed documents needed just for troubleshooting purposes might be best stored external to the database on an inexpensive storage array, with just a link or a file naming convention to identify them.

    Small documents that are frequently accesses, and have compliance-based retention requirements, retention requirement, and backup requirements may be better off stored in the database - especially if you already have a replicated highly-available database infrastructure in place.
    LVL 44

    Expert Comment

    I just did a test of an XML file and 7zip compressed it from 164k down to 10k = 6.556% of its original size.  I got this reduction using both the Ultra and Maximum compression levels, both employing the PPMd compression method.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Course: HTML5 Specialist

    HTML5 development skills are critical to all developers. HTML5 is the foundation to almost any development process. That's why business, design studios, development shops and other organizations need HTML5 developers. Get your foot in the door as a HTML5 specialist.

    Suggested Solutions

    Title # Comments Views Activity
    Help with my python script 6 135
    sum13 challenge 24 68
    VB 6.0 printer how to align 6 35
    python sqlite question 11 17
    Purpose To explain how to place a textual stamp on a PDF document.  This is commonly referred to as an annotation, or possibly a watermark, but a watermark is generally different in that it is somewhat translucent.  Watermark’s may be text or graph…
    RIA (Rich Internet Application) tools are interactive internet applications which have many of the characteristics of desktop applications. The RIA tools typically deliver output either by the way of a site-specific browser or via browser plug-in. T…
    An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
    In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

    759 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    13 Experts available now in Live!

    Get 1:1 Help Now