Solved

Best Method for Temporarly Storing and Reading Logs

Posted on 2014-04-13
12
330 Views
Last Modified: 2014-05-26
Hello,

We are currently writing the business requirements for a website logging application used in a specialist in house CRM application. The website logging application, logs the the various screen views within the CRM application, using a PHP post script on a data collection server that passes variables and values into a database server.

The question I have, is what is the best way to cache and queue the logs locally on the data collection server in the event that the connection to the database server goes down. As the variables and values can change from screen view to screen view within the CRM application, the data collect is not going to be always the same. Which make me think that a local MySQL server wouldn't be a good way to go as there is no way to predict the database schema.

In the scenario above what would be the best way to temporary cache the logs before they are passed to the database server.

Thanks
0
Comment
Question by:PlumInternet
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 3
  • 3
  • +1
12 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 39997958
Why you are reinventing the wheel? Webservers write logs and applications can add their entries to that log (And THAT log in turn can be piped to locally running program like e.g. apache rotatelogs)
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 39997976
Agreed, don't reinvent the wheel. If you want more than web server logs, look at google analytics or something similar.
0
 
LVL 62

Expert Comment

by:gheist
ID: 39997991
Yo can run your own analytics too. On logs post-factum etc
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:PlumInternet
ID: 39998009
Thanks to everyone for responding.

For reasons I cant go into here, using web logs (is server side reporting, no client side reporting server logs will miss critical user behaviors) and Google Analytics (data ownership plus data volumes will cause data to become sampled) is not a viable option.

We have moved beyond the reason why we need to solve to the problem this why as there is very valid reasons, to now trying to figure out how. Any suggestions on what would be the best way to solve this problem would be greatly appreciated.

Thanks
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 39998026
0
 

Author Comment

by:PlumInternet
ID: 39998051
Ok I am guessing that no body knows the answer to this question then.

If it will help move things along, lets say hypothetically we want to create another competitor to the solutions listed, Or lets say I am a curious guy and want to know how the solutions listed solve this problem. Forget the why, the question I am asking is how.
0
 
LVL 39

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 250 total points
ID: 39998059
Alright then, I would start with elasticsearch as my goto log storage. One cool thing about ES is that you send data with a simple json web request. So if the ES server isn't around, just save the request locally (file, db, object, whatever). This is perfectly fine for lower volume web traffic.
If you have more demanding data storage needs, look at messaging queues. The most popular one is rabbitmq.
http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/
0
 
LVL 34

Accepted Solution

by:
Slick812 earned 250 total points
ID: 39998219
Greetings  PlumInternet, , , , I have been able to save In File many data sets, and have them stacked, residual, series, and various formats or as TEXT, or as binary. In PHP I have found making an "array" to be the container as a ARRAY  or  a Key-Value data set, will produce a "Tree access" like a file system. You place into this any and all "data" you need to store, and then -
$store2file = serialize($array_Data);
and make a file.

BUT, you will need to KNOW, HOW, WHY, and WHEN, you need to access this file and READ the "Portions" of Data, stored there.

But, for me, this works, time and time again.
The serialize( ) is a native PHP, while JSON and others (XML) are foreign to PHP.
0
 
LVL 62

Expert Comment

by:gheist
ID: 39998314
While you stay hiding what "critical behaviour" webserver is not able to log, even keeping us guessing what webserver(s) you have, i would say nobody can provide you with what you want past that the BEST way is to use existing logging capabilities.
0
 

Author Comment

by:PlumInternet
ID: 39998350
Thanks Gheist, that's precisely my point. The question is not whats the best way to use existing logging capabilities is it.....To recap the question was:

"question I have, is what is the best way to cache and queue the logs (collected post request data, not web logs) locally on the data collection server in the event that the connection to the database server goes down."

I think slick812 has been right on the money with the help we are looking for. Thanks Slick812 :)
0
 
LVL 62

Expert Comment

by:gheist
ID: 39998704
Would be nice to know we talk here IIS apache nginx ot log4j .... Capabilities and options depend on that.
0
 
LVL 62

Expert Comment

by:gheist
ID: 39998709
In general :
Get log analysis tool that shows what you want
Make webserver log required input (with reservation that it will be impossible to recover missing data from the past)
Will it be OK to do hourly or nightly log transfers? -> transfer logs
No -> try syslog or SQL (slowdown but hopefully for good purpose)
0

Featured Post

[Webinar] Learn How Hackers Steal Your Credentials

Do You Know How Hackers Steal Your Credentials? Join us and Skyport Systems to learn how hackers steal your credentials and why Active Directory must be secure to stop them. Thursday, July 13, 2017 10:00 A.M. PDT

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
This article discusses how to create an extensible mechanism for linked drop downs.
The viewer will learn how to dynamically set the form action using jQuery.
The viewer will learn the basics of jQuery including how to code hide show and toggles. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery…

623 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question