Solved

Best Method for Temporarly Storing and Reading Logs

Posted on 2014-04-13
12
304 Views
Last Modified: 2014-05-26
Hello,

We are currently writing the business requirements for a website logging application used in a specialist in house CRM application. The website logging application, logs the the various screen views within the CRM application, using a PHP post script on a data collection server that passes variables and values into a database server.

The question I have, is what is the best way to cache and queue the logs locally on the data collection server in the event that the connection to the database server goes down. As the variables and values can change from screen view to screen view within the CRM application, the data collect is not going to be always the same. Which make me think that a local MySQL server wouldn't be a good way to go as there is no way to predict the database schema.

In the scenario above what would be the best way to temporary cache the logs before they are passed to the database server.

Thanks
0
Comment
Question by:PlumInternet
  • 5
  • 3
  • 3
  • +1
12 Comments
 
LVL 61

Expert Comment

by:gheist
Comment Utility
Why you are reinventing the wheel? Webservers write logs and applications can add their entries to that log (And THAT log in turn can be piped to locally running program like e.g. apache rotatelogs)
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
Agreed, don't reinvent the wheel. If you want more than web server logs, look at google analytics or something similar.
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
Yo can run your own analytics too. On logs post-factum etc
0
 

Author Comment

by:PlumInternet
Comment Utility
Thanks to everyone for responding.

For reasons I cant go into here, using web logs (is server side reporting, no client side reporting server logs will miss critical user behaviors) and Google Analytics (data ownership plus data volumes will cause data to become sampled) is not a viable option.

We have moved beyond the reason why we need to solve to the problem this why as there is very valid reasons, to now trying to figure out how. Any suggestions on what would be the best way to solve this problem would be greatly appreciated.

Thanks
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
0
 

Author Comment

by:PlumInternet
Comment Utility
Ok I am guessing that no body knows the answer to this question then.

If it will help move things along, lets say hypothetically we want to create another competitor to the solutions listed, Or lets say I am a curious guy and want to know how the solutions listed solve this problem. Forget the why, the question I am asking is how.
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 250 total points
Comment Utility
Alright then, I would start with elasticsearch as my goto log storage. One cool thing about ES is that you send data with a simple json web request. So if the ES server isn't around, just save the request locally (file, db, object, whatever). This is perfectly fine for lower volume web traffic.
If you have more demanding data storage needs, look at messaging queues. The most popular one is rabbitmq.
http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/
0
 
LVL 33

Accepted Solution

by:
Slick812 earned 250 total points
Comment Utility
Greetings  PlumInternet, , , , I have been able to save In File many data sets, and have them stacked, residual, series, and various formats or as TEXT, or as binary. In PHP I have found making an "array" to be the container as a ARRAY  or  a Key-Value data set, will produce a "Tree access" like a file system. You place into this any and all "data" you need to store, and then -
$store2file = serialize($array_Data);
and make a file.

BUT, you will need to KNOW, HOW, WHY, and WHEN, you need to access this file and READ the "Portions" of Data, stored there.

But, for me, this works, time and time again.
The serialize( ) is a native PHP, while JSON and others (XML) are foreign to PHP.
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
While you stay hiding what "critical behaviour" webserver is not able to log, even keeping us guessing what webserver(s) you have, i would say nobody can provide you with what you want past that the BEST way is to use existing logging capabilities.
0
 

Author Comment

by:PlumInternet
Comment Utility
Thanks Gheist, that's precisely my point. The question is not whats the best way to use existing logging capabilities is it.....To recap the question was:

"question I have, is what is the best way to cache and queue the logs (collected post request data, not web logs) locally on the data collection server in the event that the connection to the database server goes down."

I think slick812 has been right on the money with the help we are looking for. Thanks Slick812 :)
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
Would be nice to know we talk here IIS apache nginx ot log4j .... Capabilities and options depend on that.
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
In general :
Get log analysis tool that shows what you want
Make webserver log required input (with reservation that it will be impossible to recover missing data from the past)
Will it be OK to do hourly or nightly log transfers? -> transfer logs
No -> try syslog or SQL (slowdown but hopefully for good purpose)
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Nothing in an HTTP request can be trusted, including HTTP headers and form data.  A form token is a tool that can be used to guard against request forgeries (CSRF).  This article shows an improved approach to form tokens, making it more difficult to…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now