Solved

Why my PHP file failed to complete batch process after 4 hours of data processing?

Posted on 2012-03-23
5
360 Views
Last Modified: 2012-06-27
I have a PHP file that does some data processing work in the mysql database and I run in from a web browser each weekend.

The PHP file usually completes the job in 4 hours, but past 2 weekends, it failed to complete. Then, i need to reexecute the file.

Sometimes during the data processing, it seems like, when inserting rows into team table, there suppose to be 11 rows per team id, but some team id only have 7 rows. And this happen randomly. Each time we run the script, team id 4 can missed some rows, but second time we run, team 10 may missed some rows

Why?

This is the mem health of my db server.

             total       used       free     shared    buffers     cached
Mem:       6291456    6217556      73900          0     172192    5286472
-/+ buffers/cache:     758892    5532564
Swap:      2096440         64    2096376
0
Comment
Question by:melwong
  • 2
  • 2
5 Comments
 
LVL 20

Expert Comment

by:virmaior
ID: 37760924
I don't think this has to do with the memory health of your server, and you're not giving enough information to make sense of what's going on.

My guess as to what's happening is that you've got an assumption that is true on a short time scale but false across the total time it takes for your processing to happen.  Say for instance, you clean up a forum every weekend, but people are still posting to it during the cleaning process.
0
 
LVL 50

Expert Comment

by:Steve Bink
ID: 37760928
The easiest way to debug this is to use some profiling in your script.  Write interesting data points out to the log at key moments during execution.  If the data points do not look right, you have a clue to follow.  Interesting data points could be SQL queries about to be run, the return or row count from those queries, etc.
0
 

Author Comment

by:melwong
ID: 37761831
The problem i can't give much info on what's going on because the batch files does A LOT of data movement - select, delete, insert. We also store data into PHP array.

Is it safe to store data into array while the script is run for 4 hours? Will the data in array be lost during the process?
0
 
LVL 20

Accepted Solution

by:
virmaior earned 100 total points
ID: 37761954
If PHP runs out of memory, it crashes completely -- not partial runs.

There's no specific danger by storing things in a PHP array. By that, I mean that the only danger is if the physical hardware is so unstable that it normally fritzes and loses things.

In other words, the problem is located elsewhere.  Somewhere you've got a bad assumption running around about what's going to last until the previous step is done or a logic error that can muck things up.
0
 

Author Closing Comment

by:melwong
ID: 37761962
thx
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

In Solr 4.0 it is possible to atomically (or partially) update individual fields in a document. This article will show the operations possible for atomic updating as well as setting up your Solr instance to be able to perform the actions. One major …
This article discusses how to create an extensible mechanism for linked drop downs.
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now