Why my PHP file failed to complete batch process after 4 hours of data processing?

I have a PHP file that does some data processing work in the mysql database and I run in from a web browser each weekend.

The PHP file usually completes the job in 4 hours, but past 2 weekends, it failed to complete. Then, i need to reexecute the file.

Sometimes during the data processing, it seems like, when inserting rows into team table, there suppose to be 11 rows per team id, but some team id only have 7 rows. And this happen randomly. Each time we run the script, team id 4 can missed some rows, but second time we run, team 10 may missed some rows

Why?

This is the mem health of my db server.

             total       used       free     shared    buffers     cached
Mem:       6291456    6217556      73900          0     172192    5286472
-/+ buffers/cache:     758892    5532564
Swap:      2096440         64    2096376
melwongAsked:
Who is Participating?
 
virmaiorConnect With a Mentor Commented:
If PHP runs out of memory, it crashes completely -- not partial runs.

There's no specific danger by storing things in a PHP array. By that, I mean that the only danger is if the physical hardware is so unstable that it normally fritzes and loses things.

In other words, the problem is located elsewhere.  Somewhere you've got a bad assumption running around about what's going to last until the previous step is done or a logic error that can muck things up.
0
 
virmaiorCommented:
I don't think this has to do with the memory health of your server, and you're not giving enough information to make sense of what's going on.

My guess as to what's happening is that you've got an assumption that is true on a short time scale but false across the total time it takes for your processing to happen.  Say for instance, you clean up a forum every weekend, but people are still posting to it during the cleaning process.
0
 
Steve BinkCommented:
The easiest way to debug this is to use some profiling in your script.  Write interesting data points out to the log at key moments during execution.  If the data points do not look right, you have a clue to follow.  Interesting data points could be SQL queries about to be run, the return or row count from those queries, etc.
0
 
melwongAuthor Commented:
The problem i can't give much info on what's going on because the batch files does A LOT of data movement - select, delete, insert. We also store data into PHP array.

Is it safe to store data into array while the script is run for 4 hours? Will the data in array be lost during the process?
0
 
melwongAuthor Commented:
thx
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.