• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 402
  • Last Modified:

Why my PHP file failed to complete batch process after 4 hours of data processing?

I have a PHP file that does some data processing work in the mysql database and I run in from a web browser each weekend.

The PHP file usually completes the job in 4 hours, but past 2 weekends, it failed to complete. Then, i need to reexecute the file.

Sometimes during the data processing, it seems like, when inserting rows into team table, there suppose to be 11 rows per team id, but some team id only have 7 rows. And this happen randomly. Each time we run the script, team id 4 can missed some rows, but second time we run, team 10 may missed some rows

Why?

This is the mem health of my db server.

             total       used       free     shared    buffers     cached
Mem:       6291456    6217556      73900          0     172192    5286472
-/+ buffers/cache:     758892    5532564
Swap:      2096440         64    2096376
0
melwong
Asked:
melwong
  • 2
  • 2
1 Solution
 
virmaiorCommented:
I don't think this has to do with the memory health of your server, and you're not giving enough information to make sense of what's going on.

My guess as to what's happening is that you've got an assumption that is true on a short time scale but false across the total time it takes for your processing to happen.  Say for instance, you clean up a forum every weekend, but people are still posting to it during the cleaning process.
0
 
Steve BinkCommented:
The easiest way to debug this is to use some profiling in your script.  Write interesting data points out to the log at key moments during execution.  If the data points do not look right, you have a clue to follow.  Interesting data points could be SQL queries about to be run, the return or row count from those queries, etc.
0
 
melwongAuthor Commented:
The problem i can't give much info on what's going on because the batch files does A LOT of data movement - select, delete, insert. We also store data into PHP array.

Is it safe to store data into array while the script is run for 4 hours? Will the data in array be lost during the process?
0
 
virmaiorCommented:
If PHP runs out of memory, it crashes completely -- not partial runs.

There's no specific danger by storing things in a PHP array. By that, I mean that the only danger is if the physical hardware is so unstable that it normally fritzes and loses things.

In other words, the problem is located elsewhere.  Somewhere you've got a bad assumption running around about what's going to last until the previous step is done or a logic error that can muck things up.
0
 
melwongAuthor Commented:
thx
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

  • 2
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now