Why my PHP file failed to complete batch process after 4 hours of data processing?

I have a PHP file that does some data processing work in the mysql database and I run in from a web browser each weekend.

The PHP file usually completes the job in 4 hours, but past 2 weekends, it failed to complete. Then, i need to reexecute the file.

Sometimes during the data processing, it seems like, when inserting rows into team table, there suppose to be 11 rows per team id, but some team id only have 7 rows. And this happen randomly. Each time we run the script, team id 4 can missed some rows, but second time we run, team 10 may missed some rows

Why?

This is the mem health of my db server.

             total       used       free     shared    buffers     cached
Mem:       6291456    6217556      73900          0     172192    5286472
-/+ buffers/cache:     758892    5532564
Swap:      2096440         64    2096376
melwongAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

virmaiorCommented:
I don't think this has to do with the memory health of your server, and you're not giving enough information to make sense of what's going on.

My guess as to what's happening is that you've got an assumption that is true on a short time scale but false across the total time it takes for your processing to happen.  Say for instance, you clean up a forum every weekend, but people are still posting to it during the cleaning process.
0
Steve BinkCommented:
The easiest way to debug this is to use some profiling in your script.  Write interesting data points out to the log at key moments during execution.  If the data points do not look right, you have a clue to follow.  Interesting data points could be SQL queries about to be run, the return or row count from those queries, etc.
0
melwongAuthor Commented:
The problem i can't give much info on what's going on because the batch files does A LOT of data movement - select, delete, insert. We also store data into PHP array.

Is it safe to store data into array while the script is run for 4 hours? Will the data in array be lost during the process?
0
virmaiorCommented:
If PHP runs out of memory, it crashes completely -- not partial runs.

There's no specific danger by storing things in a PHP array. By that, I mean that the only danger is if the physical hardware is so unstable that it normally fritzes and loses things.

In other words, the problem is located elsewhere.  Somewhere you've got a bad assumption running around about what's going to last until the previous step is done or a logic error that can muck things up.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
melwongAuthor Commented:
thx
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
PHP

From novice to tech pro — start learning today.