Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win

x
?
Solved

Why my PHP file failed to complete batch process after 4 hours of data processing?

Posted on 2012-03-23
5
Medium Priority
?
399 Views
Last Modified: 2012-06-27
I have a PHP file that does some data processing work in the mysql database and I run in from a web browser each weekend.

The PHP file usually completes the job in 4 hours, but past 2 weekends, it failed to complete. Then, i need to reexecute the file.

Sometimes during the data processing, it seems like, when inserting rows into team table, there suppose to be 11 rows per team id, but some team id only have 7 rows. And this happen randomly. Each time we run the script, team id 4 can missed some rows, but second time we run, team 10 may missed some rows

Why?

This is the mem health of my db server.

             total       used       free     shared    buffers     cached
Mem:       6291456    6217556      73900          0     172192    5286472
-/+ buffers/cache:     758892    5532564
Swap:      2096440         64    2096376
0
Comment
Question by:melwong
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
5 Comments
 
LVL 20

Expert Comment

by:virmaior
ID: 37760924
I don't think this has to do with the memory health of your server, and you're not giving enough information to make sense of what's going on.

My guess as to what's happening is that you've got an assumption that is true on a short time scale but false across the total time it takes for your processing to happen.  Say for instance, you clean up a forum every weekend, but people are still posting to it during the cleaning process.
0
 
LVL 51

Expert Comment

by:Steve Bink
ID: 37760928
The easiest way to debug this is to use some profiling in your script.  Write interesting data points out to the log at key moments during execution.  If the data points do not look right, you have a clue to follow.  Interesting data points could be SQL queries about to be run, the return or row count from those queries, etc.
0
 

Author Comment

by:melwong
ID: 37761831
The problem i can't give much info on what's going on because the batch files does A LOT of data movement - select, delete, insert. We also store data into PHP array.

Is it safe to store data into array while the script is run for 4 hours? Will the data in array be lost during the process?
0
 
LVL 20

Accepted Solution

by:
virmaior earned 300 total points
ID: 37761954
If PHP runs out of memory, it crashes completely -- not partial runs.

There's no specific danger by storing things in a PHP array. By that, I mean that the only danger is if the physical hardware is so unstable that it normally fritzes and loses things.

In other words, the problem is located elsewhere.  Somewhere you've got a bad assumption running around about what's going to last until the previous step is done or a logic error that can muck things up.
0
 

Author Closing Comment

by:melwong
ID: 37761962
thx
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This post looks at MongoDB and MySQL, and covers high-level MongoDB strengths, weaknesses, features, and uses from the perspective of an SQL user.
In this blog, we’ll look at how improvements to Percona XtraDB Cluster improved IST performance.
The viewer will learn how to count occurrences of each item in an array.
In this video, Percona Solution Engineer Rick Golba discuss how (and why) you implement high availability in a database environment. To discuss how Percona Consulting can help with your design and architecture needs for your database and infrastr…
Suggested Courses

609 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question