Passing off large script processing to another script

Most serious developers often face larger projects where user forms take a long time to process, leaving the user wondering what is happening and often clicking the submit button more than once. This article is not to show you how to disable the submit button once clicked or to show a spinner or progress bar. I will show you how to let another script do all the work while your form gets an almost instant response back from the server, allowing the user to continue interacting with your website. I will also show you how to have your form "checkup" on the external process so you can show a real time progress bar (or spinner).

If you have ever had a form that takes more than a few seconds to process then this article is for you. Although I am focusing on PHP, the general idea should work for any other scripting language.

Recently I had to write a script that took a rather large user form and processed it, interacting with a database and calling an outside public API to register items on another website as well as on the local website local website meaning your online (www.yoursite.com) website. Unfortunately the public API of the company in question was very slow to respond. Asking them if they could speed up their API got no response so I came up with a better way not to leave my users hanging.

This process will work whether you are using a public API on another website or doing all local database stuff. Here is what I did.

Initially my user form submitted all the form data to a processing script which took care of all the local database stuff and did the 60 plus, API calls, and dealt with the responses. If it made it all the way to the end it would send a response back to the user form with an error message number (0 for success).

From the time the user clicked the submit button to the time they got notification of success, they had been waiting just over 1 minute.  This was unacceptable to me and simply disabling the submit button and showing a spinner was not good enough.

HOW IT WORKS:

Nothing in your form changes. The change happens in the processing script. The first part of the script was to register the new user on the websites database (our website not the public API's website). This was done in the usual way making SQL insert statements and entering in the users details. This script then needs to get the new userid and send it back to the form. You will need this userid in order to call the real processing script.

So now you insert the user sent data into the database and retrieve the new userid.

As soon as you get a userid returned you need to create a master results array to store all the information. So the master_results looked something like this to start with.

<?php
                      
                      // Current Step variable - available to all functions/methods
                      $current_step = 1;
                      
                      // Master results array needs to be available to all public and private functions/methods
                      
                      $master_results = array(
                      'total_steps' => 20,
                      'current_step' => $current_step,
                      'userid' => $userid
                      );
                      
                      // Remember we already have the new userid.
                      
                      // Json encode the results.
                      $json = json_encode($master_result<wbr ></wbr>s);
                      
                      // Save the results string into your database so other functions/methods can retrieve and update it.
                      
                      mysql_query("INSERT INTO user_data (data) VALUES ('$json')") or die(mysql_error());
                      
                      // I also updated a field I labeled 'updatedon' with the current date/time so I could use that later for other functions but this is not required.
                      ?>

Select all Open in new window

PASSING OFF:
The final step to do is to PASS OFF the rest of the work to another script to finish off the job. To call another script is as simple as doing a shell command passing the path, filename and any arguments you need to send. In my case, I only needed the userid so here is my call.

To learn more about the exec() command go here exec()

Assuming the process.php file is in the same folder as the current script. If not, then add the path to the call.

$pid = exec('php process.php '.$userid.' > /dev/null 2>&1 & echo $!');
                      
                      if (!$pid) {
                          // error here
                      } else {
                          // We are done so this script should return the userid so we can use it for "CHECKUP"
                          echo $userid;
                      }

Select all Open in new window

CHECKUP:

The checkup is a simple process. The only thing you need to pass to it is the userid which was created in step 1 in my case. The first job to do is get the user data information (json string) and decode it back into an array so you can checkup on the progress.

<?php // checkup.php
                      // Get the user data
                      
                      $json = mysql_query("SELECT data FROM user_data WHERE userid = '$userid' LIMIT 1") or die(mysql_error());
                      
                      if (!$json) {
                           // return an error code here
                           echo 1;
                      }
                      
                      // We got the results so decode them into an array.
                      
                      $master_results = json_decode($json, true);
                      // If you add the 'true' argument you will get an array instead of an object returned.
                      
                      // Get the step information from the results array and do some simple math to determine how far the script has gotten.
                      
                      // Total steps
                      if (!isset($master_results['total_steps'])) {
                            // error here
                      } else { 
                          $total_steps = $master_results['total_steps'];
                      }
                      
                      // Current step
                      if (!isset($master_results['current_step'])) {
                            // error here
                      } else { 
                          $current_step = $master_results['current_step'];
                      }
                      
                      // NOTE: you don't have to assign these values to a variable if you only need to use them once. You can just operate directly with the master_results['total_steps'] etc...
                      
                      // Do the math
                      $percent_complete = $current_step / $total_steps;
                      
                      // Return the result
                      echo round($percent_complete);
                      
                      // you could round this number if you prefer or do it in javascript it really doesn't matter. I usually round it
                      ?>

Select all Open in new window

So ajax will call "checkup.php?userid=" and the userid returned from the first form submit and it will get back the percent complete of the entire process.

LAST STEP:

This step is the process.php where all the work is done in the background (independent of your user form or the initial script which has now completed and returned the userid).

It is important to note that in order to do a checkup of progress, each step involved in the processing script needs to update the database with it's current step number as we did in the initial script. Firstly though, we need to get the results back from the database and decode them just like we did in CHECKUP so we have the current step number. I won't bother repeating the code as it is already written above.

This process script will be called through the "exec()" command in the previous script and it is passed at least 1 argument. In my case it was the userid. So we need to retrieve it inside the script. For this we use $argv which is a built in PHP array similar to $_POST and $_GET. You can read up on $argv HERE

To get the first argument we passed to this script it will always reside in $agv[1];
$argv[0] holds the script name (process.php);

$userid = $argv[1]; // There it is.

So now you can retrieve the users data from the database as we have the userid.

All your functions/methods will go in here and after each stage/step is complete, you must increment $current_step and place it back into the $master_results array and re-encode it to a json string, then save it back to the database.

In my case, I had some 40 private methods and handful of public methods that took care of all the external API calls, each one returning an id for that item, which I would record back into my database via the $master_results array.

It is important to note here that this process.php must not return anything to the screen as no one is watching it run. No echo or print statements are required. Just get the job done and if an error occurs, error_log('Place your error notice in here so you can view it later') or something along those lines. This script will get to the end (if no errors along the way) and just finish with no output.

So that is it in a nutshell.

SUMMARY:

This article is designed to show you how to pass off a slow running form due to the amount of work it is doing, to another script which does all the work. It also shows you how to do a checkup on that process in case you want to have a progress bar or similar on your webpage. If the progress bar or checkup is not required, there is no need to record the steps into the database so you can ignore those parts. The main thing is to know how to pass off or call another script from the main script and that is done with exec() command in PHP. In other scripting languages there will be a similar shell command to call files but that is beyond the scope of this article.

I hope this will prove to be helpful to the many developers who find themselves in this predicament. I will be happy to elaborate on any part of this if you don't understand it.

Regards
Mark (elvin66)