• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 195
  • Last Modified:

Browser processing time

I am creating an application that displays 45,000 rows of data or more in a report table. The actual pulling of the data from the database and processing of that data takes a few seconds, but displaying the data by means of an output string takes between 3 - 6 minutes and I can't figure out why it is taking so long. To debug it I created a debug script that displays the time at critical points along the way. I get slightly different results between Firefox and IE. Below is the debug display I get from Firefox:

1. The region is: *******
2. The region is: *******
3. The region id is: 72
The div id is: 22
Before getting the data from db: : 03:22:51
Number of rows gotten is 44395
Before setting title: : 03:22:53
Before report heading: : 03:22:53
Before report body: : 03:22:53
0. Start of row: : 03:22:53
4000. Start of row: : 03:22:53
8000. Start of row: : 03:22:54
12000. Start of row: : 03:22:54
16000. Start of row: : 03:22:54
20000. Start of row: : 03:22:55
24000. Start of row: : 03:22:55
28000. Start of row: : 03:22:55
32000. Start of row: : 03:22:56
36000. Start of row: : 03:22:56
40000. Start of row: : 03:22:57
44000. Start of row: : 03:22:57
After report body: : 03:22:57
After displaying report: 03:29:10

Here is the debug display I get from IE:

1. The region is: *******
2. The region is: *******
3. The region id is: 72
The div id is: 22
Before getting the data from db: : 04:29:42
Number of rows gotten is 44395
Before setting title: : 04:29:44
Before report heading: : 04:29:44
Before report body: : 04:29:44
0. Start of row: : 04:29:44
4000. Start of row: : 04:29:45
8000. Start of row: : 04:29:45
12000. Start of row: : 04:29:45
16000. Start of row: : 04:29:46
20000. Start of row: : 04:29:46
24000. Start of row: : 04:29:46
28000. Start of row: : 04:29:47
32000. Start of row: : 04:29:47
36000. Start of row: : 04:29:48
40000. Start of row: : 04:29:48
44000. Start of row: : 04:29:48
After report body: : 04:29:48
After displaying report: 04:32:37
(Note: browser continued processing and wasn't complete with loading for another 4 minutes)

The Script that handles this creates an array of the data pulled from the database and then from that creates the display string. I thought maybe the array of data was taking up too much memory, so I unset that after creating the string, and that didn't change the results at all. Below I'm including 3 methods in my class (PHP) that handle this:
protected function format_for_null($item){
		if($item == '' || $item == NULL){
			$item = " ";
			}
		return $item;
	}
	
	protected function report_body(){
		$this->get_color_set();	
		$output = "<div id='cmts_plan_body'><table id='cmts_plan_body_table'>";
		$i = 0;
		foreach ($this->cell_vals as $row){
			if($i%4000 == 0 ){ //DEBUG CODE
				$this->display_time($i .'. Start of row: ');//DEBUG CODE
			}
			$output .= "<tr>";
			if ($row[0] != ''){
				$dDate = $this->convert_date_for_form($row[0]);
			}
			else{
				$dDate = '&nbsp;';
			}
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthC'>".$this->format_for_null($row[1])."</td>";	
			$output .= "<td class='widthC'>".$this->format_for_null($row[2])."</td>";
			$output .= "<td class='widthC'>".$this->format_for_null($row[3])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[4])."</td>";
			$output .= "<td class='widthE'>".$this->format_for_null($row[5])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[6])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[7])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[8])."</td>";
			$cClass = $this->set_background_color($row[9]);
			$output .= "<td class='".$cClass."'>$row[9]</td>";	
			$output .= "<td class='widthB'>".$this->format_for_null($row[10])."</td>";	
			$output .= "<td class='widthC'>".$this->format_for_null($row[11])."</td>";
			$output .= "<td class='widthC'>".$this->format_for_null($row[12])."</td>";
			$cClass = $this->set_background_color($row[13]);
			$output .= "<td class='".$cClass."'>$row[13]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[14])."</td>";
			$cClass = $this->set_background_color($row[15]);
			$output .= "<td class='".$cClass."'>$row[15]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[16])."</td>";
			$cClass = $this->set_background_color($row[17]);
			$output .= "<td class='".$cClass."'>$row[17]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[18])."</td>";
			$cClass = $this->set_background_color($row[19]);
			$output .= "<td class='".$cClass."'>$row[19]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[20])."</td>";
			$cClass = $this->set_background_color($row[21]);
			$output .= "<td class='".$cClass."'>$row[21]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[22])."</td>";
			if ($row[23] != ''){
				$dDate = $this->convert_date_for_form($row[23]);
			}
			else{
				$dDate = '&nbsp;';
			}
			$output .= "<td class='widthA'>".$dDate."</td>";
			if ($row[24] != ''){
				$dDate = $this->convert_date_for_form($row[24]);
			}
			else{
				$dDate = '&nbsp;';
			}
			
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthE'>".$this->format_for_null($row[25])."</td>";
			if ($row[26] != ''){
				$dDate = $this->convert_date_for_form($row[26]);
			}
			else{
				$dDate = '&nbsp;';
			}
			
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthE'>".$this->format_for_null($row[27])."</td>";		
			$output .= "<td class='widthA'>".$this->format_for_null($row[28])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[29])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[30])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[31])."</td>";
			if ($row[32] != ''){
				$dDate = $this->convert_date_for_form($row[32]);
			}
			else{
				$dDate = '&nbsp;';
			}		
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[33])."</td>";
			$output .= "<td class='widthF'>".$this->format_for_null($row[34])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[35])."</td>";
			$output .= "</tr>";
			$i++;
		}
		$output .= "</table></div>";
		unset ($this->cell_vals);
		return $output;		
	}
	
	public function make_augment_plan(){
		$this->debug_echo .= "The div id is: " . $this->div_id . "<br>";
		$this->display_time('Before getting the data from db: ');
		$this->get_report_fields();
		$this->display_time('Before setting title: ');
		$output = $this->report_title();
		$output .= "<div id='main_display'>";
		$this->display_time('Before report heading: ');
		$output .= $this->report_heading();
		$this->display_time('Before report body: ');	
		$output .= $this->report_body();
		$this->display_time('After report body: ');
		$output .= "</div> <!-- End Main Display -->";		
		return $output;
	}

Open in new window


It would be great to know why the browsers take so long to display the output string and then why they take so long even after the data is displayed to complete the loading process and to know what I could do about that.

Thanks.
0
denewey
Asked:
denewey
1 Solution
 
Dave BaldwinFixer of ProblemsCommented:
Other than the fact that you are trying to display more than any person can go thru, I can think of a number of reasons.  Number one is allocating and re-allocating more memory as the browser tries to construct the table image.  If the browser runs out of memory, then you get to swapping to virtual memory on disk and that is a comparativley slow process.  Number two is that browsers often will not display a table until it has ALL been read and 'created' in the browser's internal structure.  Browsers do not create the display directly from the HTML code.  They first put it into an internal organization (the DOM - Document Object Model) and then they use that to display the page on the screen.

Your times are misleading you also.  You're getting the time when that statement is encountered in PHP.  That doesn't mean that the data has been sent to the browser at that point.  Both PHP and the web server will collect the info in buffers and send the data when there is enough to send or sometimes, not until it has all been collected and then the whole page gets sent at once.

A more reasonable test of the performance of the browsers is to send smaller numbers of rows and see what happens.  I would start with 20 rows, then 100, then 1000 and see how it goes.  You may find that you want to 'paginate' the rows on screenfull at a time if the page is for humans to read.
0
 
COBOLdinosaurCommented:
I'm curious, so I have to ask.  Why would ever want to 45,000 rows on a web page?  That is equal to a 700-800 page report.  How can that be of use to a user?
0
 
deneweyAuthor Commented:
COBOLdinosaur - It's what the user requested.
DaveBaldwin - I'll get back to you tomorrow when I get back to work - that's the angle I'm thinking with.
0
New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

 
liveaspankajCommented:
Your code doesnt output anything to the browsers. Its just building the $output variable. That means the actualy echo command is given after the 3-4 minutes time. This makes it clear that browser is not the culprit till this time.


Now its very clear that this 3 min time is taken is constructing the $output variable for 45k rows. One possible reason i see is that the everything is going into a variable, so it might be taking alot of memory and the memory on server is limited and it has to wait for memory to free up.

To be sure of this you can do this:

1. Output the times at each stages of the $this->report_body(); function, specially before and after distinct functions. You might get if there is any particular function that is taking more than expected time. You might need to display microtime() because each row is taking 0.250 second only

2. Check your memory and increase if required.

3. You can try to output the data or store in a file, instead of building one huge variable in memory, and passing it back and forth.







0
 
deneweyAuthor Commented:
I was able to get the requirements for the report cut down to about 10,000 rows. Even still that was unmanageable so I went to a pagination scheme and that has handled any loading difficulties I was having.
0
 
Dave BaldwinFixer of ProblemsCommented:
Good, I thought that would be the way to go.  Thanks for the points.
0
 
deneweyAuthor Commented:
You're welcome. You gave a concise but complete explanation of what I was dealing with which gave me the information I need to decide which way to go. Thanks
0
 
Dave BaldwinFixer of ProblemsCommented:
You're welcome.
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now