Browser processing time

Posted on 2011-10-31
Last Modified: 2012-05-12
I am creating an application that displays 45,000 rows of data or more in a report table. The actual pulling of the data from the database and processing of that data takes a few seconds, but displaying the data by means of an output string takes between 3 - 6 minutes and I can't figure out why it is taking so long. To debug it I created a debug script that displays the time at critical points along the way. I get slightly different results between Firefox and IE. Below is the debug display I get from Firefox:

1. The region is: *******
2. The region is: *******
3. The region id is: 72
The div id is: 22
Before getting the data from db: : 03:22:51
Number of rows gotten is 44395
Before setting title: : 03:22:53
Before report heading: : 03:22:53
Before report body: : 03:22:53
0. Start of row: : 03:22:53
4000. Start of row: : 03:22:53
8000. Start of row: : 03:22:54
12000. Start of row: : 03:22:54
16000. Start of row: : 03:22:54
20000. Start of row: : 03:22:55
24000. Start of row: : 03:22:55
28000. Start of row: : 03:22:55
32000. Start of row: : 03:22:56
36000. Start of row: : 03:22:56
40000. Start of row: : 03:22:57
44000. Start of row: : 03:22:57
After report body: : 03:22:57
After displaying report: 03:29:10

Here is the debug display I get from IE:

1. The region is: *******
2. The region is: *******
3. The region id is: 72
The div id is: 22
Before getting the data from db: : 04:29:42
Number of rows gotten is 44395
Before setting title: : 04:29:44
Before report heading: : 04:29:44
Before report body: : 04:29:44
0. Start of row: : 04:29:44
4000. Start of row: : 04:29:45
8000. Start of row: : 04:29:45
12000. Start of row: : 04:29:45
16000. Start of row: : 04:29:46
20000. Start of row: : 04:29:46
24000. Start of row: : 04:29:46
28000. Start of row: : 04:29:47
32000. Start of row: : 04:29:47
36000. Start of row: : 04:29:48
40000. Start of row: : 04:29:48
44000. Start of row: : 04:29:48
After report body: : 04:29:48
After displaying report: 04:32:37
(Note: browser continued processing and wasn't complete with loading for another 4 minutes)

The Script that handles this creates an array of the data pulled from the database and then from that creates the display string. I thought maybe the array of data was taking up too much memory, so I unset that after creating the string, and that didn't change the results at all. Below I'm including 3 methods in my class (PHP) that handle this:
protected function format_for_null($item){
		if($item == '' || $item == NULL){
			$item = " ";
		return $item;
	protected function report_body(){
		$output = "<div id='cmts_plan_body'><table id='cmts_plan_body_table'>";
		$i = 0;
		foreach ($this->cell_vals as $row){
			if($i%4000 == 0 ){ //DEBUG CODE
				$this->display_time($i .'. Start of row: ');//DEBUG CODE
			$output .= "<tr>";
			if ($row[0] != ''){
				$dDate = $this->convert_date_for_form($row[0]);
				$dDate = '&nbsp;';
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthC'>".$this->format_for_null($row[1])."</td>";	
			$output .= "<td class='widthC'>".$this->format_for_null($row[2])."</td>";
			$output .= "<td class='widthC'>".$this->format_for_null($row[3])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[4])."</td>";
			$output .= "<td class='widthE'>".$this->format_for_null($row[5])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[6])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[7])."</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[8])."</td>";
			$cClass = $this->set_background_color($row[9]);
			$output .= "<td class='".$cClass."'>$row[9]</td>";	
			$output .= "<td class='widthB'>".$this->format_for_null($row[10])."</td>";	
			$output .= "<td class='widthC'>".$this->format_for_null($row[11])."</td>";
			$output .= "<td class='widthC'>".$this->format_for_null($row[12])."</td>";
			$cClass = $this->set_background_color($row[13]);
			$output .= "<td class='".$cClass."'>$row[13]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[14])."</td>";
			$cClass = $this->set_background_color($row[15]);
			$output .= "<td class='".$cClass."'>$row[15]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[16])."</td>";
			$cClass = $this->set_background_color($row[17]);
			$output .= "<td class='".$cClass."'>$row[17]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[18])."</td>";
			$cClass = $this->set_background_color($row[19]);
			$output .= "<td class='".$cClass."'>$row[19]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[20])."</td>";
			$cClass = $this->set_background_color($row[21]);
			$output .= "<td class='".$cClass."'>$row[21]</td>";
			$output .= "<td class='widthB'>".$this->format_for_null($row[22])."</td>";
			if ($row[23] != ''){
				$dDate = $this->convert_date_for_form($row[23]);
				$dDate = '&nbsp;';
			$output .= "<td class='widthA'>".$dDate."</td>";
			if ($row[24] != ''){
				$dDate = $this->convert_date_for_form($row[24]);
				$dDate = '&nbsp;';
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthE'>".$this->format_for_null($row[25])."</td>";
			if ($row[26] != ''){
				$dDate = $this->convert_date_for_form($row[26]);
				$dDate = '&nbsp;';
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthE'>".$this->format_for_null($row[27])."</td>";		
			$output .= "<td class='widthA'>".$this->format_for_null($row[28])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[29])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[30])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[31])."</td>";
			if ($row[32] != ''){
				$dDate = $this->convert_date_for_form($row[32]);
				$dDate = '&nbsp;';
			$output .= "<td class='widthA'>".$dDate."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[33])."</td>";
			$output .= "<td class='widthF'>".$this->format_for_null($row[34])."</td>";
			$output .= "<td class='widthA'>".$this->format_for_null($row[35])."</td>";
			$output .= "</tr>";
		$output .= "</table></div>";
		unset ($this->cell_vals);
		return $output;		
	public function make_augment_plan(){
		$this->debug_echo .= "The div id is: " . $this->div_id . "<br>";
		$this->display_time('Before getting the data from db: ');
		$this->display_time('Before setting title: ');
		$output = $this->report_title();
		$output .= "<div id='main_display'>";
		$this->display_time('Before report heading: ');
		$output .= $this->report_heading();
		$this->display_time('Before report body: ');	
		$output .= $this->report_body();
		$this->display_time('After report body: ');
		$output .= "</div> <!-- End Main Display -->";		
		return $output;

Open in new window

It would be great to know why the browsers take so long to display the output string and then why they take so long even after the data is displayed to complete the loading process and to know what I could do about that.

Question by:denewey
    LVL 82

    Accepted Solution

    Other than the fact that you are trying to display more than any person can go thru, I can think of a number of reasons.  Number one is allocating and re-allocating more memory as the browser tries to construct the table image.  If the browser runs out of memory, then you get to swapping to virtual memory on disk and that is a comparativley slow process.  Number two is that browsers often will not display a table until it has ALL been read and 'created' in the browser's internal structure.  Browsers do not create the display directly from the HTML code.  They first put it into an internal organization (the DOM - Document Object Model) and then they use that to display the page on the screen.

    Your times are misleading you also.  You're getting the time when that statement is encountered in PHP.  That doesn't mean that the data has been sent to the browser at that point.  Both PHP and the web server will collect the info in buffers and send the data when there is enough to send or sometimes, not until it has all been collected and then the whole page gets sent at once.

    A more reasonable test of the performance of the browsers is to send smaller numbers of rows and see what happens.  I would start with 20 rows, then 100, then 1000 and see how it goes.  You may find that you want to 'paginate' the rows on screenfull at a time if the page is for humans to read.
    LVL 53

    Expert Comment

    I'm curious, so I have to ask.  Why would ever want to 45,000 rows on a web page?  That is equal to a 700-800 page report.  How can that be of use to a user?

    Author Comment

    COBOLdinosaur - It's what the user requested.
    DaveBaldwin - I'll get back to you tomorrow when I get back to work - that's the angle I'm thinking with.
    LVL 5

    Expert Comment

    Your code doesnt output anything to the browsers. Its just building the $output variable. That means the actualy echo command is given after the 3-4 minutes time. This makes it clear that browser is not the culprit till this time.

    Now its very clear that this 3 min time is taken is constructing the $output variable for 45k rows. One possible reason i see is that the everything is going into a variable, so it might be taking alot of memory and the memory on server is limited and it has to wait for memory to free up.

    To be sure of this you can do this:

    1. Output the times at each stages of the $this->report_body(); function, specially before and after distinct functions. You might get if there is any particular function that is taking more than expected time. You might need to display microtime() because each row is taking 0.250 second only

    2. Check your memory and increase if required.

    3. You can try to output the data or store in a file, instead of building one huge variable in memory, and passing it back and forth.


    Author Closing Comment

    I was able to get the requirements for the report cut down to about 10,000 rows. Even still that was unmanageable so I went to a pagination scheme and that has handled any loading difficulties I was having.
    LVL 82

    Expert Comment

    by:Dave Baldwin
    Good, I thought that would be the way to go.  Thanks for the points.

    Author Comment

    You're welcome. You gave a concise but complete explanation of what I was dealing with which gave me the information I need to decide which way to go. Thanks
    LVL 82

    Expert Comment

    by:Dave Baldwin
    You're welcome.

    Featured Post

    6 Surprising Benefits of Threat Intelligence

    All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

    Join & Write a Comment

    It's here again; Microsoft is launching a new version of Internet Explorer: Internet Explorer 9, with noticeable changes on its interface, functions and new tools. As they say on its promotional video: "It's time to play, on a more beautiful web", f…
    Internet is a big network which is formed by connecting multiple small networks.It is a platform for all the users which are connected to it.Internet act as platform in different fields. Such as: Internet  as a collaboration platform. Internet  as…
    Google currently has a new report that is in beta and coming soon to Webmaster Tool accounts. This Micro Tutorial will highlight new features for Google Webmaster Tools.
    This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.

    745 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now