Advice on PHP 5 Accelerators and Zend Debugger

First thing to think of is to use Zend products both for development/debug and for cache. Even Zend Optimizer should do the trick. Caches are good nowadays so you won't notice the difference between.

But if you think about speed up your application maybe it would be better to make some application level-cache so you: codC won't redo exact the same things every time. As you know what your code should do you cat tell where the cache should be placed.

You may also consider to use different web server (say nginx or lighthttpd to serve at least static content of your site (and try to convert dynamic pages into static ones, say by rendering news page into html once a minute instead of generate it on the fly every time, and serve that static page with nginx).

I read that zend optimizer wasn't an opcode cache.

Yes you're right but as of my experience I faced segmentation faults with both APC and XCache, but it was a bit ago so now these caches may be more stable. This was the reason I choose Zend Optimizer, as my script was too heavy and Zend was good on it. Lucky me and no seg. faults. But I'd say my setup was some LAN site so I could afford not to use op-cache.

I'd vote for XCache just because of its developer, who also behind Lighthttpd. I'd also try to test your code both under XCache and APC, and choose which one gives better results on _your_ code. But of course kind of watchdog should be set up to check if cache goes bad. You might tail Apache logs for 'exit signal Segmentation fault' lines and reboot server at that time.

Well the application is on the internet, and the server is doing other work as well.
So I need to look at ways of reducing CPU load.
I can't generate pages as static pages because it's a secure / members only type situation.
And at the very least the authorisation and tracking parts of the code need to run for each request.

And I figure in the future if we start to use AJAX then the problem of CPU usage on the server will be even worse.

Any ideas with Xcache and Zend remote debugger ?

First, there are a lot of content that really static (images, CSS). And if you think about you can rewrite some code so it will be built of some static (generic) parts. You can even make index.html static instead of dynamic index.php, but in the beginning just include javascript (which will be generated by script) that will contain say username and some per-user settings, and javascript on the index.html page will render personalized page just for the user.

You can even go even further and create per-user dirs under your webroot named after usernames. Put per-user files there and limit access (via HTTP auth) there .httaccess so only user with given username will have access to his personal dir. So yo3 will be able to include per-user info the page by referring to content of such a dir, but only the user will see the content that should be seen only by him. Tricky but saves a lot of web server CPU. Static files may be serverd by lightweight web server and per-user files may be created by backend asynchronously when some user data is changed.

As for opcache and debug I'd set up production server with opcache only and test server only with debug support. You can always turn debug off and turn opcache on on test server but generally there is no point in caching on test server where there are only a few users. Don't even bother with opcache + debug, this is what I'd say.

You can also investigate way to use not Apache but ngnix (http://nginx.net/ or wiki at http://wiki.codemongers.com/Main) to serve php content too. This server handles high load very well, but it uses Fast-CGI (only a fewPHP  projects won't support that). Quick search at Google given a lot of info on that. Ngnix cookbook example also exists:

http://wiki.codemongers.com/NginxFcgiExample

Yeah, I think the double edge sword is the heavy use of PHP5 OOP

It does actually run fast enough the problem seems to be actually loading the scripts rather than executing them.
Will also be caching more page components, which presents its own challenge as the application is also multilingual.
All javascript / css / images are already offloaded into a static area.

Leaning towards Xcache because of the amount of statistics plus we can use runkit with it.

Already using Apache / FastCGI not sure if Xcache will share opcode cache across FastCGI threads.

Reason I want to use opcode cache on the development machine is so that I can tweak the application to make it more cacheable.
But don't want to loose remote debug, it's so useful.
Will just have to switch it on  and off or run 2 Apache's talking to the same files just different php.ini's

My other problem I have with the Zend Studio 5.5 is that it has bad UTF8 support.
But I'm trying to keep separation from code and templates.
So using smarty which is great apart from the over head.
Like the principle but may replace it with something a little more light weight.
 

Okay making progress.
Have now got XCache installed seems to run well.
Figured my application wouldn't be very cacheable as I use __autoload() and lots of conditional includes.
But Xcache seems to take it all in it's stride.
Not sure what the performance gain is need to change the application so I can run apache bench against it.

Xcache wont seem to work with Zend Optimizer or Zend Debugger but was expecting that!

Because of the sheer amount of code and the fact that you have stated the loading of scripts seems to take longer than execution, I would suggest that you consider a few things:

1) Are you using a disk subsystem that is bottlenecking your site? A lot of people seem to have forgotten that hard disks are electromechanical devices and as such, are extremely slow relative to the rest of the computer. This is the reason servers have for some time been equipped with expensive SCSI and RAID controllers. A lot of new servers have "cheaped out" and gone with something called "fakeraid" which uses SATA hard drives to make software or hybrid hardware-software "RAID" subsystems which are not up to par with a real SCSI or SAS RAID device.

Anything less than a hardware based true RAID system with 15,000 rpm SCSI or SAS drives is going to slow your system down. In fact, even that will slow it down, just not as much.

Make sure your system drives are on a different RAID from your data drives, as well.

2) Are you falling for the "Mighty CPU" sales tactic? A lot of hardware providers and even VARs are pushing multiple Quad-core CPUs on people who don't realise that the CPU is far from the most taxed component of most servers unless you are engaged in heavy-duty scientific calculations or hypercube data mining. I have an older single-socket dual-core CPU in one of my most heavily used LAMP servers and the CPU in this machine basically sits idle all the time. It rarely goes over 1% utilization and almost never over 30%.

3) How good is your memory? Is it fast enough? Do you have enough of it? I wouldn't build a server with less than 4GB now, even knowing that it isn't all going to be used without a 64-bit OS or one compiled for PAE. RAM is cheap, even the DDR3 kind (I don't think you can get that for a server anyway but I haven't really checked) is relatively cheap for what you get out of it.

4) Are you using your CPU and RAM effectively? If you have a 64-bit CPU, and it's highly unlikely that you don't, and 4GB or more of RAM, you might gain something from using a 64-bit operating system, or alternately using a 32-bit OS that has been compiled for Intel's PAE extensions which allow a machine to fully utilize 4GB or more of RAM, up to even 64GB. Beyond 64GB you will have to go with a full 64-bit OS.

There are also many other things you can do to make your server more responsive, including MySQL configurations (query caching, etc.) and so on. System optimization is a broad subject and goes far beyond simply installing a PHP accelerator, but you get out of it what you put into it in terms of time and money.