Managing a high volume Web site has a unique challenge when you consider all the moving parts. Our latest problem turned out to be related to this:
Performance Hit Since PHP 4.3.10 / 5.0.3
Many people have noticed that vBulletin (and a lot of other PHP applications) suddenly started to run significantly slowed than normal after installing PHP 4.3.10 or 5.0.3 in order to patch the security flaw in previous versions of PHP.
This cause of this slow-down has been identified as a problem with the unserialize() function in PHP. For more details, see bugs.php.net.
This problem has now been fixed by the PHP developers, though the fixed version has yet to be released in a 'stable' version. However, the latest CVS snapshots of PHP 4.3.x and 5.0.x, available from snaps.php.net contain the fix and restore the original speed of unserialize().
While we would not recommend running a 'dev' version of PHP on any production server, we understand that the performance problem has been a major issue for some people. If you are badly affected, you may want to consider running a 'dev' version of PHP at your own risk in order to overcome the performance problem.
As it turns out, when we moved to this latest version of PHP to fix a potential security flaw, it caused the slow down (and as noted, not just for our Web site). We are currently awaiting the official release fo the new version with the fix not opting to go with the 'dev' version because there are risks associated with that and we really are not interested in becoming a statistic. Once this fix is released, we'll install, test and reopen search. etc.
Thanks for being patient as well as understanding there are a lot of moving parts on this Web site and we're not just talking about our members. HA!