Part of the goal of Flourish is to provide a solid library of code to build web apps and websites on top of. Since PHP runs on such a variety of operating systems and many users do not control the server they are installed on, operating system and PHP version compatibility is important. Thanks to recent donations by iMarc and Luke Foreman, the testing setup has been expanded to include OS X 10.6 and Windows Server 2008.
With these changes, over 60,000 tests are run with every SVN commit, on a total of 10 different versions of PHP, running on 12 different server environments. All database-related tests are run on the six supported databases, and most are run using each of the 14 supported database extensions against eight different database versions. In situations where Flourish provides a pure-PHP fallback implementation of a non-standard PHP extension, the tests are run both with the non-standard extension loaded and not.
At the end of the day, this helps to ensure that Flourish will work if you deploy it on a RHEL 4 server running PHP 5.1.6, a Windows Server 2008 machine running PHP 5.3.3 or a Solaris box with PHP 5.2.9.
Throwing around numbers is great and all, but I'd like to touch on the setup a little bit. Flourish uses PHPUnit for unit and integration tests. And yes, many of the test aren't true unit tests. Since portability and compatibility are key, I want all of the code to actually run. If I were to have written pure unit tests, I would have mocked all sorts of dependencies and missed quite a number of bugs in PHP and bizarre edge cases.
PHPUnit contains an awesome amount of functionality, however when I started with it, it did not allow for running tests with different PHP configuration. I wouldn't be surprised if Sebastian has added the functionality in 3.4 or 3.5, but I started this whole process back with
parses class config files and runs the tests once for each different configuration. It has evolved over time, and isn't the prettiest thing I've ever written, but it gets the job done.
tests.php includes some other functionality, including a column-based, color-coded shell output for interactive testing and JSON output for automated testing. It allows for configuration values to be passed at run time (through a nasty PHP ini hack) so that passwords don't need to be hard coded in the tests and inadvertantly leaked through a source code repositories. In addition, you can included or exclude specific classes and filter the configurations to be run.
Once the local test runner once was written, I went to work setting up a bunch of virtual machines. There are a total of 12 VMs running on two different physical machines, an Intel i7 quad-core with 8GB of ram and a dual-core iMac with 3GB ram. One of the virtual machines runs DB2, MySQL, Oracle and PostgreSQL, while another runs SQL Server 2005 and 2008. Every machine, including Windows, is running OpenSSH, allowing for key-based authentication and distribution of the tests. I leave the stock OS configuration alone as much as possible, but install PHP, various extensions, PHPUnit, some command line database programs and configure the various local sendmail implementations to route through an SMTP relay. Where possible I use the OS-supplied PHP packages in an effort to find bugs related to common environments.
To run the rests on the virtual machines, I use a custom bash script
scps a tarball to each machine and then invokes
remote_tests.sh allows for parallel execution of the test scripts for the different database types, however I found that performance decreased due to disk thrashing. It also allows checking the remote server configuration to show the installed PHP version and the install status of both required and optional PHP extensions.
I haven't spent much time abstracting or refining this setup for use by others, but I could see it being useful for library and framework maintainers. You can check out the code at http://github.com/flourishlib/flourish-tests.