Unfortunately, automated tests in general have a downside especially when a project matures. The number of tests increase causing the time it takes to run the test suite to increase. This either causes the test suites to become a bottleneck in the development cycle, or people just run them less frequently. Running tests less frequently has a compounding effect because tests start going stale, which increases maintenance cost on the tests, which causes them again to be run less frequently. Unfortunately, Selenium has a handicap in this realm because it runs within the actual web browser which inherently adds time to each test.
Fortunately, automated tests are a good candidate for parallel execution even though they are generally executed serially. This is great news because we can significantly reduce the time a test suite takes to execute by distributing it across a mini-cluster of computers. Both JUnit and TestNG have projects for distributing their execution, and there is the Selenium Grid project that will distribute Selenium tests. Basically the goal of the project is to build a small test cluster where Selenium tests can be farmed out onto different machines to greatly reduce the time it takes to execute the full test suite. It is definitely a good idea, and it will address the problem of stale tests and their associated maintenance cost. With it taking less time to run the suite, we can easily add it back into a CI system and run the tests more frequently.
Unfortunately, as I mentioned in my post on Elastic Build Systems, once we start to distribute our environments we open up a host of other problems. Namely, ensuring that all the machines in the cluster have been updated with the same version of Selenium, and Selenium Grid, that the browser versions being tested are properly synchronised, and in general be confident that the machines are identical: a test running on one machine is guaranteed to give the same results when executed on another machine. And of course the flip side of having tests run faster by distributing it across multiple machines means that there are more machines running idle for longer. Finally with a special consideration for Selenium it might be desirable to rerun the test suite across multiple browsers to regression test what is supported. With this in mind, your test cluster could start growing to include specialised Windows machines running IE7 and IE8, Linux machines running FireFox, etc. to ensure that you get good coverage, in addition to a speed boost. Of course, when adding more machines, the complexity of managing the cluster also increases, as well as the wasted resources of idling machines.
Fortunately, GridCentic's Copper high-performance virtualization platform makes managing a computer cluster extremely easy and allows a cluster to be re-purposed within seconds.
At the core of the Copper platform is the ability of virtual machines to live-clone themselves. In other words, Copper allows running virtual machines to instantly create multiple copies of themselves where each copy is an exact replica of the original machine, from the memory that has been loaded down to the instruction pointer in the CPU. This whole process of taking a single running virtual machine to saturating the physical limitation of your hardware with cloned virtual machines happens within seconds. In addition, the Copper platform also takes care of networking all of these virtual machines so that they can communicate with each other, and if desired they can also communicate with the rest of the network.
Lets suppose for example that we manage to scrounge together a little test cluster of 4 machines that we can use for running Selenium Grid to optimize our Selenium test runs. Ideally we would like to ensure that our web platform tests run using our supported configurations: Windows running IE8, OS X running Safari, and Linux running Firefox. Traditionally, we would divide up our cluster for these configurations with a single machine per configuration and one spare machine we could give to a single configuration. We have just lost the benefits of Selenium Grid and we have an extra computer that cannot be shared.
However, we can make use of Copper's ability to dynamically and quickly re-purpose an entire cluster. Each testing environment (operating system, browser, tools and Selenium Grid) will be encapsulated in its own virtual machine that will run on top of the little test cluster. When it becomes time to run the tests, the virtual machine will scale itself out using the fast live-cloning provided by Copper and then run Selenium Grid as normal. Once it has finished executing the tests, it will clean up its clone machines and scale back down freeing up the cluster resources. Then another test environment can scale out onto the recently freed resources, run the tests, and again scale back down. This can be done for each environment.
Moreover, supporting a new test environment is nothing more that creating a virtual machine for it and configuring the machine by installing the test environment software. This new configuration can now scale up, run tests using Selenium Grid, and scale back down.
In a previous post I showed how Hudson can be easily transformed into an Elastic Build System. Basically, it allowed Hudson to dynamically create new slave machines that are identical to the master machine and then farm out build jobs to them. Hudson could also destroy the slaves when they were no longer needed to free up resources. Now let's combine everything together to show how we can build an ultimate continuous integration environment using the Copper platform.
There will be a single virtual machine with Hudson installed that will be our master machine. We will continue to have a virtual machine per test environment that we want to support, and we'll configure Hudson to treat each of these machines as a slave. We'll then configure some Hudson jobs that will run on our test environment slaves that will basically clone out the test environment, start up Selenium Grid, run the tests and then scale back down. We'll also configure Hudson to have a couple of elastic slaves as described in a previous post. So now we have it. A developer commits some change, Hudson farms the build out to a dynamically created slave. Once the build is completed, it can trigger the Selenium Grid testing agents in turn so that they will scale out, run the regression integration tests, scale back down and produce a test report of any possible regression issues.
Amazing.
0 comments:
Post a Comment