Jolt Awards: Testing Tools
Testing tools are like annoying high-school friends. They take an absurd delight in pointing out the shortcomings in your work, but when the chips are down, they're reliably on your side. Today, these big-mouth, good buddies seem to be swarming into every step of the enterprise development process. Whereas user testing and some UATs were all that were principally used in the past, now there are literally dozens of different types of tests, each of which can indeed provide legitimate and useful information to improve the end product. Unit tests, integration tests, functional tests, UATs, loading tests, security intrusion tests, fuzz testing, exploratory testing, model-based testing, and so on. Add to these, techniques such as static code analysis, and one has to wonder whether testing might be an even larger part of the development cycle than coding.
Either way, though, what is clear is that the advent of numerous testing options has made it possible for rank and file developers to turn out far more reliable code than at any other point in history.
While this is great news for end users, it does confront developers and QA teams with the need to choose the appropriate types of tests for their products and the correct tools to run them accurately. The Jolt Awards provide one of the best ways of seeing the top products in testing. In this article, we include the top spot, the Jolt Award winner, followed by two winners of Productivity Awards. A final page lists the top three runners-up. All products were tested and reviewed by our panel of judges, most of whom have judged in this category before. They include: Andrew Binstock, Larry O'Brien, Gary Pollice, Roland Racko, and Mike Riley. The awards themselves are run by Dr. Dobb's, of course, and sponsored by Rackspace (who let us use their cloud for testing and administrative functions), and Safari On-Line Books (who kindly provide free subscriptions to the judges so as to facilitate access to the latest technical writing on development topics.)
And now the envelopes please...
Designed for cross-platform, functional GUI regression testing, Squish sports a lengthy list of features that work equally well across Windows, Linux, OSX, and embeddable systems. These include insertion and verification of points, screenshots, complex data controls, SQL databases and external files, the ability to automatically record test scripts independent of screen coordinates and resolutions (making GUI tests far less brittle), comprehensive logging in XML, HTML, XLS, and plain text, customizable object name generation, and extensive GUI object map generation. It can accept tests written in JavaScript, Perl, Python, Ruby, and Tcl; and generate tests in the same languages. Usability is excellent, as it took me only a few minutes to get oriented and start generating Python and Ruby test scripts for several diverse GUI-based applications.
The product also ships with the full source code, so you'll never have to worry about making an investment in a technology that could go dark if the company goes MIA.
froglogic offers two types of licenses (tester and runner) and license models (named user and group). While pricing is disclosed on their website, the formulas are complex enough that it's hard to know exactly how much a small test organization would pay. Roughly 1000 euros per seat per year is a valid range. The company offers useful screencasts on its site that demonstrate many of Squish's Jolting differentiators.
— Mike Riley
Jolt Productivity Award
AgitarOne, Agitar Technologies
If you're looking for ways to automate generation of your JUnit tests, AgitarOne is an excellent solution. It helps to cultivate a strong TDD orientation in an organization via the generation of JUnit tests. Additionally, the tests' new functional coverage tracker monitors test effectiveness. Couple this with a number of built-in code rules, a comprehensive project dashboard, continuous integration testing, and interactive exploratory testing and you've got quite a package. Agitar then throws in its trademarked "software agitation" (dynamically generated regression test cases for all possible outcomes). This step is probably the coolest and most effective feature I've seen in a JUnit test automation platform, and AgitarOne pulls it off effortlessly.
The server-side component of AgitarOne incorporates the open-source CruiseControl package for continuous integration testing. Combine it with AgitarOne's Maven Antrun plugin for easy AgitarOne-generated script execution and a Hudson CI export wizard plug-in, and the test can be configured as a Hudson job. The AgitarOne server also provides detailed metrics by class, package, and developer to keep management informed of issues and aid with risk prioritization. Check out the screencast demos on Agitar's site to see the product in action.
Agitar has been an aggressive player in unit test generation and management and its new parent, McCabe Software, has taken the original, foundational product and made an excellent, polished, and very useful tool for sites that value complete, top-to-bottom unit testing.
— Mike Riley
Jolt Award For Excellence in Testing Tools
The first sentence on the Selenium website succinctly sums up what the product does: "Selenium automates the web browser. That's it." As a result, a huge ecosystem of extensions and plug-ins have been created to make Selenium the default web application testing tool found in nearly all serious web developers' toolboxes.
Selenium comes in two flavors. The Selenium IDE is a Firefox add-on that macro-records user interactions for playback and analysis. The generated scripts can be edited and replayed for further customization. The second is the more comprehensive Selenium WebDriver, a language-specific set of bindings to automate web browser interactions. The Selenium API supports C#, Java, Perl, Python, and Ruby, as well as specific testing frameworks such as Bromine, Hermes, JUnit, NUnit, RSpec, and Unnittest among others. Given the level of flexible automation that Selenium has to offer, scripting tests of Web-based apps is remarkably easy. Most developers and testers should be able to learn frequently used commands in a few hours.
Selenium can drive both desktop and mobile browsers. It even offers an Android server APK that makes on-device browser testing a snap. An iOS version is also available, but requires it a bit more work to compile into the the RemoteWebDriver class into your iOS project.
If you tried Selenium a few years ago and thought it was a bit rough around the edges or that it lacked a feature you needed, you should do yourself a favor and check out the latest release. Selenium is a mature product with excellent documentation and support, and it benefits from more than 20 plug-ins that can perform everything from page coverage reports to taking screenshots of failed states for further analysis. Users seeking commercial Selenium support have several options, principally from the project's creator and sponsor, Sauce Labs.
All of this technology is available at no cost due to being open source. It's a high-quality package at a price that can't be beat; and for that, it was awarded this year's Jolt Award for Excellence in Testing Tools.
— Mike Riley
The Best of the Rest
As it is every year, several other meritorious nominees almost make it to the podium. These products are useful, important, and should be a part of any evaluation of testing tools for adoption by IT organizations.
TestComplete is a Windows-oriented automated testing tool for creating and executing tests on Windows, rich clients (Flash and Java), and Web applications. Tests can be run either directly or via scripts, the latter of which can be in any of several languages (VBscript, JScript, etc.). It also provides special support for data-driven tests that are powered from text, Excel, or DBMS data sources.
Telerik Test Studio covers the widest range of testing in the passel of products in this article. It covers functional testing (mostly geared towards Web apps), performance testing, load testing, and mobile testing (the latter on iOS only). The suite is not cheap — each element is purchased separately — but put together, they provide one of the most comprehensive suites for testing available today. Telerik products have won Jolt Awards this year that recognize their quality, and these products continue that tradition.
Gorilla Logic's FoneMonkey (recently renamed MonkeyTalk) is the most thorough testing tool available for iPhones. It greatly expands Apple's minimal testing support by being able to script, capture, replay, and test all activity on an iPhone including access to screen elements not normally available to testing tools. It does this by deep intercept of actions and high-resolution recording of movements. Probably the premier tool for testing near-final apps on the iPhone.
— Andrew Binstock
