Technical

Automated Testing Tool Comparisons

Kirsten Van Langenhoven

Automated testing is rapidly gaining popularity across the web development field, and as expected, the number of automated testing tools is growing rapidly as well. These tools can handle one or more types of testing, including visual regression, record and playback, and scripting testing.

To learn about these types of automated tests, please refer to part 1 of this blog post, Automated Testing for the Web.

Below we’ve reviewed and compared a couple of tools for Visual Regression Testing, Record and Playback Testing, and Scripting Testing.

Visual Regression Testing: Percy.io vs. Applitools Eyes

Percy.io

Price: Free - $1,449+

Percy.io is “your all-in-one visual review platform.” It integrates into your pipeline, allowing you to run a test every time you push changes.

This means Percy will take screenshots of your website before and after the changes have been made and compare the images. Then, using their sleek dashboard, you can review the differences and approve the changes before deploying.

Percy uses smart pixel diffing to compare before and after screenshots, then highlights any areas of difference, making it much easier to identify any unwanted changes. Through a “stabilization” process, Percy is smart enough to ignore dynamic elements such as animations and GIFs.

Initial integration can be somewhat difficult as it requires someone with a technical background and a website or app with a proper development workflow.

However, Percy can integrate with a variety of frameworks and once set up, it becomes part of the workflow, helping you ensure that every website update only affects the elements you want.

Percy testing dashboard.

Applitools Eyes

Price: Free - Price is determined on a case by case basis

Applitools takes visual regression testing one step further by integrating AI learning into their product. Different browsers may render a website slightly differently, which could trigger several errors in a standard pixel by pixel visual comparison.

Applitools uses “AI-powered computer-vision algorithms” to simulate the human eye and ignore small, imperceptible differences to avoid potential false-positive results.  

Applitools can also be integrated into many different frameworks and tools. However, it is designed to be an addition to your pre-existing tests, meaning it can’t be used independently. Fortunately, Applitools has an integration with the record and playback tool, Selenium IDE (discussed below), which only takes a few clicks to begin writing your first test.

Applitools testing in action.

Overall Review: Visual Regression Testing

Both tools are powerful additions to any workflow and are user friendly. Applitools Eyes may boast an AI engine to better prevent false-positives, but Percy is not without its own smart algorithms.

The primary determining factors are differences in integration options and pricing. It is crucial to check that your preferred frameworks and tools are supported by either product. Pricing is also dependent on the number of screenshots you take, which can quickly and drastically change your pricing tier.

Record and Playback Testing: Selenium IDE vs. TestCafe

Selenium IDE

Price: Free

Selenium IDE is a free browser extension for Chrome and Firefox which allows you to easily record and create automated tests. Installation is very quick and easy, so you can begin making tests in a matter of minutes.

The UI is clean and relatively simple to understand. It includes a useful reference tab that explains the different types of commands, which is essential for getting started.

One of the handiest features is when recording, you can click on an element on the webpage and it will be populated into the target section. This helps take the guesswork out of finding the appropriate class or ID of the selected element.

However, like any tool, it has its shortcomings. For example, commands are shown in a long dropdown, and while most are easily understood, it can be intimidating and daunting to scroll though at first.

The primary drawback though is its inability to do cross-browser and/or parallel testing without using Selenium’s command-line runner, which drastically increases the learning curve and technical knowledge needed. Overall, Selenium IDE might be a better solution for rapid prototyping, rather than robust testing.

Selenium IDE testing in action.

DevExpress TestCafe Studio

Price: $250 or $500/year

TestCafe Studio is a paid desktop application, that is a more polished version of Selenium IDE with a few extra features.

In addition to its more refined UI, the most prominent difference is the use of icons to visually represent the list of commands, rather than a long list of text.

Like Selenium IDE, you can select and populate a target element’s information while recording, but TestCafe goes one step further and also lists the properties and current values. These types of features help reduce the learning curve and will make it easier to write your tests.

One of the biggest perks of TestCafe is its ability to easily run cross-browser and parallel tests. Although many aspects have been simplified, it is still beneficial to have a basic knowledge of CSS classes and IDs.

TestCafe testing in action.

Overall Review: Record and Playback Testing

As usual, you get what you pay for. Selenium IDE is very powerful, relatively user-friendly, and best of all, free. It is ideal for anyone looking to explore this type of testing tool or for creating tests for more casual projects.

TestCafe is more polished and has a smaller learning curve due to its simplified UI and ability to easily run cross-browser and parallel tests, but at a cost. TestCafe’s features and support make it better suited for a company needing a more formalized tool and process.

Scripting Testing: CrossBrowserTesting vs. BrowserStack

CrossBrowserTesting

Price: $78 - $1,500+/month

CrossBrowserTesting (CBT) is a cloud testing platform that includes a variety of automated and manual testing tools, including visual regression, record and playback, scripting testing, and more.

In terms of its automated testing capabilities, you can import, create, and organize your library of tests, then run them in parallel on real browsers and devices.

Running multiple simultaneous tests, the speed of your automated testing can be greatly increased. In addition to running tests through their web application, you can also use their API to trigger your automated tests.

CBT supports many popular programming languages and has integrations with a variety of testing frameworks and CI/CD tools. This allows for CBT to be easily integrated into your pre-existing workflow.

Of the variety of features, one of the greatest benefits of using a tool like this is that you can leverage its large collection of real devices and browsers for proper cross browser testing.

CrossBrowserTesting screenshot via CrossBrowserTesting

BrowserStack

Price: $29 - $4,979+/month

BrowserStack is also a cloud testing platform where tests can be imported, created, organized, then run in parallel on real devices, similar to CBT.

BrowserStack does lack some of the tools that CBT includes, such as visual regression and record and playback testing. However, due to that, it has more robust automated testing features than its competitor. For example, BrowserStack has a larger variety of debugging tools, including numerous types of logs and an interactive session feature.

With a larger number of supported languages and framework integrations than CBT,  BrowserStack is more flexible when it comes to creating your automated tests and connecting it to your current workflow.

As with CBT, the biggest advantage of using a service like BrowserStack is the ability to access their large, highly maintained grid of real browsers and devices.

BrowserStack in action via QAboy

Overall Review: Scripting Testing

Both tools are powerful, cloud-based testing platforms that can run your tests in parallel on their grid, which includes over 2,000 real desktop browsers and mobile devices.

Of the two options, CrossBrowserTesting is generally more powerful as it has a larger variety of tools. One drawback is its lack of integrations with frameworks in comparison to BrowserStack.

BrowserStack has fewer features, but is more focused on automated testing, has more robust debugging tools, and a greater number of integrations.

The pricing can vary greatly depending on the type of plan, number of parallel tests, and number of users. However, on a similar automation-based plan, BrowserStack can be more expensive than CBT.

While both tools have a lot to offer, the decision really comes down to what your team needs and which tool better caters to those specific needs.

Conclusion

Automation is a powerful ally that can help improve the quality of your website or app. Selecting a tool or combination of tools can be a difficult decision to make, as it requires thorough analysis of different tools and your workflow. As each project is unique, the types of testing and tools needed are unique.

If you're looking for some direction to get started with automated testing, PINT can help.

Tell us about your project