Automated Testing Tool Comparisons

Automated testing is rapidly gaining popularity across the web development field, and as expected, the number of automated testing tools is growing rapidly as well. These tools can handle one or more types of testing, including visual regression, record and playback, and scripting testing.

To learn about these types of automated tests, please refer to part 1 of this blog post, Automated Testing for the Web.

Below we’ve reviewed and compared a couple of tools for Visual Regression Testing, Record and Playback Testing, and Scripting Testing.

Visual Regression Testing: Percy.io vs. Applitools Eyes

Percy by BrowserStack

Price: Free – $269/month

Percy by BrowserStack  is “your all-in-one visual review platform.” This tool will automate visual testing by comparing screenshots to a baseline and highlighting changes to prevent unintended visual regressions. It claims to have built-in support for parallelized test environments with only two simple parameters, so running complex test suites in parallelized CI services or parallel test runners is relatively easy.

The Percy Visual Engine is marketed as a fundamental AI-powered algorithm that is an upgraded version of the existing diffing algorithm to improve noise reduction and highlight important changes. It claims to use computer vision to render each pixel accurately, mitigating against false positives caused by anti-aliasing. Its snapshot stabilization technology is designed to prevent false positives by automatically freezing animations. It should integrate with existing CI/CD workflows and support test automation frameworks like Cypress, Puppeteer, WebdriverIO,  Protractor, and TestCafe.

Applitools Eyes

Price: Free – Price is determined on a case by case basis

Applitools takes visual regression testing one step further by integrating AI learning into their product. Different browsers may render a website slightly differently, which could trigger several errors in a standard pixel by pixel visual comparison.

Applitools uses “AI-powered computer-vision algorithms” to simulate the human eye and ignore small, imperceptible differences to avoid potential false-positive results.

Applitools can also be integrated into many different frameworks and tools. However, it is designed to be an addition to your pre-existing tests, meaning it can’t be used independently. Fortunately, Applitools has an integration with the record and playback tool, Selenium IDE (discussed below), which only takes a few clicks to begin writing your first test.

Overall Review: Visual Regression Testing

Both tools are powerful additions to any workflow and are user friendly. Applitools Eyes may boast an AI engine to better prevent false-positives, but Percy is not without its own smart algorithms.

The primary determining factors are differences in integration options and pricing. It is crucial to check that your preferred frameworks and tools are supported by either product. Pricing is also dependent on the number of screenshots you take, which can quickly and drastically change your pricing tier.

Record and Playback Testing: Selenium IDE vs. TestCafe

Selenium IDE

Price: Free

Selenium IDE is a free browser extension for Chrome and Firefox which allows you to easily record and create automated tests. Installation is very quick and easy, so you can begin making tests in a matter of minutes.

The UI is clean and relatively simple to understand. It includes a useful reference tab that explains the different types of commands, which is essential for getting started.

One of the handiest features is when recording, you can click on an element on the webpage and it will be populated into the target section. This helps take the guesswork out of finding the appropriate class or ID of the selected element.

However, like any tool, it has its shortcomings. For example, commands are shown in a long dropdown, and while most are easily understood, it can be intimidating and daunting to scroll though at first.

The primary drawback though is its inability to do cross-browser and/or parallel testing without using Selenium’s command-line runner, which drastically increases the learning curve and technical knowledge needed. Overall, Selenium IDE might be a better solution for rapid prototyping, rather than robust testing.

DevExpress TestCafe Studio

Price: $250 or $500/year

TestCafe Studio is a paid desktop application, that is a more polished version of Selenium IDE with a few extra features.

In addition to its more refined UI, the most prominent difference is the use of icons to visually represent the list of commands, rather than a long list of text.

Like Selenium IDE, you can select and populate a target element’s information while recording, but TestCafe goes one step further and also lists the properties and current values. These types of features help reduce the learning curve and will make it easier to write your tests.

One of the biggest perks of TestCafe is its ability to easily run cross-browser and parallel tests. Although many aspects have been simplified, it is still beneficial to have a basic knowledge of CSS classes and IDs.

Overall Review: Record and Playback Testing

As usual, you get what you pay for. Selenium IDE is very powerful, relatively user-friendly, and best of all, free. It is ideal for anyone looking to explore this type of testing tool or for creating tests for more casual projects.

TestCafe is more polished and has a smaller learning curve due to its simplified UI and ability to easily run cross-browser and parallel tests, but at a cost. TestCafe’s features and support make it better suited for a company needing a more formalized tool and process.

Scripting Testing: CrossBrowserTesting vs. BrowserStack

CrossBrowserTesting

Price: $78 – $1,500+/month

CrossBrowserTesting (CBT) is a cloud testing platform that includes a variety of automated and manual testing tools, including visual regression, record and playback, scripting testing, and more.

In terms of its automated testing capabilities, you can import, create, and organize your library of tests, then run them in parallel on real browsers and devices.

Running multiple simultaneous tests, the speed of your automated testing can be greatly increased. In addition to running tests through their web application, you can also use their API to trigger your automated tests.

CBT supports many popular programming languages and has integrations with a variety of testing frameworks and CI/CD tools. This allows for CBT to be easily integrated into your pre-existing workflow.

Of the variety of features, one of the greatest benefits of using a tool like this is that you can leverage its large collection of real devices and browsers for proper cross browser testing.

BrowserStack Automate

Price: $129 – $249+/month

BrowserStack is also a cloud testing platform marketed as a tool where teams can harness the Selenium grid to start testing on 3000+ mobile devices and desktop browsers. As the latest update to BrowserStack’s Automate test platform, its Automate TurboScale is designed to allow setting up browser automation grids on many cloud providers.

Teams can harness the power of parallelization to reduce the execution time of test suites. Automate is designed to support 50+ integrations across frameworks, CI/CD tools, project management, test case management, and workflow automation.

Each test run is intended to capture all logs, including text, console, network, screenshots, and video recordings. You should be able to access them via the dashboard or API or use breakpoints to debug interactively and pinpoint issues faster. Teams can also utilize BrowserStack’s secure WebSockets (WSS) to test web applications on internal development environments using the local binary.

Overall Review: Scripting Testing

Both tools are powerful, cloud-based testing platforms that can run your tests in parallel on their grid, which includes over 2,000 real desktop browsers and mobile devices.

Of the two options, CrossBrowserTesting is generally more powerful as it has a larger variety of tools. One drawback is its lack of integrations with frameworks in comparison to BrowserStack.

BrowserStack has fewer features, but is more focused on automated testing, has more robust debugging tools, and a greater number of integrations.

The pricing can vary greatly depending on the type of plan, number of parallel tests, and number of users. However, on a similar automation-based plan, BrowserStack can be more expensive than CBT.

While both tools have a lot to offer, the decision really comes down to what your team needs and which tool better caters to those specific needs.

Conclusion

Automation is a powerful ally that can help improve the quality of your website or app. Selecting a tool or combination of tools can be a difficult decision to make, as it requires thorough analysis of different tools and your workflow. As each project is unique, the types of testing and tools needed are unique.

If you’re looking for some direction to get started with automated testing, PINT can help.