Website Testing: Beyond QA

Website Testing: Beyond QA

When should you test your website? A stock answer might be, “During the Quality Assurance phase of a redesign.” And that is a good time to test things. But there’s a deeper answer to the question of website testing.

Turns out that answer is also a phrase we’ve heard our CEO mention more than once: “Always be testing.”

thomas powell always be testing

In a perfect world, we would do fully test-driven development. This would allow us to test various aspects of the site during every phase of web development. It takes a bit more time, but ultimately creates a solid product built to last.

Website Testing for Every Phase

test driven development

In the chart above, we’ve diagrammed how website testing could be applied to each phase of a website redesign. Such testing is designed to ensure the best possible end product.

At the project start, there is a step before any new work begins. This is to set up analytics, heatmaps, and other tracking. It’s an important step. If you don’t really know what people are doing on the current site, it will be tough to improve on it with a redesign.

In addition, baseline data helps set the stage for success. With consistent data, improvements can be proven empirically by a variety of statistics:

  • Time on site
  • Pages per visit
  • New and returning visitors
  • Events triggered
  • Conversions

Basic Analytics

Google Analytics isn’t a bad tool for gathering much of this data. The key is making a few customizations to the standard installation to track metrics unique to your particular site. The most common updates are:

  1. Adding event tracking
  2. Setting up goals and conversions

Other tools that can be helpful at this point include:

Heat Maps

Hotjar or Mouseflow are two popular options for heat map software. They record how users interact with a given page, providing trend data on:

  • Where people click
  • Where people stop scrolling
  • More

This type of data is helpful for finding where on a page users fall out of the funnel.

Other Website Tracking

Piwik, Mixpanel, or Heap Analytics offer more granular tracking than basic analytics. They can help explore usage patterns and user info. When there’s a specific question to answer, these tools can help get you to the answer faster with funnels, cohort analysis, segmentation, and more.

Testing During the Information Architecture Phase

Tree Testing

Tree testing does not actually use the website itself. During these tests, users are presented with text lists and given a task to complete. One example task would be finding an ecommerce product. Then the Information Architect observes the steps taken to navigate down to that specific item. IAs use tree testing to confirm that the structure of the information makes sense. Then they will iterate on that to make the structure as efficient as possible.

First Click Testing

This test examines what a user clicks on first in order to complete their task. It can be performed on a functioning website, a prototype or a wireframe. They are good tests to run on existing sites during the IA phase. They help provide important user needs and preferences to the Information Architect.

According to MeasuringU, a user who clicks down the right path on their first click will complete their task successfully 87% of the time. That makes first click tests useful on wireframes to evaluate (and hopefully confirm) that the layout of information supports and communicates the proposed navigation structure.

A/B Testing

A/B tests splits traffic to a page in two buckets, and serve a different option to each bucket of users. A/B tests can be used to measure interaction based on different calls to action or page layouts.

Employing the right tests at the right time help minimize waste and ensure you move forward with the best possible version of your site or app.

  • Carrie Nusbaum, PINT UX Designer

Paper Prototyping

One other way an IA or UX expert may test a proposed feature is with paper prototyping. This involves making a paper version of a wireframe. Pages and pieces can be moved around based on “user clicks on the screen” (finger presses on the paper). This is a quick way to gauge user reactions to a variety of ideas. And since it uses paper, no developer time for coding is invested on ideas that don’t thrill end users.

Gathering Requirements for Transaction Tests

One other test-related activity that may take place during IA is requirements gathering for transactional tests. Doing this early on allows the team to create designs that accommodate all the required features and processes. The tech team can also start planning tests their code must pass in order for the site to do its job.

Testing During the Design Phase

website testing design phase

As wireframes get approved, the visual design phase can begin. During this phase, high fidelity testing may take place. This differs from the paper prototyping in the IA phase. These prototypes will include design elements and will likely be on a screen and clickable. They will not be coded but may use a program like Adobe Experience Design that simulates coded interactions on design comps.

Such tests can measure user reactions to different images and layouts. For example, if there is a large image area on a homepage or landing page, the design phase is a good place to test the gaze of people in those images. In 100 More Things Every Designer Needs to Know About People, Weinschenk share a flowchart to decide which gaze is best, but this could always be tested for your site’s unique audience.

gaze flow chart

Visual elements can support a solution’s interaction design, but they can also elicit an emotional response from users. Understanding and exploiting these emotional responses can help designers to influence users appropriately.

  • Michael Hawley, MadPow Media

Desirability Testing

The design phase is also the time to do desirability testing if time and budget allows. Experience questionnaires and quick memory tests are good ways to gauge emotional responses and attitudes toward site designs.

Testing During Coding and Development

Once developers are ready to have implemented approved designs into code and added interactive features, the temptation is to start writing that code right away, and worry about testing it afterwards. This is not the best approach however. A better idea is to write the tests first. In fact, this is where the concept of “test driven development” actually originated–an idea that grew out of the test-first philosophy of extreme programming (XP) before being formalized as TDD by Kent Beck, one of the original co-authors of the Agile Manifesto.

In this conception, developers begin by creating unit tests that correspond to the requirements captured (ideally in an iterative manner) in specification work with clients & users. These tests will of course initially fail (after all, the code to make them succeed hasn’t been written yet!), but the important thing is that when they do succeed, developers know that their units of code do in fact fulfill the requirements they have been given. In short, this methodology, properly executed, can prevent code that does not meet requirements from getting into the production.

TDD is all about having a high degree of confidence that your code really does do what it was meant to do.

  • Joe Lima, PINT Solutions Architect

Building Unit Tests

It’s crucial that the Unit Tests built during TDD test the smallest modules (“units”) of a code base that can be independently exercised. Once written, such unit tests should be runnable over and over, using an automated process. That way they can be put to use during QA and after launch for regression testing, to keep the code base in a known-good state even as changes are made to it.

At this point website testing can also begin on interactive usability. This testing can be facilitated with:

  • A program that gathers data from many users
  • Directing test users to staging site with analytics
  • Anecdotally through observation in person

Good ol’ QA

quality assurance testing

Cross-browser testing usually happens during Quality Assurance. This phase of test-driven development typically includes functional testing on the latest, currently-stable versions of desktop browsers. At PINT, we include:

  • Google Chrome
  • Microsoft Edge and Internet Explorer 11
  • Mozilla Firefox
  • Apple Safari

QA should also test the functionality on the latest versions of the mobile browsers. At PINT, this covers:

  • Safari on iOS
  • WebKit on Android
  • Chrome on Android

QA should also ensure developers are using forward-compatible methodologies. This practice leads to code that will work as designed in browsers adhering to modern web standards, now and in the future.

A note about forward compatibility: it does not cover minute default browser rendering variances. Nor does this guarantee code will work in all future versions Manufacturers may introduce breaking changes from time to time. Our friends at ZingChart recently blogged about the trend of website errors after browser updates. At PINT, we monitor these industry trends and try to alert clients when their codebases should be upgraded.

Transactional Monitors

Once QA is wrapped and everything is working as it should be, it is good time to set up transactional monitors. These are scripts or services that can run through processes to make sure users can complete the actions we want them to do. Commonly tested items include:

  • Login sequences
  • Navigation paths
  • Forms
  • Other interactive processes

Transactional monitors help alert us when some necessary part of a process gets changed or fails and unexpectedly breaks a process on the site.

It is best to set up transactional monitors after everything has been tested. That way they are monitoring a known-good process. Alerts can make you immediately aware of when something starts erroring, so you will know it before your customers do.

Post Launch Testing

Three types of test-related activities occur after the site or redesign goes live:

  1. RUM or real user monitoring
  2. A/B testing on site elements
  3. Updating transactional monitors/tests

Some people may need some rum at this point in the project. But we’re talking about the RUM acronym: Real User Monitoring. RUM records user activity on the site and ensures users have a positive experience. RUM may identify:

  • Errors
  • Slowdowns
  • Other issues that actual end-users experience in real time

Good RUM tools and services can also conduct this monitoring from a number of different geographical locations, and under a number of different bandwidth and device conditions, thereby covering your entire target audience in all its diversity.

A/B Testing

This is the same type of website testing that may have occurred during the IA phase. However at this stage, the point of the test isn’t to test different new ideas against each other. Post-launch, you’re more likely to test the new site against new ideas that emerge after reviewing user activity patterns. What changes will help move users further down the funnel?

Updating Monitors and Tests

This an important step. If the web team updates processes on site but doesn’t update the monitoring tools and tests, they will all break.

Advantages of Website Testing

Based on the process laid out above, testing offers advantages to organizations in a variety of situations:

  • Building a new site for a new venture
  • Redesigning an old site
  • Exploring how to work with the design you have

But testing is also helpful if you’re trying to encourage evolution. It doesn’t matter whether your team is resistant to change or on a tight budget. Suggestions for changes backed up by user data or performance metrics are much harder to turn down.

Looking for help gathering data or setting up tests? We help people do just that every day. Let’s talk.

Related Articles


Accessibility on the Modern Web

There’s been a lot of buzz in the news lately about accessibility, specifically in reference to the dozens of ADA lawsuits that seem to be more and more...


Automated Visual Regression Testing

What is automated visual regression testing? The name sounds scary, but in reality, the idea is fairly simple. If you have a user interface (UI),...


Automated Testing Tool Comparisons

Automated testing is rapidly gaining popularity across the web development field, and as expected, the number of automated testing tools is growing rapidly as well....