Challenges in QA testing and how to overcome them

Introduction

Our firm specializes in developing and maintaining extensive websites for enterprise-level clients and government agencies. When constructing or overseeing websites on a continuous basis, ensuring optimal performance across various devices—such as mobiles, tablets, or desktops—for sites comprising hundreds or thousands of pages can be a demanding endeavor.

Considering the vast number of devices and the multitude of pages, it is essential to make strategic choices that maximize the return on investment while testing the website efficiently and effectively. In this insightful article, we delve into various business and technical considerations we take into account to achieve this goal.

‍

Analyze the customer’s website

When we begin evaluating a client's website, our initial step is to examine their Google Analytics, with a particular emphasis on the top 10 pages in terms of traffic. We have found that for many websites, up to 80% of their monthly traffic is generated from these top 10 pages. As a result, concentrating on these pages and delving deeper into the details serves as a practical starting point. It goes without saying that concentrating on, for example, ten pages rather than thousands significantly streamlines our team's efforts. A sample report is provided below for reference.

The next step is to analyze traffic distribution across different devices, such as mobile, tablet, and desktop, to determine the top 5-10 platforms that require routine testing. A sample report is provided below for reference.

In the given context, it is evident that the iPhone attracts the most traffic, followed by the iPad and older iPhone versions. Samsung Galaxy phones account for the remaining substantial portion of the traffic. 

Lastly, we do a traffic analysis to identify the kinds of browsers that users are using when they visit the site. A sample report looks like this:

By combining both device and browser analyses, it becomes apparent that Safari on the iPhone takes the lead, with Chrome on the iPhone (and presumably Samsung phones) following closely behind. While there are further secondary insights that can be derived from this data, we will set those aside for the moment.

‍

Use QA Virtualization Technology

Now that we know what pages to focus on, along with the kinds of devices and browsers, our teams use virtualization products such as Browserstack to test on these selected configurations. A sample test plan looks like this.

The QA team can test the webpage on multiple different devices and browser configurations to quickly identify any issues. 

‍

Leverage QA Automation and Machine Learning

Browserstack facilitates continuous testing by offering automated screenshot capabilities for an array of configurations, which can be viewed in one glance on their dashboard. This allows the team to promptly identify any issues and eliminates the need for ad-hoc manual checks on multiple configurations. A sample report is provided below for reference.

In addition to these methods, machine learning-based visual testing can be utilized for more efficient and accurate testing. For instance, by training AWS Rekognition using images at various resolutions and regularly sending images across these resolutions, we can receive automated feedback if any discrepancies or issues arise. This provides a powerful tool to complement traditional testing methods and helps to identify potential issues more quickly, thereby saving time and resources. 

‍

Summary

Thoroughly testing a website and providing customers with peace of mind can be a challenging endeavor. However, as demonstrated, smart analysis of top-visited pages, prevalent devices, and popular browsers can assist in making informed decisions that effectively cover 80-90% of the website. Utilizing tools like Browserstack and Machine Learning can further enhance your productivity.