Skip to content

Integrating User Testing With Agile Product Development

 By Userlytics
 May 15, 2023

Integrating User Testing With Agile Product Development

For years if not decades top management executives from large corporations have talked about how the user experience (UX) of their customers is a top priority.

However, a quick & easy user test of even some of the most important and well known companies shows that this has, until recently, been more lip service than concrete actions.

As an example, here are a few highlight reels of UX issues we identified in two recent demonstration studies:

Example User Testing High Light Reel 

Demo User Testing High Light Reel

The reason for this mismatch between statements and results is not hard to find. Until mid 2015 there was a severe lack of resources underpinning the supposed priority on the digital user experience.

As a result, there were very limited resources assigned to the discipline of user experience design, user testing and user research, whether in terms of the number of personnel or the budget that these UX professionals have at their disposal.

The situation was not much better within the agencies serving large corporates. In a typical digital agency environment, there’d often be no more than 4-5 user experience designers handling UX needs for all the clients in an office of over 100 people.

The tide is finally turning. As consumers are becoming more design-conscious and the economy is becoming more experience-driven, the demand for UX designers has skyrocketed.

Demand for UX Designers

In addition, most large corporates are quickly staffing up their UX departments, increasing the number of user experience researchers by a factor of 400% or more in the space of 12 months.

This trend reflects the rising attention paid to UX as a discipline, which is also made possible by better and more effective user testing tools and methods.

ROI of User Experience Testing

Agile and User Testing

In parallel to the increased resourcing in terms of UX professionals, user research and user testing budgets, an increasing number of companies have been adopting an Agile methodology for project management and product development.

The Agile methodology focuses on building workable prototypes rapidly and then testing them to better inform the next iteration. 

It’s a departure from the legacy waterfall software development method, a sequential, non-iterative design process. The waterfall method is often described as flowing steadily downwards (like a waterfall) through the phases of conception, initiation, analysis, design, construction, testing, production/implementation, and maintenance.

Unlike the conventional waterfall workflow, the Agile approach and sprint-based development allow more opportunities for teams to test and improve the user experience design of a product, thereby bringing the importance of good UX to the forefront of the software development process.

Agile SW Development

Since 2015, Agile UX processes and systems have been adopted by many companies, large and small, and have implied the adoption of fast paced user research, user testing, and usability studies to support the iterative nature of design > test > redesign in an Agile UX Testing process.

ROI of Agile User Testing

The Agile methodology is becoming increasingly popular because it offers tremendous cost-saving advantages in eliminating development inefficiencies when compared to the waterfall workflow

In the waterfall methodology, user research and user testing are often treated as items to check off the list at the end of a product development process to “validate” previous decisions, rather than as a “learning” process; By the time a team received the user testing results, it was often too expensive to make the required changes if they significantly altered the features, functionality or UI.

ROI if early user testing

In an Agile methodology, the process of user research and user testing has become an integral part of the design and development flow. Agile UX testing is often performed in parallel to design and development sprints.

The results of such integrated “Agile UX Testing” has a significant impact on the subsequent iteration of a product, thereby making it much easier to justify the cost of investing in the research and quantify the return on investment (ROI).

Companies aren’t adopting Agile UX process just because “everyone else is doing it.” Research has shown that user acceptance and user adoption — both of which can be improved dramatically by well-designed user testing — is critical to boosting ROI and ensuring the success of a project.

ROI of user acceptance

Challenges In Integrating Conventional User Testing Into the Agile Design & Development Process

Even just a few years ago when many companies were still following a waterfall methodology, usability testing was often conducted in a lab and required careful planning months ahead of time. This is no longer a possible or practical set up for companies that want to apply user testing as a rapid, iterative “sanity check” to assist in the design and development process, which often span only 2-4 weeks. 

In that sense, in-lab moderated usability testing falls short when it comes to providing feedback in real-time to inform the next iteration of a product, because it is an inherently unscalable process.

In-lab moderated testing

For a long time, in-lab moderated usability testing and the Agile methodology have been at odds with each other because it’s hard to reconcile the time it takes to set up and conduct an in-lab moderated usability test and the fast turnaround time required by an agile sprint. 

Moreover, the cost of moderated user testing in a lab environment can be prohibitive, especially for smaller companies. Travel expenses, lab rentals, test equipment, time away from work by employees — these factors can all add up, making it impossible to incorporate frequent user testing into product development cycles.

The challenge has been to conduct usability tests in an agile and scalable fashion, so the user testing process can support the rapid product development cycles instead of slowing them down.

Agile User Testing Flowchart

Remote User Testing Faster than In-Lab User Testing

In order to solve the challenges posed by conventional usability testing and make it work for the Agile process, remote usability testing was invented. 

With remote user testing, instead of orchestrating day-long usability tests done in labs, real users take part in user tests from their homes and offices, on their own schedule — a comfortable environment that allows them to react in a way that’s closer to how they would in real life.

Remote User Testing. User Testing and Agile Product Development

Remote user testing allows user research to be set up and completed quickly — often with a 1-day turnaround — while providing a timely and continuous feedback loop of user experience insights to inform the Agile design and development process.

Remote user testing can be either moderated or unmoderated — each with its pros and cons:

Moderated Remote User Testing

Moderated remote user testing allows UX practitioners to design instructions and pose questions based on the actions and responses of each user testing participant. The Moderator and participant interact with each other in real-time, using screen sharing platforms like webex, gotomeeting, skype etc. 

Though this method removes the travel expenses and lab rental fees and increases schedule flexibility, companies still have to allocate specific personnel and time slots for each single participant, making this an inherently non-scalable solution.

The implementation of moderated remote testing is only possible when the UX department has enough personnel and resources to provide this type of usability testing support to the organization. In addition, the organization needs to be large enough and develop enough products to absorb the cost and justify having this level of resources on “standby” all the time.

Unmoderated Remote User Testing

During unmoderated remote user testing, UX practitioners assign pre-defined tasks and questions for participants to complete on a website, mobile app or prototype asset — via desktop or mobile devices. The UX team collect and analyze the qualitative and quantitative data via software that records every interaction, including a picture-in-picture video recording of both the participant and his/her screen (desktop or mobile).

Unmoderated remote user testing

This user testing method is inherently scalable because multiple tests can be managed simultaneously by a single user testing researcher, on the schedule most convenient to participants, with no need to tie down a user testing researcher on a 1-1 basis. It also has a relatively low overhead.

However, this method doesn’t allow real-time interactions with participants. So user testing researchers cannot probe with specific instructions or questions based on the real time actions and responses they observe from participants.

That’s where the latest innovation in remote user testing, branching logic with qualitative picture-in-picture video recordings, comes in, bridging the gap between moderated and unmoderated usability testing.

Combining the Benefits of Moderated and Unmoderated Testing

The emergence of unmoderated user testing platforms with branching logic is allowing UX designers to combine participant action and respondent dependent user test scripts with scalable user testing. 

Branching logic, with the aid of advanced software, combines the scalability of unmoderated user testing with the flexibility and customization of moderated usability testing. The moderator can customize the user test script for each participant with different instructions and questions depending on their actions and responses.

This method allows companies to conduct usability and user testing to collect both qualitative and quantitative data, including picture-in-picture video recordings of user testing participants and their screens, quickly and affordably so the results can inform the rapid and responsive product development cycles based of an Agile methodology.

This innovation in usability testing is enabling UX design & testing departments to conduct large scale, multiple and iterative user testing to support the exponential growth of digital interfaces that need rapid and effective user testing.

So that Agile development and a great user experience are no longer trade-offs to be managed, but rather components of a process that deliver what senior management has defined as the best strategic differentiator and ROI multiplier; an amazing user experience.

Agile User Testing equals great UX. User Testing and Agile Product Development

Didn’t find what you were searching for?

Related posts

You have been working hard and investing a budget on your latest product -you can't wait to get it in front of end users after an arduous design process. But you're also aware that the success of a product or service depends
AB Testing UX As businesses strive to stand out in competitive markets, their success hinges upon understanding and catering to the needs of their users. After all, an exceptional user experience not only fosters customer loyalty but also drives conversions and ultimately
Designing Meaningful Experiences in a Globalized World In today's interconnected world, it is crucial to design products and services that cater to diverse audiences. To achieve this, user research must be situated within the context of global design, considering the social, economic,
As UX researchers and designers, we understand the critical role that user experience research plays in creating successful products that meet the needs of our customers. However, conducting research is only the first step in the process - effectively communicating the findings
In the world of user experience (UX) research and design, stakeholder buy-in is crucial for success, though it is often a point of contention for practitioners who aim to prioritize UX in their organization. Without the support of stakeholders, UX research can
In today's digital landscape, creating a seamless and captivating user experience (UX) has become the cornerstone of success, regardless of the industry you are in. Testing digital assets is one of the most critical aspects to staying competitive. However, it is not
It’s no secret that a well thought out and engaging digital user experience increases sales as well as a loyal customer base. But what happens when your customer base extends into different countries and cultures? Do you copy, paste, and translate your
What’s the difference between a great user experience and a tolerable one? One might say the high-level stuff, like layout and design, but ultimately it’s in the details, or microinteractions. Microinteractions often make or break a digital user experience. When done well
Accessibility is a crucial aspect of creating digital products that can be used by as many people as possible. When designing a digital product, it's important to consider accessibility so that everyone, including those with disabilities, can access and use the product
They say finding love is one of the hardest things to do as a human being. We’ve been fed this narrative our entire lives, with movies, books, and songs all focusing on how impossible it is to find “the one.” Even in
User Experience (UX) has become a critical differentiating factor for businesses. To meet users' evolving expectations, UX research plays a pivotal role in understanding user behavior, pain points, and preferences. In this context, with the rapid advancements in Artificial Intelligence (AI), UX
UX designers strive to create intuitive, seamless, and delightful experiences for users —whether for websites, mobile apps, or prototypes. And one of the key components in achieving this goal is user research, a crucial phase where UX experts strive to understand users,
Things to be Thankful for: Advantages of Professional Peer Support in UX Research Thanksgiving is a time to reflect on all the things we are thankful for. As user researchers, we are thankful for the ability to connect with our peers and
2022 has been a busy and exciting time at Userlytics. We’ve launched several new features and capabilities, including sentiment analysis, accessibility testing, the ULX score, and VR testing. These newly implemented tools were designed to enhance your experience and provide you with
As staples of nearly all research conducted today, many people think they know about the differences between qualitative (qual) and quantitative (quant) testing. However, in the context of user research, these two methodologies are often used interchangeably. So, when to use one
Understanding what democratizing user research means is not as simple as searching for the definition online, or asking a colleague for a quick summary. In a general sense, the universal definition is “providing all stakeholders with access to user insight and UX
It's no secret that a good website or app user experience is crucial for boosting sales. In an offline location such as a store or restaurant, being physically present may keep a prospect even if the experience is subpar. But in an
In the ever-evolving realm of user experience (UX) design, research acts as the compass that guides designers towards creating delightful and intuitive digital experiences. As UX designers, we understand the vital role research plays in uncovering user insights, informing design decisions, and
As a UX researcher or designer, it's important to have a clear understanding of your target customer before beginning a remote UX study. Target user research is crucial for designing user-centric products and services. It helps ensure that the end result aligns
User research is an essential part of the UX design process. It helps us to understand our users, their needs, and their pain points. It provides the foundation for ultimately shaping the success of a product or service. However, without a systematic
In remote user experience and usability testing, there are a variety of factors that contribute to valuable, authentic user data that can be used to improve your brand’s user and customer experience. Among those factors are the quality and authenticity of the
Are you thinking about doing some user testing on your website, app or prototype? Are you currently weighing up the different options available to you? Should you use a free video-conferencing tool or a paid user testing platform? What are the differences
If you visited a complex website and found everything you were looking for with ease, it was unlikely a coincidence. A well-designed website structure is often the result of careful user research and testing, following information architecture principles and best practices. So
While remote user experience testing significantly enhances researchers' ability to reach test participants, these participants’ technical skills and access to technology can often vary greatly. Therefore, it is important to design and administer remote tests that facilitate participants' varying technical skills and
Userlytics prides itself in being a truly global platform. With over one million panel testers in over 150 countries, we are happy to say that we are able to find and test your target customers, wherever they may be. With the inception
In today’s highly digital world, cybersecurity is of the utmost importance when running day to day business operations. As usability testing grows in popularity, so does the need to choose a UX platform that values your company's privacy and security. But how
Userlytics makes user experience testing easy, efficient and fun with its state-of-the-art no-download web recorder. This revolutionary user experience testing recorder allows participants to jump right into usability tests directly with a simple click of a button; this means you can more
2021 is a year dedicated to the customer. Businesses are moving away from organizational-centric strategies and thinking more and more about the needs of their customers. But what exactly do customers want? Customers want their experience with brands and digital assets to
Moderated testing sessions are a great opportunity to get thorough and instant feedback from users all around the world. This type of testing involves a live, online interview where you are face-to-face with your users via webcam, asking them questions, guiding them
Researching the ins and outs of your business is integral to setting your company up for success. Increasingly, fine-tuning and improving your customers’ brand experience has become one of the top priorities for businesses. With customer expectations being at an all-time high,
Are you a UX researcher who mainly conducts in-person studies and you’re looking for some online alternatives to get you through this unusual period of lockdown and social distancing? Or perhaps you’re someone who is starting their path down UX enlightenment, looking
Given the current world health and associated economic crisis, Userlytics ran a qualitative moderated ("Live Conversations") user experience study with participants in the US to determine how respondents’ online shopping behavior has changed as a result of the pandemic. The following is
Picture this: an online shopping user clicks on a product they like but the description does not say everything they need to know before making online purchases; key features, pricing and availability, or shipping and return information -strike 1. To find out
The Best and Worst UX in the Travel Industry Finding the best flight deals online can be an exhilarating experience. Almost every booking site today promises the lowest fares available and, in some cases, even a price match guarantee. With such a
Last November 18th our CEO was interviewed by Ditsa Keren, of Website Planet; Here are some excerpts from the interview: “If we take a step back and think about how software has evolved over the past 15 years, the cost and time
Designers are prone to falling into the trap of cluttering their interfaces. Limitations of space often push them to the wall, and they find themselves trying to fit everything they can into a tiny usable space. The unique problem that the modern
How Can Bad UX Influence Your Conversion Funnel? Source: Shengjun Shi Regardless of whether you sell a product, offer a service or run a blog, any visitor comes to your site with simple goals like consuming or gathering more information. The user experience
From initial concept to finished draft, tips on how to craft an effective unmoderated user study A user test is a little bit like cake: you know what a great one looks like, but you’re not quite sure how to go from
Apple has recently entered the race with its Homepod. Which of the two current leaders, Amazon's Alexa (Echo) or Google Home, provides the best user experience? Interactive Voice Response (IVR) systems used by banks and utility companies provide some of the clearest
February 22th, 2018 Usability testing data is a sure-fire way to eliminate guess-work and focus on making the UX design process more effective. These types of user tests can help you figure out if your audience can adequately use your website or mobile app. Additionally, they help
In theory, every company these days is customer-centric and has a mobile-responsive design process, and yet a surprising amount of companies haven't updated their online stores or platforms accordingly. Many of them are not leveraging the benefits user testing can bring to their business. Today's
Online shopping is not a new concept. According to statistics, about 80% of Americans are shopping online these days, some exclusively, with zero visits to actual physical stores. With years of experience of e-commerce design and user research, one would think major e-commerce sites
Turning visitors and prospects into paying customers is about more than good advertising and slick SEO. Users are savvy enough to tell the difference between good and bad UX, and it can often be the differentiating factor between a single visit and
Are you delivering the best user experience for your website and mobile app visitors? You may think your website or app offers seamless navigation, intuitive form usage, and an effective way for users to interact with the information being provided. However, the real question is: What
By Mark Hall, Guest Blogger.An e-commerce and Customer Research Expert who has consulted for over 20 years with a wide range of clients including AT&T, American Express, Edmunds, SpyTec and the California Lottery. When not working, Mark enjoys playing saxophone, mountain biking
Everybody working in the IT industry has seen a number of case study videos, where users talk about glitches in software, unavailable options in apps, and their ideal features and interfaces. Running a remote unmoderated usability study seems to be an easy way
The conflict between qualitative and quantitative user experience research methodologies has its origin in a time when computers were only a dream, and with the emergence of the social sciences over a century ago. It lies in a struggle that is much older than
How to tell the difference, and when to bring both into your projects In the world of User Experience Research, there’s often some confusion surrounding the difference between User Studies (and user research in general) and User Acceptance Testing (UAT). In the
Quantifying how good a user experience is, or how well a site meets the needs of its users, can be a difficult task. We can look objectively at metrics like time on task, or success/failure rates to determine user behavior, but they
By Mark Hall, Guest Blogger. An e-commerce and Customer Research Expert who has consulted for over 20 years with a wide range of clients including AT&T, American Express, Edmunds, SpyTec and the California Lottery. When not working, Mark enjoys playing saxophone, mountain

Let’s work together on your next UX study.

Create positive user experiences and keep customers loyal to your product and brand.

Analytics tells you what,
Userlytics tells you WHY.