The Term “Benchmarking”

This section covers “benchmarking against competitors”. Before diving in, let’s look at the meaning of the word “benchmark”. According to Oxford Languages, “to benchmark” means “evaluate or check (something) by comparison with a standard.”  The two actions in this definition are “evaluate” and “compare”. The “standard” is typically an industry standard, but it can also simply mean to evaluate and compare products against your product; you’re using your product as the standard, in this case. We’ll show how the focus on both evaluating and comparing can yield a lot of crucial information.

The Benefits of Reviewing Competitor Products

There are a number of benefits to reviewing your competitor products.

Features and Their User Experience

Feature Availability

A product or solution typically consists of a set of features. Similar applications can have a somewhat different set of features and functionality. You will want to understand which features and functionality competitors’ apps have. Do they have features that your product doesn’t have? Do you have features that they don’t have?

Impact of the Feature

Understanding what the feature set looks like provides some insight, but a next crucial step is to understand the value of these features. Imagine your competitor has a feature you don’t have. Is it even worth considering a similar feature? It would be wonderful if you could understand what impact that feature has on your competitor’s bottom line, their customer retention, or any other KPI. Unfortunately, acquiring this type of information is often not easy.

One way to gather this information is to conduct a competitive analysis and get user feedback on the ease of use, usefulness, and other aspects of your competitor’s features. Analyzing the user feedback can provide insights. Answers to “why” questions and open questions (those that require the user to talk or write down their thoughts rather than selecting an answer in a multiple-choice question) will help shed light on the impact. This knowledge is important, yet we need to be mindful that other factors influence whether a feature makes sense to implement, such as development cost and timeline, and whether this feature is truly in line with your brand and product direction.

Benchmarking the Feature

This competitive analysis can also be used to start benchmarking features and products. Users answer rating questions based on a scale. Careful consideration is needed to make sure that what we measure provides insights and hence creates a meaningful benchmark. What to measure depends on the product, the competitive landscape, and must align with your company’s KPIs.

Ideally, benchmarking exercises are conducted on a regular basis (e.g. every quarter, or every year) and provide a longer-term view of the state of your product and its features relative to your competitors.

Removing Barriers to a Great User Experience

Certain industries are heavily regulated. As a result, brands may be required to both gather and provide certain information at certain points in the customer journey.

For instance, certain personal information must be gathered when a user applies for a credit card. This information gathering may present a barrier to the user experience; it slows down the process and the user may find providing the information cumbersome. The fintech industry very much competes on the user experience. As a result, it is critical for credit card companies to make the experience of applying for a credit card as easy and seamless as possible.

Similarly, US airlines are required to share federal regulations stating that hazardous materials are not allowed in baggage and outline what is defined as a “hazardous material”.  This information must be “prominently displayed”.

UX designers and UX writers often spend a lot of time trying to optimize the experience for these types of instances. As part of their effort, it can be helpful to look at how others have tried to overcome these challenges. And getting user feedback on these different approaches is necessary to be informed. Just as the existence of a feature is not insightful without user feedback, user feedback provides additional insight here as well. Another, perhaps quicker way to get examples of experiences that are likely good, is to review the applications that are deemed “best in class” apps in the relevant vertical. There is an assumption that the “best in class” experiences are indeed that, of course.

The Viability of a Conversational Approach

We’ve largely gotten used to the fact that a digital consumer app will have a web version and a mobile web version or mobile app. Whether or not there is a desktop client version will depend on the type of application, and whether it makes sense to develop. In other words, it is important to consider and evaluate whether a desktop client is a useful and therefore viable approach.

The same holds for a conversational approach, and it is important to consider whether a conversational approach makes sense. More on this topic can be found on this site in Analyze Existing Communication Channels.

To get closer to understanding whether a conversational approach would be a good addition to ways in which a customer can communicate with the brand, a competitive review of conversational applications is an option. However, it may not tell the full story. Imagine you review the competitive space you operate in, and no competitors have a conversational application. Does that mean that they look into the option and decided, based on solid data, that this was not a worthwhile effort? And if that was the case, did this research effort happen two years ago when it truly wasn’t viable, but now times have changed? Or has no one truly considered a conversational application to your space? Based on the information, you have little way of knowing whether you are a trailblazer who’s stumbling upon the gold, or if you’re just throwing money and time out of the window. Again, the existence or lack of a feature, or in this case, another way to communicate, does not provide you with all the needed information.

Methods

Functionality Matrix

The main goal of a functionality matrix is to catalog the features, in other words, their presence or absence.

Other information that can be gathered includes:

• Subjective ratings on aspects of the feature, e.g. ease of use. This is the perspective of a single reviewer only (unless additional reviewers are added for this phase) and must be viewed within the context of this person as a reviewer (e.g. rather than a target user in a more natural setting)
• For forms or complex workflows, it may be useful to gather which information is requested at which step, and how that may influence the user experience. E.g. consider a telehealth app that asks for all user information before allowing to schedule an appointment, versus asking for the necessary information, allowing the user to schedule an appointment, and subsequently gather additional information

Competitive UX Study

Qualitative Study

The goal of a qualitative study is to gather user feedback on what users like, dislike, what is easy, what is cumbersome; the information that helps you understand the value of a feature, and how to best implement features, especially in environments that are heavily regulated.

Study participants are asked to complete a number of tasks in one or more applications and provide their feedback. The feedback may consist of thinking aloud, answers to open questions, and answers to ratings questions. Qualitative studies use a small group of participants.

Two common methods for competitive studies are within-subjects studies and between-subjects studies. The former has the same set of participants and all participants use all the different applications under review. It is important to counterbalance: if 3 applications are reviewed, the order must be randomized across participants. This technique combats bias. Within-subjects studies allow the participants to self-report a product comparison. It is generally the cheaper of the two methods, but can only be used if few (max 3) applications are reviewed. More applications would extend the session time and lead to participant fatigue, which in turn leads to lower quality results.

Between-subjects studies use a different set of participants for each application. This is the preferred method if the application is extensive as it makes for longer sessions for the participants. It is also a better option if more than 3 applications are reviewed. It is the more expensive option since more participants need to be recruited and compensated. Results may be more extensive, and the UX researcher covers the comparison, in addition to the analysis.

Regardless of the type of competitive study, it is important to run competitive studies as a blind study; participants must be unaware of the brand that is sponsoring the study, in order to avoid bias.

Quantitative Study

The goal of a quantitative study is to gather data, in this case, benchmark data. Quantitative studies are typically surveys and use a larger group of respondents.

As mentioned above, asking questions that will provide relevant insights is crucial to the usefulness of these results. The size of the respondents group depends on a number of factors. For more information on quantitative studies, please visit the resources section or google “quantitative UX research methods”.

Other Resources

• UX Strategy, Jaime Levy. O’Reilly. 2021. A great book on product strategy for digital applications (not focused on conversational AI). Chapter 4 Conducting Competitive Research and chapter 5 Analyzing the Competition have a lot of great information to help you with your competitive research.
• To read more about UX and UX studies and methodologies, there are many great resources on the Internet. One, in particular, is Nielsen Norman Group. They have an extremely extensive library of articles, videos, etc.