Quantitative vs. Qualitative UX Research
In this article, we’ll compare quantitative UX research and qualitative UX research. Both of these types of research have their own unique pros and cons, required resources, and available methods. Therefore, knowing the differences and similarities of both of them will help you to improve your user experience (UX) research projects.
Quantitative UX research
According to Wikipedia, quantitative research is “the systematic empirical investigation of observable phenomena via statistical, mathematical or computational techniques”. More practically, it refers to (UX) research based on numbers (often in high volumes) and facts, wherein participants are seen as anonymous data points.
Pros compared to qualitative research
- No recruiting of participants needed
- Statistical methods can be used due to high volume
- More based on ‘facts’ instead of ‘opinions’
Cons compared to qualitative research
- Only insights into ‘what’ not ‘why’
- Requires high visitor numbers
- Translating raw data into findings requires expertise
Traffic: Most of the methods outlined below rely on you having fairly high volumes of traffic on your website. Without these numbers, the statistical techniques used in quantitative research simply aren’t reliable. The exact amount of visitors that you need depends on the specific method.
Money: Gathering the quantitative data is relatively cheap in most cases, as it comes from visitors that are on your website anyway. The analysis of this data, however, is probably quite a bit more costly. Namely, because this data will need to be analyzed by specialists.
People: Analyzing the data will often be performed by academically trained specialists such as data scientists or web analysts.
Maturity: Many of the methods outlined below rely on either having quite a bit of traffic and/or having trained specialists to analyze the data. Therefore, quantitative methods tend to require slightly more digitally mature organizations compared to most qualitative UX research methods.
Time: Similar to the cost analysis, the bulk of the resources in terms of time will also be spent on analysis instead of gathering the data.
Each of the quantitative research methods listed below has its own characteristics. Before jumping head-first into one of the methods or start using specific user feedback tools, determine which of them is most relevant for your unique situation. Especially important for quantitative methods is that you preferably always be collecting data, even when you don’t have the time to analyze it right away. Also, as with all research methods, keep in mind relevant privacy laws such as GDPR when gathering your data.
By far the most popular tool in this category is Google Analytics. Other tools include Adobe Analytics, Webtrekk, Clicky, and Piwik. With these tools, you can log the behavior of visitors on your website. Some examples of available metrics are bounce rate, time on site,
With methods such as heatmaps, movement maps, and scroll maps, you can get an aggregated view of what parts of a website visitors view and click on. Heatmaps will show elements where people click on in warmer colors and elements that aren’t clicked on in colder colors. Popular tools to create visual feedback on your website like this include Hotjar and CrazyEgg.
Public data sources
An often overlooked approach to gathering useful quantitative data to improve your website is public data sources. For the USA these include Data.gov, Census.gov, and Usa.gov. For other countries, alternative resources are likely available online too. These data sources can help you to get a clearer picture of your target audience, or aid you in the creation of your personas.
For larger organizations, analyzing your back-office sales and leads data can generate valuable insights. For instance, it can allow you to identify in what regions or cities your organization currently is most successful. It might also help to clarify what demographics are currently buying which products, something that might help you to improve targeting. When doing such analyses, keep in mind that there are privacy laws such as GDPR that prohibit freely linking data sources to each other.
In order to learn more about what questions visitors have on your website, you could analyze your chat logs. This will likely uncover common issues or frustrations and might lead to new FAQ items or a clearer view of what the top tasks of your visitors are.
If your brand is active on social media channels such as Twitter, Facebook or Instagram, then you might be able to perform sentiment analysis. With this technique, you could learn more about what content resonates well with visitors and what doesn’t. At the same time, you can also learn about the attitude of your customers towards your brand over time.
A lesser known form of quantitative UX research is form analysis. With this method, you analyze the form on your websites and look for difficult or redundant fields. Software for form analysis will show you metrics such as dwell time, return percentage, drop-off percentage, and much more.
Qualitative UX research
Wikipedia defines qualitative research as “a scientific method of observation to gather non-numerical data.”. More practically, it refers to (UX) research based on “‘meanings, concepts definitions, characteristics, metaphors, symbols, and description of things’ and not to their ‘counts or measures.'”.
Pros compared to quantitative research
- You’ll get much more detail per tester
- The ability to ask specific usability questions during the sessions
- The analysis doesn’t have to be performed by data analysts
Cons compared to quantitative research
- Recruiting of participants can be time-consuming and expensive
- Analyzing the individual feedback can take a lot of time
- Because of the low volumes, no statistical generalization is possible
Traffic: Most of the listed methods don’t require large numbers of visitors on your website as they recruit testers externally.
Money: A large part of the cost of qualitative user research comes from recruiting the testers and setting up the testing sessions. Especially when you’re just starting out with UX research, try to go for a ‘guerrilla’ approach to save money. With guerrilla UX testing you use simple, effective and cheap techniques (such as user testing in a café or interviews via Skype) to get quick and dirty insights.
People: In most cases, setting up UX research experiments properly actually takes quite some expertise. After all, you want to choose the rights tasks for the testers to complete and ask the optimal questions afterward.
Maturity: Qualitative research tends to be easier to start with than quantitative research. Therefore, the required digital maturity level of an organization can be lower. However, keep in mind that setting up more complex experiments will ramp up the required maturity level pretty quick.
Time: For most of the qualitative research methods, significant time should be invested in both the setup of the experiments and the analysis of the results.
The different types of qualitative research methods listed below have their own characteristics. For instance, some of the methods require the recruiting of participants, while others don’t. In particular, for these qualitative methods, keep in mind relevant privacy laws such as GDPR when processing (personal) data of your testers to avoid legal issues.
Collecting feedback from real people about your website can be incredibly valuable. With user feedback, you can learn about matters such as trust, clarity, and usability of your website.
Another way to gather qualitative feedback on your website is by having your visitors or customers complete online surveys. This enables you to ask questions about their preferences or obstacles they encountered on your website.
With user testing you can see and hear visitors interact with your website, performing specific tasks that you set out for them. User testing can be done remove or in-house and moderated or unmoderated. Each of these choices carries their own benefits and challenges.
If you want to learn what your visitors think of your organization, you could ask them to write a review about you. Many companies display these reviews (or the aggregated scores from them) on their website to establish social proof.
One of the classical ways of collecting qualitative user feedback is to interview people. You can choose to interview people in real life or via digital methods such as Skype. In order to make the most of the valuable interview time, try to prepare the interviews as best as you can. Make sure you have a list of questions ready and decide whether or not you want to show your website during the interview.
Via tools such as Hotjar, ClickTale, and SessionCam you can see an actual website visitor interact with your website. These tools allow you to see where visitors click, where they move their mouse, and what pages they look at.
Both quantitative and qualitative research methods have their own unique upsides and downsides. Therefore, it’s important to consider both of these types of research first, before starting to gather data. Such an inventory analysis will most likely result in better findings from the usability testers.