Menu
Qualitative Analysis
This analysis is part of the design process - 01 Learn & 05 Refine - of the Case Study Design for Learning - AI Research Problem.

#Survey #ThematicAnalysis #QualitativeData
Summary
The Case Study Design for Learning - AI research Problem is based on the problem space of How do we shift from a concept of tracking student's progress to start tracking student's proficiency?

To understand how users perception variate for progress and proficiency according to certain types of graphic representations. I conduct a survey study.

01 Learn - My Approach & Survey Process

Contextual Inquiry
(empirical process)
Users & Business Goals
Data Gathering
raw work activity data
organized & structure work activity data
Needs & Requirements
(deductive analytic process)
Extraction
Design Informing Models
(e.g., flow models, usage scenarios)
Extraction
DESIGN
(integrative process)
Contextual Analysis
(inductive analytic process)
Data Interpretation & Consolidation
Hartson, Rex, and Pardha Pyla. 2012. The UX Book. San Francisco, United States: Elsevier Science & Technology.
For a more detailed understanding, feel free to download the full survey report here.
Contextual Inquiry / Data Gathering
Survey Process

1. I gathered 15 different graphic representations normally attribute to the concept of progress and/or proficiency.

2. I asked the users how they perceive the respective graphic individually.

3. I gather all the graphic representation on a grid and ask which of them they thought to better represent the notion of progress and the notion of proficiency

Goal

Test for Distinction Bias.
Interesting Fact: Our users are all accountants. So, I found out with this study that they have an intrinsic need for every graphic representation to be followed by numbers, results or percentage description. They just love numbers :D
Contextual Analysis / Data Interpretation & Consolidation
The most voted graphic representation for Progress
20,69% of users voted with a significant expression for this representation as to the most recognizable for tracking progress.
The most voted graphic representation for Proficiency
30,43% of users voted with a significant expression for this representation as to the most recognizable for tracking proficiency.
Needs & Requirements / Extration
The two concepts are already complex to distinguish, so upfront, I exclude all the graphic representation that didn't have a significant result. I didn't want to fall in the mistake of creating an ambiguous representation. Accordantly, I immediately excluded the following representation:
Final Design Decision
Survey Study showed us that users associate proficiency more to starts representations and percentages and pie charts to the concept of progress. Accordantly, I followed the following graphic representation for the CPA exam review application. On the left for tracking proficiency and on the right for tracking progress.

05 Refine - My Approach & Survey Process

PART I
Open-End Survey
After finishing this first MVP, we conduct a Beta A/B Test with ~3 000 students. The AI algorithm needed to learn from individual users but also from the collective usage. To give time to test the algorithm, we launch the A/B test during three months and the end surveyed the students.
For a more detailed understanding, feel free to download the full outline survey here.
PART II
Thematic Analysis
To begging analysis the exploratory research, I conduct a qualitative analysis to uncover themes in the data.

I start it by doing a thematic analysis to explore similarities and relationships between the different chunks of the survey data. The goal was gather more visual means of analysis to present to the stakeholders in the final session review of the design process.
Important Note: Visualization helps my team to understand better the similarities and relationships in the data more clearly to discloser better design and business decision.
Thematic Analysis Process

1. I like to print the results transcripts and start to highlighting important keywords and sections that are relevant to the study.

2. Next, I start cutting the paper texts sections that have been highlighted and start organizing them on an Affinity Diagram and making some side notes to them (descriptive labels).
Important Note: Ideally this process shouldn't be done alone. But I am a team of one and my stakeholders are on the other side of the ocean. I had to remedy the situation and conduct the analysis myself and be subject to my own bias.
3. I organize the data into blocks of themes based on their relationships, and start to figure out some categories. At the end I found 4 major themes:
User Interface (UI)
Algorithm
Features
Behavior
Note: This was a time consuming and difficult task because I had to analyze the feedback from 300 students. It was an interesting challenge to reduce to 4 categories.
4. Once I found my 4 categories, I start thinking about how these all relate to each other.

5. Since we are a remote team, I used Miro platform to build a visualized representation of the categories relationships to present to my stakeholders.

6. With the Product Owners we explore the results from the 4 points of view - Behavior, Algorithm, User Interface and Feature. We analyse the positive and negative feedback and come up with solutions and strategies to improve the feature efficiency and experience.
Important Note: For privacy and competitor reasons, I am not allowed to share this dashboard real content.
The findings from this analysis where implement in the Design Review in the design process phase 05 Refine on the Case Study Design for Learning - AI Research Problem. Follow the case study below.

Case Study
Design for Learning - AI Research Problem

The short turn strategy of Becker, is to move from a fill-to-all experience to a personalized one. With the support of Artificial Intelligence (AI), we are moving from the concept of Progress to Proficiency
Made on
Tilda