Educational Testing Service / Product Concept Validation / 2022
Exploring & Testing Solutions for Unmet Needs in the Higher Ed and Workforce Space
Overview
I conducted strategic and generative research as well as concept testing of the ‘Student-Employer Match’ product concept: a matching service incorporating advanced analytics on student and employer data to predict the probability of employment and good-fit. My insights from user research, combined with my team's thorough market research and go-to-market strategy investigations, were notably impactful in illustrating to leadership the crucial role of collaborative early validation work. By leveraging our learnings, the organization was able to pivot from this initial concept and move in a direction better aligned with market needs while minimizing time, cost, and risk associated with product development. As a result, research garnered recognition at the highest levels of the organization, fostering trust in the process and its ability to deliver impactful insights and drive business value.
Role
UX Researcher
Discovery Research
Primary Research
Workshop Facilitation
Data Synthesis
Concept Testing
Team
Market Research
Psychometric Research
New Product Development
Go to Market
UI/UX Design
Year
2022
Summary
01.
Challenge
ETS was uniquely positioned to explore this problem space. Bridging ETS’s internal Global Higher Education initiative with its internal Work Skills initiative, the idea of a “Student-Employer Match” was created, and the Grad Career team was born.
Opportunity
What we knew to be true is that both sides of the labor market are struggling. Better signaling is critical, now more than ever.
How might we seamlessly connect job seekers with positions that align to their education AND their skills, empowering them to realize their full potential, while simultaneously providing employers with the tools to identify and make informed skill-based hiring decisions?
Impact
Value of Early Research: Our research revealed a crucial truth: the initial Grad Career concept, while technologically innovative, lacked a strong problem-solution fit. User data indicated the product didn't effectively address needs of our target audiences.
Improved Product Direction: Based on research insights, a data-driven pivot was recommended. Senior leadership heeded this advice, prioritizing further exploration towards a superior solution currently in pilot.
Avoided Costly Mistakes: By identifying this issue early, we steered leadership away from potentially significant development costs associated with a product that wouldn't resonate with users.
Increased Trust in Research: The success of this project garnered recognition at the highest levels, highlighting the power of early user research and its ability to deliver valuable insights that drive business value.
Collaboration is Key: This successful outcome demonstrates the impact of collaborative efforts across research, market research, and go-to-market strategy teams.
Approach
02.
Overview
Over the course of four months, the Grad Career team - an interdisciplinary agile team - executed a product validation exploration of a concept idea in the higher education-to-workforce space. We set out to understand the problem space, shape the product concept, and find problem-solution fit. Could our product concept offer sufficient differentiated value to end-users and customers?
Problem Discovery
I.
Understanding the landscape: We investigated the state of the market to identify key market trends and begin to understand the problem space that we would be entering. As we progressed into Problem Definition & Validation, so did our understanding of the market, with specific attention to our competitors and their differentiators.
Problem Definition
II.
Exploring the problem space: My team and I defined the problem and our target audience, which allowed us to frame who we were solving for, what they were experiencing, and how they are currently satisfying any unsolved needs.
Problem Validation
III.
Determining how to solve the problem: This stage was critical, as we built our problem hypotheses and product assumptions in order to begin experimenting. This enabled us to craft strategies around how to approach the problem, to execute, and to begin gathering evidence around what works and what works best.
Product Validation
IV.
Validating for problem-solution fit: We tested if the product idea could solve the critical problem at hand. Testing the product concept for validity, we were able to refine our differentiators and our potential business models.
At the closing of our exploration, reflecting on the data we had collected, our understanding of the problem space and market landscape, and our assessment of problem-solution fit, we had to ask ourselves a critical question: pivot or persevere?
Research
03.
I. Problem Discovery
According to an analysis by the Federal Reserve Bank of New York, recent college graduates face greater employment challenges post-pandemic than the general population. The Economic Policy Institute reported that 43 percent of college graduates were underemployed in 2020, while a report from Strada Institute for the Future of Work and the Burning Glass Institute also revealed that 52 percent of four-year college graduates are underemployed just one year after graduation.
On the hiring side, employers faced difficulties attracting and retaining the right candidates. Hiring managers and talent teams recognize the need to prioritize the quality of hiring practices to find better candidates and improve retention, with 62 percent of recruiters indicating that improving quality of hire as top priority in the Employ Quarterly Insights Report 2022.
What was undeniable to the Grad Career team was that the labor market was chaotic, and that both sides faced unique challenges that affected the other.
II. Problem Definition
Stakeholder Engagement: To ensure alignment and manage expectations, I co-facilitated a kick-off stakeholder workshops alongside the Product Owner (PO). These workshop sessions were instrumental in uncovering internal assumptions about the problem space, customer base, major needs, and the initial product concept. The value extended beyond clarity on project outcomes; it also established a realistic understanding of what we could achieve within the project scope.
Identifying our target audience: My exploratory research into the problem space began with setting out to identify and understand our target audience. Leveraging our early market research learnings and assumptions unearthed from our stakeholder workshops, the team empathy mapped for the following segments: learners & job seekers, employers & recruiters, and HEI’s Career Service Centers.
We want to learn about
Learners
What unmet needs do undergrads and grad students have, related to career exploration?
How do learns & jobseekers identify and improve gaps in skills required for occupations of interest?
Employers
Across industries, what are the greatest gaps in skills for candidates and new hires?
How effective are current methods of recruitment, screening, and hiring to employers?
Career Services (HEI)
How satisfied are Career Services with their current processes of helping students explore career interests and find opportunity?
In what ways do the tools Career Service Centers use serve them well?
Unlocking user needs: I launched a user survey with our learner population to understand current needs, goals, and behaviors. For our first user outreach activity, I leveraged ETS’ test-taking population and recruited GRE test takers as well as TOEFL test takers who had taken the TOEFL for an undergraduate program within the last two years, no renumeration. By focusing on learners that were still in undergrad, especially those on the cusp of graduation, we wanted to hone in on a critical moment in the learner journey: the ‘Exploration stage’, the moment after initial awareness where a learner begins to contemplate their future, but before any initial decisions are made and a path is pursed.
Learner Study
Study
User Surveys with Learner population via Qualtrics, to understand needs and behaviors
Sample
GRE and TOEFL test takers, in undergrad program at freshman-junior year level (n=133)
Sample Questions
What skills are you most interested in improving in yourself for academic/career success?
How willing would you be to take additional courses for specific skills of interest to target employers after completing your education?
What do you want to be doing in 5 years?
What are your main concerns about meeting this 5 year goal?
Shaping problem hypotheses: At this stage in our product concept validation journey, I played a crucial role in facilitating a cross-functional team exercise. We utilized the Lean Canvas framework to identify our riskiest assumptions, which we then transformed into working hypotheses. However, we didn't stop there. For each target audience, I guided the team in building experiment templates that detailed the methodology for testing each hypothesis. This involved not only defining our success criteria, but also determining the most efficient and accessible validation methods given our resource and time constraints. This focus on building testable hypotheses and corresponding experiments ensured a data-driven approach to defining the core problem and its potential solutions.
III. Problem Validation
Understanding the Learner Journey: To truly understand the learner experience, we embarked on a journey mapping exercise. This process wasn't just about documenting touch points; it was about making intangible experiences visible and empathizing with the student perspective. Through this exercise, we were able to identify key pain points, pivotal moments in their journey, and experience gaps that our solution could potentially address. One of the most pivotal moments we’d isolated was the Initial Decision stage, which is when the learner commits to one path. This stage was explored further at the Product Validation stage as we transitioned from discovery to design; we mapped out the potential paths a learner could walk at the Initial Decision stage, what is most influential in their decision making, and what the touch points they come across when committing down said path.
Market Evaluations: I collaborated closely with our market researcher to conduct a comprehensive analysis of the industry landscape. This included identifying not only direct competitors but also potential partners and complementary solutions. By uncovering the competitive landscape and potential synergies, we were well-positioned to develop a differentiated value proposition. This involved crafting a clear and concise concept statement that effectively communicated the core benefit of our product. Additionally, I played a key role in refining our value proposition, ensuring it resonated with both learners and our target market segments. This comprehensive market evaluation ensured our solution addressed a genuine need and offered a compelling value proposition in the competitive landscape.
IV. Product Validation
Prioritization and Solution Development: At the crosspoint of discovery and design, I organized and facilitated strategic workshops with my core team. The first exercise was a deeper dive into our learner audience’s journey. We wanted to better understand the potential touch points along their path and where the greatest opportunities between ETS and said audience truly lied. I mocked up an original activity, inspired by traditional design thinking principles, to organize these opportunity moments and brainstorm ways to reach students effectively. This exercise not only defined the voice for our concept description and value proposition but also set the stage for feature prioritization. Leveraging our research findings and the touchpoint exercise, we then prioritized features crucial for our minimum viable product (MVP), the "Student-Employer Concept." These prioritized features were then translated into solution concept sketches, informed by both market research and user insights.
Validation through Testing: To validate our hypotheses, we tested these concept sketches with learners through a 3-day discussion board. We also conducted testing with key stakeholders – Directors of Recruiting (employers) and Directors of Career Services (Higher Education Institutions). Feedback from these tests fueled iterative refinements, strengthening the product concept's vision, value proposition, and business model. This project showcases the value of user-centered design. By marrying design thinking with research and hypothesis testing, we were able to develop a communicable concept addressing the needs we heard from both learners and employers.
LEARNERS
Concept Test
Overview
Learners participated in a three-day online discussion board in late January of 2022.
Activity required learners to analyze concept description and MVP, and provide feedback on concept
Sample
N=18 learners across three learner segment types, and from diverse backgrounds. Industries of interest were Business, Education, Engineering, Healthcare, and Social Services.
Note: we defined ‘Learners’ as students or employed individuals who are nearing the end of their educations and have yet to embark on their careers
LEARNERS
Concept Feedback
The Student-Employer Match concept is not seen as unique amongst Learners. It’s perceived as ‘just another tool used to search for jobs and investigate careers’. Training programs and soft skills evaluations are not desired enough; that is, without push from hiring side of market, learners do not prioritize these offerings.
Learners had a positive reaction to concept overall because it unites interests with career paths and positions
The platform is seen as being similar to other existing tools. Although nice seeing many tools in one place, this is a minimal benefit and not an important differentiator
Little interest in online training programs, especially in soft skills. Belief is that best training will come from grad programs or directly from future employer.
Learners were less interested and trusting of automated capabilities as concept revealed potential for AI tools and functions.
Progression of MPV Product Concept
Round 1: Ideation Sketches
Flows & Opportunities
Round 2: Early Concept
Concept test
Round 3: Iterated Concept
Updated to reflect feedback & market input
Final Findings
Our market learnings and user insights revealed a critical need to refine the product concept's direction. While we identified a critical problem worth solving for, we could not definitively prove that our proposed solution truly addressed it, nor did we find sufficient evidence of user willingness to pay. Desirability was proven and validated, but we were not able to prove viability; feasibility, which would fall under solution-validation, was not investigated.
Therefore, we recommended a pivot to leadership to achieve a stronger problem-solution fit and better align with market needs, business needs, and our overall mission. Beyond the pivot, we identified the need to strengthen the value proposition across all user segments, highlighting our unique offerings. This led to a reassessment of the assumed business model, recognizing the crucial role of employers and the importance of strategic partnerships in this competitive landscape.
Conclusion
04.
Impact
Our recommendation to pivot was accepted by leadership and the Grad Career product concept shifted in strategic direction. Our team’s research, insights, and final recommendation informed the development of Apprize, an ETS innovation that “connects employers interested in skill-based hiring with job seekers looking to level up” (in pilot as of 2023).
Overall, this case study emphasizes the significant value of conducting quality research early in the product development lifecycle. By leveraging insights from comprehensive user research, market research, and psychometric research, we not only identified a potential product misstep, but also saved the company valuable time, cost, and risk. This success story fosters trust in the research process and its ability to drive better business decisions.
Reflections
This project at ETS stands out as one of my most rewarding experiences. Collaborating within a cross-functional team, I had the unique opportunity to work with researchers from market research and psychometrics. By integrating insights from all these areas alongside user research, we ensured evidence-driven investigation and decision-making that truly aligned with the needs of the target population.
Reflecting on the project, there's always room for improvement. With more time and resources, I would have liked to conduct further concept testing, such as sequential monadic or comparative testing. This would have allowed us to refine future iterations of the concept and address identified challenges even more effectively.
Ultimately, I'm incredibly proud of my team. Our curiosity, determination, and willingness to push boundaries were key. We actively sought support from leadership and were committed to thorough documentation and evidence organization to ensure the integrity of our work. Our success hinged on this evidence-led approach, empowering us to confidently propose a directional pivot, even when it challenged traditional business perspectives.