User Acceptance Testing for New Jersey Career Seeking App

*User testing portion of UAT

You have a beta version of an application.

Having a user researcher is essential assess users attitudes and accept the minimum viable product (MVP) version of an application.

Project Scope

  • Goals: Identify how New Jersey career seekers who are exploring the best route to reach their career destination interact with the features available on the career exploration application. 

  • Client: New Jersey Department of Labor (NJ DOL)

  • Method: Screener questionnaire, User testing interviews, Satisfaction questionnaire

  • Duration: 12 weeks

  • Participants: 16 New Jersey career seekers; 5 interviewees

  • Recruitment: Useberry application participant pool, email scheduling

  • Tools: Miro, Zoom, Dovetail, site test environment

Background

The application intends to help New Jersey (NJ) career seekers explore the best route to reach their career destination. The application was in its beta version.

I conducted a user testing initiative to identify how career seekers who are exploring the best route to reach their career destination interact with the features available on the application.

User Testing

The user testing initiative was conducted to answer these questions:

  • How do users explore careers and how will NJ Career Pathways help in doing so?

  • Can users effectively complete expected tasks on NJ Career Pathways?

  • What are users’ attitudes towards NJ Career Pathways?

Method

The initiative included a screener questionnaire to recruit participants, user testing interviews, and a post-interview satisfaction questionnaire.

Participants included:

  • Those unfamiliar with NJ Career Pathways

  • Desktop, laptop, tablet, and mobile users were welcome

  • NJ career seekers who are looking to explore and gain a depth of knowledge in careers

  • NJ career seekers who are actively looking for work

  • NJ career seekers who are actively seeking training

Screener Questionnaire

I used Useberry, a UX research platform to screen for NJ residents who were actively looking for careers

Participants were scheduled based on the availability they listed on the screener

I surveyed 16 participants

The screener included participants’:

  • Name and email

  • Career goal experiences

  • Availability

  • Devices typically used to explore careers

Career goal questions included:

  • What do you aim to do in exploring career paths?

  • What careers interests you?

  • How comfortable are you in exploring careers path(s)?

  • Why are you exploring careers?

  • Where are you with your career exploration?

  • What types of information are you exploring?

  • What platforms have you used to find information about career paths? 

  • Have you used My Career NJ to find career opportunities?

Screener Questionnaire Findings

Participants primarily were looking to find job opportunities that match their career interests and explore different career paths.

Participants had varied in the careers they were interested in.

Many were exploring careers because they felt a need to change their career path, wanted to make more money, and felt their job did not use their skills.

(See Screener Findings in PDF)

User Testing Interview Method

I conducted user testing interviews to:

  • Identify the content relevance and intuitiveness of NJ Career Pathways 

  • Determine user preferences in the design and aesthetics of NJ Career Pathways

  • Identify errors user encounter and how responsive NJ Career Pathways performs

I interviewed five NJ career seekers.

User Testing Set-Up Guides

I created user testing set-up guides to help users prepare before the session.

Participants used a desktop/laptop or a mobile device to interact with the application during interviews which was determined by the devices the participants identified having access to on the screener. Participants were provided with set-up guides to allow them to prepare for the interviews.

(See Appendix Set-Up Guides in PDF)

Types of Data Collected from the Testing Session

Expectations provide qualitative data where participants expressed what  they would expect would come of their interaction with the tools within NJ Career Pathways

E.g. “Popular opportunities in healthcare”

Interaction comments provide notes and quotes from participants shared thinking aloud while using of tools within NJ Career Pathways  

E.g. “I almost went past the next section without reading”

Takeaways provide qualitative data summarizing the insights users provided while interacting with NJ Career pathways 

E.g. “Participants liked ‘Salary’ range but aren’t exactly sure how it’s determined”

Recommendations provide concrete examples of changes that can be implemented to improve NJ Career Pathways  after taking into account user insights. 

E.g. “Make occupation box hovering highlight a brighter color to help users notice the box is clickable”

Features in the application to test

  • The host website page that guides you to the application

  • The landing page of the application

  • A dropdown allowing users to select an occupation of their interest

  • A visual display of the progression of their chosen occupation growth

  • The description of the occupation including salary, education and skills

  • Links to support resources such as a job and training posting page

  • Wrap-up thoughts including how to save info and who are the ideal users

Example of a Select Occupation with interaction comments

Example of a Pathways Map with interaction comments

User Testing Interview Recommendations

Organization Logic & Navigation

  • Single click application buttons on the homepage

  • Continue using goal buttons to guide users to explore career

  • Continue to use the Manufacturing version of the drop-down, Grouping sub-fields of in the drop-down was helpful

  • Design tip that suggests mobile users view the map rotated to landscape mode

  • Discuss conducting an A/B test for vertically displayed map

  • Design the ability to click on the occupation boxes to be more obvious with more noticable hover highlights, an instruction, and a suggestion symbol

  • Design the ability to expand the map to be more obvious. Users did not know that they could interact with Pathways Map by clicking on “Expand” and the occupation boxes

  • Discuss how users are intended to interact with the 2-path expanded map

  • Continue manufacturing version of “Education” and “Other Experience”

  • Develop side window link (like the “Learn More”) with “Salary”, “In Demand”, and “Job Count” to better explain these and avoid the tooltip stickiness

  • Ensure related training cards are the top recommended training programs when clicking “See more training on Training Explorer”

Minimalistic Design & Aesthetics

  • Avoid or quickly skim passed large chunks of text  

  • Add images that correspond to the field

  • Displaying graphs of stats of growth in the industry is an interesting idea

  •  Like the visual of the Pathways Map

  • Adding images, colored headers, and the map helped improved users satisfaction with the Manufacturing design vs the Healthcare & TDL design as well as the Related occupation page (In-Demand occupations) and Related Training page (Training Explorer)

  • Update all pages to the Manufacturing version design

Relevance of Results & Information

  • Reduce the Welcome Page description

  • List a few essential education requirements

  • List 2-3 unconventional sectors 

  • Discuss how to include descriptions about how an occupation operates within industries

  • Continue manufacturing version of “Education” and “Other Experience”

  • Discuss having Related Occupations return as they are not included in the Manufacturing design. Related Occupations shoes send you to another occupation page on NJ Career Pathways, not In-Demand Occupations

  • Adding if a training is remote or in-person on the related training card

  • The information is helpful particularly for those who have skills but are missing or unclear about the steps to move up

  • List how NJ Career Pathways will help the target users on the Welcome Page and maybe very briefly on the homepage CP button

  • Continue to add industries and occupations

  • Market NJ Career Navigator’s “Career Paths” as the authenticated experience to save careers. Both NJ Career Navigator’s “Career Paths” and NJ Career Pathways should display the same occupations

  • Provide testimonials from those in the selected occupation, especially if they used Career Pathways to align their career

Error Handling & Responsiveness

  • Develop side window link (like the “Learn More”) with “Salary”, “In Demand”, and “Job Count” to better explain these and avoid the tooltip stickiness

  • Healthcare & TDL versions offer Related Occupations which take you to In Demand Occupations confusing users wanting to click “Back”

  • Ensure Jobs Open in NJ counts match what is listed on CareerOneStop or explain the difference in numbers 

  • Discuss limiting job postings to x number of days on CareerOneStop 

  • Ensure all occupations consistently display related training cards or note “No recommended trainings available”.

  • Ensure related training cards are the top recommended training programs when clicking “See more training on Training Explorer”

  • Users were able to return to selected occupation easily when they accidentally clicked off or back

Post-Interview Questionnaire

I used Google Forms, a survey platform to evaluate participants’ satisfaction and final impressions using the application

Participants were tasked to complete the questionnaire after their interview session.

The five participants competed the questionnaire

The questionnaire included participants’:

  • Satisfaction using the application

  • Belief of helpfulness using the application

  • Comfort using the application

  • Likeliness to interact with the application again

  • Likeliness to recommend with the application to others

Post-interview questionnaire questions included:

  • How satisfied were you in navigating to NJ Career Pathways from the homepage?

  • How helpful do you think the career pathway map is in helping you explore within that career path?

  • How helpful do you think the description information about the career is in helping you explore careers?

  • How helpful do you think the links that were available are in helping you explore careers?

  • How effective do you think the NJ Career Pathways tool would be to help you explore careers?

  • How confident were you in using the NJ Career Pathways tool?

  • How likely are you to use the NJ Career Pathways tool?

  • How likely would you recommend our NJ Career Pathways tool to other people?

  • How likely would you recommend our NJ Career Pathways tool to other people?

Post-Interview Questionnaire Findings

Participants were generally satisfied with interacting with the application’s features and found the application helpful.

Participants were generally likely to use the application and very likely to recommend the application to others

(See Post-session Questionnaire Findings in PDF)

Conclusion

The insights from the user testing initiative resulted in some immediate changes such as hover shadows, tooltip adjustments, and improving related training cards. After immediate changes the MVP version was accepted for production.

The initiative also fostered product discussions and further research initiatives and design thinking workshops surrounding the Pathways Map, redesigns of the Welcome and Learn More pages, and future states of the application.