User Testing: Our Team’s Process for Virtual and In Person User Testing

Photo of post-its on clear board, by rawpixel-com-284722 from unsplash.com

User testing is something that we feel strongly about at Raizlabs. Both design and development teams conduct user tests to determine how target users will respond to and interact with a product. We’ve been designing and building products since 2003, thus user testing isn’t something that’s new to us, but we’re constantly trying to come up with new ways to organize and execute our process. A part of our process includes sharing how we work with both our clients and the design and development communities. Below is our example setup for in-person, moderated user testing.

Sourcing Users

There are many ways to source users for rounds of testing. In this post, we’ll talk specifically about:

Userinterviews.com

What is it?

Userinterviews.com is a website that allows you to source users from your area for in person or remote testing.

  • Userinterviews charges $20 per sourced tester and you need to give an extra incentive pay to each user (gift cards work well!).

How it works

  1. Set-up screener questions and select times you are available. (leave time in between tests as a buffer in case you go over and to greet the next tester – at least 15 minutes. Also, request more testers than you need incase someone cancels.)
  2. It goes live and people will go through the screening questions.
  3. You will receive a list of screened possible testers, and then you can approve them. 
  4. Approved participants will be notified and select a time – their status will then become “CONFIRMED” on the dashboard.
  5. Userinterviews will give you a schedule with contact information & demographics for each participant.

Benefits of Userinterviews

  • Includes Screener Questions
  • Vetted Participants
  • Automatic Scheduling
  • Users are charged after the interview is complete (no charge for no shows)
  • Sources users within a few days

Recording

Usertesting.com

What is it?

Usertesting.com is a website used to conduct unmoderated, remote testing. They also have mobile technology (which we helped build here at Raizlabs). While the user goes through the test, it records audio and the screen with touch points – this allows you to see and better understand the actions users are taking with your product.

Setup for In Person testing

screenshot of usertesting.com showing usertesting panel and My Recruit

  1. Create a new test and select “My Recruit.” This will create a link that you can send to specific people.
  2. When setting up the test for moderated testing, don’t worry about filling out the tasks – we wrote, “listen to moderator” (the user will never even see this). Questions will be asked by moderator rather than UserTesting.
  3. It will generate a link – send the link to an email address that you can access on the phone that will be used for testing.
  4. Before the interview, click the link in the email and login (you will need a different email for every test, so send as many beforehand as you’ll need). 
  5. Click through the initial onboarding of the test. Hit the “Start” button (recording will begin) and then pause the recording. Load InVision. – Leave like this until your user comes in.
  6. When your user is ready, hit the “Resume recording” button and record your initial background questions, even if they aren’t interacting with the app.
  7. When the user test finishes, conclude the test on the phone, and you are done!

Benefits

  • All user testing information is cataloged in one place (Raizlabs often uses UserTesting for unmoderated testing)
  • Shows touch points in recording
  • Records audio and screen

Technical Setup

  1. Create prototype in InVision (or the design software of your choice).
  2. Test prototype through Usertesting.com to record and save information.
  3. If you want others to listen live:
    -Hook up the device to a laptop and use Quicktime to show the device’s screen on your laptop.

    – Setup Google Hangouts and screenshare the window to broadcast to outside listeners. (We recommend that you only have 1-2 people in the room with the user to ensure a clear recording. Everyone else can listen at their desk with headphones.)

Physical Setup

  • Print the following:

    • Consent forms – Draw up and send these to users ahead of time so they can read through them! Having an email connection with the users before the test day is also a great way to start building a solid foundation for rapport.
    • Signs to make navigation the day of the test easier. We usually have these in the elevator banks, leading to testing rooms, to the restrooms, and most importantly, on the doors of testing rooms so that testers aren’t disturbed.
    • Surveys – These will differ depending on the product being tested, but we typically ask users to strongly agree or strongly disagree with key statements about the product in general and about key features, e.g. “I found this app too complex” or “I would use this feature.”
    • List of testers to give to front desk – this is an important organizational element. Not only does it make the process of the users getting to the office easier (through the lobby security) but also helps to keep things on schedule throughout the day. Feel free to give blocks of time for each tester a bit of padding so your team can organize thoughts before the next user arrives.
  • Materials

    • Laptop
    • Charger
    • Phone (big screen preferred)
    • Lightning cord ( or whatever is needed to connect phone to laptop)
    • Phone stand
    • Clipboard for surveys
    • Pens for surveys and consent forms
  • Buy

    • Payment for users (we use Amazon.com giftcards)
    • Water bottles for users
    • Snacks to grab
  • Other:

    • Sticky notes, pens, & dry erase would be smart for outside listeners to start generating thoughts for affinity diagrams.
    • Business cards, in case any of the users have follow-up questions.

Things we’ve learned from doing this

  • Send a personal email the day before testers come in. It was nice to have some responses and connections to the people coming in – and remember, be friendly!
  • Send consent forms prior (as well as having them in person) so you don’t waste time with them having to read it during your session. Testers also need a heads up so they aren’t asked to sign blindly.
  • Schedule AT LEAST 15 minutes between users to allow for transition time, running over, etc. This is the padding time we mentioned earlier.
  • Have a person other than the moderator be ready to greet and walk people out incase they go overtime is helpful. The 2nd person can also give out a survey and do a closing talk with testers as they leave.
  • 33% of our users canceled, so schedule a few more testers than you need (more data is better than less!).

Make sure you

  • Have the moderator listen to all of the recordings to take notes
  • Have all listeners write quotes, findings, pain points on sticky notes
  • All Listeners come together with sticky notes and create an affinity diagram (even if it’s just 1 listener!)

    • Group stickies with common themes. For example, we uncovered patterns in people’s thoughts about trust within a recent app we tested, as well as self reflection.
  • Talk about each sticky and groupings as a team . This creates good discussion and gets everyone on the same page.
  • Create a writeup

This is a helpful exercise when there is so much information and lets you compare tests in a more visual and interactive way to find patterns.

If you and your team utilize other tools or practices as part of your user testing process, we’d love to hear from you. Feel free to contact our Design team via Dribble or our business development team to learn more about our design services.


Do you have a project in mind? We’d love to work with you. If you’d like an opportunity to work on projects with us, check out our Careers page. We’re hiring!