4b needs tweaking: Cimex Usability Study on Skype 4.0 Beta 1

Tuesday, July 29th, 2008 7:34 pm from Skype Journal by Phil Wolff
Read on original site

Topics: Developer Zone, Developers, Marketing, Technology, design

How do you discover what people really do with your software? Our behavior often differs from what we think we do. The human-computer interaction research method eye tracking can reveal where a user’s attention goes when they use a product.

Two days after Skype launched the Skype for Windows 4.0 Beta in June 2008, Cimex, a British firm with a usability and accessibility lab, spent a day watching people use Skype. Then wrote them up. I’m including the full text of their report below the fold. From their blog:

A usability study by Cimex has revealed Skype’s new Beta release may prove difficult to use, due to weaknesses in presentation, navigation, structure and functionality.

The aim of this study was to capture the users’ expectations, needs and preferences, all of which we believe can be used as a base to create a more engaging, efficient and effective Skype application.

Caveats:

Only six users were tested: 25-50 years’ old; British, French, Greek; three newbies, three experienced Skypers; three men, three women. This is not representative of Skype’s worldwide user community. Skypers come in all ages, speak many languages and bring wildly different cultural norms to software use.

The team said "A larger sample would allow us to perform more detailed quantitative data analysis of the performance rates." A larger sample would also "allow us to create more targeted scenarios and cover all the range of Skype user profiles."
The study was relatively shallow since few user goals were tested in the time available, and many product features were unexplored. See the sidebar for four user goals.

Four of the twelve tasks used in the study

Could you please have a look at this welcome screen and tell me what your first impressions are? What would be the first thing you would like to do from here? (Users were allowed to explore the options of the welcome screen and then asked to start using the application.)

What are you impressions by looking at this page of the application? Do you like the way the information is presented?

Let’s have a look at the top menu (Skype, Contacts, Conversation, etc). Are these options meaningful enough to you? Without clicking on them yet, could you please tell me what do you expect to find under each of these menu options?

Imagine now, that you want to invite a friend who hasn’t got a Skype account, to join Skype. How would you do that?

Observing users with eye tracking is best applied iteratively and with a specific problem in mind. Feedback from one round can reveal a problem which, once cleared, reveals other obstacles to people doing what they set out to do.

I’d be curious how these results compare with the 3.8 release; has 4’s redesign addressed many of 3.X’s problems with complexity and discovery?

Unlike many other products, Skype comes in many flavors, like the mobile edition on the Skypephone or Skype for the Nokia tablets. You might learn a great deal by testing goal success (e.g. "Call this phone number", "IM a friend on your buddy list", "start a video call") across operating systems and devices.

So here’s the Skype Journal Daily Question: What are the top three design goals you see in Skype 4b?

Skype 4.0 Beta usability review

1. Introduction

This report contains the results of a short series of user testing sessions, carried out by Cimex on the new version of Skype, namely the Skype 4.0 Beta 1 release. Our brief usability study has revealed many of the difficulties users faced, due to weaknesses in the presentation, navigation, structure and functionality of this widely used application.

Figure 1. The first page of Skype application.

2. Aims and Objectives

The aim of this study was to capture the users’ expectations, needs and preferences all of which can be used as a base to create a more engaging, efficient and effective Skype application. The key objectives were to:

Explore users’ experiences and expectations based on their previous use of Skype.
Assess the usability of Skype 4.0 Beta 1 version by users performing typical tasks with the application.
Assess if the information presented and the functionality are appropriate for the target audience.
Capture the users’ reactions and preferences for style and design ideas for the application.

3. Methodology

For the purpose of this study, the qualified User Experience team at Cimex carried out six individual user testing sessions, which took place at the Cimex specialised usability labs in Central London. Each session lasted 60 minutes and consisted of four main areas:

Pre-session interview

The purpose of this first exercise was to make the user feels comfortable with the testing and to gather useful information about the user’s expectations and their previous experience with Skype, or other similar applications which are based on VoIP (Voice translated into data) or IM (Instant messaging) technology.

Task performance with Eye Tracking

In the next part of the session, each participant was asked to complete a set of twelve tasks on Skype, while the eye tracking device recorded their initial eye movements. These were typical tasks based on “real-life” scenarios, for example “Imagine that you want to invite a friend to join Skype. How would you do that?” The user was encouraged to think aloud and share their thoughts as they interacted with the application.

Figure 2. A user testing session at Cimex usability lab

Post-session interview

After the task performance with the eye tracking, the post-session interview was conducted to learn the user’s overall thoughts and conclusions on using the application.

Design

The final section of the session was to gather feedback on the design of the application by capturing the user’s opinion and reactions about the overall “look and feel”, style and colour of the Skype application.

4. Participants

A total of six participants took part in the current testing. The participants were representative of typical Skype users and those who were keen to learn more about Skype. In particular, two of the users had never used Skype before, two had used Skype a few times, and the other two used Skype on a regular basis. The sample included three male and three female, and they were between 25 to 40 years old.

5. Data Analysis

Qualitative analysis was used to analyse the information obtained from the user testing sessions, and gaze plot analysis was used to analyse the data collected with the eye tracker. The data analysis was based on the users’ reactions and comments during the interviews and the interaction with application, as well as on their ability to perform the given tasks.

6. Results

Pre-session Interview

People who had never used Skype before expected to find clear instructions on how to use Skype services, a simple interface and information about the costs.

Free Skype-to-Skype calls and cheap international calls were considered the most popular services by the existing Skype users, whereas conference calls and file transferring were described problematic.

Task Performance

Task performance revealed many important findings which indicated the numerous problematic areas in the presentation, navigation, structure and functionality of Skype 4.0 Beta. Some of the key findings are described below:

Figure 3. Content was overlapped when the window size was decreased.

a. Finding: The size of the application’s window was described as unnecessarily big and the content was overlapped when the users tried to decrease it.

Recommendation: Allow users to select a desirable window size without the content being overlapped.

Figure 4. The conversation window had many unclear icons.

b. Finding: The purpose of many icons was considered not obvious and unclear.

Recommendation: Provide intuitive and distinct icons, and ensure that their purpose is not misinterpreted by looking similar to other buttons.

Figure 5. The search for contacts had no option for advanced searching or sorting the search results.

c. Finding: Users wanted to be able to narrow down the search for contacts and also to sort the search results easily.

Recommendation: Ensure the contacts search facility is effective and provides intuitive advanced search and filtering methods.

d. Finding: It was not clear when a contact was finally added to the user’s list as no confirmation message was given.

Recommendation: Provide appropriate feedback through clearly defined messages to keep users informed about what is going on.

e. Finding: All the users wondered how they could view their credit balance.

Recommendation: Make important information, such as the user’s credit balance, visible and accessible at any point throughout the application. Eye Tracking The image below demonstrates an example scan path extracted by the eye tracking analysis. The blue dots are fixations that indicate where the eyes looked to process information. Larger dots show that more time has been spent looking at the elements on the screen.

Figure 6. An example of an eye tracking scan path for the first page.

f. Finding: During the first seconds of viewing the first page, all of the users’ attention has been attracted by the right area of the screen, without noticing the important options on the left.

Recommendation: Enhance the application’s presentation so that users will not spend unnecessary time and get disappointed while trying to detect the options that will allow them to start using the application.

Post-session interview

Overall the application was considered very useful and easy to use but with a potential for improvement. In general, users wanted a better interface design, clearer navigation mechanisms, and also highlighted the importance of accessing their credit balance at any time. Design Although some users supported that they did not have many design expectations from this kind of applications, a better layout with smaller window size and more intuitive icons was considered necessary by everyone.

Figure 7. Users’ comments on the design of the conversation page.

7. Conclusion

Our usability study demonstrated how the contribution of real users can reveal a number of significant findings and recommendations. The Skype 4.0 Beta 1 was overall described useful and easy to use, but with poor layout, not obvious icons, unclear navigation and information structure. For that reason, an exploration of alternative solutions to the site structure and naming conventions to provide a more intuitive and efficient user experience is considered essential. The outcomes of this study can be used as the foundation for building a more intuitive, engaging and efficient new version of the popular web application.

If you want more information about Cimex’s usability lab, contact:

Andrew Japp, Business Development Manager
Cimex, 53-55 Scrutton Street, London UK, EC2A 4PJ
T: +44 (0)20 7324 7780
E: andrew.japp AT cimex DOT com
www.cimex.com