What is SUMI?

The Software Usability Measurement Inventory is a rigorously tested and proven method of measuring software quality from the end user's point of view. This used to be called simply user satisfaction but in fact, since when it first came out in the 1990s, SUMI has always actually been measuring user experience.

SUMI defined user experience with work-based software products in 1995! SUMI uses a rigorous scientific method of analysis and is backed up by over 25 years of industrial application.

SUMI is supported by an extensive reference database and an analysis and report generation tool called SUMISCO.

Take a look at the detailed article about how SUMI was constructed. If you put "SUMI questionnaire" into any decent search engine you'll find many pages about how SUMI has been promoted by many orgainsations, and used in applications and research.

Point of contact for SUMI

SUMI is the brainchild of Dr Jurek Kirakowski who can be contacted at jzk@uxp.ie.

Who should use SUMI?

SUMI is recommended to any organisation which wishes to measure the perceived quality of end-user experience of software. Organisations may be:

• developers,
• consumers of software,
• purchase consultants,
• usability or user experience consultants.

SUMI has also been used to set user experience requirements by software procurers in a testable manner.

It has been well documented that if staff have quality tools to work with, this contributes to overall efficiency of staff and the quality of their work output. In the domain of business systems, low SUMI Global scores have been shown to increase operating costs due to loss of staff satisfaction and poor effectiveness and efficiency.

SUMI has been used effectively to:

• assess new products during product evaluation
• make comparisons between products or versions of products
• set targets for future application developments.

SUMI has been used specifically within development environments to:

• set verifiable goals for user experience
• track achievement of targets during product development
• highlight good and bad aspects of an interface.

SUMI is the de facto industry standard questionnaire for analysing users' experience of internet and desktop software applications. It is backed up by a report generator which refers to a large standardisation database. SUMI isn't just a questionnaire. It's a service. Take a look at an example report generated by SUMISCO

Why use SUMI?

SUMI is the only commercially available questionnaire for the assessment of user experience with software which has been developed, validated, and standardised in a wide selection of languages. Each language version has been carefully translated, back translated, and validated by teams of native speakers of the target language.

SUMI enables measurement of some of the user-orientated requirements expressed in the European Directive on Minimum Health and Safety Requirements for Work with Display Screen Equipment (90/270/EEC).

SUMISCO is based on profiles of over two thousand responses to commercially available software. This standardisation base is updated annually. The SUMISCO report will tell you how your product compares to this standardisation base: whether you are about average for the market, below, or above. Because of the background statistics in our database, you can find out very precisely how you compare to the rest of the market.

What does SUMI look like?

Take a look at the English language questionnaire online at sumi.uxp.ie/en/. You're invited to try it with the throw-away password "999".

SUMI consists of 50 statements to which the user has to reply that they either Agree, Don't Know, or Disagree.

Here are some example statements:

Item No. Item Wording
1.       This software responds too slowly to inputs.
3.       The instructions and prompts are helpful.
13.      The way that system information is presented is clear and understandable.
22.      I would not like to use this software every day.

What does the client get from SUMI?

The SUMISCO report generator is the interface to the extensive standardisation data base developed for SUMI and which is regularly up-dated and maintained.

Take a look at the report generated from an anonymous example. This is sent to the client as a zipped, encoded file when the evaluation has been completed.

What languages is SUMI translated into?

Currently we have as follows although some language versions are not yet online at UXP:

Each has been carefully translated and validated in the target language by several native speakers of the language working in a back-translation paradigm. Please do not attempt to translate SUMI into another target language without first consulting Dr Kirakowski.

Constructive critique of the wording of SUMI items in translation is also greatly appreciated.

How many respondents are required?

Online SUMI might require sample sizes with a minimum of about 20 unless your respondents are well selected. This is because anonymous online samples tend to be less well controlled than samples which you have recruited in person.

However, we know that a personally recruited SUMI will give you reliable results with as few as 12 respondents. This is because you are able to control the quality of your sample.

You can use fewer repondents if you wish, but beware that your results may not be as representative of the true user population. In fact, SUMI has yielded useful information with sample sizes of four or five.

However, this question is a bit like 'how long is a piece of string?' Basically, you should try to get as many respondents as you can within your timeframe and budget.