How successful have we been?

Our project plan identified both qualitative and quantitative measures with which to gauge success.

Quantitative

The learnability investigation was split across two interfaces with discrete audiences, the querying interface ("public") and the XML document editing interface ("editor").


Issue list for 'Public' interface

• 'Users do not understand the difference between querying and searching'
  Issue 1: partially met
• 'There is no method for users to initiate queries using statistical tools'
  Issue 2: partially met
• 'Users accidentally left a previous search filter active when starting a new search'
  Issue 3: partially met
• 'Users expected highlighted terms to extend their current search'
  Issue 4: partially met
• 'Users have no way of creating a citation, nor any help for saving their search'
  Issue 5: met
• 'There is not enough information about the source being searched in the results pages'
  Issue 6: met
• 'Variant spellings currently have to be searched on separately'
  Issue 7: met
• 'There is currently no keyword search function'
  Issue 8: partially met

The fact that the redesigned interface is unrecognisable from the version with which the investigation began indicates that the usability process was able to give strategic direction as well as feedback on finer details such as nomenclature; this is made more powerful by the fact that this was the first time that many users had seen the product.


Issue list for 'Editor' interface

• 'Users need constant feedback when using something new'
  issue 1: unmet
• 'Some parts of articles are repetitively structured and would suit some automation'
  issue 2: met
• 'Users do not check work by looking at mark-up'
  issue 3: partially met
• 'Users need standardised visual cues to use devices which move or transform'
  issue 4: met
• 'Users need to be able to specify whether their comments are intended for publication'
  issue 5: partially met
• 'There is a need to provide guidance that is tailored both to the source, and to the role of the current user'
  issue 6: unmet
• 'Users like to navigate documents quickly using search'
  issue 7: met
• 'Some users will need to be able to track the changes that have been made to a document'
  issue 8: met

You would be forgiven for thinking that an interface which relied heavily on interactive menus and controls would be difficult to investigate using click testing; however, given that all user activity starts with one click, it still needed to accurately signpost the correct entry points for interactivity. Overall, although the exercise created positive results, there were several areas in which the redesign failed; most frustratingly, issue 6, a simple case of inappropriate wording.

Qualitative

It is much harder to report on qualitative feedback – a lot of it is confidential and cannot be 'transcribed'; in addition, the compressed nature of the project plan means that there is little time for changes to take effect and be measured again. As an observation, it is more likely that users will view a project team as acting more deliberately (i.e. with a greater sense of strategy) if they communicate changes in system delivery personally. Interviews and focus group, and to some extent, email surveys can carry this sense of purpose.

Approach

Each primary issue within the public and editor interfaces now has a fixed point within system documentation which can track its development. By opening this up to the general public, further scrutiny of the issues is enabled and the project can appear more confident in its action.

By publishing the life cycle of each issue, the project cannot hide any aspect of development and its decisions are open to the standard academic techniques of review and criticism. It is a deeply honest approach to system design whose success can be judged by the range of people, including non-specialists, who are empowered to comment on it.