No discussion of the transformation of system design is possible at the moment without recourse to the severe nature of the research funding environment; currently, the recession has replaced necessity as the mother of invention. Given this, what practical change could take place?
System development could take a more open, public approach, and usability techniques could prove extremely useful here as they enable interaction with users in multiple ways, from surveys and interviews to blog postings of results. This would create more information which a project could publish through its lifecycle rather than aggregating and (to a certain extent) generalising in a final report.
Publishing the project plan and outcomes provides a useful starting point for the project; however, the language used in them will likely come directly from the original application for funding which will be highly specialised and reliant to a certain degree on jargon and technical understanding. Far more preferable would be to ensure that all public project communications could be accessible to as wide an audience as possible (this should include, of course, budget holders within the projects' own organisation).
To further exploit the work undertaken, an open library of evidence collected at first hand could be created from each project. Put simply, if a project undertakes a survey of, say, 100 researchers in a specific academic discipline then that survey must be made available for re-use. The same concept could be applied to interviews, focus groups or transect walks provided that data protection legislation was adequately followed. This would make available a knowledge base upon which other projects could instantly draw for informing their own usability strategies. Furthermore, with much research being ploughed into text mining, the first hand evidence materials here would form an excellent candidate for approaches such as sentiment analysis.
By applying research practice, this field could be transformed from one predominantly made up of narrative reflections of practitioners to a data store (perhaps accessible via an API) on which projects could draw for the modelling and evolution of user behaviour; thus corroborating or challenging results, and even for constructing funding applications (for instance, if particular sections of the academic community have not been adequately served or covered in prior research).
Showing posts with label usability. Show all posts
Showing posts with label usability. Show all posts
Wednesday, 14 March 2012
Monday, 12 March 2012
How successful have we been?
Our project plan identified both qualitative and quantitative measures with which to gauge success.
Issue list for 'Public' interface
The fact that the redesigned interface is unrecognisable from the version with which the investigation began indicates that the usability process was able to give strategic direction as well as feedback on finer details such as nomenclature; this is made more powerful by the fact that this was the first time that many users had seen the product.
Issue list for 'Editor' interface
You would be forgiven for thinking that an interface which relied heavily on interactive menus and controls would be difficult to investigate using click testing; however, given that all user activity starts with one click, it still needed to accurately signpost the correct entry points for interactivity. Overall, although the exercise created positive results, there were several areas in which the redesign failed; most frustratingly, issue 6, a simple case of inappropriate wording.
By publishing the life cycle of each issue, the project cannot hide any aspect of development and its decisions are open to the standard academic techniques of review and criticism. It is a deeply honest approach to system design whose success can be judged by the range of people, including non-specialists, who are empowered to comment on it.
Quantitative
The learnability investigation was split across two interfaces with discrete audiences, the querying interface ("public") and the XML document editing interface ("editor").Issue list for 'Public' interface
- 'Users do not understand the difference between querying and searching'
Issue 1: partially met - 'There is no method for users to initiate queries using statistical tools'
Issue 2: partially met - 'Users accidentally left a previous search filter active when starting a new search'
Issue 3: partially met - 'Users expected highlighted terms to extend their current search'
Issue 4: partially met - 'Users have no way of creating a citation, nor any help for saving their search'
Issue 5: met - 'There is not enough information about the source being searched in the results pages'
Issue 6: met - 'Variant spellings currently have to be searched on separately'
Issue 7: met - 'There is currently no keyword search function'
Issue 8: partially met
The fact that the redesigned interface is unrecognisable from the version with which the investigation began indicates that the usability process was able to give strategic direction as well as feedback on finer details such as nomenclature; this is made more powerful by the fact that this was the first time that many users had seen the product.
Issue list for 'Editor' interface
- 'Users need constant feedback when using something new'
issue 1: unmet - 'Some parts of articles are repetitively structured and would suit some automation'
issue 2: met - 'Users do not check work by looking at mark-up'
issue 3: partially met - 'Users need standardised visual cues to use devices which move or transform'
issue 4: met - 'Users need to be able to specify whether their comments are intended for publication'
issue 5: partially met - 'There is a need to provide guidance that is tailored both to the source, and to the role of the current user'
issue 6: unmet - 'Users like to navigate documents quickly using search'
issue 7: met - 'Some users will need to be able to track the changes that have been made to a document'
issue 8: met
You would be forgiven for thinking that an interface which relied heavily on interactive menus and controls would be difficult to investigate using click testing; however, given that all user activity starts with one click, it still needed to accurately signpost the correct entry points for interactivity. Overall, although the exercise created positive results, there were several areas in which the redesign failed; most frustratingly, issue 6, a simple case of inappropriate wording.
Qualitative
It is much harder to report on qualitative feedback – a lot of it is confidential and cannot be 'transcribed'; in addition, the compressed nature of the project plan means that there is little time for changes to take effect and be measured again. As an observation, it is more likely that users will view a project team as acting more deliberately (i.e. with a greater sense of strategy) if they communicate changes in system delivery personally. Interviews and focus group, and to some extent, email surveys can carry this sense of purpose.Approach
Each primary issue within the public and editor interfaces now has a fixed point within system documentation which can track its development. By opening this up to the general public, further scrutiny of the issues is enabled and the project can appear more confident in its action.By publishing the life cycle of each issue, the project cannot hide any aspect of development and its decisions are open to the standard academic techniques of review and criticism. It is a deeply honest approach to system design whose success can be judged by the range of people, including non-specialists, who are empowered to comment on it.
Wednesday, 21 December 2011
'Some users will need to be able to track the changes that have been made to a document': ReScript editing issue 8
Page section
Article editor
Heuristic
Memorability
Description
Some users (for instance editors) will need to assess other users' work and will need to compare different versions of the same document.
Impact severity
High
Recommendation
Insert links to the entire version history of a document, in reverse chronological order.
Examples
The transcription of a document will lead to certain decisions being made by the user on structure as well as mark-up. A version feature would enable editors to see how users have interpreted guidance, and provide practical feedback for future work.
Quantitative measure
Actual question
Development change
Inclusion of a folder tab entitled 'version history', appearing third in the folder structure, and giving links to previous versions.
Reflections
Article editor
Memorability
Description
Some users (for instance editors) will need to assess other users' work and will need to compare different versions of the same document.
Impact severity
High
Insert links to the entire version history of a document, in reverse chronological order.
Examples
The transcription of a document will lead to certain decisions being made by the user on structure as well as mark-up. A version feature would enable editors to see how users have interpreted guidance, and provide practical feedback for future work.
 |
| Figure 8—1: we learn from our mistakes, and currently there is no way for a user to see the exact changes which an editor has made on one of their documents. |
Click where you would expect to access and view a previous version of this document
Actual question
Click where you would expect to access and view a previous version of this document
Initial click test result ('before')
November 2011: 120 responses.
 |
| Figure 8—2: before |
Inclusion of a folder tab entitled 'version history', appearing third in the folder structure, and giving links to previous versions.
 |
| Figure 8—3: new folder tab |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 8—4: after |
Clear grouping of clicks on the appropriate folder tab, objective has been met. The alignment of question wording with interface seems to improve the chance of click test success - however, this wording is generated from qualitative interviews therefore it is fair to assume that by institutionalising it at an early stage, we have made the process more robust.
Tuesday, 20 December 2011
'Users like to navigate documents quickly using search': ReScript editing issue 7
Page section
Article editor
Heuristic
Learnability, memorability
Description
At times, users need to locate specific parts of sometimes quite lengthy and heavily marked-up documents, either to check the style of mark-up employed, or to look for a particular phrase which might be a candidate for mark-up. They may also wish to have a quick way of comparing the mark-up of matching items. Without a search function, this is problematic.
Impact severity
High
Recommendation
The inclusion of a search function in the control pane, with matches highlighted.
Examples
Parliamentary texts may continually refer to the same individuals, and users may choose to find all instances and mark them up consecutively.
Quantitative measure
Actual question
Initial click test result ('before')
November 2011: 120 responses.
Development change
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
Using search is a necessary skill to add sohphistication into web browsing; users recognise and understand it as a function and the click groups above show that the revised interface is resoundingly clear in meeting this objective.
Article editor
Heuristic
Learnability, memorability
Description
At times, users need to locate specific parts of sometimes quite lengthy and heavily marked-up documents, either to check the style of mark-up employed, or to look for a particular phrase which might be a candidate for mark-up. They may also wish to have a quick way of comparing the mark-up of matching items. Without a search function, this is problematic.
Impact severity
High
Recommendation
The inclusion of a search function in the control pane, with matches highlighted.
Examples
Parliamentary texts may continually refer to the same individuals, and users may choose to find all instances and mark them up consecutively.
 |
| Figure 7—1: the more heavily the text is marked up, the less easy it becomes to scan for key words. |
You are looking for the name “Wallworth”. Click where you would expect to find a function which allowed you to search for this name within the current document.
Actual question
You are looking for the name “Wallworth”. Click where you would expect to find a function which allowed you to search for this name within the current document.
November 2011: 120 responses.
 |
| Figure 7—2: before |
Inclusion of a search function limited to the editing window, appearing second in the new toolbar.
 |
| Figure 7—3: matches within the text are highlighted and the document position is automatically moved to the first match |
December 2011: 90 responses.
 |
| Figure 7—4: after |
Using search is a necessary skill to add sohphistication into web browsing; users recognise and understand it as a function and the click groups above show that the revised interface is resoundingly clear in meeting this objective.
Monday, 19 December 2011
'There is a need to provide guidance that is tailored both to the source, and to the role of the current user': ReScript editing issue 6
Page section
Article editor
Heuristic
Memorability
Description
Guidance about the source itself will be essential for all users. In addition, different levels of guidance will be needed depending on whether the user is editing or transcribing. This guidance should extend into practical advice about mark-up, preferably including worked examples.
Impact severity
High
Recommendation
Creation of a persistent page object into which guidance may be tailored to both the source and role of the current user.
Examples
The names of some items are likely to be somewhat generic or esoteric depending on the source, e.g. event in the Alumni Oxonienses draws together related details within education and occupation.
Quantitative measure
November 2011: 120 responses.
Development change
Creation of a tab folder, entitled 'About the source' for editorial guidance. Easy to switch between guidance and editing.
Article editor
Heuristic
Memorability
Description
Guidance about the source itself will be essential for all users. In addition, different levels of guidance will be needed depending on whether the user is editing or transcribing. This guidance should extend into practical advice about mark-up, preferably including worked examples.
Impact severity
High
Recommendation
Creation of a persistent page object into which guidance may be tailored to both the source and role of the current user.
Examples
The names of some items are likely to be somewhat generic or esoteric depending on the source, e.g. event in the Alumni Oxonienses draws together related details within education and occupation.
 |
| Figure 6—1: there appears to be no linear flow for the editing process meaning it is difficult for people to know how and where to start |
Quantitative measure
Click where you would expect to find guidance on how to mark up the entity "East india Company"
Actual question
Click where you would expect to find guidance on how to mark up the organisation "East india Company"Initial click test result ('before')
November 2011: 120 responses.
 |
| Figure 6—2: before |
Development change
Creation of a tab folder, entitled 'About the source' for editorial guidance. Easy to switch between guidance and editing.
 |
| Figure 6—3: new folder tab structure enables guidance to appear alongside editing function without changing pages |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 6—4: after |
Reflections
This issue remains unmet as there is no particularly strong concentration of clicks on the 'About the Source' folder tab, and the generic Help link in the top right attracts a similar amount of attention in both versions.
On reflection, this may be a simple labelling issue. 'About the source' is a general term but inappropriate in this context. Results would likely have been better if it were titled 'Mark-up guidelines', 'Editing guidelines' or even 'How to edit this'. This is a frustratingly small error at the final execution phase which has led to the issue being unmet and should have been resolved by QA.
Sunday, 18 December 2011
'Users need to be able to specify whether their comments are intended for publication': ReScript editing issue 5
Page section
Article editor
Heuristic
Memorability
Description
A formal distinction exists between phases of editorial work – the initial transcription and mark-up procedure may uncover ambiguities in the original of which the reader must be made aware. In addition, further secondary or explanatory material (for instance, footnotes) may be added by another user in a subsequent phase.
Impact severity
High
Recommendation
Although the semantic structure of the underlying content mark-up is flexible enough to support the presence of notes, users conceive of marking-up and editing as two separate functions which, in the course of a project, may be carried out by separate users. Support these functions distinctly and enable editors to specify whether their comments are intended for publication (Typical categories include: Added, Deleted, Gap, Supplied, Critical apparatus (e.g. footnote)). This solution needs to be extensible, will rely on editorial style and is expected to differ between sources.
Examples
During the editorial process, a user may comment on difficulties encountered during the transcription of the text, e.g. "This appears to read Botolph, but could possibly read Butolph." They may also add explanatory material, e.g. "The Botolph referred to is the parish of St Botolph Aldgate". At present, they cannot specify which of their comments are intended for publication, e.g. as footnotes, and which are not.
Quantitative measure
November 2011: 120 responses.
Development change
Creation of dedicated toolbar function, entitled 'Commentary' for editorial notes. Works in the same material way that the ready mark-up function does.
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
This objective was only partially met in spite of a new dedicated device being created for this purpose. The mark-up function to the left attracted a substantial number of clicks. It should have been clearer that the applying critical apparatus was distinct from the more formal marking up of content and so a change in labelling may be useful, from 'Commentary' to 'Footnotes, comments'.
There seems to be a risk here that, because the actual enhancement suggested is altering a few words, that that would undermine the substantial investment in time and resources in usability analysis. By its nature, usability reviews pick up on aspects of the interface which are likely to have a negative effect on the user experience so care needs to be taken to prevent the only output from this process appearing slight or subjective.
It feels frustrating that the recommendation here is to change a few words and that that is the difference between assigning a Met and a Partially Met classification to the work. It's almost as if we need a pre-usability-test usability test.
Article editor
Heuristic
Memorability
Description
A formal distinction exists between phases of editorial work – the initial transcription and mark-up procedure may uncover ambiguities in the original of which the reader must be made aware. In addition, further secondary or explanatory material (for instance, footnotes) may be added by another user in a subsequent phase.
Impact severity
High
Recommendation
Although the semantic structure of the underlying content mark-up is flexible enough to support the presence of notes, users conceive of marking-up and editing as two separate functions which, in the course of a project, may be carried out by separate users. Support these functions distinctly and enable editors to specify whether their comments are intended for publication (Typical categories include: Added, Deleted, Gap, Supplied, Critical apparatus (e.g. footnote)). This solution needs to be extensible, will rely on editorial style and is expected to differ between sources.
Examples
During the editorial process, a user may comment on difficulties encountered during the transcription of the text, e.g. "This appears to read Botolph, but could possibly read Butolph." They may also add explanatory material, e.g. "The Botolph referred to is the parish of St Botolph Aldgate". At present, they cannot specify which of their comments are intended for publication, e.g. as footnotes, and which are not.
 |
| Figure 5—1: at times, the transcriber may need to advise the reader that the originals were unclear |
Quantitative measure
Where would you expect to click in order to add a footnote to this paragraph?
Actual question
Initial click test result ('before')Where would you expect to click in order to add a footnote to this paragraph?
November 2011: 120 responses.
 |
| Figure 5—2: before |
Creation of dedicated toolbar function, entitled 'Commentary' for editorial notes. Works in the same material way that the ready mark-up function does.
 |
| Figure 5—3: commentary device added to toolbar and separated from mark-up device by search function |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 5—4: after |
This objective was only partially met in spite of a new dedicated device being created for this purpose. The mark-up function to the left attracted a substantial number of clicks. It should have been clearer that the applying critical apparatus was distinct from the more formal marking up of content and so a change in labelling may be useful, from 'Commentary' to 'Footnotes, comments'.
There seems to be a risk here that, because the actual enhancement suggested is altering a few words, that that would undermine the substantial investment in time and resources in usability analysis. By its nature, usability reviews pick up on aspects of the interface which are likely to have a negative effect on the user experience so care needs to be taken to prevent the only output from this process appearing slight or subjective.
It feels frustrating that the recommendation here is to change a few words and that that is the difference between assigning a Met and a Partially Met classification to the work. It's almost as if we need a pre-usability-test usability test.
Saturday, 17 December 2011
'Users need standardised visual cues to use devices which move or transform': ReScript editing issue 4
Page section
Article editor
Heuristic
Memorability
Description
Users were unanimous in liking the manuscript image viewer but none of them understood its potential for movement/resizing without interviewer intervention.
Impact severity
High
Recommendation
That design conventions for resizing objects be researched and employed on the manuscript image viewer, with some accompanying help text.
Examples
The user engaged with the controls inside the image viewer (zoom in/out, normal size and shrink to fit, but failed to resize the whole box, meaning their image viewer stayed the same size throughout their session, unnecessarily restricting its utility.
Quantitative measure
Initial click test result ('before')
November 2011: 120 responses.
Development change
Consolidate manuscript images into one 'viewer' with a menu to select different images. Note that the design of the extensible border was not changed.
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
The objective was met through the convincing grouping of clicks; however, it is not possible to discern the effectiveness of the extensible border because this type of test is static, i.e. users interact with an image of the page, not the page itself, so they cannot hover over the border, see the mouse cursor change to a resizable icon. Although impossible to test in this way, it should still be possible to include this functionality in a demonstration video, or a prior training course.
The objective of the question is so clear that it reveals a certain number of clicks from users who are simply not engaged with the test - this helps to give the analyst a sense of the amount of disinterested or wasted replies and think about whether the people approached to take the survey were properly identified and communicated with. In addition, the sequencing of difficult questions at the start may have led some users to think that they could not manage the rest of the survey.
Article editor
Heuristic
Memorability
Description
Users were unanimous in liking the manuscript image viewer but none of them understood its potential for movement/resizing without interviewer intervention.
Impact severity
High
Recommendation
That design conventions for resizing objects be researched and employed on the manuscript image viewer, with some accompanying help text.
Examples
The user engaged with the controls inside the image viewer (zoom in/out, normal size and shrink to fit, but failed to resize the whole box, meaning their image viewer stayed the same size throughout their session, unnecessarily restricting its utility.
 |
| Figure 4—1: users controlled the size of the manuscript within the box - it wasn't apparent that they could extend the viewer's size as well by dragging the box border. |
Quantitative measure
Click where you would expect to be able to expand the size of the manuscript images
Actual question
Click where you would expect to be able to expand the size of the manuscript images
Initial click test result ('before')
November 2011: 120 responses.
 |
| Figure 4—2: Before |
Development change
Consolidate manuscript images into one 'viewer' with a menu to select different images. Note that the design of the extensible border was not changed.
 |
| Figure 4—3: single image viewer concentrates attention on one spot and eliminates the possibility of the image appearing below the fold and so being missed |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 4—4: After |
The objective was met through the convincing grouping of clicks; however, it is not possible to discern the effectiveness of the extensible border because this type of test is static, i.e. users interact with an image of the page, not the page itself, so they cannot hover over the border, see the mouse cursor change to a resizable icon. Although impossible to test in this way, it should still be possible to include this functionality in a demonstration video, or a prior training course.
The objective of the question is so clear that it reveals a certain number of clicks from users who are simply not engaged with the test - this helps to give the analyst a sense of the amount of disinterested or wasted replies and think about whether the people approached to take the survey were properly identified and communicated with. In addition, the sequencing of difficult questions at the start may have led some users to think that they could not manage the rest of the survey.
Friday, 16 December 2011
'Users do not check work by looking at mark-up': ReScript editing issue 3
Page section
Article editor
Heuristic
Learnability
Impact severity
High
Recommendation
Extend the functionality of the preview pane so that double-clicking on text there simultaneously selects the same text in the XML editing window as well.
Examples
The user looks through and selects text in the preview pane, and then uses the toolbar to insert the relevant attribute. As the cursor position in the XML editing window is not in sync with the cursor position in the preview pane, the attribute code is inserted in the text in the XML editing window in the wrong position, as illustrated below.
Quantitative measure
Actual question
Initial click test result ('before')
November 2011: 120 responses.
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
The wording of the question could have been better: the term 'attribute' is quite generic and would normally be replaced with a worked example, e.g. "where would you expect to mark-up Evance Jones' surname". This had the effect of confusing users of the test, leading to a group of clicks on the third toolbar option, titled 'Commentary'. This would be mitigated if this toolbar option were to be renamed along the lines of 'Footnotes, Comments'.
Overall, there has been a movement of clicks away from the text and into the toolbar but some still remain. By this point in the sequence of tests, I am beginning to think that a further context menu might be necessary which pops up at the cursor position and also permits access to the 'insert mark-up' functions. This would be a different means to support people but it would not be easy to investigate using this type of click test.
Article editor
Heuristic
Learnability
Description
Many users will be unused to editing XML marked-up text, and may find this daunting. When checking the document, users will likely read it through using the preview pane rather than the XML editing window, and will expect to be able to select parts of the text that they wish to edit from within the preview pane.
Impact severity
High
Recommendation
Extend the functionality of the preview pane so that double-clicking on text there simultaneously selects the same text in the XML editing window as well.
Examples
The user looks through and selects text in the preview pane, and then uses the toolbar to insert the relevant attribute. As the cursor position in the XML editing window is not in sync with the cursor position in the preview pane, the attribute code is inserted in the text in the XML editing window in the wrong position, as illustrated below.
 |
| Figure 3—1: the real-time preview pane allows text selection as does the main editing window. If the text selection is different, the user could assume they are inserting the relevant mark-up given the preview pane position but the system would proceed using the text window. |
Click where you would expect to add an attribute to the name "Evance Jones".
Actual question
Click where you would expect to add an attribute to the name "Evance Jones".
November 2011: 120 responses.
 |
| Figure 3—2: Before |
Development change
Removing the preview pane from simultaneous viewing alongside the text editing window, and limiting the toolbar to only appear alongside the text window.
 |
| Figure 3—3: de-coupling the preview pane leaves more room for the editing window and new toolbar, presented within a tab folder structure |
December 2011: 90 responses.
 |
| Figure 3—4: After |
The wording of the question could have been better: the term 'attribute' is quite generic and would normally be replaced with a worked example, e.g. "where would you expect to mark-up Evance Jones' surname". This had the effect of confusing users of the test, leading to a group of clicks on the third toolbar option, titled 'Commentary'. This would be mitigated if this toolbar option were to be renamed along the lines of 'Footnotes, Comments'.
Overall, there has been a movement of clicks away from the text and into the toolbar but some still remain. By this point in the sequence of tests, I am beginning to think that a further context menu might be necessary which pops up at the cursor position and also permits access to the 'insert mark-up' functions. This would be a different means to support people but it would not be easy to investigate using this type of click test.
Thursday, 15 December 2011
'Some parts of articles are repetitively structured and would suit some automation': ReScript editing issue 2
Page section
Article editor
Heuristic
Efficiency
Description
Articles within a specific source typically have structures which are repeated several times. Users often mark up structures by hand and/or use 'copy and paste'. This sometimes leads to incomplete and/or inaccurate mark-up, slowing the editorial process down and making additional editorial intervention necessary.
Impact severity
High
Recommendation
Part of the set-up process for a project should be the definition of repetitive code structures. It should then be possible for users simply to drop copies of these accurate and complete structures into the article wherever they are needed.
Examples
When attempting to copy the mark-up structure used for one instance of the item person to another instance, users sometimes accidentally omitted parts of the tagging code.
Quantitative measure
Actual question
Initial click test result ('before')
November 2011: 120 responses.
Development change
Toolbar now includes a single menu for inserting mark-up with elements arranged hierarchically.
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
This issue has been met, as shown by the movement of large click groupings from the text and preview 'contexts' to the toolbar function. The extended manuscript image was a red herring, designed to confuse those people who have very little grounding in this type of work - the amount still clicking on the image suggests that there is a need for an initial period of training for quite a number of potential contributors.
Regarding the efficacy of the test, the revised function itself uses a turn of phrase from the test itself ('...ready made mark-up'). That may to a degree, invalidate the findings, if users are mentally 'matching' the question to a part of the page. However, by using as little jargon as possible in this type of test, the software benefits far more from having labelling which is easy to understand, than the test loses in fairness. After all, the questions emanate from qualitative research where the analyst is able to reflect language which they think will be well understood
Article editor
Heuristic
Efficiency
Description
Articles within a specific source typically have structures which are repeated several times. Users often mark up structures by hand and/or use 'copy and paste'. This sometimes leads to incomplete and/or inaccurate mark-up, slowing the editorial process down and making additional editorial intervention necessary.
Impact severity
High
Recommendation
Part of the set-up process for a project should be the definition of repetitive code structures. It should then be possible for users simply to drop copies of these accurate and complete structures into the article wherever they are needed.
Examples
When attempting to copy the mark-up structure used for one instance of the item person to another instance, users sometimes accidentally omitted parts of the tagging code.
 |
| Figure 2—1: the name is repeated in the text with a slight variation in spelling. The risk of transcription inaccuracy is increased with the temptation to 'cut and paste' and move on. |
Quantitative measure
You want to mark-up the text "Evance Jones" as a person. Rather than marking-up by hand, click where you would expect to instruct the system to insert some ready-made mark-up for a person.
Actual question
You want to mark-up the text "Evance Jones" as a person. Rather than marking-up by hand, click where you would expect to instruct the system to insert some ready-made mark-up for a person.
Initial click test result ('before')
November 2011: 120 responses.
 |
| Figure 2—2: Before |
Development change
Toolbar now includes a single menu for inserting mark-up with elements arranged hierarchically.
 |
| Figure 2—3: prominent toolbar with markup device first |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 2—4: After |
Reflections
This issue has been met, as shown by the movement of large click groupings from the text and preview 'contexts' to the toolbar function. The extended manuscript image was a red herring, designed to confuse those people who have very little grounding in this type of work - the amount still clicking on the image suggests that there is a need for an initial period of training for quite a number of potential contributors.
Regarding the efficacy of the test, the revised function itself uses a turn of phrase from the test itself ('...ready made mark-up'). That may to a degree, invalidate the findings, if users are mentally 'matching' the question to a part of the page. However, by using as little jargon as possible in this type of test, the software benefits far more from having labelling which is easy to understand, than the test loses in fairness. After all, the questions emanate from qualitative research where the analyst is able to reflect language which they think will be well understood
'Users need constant feedback when using something new': ReScript editing issue 1
Page section
Article editor
Heuristic
Learnability
Description
Users often mark up data using attributes without identifying the item to which they belong. This makes it impossible to extract certain types of meanings from the article and so making additional editorial intervention necessary.
Impact severity
High
Recommendation
When applying an attribute, visually emphasise the item to which it relates, and report if the appropriate logical parent is not marked up.
Examples
Users often marked up attributes, such as forename or surname, but omitted to nest these within the relevant item, e.g. person, which would correctly limit the scope/context of the attributes.
Quantitative measure
Actual question
Initial click test result ('before')
November 2011: 120 responses.
Development change
Toolbar has been reconfigured to arrange attributes directly next to its correct parent item, in a single drop down menu. In addition, a status bar at the bottom warns of any incorrect XML.
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
This was quite an involved question to begin the test with, and hopefully set the tone for the remainder of the survey, giving users an idea of what level of question to expect. The correct answer for the follow-up was the status bar message which appears beneath the text editor which did attract some clicks; the code around the problem mark-up also turns red when incorrect so those that clicked on the text itself are also correct.
There are other incorrect groups of clicks though, the right sidebar and top toolbar most prominent. IN hindsight, this particular test may have been better positioned later on in the test, when users had had a chance to better acquaint themselves with the system. However, this is still a difficult question to answer as it asks to draw on an idea of interactivity for a software programme of which they probably have no experience.
Article editor
Heuristic
Learnability
Description
Users often mark up data using attributes without identifying the item to which they belong. This makes it impossible to extract certain types of meanings from the article and so making additional editorial intervention necessary.
Impact severity
High
Recommendation
When applying an attribute, visually emphasise the item to which it relates, and report if the appropriate logical parent is not marked up.
Examples
Users often marked up attributes, such as forename or surname, but omitted to nest these within the relevant item, e.g. person, which would correctly limit the scope/context of the attributes.
 |
| Figure 1—1: here the properties forename, surname and occupation appear without a parent element to delimit their scope to the individual Richard Casye |
Quantitative measure
You have just marked up "Richard Casye". If there was a problem with your mark-up, where would you expect to see a warning message?
Actual question
You have just marked up "Richard Casye". If there was a problem with your mark-up, where would you expect to see a warning message?
Initial click test result ('before')
November 2011: 120 responses.
 |
| Figure 1—2: Before |
Development change
Toolbar has been reconfigured to arrange attributes directly next to its correct parent item, in a single drop down menu. In addition, a status bar at the bottom warns of any incorrect XML.
 |
| Figure 8—3: the toolbar is much more prominent, and the preview device has been moved to a different folder tab |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 1—4: After |
Reflections
This was quite an involved question to begin the test with, and hopefully set the tone for the remainder of the survey, giving users an idea of what level of question to expect. The correct answer for the follow-up was the status bar message which appears beneath the text editor which did attract some clicks; the code around the problem mark-up also turns red when incorrect so those that clicked on the text itself are also correct.
There are other incorrect groups of clicks though, the right sidebar and top toolbar most prominent. IN hindsight, this particular test may have been better positioned later on in the test, when users had had a chance to better acquaint themselves with the system. However, this is still a difficult question to answer as it asks to draw on an idea of interactivity for a software programme of which they probably have no experience.
'Index filters sometimes produce prohibitively long lists': ReScript querying issue 9
Page section
Index
Heuristic
Efficiency
Description
Index filters sometimes produce very long lists, which are unwieldy to navigate.
Impact severity
Medium
Recommend
Remove the Index section. The new design will incorporate a search function with either predictive text, or alphabetical sections.
Examples
There are currently 4,000 surnames listed in the Parish Clerks’ Memoranda Books.
Quantitative measure
Reflections
This issue was not tested because of it was covered within the tests for the other search-based issues. It reminds us of issue #1, in which historians did not adopt the functionality which comes from seeing text as a 'database of words'. For many users, free-text search is an essential (non-negotiable?) part of the research process.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Index
Heuristic
Efficiency
Description
Index filters sometimes produce very long lists, which are unwieldy to navigate.
Impact severity
Medium
Recommend
Remove the Index section. The new design will incorporate a search function with either predictive text, or alphabetical sections.
Examples
There are currently 4,000 surnames listed in the Parish Clerks’ Memoranda Books.
 |
| Figure 9—1: the length of site menus is subject to every variation within the text |
Quantitative measure
Click where you would expect to search for the surname “Gaveston".
Reflections
This issue was not tested because of it was covered within the tests for the other search-based issues. It reminds us of issue #1, in which historians did not adopt the functionality which comes from seeing text as a 'database of words'. For many users, free-text search is an essential (non-negotiable?) part of the research process.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Wednesday, 14 December 2011
'There is currently no “keyword search” function': ReScript querying issue 8
Page section
Index; Search legend
Heuristic
Efficiency
Description
It is currently not possible for users to conduct full text searches using keywords.
Impact severity
High
Recommendation
Replace the current Search legend with a “Super Search” form which includes a simple keyword search function here.
Examples
It is currently not possible for users to search on terms that are not included in the Index, e.g. scripture references such as “1 Peter”.
Quantitative measure
Actual question
Initial click test result ('before')
September 2011: 89 responses.
Development change
The publication's home page is now quite configurable, and contains a box to search on article titles (with a jQuery powered auto-complete function), as well as a conventional free-text search.
Follow-up click test result ('after')
November 2011: 330 responses.
Reflections
The article title search box on the main left panel attracted some clicks (erroneously) but the clear majority plumped for the correct sidebar option, with a few going for the wider site search at the head of the page. Search is so important that having a couple of options is a strength, but there must be swift feedback if a user is entering the wrong kind of information; for instance, using the article title search to look for a keyword.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Index; Search legend
Heuristic
Efficiency
Description
It is currently not possible for users to conduct full text searches using keywords.
Impact severity
High
Recommendation
Replace the current Search legend with a “Super Search” form which includes a simple keyword search function here.
Examples
It is currently not possible for users to search on terms that are not included in the Index, e.g. scripture references such as “1 Peter”.
 |
| Figure 8—1: no free-text search feature |
Quantitative measure
Where would you click to search on all inclusions of scripture verses from 1 Peter?
Actual question
Where would you click to search for all records with scripture verses from 1 Peter ?
Initial click test result ('before')
September 2011: 89 responses.
 |
| Figure 8—2: Before |
Development change
The publication's home page is now quite configurable, and contains a box to search on article titles (with a jQuery powered auto-complete function), as well as a conventional free-text search.
 |
| Figure 8—3: not one but two search boxes (three if you include the full site search) |
Follow-up click test result ('after')
November 2011: 330 responses.
 |
| Figure 8—4: After |
Reflections
The article title search box on the main left panel attracted some clicks (erroneously) but the clear majority plumped for the correct sidebar option, with a few going for the wider site search at the head of the page. Search is so important that having a couple of options is a strength, but there must be swift feedback if a user is entering the wrong kind of information; for instance, using the article title search to look for a keyword.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Tuesday, 13 December 2011
'Variant spellings currently have to be searched on separately': ReScript querying issue 7
Page section
Index
Heuristic
Efficiency
Description
All spelling variants currently have to be searched on separately.
Impact severity
High
Recommend
In the redesigned search form, provide two options: one for searching individual spelling variants, and one for searching all variants together.
Examples
There are currently several spellings for many search items, e.g. agewe.
Quantitative measure
Actual question
Initial click test result ('before')
September 2011: 89 responses.
Development change
Creation of a thesaurus model for each publication which would permit suggested expansions for queries. These will also be filterable for particular predicates, e.g. suggest farmer only when marked up as an occupation, not a surname.
Follow-up click test result ('after')
November 2011: 330 responses.
Reflections
These will be opt-in; the user must choose to add them to their query, forcing them to think about how the source is written. It also indicates where the source has particular areas of strength which if simply narrated in the 'About the source' section could be missed by the majority of users. The objective has been pretty clearly met.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Index
Heuristic
Efficiency
Description
All spelling variants currently have to be searched on separately.
Impact severity
High
Recommend
In the redesigned search form, provide two options: one for searching individual spelling variants, and one for searching all variants together.
Examples
There are currently several spellings for many search items, e.g. agewe.
 |
| Figure 7—1: each user must combine different spellings by hand |
Quantitative measure
Where would you click to search on all variant spellings of “agewe”?
Actual question
Where would you click to search on variant spellings of the word 'widow' ?
Initial click test result ('before')
September 2011: 89 responses.
 |
| Figure 7—2: Before |
Development change
Creation of a thesaurus model for each publication which would permit suggested expansions for queries. These will also be filterable for particular predicates, e.g. suggest farmer only when marked up as an occupation, not a surname.
 |
| Figure 7—3: a search for widow here has returned a couple of editor-controlled expansions |
Follow-up click test result ('after')
November 2011: 330 responses.
 |
| Figure 7—4: After |
Reflections
These will be opt-in; the user must choose to add them to their query, forcing them to think about how the source is written. It also indicates where the source has particular areas of strength which if simply narrated in the 'About the source' section could be missed by the majority of users. The objective has been pretty clearly met.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Wednesday, 7 December 2011
'There is currently not enough information about the source being searched in the results pages': ReScript querying issue 6
Page
Source page
Heuristic
Learnability, Memorability
Description
There is currently no information, apart from its title, about the source being searched.
Impact severity
Medium
Recommend
Examples
Users reported that it would be useful to have information about the source documents being searched on the individual source search pages.
Quantitative measure
Actual question
Initial click test result ('before')
September 2011: 89 responses.
Development change
Inserted a new link to the publication in the title bar, a short bibliographic description in the first sidebar, and a link to create a citation.
Follow-up click test result ('after')
November 2011: 330 responses.
Reflections
The response was split between the title and sidebar and was very successful overall. The sidebar has the further attribute of leading into the contextually relevant calendar function which permits highlighting of matching attributes in the text. By associating the publication with these functions, users should have an accurate understanding of the kinds of queries which will be successful with this source.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Source page
Heuristic
Learnability, Memorability
Description
There is currently no information, apart from its title, about the source being searched.
Impact severity
Medium
Recommend
- Include a brief description of the source directly underneath the name; and
- create a clickable image that takes users to a stand-alone page with comprehensive information about the source.
Examples
Users reported that it would be useful to have information about the source documents being searched on the individual source search pages.
 |
| Figure 6—1: the scope and provenance of this source is absent |
Quantitative measure
Where on this page would you expect to find background information about the Alumni Oxonienses?
Actual question
Click where you would expect to find background information about the 'Alumni Oxonienses'.
Initial click test result ('before')
September 2011: 89 responses.
 |
| Figure 6—2: Before |
Development change
Inserted a new link to the publication in the title bar, a short bibliographic description in the first sidebar, and a link to create a citation.
 |
| Figure 6—3: bibliographic information appears in two locations, the title and sidebar |
Follow-up click test result ('after')
November 2011: 330 responses.
 |
| Figure 6—4: After |
Reflections
The response was split between the title and sidebar and was very successful overall. The sidebar has the further attribute of leading into the contextually relevant calendar function which permits highlighting of matching attributes in the text. By associating the publication with these functions, users should have an accurate understanding of the kinds of queries which will be successful with this source.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Tuesday, 6 December 2011
'Users have no way of creating a citation, nor any help for saving their search': ReScript querying issue 5
Page
Source page
Heuristic
Efficiency
Description
Once users have conducted a search, there is no way for them to save either the search path or the results.
Impact severity
Medium
Recommend
Create an option for saving searches.
Examples
If users create a search and later want to re-run it either as-is, or with amendments, they have to start from scratch.
Quantitative measure
September 2011: 89 responses.
Development change
Search results pages now have a Remember this search facility, which will be accompanied by a drop down list of those recently saved.
Follow-up click test result ('after')
November 2011: 330 responses.
Reflections
The grid layout of the new design is no less complex compared to the previous version, but the Remember function seems to be well positioned given the clear grouping of clicks. It could have been named 'Save' but that has connotations of downloading / exporting to files which isn't what was meant. 'Remember' is a more informal word, and the function itself uses the HTML5 localStorage feature, which ought to make it compatible with mobile devices.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Source page
Heuristic
Efficiency
Description
Once users have conducted a search, there is no way for them to save either the search path or the results.
Impact severity
Medium
Recommend
Create an option for saving searches.
Examples
If users create a search and later want to re-run it either as-is, or with amendments, they have to start from scratch.
 |
| Figure 5—1: complicated queries cannot be saved when viewing |
Quantitative measure
Where would you expect to click in order to save the results of this search?Actual question
Where would you expect to click to save the results of this search?Initial click test result ('before')
September 2011: 89 responses.
 |
| Figure 5—2: Before |
Development change
Search results pages now have a Remember this search facility, which will be accompanied by a drop down list of those recently saved.
 |
| Figure 5—3: New 'My Links' section in title bar will also link to saved articles |
Follow-up click test result ('after')
November 2011: 330 responses.
 |
| Figure 5—4: After |
Reflections
The grid layout of the new design is no less complex compared to the previous version, but the Remember function seems to be well positioned given the clear grouping of clicks. It could have been named 'Save' but that has connotations of downloading / exporting to files which isn't what was meant. 'Remember' is a more informal word, and the function itself uses the HTML5 localStorage feature, which ought to make it compatible with mobile devices.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Friday, 2 December 2011
'Users expected highlighted terms to extend their current search': ReScript querying issue 4
Page section
Results
Heuristic
Learnability
Description
Users expected to be able to click on a highlighted term and have all results for that term appear. They were surprised to have search symbols appear instead.
Impact severity
Medium
Recommend
Remove search symbols and create an instantaneous link to the appropriate search filter from highlighted text.
Examples
To find all instances of “Richardson”, users currently hover over the text “Richardson” and then have to choose either the blue circle or red triangle search option.
Quantitative measure
Click where you would expect to add “Richardson” to your current search.
Actual question
Click where you would expect to add "Richardson" to your current search.
Initial click test result ('before')
September 2011: 89 responses.
Follow-up click test result ('after')
Reflections
The most correct answer to this question, i.e. ‘Richardson’ in the surnames list, appeared so far down the page when viewed in the actual test format that users were not able to see it without scrolling down. However, in spite of this, there were still a large number of clicks at the bottom centre of the page, indicating that although they could not actually see the surname ‘Richardson’ in the sidebar, a great many users still recognised that they needed to navigate further down the page in order to find it in the list of surnames.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Results
Heuristic
Learnability
Description
Users expected to be able to click on a highlighted term and have all results for that term appear. They were surprised to have search symbols appear instead.
Impact severity
Medium
Recommend
Remove search symbols and create an instantaneous link to the appropriate search filter from highlighted text.
Examples
To find all instances of “Richardson”, users currently hover over the text “Richardson” and then have to choose either the blue circle or red triangle search option.
 |
| Figure 4—1: although explanations were provided at the head of the page, the difference between UNION and INTERSECT meant nothing to historians |
Quantitative measure
Click where you would expect to add “Richardson” to your current search.
Actual question
Click where you would expect to add "Richardson" to your current search.
Initial click test result ('before')
September 2011: 89 responses.
 |
| Figure 4—2: Before |
Development change
Inclusion of a calendar function in the first sidebar enables categories of marked-up information to be presented alongside the original transcript, acting as an informal digest of the type, language and style of the content (e.g. http://www.rescript.org/article.aspx?p=2&a=43).
 |
| Figure 4—3: A calendar function helps the user navigate the document – each entry is clickable and highlights instances of it in the main text section itself. Those highlighted areas also become direct hyperlinks to the search engine |
Follow-up click test result ('after')
November 2011: 330 responses.
 |
| Figure 4—4: After |
Reflections
The most correct answer to this question, i.e. ‘Richardson’ in the surnames list, appeared so far down the page when viewed in the actual test format that users were not able to see it without scrolling down. However, in spite of this, there were still a large number of clicks at the bottom centre of the page, indicating that although they could not actually see the surname ‘Richardson’ in the sidebar, a great many users still recognised that they needed to navigate further down the page in order to find it in the list of surnames.
Bruce Tate, Project Manager
Donna Baillie, Project Officer
Subscribe to:
Posts (Atom)