We're pleased to announce that, with the help of funding from the Joint Information Systems Committee (JISC), the raw transcripts of the Parish Clerks' Memorandum Books have now been fully uploaded onto ReScript.
These books comprise a journal of events in the parish of St Botolph Aldgate in the period 1583-1625, as recorded by the parish clerks. Significantly wider in scope than the parish registers, these sources offer a uniquely rich and detailed view into the social and economic lives of the parishioners.
The project will now begin looking for editorial input to help mark up the texts which it is envisaged will offer greater support for deeper analysis of trends. Since the books record the rights of passage of the parish, they offer the opportunity to follow the lives of individuals: their birth, marriage, baptism of their children and their death may all be recorded within the Memorandum Books.
If you are interested in volunteering to mark up these early modern texts, please contact us through the British History Online contact form, using the subject line 'PCM contributor'.
Thursday 13 September 2012
Wednesday 14 March 2012
Lessons Learned
No discussion of the transformation of system design is possible at the moment without recourse to the severe nature of the research funding environment; currently, the recession has replaced necessity as the mother of invention. Given this, what practical change could take place?
System development could take a more open, public approach, and usability techniques could prove extremely useful here as they enable interaction with users in multiple ways, from surveys and interviews to blog postings of results. This would create more information which a project could publish through its lifecycle rather than aggregating and (to a certain extent) generalising in a final report.
Publishing the project plan and outcomes provides a useful starting point for the project; however, the language used in them will likely come directly from the original application for funding which will be highly specialised and reliant to a certain degree on jargon and technical understanding. Far more preferable would be to ensure that all public project communications could be accessible to as wide an audience as possible (this should include, of course, budget holders within the projects' own organisation).
To further exploit the work undertaken, an open library of evidence collected at first hand could be created from each project. Put simply, if a project undertakes a survey of, say, 100 researchers in a specific academic discipline then that survey must be made available for re-use. The same concept could be applied to interviews, focus groups or transect walks provided that data protection legislation was adequately followed. This would make available a knowledge base upon which other projects could instantly draw for informing their own usability strategies. Furthermore, with much research being ploughed into text mining, the first hand evidence materials here would form an excellent candidate for approaches such as sentiment analysis.
By applying research practice, this field could be transformed from one predominantly made up of narrative reflections of practitioners to a data store (perhaps accessible via an API) on which projects could draw for the modelling and evolution of user behaviour; thus corroborating or challenging results, and even for constructing funding applications (for instance, if particular sections of the academic community have not been adequately served or covered in prior research).
System development could take a more open, public approach, and usability techniques could prove extremely useful here as they enable interaction with users in multiple ways, from surveys and interviews to blog postings of results. This would create more information which a project could publish through its lifecycle rather than aggregating and (to a certain extent) generalising in a final report.
Publishing the project plan and outcomes provides a useful starting point for the project; however, the language used in them will likely come directly from the original application for funding which will be highly specialised and reliant to a certain degree on jargon and technical understanding. Far more preferable would be to ensure that all public project communications could be accessible to as wide an audience as possible (this should include, of course, budget holders within the projects' own organisation).
To further exploit the work undertaken, an open library of evidence collected at first hand could be created from each project. Put simply, if a project undertakes a survey of, say, 100 researchers in a specific academic discipline then that survey must be made available for re-use. The same concept could be applied to interviews, focus groups or transect walks provided that data protection legislation was adequately followed. This would make available a knowledge base upon which other projects could instantly draw for informing their own usability strategies. Furthermore, with much research being ploughed into text mining, the first hand evidence materials here would form an excellent candidate for approaches such as sentiment analysis.
By applying research practice, this field could be transformed from one predominantly made up of narrative reflections of practitioners to a data store (perhaps accessible via an API) on which projects could draw for the modelling and evolution of user behaviour; thus corroborating or challenging results, and even for constructing funding applications (for instance, if particular sections of the academic community have not been adequately served or covered in prior research).
Monday 12 March 2012
How successful have we been?
Our project plan identified both qualitative and quantitative measures with which to gauge success.
Issue list for 'Public' interface
The fact that the redesigned interface is unrecognisable from the version with which the investigation began indicates that the usability process was able to give strategic direction as well as feedback on finer details such as nomenclature; this is made more powerful by the fact that this was the first time that many users had seen the product.
Issue list for 'Editor' interface
You would be forgiven for thinking that an interface which relied heavily on interactive menus and controls would be difficult to investigate using click testing; however, given that all user activity starts with one click, it still needed to accurately signpost the correct entry points for interactivity. Overall, although the exercise created positive results, there were several areas in which the redesign failed; most frustratingly, issue 6, a simple case of inappropriate wording.
By publishing the life cycle of each issue, the project cannot hide any aspect of development and its decisions are open to the standard academic techniques of review and criticism. It is a deeply honest approach to system design whose success can be judged by the range of people, including non-specialists, who are empowered to comment on it.
Quantitative
The learnability investigation was split across two interfaces with discrete audiences, the querying interface ("public") and the XML document editing interface ("editor").Issue list for 'Public' interface
- 'Users do not understand the difference between querying and searching'
Issue 1: partially met - 'There is no method for users to initiate queries using statistical tools'
Issue 2: partially met - 'Users accidentally left a previous search filter active when starting a new search'
Issue 3: partially met - 'Users expected highlighted terms to extend their current search'
Issue 4: partially met - 'Users have no way of creating a citation, nor any help for saving their search'
Issue 5: met - 'There is not enough information about the source being searched in the results pages'
Issue 6: met - 'Variant spellings currently have to be searched on separately'
Issue 7: met - 'There is currently no keyword search function'
Issue 8: partially met
The fact that the redesigned interface is unrecognisable from the version with which the investigation began indicates that the usability process was able to give strategic direction as well as feedback on finer details such as nomenclature; this is made more powerful by the fact that this was the first time that many users had seen the product.
Issue list for 'Editor' interface
- 'Users need constant feedback when using something new'
issue 1: unmet - 'Some parts of articles are repetitively structured and would suit some automation'
issue 2: met - 'Users do not check work by looking at mark-up'
issue 3: partially met - 'Users need standardised visual cues to use devices which move or transform'
issue 4: met - 'Users need to be able to specify whether their comments are intended for publication'
issue 5: partially met - 'There is a need to provide guidance that is tailored both to the source, and to the role of the current user'
issue 6: unmet - 'Users like to navigate documents quickly using search'
issue 7: met - 'Some users will need to be able to track the changes that have been made to a document'
issue 8: met
You would be forgiven for thinking that an interface which relied heavily on interactive menus and controls would be difficult to investigate using click testing; however, given that all user activity starts with one click, it still needed to accurately signpost the correct entry points for interactivity. Overall, although the exercise created positive results, there were several areas in which the redesign failed; most frustratingly, issue 6, a simple case of inappropriate wording.
Qualitative
It is much harder to report on qualitative feedback – a lot of it is confidential and cannot be 'transcribed'; in addition, the compressed nature of the project plan means that there is little time for changes to take effect and be measured again. As an observation, it is more likely that users will view a project team as acting more deliberately (i.e. with a greater sense of strategy) if they communicate changes in system delivery personally. Interviews and focus group, and to some extent, email surveys can carry this sense of purpose.Approach
Each primary issue within the public and editor interfaces now has a fixed point within system documentation which can track its development. By opening this up to the general public, further scrutiny of the issues is enabled and the project can appear more confident in its action.By publishing the life cycle of each issue, the project cannot hide any aspect of development and its decisions are open to the standard academic techniques of review and criticism. It is a deeply honest approach to system design whose success can be judged by the range of people, including non-specialists, who are empowered to comment on it.
Wednesday 21 December 2011
'Some users will need to be able to track the changes that have been made to a document': ReScript editing issue 8
Page section
Article editor
Heuristic
Memorability
Description
Some users (for instance editors) will need to assess other users' work and will need to compare different versions of the same document.
Impact severity
High
Recommendation
Insert links to the entire version history of a document, in reverse chronological order.
Examples
The transcription of a document will lead to certain decisions being made by the user on structure as well as mark-up. A version feature would enable editors to see how users have interpreted guidance, and provide practical feedback for future work.
Quantitative measure
Actual question
Development change
Inclusion of a folder tab entitled 'version history', appearing third in the folder structure, and giving links to previous versions.
Reflections
Article editor
Memorability
Description
Some users (for instance editors) will need to assess other users' work and will need to compare different versions of the same document.
Impact severity
High
Insert links to the entire version history of a document, in reverse chronological order.
Examples
The transcription of a document will lead to certain decisions being made by the user on structure as well as mark-up. A version feature would enable editors to see how users have interpreted guidance, and provide practical feedback for future work.
 |
| Figure 8—1: we learn from our mistakes, and currently there is no way for a user to see the exact changes which an editor has made on one of their documents. |
Click where you would expect to access and view a previous version of this document
Actual question
Click where you would expect to access and view a previous version of this document
Initial click test result ('before')
November 2011: 120 responses.
 |
| Figure 8—2: before |
Inclusion of a folder tab entitled 'version history', appearing third in the folder structure, and giving links to previous versions.
 |
| Figure 8—3: new folder tab |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 8—4: after |
Clear grouping of clicks on the appropriate folder tab, objective has been met. The alignment of question wording with interface seems to improve the chance of click test success - however, this wording is generated from qualitative interviews therefore it is fair to assume that by institutionalising it at an early stage, we have made the process more robust.
Tuesday 20 December 2011
'Users like to navigate documents quickly using search': ReScript editing issue 7
Page section
Article editor
Heuristic
Learnability, memorability
Description
At times, users need to locate specific parts of sometimes quite lengthy and heavily marked-up documents, either to check the style of mark-up employed, or to look for a particular phrase which might be a candidate for mark-up. They may also wish to have a quick way of comparing the mark-up of matching items. Without a search function, this is problematic.
Impact severity
High
Recommendation
The inclusion of a search function in the control pane, with matches highlighted.
Examples
Parliamentary texts may continually refer to the same individuals, and users may choose to find all instances and mark them up consecutively.
Quantitative measure
Actual question
Initial click test result ('before')
November 2011: 120 responses.
Development change
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
Using search is a necessary skill to add sohphistication into web browsing; users recognise and understand it as a function and the click groups above show that the revised interface is resoundingly clear in meeting this objective.
Article editor
Heuristic
Learnability, memorability
Description
At times, users need to locate specific parts of sometimes quite lengthy and heavily marked-up documents, either to check the style of mark-up employed, or to look for a particular phrase which might be a candidate for mark-up. They may also wish to have a quick way of comparing the mark-up of matching items. Without a search function, this is problematic.
Impact severity
High
Recommendation
The inclusion of a search function in the control pane, with matches highlighted.
Examples
Parliamentary texts may continually refer to the same individuals, and users may choose to find all instances and mark them up consecutively.
 |
| Figure 7—1: the more heavily the text is marked up, the less easy it becomes to scan for key words. |
You are looking for the name “Wallworth”. Click where you would expect to find a function which allowed you to search for this name within the current document.
Actual question
You are looking for the name “Wallworth”. Click where you would expect to find a function which allowed you to search for this name within the current document.
November 2011: 120 responses.
 |
| Figure 7—2: before |
Inclusion of a search function limited to the editing window, appearing second in the new toolbar.
 |
| Figure 7—3: matches within the text are highlighted and the document position is automatically moved to the first match |
December 2011: 90 responses.
 |
| Figure 7—4: after |
Using search is a necessary skill to add sohphistication into web browsing; users recognise and understand it as a function and the click groups above show that the revised interface is resoundingly clear in meeting this objective.
Monday 19 December 2011
'There is a need to provide guidance that is tailored both to the source, and to the role of the current user': ReScript editing issue 6
Page section
Article editor
Heuristic
Memorability
Description
Guidance about the source itself will be essential for all users. In addition, different levels of guidance will be needed depending on whether the user is editing or transcribing. This guidance should extend into practical advice about mark-up, preferably including worked examples.
Impact severity
High
Recommendation
Creation of a persistent page object into which guidance may be tailored to both the source and role of the current user.
Examples
The names of some items are likely to be somewhat generic or esoteric depending on the source, e.g. event in the Alumni Oxonienses draws together related details within education and occupation.
Quantitative measure
November 2011: 120 responses.
Development change
Creation of a tab folder, entitled 'About the source' for editorial guidance. Easy to switch between guidance and editing.
Article editor
Heuristic
Memorability
Description
Guidance about the source itself will be essential for all users. In addition, different levels of guidance will be needed depending on whether the user is editing or transcribing. This guidance should extend into practical advice about mark-up, preferably including worked examples.
Impact severity
High
Recommendation
Creation of a persistent page object into which guidance may be tailored to both the source and role of the current user.
Examples
The names of some items are likely to be somewhat generic or esoteric depending on the source, e.g. event in the Alumni Oxonienses draws together related details within education and occupation.
 |
| Figure 6—1: there appears to be no linear flow for the editing process meaning it is difficult for people to know how and where to start |
Quantitative measure
Click where you would expect to find guidance on how to mark up the entity "East india Company"
Actual question
Click where you would expect to find guidance on how to mark up the organisation "East india Company"Initial click test result ('before')
November 2011: 120 responses.
 |
| Figure 6—2: before |
Development change
Creation of a tab folder, entitled 'About the source' for editorial guidance. Easy to switch between guidance and editing.
 |
| Figure 6—3: new folder tab structure enables guidance to appear alongside editing function without changing pages |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 6—4: after |
Reflections
This issue remains unmet as there is no particularly strong concentration of clicks on the 'About the Source' folder tab, and the generic Help link in the top right attracts a similar amount of attention in both versions.
On reflection, this may be a simple labelling issue. 'About the source' is a general term but inappropriate in this context. Results would likely have been better if it were titled 'Mark-up guidelines', 'Editing guidelines' or even 'How to edit this'. This is a frustratingly small error at the final execution phase which has led to the issue being unmet and should have been resolved by QA.
Sunday 18 December 2011
'Users need to be able to specify whether their comments are intended for publication': ReScript editing issue 5
Page section
Article editor
Heuristic
Memorability
Description
A formal distinction exists between phases of editorial work – the initial transcription and mark-up procedure may uncover ambiguities in the original of which the reader must be made aware. In addition, further secondary or explanatory material (for instance, footnotes) may be added by another user in a subsequent phase.
Impact severity
High
Recommendation
Although the semantic structure of the underlying content mark-up is flexible enough to support the presence of notes, users conceive of marking-up and editing as two separate functions which, in the course of a project, may be carried out by separate users. Support these functions distinctly and enable editors to specify whether their comments are intended for publication (Typical categories include: Added, Deleted, Gap, Supplied, Critical apparatus (e.g. footnote)). This solution needs to be extensible, will rely on editorial style and is expected to differ between sources.
Examples
During the editorial process, a user may comment on difficulties encountered during the transcription of the text, e.g. "This appears to read Botolph, but could possibly read Butolph." They may also add explanatory material, e.g. "The Botolph referred to is the parish of St Botolph Aldgate". At present, they cannot specify which of their comments are intended for publication, e.g. as footnotes, and which are not.
Quantitative measure
November 2011: 120 responses.
Development change
Creation of dedicated toolbar function, entitled 'Commentary' for editorial notes. Works in the same material way that the ready mark-up function does.
Follow-up click test result ('after')
December 2011: 90 responses.
Reflections
This objective was only partially met in spite of a new dedicated device being created for this purpose. The mark-up function to the left attracted a substantial number of clicks. It should have been clearer that the applying critical apparatus was distinct from the more formal marking up of content and so a change in labelling may be useful, from 'Commentary' to 'Footnotes, comments'.
There seems to be a risk here that, because the actual enhancement suggested is altering a few words, that that would undermine the substantial investment in time and resources in usability analysis. By its nature, usability reviews pick up on aspects of the interface which are likely to have a negative effect on the user experience so care needs to be taken to prevent the only output from this process appearing slight or subjective.
It feels frustrating that the recommendation here is to change a few words and that that is the difference between assigning a Met and a Partially Met classification to the work. It's almost as if we need a pre-usability-test usability test.
Article editor
Heuristic
Memorability
Description
A formal distinction exists between phases of editorial work – the initial transcription and mark-up procedure may uncover ambiguities in the original of which the reader must be made aware. In addition, further secondary or explanatory material (for instance, footnotes) may be added by another user in a subsequent phase.
Impact severity
High
Recommendation
Although the semantic structure of the underlying content mark-up is flexible enough to support the presence of notes, users conceive of marking-up and editing as two separate functions which, in the course of a project, may be carried out by separate users. Support these functions distinctly and enable editors to specify whether their comments are intended for publication (Typical categories include: Added, Deleted, Gap, Supplied, Critical apparatus (e.g. footnote)). This solution needs to be extensible, will rely on editorial style and is expected to differ between sources.
Examples
During the editorial process, a user may comment on difficulties encountered during the transcription of the text, e.g. "This appears to read Botolph, but could possibly read Butolph." They may also add explanatory material, e.g. "The Botolph referred to is the parish of St Botolph Aldgate". At present, they cannot specify which of their comments are intended for publication, e.g. as footnotes, and which are not.
 |
| Figure 5—1: at times, the transcriber may need to advise the reader that the originals were unclear |
Quantitative measure
Where would you expect to click in order to add a footnote to this paragraph?
Actual question
Initial click test result ('before')Where would you expect to click in order to add a footnote to this paragraph?
November 2011: 120 responses.
 |
| Figure 5—2: before |
Creation of dedicated toolbar function, entitled 'Commentary' for editorial notes. Works in the same material way that the ready mark-up function does.
 |
| Figure 5—3: commentary device added to toolbar and separated from mark-up device by search function |
Follow-up click test result ('after')
December 2011: 90 responses.
 |
| Figure 5—4: after |
This objective was only partially met in spite of a new dedicated device being created for this purpose. The mark-up function to the left attracted a substantial number of clicks. It should have been clearer that the applying critical apparatus was distinct from the more formal marking up of content and so a change in labelling may be useful, from 'Commentary' to 'Footnotes, comments'.
There seems to be a risk here that, because the actual enhancement suggested is altering a few words, that that would undermine the substantial investment in time and resources in usability analysis. By its nature, usability reviews pick up on aspects of the interface which are likely to have a negative effect on the user experience so care needs to be taken to prevent the only output from this process appearing slight or subjective.
It feels frustrating that the recommendation here is to change a few words and that that is the difference between assigning a Met and a Partially Met classification to the work. It's almost as if we need a pre-usability-test usability test.
Subscribe to:
Posts (Atom)