Evaluation Methods: Quantitative & Qualitative
A. Testing EXPERIENCE: checking its quality
Heuristic Evaluation VR (expert review)
This is not a user test, but an expert review.
A Heuristic Evaluation is performed with 4-6 people from your team or 4-6 designers-developers from other teams.
The idea is that each of you will take the headset and go through the experience. After this, each of you will list challenges & problems that you have experiences for each heuristic (so Natural Engagement can show a list of multiple challenges that need to be solved, and then the next heuristic as well).
When everyone has created their list of issues, problems and challenges, you will start discuss each issues and then rate each one of them. (1= simple problem, easy solvable within project - 5= complex problem, unsolvable within project). You will create a Heuristic Evaluation Matrix to receive an overview of the amount and complexity of usability problems in total.
These heuristics were motivated by the different nature of VEs, in particular, the need for intuitive interaction and the sense of immersion, which is important for many VR applications that aim to simulate reality as faithfully as possible (Stone, 2002).
Heuristics for VR
Tip: try to come up with some of your own heuristics, specified for your project.
Natural engagement. Interaction should approach the user’s expectation of interaction in the real world as far as possible. Ideally, the user should be unaware that the reality is virtual. Interpreting this heuristic will depend on the naturalness requirement and the user’s sense of presence and engagement.
Compatibility with the user’s task and domain. The VE and behaviour of objects should correspond as closely as possible to the user’s expectation of real world objects; their behaviour; and affordances for task action.
Natural expression of action. The representation of the self/presence in the VE should allow the user to act and explore in a natural manner and not restrict normal physical actions. This design quality may be limited by the available devices. If haptic feedback is absent, natural expression inevitably suffers.
Close coordination of action and representation. The representation of the self/ presence and behaviour manifest in the VE should be faithful to the user’s actions. Response time between user movement and update of the VE display should be less than 200 ms to avoid motion sickness problems.
Realistic feedback. The effect of the user’s actions on virtual world objects should be immediately visible and conform to the laws of physics and the user’s perceptual expectations.
Faithful viewpoints. The visual representation of the virtual world should map to the user’s normal perception, and the viewpoint change by head movement should be rendered without delay.
Navigation and orientation support. The users should always be able to find where they are in the VE and return to known, preset positions. Unnatural actions such as fly-through surfaces may help but these have to be judged in a trade-off with naturalness (see heuristics 1 and 2).
Clear entry and exit points. The means of entering and exiting from a virtual world should be clearly communicated.
Consistent departures. When design compromises are used they should be consistent and clearly marked, e.g. cross-modal substitution and power actions for navigation.
Support for learning. Active objects should be cued and if necessary explain themselves to promote learning of VEs.
Clear turn-taking. Where system initiative is used it should be clearly signalled and conventions established for turn-taking.
Sense of presence. The user’s perception of engagement and being in a ‘real’ world should be as natural as possible.
…….. you can create some of your own heuristics as well.
Sutcliffe, A. and Gault, B. (2004). Heuristic evaluation of virtual reality applications. [online] Available at: https://www.researchgate.net/publication/222086342_Heuristic_evaluation_of_virtual_reality_applications [Accessed 29 Aug. 2019].
Cognitive Walkthrough
Cognitive Walkthrough is a formal method for evaluating a UI without users.
Focuses on first time use
Task oriented: requires tasks and walkthrough scenarios
Will users be able to follow this scenario? Can you tell a believable story?
Must be aware of user capabilities
article 4: Cognitive walkthrough procedure
Stages of Action Model, Norman (2001)
Formative Evaluation
Formative Evaluation is an observational empirical evaluation method, applied during evolving stage of design, that assesses user interaction by iteratively placing representative users in task-based scenario’s in order to identify usability problems, as well as to assess the design’s ability to support user exploration, learning and task performance. Can be done in a formal and informal way.
article 9: Case study Evaluation of the therapist manual and interfaces of the Rutgers Ankle Rehabilitation System (RARS)
B. Testing USER TASKS: checking its quality
Navigation & Wayfinding
System Control
Selection & Manipulation
Feedback, Feedforward & Force feedback
Tools: Hierarchical Task Analysis - User Task Flow - User Testing - Heuristic Evaluation - Biometric measuring - Questionnaires - Interviews
Hierarchical Task Analysis
Identify the task to be analyzed: Pick a persona and scenario for your user research, and repeat the task analysis process for each one. What is that user’s goal and motivation for achieving it?
Break this goal (high-level task) down into subtasks: You should have around 4–8 subtasks after this process. If you have more, then it means that your identified goal is too high-level and possibly too abstract. As Don Norman (1998) said, users are notoriously bad at clearly articulating goals: e.g., ”I want to be a good mom” – where do you even begin? Each subtask should be specified in terms of objectives. Put together, these objectives should cover the whole area of interest—i.e., help a user achieve a goal in full.
Draw a layered task diagram of each subtask and ensure it is complete: You can use any notation you like for the diagram, since there is no real standard here. Larry Marine shares some helpful advice on the notation he uses, which is examined below.
Write the story: A diagram is not enough. Many of the nuances, motivations and reasons behind each action are simply lost in the diagram, because all that does is to depict the actions and not the reasons behind them. Make sure you accompany your diagram with a full narrative that focuses on the whys.
Validate your analysis: Once you’re happy with your work, review the analysis with someone who was not involved in the decomposition, but who knows the tasks well enough to check for consistency. This person can be another team member working on the same project, but you could also enlist the help of actual users and stakeholders for this purpose.
Heuristic Evaluation VR
Heuristic or guidelines-based expert evaluation is a method in which several usability experts separately evaluate a UI Design by applying a set of heuristics or design guidelines that are either general enough to apply to any UI of are tailored for 3D UIs in particular. No representative users are involved. Below a few different Heuristic setups are presented. Have a look at them and then choose which one suits your needs. Heuristic Evaluation for Gameful DesignHeuristic-Evaluation-for-Gameful-Design.pdf article 5: Heuristic Evaluation proces article 6: Comprehendable Heuristics for VR systems, Murtza, Youmans, Monroe, 2018 article 7: Online VR Heuristic Evaluation Tool article 8: Heuristics specified for Virtual Reality, 2004, accessed Jul 18 2018
Summative Evaluation
Summative evaluations typically occur after user interface designs are complete, and are comparing specific differences between configurations. As such, they are most appropriate for latestage prototypes in which general usability has been established, but specific interaction or interface questions persist. Comparing 3D UI’s requires a consistent set of user task scenarios.
Tools: User Journey - Touchpoints - Proxemic areas - Action spaces - Rapid Prototype - Play-acting (see Class 1: Design Tools)
Last updated
Was this helpful?