Evaluation Methods: Setting up an Experiment

A scientific method is a plan that is followed in performing a scientific experiment and writing up the results. It is not a set of instructions for just one experiment, nor was it designed by just one person. The scientific method has evolved over time after many scientists performed experiments and wanted to communicate their results to other scientists. The scientific method allows experiments to be duplicated and results to be communicated uniformly.

Before you start to define your experiment

1: Explore, think about your personal motivations and your team's ambitions.

• What kind of human interaction is necessary to explore: hands, feet, head, fingers, body?

• What kind of devices do you have and what user tasks can you explore? See Gabbard's taxonomy of usability characteristics below:

Taxonomy of usability characteristics, Joseph Gabbard, 1997

Defining your experiment

2: Form a hypothesis

When preparing to do research, a scientist must form a hypothesis, which is an educated guess about a particular problem or idea, and then work to support it and prove that it is correct, or refute it and prove that it is wrong.

3: Define your research question

Main objective for research: what do I want to find out?
Main research question
Subquestions

4: Think through your research method

- What method will I use for testing? 
- What is the goal of this method?
- What are benefits of using this method? Any possible counter effects? 

5: Have a real good look at your evaluation metrics

• Independent and Dependent Variables A well-designed experiment needs to have an independent variable and a dependent variable. The independent variable is what the scientist manipulates in the experiment. The dependent variable changes based on how the independent variable is manipulated. Therefore, the dependent variable provides the data for the experiment.

Validity and Reliability

Validity

Validity implies the extent to which the research instrument measures, what it is intended to measure. What is the accuracy of the measurement. A valid instrument is always reliable.

Reliability

Reliability refers to the degree to which scale produces consistent results, when repeated measurements are made. Is the experiment executed in a precise manner and is it consistent with former experiments. A reliable instrument need not be a valid instrument.

Additional info about Validity and Reliability