A Quick Guide to Conducting Statistical Analysis of HRI Data¶

How Many Participants Should I Recruit? A Guide to Conducting a Power Analysis to Determine Experiment Sample Sizes¶

Conducting a power analysis is the best way to answer the question "How many participants should I recruit for my HRI study?" A good tool to use is the free and well-used G*Power software.

When calculating sample size using a power analysis, G*Power is pretty easy to follow. You select the statistical test you plan to run, the type of power analysis you want to conduct, and input some parameters, and you have the number of participants you should run in order to ensure that your study is sufficiently powered. Some notes on how to determine your parameter values:

Effect size: the best way to determine an effect size is to find a human subjects study examining a similar research question to yours and use the effect size reported in that other study. If you cannot find a listed effect size, it's best practice to assume a medium effect size. A "Rules of thumb on magnitudes of effect sizes" web page can be helpful for determining small/medium/large effect sizes and converting between effect sizes.
α err prob: typically, we use a value of 0.05 here, representing the fact that statistical significance is determined by p-values less than 0.05.
Power (1-β err prob): this represents the desired statistical power of the study. For in-person human subjects studies, a power of 0.80 (80%) is recommended. For online studies (e.g., Prolific, AMT), you may want to go with a higher power of 0.90-0.95.
Number of groups: the number of experimental between-subjects groups your study contains.
Numerator df: the number of degrees of freedom your study contains; a Degrees of Freedom Tutorial can help with this.
Number of measurements: if you are conducting a study where you repeatedly take measurements of the same participant, this value represents how many measurements each participant receives.

As an example: a power computation for a 1-way ANOVA analysis with a medium effect size and 80% power for a 3-condition between-subjects design gives a total sample size of 159. Since 159/3 = 53, you would recruit and run 53 participants in each of the 3 between-subjects conditions.

Which Statistics Test Should I Run? Independent Samples¶

Software Setup¶

You can use the R programming language and RStudio IDE to perform data analysis. R is free, has solid online documentation, and enables users to perform a wide array of statistical tests that are not always available on other platforms (e.g., multi-level mixed-effects models). If you're doing something simple, you could look at using SPSS, which is provided free for student use at Queen's.

Working with an Example¶

This guide uses an example from another researcher's (Sarah Sebo's) prior studies. The main focus of the study was a 30-minute period where 3 human participants (2 in-group members, 1 out-group member) tried to decide which survival items they would want to bring with them in an imaginary survival scenario. One participant was designated as the robot liaison, the only one who could ask the robot for information about the survival items and environment. The study had a 2 (robot liaison: in-group or out-group member) x 2 (robot backchannels: present or absent) between-subjects design. Researchers were interested in whether the robot liaison role and whether the robot backchanneled would influence how included and psychologically safe people felt in the group, as well as participants' backchanneling behaviour. Participants were surveyed before the experiment (age, gender, prior familiarity, extraversion, emotional intelligence) and after the interaction (inclusion, psychological safety). Researchers counted the number of backchannels produced by the robot and by the group from interaction logs. See the HRI 2020 and Frontiers in Psychology 2020 publications for more details.

Each group is considered one data point (examining the influence of independent variables on the aggregated behaviour of the entire group), ensuring that each data point is independent of all others in the analysis. All of the tests described in this section are only valid if each sample is independent of the other samples in the dataset. If each participant were considered one data point instead, participants in the same group would not be independent of one another, violating the assumption of independent samples.

Selecting the Right Statistical Test¶

First, identify the following variables:

Independent Variable(s): any variables that the researchers directly manipulate; in this example, there are 3 possible independent variables: robot liaison identity (in-group or out-group member), the presence of robot backchannels (present or absent), and the number of robot backchannels.
Dependent Variable: the variable being studied, also called the outcome variable.
Covariate(s): a subclass of independent variables that are not the primary variables manipulated by researchers but may still influence the dependent variables; in this example: gender composition, average age, average prior familiarity, average extraversion, and average emotional intelligence of the group.

Next, consider whether the independent and dependent variables are categorical or continuous. Categorical variables can be sorted into categories and do not hold numeric value even if represented by a number (e.g., gender, presence of robot backchannels, any yes/no variable). Continuous variables hold numeric values that vary along some numeric range (e.g., Likert ratings, counts of behaviour occurrences, a participant's age).

With this in hand, determine which test to run:

Independent variable(s) categorical, one independent variable, dependent variable categorical: no covariates → Chi-Square test; any covariates → logistic regression.
Independent variable(s) categorical, one independent variable, dependent variable continuous: no covariates → t-test or 1-way ANOVA (analogous); any covariates → 1-way ANOVA.
Independent variable(s) categorical, multiple independent variables: dependent variable categorical → logistic regression; dependent variable continuous → 2-way ANOVA.
Independent variable(s) continuous, dependent variable categorical: no covariates → Pearson correlation or logistic regression; any covariates → logistic regression.
Independent variable(s) continuous, dependent variable continuous: no covariates → Pearson correlation; any covariates → linear regression.

Examples of Each Statistical Test in R¶

Data file: jibo_survival_experiment_data_independent_samples.csv
R file: stats_test_for_independent_samples.R

Migration note: these two files were referenced as downloadable attachments on the source page; re-attach them to the corresponding wiki page once migrated (a GitLab wiki page can carry file attachments directly).

Which Statistics Test Should I Run? Grouped (Non-Independent) Samples¶

Software Setup¶

As with independent samples, R and RStudio are used, since R has strong documentation and support for statistical tests for grouped samples not always available elsewhere (e.g., the multi-level mixed-effects models described below).

Working with an Example¶

This example uses another of Sara's prior studies, where each person is treated as one individual data point "grouped" with the other participants they interacted with. Three participants and one Nao robot completed a collaborative task of constructing railroad routes on a tablet screen; the team succeeded a round only if every member built a valid section. There were 30 rounds; at the end of each, the robot either made a vulnerable utterance, a neutral utterance, or said nothing, forming 3 between-subjects conditions (vulnerable robot, neutral robot, silent robot). Researchers were interested in how the robot's utterances shaped how people viewed the robot and how participants interacted during the game (e.g., whether they explained a mistake to the group). See her HRI 2018 paper, PNAS 2020 paper, and/or her dissertation for more detail.

Mixed-Effects Models¶

Because the data associated with each participant is not independent of the rest of the participants' data, t-tests and ANOVAs (outlined above) cannot be used; instead, a mixed-effects model is needed, which captures both fixed effects (non-random effects such as experimental condition variables and covariates) and random effects (random variables in a statistical model, here the groupings of participants). For example, if participants 1, 2, and 3 are in group A and participants 4, 5, and 6 are in group B, one variable in the model captures participant groups.

When specifying the random effect, you also need to specify how it relates to the intercept and slope of the model; a random intercept with a fixed mean is typical for random effects applied to participants in groups. In R, this is specified as (1 | group), assuming "group" is the variable name representing the random effect.

Other recommended resources on mixed-effects models:

Mixed Models with R by GitHub user m-clark
LME4 Tutorial: Popularity Data by Rens van de Schoot

Selecting the Right Statistical Test¶

The selection process is simpler for grouped samples, though fitting the model is more complex. The main determining factor is whether the dependent variable is categorical or continuous:

Categorical dependent variable → generalized linear mixed-effects model with a binomial family and logit link (the glmer function from the lme4 R package, with family=binomial(link = "logit")).
Continuous dependent variable → linear mixed-effects model (the lmer function from the lme4 R package).

Achieving a Good Model Fit¶

One way to refine the statistical model is to find the model with the best fit by systematically eliminating covariates that do not have a significant influence on the model's outcomes: run the mixed-effects model (with REML=FALSE if using lmer) and eliminate covariates one by one, starting with the largest p-value, as long as it's not lower than 0.05. Compare the model fits by running an ANOVA test on the models, which outputs AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion); the lower these values, the better the model fit (BIC imposes a larger penalty for additional variables and prefers a less complex model). Choose the model with the lowest value of your chosen criterion, then run the final version of that model (with REML=TRUE if using lmer).

A few other checks are worth doing to ensure a good model fit:

Ensure plots of the residuals don't contain values greater than 3 standard deviations from the mean and that there aren't any trends.
Plot the dependent variable (y-axis) against the model predictions (x-axis) to ensure the model seems to be fitting well (data points clustered around the prediction line).

Examples of Each Statistical Test in R¶

Categorical dependent variable, generalized linear mixed-effects model:

Data file: nao_vulnerability_experiment_repeated_measures_data.csv
R file: stats_test_for_grouped_samples_categorical_dv.R

Continuous dependent variable, linear mixed-effects model:

Data file: nao_vulnerability_experiment_survey_data.csv
R file: stats_test_for_grouped_samples_continuous_dv.R

Migration note: these four files were downloadable attachments on the source page; re-attach them to the corresponding wiki page once migrated.

Additional Resources¶

Jacob Wobbrock's "Practical Statistics for HCI" is a helpful resource for foundational data analysis skills in HCI.
A workshop-provided Excel spreadsheet for figuring out what kind of statistics to run based on experimental design was also linked on the source page as a download; re-attach it once migrated.