Experiments on the web¶

by Christophe, Sander, & Lore

Retreat 2020

Find this tutorial (and more) on https://github.com/gestaltrevision/Reproducible_workflow_tutorial

Intro¶

• Our lab already has a history in doing online experiments (check L-POST for an example, see also: psytests).

• So far most of those tests were developed by Christophe (or his predecessor).

• Indeed, in general experiments in the browsers are more complex than regular offline experiments: Christophe's intro to the logic of the web

• However, increasingly tools are becoming available that facilitate experiment building for the web for non-professional web-builders: I will introduce a few

• Browsers have advanced a lot in capabilities, allowing very complex, dynamic environments and tasks (like video games). (examples)

• Online recruitment platforms have become easy to use: Lore and I will introduce two popular options

Why can’t we just ask Christophe to continue to do this for us?¶

• That’s still a possibility but experimenters generally need a lot of control over things and go through iterative development
• If you can do it yourself, it's faster (Christophe will have a waiting list)
• Understanding how it works is important when conceiving the design and when interpreting your data

Note: we focus on actual experiments only here, if you need options for questionnaire-only studies:¶

• FormR (Eline has experience with this)
• LimeSurvey
• Qualtrics
• Experiment factory (paper)
• Google forms

Why experiments on the web?¶

• Larger samples (bigger pool), easy recruitment (potentially more diverse, more tunable to study).
• Reaching less accessible populations (cf. autism, social anxiety, visually impaired populations,...)

• Inexpensive though it is recommended to give ethical rewards
• Increased anonymity, cf. sensitive research topics (but also potential for depersonalization: cf. low wage workers)
• No face-to-face interactions -> standardization & clarity of instructions. Reducing undocumented experimenter bias (e.g. additional instructions, subtle reinforcing cues, repeating a question) and demand characteristics

• Easy (direct) replication
• Easy sharing, ‘forking’: Increasing reproducibility & transparency
• Faster data collection (many people do your study at the same time)

Advantages (contd.)¶

• Phone or tablet-based surveys (cf. experience sampling) or experiments
• Leverage the ideas/tools of an immense web development community may help you develop better/more engaging experiments (for larger samples, more data per participant)
• You can still run the exact same experiment in the lab as well!
• You learn to build web apps using JS/HTML/CSS/db: A valuable, transferable skill outside of academia

Disadvantages:¶

• Attention & motivation (e.g. participants might be listening to music, watching tv/youtube/social media at the same time, or be interrupted by others): fierce online competition for attention: increased probability of drop-out or low-quality data.

(Partial) solutions:

• Recording how often participants change windows/take breaks might help clean data
• Catch questions could be useful
• Optimize reward scheme
• More thought into how to engage participants (actually a plus)
• Forcing fullscreen

Disadvantages (contd.)¶

• Low social control & no face to face interactions (make sure participants' understand the task)
• Honesty & naiveness (not necessarily worse than lab, cf psych students): excluding participants based on participation in previous studies
• Contextual factors are not (easily) controlled: noise, lighting, distractions
• Data analysis: dealing with extreme response times because of lapses, distraction, hardware/software issues (outliers)
• Learn the new environment: (basics of) JS/HTML/CSS/(db), web logic

Disadvantages (contd.)¶

Hardware/software (browser) heterogeneity and feature support (old versions, old hardware)

(Note: speed of internet is usually not an issue: once (down)loaded, it runs locally).

Avoiding variability:

• Enforcing versions, browsers, excluding tablets/mobile (when desired)
• Checking refresh rates (checking other browser capabilities)
• Preloading media
• Size on screen: matching real-life objects, rule of thumb, participants can make matching between stimuli themselves. See this article for precise methods.

So when are web experiments (un)suitable?¶

Despite the mentioned problems, direct comparisons often show similar quality data and many classic psychology findings replicate with reasonably high fidelity. In practice, whether web experiments are a good idea will depend on your research question(s) and methods.

• When large(r) samples are needed (e.g. for individual differences, norming, etc., but don't we all want them?)
• But research that relies on fewer participants should still aim for the highly controlled lab environment
• Obviously unsuitable when using physiological measures, fMRI/MEG/EEG (though who knows in the future)
• Even psychophysical tasks with brief exposure durations may be possible with high precision according to recent benchmarks (see refs below)
• Comparison papers suggest the reaction times are often slower and noisier but that 1) usually doesn't compromise comparisons between conditions, 2) can be (partly) compensated by larger samples and/or trial numbers
• When very high precision of response times is required, or when absolute response times are important, stick to the lab
• Trade-off between technical complexity/time investment (when learning/designing/implementing) vs ease (& speed) of data collection

Benchmarking and direct comparison studies¶

• Online psychophysics: Reaction time effects in cognitive experiments
• Online versus offline: The Web as a medium for response time data collection
• Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments
• Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments
• Online Timing Accuracy and Precision: A comparison of platforms, browsers, and participant's devices
• The timing mega-study: comparing a range of experiment generators, both lab-based and online

Switch to: Christophes intro to the logic of the web...¶

Existing libraries & frameworks to build/deploy your experiments online¶

Many tools exist, most are pretty recent. Some are builders (graphical point-and-click interface, wysiwyg), some also provide the coding libraries for more flexibility. Some are client-side only ('front-end'), others (usually commercial ones) provide integrated solutions (server space for deploying and storing data). Some 'natural selection' of the field still has to happen. New, more succesful tools may pop up.

This can be overwhelming, but it is intrinsic to working on new frontiers (+ competition is good in software). Don't take my choices as gospel. Choose based on your needs. For me the tool needs to be:

• maximally flexible (allow me to use custom javascript code to implement unconventional structure, custom, dynamic/interactive stimuli,...),
• free of cost,
• open source,
• with some track record,
• maintained by multiple developers (preferably psychologists)
• well-documented (online tutorials + examples).

So I'm putting aside:¶

Integrated (commercial) frameworks with more limited experiment building capabilities (but they might be sufficient for your needs)

• Psytoolkit (paper): low threshold (no js knowledge needed), but very limited, bitmap-based functionality. Aimed at cognitive psychologists. Many classic experiments and questionnaires already implemented and usable through (free) web interface.
• Labvanced: commercial tool, integrated development environment, no js needed, bitmap or questionnaire-based. Includes deploying & data storage (you don't need your own server). No js library
• Gorilla: commercial builder tool (examples). No js library.

Frameworks¶

Agnostic to frontend experiment building (js) tools. Tools for deploying and sharing your experiments. Noncommercial, developed by researchers.

• Experiment factory (paper; Poldrack lab): library of reusable experiments to increase reproducibility.
• Psiturk: framework (dashboard) to manage connection to Mturk. Example experiments available.
• Pavlovia: From the makers of psychopy. Still under development (documentation lacking). Provides a server to deploy and share your experiments + saving the data. Based on git (web interface is similar to Github), but straight uploading from the psychopy builder is possible. Noncommercial (funded by Wellcome Trust) but pay per run/participant (0.20 GBP) or unlimited institution license (1500GBP). Server in UK, fully GDPR compliant. Stroop example, code. Intro/tutorial video

Focus on frontend (client-side) experiment building¶

Most basic option: Vanilla javascript from scratch: most flexible but time-consuming, technically challenging. Building your own timeline, data structures and stimuli. Only for experienced (or ambitious) web developers. Js is an ugly, bastardized language (especially if you come from python, R fans are already used to some self-flagellation ;-) but you'll learn to love it because of its powers and expressivenes in the browser.

Here the real race (so far) is between these three¶

These are free, open source js libraries and all include a builder. They provide a timeline/scheduler and tools for controlling the flow of an experiment and an (extensible) set of stimuli.

• lab.js (builder, documentation)
• jsPsych: (builder). Also: tools for R/Rstudio (jspsychr; jaysire).
• psychojs: builder (same as psychopy builder). Support for building online experiments is under development: (features currently available). Has the psychopy reputation/resources behind it. Will eventually probably be more advanced in stimulus presentation and recording data. Extra benefit: structure is similar to psychopy.

A brief intro to labjs (demo)¶

• Online WYSIWYG tool, includes instant test runs and data inspector.
• You need to know (or be able to look up) the basics of html and a bit of CSS
• Basic components:
  • 'Screen'
  • flow control: sequences, loops
• Placeholders: ${ variable }
• Data saved (row) per component
• Using the canvas: an html5 element that allows drawing anything (including complex, animated stimuli) using javascript (js).
• Adding js code: possible in every component (in the 'script' tab) e.g.
  • Getting prolific id from URL
  • Sending data to server
  • Sending the completion confirmation to prolific/amazon

Find the demo in https://github.com/gestaltrevision/Reproducible_workflow_tutorial/onlineExp

I've built my experiment, now what?¶

• Export for local data collection: download data link at the end
• Download to deploy on a generic web server (e.g. psytests.be)
• Download and push to github (requires data saving to other server, e.g. psytests.be).

Advanced:¶

• Using github to host/serve (static) pages, why?
  • Iterative development (you don't need to bother Christophe with every new version)
  • You can track changes in time
  • You don't need a separate server to deploy the study
• You can also download the code to edit it offline (recommended editor: Visual Studio Code: free, open source, maintained by Microsoft, many tools for web developers), but that obviously requires knowing the basics of js and of the lab.js library.
• Debugging in console (in Chrome: right-click -> inspect -> console)

Some important considerations¶

• Give explicit instructions and show warnings about closing the page, going back in the browsers, etc.
• Consider 'sanity checks' to avoid bots (especially for Mturk)
• Test your experiment on different browsers/screens!
• Pilot extensively before moving to larger pools!

Switch to: Lores intro to recruitment platforms on the web...¶

References¶

• A primer on running human behavioural experiments online
• Conducting perception research over the internet: a tutorial review
• The Experiment Factory: Standardizing Behavioral Experiments
• Law and Psychology Grows Up, Goes Online, and Replicates
• Retention of participants recruited to a one-year longitudinal study via Prolific
• An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use
• Prolific.ac—A subject pool for online experiments
• Beyond the Turk: Alternative platforms for crowdsourcing behavioral research
• The state of web-based research: A survey and call for inclusion in curricula.
• The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions.
• Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires.
• Running experiments on Amazon Mechanical Turk.
• Amazon's Mechanical Turk A New Source of Inexpensive, Yet High-Quality, Data?
• Using Mechanical Turk to Study Clinical Populations
• Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research.
• Controlling for Participants’ Viewing Distance in Large-Scale, Psychophysical Online Experiments Using a Virtual Chinrest