Outline

1. Part 1: The Phisolophical Multiverse approach
   

2. Part 2: MB2-Pupillometry Spin-Off Case Study

Thanks so much to these incredibly collaborative people

Co-authors:
Melanie S. Schreiner • Alvin Tan • Marlena Mayer • Robert Hepach and the MB2-P Team

What is Multiverse Analysis?

• “A systematic exploration of all reasonable analytical paths” (Steegen et al., 2016)
  
• Enhances transparency and systematically organizes researcher degrees of freedom
  
• A garden of analytical forking paths: grounded in theory

Why It Matters for Developmental Science

• Universal Applicability: works across contexts and methods
  
• Flexible Implementation: supports exploratory and confirmatory designs
  
• Credibility Enhancement: addresses replication challenges
  
• ManyBabies Alignment: fits with goals of transparency and rigor

Guiding Principles

• Think Before Testing
  Evaluate choices before collecting data.

• Theoretical Grounding
  Base decisions on prior evidence.

• Complete Transparency
  Present all plausible paths.

• Computational Implementation
  Use tools to explore the full analytic space.

Types of Multiverse Approaches

Exploratory Methods

• Vibration of Effects (VoE)
  
• Summary indices
  
• Graphical representations

Focus: Identify patterns and generate hypotheses.

Inferential Methods

• PIMA (Post-selection Inference)
• Sign-flipping score test

Focus: Maintain statistical rigor while exploring variation.

Addressing the Credibility Crisis

Problem: P-hacking

• Selective analysis for significance undermines integrity.

Problem: HARKing

• Hypothesizing after results are known (post hoc as a priori).

Solutions via Multiverse Analysis

• Multiverse Transparency
  Makes all choices visible to prevent selective reporting.

• “Legal P-hacking”
  Exploring multiple paths is valid if systematic and transparent.

The Multiverse Landscape

• Data Inclusion/Exclusion: who/what to include?
  
• Preprocessing Methods: cleaning, transforming
  
• Variable Operationalization: defining constructs
  
• Statistical Models: which methods and why

Visualizing the Multiverse

Specification Curve

Volcano Plot

Volcano Plot

Study Design Overview

• Cross-lab pupillometry with over 30 labs worldwide
• Age cohorts: toddlers and adults
• Condition × Outcome × Time modeling
• From: Calignano, Girardi, Altoè., 2024 First steps into the pupillometry multiverse of developmental science. Behav Res Methods.

ManyBabies 2: Theory of Mind in Infancy

MB2 – Do toddlers and adults engage in spontaneous Theory of Mind (ToM)?

MB2 – ToM: Paradigm

MB2 – ToM

MB2 – ToM

MB2P – Secondary Data Analysis

MB2P – Secondary Data Analysis

MB2P – Secondary Data Analysis

• MB2P investigates responses to goal-congruent vs. goal-incongruent outcomes
• Is the child surprised if the protagonist (bear) acts incongruently with the goal (following the mouse)?
• Measures: pupil dilation and looking time during events

The MB2P Preregistration and the Multiverse Approach

The MB2P Preregistration and Data Simulation

• Data Collection by MB2 completed in 2023. Raw data will be shared after Stage 1.
• Analyses will focus on the second test trial.
• Data Simulation mirrors expected data structure to pre-register processing and analysis strategies.
• Simulated dataset includes:
  • Participant ID, Two age cohorts , Timestamps or durations, x/y coordinates, Pupil size (left/right), Lab ID, Conditions and outcomes

MB2-P Multiverse Structure

• Five Forking Dimensions:
  • Pupil plausibility
  • Gaze within screen
  • Filtering via moving average
  • Baseline correction window (1s, 0.5s, 0.25s)
  • Participant exclusion (none, trial 2, trial 1&2)

Simulated Data

Density Plot by Condition × Outcome × Age

DF1: Pupil Plausibility

DF1: Pupil Plausibility

DF2: Gaze Filtering

DF2: Gaze Filtering

DF3: Moving Average Filter

DF3: Moving Average Filtering

DF4: Baseline Correction

`geom_smooth()` using formula = 'y ~ x'

DF4: Baseline correction from a single Universe

DF5: Participant Inclusion Criteria

# A tibble: 2 × 2
  df5          n
  <chr>    <int>
1 excluded 47200
2 included   800

Specification Curve of the Multiverse’ Models

Specification Curve of Model Estimates

Volcano plot of the Multiverse’ Models

Volcano Plot of Effect Estimates

Thank You

• Questions? Feedback?