henhaobio
Sign in

bioRxivNeurosciencePreprint

Application of the hierarchical bootstrap to multi-level data in neuroscience

Varun SaravananiD, Gordon J. Berman, Samuel J. SoberiD

PDFDOI

In-depth summary

Claude Opus 5.5 · xhigh effort · 17 Oct 2026 · from the full text of version 2 (28 Apr 2020) · CC BY 4.0

Treating neurons as independent gave about 46–96% false positives in simulated nested data; a hierarchical bootstrap stayed near 5% with many subjects and kept more power than per-subject averaging.

Neuroscience data usually nest trials within neurons within animals, yet t-tests and ANOVA treat every measurement as independent, while averaging per animal costs power and linear mixed models (LMMs) assume linear structure and hinge on user choices about random effects. The authors bring a hierarchical bootstrap, common in statistics but rare in neuroscience, to this setting and test it against these alternatives in simulations and two reanalysed datasets.

Key results

  1. SuggestedWhen neurons nested in subjects are treated as independent, simulated t-test false positives climb with the number of neurons per subject, whereas subject-level averaging, the hierarchical bootstrap and an LMM stay at the nominal rate.

    1000 simulated subjects (Poisson firing at 5 Hz, 3 Hz Gaussian per-subject offsets) split at random into groups of 500, 1000 repeats per setting, α = 0.05: the traditional test's false-positive rate rose from around 46% at 10 to almost 96% at 3000 samples per subject; the other three did not differ significantly from 0.05 (Bonferroni, 3 comparisons) and kept a stable SEM. · 3.1.1 The hierarchical bootstrap does not have a…

  2. SuggestedWith few subjects per group, the hierarchical bootstrap and the LMM keep more power than subject-level averaging; the bootstrap has marginally more power than the LMM but a higher false-positive rate when subjects are very few.

    1–16 subjects per group, 100 neurons each, second-group means of 6.5, 8 or 9.5 Hz vs 5 Hz: traditional tests had the most power but stayed around or above 80% false positives; averaging held near 5% with the lowest power; at N=1 the bootstrap matched the traditional false-positive rate while the LMM gave 16 ± 1%. Power values are plotted, not stated, in this text. · 3.1.3 The Bootstrap and LMM balance intended…; 4 Discussion

  3. ShownReanalysing the authors' earlier songbird pitch-shift study, the hierarchical bootstrap does not support the anti-adaptive pitch change in different-type syllables that traditional and syllable-level summarized tests reported, while still confirming the target-syllable shift.

    Different-type syllables: −0.087 ± 0.003 semitones, p = 4×10^-125 (traditional); −0.12 ± 0.06, p = 0.053 (summarized; 0.048 originally); −0.09 ± 0.09, pboot = 0.25 (bootstrap). Target: 0.34 ± 0.12, pboot = 0.995. An LMM called the different-type shift significant (−0.12 ± 0.06, p = 0.036), which the authors attribute to likely misspecified random effects. · 3.2.1 The bootstrap highlights the risk of false…; 4 Discussion

  4. SuggestedThe hierarchical bootstrap shows no conservative bias on hierarchical data but is clearly conservative when all data points are independent, where the other three methods sit at about 5% false positives.

    Independent-data simulation (100 neurons for each of 1000 subjects, no subject offset, 10000 repeats): false-positive rate 0.66 ± 0.08% for the bootstrap vs 4.80 ± 0.21% traditional, 4.99 ± 0.22% summarized and 4.80 ± 0.21% LMM; bootstrap error bars roughly 1.4 times larger (1.407 ± 0.035 vs 1.002 ± 0.002 ×10^-2 for the traditional test). · 3.1.2 The hierarchical bootstrap is more…; 3.1.1 The hierarchical bootstrap does not have a…

  5. SuggestedOn fly optogenetic behaviour maps, the hierarchical bootstrap outlined a compact head-grooming region, whereas the original traditional analysis added a separate region judged false positive and summarized statistics found no significant region.

    Six experimental and six control flies, 30 trials each, descending neurons G7/G8 (line SS02635); traditional = Wilcoxon rank-sum tests with Sidak correction. The false-positive call rests on visual inspection of every trial; an LMM with fly random effects picked a smaller, partly discontinuous region, read by the authors as false negatives. · 3.2.2 The hierarchical bootstrap captures the…

Methods, data and reproducibility

Simulated Poisson firing of neurons nested in subjects (usually 100 per subject, with or without Gaussian subject offsets) and compared false-positive rate, SEM and power of four tests (t-test on all neurons, t-test on subject means, a hierarchical bootstrap resampling subjects then neurons, an LMM with a subject random effect), then reanalysed songbird pitch-shift and fly optogenetic behaviour data with them.

  • Simulated neuronal firing (Poisson, 5 Hz mean, 3 Hz Gaussian per-subject offsets or none), usually 100 neurons per subject (number varied in the first simulation) · n = 1000 subjects in groups of 500; 1–16 subjects per group in the power simulations
  • Reanalysed vocal generalization data: 1-semitone auditory pitch shift of a target syllable via headphones, last 3 days of the shift period · Bengalese finch (Lonchura striata var. domestica), male
  • Optogenetic activation of descending neurons G7 and G8 (line SS02635), behaviour mapped onto a two-dimensional density of stereotyped behaviours · Drosophila melanogaster · n = six experimental and six control flies, 30 trials each
Methods & tools
hierarchical bootstrap, linear mixed models (LMM), Student's t-test, Wilcoxon rank-sum test with Sidak correction, Python, NumPy, SciPy, Matplotlib, StatsModels, Pandas, MATLAB, Jupyter Notebooks
Reagents & environment
Simulations in Jupyter Notebooks with Python 3.7.2, NumPy 1.15.4, SciPy 1.1.0, Matplotlib 3.0.2, StatsModels 0.11.0 and Pandas 0.23.4; reanalysis in MATLAB 2017a.
Reproducibility gaps
Simulation and analysis code is said to be on GitHub, but no address survives in this text; there is no availability statement for the songbird or fly data.
Limitations (6)
  • The bootstrap is conservative when data points are truly independent: its false-positive rate fell to 0.66 ± 0.08% against the nominal 5%, with error bars roughly 1.4 times those of the other methods. · stated by the authors
  • With very few upper-level units the bootstrap treats lower-level samples as the population, so its false-positive rate approaches the traditional test's (equal with one subject per group), above the LMM's 16 ± 1% at N=1. · stated by the authors
  • The bootstrap suits only metrics that obey the law of large numbers and are linear in probability space (entropy-like quantities are excluded), and the resampling scheme at each level must be chosen to mimic real repeats of the experiment. · stated by the authors
  • Because of a data-archiving discrepancy, the recomputed summarized p-value for different-type syllables (0.053) differs from the originally published 0.048. · stated by the authors
  • All simulations use one additive design (Poisson firing with or without Gaussian subject offsets); the claimed edge over LMMs for non-linear hierarchies and the ease of scaling to large datasets are argued, not simulated. · noted in this summary
  • pboot is read as the probability that a hypothesis is true (0.85 taken to mean generalization would appear in 85% of new birds); this treats the bootstrap distribution as a posterior and is asserted rather than checked in the simulations. · noted in this summary

Read this if you analyse trials nested in neurons nested in animals and want a simulation-backed comparison of t-tests, per-animal averaging, LMMs and a hierarchical bootstrap, with worked reanalyses of real behavioural data.

Citations
170

Abstract

AO_SCPLOWBSTRACTC_SCPLOWA common feature in many neuroscience datasets is the presence of hierarchical data structures, most commonly recording the activity of multiple neurons in multiple animals across multiple trials. Accordingly, the measurements constituting the dataset are not independent, even though the traditional statistical analyses often applied in such cases (e.g. Students t-test) treat them as such. The hierarchical bootstrap has been shown to be an effective tool to accurately analyze such data and while it has been used extensively in the statistical literature, its use is not widespread in neuroscience - despite the ubiquity of hierarchical datasets. In this paper, we illustrate the intuitiveness and utility of this approach to analyze hierarchically nested datasets. We use simulated neural data to show that traditional statistical tests can result in a false positive rate of over 45%, even if the Type-I error rate is set at 5%. While summarizing data across non-independent points (or lower levels) can potentially fix this problem, this approach greatly reduces the statistical power of the analysis. The hierarchical bootstrap, when applied sequentially over the levels of the hierarchical structure, keeps the Type-I error rate within the intended bound and retains more statistical power than summarizing methods. We conclude by demonstrating the effectiveness of the method in two real-world examples, first analyzing singing data in male Bengalese finches (Lonchura striata var. domestica) and second quantifying changes in behavior under optogenetic control in flies (Drosophila melanogaster).

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISKolpakova '22Wilcox '18Alger '19Coventry '24Dimmock '25Frankot '23
66% semantic matchNeuroscience
Jenya Kolpakova, Jonathan D. Marvel-Zuccola, Kensuke Futai+2 more·bioRxiv·Mar 23, 2022