Skip to content

Add 95% error bars to simulation examples (exact binomial + bootstrap SUE)#51

Open
endolith wants to merge 1 commit into
masterfrom
cursor/error-bars-monte-carlo-ci-1dd7
Open

Add 95% error bars to simulation examples (exact binomial + bootstrap SUE)#51
endolith wants to merge 1 commit into
masterfrom
cursor/error-bars-monte-carlo-ci-1dd7

Conversation

@endolith
Copy link
Copy Markdown
Owner

Summary

Addresses issue #13 by documenting uncertainty explicitly on the plots that estimate proportions or ratios from Monte Carlo simulation.

Accuracy relative to #13 / #14

  • Issue Add uncertainty to estimates #13 correctly warns that SD, standard error, and CI are different quantities; anything plotted should be labeled. PR Add 95% CI error bars to CE and CW examples #14’s direction (explicit 95% confidence intervals for binomial counts rather than a vague “variance” bar) matches standard practice for yes/no simulation outcomes.
  • Binary / count plots (Condorcet efficiency conditional on a CW existing, Condorcet paradox rate, Niemi comparison): two-sided exact (Clopper–Pearson) intervals from scipy.stats.binomtest(...).proportion_ci(method='exact'), i.e. the inverse binomial test. This is an accepted frequentist CI for a binomial proportion with fixed trial count per point.
  • Social utility efficiency is not binomial; it is implemented here as the ratio of sums actually accumulated in each script: (\hat\theta=\sum_i W_i/\sum_i Z_i) with paired ((W_i,Z_i)) per simulation draw (definitions of (W,Z) match each figure: random-society baseline, spatial-model (A/B), etc.). Uncertainty uses a paired percentile bootstrap on that same ratio statistic—appropriate for the plug-in estimator being displayed.

Shared helpers and rationale live in examples/plot_uncertainty.py.

Examples updated

Plotted examples that expose a scalar estimate per (x)-value now carry asymmetric vertical intervals:

  • Merrill CE / Wikipedia paradox / Niemi: binomial CIs
  • Merrill Fig 3 & 4 (classic + updated), Weber effectiveness tables: bootstrap CIs for SUE

Histogram-style scripts (e.g. distributions_by_*, Tomlinson figures, Merrill Fig 2a–2b scatter) are unchanged—density/scatter visuals do not use the same single-point CI semantics.

Tests

pytest (191 tests) passes locally.

Open in Web Open in Cursor 

- Exact Clopper-Pearson (binomtest) 95% CIs for binomial proportions:
  Condorcet efficiency, paradox rates, Niemi comparison.
- Percentile bootstrap 95% CIs for social utility efficiency ratios
  sum(W)/sum(Z) with paired resampling of simulation draws, matching the
  plotted plug-in estimators.
- Shared helpers in examples/plot_uncertainty.py with brief rationale.

Co-authored-by: endolith <endolith@gmail.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 11, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.31%. Comparing base (8a7213f) to head (0029212).

Additional details and impacted files
@@           Coverage Diff           @@
##           master      #51   +/-   ##
=======================================
  Coverage   96.31%   96.31%           
=======================================
  Files          19       19           
  Lines         489      489           
=======================================
  Hits          471      471           
  Misses         18       18           
Flag Coverage Δ
no-numba 95.50% <ø> (ø)
numba 89.16% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants