Add uncertainty to estimates

[endolith]

"Estimated variance of estimated probability"

https://gist.github.com/endolith/9e73ea30511f0befb9b336526f34d0a6

https://stats.stackexchange.com/q/430409/11633

https://stats.stackexchange.com/q/576491/11633

[endolith]

Error bars often represent one standard deviation of uncertainty, one standard error, or a particular confidence interval (e.g., a 95% interval). These quantities are not the same and so the measure selected should be stated explicitly in the graph or supporting text.

🤔

[endolith]

https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.errorbar.html

[endolith]

Figured out how to do 95% confidence intervals and plot error bars for binary tests like Condorcet cycle likelihood and Condorcet efficiency using scipy.stats.binomtest(n_successes, n_trials).proportion_ci():

https://github.com/endolith/elsim/compare/error_bars

Adding confidence intervals for total utility or Social Utility Efficiency, on the other hand, is much less obvious and I've spent far too long trying to figure it out. scipy.stats.bootstrap is probably appropriate, but very slow. Generating more data could provide credible intervals(?) but is also extremely slow. Giving up for now.