misskappa

A C++17 library and R package for weighted agreement coefficients with any number of raters, arbitrary pairwise loss matrices, and support for missing ratings.

We support weighted variants of Cohen’s kappa (1960, 1968), its multirater variant Conger’s kappa (1980), Fleiss’ kappa (1971), and the Brennan–Prediger coefficient (1981).

Missing data is handled by several estimators. For categorical data, Cat-FIML is efficient under MAR and MCAR assumptions. For quadratic weights, pairwise is consistent under MCAR, while robust NT-FIML (Yuan & Bentler, 2000) is consistent under MCAR and efficient under MCAR and MAR under normality. For general weights and arbitrary data, IPW is consistent under MCAR.

In addition to agreement coefficients, misskappa estimates Cronbach’s coefficient alpha (1951) under missing data, with pairwise-available, robust NT-FIML (Zhang & Yuan, 2016), and categorical Cat-FIML estimators.

Installation

Install the R package from GitHub with remotes. It builds from source, so a C++17 toolchain is required (Rtools on Windows, the Command Line Tools on macOS).

# install.packages("remotes")
remotes::install_github("JonasMoss/misskappa", subdir = "r-package")

Usage (R)

library(misskappa)

Basic agreement

The workhorse is kappa(). Here five raters sort subjects into three categories, and some of the ratings are missing (Klein, 2018). The IPW estimator stays consistent under MCAR:

kappa(dat.klein2018, estimator = "ipw")
#> misskappa: estimator=ipw, weight=nominal
#>                  estimate     se  lower  upper
#> Conger             0.4301 0.1050 0.2244 0.6358
#> Fleiss             0.4054 0.1196 0.1710 0.6399
#> Brennan-Prediger   0.4204 0.1136 0.1978 0.6430

You get back the weighted Conger, Fleiss, and Brennan–Prediger coefficients, each with a standard error. (With two raters you would get Cohen’s kappa instead of its multirater cousin Conger.)

Want to know whether two studies agree to the same degree? kappa_test() checks it for you. The Gwet (2014) and Klein (2018) data come from different subjects, so the two samples are independent and we set paired = FALSE:

kappa_test(
  gwet  = kappa(dat.gwet2014,  estimator = "ipw"),
  klein = kappa(dat.klein2018, estimator = "ipw"),
  coef = "Conger", paired = FALSE
)
#> 
#>  Independent-sample test of equal Conger across 2 fits
#> 
#> data:  gwet, klein
#> X-squared = 0.0019646, df = 1, p-value = 0.9646
#> sample estimates:
#>      gwet     klein 
#> 0.4233051 0.4301049

kappa() also does g-wise kernels, where the disagreement function compares g > 2 raters at once instead of averaging over pairs:

kappa(dat.gwet2014, estimator = "ipw", weight = "nominal", g = 3)
#> misskappa: estimator=ipw, weight=nominal, g=3
#>        estimate     se  lower  upper
#> Conger   0.4369 0.0850 0.2703 0.6034
#> Fleiss   0.4308 0.0861 0.2620 0.5997

Coefficient alpha

misskappa estimates Cronbach’s alpha under missing data too. Here it runs on the Neuroticism items of psych::bfi. That is 2800 respondents from the SAPA project (Revelle, Wilt & Rosenthal, 2010), and 106 of them skipped at least one answer. Robust normal-theory FIML keeps every respondent:

data(bfi, package = "psych")
N <- paste0("N", 1:5)
alpha(bfi[, N], estimator = "nt_fiml")
#> misskappa: estimator=nt_fiml, weight=score
#>       estimate    se lower  upper
#> alpha   0.8138 0.006 0.802 0.8256

alpha_test() compares reliabilities the same way kappa_test() compares agreement. Is the Neuroticism scale equally reliable for men and women?

Different people answer in each group, so the samples are independent again:

g <- split(seq_len(nrow(bfi)), bfi$gender)
alpha_test(
  men   = alpha(bfi[g[["1"]], N], estimator = "nt_fiml"),
  women = alpha(bfi[g[["2"]], N], estimator = "nt_fiml"),
  paired = FALSE)
#> 
#>  Independent-sample test of equal alpha across 2 fits
#> 
#> data:  men, women
#> X-squared = 3.3766, df = 1, p-value = 0.06613
#> sample estimates:
#>       men     women 
#> 0.7959317 0.8209147

Frontier / experimental

There is early support for vector-valued ratings, where each rating is a vector and the weights act one component at a time. This part is still active research and the paper is in preparation, so expect the surface to shift. The vector-valued agreement article walks through it.

Every result is a misskappa_estimate object holding the estimates and their covariance matrix. Reach for coef(), vcov(), and confint() when you want the pieces.

Documentation

The package website has the full reference and the worked-example articles:

• Getting started
• g-wise agreement
• Agreement coefficients and loss matrices
• Missingness and estimators
• Testing equality of agreement coefficients
• Validation strategy
• Vector-valued agreement (frontier / experimental)
• C++ API reference

Contents

• C++17 library in include/misskappa/ and src/
• R package in r-package/, wrapping the library via Rcpp
• Manuscripts in papers/, the completed papers accompanying this package

Quick build

cmake --preset dev
cmake --build --preset dev
ctest --preset dev

Or via just:

just test            # dev build + ctest
just r-install       # build opt + install R package
just r-check         # R CMD check + testthat
just paper <slug>    # build the manuscript PDF for papers/<slug>/

See AGENTS.md for the project contract and each paper’s AGENTS.md for manuscript-specific direction.

License

Code under the MIT License, Papers under CC BY 4.