nonprobsvy Coverage

What this is

Empirical evidence that the confidence intervals produced by the nonprobsvy R package achieve their nominal coverage levels under the data-generating processes from the methods literature. Modeled after doubleml-coverage.

Every coverage table on this site is computed automatically by GitHub Actions and updated whenever the package or this repo’s simulation code changes.

How to read the tables

Each row is one parameter cell — one combination of sample sizes, link function, penalty, variance method, etc. The columns that vary across rows (e.g. penalty for SCAD/lasso/MCP, k for nearest-neighbour, var_method for analytic vs bootstrap) are shown on the left of the table; columns that are constant across the whole table are hidden.

What the numbers mean

  • n_reps — how many Monte Carlo replicates produced this row.
  • bias — mean of point estimate − true population mean. Closer to zero is better.
  • rmse — root mean squared error of the point estimate.
  • mc_se — Monte Carlo standard error of the point estimate itself (≈ sd(point) / sqrt(n_reps)). Captures how precisely the simulation pins down the bias and coverage values.
  • mean_se — mean of the estimator’s reported standard error across replicates. The estimator is well-calibrated when mean_se ≈ mc_se × sqrt(n_reps) (the empirical sd of the point estimates).
  • coverage — proportion of replicates whose confidence interval contained the true population mean. This is the headline.
  • ci_width — mean width of the confidence interval.

What the cell colors mean

A coverage estimate from n_reps simulations isn’t infinitely precise — it has its own Monte Carlo SE of sqrt(p(1-p) / n_reps) (about 0.010 at p = 0.95 with n_reps = 500). The color compares the gap between empirical and nominal coverage to that Monte Carlo SE:

  • green — the gap is within 1 × MC SE. The estimator looks correctly calibrated; any residual gap is consistent with simulation noise.
  • amber — gap is between 1 and 2 × MC SE. Borderline; could plausibly be either real miscalibration or sampling noise. Worth re-running with more replicates to disambiguate.
  • red — gap is more than 2 × MC SE. Almost certainly real miscalibration, not simulation noise. Under-coverage (red and low) means the CI is too narrow; over-coverage (red and high, e.g. 1.000) means the CI is too wide / conservative.

Concrete examples: at n_reps = 500 and α = 0.05 (nominal coverage 0.95, MC SE ≈ 0.010):

  • 0.945 → green (gap 0.005, within 1 MC SE)
  • 0.930 → amber (gap 0.020, between 1 and 2 MC SE)
  • 0.890 → red (gap 0.060, beyond 2 MC SE → CI too narrow)
  • 1.000 → red (gap 0.050, beyond 2 MC SE → CI too wide)