Joint with Ryan Copus.
Discretion to semi-autonomous administrators is a fact of life for any government. Measurements of inconsistency are important tools for assessing the quality of these administrators’ decisions, and yet high quality measures are rarely available. The core dilemma is that mean differences between decision makers on outcomes—that is, disparity statistics—are not a good proxy for inconsistency, since they systematically understate the level of disagreement. Our contribution is to demonstrate how this downward bias can be mitigated through targeted estimation of heterogeneous treatment effects using machine learning on administrative data. We validate our approach on a dataset of all civil appeals decided by the Ninth Circuit between 1995 and 2013. We also apply our method to evaluate the Ninth Circuit’s use of unpublished opinions. Evidence strongly suggests that, contrary to court policy and conventional wisdom, opinions are more likely to be unpublished where inter-panel disagreement over case outcomes is highest.