I
will introduce the concept of calibration of
classifiers, and present some impossibility and
possibility results for probabilistic
classifiers in ML. I will discuss a tractable
extension of post-hoc calibration for multiclass
classification problems that is achievable
without making any distributional assumptions on
the data (beyond being iid), or any assumptions
on the initial classifier. I will also mention
how to perform binning without sample splitting,
and how to handle label shift and covariate
shift, and how to recalibrate with data arriving
sequentially.