Understanding reliability diagram for classification

@syed.danish wrote:

Hi everybody,
I was going through a research paper(attached below if needed) on predicting good probabilities and it suggests two methods to do the same.
1. Platt scaling
2. Isotonic regression

It was mentioned there that on real problems where the true conditional probabilities are not known, model calibration can be visualized with reliability diagrams. First, the prediction space is discretized into ten bins. Cases with predicted value between 0 and 0.1 fall in the first bin, between 0.1 and 0.2 in the second bin, etc. For each bin, the mean predicted value is plotted against the true fraction of positive cases. If the model is well calibrated the points will fall near the diagonal line.

Ques : Please help me understand the highlighted line in the above paragraph.

Another thing that was mentioned here was Platt Scaling is most effective when the distortion in the predicted probabilities is sigmoid-shaped. Isotonic Regression is a more powerful calibration method that can correct any monotonic distortion.

Ques : Please help me understand the highlighted line in the above paragraph.

Thanks in advance,
Syed Danish

Predicting good probabilities with supervised learning.pdf (399.5 KB)

Posts: 3

Participants: 2

Read full topic