Is This Difference Real?

The bar is the likely range. It shows where the true difference most likely falls (95% confidence). The dot is the difference you entered. "Clear difference" means the whole range sits on one side of the no-difference line. "Too close to call" means the range crosses it.

Two different groups. The range uses the Newcombe score method, which holds up with small groups and with rates near 0% or 100%. The chance figure comes from the same method, so the status, the range, and the chance figure always agree.

Same students twice. When the same students are measured twice (fall and spring), only the students who switched carry information. This uses the exact McNemar test (McNemar, 1947) and its matching exact range. If you can match students, use this even for year over year: a school's students this year and last year mostly overlap, so the two-group test is a little cautious there.

What "chance" means when you have every student. Your rates count every student, so they are not a survey. The test asks whether the difference is bigger than the ups and downs you would expect if a different set of students had gone through the same school.

The chance figure, read correctly. It is how often a gap this big or bigger would show up if there were no real difference. It is not the chance that the difference is real.

Small groups. Under 10 students in a group, the tool shows the rates and the range but makes no call. Under 30, read the range as rough.

Clear is not the same as important. With big groups a small gap can be clear. Decide whether the size of the change matters for your students.

Sources: Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine, 17, 873-890. McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12, 153-157.