How we grade

Every credible range here is a judgement call, so every one carries a written rationale. A scorecard that cannot be disputed line by line is not a scorecard, it is an accusation.

What is being measured

Not quality. The grade answers one question: of the things this institution was in a position to tell the public, how much did it actually tell them? A well-funded university with poor outcomes that reports every field completely scores higher here than a good school that reports nothing. Outcomes are graded elsewhere, by other people, using the very data this project is checking the existence of.

How an absent value is classified

A field an institution did not report, a field suppressed to protect a small cohort, and a field whose true value is zero are three different facts. Rendered carelessly they all become 0 on a page, and a reader cannot tell a college that admits nobody from a college that declined to say. Every value is classified before anything else touches it.

Suppression is a policy decision made for good reasons and is never held against anyone. Penalising an institution for protecting a twelve-person cohort would push publishers toward disclosing things they should not, which is the opposite of the point. Suppressed and not-applicable fields leave the denominator rather than scoring zero, and an institution with an empty denominator gets no grade at all, not a zero.

Why a published value can still be a failure

A value being present is not sufficient; it also has to be credible for the field it sits in. A published zero survives in federal data because zero is a legal number, not because anyone measured it. A wrong number published as fact is worse than a gap, because the gap is visible to a reader and the wrong number is not.

Peer comparison

A fixed credible range is a blunt instrument and an institution can reasonably object to one. So every implausible finding carries its peer group: sector, level, and state, with the institution excluded from its own comparison. A reader who thinks a grade is unfair can see what comparable institutions published and attack the peer definition, the sample, or the conclusion. All three are better arguments to be having than one about where a constant was set.

Where the peers turn out to publish the same value, the verdict says so and the finding argues against itself. That is intended and is not suppressed.

A peer claim is only made where at least 10 comparable institutions exist and at least 10 of them published the field. Both counts, because the second is what the comparison is made out of: a group of fifty in which six published a number supports a claim about six, and the finding says so instead of borrowing the confidence of the fifty. Below either bar the finding is still reported and still links here; it simply arrives without a peer comparison attached, because an unsupported comparison is worse than none.

Letter bands

A, 95% and above. B, 85%. C, 70%. D, 50%. F below that. Fixed rather than curved, because grading on a curve would hide a field-wide collapse in reporting: if everyone stopped publishing graduation rates tomorrow, a curve would report that as normal.

Drift

A single snapshot cannot distinguish a field that was never collected from one that was collected until recently and then stopped. The first is a gap in the data model; the second is a change in what the public is allowed to know. Only the comparison between runs can tell them apart, so per-field counts are committed to version control on every run.

Drift is a change in rate, not a change in count. Every comparison divides by the institutions the field applied to in that run, and the reason is a mistake this project made and published to itself. Measured on counts, three IPEDS collection years produced three confident systemic findings and all three were false: between 2021 and 2023 the directory shrank from 6,289 institutions to 6,163, so 130 fewer published a web address, and that was reported as a systemic 2.1% collapse. The share publishing one had gone up, from 99.93% to 99.95%. Colleges closed; they did not stop reporting. The one real movement in the period, the athletics disclosure rising from 57.1% to 59.4%, ranked fourth and was never flagged, because 52 is a small number next to 130.

A change is called systemic when that rate moves by at least 2 percentage points. The threshold is a judgement call, stated here so a reader can disagree with it, and set low because a coordinated stop-reporting event is newsworthy well before it touches a majority of institutions. Three real collection years say it is roughly right: every year-on-year movement in these six disclosures sits under one point except the athletics disclosure, which rose 1.75 points in a year and 2.26 across two. At 2% the bar flags that and nothing else. At 1% it would report ordinary annual churn as policy; at 5% it would have found nothing in three years of federal data, which is not a measurement but a way of never having to say anything.

Drift is reported in both directions: fields that started being reported are as real a finding as fields that stopped, and reporting only the losses would make this an argument rather than a measurement. A field whose rate cannot be computed in either run is reported as unmeasured and is never called systemic, because an unknown is not a large movement.

The fields: College Scorecard

The fields: IPEDS

IPEDS records public disclosures the Scorecard does not carry, and states absence three different ways, all of them negative integers: -1 not reported, -2 not applicable, -3 not available. They are not interchangeable, and only the first counts against an institution.

The athletics disclosure below was ungraded until a second IPEDS file arrived, and the reason is worth stating. It is blank for 4,469 of 6,163 directory rows, and almost every one of those is a college with no athletics programme, so grading the column against the directory alone would have produced four thousand confident and entirely fabricated violations. The institutional characteristics file carries each institution's own answer about whether it competes, and that answer moves the denominator from 6,163 to 1,998. The finding did not need a better threshold. It needed to know who the rule applied to.

One candidate is still deliberately not graded. The veterans information page is blank for 2,377 institutions, and the same characteristics file would now supply an applicability rule for it. It stays ungraded because applicability was never the obstacle here: no universal requirement obliges an institution to publish a veterans page, so a rule about who it applied to would be a rule about a duty that does not exist. Knowing who would owe a disclosure is not the same as there being one to owe.

If this is wrong about your institution

The rules above are the whole of it; there is no model anywhere in the grading path and no judgement that is not written down on this page. If a field is marked wrongly, the disagreement will be with a stated bound, a stated peer group, or the underlying federal record, and all three can be checked.