Transparent by design
How we grade
Every finding is deterministic, cites a spec clause, and can be explained in one sentence. There is no model anywhere in the grading path.
Weighted score
Dimensions
An endpoint no vantage could reach is not graded at all. It is published as not observed, with the reason and the vantages that tried, because nothing else could be observed and grading a document nobody retrieved would be an accusation this project cannot support. F means the opposite: the endpoint answered, and what it declares falls short across the checks below.
Findings
Reachability
Does /metadata answer with HTTP 2xx over HTTPS?
An endpoint that cannot be reached is published with the reason stated, rather than dropping out of the dataset, and it is not graded. Causes are distinguished: DNS non-resolution, TLS failure, timeout, and refusal are different facts, and only some of them are about the endpoint. Reaching an endpoint from any vantage settles that it is up; failing from every vantage we have is reported as not reached from those vantages, which is a weaker statement than down.
Not observed
What happens to the checks that could not run?
When no vantage retrieved a document, the checks that read it do not run, score nothing, and publish nothing about the endpoint. The dimension shows no number, because zero is a measurement and this is the absence of one. An unreachable endpoint used to publish four findings describing what the payer had not declared, each with a spec citation, from a run that had received no document at all; the project's own history file disproved every one of them for the endpoint it happened to.
Response time
How long did /metadata take?
The median across the vantages that answered, which today share one network, so bands are deliberately coarse: full credit under 3s, partial under 8s. The raw milliseconds and the vantages are always shown. A network path difference must never flip a grade.
FHIR version
Does the server declare the release it intends to serve?
Checked against the endpoint's registered intent, not against R4 unconditionally. An R5 server declaring 5.0.0 is correct. R4 is the default because the CMS interoperability rules require it of payer APIs.
Software identity
Are software name and version declared?
Knowing what is running is part of what a CapabilityStatement is for.
Declared breadth
How many resource types are declared?
Five or more earns full credit, and so does narrow-but-complete: two to four resource types with every one documenting its interactions. CMS Blue Button 2.0 is deliberately scoped to three, which is a design decision, not a deficiency.
Interaction coverage
Do declared resources document their interactions?
A resource listed with no interactions tells a client nothing it can act on.
Interoperability profiles
Are US Core, CARIN, or Da Vinci canonical URLs declared in any conformance element?
Declared profiles are how a client knows which implementation guide the server follows. Five elements are read before anything is concluded: rest.resource.supportedProfile, rest.resource.profile, instantiates, imports, and meta.profile. The finding names the element the declaration was found in, or names all five when none carries one, because "no recognized interoperability profiles declared" used to be asserted after reading exactly one of them.
Named in prose only
Does the document name a guide it does not declare?
Worth zero points in either direction, and shown only when I1 found no declaration. Prose is not a conformance claim: a title reading "CARIN PatientAccess Implementation" tells a client nothing it can act on, which is exactly what supportedProfile is for. But a flat denial next to a document that says CARIN three times invites a reader to conclude something the document contradicts, so the note says what is actually the case and which element would fix it.
SMART discovery
Is .well-known/smart-configuration present and complete?
Not applicable to Provider Directory APIs, which are required to be reachable without authentication and are not scored on an authorization surface they must not have.
Declared security
Does the CapabilityStatement declare an OAuth/SMART service?
Not applicable to Provider Directory APIs, for the same reason as I2.
Where the measurement comes from
Every published grade reconciles probes from more than one vantage, on the rule that reaching an endpoint from anywhere settles that it is up, while failing from one place settles nothing. That rule exists because a live payer endpoint was once recorded as dead when a middlebox on the probing network intercepted TLS.
What the vantages are, precisely: three GitHub-hosted runner images, Ubuntu, macOS and Windows. They are three hosts on one provider's network. They are not three independent networks, and nothing here calls them that. Three hosts catch a fault local to one host or one trust store; they cannot catch a source-address rule, bot filter, geo rule, or rate limit applied to that provider's address space, because such a rule reaches all three at once. So a run where every vantage failed publishes that the endpoint was not reached from that network on that day, and says why it cannot separate that from an endpoint being down. A genuinely independent vantage is an open item, and until one exists this page will keep saying one network.
Each vantage counts once. The publishing run makes no probe of its own; it grades the documents the probing runs retrieved, which is also why a scheduled day costs an endpoint at most six requests.
What a grade is not
It is not an audit, a compliance determination, or a statement about care quality. It describes what a public document declared on a given day, from a handful of hosts on one network. Grades are comparable within a category only.
Corrections
This project has made and published several measurement errors, including grading narrow APIs as deficient, penalizing a public-by-design API for having no authorization surface, and recording a live endpoint as dead because of TLS interception on the probing network. Each is documented in the write-up. If something here is wrong, please open an issue.