Most program review templates ask for a score. A weighting is agreed, each factor is rated, and the factors are summed into a number that sorts programs into a list. The appeal is obvious: it is auditable, it is consistent, and it produces a ranking.
It also produces answers like this one. An early version of our own model used a weighted sum across five factors. It scored a program that had lost 15% of its completions at 69 out of 100 — “Favorable.” The arithmetic was correct. The model had treated shrinking supply as reduced competition, which is a reasonable thing to believe about a market and a catastrophic thing to believe about an academic program.
That failure is not peculiar to our model. It is structural. A weighted sum assumes the factors are substitutable — that enough strength on one can compensate for weakness on another. For program review that assumption is false. A program with a collapsing completion count and strong occupational demand is not “averagely healthy.” It is a specific situation with a specific set of follow-up questions, and averaging it destroys the information that made it interesting.
What replaced the score
What follows is the structure we use now. It is not a scoring rubric and it deliberately does not produce a single number. It is five questions, each answered from a named public source, each shown with its underlying figure so a reader can disagree with the reading rather than having to trust it.
The order matters. The first two are usually enough to separate the obvious cases, and the last three exist to stop the first two being over-read.
Question 1
What is the completions trend, over two horizons?
IPEDS award counts, two-year and eight-year. Two horizons because they disagree often and the disagreement is informative: a program down 8% over two years but up 6% over eight is having a different conversation from one down on both.
Question 2
Is the occupation growing, and is the degree the route into it?
BLS Employment Projections for the occupations the NCES CIP–SOC crosswalk assigns to the program, weighted by employment. The second half of the question is the one usually skipped. A growing occupation only supports a program if employers hire from that credential.
Question 3
Is the base large enough for the percentage to mean anything?
A 200% increase from a 490-award base is not evidence of durable demand. A 3.6% decline on 256,911 awards is not evidence of a closure risk. Percentages without a base are the most common way a program review misleads itself.
Question 4
Is the decline in the number of programs, or in the size of them?
IPEDS records the institution count alongside the award count. If institutions are leaving, the field is contracting at the point of provision. If the institution count is flat and awards are falling, the same programs are producing fewer graduates — a different problem with different remedies.
Question 5
What does the data not know?
Mission fit, quality, cost per graduate, replacement demand, and whether graduates work in the mapped occupation at all. These are not footnotes. They are the questions that decide the case, and no federal dataset answers them.
Why the questions have to be asked together
The value is not in any single question. It is in the combinations, and in the fact that the same completions number means different things depending on the answers to the others. Four cases from our own label set make this concrete.
| Program | Awards 2yr | Awards 8yr | Occupation | Awards 2023 | Reading |
|---|---|---|---|---|---|
| Dental Assisting | −9.7% | −30.6% | +11.4% | 15,557 | Healthy — licensure-gated route into a growing occupation |
| Pharmacy | −22.2% | −22.9% | +17.4% | 12,167 | Conditional — strong demand does not override here |
| Accounting | −19.4% | −29.6% | +4.7% | 57,564 | Healthy — large, and the degree is the required route |
| Physics | −7.1% | +2.0% | +0.7% | 11,238 | Declining — weak demand and a negative recent trend |
Award change: IPEDS completions, all award levels. Occupation growth: BLS Employment Projections 2025–35, employment-weighted across the NCES CIP–SOC crosswalk. Readings are expert judgment recorded before the model was tuned to them.
Dental Assisting lost 30.6% of its awards over eight years and is still healthy, because the occupation is growing 11.4% and the credential is the licensed route in. Pharmacy lost 22.9% over eight years against stronger occupational growth of 17.4% and is only conditional. The two cases are not consistent with any single rule about demand overriding trend, and a weighted sum would have to give them the same answer.
That inconsistency is the point. It is not noise to be averaged away. It is the judgment that makes the analysis worth paying for, and it is why the output is a reading with its inputs shown rather than a number.
How the readings were validated
A model tuned against its author's intuition is a liability. Ours is tuned against a domain expert's judgment: 60 programs labelled healthy, conditional, or declining by a sitting provost, with a written rationale for each, recorded before the model was fitted to them.
The labels are the ground truth. Where the model disagrees, the model is wrong and the rule changes. That produced four rules we would not have written from intuition alone:
- Occupational demand can override a declining trend — but only sometimes, and the boundary is a judgment call rather than a threshold.
- Growth from a small base must be discounted. A 288% eight-year increase on 4,255 awards is a new field, not a proven one.
- Large programs need a size-scaled materiality floor. A 3.6% decline on 256,911 awards is a reallocation question, not a closure risk.
- Trend alone cannot separate cases. Accounting and International Studies have similar completion declines and opposite verdicts.
On 18 programs the rule had never seen, it agreed with the expert's judgment 12 times — 67%. On 11 programs scored blind, 7 times — 64%. Both figures are modest, and they are reported on every report we produce. A rule that agrees with an expert two times in three is useful for sorting a long list and useless for deciding a single case.
Where it is still wrong
Being explicit about this is not modesty; it is the only way the tool is usable. The model currently disagrees with the expert on several programs in the label set, including cases we would want to get right:
- Accounting — expert says healthy, model says conditional. The model's size floor is not catching a 57,564-award program with a strong occupational anchor.
- Pharmacy — expert says conditional, model says healthy. The model lets a 17.4% demand figure override a 22.9% eight-year decline, which the expert explicitly rejected.
- Physics — expert says declining, model says healthy. A positive eight-year figure is masking weak demand and a negative recent trend.
- Sustainability Studies and Barbering — expert says conditional, model says healthy. Both are small-base growth stories the model reads as strength.
These are published here because a reader who knows the field will find them anyway, and because a tool that hides its error rate is asking to be trusted rather than checked.
What this means for a review committee
If you are running a program review, the practical implication is not to abandon scoring. It is to stop treating the score as the finding.
- Show the inputs next to the verdict. A provost who cannot see the completion figure behind a rating cannot challenge it, and an unchallengeable rating will be ignored or, worse, obeyed.
- Do not let factors cancel. If one factor is damning, report it as damning and report the others separately. A sum that hides a collapse behind three strengths is worse than no sum.
- State the base with every percentage. It is the single cheapest protection against a misleading trend.
- Separate what the data knows from what it does not. Mission fit and quality are not in IPEDS, and a review that implies otherwise has quietly changed the question.
- Treat the output as decision support. A provost who closes a program on a number we generated is a real risk, and the report should be built to make that harder rather than easier.
None of this is an argument against quantitative review. It is an argument for keeping the judgment visible, because the judgment is the part that cannot be bought off the shelf — and the part that a weighted sum silently replaces.
Check your own program
We produce a written viability assessment for a single program: completions against occupational demand, the institution count and per-institution yield, state-level projections, graduate earnings where they exist, and every figure traced to its source and vintage. Every component shows its underlying number, and the report states its measured agreement with expert judgment. The first one is free.
Request an assessment for your program →
About this analysis
Award counts and institution counts are all award levels, 2015–2023, from IPEDS completions via the Urban Institute Education Data Portal (ODC-By v1.0). Occupation growth is employment-weighted projected change 2025–35 from BLS Employment Projections, using the NCES CIP–SOC crosswalk. The occupational anchor is the share of mapped employment in occupations that typically require a bachelor's degree or above. Expert labels were recorded by Dr. Brian Thomas across three rounds before the scoring rule was fitted to them; the rule was frozen and checksummed before the holdout was scored. Holdout agreement is 12 of 18 (67%); blind agreement is 7 of 11 (64%). Both are reported on every assessment. IPEDS 2020 is excluded because it is a near-exact duplicate of 2019 in the published data; 2023 is provisional. Prepared by True Heading Collective LLC. Every figure is reproducible from the public sources named.