Benchmarking for training organisations
See where you sit against comparable providers.
Benchmarking on the measures that actually get examined, against providers of similar size, scope and delivery mode, with the method published so you can argue with it.
Comparable, not merely available
A benchmark drawn against every provider in the country tells you almost nothing. A provider delivering three qualifications to two hundred students in one location has little in common with one delivering sixty across four states, and averaging them produces a number that describes neither.
Comparison groups are therefore built on size, scope breadth, delivery mode and cohort profile. Where a group would be too small to be meaningful or to protect the identity of the providers in it, no comparison is shown rather than a weak one.
The method is published
Every measure states how it is calculated, what population it is drawn from, and the period it covers.
That is unusual in benchmarking products and it is the point. A number you cannot interrogate is a number you cannot act on, and the first question a sensible manager asks about an unflattering figure is how it was arrived at. If the answer is proprietary, the figure is worthless to them.
What benchmarks can and cannot tell you
A benchmark tells you that you differ from comparable providers. It does not tell you that you are wrong.
A completion rate below the group may reflect a harder cohort you chose deliberately to serve, and treating that as a failure would push you towards enrolling easier students, which is the opposite of useful. Equally, a figure well above the group is not automatically good news and is occasionally the first sign of assessment that is not sufficiently rigorous.
The value is in the question a difference prompts, not in the difference itself.
Measures that can be gamed, marked as such
Any measure attached to consequences will eventually be managed rather than met. That is not cynicism, it is a well-observed property of performance indicators.
Where a measure is easy to move without improving anything underneath, it is marked. Completion rates respond to enrolment selection. Satisfaction responds to when and how you ask. Time to completion responds to how generously you record it. Knowing which numbers are soft is part of reading them honestly.
Nobody is named
Comparisons are against aggregated groups. No individual provider is identified, and group sizes are set so that no provider can be inferred from the aggregate.
Where a provider supplies their own data for comparison, it is used for their comparison and not added to anybody else's reference group without explicit agreement.
The measures that are actually examined
- Completion and withdrawal
- Split by cohort and by delivery mode, because an aggregate hides the pattern that matters
- Time to completion
- Against the volume of learning the qualification anticipates, which is where compressed delivery becomes visible
- Assessment outcomes by assessor
- Wide variation between assessors on comparable cohorts is a validation question before it is anything else
- Trainer load
- Students per trainer, which constrains how much supervision of judgement is realistically possible
- Scope utilisation
- Qualifications on scope that have never been delivered, which invite an obvious question
Using a benchmark without being managed by it
The useful sequence is to look at a difference, form a hypothesis about why it exists, and then go and check the hypothesis in your own records. The benchmark points; your files answer.
The unhelpful sequence is to set the benchmark as a target and manage towards the number. That reliably improves the number and frequently worsens the thing the number was standing in for.
Where the data comes from
Two sources, kept distinct because they carry different weight.
Published national collections, which are consistent, comparable and usually a year or more behind. And data a provider supplies about itself, which is current and unverified, and is labelled as self-reported wherever it appears.
Mixing the two silently would produce a tidier product and a less honest one, so a comparison drawing on both says which parts come from where.
General information. Benchmarks are context for your own judgement, not a substitute for it, and they do not replace regulatory requirements.