27 January 2026 · Journal
Building a session quality score without vanity metrics
A score is a claim. Claims need owners, populations, and an expiry date. Formulae come last.
Most failed quality indices we see in Bangkok product orgs were born in a dashboard workshop. Someone asked for “one number like NPS.” A well-meaning analyst mixed crash-free rate, time-on-task, and a survey snippet. The number moved. Nobody could say why. Session Quality Analytics then became a political object instead of an evidence practice.
Write the charter first
Before any SQL, write one page: who is in the population, what unit you are scoring (session or task attempt), what would falsify the score, and who may retire it. If that page cannot be agreed, do not ship a number. Ship the disagreement.
Keep three questions separable
Did the system work? Did the person finish or abandon with intent? Were they treated with respect (consent, accessibility, no PII in replay)? You may later weight them, but each must plot alone. A composite that cannot be decomposed is vanity with extra steps.
When to refuse a composite
Refuse when capture is incomplete, when one component is a survey with a 4% response rate, or when leadership intends to bonus engineers on the blend. In those cases Cloudpacketbase teaches a “score deferred” label. It is not glamorous. It prevents a year of fake movement.
Version and expire
Every time the event contract changes, increment the score version or kill it. Silent drift is a quality failure of the analytics programme. We put expiry dates on charters the way we put them on legal policies: not because the world ends, but because someone must look again.
The flagship course spends two full weeks on this artefact. If you only steal one template from Cloudpacketbase, steal the charter, not the formula.