12 March 2026 · Journal
Why p95 session duration is a misleading quality proxy
Duration is easy to compute and almost never means what leadership thinks it means. This note walks through three checkout studies from Bangkok teams we taught last year.
A session that lasts a long time can be a person comparing installment plans with care. It can also be a tab left open during lunch, a spinner that never resolved, or a customer who could not find the OTP field. Collapsing those into a p95 “experience” number is how Session Quality Analytics loses trust.
Study A — bill pay, shorter and worse
A utilities partner shortened median session time after a redesign. Task success on “pay this bill without calling support” fell. The missing seconds were the confirmation screen copy in Thai that used to explain a delayed posting window. People finished faster because they gave up earlier and called the call centre, which was not in the session trace.
Study B — grocery, longer and better
A grocery checkout added a substitution picker. p95 duration rose. Completion rose with it. The quality conversation almost killed the feature because duration had been wired into a “health” ring. Once we split completion from duration, the ring was retired.
Study C — background tabs
A wallet webview counted visibility-hidden time as session time. p95 looked catastrophic on Android. Reliability was fine. The fix was a visibility contract, not a performance project. Instrumented Session Quality now treats this as a week-one exercise.
What to publish instead
Publish task success, explicit abandon, fatal errors, and — if you must — duration as a diagnostic, never as the score. If leadership wants one number, give them a versioned composite with an expiry date and a paragraph that says duration is not in it.