As an executive or clinician in a hospital, there are metrics that you monitor to understand how well you are doing operationally and clinically. Your performance may even be partially determined based on some of them. At a surface level, most metrics seem obvious and there may be an implicit assumption that they are obviously correct. If everyone else in the hospital uses them, why wouldn’t they be correct? But do you really know the definitions of the metrics that matter most to you? Do you know where the data is coming from? Do you know if there are outliers? Do you know who enters the data?
My takeaway? Nobody knows the data as well as they think they do. At best, hospital users ignore these metrics because they don’t understand them, but more likely, users are making important decisions based on misguided understanding or inaccurate metrics. Because of this, the most important metric is one that no hospital actually tracks: a “meta” metric for metric quality that underlies EVERY hospital’s performance tracking initiatives.
When I recently met with a hospital to discuss a balanced scorecard, the Director of the ED, the VP of Operations, and the IT analyst each gave me a different definition of “Emergency Department (ED) Boarding Time.” I’m willing to bet that if I asked 10 other people, I would get 10 more distinct answers. Let’s use this example to explore how well you know your hospital’s metrics. How does your hospital define boarding time? Are you confident it is tracked and calculated correctly? Let’s start with a basic definition – “the amount of time between when an admitted patient is ready to leave the ED and when they actually leave the ED.” Now let’s break that down into its components:
- […an admitted patient] There may be good reasons to exclude certain patient populations from the boarding metric. What about psych patients, pediatrics, patients that go to the clinical decision unit, etc.?
- […ready to leave the ED] The timestamp we use to determine when a patient is ready to leave the ED can have a significant impact on total boarding time. Does boarding begin when the patient is clinically ready to leave? What about when the bed assignment is completed? There are likely at least four different timestamps that could be used. Additionally there may be different timestamps that should be used based on some of the unique patient populations listed above.
- […when they actually leave the ED] This part can be the most complicated. Similar to the last bullet point, there are multiple timestamps that may be used. But in addition to this, no timestamp may be captured at ALL! There are some patients who just disappear. They obviously don’t physically disappear; there is a perfectly good explanation for where they went, but for some reason the timestamp is not captured when they leave the ED. This is so common that we have a special term for this population – “ghost patients.” What do we do with patients with bogus or incomplete data?
There’s now hopefully a good shadow of doubt in your mind that the metrics you track are not what they seem. Managing good metric quality requires three areas of focus:
- Governance – There is a lot that goes into good metric governance, but at a basic level, your hospital should ensure there is one source of truth for each metric, metric definitions are well defined, there is an owner or subject matter expert for each metric, and there is a periodic review of which metrics should be calculated and disseminated.
- Transparency – At a minimum, the owner of each metric should be able to drill into the metric to get the underlying data used for the calculation.
- Continuous auditing – Unfortunately, data is not well behaved. Changes such as a system upgrade, a decision to change a business workflow, or a change to a report script can all throw a serious wrench into the accuracy of a metric. Metric errors WILL occur, but they should be caught and addressed quickly. Automated testing of data and metrics is the best way to protect against these occasional issues
Good governance can only be solved by good business practices in the hospital. Transparency and continuous auditing should be important evaluation criteria when considering any IT system or method of reporting. If your current reports don’t support these items, you should take those metrics with a large grain of salt.
With metric quality in mind, I challenge you, the reader, to select a metric that is important to you and that you view every day or week. Come up with 3-5 questions you can ask to tease out whether this metric has good quality and try to find someone that can provide the answers. You may be surprised at what you find.