Episode 33: Data Quality in Healthcare: How PIQI Measures and Assesses Data

Are you tired of dealing with unreliable data that hinders your decision-making process? Join Charlie Harp, the host of the Informonster Podcast, as he explores issues impacting data quality and explains how Simple Assessment Modules (SAMs) function within the PIQI framework.

In the final episode of our Data Quality in Healthcare - PIQI Framework series, Charlie guides you through the intricacies of SAMs which are specialized tests designed to evaluate the quality of your data entities for specific use cases. Discover how these plug-and-play modules can seamlessly integrate into evaluation profiles, allowing you to assess your data against a battery of relevant criteria.

Charlie also explains the concept of evaluation profiles, which establishes a comprehensive framework for measuring and assessing your information entities. You'll learn how these profiles produce quality scores and provide detailed statistics, empowering you to make informed decisions based on reliable data. 

Contact Clinical Architecture

• Tweet us at @ClinicalArch
• Follow us on LinkedIn and Facebook
• Email us at informonster@clinicalarchitecture.com

Thanks for listening!

Transcript

Charlie Harp (00:10):

Hi, I am Charlie Harp and this is the Informonster Podcast. On today's Informonster podcast, we're going to wrap up the series on the PIQI framework, this being the final part of a four-part series. If you haven't listened to the first three parts, it would be better if you did just saying, but I will try to make it interesting regardless. Now, in the previous episodes in this series, I gave a PIQI overview. I talked about the PIQI model, which is simplified data-oriented lease comm denominator model to help us focus on the content and not the container and the PIQI taxonomy, which provides a categorical lens to help us identify the nature of quality issues. In this episode, we're going to focus on the concept of a simple assessment module or SAM and how they are organized within something I like to call an evaluation profile.

(01:01):

In other words, how does it all work? But first, let's talk about automated looms. That's right, ladies and gentlemen. Automated weaving machines.

(01:11):

In the mid-20th century Sakichi Toyoda, often referred to as the “King of Japanese Inventors”, founded Toyota Automatic Loom Works limited where he introduced the concept of production automation. After his death in 1930 his eldest son, Kiichiro, spun off fledgling automobile production department into what we now know as Toyota Motor Corporation. I imagine that at this point you are a little confused. You are wondering why they changed the name from toyoDA to toyoTA in 1937. I was wondering that too. It turns out that the word ‘toyoda’ uses ten Japanese strokes to write while ‘toyota’ uses only eight. In Japanese culture eight is considered a lucky number. Also, toyoDA in Japanese literally means “fertile rice patties”, so there’s that.

(02:04):

Actually, you're probably wondering why I'm talking about Toyota in an Informonster podcast. Well, I'm going to tell you why. Toyota is known for something called the Toyota Production System, and this socio-technical management system has been adapted and implemented by many companies over the last 50 years. While the Toyota production system or "Toyota Way is not perfect, it does have a few principles that are worth exploring in the world of healthcare data quality. The first many of us have read about or heard of is Kaizen. Kaizen is the principle of continuous improvement. Now, it's a broad concept that can apply to individuals, teams, and processes, but what I would like to focus on is process and the mindset of continually monitoring what you were doing, identifying issues and resolving not only the instance of the issue, but the circumstances that created the issue in order to evolve the process beyond those circumstances.

(03:01):

So that's the first principle that I think is interesting when it comes to how we should be thinking about healthcare data quality. The second principle was introduced by Sakachi Toyota himself, and that is the principle of Jidoka, which promotes automation where the automated process can recognize a problem is occurring and mitigate the problem before it affects the production process, whether that involves humans or it does not. Now, this brilliant insight into the risk of automation by one of the originators back in the mid 20th century reminds us that even then it was recognized that automation is a way to accelerate an activity and sometimes that activity can be creating problems. This is an important principle, this idea, especially today as we embrace tools like artificial intelligence, which is really just another form of automation.

(03:52):

The third principle is “Genchi Genbutsu”. This literally translates to "real location, real thing," and it suggests that you truly understand a situation, you need to see it for yourself.

(04:04):

I think of this as transparency, and it goes back to something I said in a previous podcast, healthcare data quality as the product of a lot of individual collections of data. It's not a big thing. It's like people, a team is not a thing. The people on the team are the thing that make the team work. It's the same with healthcare data quality. Healthcare data is not a thing until you get down to the level where you're actually looking at the data. So when you're dealing with data quality, you can't ignore the actual data and what's happening at an atomic level. So "Genchi Genbutsu" real thing in the real location. And the last principle doesn't have a Japanese phrase to go with it. So I'm going to call it "sonkei", which is respect in Japanese. This is the 11th principle in the Toyota production system, which states the following, "respect your extended network of partners and suppliers by challenging them and helping them improve."

(05:01):

When you are operating in an ecosystem where your success is dependent upon the success of others, the wise thing to do is help them be successful. This is one of my favorite principles when I did the research on this, and it's a big reason why I began working on the PIQI framework almost a year ago. We cannot fix data quality on the receiving end, nor can we fix it in transit. It must be fixed at the source. So if we really want to improve data quality, we have to find a way to identify what's broken, stop the machine, examine the root cause, and communicate it back to our partner and evolve our ecosystem. Which are the principles of Jidoka, genchi genbutsu, and sonkei for kaizen. And if we start thinking about patient information as a product and the information we receive and aggregate from our own systems and other systems as components, that will become a vital part of that product.

(06:05):

My reason for talking about Toyota and for creating the PIQI framework become pretty clear. Does that work?

(06:12):

So in previous episodes, like I said, we talked about the model, the taxonomy, understanding the nature of what we're finding. This episode is about the mechanisms that we will use to measure and ultimately assess the patient information itself.

(06:26):

So if you go back to a year ago when I was designing the HDQT, the healthcare data quality taxonomy, I was thinking about what is a meaningful way to evaluate the data itself and not just categorize it. The problem I had was how do I get it into someplace where I can align it to this category? And that led me down a rabbit hole of research on assessment methodologies for structured data and approaches. And some were really convoluted, some were schema specific. And just to save you a bunch of time so you don't have to do this yourself, I built the PIQI framework, or at least the design for the PIQI framework.

(07:04):

Now at the core of this framework, when you think about measurement and assessment at the core of this framework is a very basic concept of counting every test we perform, the denominator, and counting each time that test is passed, the numerator, and using the numerator over denominator as the basis for a percentage score. Yes, it's true. This amazing- innovated invention that I have created will revolution. Wait a minute, this is how we've been doing quality measures forever. That's right at the core of the PIQI framework is the same tried and true mechanism that we've been using for decades. And for me, when I arrived at that place, it was reassuring that I wasn't creating something that was crazy or different and that we could all understand what it represents when we're looking at something. Now there are some nuances and what I found is I delved into patient information data quality is that not everything is straightforward.

(08:05):

There are things that are straightforward, but not everything is straightforward. And so there are some nuances to how we apply this very basic principle of pass fail against something like patient information.

(08:16):

So let's start by this mechanism that does simple pass-fail and this mechanism in PIQI is called a simple assessment module or SAM. Now you can think of a SAM as a test. This takes a standard input, which can be either an attribute, which is just a string, a codeable concept, a codeable value, a range value, a specific data class like a lab element or the entire patient message, and it runs its logic against that input and it returns a pass or fail. Now there are a fixed number of inputs that you can give. So a PIQI falls into a category of "I'm a lab element", "SAM" or "I am an attribute SAM", but the interface is the same.

(09:00):

You have an input, you have parameters, and you have the output, which is a pass or fail.

(09:07):

What's important about this approach is a SAM has a signature and as long as I adhere to that signature, the input and the output signature, I can create new SAMs that are plug and play in the assessment process. If I want to create a SAM that evaluates an attribute to see if it's a date, I create a SAM called attribute is date format, I pass in an attribute, I pass in a parameter if it's necessary and I get back a pass or fail. That's it. And so I can create new SAM modules and plug them into my architecture, and as long as they fit the signature, they can do that. Now this creates an opportunity for me to share SAMs so that not everybody has to reinvent the wheel.

(09:50):

I can also create SAMs that don't care what the entity is, like a SAM called the "attribute is populated", which just assesses if the thing I passed into it is not empty or null and return a simple pass if it is populated or a fail. If it isn't, SAMs can also have dependencies on other SAMs. For example, for a test code to be compatible, it needs to be in the loin code system. For that to happen, it needs to be a valid concept to begin with. For that to happen, the codeable concept data itself has to be complete. You need a code system, a code and a display, and for that to happen the test attribute has to be populated. You get the drift. It's like the song, "There's a hole in the bucket", but in JSON. The nice thing about the SAM dependency chain is that if you assign a SAM at the top of the chain, so I assign concept is compatible with lo, it automatically gets its predecessors as part of the deal.

(10:48):

So it won't just run against the entity that I assign it to and I'll get to that in a minute, but it's going to rely on all of its predecessors passing because for all intents and purposes, I can't even run that SAM unless it's passed all of its predecessors. So they have to be built into the SAM as dependencies.

(11:08):

Now, I can also build SAMs that are more complicated, so "is populated" is pretty basic, but for example, I can build a SAM that accepts a lab result element, which gives it access to all the parts of the lab result and checks to see if the lab result value is plausible given the test code and the unit of measure.

(11:28):

When I create a SAM that's required that I assign a taxonomy dimension from the PIQI taxonomy. This means that when the SAM either fails or passes, it's going to populate the statistics for that dimension.

(11:45):

So I don't need to do that independently. For example, if I have a SAM that says "is populated", I'm going to want to sign it the dimension of available or availability rather. What that means is I don't have to worry about what the dimension is because whoever created the SAM was already assigned that.

(12:03):

Now using this approach, one could build out a comprehensive collection of SAMs that can be used across a growing list of elements and entities in their patient information. So essentially SAM's are tools in my toolbox that can be used to measure and assess information entities that are moving through my pipeline. But how do I decide which tools to use and against which entities and for what use case? Well evaluation profiles. That's how an evaluation profile is a configuration that establishes a battery of SAM's to evaluate relevant data entities and determine if the data is fit for use.

(12:43):

An example of an evaluation profile, use cases USCDI v3, which provides a standard list of criteria that are recommended for nationwide interoperable health information exchange. So I have my SAMs in the toolbox. I have a collection of patient entities and I need to assess those entities based upon use case rules like USCDI v3, I establish an evaluation profile for USCDI v3 and define the SAMss that I want to use to evaluate each entity in any given patient message. These entity or field level definitions, we'll call evaluation criteria. Each criteria is essentially the SAM that will provide the score for a given entity in the patient message. How about an example? Let's say that in my USCDI v3 evaluation profile, I want to verify that the test concept in a lab result record is interoperable. In other words, is it a LOINC code or does it have a LOINC code associated with it in the data?

(13:48):

Now to do this in the evaluation criteria definition, I select the test concept attribute as my entity out of the lab result element. The lab result test concept attribute is a codeable concept in the model and I assign the 'codeable concept is compatible' SAM and populate the code system list parameter with the acceptable code system identifiers for LOINC. I'll also say that this criteria is being scored as opposed to just being informational because in an evaluation criteria you can measure things that don't go against the quality score. You're just curious how often have they done this. So that's another feature of the PIQI framework is you don't have to score. You can also just kind of assess and look at statistical data on something you're curious about. It's worth noting that beyond scoring the data at an element level or for the entire message, there's also a concept of gating, which means you can decide if the failure of a given SAM at an element or a patient message level is bad enough to flag the element or message for quarantine to keep it from junking up your clinical repository.

(15:05):

For example, I can have a bunch of messages and the aggregate score is 20%, but I can also look at that and say that the lab results are so bad that none of the messages should be allowed to proceed and go into the data warehouse. The data quality issues are so bad that I'm not going to let them through. There might be cases where you won't worry so much about that depending on what you're trying to do with the data and you might say that certain failures I'm going to let slide through and other fails are unacceptable. The gating allows you to do that, but we've kept that independent from the score in the PIQI model. Whether something scores is one thing, whether it gates is something totally different. In our example, we've gone through and we've flagged that if we take our lab result test record scenario where we're saying the lab result test record needs to be compatible with link, the following things are going to occur relative to the test concept attribute first.

(16:05):

The fact that the attribute has a scoring, SAMs means that I increment the denominator for the entity. So I've identified this as a scoring scenario, not just a statistical scenario. Next I look at the SAM and I look at its definition and I see that it has a list of predecessors, it must pass. So I go to the most basic element on that list, which in this case is the attribute is populated. I take the test code, concept attribute as an attribute, which is a string, and I say, is it populated? If that passes, I move on to the next dependent SAM, which says, is the code complete? So is it a codeable concept? Essentially if that passes, I go to the next one which says, is it a valid code? So given the code system, the code and the display, is it a valid code in the code system that it gave you?

(16:54):

If that passes, it says, is that valid code, a LOINC code and that's the final gate. If all of those pass, I get my point, I get my numerator point, so I'm one for one. If any of them fail, so let's say the code is not populated. If the code is not populated fails, then the entity does not get a point. It is a failure and the fact that the failure that occurred on the test concept attribute is not populated, goes into the statistics. So I know that I had one instance where the test code concept attribute was not populated. By the way, that has a dimension of unpopulated, which is part of the availability category in the PIQI taxonomy. So when a failure occurs, the statistics happen. If no failure occurs, the entity passes and gets its point. This is important because it's easy to think that every SAM that runs against an entity counts as a point, but it doesn't.

(17:55):

Only the assigned SAM against the entity determines whether it gets a point and I either get to that last SAM and pass it or I don't. Any failure in the chain counts as no point, which is a zero over one regardless of how many SAMs I run against an entity. I hope that makes sense and it's as simple as that. So I run through my SAMs, the entity gets appointed or doesn't get a point. If I have a message where I am performing a thousand assessments on that message from beginning to end and of all those assessments I pass 80% of the ones that are scoring because I only care about the ones that are scoring for the score purposes, then I'd get an 80%. So I'd have 800 out of a thousand that passed 80%. Make sense? Alright, the other thing that's important to talk about here is that there are SAMs that are only run if the entity I pass to it satisfies a certain condition.

(18:59):

For example, in USCDI v3, there is a standard that says a result value should be a SNOMED code. So if you say negative as your result, it should be coded to SNOMED, but that test result value is a SNOMED code for interoperability does not make sense if the result is numeric. If it's a hemoglobin A1C and it's 7%, then that's not a SNOMED code. That's a numeric result. All I care about at that point is does the result type match the value? A SAM checking SNOMED would be ignored in this scenario if the value isn't numeric. So you have a condition that says this, SAMs only counts if the results a code. Just like if you're looking at the result unit, you want the result unit to be in the UCUM nomenclature, but only if it's populated. If it's an arbitrary result or it's a result like negative, you're not going to have a result unit, but you don't want that to fail because you don't need a result unit.

(20:02):

Does that make sense? So this notion of conditionality is also conditionality is that even a word, is something that is important when you're doing this assessment so you don't get false quality issues when they're not supposed to be there in the first place. Alright, so when you look at the net results, an evaluation profile produces a quality score on a message and detailed statistics about the nature of the issues that it found relative to the profile. Different profiles are going to care about different things or maybe one will be a subset of something else. And each message is a data point where in the aggregate you can assess using average or median to determine the aggregate quality or the average or median quality of the data coming from a given source. And you can also look for patterns in the failure information, which entities fail, what SAM did they fail, what category was it in to help you identify the root cause of those issues so you can communicate them back to the source of the data and say, you're missing the code system and that's why you're failing.

(21:17):

So if you can fix that, we can get good data and everybody's happy. It's also worth noting that a given implementation of the PIQI framework can have multiple evaluation profiles at a given point in time. So this allows you to evaluate the same messages, ideally not running their overlapping SAMS more than once, but you can evaluate the same messages against different use cases. So for example, you can have a profile that evaluates the current state like USCDI v3 for today, and also implement an evaluation profile that evaluates against a future state like USCDI v4 for the future to help measure your progress. How kaizen is that? Alright, Hey, I know that this is not super easy to follow in a podcast, so I'm going to spare your mind's eye and stop it right here. We built an initial version of the PIQI framework and we are putting it to the test.

(22:16):

As I speak, I'm biased, but the results we're seeing are I think are very exciting and very illuminating and I'm hoping to put together a webinar where we can actually go through it and show it to people in action. So stay tuned for that. In the meantime, if you want to learn more about the PIQI framework, please don't hesitate to reach out. I hope you've enjoyed this series about the PIQI framework and I hope I get to work with you to see it become a reality. With that, I am Charlie Harp and this has been the Informonster podcast. Thank you so much for listening and I'll see you out there.