September 26, 2019
From Stephanie Sundborg, PhD, Research and Evaluation Coordinator, Trauma Informed Oregon
“How might we measure that?” Believe me, I ask this question so often that I’m sounding like a broken record. But in fairness, this question is really at the heart of our attempt to find trauma informed care (TIC) measures. As we talk about training in this newsletter, it’s important to think about the questions we should ask. Here are a few that come to mind: How much training is enough? What does competence look like? How might we measure it? Does everyone need the same type and amount of TIC knowledge?
Competence is generally thought of as the integration of knowledge, skills, and attitudes. For individuals and organizations wishing to adopt a trauma informed approach, formal training is often recommended as the first step to acquiring knowledge and competence.
Training builds awareness about the impact of trauma and trauma informed care, and is an effective strategy in building influential change agents. That said, competence and proficiency are different from hours attending training. Furthermore, there may be different knowledge needs within the organization and overtime.
As the TIC training developmental plan lays out, the first goal may be to increase awareness of the impact of trauma, in which case a measure of attitudes and beliefs may be the most appropriate. If the goal is to have knowledgeable staff, it may be necessary to test content knowledge.
Measuring knowledge is not without challenges, however. Knowledge can be understood in terms of what is known (type) and how well the information is understood (quality). Both of these factors influence how knowledge is tested. TIC is a very broad topic consisting of precise and technical information as well as imprecise conceptual information. While some of the technical information would work on a test of knowledge, much of the conceptual and abstract information would be difficult to test. A quick overview of each of these considerations may shed light on the challenges with knowledge measurement.
One of the first considerations is whether the knowledge needs to be factual or whether it can be gist memory. Gist memory reflects the essence or essential meaning of the information rather than the declarative facts. As an example, is it necessary for people to know the names of the brain structures impacted by trauma or will it be sufficient that they know brain structures are impacted by trauma? As we develop and gain experience we increasingly rely on gist memory in order to make decisions and form judgment. This type of processing and memory may be more closely representative of what can be retained following TIC trainings. While gist differs in terms of detail, it incorporates both intuition and attitude. Adults increasingly rely on gist memory especially in cases of information overload. Additionally, as information becomes more distant in terms of time and space, verbatim memory deteriorates more quickly while gist memory is more likely to remain.
Another limitation pertains to content. Knowledge may be described as conceptual consisting of principles and definitions or procedural consisting of action sequences. Although there is some guidance, as stated above, to the topic areas that should be included, the abundance of information from which to chose testable content is overwhelming. The field has not specified which information is essential to a trauma informed approach; therefore, selecting questions that are both meaningful for assessment yet manageable for participants in number is extremely difficult. A suitable alternative is to ask about knowledge more generally. As noted, gist memory is reliable and long lasting, and may be sufficient in order to show TIC commitment.
As if knowledge quality and type aren’t difficult enough, we also have to think about challenges related to measurement. As we grapple with this issue, it can be helpful to draw from what is known by people who study learning and knowledge translation. According to experts, there are stages of learning, beginning with knowledge and comprehension (or the simple recall of information) and progressing to application, analysis, synthesis, and evaluation. Knowledge and comprehension are the least developed stages as they reflect rote awareness of the facts. It isn’t until application and analysis that people use, process, and apply the information. In many situations, however, application and analysis can be too difficult and time consuming to measure. Tests of knowledge and comprehension are much easier to administer but they can be unreliable and flawed. For example, as we all remember from our days in school, T/F questions can stump those who know the information due to the tricky wording embedded in the question. Correctly guessing is also possible, especially when the alternative choices are too easy. Discrimination refers to the ability to differentiate between those with the knowledge and those without. Difficulty reflects the likelihood that those with the knowledge will get the answer correct. Both of these are important to consider when testing knowledge. Self-reports of knowledge are also a common method of assessment given the difficulty administering factual tests. These, too, are not without challenges, including an under or overestimation of knowledge. Further, self-reported knowledge is often found to be unrelated to tested factual knowledge.
Hopefully, this gives you some things to think about as you consider ways to better understand TIC competence and knowledge. Please see Phase 2 and Phase 3 of our Road Map to Trauma Informed Care for more information.
As we continue to figure out what TIC competence looks like, we’d love to hear from you. What knowledge is needed to apply trauma informed care in practice? You may think a topic area is important, e.g., the neurobiology that explains the impact of trauma; or you may have a specific factoid that is essential, e.g., the principles of trauma informed care. Please fill out this one-question survey: What knowledge is needed in order to apply trauma informed care in practice?
While you are at it, we are also starting to think about what trauma informed leadership looks like. We’d love to hear your thoughts about this too. Please fill out this one-question survey: What do trauma informed leaders look like? What do they do or say?
Thank you for all that you do to forward TIC and measure progress!!