In 2020, Flake and Fried published an article entitled, "Measurement Schmeasurement," where they highlighted the threat posed by questionable measurement practices (QMPs) to psychological science. They argue that the ubiquitous use of QMPs raises doubts about the validity of our measurement of psychological phenomena which in turn challenges the validity of empirical knowledge in psychology.
Turning to our own subfield of traumatic stress research is pertinent. We can ask whether the tools we use allow us to measure the phenomena that we purport to study. In other words, are we defining and measuring trauma exposures, PTSD and other post-trauma disorders consistently and accurately? If we do this well across studies, then the body of knowledge in our field can be considered robust, and we can confidently compare the findings of studies conducted with a wide variety of groups in different settings, locations and even languages. The concern is that if we conclude that our measurement practices are problematic, it raises fundamental questions about whether we really know what we think we know about assessment and treatment of trauma-exposed individuals.
Classical measurement theory posits that an attribute needs to have some quantitative structure to be measurable (Trendler et al., 2009). In psychological science, it is well-recognized that psychological phenomena are hard to capture, and that not all such experiences are necessarily quantifiable. There is not a precise or objective way to measure psychological states, and within our field, even our “gold standard” approaches are imperfect tools.
Perhaps most obviously, we should note that our fundamental definitions regarding the PTSD construct, while sharing core features, are not invariant. Both the DSM and ICD criteria have evolved over time and are not consistent with each other. Thus, any measure that we use to assess PTSD will have variable construct validity, depending on the version of the construct that is being measured.
Even attempting to measure trauma exposure can be tricky. Karstoft and Armour’s paper (2022), “What we talk about when we talk about trauma,“ compared eight commonly used self-report trauma exposure scales. The authors found very heterogeneous assessment across scales; there was only modest content overlap between the scales. Notably, 18 trauma exposures were only assessed by one scale, and no exposure was assessed across all eight scales. There was also a great deal of variation regarding whether respondents were asked to specify the frequency, intensity, duration, and timing of trauma exposure.
Another issue is conducting research in different countries and cultures. Concepts can have different meanings in different contexts. Yet, most of the measures used in the field of traumatic stress were originally developed in English, most commonly in high-income countries such as the US. Furthermore, as noted by Hoffman and colleagues (2022), the quality of translations of PTSD screening measures is highly variable, with a lack of transparency about the translation processes used, and insufficient information about the validity of the translated questionnaires. Without this information we cannot reliably compare “PTSD” between studies using these different translations.
So, what do we do? Beyond acknowledging these issues, Flake and Fried (2020) present a framework for avoiding QMPs, recommending clear definition and operationalization of the construct, justifying the selection of a particular measure, describing the methods for quantification and transparently reporting any modifications. Taking these kinds of steps to improve our measurement practices is fundamental to improving assessment and treatment of people affected by trauma.
About the author
Talya Greene, PhD, is Associate Professor in the Department of Clinical, Educational and Health Psychology and University College London. Her research focuses on investigating the dynamics of traumatic stress symptoms in daily life using ecological momentary assessment, and on psychopathological symptom networks.
References
Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465.
Karstoft, K. I., & Armour, C. (2023). What we talk about when we talk about trauma: Content overlap and heterogeneity in the assessment of trauma exposure. Journal of Traumatic Stress, 36(1), 71-82.
Hoffman, J., Ben-Zion, Z., Arévalo, A., Duek, O., Greene, T., Hall, B. J., Harpaz-Rotem, I., Liddell, B., Locher, C.,
Morina. N., Nickerson, A., Pfaltz, M. C., Schick, M., Schnyder, U., Seedat, S., Shatri, F.,
Fong Sit, H., von Kanel, R., & Spiller, T. R. (2022). Mapping the availability of translated versions of posttraumatic stress disorder screening questionnaires for adults: A scoping review. European Journal of Psychotraumatology, 13(2), 2143019.
Trendler, G. (2009). Measurement theory, psychology and the revolution that cannot happen. Theory & Psychology, 19(5), 579-599.