Valid or not valid that’s the question: the limited validity of measurement instruments ‘proven valid’

Abstract

Researchers often make use of existing measurement instruments that have proven valid through the statistical testing of psychometric characteristics. However, this is insufficient to establishing content and construct validity. Validity is the extent to which a measurement instrument measures what it claims to measure. This means that the construct concerned should be measured completely, whilst all its elements should be processed by the respondents as intended. New analysis techniques, e.g. IRT, already showed that often constructs measured are only partially represented. Moreover, whether the elements of measurement instruments are interpreted as intended is rarely ever verified: We hardly know how respondents interpret and understand our questions. Cognitive interviewing provides a way to better establish content validity. This is an iterative procedure in which 2-3 small rounds of interviews are conducted with as few as 6-10 members of the target population using ‘Think Aloud’ and ‘Probing’. Through this it becomes clear how items are being understood and interpreted. We present results of cognitive interviews, demonstrating that popular, validated questionnaires, consist of questions that are not well understood by the target population. Moreover, the language used often does not fit the way the respondents would express themselves. Items like: “Are you able to engage into social activities?â€, really puzzles people. Thus, we might not know whether we measured what we aimed to measure. Therefore we conclude to outlining a procedure to employ cognitive interviewing and to better involve the target population, in order to establish real (content and construct) validity.