Assessing and Improving the Reliability of our PROs in Complex Multifaceted Designs – An Introduction to Generalizability Theory
Organized By: Psychometrics SIG
Organized By: Psychometrics SIG
Description
Reliability is a key property that needs to be assessed with all PROs, as outlined in regulatory guidance documents as well as psychometric best practice. That is, to what extent are my PRO scores reproducible or consistent across facets of interest, e.g. time? However, common means of estimating reliability may be misleading and should not be applied in complex multifaceted designs where multiple sources of measurement error can detract from reliability. To illustrate, imagine a PRO scale that is administered both via paper & computer, as well as at home and/or a clinical site, and at multiple time points. In this example, there are three sources of measurement error (plus interactions) that could impact reliability, namely, the sample of items in the scale, the modality through which the PRO is administered (paper or computer), location (home or clinic) as well as time. Common reliability indices (e.g. Cronbach’s alpha) should not be employed to estimate reliability with multifaceted designs (i.e. where there are several potential sources of measurement error). G-theory, on the other hand, provides a useful framework to partition these sources of error and properly estimate reliability. This workshop will illustrate the immense potential of g-theory in not only estimating PRO reliability but also improving the quality of our measurements.
Intended AudienceReliability is a key property that needs to be assessed with all PROs, as outlined in regulatory guidance documents as well as psychometric best practice. That is, to what extent are my PRO scores reproducible or consistent across facets of interest, e.g. time? However, common means of estimating reliability may be misleading and should not be applied in complex multifaceted designs where multiple sources of measurement error can detract from reliability. To illustrate, imagine a PRO scale that is administered both via paper & computer, as well as at home and/or a clinical site, and at multiple time points. In this example, there are three sources of measurement error (plus interactions) that could impact reliability, namely, the sample of items in the scale, the modality through which the PRO is administered (paper or computer), location (home or clinic) as well as time. Common reliability indices (e.g. Cronbach’s alpha) should not be employed to estimate reliability with multifaceted designs (i.e. where there are several potential sources of measurement error). G-theory, on the other hand, provides a useful framework to partition these sources of error and properly estimate reliability. This workshop will illustrate the immense potential of g-theory in not only estimating PRO reliability but also improving the quality of our measurements.
Measurement scientists and COA specialists with an interest in advanced psychometrics who routinely work with PROs, either in industry or consulting
Audience Benefits
This webinar will provide the audience with an overview of:
- Generalizability theory (g-theory), common terminology as well as an overview of why it’s better suited to assess the reliability of our PRO measures in a multifaceted design.
- Practical questions that g-theory can answer with PROs and COAs and how this framework can help us to improve our measures as well as design via variance component estimation (generalizability or g-studies) and decision studies (d-studies).
Learning Outcomes/Objectives
- Identify scenarios where traditional reliability measures may be inappropriate and better suited to generalizability analysis.
- Understand terminology commonly used in g-theory with practical illustrations.
- Gain confidence in reading journal articles that estimate reliability based on generalizability theory
Presenters
Recording Policy
Photography, video or audio recording (including screen capture) of ISOQOL courses, sessions, digital materials, or ISOQOL graphics without written permission from ISOQOL is strictly prohibited.
Note that photographs and video recordings taken by or on behalf of ISOQOL is the property of ISOQOL.
ISOQOL members receive a discounted rate. To view membership rates, click here.