Page 72 of 76
SR10. Was the choice of synthesis suitable for the data?
results were broadly similar from study to study, variation is discussed, appropriate
weighting where appropriate, heterogeneity discussed
SR11. Was the quality of the studies taken into account in any synthesis?
SR12. Overall assessment of reliability of the findings (SR)
taking into account the answers to questions in domain 1 and 2 and the risk of bias
questions, how reliable are the findings of the study?
Domain 3. How generalisable are the results? SRs
SR13. Did the review consider contextual factors?
SR14. Overall assessment of reliability of the findings
taking into account all the questions on reliability of findings and scores for bias, what is the
overall assessment of the reliability of the findings?
DFE SAGE QA SURVEYS
Domain 1. what are the methods? (surveys)
SY1. Are the objectives of the study clearly stated?
SY2. Is the survey research design appropriate for the objectives of the study?
consider whether the survey designs are the most appropriate for the research question - e.g.
is the survey design the most appropriate for a question of effectiveness? or is a predefined
survey design the appropriate designs for exploring people views and experiences?
SY3. Are there references made to original work if existing tool made?
if the study uses an existing tool, do the authors make explicit references to the original
source?
SY4. Is the reliability and validity of the new tool reported?
if the study uses a new tool, did the authors report on reliability and validity tests (factor
analysis, pilot testing) of this new tool?
SY5. Is any statistical analysis correctly performed and interpreted?
SY6. Are these measures of direct experience?
measures that ask people to reflect on hypothetical scenarios, or their hypothetical behaviour
are generally less reliable than those based on experience
SY6. Were steps take to improve accuracy of recall?
Participants who are asked to reflect on experience that happened a long time ago may suffer
from degraded recall. steps to avoid this are activity diaries or participants are asked to
recall events from recent memory.
SY7. Risk of measurement bias (surveys)
the risk of the study not being able to measure the factors it sets out to measure.
SY7. Risk of questionnaire bias (surveys)
Unintended communication, difficulties in questions designs and factor bias in untested
questionnaires.
Domain 2. Are the results valid? (surveys)
SY8. Is the survey population and sample frame clearly described?
How was this sample identified, and from what whole population are they drawn?