Using theory of change in child health service interventions: a scoping review protocol

Background: The objective of this scoping review is to map the evidence of how child health service interventions use their theory of change. A theory of change is a hypothesis of how and why an intervention is intended to bring about change. It can be used as a program design, implementation, and evaluation tool. This scoping review will provide an overview of the evidence base for, and identify the way in which, theories of change in child health service interventions are defined, rationalised, developed, presented, and refined. Methods: The inclusion criteria for this scoping review is any child health service intervention globally, that describes their theory of change or theory of change development process. Relevant exclusions include: logic models or logic frameworks that do not meet this review’s definition of theory of change, systematic reviews, behavioural change interventions that target patient’s behaviour, school-based interventions, and maternal health interventions not related to child health outcomes. This scoping review will follow the Joanna Briggs Institute Reviewer’s manual. Relevant publications will be first searched on selected electronic databases and grey literature. A search strategy will be developed. The search will be limited to articles written in the English language. Results of the search will be curated using Endnote and duplicates removed. Results will be imported to Rayyan. The inclusion criteria will be applied during the process of title and abstract screening, by two independent reviewers and disagreements resolved by a third independent reviewer. Full-texts will have the inclusion criteria applied via the same reviewer process. Data relevant to the research sub-questions will be extracted, analysed, charted and discussed. Ethics and dissemination: Ethical approval is not required for this review as we will make use of already published data. We aim to publish the findings of our review in a peer-reviewed journal.


Introduction
Health service interventions are interventions targeting the access to, and the use, costs, quality, delivery, organisation, financing, and outcomes of health care services such as hospitals and primary healthcare services (Institute of Medicine Committee on Health Services Research, 1994;Lohr & Steinwachs, 2002;Steinwachs, 1991). An intervention is any organised activity, program, project, or initiative, that is supported by resources and established with the purpose of enacting change. This scoping review will focus on chid health service interventions, defined as any such intervention with a target population of children aged 0-19 years old. Health service interventions are often complex by nature, involving multiple stakeholders, diverse objectives, and are dependent in context. A theory of change (ToC) is a tool utilised by some health service interventions to navigate this complexity.
A theory of change (ToC) is a theory or hypothesis of how and why an intervention is intended to bring about change (Center for Theory of Change, 2021;Connell & Kubisch, 1998;De Silva et al., 2014;Weiss, 1995). When a ToC is reported in internal or external organisation publications, it is an articulation of the hypothesised pathways that lead to change. It presents the cumulative study of an intervention's assumptions, activities, mechanisms, measurement indicators, outcomes, and context, as well as the linkages between these various components (Connell & Kubisch, 1998).
A ToC can be presented in narrative form or illustrated visually using ToC diagrams. These diagrams are referred to by a number of different terms, depending on subtleties in their form and contents. These terms include, but are not limited to: logic models, logic frameworks, logframes, outcome hierarchies, and result chains. Inappropriate use of these terms can create confusion amongst the ToC literature (Weiss, 1997b). It is therefore important in this scoping review to explicitly outline what is meant and not meant by the term theory of change. For example, perhaps the most common term used incorrectly as a ToC synonym is logic model. De Silva et al. (2014) outline that logic models, strictly speaking, are more simple, linear, and rigid diagrams than true ToC diagrams, that do not outline assumptions, measurement indicators or describe a consideration of the intervention's causal relationships. 'Indicators' are the ToC terminology for the specific measurable marker used to determine if each activity performed in the proposed causal pathway and associated outcomes in a ToC, are achieved. This data can be either qualitative or quantitative. Indicators should include the population group related to the activity or outcome and the threshold that is the minimum accepted achievement of the outcome (Taplin & Clark, 2012). An example a ToC diagram and a logic model have been included as Extended data to aid understanding (Jones, 2022).
The terms ToC, and the synonymous theory-based evaluation, are generally accepted to have been popularised in the 1990s, stemming from decades of work in program evaluation and program theory (Connell & Kubisch, 1998;Connell et al., 1995;Weiss, 1995;Weiss, 1997b). They have been used in many fields, including health, education, business, community development and social welfare (Center for Theory of Change, 2021;De Silva et al., 2014;Weiss, 1997b). Despite this seemingly broad application base, the value that a ToC brings to an intervention needs to be considered if more organisations are to invest the resources required to articulate and present a comprehensive ToC. Even as early as 1987, Huey-Tsyh Chen and Peter Rossi argued for the internal and external validity of a theory based evaluation approach (Chen & Rossi, 1987). Then in 1997, Carol Weiss, one of the architects behind the development of ToC articulation, noted that the literature was "replete with paeans to the value of the theory based approach" to program evaluation (Weiss, 1997a). That is, a number of studies speak to the benefits of the ToC development process such as helping to engage stakeholders, clarify focus, enhance connection to the intervention, and engage team thinking around the process that underpins interventions. Furthermore, these studies report the value that a ToC provides once completed, including providing a roadmap for the intervention, a clear anchor for evaluation, a base in which to refine thinking and monitor the intervention as context changes overtime, scale-up utility, and identify intentional and unintentional consequences (Anderson, 2006;Blamey & Mackenzie, 2007;Connell & Kubisch, 1998;De Silva et al., 2014;Rogers et al., 2004;Weiss, 1997a;Weiss, 1997b). However, it is important to consider that there is a selection bias of papers in the literature that present beneficial ToC articulations, rather than those that failed and were stopped short. Some issues with ToC identified in the literature include: the confusion around definition, that they are reductionist models of more open systems that give a false sense of control to the intervention implementers, they are not developed with enough rigour, and provide an excuse not to adapt when context changes (Connell & Kubisch, 1998;De Silva et al., 2014;Weiss, 1995;Weiss, 1997a). Overall, this scoping review will aim to answer the question: how are ToCs for child health service interventions developed, utilised, and refined? It will also aim to answer the following research sub-questions: A scoping review was selected for this study as it will provide a mapping opportunity as well as an overview of the available research evidence; allowing for the identification of the way in which ToC in child health service interventions are defined, rationalised, developed, presented, and refined. This scoping review is intended to inform organisations working in the field of child health about how they may design and utilise their own ToC. Ultimately, the objective of this scoping review is to map the evidence how child health service interventions around the world develop, use, and refine their theory of change.

Methods
The proposed scoping review will be conducted in accordance with the JBI methodology for scoping reviews (Joanna Briggs Institute, 2015;Peters et al., 2020) and was also used for the development of this protocol.

Eligibility criteria
Participants. Health service interventions targeted at children and youth aged 0-19 years will be included. Health service interventions will be defined as interventions targeting the access to, and the use, costs, quality, delivery, organisation, financing, and outcomes of health care services. The term 'intervention' will be defined as any organised activity, program, project, or initiative, that is supported by resources and established with the purpose of enacting change. This scoping review will exclude studies in which the main target of the intervention was individual patients rather than the health service more broadly. For example, an intervention using behavioural change theory aiming to influence a patient's cognition and behaviour would be excluded. However, if an intervention used behavioural change theory aiming to change the behaviour of staff members in order to more effectively deliver a healthcare service, this will be included. Settings outside healthcare facilities such as schools, will be excluded. Obstetric health service interventions that aim to improve maternal health outcomes rather than newborn health outcomes will also be excluded. Some health service interventions that overlap with public health interventions such as vaccination programs will be included in this scoping review.

Concept.
Interventions that discuss their ToC will be included. Specifically, papers will be included if they a) describe how a ToC was utilised throughout any stage of a child health service intervention such as design, implementation, or evaluation. Or b) describe the development process for a ToC planned to be used in a child health service intervention. A ToC will be defined as a theory or hypothesis of how and why an intervention is intended to bring about change. As highlighted by Breuer et al. (2016), it may be difficult to ascertain if a study has discussed a ToC as per this definition purely from the title and abstract alone. Therefore, during title and abstract screening a study will be included for full-text screening if it refers to the development of their own ToC, theory-based evaluation, theory informed evaluation, program theory, outcomes hierarchy, results chain logic model, or logic framework or, if the title/abstract include comments on the steps of how, or mechanisms behind why, an intervention worked. Then, during full-text screening, when a more thorough analysis of how these studies used the various terms above can be conducted, the specific definition of ToC above will be applied. The included studies should include a consideration of at least one of the following ToC components; assumptions, activities, mechanisms, measurement indicators, outcomes, and context, or the linkages between these various components.
Sources. This scoping review will consider a number of different study designs including both quantitative, qualitative and mixed-methods study designs. From the grey literature, organisational e.g., private organisations and NGO's as well as government ToC documents will be included. Systematic reviews will not be included. There will be no other limits of date, study design, or type of publication.

Search strategy
The search strategy will aim to locate both published and unpublished studies. An initial limited search of MEDLINE was conducted to identify articles on the topic. The text words contained in the titles and abstracts of relevant articles, and the index terms used to describe the articles were used to develop full search strategies for each database (see Extended data). The search strategy, including all identified keywords and index terms, will be adapted for each included database and/or information source. Broadly, the database search will combine terms from the two themes: childhood age range and ToC. The reference list of all included sources of evidence will be screened for additional studies.
Studies published in English will be included. This is due to resourcing constraints within the research team. No publication year restrictions will be used in the search in order to gain an understanding of how different terminology has been used over time.
The databases to be searched include, MEDLINE, EMBASE, Global Health, WHO Global Index Medicus, CINAHL and SCOPUS. Sources of unpublished studies/ grey literature to be searched include the first 10 pages of a specific google search 'Child* "theory of change" filetype:pdf'. The titles and then full-texts of these pdfs will then have the same criteria applied as the literature. The most recent search of these sources in formulating the search strategy was completed on 14 th December 2021.

Evidence selection
This will be an iterative process whereby the literature will be searched and the search strategy refined. Following the search, all identified citations will be collated and uploaded into Endnote V20 and duplicates removed before being uploaded to Rayyan, an online review management software. Following a pilot test, titles and abstracts will then be screened by two or more independent reviewers for assessment against the inclusion criteria for the review. The inclusion and exclusion criteria were refined and agreed by all three researchers. As outlined in the introduction, there are a number of terms used to describe diagrams that are similar to, but often lack the depth of, a complete ToC. This presents a challenge during particularly the title and abstract screening stage, where a paper may not explicitly acknowledge an articulation of a ToC and label it as such. Therefore, careful analysis of the abstracts to identify if comments are made on the 'how' and 'why' an intervention has worked, will be critical when there is an absence of specific phrases such as; ToC, theory based evaluation, theory informed evaluation, logic model, logic framework, or program theory. This approach to study selection diverges from that of the review conducted by Breuer et al. (2016) who limited the criteria to needing the specific term 'theory of change' to be explicitly mentioned in the title or abstract order for the paper to meet the inclusion criteria.
Following this, potentially relevant sources will be retrieved in full. The full text of selected citations will be assessed in detail against the inclusion criteria by two or more independent reviewers. Reasons for exclusion of sources of evidence at full text that do not meet the inclusion criteria will be recorded and reported in the scoping review. Any disagreements that arise between the reviewers at each stage of the selection process will be resolved through discussion. The results of the search and the study inclusion process will be reported in full, in the final scoping review and presented in a Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for scoping review (PRISMA-ScR) flow diagram.

Data collection
Data will be extracted from papers included in the scoping review by two or more independent reviewers using a data extraction tool developed by all reviewers, to aid with consistency of which data will be extracted (see Extended data). The data extracted will include specific details about the participants, concept, context, study methods and key findings relevant to the review questions. A draft extraction form is provided (see Extended data). The draft data extraction tool will be modified and revised as necessary during the process of extracting data from each included evidence source. Modifications will be detailed in the scoping review. Any disagreements that arise between the reviewers will be resolved through discussion, or with an additional reviewer(s). If appropriate, authors of papers will be contacted to request missing or additional data, where required.
Data analysis and presentation. Data will be presented in table form. Furthermore, a narrative summary will accompany the tabulated information and will describe how the results relate. Data will be analysed using both qualitative and quantitative methods. Qualitative data analysis will involve all reviewers engaging in an in-depth discussion about the ToC data and describing the major concepts arising. Literature will be analysed to study location, type of health service intervention, and ToC definition (using a checklist of key ToC components such as assumptions, activities, mechanisms, measurement indicators, outcomes, context, and linkages), ToC presentation. Data extracted from the data extraction tool (see Extended data) will then be charted as per the JBI. Firstly, in the data charting process, a calibration exercise with the full team will be implemented. This will involve a random sample of 20 citations for title and abstract screening. A roundtable discussion will be had to clarify any issues. After the initial calibration exercise, one author (BJ) will chart the data independently and two authors (ME and SN) will verify the data for accuracy. Inconsistencies and disagreements will be resolved through discussion. An iterative approach will be taken with the data charting process and major revisions with rationale will be included in the final report. Critical appraisal of individual sources of evidence will not be completed as it is out of the scope of this review.
This scoping review has a number of potential impacts. Firstly, it will further elucidate to implementation researchers how ToC are described in the literature, what is meant by the term ToC, the value that ToCs bring to interventions, what is missing from these descriptions, and encourage consideration of how these descriptions could be improved. Secondly, for health service practitioners working within the field of child health service interventions, it will map the available evidence on how these interventions may use a ToC. Thirdly, this scoping review may encourage those in fields outside of child health service interventions to consider using a ToC as a program design, implementation, and evaluation tool.

Study status
This scoping review will be conducted in early 2022. Thank you for the opportunity to review this paper I have read the paper with interest and I think the background idea is interesting. Child health is a growing global concern, and implementing and/or evaluating health service programmes with the promise to improve child health become imperative. While thinking of programmes that may provide better gains for child survival, developing a good theory of change (TOC) in the implementation and/or evaluation of such programmes could bear multiple benefits as enumerated by the authors. Hence, the need to better understand how TOC is defined (clarity of concept), developed and used (the object of this review protocol).

Comments to the authors:
I have found that the rationale for, and the study were clearly described. I would like to subject that the heading 'objective' be included in the paper to guide the reader on the flow of the steps followed in the review. As it stands, the objective seems lost in the introduction.

○
The sentence "This scoping review is intended to inform organisations working in the field of child health about how they may design and utilise their own ToC", seems restrictive. There may be individuals e.g. students, public health practitioners, health promoters who may be interested in designing their own TOC for particular interventions and not be part of a given organization.
If "this scoping review may encourage those in fields outside of child health service interventions to consider using a ToC as a program design, implementation, and evaluation tool" as stated by the authors on the potential impacts of this review, then the statement in the above paragraph should be corrected to expand to include a broader audience. In this sentence, I wonder whether the development, refining and use of TOC is done by researchers (designers of interventions, or interventions) "Ultimately, the objective of this scoping review is to map the evidence of?? how child health service interventions around the world develop, use, and refine their theory of change" ○ The authors opted for the JBI methodology for scoping reviews (Joanna Briggs Institute, 2015;Peters et al., 2020) although appropriate for the scope of study proposed by the authors, I did not find the rationale for this choice, given the existence of other approaches to scoping reviews.The authors have sufficiently provided details of the methods to allow replication by others.