Development of indicators to measure health system capacity for quality abortion care in 10 countries: a rapid assessment of a measurement framework and indicators

Introduction

Strong health systems are required to achieve Sustainable Development Goal targets 3.1, 3.7 and 5.6, which ensure sexual and reproductive health services and rights. However, there are severe weaknesses in health system capacity to deliver quality abortion care in many countries, such as low facility readiness or poor availability of essential equipment and skills.1 2

Effective monitoring and evaluation is essential to inform evidence-based programme design, but monitoring of health system capacity to provide quality abortion care is lacking. Monitoring health system performance requires an assessment of inputs and processes (eg, governance), outputs (eg, service readiness), outcomes (eg, intervention coverage) and impact (eg, health outcomes).3 A recent extensive scoping review identified that most abortion indicators focus on outputs, outcomes and impact (eg, access, availability and physical health),4 and do not pay adequate attention to inputs and processes, although these are critical elements of health system performance.3 5 The scoping review highlighted the need to develop and test new indicators for monitoring health system capacity to provide quality abortion care.4

General health system inputs and processes are well-defined. The WHO health system building blocks are essentially inputs and processes, and these include leadership and governance, health workforce, health information systems, medical products and technologies, health financing and service delivery.6

Although existing health system framework indicators can monitor overall health system capacity,3 6 7 they are not sufficient to effectively monitor health system capacity to deliver quality abortion care, as abortion is often sidelined or excluded from mainstream health education, planning, service provision and financing.8 9 For example, health worker density and distribution indicators do not tell us whether there is adequate density and distribution of health workers that have essential competencies or legal authorisation to provide abortion care. For stigmatised or neglected areas of health, it is particularly critical for disease-specific or topic-specific indicators to be integrated into common platforms, data process and management systems.10 Abortion has often historically been excluded from monitoring indicators and initiatives11 12 and even monitoring frameworks for sexual and reproductive health (SRH) often exclude abortion.13 14

A significant gap therefore exists in the availability of indicators and tools to monitor health system capacity to provide quality abortion care at the input and process levels. In this paper, we describe the process and results of developing input and process level indicators designed to measure health system capacity to provide quality abortion care. We define quality abortion care as care that is effective, efficient, accessible, acceptable, equitable and safe and includes abortion care, management of complications after an abortion and services after abortion such as contraceptive services and linkages to other relevant services.15

Methods

We describe the process of developing indicators to monitor a 4-year project focusing on health system strengthening to prevent unsafe abortion in ten countries. This is followed by a descriptive analysis of an internal evaluation of the indicators.

Results

A total of 30 new indicators were developed across the 5 health system building block domains (table 1). Key definitions, data source suggestions and reference documents are included in the metadata document in online supplemental appendix 1.

All indicators could be constructed with information from existing sources with the expectation that no primary data collection would be required from facilities, providers or patients and no confidential information would be collected. Depending on the indicator, existing sources could include administrative documents such as policy papers, clinical training curricula, health management information system databases and financial tracking systems. Many of the indicators were designed to facilitate collection of information separately for abortion, postabortion care and family planning, rather than for quality abortion care as a single compound unit, which enabled interpretation and identification of specific areas for improvement (online supplemental appendix 2). The newly developed indicators were specific to abortion but were similar to those used for other health topics, to ensure alignment with existing health system monitoring processes and avoid siloed abortion care monitoring processes.

Six leadership and governance indicators were created to assess the level of political commitment to SRHR, including abortion-related rights, through policies, strategies and approaches for health system governance. The five health workforce domain indicators assess the extent to which systems are in place to support adequate production, availability and distribution of health workers trained in provision of abortion-related care. The six health information system domain indicators focus on inclusion of SRH indicators in the national health information system. The eight access to medicines and health products domain indicators assess the systems in place to promote availability of quality-assured essential abortion medicines. The five health financing domain indicators assess steps toward better tracking of financing flows allocated for SRH service provision, the level of inclusion into the benefits package of major national health financing mechanisms and the way these mechanisms allocate resources to health service providers. Not all indicators were specific to abortion. However, all indicators included were understood to be representative of the enabling environment for quality abortion care.

Table 1 summarises the results from the evaluation of indicators completed by 10 assessment teams, including average scores for each criterion, and specific challenges identified by teams. Average scores were lowest for feasibility (7.7 out of 10) compared with importance (8.5), usefulness (8.9) and validity (9.3). A key feasibility issue was that some data were not available or were difficult to access, and this was reflected in high levels of missing data for some indicators (such as 2.3, 4.8 and 5.3 in table 1) during project monitoring.

Of the domains, average scores were lowest for health financing (ranging from 6.0 for feasibility to 8.4 for validity) and highest for leadership and governance (ranging from 9.0 for importance to 9.7 for validity and usefulness) (data not shown). Text feedback on the health financing indicators mostly related to the perceived complexity of these indicators, challenges with interpretation and difficulties experienced with measurement. Feasibility scores were also below average for health workforce indicators (6.6 on average), and text feedback highlighted that some of the workforce indicators (such as 2.1, 2.2 and 2.3 in table 1) were difficult to track, involved requesting information from multiple education institutions or non-health government departments such as Ministries of Education or Labour, and required multiple judgements about whether facilities are accredited, and whether training is competency based. As staffing needs assessments are conducted by cadre, rather than by the specific functions staff serve, assessors found that a lack of relevant data also made workforce indicators less feasible to measure, highlighting the need for more granular data at lower levels of the health system.

Broader feedback included the need for fewer and less complex indicators. Assessors highlighted the challenge of interpreting numbers-based indicators (such as 1.5 and 5.2 in table 1), as clear benchmarks do not exist for the desired number of policies or institutions. Assessors also noted the challenges of interpreting results from the more subjective, descriptive indicators (such as 1.6 and 2.5). During project monitoring, in some countries leadership and governance indicators (such as 1.2 and 1.4) were recorded as supportive of an enabling environment despite existing in the context of restrictive abortion policies. Some of these indicators referred to SRHR broadly rather than abortion specifically, which may explain this seeming contradiction. Some assessors felt it was important to include a greater focus on family planning, as access to family planning may be an upstream determinant of unsafe abortion.

In the feedback responding to specific domains and the set of indicators as a whole, assessors noted additional areas to potentially include. The proposed additional areas for each health system building block domain are listed in table 1. For the framework as a whole, some assessors felt that service delivery indicators and research and innovation should be included.

Discussion

Strong health system capacity for abortion care is needed to ensure that high-quality abortion care is available and accessible. To successfully strengthen health system capacity, effective indicators are required to monitor progress, identify gaps and inform action. In this paper, we present 30 indicators to measure health system capacity for quality abortion care across five health system domains, alongside feedback reflecting the experience of implementing these indicators in 10 countries. The five health system building block domains were found to have relevance for monitoring health system capacity to provide abortion care: governance and leadership, health workforce, health information, access to medicines and health products, and health financing. Indicators that scored highly against selected criteria (validity, feasibility, usefulness, importance) may be useful for inclusion within future efforts to monitor and strengthen health system capacity for quality abortion care. The indicators have the strengths of drawing on existing data sources, not requiring primary data collection and aligning with existing health system monitoring. These characteristics reduce the resource requirements and intensiveness of the indicators and can increase motivation to use them. Many of the indicators can also inform assessments of health system capacity to support self-care for abortion, as evidence-based protocols, trained workforce and medicine registrations are essential for both provider-led and patient-led models of care.31 Drawing on feedback from implementation of these indicators, we identified challenges which can be remedied and areas that can be strengthened; these are discussed below.

As a set, the indicators presented in this paper have the limitation that they do not cover all six health system building block domains. The indicators were developed for a specific project and the selection process was therefore guided by the project activities we intended to monitor. We developed indicators for five of the WHO health system building blocks, but not for the area of service delivery, as the project did not include activities in this area. A recent scoping review of abortion metrics4 identified that among abortion care indicators, those of access, availability and provision of abortion care were the most common. This suggests service delivery indicators for abortion are already well-developed, even though they may not be commonly used.

Other key primary healthcare (PHC) operational framework levers were not included in our framework, such as engagement with communities and other stakeholders, PHC-oriented research, physical infrastructure and digital technologies for health.32 This point was highlighted by assessors in their feedback on the indicators, as it was noted that research and innovation indicators should be included in future work. Digital technologies have become particularly important for abortion care in recent years due to the growth in self-management of medical abortion, with telemedicine,33 hotlines34 and other digital interventions35 36 providing new modes to expand access to patient-centred care. Future work could develop indicators within some of these additional domains and could include some of the indicators proposed by assessors in their feedback, for example, whether competency-based training guidelines for SRHR have been endorsed at the national level in preservice education of healthcare workers. Other areas to consider in further indicator development include policies on conscientious objection to abortion care provision, policies relating to consent for abortion care based on age and marital status, the level of the health system at which abortion care and abortion medications specifically can be offered and the regulated availability of medical abortion drugs such as prescription requirements and regulatory class. By adding further detail, indicators could also assess capacity for first and second trimester abortion separately.

Assessors highlighted the need for clear indicator specification. Feedback from assessors identified the need for terms (such as competency-based training) to be clear and consistent across settings to assess whether a specified standard has been met. Some descriptive indicators, particularly those measuring the enabling environment for abortion care, were considered by assessors to be too subjective to generate comparable measurement across countries and over time or were difficult to interpret due to a lack of obvious benchmarks. Not all indicators were designed for cross-country comparison and instead were intended to describe and analyse the present situation in a country so that this information could inform future action plans (eg, 5.3: number of health financing instruments). Some indicators do not necessarily require a benchmark (eg, 1.5: number of laws and policies) as there is not an ideal number of laws and policies, but tracking the number that align with global or WHO guidelines can indicate whether the national situation is improving or declining over time. There is a need to balance the inclusion of richer, more descriptive assessments which can inform decision-making in one context, with the need for quantitative indicators that minimise subjectivity and allow tracking of change over time and between contexts. While health system monitoring is primarily useful at a national level to inform policies and programmes, cross-country comparisons can help to identify effective strategies for improving abortion access or to motivate reform. Additional precision for such comparisons could be achieved by including structured subcomponents to descriptive indicators for scoring and structured narrative reporting.

Most of our indicators were measured specifically for abortion, postabortion care and family planning. However, some indicators referred more broadly to SRHR and its integration into country cooperation strategies, health benefit package assessments and health financing instruments. Since abortion care is often excluded from efforts to improve SRHR,13 14 it is possible that these indicators may not be specific enough to capture whether quality abortion care has been sufficiently addressed within these mechanisms. Design of future indicators to monitor quality abortion care should consider the need for specificity, given the politicised and stigmatised nature of abortion.37 Additionally, although it is important to improve the visibility of health system capacity to deliver abortion care, this need must be balanced with the potential risk of siloed disease-specific or topic-specific monitoring. The indicators presented in this paper were developed to align with existing health system monitoring and tools, to facilitate integration and avoid siloed abortion care monitoring; this tension requires further consideration in future work.

Feasibility of measurement was a common challenge listed by assessors. To be used, indicators must be feasibly measurable from accessible sources. We found our indicators scored lower on feasibility than on any other criteria, and several of the indicators were only considered to be feasible by 5 or fewer out of 10 assessors. In some cases, this was because measurement was found to be complicated, involving consultation of multiple information sources. For many of the indicators, we assumed the information was available in the public domain or via communication with ministries, but this was not always the case. The feedback from assessors therefore highlighted the need for more granular data to be available at lower levels of the health system, for example, from educational institutions or health financing assessments, which could support efforts to measure topic-specific health system capacity. We intended for indicators to be measured annually in our project, but some indicators may change more rapidly than others, and future work could also consider the regularity with which each indicator should be assessed.

Health financing indicators scored particularly low in our indicator assessment, in part due to feasibility concerns but also due to perceived over-complexity. This may partially reflect the fact that most assessors were SRH specialists and did not have expertise in health financing. Similarly, some of the challenges highlighted by assessors in the domains of access to medicines and health products and health workforce could reflect a lack of expertise in these technical areas among most of the assessors. This highlights the need for collaborative working in health system strengthening, as siloed expertise can limit programme effectiveness as well as monitoring and evaluation efforts. The reliance on other sectors of government (legislation, education, finance) to measure certain indicators also highlights the need for multisectoral collaboration when monitoring and addressing health system inputs for abortion care.

Challenges with tracking the proposed health financing indicators may also highlight the lack of evidence to support health financing for quality abortion care. Health financing is a critical area for abortion care and there is a need to strengthen the capacity of SRH programmes in this area. Out-of-pocket payments are the dominant source of funding for reproductive healthcare in many countries,38 and economic factors can thwart care-seeking, affect the type of care sought and impact the gestational age at which care is reached.38 However, the fragmentation of health financing in many countries and the lack of transparency in financing data can make progress hard to measure. The 2020/2021 WHO Health Benefit Package Survey,39 implemented separately from the project reported on in this paper, identified services covered in health benefit packages of the largest government-financed scheme in each of 115 countries and areas. The 2020/2021 survey included abortion care for the first time, and now this indicator can be used in future work.40 Future indicators could also assess whether any specific conditions exist for the inclusion of abortion care in health benefit packages.40

As next steps, we suggest these indicators should be further reviewed and amended by the wider SRHR measurement community, to address the gaps we have identified, strengthen definitions where needed and ensure adequate balance across health system capacity domains. This process will also require consultation with broader health experts in the areas of medicines and health financing, as well as experts involved in education, labour, legislation and public finance. Indicators from project monitoring have been adopted into research projects and monitoring systems, including the Sexual and Reproductive Health Self-Care Measurement Tool.31 This new tool includes several of our high-scoring leadership and governance indicators to measure the enabling environment for self-managed abortion. The adoption of project indicators into research projects and tools shows the demand for indicators in this area and the urgency to move forward with next steps.

This process of developing and assessing indicators to measure health system strengthening for quality abortion care had limitations. Although a large team with broad knowledge in relevant areas developed the indicators, and there was some input from Ministries of Health at the indicator development stage, we did not then seek further validation through external review or involvement of wider stakeholders. Most implementers and assessors of the indicators were specialists in SRH and did not have technical expertise in health financing, access to medicines and health products or health workforce, which may have influenced indicator scores in these domains. We did not receive feedback from all project team members, but the full team discussed the results in several project meetings to ensure overall agreement. Criteria statements in the evaluation of indicators could also have been more specific and distinct to ensure meaningful responses. Most indicators were scored highly, and it is possible there was some confirmation or social desirability bias involved in these responses, given that assessors were involved in the original development of these indicators and responses were not anonymised. To mediate bias, assessors were requested to review the indicators based on their experiences with implementation, with the clear understanding that the indicators needed constructive feedback and revision. Resulting scores ranged widely—from three to 10—and text responses included constructive critique. The indicators were used for project monitoring by teams during the COVID-19 pandemic, which may have made the collection of information more challenging than it would have been otherwise, so the low feasibility rating of indicators may reflect some of these COVID-19 related challenges. Yet, we also recognise that WHO’s close collaboration with Ministries of Health may have facilitated access to information and without these relationships, feasibility scores may have been lower. Finally, while we consider the indicators to be applicable for a federalised system, they may require some adaptation in future work to be specified for each region or area.

The initial goal of this work was to monitor a health system strengthening project for quality abortion care, not to develop a normative set of indicators, and we do not intend for our indicators to be adopted verbatim elsewhere. In this paper, we present an initial attempt to develop indicators to measure health system capacity for quality abortion care, along with clear suggestions for how these indicators could be strengthened in future work. Our work had several strengths. The indicators, structure and criteria were developed by a group with broad knowledge in the relevant areas, were implemented as a baseline with two follow-up rounds, and were implemented in multiple countries with diverse contexts. Structured feedback was obtained from the experience of implementation. Most of the indicators we developed scored highly against predetermined criteria, and the indicators were drawn from existing data sources and aligned with existing health system monitoring efforts. These indicators and the future directions we have identified therefore fill a significant gap in abortion monitoring and evaluation and will serve as building blocks for future projects.

Conclusion

Meeting Sustainable Development Goal targets 3.1, 3.7, 5.6 and other commitments to advance SRHR depends on evidence-based planning by health system leaders, policy-makers and others. Having identified a gap for indicators in health system capacity to provide quality abortion care, in this paper we shared a set of indicators developed for a multicountry, 4-year project focusing on health system strengthening for abortion. After multiple rounds of implementation, we have made progress toward identifying appropriate domains and indicators that are valid, feasible, useful and important. We have also identified challenges associated with these indicators and potential areas for strengthening. We have outlined further work that needs to be done in the area before a core set of indicators can be recommended. Establishing a core set of indicators to facilitate measurement of health system capacity for quality abortion care is critical for motivating action and accountability in reproductive health and rights and meeting Sustainable Development Goal commitments.