We are proud to report that over the past decade RESEP has made a significant contribution to the capacity for health policy analysis on the continent. In response to identifying a substantial need for more granular evidence on the weaknesses of the often dysfunctional, wasteful and polarized health systems in African countries, RESEP has invested in training young researchers and developing the first health economics elective for Economics graduate students. In parallel, RESEP has expanded its team of health economics researchers and established a reputation for useful, careful and rigorous health economics research. The RESEP health policy team has differentiated themselves from the pack by focusing on the demand side and understanding healthcare from the perspective of the patient. We have over the past ten years been involved in a variety of exciting healthcare projects and they have all been attempts to listen closely and anew to people’s stories – whether it be through primary data gathering or secondary data analysis.
This resonates with recent shifts in global and local policy towards more client-responsive services – as also reflected in South Africa’s Health Market Inquiry and the planned rollout of National Health insurance – but has a different origin. It has always been important for our team to do work that is responsive to the daily battles of the people of South Africa. It needs to resonate with what matters, feed into their lives and connect with that which they care about.
Consequently, we have sought to tell stories of daily struggles: we have told devastating stories about how the high level of trauma and death affects the mental health and well-being of South African households. We have shared stories about pregnant women who delayed their clinic visits, unavailable TB medicine, rude nurses and severely ill children. We have told stories about the sacrifices made by the caregivers of young kids with MDR-TB. We have shared stories about nurses not following protocols, nurses not giving community members the care they need and deserve.
The last set of stories were particularly hard-won, so more detail and depth are warranted. In 2016 our team embarked upon a standardized patient or mystery patient study to assess the quality of clinical services at public primary care facilities in the Cape Town and East London metros. First, we need to explain, a standardized or mystery patient is similar to a mystery shopper – it is someone who is pretending to be an average shopper or patient, but who is, in fact, assessing the quality of the services offered. In the case of primary health care, the idea is that you choose a very standardized clinical protocol so that it is straightforward to assess. For our study, we included hypertension screening, contraception advice and TB screening.
The reasons why we choose the standardised patient approach over other easier and cheaper alternatives is because no other method can produce data that is so granular, personal and honest. Direct observations come the closest but they are affected by the Hawthorne effect (a reactive response in which individuals, when they are aware of being observed, adjust aspects of their behaviour) and because they are a third-party account they are also less personal and granular. Patient file audits are not useful in South Africa because nurses take limited notes and the lack of note-taking tends to be worse in poor communities, which means we are more likely to lack information for the facilities that we are most concerned about.
The way this approach works is that the standardized patients present to the health worker with a specific opening sentence such as “I have been coughing a lot recently” or “I need advice on contraception” that should predictably lead to a series of further questions and tests as outlined by national guidelines and good clinical practice. The methodology requires that you select clinical cases where specific symptoms map very clearly and simply to a set of questions and procedures.
After the visit, the standardized patients capture the questions asked by the nurses and the tests conducted by the nurses on a score sheet. Our standardized patients were recruited from local communities and they received extensive training to ensure that they know how to answer nurse questions and also to not disclose any information without being asked for it.
This is the first time that a standardized patient study has been conducted in South Africa on this scale, so it was pioneering work, but worthwhile. Across all three clinical domains, we found that service delivery was lacking, that nurses were not complying with protocols and that clinics were missing opportunities. What is useful about this methodology is also how specific and granular we can be about where and how problems occur, which provide very useful inputs for policymaking and implementation. The graph below shows that evidence of dysfunctional hypertension management and missed opportunities in patient risk assessment, diagnosis and advice. All of these cases are interactions where patients tested positive for stage 1 hypertension. Yet, in less than half of cases did nurses explain the importance of regular blood pressure monitoring and testing and in only 36% of cases did nurses highlight the risks and severity of high blood pressure. One in five visits were classified as failed: even though nurses documented a blood pressure reading in the stage 1 hypertension zone, they did not diagnose this as potential hypertension or schedule a follow-up visit to confirm the reading. Nurses also failed to provide advice on how to control hypertension via stress management, smoking, moderate alcohol consumption, regular exercise, weight control and a healthy diet.
Through careful and incremental work, we have started to understand just how large the divide is between doctors and their patients, but also the extent to which traditional sources of patient information may often not be an accurate reflection of their experiences. There are a couple of reasons for this, including that patients from poor communities often have very low expectations; in fact, many of the users of public clinics know nothing else than their home clinic. Our standardized patient work also found that patients are very unlikely to report dissatisfaction even if they waited long and nurses were rude to them and they also tend to underreport problems with case management.
We found that our standardized patients tend to be more critical, but also that their satisfaction was more consistently related to the case management quality. Our standardized patients who have benefited from our training on protocols were more likely to say that they were satisfied than real patients that we interviewed as they were leaving the clinic – even when our analysis controlled for differences in socioeconomic status, education levels, age and gender. We also found that our standardized patients tend to be more discerning than the real patients we interviewed: we asked them about the detail of their consultation with the nurses and their experiences in the clinics, but often there were few meaningful patterns. The graph below shows the relationship between satisfaction scores and the experiences of real and standardized patients that visited clinics for contraception advice. It is clear that in most cases there did not seem to be good clinical reasons for why real patients were satisfied or dissatisfied: the coefficients are very low for real patients and in most cases also not significant. By contrast, with our trained standardized patients we observed stronger relationships between satisfaction, protocol compliance and good clinical practice. Asking patients about life circumstances, talking through options and explaining the disadvantages and advantages of options all tend to dramatically increase the satisfaction levels of standardized patients – and far less so for real patients.
We interpreted this as evidence that these patients were better able to discern good and poor quality health services and their satisfaction scores were, therefore, more useful. The research provides evidence of the important role of expectations and knowledge on satisfaction scores. Without more widely available information on government medical protocols and better training on citizen rights, satisfaction scores may not provide much meaningful information on the clinical quality of our health services.
Do patients reliably observe weaknesses in health services?
The stories we hear and tell as a society matter because they are often the stories we live into. As RESEP researchers, we tell our stories for a very specific purpose: we want to cast a light on the plight of poor service delivery in vulnerable and marginalized communities and urge the government to pay more attention to them. There is a Malawian saying that says that outsiders cut with a sharper blade, that newcomers can deliver fresh insights. This is our wish as researchers, to come with a sharp blade that can observe closely and document accurately.
In telling these stories, our team has earned a reputation for credible and relevant research with papers in high-impact and high ranking field journals. Two of our standardized patient stories have already been published and we provide the links to the journal papers below.
However, we write our stories also because we want to influence and change society to ensure that communities and
citizens are at the centre of the government’s agenda, not at the periphery. It has therefore always been important for us to get our messages and stories to government, but increasingly, we have also become more aware of the important role of direct communication with and empowerment of citizens and grassroots NGOs. It is crucial that researchers also help to empower and strengthen communities and community-based organizations – both as an end in itself, and also because this enables communities to do better at holding government accountable.
For those who want to read more about our standardized patient work:
Burger, R. Gerdtham, U.G., Haal, K. Hompashe, D., Smith, A. & Schutte, A.E. 2020. Use of simulated patients to assess hypertension case management at public healthcare facilities in South Africa. Journal of Hypertension, 38 (2): 362-367.
Christian, C.S., Gerdtham, U.G., Hompashe, D., Smith, A. & Burger, R. 2018. Measuring Quality Gaps in TB Screening in South Africa Using Standardised Patient Analysis. International Journal of Environmental Research and Public Health, 15(4): 729.
