How to ask the wrong question to get the 'right' answer

ISSN 0256-5004 (Print)

AIMS Journal, 2007, Vol 19 No 4

Debbie Chippington Derrick questions the Healthcare Commission's Review of Maternity Services 2007

As an exercise in obfuscation the Healthcare Commission's Review of Maternity Care is a classic example. Instead of evaluating the quality of maternity care in a meaningful way the Healthcare Commission has used a similar scoring system to that used by the Eurovision Song Contest where there are a set number of points to be assigned however bad all of the songs are, with the result that the 'less worse' songs still obtain a high score.

The Review of Maternity Services 2007 was published by the Healthcare Commission, England's healthcarewatchdog, on 25th January 2008. It claims 'Our review explored how organisations in the NHS are improving the way in which they deliver maternity services.' As my local unit was rated as a 'best performing' Trust I decided to have a look at what this actually meant, and how helpful the Review is in practice at illustrating how a Trust is performing.

There are 148 NHS Trusts detailed in the Review; 38 Trusts receive the classification 'best performing' (26%), 42 as 'better performing' (31%), 37 as 'fair performing' (22%) and 31 as 'least well performing' (21%). Interestingly the majority of the 'least well performing' Trusts were in London, where there were no 'best performing' and only one 'better performing' Trusts.

The Review emphasised that:'An assessment of "least well performing" does not mean that a trust is providing care which is unsafe. If we have concerns that a trust is unsafe, we do not hesitate to use our powers of enforcement by, for example, carrying out investigations.' The Review did not acknowledge that Trusts categorised as 'best performing' may also have significant failings that need to be addressed. I am also left with concerns; if the standard of this Review is an indication of how effective the investigations are, how is it going to be able to identify when a Trust is unsafe?

Trusts were rated on 5 indicators in 25 different areas. My local Trust, which ranked as a 'best performing' Trust, achieved an average score of just under 3.5 across all these indicators. The classification of performance was based on each Trust's averages of all 25 scores with

best performing greater than 3.28 average score
better performing less than 3.28
but greater than 3.00 average score
fair performing less than 3.00
but greater than 2.76 average score
least well performing less than 2.72 average score
(worst score was 1.96)

Measurement of performance

It quickly becomes apparent that the measures used are not assessing any standards of care that are considered desirable or appropriate, but that the performance of each Trust is measured solely against the performances of all other Trusts. So the outcome of the study's finding that some Trusts are performing well and others less well is a design feature of the study, not a reflection of the quality of our maternity services.

Since the study does not attempt to measure the performance of Trusts against standards that should be expected from such a ser vice or might be considered desirable, were it to be repeated in the future it would give absolutely no indication of whether our maternity services had improved or declined; it would simply show how each Trust is performing against the others, at that time, on the chosen set of questions.

For most of the indicators the scoring system was set up so that the worst 6.25% of Trusts would receive One point, then next worst 25% of Trusts receive two points, the middle 37.5% of Trusts three points, then next best 25% four points and the best 6.25% five points; as illustrated in Figure 1 below. There were some slight adjustments of this method for some of the indicators, as will be seen in those described later where more than one aspect was included in the indicator.

vol19no4blockdiagram.gif

Figure 1

What was assessed?

The 25 indicators are listed below and although there are some important aspects that are clearly missing such as place of birth, normal birth rates and infant morbidity, on the surface this looks as though this addresses important issues.

  1. Women not receiving NICE recommended number of antenatal appointments
  2. Availability of NICE recommended screening
  3. Appropriate use of caesarean sections
  4. Maternal morbidity
  5. Postnatal care of women and babies
  6. Progress on implementing Mental Health NICE guidance
  7. Extent that staff are trained in core maternity skills
  8. Safety culture
  9. Average time between first making contact and booking appointment
  10. Choice and continuity for antenatal care
  11. Percentage of women offered an informed choice for screening tests
  12. Percentage of women attending NHS antenatal classes who wanted to
  13. Extent of choice in labour
  14. Support for infant feeding
  15. Quality of support in caring for the baby after discharge
  16. Stakeholder involvement in service planning and evaluation
  17. Staffing levels
  18. Integration of support workers
  19. Average cost per delivery
  20. Delivery of hospital based antenatal care
  21. Data quality
  22. Appropriate involvement of obstetricians and midwives in antenatal care
  23. Percentage of women who considered their length of stay was about right
  24. Homeliness of delivery rooms
  25. Women's view of cleanliness of delivery and postnatal areas

Details of a few of the indicators

Below are details of four of the indicators, and how my Trust performed on each; other performance indicators are similarly structured around how well the Trust did in meeting a particular aspect of care in that area.

Indicator 1 -Women not receiving NICE recommended number of antenatal appointments

This is not based directly on whether a Trust is succeeding in meeting the recommendation of the NICE Guideline,1 but how it ranks against other Trusts in meeting this target.

For each Trust the results from the women's survey carried out in November 20071are used. It considers the percentage of women who made contact with the service prior to week 16 and had their baby after 37 weeks, who clearly did not get the number of appointments they should have. The Review acknowledges this is an underestimate due to lack of detail in the way the data was collected from the women.

The performance of all the Trust were used to generate a scoring system for this indicator based on what percentage of women clearly had not had the recommended number of antenatal appointments:

  • 5 points for less than 11.54% of women
  • 4 points for more than 11.54%, but less than 16.79%
  • 3 points for more than 16.79%, but less than 22.08%
  • 2 points for more than 22.08%, but less than 33%
  • 1 point for more than 33%

My Trust scored 4 points for this, as the survey showed that just under 16% of women were clearly not receiving the recommended number of appointments. This is a high score despite the fact that the Trust is failing to provide the appropriate number of antenatal appointments for about 1 in 6 women.

Indicator 2 - Appropriate use of caesarean section

The scoring system for this was as follows

  • 1 point regardless of performance
  • 1 point if it knew either the CS rates for primips, VBAC or ECV rates
  • 1 point if the CS rate for first time mothers did not exceed 24.38%
  • 1 point if the VBAC rate was above 31.53%
  • 1 point if the ECV rate for those with diagnosed with a breech baby after 36 weeks was above 27.09%

The 24.38% CS rate, 31.53% VBAC rate and 27.09% ECV rate are the median (middle rate, when the rates for all the trust are put in order) of the rates of the Trusts that could provide this data.

My local unit scored three, one automatic point, one point for a very slightly lower primip CS rate of 23.84%, and one for knowing some of these rates. It was unable to provide the ECV rate, and the VBAC rate is an unbelievable low of 10.32%. The report documents that 92 Trusts of the 148 were unable to provide full data for this indicator ; it really seems quite shocking that only 56 Trusts were even able to provide answers to these questions.

The World Health Organisation recommends that caesarean section rates should not rise above 10-15%, with the evidence being that rates higher than 10% confer no benefit; so scoring a point for a rate of under 24.38% for first time mothers seems a little meaningless in terms of good performance.

Indicator 4 - Maternal morbidity

This indicator focused solely on blood loss and the time scale in which women had any perineal stitching carried out; no other aspects of maternal morbidity were considered.

One point was given regardless of performance with an additional point for each of:

  • more than 84.43% of women who need perineal stitches receive them within an hour
  • more than 91.11% of women who need perineal stitches receive them within an hour
  • the trust was able to provide data about haemorrhage in excess of 2500ml
  • if less than 0.1939% of women experienced blood loss of more than 2500ml

The percentages used for the scoring system were obtained as follows: 25% of Trusts carried out 91.11% or more of stitching with in an hour, and 75% of Trusts carried out 84.43% of stitching within an hour. The median rate for haemorrhage in excess of 2500ml was 0.1939%.

My Trust only scored 2 showing there was room for improvement, as the suture rate was under 70% within an hour, one of the worse performances of all the Trusts, and the haemorrhage rate was over 0.37%, nearly twice the average rate.

However, I am left wondering what this really tells us about morbidity. These very specific aspects seem to fail to capture the concerns of most women. What about the actual rates of women needing stitching for example?

Indicator 17 - Staffing levels

One point was given regardless of performance and one point for each of:

  • more than 28.13 full time equivalent midwives per 1000 births
  • more than 34.90 full time equivalent midwives per 1000 births
  • 40 hours + consultant presence for units with <5000 births, or 60 hours+ consultant presence for units with >=5000 births
  • If all consultant units have at least 10 consultant anaesthetist programmed activities (which is to do with their availability to the labour ward).

The rates used for the scoring were obtained as follows: 25% of Trusts had less than 28.13 midwives per 1000 births, and 75% of Trusts had less than 34.90 midwives per 1000 births. The consultant obstetric and anaesthetist requirements were one of the few based on guidelines for numbers required, because this is one of the few areas where recommended staffing levels seem to be being met. It seems amazing that funding can be obtained when it comes to recommendations for these high level staff, but almost nothing else.

My local trust scored 4 on this indicator, losing one point as it only managed 31.87 full time equivalent midwives per 1000 births. One of the most important factors for women during pregnancy, birth and the postnatal period is midwifery support, so to see the number of midwives available brought down to just 2 points in only one of 25 indicators (less than 0.08 of the total score between 1 and 5) is very depressing.

Indicator 18 - Integration of support workers

One point was awarded regardless of performance and then:

  • 1 point when maternity support workers were at more than 5.12/1000 births
  • 2 points when maternity support workers were at more than 7.534/1000 births
  • 3 points when maternity support workers were at more than 10.68/1000 births
  • 1 point if maternity support workers were carrying out more than 12 different midwife roles.

The rates used for the scoring where obtained were as follows: 12.5% of Trusts had maternity support workers at less than 5.12/1000 births, half had them at more than 7.534/1000 and 12.5% had them at more than 10.68/1000. 12 was the median value across all the Trusts of the number roles maternity support workers were carrying out.

My Trust scored only 2, so should I be concerned or relieved about this? It would actually seem that its low score may be an indication that it has not been led into the erosion of midwifery in the same way that many other Trusts have.

Surely before we start making these aspects into targets we need to know whether they confer any advantage to bir thing women, or whether they are a marker of less midwifery support? How many roles is it appropriate to delegate to a maternity support worker? Should maternity support workers really be under taking tests, giving health advice, carrying out vital sign observation, carrying out antenatal classes, supporting women in labour, being theatre assistants, being the second person at home birth, etc? Are they simply a cheaper, less well trained version of a midwife? Is this indicator evidence of a Trust doing better or one that has hit crisis point with midwife numbers?

Conclusion

There seem to be three main failings of this Review

  • There are no meaningful standards against which each Trust is measured, with only a relative ranking between Trusts being used.
  • The measures of performance used are not well focused on what constitutes good maternity care.
  • The Review fails to separate out the different service providers within each Trust.

These shortcomings mean that the Review fails to assess whether the Trusts are providing what mothers and babies actually need from a maternity ser vice. It fails to provide a method that can be used in the future to measure whether our maternity services have improved or not. Nor does it provide information about how different service providers within a Trust are performing, so data from a midwifery unit or a group of community midwives providing a good home birth service will be swallowed up within the data for the whole Trust.

This Review seems to be another wasted opportunity to address the changes that are really needed to improve care for mothers and babies, and we are left to ask how much of tax payers' money has been wasted on this empty exercise. We have to hope that the data that has been generated and is to be made available on CD next month will be put to better use.

AIMS comment

To find out how your Trust did and what questions were asked to assess the other area of competency of Trusts, go to
http://www.nhssurveys.org/Filestore/CQC/2007_Maternity_services_survey_report.pdf
Or look at www.birthchoiceuk.com/HealthCareCommissionSurvey/T130.htm.

Where you choose to have your baby and whom you choose to look after you in labour will affect the type of birth you have and the care you receive. The BirthChoiceUK.com (www.birthchoiceuk.com) explains your options and gives information to help you make those choices.

You have the right to choose where to have your baby.
"All women should be involved in planning their own care with information, advice and support from professionals, including choosing the place they would like to give birth....." National Service Framework for Children,Young People and Maternity Services, pg 5. Department of Health 14/9/2004

References

  1. NICE (2003) Antenatal care, routine care for the healthy pregnant woman. Clinical Guideline 6. National Collaborating Centre for Women's and Children's Health, October 2003 www.nice.org.uk/cg6

Glossary

  1. CS - caesarean section
  2. Primip - primipara, first time mother
  3. VBAC - vaginal birth after caesarean
  4. ECV - External Cephalic Version, turning a breech baby to a head down position

Latest Content

Journal

« »

Reviews: Trust your Body, Trust you…

Complete list of book reviews on the AIMS website Trust your Body, Trust your Baby: How learning to listen changes everything Why Mothers' Medication Matters Trust your B…

Read more

Conference Report: Doula UK 2018 Co…

AIMS Journal, Vol 29, No 4 By Jo Dagustun Wow – what another great conference put on by the team at Doula UK! I was keen to get to this annual conference again, having be…

Read more

Babywearing Twins and Multiples: Ho…

AIMS Journal, 2017, Vol 29 No 4 Nicola Lawson shares her knowledge on carrying one - two - three babies! The idea of transporting two babies at once can be daunting, and…

Read more

Events

« »

MBRRACE-UK ‘Saving Lives, Improving…

To register your interest please email conference@npeu.ox.ac.uk or keep an eye on our website https://www.npeu.ox.ac.uk/mbrrace-uk/bookings . Earlybird bookings will open…

Read more

Midwifery Today Conference “Birthin…

17–21 October 2018 Further Details

Read more

AIMS AGM

AIMS AGM 2018 All members welcome! Please email secretary@aims.org.uk if you plan to attend to help us to judge numbers, or if you wish to send apologies 10 for 10.30 sta…

Read more

Latest Campaigns

« »

Press Release: Jeremy Hunt announce…

AIMS is delighted that the Government has recognised the importance to the safety of women and babies of the continuity of carer model of midwifery. Having a midwife that…

Read more

Dr. Ágnes Geréb, Hungarian Midwife…

Dr. Ágnes Geréb is a Hungarian obstetrician and midwife who has been under house arrest following her support for women outside of the obstetric system. March 2018: ENCA…

Read more

"Promoting professionalism, reformi…

AIMS submitted our response to this consultation on the 23 January 2018. A number of regulators, including the Care Quality Commission (CQC), the Professional Standards A…

Read more