Ofsted Short Inspections

7 March 2017

Commenting on a reliability study of short inspections, published by Ofsted, Kevin Courtney, General Secretary of the NUT, the largest teachers’ union, said:

“This is a thorough and honest report from Ofsted which seeks to examine the reliability of its short inspections for ‘good’ schools that were introduced in September 2015. As Ofsted itself points out, it was neither an evaluation of the validity of Ofsted inspections, nor was it a reliability indicator of full section 5 inspections.*

“School inspection is a high-stakes process with potentially extreme consequences for senior leaders, teachers and school communities. It is to be welcomed that Ofsted is committed to making the current school inspection process as robust and independent as possible and studies of this sort are a necessary part of that process.

“However given the limitation of this research the NUT concurs with Ofsted that further research is required in order to establish the reliability of inspection more broadly as well as investigating the overall validity of school inspection for determining school quality.

“It is frankly astonishing that reviews of reliability and validity have not been carried out previously. Many head teachers will tell you that it is the fear of Ofsted that is driving the huge overwork of teachers and in turn driving the teacher shortage.

The NUT believes that there is a better approach to school accountability in which the primary goal of school evaluation should be to help schools move forward through critical self-evaluation. Head teachers and teachers should be at the forefront of evaluating their school, with the inspectors' role being to scrutinise the self-evaluation rather than make judgements themselves without sufficient time and evidence.

“Effective evaluation should equip teachers with the know-how to evaluate the quality of learning in their classrooms so that they do not have to rely on an external view, whilst welcoming other perspectives because they can enhance and strengthen good practice.”

*Editor’s Notes

The report aimed to evaluate how frequently two inspectors independently conducting a short inspection of the same school on the same day agreed whether the school remained ‘good’ or whether they needed further evidence via a full section 5 inspection in order to reach a secure judgement. The study involved 24 primary schools that had been judged as ‘good’ at their previous inspection and which had between 250 and 500 pupils on roll. All the schools involved in the study had agreed to take part. The schools were given a ‘live’ inspection on the same day by a ‘lead’ and ‘methodology’ inspector who were expected to conduct their inspections independently.

While the report concludes that in 22 of the 24 completed inspections, inspectors agreed on their final judgement at the end of the short inspection, the report itself highlights a number of limitations of the study:

  • First, the sample size was just a third of that originally intended and, as Ofsted states, “the small sample size means there is limited external validity to these findings. It is someway short of the calculated sample required for statistical validity.” (paragraph 60, page 28-9)
  • Second, only four of those inspections in the sample were overseen by independent observers who were tasked with ensuring the inspectors did not confer during the inspection. And in two of these four inspections, as Ofsted admits, the observers were not fully independent as they were former HMI. (see paragraph 53, page 23)  Ofsted itself concludes that, “we cannot be completely sure that all of the inspectors in the test inspections arrived at their judgements independently.” (paragraph 86, page 36)
  • Third, the study does not seek to address whether the section 5 Ofsted framework was reliable since it has not itself been subject to a methodology test. (paragraph 99, page 39).
  • Fourth, there has been no analysis of the impact of Ofsted’s quality assurance work on the reliability of inspection outcomes. (paragraph 100, page 40).
  • In addition, the NUT believes that there are other aspects of the study which were flawed and may have influenced the overall findings. These include the fact that:
  • Both the lead and methodology inspector jointly attended the initial meeting with the head teacher where the logistics of the inspections were agreed, the head and senior leadership team explained their self-evaluation and key lines of enquiry were agreed. (paragraph 40, page 17) It is difficult to accept that the two inspectors would begin their independent inspections without some shared assumptions at the outset.
  • Second, the post-inspection interviews with the lead and methodology inspectors were conducted several months after the inspection took place. (paragraph 54, page 23) This is problematic in terms of being able to make a robust evaluation of the inspectors’ genuine experience of the process since these are highly likely to have been influenced by hindsight and possible discussion with colleagues so long after the event.
  • Third, one inspector indicated a feeling of nervousness when discussing with their fellow inspector who was going to reveal their independent judgement first. The desire to confirm one another’s findings is perhaps not surprising and while this was expressed by just one inspector, it is quite likely to have influenced the behaviour of others.  As the report notes: “It is unlikely similar views would be expressed if there had not been something important riding on the eventual independent outcomes of both inspectors.” (paragraph 84, page 35-6)
  • Fourth, the research was conducted by Ofsted itself. To ensure full confidence in the outcome, Ofsted should in future commission independent third party researchers for such an important exercise.
Teachers Building Society NQT mortgages