A Review of the Education External Evaluation ServicesBrian Donnelly Education Review Office
APPENDIX 3: SUMMARY OF WRITTEN & PUBLIC CONSULTATION
is intended to provide the essence of the main issues raised in the
written submissions and referred to at the oral consultations and meetings.
of Reference 1: Examine the nature and scope of the authority of the
Education Review Office and assess the degree to which such authority
impacts on and contributes to improvement in the
quality of educational outcomes.
Role of the Education Review Office, and its objectives
In their written
submission, ERO identified itself as the external education evaluation
body that "aims to provide evaluative reports to inform the decision
making of core stakeholders in education and to be a recognised professional
authority on education evaluation."
The key message
in submissions is that ERO should focus on the quality of education,
and educational effectiveness. NZEI states, "Any system of education
external evaluation should be aimed at improving the quality of education
received by students."
There was divergence
of view on what the ERO's objectives should be:
- "It would be
more worthwhile to schools if they looked at the overall picture of
what makes an 'effective school"
- Boards "were
under the impression that the purpose of the review was to determine
the effectiveness of the Board as a governing body"
of the minimum standards and regulations is the national role of the
Education Review Office"
- "I would like
to recommend that ERO be given greater responsibility for the evaluation
of teacher performance in the classroom"
- "We would recommend
review which aims at [school] improvement", and pertinently
- "Schools are
confused as to what exactly ERO is focusing on"
The oral consultations
- It is essential
to audit the schools and early childhood centres even if it is traumatic
and you always dread the auditor. ERO has an audit and not an enforcement
- ERO has reinforced
the idea that schools are responsible for themselves. They are
accountable to their public and to the taxpayer. Schools and communities
want and use ERO reports. There was, however a feeling that ERO
is there to serve the Minister, not the children. The school, students
and community should be ERO's clients. The Crown has an interest
in its effectiveness including the health and well being of the system.
- Reviews motivate
and advance school improvement but do not take account of lack of resources
or local government charges.
- Audits ensure
you meet regulations and no one can abdicate from this responsibility.
- ERO is involved
in monitoring teacher performance, identifying incompetent teachers
in its reports but ERO is not the employer so it has no role in talking
- ERO's role should
be to test the system and not the teacher.
- ERO involvement
with a parental complaint should only happen after a complaint has been
dealt with by the principal and board.
of the Review Office
Those who made
submissions in this area tended to focus their comments on two aspects:
- The key stakeholder
organisations valued the regular liaison with the Chief Review Officer
and her organisation. "Te Kohanga Reo National Trust acknowledge and
appreciate the relationship that has been developed over the years between
the National Trust and the Education Review Office".
- Individual schools
and centres regarded the comments made by the Chief Review Officer,
particularly in relation to teacher performance, as negative, unsubstantiated,
unhelpful and tactless. She is seen as a control agent.
the public meetings included:
- ERO has a credibility
and image problem and is seen as a threat e.g. "the dreaded ERO review".
- An ERO visit requires
a lot of energy. Very often schools feel they are working for
ERO to please them and this can lead to conflict and lack of respect.
- ERO is "a good
thing" - positive and beneficial. It is essential for the image
of ERO to be open as well as forthright and independent
- The workload of
the reviewers is very high.
Teaching and Learning
were mixed about whether ERO reviews resulted in improvements in teaching
- "Most schools
received favourable reports" but "did not find the recommendations
likely to contribute to improved learning for children."
for no improvement included:
- "The school
in my view gained very little from the experience and resources had
to be diverted from our main purpose - i.e. teaching - to prepare for
and deal with the visit"
- Stress of visit
on principal, board and school staff which had an effect on staff morale
- "ERO reports
are often difficult for schools to interpret"
are more likely to alter school practices when recommendations are made
by reviewers they respected and found to be credible"
there were major improvements resulting from the work of ERO.
- "ERO has already
made a difference to the standard of education in NZ and will continue
to do so".
reasons were given to support the belief that there were improvements:
- "The contents
of the report have formed part of the planning objectives for our school
and led to more focused planning"
- "The Education
Review Office has done more to raise the standard of education in New
Zealand than any other body in the last 30 years".
- "Schools have
improved policy making and increased the efficiency of administration
and management procedures"
- "Although our
last review was far from glowing it gave our new board a direction to
head in which to rectify our problems".
from those schools which are well advanced in using and advocating self
review appeared to be more positive in saying that ERO reviews led to
improvements in teaching and learning.
the submissions stated the need for external, independent audit.
Two basic reasons emerged.
- To facilitate
accountability of the entity to its local stakeholders.
Council maintains that an external evaluation and review system is an
integral part of validating professional accountability"
- To facilitate
Government in determining whether its policies are being delivered as
intended and are being effective. "They provide an important
accountability function regarding the spending of public money, affirming
the quality of learning and teaching and providing staff with an objective
viewpoint on their achievements."
voiced these concerns:
- There is a lack
of flexibility in the methodology. ERO needs greater understanding
of school context; to value cultural initiatives, the philosophy and
character of a school; its charter; the socio-economic status of the
community; and to take account of the system that creates the problems
that schools deal with and ERO judges.
- There should be
a holistic, single review once every three years
- In early childhood
review every 2 to 3 years is best with ERO continuing to come in without
- ERO picks up little
issues and not always the broad picture. Reviewers are not in
a school long enough to 'see under the paper'. A number
of small negatives overshadow the positives.
- Need continuity
of reviewers visiting a school with more reviewers spending more time
- It is difficult
to measure achievement in neat chunks, i.e. valued added. ERO is accountable
for the information it gathers and uses, a school should be able to
challenge and seek verification of a written view.
- Current reports
are user unfriendly, as are some reviewers.
- Differing opinions
were expressed on the variable time ERO spent with parents.
of Reference 2: Assess the methodology and frequency of ERO's current
external evaluation services, and how these can more effectively contribute
to improvement in the delivery of education by schools and early childhood
to reflect experience. Some comments related to the behaviours
and activities of reviewers.
- "At all times
I found their comments constructive, non-threatening and of considerable
assistance. The report itself was detailed and where recommendations
were made they were supported by clearly understandable reasons and
accurate recitations of fact."
- "The professionalism
of the officers reviewing this school has always been of the highest
a lack of professionalism.
- "Some reviewers
demonstrate a real lack of interpersonal skills and there seems to be
an almost obsessive desire to identify areas of weakness"
- "One of the
reviewers who came to the school had a very high-handed and intimidating
The key concern
appears to be reviewer credibility, their knowledge on aspects of the
curriculum, philosophy, sensitivity to school needs and whether they
have had recent teaching/principal or manager experience in the sector
that they are reviewing.
- "For the team
to be credible it requires: management expertise in the areas of boards
of trustees and principalship with up to date experience in Tomorrow's
Schools; curriculum expertise and current curriculum knowledge delivery;
legislative expertise; assessment expertise; learning experience."
- Teachers resent
being reviewed by people who are not specialists or ex principals in
their sector. There is a lack of trust in the professional judgement
of ERO reviewers reinforced by perceptions that some schools have hoodwinked
ERO into giving a better report than the school deserves.
- "Concern was
expressed about reviewers who have no background in early childhood
and/ or with no understanding of the philosophy of the centre being
to consistency, the public meetings provided the following views:
- There is an issue
of consistency across the country. For example, a cluster of 3
schools developed a child abuse policy. When ERO carried out assurance
audits, one school got a compliance, one got no mention and the third
was marked as unsatisfactory.
- Schools in the
same area have different experiences - teams focus on different issues
and priorities, terms of reference, focus, issues to be examined.
There are variations in the level of investigation and a lack of evenness
Two types of
workloads were drawn to the Panel's attention.
- The workload of
ERO staff. They are not resourced adequately for the desired frequency
of reviews and the time spent on site during a review.
is seriously under-resourced in terms of funding ... It is a credit
to the professionalism and genuine commitment of the management and
personnel of ERO that it is able to operate with reasonable success
under such stressful and constraining conditions."
- The additional
stress on school staff, particularly for teaching principals and small
rural schools, in preparing for an ERO visit and during the time the
review team is on site.
"ERO have exerted huge pressure on rural schools who have much more
limited human resources to meet the same requirements as urban schools.
... we have always had excellent ERO reports but it is only through
the staff in particular working very long hours and being extraordinarily
dedicated beyond the norm."
of great interest to all those who made submissions and who attended
public meetings. Comments included:
- Reports are regarded
as bland and non-specific, don't notice the positives and do not record
student and school achievements.
- Reports seem to
use the same format providing little sense of the school's ethos and
- It is difficult
to read ERO reports - full of jargon, often have no definitive evidence
to support statements, with opinion set out as fact, problems of sweeping
generalisations in reports eg, "some teachers", "most teachers".
- There are dramatic
differences between oral feedback and draft report. When positives are
reported orally and fail to appear in the written report the credibility
of the report and the reviewer is undermined. Frustration was
expressed in that the ERO oral report was excellent, the written report
very good and the newspaper report very damning.
- No dialogue with
ERO or option of discussing/giving further information with the unconfirmed
- There is a power
imbalance between ERO and the school here. There is no appeal
process re what is in the final confirmed report. "Let there
be no doubt that ERO do not alter school reviews nor do they apologise
when they are found to be subsequently wrong."
- There can be delays
between ERO visits and reports. Where a school has rectified a
problem they can still be pilloried for something that is now fixed.
- The suggestion
was made that the audience of the report must be clarified and perhaps
it was necessary to write multiple reports designed for different audiences
e.g. Boards, Principal, teachers, parents, media.
- ERO reports have
improved over time.
Publication of Review Reports
There was widespread
but reluctant agreement that ERO reports are public documents.
- "We agree that
the reports are public documents and should be published and made available
to the media in the interest of accountability"
There was recognition
by the public that ERO reports have a tremendous impact when printed
in the newspaper and negative reporting can be damaging.
- "We do not believe
it is productive to a school and its community to be sensationalised
by the media due to a perceived poor ERO report. The current media
use of ERO reports appears to focus on negative reports or aspects of
reports in ways which produce trauma for the school community concerned."
- Reports, "should
never be made public to be exploited in the press and other media.
This can only lead to further problems. The board of the school
is the democratic representation of the local community and they should
be informed and asked to correct faults."
- "All findings
and reports should be available to the board, staff and parents without
all the newspapers selective reporting that presently happens."
- Release to media
by school, not by ERO.
be available only on request from the school, that way the school has
an opportunity to put things in perspective and has prior knowledge
something will be published and not be subject to the sneak attack".
- "In relation
to the timing of confirmed reports it seems common sense that reports
should not be made public until a school has had time to comply with
ERO instructions/act on its recommendations."
- ERO should prepare
a separate summary report for the media, with a more detailed report
to the school "The ERO reports should be confidential documents
that are as helpful as possible to schools. A shorter (two paragraph)
summary report using a standardised format could be available for the
- The Commonwealth
Press Union however, representing many of the NZ journalists, however,
"wholeheartedly supports the current ERO practice of publicly releasing
school review reports for the following reasons: the release
of reports is in compliance with the intent of the Official Information
Act; the information is of considerable public interest; the level of
Crown investment in schools is high, which justifies a high level of
public involvement and information; the approach is in line with Tomorrow's
Schools in enhancing the performance of schools; this approach is likely
to lead to more informed parental choice."
- Some Principals
argued that journalists need to be educated as to what reports mean
and not used out of context and prepare balanced articles. It
was suggested that ERO should follow up negative media reports.
The press usually gives the school opportunity to comment but
often they do not give it, so an article may be unbalanced. "Schools
should be looking for skills on how to handle the media and how to handle
- "The media shackles
ERO too - perhaps that's why the reports are so insipid."
- Media could get
reports from schools, not direct from ERO.
- ERO reports could
be delayed 3-6 months before publication or not published.
were made on the national evaluation reports although more had positive
than negative comments. Comments from schools indicated they found
- "Booklets produced
by the Review Office such as Professional Leadership in Primary Schools
and Reporting Student Achievements are well worth reading. They
should be required reading for all aspiring or present principals."
- "ERO publications
are excellent vehicles for promoting best educational practice."
organisations, however, questioned the validity and reliability of the
research methods used to produce the findings.
- "Generally speaking,
ERO publications are too esoteric to be of any use to anybody except
politicians and educationalists and as already mentioned are not rigorously
researched documents and are often misconstrued."
There is an
unmet demand for advice, guidance and support for schools.
- "Staff would
like comprehensive advice and follow-up assistance to go with the requirements
provide advice, and the role of ERO in this area was debated.
also want advice and support to help them follow through on requirements
- "In our view,
this should not be the function of the Review Office. Their evaluative
role should be maintained and not compromised by also assuming a development
- "Reviewers need
to be able to give advice to teachers. They view many good things
happening in other schools - share this."
came through in submissions:
- "NZFKA considers
that the Education Review Office should not be involved in offering
guidance and advice ... the Office would: develop a conflict of interest
as both adviser and reviewer."
- There should be
a separate arm of ERO to give advice. "An additional arm of the ERO
should be created to assist governing bodies implement review recommendations."
- "Reviewers are
in a unique position to be able to provide quality advice, guidance
and assistance after being in the school for a period of time.
It is a waste of a valuable resource not to be able to utilise their
knowledge and expertise."
- ERO should direct
schools to sources of advice. There is a lack of knowledge about where
to get help for problems, especially principal or board problems faced
by the board. This is compounded by a perceived reduction in professional
contact with other schools.
- ERO should have
the power to direct advisors to approach schools to give support.
- ERO does not have
a function in advice and guidance.
- The Government
needs to give more professional support for schools. Concern came
through that access to advisory services was limited. The number
of school advisors needs to be increased and their role expanded.
There appears to be a need to formalise the relationship between ERO
and the Advisory Services. More advice also needs to be given
via the NZ School Trustees Association.
There was a
wide range of suggestions as to the format, processes, approach and
criteria suggested for reviews. Some of the repeated themes were
- There needs to
be more emphasis on self review for both schools and ERO.
- "Some schools
do good self review, ERO could just check the self review and suggest
whether it can stand or needs changing by spot checking the evidence."
less confident, stating:
- Self review has
only started coming through in the last two years. Self review should
lead to better information that schools and ERO can use. ERO is
a motivator for self review. Schools don't want an imposed system
from the outside.
- There was concern
expressed as to whether or not self review would be rigorous, valid
and reliable? Will it tell the public what schools are doing and
how well they are doing it? Self review is certainly the way things
must head in the future but there is not enough expertise, training
and skill to do this at present.
- ERO must declare
its intentions and provide clear definition about what it is looking
for. An audit is imperative but it must be professional, comprehensive,
acknowledge diversity, be user-friendly and draw attention to sources
of advice. A 3-year cycle is regarded as adequate so long as it
is adhered to. Having the assurance manual was good - saved time,
helped school to focus on problems. ERO should spend less time
looking at documents and more time in the classroom visit.
- The amount of
time the team is given to review the school is unreasonable. Assurance
audits could be done by checklist records by a school or centre.
ERO is involved in trying to do too many reviews. Schools can
check it all themselves, the procedural stuff doesn't need ERO.
- ERO does not acknowledge
anything which is not documented. "ERO only put back into a
report what we have told them - they don't tell us anything of use";
there is a sense that ERO don't audit everything they should.
- Schools do not
know what ERO expects - there is a lack of documentation as to the criteria
and standards used by ERO, the information required and how it is to
be used, the focus of the review, the processes used. Different
criteria appear to be being used depending on the ERO team e.g. the
definition of self review, the level of detailed assessment required
- Schools want a
set of national standardised measures for education outcomes yet at
the same time they want ERO to recognise the uniqueness of their school,
the student experiences, the overall picture and context.
- Some of the difficulties
for ERO arise from a lack of direction and guidelines from the Ministry
of Education, with ERO giving critical direction in the absence of the
Ministry of Education.
- Reviews should
expand to comment on quality achievements of the school and students,
school context, the holistic picture of the school, the school charter,
the needs of special education students and Maori students.
- "The external
evaluation systems should ... be required to make an assessment of student
performance against fundamental national and international benchmarks,
be required to include formal consideration of teacher, student and
parental satisfaction ... and report on significant factors as they
impact on the overall quality of education outcomes"
three main messages from the written submissions.
- In the early childhood
sector people wanted more frequent reviews, spending more time in centres.
Two years was the most common time frame mentioned.
- The school sector
wanted reviewers to spend longer time in schools when reviewing.
"ERO visits should be held over a period of a few weeks so they can
build up a relevant picture."
- Schools were generally
happy with a frequency of reviews every three years although some wanted
annual reviews and others only reviews every four or five years.
"Every three years is a good time frame."
public meetings suggests:
- Reviewers must
be up to date with practice, knowledgeable, have specialist current
curriculum knowledge, must have worked in the Tomorrow's Schools environment
and have experience of the new curriculum. ERO teams are professional
in behaviour and conduct but their level of expertise is questioned
i.e. knowledge, this influences their ability to review accurately.
There is a perception that ERO needs reviewers who are experienced in
the sector being reviewed - pre-school, primary and secondary.
Anything less reduces their credibility.
- Secondment of
principals to ERO teams is a good idea and they work well in a team
but they should not work in their own area as there is a conflict of
roles. However, reviewer salaries have not kept pace and are well
below the level of principals so ERO is not likely to attract experienced
and respected principals.
Partnership and Policing
- ERO needs a more
active, ongoing relationship with schools. Rarely do the same review
members come back in the next audit.
- Schools must provide
reviewers with a professional environment in which to work. The
Principal should be the reviewers' friend, accompanying the team.
- The empowered
schools concept is a myth - there is still a lot of central control
and schools are sandwiched between ERO and the Ministry of Education.
ERO and the Ministry do not talk to each other and need to. ERO needs
to work as a team with NZQA and the Ministry of Education.
- The question was
asked "Who is the ERO report for?" - the Minister, the Board, the
Principal, the parents - the client base needs to be identified.
- ERO should investigate
exemptions for home schooling.
- ERO should be
an avenue for parental complaints if the Board has given no satisfaction.
- ERO should expand
its focus to look at the impact of Government policies on schools, the
local community and issues out of the control of the school.
- An alternative
view is that ERO should not be handling issues beyond education.
A health and safety agency could be appointed and spot check for compliance
with non-education requirements.
- The functions
of each of the agencies - Ministry of Education , ERO, NZQA etc. need
to be clarified and the specific criteria against which the audit was
conducted e.g. ERO, ACC, OSH.
- The functions
of ERO should be contestable.
- To give schools
choice in who carries out reviews "a. Each education district
would have a number of reviewers to choose from ... b. These reviewers
would still apply the national education and administration guidelines
... c. The reviewers would be accredited by a section within the Ministry
of Reference 3: Recommend any procedural or legislative changes that
optimise the educational benefits and maximise the effectiveness of
the current usage of the ERO's resources.
- "There should
be a statutory requirement for a follow up meeting between ERO and the
school to discuss implementation of changes necessary to remedy any
failing or deficiencies identified in the report."
- In relation to
compliance issues in the early childhood sector, "tougher more
stringent requirements be enforced," including withdrawal of government
funding where there is failure to respond.
- Links to advice
and support "The need for an agency to follow up with professional
development in areas recommended."
- If there is a
poor report and no change, where does the community go for action?
ERO should be able to insist on the recommendations being followed up.
Time limits should be in place for ERO's recommendations to be acted
- If ERO is not
satisfied, then another agency can then come in and assist the school
to meet the needs as required by ERO. There is a need for advisors to
go into schools to give support and advice.
- Boards are aware
of the importance of professional development but its costs for remote
schools is prohibitive. Resourcing must recognise this. Boards
need to know where to go for assistance and sources of best practice.
- The leadership
of the Principal is crucial and may be the problem in some instances.
- There is no need
for ERO to have teeth. The Minister should act on ERO advice.
ERO should have more clout with the Ministry of Education re health
and safety concerns identified in recommendations. If there are compliance
issues, ERO should come back and check. Where non-compliance is
serious it should be possible to cut the funding.
- ERO should comment
on Board-Principal relationship. Much of the work of a Board is
delegated to a Principal and staff to carry out, including implementation
of ERO compliance and recommendations. It is unclear what should
be left to the Principal and what to the Board.
- Review teams may
not have the expertise or competence to guide but they should be able
to direct to sources which are appropriate.
of Reference 4: Examine current methodology for evaluating the impact
of policy and delivery of services of other state education agencies
on educational outcomes for students, and consider how this information
should be fed back into the policy process.
A lack of confidence
was expressed at public meetings in the differing interpretations of
the Ministry of Education and the Education Review Office on some issues
e.g. assessment. ERO appear to be imposing their interpretation
on schools, contrary to the self management school model. When
ERO uses a comply/not comply model for policies under development or
partially implemented different reviewers have different standards on
what constitutes compliance.
of Education and ERO have different expectations, with the school in
the middle. There is considerable concern about the differences
between advice received from the Ministry and advisory services and
ERO expectations. Sometimes the recommendations made in a report
are contrary to known good practice. There was a repeated call
for clear expectations from ERO, unambiguous language and confluence
between Ministry and ERO statements.