A Review of the Education External Evaluation Services

Brian Donnelly Education Review Office

APPENDIX 3: SUMMARY OF WRITTEN & PUBLIC CONSULTATION

This summary
is intended to provide the essence of the main issues raised in the
written submissions and referred to at the oral consultations and meetings.

Term
of Reference 1: Examine the nature and scope of the authority of the
Education Review Office and assess the degree to which such authority
impacts on and contributes to improvement in the
quality of educational outcomes.

Audit,
Role of the Education Review Office, and its objectives

In their written
submission, ERO identified itself as the external education evaluation
body that "aims to provide evaluative reports to inform the decision
making of core stakeholders in education and to be a recognised professional
authority on education evaluation."

The key message
in submissions is that ERO should focus on the quality of education,
and educational effectiveness. NZEI states, "Any system of education
external evaluation should be aimed at improving the quality of education
received by students."

There was divergence
of view on what the ERO's objectives should be:

  • "It would be
    more worthwhile to schools if they looked at the overall picture of
    what makes an 'effective school"
  • Boards "were
    under the impression that the purpose of the review was to determine
    the effectiveness of the Board as a governing body"
  • "Monitoring
    of the minimum standards and regulations is the national role of the
    Education Review Office"
  • "I would like
    to recommend that ERO be given greater responsibility for the evaluation
    of teacher performance in the classroom"
  • "We would recommend
    review which aims at [school] improvement", and pertinently
  • "Schools are
    confused as to what exactly ERO is focusing on"

The oral consultations
confirmed that:

  • It is essential
    to audit the schools and early childhood centres even if it is traumatic
    and you always dread the auditor. ERO has an audit and not an enforcement
    role.
  • ERO has reinforced
    the idea that schools are responsible for themselves. They are
    accountable to their public and to the taxpayer. Schools and communities
    want and use ERO reports. There was, however a feeling that ERO
    is there to serve the Minister, not the children. The school, students
    and community should be ERO's clients. The Crown has an interest
    in its effectiveness including the health and well being of the system.
  • Reviews motivate
    and advance school improvement but do not take account of lack of resources
    or local government charges.
  • Audits ensure
    you meet regulations and no one can abdicate from this responsibility.
  • ERO is involved
    in monitoring teacher performance, identifying incompetent teachers
    in its reports but ERO is not the employer so it has no role in talking
    about competency.
  • ERO's role should
    be to test the system and not the teacher.
  • ERO involvement
    with a parental complaint should only happen after a complaint has been
    dealt with by the principal and board.

Image
of the Review Office

Those who made
submissions in this area tended to focus their comments on two aspects:

  • The key stakeholder
    organisations valued the regular liaison with the Chief Review Officer
    and her organisation. "Te Kohanga Reo National Trust acknowledge and
    appreciate the relationship that has been developed over the years between
    the National Trust and the Education Review Office".
  • Individual schools
    and centres regarded the comments made by the Chief Review Officer,
    particularly in relation to teacher performance, as negative, unsubstantiated,
    unhelpful and tactless. She is seen as a control agent.

Comments from
the public meetings included:

  • ERO has a credibility
    and image problem and is seen as a threat e.g. "the dreaded ERO review".
  • An ERO visit requires
    a lot of energy. Very often schools feel they are working for
    ERO to please them and this can lead to conflict and lack of respect.
  • ERO is "a good
    thing" - positive and beneficial. It is essential for the image
    of ERO to be open as well as forthright and independent
  • The workload of
    the reviewers is very high.

Improving
Teaching and Learning

Submissions
were mixed about whether ERO reviews resulted in improvements in teaching
and learning.

  • "Most schools
    received favourable reports" but "did not find the recommendations
    likely to contribute to improved learning for children."

Reasons cited
for no improvement included:

  • "The school
    in my view gained very little from the experience and resources had
    to be diverted from our main purpose - i.e. teaching - to prepare for
    and deal with the visit"
  • Stress of visit
    on principal, board and school staff which had an effect on staff morale
  • "ERO reports
    are often difficult for schools to interpret"
  • "Principals
    are more likely to alter school practices when recommendations are made
    by reviewers they respected and found to be credible"

Others believed
there were major improvements resulting from the work of ERO.

  • "ERO has already
    made a difference to the standard of education in NZ and will continue
    to do so".

The following
reasons were given to support the belief that there were improvements:

  • "The contents
    of the report have formed part of the planning objectives for our school
    and led to more focused planning"
  • "The Education
    Review Office has done more to raise the standard of education in New
    Zealand than any other body in the last 30 years".
  • "Schools have
    improved policy making and increased the efficiency of administration
    and management procedures"
  • "Although our
    last review was far from glowing it gave our new board a direction to
    head in which to rectify our problems".

Submissions
from those schools which are well advanced in using and advocating self
review appeared to be more positive in saying that ERO reviews led to
improvements in teaching and learning.

Accountability
and Responsiveness

Almost all
the submissions stated the need for external, independent audit.
Two basic reasons emerged.

  • To facilitate
    accountability of the entity to its local stakeholders.

"The Principals'
Council maintains that an external evaluation and review system is an
integral part of validating professional accountability"

  • To facilitate
    Government in determining whether its policies are being delivered as
    intended and are being effective. "They provide an important
    accountability function regarding the spending of public money, affirming
    the quality of learning and teaching and providing staff with an objective
    viewpoint on their achievements."

Public meetings
voiced these concerns:

  • There is a lack
    of flexibility in the methodology. ERO needs greater understanding
    of school context; to value cultural initiatives, the philosophy and
    character of a school; its charter; the socio-economic status of the
    community; and to take account of the system that creates the problems
    that schools deal with and ERO judges.
  • There should be
    a holistic, single review once every three years
  • In early childhood
    review every 2 to 3 years is best with ERO continuing to come in without
    notice.
  • ERO picks up little
    issues and not always the broad picture. Reviewers are not in
    a school long enough to 'see under the paper'. A number
    of small negatives overshadow the positives.
  • Need continuity
    of reviewers visiting a school with more reviewers spending more time
    in schools.
  • It is difficult
    to measure achievement in neat chunks, i.e. valued added. ERO is accountable
    for the information it gathers and uses, a school should be able to
    challenge and seek verification of a written view.
  • Current reports
    are user unfriendly, as are some reviewers.
  • Differing opinions
    were expressed on the variable time ERO spent with parents.

Term
of Reference 2: Assess the methodology and frequency of ERO's current
external evaluation services, and how these can more effectively contribute
to improvement in the delivery of education by schools and early childhood
centres.

Reviewers
as Professionals

Comments seem
to reflect experience. Some comments related to the behaviours
and activities of reviewers.

  • "At all times
    I found their comments constructive, non-threatening and of considerable
    assistance. The report itself was detailed and where recommendations
    were made they were supported by clearly understandable reasons and
    accurate recitations of fact."
  • "The professionalism
    of the officers reviewing this school has always been of the highest
    order"

Others perceived
a lack of professionalism.

  • "Some reviewers
    demonstrate a real lack of interpersonal skills and there seems to be
    an almost obsessive desire to identify areas of weakness"
  • "One of the
    reviewers who came to the school had a very high-handed and intimidating
    manner"

The key concern
appears to be reviewer credibility, their knowledge on aspects of the
curriculum, philosophy, sensitivity to school needs and whether they
have had recent teaching/principal or manager experience in the sector
that they are reviewing.

  • "For the team
    to be credible it requires: management expertise in the areas of boards
    of trustees and principalship with up to date experience in Tomorrow's
    Schools; curriculum expertise and current curriculum knowledge delivery;
    legislative expertise; assessment expertise; learning experience."
  • Teachers resent
    being reviewed by people who are not specialists or ex principals in
    their sector. There is a lack of trust in the professional judgement
    of ERO reviewers reinforced by perceptions that some schools have hoodwinked
    ERO into giving a better report than the school deserves.
  • "Concern was
    expressed about reviewers who have no background in early childhood
    and/ or with no understanding of the philosophy of the centre being
    reviewed" .

In relation
to consistency, the public meetings provided the following views:

  • There is an issue
    of consistency across the country. For example, a cluster of 3
    schools developed a child abuse policy. When ERO carried out assurance
    audits, one school got a compliance, one got no mention and the third
    was marked as unsatisfactory.
  • Schools in the
    same area have different experiences - teams focus on different issues
    and priorities, terms of reference, focus, issues to be examined.
    There are variations in the level of investigation and a lack of evenness
    of analysis.

Workloads Imposed

Two types of
workloads were drawn to the Panel's attention.

  • The workload of
    ERO staff. They are not resourced adequately for the desired frequency
    of reviews and the time spent on site during a review.

"The organisation
is seriously under-resourced in terms of funding ... It is a credit
to the professionalism and genuine commitment of the management and
personnel of ERO that it is able to operate with reasonable success
under such stressful and constraining conditions."

  • The additional
    stress on school staff, particularly for teaching principals and small
    rural schools, in preparing for an ERO visit and during the time the
    review team is on site.

    "ERO have exerted huge pressure on rural schools who have much more
    limited human resources to meet the same requirements as urban schools.
    ... we have always had excellent ERO reports but it is only through
    the staff in particular working very long hours and being extraordinarily
    dedicated beyond the norm."

Reporting

Reporting was
of great interest to all those who made submissions and who attended
public meetings. Comments included:

  • Reports are regarded
    as bland and non-specific, don't notice the positives and do not record
    student and school achievements.
  • Reports seem to
    use the same format providing little sense of the school's ethos and
    uniqueness.
  • It is difficult
    to read ERO reports - full of jargon, often have no definitive evidence
    to support statements, with opinion set out as fact, problems of sweeping
    generalisations in reports eg, "some teachers", "most teachers".
  • There are dramatic
    differences between oral feedback and draft report. When positives are
    reported orally and fail to appear in the written report the credibility
    of the report and the reviewer is undermined. Frustration was
    expressed in that the ERO oral report was excellent, the written report
    very good and the newspaper report very damning.
  • No dialogue with
    ERO or option of discussing/giving further information with the unconfirmed
    written report.
  • There is a power
    imbalance between ERO and the school here. There is no appeal
    process re what is in the final confirmed report. "Let there
    be no doubt that ERO do not alter school reviews nor do they apologise
    when they are found to be subsequently wrong."
  • There can be delays
    between ERO visits and reports. Where a school has rectified a
    problem they can still be pilloried for something that is now fixed.
  • The suggestion
    was made that the audience of the report must be clarified and perhaps
    it was necessary to write multiple reports designed for different audiences
    e.g. Boards, Principal, teachers, parents, media.
  • ERO reports have
    improved over time.

Publication of Review Reports

There was widespread
but reluctant agreement that ERO reports are public documents.

  • "We agree that
    the reports are public documents and should be published and made available
    to the media in the interest of accountability"

There was recognition
by the public that ERO reports have a tremendous impact when printed
in the newspaper and negative reporting can be damaging.

  • "We do not believe
    it is productive to a school and its community to be sensationalised
    by the media due to a perceived poor ERO report. The current media
    use of ERO reports appears to focus on negative reports or aspects of
    reports in ways which produce trauma for the school community concerned."
  • Reports, "should
    never be made public to be exploited in the press and other media.
    This can only lead to further problems. The board of the school
    is the democratic representation of the local community and they should
    be informed and asked to correct faults."
  • "All findings
    and reports should be available to the board, staff and parents without
    all the newspapers selective reporting that presently happens."
  • Release to media
    by school, not by ERO.

"They should
be available only on request from the school, that way the school has
an opportunity to put things in perspective and has prior knowledge
something will be published and not be subject to the sneak attack".

  • "In relation
    to the timing of confirmed reports it seems common sense that reports
    should not be made public until a school has had time to comply with
    ERO instructions/act on its recommendations."
  • ERO should prepare
    a separate summary report for the media, with a more detailed report
    to the school "The ERO reports should be confidential documents
    that are as helpful as possible to schools. A shorter (two paragraph)
    summary report using a standardised format could be available for the
    media".'
  • The Commonwealth
    Press Union however, representing many of the NZ journalists, however,
    "wholeheartedly supports the current ERO practice of publicly releasing
    school review reports for the following reasons: the release
    of reports is in compliance with the intent of the Official Information
    Act; the information is of considerable public interest; the level of
    Crown investment in schools is high, which justifies a high level of
    public involvement and information; the approach is in line with Tomorrow's
    Schools in enhancing the performance of schools; this approach is likely
    to lead to more informed parental choice."
  • Some Principals
    argued that journalists need to be educated as to what reports mean
    and not used out of context and prepare balanced articles. It
    was suggested that ERO should follow up negative media reports.
    The press usually gives the school opportunity to comment but
    often they do not give it, so an article may be unbalanced. "Schools
    should be looking for skills on how to handle the media and how to handle
    the reviewer."
  • "The media shackles
    ERO too - perhaps that's why the reports are so insipid."
  • Media could get
    reports from schools, not direct from ERO.
  • ERO reports could
    be delayed 3-6 months before publication or not published.

Evaluation
Reports

Few comments
were made on the national evaluation reports although more had positive
than negative comments. Comments from schools indicated they found
them useful.

  • "Booklets produced
    by the Review Office such as Professional Leadership in Primary Schools
    and Reporting Student Achievements are well worth reading. They
    should be required reading for all aspiring or present principals."
  • "ERO publications
    are excellent vehicles for promoting best educational practice."

Some national
organisations, however, questioned the validity and reliability of the
research methods used to produce the findings.

  • "Generally speaking,
    ERO publications are too esoteric to be of any use to anybody except
    politicians and educationalists and as already mentioned are not rigorously
    researched documents and are often misconstrued."

Advice
and Guidance

There is an
unmet demand for advice, guidance and support for schools.

  • "Staff would
    like comprehensive advice and follow-up assistance to go with the requirements
    and recommendations."

Who should
provide advice, and the role of ERO in this area was debated.

  • "Principals
    also want advice and support to help them follow through on requirements
    and recommendations."
  • "In our view,
    this should not be the function of the Review Office. Their evaluative
    role should be maintained and not compromised by also assuming a development
    role."
  • "Reviewers need
    to be able to give advice to teachers. They view many good things
    happening in other schools - share this."

Several themes
came through in submissions:

  • "NZFKA considers
    that the Education Review Office should not be involved in offering
    guidance and advice ... the Office would: develop a conflict of interest
    as both adviser and reviewer."
  • There should be
    a separate arm of ERO to give advice. "An additional arm of the ERO
    should be created to assist governing bodies implement review recommendations."
  • "Reviewers are
    in a unique position to be able to provide quality advice, guidance
    and assistance after being in the school for a period of time.
    It is a waste of a valuable resource not to be able to utilise their
    knowledge and expertise."
  • ERO should direct
    schools to sources of advice. There is a lack of knowledge about where
    to get help for problems, especially principal or board problems faced
    by the board. This is compounded by a perceived reduction in professional
    contact with other schools.
  • ERO should have
    the power to direct advisors to approach schools to give support.
  • ERO does not have
    a function in advice and guidance.
  • The Government
    needs to give more professional support for schools. Concern came
    through that access to advisory services was limited. The number
    of school advisors needs to be increased and their role expanded.
    There appears to be a need to formalise the relationship between ERO
    and the Advisory Services. More advice also needs to be given
    via the NZ School Trustees Association.

Conducting
reviews

There was a
wide range of suggestions as to the format, processes, approach and
criteria suggested for reviews. Some of the repeated themes were
as follows:

Self
Review

  • There needs to
    be more emphasis on self review for both schools and ERO.
  • "Some schools
    do good self review, ERO could just check the self review and suggest
    whether it can stand or needs changing by spot checking the evidence."

Others were
less confident, stating:

  • Self review has
    only started coming through in the last two years. Self review should
    lead to better information that schools and ERO can use. ERO is
    a motivator for self review. Schools don't want an imposed system
    from the outside.
  • There was concern
    expressed as to whether or not self review would be rigorous, valid
    and reliable? Will it tell the public what schools are doing and
    how well they are doing it? Self review is certainly the way things
    must head in the future but there is not enough expertise, training
    and skill to do this at present.

Criteria

  • ERO must declare
    its intentions and provide clear definition about what it is looking
    for. An audit is imperative but it must be professional, comprehensive,
    acknowledge diversity, be user-friendly and draw attention to sources
    of advice. A 3-year cycle is regarded as adequate so long as it
    is adhered to. Having the assurance manual was good - saved time,
    helped school to focus on problems. ERO should spend less time
    looking at documents and more time in the classroom visit.
  • The amount of
    time the team is given to review the school is unreasonable. Assurance
    audits could be done by checklist records by a school or centre.
    ERO is involved in trying to do too many reviews. Schools can
    check it all themselves, the procedural stuff doesn't need ERO.
  • ERO does not acknowledge
    anything which is not documented. "ERO only put back into a
    report what we have told them - they don't tell us anything of use";
    there is a sense that ERO don't audit everything they should.
  • Schools do not
    know what ERO expects - there is a lack of documentation as to the criteria
    and standards used by ERO, the information required and how it is to
    be used, the focus of the review, the processes used. Different
    criteria appear to be being used depending on the ERO team e.g. the
    definition of self review, the level of detailed assessment required
    by ERO.
  • Schools want a
    set of national standardised measures for education outcomes yet at
    the same time they want ERO to recognise the uniqueness of their school,
    the student experiences, the overall picture and context.
  • Some of the difficulties
    for ERO arise from a lack of direction and guidelines from the Ministry
    of Education, with ERO giving critical direction in the absence of the
    Ministry of Education.
  • Reviews should
    expand to comment on quality achievements of the school and students,
    school context, the holistic picture of the school, the school charter,
    the needs of special education students and Maori students.
  • "The external
    evaluation systems should ... be required to make an assessment of student
    performance against fundamental national and international benchmarks,
    be required to include formal consideration of teacher, student and
    parental satisfaction ... and report on significant factors as they
    impact on the overall quality of education outcomes"

Timing
of Reviews

There were
three main messages from the written submissions.

  • In the early childhood
    sector people wanted more frequent reviews, spending more time in centres.
    Two years was the most common time frame mentioned.
  • The school sector
    wanted reviewers to spend longer time in schools when reviewing.
    "ERO visits should be held over a period of a few weeks so they can
    build up a relevant picture."
  • Schools were generally
    happy with a frequency of reviews every three years although some wanted
    annual reviews and others only reviews every four or five years.
    "Every three years is a good time frame."

Composition
of Teams

Feedback from
public meetings suggests:

  • Reviewers must
    be up to date with practice, knowledgeable, have specialist current
    curriculum knowledge, must have worked in the Tomorrow's Schools environment
    and have experience of the new curriculum. ERO teams are professional
    in behaviour and conduct but their level of expertise is questioned
    i.e. knowledge, this influences their ability to review accurately.
    There is a perception that ERO needs reviewers who are experienced in
    the sector being reviewed - pre-school, primary and secondary.
    Anything less reduces their credibility.
  • Secondment of
    principals to ERO teams is a good idea and they work well in a team
    but they should not work in their own area as there is a conflict of
    roles. However, reviewer salaries have not kept pace and are well
    below the level of principals so ERO is not likely to attract experienced
    and respected principals.

Relationships,
Partnership and Policing

  • ERO needs a more
    active, ongoing relationship with schools. Rarely do the same review
    members come back in the next audit.
  • Schools must provide
    reviewers with a professional environment in which to work. The
    Principal should be the reviewers' friend, accompanying the team.
  • The empowered
    schools concept is a myth - there is still a lot of central control
    and schools are sandwiched between ERO and the Ministry of Education.
    ERO and the Ministry do not talk to each other and need to. ERO needs
    to work as a team with NZQA and the Ministry of Education.
  • The question was
    asked "Who is the ERO report for?" - the Minister, the Board, the
    Principal, the parents - the client base needs to be identified.
  • ERO should investigate
    exemptions for home schooling.
  • ERO should be
    an avenue for parental complaints if the Board has given no satisfaction.
  • ERO should expand
    its focus to look at the impact of Government policies on schools, the
    local community and issues out of the control of the school.
  • An alternative
    view is that ERO should not be handling issues beyond education.
    A health and safety agency could be appointed and spot check for compliance
    with non-education requirements.
  • The functions
    of each of the agencies - Ministry of Education , ERO, NZQA etc. need
    to be clarified and the specific criteria against which the audit was
    conducted e.g. ERO, ACC, OSH.
  • The functions
    of ERO should be contestable.
  • To give schools
    choice in who carries out reviews "a. Each education district
    would have a number of reviewers to choose from ... b. These reviewers
    would still apply the national education and administration guidelines
    ... c. The reviewers would be accredited by a section within the Ministry
    of Education."

Term
of Reference 3: Recommend any procedural or legislative changes that
optimise the educational benefits and maximise the effectiveness of
the current usage of the ERO's resources.

Post
review action

  • "There should
    be a statutory requirement for a follow up meeting between ERO and the
    school to discuss implementation of changes necessary to remedy any
    failing or deficiencies identified in the report."
  • In relation to
    compliance issues in the early childhood sector, "tougher more
    stringent requirements be enforced," including withdrawal of government
    funding where there is failure to respond.
  • Links to advice
    and support "The need for an agency to follow up with professional
    development in areas recommended."
  • If there is a
    poor report and no change, where does the community go for action?
    ERO should be able to insist on the recommendations being followed up.
    Time limits should be in place for ERO's recommendations to be acted
    upon.
  • If ERO is not
    satisfied, then another agency can then come in and assist the school
    to meet the needs as required by ERO. There is a need for advisors to
    go into schools to give support and advice.
  • Boards are aware
    of the importance of professional development but its costs for remote
    schools is prohibitive. Resourcing must recognise this. Boards
    need to know where to go for assistance and sources of best practice.
  • The leadership
    of the Principal is crucial and may be the problem in some instances.
  • There is no need
    for ERO to have teeth. The Minister should act on ERO advice.
    ERO should have more clout with the Ministry of Education re health
    and safety concerns identified in recommendations. If there are compliance
    issues, ERO should come back and check. Where non-compliance is
    serious it should be possible to cut the funding.
  • ERO should comment
    on Board-Principal relationship. Much of the work of a Board is
    delegated to a Principal and staff to carry out, including implementation
    of ERO compliance and recommendations. It is unclear what should
    be left to the Principal and what to the Board.
  • Review teams may
    not have the expertise or competence to guide but they should be able
    to direct to sources which are appropriate.

Term
of Reference 4: Examine current methodology for evaluating the impact
of policy and delivery of services of other state education agencies
on educational outcomes for students, and consider how this information
should be fed back into the policy process.

Interpretation

A lack of confidence
was expressed at public meetings in the differing interpretations of
the Ministry of Education and the Education Review Office on some issues
e.g. assessment. ERO appear to be imposing their interpretation
on schools, contrary to the self management school model. When
ERO uses a comply/not comply model for policies under development or
partially implemented different reviewers have different standards on
what constitutes compliance.

The Ministry
of Education and ERO have different expectations, with the school in
the middle. There is considerable concern about the differences
between advice received from the Ministry and advisory services and
ERO expectations. Sometimes the recommendations made in a report
are contrary to known good practice. There was a repeated call
for clear expectations from ERO, unambiguous language and confluence
between Ministry and ERO statements.