A Review of the Education External Evaluation Services

Brian Donnelly Education Review Office

B COMPONENTS OF THE REVIEW PROCESS

10 Notification

The review
team notifies the school of its intention to undertake a review indicating
the type of review and the documentation required before and during
the on-site stage of the review. Depending on the type of review
a series of questions which will be explored by the review team are
noted. The initial letter is usually sent out at least two months
before the visit.

It is important
to establish that the visit is one part of the review. Documentation
sets the school in perspective for the reviewers who will then visit
the school for a number of days. This is followed by analysis
of the information gathered in order to make judgements and to form
opinions before reporting formally to the Board of Trustees within 15
or 20 working days (depending on the type of review) of the completion
of the on-site visit. The Board has a further 20 working days
to respond to the unconfirmed report and to discuss errors of
fact and to suggest amendments. The report is then confirmed and
becomes a public document.

11 Documents required

The Panel has
had numerous complaints about the workloads involved in preparing the
documentation required for a review. The Panel is convinced that,
without the documentation, it is not possible to determine whether or
not schools and centres have in place the policies and practices which
enable them to meet their obligations, respond to emergencies or deal
with conflict or provide the continuity for the school in the event
of changes in personnel at any level. Without the documentation
schools are not in a position to make informed decisions. The
Panel supports the use of the Board Declarations for its legal compliances
and believes that this will reduce the workload at the time of an Education
Review Office visit because it will have to be done annually.
It will also allow the Education Review Office to concentrate its resources
on educational effectiveness and the quality of outcomes to a greater
extent, with spot checking of the Board Declaration.

12 Meetings

An on-site
visit usually commences with a meeting of the senior management staff
and often members of the Board of Trustees. Review procedures
are clarified, issues are raised by the review team and the senior staff,
questions are asked and an overview of the school conveyed. Panel
members had the opportunity to join several of these sessions and found
them to be open, frank and informative.

At the conclusion
of the on-site visit an oral report is given to the Board and Principal,
although staff may also be involved. The oral report has come
to be regarded as an indication of the likely contents of the unconfirmed
report when the reality is that it is a means of initiating discussion
and providing feedback to those present before the team leaves the school.
It is an exit meeting for reviewers at the conclusion of the on-site
visit. The confusion over the differences between the oral and
written reports has lead to accusations of inconsistency and lack of
credibility. It is clear to the Panel that a great deal of effort
must go into clarifying the purpose of the oral report meeting.

13 Interviews

Interviewing
the Board of Trustees Chair and members, the Principal and senior managers,
heads of department or syndicate leaders, classroom teachers, and sometimes
students and their parents, is important in the process of gathering
information and drawing valid conclusions. It should be a regular
and routine part of the process to conduct focus group interviews with
staff, support staff, students and parents. If such interviews
were included in the review process, then alleged unverified individual
comment should not find its way into reports.

14 Classroom visits.

Classroom visits
are an integral part of the on-site work of the reviewers.
Along with the written material provided and other observations made,
such visits contribute to determining whether or not the curriculum
is being delivered effectively. Teachers provide the documentation
relating to planning, assessment of learning, records, student portfolios
and profiles to support what they are doing and what their students
are learning. Here again, confusion has arisen through lack of
appreciation of the purpose of classroom visits. The gap in perception
is very great. Principals and teachers have yet to fully comprehend
that the focus is on delivery of programmes rather than on individual
teachers
. If the Accountability Review is to be effectively
introduced, then the Education Review Office must do more to take the
mystery
out of the process and reviewers must be able to articulate
what they are observing and why. This requires the production
of a comprehensive manual covering procedures.

While teachers
are crucial to effective curriculum delivery their individual competence
is not being examined by reviewers. It is the task of the
Principal to evaluate teacher competence by using the Performance Management
System as the basis on which to advise the Board of Trustees on the
nature of professional development which may be required. Teachers
are responsible for classroom management, motivating their students
and monitoring student progress in a supportive school environment.
The reviewers will be asking what steps the school takes to support
the teacher, whether or not the principal provides professional leadership,
how often a teacher's classroom has been formally visited and if there
are good working relationships in the school. They can also investigate
the degree to which the school is providing the resources needed to
meet teachers' needs, and they can inquire, for example, about whether
the teacher has been trained in information technology and how the internal
communication system works.

The Panel concluded
that much of the feeling about to the Education Review Office is based
on lack of knowledge and understanding about a review's nature and
purpose in schools and centres as well as ineffective communication
from the Office. It is an issue which must be addressed as a matter
of urgency by all the parties involved. There is a dual responsibility
for information to be provided by the Office and to be understood by
people in schools and centres.

15 Timing, length and frequency of
visits

A number of
submissions were received which expressed concern that a 3-day visit
was an inadequate time to spend in a school or centre in order to make
informed and accurate judgements. What is at issue seems to be
a lack of understanding, as previously discussed, about the purpose
of the on-site work.

When an on-site
visit is scheduled and conducted, it is essential to the credibility
of the Education Review Office that reviewers adhere to the timetable
and give their undivided attention to the school or centre during
the time allocated.

Submissions
from the Early Childhood Education sector called for greater frequency
of reviews. They believe, and the Panel concurs, that a 4-year
interval is too long between reviews. Early Childhood educators
welcome reviewers and want the "spot" reviewing to continue, although
concern was expressed that the Ministry of Education had moved into
this area as well. (See Section 7 of Terms of Reference 4.)
There was also a call for greater attention to be given to curriculum
development and delivery, whether Te Whariki was being implemented and
the quality of adult-child interactions. The Panel is reluctant
to specify the frequency of reviews of early childhood centres but believes
it must be reduced to a 3-year interval as a minimum. Four years
could be the entire life of a child in a centre, which is too long,
particularly if there are risk factors which may go undetected.

The introduction
of Accountability Reviews should allow the Education Review Office to
rethink their scheduling of on-site visits. Where a school has
already established a high level of legal compliance with effective
curriculum delivery and reporting of student achievement, with a robust
self-review system in place, the frequency of review can be extended
to 4 years. This would allow more frequent reviews where there
are indicators which raise concerns about a school's performance
(e.g. a change of Principal, high staff turnover, a transient population,
poor student achievement, high truancy, having to co-opt a number of
Board of Trustees members or other risk factors). The panel believes
the Office should advise the School or Centre of the appropriate timing
of its next review in the confirmed report.

RECOMMENDATIONS:

    7. That
    the Board Declaration be used by the Education Review Office to provide
    assurance that Boards of Trustees are meeting their legal obligations;
    along with spot audit during on-site visits being used to ensure such
    compliance exists.
    8. That,
    in adopting Accountability Reviews, the Education Review Office concentrates
    on educational effectiveness and outcomes in its review processes.
    9. That
    the Education Review Office prepare and publish a manual for schools
    providing details of the protocols which will guide the Accountability
    Reviews.
    10. That
    the Education Review Office implement a communications strategy to ensure
    understanding of the criteria against which performance is being evaluated.
    11. That
    the review process should include focus group interviews with the Board
    of Trustees, the Principal and senior management staff,
    heads of departments or syndicate leaders, teaching staff, support staff,
    students and parents.
    12. That,
    once a school or centre is notified of a review, the timetable be adhered
    to and reviewers give their undivided attention in the school or centre
    during the time allocated.
    13. That
    spot reviews by the Education Review Office be retained for early childhood
    centres with the frequency being increased to at least 3 year intervals
    and, over time, to 2 year intervals.
    14. That
    the frequency of review for schools be retained at 3-4 years, with more
    frequent reviews where warranted by the indicators and with less frequent
    reviews being conducted where educational effectiveness has been assured.
    15. That
    the timing of the next review be included in the confirmed report.

C CONTRIBUTING TO IMPROVEMENT
& DELIVERY OF EDUCATION

Three issues
dominated the Panel's public consultation process and to a large extent,
the written and oral submissions. They were reporting, publication
and the provision of advice and guidance. If the external evaluation
services are to contribute effectively to improving the delivery of
education and to have an sustainable impact for good, then these issues
must be considered seriously.

Comments ranged
from 'Schools are paralysed by a bad report' to 'If the purpose
is to improve education then publishing reports has little to do with
it even though it seems parents welcome them'. The perceived
differences between the oral briefing and the written report has already
been noted and acknowledged in an Otago-Southland regional report to
the Chief Review Officer. Calls were made repeatedly not only
for all reports to be specific, blunt and constructive but also to include
positives about a school or centre. Claims were made that media
reports were sensationalised but this point was not borne out in examination
of the media clippings.

16 Reporting

The process
of reporting must be improved if school improvement is to follow.
The Panel is concerned about the differences between the oral briefing,
when this occurs, and the written report. Consideration was given
to requiring the written report to be the basis of the oral briefing,
but on reflection this would prevent an important part of the review
process from being completed, that is, the analysis of findings and
making judgements before preparation of the written report. On
balance, reviewers must endeavour not to mislead and to be entirely
objective in their dealings with Boards of Trustees and Principals during
the oral report at the end of an on-site visit.

One argument,
heavily supported, which the Panel found persuasive, was the need for
a realistic time-frame to be put in place to allow schools and centres
to respond to the recommendations and to develop an action plan to implement
the changes required to improve their effectiveness. Accordingly,
we are recommending changes to the period between receipt of the unconfirmed
and confirmed written reports so that the actions taken by the school
and centre can be included in the final report. Our proposal is
more in line with the Picot and Tomorrow's Schools plan and extends
the present time-frame for the confirmed report. We believe the
proposal will be seen as fair and reasonable and also will give greater
incentive to respond positively to the recommendations.

Once the unconfirmed
report is received, and any errors of fact in it are negotiated to a
satisfactory conclusion, then, a period of three months should elapse
to allow the school to develop an action plan to address the review
report recommendations. This would allow time for the Board to
seek advice (see page 50-51) and the Education Review Office to alert
the Ministry of Education. Plans developed would be incorporated
in the report which would be confirmed and become a document in the
public domain.

17 Dispute Resolution

Schools have
been informed that they can challenge the factual accuracy of a review
report but, in practice, changes are made very rarely. There is
no real appeal system in place for schools and centres. The only
recourse which schools and centres have when a dispute arises is to
the Courts and this is beyond the resources of almost all parties.

In the interests
of natural justice, the Panel concluded that, in the first instance,
there is an onus on the parties involved to find solutions where conflict
emerges between a school or centre and the Education Review Office over
matters of fact or the interpretation of them in the unconfirmed report.
This would probably involve the Regional Manager of the Educational
Review Office. If the purpose of the oral report was clearly understood
and reviewers were more direct in their presentation then such events
would be rare. The Education Review Office should make available
to a school or centre the evidence they hold in support of their statements
in the report, when requested to do so.

In the event
of a dispute being unresolved, the view of the Panel is that it is the
responsibility of the Education Review Office to resolve the matter
with the National Manager, Reporting Services, making a judgement on
the basis of evidence provided by the parties. This approach assumes
that this manager would have no part in reporting on individual schools
or centre. The final authority for dispute resolution must rest
with the Chief Review Officer. The key to reducing differences
of opinion and the potential for disputes is transparency. The
final element in the dispute resolution process is that all disputes
requiring intervention of the National Manager Reporting Services, should
be reported to the Advisory Council on Quality in Education of the Education
Review Office in the interests of transparency.

18 Parents as consumers / clients
of the Education Review Office

Assuring parents
as to the effectiveness of the education received by their children
is an important adjunct of the reporting process adopted by the Education
Review Office. It is noted that the Office for Standards in Education
(OFSTED) in the United Kingdom is now including a summary of its inspection
report covering the main findings, student achievement, the quality
of education provided, efficiency, student spiritual, moral and cultural
development and what the school should do next, as part of its reporting
process to schools. It is expected that the summary will be sent
by the school or centre to every family with a child at the school or
centre. The Panel was pleased to be informed that the Education
Review Office is intending to follow a similar procedure.

19 The school context

In the interests
of the school or centre and their communities, and also to ensure they
have ownership and input into a review report, the Panel believes that
the institution should provide a narrative description of its location
and the context in which it operates. Once the proposal for mandatory
inclusion of a 3-year Strategic Plan and an annual Statement of Performance
Indicators in Charters is implemented, then the narrative description
envisaged can be lifted straight from the Strategic Plan into the review
report. Further, the reviews will take place against the background
of the plan and the performance indicators. Schools and centres
have significant achievements between reviews and they ought to have
the opportunity to include these successes, and the issues currently
under review or development, in the review report.

20 Value added education

The social
and cultural environment of students is a factor which must be considered
in educational review but so, too, are teacher attitudes and expectations.
Gibbs (1997) made some acute observations when he said that,

    "when
    we invest more into understanding the thinking of teachers, and helping
    them to be more aware of their own thinking as they teach, we will begin
    to unlock many of the mysteries of what makes the difference between
    effective teachers and less effective teachers.' We want 'teachers
    who will approach their teaching with an eagerness and commitment that
    they not only can make a difference, but they do." (Education
    Review 19.9.97)

The Panel is
convinced that the barriers of socio-economic status and culture cannot
be an excuse for not making a difference and, indeed, Harker and Nash
(1996) show that some schools are significantly more effective than
others when School Certificate results in Mathematics, Science and English
are considered and that variability is substantially reduced when the
initial ability of students, their gender, SES and ethnicity as well
as school mix are taken into account. The value-added estimate
reveals a high level of success among many schools which are regarded
as not performing. As David Lange (1995) commented,

    "We never
    talk about success in terms of the numbers of children whose English
    vocabulary has expanded by 50 per cent in two years. Yet (many
    poor) schools would beat any other school in the country on that count."

It seems that
administrators and educators in New Zealand are not persuading teachers
to engage in, and commit to applying indicators which reveal student
learning and not just observer or teacher beliefs about them.
In his submission to the Panel, Professor Nuthall (University of Canterbury)
advocated that there is a substantial body of research to guide how
the quality of teaching services and student achievement should be evaluated.
In isolation from other factors, observation is not a good tool and
neither are checklists. Direct assessment is essential and related
to data-bases of normative data about what can or ought to be expected
of students in specific circumstances. It is not acceptable, as
many teachers have told the Panel, that their professional experience
tells them whether or not their students are making progress at
school.

It is against
this background that the Education Review Office work must be based,
not just on the questions, "What do you expect your students to achieve?"
and "What assessment practices are used to determine learning?"
but also on results of assessment, analysis of performance against expectation
and how the information is used to improve teaching and learning.
If teaching is to be a truly "research informed profession"
then a great deal of effort must be invested in improving teacher belief
in applying the indicators which will give them greater certainty about
student learning.

Education in
New Zealand is highly regarded by the international community for its
innovation and responsiveness as well as for its willingness to lead
and to take risks. Examples would include curriculum development,
self-management, direct resourcing and the qualifications framework.
One of the great strengths of New Zealand teachers is their independence,
and their ability to develop teaching strategies which suit them and
the children's learning style. Such teachers think for themselves
and reflect on their practices.

Within the
existing accountability framework, the Panel would welcome greater acknowledgement
of the diversity of approaches within the present system. Diversity
is celebrated within the early childhood sector where the different
approaches have developed collaborative mechanisms for developing and
implementing guidelines for improving the quality of education and care.
The prime example for this is Te Whariki, the Early Childhood Education
Curriculum statement. The same ought to be true for the compulsory
sector where there is a need for better dissemination of best practice
models.

21 Publication

The position
adopted by the Education Review Office is that, once reports are confirmed,
they are public property. Lists of confirmed reports are made
available at the end of each month to the media who are aware that the
reports are available on request. Some newspapers have a standing
order for the full report to be sent at the time of its release.

Boards of Trustees,
Principals and Centre Managers understand that accountability and transparency
go together and that the public have a right to know what the external
evaluator has found in reviewing a school or centre. There is
an acceptance that the Education Review Office is not responsible for
the media but there is also concern that media reports may sometimes
be less than accurate in the way the report is presented, especially
if there are a number of problems to be addressed. As was observed
earlier, the vast majority of media reports in newspapers have cast
schools and centres in a positive light with about ten percent getting
negative headlines. Most reports are condensed summaries of findings
and appear to be both fair and balanced. There are some clippings
which show that schools have acknowledged that the publication of the
report was a positive stimulus to addressing their problems and in turning
the schools into effective learning environments. The Panel readily
acknowledges that this is not always the case.

The Education
Review Office has no powers to enforce its recommendations. In
making its reports publicly available, it concedes that it is using
them as an incentive for action by the school or centre concerned.
Transparency is regarded by the Office as an instrument of accountability.
Its powers are limited to returning to the institution for a follow
up review to determine the extent of changes and to report to other
agencies on the findings in the expectation of action. The Panel believes
that its recommendation to extend the time-frame for action, with subsequent
inclusion of the implementation of the recommendations in the confirmed
report, will add an incentive and reduce the threat which can accompany
the early release of review reports.

Some schools
have developed a good relationship with their local media and are proactive
in informing local papers about their activities. The School Trustees
Association has acknowledged that managing publicity has become an issue
to which it is directing attention in its Boards of Trustees training.

22 Advice and guidance

As the impartial,
independent external evaluator of schools and centres, the Education
Review Office must be free from direct intervention functions.
The Education Review Office has been careful not to compromise its independence
by allowing its officers to give advice and guidance as part of their
review function. It can be argued that the review report is itself
an advice document in that it will recommend what needs to be done to
remedy non-compliance problems. However, the Education Review
Office in its Vision Statement (1996-97) states that, "it seeks to
be a source of useful guidance for schools and centres interested in
good practice and improvements to education." How this statement
is realised is a problem. The Education Review Office claims that
its evaluation reports are the vehicle for delivery of advice and guidance
and to a large extent they are, although uptake and dissemination is
not as widespread as the Panel expected.

It is claimed
by Boards of Trustees, Principals and Centre Managers that the reviewers
are among the few in the Education Sector who have an overview of practice
in the sector and they are looked to for advice and guidance.
Some reviewers, the Panel heard, will respond to requests orally but
most will not. The Panel has formed the view that the Education
Review Office should encourage reviewers to point to examples of best
practice where there is a need and a request is made to do so.

The availability
of advice, guidance and support appears to be variable and, in a country
where many schools and centres are isolated in small communities, the
absence of professional colleagues can cause acute loneliness.
Many teachers do not know how to access advice and guidance, peer support
is minimal and subject associations may not be as active as formerly.

The Panel acknowledges
that the gap is a very real one and believes there is an urgent
need for both proactive and reactive advice, guidance and support systems
to be put in place. As a first step, it is suggested that the
Education Review Office act as broker by inserting a page in its
reports to schools and centres providing information about the sources
of advice which are available in the district in which the school or
centre is located. This would include the Advisory Services at
Colleges of Education; the Early Childhood Development Unit; consultants
in the field of governance, management, curriculum, assessment, personnel,
finance, property and evaluation; possible resource personnel who would
give peer support and relevant Evaluation publications.

The suggestions
we have made would neither challenge the principle of self-management
nor the independence of the Education Review Office. They would
meet a demand from the professionals in the schools and centres who
say, "It would be nice to know that there is support there, when we
want it."

The position
of small and rural schools is of particular concern. School clusters
working together seem to the Panel to be a way of securing improvement
in educational opportunity. They would be able to produce solutions
to identified needs with the advantage of peer support. The consequence
would be an increase in confidence and reduced workload for those involved.
There are important issues. Some schools, principals and
teachers may be unwilling to participate or may feel that undue advantage
may be taken of their willingness to do so or it is believed it is the
Ministry of Education's responsibility to provide the support.
However, the Panel recommends that, wherever possible, clusters of schools
work together on the issues of concern to them. It may be necessary
for the Ministry of Education to provide facilitators to assist the
clusters.

We endorse
the practice of publishing Evaluation Reports which are based on best
practice on topics of relevance to schools and centres.

Wylie (1997)
identifies advisers as the primary sources of information and advice
on curriculum, assessment policy and advice and staff development.
Other major sources are other teachers in the school and curriculum
contracts. Consultations with leaders of the Advisory Services
in Dunedin, Christchurch and Hamilton revealed a variety of responses.
It was reported that, after an Effectiveness Review, a Principal often
calls in advisers to help to deal with the non-compliance issues.
In the Waikato, the service has been renamed the "Teacher Support
Service" aiming for school development, offering leadership training
and facilitating school development through exploring questions such
as, "Where are you at, where do you want to be, when do you want to
get there, how can we help you to get there, and how will you know when
you've arrived?" Initiatives such as this are very important
in moving through self-review to self-improvement.

The model to
which the Panel has responded positively has been developed by the Christchurch
College of Education Advisory Team and the Education Review Office.
A draft protocol is in place to ensure that each service has an accurate
understanding of the other's work and to allow the easy access and
exchange of knowledge and expertise from one service to the other.
In practice, it means the Advisory Services can be alerted where a review
highlights issues where the Service may need to work with a school or
centre. The Panel recommends that this initiative be followed
in other regions as a most productive way of involving the Advisory
Service in a proactive role.