A Review of the Education External Evaluation Services

TERM OF REFERENCE 1

THE NATURE AND SCOPE OF THE EDUCATION REVIEW
OFFICE'S AUTHORITY

Ko te tamaiti
te putake o te kaupapa.

The child
is the heart of the matter:

1 The legislative background

The Education
Review Office derives its authority from Sections 323 - 328 of the Education
Act (1989) under the title Review of Educational Services.
Educational services include services provided by an organisation which
is owned or operated by the Crown or whose provision is funded by public
money or regulated by or under a statute. The Act sets out the
functions of the Chief Review Officer with respect to conducting reviews
of such organisations:

• At the direction
  of the Minister,
• Initiated independently
  by the Chief Review Officer,
• Administering reports
  to the Minister, and
• Giving the Minister
  assistance and advice.

The Chief Review
Officer is designated to appoint review officers who are obliged to
carry and present proof of identity. Reviewers are given powers
of entry to organisations, but not dwelling places, and they are able
to require any person in an applicable organisation to produce documents
and information relevant to the review and to inspect the work, meet
with and talk to any person in the organisation being reviewed.

The Education
Review Office is required to report on exemptions from enrolment s.21
and s.26 of the Education Act (1989), and to report on registered private
schools. (s.35A of the Education Act (1989). The Office also has
a statutory role in reporting on the performance of Boards of Trustees'
proposals to split or form joint Boards and on the enforcement of the
school Charter, at the request of the Secretary for Education (s.64;
s.112 and s.313 Education Act 1989).

The State Sector
Act (1998), s.77, requires the Office to report annually to Parliament
on the performance of the pre-tertiary sector in relation to equal employment
opportunities.

2 Interpreting the law

The Education
Review Office has to interpret its role arising from the legislation
and to take account of the probable intentions of the Crown in developing
the criteria for auditing education delivery, a requirement placed by
the Crown on schools and early education centres. Despite having
set out:

• The National Education
  Guidelines (NEGs) including the National Education Goals,
• The New Zealand
  Curriculum Framework which embraces the National Curriculum Statements,
  and
• The National Administration
  Guidelines (NAGs), (see Appendix 5)

the Crown has
not been specific regarding its expectations and this has created problems
for all the partners involved in education delivery. The Panel
resolved to make recommendations on this issue as part of its deliberations.

It appears
that, in the absence of the Crown's willingness to define general
benchmarks and standards, the responsibility for this would be assumed
by Boards of Trustees. However, they, too, in the main have either
abdicated, or simply are not aware, of this expectation. The result
has been the emergence of the Education Review Office as the minder
who has assumed the responsibility by default, although the criteria
and evaluation methodology which are used are not widely understood.
This has led to unwarranted but understandable criticism.

3 Compliance issues

There are 34
Acts and Regulations listed in the Office's Handbook of Contractual
Obligations and Undertakings (1993) (Appendix 6). The burden of
complying with this raft of legal requirements falls on the Boards of
Trustees and Early Childhood Centre Managers. They are important
to the provision of a safe environment, the health of the children,
effective property management and being a good employer.
The Panel welcomes the continuing development of the Board Declaration
in which compliance is recorded and against which an audit of aspects
of the compliance requirements can occur.

4 Who is the client?

A fundamental
issue has emerged as to; "Who is the Education Review Office's
client and to whom is the Office accountable?" The Act identifies
only one client and that is the Minister responsible for the Education
Review Office (s.325 Education Act, 1989). This means that the
Office's focus of attention lies in determining whether Government
policies are being delivered as intended and on their effectiveness.
Omitted from the Act is recognition of the role of the Office in facilitating
the accountability of schools and centres to their local communities.

The Education
Act (1989), s.100, details requirements and procedures for annual reporting
by Boards of Trustees. Implicit in this section of the Act is
the Boards' accountability to the local community through the Annual
General Meeting requirements.

Submissions,
hearings, and discussion at meetings with the Panel confirm that the
local communities look to the Education Review Office's evaluation,
audit and reporting processes to provide them with accountability and
assurance of quality educational delivery and outcomes. They do
not seem to consider the Annual General Meeting to be the mechanism
for accountability.

The significantly
increased number of requests for school reports (an average of 3 per
week in 1992 compared to 70 per week in 1997) from the Education Review
Office would support the views expressed above.

There is a
misunderstanding of the lines of accountability prescribed in the Act.
The Education Review Office uses its reviews of schools and centres
to inform the Minister and, in reporting to Boards of Trustees and Centre
Managers, is also acknowledging a responsibility to them. The
proper channel of accountability to the local community is through the
Board of Trustees' Annual Report. This is required to be given
at the Board's Annual General Meeting. The local community perceives
the Education Review Office reports as the accountability mechanism
but this is incorrect. The Ministry has a responsibility to remind
Boards of their obligations and to clarify the lines of accountability.

Administering
for Excellence (1988) and Tomorrow's Schools (1988) stated
that the Education Review Office would:

• Ensure that institutions
  are accountable for the government funds they spend and for meeting
  the objectives set out in their Charter,
• Be accountable
  through the Chief Executive to the Minister,
• Ensure that regular
  reviews will be a co-operative endeavour, aimed at helping boards to
  meet their objectives and review their own performance,
• Comment on the
  performance of other elements in the system - the Special Education
  Service, the Colleges Of Education supply of Advisory Services and the
  Ministry of Education's provision of policy advice and overseeing
  of policy implementation.

Picot's Task
Force (Administering for Excellence 1988) claimed that genuine
accountability involves three major elements:

• Clear and specific
  aims and objectives, expressed as outcomes,
• Control over the
  resources available to achieve those objectives, and
• Monitoring by an
  outside agency of how well the objectives are met.

Picot proposed,
and Tomorrow's Schools (1988) reinforced, that review reports
on the performance of the institutions would be public documents.

It is clear
to the Panel, as has been stated previously, that the lack of clear
aims and objectives expressed as outcomes has hampered the Office's
work. That schools do not have complete control over their resources
has not had the same effect.

The Panel has
no doubt that schools and centres are independent institutions and that
reviewers are entitled to form an objective, unbiased opinion about
them. The Office, moreover, is at liberty to determine its own
methodology, has access to the information it requires to do so and
is reviewing to internationally accepted standards as discussed in the
relevant literature.

The acceptance
by schools and centres of the need for accountability was reinforced
at all the consultation meetings and in the written statements received
by the Panel. However, accountability in New Zealand requires
that an entity accounts for its performance through the provision of
information to the Crown and it is this information which is subject
to audit assurance. This does not seem to be well understood by schools
and centres who have commented on what they consider to be a time consuming
business that involves an unnecessary amount of documentation which
has little if any bearing on educational outcomes for the students.
They believe the requirements are instead, "designed to meet reviewers'
insatiable desire for documents."

5 Compliance with the Public Finance
Act.

A significant
issue emerged in examining the Public Finance Act. Sections 41
and 42 set out the obligations of Crown Entities in preparing their
Annual Financial Statements and Statements of Intent which are to include
objectives, the nature and scope of activities to be undertaken, performance
targets against which the entity may be judged and accounting policies.
Currently, Boards of Trustees are exempt from the preparation of statements
of service performance required under s.41(2)(f) of the Public Finance
Act. This exemption is granted by the Minster of Finance under
s.41A of the Act and has been granted annually for the past few years.
This matter is re-visited under Term of Reference Three, as part of
a discussion on the way forward. While early childhood centres
are not Crown Entities in the Fifth Schedule of the Public Finance Act,
they should be encouraged to adopt the proposals recommended to improve
their Charters.

The Panel is
not satisfied that, in general, the Charter is a functional, operational
document at the present time and has resolved that schools and centres
should be required to develop Three Year Strategic Plans along with
annual Statements of Performance Indicators. This will achieve
these ends:

• Eliminating the
  misunderstanding about the nature and importance of accountability,
• Schools and centres
  will move through self-review to self-improvement, and
• Overcoming the
  difficulties associated with s.41(2)(f) of the Public Finance Act.

These proposed
developmental initiatives invite tightly focused reviews based on the
schools' and centres' own objectives, targets and expected outcomes
as envisaged by the Lough Committee (Today's Schools 1990).
Schools, in particular, will set out their expectations of the students'
behaviours and achievements and will be able to show the extent to which
they have achieved those expectations using their data-bases (of both
qualitative and quantitative data) and profile information to verify
and validate reported outcomes.

The Education
Review Office's principal client is the Crown as the owner and funder
of schools and funder of centres. The Crown must be assured
by schools and centres that its investment in education is delivering
quality educational outcomes. This will be achieved through accountability
processes to the local communities and the review reports of the Education
Review Office.

There is, therefore,
no need to amend the current legislation with respect to the functions
of the Chief Review Officer (s.325). There is, however, a very
compelling imperative to improve the Office's reporting process against
the Charter intentions of the schools and centres which focus on the
quality of learning and learning outcomes.

The Education
Review Office's functions are summarised by the Ministry of Education
as follows: (submission to the Panel)

• To review compliance
  with legal requirements,
• To review performance
  in discharging the obligations detailed in the Charter,
• To review students'
  achievements from quantitative data, and
• To review qualitative
  information about student achievement.

The Panel is of the unequivocal view that the primary focus of review has to be on students' achievements, supported by valid quantitative and qualitative data.

6 The parent community

The Education Review Office has a function in assuring the parent community that the quality of education received by their children is of the highest standard and that their schools and centres are safe places in which to learn. The Panel was informed that parents are requesting Education Review Office reports in ever increasing numbers and it endorses the intention of the Office to include an Executive Summary, specifically designed for parents, as part of review reports in the future.

RECOMMENDATIONS:

1. That the exemption under s.41 of the Public Finance Act from the preparation of statements of service performance (required under s.41 (2) (f) of the Act) be granted yearly for Boards of Trustees.

2. That,
in developing the protocols for Accountability Reviews, the focus of
attention include processes for evaluating the quantitative and qualitative
achievements of schools and centres.

B. ERO'S IMPACT ON IMPROVING EDUCATIONAL OUTCOMES

Schools should
be dynamic, interactive, social entities which are committed to learning
in the broadest sense. They thrive on positive feedback and recognition
of the professionalism of their staff and they look for success and
achievement for their students. They are not averse to criticism
but they expect it to lead to improving their practice and performance.
It is apparent from the submissions received that there is a widely
held belief that the Education Review Office often fails to recognise
success and the efforts being made, regards teachers with suspicion
and does not regularly reinforce what schools are striving to
achieve.

The Office's
determination, 'to provide evaluative reports which help to ensure
that every New Zealand child's educational experience enables him/her
to reach the highest possible level of individual cognitive development,
intellectual strength and social confidence,' is not disputed.
The problem lies in the image created in the pursuit of these goals.
Auditors are rarely going to be popular - there will always be a tension
as part of the process but, to be effective, the auditor needs the support
and co-operation of the sector it seeks to influence and improve.
Schools have stated that audit often leads to anxiety, complaint and
fault finding which can be humiliating. Some schools describe
a lack of co-operation and a climate of conflict and confrontation with
the audit experience. Schools want the Office to be an ally rather
than being perceived in a policing role. The Education Review
Office should seek to establish and build a culture of improvement
that leaves schools and centres inspired and pleased with their review
experience but in no doubt about what needs to be done next.

7 Critical friends

The Education
Review Office must be impartial and independent. Where it is critical,
its evidence must stand up to scrutiny. Schools and centres need
critical friends who diagnose difficulties and their possible causes
and set out recommendations requiring action:

'friends
who at times will listen and help sort out their thinking and make sound
decisions, who are not afraid to tell them when expectations for themselves
and others are too low and when their actions do not match their intentions.'
(Stoll and Fink 1996)

To achieve
the above, reviewers must be highly trained professionals, generating
confidence in their ability to evaluate, articulating clearly the criteria
against which they are reviewing, and reporting findings accurately
and honestly. Often the Panel has heard about inconsistency amongst
reviewers in one team and in and among regions; of reviewers acknowledging
their own inadequacy when teams in schools and centres have reviewers
without sector specialist experience. To rekindle confidence,
the training needs of reviewers must be identified and met, so they
are steeped in evaluation theory which is allied with a sound understanding
of children's learning processes and able to generate the confidence
in themselves and the review process in practice. To some extent
the limitations outlined can be attributed to a perception of the Education
Review Office being under-resourced and carrying unacceptably high workloads
at regional levels. These factors, however, cannot be regarded
as excuses or even explanations for the frequency with which concerns
about reviewer credibility and consistency were repeatedly expressed
across the country.

Not withstanding
these criticisms of the Education Review Office, a recent survey Wylie,
(1997) indicate that schools are largely satisfied with the accuracy
of their reports and found them positively reinforcing although they
did not expect the review to have great impact on the school, a perspective
which the Panel finds disturbing.

The consensus
among those who made submissions seems to be that the Education Review
Office is having an impact on the quality of educational delivery and
outcomes. It may be too early to judge the success of the review
process (OECD 1995) but, without doubt, reviews have made Boards more
aware of their responsibilities. In this sense, the Panel agrees
with a recent OECD statement that external evaluation appears to be
an effective stimulant to:

• Improved service
  management performance,
• Improved decision
  making by policy makers,
• Improved accountability,
  and
• Greater sensitivity
  to client's expectations.

(N.Z. Country
Paper - March 1995)

Newspaper clippings
taken from nine daily papers over the period November 1996 to August
1997 show that reports were published on 144 schools and centres.
Of these reports, 97 were favourable to very favourable, 34 were positive
but identified areas in need of attention while 13 media reports reflected
a poor Education Review Office review. Some schools or centres
were reported to have used a previously poor review to turn their school
around, with considerable success, while others felt their work was
confirmed by the report.

The impact
of the Education Review Office is intermittent and uneven but it would
be sustained through ongoing institutional self-review and refining
of the Performance Management System in schools and centres. As
described by the School Trustees Association, 'The Education Review
Office has been a driving force, encouraging schools in the process
of Self Review.'

The Panel has
drawn the following conclusions from the submissions:

• The Education Review
  Office has had a significant effect in ensuring that Boards are complying
  with their legal obligations.
• Schools have been
  motivated to organise their documentation and policies. Often,
  this has rapidly turned around poorly performing schools despite the
  workload involved.
• Parents seek access
  to the reports on their school's effectiveness, indicating acceptance
  of the report's worth.
• Teachers have been
  challenged to debate the process of review, its validity and effectiveness.
  Teachers remain sceptical of the Office's impact on improving curriculum
  delivery. They want the qualitative information collected by teachers
  to be recognised as valid as well as the more easily determined quantitative
  data. This debate on the nature of data, in itself, is evidence
  of the impact of the review process.
• Government departments
  (e.g. the Ministry of Education) have been obliged to respond to Review
  Office reports as demonstrated by the recent South Auckland initiatives
  and the School Support and AIMHI projects.
• The impact on the
  Crown is evidenced in the scrutiny of reports from the Education Review
  Office by the Education and Science Select Committee, the number of
  Parliamentary Questions and responses, and the solicited and unsolicited
  inputs from the Education Review Office into policy development from
  time to time.

8 Can the Education Review Office
contribute more to improving the quality of Educational Outcomes?

With the administrative
references now well embedded in the education system, the Panel considers
the time is appropriate to firmly position the focus of the Education
Review Office on curriculum delivery - teaching, learning and assessment.

The Panel is
convinced that the attitude and role of the Principal is critical to
the impact of a review in improving the quality of education outcomes.
The Principal's perceptions of the reviewers, and indeed the review
process itself, has a direct effect on the nature of their reception
in a school and the success of the review. If the Principal is
positive, then the review is likely to verify what is already known
and staff will be informed and positive about the presence of the review
team. If, however, the Principal is defensive, belligerent or
dismissive, then the tone and climate for the review and its findings
will be of questionable value. The Principal's sphere of influence
can also extend to the Board of Trustees whose vulnerability is perhaps
greater than others.

Accountability
is a two-way process and the relationship between the educational leadership
in the school and the evaluator has to be nurtured. The Education
Review Office must take every opportunity to improve its image with
the professionals in schools and centres through an effective public
relations campaign. This might include good communication through
newsletters and papers such as Education Review, the New Zealand Education
Gazette, Education Today and similar publications, seminars, workshops
and clinics to explain procedures and to develop a positive interface.
The understanding from such communication will be critical to the acceptance
and success of Accountability Reviews. Confusion, and, therefore,
the opportunity for criticism, must be avoided. It is incumbent
on Education Review Office Area Managers to maintain personal contact
with providers wherever this is possible. In addition, the Panel
believes that seconding practising principals or deputy-principals to
review teams would strengthen their curriculum expertise and credibility.

The Panel believes
the Education Review Office must clearly position itself as a key contributor
to educational improvement in our schools and centres.

RECOMMENDATIONS:

3. That greater
emphasis be given to training and retraining reviewers so they are skilled
in the process of evaluation and knowledgeable about the essential learning
areas, skills and attitudes set out in the New Zealand Curriculum Framework.

4. That the
composition of review teams reflect the interests of the institution
being reviewed.

5. That consideration
be given to seconding external reviewers (e.g. principals or deputy-principals)
on a regular basis for specialist input rather than where a shortage
occurs.

6. That the
Education Review Office allocate a significant percentage of its PR
resources to developing quality relationships with its clients - the
schools and centres.