A Review of the Education External Evaluation Services

Brian Donnelly Education Review Office

23 Schools Support Project

The Ministry
of Education appropriates $23.8M annually to fund contracts for in-service
support. The priority areas for 1997-98 are: support for school
management, support for curriculum and support for Maori Education.
In addition the Schools Support Project is in place where schools are
found to be "at risk". Boards of Trustees, Principals or community
groups may signal serious problems to the Ministry to trigger action.
The Education Review Office or local office of the Ministry may also
seek action. The Schools Support Project has worked exceedingly
well where it has been used, but schools regard it as a last resort
and only to be utilised where dysfunction is apparent.

The Panel came
to the view that a hierarchy of responses are called for.

  1. Providing that the
    review report contains sources of advice and guidance, most Boards of
    Trustees and Principals will invoke self-help by choosing the most appropriate
    source for their situation.
  2. Clusters of schools
    should be encouraged to work together to avoid reinventing the wheel
    in school or centre improvement and to provide collaborative support
    in dealing with problems.
  3. Protocols should
    be developed between the Advisory Services and the Regional Education
    Review Offices or other appropriate agencies to provide alert systems
    where assistance is needed.
  4. A web-site should
    be developed by the Ministry of Education which identifies sources of
    best practice, advice and guidance and a chat service. The Ministry
    of Education should also consider establishing an 0800 help-line
    to deal with issues as they arise.
  5. Consultants could
    be identified who offer a range of services which schools might use.
  6. School Support Services
    should continue to be available for long-standing problems.

The Panel has
noted that the recent Green Paper on Teacher Education proposes to allocate
a significantly greater share of the professional development funding
directly to schools through their operations grants. There are
some advantages in such a move as long as rural and Maori communities
are serviced and curriculum contracts are available to ensure coverage
of the country and successful implementation of the New Zealand Curriculum
initiatives. Commenting on the contestability of advisory services
is outside the terms of reference of this Panel. Nevertheless
it is bound up with the provision on Advice and Guidance on which we
have commented.

RECOMMENDATIONS:

    16. That,
    once the unconfirmed report is received, commented on and points of
    dispute negotiated, then a period of three months should elapse during
    which the school or centre would develop, implement and report on its
    action plan in response to the recommendations for action which are
    in the review report. These actions would be incorporated in the
    confirmed report which would then become a document in the public domain.
    17. That
    the Chief Review Officer has responsibility to ensure that a dispute
    resolution process is in place and effectively communicated to schools
    and centres to cover disputes which arise over statements in the unconfirmed
    reports.
    18. That,
    when unresolved issues in the report are subject to dispute, the report
    will remain unconfirmed until the National Manager Reporting Services
    has made a decision.
    19. That
    the dispute resolution process should involve negotiation between the
    parties, involvement of the Regional Manager, availability of evidence
    collected by reviewers to support their findings and, where unresolved,
    the National Manager Reporting Services would examine the evidence and
    make a judgement. The ultimate authority would rest with the Chief
    Review Officer.
    20. That
    all disputes which require intervention from the National Reporting
    Manager would be reported to the Advisory Council on Quality in Education
    of the Education Review Office.
    21. That
    the review report include a narrative description provided by the school
    or centre on its location and the context in which it operates, a list
    of significant achievements since the last review and the issues under
    consideration within the institution.
    22. That
    the Education Review Office proceed with its intention to produce a
    parent friendly summary of its review findings for schools to send to
    all families with children at the school or centre.
    23. That
    the Ministry of Education initiate further research and in-service training
    for teachers in the craft of assessment and its application.
    24. That
    the policy of releasing the results of reviews to the schools, centres
    and the media be continued.
    25. That
    schools and centres should develop strategies for media management.
    26. That
    the Education Review Office personnel working in the field are able
    to point to examples of best practice where there is an identified need
    and request for information.
    27. That,
    as a normal part of its reporting procedure, the Education Review Office
    include a section outlining sources of advice and guidance available
    in the district and from which schools and centres would be able to
    choose the most appropriate sources.
    28. That
    the Education Review Office continue to publish evaluation reports based
    on best practice and which are relevant to the problems being encountered
    by schools and centres.
    29. That
    protocols be developed in each Education Review Office region with local
    advisory services to facilitate easy access to advice, guidance and
    support for schools and centres.
    30. That
    the Ministry of Education establish a range of actions to assist schools
    where action is required to improve the management and delivery of education.

TERM OF REFERENCE 3

Managing
with confidence in an age of uncertainty

A PROCEDURAL OR LEGISLATIVE
CHANGES TO OPTIMISE EDUCATIONAL BENEFITS.

1 The Education Act (1989)

The Education
Act s.325 and the State Sector Act s.32 are explicit in stating that
the Chief Review Officer is accountable to the Minister responsible
for the Education Review Office. This Minister can direct the
Chief Review Officer to conduct reviews of the applicable organisations,
in this case schools or centres providing an educational service.
The Chief Review Officer must report and give advice and assistance
to the Minister.

The legislation
is enabling in that it does not specify the approach to be taken by
the Chief Review Officer in carrying out the duties. The Departmental
Forecast Report (1997-98) of the Education Review Office identifies
its three medium term intentions as:

  • To add value to
    the quality of New Zealand Education,
  • To increase stakeholder
    satisfaction, and
  • To maintain recognised
    professional autonomy.

The stakeholders
are identified as the Minister responsible for the Education Review
Office, Boards of Trustees, Principals and teachers, parents and others
with a significant interest in education. Because of the decentralised
approach which has been adopted, the Chief Review Officer has no power
to intervene in the decision making of a Board of Trustees. The
Education Review Office can review individual providers and advise the
Minister and Ministry of its findings. In reporting to the
stakeholders, it considers it is fulfilling its function to review and
inform. The Panel is firmly of the view that the Education Review
Office cannot be mandated to advise or intervene without compromising
its independence. Given this premise the Panel considers that
the Education Act provides a satisfactory framework in which the Education
Review Office can operate.

It can be argued
that the Education Review Office facilitates accountability to the Crown,
as the owner and/or funder of the schools and centres which it reviews,
through its reports to schools and centres, evaluation reports, and
its annual report to Parliament. The Crown Entities (the schools
and the centres) provide the information so that the Education Review
Office can report to the Crown. The problem is the lack of specification
against which reviews are to occur. At present, it is the Charter,
the Education Review Office's interpretation of the law and their
unspecified indicators of performance and standards which are the basis
of the review process. While this certainly justifies the Education
Review Office allocating 90 percent of its resources to information
gathering from the schools and centres, the Panel is concerned that
schools and centres remain confused and uncertain about what indicators
the Education Review Office is using to make its judgements.

2 Other issues

Several commentators
have argued that the original intention of Picot and the Tomorrow's
Schools policies was that the Education Review Office would comment
on,

    "the
    Special Education Service's supply of services to institutions, Teachers'
    Colleges supply of general advisory services to institutions and the
    Ministry of Education's provision of policy advice and overseeing
    of policy implementation as it affects the performance of institutions".
    (Cab. SEQ (88) 181).

The Cabinet
paper (para.34) indicated that these services would be reviewed in the
context of reviews of institutions. The institution's Charter
would include a requirement for teachers to receive support and students
needing special services to be provided for.

Since 1993,
the Special Education Service has negotiated an annual Document of Accountability
which requires its Board to report to the Minister on the delivery of
services. Since 1990, the Ministry of Education has negotiated
two year contracts with the Colleges/Schools of Education to deliver
Advisory Services. The Ministry of Education contracts include
quarterly reporting on the delivery of services against specified performance
measures. These contracts expire at the end of 1998 after which
provision of these services will depend on the outcome of the Teacher
Education Review. The Panel is aware of unmet needs in curriculum
delivery, assessment, self-review and performance management in the
schools and centres. As policy emerges from the Teacher Education
Review, these issues must be addressed in order to ensure that rural
as well as urban schools and centres have the advantage of quality in-service
education. It is incumbent on the Ministry of Education to ensure
that all schools and centres have access to the knowledge, skills and
data-bases which are available to improve the delivery of education.

The Education
Review Office argues that it fulfils the function of reviewing the Ministry's
provision of policy advice and its implementation through the regular
publication of its Evaluation Reports and in the interactions it has
with Ministers, the Ministry of Education and other agencies, such as
Te Puni Kokiri, as policy is being developed.

In the context
of this discussion, the Panel accepts that the Education and State Sector
Acts are robust and adequate in their statutory requirements of the
Education Review Office.

3 The Public Finance Act (1989)

The Education
Act (1989) s.90 states that every Board (of Trustees) is a Crown Entity
for the purposes of the Public Finance Act (1989). The Panel has
already argued against removal of the exemption from s.41 (2) (f) of
the Public Finance Act on the grounds that it would require an annual
educational audit of 2,700 Crown Entities in addition to the current
financial audit. The Boards of Crown Entities and Chartered Early
Childhood Services are required to have Charters (s.61 and s. 312 Education
Act 1989).

The OECD observes
in Schools Under Scrutiny (1995) that the New Zealand approach is,

    "that
    if the institution itself is involved in setting its own standards,
    .... the principal and staff will be motivated by a feeling of ownership
    which will make them perform better."

The Panel has
concluded that the Education Act should be the vehicle for achieving
the requirements of the Public Finance Act by making Charter documents
functional and operational and by stimulating local innovation and the
emergence of high standards. They are, after all, binding documents
on the Boards of Trustees given that s.64(2) empowers the Secretary
to take proceedings to enforce the Charter or to constrain a Board from
taking any action which is contrary to a Charter. An amendment
to s. 61 and s. 312 requiring schools and centres to have a three
year strategic plan and an annual statement of performance indicators

in their Charters would close a gap in the current system. At
the same time, it would provide the Education Review Office with a clear
statement of the school or centre's intentions and against which the
review would be conducted. This will remove the 'mystery'
and reinforce schools and centres as being self managing.

There has been
considerable comment in the literature on the meaning of 'standards'
in education (OECD 1989) along with warnings against judging success
against narrow mechanistic criteria. There is nothing to be gained
from endless debate about whether standards have risen or fallen.
What would be helpful is the clarification of the educational aims and
values on which the standards can be based, the use of available research
to develop indicators and set targets for achievement which are both
quantitative and qualitative.

4 Absence of service quality

The Education
Review Office has drawn attention to "the relative absence of any
service quality requirement such as those which might guide the performance
of trustees and their professional teaching staff." (Annual Report
1996). The Panel notes also that the Ministry of Education used
the New Zealand Education Gazette (30 April 1993) to inform schools
of the revised National Education Guidelines which were to be incorporated
into Charters, but it makes no mention of performance measures
which will be used in arriving at judgements.

In an overview
of the State Sector Reforms in New Zealand, Professor Schick (1996)
comments that while,

    "NZ
    has passed with flying colours and a culture of performance has penetrated
    public management", nevertheless "in spite of considerable accountability
    requirements the Crown entities' operations and finances are not as
    transparent as they should be."

5 Strategic plans and performance
indicators

    'If
    you don't know where you are going, any destination will do.'

It is for these
reasons that the Panel has reached the conclusion that there is a gap
in the present system which is inhibiting "self-managing" schools
from moving through "self-review" to "self-improvement".
There is an absence in most cases of planning, performance indicators,
measurement, review and reporting. The Panel is aware that a significant
number of schools and centres are engaging in the planning we propose.
This is endorsed and applauded by the Panel. Now there must be
a deliberate move to adopt the process throughout the education community
by the year 2000. The Panel has resolved that it is essential
for schools to develop performance indicators for each of the National
Administration Guidelines as set out in the recent Ministry of Education
Governance and Management package (July 1997) with the primary focus
being on curriculum delivery to improve educational outcomes.

These indicators
must be worthwhile, specific, measurable, achievable, realistic and
set against a time-frame for achievement. Included in the planning
would be the staffing required, the allocation of available resources
and the use of plant and equipment. While it would be the responsibility
of the Board of Trustees to develop the Strategic Plan and Performance
Indicators for National Administration Guidelines 2-6, the Principal
and staff would be responsible for developing those for Guideline 1
(the curriculum) for Board approval. (Appendix 5.)
As for any change in the Charter document the approval of the Secretary
of Education would be required (s.61 Education Act 1989).

This process
will require time and should be approached with caution in order to
inspire trust and confidence within the education community. It
may require a change in the attitudes of administrators, teachers, students,
parents and communities during the development phase but once in place
the benefits will flow through with an improved sense of purpose, understanding
of processes and monitoring. A further outcome will be that, for
the first time, it will be possible for New Zealand schools and centres
to demonstrate how they have used their available resources and what
resources are required to meet identified and accepted commitments.

6 The qualities of a good educational
place

The concept
of effectiveness has been highlighted in the educational administration
literature over the past two decades. This research suggests that
schools do make a difference in terms of the student outcomes
which schools achieve, after consideration has been given to the learning
histories and family backgrounds that students bring with them on entry
to their learning places. (Hopkins, Ainscow and West 1994). The
concept focuses attention on the question, "What are the qualities
of a good educational place?"

    "An effective
    school is one which sets performance objectives acceptable to its system
    and to its community and improves its achievement of these over time"
    (Braithwaite and Low).

They conclude
that performance indicators should:

  • Help schools and
    teachers improve their performance,
  • Provide guidance
    and management information to systems,
  • Provide the consumers
    and the public with indicators of school effectiveness, and
  • Guide policy formulation
    and research.

7 Performance indicators

The Panel has
concluded that the degree of effectiveness can be managed by way of
specific performance indicators. Braithwaite and Low distinguish
between three kinds of indicators:

  • Input indicators
    which include such items as resources expended in an organisation (e.g.
    dollar expenditure per student),
  • Process indicators
    which include aspects of curriculum provision, pastoral care or school
    atmosphere, and
  • Output indicators
    which include skill achievement levels, attendance and retention rates.

(Appendix 10
provides an outline of the types of indicators which might be useful.)

The Panel is
encouraged by the work already done by the Ministry of Education
in, "A Guide to Good Practice" (1995) and wishes to see this work
further developed to assist schools and centres to develop their own
plans.

8 Criteria for review clarified

The Panel is
convinced that, once these plans are in place, the criteria for review
would be clarified and the Education Review Office would use the performance
indicators as standards for its review purposes. The Charter would
once again be a functional, operational document because it would specify
educational objectives, performance indicators and expected outcomes.
It would also be the basis on which the Annual Report of the school
or centre would be written.

These developments
should offset the criticism coming from schools that they do not know
the criteria for review, that there is mystery surrounding the Education
Review Office's work and that the context in which the school operates
is ignored. The Panel believes that these proposals will emphasise
the flexibility and scope of local involvement, enhance local control
and governance and provide schools and centres with an operational plan
for the delivery of its educational function. Schools or centres
will have certainty and know what they are setting out to achieve and
will be able to report with clarity on what has been achieved.

9 Critical factors

There is a
risk that under-performing schools may write strategic plans which deal
with lower order issues and not maximise the educational potential of
the students enrolled. The Panel regards this work as critical
to the process of change and school improvement. At the same time
the Panel does not minimise the demands it will make, nor the time which
will have to be devoted to it. The task requires a high level
of professionalism from those involved in education in New Zealand but
the Panel is convinced that the commitment and challenge of the task
in the short term will produce more satisfying professional outcomes
long term. Guidance from the Ministry of Education will be essential.
Some schools may wish to adapt a standard model which has the approval
of the Ministry; others may seek the help of a consultant to work to
a brief to develop the Strategic Plans and Statement of Performance
Indicators. It is especially important that small schools are
encouraged to form clusters to work co-operatively on producing these
documents.

RECOMMENDATIONS:

    31. That
    the Education Review Office provide schools and centres with the indicators
    and standards which they use during a review in order to make judgements
    about them until such time as schools and centres develop their own
    Strategic Plans and Annual Statements of Performance Indicators.
    32. That
    schools and centres be required to include a Three-year Strategic Plan
    for the delivery of the National Administration Guidelines and an Annual
    Statement of Performance Indicators in their Charters against which
    they would report annually.
    33. That
    the same requirements for amending Charters as set out in s.61 of the
    Education Act (1989) be applied to the Strategic Plan.
    34. That the Ministry of Education
    identify the production of guidelines for strategic planning and statements
    of performance indicators as a priority; and the Minister of Education
    consider professional development to assist the implementation as key
    issues for the 1998/99 and 1999/2000 budgets.


10 Self-review and self-improvements

The National
Administration Guideline 4 requires Boards of Trustees to:

  1. Document how the
    National Education Guidelines are being implemented, and
  2. Maintain an on-going
    programme of self-review.

The Education
Review Office published an Evaluation Report on Self Review in 1994
and the Ministry of Education package 'Governing and Managing
Part Two' (1997) has a guide to self review which has been very well
received. It became clear to the Panel, during its hearings and
public meetings, that there was considerable goodwill to becoming involved
in self-review. It was seen as 'empowering, motivating, and
satisfying' but 'currently a fragile plant with great potential.'

In some respects,
the Early Childhood Sector is well ahead of the Compulsory Sector in
implementing robust self-review. Some centres have developed their
own indicators, while others have called in a consultant to assist with
the process. It is seen as a melding of external and internal
quality assurance processes that has led to improved provision of quality
education and care. It is also acknowledged within this sector
that there is still a great deal of development work to be done.

11 School improvement

The model which
the Panel is advocating includes:

  • Strategic thinking,
  • Strategic planning,
  • Statements of performance
    indicators,
  • Quantitative and
    qualitative measurement,
  • The Board document
    assuring compliance,
  • Routine assurance
    checks by the Education Review Office,
  • On-going self-review
    monitoring and analysis,
  • Accountability
    reviews focusing on educational effectiveness and quality of outcomes,
    and
  • Reporting.

Documents are
shown in squares and organisations in circles.

The Panel does
not underestimate the development work which will need to be done and
the commitment required to put in place robust, informative processes.
This work relies heavily on staff confidence, competence, and also effective
leadership. However, the Panel concurs with the observation of
Hargreaves and Hopkins (1991) that school improvement will result from,

    "systematic,
    sustained effort aimed at change in learning conditions and other related
    internal conditions in one or more schools with the ultimate aim of
    accomplishing educational goals more effectively,"

12 Small and rural schools

As with the
Strategic Plan development, small and rural schools are likely to experience
difficulty if they are expected to work in isolation. The Panel
again recommends they would be well advised to form clusters with the
help of facilitators to develop the protocols to undertake self-review
with the support of colleagues engaged in the same task in similar circumstances.
Professional development training would need to be available through
the Advisory Services, consultants etc., with guidelines available on
the Internet, floppy disc, or hard copy on request from the Ministry
of Education so that no school or centre feels they are working on their
own.

13 Quality education outcomes

Over the next
five years, schools will have access to the outcomes of the National
Monitoring Programme: the benchmarks and expectations for each level
will be clarified through the processes the Panel has outlined and the
New Zealand Council for Educational Research's item banks should be
available. These sources will be helpful in assisting schools
to undertake the onerous development work which is required. The
expectation of the Panel is that the platform outlined will facilitate
excellence. Schools, their communities and the Crown will be better
informed about the effectiveness of the delivery of education and the
quality of educational outcomes than is currently the case. Confidence
in the system and heightened professionalism will also be outcomes.
In addition it is likely that the Educational Review Office may be able
to reallocate its resources to more frequent review of schools and centres
on notification or on request where problems or risks have been identified.

RECOMMENDATIONS:

35. That self-review should drive both school improvement and external evaluation.

36. That schools and centres be required to plan and implement self-review protocols,
assisted by guidelines, and in-service professional development.