A Review of the Education External Evaluation ServicesBrian Donnelly Education Review Office
TERM OF REFERENCE 2
create the environment for teachers to teach and students to succeed
A METHODOLOGY & FREQUENCY OF ERO'S
EXTERNAL EVALUATION SERVICES
Review Office's briefing notes to the Panel illustrate the accountability
framework for school governance in the following diagrammatic form.
documents on which Reviews are based are the National Education Guidelines
which include the National Education Goals, the National Curriculum
Statements and the National Administration Guidelines. (Appendix
5) Schools and Early Childhood Education Centres are required
to have a Charter under the Education Act (1989) s.61-64 and s.312-313.
These documents are integral to a self-managing system because they
encapsulate the vision of the institution and the local circumstances
which influence it, including Maori in the community and special features.
Charters may be amended after consultation with the community and with
the approval of the Secretary for Education who is also empowered to
enforce the Charter but not without consulting the Chief Review Officer.
Notwithstanding the intention of the Act that the Charter is a contract
for educational delivery, the Panel has concluded that Charters are,
by and large, no longer functional documents that guide the schools'
and centres' operations, despite a number being presented to the Secretary
for modification during 1997.
For the Early
Childhood Education sector there is also provision for Statements of
Desirable Objectives and Practices (DOPs) to be set by Regulation.
The 1996 revision of the Statements will become operational on 1 August
1998 (as stated in the New Zealand Education Gazette 13.10.97).
The purpose of the DOPs is to establish national criteria for
the provision of quality early childhood education and care. They
are deemed to be part of the Charter of every provider.
1 Types of review
Review Office undertook a number of different types of review (1990-1997).
- Assurance Audits:
in which the governance and management of the school or centre are examined
in terms of meeting their obligations and undertakings to the Crown.
These audits are undertaken in all state and privately owned schools
and in all licensed or chartered early childhood centres. There
are three broad conceptual phases to be examined in order to reach each
judgement of compliance or non-compliance during an assurance audit.
- Phase 1: Compliance
- Phase 2 Preliminary
- Phase 3 Compliance
is objective, clear and concise and the tone is positive. The
actions required for compliance are clearly stated and written in ways
which make it clear to governing/ managing bodies what is required of
them. Suggested developments are stated which should lead to improved
practices. (Further details are set out in Appendix 7.)
Reviews: in which there is an evaluation of student achievement
and the impact of the teaching services and management practices within
a school on that achievement. Such a review is concerned with
determining the extent to which the school knows about student achievement
and uses that information to impact positively on students' learning.
- Evaluate the schools'
expectations of student achievement,
- Verify the difference
demonstrated in student achievement,
- Evaluate the impact
of the school on student achievement during the time the students have
been enrolled at the school,
- Evaluate the progress
made by students against the achievement objectives of the New Zealand
curriculum and local objectives,
- Evaluate the competence
of the school in monitoring, assessing and analysing information to
improve student learning,
- Identify and evaluate
a range of factors present within the school which impact on student
details are set out in Appendix 8.)
Reviews (1998): These are being developed during 1997 for introduction
in 1998. It is believed that Accountability Reviews will be more
focused on evaluating the extent to which early childhood centres,
schools and other education providers (such as home schoolers) discharge
the various accountabilities placed on them by their Charters, legislation
and regulations. The aim is to concentrate on the quality of performance,
delivery of education and areas of risk to pre-school children and students,
focusing the professional evaluation skills of the review officers accordingly.
This approach involves improved information handling and analysis, changes
to the process of scheduling reviews, deploying the specialist skills
of review officers as a national - rather than just a local - resource,
and changes to the way in which the Office reports on its reviews.
Accountability Reviews Focused on Stakeholder Interests
||The consistency of service
delivery with purchase/ contractual requirements (e.g. as may be set
out in charters)
||The quality with which service
providers manage the Crown's ownership interests i.e.
and the capability
||The ability of governing/
managing bodies to:
Education Briefing notes to the Panel. (Further details on Accountability
Reviews are set out in Appendix 9.)
the Assurance Audit process has been used as an incentive to schools
and centres to meet the legal requirements prescribed by statute and
associated regulations. That this was the right place to start
is not questioned. In 1993, only about 6% of schools and centres
met the compliance standards as opposed to 88% compliance in 1996.
This indicates that the incentive has been effective in terms of legal
requirements and, while the primary focus on this in past years may
have been appropriate, it is now time to focus more clearly on the effectiveness
of education delivery and outcomes. In this regard the Panel and
compulsory sector applaud the introduction of the Board of Trustees
Declaration document and the random on-site audit of compliance issues.
This declaration encompasses the legal compliances for which Boards
are responsible and requires them to sign off that their policies are
The Panel is
also aware that the Office recognises the need to be more firmly focused
on education effectiveness, while retaining the requirement for schools
to meet legal compliance hence the introduction of Accountability Reviews
in 1998. The Panel does, however, have serious reservations about
the transition from Effectiveness Reviews to Accountability Reviews.
These concerns are largely to do with effective communication.
not appear to have had the same exposure and experience of Effectiveness
Reviews as they have had with Assurance Audits. Many schools report
that, in the past seven years, they have had two Assurance Audits and
as recently as in the past 12-18 months had their first Effectiveness
Review. A small number of schools report that they are due to
have their first Effectiveness Review this year although the 1996/97
Education Review Office Annual Report indicated that 359 of 1243 reviews
were Effectiveness Reviews (28%) the timetables for reviews in each
region for the first part of Term 3 1997 which were provided for the
Panel show that, of 123 reviews to be undertaken, only 22 (18%) were
Effectiveness Reviews. Only the Central South Region showed a
preponderance of Effectiveness Reviews (10/15). This is surprising
when the Office is aware of the degree of compliance which exists
and reinforces the need to move rapidly to Accountability Reviews.
3 Is the current approach appropriate?
One of the
single greatest areas of discontent expressed by practitioners was what
they described as the Education Review Office's preoccupation with
what can be measured to show "value added" performance. Many
practitioners believed that the Office's fixation with quantitative
assessment data was too narrow in evaluating the performance and, therefore,
the school's effectiveness in terms of students' achievements.
Criticism also extended to the questioning of the reviewers' interpretation
and professional judgement of what constituted valid data in the assessment
of curriculum objectives.
repeatedly referred to schools being diverse communities and collectives
of learners that frequently acted in synergy though not always exclusively
so. While academic achievement is unequivocally an important component,
a range of other significant factors and complex interactions contribute
to a student's achievements, many of which are not tangible and, consequently,
are not directly measurable. That this is so should not preclude
the value of such factors being considered in the evaluation process.
Indeed, it is the Panel's opinion that qualitative data is as important
and as valid as the quantitative data which is gathered to provide a
profile on a student, programme or school. The value of any piece
of data will, however, be determined by a range of factors, including
context and relevance.
The Panel further
subscribes to the view that it is not a matter of either/or but that
a combination of both would produce the richest and most satisfying
results for all interested parties. It should of course be noted
that either data base is of little value to a school if it is only being
collected for the purposes of meeting the requirements of the external
evaluation agency rather than to inform the school, the students and
Directions for Better Education
OECD countries have made tremendous economic efforts during the past
twenty years to invest in the material provision of schools and to carry
out sweeping structural, organisational and curricular reforms. These
efforts have brought considerable success. So far, however, success
has been measured largely in material terms."
The OECD (1989)
warns that "the next phase will call for emphasis on less tangible
improvements which will necessarily prove more difficult to achieve
than the fulfilment of quantitative targets. It will require educational
authorities at all levels to consider now what compulsory schooling
should look like in ten or twenty years' time. What will society
then be wanting from the schools and what will schools be wanting from
When the Ministers
of OECD countries met in Paris in 1984, " Quality in Basic Schooling"
figured as one of three major items on their agenda. The Ministers
concluded their deliberations by issuing a communiqu&eacut; in which they
declared that a priority for future OECD work in the education field
quality of basic schooling, particularly basic education related to
modern needs in increasingly pluralistic societies, including matters
- better preparation
for adult life
- measures to improve
the status, effectiveness and professional role of teachers
- the organisation,
content and structure of the curriculum and methods of evaluation
- qualitative factors
affecting the performance of schools, including school based leadership
- programmes designed
for the disadvantaged and the handicapped."
were embraced in the literature on schools during the 1980s. They
promoted an emphasis on identifying the characteristics of "good schools".
Downer (1991) for example, summarised a wide range of literature on
"effective" schools. The more current perspective focuses
on "doing something about it" - i.e. seeking to "improve"
the quality of the learning phase. The essence here lies in commitment
to action and acknowledging that planning and implementation strategies
for improvement go far beyond simply knowing about effectiveness.
the 1990s such as Hargreaves & Hopkins (1991) Southworth (1993)
and Reynolds, Hopkins and Stoll, (1993) offer advice on linking knowledge
of effectiveness with practices designed to result in "better" schools
and centres. The contemporary improvement movement assumes, e.g.
that the individual institution should be the centre of change; that
improvement should be carefully planned and managed; that educational
goals are more than test scores; that leadership support and staff energy
are both important; and that change succeeds when it becomes part of
The Panel further
develops this approach in the report.
5 Teacher observations
and evaluation practices embrace a wide range of instruments.
One, perhaps, that has not been given the respect or value due to it,
is teacher observational expertise, experience and description.
themselves must depend on their observation of students to determine
their effectiveness on a day by day basis. Research has shown
that teachers depend on available signs, involvement and motivation
to determine whether students are learning or not. Since teachers
have no direct access to the minds of students, and they must adjust
their teaching to the apparent level of involvement of students in the
learning process, teachers have to use all available indirect signs
of their effect on students' minds...." (Nuthall, submission
on to remind us that observation of student behaviours in the classroom
is not a reliable substitute for direct assessment of student learning
even though such evaluation is always relative. He comments that
"the observational expertise has been translated into a myth and that
research has shown it is rarely possible to tell by observation when
students are learning." In Nuthall's view, observations
need to be regular and set in context to allow for meaningful analysis
and conclusions to be drawn.
constitute some of the reasons for reviewers requiring assessment data,
its aggregation and analysis in making judgements about the effectiveness
of programmes delivered by teachers. There are no shortcuts.
This does not mean that descriptive information should not be kept.
The Panel has already indicated that both quantitative and qualitative
information are desirable.
6 Socio-economic status of the school
community as a contextual variable
the debate about the relevance of qualitative data and 'value added'
in the external evaluation of schools is the bearing of the decile ranking
of a school. Comments made to the Panel ranged from the decile
ranking being used as an excuse for the school's or students' poor
performance, to it being a factor that is a barrier to effective learning
and consequently achievement, to it having little if any bearing at
from Principals and Trustees from the low decile schools were particularly
vociferous on the need for review reports to include comment on the
contextual issues that impacted on their schools. This was supported
by others who added that comment should also cover government social
sector policy generally, while others believed it should be restricted
to the Ministry of Education's policies and practices.
the message the Panel received is that schools want contextual issues
to be included in the review process as an acknowledgement that there
are factors external to the school over which they have no control;
but that it is likely these factors have an impact on student achievement
levels and the effectiveness of the school's performance.
researchers have varying views on the impact of external factors on
a school's effectiveness. The researchers commissioned by PPTA
to Review the Education Review Office (in an informal communication
with the Panel) refer to Metz (1990) who observes:
the degree that the educational reform movement sets aside class differences
as unimportant, it brackets and overlooks one of the major influences
on schools. It consequently relies for all of its impact on attempting
to change patterns that exert much weaker influence. To ignore
the most forceful influences in a situation is rarely a prescription
for effective reform."
On the other
hand, Thomas, Simmons and Mortimore (1995) comment that:
research has shown that data relating to factors outside the control
of the school (for example, gender and social class, and neighbourhood
characteristics of pupil's homes) are useful only in fine-tuning schools'
value added results."
two major American studies on socio-economic status as a context variable,
one by Hallinger & Murphy and one by Teddlie and Stringfield (in
Reynolds et al 1994) confirm that there were some characteristics which
distinguished effective from ineffective schools,
regardless of the socio-economic status (SES) of the schools.
They included: clear academic mission and focus, orderly environment,
high academic engaged time on task, and frequent monitoring of student
reports that the differences in characteristics associated with effectiveness
in middle and low SES schools revolve around six areas:
- Promotion of educational
- Principal leadership
- The use of external
- Emphasis on the
- Parental contact
with the school, and
- Experience level
In a recent
British study (Blakey & Heath, 1992) the authors concluded that,
when explaining a pupil's performance, "we need to take account
both of his or her own social class background and the background of
the other children in the school." The authors noted that student
subcultures within schools were moulded more by social class than by
7 Can effective schools compensate
This is the
title of a chapter included in research undertaken by Peter Mortimore
which he prefaces with, 'Why ask the question? According to
Silver (1994), schools, the philosophers, and providers of schooling,
have always been concerned with outcomes.' Indeed quality educational
outcomes are at the very heart of this review. In attempting to
define contextual issues the Panel believed that a range of variables
might be considered including the effects of the school's culture
on the learning outcomes of their students. It is the examination
of this issue that has inspired considerable research in the last 20
years, notably within the School Effectiveness Movement.
that, in the USA, 'the research on the potential differential effects
of individual schools has been closely bound up with concern about the
plight of disadvantaged children and the limited opportunities available
to them.' In the United Kingdom, detailed research (Reynolds
1976, Rutter 1979, Mortimore 1988, Tizard 1988, Smith & Tomlinson
1989), found clear evidence of school differences in students' outcomes.
Each of the research teams used this evidence to argue that these differences
were not simply due to the effects of schools receiving different types
of students but, rather, that they were associated with differences
in the way the schools, through management and the quality of the teaching
and learning environment, promoted achievement.'
8 Schools a mediating lever
whose lives are disadvantaged in relation to their peers, schools remain
one of the few mechanisms that are able to provide a compensating boost.
The more effective the school the higher the proportion of students
who will get to the starting-line in the competition for favourable
9 So: can effective schools compensate
judgement is, "Yes", to a certain extent. The Panel agrees
with this. What needs to be clarified by each school and centre is
the specification of the extent to which it can compensate, given the
resources (personnel, plant and finances) it has at its disposal.
The mechanism for this process is discussed in greater detail under
Charters and Strategic Plans. The Panel wishes to emphasise that
schools and centres need to take account of those external factors in
their initial planning and not at the end of the cycle when they become
a convenient excuse for performance that is less than acceptable.