A Review of the Education External Evaluation Services

Brian Donnelly Education Review Office

TERM OF REFERENCE 2

Schools must
create the environment for teachers to teach and students to succeed

A METHODOLOGY & FREQUENCY OF ERO'S
EXTERNAL EVALUATION SERVICES

The Education
Review Office's briefing notes to the Panel illustrate the accountability
framework for school governance in the following diagrammatic form.

The essential
documents on which Reviews are based are the National Education Guidelines
which include the National Education Goals, the National Curriculum
Statements and the National Administration Guidelines. (Appendix
5) Schools and Early Childhood Education Centres are required
to have a Charter under the Education Act (1989) s.61-64 and s.312-313.
These documents are integral to a self-managing system because they
encapsulate the vision of the institution and the local circumstances
which influence it, including Maori in the community and special features.
Charters may be amended after consultation with the community and with
the approval of the Secretary for Education who is also empowered to
enforce the Charter but not without consulting the Chief Review Officer.
Notwithstanding the intention of the Act that the Charter is a contract
for educational delivery, the Panel has concluded that Charters are,
by and large, no longer functional documents that guide the schools'
and centres' operations, despite a number being presented to the Secretary
for modification during 1997.

For the Early
Childhood Education sector there is also provision for Statements of
Desirable Objectives and Practices (DOPs) to be set by Regulation.
The 1996 revision of the Statements will become operational on 1 August
1998 (as stated in the New Zealand Education Gazette 13.10.97).
The purpose of the DOPs is to establish national criteria for
the provision of quality early childhood education and care. They
are deemed to be part of the Charter of every provider.

1 Types of review

The Education
Review Office undertook a number of different types of review (1990-1997).

  • Assurance Audits:
    in which the governance and management of the school or centre are examined
    in terms of meeting their obligations and undertakings to the Crown.
    These audits are undertaken in all state and privately owned schools
    and in all licensed or chartered early childhood centres. There
    are three broad conceptual phases to be examined in order to reach each
    judgement of compliance or non-compliance during an assurance audit.
  • Phase 1: Compliance
    Intentions,
  • Phase 2 Preliminary
    Compliance Conclusions,
  • Phase 3 Compliance
    Verification.

The report
is objective, clear and concise and the tone is positive. The
actions required for compliance are clearly stated and written in ways
which make it clear to governing/ managing bodies what is required of
them. Suggested developments are stated which should lead to improved
practices. (Further details are set out in Appendix 7.)

  • Effectiveness
    Reviews: in which there is an evaluation of student achievement
    and the impact of the teaching services and management practices within
    a school on that achievement. Such a review is concerned with
    determining the extent to which the school knows about student achievement
    and uses that information to impact positively on students' learning.
    Effectiveness
    reviews, therefore:
  • Evaluate the schools'
    expectations of student achievement,
  • Verify the difference
    demonstrated in student achievement,
  • Evaluate the impact
    of the school on student achievement during the time the students have
    been enrolled at the school,
  • Evaluate the progress
    made by students against the achievement objectives of the New Zealand
    curriculum and local objectives,
  • Evaluate the competence
    of the school in monitoring, assessing and analysing information to
    improve student learning,
  • Identify and evaluate
    a range of factors present within the school which impact on student
    achievement.

(Further
details are set out in Appendix 8.)

  • Accountability
    Reviews (1998): These are being developed during 1997 for introduction
    in 1998. It is believed that Accountability Reviews will be more
    focused on evaluating the extent to which early childhood centres,
    schools and other education providers (such as home schoolers) discharge
    the various accountabilities placed on them by their Charters, legislation
    and regulations. The aim is to concentrate on the quality of performance,
    delivery of education and areas of risk to pre-school children and students,
    focusing the professional evaluation skills of the review officers accordingly.
    This approach involves improved information handling and analysis, changes
    to the process of scheduling reviews, deploying the specialist skills
    of review officers as a national - rather than just a local - resource,
    and changes to the way in which the Office reports on its reviews.

Accountability Reviews Focused on Stakeholder Interests

Stakeholder Performance Interest
    Crown
    (purchaser)
The consistency of service
delivery with purchase/ contractual requirements (e.g. as may be set
out in charters)
    Crown
    (owner)
The quality with which service
providers manage the Crown's ownership interests i.e.
  • Financial capital
    investment
  • Physical capital
    investment (plant & equipment)
  • Human capital investment

and the capability
of these assets to sustain a return on the Crown's investment

    Students and Parents
    (consumer)
The ability of governing/
managing bodies to:
  • Identify students'
    needs, requirements & entitlements
  • Add value in terms
    of student achievement
  • Provide and sustain
    high quality educational services

From
Education Briefing notes to the Panel. (Further details on Accountability
Reviews are set out in Appendix 9.)

2 Discussion

Essentially
the Assurance Audit process has been used as an incentive to schools
and centres to meet the legal requirements prescribed by statute and
associated regulations. That this was the right place to start
is not questioned. In 1993, only about 6% of schools and centres
met the compliance standards as opposed to 88% compliance in 1996.
This indicates that the incentive has been effective in terms of legal
requirements and, while the primary focus on this in past years may
have been appropriate, it is now time to focus more clearly on the effectiveness
of education delivery and outcomes. In this regard the Panel and
compulsory sector applaud the introduction of the Board of Trustees
Declaration document and the random on-site audit of compliance issues.
This declaration encompasses the legal compliances for which Boards
are responsible and requires them to sign off that their policies are
in place.

The Panel is
also aware that the Office recognises the need to be more firmly focused
on education effectiveness, while retaining the requirement for schools
to meet legal compliance hence the introduction of Accountability Reviews
in 1998. The Panel does, however, have serious reservations about
the transition from Effectiveness Reviews to Accountability Reviews.
These concerns are largely to do with effective communication.

Schools do
not appear to have had the same exposure and experience of Effectiveness
Reviews as they have had with Assurance Audits. Many schools report
that, in the past seven years, they have had two Assurance Audits and
as recently as in the past 12-18 months had their first Effectiveness
Review. A small number of schools report that they are due to
have their first Effectiveness Review this year although the 1996/97
Education Review Office Annual Report indicated that 359 of 1243 reviews
were Effectiveness Reviews (28%) the timetables for reviews in each
region for the first part of Term 3 1997 which were provided for the
Panel show that, of 123 reviews to be undertaken, only 22 (18%) were
Effectiveness Reviews. Only the Central South Region showed a
preponderance of Effectiveness Reviews (10/15). This is surprising
when the Office is aware of the degree of compliance which exists
and reinforces the need to move rapidly to Accountability Reviews.

3 Is the current approach appropriate?

One of the
single greatest areas of discontent expressed by practitioners was what
they described as the Education Review Office's preoccupation with
what can be measured to show "value added" performance. Many
practitioners believed that the Office's fixation with quantitative
assessment data was too narrow in evaluating the performance and, therefore,
the school's effectiveness in terms of students' achievements.
Criticism also extended to the questioning of the reviewers' interpretation
and professional judgement of what constituted valid data in the assessment
of curriculum objectives.

Submissions
repeatedly referred to schools being diverse communities and collectives
of learners that frequently acted in synergy though not always exclusively
so. While academic achievement is unequivocally an important component,
a range of other significant factors and complex interactions contribute
to a student's achievements, many of which are not tangible and, consequently,
are not directly measurable. That this is so should not preclude
the value of such factors being considered in the evaluation process.
Indeed, it is the Panel's opinion that qualitative data is as important
and as valid as the quantitative data which is gathered to provide a
profile on a student, programme or school. The value of any piece
of data will, however, be determined by a range of factors, including
context and relevance.

The Panel further
subscribes to the view that it is not a matter of either/or but that
a combination of both would produce the richest and most satisfying
results for all interested parties. It should of course be noted
that either data base is of little value to a school if it is only being
collected for the purposes of meeting the requirements of the external
evaluation agency rather than to inform the school, the students and
their caregivers.

4
Directions for Better Education

    "All
    OECD countries have made tremendous economic efforts during the past
    twenty years to invest in the material provision of schools and to carry
    out sweeping structural, organisational and curricular reforms. These
    efforts have brought considerable success. So far, however, success
    has been measured largely in material terms."

The OECD (1989)
warns that "the next phase will call for emphasis on less tangible
improvements which will necessarily prove more difficult to achieve
than the fulfilment of quantitative targets. It will require educational
authorities at all levels to consider now what compulsory schooling
should look like in ten or twenty years' time. What will society
then be wanting from the schools and what will schools be wanting from
society?"

When the Ministers
of OECD countries met in Paris in 1984, " Quality in Basic Schooling"
figured as one of three major items on their agenda. The Ministers
concluded their deliberations by issuing a communiqu&eacut; in which they
declared that a priority for future OECD work in the education field
should be:

    "The
    quality of basic schooling, particularly basic education related to
    modern needs in increasingly pluralistic societies, including matters
    such as:
  • better preparation
    for adult life
  • measures to improve
    the status, effectiveness and professional role of teachers
  • the organisation,
    content and structure of the curriculum and methods of evaluation
  • qualitative factors
    affecting the performance of schools, including school based leadership
    and
  • programmes designed
    for the disadvantaged and the handicapped."

Such qualities
were embraced in the literature on schools during the 1980s. They
promoted an emphasis on identifying the characteristics of "good schools".
Downer (1991) for example, summarised a wide range of literature on
"effective
" schools. The more current perspective focuses
on "doing something about it" - i.e. seeking to "improve"
the quality of the learning phase. The essence here lies in commitment
to action and acknowledging that planning and implementation strategies
for improvement go far beyond simply knowing about effectiveness.

Writers of
the 1990s such as Hargreaves & Hopkins (1991) Southworth (1993)
and Reynolds, Hopkins and Stoll, (1993) offer advice on linking knowledge
of effectiveness with practices designed to result in "better" schools
and centres. The contemporary improvement movement assumes, e.g.
that the individual institution should be the centre of change; that
improvement should be carefully planned and managed; that educational
goals are more than test scores; that leadership support and staff energy
are both important; and that change succeeds when it becomes part of
natural behaviour.

The Panel further
develops this approach in the report.

5 Teacher observations

Assessment
and evaluation practices embrace a wide range of instruments.
One, perhaps, that has not been given the respect or value due to it,
is teacher observational expertise, experience and description.

    "...teachers
    themselves must depend on their observation of students to determine
    their effectiveness on a day by day basis. Research has shown
    that teachers depend on available signs, involvement and motivation
    to determine whether students are learning or not. Since teachers
    have no direct access to the minds of students, and they must adjust
    their teaching to the apparent level of involvement of students in the
    learning process, teachers have to use all available indirect signs
    of their effect on students' minds...." (Nuthall, submission
    104.)

Nuthall goes
on to remind us that observation of student behaviours in the classroom
is not a reliable substitute for direct assessment of student learning
even though such evaluation is always relative. He comments that
"the observational expertise has been translated into a myth and that
research has shown it is rarely possible to tell by observation when
students are learning." In Nuthall's view, observations
need to be regular and set in context to allow for meaningful analysis
and conclusions to be drawn.

These comments
constitute some of the reasons for reviewers requiring assessment data,
its aggregation and analysis in making judgements about the effectiveness
of programmes delivered by teachers. There are no shortcuts.
This does not mean that descriptive information should not be kept.
The Panel has already indicated that both quantitative and qualitative
information are desirable.

6 Socio-economic status of the school
community as a contextual variable

Included in
the debate about the relevance of qualitative data and 'value added'
in the external evaluation of schools is the bearing of the decile ranking
of a school. Comments made to the Panel ranged from the decile
ranking being used as an excuse for the school's or students' poor
performance, to it being a factor that is a barrier to effective learning
and consequently achievement, to it having little if any bearing at
all.

Submissions
from Principals and Trustees from the low decile schools were particularly
vociferous on the need for review reports to include comment on the
contextual issues that impacted on their schools. This was supported
by others who added that comment should also cover government social
sector policy generally, while others believed it should be restricted
to the Ministry of Education's policies and practices.

Essentially
the message the Panel received is that schools want contextual issues
to be included in the review process as an acknowledgement that there
are factors external to the school over which they have no control;
but that it is likely these factors have an impact on student achievement
levels and the effectiveness of the school's performance.

Even education
researchers have varying views on the impact of external factors on
a school's effectiveness. The researchers commissioned by PPTA
to Review the Education Review Office (in an informal communication
with the Panel) refer to Metz (1990) who observes:

    "To
    the degree that the educational reform movement sets aside class differences
    as unimportant, it brackets and overlooks one of the major influences
    on schools. It consequently relies for all of its impact on attempting
    to change patterns that exert much weaker influence. To ignore
    the most forceful influences in a situation is rarely a prescription
    for effective reform."

On the other
hand, Thomas, Simmons and Mortimore (1995) comment that:

    "Recent
    research has shown that data relating to factors outside the control
    of the school (for example, gender and social class, and neighbourhood
    characteristics of pupil's homes) are useful only in fine-tuning schools'
    value added results."

Results from
two major American studies on socio-economic status as a context variable,
one by Hallinger & Murphy and one by Teddlie and Stringfield (in
Reynolds et al 1994) confirm that there were some characteristics which
distinguished effective from ineffective schools,
regardless of the socio-economic status (SES) of the schools.
They included: clear academic mission and focus, orderly environment,
high academic engaged time on task, and frequent monitoring of student
progress.

Teddlie further
reports that the differences in characteristics associated with effectiveness
in middle and low SES schools revolve around six areas:

  • Promotion of educational
    expectations,
  • Principal leadership
    style,
  • The use of external
    reward structures,
  • Emphasis on the
    school curriculum,
  • Parental contact
    with the school, and
  • Experience level
    of teachers.

In a recent
British study (Blakey & Heath, 1992) the authors concluded that,
when explaining a pupil's performance, "we need to take account
both of his or her own social class background and the background of
the other children in the school." The authors noted that student
subcultures within schools were moulded more by social class than by
ability.

7 Can effective schools compensate
for society?

This is the
title of a chapter included in research undertaken by Peter Mortimore
which he prefaces with, 'Why ask the question? According to
Silver (1994), schools, the philosophers, and providers of schooling,
have always been concerned with outcomes.' Indeed quality educational
outcomes are at the very heart of this review. In attempting to
define contextual issues the Panel believed that a range of variables
might be considered including the effects of the school's culture
on the learning outcomes of their students. It is the examination
of this issue that has inspired considerable research in the last 20
years, notably within the School Effectiveness Movement.

Mortimore notes
that, in the USA, 'the research on the potential differential effects
of individual schools has been closely bound up with concern about the
plight of disadvantaged children and the limited opportunities available
to them.' In the United Kingdom, detailed research (Reynolds
1976, Rutter 1979, Mortimore 1988, Tizard 1988, Smith & Tomlinson
1989), found clear evidence of school differences in students' outcomes.
Each of the research teams used this evidence to argue that these differences
were not simply due to the effects of schools receiving different types
of students but, rather, that they were associated with differences
in the way the schools, through management and the quality of the teaching
and learning environment, promoted achievement.'

8 Schools a mediating lever

For families
whose lives are disadvantaged in relation to their peers, schools remain
one of the few mechanisms that are able to provide a compensating boost.
The more effective the school the higher the proportion of students
who will get to the starting-line in the competition for favourable
life chances.

9 So: can effective schools compensate
for society?

Mortimore's
judgement is, "Yes", to a certain extent. The Panel agrees
with this. What needs to be clarified by each school and centre is
the specification of the extent to which it can compensate, given the
resources (personnel, plant and finances) it has at its disposal.
The mechanism for this process is discussed in greater detail under
Charters and Strategic Plans. The Panel wishes to emphasise that
schools and centres need to take account of those external factors in
their initial planning and not at the end of the cycle when they become
a convenient excuse for performance that is less than acceptable.