A Review of the Education External Evaluation Services

Brian Donnelly Education Review Office

14 Contestability

A small number
of submissions advocated that the evaluation of schools and centres
should be a contestable service. The Office for Standards in Education
(OFSTED) in the United Kingdom was cited as a possible model.
There, teams of independent inspectors are invited to tender for the
inspection of individual schools by OFSTED; if they meet a minimum quality
threshold, contracts are awarded on the basis of value for money.
Every inspection team must include one "registered inspector" who
takes ultimate responsibility for the satisfactory completion of the
contract and one "lay inspector" who should not be professionally
involved in education. The Panel has considered the proposal and
has resolved that there is no groundswell of support for it.

Some of the reasons for this conclusion include: the small size of the New Zealand
market; the difficulty of covering the country, particularly its more
remote parts; the need to achieve national consistency where there are
disparate entities conducting reviews (OFSTED's principal problem);
the need to ensure that the teams have the required knowledge and background
to evaluate effectively; and the training costs where there is a constant
or periodic changeover of personnel.

Reviewers require
a substantial background of experience and knowledge and, even when
competent educational professionals are recruited, they still need extensive
training before going into the field and, even when they do, further
support is required.

15 Maori education

The Education
Act s.62 and s.63 make provision for policies and practices that "reflect
... the unique position of Maori culture" and that "all reasonable
steps to ensure that instruction in tikanga Maori and Te Reo Maori are
provided for full time students whose parents ask for it".

Great concern
about Maori underachievement was reflected in submissions received about
Maori education and the education of Maori. These concerns are
shared by the Panel.

It is evident
that the Education Review Office has a responsibility to investigate
and report on the policies and procedures adopted by schools and centres
and that the rigour with which the Education Review Office approaches
this task needs to be intensified. Doing so would signal to schools
and centres that responsiveness to Maori and tangata whenua is a priority
that is critical to improving the effectiveness of educational outcomes
and results that are of an enduring nature. Rigorous, robust review
must occur of the total learning experience of Maori students and must
not be restricted to Maori language and custom. The review process
would take into account a range of features which could include some
or all of the following: resourcing that facilitates responsiveness;
the policy framework that sets direction and accountability; data collection
and analysis that informs and improves; professional development; appropriate
cultural practices; stakeholder satisfaction; pastoral care provision;
representation; liaison and consultation; and attendance, attrition
and retention rates of Maori students.

It is important
to acknowledge that the wide range of performance indicators set out
in Appendix 10 also apply to Maori Education.

The Panel noted
the emerging trend for students to move to and from immersion and mainstream
programmes. Anecdotal information suggests that the numbers of
learners in this category are significant and growing. The implications
of this flow needs to be examined, analysed and evaluated to ensure
that no barriers to learning are developing. A systematic collection
of data in this area is required to establish the extent and nature
of any problems, the risk factors which are apparent and how schools
and centres are currently responding as well as what may be required
in the future by students who are making these transitions.

The Panel makes
a number of recommendations which are believed to be first steps in
addressing some of the issues for Maori Education.

16 Home schooling

The Education
Act (1989) s.21 provides for parents to apply to the Secretary of Education
for long-term exemption from enrolment. The Secretary must be
satisfied that the 'person will be taught at least as regularly and
as well as in a registered school'. A registered school is a
State School, or is an independent school (to which the National Curriculum
Guidelines do not apply). The latter however, are required to
provide a suitable curriculum. To comply with these conditions
the Secretary must have evidence of the routines to be followed in the
home-schooling environment and the approach, planning and balance in
the curriculum adopted.

Since 1994,
when Government withdrew funding for reviewing home schooling, the Ministry
of Education notified all home schoolers that they must provide a detailed
written report annually on all aspects of their home schooling programmes.
Mathematics and Language and Reading are to be commented on fully in
1997 against the following questions:

  • What knowledge,
    skills and attitudes have you concentrated on during the year?
  • What activities
    have you used to build up the knowledge, skills and attitudes?
  • What progress has
    your child made and how has this been revealed?
  • Are there areas
    where progress was unsatisfactory and what do you plan to do about this?
  • What does your
    child enjoy most?

At present
some 5,500 young people of school age are being home-schooled.
Given this level of activity, a decision was made to reinstate home
schooling audits in the 1997-98 financial year, with 940-980 reviews
to be undertaken. These reviews are a form of Accountability Review
and will concentrate on quality of programmes, their effectiveness and
the educational outcomes.

Strong submissions
were made by home schoolers that the current restriction on access to
private homes should be retained. This has the disadvantage of
preventing reviewers from observing the educational programme being
provided unless access is volunteered by parents. It can also
mean that there is no access to the student which is more serious.
While the Panel accepts the prohibition on entry without consent to
private dwellings, the view was formed that, in the interests of the
student, entry should not be denied without valid reason. A reviewer
is, however, entitled under s.327(b)(ii) and (d) and (e) to meet with,
inspect the work of, and make copies of any work documents of a student.
The Panel believes that the Ministry should continue to require home
schoolers to provide documented evidence of curriculum provision through
their annual written reports on specified areas of knowledge.

17 School hostels

A number of
New Zealand schools own and manage student hostels and, so, they are
in effect 'boarding schools' where the hostel is regarded as an
integral part of the educational provision. At present, the Educational
Review Office has no powers to audit the hostel where it is possible
the safety of students may have a material influence on the quality
of their educational experience. While some independent schools
have requested the Education Review Office to include the hostel in
their review, some School Trustees have rejected the possibility that
the Education Review Office should have any interest in student accommodation.
The Independent Schools Council made a strong submission that the boarding
component of schools had been neglected; yet it is critical to the health,
safety and culture of a school. The Panel has reached the conclusion
that the Education Review Office ought to have powers of entry to hostels
where they are an integral part of the educational provision of the
school.

18 Special education

The Education
Act (1989) s.8 and 9 explicitly refer to the equal rights of those with
disabilities to enrol and receive education at state schools.
It was brought to the Panel's attention that negative attitudes to
children with disabilities persist in many schools and parents are often
dissatisfied both with the reception they receive when applying for
enrolment and the educational provision their children receive.
The Education Review Office refers to special education in its reports
but it remains clear that an in-depth review of school and centre policies,
enrolment procedures, education provision and its effectiveness needs
to be undertaken to assure parents and the community that the intentions
of the Act are being adhered to.

19 Services for students

The National
Administration Guidelines are explicit in their intention to ensure
that the nation's young people receive a balanced curriculum covering
the areas of knowledge, skills and attitudes detailed in the National
Curriculum Statements along with monitoring progress and achievement
and overcoming barriers to learning. They also require Boards
of Trustees to provide appropriate career information and guidance for
all students. To do so, Boards are expected to have strategies
and policies, appropriate to the background and age of the students
that include goals, processes and reporting procedures. Services
to students such as guidance, counselling, careers awareness and advice,
transition services and health and well-being advice are all important
aspects of school life and on which the Education Review Office should
comment in its reports.

20 The Corporate Office of the Education
Review Office

Questions have
been raised regarding the size and costs of the Corporate office of
the Education Review Office in Wellington. What at first sight
appears to be a top heavy structure emerges as a very lean operation
in practice. The Office has three Outputs with $14.871M (90 percent)
of its allocated budget for the 1997/98 financial year devoted directly
to Reporting Services, that is (Outputs 1 and 2) Accountability Reviews
in Early Childhood Services and Accountability Reviews in Schools and
other education providers.

The remaining
$1.603M is allocated to (Output 3) the Quality of Education Reports
and Services, which includes Evaluation Reports Policy Advice and Ministerial
Services.

The Corporate
Office includes the Chief Review Officer's Unit, Legal and Financial
Services and Evaluation Services. Also located at Corporate Office
are Te Uepu a Motu, the National Manager Reporting Services, and the
Development and Human Resources Unit. Many of these are services
to the organisation as a whole and are part of a strategy which brings
the management team together for greater efficiency and effectiveness.
Given this breakdown, it is clear that the Education Review Office is
an extremely lean operation. Nevertheless, if in the future the
Education Review Office were to be allocated an increase in funding
resource, then the Panel believes priority must be the appointment of
reviewers rather than expansion of the Corporate Office.

RECOMMENDATIONS:

    37. That
    there be no change in the arrangements for recruiting reviewers to the
    Education Review Office.
    38. That
    the Education Review Office focus on Maori Education in 1998 and publish
    an Evaluation Report on its provision and effectiveness in mainstream
    schools and centres as well as the entry and exit circumstances of students
    to Kura Kaupapa and Te Kohanga Reo.
    39. That
    the Statements of Performance Indicators proposed in this review include
    targets for Maori education and education of Maori.
    40. That
    the Education Review Office ensure that review teams include a speaker
    of Te Reo wherever appropriate and, if not possible, a reviewer with
    an appreciation of Tikanga Maori.
    41. That
    the Education Review Office work with Te Puni Kokiri to establish an
    appropriate protocol for the review of the education of Maori and Maori
    education.
    42. That
    home schooling caregivers be reviewed on an ongoing basis and continue
    to be required to provide a written annual report to the Ministry of
    Education as the exempting authority.
    43. That
    the Education Review Office prepare protocols for the audit of home
    schooling providers, including access to the learner and the learning
    environment, where caregivers are willing to give access and make them
    available to those included in the review each year.
    44. That
    the Education Act be amended to give the Education Review Office powers
    of entry to school hostels where these are an integral part of the education
    provision of the school.
    45. That
    the Education Review Office take particular note in its Accountability
    Reviews of the provision of education for students with disabilities
    in schools and centres.
    46. That
    the Education Review Office focus on Special Education in 1998/99 and
    produce an evaluation report on the provision and practices in relation
    to students in schools and centres.
    47. That
    the Education Review Office include and comment on the range of guidance
    services available for students and their effectiveness in its reports
    on schools.
    48. That
    any increase in funding for the Education Review Office should be allocated
    to the appointment of additional qualified reviewers.

TERM OF REFERENCE 4

METHODOLOGY FOR EVALUATING THE IMPACT OF POLICY
AND DELIVERY OF SERVICES OF OTHER STATE EDUCATION AGENCIES

The Mission
Statement of the Education Review Office is "High quality evaluation
contributing to high quality education." It follows that, to
fulfil this mission, the Education Review Office should sit at the table
when strategies for education are being developed. The feedback
which the Education Review Office can provide will be based on its knowledge
of and experience in the field, its links with a wide range of educational
agencies (the Education Review Office consulted with 31 different agencies
and working parties in 1996/97 and, in the first six months of 1997,
with 16 major education stakeholders) and its independent Evaluation
Reports.

<1 Improving communication

The Ministry
of Education recognises the importance of the Education Review Office's
contribution and acknowledges that there have been communication problems
in the past. Policy development and implementation depends to
a large extent on the effective use of people and resources. That
there has been 'stand-off' in the past is regrettable and may have
been the result of the separation of the Ministry of Education from
the education service agencies like the Education Review Office which
were part of the Tomorrow's Schools development. There has been a
lack of accountability to one another. Communication was not well
developed and resulted in poor relationships. These may have been
inhibiting factors in making good use of people with a wealth of knowledge,
experience and expertise from the field and who could make a useful
contribution to policy development. The Ministry of Education
was perceived by the Education Review Office, Boards and Principals
as being unreceptive and unresponsive.

It may be that
the linkages between the independent agencies were inadequately detailed
in the Education Act (1989). Given the emerging goodwill and evidence
of better communication and involvement in policy development, the Panel
suggests that the greater co-ordination and coherence within the education
sector which has always been necessary, is now possible. Contributing
factors to this improvement have been the appointment of a Minister
responsible for the Education Review Office, separate from the Minister
of Education, regular combined meetings of Chief Executive Officers
together and with the Minister of Education and Minister responsible
for the Education Review Office, a conscious acceptance of the
Education Review Office's ability to alert the Ministry to real and
potential problems and the establishment of the Advisory Council on
Quality in Education by the Education Review Office.

The Panel has
already called for a communication strategy to be developed between
the Education Review Office and the schools and centres. Expanding
this strategy to include the Ministry of Education and Te Puni Kokiri
and other agencies such as the Teacher Registration Board, New Zealand
Qualifications Authority and Children and Young Persons Service would
be welcomed by educators.

2 Policy development

The Education
Review Office must remain as an independent Department of State actively
making inputs into policy through formal consultation with the Ministry
of Education. Further, the Panel is satisfied that initiatives
which have been taken already will have the effect of ensuring that
protocols are in place and that formal and informal lines of communication
are open and operating. It is the Ministry of Education which
has the primary responsibility for policy development and its implementation
once the approval of Cabinet is given. It is incumbent, therefore, on
the Secretary for Education to continue to develop the consultation
protocols with the agencies delivering education services or which have
a direct impact on education. These include Te Puni Kokiri, the
Education Review Office, New Zealand Qualifications Authority, the Early
Childhood Development Unit, Te Kohanga Reo National Trust, the Education
Training and Support Agency, the Careers Service, the Special Education
Service, the Teacher Registration Board, the Colleges of Education through
the Advisory Services, Children and Young Persons Service and the Ministry
of Health.

The Panel believes
that the Crown and other stakeholders need to be assured that consultation
occurs among these agencies and with the Ministry of Education in the
development and implementation of policy.

As the evaluator,
the Education Review Office has no direct or primary role in designing
or implementing the law, regulations and official education policies
against which it must review schools, centres and other educational
providers. Its input is confined to information, advice and consultation;
this is appropriate, given the Office's function.

3 Channels of communication.

The Education
Review Office has direct lines of communication with the Government
through its Minister and with the Ministry of Education. The Minister
may direct the Office to initiate action or to produce an Evaluation
Report on some specific topic. The Office's reports to the Ministry
may result in action being initiated as in the case of the Mangere-Otara
project. The open lines of communication with other agencies should
result in increased confidence among Boards, Principals, teachers, parents,
the community and the education interest groups.

The Education
Review Office has had a significant impact through its reports on schools
and centres which are all sent to the Ministry of Education. Evaluation
Reports, Annual Report to Parliament and comment from time to time on
issues which are emerging as a result of its work are all conveyed.
The Education Review Office draws the attention of the Minister to areas
which may become priorities for work to be done.

4 Significant achievements

Over the last
seven years respect for the Education Review Office has grown.
The Office is acknowledged as an independent evaluator with integrity
and with a staff of committed people. Some important achievements
include:

  • Its significant
    impact in ensuring that strategies were developed in 1996 to address
    the pressing and long-standing issues in Mangere and Otara schools,
  • It was instrumental
    in the development of the Performance Management Systems in schools
    following its report on Managing Staff Performance in Schools in 1995,
    and
  • Arising out of
    the Office's concerns about Boards of Trustee expertise, the Ministry
    has contracted trainers to work with Boards in specific areas of need.

5 Emerging issues

Recently, the
Education Review Office has drawn attention to emerging issues which
include:

  • The quality of
    professional leadership in New Zealand schools and the importance of
    training as a pre-requisite for principal and senior management positions,
  • The exemption
    of teachers in Kura Kaupapa Maori from compulsory registration or carrying
    a Limited Authority to Teach,
  • The potential
    for the Limited Authority to Teach to be used inappropriately in a time
    of teacher shortage, and
  • The teacher supply
    problem and parent and community demand for high quality teaching.

Other questions
to which attention is drawn are recorded in the Education Review Office's
Annual Report (1997) and include:

  • What are regarded
    as core competencies of capable teachers?
  • What do policy
    makers regard as a 'balanced curriculum'?
  • What are the critical
    stages of students' learning and how might these best be assessed?
  • What are regarded
    as tolerable levels of inconsistency in students' school experiences?

In identifying
these issues and questions, the Education Review Office is exercising
the function of 'diagnosis' of problems in the system. It
is outside the scope of the Review to do anything other than list these
issues in the expectation that there will be appropriate reaction from
the Minister and the Ministry of Education.

6 Extension of system diagnosis function

The Panel believes
that the diagnosis function should be extended to identifying the causes
of performance problems in schools and centres and on which the Education
Review Office would be expected to report. This is quite separate
from the Education Review Office having a role in providing advice and
guidance or in enforcing compliance which are correctly included in
the functions of the Ministry of Education.

<7 Two issues of concern

Early childhood centres - short notice visits

The Early Childhood
Sector has drawn to the Panel's attention the problems which have
emerged due to both the Education Review Office and the Ministry of
Education being involved in short notice visits commonly called spot
checks of Early Childhood Centres. There is universal support
within the sector for spot-checking to continue. The Panel was
given examples of centres who, when notified of a visit, will transfer
equipment and staff to the centre for the duration of the visit in order
to get a good report. The Education Review Office claims that
they were not briefed prior to the implementation of the policy by the
Ministry while the Ministry says it is liaising with the Office at the
local level. The Ministry report that its spot checks were instituted
because of a major risk of non-compliance with the Regulations or Charters
due to the Education Review Office extending out its cycle of visits
from three to four years. This 'double checking' has created
confusion and annoyance within the sector because the practitioners
are unaware of the motivation for this action. While the Ministry
has responsibility for issuing licences and approving Charters, it is
encroaching on the role of the Education Review Office in instituting
spot checks without consultation and direct liaison with the Chief Review
Officer. The Ministry concede that, because the Education Review
Office has increased its activity in the Early Childhood Centres, (935-980
reviews in 1997/98 as opposed to 610-650 in 1996/97) the need for their
involvement is no longer necessary.

Assessment in schools

The school
sector has reported its confusion over the expectations of assessment
that are part of the National Curriculum Statements. Submissions
have been made that the Education Review Office expects each objective
to be assessed and records kept of student achievement against all objectives.
The Ministry of Education has stated (New Zealand Education Gazette
March 1995) that 'it is concerned that assessment does not dominate
the teaching and learning process, rather it should be an integral part
of teaching and learning.' They say, 'The schools should aim
for assessment procedures which are manageable for teachers, non-intrusive
for students and focused on promoting learning.'

The Ministry
of Education advocates the 'key' as being for 'schools to plan
in advance what they intend to report on and how they might best collect
and aggregate the information.' Too many teachers are engaged
in over-assessment and the planning advocated by the Ministry is frequently
not in evidence.

Assessment
is an integral part of teaching and learning and, therefore, must be
included in every teacher's learning plan. It is the Panel's
view that the process it has recommended, of a 3-year Strategic Plan
and an annual Statement of Performance Indicators, will greatly assist
teachers to plan, monitor and report effectively against stated targets.
With these benchmarks in place and the Education Review Office evaluating
against them, there will be greater certainty of expectation along with
manageable assessment practices and reduced workload for teachers and
principals.

RECOMMENDATIONS:

49. That the Education Review Office and other agencies are to be commended on
their improved working relationship.

50. That protocols be developed between the Ministry of Education and the Education
Review Office and Te Puni Kokiri that will establish relationships,
functions and involvement in policy development which will survive changes
in personnel and operate effectively on a continuing basis.

51. That the Education Review Office continue to have a direct input into policy
development through the Ministry of Education.

52. That the Education Review Office include diagnosis of the causes of performance
problems in its review reports on schools and centres.

53. That the Ministry of Education withdraw from spot-checking in the early childhood
sector.

54. That the Education Review Office continue its practice of spot-checking early
childhood education centres and reporting any non-compliance to the
Ministry of Education for action.