Fourth Report of the Tertiary Education Advisory Commission - full report 38/58

Shaping the Funding Framework
Fourth Report
of the Tertiary Education Advisory Commission

Chapter 12: Accountability for Quality and
Performance

Quality lies at the heart of the Commission's desire for a tertiary education
system that ensures equitable access to lifelong learning and enables the
achievement of excellence in teaching, learning and research. In Shaping the
Strategy, the Commission noted that the current funding and regulatory
frameworks, do not provide 'sufficient direction, rewards, or support for
quality improvement'.²⁵⁹ Furthermore,
several submissions²⁶⁰ to the Commission
expressed a desire for greater accountability on the part of institutions for
their performance against their core outputs, and more consistent accountability
measures across the system.

Chapters 10 and 11 of this report focused on improving the quality of
research in the tertiary education system. This chapter examines the quality of
tuition and service delivery.

The Commission recognises the achievements of the last twelve years in
assuring the quality of tertiary education delivery. As noted in Chapter 3,
however, current funding arrangements include only weak incentives for
maintaining and improving the quality of teaching and learning environments, and
the Commission therefore believes there is scope for enhancing performance
accountability. This is particularly important if New Zealand wishes to attract
more international students. The Commission is conscious, however, that
quality-assurance processes should not cause providers to focus on simply
meeting specific requirements of the process at the expense of providing
high-quality teaching and research.

The current quality-assurance systems are described in Appendix 3. Under
these systems, the criteria for the approval of all degrees are the same, but
criteria for sub-degree programme approval and accreditation vary. For example,
those used by the New Zealand Qualifications Authority (NZQA), differ from those
used by the New Zealand Vice-Chancellors Committee (NZVCC).²⁶¹ Similarly, the New Zealand Polytechnic
Programmes Committee (NZPPC)²⁶² and the
Colleges of Education Academic Committee (CEAC)²⁶³ use the same criteria as NZQA for programme
approval and accreditation, but have different approaches to audit.²⁶⁴ Furthermore, the NZVCC does not have a
systematic approach to provider accreditation.

The Commission's proposed funding framework will remain largely demand
driven. If such a system is intended to provide appropriate incentives for
increasing the quality of teaching, then it is essential that learners are able
to access adequate information about the quality of courses, programmes and
providers. The Commission discusses this issue further in Chapter 15.

12.1 Using the Funding Framework to Promote
Quality

There are three key steps in determining how the framework for tertiary
funding can be used to promote quality. Firstly, it is necessary to define
precisely what constitutes quality and how it relates to the quality test used
to assess charters and profiles. Having done this, consideration must be given
as to what indicators would be used to measure and determine acceptable levels
of performance on an ongoing basis. Finally, there is a need to examine whether
sanctions and/or rewards should be built into the funding framework to ensure
acceptable performance and to achieve desired outcomes.

12.1.1 Improving the Quality of Academic
Staff

In addition to using funding as a lever to maintain and promote quality, the
Commission also recognises that a key way to enhance quality in teaching is to
ensure that academic staff attain basic competency in teaching and assessment.
The clearest way of achieving this is through requiring academic staff to be
trained as educators and assessors. For example, England has recently introduced
a requirement that all new teachers in tertiary institutions are trained and
accredited.

There exist a range of options for improving teaching and assessment quality
amongst academic staff. These include:

• requiring all academic staff to be registered teachers (as in the compulsory
  education system);

• requiring all academic staff to go through a basic level of training in
  teaching and assessment techniques;²⁶⁵ or

• requiring all institutions to have in place an effective in-house training
  system as a pre-requisite for accreditation or course approval.

Requiring academic staff to be registered would require the establishment of
approved qualifications and standards for registration along with a registration
board. While this is a worthy long-term aim, it is not achievable in the
short-term. Similarly, requiring all academic staff to go through training would
require wide inter-sector consultation and negotiation with staff unions.

The Commission believes, therefore, that priority should be given by
quality-assurance agencies to developing requirements for assuring the skills
and knowledge of academic staff in teaching and assessment, as part of programme
approval and accreditation processes. The Commission further believes that the
funding framework should have mechanisms to support the building of teaching
capacity amongst all academic staff, particularly with regard to the needs of
groups such as Maori and Pacific peoples (who are traditionally underrepresented
in the tertiary education system). The Commission has already recommended in
Chapter 5 that funds from the SDF be available for this purpose.

Recommendation 53

The Commission recommends that the quality of teaching in tertiary providers
be addressed as a priority issue with consideration being given to the
introduction of a requirement for new academics to undertake some teaching
training, and assistance being made available for those already in the tertiary
education system to move towards attaining competency and basic teaching skills,
within a specified period.

12.1.2 The Quality Test

The Commission has already stated that it wishes to see all providers,
programmes and activities meet quality criteria before public funding is made
available. Any decision not to fund a provider on quality grounds will clearly
need to be defensible, and this will require some modification of existing
quality-assurance and audit systems. In particular, it will be necessary to
determine appropriate quality thresholds to apply to accreditations and audits
which must be passed to gain funding. These criteria may vary for each component
of the tertiary education system. The Commission has called this process of
determining which level of quality it will fund, the quality test.

Introducing the proposed quality test will clearly require the setting of
specific quality benchmarks for levels of programme, delivery modes, and
provider type against which the TEC is able to make relevant judgements. The
Commission believes that it would be possible (and even desirable) to use these
benchmarks to set different threshold criteria for different types of providers
and it has examined existing audit processes with in this in mind.

Current NZQA and NZPPC audit reports contain requirements (issues that must
be resolved before teaching can commence) and recommendations (suggestions for
improvement). Requirements technically prevent a programme from being delivered,
but in reality providers are given a period (usually around three months) to
correct identified problems. These audits are also used for PTEs seeking
registration. To gain initial registration, PTEs must undergo an audit. If this
results in any requirements, registration is denied.

The period for which the quality audit is valid is determined on the basis of
the number of requirements and recommendations that are contained in the audit
report. The current maximum period before another audit is required is three
years, but the majority of providers undergo re-auditing before the end of this
period.

The current academic audit of universities (administered by the AAU) does not
have any mechanism for withholding quality approval, and there is no
accreditation process in place for universities. While there is a moral
obligation to address the issues raised in an AAU audit, there are no sanctions
available to enforce the Unit's recommendations.

The Commission believes that the audit systems operated by NZQA and NZPPC
provide a firm basis for qualitytest criteria. Under the current systems, a PTE
could be required to pass an NZQA Quality Assurance Standard 1 audit with a
one-year reassessment period before funding is awarded. Setting a one-year
re-audit period ensures the provider is of sufficient quality to deliver its
programmes for the period over which funding is awarded. The Commission
believes, however, that this standard should be raised progressively with the
aim of requiring that by 2005 all PTEs accessing public funding have a two-year
timeframe for re-audit.

For providers audited by NZPPC, the threshold criteria for quality could be
initially set at a specified standard with a two-year timeframe for
reassessment. This recognises the greater diversity of programmes offered by
these providers, particularly multi-year programmes at degree and diploma
levels. To support continuous quality improvement, the Commission wishes to see
this standard progressively raised.

Using threshold criteria for quality in this way is not entirely suitable for
universities, however. The AAU's audit systems are looser than those of other
quality-assurance bodies, and are not designed to result in direct sanctions.
Consequently, they do not provide an adequate basis for setting quality-test
criteria. This is an issue that urgently needs attention. The Commission is
aware of, and endorses, the current review of criteria for course approval and
accreditation being undertaken by inter-institutional quality-assurance bodies.
The Commission urges these bodies to work with the TEC to ensure that
appropriate approval, accreditation and audit processes are in place for all
providers (particularly universities), and that appropriate threshold criteria
are developed which can be used by the TEC to withhold funding on the basis of
insufficient quality.

It should also be noted that the Commission's proposals for the future
funding of post-graduate programmes (see Chapter 11) would act as a de facto
quality test in terms of the role of universities (and some other providers) in
providing education at this level.

Recommendation 54

The Commission recommends that no funding be extended to private tertiary
providers that fail to gain at least a one-year reassessment on their New
Zealand Qualifications Authority Quality Assurance Standard Audit.

12.2 Defining 'Quality'

Any scheme of rewarding or promoting quality, must be based upon a clear
notion of what constitutes 'quality'. Perspectives on what quality means can be
organised into three broad approaches:²⁶⁶

• quality as excellence;
• quality as value for money; and
• quality as fitness for purpose.

12.2.1 Quality as Excellence

From this perspective, quality is a comparative attribute determined in
relation to similar qualifications, providers or activities. Those providers,
programmes and activities scoring comparatively highly on a determined scale are
judged as being excellent and therefore of high quality. Establishing a quality
system on this definition would require the development of objective benchmarks
or standards against which qualifications, providers and activities could be
assessed and ranked. A version of this approach has, up until recently, been
used in the United Kingdom for Teaching Quality Assessment (TQA) in 'higher
education's(through the Quality Assurance Agency) and 'further'sand 'workplace's
education (through the Adult Learning Inspectorate and Office for Standards in
Education).

The key advantage in using an excellence definition is that it provides clear
and easily comparable information about the performance of providers and
qualifications. This can provide significant incentives for quality improvement,
particularly given the definition's potential influence on learners'schoice of
provider if it is made public.

There are, however, several problems with this approach. The first relates to
its expense. Maintaining a robust and 'up-to-date'sassessment system would
require both frequent updating of the standards and benchmarks, and an external
inspection organisation.

Secondly, concerns have been raised about the suitability of such a
standardised system for all providers and programmes. Many providers and
analysts complain that the system fails to take account of the specific context
and goals of particular institutions.²⁶⁷

Finally, there exists a possibility that using a 'quality-as-excellence's
definition may encourage perverse behaviour on the part of providers. For
example, using learners'scourse marks as an indicator of high-quality teaching
may cause providers simply to lower standards in order to score better on the
objective assessment process. Although all systems of performance assessment
have the potential for this, a quality-as-excellence definition encourages it
more than other definitions because of its emphasis on universal and
standardised assessment.

12.2.2 Quality as Value for Money

Value-for-money definitions of quality focus on customer and stakeholder
perceptions of whether a programme meets or exceeds expectations in relation to
the time and money invested. This approach implicitly underlies the funding of
Industry Training in New Zealand, where government subsidies can only be
accessed once industry has made a cash contribution.²⁶⁸

Depending on how it is implemented, a value-for-money approach can encourage
provider responsiveness to stakeholders (such as employers and learners). For
example, some element of funding could be made dependent upon demonstrating that
students were satisfied with the quality of their programme. This potential for
clear and direct relationships between standards and sanctions is the chief
advantage of this definition of quality.

A significant problem exists, however, in determining which stakeholder (or
set of stakeholders) should be consulted as part of evaluating quality -
especially given their differing perceptions of what constitutes value for
money. For example, some employers may wish to see programmes provide only
specialised work-focused skills, rather than generic and transferable knowledge
and attributes. On the other hand, some learners may view a lighter workload and
easier assessment as key indicators of 'quality'sin this regard.

Furthermore, this approach essentially treats education as a form of
investment in which returns are clear. In practice, this is not the case - and
the benefits of a particular programme often take some time to manifest.
Consequently, stakeholders may not be able to judge educational quality
accurately.

That said, the inclusion of stakeholder views is, and should be, an important
element in all forms of quality evaluation. Both internationally and in New
Zealand, most providers use surveys of student satisfaction as part of their
annual reporting against Statements of Service Performance. Furthermore,
value-for-money decisions are made by learners in the EFTS system and by
industry in the industry-training system, with regard to acceptable fee levels
and the level of industry contribution respectively. Issues surrounding fee
setting are discussed further in Chapter 8.

12.2.3 Quality as Fitness for Purpose

Fitness-for-purpose definitions of quality assess the performance of a
qualification, programme or activity against its stated outcomes or intentions.
Broadly speaking, this is the approach currently used by most tertiary education
quality-assurance agencies in New Zealand.

Depending on how they are implemented, fitness-for-purpose approaches have
the potential to support high levels of innovation and responsiveness, since
they allow for quality assessment to take account of the particular
characteristics and aims of individual providers.

Given the diversity of providers and potential programme-delivery methods in
the tertiary education system, however, such a definition may make it difficult
to accurately compare and contrast similar qualifications or providers with any
degree of accuracy. Consequently, it is important to have common criteria
underpinning approval, accreditation and audit systems.

12.2.4 The Commission's View

In considering appropriate definitions of quality, the Commission is aware of
the trade-off between minimising transaction and compliance costs, and ensuring
that sufficient rigour exists in the system(s) used for quality assurance.
Although a 'quality-as-excellence'sdefinition is seen to have the potential to
increase rigour, the costs associated with such a direction would be excessive.
Furthermore, there is concern that adopting this definition could cause perverse
activity on the part of providers.

With regard to the 'value-for-money'sdefinition, the Commission recognises
that it is important for the views of students, industry and the wider community
to be recognised in systems designed to measure performance. It considers,
however, that making this definition the fundamental basis of performance
measurement systems would be impractical given the diverse and conflicting
desires of different stakeholders.

The Commission consequently believes that defining quality as 'fitness for
purpose'srepresents the most acceptable and workable approach. This definition
is flexible enough to allow for differentiation between providers, yet retains
overall coherence. The Commission has also noted arguments that all definitions
of quality can be included under the fitness-for-purpose
characterisation.²⁶⁹ Whether the aim is
perfection, value for money, or transformation, 'quality'srefers to how
effective a provider is at fulfilling that purpose.

---

Footnote(s):

259

Shaping the Strategy, p. 19.

260

New Zealand University Students'sAssociation (2000); Aotearoa
Post-compulsory Student Union (2000); and Industry Training Federation (2000).

261

The NZVCC has two quality-assurance bodies: the Committee on University
Academic Programmes (CUAP) which approves university programmes and the Academic
Audit Unit (AAU), which is undertakes quality audits for universities.

262

The quality-assurance agency for polytechnics.

263

The quality-assurance agency for colleges of education.

264

CEAC are in the process of developing their audit criteria and processes.

265

For example, all tertiary teaching staff could be required to possess a
Certificate in Adult Teaching or similar qualification.

266

See, for example, Education Directions (2000).

267

See, for example, The Guardian (1999).

268

All Industry Training programmes must also meet NQF standards and be
delivered by accredited providers.

269

Woodhouse (1995).