What is external evaluation and review?

External evaluation and review (EER) is the name given to the periodic review of tertiary education organisations (TEOs), conducted by NZQA. It is one essential element in NZQA’s Evaluative Quality Assurance Framework.

Every EER produces a report on the relative quality of a TEO. These reports are published on the NZQA website.

1.1 The context of EER

Purposes of EER

EER has two interlocking goals: TEO accountability and TEO improvement.

EER holds each TEO accountable for the quality of its performance, on behalf of its learners and the community at large. As such, EER is a regulatory instrument.

EER also enables TEO improvement. Every EER offers an independent view of what is working well and where there are further opportunities for positive change. In addition, the growing store of online EER reports enables TEOs to learn from the practice of others.

Other public benefits accrue from EER. For example, over time the findings of EER reports help inform government policy. Analysis of reports to date has already identified trends, gaps and progress in important areas (such as Māori learner achievement).

The TEO sector

EER encompasses a wide range of organisations. Most recognised or registered TEOs participate. The notable exception is the university sector. Universities are periodically reviewed through the Academic Quality Agency for New Zealand Universities.

Types of TEOs undertaking EER include:

  • subsidiaries of Te Pukenga
  • wānanga
  • government training establishments
  • private training establishments.

TEOs are notable for their diversity. They range in size from the very small to the very large. They cover all types and levels of education or training, from foundation programmes to degrees. They encompass many subjects, industries and fields of study.

Learning may occur on-job or off-job, in the classroom, online, or through practicums. Students can be local (domestic) or international, young or old.

What can you expect from EER?

Every EER is guided by a set of operating principles.

EER aims to be fair and balanced. NZQA needs to ensure that its judgements are defensible, and that its own quality standards have been properly followed.

TEOs ‘own’ the quality of their education and training. NZQA cannot ‘police in’ success at a TEO. That is the TEO’s duty. A high-performing TEO assumes full responsibility for the effectiveness of its own processes and outcomes.

TEO self-assessment is primary. EERs glimpse a TEO’s performance for a short period, and over relatively long intervals. The TEO, presumably, knows itself better than any outsider can, and uses its organisational self-assessment for the benefit of its stakeholders.

Reciprocity. In every EER, NZQA undertakes to work in a professional manner with the TEO. NZQA expects the TEO to do the same. Both parties should engage collegially. Both should do all they reasonably can to manage the engagement efficiently and openly.

EER can never take a one-size-fits-all approach. NZQA acknowledges the distinctive features of the TEO under review, and the sector or environment in which it operates. Every evaluator tailors their practice accordingly.

Types of EER

There is a ‘standard’ model of EER which is explained in this guide. This model is flexible enough to respond to TEO diversity.

For some sectors, however, additional or modified resources are needed. The English language sector, for example, is supported by specialist guidelines.

In one field, however, a wholly distinct approach has been developed. TEOs who wish to have their performance reviewed within a specifically kaupapa Māori framework engage with NZQA by means of Te Hono o Te Kahurangi EERs.

1.2 What is evaluation?

Evaluation is the act of judging the merit, value or worth of something.

The object judged could be an action, a process, an object, a person, or an organisation.

Evaluation is a common human activity. We do it all the time, in every walk of life. For example, the act of preferring one brand of soap over another is itself evaluative. We may be guided by considerations of cost, quality or accessibility. But when we decide to put that particular brand of soap in our shopping basket, we have made an evaluative decision.

Evaluation in practice

We watch a game of netball. When the game finishes, we turn to a friend and say, ‘Lee played really well tonight’. What does this single act of judgment involve?

Clearly, it is based on specific examples of Lee’s activity on the court. We saw many instances of Lee passing and (maybe) shooting. We believe that Lee followed the rules throughout.

More than that, our experience of watching many netball games gives us a series of useful reference points to evaluate performance. Shooting a goal, for example, is always desirable – and Lee shot goals. We noted too that Lee’s accuracy throughout the game was better than ‘normal’. Of course, our view of Lee’s performance may change. Replays of the game may indicate a number of unforced errors that we missed at the stadium (‘Lee was actually inconsistent’). Or this game may prove to be the single highlight of Lee’s season (‘everything considered, Lee was pretty poor this year’).

But on the night, we believed that we had enough information on the better aspects of Lee’s performance to make a more positive judgment, with reference to our standards of what ‘good’ looks like.

EER as applied evaluation

Evaluation as an applied practice follows much the same steps – though to a more sophisticated degree – across a range of disciplines. Every evaluation:

  • works within a framework of rules
  • identifies and gathers information
  • analyses and interprets that information
  • produces a series of value judgments (‘good’, ‘poor’, etc.) on aspects of performance
  • expresses an overall view of the quality of the subject being evaluated (the ‘evaluand’)
  • communicate these findings.

These common evaluative features guide our practice in EER. They also inform the shape and content of these guidelines.

The building blocks of EER

Our Rules are the regulatory settings which enable tertiary education to occur. They comprise legislation (such as the Education and Training Act 2020), NZQA’s quality assurance Rules, and many sector-specific or industry requirements. Taken together these Rules tell us what is allowable. They underpin policy. They clarify which outcomes are desired and which are discouraged.

Our enquiry methods teach us how to be understand context. We already know a lot about TEOs of ‘this kind’. We often have a good deal of information on a specific TEO’s prior performance before the EER begins. Enquiry methods also teach us how to identify, gather and understand relevant information.

Our judgement tools assist us to reach defensible ratings. In EER there are two primary sets of tools: the Tertiary Evaluation Indicators (PDF, 1.1MB) (indicators) and the Rubrics (PDF, 79KB) (refer Appendices).

The indicators are merit criteria. They constitute a database of what ‘good’ often looks like in tertiary education, based on research and experience. For example, the indicators remind us that motivated students tend to learn better. A TEO in which the students are motivated has at least one of the features of high-quality performance.

At each phase of an EER, evaluators consult the Indicators as points of reference. The indicators direct our attention to what matters most, and why. The Tertiary Evaluation Indicators can be found on the NZQA website.

The rubrics are performance criteria. They help us decide where the performance of an organisation or a programme or a topic lies on a quality scale. Is performance good, bad, or something in-between?

In making judgments on quality, NZQA evaluators aim to align their findings with the descriptors for each of the relevant rubrics.

Our communication extends throughout the EER process. TEOs participating in an EER are repeatedly kept informed of what is happening, and why. They are alerted if there are changes to the process and timelines. They are advised by means of the draft EER report when outcome decisions are reached following the on-site visit. The EER report is the most important means of communication, and the final version is published on the NZQA website.

Conclusions

  • Evaluation is a common human activity. It has also developed over time as a formal practice, embedded across many disciplines.
  • EER is the specific application of the evaluation method to tertiary education in New Zealand.
  • EER is always guided by background information and the context of the education or training under review.
  • EER uses standard methods of inquiry to gather and understand the relevant evidence.
  • In making judgments, EER refers to standardised notions of merit and performance.
  • EER findings culminate in reports published on the NZQA website.

1.3 What EER covers

In scope

All EER reports include two statements of NZQA’s confidence in a TEO. One statement covers educational performance; the other, the TEO’s capability in self-assessment.

Educational performance means the relative quality of the outcomes achieved by a TEO as a whole, on behalf of its learners and community. It also takes into account the key supporting processes of the TEO and the resources it holds.

Some common features of high-quality performance are:

  • high learner achievement rates
  • strong pastoral support
  • good educational practices consistently used within the TEO
  • effective management of compliance responsibilities

Capability in self-assessment refers to the TEO’s relative effectiveness in understanding its own mission (or kaupapa), the needs of its learners and other stakeholders, taking appropriate actions to meet those needs, and how it responds to change. It includes data-gathering and evidence-based improvements resulting from the same.

Some common features of effective TEO self-assessment are:

  • timely and comprehensive programme reviews
  • ongoing engagement with industry (or community)
  • improvements to design, delivery or provision as a result of stakeholder feedback
  • proactive self-management of ‘risk’.

Out of scope

Although every EER aims to be rigorous in its findings, especially with respect to the quality of education and training at a TEO, the EER system has certain inevitable limitations in its coverage.

EER is time-bound. It provides assurance based on information available at a point in time. Policies, markets and TEOs themselves change, sometimes rapidly. A TEO rating from four years ago may not match the reality of the present moment.

There are limitations of scope. EERs are restricted in terms of time and money. We look at some programmes closely and may look less closely at others. Sampling exercises may miss key data (although our system is designed to minimise such omissions).

EER is not designed to detect organisational fraud. Fortunately, this is a relatively rare event, and NZQA has other means of identifying fraud when it does occur.

EER cannot be used to predict the outcomes of other reviews of the same TEO which, by posing different questions or examining different information, could reasonably arrive at different conclusions.

1.4 Overview of the EER process

An EER runs from the moment it has been scheduled and first contact made until all work has been completed and a report is published.

Every EER aims to gather enough information to reach an accurate view of the current overall quality of a TEO.

The following EER phases are the norm. They will usually occur as described and in sequence, whatever the TEO under review.

Under special circumstances, however, NZQA may need to change the process or timing of one or more of the phases. Whenever this happens, NZQA will give the TEO advance warning and explanation, and invite a response (as appropriate).

EER phases

  1. Scheduling: NZQA will advise a TEO of when it will conduct enquiry for their next EER. NZQA will allocate a team to conduct the EER.
  2. Scoping: NZQA and the TEO will work together to confirm what will be evaluated in the EER, and how.
  3. Enquiry: NZQA will collect or validate information on the performance and capability in self-assessment of the TEO. This will usually include spending time at the TEO itself.
  4. Judgement: NZQA will reach views on the quality of the TEO. These will take the form of findings, ratings and statements of confidence.
  5. Reporting: NZQA will issue draft and final reports on the TEO. The reports will explain the reasons for NZQA’s judgment, and the TEO is invited to provide feedback on the draft report.
  6. Closure: all final EER reports are published on the NZQA website.
  7. Categories: once an EER report is published, a TEO category will be assigned. The category derives from the two statements of confidence. Every category bestows incentives or sanctions on the TEO.

EER timelines

Timelines should be reasonable, follow due process, and be responsive to the principle of natural justice.

For indicative timelines, and more details on the step-by-step process of an EER, see The process of external evaluation and review.

Links to other sections of the guide

How does EER begin?

How are EERs planned?

How does EER enquiry occur?

How are EER judgements made?

How are EER findings reported?

What happens next?

The process of external evaluation and review

Back to External evaluation and review

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz