How are EERs planned?

The planning process for EERs begins with scoping, which determines which aspects of a TEO’s performance will be given primary attention during the EER.

3.1 The scoping process

NZQA must first assess the size, complexity and nature of the TEO under review. This assessment is guided by reviewing:

  • information held on NZQA databases
  • information made available by other Crown agencies and standard-setting bodies
  • the TEO’s own self-assessment.

Of these three sources, the TEO’s self-assessment is the most important. The TEO knows its own business more fully than any outsider could. Through its self-assessment, the TEO should be able to demonstrate ‘what matters most’ to its learners and other stakeholders.

The TEO’s scoping submission will always include, as a minimum:

  • a self-assessment summary
  • orimary evidence
  • a TEO Details form
  • an updated Improvement Plan (for Category 3 and 4 TEOs)
  • a Code of Practice self-assessment (for Signatories to the Education (Pastoral Care of International Students) Code of Practice 2016)
  • self-analysis of performance and outcomes.

After receipt of the TEO’s self-assessment material, the Lead Evaluator will discuss the scoping options with the TEO. The discussions will concentrate on what should be covered, when and how, and with what resources.

If NZQA has any significant concerns about the TEO, these will be disclosed before the scope is set.

Discussions on scope may proceed by email or phone contact, or face-to-face meetings, as seems most reasonable and practicable.

Confirming the scope

In addition to the submissions from the TEO, NZQA will also review information held internally or received from third parties. Such information can:

  • supplement or correct TEO data
  • offer an interpretation of the TEO’s performance that is different from the TEO’s
  • identify new areas of priority or risk.

Once all key information has been collected and analysed, NZQA will proceed to set the scope. In this context, ‘scope’ covers three linked decisions:

  • What NZQA’s inquiry will concentrate on (i.e. the focus areas)
  • What resource will be required (the team, and the cost)
  • How the inquiry process will be managed (agenda and plan of enquiry).

All scopes are put through rigorous review to ensure they are accurate, correctly sized and consistent with recent EERs of similar TEOs.

Mandatory focus areas

Occasionally, NZQA will set mandatory focus areas. These apply for limited periods across all TEOs and within specific sectors, as NZQA deems best. NZQA will advise TEOs of upcoming mandatory focus areas before they are set in any given EER.

The confirmation letter

The completion of the scoping phase is marked by the release of a confirmation letter. This document includes a list of the focus areas and gives an overall estimated cost for the EER. A draft agenda is also attached if there is an on-site component to the enquiry.

Focus areas can be either ‘horizontal’ or ‘vertical’. If horizontal, a focus area runs across much or all of a TEO’s operations. ‘Pastoral care of second-chance learners’ would be an example of a horizontal focus area. If vertical, a focus area looks in depth at a programme, field or faculty. ‘Diploma in Accounting’ would be an example of a vertical focus area.

Cost is a by-product of resource. That is, based on our previous experience of similar cases, NZQA will determine how much time is likely to be required to complete the EER. NZQA will also ensure that appropriate people are allocated to the case.

Agendas are developed for all EERs that involve on-site enquiry. They anticipate who will be required, when, where and why, throughout the period of on-site enquiry.

Most agendas are prepared in collaboration with the TEO. If there are any unanticipated problems in the agenda, the TEO should bring them to NZQA’s attention as soon as possible.

Other planning outputs

NZQA is committed to carrying out all its EERs in an open and direct manner. NZQA will always advise the TEO of any significant concerns as they come to our attention.

For this reason, the scoping phase of an EER will often include other communications from NZQA to the TEO, either by phone or email (or both). For example, NZQA scoping emails may:

  • summarise any current data gaps
  • declare any emerging NZQA concerns
  • identify stakeholders whom NZQA will need to contact (and who are ‘outside’ the agenda)
  • list follow-up questions which NZQA proposes to ask during the enquiry phase.

Throughout its scoping, NZQA aims to provide the TEO with clear guidelines on what information the EER team still needs to gather, and how it proposes to do so.

If a TEO is uncertain as to what NZQA is proposing to do or why during the enquiry phase, it should promptly seek an explanation – which the EER team will be happy to give.

3.2 The scoping guidelines

TEO self-assessment

TEO self-assessment practices must be comprehensive, reliable and embedded. In theory, only a TEO can fully understand its own operations; hence, NZQA cannot prescribe how a TEO should carry out its self-assessment. However, NZQA provides guidance on the common characteristics of high-quality self-assessment on the NZQA website.

Sustainable success is enabled by high-quality internal review. Unless a TEO knows, systematically and comprehensively, how effectively it is managing its business, it cannot guarantee future success. Without robust self-assessment, strong current performance may be accidental (or not even attributable to the TEO).

In preparing for an EER, a TEO should remember that NZQA will be looking for evidence of:

  • valued outcomes enabled by the TEO
  • supporting processes that have made these possible
  • ongoing, effective self-review.

TEOs must therefore provide sufficient analysis, with supporting evidence, for NZQA to be able to answer each of the key evaluation questions and arrive at organisational statements of confidence in both educational performance and organisational capability in self-assessment.

A TEO’s self-assessment summary itself should be relatively brief. It should present an analysis of what matters most to the TEO at the current moment.

An effective self-assessment summary should include information on the following:

  • The TEO’s context, kaupapa and priorities.
  • What has changed since the last EER, and why these changes matter.
  • How effective current TEO performance is, and how the TEO knows.
  • What have been the key findings of the TEO’s ongoing self-assessment, and how the TEO has responded to these in order to strengthen performance.

Supporting evidence

Some kinds of evidence are considered ‘primary’. Primary evidence must always be provided at the beginning of the scoping phase, and for every EER.

Other kinds of evidence are considered ‘secondary’. NZQA will request secondary evidence on an as-needs basis, at any point of the EER process. Evidence of this kind will often be context-specific or will provide NZQA with further substantiating detail on TEO quality.

Whenever NZQA requests specific information or documents, it will always explain why and indicate how this data will be used in the EER.

Primary evidence (mandatory) Secondary evidence (examples)
Cohort information (the number of learners, actual and EFTS (equivalent full-time students), disaggregated by domestic and international, Māori and Pasifika, and other) Programme information (conditions of approval for all programmes selected as focus areas)
Progression results, such as labour market outcomes and further study or training (if relevant) Organisational documents (e.g. the QMS (quality management system), staff qualifications, internal meeting records)
Achievement results (e.g. completion rates), with Tertiary Education Commission investment plan and target details (if relevant) Learner results (disaggregated data on learner achievement for all programmes or fields selected as focus areas. For example, information on the relative performance of Māori within a particular programme)
Māori and Pasifika outcomes  
A list of key compliance accountabilities (e.g. a subcontract with another TEO) Compliance documents (e.g. attestations of compliance, subcontracts, external moderation results)

Provide an evidential checklist

Whenever a TEO provides primary or secondary evidence to NZQA, the TEO should include a supporting checklist which:

  • identifies the evidential source (e.g. meeting minutes held in TEO archives)
  • provides date, version and status (e.g. draft internal audit report, 23 September 2018)
  • states its significance.

Where possible, all performance data will be jointly confirmed as complete and accurate before the enquiry phase begins.

Setting the scope

In setting the scope, NZQA aims to gain a typical sample of the TEO’s overall provision of education, training and supporting services.

This sample must be comprehensive enough for NZQA to draw reasonable conclusions about the quality of the TEO overall.

For example, a TEO offers 12 programmes, of which three are chosen as focus areas in an EER. In making that decision, NZQA will assure itself that these three programmes are likely to throw light on the performance of the TEO as a whole.

This assurance is strengthened by the use of the key evaluation questions, which run ‘across’ a TEO, implicitly covering programmes outside those designated as focus areas.

‘Typical’ scope

In arriving at a view of what would be a ‘typical’ selection, NZQA, in discussion with the TEO, will consider factors including:

  • The size of the TEO: as a rule, the smaller the TEO, the more straightforward the scoping exercise will be. For example, a ‘niche’ PTE that delivers a single training scheme will probably have that training scheme as its sole focus area.
  • The TEO’s own priorities: what changes, challenges or improvements has the TEO itself identified?
  • National priorities: are there any pressing strategic themes which the EER might explore? For example: postgraduate programmes in business.
  • Diversity of programmes: how can the EER encompass the TEO’s range of programme levels, fields and delivery sites?
  • Learner concentration: where (in programmes, fields or sites) are the majority of the students or trainees to be found?
  • Performance variation: which of the TEO programmes have been the highest performing? Which (if any) have had quality problems? Are there new programmes with limited evidence of learner achievement?
  • Risks: for example, are any programmes under statutory conditions? Have any programmes been the subject of recent learner complaints?

Selection of focus areas

As an example of the way focus areas are selected, ‘Acme College’ runs multiple programmes in several industry fields, and from levels 1-7; enrols domestic and international students; and had a poor result in its previous EER. The scope for the current EER of Acme College will probably therefore select focus areas that cover:

  • a cross-section of levels
  • programmes with international as well as domestic students
  • any improvements undertaken since the last EER
  • the largest programme (in terms of EFTS or funding).

Usually, NZQA and the TEO will discuss and agree on the scope of an EER. In a few instances, this may not be possible, and in these cases NZQA will set the scope.

In setting an EER scope, NZQA will do all it reasonably can to explain its decision-making to the TEO. If a TEO has any remaining questions or concerns about the rationale of the set scope, these should be addressed as soon as possible to the assigned Lead Evaluator.

Other scoping assumptions

Focus areas act as ‘windows’ into the quality of the TEO. The selection of focus areas should be seen as enabling, rather than limiting, NZQA’s enquiry.

Thus, over the course of an EER, NZQA may look in detail at any aspect of a TEO’s performance, including programmes not selected as focus areas, if it believes this kind of additional scrutiny is necessary to provide a balanced view of the TEO. Any changes of scope this entails will be discussed with the TEO.

Every EER covers all TEO activity from the previous EER to the present moment. In preparing for an EER, a TEO will therefore need to consider everything that is known, or knowable, about its performance over the preceding period.

Accurate information is vital

TEO failure to provide relevant and reliable information during the scoping phase may impact on the final EER ratings and statements of confidence, in so far as it indicates weaknesses in organisational self-assessment. Likewise, the provision of data that has not been accurately analysed and understood may limit the degree of confidence that NZQA can express in the TEO overall.

In extreme cases, gaps in TEO data or self-assessment identified during the scoping phase may delay (or extend) the timing of NZQA’s enquiry. If this situation arises, NZQA will advise the TEO directly of the action it will be taking, and any consequences arising.

TEOs should always bear in mind that scoping is an essential step in the overall EER process. NZQA will already be forming working hypotheses of TEO quality during the scoping phase, and any limitations on TEO self-assessment which arise at this stage may be noted as material.

If a TEO is in any doubt about what it needs to do or produce during the scope, it should contact the Lead Evaluator directly.

3.3 Time and cost

Estimating the cost

All evaluation time expended on a job by NZQA staff and contractors is on-charged to the TEO at a standard hourly rate.

Incidental costs incurred (e.g. travel or accommodation) are not on-charged. Nor is time spent by NZQA on administration or management of the work.

When the scope has been set, NZQA will estimate the likely amount of chargeable work needed to complete the EER.

Our estimates are by band, with an upper and lower figure. The bands are based on our experience of the time involved in EERs of similar size and complexity.

NZQA will advise you of the estimated cost by confirmation letter, in advance of the enquiry phase.

Managing the cost

Our estimates assume that the TEO will participate promptly in each EER stage and with a reasonable degree of efficiency.

If there are delays or information gaps at the TEO’s end, costs may increase. But if the TEO provides NZQA with comprehensive material, costs may decrease.

In some instances, EERs will involve more work than first expected, because of difficulties that arise during or after the on-site phase.

Whenever NZQA becomes aware that the estimate may be significantly exceeded, we will alert the TEO to the possibility.

Billing

A TEO will receive two invoices for its EER.

The first will be sent after completion of the enquiry phase. This invoice will be for 50 per cent of the upper end of the estimated cost.

The second will be sent after completion. This invoice will be for the total actual costs, minus any payment made on the first invoice.

The second invoice groups all time spent into three fields: preparation and planning, on-site, and reporting. The first and third fields encompass all time spent before and after the enquiry phase.

Before sending invoices to TEOs, NZQA carefully checks that the invoices are accurate, complete and reasonable. If the estimated costs have been exceeded, NZQA will check that the TEO has been advised.

If a TEO wishes to query either of its invoices, it should write to qaadmin@nzqa.govt.nz.

Links to other sections of the guide

What is external evaluation and review?

How does EER begin?

How does EER enquiry occur?

How are EER judgements made?

How are EER findings reported?

What happens next?

The process of external evaluation and review

Back to External evaluation and review

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz