How are findings reported?

The EER report provides a public record of the most important and relevant outcomes of the evaluation process.

6.1 The reporting process

On-site briefings

Every on-site phase of an EER begins with a meeting between NZQA and the TEO. NZQA will formally confirm the process and timelines that it intends to follow on site. The TEO should inform NZQA of basic ‘housekeeping’ arrangements (e.g. civil emergency procedures).

If the TEO is uncertain on any points of process or timelines, or wishes to revise the on-site arrangements, it should declare them as soon as possible.

NZQA aims to be flexible and will try to minimise any disruption to the TEO’s educational delivery caused by its on-site inquiry.

Keeping the TEO updated

If NZQA’s on-site phase lasts more than a day, NZQA will offer periodic updates to the TEO, usually at the close of each working day. These will identify any information gaps, any serious concerns (by exception), and confirm the process for the following day.

At the close of the on-site phase, NZQA will hold a formal meeting with the TEO. This meeting will be recorded and the audio file will be made available to the TEO.

The primary aims of the closing meeting are to:

  • summarise NZQA’s formative findings (this does not include the formative ratings, which are released in the draft report)
  • state any serious concerns (by exception)
  • note any remaining information gaps, and invite further TEO submissions
  • confirm the next steps in the EER reporting process
  • invite a brief TEO response, including any follow-up questions.

On occasion, information gaps identified at this stage may be so significant that the EER team cannot yet begin to draft a report. For example, key moderation data may only become available in two weeks’ time.

Whenever a situation of this kind arises, NZQA will notify the TEO of the likely delay and explain why it is necessary and important to receive the ‘missing’ information.

Drafting the EER report

After receiving and considering all necessary information, and completing a synthesis, NZQA will proceed to draft the EER report.

In the act of drafting an EER report, NZQA will sometimes change its initial findings, ratings and statements of confidence in the light of further reflection and newly received information.

When the EER report has been drafted, it undergoes peer review to ensure that its conclusions are fully supported by the evidential file, and an editing check.

The draft report is then circulated across the other relevant knowledge-holders within NZQA to confirm its accuracy.

The last step is management review. The Manager, Evaluation holds delegation for EER reports. Hence, the Manager, Evaluation makes the final decision on the content and findings of all draft EER reports.

Once it has been authorised by the Manager Evaluation, the draft EER report will be sent to the TEO and any relevant Crown agencies for review and comment. Any third parties identified in a report (e.g. ITOs) will be invited to comment on those sections of the report that relate to them.

Draft EER reports present NZQA’s provisional views. The findings, ratings and statements will only be confirmed in the final report after NZQA has considered all stakeholder submissions.

The TEO response

NZQA strongly encourages TEOs to provide full and considered feedback on draft EER reports.

Though a TEO may provide its feedback in any form, a TEO submission should always aim to meet the following standards:

  • The TEO response should never be longer than the original report.
  • All references to the report should include section and page numbers.
  • Any errors of fact in the report should be noted, with supporting evidence.
  • Any errors of interpretation in the report should be noted, with supporting evidence.
  • Any significant omissions in the report should be noted, and a clear rationale supplied as to why they should be included, with supporting evidence.
  • Any disagreements on matters of judgement should be noted, and a clear rationale given, with supporting evidence.
  • Any complaints about NZQA conduct should be separately addressed.
  • The tone of the response should be professional and courteous.

NZQA will carefully consider any TEO response to a draft EER report on its merits. Ratings may change depending on the merits of the TEO response.

Finalising the EER report

During the course of an EER, a TEO will sometimes take corrective action in relation to one or more instances of non-compliance.

If the action taken is significant, NZQA will note this fact within the EER report. In every such instance, the report will indicate if the correction:

  • is a plan only, or a completed action
  • addresses a problem that the TEO itself had identified
  • addresses a problem that is the result of human error, or more systemic failure
  • is major or minor in its impact.

Depending on the nature of the correction, as indicated above, NZQA may change one or more of the associated ratings. Please note that this possibility relates to correcting compliance matters (not filling long-term performance gaps).

In finalising an EER report, NZQA will always consider the quality of all submissions received, including those from third parties (e.g. Immigration New Zealand). If NZQA concludes that the evidence warrants it, contents from these submissions may be included in the final report.

Impact of new information

Between the completion of the draft and final reports, NZQA may receive significant new information about the TEO that has a potential impact on the findings of the report. If so, NZQA will advise the TEO of the new information and invite comment.

Once comment has been received, NZQA will consider whether new information needs to be included in the final version of the EER report, and any impact on findings.

If significant changes are made to an EER report between its draft and final version, NZQA will undertake a second peer review before the report progresses to management review. In this context, NZQA deems ‘significant changes’ to be:

  • the inclusion of any new information that has a major impact on the overall findings
  • changes to one or more ratings
  • changes to one or more of the statements of confidence.

Final report

The final version of the EER report, when released, represents NZQA’s summative view of the quality of a TEO’s performance and its capability in self-assessment, as these have been determined by a quality-assured process and signed off by the Manager, Evaluation.

Finalised EER reports include a separate document or letter summarising the reasons that NZQA has made or not made changes in response to the TEO’s feedback.

Next steps

In a few instances, TEOs will still disagree with the accuracy or fairness of the final report. A TEO may also have unanswered questions about the findings or the process that led to them.

There are four main options for addressing concerns about accuracy or fairness:

  • Query.
  • Correction.
  • Debrief.
  • Reconsideration.

If a TEO has some queries about the process or outcomes of the EER, one easy solution is to contact the Manager, Evaluation, directly. The manager or one of the evaluation team will try to answer your concerns fully, either by phone or in writing. Please note that this option is for clarification purposes only.

Errors of fact

If a TEO believes that some errors of fact remain in the final report, it should write immediately to the Manager, Evaluation, providing supporting evidence. If the manager agrees that these are in fact errors, they will be corrected in the final report at no on-charged cost to the TEO.

If a TEO believes that its staff could benefit, it may ask NZQA to provide a full verbal debrief on the findings made in the final report. Depending on the size of the TEO, this debrief will be done by phone, videoconference or direct visit. Please note that this option is for capability-building purposes only.

If all else fails, a TEO may seek a reconsideration of one or both of the statements of confidence in an EER report. The guidelines for lodging a reconsideration application and the indicative process followed are given on the NZQA website.

In deciding whether or not to seek a reconsideration, the TEO should remember that time spent on a reconsideration is outside the original budget estimate. As a rule, reconsiderations add an additional 30-40 per cent to the overall EER costs. They are on-charged in full to the TEO unless the reconsideration leads to a change to one or both of the statements of confidence (in which case the additional costs are waived).

6.2 The reporting guidelines

NZQA never gives ratings or statements of confidence during the enquiry phase. The reason is simple: the ratings and statements of confidence are still ‘under development’.

Instead, NZQA will share information on any emergent findings. As indicated earlier, ‘findings’ are important processed information that contribute directly to the judgments the team will eventually make.

The primary source of the EER team’s judgment is the report intended for publication. Any additional comment from NZQA aims only to clarify the intent of the report – not to supplement or replace its findings.

By the same token, while a TEO can provide feedback on a draft report in any form it chooses, NZQA will only classify a TEO response as formal if:

  • it is received within a timeline set by NZQA
  • it is authorised by the TEO’s chief executive or delegated authority
  • it relates to the content of the report or the process which led to its production.

Report: structure and style

EER reports are a summary only. That is, they are based on a larger evidential file, and select which findings should be given most weight and explanation in a defensible, public document.

The readership of any EER report varies widely. A reader could be a student, a staff member at a TEO, an industry representative, a government official or someone else. For that reason, NZQA aims to produce reports that are directly and simply written.

Whoever reads an EER report will find essential information on:

  • the context of the TEO
  • the scope of the EER
  • NZQA’s view on the relative quality of the TEO (as expressed though the statements of confidence and ratings)
  • why NZQA arrived at these views (i.e. the main supporting evidence)
  • what further actions, if any, the TEO might or must undertake, as a result of this report.

To that end, NZQA uses a standardised reporting format which describes the background of the TEO, then provides the evaluative judgements made in the course of the EER.

TEO details

This preliminary section of the report sets the context. It briefly describes what makes the TEO distinctive (for example, ‘Academy X is a small flight school in Gisborne’) and provides basic information on its operations.

This basic information covers:

  • the status of the TEO (e.g. PTE, Signatory to the Education (Pastoral Care of Tertiary and International Students) Code of Practice 2021 – 'the Code')
  • the TEO’s address and delivery sites
  • staff and student numbers
  • recent significant changes at the TEO (e.g. started a new hairdressing programme last year).

Student numbers are usually disaggregated by:

  • EFTS and actual (current) students
  • domestic and international provenance
  • Māori and Pasifika and other high priority learners.

This section also notes the confidence statements made in the previous EER report, and the scope of the current EER. Where the scope includes programmes as focus areas the report specifies:

  • whether they are NZQA-approved programmes, NZQA-approved training schemes, or TEO-managed programmes
  • their formal NZQA programme title and number (where relevant)
  • the qualification to which they lead (where relevant).

The only information included in this section of the report is confirmed matters of fact. No evaluative conclusions or judgments can be drawn from this section.

Summary of Results

This section of the report outlines the two statements of confidence. It includes an overarching summary of the key findings which led to these statements, then provides more detail by way of a series of bullet points.

If NZQA believes that a finding is significant enough to influence a statement of confidence, that finding will be noted in this section. On the other hand, if NZQA does not consider that a finding carries quite that degree of weight, the finding in question will only be mentioned in the main body of the report.

This distinction is important. The statements of confidence cover the most important matters as determined by NZQA. If a finding has been relegated to another part of the report, then NZQA certainly considers it relevant and important within that part of the report – but not in isolation as directly impacting on either of the statements of confidence.

The Summary of Results should always present an evidenced case. It should be clear to an impartial reader, for example, why NZQA is expressing itself Highly Confident rather than Confident in the performance of a TEO.

Key evaluation questions (general)

The main body of most EER reports covers the six key evaluation questions (KEQs). Each KEQ carries two ratings (one for performance, the other for capability in self-assessment) and explanatory text.

The KEQs themselves fall into two main groups: those relating to outcomes (KEQs 1 and 2) and those relating to the TEO’s supporting processes, systems and resources (KEQs 3-6).

There is no inbuilt ‘hierarchy’ of KEQs. No KEQ has intrinsically more importance than another. If in the course of an EER the evaluators place special weight on one or more of the KEQs, the report will explain this decision.

The six key evaluation questions used in EER:

  1. How well do students achieve?
  2. What is the value of the outcomes for key stakeholders, including students?
  3. How well do programme design and delivery, including learning and assessment activities, match the needs of students and other relevant stakeholders?
  4. How effectively are students supported and involved in their learning?
  5. How effective are governance and management in supporting educational achievement?
  6. How effectively are important compliance accountabilities managed?

The content of KEQs

The structure of each KEQ section is the same: the two ratings are stated, followed by the findings and supporting evidence, then a summarising conclusion is drawn. This conclusion should be understandable and clear in isolation (as well as congruent with the assigned ratings).

How evidence is cited

Some Tertiary Evaluation Indicators are more commonly associated with some KEQs than with others. For example, one indicator of educational success is learner completion of programmes. Many answers to KEQ1 take this indicator (among others) into account.

Likewise, different kinds of evidence tend to cluster around this or that KEQ, for common-sense reasons.

For example, answers to KEQ3, which looks at the effectiveness of programme design and delivery, will often note the relevance of TEO programme review documents or processes. It is less likely that this kind of evidence will also be cited under KEQ4 (which considers how well learners are supported).

The ratings for each KEQ are determined with reference to:

Repetition of findings

In some instances, however, the same evidence, findings or indicators will be cited in different KEQs within the same report. This is not ‘double jeopardy’. Rather, in NZQA’s opinion, some information is relevant in more than one context, in order to adequately answer multiple KEQs.

For example, a TEO may have a poor moderation track record for one of the programmes in scope. Depending on the details of the case, this fact may be considered in answering:

  • KEQ1: because poor moderation potentially undermines the validity of reported learner completion results
  • KEQ3: because poor moderation may indicate failures in the training or skills of the TEO’s academic staff
  • KEQ3: because the poor moderation in this instance is not a one-off lapse but an ongoing pattern, which reflects on the oversight of the academic management
  • KEQ6: because acceptable moderation results and processes are compliance requirements.

In itself, this repetition may – or may not – have a serious effect on the overall outcomes of the EER. As always, NZQA will consider the specific materiality, representativeness and weighting before passing judgment.

As with the Summary of Results, each KEQ answer should always present an evidenced case. It should be clear to an impartial reader, for example, why NZQA is rating the performance of a particular KEQ Marginal rather than Good.

TEO management of compliance

NZQA expects all TEOs to comply with regulatory requirements. These include New Zealand legislation, NZQA Rules, other Crown agency regulations, and standards set by industry bodies.

Under KEQ6, NZQA considers how effectively a TEO is managing these responsibilities. The evaluators examine the TEO’s relevant processes, actions and outcomes, and review a selection of evidence (e.g. student enrolment records). Based on this information, a rating is assigned by the same evaluative process used to answer all the other KEQs. Because of limitations of time, sample size and the wording of KEQ6, EER reports never affirm that a TEO is ‘fully compliant’. But they note whenever serious problems have been identified and decide, on the evidence available, whether the TEO has reliable internal processes for identifying and mitigating non-compliances.

Other Crown mechanisms, such as risk monitoring, provide the public with more detailed ongoing assurance that no ‘unmanaged’ breaches are occurring in the TEO sector.

Evaluating TEO compliance management

An EER enquiry might reveal that two international students entered the organisation without proper insurance. This would be considered a breach of the Code (i.e. The Education (Pastoral Care of International Students) Code of Practice).

For the NZQA evaluators, this would of course be an important fact. But it would not in itself determine the KEQ rating. To reach an evaluative judgment under KEQ6, the evaluators would ask themselves how important this breach was in the larger context. Were these two students sampled from a cohort of four – or of 400? Was the problem identified by the TEO itself and properly managed? Was the failure systemic, or an isolated example of human error?

Focus Areas

All EER reports rate focus areas by the same methodology as used to rate KEQs and arrive at statements of confidence.

As a rule, separate text on focus areas will usually be found only in reports on large and complex entities (such as wānanga and institutes of technology and polytechnics).

For smaller and less complex EERs, where all the necessary contextual information has already been provided through the KEQs, focus area comment will be omitted.

Requirements and Recommendations

Some EER reports include text under Recommendations and Requirements.

Recommendations are added whenever NZQA has found specific opportunities for improvement that it believes the TEO should consider.

Recommendations are optional. A TEO will be expected, however, to reflect on the merits of each recommendation as part of their ongoing self-assessment. The next EER of the TEO may therefore include follow-up questions such as: ‘what (if anything) did you do about our recommendations? ‘Why (or why not)?’, ‘How effective were the steps you took?’

Recommendations aim to provide guidance on what might change in a TEO, rather than directing how change should occur. Decisions on the ‘how’ rests with the TEO itself.

Requirements are mandatory. They are included whenever NZQA has identified a breach of an essential regulatory requirement (such as the NZQA Rules). A TEO must comply quickly and effectively with any report requirements. NZQA will check that requirements have been met either at the time of the next EER or sooner through one of its other monitoring activities.

Links to other sections of the guide

What is external evaluation and review?

How does EER begin?

How are EERs planned?

How does EER enquiry occur?

How are EER judgements made?

What happens next?

The process of external evaluation and review

Back to External evaluation and review

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz