How does EER enquiry occur?

EER enquiry involves a number of processes of collecting and analysing information about a TEO’s performance.

4.1 The enquiry process

Purpose

During the enquiry phase, the evaluators complete the assembly of the evidential file. They aim to:

  • validate data received during the scoping phase
  • find then validate new and relevant data
  • interpret all data correctly.

Enquiry can take one or more of three forms: desk review, on-site enquiry, or off-site enquiry. Time spent on each of these is on-charged to the TEO at the same rate.

The enquiry phase is completed when, in NZQA’s view, all necessary information has been gathered to reach judgments about the current quality of the TEO’s performance and its capability in self-assessment.

Desk review

A desk review involves the collection, validation and analysis of documents received from the TEO and other parties.

Desk reviews may be supplemented by discussion with the TEO by email, phone or videoconference.

For small TEOs, especially those engaged in on-job training, NZQA will sometimes conduct a desk review only, without ever directly visiting the TEO.

This decision will only be made, however, if NZQA is convinced that a desk review alone will provide sufficient evidence to reach judgments about the TEO under review.

On-site enquiry

On-site enquiry is sometimes referred to as ‘the EER’. This is not so: it is but one phase in the larger EER process. Further, though most EERs do include on-site enquiry, it is not mandatory or necessarily the most important phase.

On-site enquiry involves a visit by the EER team to the TEO’s head office, or to one or more of the TEO’s delivery sites.

Time spent on site ranges from one day to two weeks, depending on the scale and complexity of the EER.

Time and resources required on site will be laid out in the agenda, which is sent to the TEO in advance. In allocating meetings on site, NZQA will make every reasonable effort to minimise disruption to the daily operations of the TEO, especially with respect to its learners.

Information will be gathered or confirmed on site by a range of processes, such as interviews, presentations, document review and discussions.

Off-site enquiry

Off-site enquiry covers contact made with stakeholders absent during the on-site phase.

Examples of relevant stakeholders include at-home students, graduates, industry advisory groups, standard-setting bodies and other Crown agencies.

Off-site enquiry can involve face-to-face meetings, but more often will be conducted by phone calls, emails or videoconference.

Sometimes off-site enquiry overlaps with on-site enquiry; in such cases, this will be noted in the agenda. More often, however, off-site enquiry will occur as a follow-up to information received or considered on site. For example, the EER team may discuss a TEO’s performance against its investment plan with a Tertiary Education Commission representative.

Clarifying the data

Though NZQA expects that a TEO will already have identified its most important data during the scoping phase, on-site enquiry allows TEOs a further opportunity to illustrate or clarify any claims relating to that data, or how the data has been used by the TEO.

Whenever student interviews are carried out, NZQA selects who will participate in consultation with the TEO. To ensure open dialogue, all students in an EER interview will be guaranteed anonymity. Interview notes will, therefore, not record student names but will summarise key comments made.

As a rule, NZQA does not engage in controlled observations (i.e. watching a tutor or trainer in the classroom), because we believe that such activities are of limited value.

On the other hand, evidence gathered by the team first-hand through ongoing observation can be of value. If an EER team notices anything of particular significance while on site, this will be recorded in the evidential file and considered as an input during the judgment phase.

Any TEO can include a support or advisory person during the on-site enquiry, without the need to seek permission from NZQA.

4.2 The enquiry guidelines

In the course of its enquiry, the EER team will always check that data is relevant, accurate and complete. It will also confirm how the data has been used, and why.

Wherever possible, a TEO should present NZQA with ‘processed’ (rather than ‘raw’) data. That is, any data submitted should already have been collected, interpreted and analysed by the TEO as part of its organisational self-assessment.

For example, if you state that ‘our completion rates are 80 per cent’, you should also indicate how you know this claim is correct and why you believe this is a good (or modest or excellent) outcome.

If the evaluation team is forced to explain or ‘correct’ the TEO’s own data, this in itself points to a serious performance or self-assessment weakness in the TEO.

What is ‘data’?

Data refers to all information on the performance or quality of a TEO.

Data can be ‘soft’ (e.g. student testimony of improved well-being), or ‘hard’ (e.g. qualification completion rates).

For most TEOs, both hard and soft data sources will be relevant indicators of performance. The guidelines to the Tertiary Evaluation Indicators refer to a wide range of data sources that can be drawn on for self-assessment (and EER) purposes.

EERs are generally not interested in data in isolation, but in how the data has been used by the TEO under review to understand and improve performance.

For this reason, the primary data guiding every EER should be whatever has been identified by the TEO, and confirmed by NZQA, as significant.

Sampling

As already noted, all EERs are based on sampling, that is, by a series of selections – and exclusions.

Thus, some programmes will be selected as focus areas for review, some stakeholders will be interviewed, and some documents will be examined. Conversely, other types of potentially relevant evidence (other programmes, stakeholders, documents) will not be considered, or at least not considered in as much detail.

Well-done sampling should provide some degree of insight into the quality of the TEO as a whole. Though the information accessed through an EER will always be partial, NZQA should still be able to draw reasonable larger inferences from it.

Targeted/random enquiry

During the enquiry phase, sampling will be targeted or random (or both). For example: the EER team might decide to meet directly with a group of students who had previously complained to NZQA about the quality of a programme; this would be an example of targeted enquiry. Alternatively, the EER team might choose to speak with every tenth student listed on the TEO’s roll; this would be an example of random enquiry.

Limitations of time and resource can still mean that some important information gets overlooked, or less important information over-emphasised. For example, it is possible that the sampled data, while accurate, is unfairly weighted towards ‘outlier’ results.

Gaining assurance

Despite these important caveats, the EER process as a whole has several inbuilt controls to mitigate sample error during the enquiry phase. These are:

  • well-informed scoping decisions, including any emergent view by the EER team on the quality (hence, the reliability) of the TEO’s self-assessment
  • the ongoing opportunity afforded the TEO to provide information missed during the enquiry phase
  • third-party hard data sources (e.g. NZQA moderation results) to supplement or correct TEO information claims
  • the in-house quality assurance carried out by NZQA before it confirms any judgements or issues any reports.

Issues of scale

As a rule, the smaller the TEO, or the less complex the EER, the more weight can usually be placed on the reliability of the sampled data. By the same token, the larger the TEO, or the more complex the EER, the more cautious are the conclusions that can reasonably be drawn from limited sample sizes.

Triangulation

The enquiry phase first identifies data, then checks its accuracy. While some data can be taken as a given (e.g. the contact details for a TEO), other data claims need to be verified, either to confirm a fact or to assess how reliably the data has been interpreted.

The most common term for this verification process is ‘triangulation’. Triangulation aims to mitigate the possibility of unintentional bias and the limitations of relying on single source material.

As the word indicates, triangulation involves looking for three (or more) confirming points of evidence. This can occur in various ways. For example:

  • Three examples of the same thing
  • Three types of sources
  • Data gathered over three points in time
  • Data looked at in three different ways
  • Three observers.

Triangulation cannot guarantee that every data claim made in a report will be accurate, but it much reduces the likelihood of error.

EER records will always indicate on what basis a factual claim has been made. If this claim relies on triangulation to any degree, the evidence (of methods, samples, sources, etc) will be stated.

Links to other sections of the guide

What is external evaluation and review?

How does EER begin?

How are EERs planned?

How are EER judgements made?

How are EER findings reported?

What happens next?

The process of external evaluation and review

Back to External evaluation and review

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz