Guidance on external evaluation and review enhancements

On 1 July 2016, NZQA introduced several changes to external evaluation and review (EER). These changes are intended to build on the principles of evaluative quality assurance. They strengthen the existing system through sharpening aspects of its design, tools and processes.

In making these changes, NZQA consulted closely with the tertiary education sector. Sector feedback has been very useful, and has contributed to a number of further refinements. NZQA remains committed to working with the sector and other expert stakeholders in developing guidelines for implementing the enhancements.

This resource will be the first of an ongoing series of engagements with the sector, to test the effectiveness of the changes and invite suggestions for further improvement.

For that reason, this guidance should be as read ‘draft’ or ‘provisional’. It reflects NZQA’s current view of how the EER enhancements will best work. The approach is iterative. NZQA anticipates that ongoing reflection, experience and feedback will further refine the enhancements. When that occurs, an updated set of guidelines will be posted on the NZQA website.

What is in scope

This guidance material covers the following EER enhancements

1. A summary of the revised key evaluation question (KEQs)
2. The addition of a new compliance KEQ
3. Refinements of other KEQs
4. Update on other EER initiatives

A. The revised KEQs

Here is the full list of revised KEQs, with the changed KEQs highlighted in italics, and the new KEQ highlighted in bold.

1. How well do students achieve?
2. What is the value of the outcomes for key stakeholders, including students?
3. How well do programme design and delivery, including learning and assessment activities, match the needs of students and other relevant stakeholders?
4. How effectively are students supported and involved in their learning?
5. How effective are governance and management in supporting educational achievement?
6. How effectively are important compliance accountabilities managed?

B. The compliance question (KEQ 6)

The new question (KEQ 6) responds to the need to incorporate ‘compliance’ management and performance into each EER. Like all the other KEQs, it proceeds from the assumption that the primary knowledge needed to answer the question is held by the tertiary education organisation (TEO) itself, though its self-management and self-assessment systems.

By ‘compliance’, NZQA proposes to examine how well the TEO manages its regulatory responsibilities under the Education and Training Act 2020, and associated rules. In practice this will mean effective self-management of:

• Compliance management processes
• Conformance with relevant NZQA rules - including rules for approvals, accreditation EER and registration, and others as may be appropriate (see NZQA Rules).
• Conformance with relevant regulations, such as The Education (Pastoral Care for International Students) Code of Practice
• Programme delivery and programme hours, including consistency with the programme approval (including EFTS (equivalent full-time student) values based on credits, learning hours approved and delivered, and full-time teaching weeks)
• Significant legal and/or ethical issues (which, ideally, will have been identified and mitigated by the TEO in advance of the EER).  

Evidence that a TEO can use to demonstrate compliance can come from a variety of sources: meeting minutes, internal quality assurance documents, records of student attendance, programme approval and delivery documents, audit reports, survey results (e.g. of students or employers), timetables and delivery schedules, student or course handbooks, TEO risk management documents and any other relevant information.

In deciding what evidence it considers most relevant to the question, each TEO should take into account the ‘merit criteria’, that is, those tertiary evaluation indicators that relate to ‘compliance’ (PDF, 62KB).

Evaluators will also include in the EER topic-specific interview questions for staff and students, and/or direct observation of performance, as appropriate. Like the TEO, the evaluators will be guided in their judgments by reference to the relevant tertiary evaluation indicators.

The onus will be on the TEO to demonstrate its compliance in the first instance, with independent verification by evaluators as required to complete the picture.

C. The changed questions (KEQs 3 and 4)

The wording of the changed KEQs better responds to the needs of the primary stakeholders: students. Current research suggests that educational achievement is strongly influenced by student engagement and motivation.

Other key factors include the active involvement of students in their own learning; the support provide by the TEO which enables them to be so; and the teacher-led activities of programme design, delivery and assessment that matches stakeholder needs.

No new actions should be required by any TEOs due for EER. Standard internal review processes within each TEO should already indicate how well student learning is supported and the role played by the teaching staff in building an effective learning culture.

If TEOs have any concerns about how the changed KEQs might affect the scope of their own self-assessment, they should contact their assigned Lead Evaluator as soon as possible to discuss this. NZQA will work constructively with each TEO, on a case-by-case basis, to identify relevant sources of evidence that will assist the TEO answer that question.

The TEO should also refer to the full set of existing tertiary evaluation indicators. Many of these are already directly relevant to the revised KEQs. The upcoming review of the tertiary evaluation indicators (below) will also assist this process

D. Other EER initiatives

1. Review of indicators

Over the next four months, NZQA will work with representatives across the sector to ensure that all our tertiary evaluation indicators remain comprehensive, useful and current and can be aligned (where feasible) with particular KEQs. During this engagement, NZQA will review the adequacy of the draft ‘compliance’ indicators, and refine these as necessary. If you wish to be involved in the TEI Working Party, please send an expression of interest to eer@nzqa.govt.nz by 1 August 2016.

2. Self-assessment

NZQA is currently developing more comprehensive guidelines on what is expected from a TEO’s self-assessment at the time of its EER, especially as this relates to programme design, approval and delivery. While these are under development, any TEO that is uncertain about what documentation NZQA expects in its own case should contact its assigned Lead Evaluator.

3. International students

NZQA is committed to the responsible development and delivery of education to international students. It is essential to New Zealand’s ‘brand’ to ensure that all students are adequately supported in their programmes of study.

To that end, from 1 August 2016, NZQA will propose a standard focus area for all EERs of TEOs that enrol international students. This focus area will be entitled ‘International Students: Support and Well-Being’ and will examine how effectively the TEO under review is discharging its pastoral care responsibilities towards is international students.

This focus area will be proposed during the scoping process of each EER. If a TEO has any questions or concerns about the application of this area to its own delivery, it should discuss them with the assigned Lead Evaluator at this stage of the EER process.