It is expected most education organisations will provide convincing self-assessment with sufficient supporting evidence during the review meeting and receive a rating of ‘sufficient’.

Education organisations unable to do so will have an opportunity to submit further evidence that the organisation is sufficient (i.e. their graduates meet the graduate outcomes).

• NZQA will not make the final decision on consistency of the qualification until they have considered the further evidence.
• An interim rating of ‘not sufficient’ may be changed to ‘sufficient’ as a result of the further evidence.
• NZQA may provide recommendations to help the organisation to improve evidence.

NZQA will take further action if organisations repeatedly cannot provide convincing self-assessment evidence.

NZQA develops and publishes a schedule of qualifications for consistency review annually in conjunction with the relevant qualification developers.

All organisations that have awarded the New Zealand qualification scheduled for a consistency review are required to participate in the scheduled review.

NZQA will allocate appropriate reviewers to conduct the reviews.

If an education organisation has delivered an approved and accredited Te Hono o Te Kahurangi programme that leads to the relevant qualification, the allocated reviewer will have expertise in te reo Māori, tikanga Māori and applying Te Hono o Te Kahurangi evaluative principles.

Conclusions at education organisation level

The first step is the evaluation of the self-assessment and supporting evidence that the individual education organisation provides to the consistency review to show how well its graduates meet the graduate outcomes of the qualification.

Education organisations must be able to demonstrate with high quality, convincing evidence that their graduates demonstrate the graduate profile of the qualification.

Using the evaluation question

Consistency of graduate outcomes is determined by answering the evaluation question:

How well does the self-assessment and supporting evidence provided by the education organisation demonstrate that its graduates match the graduate outcomes at the appropriate threshold?

The question applies to each individual education organisation and will be answered in the context of the graduate profile of the individual qualification that is the subject of the consistency review.

The answer to the question is informed by the following criteria:

• the nature, quality and integrity of the self-assessment and supporting evidence presented by the education organisation
• how well the organisation has analysed, interpreted and validated all the available evidence, and used the understanding gained to achieve actual or improved consistency
• the extent to which the education organisation can reasonably justify and validate claims and statements relating to the consistency of graduate outcomes, including as appropriate, in relation to other providers of programmes leading to the qualification.

The conclusion, based on the answer to the evaluation question, is a judgement about the extent and validity of the self-assessment and evidence the education organisation uses to demonstrate how well its graduates meet the graduate outcomes.

Rating the answer to the evaluation question in relation to each education organisation

Rubric one sets out the expected levels of performance in relation to the evaluation question for an individual education organisation.

In the context of the rubric – ‘good evidence’ is defined as:

• relevant to answering the evaluation question
• obtained from more than one source (i.e. is corroborated or triangulated)
• of more than one type (e.g. quantitative and qualitative data)
• making sense in the context of the qualification and the education organisation programme.

The quality of the answer to the evaluation question is rated according to the rubric.

Rubric one: Performance criteria for rating answers to the evaluation question for individual education organisations

The critical outcomes are determined during the assuring consistency process, when the participants develop an accepted ‘threshold statement’ for the qualification.

The essential evidence requirements are related to the qualification specific ‘threshold statement’.

Conclusions about the qualification across participating education organisations

A consistency review reaches a conclusion about how well graduates match the graduate outcomes of the qualification, at the appropriate threshold, across all education organisations providing programmes leading to the qualification.

The conclusion is based on synthesising the evidence available from participating education organisations. It is an overall, well-informed professional judgement of the national consistency of graduate outcomes relating to an individual qualification.

The decision on the national consistency of graduates of the qualification is based on the synthesis of the answers to the evaluation question answered for each education organisation participating in the consistency review. The performance criteria for reaching a conclusion about the national consistency of the achievement of graduate outcomes are set out in rubric two.

A decision on the consistency of the graduates of the qualification will be made once:

• there is confidence in the agreed threshold, and
• education organisations have had the opportunity to demonstrate the quality and sufficiency of their self-assessment evidence if this wasn’t available at the time of the review.

Rubric two: Criteria for judgements about the national consistency of graduate outcomes of qualifications across all education organisations providing relevant programmes

Travel and other expenses for participating in consistency reviews are the responsibility of the participating education organisations.

NZQA annually collects a fee per graduate from education organisations. The fee is intended to cover the costs of managing the overall system.

The fee is published on the NZQA fees page and is periodically reviewed.

The qualification developer is responsible for:

• determining, in conjunction with relevant stakeholders, the examples of convincing evidence required for demonstrating consistency for the qualification (this information is included in the listing details of the qualification)
• clarifying the evidence requirements for assuring national consistency in the qualification when requested by programme developers and other stakeholders
• advising the consistency reviewer as a subject matter expert, on the qualification outcomes and contributing to the ‘threshold statement’ discussion
• assisting with arrangements for consistency reviews
• collating feedback from the assuring consistency process to inform qualification review.

The education organisation is responsible for:

• participating in periodic events for assuring national consistency
• consulting with the qualification developer (the organisation that listed the qualification) as their programme is developed
• ensuring that all the graduates have achieved the graduate outcomes to an equivalent and appropriate standard
• demonstrating, in a written self-assessment, with supporting evidence, how graduates meet the qualification outcomes
• in response to the evaluation question, providing an oral summary of evidence that supports their self-assessment
• ensuring that the self-assessment and supporting evidence they provide is sufficiently representative, both of the graduates and of the range of knowledge, skills and attributes within the graduate outcome statement
• responding to and complying with recommendations arising from the consistency reviews.

Consistency reviewers are responsible for:

• working with the range of qualification developers and programme owners on behalf of NZQA
• using an appropriate approach when working with qualifications developed to meet the needs and aspirations of Māori learners
• using a Te Hono o Te Kahurangi kaupapa lens when working with education organisations with programmes approved and accredited through this framework
• preparing for and facilitating the assuring consistency event
• identifying good quality, convincing organisational self-assessment and supporting evidence of graduates meeting the graduate outcomes
• leading and facilitating the discussion and guiding the participants to develop the ‘threshold statement’ for the qualification
• using an evaluative approach to decide and report on the extent to which graduates from different programmes are achieving qualification outcomes
• writing the consistency review report listing the ratings for each education organisation and the reasons for their decision
• the final judgement on national consistency of the qualification, assisted as required by subject matter experts
• making robust decisions and reporting appropriately to NZQA
• producing clear and credible written reports that make explicit the evidence used to reach their judgement.

NZQA is responsible for:

• oversight of the effectiveness of the processes for assuring national consistency of graduate outcomes
• oversight and co-ordination of consistency reviewers by contracting, training, supporting and monitoring reviewers
• publishing an annual schedule of qualifications for ongoing national consistency events in consultation with qualification developers
• publishing the results of assuring consistency events
• following up with education organisations where their evidence is judged ‘not sufficient’.

Steps in preparing for the consistency review

1. NZQA publishes a schedule of qualifications to undergo consistency review and allocates a consistency reviewer to each event.
2. Consistency reviews involving organisations that have programmes approved and accredited through NZQA’s Te Hono o Te Kahurangi framework will be allocated a reviewer who has expertise in te reo Māori, tikanga Māori and applying Te Hono o Te Kahurangi evaluative principles.
3. Education organisations who have reported graduates for the qualification involved will be sent details of the review no later than two months before the review.
4. Each education organisation with graduates prepares and submits a self-assessment report based on evidence from its self-assessment processes. The report will demonstrate why the organisation is confident its graduates have met the graduate outcomes for the qualification.

Scheduling qualifications for assuring consistency review

Qualifications will typically be scheduled for consistency review 12-18 months before the qualification review date.

The schedule will be published and updated as required.

Arranging the consistency review meeting

The following factors are considered in arranging the details of a consistency review:

• the number of education organisations with graduates of a programme
• the geographical spread of the education organisations
• identifying the approach required, in particular using a specific approach for qualifications designed to meet the distinctive needs and aspirations of Māori (including the selection of a suitable reviewer)
• identifying the appropriate mechanism for conducting the review (i.e. a cluster meeting at a specific venue, teleconference or videoconference or through suitable electronic media).

Education organisation report

Education organisations that have reported graduates are required to prepare and submit a self-assessment report that addresses how they know that graduates have met the qualification graduate outcomes.

The process involves:

• reporting on the evidence collected during self-assessment activities to support conclusions and judgements
• reviewing the self-assessment evidence to enable organisations to answer the evaluation question:

How well does the self-assessment and supporting evidence provided by the education organisation demonstrate that its graduates match the graduate outcomes at the appropriate threshold?

Education organisations should:

• explain why they are convinced that their graduates have met the graduate outcomes
• justify the nature, quality and integrity of the supporting evidence
• identify any gaps in the evidence
• identify areas of possible improvement
• determine a self-rating.

The education organisation uses the Consistency report template (DOCX, 89KB).

Early consistency reviews

Qualifications may be subject to a consistency review earlier than scheduled if there are concerns about any of the following factors:

• the qualification has had a previous judgement of ‘National consistency not confirmed’
• an unexpectedly high volume of graduates from the qualification
• the qualification involves high risk or high-profile activities
• a request from the qualification developer or another key stakeholder
• an identified risk or issues or concerns about the qualification
• a diverse range of qualification awarders
• multiple education organisations are assessing/participating in the same approved programme.

Steps for the consistency review meeting

1. Review meetings will be facilitated by a reviewer allocated by NZQA.
2. Participants make their own arrangements to attend the consistency review meeting.
3. Participants verbally present their evidence and summary self-assessment report at the meeting.
4. The reviewer facilitates the review meeting, assists the participants to develop the threshold statement and reports the results to NZQA.

Who needs to participate?

All education organisations with graduates of the qualification must participate. Each education organisation must be ready to send authorised representatives who have sufficient knowledge of the qualification graduates to present and discuss self-assessment and supporting evidence on behalf of their organisation.

The qualification developer is a mandatory participant in the consistency review. They can provide clarification and guidance on the standard expected by employers and industry.

Expectations of the consistency review

Successful reviews will be conducted in a spirit of co-operation and collaboration – kotahitanga, manaakitanga and whanaungatanga – between the participants.

The consistency review will take an educational and evaluative approach.

Consistency review meeting

Each participating education organisation will present a summary of their self-assessment evidence to demonstrate why they are confident that graduates have achieved the graduate outcomes for the qualification.

There are three distinct activities to be undertaken by the review:

a. Present the summary report

The first task is to systematically review, compare and discuss the self-assessment evidence presented by each education organisation. Each participating education organisation will explain how their evidence demonstrates that graduates meet the graduate outcomes.
 
b. Use the graduate profile and the range of evidence to agree the threshold to be achieved

The second activity in the review is to develop a ‘threshold statement’ for the qualification under consistency review. The aim is to understand and agree the content of the graduate outcomes all stakeholders can reasonably expect education organisations to have evidence of. The intent of the meeting is to connect the graduate outcome statement with the convincing evidence the education organisations could gather.

The threshold includes a clear definition of the level of skill, knowledge and attributes required. Some parts of the threshold may be very similar to aspects of the graduate outcome statement, but the threshold is focused on the current cohort of graduates and the skills, knowledge and attributes it is reasonable to expect the education organisations to be able to have evidence of.

This part of the meeting requires the participants to consider and agree on statements that describe the outcomes of the qualification and the essential self-assessment evidence related to these. These statements are used by the reviewers when making their judgements on the nature, quality and integrity of education organisations evidence.

Depending on the size of the group this activity could be done with all participants or in small groups.

c. Reflect on their initial decision about the consistency of their graduates

At the end of the review meeting, each participant will be asked to reflect on the quality and sufficiency of their self-assessment and supporting evidence and identify:

• what they will continue to do
• what they will change in their self-assessment and supporting evidence in the future
• any change to their initial rating on the evaluation question – sufficient/not sufficient.

The consistency reviewer will retain all material provided to NZQA for the review meeting. This material is confidential to NZQA.

Examples of sources of evidence

The individual education organisation’s self-assessment is the primary source of evidence. Conclusions should be supported with multiple sources of convincing evidence: for example, triangulation between what the education organisation knows about their programme, what the graduates say and what the end-user says.

Self-assessment evidence should clearly link to the graduate outcomes and may reference:

• employer feedback
• graduate feedback
• community feedback surveys
• whānau, hapū, iwi feedback
• destination or end-user data (including feedback from the providers of the graduate’s next level of study)
• portfolios of work or capstone event reports
• external benchmarking activities
• results of internal and external moderation of assessment
• other relevant and reliable evidence.

Methods of validation of the self-assessment and supporting evidence may include:

• comparison with formally established internal or external benchmarks or expectations, or professional, licensing or discipline standards
• data on employment outcomes or progression to further training over time
• comparison with similar graduates from other education organisations.

One qualification awarder

When there is only one organisation awarding the qualification, there is still a requirement to assure national consistency of the graduate outcomes.

This independent assurance is important if the qualification is offered, for example, in multiple workplaces.

An arrangement will be made that is appropriate to the context and circumstances that apply to the organisation.

Alternative mechanisms

For some qualifications, an existing panel or other body such as a registration body has been established for the purposes of moderation or other quality assurance. In these instances, NZQA may, in discussion with the qualification developer, determine the relationship between assuring consistency and existing arrangements.

The mechanism will be agreed on a case-by-case basis.

Steps for reporting the results of a consistency review

1. Consistency reviewer reports to NZQA at the completion of the review meeting(s) for the qualification.
2. NZQA reviews and accepts the report then publishes either an interim or a final report on the NZQA website.
3. NZQA follows up with individual education organisations and the qualification developer on specific issues or recommendations identified during the review.

The consistency review report

The report will address the overall quality and sufficiency of the self-assessment and supporting evidence provided for the reviewer to reach a judgement of ‘National consistency confirmed’ or ‘National consistency not confirmed’ on the qualification outcomes.

The decision on the quality and sufficiency of self-assessment and supporting evidence for each education organisation (sufficient/not sufficient) will be included in the report the reviewer provides to NZQA. The report is quality assured internally by NZQA prior to publication.

The report produced by the reviewer for NZQA will include discussion of education organisations that:

• have participated in the process and been judged ‘sufficient’
• those who have engaged but their self-assessment information or evidence is not sufficient
• those not engaged in the process.

The focus for the report is on improving both the qualification and the education organisation’s understanding of the outcomes of the qualification.

The report will also include:

• the threshold developed at the review meeting
• an overview of the self-assessment and evidence provided by the education organisations
• examples of the convincing and credible evidence used by the reviewer in their evaluation
• findings and recommendations on improvements to the qualification: for example, there may be evidence that some outcomes are not fit-for-purpose based on the supplied evidence
• examples of good practice.

Following the consistency review, NZQA will publish the names of education organisations where the decision is that the self-assessment and evidence was sufficient.

The names of education organisations that were not able to present sufficient convincing evidence at the consistency review will not be listed. These organisations will have a defined period after the consistency review meeting to present further evidence, or to demonstrate through an organisational improvement plan how they will address their self-assessment evidence issues in a prompt, agreed timeframe.

Follow-up actions

If the final decision in relation to the qualification is ‘National consistency not confirmed’ then the next consistency review or qualification review may be scheduled.

It is NZQA’s responsibility to follow up with individual education organisations where the decision on the evidence is ‘not sufficient’ or where education organisations do not engage in the consistency review process.

Possible actions include:

• requiring a review of the accredited programme
• NZQA monitoring of the programme with a focus on assessment practice
• NZQA investigation of the education organisation
• imposing conditions on education organisation accreditation, or other action, on confirmation that the organisation is not complying with the relevant Act or NZQA Rules.