NZQA's approach to programme monitoring at levels 1-6 and level 7 diplomas

The purpose of this programme monitoring information is to clarify NZQA’s approach to monitoring programmes at levels 1-6 of the NZQF, and level 7 diplomas.

NZQA’s approach to programme monitoring

To maintain approval and accreditation to deliver an NZQA-approved programme, all TEOs must continue meeting the requirements set out in Rules 11 and 12 of the NZQF Programme Approval and Accreditation Rules 2018.

The requirements state that to maintain approval and accreditation, TEOs must ensure that the criteria set out in Rule 4.1 and Rule 6.1 continue to be met:

Criteria to maintain programme approval Rule 4.1
Criteria to maintain accreditation Rule 6.1

Where NZQA requests is, TEOs must participate in programme monitoring.

NZQA has a range of programme monitoring activities, which focus on one or more of the criteria to maintain programme approval and accreditation.

For example, NZQA may carry out:

  • a visit to the TEO and moderation of learner work samples
  • moderation only
  • internal moderation systems monitoring
  • programme review monitoring
  • validation of offshore online delivery

NZQA carries out the majority of its current monitoring activities at Levels 1-6 and Level 7 diplomas remotely.

NZQA determines an overall rating for the programme

TEOs provide evidence relating to the relevant approval and accreditation criteria. NZQA uses this evidence to answer a key evaluation question (KEQ) for programme monitoring, for example:

To what extent does the evidence provided demonstrate that programme delivery meets approval and accreditation criteria?

After answering the relevant KEQ, NZQA determines an overall rating for the programme.

The rating is based on:

  • NZQA’s evaluation of the evidence provided and any issues identified
  • the TEO’s self-identification of any gaps or weaknesses
  • the level of action required to rectify the issues.

How does NZQA select programmes and TEOs for monitoring?

NZQA takes an intelligence-led, risk-based approach to programme monitoring: e.g.

  • Patterns of concerning performance identified in programmes across the tertiary education sector
  • TEO-specific issues such as poor programme delivery and/or assessment practice; a change of ownership or governance
  • Programmes with specific risk factors such as high growth rates, low fees.

Programme monitoring priorities

The current programme monitoring priorities are:

  • Business and Management focused programme from levels 5-9, including Hospitality Management and Healthcare Management programmes
  • Information Technology focused programmes at levels 5-9
  • programmes deliveref offshore and online
  • Level 7 diplomas

NZQA is also monitoring the effectiveness of internal moderation systems, and programme review processes across a range of TEOs and programme subject areas.

Who monitors the programmes?

Monitors may be NZQA staff, external (contracted) monitors, or a mixture of both.

All monitors have in-depth knowledge and experience in assessment and moderation, curriculum design, and programme development and delivery.

The monitors are responsible for:

  • leading the programme monitoring activity
  • evaluating the evidence provided
  • drafting the monitoring report
  • quality assuring moderation reports (where applicable).

NZQA also contracts moderators to verify assessment decisions in relation to programme learning outcomes. The moderators produce detailed moderation reports and NZQA summarises the findings in the overall programme monitoring report.

How does NZQA choose moderators?

NZQA moderators are contracted based on their subject area expertise, assessment and moderation experience and educational qualifications.

NZQA has a robust conflict of interest process to ensure moderators are neutral.

How does a programme monitoring activity work?

NZQA notifies the designated contact person at the TEO

NZQA contacts the person listed as the TEO’s Quality Assurance contact on the NZQA website. The TEO can nominate another staff member to be the contact person if they wish.

This person will be NZQA’s contact for confirming the programme monitoring activity and, as required, confirming the agenda, supplying documents, and liaising with the monitors.

Documentation requirements

Documentation

NZQA requests documentation associated with programme delivery and quality management. See the Appendix to the notification letter for a list of what documentation NZQA needs.

If the monitoring activity has a moderation component, NZQA also requests academic results so it can select marked learner work for verification of assessment decisions.

NZQA typically selects assessments with the heaviest course weighting and assessments conducted later in the course, if possible.

The agenda

When the monitoring activity involves interviews with TEO staff, these usually include academic staff, programme management and learners.

The TEO can adjust the timings or rearrange the order of interviews and any class observations for minimal disruption.

Interviews and class observations

Interviews

Who might be interviewed?

  • Quality and academic manager
  • Programme leaders responsible for the programme(s)
  • Academic records/administration staff
  • Key academic staff members for the programme(s) (e.g. programme delivery, teaching staff, assessors)
  • Learners currently enrolled on the programme(s).

What will NZQA ask them about?

Questions for academic manager, quality manager, programme leader, programme developer, and academic staff members may cover:

  • programme length and structure
  • changes to term/semester/block delivery
  • timetables (staff/internal and published)
  • approved learning hours
  • programme regulations
  • integration of practical and work-based components
  • rules for award of qualification
  • moderation systems
  • review of programme performance and self-assessment.

Questions for academic registrar and administration staff may cover:

  • English language requirements for international students
  • credit recognition and transfer
  • results and certification.

NZQA may also select a sample of student enrolment files to check admission processes are being applied appropriately. The monitors bring this list with them on the day.

Questions for academic staff members may cover:

  • qualifications
  • professional development
  • assessment materials
  • moderation practice
  • plagiarism prevention, detection and management
  • assessment methodology.

Questions for support staff may cover:

  • support services
  • educational resources
  • pastoral care.

Questions for current learners may cover:

  • learning experience
  • assessment experience
  • programme requirements and expectations.

Recording the interviews

The monitors ask permission to record staff interview sessions before the discussion starts. TEOs can request copies of the recordings. The monitors do not record interviews with learners.

NZQA uses the recordings only to assist in writing the programme monitoring report and destroys the recording six months after finalising the report.

Monitors observing TEO classes

ZFor onsite visits, monitors may ask to sit in on a class. The monitors are checking that the activity reflects the programme documentation and the information that the TEO publishes about the programme.

For example, that learners are attending as required, English is the medium of instruction, and qualified staff are leading the teaching activity.

After the monitoring activity

NZQA provides the TEO with a draft programme monitoring report, including a summary of moderation results.

If there is any delay NZQA will notify the TEO.

TEO and NZQA follow-up actions

If NZQA has concerns with the integrity of qualifications awarded to learners, it will decide if these concerns require immediate remedial actions or specific actions in a short timeframe.

If the report identifies areas for improvement, NZQA may agree an action plan with the TEO. NZQA follows up with the TEO to ensure the action plan is effective. Where the issues are significant, the NZQA Risk Team will also be involved.

Where the issues identified are relatively minor, NZQA expects the TEO to address any requirements or recommendations without further follow-up by NZQA.

Student data and learner work held by NZQA

NZQA holds all submitted materials, including learner work samples and enrolment data, securely and only uses it for completing the programme monitoring report.

NZQA holds copies of learner materials on file for two years (i.e. to finalise the report and allow any follow-up). Then the records are destroyed.

Learner IDs (not names) are used instead of names in the moderation reports.

The programme monitoring report

The programme monitoring report shows if the delivery and assessment of the programme continues to reflect the NZQA-approved documentation and the relevant criteria. The programme monitoring report also identifies any rule breaches.

The monitor completes the draft programme monitoring report and asks the TEO for any comments or factual accuracy. After considering this feedback, NZQA finalises the report.

Focus and structure of the draft programme monitoring report

NZQA summarises programme monitoring outcomes in a report with sections relating to one or more of the following criteria.

1. Programme structure and delivery 4.1 Criterion 3 Delivery methods
The delivery methods are adequate and appropriate, given the stated learning outcomes for the programme.
4.1 Criterion 5 Regulations
There are clear, relevant, and appropriate regulations that specify requirements for: programme length and structure.
2. Programme regulations 4.1 Criterion 5 Regulations
There are clear, relevant, and appropriate regulations that specify requirements for:
- admission
- credit recognition and transfer
- recognition of prior learning
- integration of practical and work-based components
- normal progression within the programme.
18.1 – 18.6 English language requirements for international students
3. Assessment and moderation 4.1 Criteria 5 Regulations
There are clear, relevant, and appropriate regulations that specify requirements for: assessment procedures, including authenticity of student work.
4.1 Criteria 6 Assessment and moderation
Assessment methodology is fair, valid, consistent and appropriate given the stated learning outcomes. There is an effective system for moderation of assessment materials and decisions.
6.1 Criterion 1 Assessment and moderation
The institution has the capability and capacity to ensure assessment materials and decisions are fair, valid, consistent and appropriate for the level, given the stated learning outcomes.
4. Resources 6.1 Criterion 2 Resources
The institution has the capability and capacity to support sustained delivery of the programme through appropriate academic staffing, teaching facilities, educational and physical resources, and support services.
5. Programme review 4.1 Criterion 7 Assessment and review
The institution:
- assesses the currency and content of the programme
- has adequate and effective processes for the ongoing review of the programme, taking account of the results of any review of the qualification
- has adequate and effective processes for monitoring the quality of outcomes for learners and other stakeholders, and for reviewing programme regulations and content
- updates the programme accordingly.
6.1 Criterion 4 Assessment and review
There must be adequate and effective review of programme performance and the institution’s capability to support the programme.
There must be monitoring of improvement following review, and processes for determining whether the programme should continue to be delivered.

 

The programme monitoring report recognises and acknowledges effective delivery, progress and achievement. NZQA also acknowledges development during the programme’s first delivery and the TEO’s efforts in continuous improvement.

In most monitoring report templates, NZQA categorises findings under four headings:

  1. Commendations
  2. Issues
  3. Requirements
  4. Recommendations

Commendations reinforce good practice, whereas issues indicate there is insufficient evidence for one or more of the criteria to maintain approval or accreditation.

Requirements reflect the issues identified in the report and must be actioned by the TEO.  NZQA follows up with the TEO to ensure the requirements are addressed.

TEOs do not have to implement recommendations in the report but NZQA encourages TEOs to do so.

Reporting the overall rating of the programme monitoring report

Based on the report findings, NZQA proposes an overall rating to answer therelevant programme monitoring KEQ.

For example:

Programme monitoring outcome Indicators
Meets programme criteria overall - Programme delivery continues to meet overall approval and accreditation criteria
- Education organisation is proactively managing programme quality
- Improvements have been self-identified and implemented
Meets some programme criteria - Programme delivery has some weaknesses
- Education organisation is identifying and addressing some gaps or weaknesses
- Some improvements required to meet approval and accreditation criteria
Does not meet programme criteria - Programme delivery has serious weaknesses
- Education organisation is not identifying or addressing gaps or weaknesses
- Significant improvements required to meet approval and accreditation criteria

If the overall outcome is ‘meets some’ or ‘does not meet’ programme criteria, the report will indicate how serious NZQA considers the findings to be.

Who is the intended audience of the programme monitoring report?

The primary audience for the programme monitoring report is the TEO’s management and programme leadership. It is not intended for the learners.

The following Quality Assurance Division (QAD) business units view the programme monitoring report in both draft and final states:

  • Approvals and Accreditation
  • Consistency
  • External Evaluation and Review
  • Quality Assurance Māori
  • Risk.

Programme monitoring reports are not published on the NZQA website but feed into other quality assurance processes, including Approvals and Accreditation and External Evaluation and Review.

Responding to the draft programme monitoring report

NZQA asks the TEO to check the programme monitoring report and point out what they believe to be any errors of fact. This may include disagreeing with the moderation findings.

Where applicable, the TEO may include in its response an outline of what it will do to address any concerns or areas for improvement raised in the programme monitoring report.

NZQA evaluates the TEO response and issues the TEO with a final programme monitoring report, including any amendments.

Impact on TEO applications for new approvals

Where the report identifies serious quality assurance concerns, NZQA may consider putting any new approvals on hold until the identified issues are resolved.

How does moderation for programme monitoring work?

The purpose of moderation for programme monitoring is to evaluate whether, overall, the assessment evidence provides assurance that learners meet the graduate profile.

Process

Moderators establish to what extent assessment materials and assessor decisions are fair, valid, consistent and appropriate for the level, given the stated learning outcomes. The learning outcomes are those developed by the TEO and approved by NZQA against the graduate profile.

NZQA makes an overall judgement on the TEO’s assessment evidence, based on the moderation results and the intended graduate profile.

Moderation focuses on learner achievement

NZQA only moderates learner work that has been awarded a pass/achieved grade because the focus is on providing assurance that any credentials or qualifications awarded to learners are credible.

Moderation reports

NZQA moderation reports include feedback on whether there is sufficient evidence that:

  • the learner has achieved the learning outcome(s)
  • the learner work is authentic
  • assessment materials allow learners the opportunity to achieve the learning outcome(s).

How to read moderation reports

Section 1: Assessment details

NZQA populates the moderation report template with key data from the TEO’s approved programme, including the level and credits of the components, the associated learning outcomes, the assessment type, conditions and weighting.

Section 2: Summary of moderation outcomes

This section states the overall outcome for the moderation of that component and includes:

  • the ‘agreement rate’, which is the moderator’s agreement with the assessor decisions on the achievement of learning outcomes; and
  • a brief description of the main issues (if any).

Section 2 also includes a moderation outcome summary for assessor decisions and assessment materials (assessment tasks and marking guidance). Moderators select one of three options for assessor decisions and one of three options for assessment materials:

Assessor decisions outcome options and what they mean
1 Overall, assessor decisions are fair, valid and consistent given the stated learning outcomes

This means the moderator has confidence in the assessor decisions.

Any issues are minor in nature.

2 There are some issues with the assessor decisions

This means the moderator has concerns with some assessor decisions, but the nature and scope of the issues do not undermine the credibility of learner achievement overall.

The moderator envisages that the TEO can address the issues before the next delivery of the module/course.

3 There are significant issues with the assessor decisions

This means the moderator has disagreed with all/the majority of assessor decisions.

The nature and scope of the issues undermine the credibility of learner achievement in this module/course.

 

Assessment materials outcome options and what they mean
1 Overall, assessment materials provide learners with the opportunity to achieve the learning outcomes

This means:

- assessment tasks provide learners with the opportunity to achieve the learning outcomes

- marking guidance supports fair and consistent assessment, and clearly states minimum evidence expectations for achievement of each learning outcome.

Any issues are minor in nature.

2 There are some issues with the assessment materials

This means the moderator has concerns that:

- the assessment tasks do not always provide the opportunity for learners to achieve the learning outcomes; and/or

- marking guidance does not always support fair and consistent assessment or does not clearly state minimum evidence expectations for achievement of each learning outcome.

However, the nature and scope of the issues suggest that the TEO will be able to address the issues before the next delivery of the module/course.

3 There are significant issues with the assessment materials

This means there are serious flaws in the assessment tasks or marking guidance – i.e.

- the assessment tasks do not provide learners with the opportunity to achieve the learning outcomes; and/or

- marking guidance does not support fair and consistent assessment or does not clearly state minimum evidence expectations for achievement of each learning outcome

The moderator is concerned about the capability and capacity of the TEO to ensure assessment materials are valid.

Section 3: Verification of assessor decisions

This section includes the moderator’s comments on the learner work and the moderator’s judgement about whether the learning outcomes were achieved.

Moderators note any issues in order of significance in relation to achievement of the learning outcomes. Moderators may also comment on depth of learner responses in relation to NZQF Level, and authenticity of learner work as appropriate. 

Section 4: Assessment tasks and marking guidance

Moderators check that the assessment tasks provide the opportunity for learners to achieve the learning outcomes, at the stated NZQF level.

Moderators also check that marking guidance supports fair and consistent assessment, and clearly states minimum evidence expectations for achievement of each learning outcome.

Section 5: Learner work examples

Moderators may select excerpts of learner work that support their judgement and to exemplify the issues outlined in Section 3.

 

 
Skip to main page content Accessibility page with list of access keys Home Page Site Map Contact Us newzealand.govt.nz