He whakataki | Introduction
Download a PDF version of this report [PDF, 308 KB]
NZQA is responsible for monitoring the quality and results of Transitional Industry Training Organisation (TITO) national external moderation (NEM) systems. Each year, NZQA surveys TITOs on their NEM systems and meets with TITOs to discuss the moderation data and identify common themes and good practice.
In 2018, NZQA and the 11 TITOs agreed to build on the annual survey process by developing NEM principles to provide a common framework of reference for implementing and monitoring NEM systems. NZQA established the monitoring framework with TITOs, including what evidence TITOs could use to demonstrate how they apply the NEM principles in practice.
NZQA monitored all TITOs’ NEM systems under the principle-based approach from 2019-2021.
The purpose of this summary is to:
- share findings and examples of good practice from the first round of monitoring under this framework
- identify opportunities for strengthening NEM systems and processes
- summarise recommendations for an effective NEM system.
On this page
Tukanga Arotake | The monitoring process
The principle-based monitoring approach enabled NZQA to develop a more in-depth understanding of how each TITO operates its NEM system. TITOs provided qualitative data for NZQA to evaluate each TITOs’ NEM processes and policies, how effectively these were applied and how each TITO analyses and utilises the moderation results and other findings from its moderation system. NZQA also interviewed TITO staff and organisations that participate in the TITO’s NEM system.
TITOs continued to complete the annual survey which provided quantitative data and complemented the monitoring process to inform the final summary report.
Ngā putanga | Monitoring outcomes
In relation to the TITOs NEM performance, three TITOs were rated Excellent, and eight TITOs were rated as Good.
There is convincing evidence that NEM systems operated by all 11 TITOs provide confidence in the overall quality of assessment against standards for which each TITO is the Standard Setting Body. Refer Appendix 1 for rating indicators.
TITO representatives agreed that the feedback they received through the monitoring process was very useful for their self-review and provided valuable information towards refining their NEM systems.
Ngā kitenga | Monitoring findings
Areas of good practice identified through the monitoring process
Clear processes and policies
Overall, TITOs submitted comprehensive and relevant documentation on their moderation policies, processes, and organisational structures. All TITOs provided evidence that moderation processes are well managed. Some TITOs submitted flowcharts of their consent to assess, moderation and appeal procedures which clearly outlined the responsibilities of all participants. Staff selection policies and responsibilities of the quality assurance staff were clearly outlined. Almost all organisations interviewed by NZQA confirmed that TITOs’ moderation policies and procedures were clear on what organisations need to do to meet national external moderation requirements.
Risk-based moderation approach
Most TITOs aim to post-assessment moderate their unit standards according to a 5-year plan and ensure that all organisations have been engaged in moderation within a two-year timeframe. There is convincing evidence that TITOs know their high-risk organisations and high usage standards. These are selected more frequently for moderation, for example, every year or at least every second year. NZQA recommends a risk-based post-assessment moderation approach and encourages TITOs to focus their sampling on identified high-risk organisations and high-use/high-risk standards. This is where most value can be gained from the moderation process.
Flexible post-assessment moderation methods
TITOs are implementing moderation methods that suit their providers and the assessment context. Desk-based moderation is included in all TITOs’ moderation planning and has proven to be efficient and cost-effective. Cluster group moderation has also proven to be a valuable method for engaging schools and tertiary providers in productive post-assessment moderation. Cluster moderation assists in ensuring that the large school sector is adequately engaged in the post-assessment moderation process. Furthermore, cluster moderation is a model that most schools value to maintain relationships with other teachers and it drives a community of practice model that promotes quality assessment. Post-assessment verification visits and live observations are more costly. However, these may be most appropriate for specific industry contexts. Visits also help TITOs manage the relationship with non-compliant providers.
Overall, NZQA commends TITOs who use a variety of post-assessment moderation approaches to accommodate their various provider and industry sectors.
Ensuring moderation consistency
Most of the larger TITOs employ contract moderators to ensure targeted moderation of their various industry sectors by industry experts. TITOs have robust new moderator induction and peer moderation monitoring policies in place.
Most TITOs review reports completed by new moderators before allowing them to make moderation judgements independently. NZQA supports this as an added layer of quality assurance within the NEM system.
Smaller and medium-sized TITOs employ contract moderators only for specialised industry sectors. Most moderation activities are completed instead by the TITO’s quality assurance staff. Assistance is sought from industry experts, where necessary, to ensure accurate moderation decisions. A proportion of moderation reports are usually peer-moderated by another team member or at internal verification meetings which helps to ensure consistency of moderation judgements.
Several TITOs have developed moderator manuals or guidance materials, and some host professional development events for contracted moderators to keep up to date with assessment and moderation best practice.
Promoting good practice assessment
Several TITOs facilitate annual assessor workshops which are well-received by organisations. TITOs also promote good practice assessment through guides and manuals assisting new and current assessors. Interviews with organisations support that they are in favour of attending assessor workshops wherever possible to learn about good practice examples. Cluster group moderation has also helped providers to share their assessment practices.
NZQA encourages TITOs to investigate opportunities for providers to collect assessment evidence through modern digital technology. For example, assessors can capture video evidence through a mobile phone app that moderators can access for verification. One TITO operates a Learning Management System (LMS) where assessments are submitted as essays or portfolios. These are directly moderated in the LMS. The TITO also provides its contracted moderators with specialised training for conducting moderation of assessments submitted on-line through the LMS.
TITOs also submitted suitable examples of engaging with non-compliant organisations or those with poor moderation outcomes. In most cases, this has been effective in improving their assessment practice and helped the organisations to become compliant.
Constructive pre-and post-assessment moderation reports
In general, pre-and post-assessment moderation samples provided by each TITO showed constructive and detailed moderator feedback on the validity of assessment materials and assessor decisions, and for meeting all evidence requirements of the unit standard.
Interviews with organisations confirmed that they receive useful and detailed feedback from their respective TITOs on their assessment practices, and where improvements are required.
Moderation informs the review of unit standards
Examples of standard review processes implemented following post-assessment moderation are highly encouraging. It shows that the TITO sees moderation outcomes and stakeholder feedback as a valuable part of unit standard review.
Moderation outcome monitoring
Overall, each TITO provided NZQA with evidence of suitable systems for monitoring pre- and post-assessment moderation outcomes of organisations. Pre-and post-assessment moderation data spreadsheets showed how non-compliant and at-risk providers were tracked and what follow-up actions were planned.
Areas that need strengthening
Exempt versus not moderated organisations
The exempting of organisations for consistently compliant moderation results is appropriate for acknowledging and encouraging good practice, as well as ensuring TITO moderation resources are focused more heavily in areas of higher risk. The decision to not moderate an education organisation due to ‘low’ credit reporting volumes or ‘other moderation priorities', as stated by some TITOs, is not the same as a performance-based exemption.
Each TITO needs to establish clear and transparent risk-based guidelines to ensure consistency in decision making to exempt or not moderate an education organisation in each moderation cycle.
Documenting follow up actions for non-compliant providers
Overall, NZQA is confident that TITOs are following up effectively with non-compliant providers. However, NZQA recommends TITOs maintain more detailed record keeping for non-compliant organisations or organisations with poor moderation results. Records need to show evidence that the policies and processes are being implemented appropriately. Therefore, TITOs need to ensure that supporting evidence is available if escalation to NZQA is required following CMR policies regarding organisations with ongoing and unresolved non-compliance issues. Follow up activity records should include how and when non-compliant organisations were followed up, records of all communication about the non-compliance or poor moderation, and the results of any actions implemented by the education organisation and/or TITO to rectify the issues.
Effective moderation for schools
TITOs using cluster group moderation tend to complete a higher volume of post-assessment moderation activities for schools. TITOs with lower amounts of post-assessment moderation completed could consider cluster group moderation to increase their moderation coverage.
However, cluster group moderation needs to be focused and rigorous to achieve reliable moderation outcomes, with moderation reports that are relevant to each participating organisation.
There is also an opportunity to ensure that all assessing organisations participate in the national external moderation process. This is particularly important where there are memoranda of understanding between schools reporting the results to NZQA for assessments carried out by tertiary education organisations.
In these instances, TITOs need to ensure that the relevant tertiary education organisation is moderated for these standards. All TITOs are able to see the relationship between assessing and reporting organisations by downloading the ‘quarterly report’ from the NZQA website.
Self-assessment and reflection for improvement of the NEM
Policies and processes relating to self-assessment were comprehensive for all TITOs. However, in most cases there was limited evidence to show how moderation trends and outcomes are evaluated and used to support NEM system review.
NZQA encourages TITOs to use the qualitative and quantitative outcomes of the NEM for self-assessment and reflection. Two effective examples seen through the monitoring were:
a. annual outcome reports or sector summary reports. The sector summary reports showed an analysis of moderation activities and discussion on identified issues arising from moderation outcomes.
b. a self-assessment report against the monitoring principles showed thoughtful practice on moderation processes and outcomes to inform NEM system improvement.
Supporting Mātauranga Māori based learning and assessment
Most TITOs have included in their strategic plans the aim to increase success for Māori in vocational education. A few examples were submitted by TITOs on embedding Mātauranga Māori based learning and assessment within the NEM system, such as training resources or assessment materials translated into te reo Māori. However, there are still more opportunities for TITOs to further develop Mātauranga Māori strategies in vocational training, assessment and moderation.
Examples of good evidence to demonstrate the NEM principles in practice
A key part of the first round of monitoring using the NEM principles was for TITOs to determine what evidence they could provide that demonstrated the principles in practice. Examples of good evidence for each of the principles are outlined below:
Ensure assessment practice is fair, valid and consistent
TITOs demonstrated well-established processes and procedures which include:
- Clear pre-and post-assessment moderation and consent to assess evaluation policies.
- Reports confirming that policies and procedures are consistently applied.
- Well-structured processes for moderator training and monitoring.
- TITOs with a large number of contracted moderators using manuals as guidelines to ensure consistency.
- Peer-review of moderator reports which ensure consistency around verification of assessor decisions, and review of provider appeals.
- Pre- and post-assessment reports showing constructive moderator feedback on the validity of the education organisation’s assessment practice, and sufficient assessment evidence to ensure unit standard requirements are fully met.
- Evidence of effective communication and follow-up with organisations that have not met the required standard.
Appropriate to the nature of the learning outcomes and assessment evidence collected
TITOs used a range of appropriate moderation approaches for assessment type or according to the needs of their various industry sectors, e.g.:
- Document-based moderation as an efficient method to moderate a wide variety of theory-based ('Demonstrate knowledge of/DKO’) standards and some lower level applied unit standards without compromising the validity of the moderation.
- TITOs are flexible for providers submitting their assessment materials for document-based moderation, e.g. electronic or digital submissions, or hard-copy submissions for portfolio type assessment evidence.
- Combining document-based moderation with on-site visits, as an in-depth post-assessment approach to ensure that the assessment practice for unit standards leading to trade qualifications is within industry contexts. Learner evidence is produced or collected in a document or workbook format for postal or digital moderation and followed up with on-site moderation.
- Live moderation is a more time-consuming and costly moderation activity. However, in some cases, it is essential depending on the specific industry and unit standards (e.g. high-risk/ high-level standards).
- Assessment practice must be culturally responsive, with embedding of Mātauranga Māori values where necessary.
Provide confidence that learners have achieved the required standard
TITOs presented a range of documents and verbal accounts of effective NEM practices assuring NZQA, learners, employers, communities and other stakeholders that learners have achieved the standard. These include:
- Moderator agreement with assessor decisions (agreement rates) are consistent over the last three years or showing improvement. High agreement rates provide confidence that the learner's work is meeting the required standard.
- Good evidence of effective moderator feedback and follow-up processes for non-compliant organisations, resulting in improved outcomes in the next moderation cycle.
- Evidence of unit standard review processes incorporating moderation outcomes, and interim rulings published on TITO websites to ensure assessors are implementing evidence requirements correctly.
- Evidence that TITOs employ subject matter experts to moderate in the relevant, specialised field.
Provide confidence in the reliability and consistency of assessor judgements about learner performance
TITOs presented the following examples which help to maintain reliable and consistent assessor judgements on learner performance:
- Long-term and annual moderation plans clearly show that moderation is strongly focused on high-risk providers and high-risk/high usage standards.
- Sufficient sampling across the industries to ensure a national perspective on the consistency of assessment.
- Annual moderation covers all education sectors and minimises the number of providers who are not moderated unless they are exempt due to sustained good practice assessment.
- Thorough pre-assessment moderation of provider developed assessment materials.
- Effective detection, recording and follow-up of providers using unapproved assessment materials.
- Annual moderation data reports verify that the annual moderation is completed according to moderation plan.
Cost effective
TITOs demonstrated cost effectiveness of their national external moderation system, where the volume of pre- and post-assessment moderation activity is relative to expenditure. The following aspects are indicators to support cost-effectiveness:
- an awareness of expenditure per assessment sample and per post-assessment moderated unit standard
- whether expenditure is justified, i.e. reflects the volume and/or method of moderation completed
- how each TITO’s moderation costs and activities completed compare to the previous two years
- how each TITO’s moderation expenditure compares with other TITOs.
Focus on improving assessment practice
TITOs focused on improving organisations’ assessment practices by using a variety of approaches, such as:
- Effective communication with organisations that had unsatisfactory moderation outcomes.
- Evidence of following up and supporting non-compliant organisations with targeted one-on-one guidance with well-documented results of improving assessment practice and compliance.
- Providing examples to organisations of good practice assessment through workshops, webinars and written publications.
Revision of the eight NEM principles
NZQA revised the NEM principles to incorporate findings from the TITO NEM monitoring process. The main change was to remove principles g) and h) to decrease duplication of information within the principles, and between the principles and Consent and Moderation Requirements documents. NZQA also made some minor wording changes in some of the criteria statements under the principle headings.
Tūtohutanga | Summary of recommendations for an effective NEM system
The new monitoring process allowed NZQA the opportunity to evaluate the variety of systems and processes each TITO operates. NZQA also identified some common themes that indicate well-managed NEM systems.
An effective, well-managed NEM system:
- ensures that moderation sampling focuses primarily on the provider and its ongoing capability in valid and consistent assessment of the TITO’s standards.
- ensures that underperforming providers are moderated in more depth and breadth than those with proven good practice assessment and positive NEM outcomes.
- ensures that high risk, high usage, high-level and high-credit value unit standards are prioritised for moderation.
- covers all education sectors and industry domains.
- includes clear and transparent policies and procedures for exempting providers from moderation, or when not to moderate providers.
- uses pre- and post-assessment moderation report templates that prompt the moderator to comment on all evidence requirements, including range items and special/explanatory notes.
- reliably tracks provider moderation history, to easily identify non-compliant providers.
- systematically shares good practice principles with assessing organisations.
- values the importance of pre-assessment moderation to minimise the risk of poor assessment practice.
- delivers accurate and consistent moderation judgements, assists moderators with training and opportunities for check-moderation and peer-review.
- uses technology and software solutions effectively for moderation planning, assessment material submissions, moderation result reporting, and data analysis of moderation outcomes.
- provides actionable feedback to education organisations and assessors and assists them in developing fair, valid and consistent assessment practice.
- promotes and supports Mātauranga Māori based learning and assessment.
- is reviewed regularly to evaluate overall effectiveness and trends in moderation outcomes and ensures that self-review informs moderation planning for the following year/moderation cycle.
- feeds moderation findings into standard and qualification reviews.