Digitisation of a public benefit project on a nationwide scale remains a lofty goal today – and in 2015, when the partnership between ACARA, ESA and Janison began, it was even more so. There would be zero tolerance for failure with such a public undertaking. Therefore, Janison needed a cloud partner with the same professionalism and quality. That partner would be Microsoft, bringing its ability to host a proposed cloud solution in regional Azure data centres.

The idea took shape to the credit of the visionaries in the room back in 2015. It became the incredible journey we know today as the digital transformation of NAPLAN.

Students of the early 2000s and before that time would undoubtedly be familiar with the typical conversation on leaving an exam room.

“Did you find that hard? I thought I wasn’t going to finish on time!”
“No, I found that one easy. I had nothing to do for the last 30 minutes.”

Perhaps these two theoretical students achieved the same results. Yet, no educator would know that one maybe needs a different level of support from the other.

For many years before digitisation, for both high and lower performers, paper-based school examinations, in their static physical format, had inherent limitations when it came to measuring a more nuanced grasp of what students knew or didn’t know. Every student sat the same test. Therefore, it was necessary to pitch the pen-and-paper NAPLAN test questions at the middle of the scope of students’ capability, with some extension at either end.

Add to this that students were already overwhelmingly digital in their everyday lives, choosing texts or instant messaging over writing or calling, streaming content over watching television, et cetera, to the point that educators could no longer ignore these preferences.

“It’s clear that young people are more engaged by technology than by pen and paper,” says Andrew Smith, ESA Chief Executive. “Therefore, we understood that we could better engage students in the test using technology.”

Modernising student engagement was just one of the objectives for the digitisation of NAPLAN, which was first introduced in 2008 to consistently monitor literacy and numeracy levels of Year 3, 5, 7 and 9 students. As years passed, the limitations of manual exam delivery grew, falling into a number of broad categories:

• Equity: Equity of access is imperative for a national testing program, especially in a country the size of Australia, with its dispersed population. That said, its varying degrees of internet bandwidth meant equity would not automatically be a given with digitisation either – a point that would require bespoke solutions for certain locations.
• Performance: The pressing matter of NAPLAN results turnaround times had always been frustrating for schools, parents and students. It could take up to three to four months for students to receive their results from the pen-and-paper format of the test – delaying teachers’ ability to plan and implement targeted teaching interventions.
• Security: Of course, there was also the opportunity to address the vulnerability of paper forms to breaches in security protocols, such as exams falling into the wrong hands or test booklets going missing in transit altogether.
• Scalability and futureproofing: Purposefully designing the system to be able to scale and flex to meet increasing student populations and to incorporate new test requirements was necessary for futureproofing, not to mention avoiding the need for a costly reconstruction further down the line.

All these factors became critical drivers for digitisation.

Of course, such a significant change affecting the nationwide schooling system called for a substantial technological effort requiring cooperation across numerous parties. Eight jurisdictions, states and territories, plus the Commonwealth, would need to agree on key aspects of the program collectively, or else it could not move forward. And while this undertaking would be huge at the system level, schools’ individual change management processes would arguably be more significant.

Still, a countrywide NAPLAN examination system, delivered in a relatable way for students and able to adapt to the style and aptitude of individuals, represented a compelling opportunity. ESA, with the support of Janison and Microsoft, forged ahead.

ESA put out feelers for an organisation capable of delivering a digital NAPLAN test and selected Janison as its delivery partner. Together they engaged multiple jurisdictions to gather functional specifications and agree on priority processes. Educators and governments already trusted Janison’s digital assessment platform Insights to deliver, mark and analyse tests for hundreds to millions of test takers across the years, which made customising it the perfect starting point.

Consistency would be essential, even though to some extent states and territories had evolved their own NAPLAN delivery processes independently over time. However, the top priority would be to ensure, through technology, that the student experience would be as consistent and smooth as possible with no interruptions nor distraction from the tools used to deliver it.

“The student experience has been the priority since the program’s inception,” says Smith. “Some children enjoy a test setting; others find it stressful. We needed to accommodate them all.”

“We’re also conscious that teachers administering the test within classrooms and within schools have varying levels of comfort with using technology, so it was also a significant priority to create interfaces and administrative processes that could be easily managed by a teacher.”

Equally, ensuring the project’s success meant individual schools upgrading their internet connectivity, local infrastructure and hardware to handle many simultaneous students sufficiently. They would also need to train administrators and implement programs to raise digital literacy among school staff. Therefore, a part of the challenge would be coordinating test-ready preparation nationwide.

“So as much as there was great excitement about the opportunity, there was certainly an understanding that this was not going to be a simple task because of the amount of change it might require across education at all levels,” says Smith.

Rising to the challenge, Janison sat down with its technology partner Microsoft to architect a platform that would meet these current needs and scale for the future – whatever that might bring. “Most solutions get re-architected and retooled every couple of years. For us, it’s a constant process with Microsoft to figure out what will be the next big technology development and how we should incorporate it,” says Welsh.

Scalability is a big part of that. With populations increasing and digital tests evolving, a digital NAPLAN absolutely must be able to scale seamlessly without refactoring the underlying technology infrastructure. “We used around 35 different types of Azure services to build the NAPLAN Online platform. One of those is a microservices layer that allows us to achieve elastic scale to support NAPLAN far into the future,” Welsh says.

“We leaned on Microsoft’s capacity for hyper-scale clouds that could enable the heavy lifting required for NAPLAN. And we tested far beyond current expectations for concurrent student access, to 450,000. In some cases, we went even further, testing to 600,000 while leveraging the scalability of Microsoft Azure to deliver that sort of volume. We can’t think of anybody else in the world managing that volume of simultaneous delivery for a school exam,” he adds.

Aside from delivery, there was test content to consider.

The Australian Curriculum, Assessment and Reporting Authority (ACARA), is responsible for developing, delivering and reporting on the national curriculum and national assessment program (NAPLAN) for Australian students.

Digitising NAPLAN provided the opportunity to do more than simply lifting and dropping existing tests. ACARA had the chance to modernise the test format significantly; therefore, Janison would need to deeply understand factors like test design and security requirements, which are a top priority in education delivery.

ACARA was most excited by the prospect of an adaptive test, capable of responding to the answers students gave by serving them a set of questions that were either a bit harder or a bit easier, depending on how they were progressing. NAPLAN could achieve adaptive testing via an algorithm embedded within the platform software that intelligently adjusts the difficulty of test questions by interpreting answers at crucial points.

“There are always students breezing through their test, completing it in no time and getting everything right. Yet, we don’t know what they’re fully capable of. Adaptive testing allows us to stretch those students by serving them more challenging questions to get a better understanding of exactly what their level of attainment can be,” says Smith.

“Just as importantly, there are plenty of students who, for various reasons, might not answer a single question on a pen-and-paper test, or they might answer just one or two. The adaptive test engages them far more than pen and paper and boosts their confidence by directing them into an easier set of questions if that’s what their responses indicate.”

Transitioning NAPLAN online presented the challenge of what to do in the event of a temporary loss of internet services. Anyone who has neglected to switch on autosaving of a Microsoft Word document on a personal computer can relate to the stress of losing unsaved work. However, loss of work is unimaginable within such an important national assessment event as NAPLAN. Therefore, Janison, on behalf of ESA, prioritised test recoverability in architecting the NAPLAN Online solution. This meant introducing protocols for auto-saving students’ test responses offline at high frequency – approximately every 30 seconds – so that only seconds of work would be missing in the event of an internet dropout, if any at all.

Meanwhile, other schools in more remote parts of Australia suffer from a near-permanent lack of internet connectivity. So being able to develop the low bandwidth solution for them was also an important part of the data-loss prevention effort.

In such a situation, Janison’s ‘low and no bandwidth’ test player continues to run students’ exams even if the school’s internet or network connection drops, and then will sync back after the test when the network connection can be restored, all whilst securely storing the encrypted test data to assure its validity. It gives the student an uninterrupted, distraction-free experience no matter the connectivity. As such it not only protects schools against data loss but fosters equity, providing a level playing field for test-takers in rural and regional areas which have less dependable internet coverage.

“Somebody might dig up the internet pipe that supplies a whole lot of schools by accident, or there could be outages from various providers at different times, causing computers to drop off their local Wi-Fi network and reconnect later in time. So because there was an understanding that this could be the case, the system was designed to upload information every 30 seconds. Should something go wrong, you lose no more than 30 seconds of students’ work,” says Smith. “Students can pick up where they left off.”

This provision would become a significant driver of equitable education in lower or patchy connectivity areas, enabling them to benefit from a digital test solution.

Rigorous field research and testing were key to user centricity, increasing the odds of successful adoption. Aspects of the digital test delivery prototype as it progressed worked well on paper but not in reality. Therefore, it was important for the NAPLAN Online team to visit schools with non-standard environments physically.

“A few of us travelled to remote schools in the Northern Territory to see their learning environments first-hand. Internet connectivity was one of the things we were looking at, and how confident they were with technology. We would come away from visits and redesign part of the solution. First-hand knowledge of the environment in which students would be taking their test showed us that some of the ideas that we had developed were not going to work in these environments,” says Smith.

Visits like these ensured that Smith and the team could put their heads together to develop the right combination of flexibility, processes and system functionality to make remote test-taking viable for everyone. This includes the solution being device-agnostic, supporting a wide range of device types and operating systems.

“This approach, known as user-centred design, was about listening to communities and working with the local education department to meet its needs,” says Smith.

Two years of planning and building resulted in a staged rollout, which acknowledged that some schools were more ready than others to embrace NAPLAN Online. Smith says: “The thinking behind a staged approach was that we could build confidence among stakeholders in the test itself, but also different schools or different stages of readiness. Some schools were equipped to come online early, so we wanted to give them that opportunity. Others required more time to build their infrastructure and confidence to the point where they could undertake the online test seamlessly.”

2016: In 2016, Janison entered into a seven-year agreement with Education Services Australia (ESA) on behalf of the Australian Federal Government to become the software platform provider for the online version of the National Assessment Platform – Literacy and Numeracy (NAPLAN), a national, annual assessment for all school children in Years 3, 5, 7 and 9.

2018: In a world-first, in May 2018, the platform successfully delivered the NAPLAN Online assessment to 1,400 schools with a total of 200,000 school students over a nine-day assessment window nationwide. Speaking at the time of the test delivery period in May 2018, Janison’s then-head of operations Sandy MacDonald said: “An assessment event such as this requires a highly scalable solution, developed in a way where no individual computer is overloaded. We’ve spread the load-balancing across disparate nodes of computers.”

Welsh reflects on this crucial moment in the project: “Our first measure of success at that point of going live for the first time was whether students could log on and navigate tests consistently, without the system slowing down. That success happened.

“Many had thought what we were aiming to achieve impossible, but we proved them wrong with NAPLAN Online, showing that a platform of our scale can work in the assessment space. I think we all breathed a sigh of relief and gained the confidence to pursue our plans for a countrywide rollout.”

2019: In 2019, Janison continued the momentum, delivering NAPLAN Online to approximately 660,000 students, with the remainder of students continuing to take the test on paper. This hybrid approach and measured rollout enabled the consortium to monitor the transition for authenticity.

The subsequent decision by education ministers to cancel NAPLAN in 2020 due to COVID-19 allowed more time for Janison, in partnership with ESA, to work on improving and strengthening the platform in readiness for NAPLAN 2021.

2021: In May 2021, still in the thick of the pandemic, approximately 70 per cent of Year 3, 5, 7 and 9 students sat the new digital NAPLAN tests. That’s 800,000 school students and more than three million tests, all completed within two weeks. Janison supported more than 195,000 concurrent NAPLAN tests at its peak, with more than 32,000 transactions per second.

2022 and beyond: Test statistics in 2022 set a record for peak concurrent student numbers – believed to be larger than any other online school assessment globally. In the second week of May 2022, the platform delivered 4.3 million tests (1m+ in a single day) for 1.2 million school students. At its peak, more than 315,000 students simultaneously sat the exam without issue. This has reaffirmed Janison’s market-leading position of seamlessly delivering the highest level of security, reliability and scalable infrastructure for online exams in an equitable and accessible manner for all students.

In fact, ministers had planned the 100 per cent milestone for 2021, but COVID-19 and lockdown conditions pushed the date forward by one year. It was an acid test of both the digital platform developed by Janison and the Microsoft Azure cloud computing platform. Educators everywhere held their breath before learning the solution had passed with flying colours nationwide.

Reflecting on the task of bringing such a complex test online for everyone everywhere, Smith attributes success to the strength of what he describes as “a genuine partnership” between ESA, Janison and Microsoft project team members. Each had a stake in open and regular communications.

“The dedication, the skill, the commitment that the teams show to this program amazes me every time I see them working together,” Smith says.

“You also can’t underestimate the level of monitoring needed to ensure that we understand what’s happening within the system at any given time, to the greatest extent possible,” Smith says. “So a level of monitoring becomes important because of the variety of environments we’re delivering into.”

While it’s early days in the 100% nationwide digital life of NAPLAN, the collaboration has brought welcome results.

Algorithms built into the test flow make tests adaptive, allowing every student a chance to give their best effort on the day because NAPLAN is serving up the appropriate following questions based on their answers. “Janison was founded on learning management systems, and we built some of that quizzing technology within the platform,” says Welsh.

Accessibility is another critical equaliser. NAPLAN Online was designed for multiple accessibility challenges, even accounting for those students who are more comfortable using an iPad than a laptop. Janison leveraged some of the accessibility features in Microsoft to ensure tests were openly available. Artificial intelligence will become key in helping ACARA understand the impact of those accessibility measures and refine them for the future.

“It’s not just about providing for people who might have visual impairments, different colour perception and other access needs such as Braille keyboards. It’s also about having the same privileges as others. For example, an Indigenous school in remote Australia may have low device numbers and poor connectivity. We designed NAPLAN Online with those challenges in mind,” says Welsh.

For example, Janison’s low and no bandwidth solution is proving essential to the ability of rural and regional schools to participate in NAPLAN in times of unreliable connectivity. It has made all the difference to some schools whose remote location had even precluded them from participating in the paper version of NAPLAN.

“This all comes back to equity, meaning young people all having the opportunity to do the same test under the same conditions so that we get a better understanding of what they can and can’t do and give teachers that guidance about what the next learning steps for those young people are,” says Smith.

The risk of physical exam booklets going missing in transit has also been resolved. Lost mail was always a possibility wherever there were manual transport logistics, and the risk is higher for schools with long and multi-phase mail journeys. Students are now not only assured of an uninterrupted test experience and safe treatment of their academic records, but also a shorter wait for their results being returned to schools.

Seamless performance is vital in test scenarios. The ultimate performance measure is that the platform continues to operate well throughout the exam period. With NAPLAN now digitised, students can sit tests online on a range of devices, wherever they are based. Microsoft Azure locations around Australia help avoid latency issues, and there are measures in place to ensure test continuity and prevent data loss.

Speaking of the procedures for managing any incidents, including internet outages, ACARA says:

“In the event of disruptions during testing, schools can pause and resume testing and, if required, reschedule testing. All student work is automatically backed up and saved every 30 seconds. The test administration authorities in each state and territory are well placed to manage any local issues that arise.”

Another key performance indicator is more granular data and results, delivered at speed so that schools are not waiting for critical teaching direction. Automated scoring of all but the writing test means ACARA can produce results and report back to teachers in a few weeks versus a few months. “From next year, the expectation is results will be back with the schools within a month,” says Smith.

Speed guarantees have enabled ministers to move NAPLAN tests forward from May to March, commencing in 2023. “Bringing the test forward into Term One allows even more time for teachers to work with the results,” says Smith. “We probably wouldn’t have done that before due to logistical challenges. For example, distributing test papers to remote parts of the country during the wet season.”

Due to the incredibly sensitive nature of data and privacy concerns relating to young students, the assessment platform must meet rigorous privacy and security requirements. The NAPLAN Online platform complies with Europe’s General Data Protection Regulation (GDPR), the Australian Government’s Information Security Manual, and the ISO 27001 information security standard.

“Delivering NAPLAN involves the collection and distribution of student test results on a significant scale. I think protecting young people’s data is essential, and much work goes into securing that information,” says Welsh, “so that means we’re entrusted with a monumental responsibility to safeguard this.”

Equally important was ensuring test submissions could not be lost or tampered with. Janison Insights’ in-built audit trail for recording every touch point would effectively eliminate the human error element of misplaced papers.

As with any major change in process there are additional impacts that sit outside of primary priorities. In this case it was environmental benefit. A high-level audit of paper saved and environmental cost avoided through lack of transportation presented impressive numbers.

“You’re talking about 30 million pieces of paper saved from digital delivery by going online. And from a sustainability perspective, it’s not just the pieces of paper; it’s shipping the pieces of paper that is also incredibly detrimental to the environment. None of that exists anymore. So that’s quite an achievement,” says Welsh.

The incredible undertaking of transitioning NAPLAN online yielded several learnings. “I think we’ve learned the value of being more flexible in designing and delivering a complex program like this,” says Smith. “It also reminded us of the critical importance of people working well together in the same space. And I think we learned how to work remotely while still collaborating closely.”

“NAPLAN Online is not a technology program. It’s fundamentally an education program,” says Welsh. “It’s about understanding what young people in Australia can and can’t do regarding their education. It assesses their knowledge and skills, allowing teachers to work directly with that understanding. Bringing NAPLAN online opens the door to all sorts of possibilities regarding how we give young people and their teachers even better information about students’ literacy and numeracy.”

The most meaningful marker of NAPLAN’s digital success is the feedback received from state and territory education departments, whose schools report an overwhelmingly positive experience for students and teachers alike.

Wendy Powson, principal of Lilydale High School, reported that her school had an encouragingly positive experience with the online tests:

“It took an awful lot of organisation, as you can imagine. We had to make sure kids who didn’t have devices had one,” she said. “What we are looking forward to is having the results coming back a bit early, so it can inform teaching.”

Principal of Mount Druitt School, Glenn Patchell, said the school, located in Sydney’s outer western suburbs, never let its postcode determine its success.

“We still have high expectations of the kids, we have focused teaching, we have engaged learning,” and “we always use the data to inform our teaching.”

The Department was looking for a specialist digital assessment provider to help it achieve the key aim of the new assessment: to quickly identify students at risk of falling behind and pinpoint what type of support they need. The window for the opt-in assessment, called Best Start Year 7, would run over 31 school days in February-March 2019, for more than 40,000 students across the state.

As well as offering government-grade reliability and security, the solution needed to be Software-as-a-Service (SaaS) that scales easily year to year and seamlessly integrates with all schools’ existing systems. Examiners also required the ability to author and deliver a number of different question types to create an engaging test for school students.

As acknowledged by former Minister for Education, Adrian Piccoli, in the strategy document, literacy and numeracy skills are the foundation for the next generation’s success in learning and in life. It’s critical for students to develop confidence in these skills, and falling behind can have devastating consequences for their secondary schooling.

The Department entrusted Janison to deliver technology that could provide precise evaluation of students’ skills at a critical juncture in their education – the first weeks of Year 7 – and quickly provide intricate diagnostic data to inform teaching intervention.

As sophisticated as the technology needed to be, it also needed to be user friendly to cater to all levels of technical literacy among teaching staff. It also had to be built to operate seamlessly within greatly varying levels of IT infrastructure between the 321 schools across the state taking part.

Best Start Year 7 demanded a sophisticated online platform that could efficiently deliver and process a considerable daily number of student test completions and feed a large volume of real-time data into NSW DoE systems, making valuable analytical information available for teachers almost instantly.

Janison partnered with the NSW DoE and tailored our cloud-based platform, Janison Insights, which delivers assessments for governments around the world, to deliver on these key needs.

Through collaborative weekly video conferences and conversations with key NSW DoE stakeholders, Janison’s project team pinpointed the necessary features and adaptations to allow teachers to more easily perform tasks within the system, including authoring tools that allowed for a wide selection of question types. During the project meetings, Janison and NSW DoE clearly demarcated responsibility in managing the school IT networks, to allow for the smoothest possible running of the exam window.

A NSW DoE survey of 293 early adopter schools who took part in a Best Start Year 7 professional training pilot, found that more than 95 per cent agreed that the exam will help to identify students requiring additional support in literacy and numeracy.

Janison hosted the live Best Start Year 7 assessment event over 31 school days, which comprised three streams: reading, numeracy and writing. We delivered the end-to-end online assessment solution to schools across the state, including the set-up, configuration, hosting, data extract, support and marking.

A total of 41,000 students sat these assessments, with an average of 2,400 tests completed daily. The highest number of tests in one day during the exam period was 6,673.

The system auto-marked the literacy and numeracy portions, enabling real-time data delivery to the NSW DoE, while the platform’s discrete marking features streamlined the marking process for the teachers who marked the writing portion of the exam. The platform operated seamlessly with schools’ existing IT infrastructure, including student devices such as laptops, desktops, PCs, Chromebooks, iPads and Android tablets.

Janison’s operations team carried out real-time monitoring of the infrastructure and application performance throughout the 31-day window, to ensure that any potential issues could be dealt with instantly. There were no delivery problems or slowdowns reported and NSW DoE stakeholders commended the smooth end-to-end delivery of the assessment and analysis data.

Speaking of the assessment event, a Department spokesman said: “Janison has worked collaboratively with the Best Start Year 7 team to successfully deliver an online literacy and numeracy assessment to NSW Department of Education students.”

“Teachers administering the test found the system easy to use and the online format successfully engaged students to demonstrate their skills and understanding.”

At the time, delivering a large scale online school assessment was unprecedented. Supporting tens of thousands of students on the platform simultaneously posed a variety of technological challenges, particularly managing data loads. Stable, consistent delivery was vital. The platform would also need to meet the Government’s strict data sovereignty requirements.

For the platform, we addressed key scalability, delivery and security challenges by taking a unique approach to services and architecture. We addressed data sovereignty by designing the application so that a minimum and acceptable amount of student data was stored in the cloud server environment, and only for the duration of the event.

We built multi-media tools into the exam authoring features. These allowed educators to author questions using animation that simulates practical classroom exercises, allowing students to work with virtual beakers, test tubes and measurement tools on the screen.

In October 2010, Janison delivered what was then a world-first: the first standardised, successful, large-scale online assessment to nearly 40,000 students from 650 schools in one day. In 2015, more than 100,000 test attempts were delivered and marked. By 2016, the test program had doubled to 81,300 students.

Feedback from teachers involved in the early stages of the project was that the Validation of Assessment for Learning and Individual Development (VALID) had transformed from being seen by students as a dull and dreaded exam into a test that pupils now actually looked forward to. Teachers report that they’ve never seen some of their students as engaged in a classroom activity as they are when they’re participating in VALID.

The VALID online assessment platform set the standard for Janison’s digital exams solutions for schools and we’re continually advancing its authoring, delivery, analysis and marking features. Due to the success of the approach, the NSW DoE has asked us to apply the VALID solution framework to a number of its other schools assessment projects.