How to Measure the Quality of Learning in Courses: Netology's Approach

The online education market is actively searching for universal methods for assessing the quality of courses that will serve as a benchmark for both educational companies and clients of online schools. In early April, at the Moscow International Education Fair (MIEF), a major joint project between the HSE Institute of Education and the Skillbox team was announced. The project aims to develop a new model for assessing the quality of online education. As part of this project, participants presented the results of the research phase, which included an analysis and synthesis of existing global models for assessing the quality of educational technologies and products. This important undertaking aims to improve online learning standards and ensure a higher degree of user trust.

The relevance of work in the field of online learning cannot be underestimated, as there is currently no single, generally accepted definition of quality in this field. The concept of quality covers a wide range of aspects, and each company in the EdTech market evaluates its courses according to unique criteria. This diversity of approaches to quality assessment creates a complex situation for users seeking reliable and effective educational resources. Therefore, it is important to develop universal standards that will help improve the quality of online learning and make it accessible to everyone.

Within a single company, different departments may interpret the concept of a "quality educational product" differently. Students also have their own perceptions of quality online learning, which influences their course evaluations. Is it important to create a unified system that integrates these different approaches and allows for objective measurement of the quality of the educational process?

Maria Rakultseva, an educational product methodologist in the Netology Strategy and Results Team, shared her approach to solving a pressing problem at the conference "Quality EdTech: Designing the Student Experience," organized by the School of Education, Otus, and Netology. In her presentation, the speaker highlighted key points regarding the development of effective educational solutions aimed at improving the student experience. The focus was on practical methods and strategies that help improve the quality of online learning and adapt it to student needs.

How Netology Researched Perceptions of Quality

Maria shared that she joined the Netology team six months ago. At the time, the company lacked a unified product quality assurance process or a data-driven product decision-making system.

The online school offers a wide range of courses, varying in format, topic, duration, student level, and goals. Teams working in different areas evaluate and interpret the quality of the educational product differently. Therefore, as Maria emphasized, it is important to create a universal evaluation framework while taking into account the unique features of individual courses. This will ensure high educational standards and meet student needs.

Maria and her team initiated research to align the company's understanding of online course quality and ensure that they meet the expectations of the target audience—students. Two studies were conducted as part of this work: an internal study focused on employee opinions and an external study focused on student views. This provided a deeper understanding of what students consider quality in online education and what aspects needed improvement. The research results will help the company optimize courses, increase their attractiveness for students and improve the overall quality of educational services.

Photo: fizkes / Shutterstock

The internal study included 22 in-depth interviews with the company's managers and employees. The surveys included producers, product managers, methodologists, expert relationship managers, and members of the support and maintenance team. This research provided valuable insights into the company's internal structure and processes, which contribute to improved collaboration and overall efficiency.

The external study consisted of two stages. In the first stage, Maria's team conducted 13 in-depth interviews with students. The sample included both students who positively evaluated their studies at Netology and those who had decided to return the course, as well as students from other online schools. This diversity of opinions allows for a complete understanding of the quality of the educational programs and student satisfaction.

In the second stage of the study, Maria and her colleagues conducted a survey of 1,019 students of the Netology online school. Survey participants were those who had been studying at the school for at least one month or completed their studies no more than six months ago.

According to the research conducted, the company's employees believe that a high-quality educational product must meet certain criteria. It is important that it is relevant, accessible, and practical, providing the opportunity to apply acquired knowledge in practice. Learning materials should be structured and easy to digest, and also offer a variety of learning formats, including interactive elements. Furthermore, the product should be tailored to the individual needs of students and provide support throughout the learning process. These aspects play a key role in creating an effective educational experience and enhancing the professional skills of employees.

it leads the student to the goal;
meets market demand;
meets student expectations and provides good feedback;
solves educational problems;
contains competent materials;
the course is attended by good experts;
the result stated in the course positioning corresponds to reality.

The results of interviews and a student survey demonstrated that the key factors for a successful course are: the quality of materials, accessibility of teachers, interactive teaching methods, and the opportunity to receive feedback. Students noted that the relevance of the information and its compliance with modern requirements are also important aspects. Paying attention to these aspects significantly increases the effectiveness of learning and student satisfaction with the course.

relevance of the content;
good teacher;
detailed feedback during the learning process;
accessible presentation of the material;
plenty of practice and work on real projects;
optimal pace and format of learning;
good support.

Students were also asked about the degree of expression of various factors in the Netology courses. Maria noted that the data obtained became a useful tool for identifying the product’s strengths, as well as areas requiring further improvement.

Find out more:

Before designing a course, a methodologist needs to conduct several key studies to ensure successful training. First of all, it is important to determine the target audience. This will help understand what knowledge and skills students need, as well as their level of preparation. It is also necessary to study the needs and interests of students to ensure the course is as relevant and useful as possible.

The next step is to analyze existing educational resources and courses. This will help identify best practices and avoid duplication of materials. The methodologist can conduct a comparative analysis to determine the strengths and weaknesses of competing courses.

It is also worth paying attention to trends in education and technology. Studying the latest teaching methods and approaches will help introduce innovations into the designed course.

Research into learning formats is equally important. The methodologist must determine which teaching methods and tools will be most effective for the target audience. This may include both traditional and modern approaches, such as blended learning or online courses.

Conducting surveys or focus groups among potential students will provide valuable information about preferences and expectations. The data obtained will help make the course more tailored to the needs of learners.

So, to successfully design a course, a methodologist needs to conduct target audience research, analyze existing resources, and study educational trends and learning format preferences. These steps will help create a relevant and effective educational product.

Maria and her colleagues compared the factors identified through internal and external research to create a complete picture of the quality of the educational course. They also associated each key factor with a measurable indicator. According to the speaker, for some factors, obvious metrics can be used, such as the Customer Satisfaction Index (CSI), which helps assess student satisfaction with various aspects of learning—from interaction with instructors to the speed of homework assessment. However, for other factors, such as assessing the relevance of content, choosing an appropriate metric proved more difficult, as a specific metric for this purpose did not exist.

At Netology, we reviewed all previously recorded quality metrics to assess their validity and representativeness. Maria's team identified the most significant and reliable metrics. These metrics were integrated into a universal SJM—a student journey map. This map clearly shows at what stage of the student's interaction with the course each metric is measured and the conclusions that can be drawn from the information obtained. This helps improve the quality of educational processes and enhance learning effectiveness.

Maria and her team analyzed the metrics that are important to Russian and international EdTech markets. They examined current trends in assessing the quality of educational products, which allows for a better understanding of the methods used to analyze the effectiveness of educational technologies and programs.

The results of this work led to the development of a new quality control system based on a hierarchical metrics matrix. This system ensures effective monitoring and quality analysis, allowing for a more accurate performance assessment and identification of areas for improvement. Using a hierarchical metrics matrix facilitates a structured approach to quality control, which in turn improves the accuracy and reliability of the data obtained.

How the new quality control system works at Netology

The new evaluation system focuses on the quality of the educational product and its connection to revenue. As Maria noted, quality and financial components are inextricably linked, as a high-quality product promotes long-term student retention. This, in turn, encourages repeat purchases; for example, after completing a professional retraining course, they may be interested in courses for further professional development or skills development. Furthermore, course quality has a significant impact on referral rates, which reflects the percentage of new students who come to the school based on recommendations from graduates. High-quality educational programs not only increase revenue but also build a strong reputation for the educational institution.

The developed metrics matrix is a tree-like structure with horizontal levels and vertical branches. The primary target metric is revenue, which is influenced by first-level metrics—referrals and repeat purchases. These metrics, in turn, depend on the second-level metrics, which include:

the percentage of returns due to students being dissatisfied with the product quality;
NPS (Net Promoter Score) of the product - an indicator of students' willingness to recommend courses to others;
overall CSI - an index of student satisfaction with the school's courses;
retention rate - a user retention rate that reflects student engagement and their activity in interacting with the courses;
students' achievement of their goals as a result of their studies.

Maria noted that these metrics are the most relevant to the company's current objectives and accurately reflect the results of the research conducted.

As you move from one level to another, the indicators become more detailed. It starts with general data for the school and gradually goes deeper into individual areas, sets of courses, specific courses, classes, modules, and content units. Responsible employees monitor the indicators at each level. At the first level, these are product managers, at the second, educational product managers and team leads, and at the third and subsequent levels, instructional designers, expert managers, and coordinators. This approach ensures high-quality education and promptly responds to changes in the educational process. There is an interconnectedness between various groups of metrics, which are organized into separate branches. For example, the Customer Satisfaction Index (CSI) is subdivided into more detailed metrics related to expert performance, content quality, and support level. At subsequent levels, the level of detail increases, allowing for CSI metrics to be derived for individual modules and content units. This structuring of metrics helps to analyze and improve each component in more depth, which ultimately affects the overall quality of services provided.

Image: slide from the presentation by Maria Rakultseva / "Netologia"

In Netology has developed core principles for operating the system and making product decisions. These principles help optimize processes, improve user interactions, and enhance product development efficiency. By following these guidelines, the team ensures high-quality service and customer satisfaction.

Work on improving quality can be done both top-down and bottom-up. For example, if an indicator at the upper levels has dropped or increased unexpectedly, you can drill down along the corresponding "branch" and determine the cause. You can also track how changes at lower levels of the system affect the indicators above.
Each employee responsible for product quality at a given level makes decisions based on the relevant indicators, metric links, and their dynamics, and acts within their area of responsibility.
When making decisions, the corresponding level and "branch" are taken into account—that is, again, to solve a problem, you need to see its place in the overall structure and consider what influences the indicators at a particular level.

Netology is currently actively researching the effectiveness of the new system. The speaker emphasizes the importance of increasing the speed of decision-making and the impact of these changes on the quality of educational products. In 2024, the online school team intends to learn to predict the behavior of key metrics. This means that when changing one metric, for example, raising or lowering it, they want to understand how it will affect other indicators. The third part of the study also examines employer perceptions of course quality. The speaker noted the importance of viewing students not only as consumers of educational products, but also as "products" that schools bring to the labor market. This will allow for better tailoring of courses to employer requirements and increase graduate competitiveness.

What's wrong with the COR metric?

Owners of online schools familiar with key metrics noticed the absence of the popular COR (Completion Rate) metric in the model proposed by Maria Rakultseva. This metric reflects the percentage of students who complete their studies. During another discussion at the "Designing Reskill vs. Designing Upskill" conference, the topic of using COR at Netology was raised. Arina Pantina, Educational Product Manager for Creative Industries at Netology, shared her thoughts on this issue. Discussing metrics such as COR is important for optimizing educational processes and improving the quality of online learning. Arina noted that at Netology, COR is used as a supporting tool for analyzing changes in other metrics. She emphasized that this metric has several significant limitations. First, there is no unified methodology for calculating COR, which leads to the possibility of manipulating the results. Arina also mentioned that she is aware of several "relatively honest" methods for artificially inflating COR to 100%. This emphasizes the importance of a careful and critical approach to data interpretation to avoid biased analysis and decision making. For some types of courses, achieving maximum COR is not always justified. This is especially true for long-term professional retraining courses, which students enroll in to master a new profession and find employment or start freelancing. Students often drop out mid-course, achieving their goals while still in training. In such cases, a low completion rate (COR) is not a relevant indicator of the quality of the educational product, as students remain satisfied—they acquire the desired knowledge and skills. Nikita Podlipsky, Managing Director of Ultimate Education, expressed this point of view at a seminar held by the HSE Institute of Education on the future of EdTech.

Arina Pantina noted the importance of measuring the changes students experience during their studies and after completing the course, especially in the context of their career development. However, such assessment can be challenging, as achieving significant results requires both time and effort from the students themselves. This underscores the importance of a personalized approach to learning and supporting students in their professional development aspirations.

Learn more about education by subscribing to our Telegram channel. We share relevant information, useful tips, and interesting materials. Don't miss the opportunity to expand your knowledge and stay up-to-date on educational news.

Optimized text for SEO:

How to Measure the Quality of Learning in Courses: Netology's Approach

Contents:

How Netology Researched Perceptions of Quality

How the new quality control system works at Netology

What's wrong with the COR metric?