Education

How to evaluate the effectiveness of an online course

How to evaluate the effectiveness of an online course

Course with employment: "Profession Methodologist from scratch to PRO»

Find out more

Product Director, teacher and mentor at EdMarket. I am a mentor at the Netology accelerator. I am the author of the course "eLearning: From Implementation Strategy to Content Development and Performance Evaluation" and the book "The Architecture of Corporate Online Learning." I run the Telegram channel "PRO Online Learning," where I share relevant knowledge and practical experience in online learning. My work is focused on developing effective educational solutions and improving the quality of corporate training. Online course developers strive to create useful and engaging learning experiences. However, how can you determine whether the desired result has been achieved? This isn't always obvious, and without specific metrics, it's impossible to make an informed decision about whether to leave a course unchanged, make improvements, or discontinue it entirely. The answer to this question often impacts not only the fate of a single course but also the entire product line, underscoring the importance of thoroughly evaluating the effectiveness of educational programs. Performance evaluation is a crucial aspect of any business. In this article, we will discuss what performance evaluation involves, what metrics can be used for analysis, and what mistakes to avoid in this process. Effective evaluation allows you to identify strengths and weaknesses, optimize processes, and improve overall productivity. It's important to consider metrics such as ROI, conversion, customer retention, and other key indicators to help you make informed decisions. By avoiding common mistakes like ignoring data or misinterpreting results, you can achieve more accurate evaluations and improve your business results.

What is included in training evaluation

Evaluating effectiveness requires several key steps. First, you need to establish clear objectives and evaluation criteria. Next, you need to collect and analyze data to identify strengths and weaknesses. It's also important to evaluate the impact of external factors on results. Benchmarking against similar projects or standards will help you better understand the results achieved. Finally, regular monitoring and strategy review will allow you to adapt to changes and improve overall effectiveness.

  • Evaluate student learning outcomes (pre- and post-course performance, midterm and final assignment results).
  • Calculate course-specific metrics (midterm and final CSI and COR, final NPS).
  • Collect feedback on learning outcomes from students, instructors, auditors, and fellow methodologists.

Based on the metrics analysis, a decision must be made about the course's future. This may include maintaining it in its current form, deleting it, changing the format, content, or frequency of assignments. If a decision is made to revise the course, a change plan should be developed based on the feedback collected from participants. This approach will improve the quality of training and increase student satisfaction, as well as adapt the course to the current needs of the audience.

By what metrics is an educational product evaluated?

Before discussing the quality metrics of an educational product, it is important to define the perspective from which we analyze the course. Assessing success can vary significantly for students, methodologists, and training organizers. Each of these participants in the process has their own criteria and expectations that influence the perception of the course and its effectiveness.

A course is valuable to a student if it helps achieve set goals related to career growth or a change in profession. Typically, such goals extend beyond the educational process and are aimed at improving a professional position or developing new skills that contribute to increased competitiveness in the labor market.

A course is considered successful if, after its completion, changes in the attitudes and behavior of participants are observed that can be quantified. For example, after training in Excel, employees began to spend 30% less time on operations with formulas. This demonstrates the effectiveness of the course and its impact on work productivity. Measurable results confirm that the training has indeed led to improved skills and more effective use of the software.

From the perspective of the training provider, the effectiveness of an educational product is determined by the ratio of invested resources to the benefits received. To achieve optimal results, it is important that the costs of creating and supporting the educational project are justified by its results. Therefore, a successful educational product must ensure a balance between investment and results, which maximizes its value for all participants.

Corporate training is an important tool for improving employee performance. Training methodologists and organizers strive to achieve common goals with the client, for example, the company's management. The main objective is to ensure that employees begin to demonstrate higher performance in their work after completing the course or training. At the same time, it is important that the training costs are economically justified. Effective corporate training contributes not only to skill development but also to an increase in the overall productivity of the company.

Having determined the perspective from which we will evaluate the success of the educational product, we will consider what indicators can be measured. It's important to consider both quantitative and qualitative metrics. Quantitative data can include the number of participants, course completion rates, and satisfaction levels. Qualitative metrics include user feedback and changes in knowledge and skills after completing the program. These measurements will help more accurately assess the effectiveness of the educational product and identify areas for improvement.

  • Achieving intermediate and final learning goals: students completed homework, passed a test, and prepared a thesis.
  • The rate of attendance at classes or watching webinars in an online course.
  • The rate and speed of homework completion—the intensity with which your students learn.
  • The speed and quality of homework checking. The quality of feedback.
  • The quality of the educational content—both primary and secondary. To assess it, you can focus on student likes/dislikes (if your LMS supports this feature), ask them to complete feedback forms, or establish your own quality standards.
  • Achieving the final goal of the student or customer, which is beyond the scope of training. For example, people purchase a course to obtain a relevant profession and subsequently find employment. This means that here we need to measure the percentage of students who are employed after completing their studies.

Photo: Monkey Business Images / Shutterstock
  • Student satisfaction (CSI) with classes, content, support, teachers. You can measure satisfaction with the course as a whole, but it's more useful to request ratings for individual components: areas of study, instructors, theoretical material and homework, and learning support. CSI is usually rated on a scale of 1 to 6, sometimes 1 to 5 or 1 to 10. Student engagement and activity (chat interactions and feedback are assessed). Net Promoter Score (NPS). After completing the course, ask students how likely they are to recommend the course—on a scale of 1 to 10 (sometimes a scale of 0 to 10 is used). It's also useful to ask why they rated it that way. Completion Rate (COR). When measuring, it's important to rely on your company's accepted standards. For example, at EdMarket, a student is considered to have completed the course if they have completed 70% of the homework assignments and the thesis project.
  • The return rate, that is, cases where the course was not suitable for students, is high. If it is high, this is a reason to re-analyze the information about your target audience and consider why the person was enrolled in an unsuitable course, what other factors could have contributed to their abandonment, and what can be done to improve the situation.
  • The "Insightometer" is the number of insights that prompted students to take action. We haven't yet fully figured out how to measure this metric. However, in each thesis, we invite students to share their conclusions, impressions of the course, and plans for the future. This helps graduates plan their future educational trajectory, and helps us understand the knowledge, skills, and aspirations of students upon completion of a particular course.

Corporate training effectiveness metrics are an important tool for evaluating results and improving educational programs. One of the most well-known approaches to evaluation is the Kirkpatrick-Phillips model. This model helps systematically analyze various levels of training effectiveness, from participant satisfaction to the actual impact on business metrics. Using this model, companies can not only identify the strengths and weaknesses of their educational initiatives but also optimize them for better results. Effective analysis of training metrics helps improve the quality of corporate education, which in turn affects the growth of business productivity and competitiveness.

  • Reaction. How do participants react to a training event or other educational product? Here, the CSI and NPS metrics are again assessed.
  • Knowledge and skills. Did participants acquire the desired knowledge and skills? Here, the COR metric is used, as well as student grades for homework and tests.
  • Behavior change. Do employees apply the acquired knowledge and skills in the workplace? This is assessed by personnel KPIs, internal ratings, and so on.
  • Impact on business results. Did the training contribute to the achievement of company goals? KPIs for a department or business process are analyzed.
  • Return on investment in training is ROI, an abbreviation for Return on Investment. It measures the return on investment in an educational event or product.

Modern LMS platforms offer a wide range of tools for collecting data and analyzing metrics. Services such as Google Forms and Mentimeter can be effectively used to evaluate metrics such as CSI, NPS, and others that require feedback. These survey tools allow you to obtain valuable data on user impressions of the course and track their reactions to various training elements. The use of such solutions helps improve the quality of the educational process and increase user satisfaction.

What mistakes are often made when evaluating training?

There are two main omissions that many people often encounter:

  • Feedback is requested only at the end of the training. If you don't request feedback at the beginning and middle of the course, you won't know the reaction of those who dropped out for whatever reason. And this is very useful information: if you understand what exactly in the course students didn't like, you can fix it.
  • Homework checking is assigned to the course author. Due to the expert's busy schedule, students sometimes have to wait a long time for feedback on homework. In this case, it is better to delegate the checking to a curator or tutor - at least partially. It is useful to organize a chat where students can promptly receive answers to their questions. This will positively impact student satisfaction.

What is a methodological audit, why and how is it conducted?

There are many courses created without taking into account the specifics of online learning and knowledge of instructional design. Their authors strive to improve efficiency, but do not always know how to do this. In such cases, many turn to methodological audit services. This task combines methodology, marketing, and business, taking into account the interests of both students and training providers. A methodological audit allows us to identify weaknesses in a course and propose solutions to address them, thereby improving the quality of the educational process and student satisfaction.

Photo: Stock-Asso / Shutterstock

For a high-quality course audit, the methodologist must have professional experience and sufficient observation, which is best developed under the guidance of a mentor. It's important to clarify the initial requirements in advance to prevent the methodologist from focusing solely on learning theories and online course design, while the client may have completely different priorities. Client goals may include improving the quality of instruction, increasing student engagement, optimizing course content, or adapting the program to the specific needs of the target audience. Understanding these goals will allow the methodologist to more effectively evaluate the course and offer recommendations that meet the client's expectations.

  • increase COR, CSI, NPS;
  • increase LTV (Lifetime Value) - the total profit received from the client during their interaction with the product;
  • expand the product line;
  • attract more students;
  • get more positive reviews;
  • reduce the number of negative reviews;
  • understand why people do not complete the course;
  • find out why there are no visible results in the work of employees after training on the course.

If you plan to independently audit your course or have received a request to analyze someone else's course, we have prepared a checklist with key questions and the necessary data for evaluation. This list is structured into seven main blocks that will help you conduct a comprehensive course audit. Using this checklist, you can systematically evaluate the content, structure, and effectiveness of a course to identify its strengths and weaknesses.

Understanding the target audience for a course is key for training providers. This allows them to create programs that best meet the needs and expectations of participants. Target audience analysis includes studying their interests, level of preparation, and learning goals. By considering these factors, providers can develop more relevant and effective learning materials and teaching methods. This focus on the target audience increases participant motivation and improves learning outcomes, ultimately leading to successful course completion.

Review course participant questionnaires, Custody Development results, and feedback from graduates. This will give you an idea of ​​how effectively the course achieves its objectives for a specific target audience with specific knowledge, skills, and learning capabilities. Analyzing this data will help you better understand how the course meets student needs and expectations.

The course's methodological concept and learning logic play a key role in the educational process. They determine the structure and sequence of material presentation, as well as the approaches to its assimilation. The main goal of the methodological concept is to create an effective educational environment that promotes the development of critical thinking and practical skills in students. The logic of learning covers the choice of methods and forms of work that help achieve the set goals. It is important to take into account the individual characteristics of students and adapt the course to their needs, thereby ensuring a deep understanding and consolidation of knowledge. An effective methodological concept allows for the integration of modern technologies into the educational process, which makes learning more interactive and engaging.

  • Are the final and intermediate learning objectives described?
  • How is the logic of learning structured and is it clear?
  • What topics and subtopics is the curriculum divided into and why this way?
  • What is the meaning of the headings and are they well formulated?
  • What is the total duration of the training and why is it this way?
  • Is the study load evenly distributed, are there any breaks?

The educational environment of the course is the structure and resources that promote student learning and development. It includes both physical and virtual elements that facilitate interaction between course participants. It is important to consider that the educational environment should be adapted to different learning styles to meet the needs of all students.

Key aspects of the educational environment include the accessibility of materials, the quality of interaction with instructors, and the opportunity for students to exchange experiences and opinions. The use of modern technologies, such as online platforms and interactive tools, plays a significant role in creating an effective educational environment.

Thus, a properly organized educational environment of the course not only promotes the acquisition of knowledge, but also forms an active and involved community of students.

  • Does the student have the opportunity to interact with other students and teachers?
  • Is there an introductory lesson?
  • Are students given recommendations on how to study effectively and work with intrinsic motivation?
  • Is it clear to whom and when they can ask questions?
  • Is the interface of the selected platform user-friendly, is the course structured clearly with all its elements and means of communication, does everything work (e.g., videos, links, interactive elements)?
  • Is it clear how to receive educational content, complete and submit practical assignments, where does the feedback from the assessor go?
  • Does the student have the opportunity to leave feedback and how do the training organizers work with it?
  • Is it possible to simplify the educational environment?
  • Is the training accessible on mobile devices, and is it convenient to complete the course in this format?

The quality of educational content is a key factor influencing the effectiveness of learning. High-quality content must be relevant, accurate, and present information in an accessible form. It must meet modern educational requirements and standards, and also take into account the needs of the target audience. It is important that the materials are well-structured and include a variety of formats, such as text, video, and interactive elements, which facilitates better knowledge acquisition. In addition, regularly updating the content helps keep it relevant and in line with the latest advances in science and technology. Investing time and resources in creating high-quality educational content not only increases the knowledge of users but also promotes their motivation to learn.

  • What format of educational materials was chosen and why?
  • How convenient is it to perceive the information? Here the font, contrast, and so on are assessed.
  • Is a common style or template used to present the content?
  • What is the volume of the main theoretical part and additional materials, and why this particular volume?
  • Do the lessons have a unified structure and, if so, what does it include?
  • Are real cases provided?
  • Is additional literature recommended?
  • What do we know about students' reactions to the course content?
  • What do we know about students' reactions to the instructors and the presentation of the material?
  • How well does the instructor convey the course content in the chosen format (speech, communication style, quality of presentation material, sound and video are assessed) and how does he or she interact with students?
  • Is it possible to reduce the volume of theory and independent study without sacrificing practical experience?
  • Is it clear to students what material is mandatory to study and what is optional?
  • Are there any errors and typos in the texts Course?

The quality of practical classes plays a vital role in the educational process. Effective practical classes promote deepening knowledge and developing the skills necessary for successful professional work. They provide opportunities to apply theoretical knowledge in practice, which significantly improves the level of material acquisition. High-quality practical classes build students' confidence and promote the development of critical thinking. It is important that classes are conducted in a supportive and collaborative atmosphere, allowing each participant to feel valued and involved. The integration of modern technologies and teaching methods also enhances the effectiveness of practical classes, making them more interactive and engaging. Thus, an emphasis on high-quality practical classes is a prerequisite for training qualified specialists.

  • What methods of assessing acquired knowledge have been chosen (midterm and final)?
  • What is the intensity of practical assignments and do students cope with it?
  • Is there sufficient theoretical content to complete the assignments?
  • How are homework assignments and instructions for completing practical classes structured?
  • What is the feedback format?
  • What do we know about students' reactions to the practical classes?
  • To what extent do the practical assignments correspond to the real-life tasks that the student will subsequently face?
  • Is the assessment system for completed assignments clear to the student?

Data collected during the training of previous cohorts is a valuable resource for analyzing and improving educational methods. They allow us to identify successful approaches and optimize training programs, taking into account the experience of previous participants. These data help adapt course content, improve interaction with students, and increase the overall effectiveness of the educational process. Using information from previous streams helps create a more holistic and adaptive learning experience, which in turn leads to better outcomes for new students.

  • completion of midterm and final homework assignments designed based on learning objectives (it’s good if the LMS online platform’s data analytics allows for detailed information to be collected);
  • speed of responses to student questions;
  • speed of homework checking;
  • attendance at webinars or viewing other educational content;
  • CSI — satisfaction with the process, content, and learning outcomes (assessed through feedback through questionnaires and interviews);
  • NPS — student loyalty, willingness to recommend the course to others;
  • COR — course completion rate;
  • return rate in cases where students were not satisfied with the course itself, the learning format, or something else;
  • percentage of students placed in internships or graduates employed;
  • percentage of positive Reviews.

Learning support is an important aspect of the educational process. It includes supporting students at all stages of their learning, which helps improve the effectiveness of material absorption and facilitates successful course completion. Support may include individual consultations, organizing additional classes, and providing necessary resources. High-quality learning support provides not only academic assistance but also the development of critical thinking, self-regulation skills, and responsibility for one's own education. The integration of modern technologies and support methods makes the learning process more accessible and effective.

The distribution of roles in the course support process plays a key role in its successful implementation. It is important to determine who is responsible for organizing the educational process, supporting students, and administering the platform. Effective distribution of responsibilities between teachers, coordinators, and technical specialists contributes to higher quality learning and increased student satisfaction.

In addition, it is necessary to assess the availability of resources for scaling the course. This includes the ability to launch new streams and conduct simultaneous training for several groups of students. Key aspects requiring attention are the availability of materials, instructor qualifications, and technical support. Ensuring sufficient resources will not only increase the number of students but also maintain a high level of educational process.

A methodological audit plays an important role in identifying shortcomings in the educational product. It can uncover both minor errors that require minimal effort to correct, as well as serious problems that require significant time and resources to resolve. After the methodologist's recommendations are presented, the client decides on the further development of the course. If the client chooses to refine the educational product, they can make the changes themselves or again contact the methodologist for assistance. It should be emphasized that course improvement is an independent task and should not be confused with the audit process. An effective methodological audit contributes to improving the quality of education and ensures the successful development of educational programs.

Learn more about education on our Telegram channel. Subscribe to stay up-to-date with the latest news and helpful tips!

Be sure to check out these additional resources:

  • How to choose a topic for an online course that will "work well"
  • How to motivate adults to learn
  • 6 ways to make a course engaging
  • 5 ways to increase the success rate of an online course

Learn more about education in our Telegram channel. Subscribe to stay up to date with interesting news and useful content!