How Evidence-Based Education Works in EdTech: The Yandex Practicum Experience

Yandex's annual online education conference, YaC/e, took place on November 15. During the event, representatives of Yandex Praktikum shared their experience in applying an evidence-based approach to education. This topic was discussed in detail in two presentations, which focused on methods for improving the effectiveness of the educational process and achieving results.

Maria Kovaleva, Head of the Educational Technologies Laboratory at Yandex Praktikum, presented the report "Methodological Expertise and Data Science in Search of Evidence in Education";
Dmitry Abbakumov, the company's Chief Psychometrician, presented the report "The Curious Case of Benjamin Bloom and Other Secrets of Evidence-Based Education."

In this material, we briefly present the key ideas and main points expressed during the presentations. We strive to highlight the most significant points so that readers can quickly familiarize themselves with the content and understand the essence of the topics discussed. This will allow them to better absorb the information and apply it in practice. We focus on the key findings and recommendations presented by the speakers.

What is evidence-based education and what "evidence" is used

The speakers presented the concept of evidence-based education, which is based on the use of scientific data to shape educational processes. An important aspect is the regular conduct of statistical studies that allow us to determine how different categories of students learn, which educational methods are the most effective, and how to improve the quality of education. The collection and analysis of digital data plays a key role in this process. According to Maria Kovaleva, this approach includes scaling up successful methods, as well as the use of artificial intelligence and machine learning to optimize educational practices.

In the process of creating an evidence-based education system, Maria uses the metaphor of a pyramid. At the base of this pyramid are data, which serve as the foundation. At the next level is the experience of applying this data in real-world conditions, and the top of the pyramid is statistical research that allows us to identify new insights. Eliminating even one of these levels can lead to the collapse of the entire structure. For example, if an educational product is developed without data and is based solely on subjective opinions about what effective learning should be (even if these opinions come from an experienced teacher or methodologist), this can lead to errors, as these opinions may be based on false stereotypes. Similarly, using data without statistical analysis does not provide a complete picture, since the same data can be interpreted in different ways.

Data: presentation by Maria Kovaleva Infographics: Maya Malgina for Skillbox Media

Dmitry Abbakumov emphasizes the importance of evidence-based education, which aims to develop pedagogical technologies. These technologies should be aimed at ensuring that educational material and methods of explanation contribute to the successful achievement of each individual student's educational goals.

An evidence-based approach significantly improves communication with students and builds their trust. A study conducted by an American educational platform confirms this: when suggesting that students practice before a test, if they are told that such preparation leads to better results in 60% of cases, this information proves more effective. This confirms that the use of data and facts in communication with students contributes to more productive learning and increases their motivation.

Dmitry identified three key aspects on which evidence-based education is based.

high-quality data, without which there can be no evidence;
valid mathematical models, that is, those that represent reality in a mathematical formula;
interpretability and transparency.

Maria Kovaleva noted important aspects that must be taken into account when conducting research and analyzing data in order to improve the educational product. These phenomena can significantly affect the quality of the results obtained and their interpretation. Paying attention to the context of the data, the choice of analysis methods, and the correct interpretation of the results will help create more effective educational solutions and increase their value for users.

The paradox of research interests. The point is that both desirable and undesirable values are important for research. For example, to analyze students' willingness to recommend a course, you need to study the attitudes of not only those who recommend it but also those who don't. Although, for business as a whole, it is, of course, important to have as few undesirable values as possible.
The paradox of applicability of findings. Every course development team wants research to be conducted specifically among their students, because students in different programs differ. And when data is obtained for a specific course, you want to compare it - what about others?
The paradox of individual actions. Educational product developers always want to know the effect of a single action on a business metric - but in most cases, a result is achieved through a combination of actions, not a single one.
The paradox of data fascination. When working with data, you must remember that there is something beyond it. That is, there is no need to cross out the methodological expertise.

How an Evidence-Based Approach Helps Students Complete Their Studies

Maria Kovaleva emphasizes how evidence-based education can improve the effectiveness of learning. For example, research shows that confidence in one's own knowledge helps students persist in courses and complete them successfully. Therefore, it is important to work on overcoming impostor syndrome in students, which often arises from a feeling of insufficient preparation and fear of the amount of information that needs to be mastered. This state "immobilizes" students and prevents them from reaching their full potential. Improving self-confidence can be a key factor in successfully completing the educational process.

Maria emphasized that students' confidence in their abilities can be increased by sharing real-life stories of graduates who also experienced doubts but ultimately achieved success. Mentors with expert knowledge also share their experiences with impostor syndrome. These revelations show students that such feelings are normal and can occur in anyone, which in turn helps them overcome their doubts and develop self-confidence.

It is important to improve specific areas of the curriculum that may be weakening students' self-confidence. This applies to the most complex topics, where students encounter difficulties and may consider dropping out. To avoid this, it is important to break down complex topics into simpler steps and gradually increase the difficulty of assignments. This approach will help students better absorb the material and increase their confidence in their abilities.

It is important to increase the practical value of the knowledge acquired, as this contributes to students' confidence in their skills. An example of this is a case described by Maria Kovaleva, where a significant decrease in student confidence was observed during a data analysis course. While analyzing the content, the instructors discovered that a complex term was explained using abstract concepts, which made it difficult for students to apply their knowledge to real-world projects. After adding real-world examples to the explanation, students' confidence increased significantly. This underscores the importance of using real-world examples to enhance understanding and retention of the learning material.

It's important to demonstrate to students how various terms and ideas are applied in real life. Practical tasks and projects, as well as collaboration with real clients, help reinforce understanding and the value of the knowledge they gain. However, it's important to keep in mind that exposing newcomers to complex, real-world projects too early can be demotivating. Students who encounter difficulties may realize not only their own accomplishments but also a significant amount of missing knowledge. Therefore, it's important to find a balance between practical experience and students' level of preparation to maintain their motivation and promote successful learning.

How to achieve balance in learning? Maria shared her experience in the data analysis department, where she observed a decline in students' confidence in their knowledge. This problem stems from the fact that data analysis is a complex field in which knowledge accumulates in ways that are difficult to grasp. Students struggle to organize and integrate information as they progress from one topic to the next. In response, the course designers implemented not only projects within each sprint but also a consolidated project that simulates real-world conditions and builds on the skills acquired in previous sprints. Results showed that students' confidence in their knowledge declines until the completion of the consolidated project, after which it increases significantly. This suggests that the consolidated project helps students overcome their confidence deficits and consolidate their acquired knowledge. The question of how Yandex Practicum measures students' confidence in their knowledge is becoming increasingly relevant. According to the speaker, this process is carried out using questionnaires. The questionnaires provide objective data on students' self-assessment and their perception of the material they have learned. Thus, Yandex Praktikum uses a systematic approach to assessing the level of confidence of students, which helps to optimize the educational process and improve the quality of training.

How an evidence-based approach makes education student-centered

Dmitry Abbakumov noted that Yandex Praktikum drew inspiration from the Mastery Learning approach. This method was developed by American psychologist Benjamin Bloom, who also became famous for his taxonomy of learning goals. This system classifies learning goals, arranging them in order of increasing difficulty—from simple to more complex. This approach allows for more effective learning and a deeper understanding of the subject.

Of course, I'm happy to help you with editing the text. Please provide the actual text you wish to change.

Mastery Learning, or mastery-based learning, is a methodology that emphasizes the complete mastery of the learning material before moving on to the next topic. This approach implies that each student has the opportunity to study the material at their own pace. If a student encounters difficulties and does not achieve the required results, they are provided with additional support and resources to deepen their understanding of the topic. This approach promotes deeper assimilation of knowledge and skill development, which in turn improves the overall level of education and promotes successful learning.

Dmitry Abbakumov explained that Bloom concluded that even the most qualified teachers and the highest-quality educational materials do not provide the same effectiveness for all students. Each person has their own unique learning style, which leads to different results in the educational process.

The psychologist proposed introducing formative diagnostics—a key stage at which the level of assimilation of the learning material provided for study and practice is assessed. The results of this assessment determine the student's further educational path. The process is as follows: students who successfully master the material move on to the next topic or receive additional assignments to deepen their knowledge. Those who fail to master the material are offered support, which Dmitry Abbakumov called "intervention" in his report. This support may include alternative educational resources that enable a better understanding of complex topics and improve overall academic performance. The implementation of formative assessments promotes a personalized approach to learning and helps each student achieve their educational goals.

The Mastery Learning strategy is based on two key components. The first component includes formative assessments, which identify differences in student knowledge and identify alternative learning approaches. This is achieved through the use of a variety of learning materials, such as retellings, additional assignments, and examples that correspond to the educational content. Additionally, additional educational interventions may utilize various forms of organizing the learning process, including group work, partner assignments, and individual projects. The variability of the content and format of the training helps ensure that every student achieves high results.

Dmitry emphasized that over its forty-year history, the Mastery Learning method has proven itself to be an effective teaching approach. Unlike the traditional method, where most students demonstrate average results, with only a few achieving high or low scores, Mastery Learning enables a significant proportion of students to achieve excellent results. This method promotes deeper assimilation of the material and an increase in the general level of knowledge.

Data: presentation by Dmitry Abbakumov Infographics: Maya Malgina for Skillbox Media

However, this approach has its limitations. It is important to consider that the effectiveness of the method may be reduced in certain situations. For example, if data is insufficiently prepared or a clear strategy is absent, the result may be far from expected. It is also worth noting that changes in the external environment can affect the relevance of the chosen approach, which requires constant adaptation and analysis. It is important to be prepared for possible difficulties and find alternative solutions to achieve the set goals.

Alternative interventions. For many years, it was believed that Bloom's approach is limited by the impossibility of organizing a wide variety of interventions for different students, since personalizing learning to such an extent is very expensive.
Cubicness. To obtain a high-quality, reliable, and valid result from formative assessments, a voluminous test is required, which significantly complicates the application of the approach.
Untimely. Mastery Learning assumes that a student must cover a significant portion of the educational material (for example, a topic) before undergoing assessment. It turns out that, faced with difficulties and disappointed with their educational experience, a person may drop out of education even before they receive support, and even before they figure out what kind of help would be useful.
Lack of focus. Testing based on the results of one large topic does not allow us to accurately identify the part that caused difficulties for the student.

Read also:

Managing cognitive load in learning is an important aspect of an effective educational process. Cognitive load refers to the amount of information our brains can process simultaneously. To optimize learning, it's important to consider the various aspects that influence cognitive load. The first step in managing cognitive load is simplifying the material. Structuring information and using visual examples help students better grasp complex concepts. It's important to break the material into manageable chunks to avoid overwhelming the learner.

The use of active learning methods also helps reduce cognitive load. Engaging students in discussions, practical tasks, and group projects allows them to actively process information and apply their acquired knowledge in practice.

Controlling the level of task difficulty is an equally important aspect. Tasks should be appropriate for the level of students' preparation to maintain their interest and motivation. This can be achieved by gradually increasing the difficulty based on student progress.

It is also worth paying attention to the individual characteristics of students. Each person has their own preferences and learning style, which can affect the perception of information. Taking these factors into account, you can adapt the teaching approach, which will allow you to more effectively manage cognitive load.

In conclusion, managing cognitive load in learning requires a careful approach and the use of various methods and strategies. Optimizing the educational process with cognitive load in mind promotes deeper understanding and assimilation of the material, which ultimately leads to an improvement in the quality of education.

Artificial intelligence and data analysis open up new possibilities for overcoming the limitations of the Mastery Learning model. Yandex Praktikum developed an innovative diagnostic technology called "Delta." Dmitry Abbakumov explained its operating principles in detail. This technology allows for more efficient tracking of student progress and tailors learning materials to their individual needs, significantly improving the quality of learning and promoting better learning.

The course includes topics comprised of lessons. Each lesson is divided into various elements, such as tasks, questions, quizzes, and learning materials. For each of these elements, from the smallest to the largest, a cognitive accessibility metric is calculated. This metric indicates how accessible a given element is for the average student. This ensures an optimal level of difficulty and supports effective learning.

The product team and methodologists develop an understanding of the typical student for a specific course and develop educational content based on this analysis. As real students complete a course, cognitive performance is assessed to determine how well the team's perceptions of student abilities match reality. This means it's important to understand how well students are mastering the course material. Deviations between expectations and actual results are designated as deltas. These deviations can be either positive or negative. Negative deviations indicate that the material was more difficult than expected, while positive deviations indicate that students found it easier than expected. If certain elements of the course produce significant deviations, this signals the need for content revision. This allows the course material to be fine-tuned to the actual needs of the typical student, improving the quality of the educational process and increasing its effectiveness.

Deviation analysis, conducted for each student, serves as the basis for further support. Furthermore, the analytical algorithm can predict course completion rates. According to Dmitry Abbakumov, the proportion of students who dropped out of their studies with a negative delta is 23%, compared to only 1% of those who completed their studies. This underscores the importance of an individualized approach to each student and the need for timely intervention to improve learning success.

The speaker emphasized that the problem of cumbersomeness in the Delta system is solved by analyzing data on how students cope with each element of the course. Assessment occurs directly during the learning process, allowing for more accurate tracking of progress and the effectiveness of the educational process. This approach facilitates adapting the course to student needs and improves the quality of learning.

The problem of lack of focus can be effectively addressed by collecting data on numerous small elements. This allows for the identification of specific moments when students are experiencing difficulties. This approach helps to better understand the learning process and target improvements. It's important to analyze data to determine which aspects require additional attention and support, which in turn facilitates more effective learning.

The problem of untimeliness has not yet been fully resolved, as student difficulties are often identified only after they arise. However, Dmitry Abbakumov announced a new version of the technology, called "Delta+." The basic idea of this technology is to use student delta data for each task to predict potential difficulties in subsequent topics. This will allow students to work proactively and prevent problems.

Yandex Praktikum claims that "Delta" is universal. The speaker emphasized that the technology was developed with the need for effective work in both small groups of students and large EdTech companies in mind. This makes Delta a suitable solution for various educational formats and scales.

Explore additional materials:

"We have little evidence-based approach in the daily work of teachers"
6 reasons why psychometrics is needed in EdTech
Checklist: 5 ways to increase the completion rate of an online course
11 metrics to evaluate the quality and effectiveness of an online course
Improvement loop: how to improve an already launched online course

The Profession of Methodologist from Scratch to PRO

You will improve your skills in developing curricula for online and offline courses. Master modern teaching practices, structure your experience, and become a more sought-after specialist.

Find out more

How Evidence-Based Education Works in EdTech: The Yandex Practicum Experience

Contents:

What is evidence-based education and what "evidence" is used

How an Evidence-Based Approach Helps Students Complete Their Studies

How an evidence-based approach makes education student-centered