Battle: Who will develop a better course—a pedagogical designer with or without a neural network?

Contents:

• What the experiment consisted of
• How the development of learning objectives was evaluated
• How the teaching strategy was evaluated
• How the course plan was evaluated
• The author's conclusions

What the experiment consisted of

Three people took part in the experiment, conducted by Philippa Hardman.

• two experienced instructional designers, one of whom performed all the tasks during the experiment independently, and the other used tools based on generative neural networks (ChatGPT 4o and Consensus GPT);
• a person with no experience in instructional design - he also resorted to the help of ChatGPT 4o and Consensus GPT.

It should be noted that the author of the experiment does not provide information about what kind of experience the participants had, how long they have been working in the profession, and whether it can be considered that the level of professionalism of the two experienced instructional designers is comparable. These aspects are important for understanding the context and validity of the experiment's results.

Philip's participants were given three identical tasks.

• write the educational objectives of the online course;
• choose a teaching strategy for the course;
• think through the overall course plan (that is, describe the modules it will contain).

The course is designed for entrepreneurs and marketers, providing them with the practical skills and strategies needed to successfully sell and develop innovative products and services in a rapidly changing environment. Participants will receive tools and methods to help them adapt to market changes and effectively engage with consumers.

Experiment participants demonstrated their skills in completing the tasks, and the results were evaluated by Philippa Hardman's subscribers on the Substack and LinkedIn platforms. Philippa presented the results anonymously, which allowed subscribers to guess which participants were experienced instructional designers or novices, who used neural networks or acted independently. According to the author of the experiment, approximately 200 instructional designers took part in the survey, but the exact data was not provided. This study has drawn attention to various approaches in educational design and the possibility of applying modern technologies in this area.

As a result of the anonymous evaluation of all three tasks, the participants took equal positions.

• According to subscribers, the experienced educational designer who used neural networks performed the best;
• The novice, who completed the tasks using neural networks, came in second;
• And the experienced educational designer, who worked independently, without the help of neural networks, came in third, that is, his result was rated worse than the results of the others (even the novice).

This experiment revealed interesting stereotypes that have formed among educational program developers regarding neural networks. In particular, educational designers often underestimate the potential of neural networks, showing distrust of their capabilities, which is similar to the attitude of students. At the same time, they tend to overestimate human abilities. Let's take a closer look at this phenomenon and its implications for the educational process and the development of technologies.

How was the development of learning objectives assessed?

An experienced instructional designer and a novice who used neural networks successfully formulated course objectives. 70% of respondents, subscribers of Philippa Hardman, rated the experienced instructional designer's results as good or very good, and 30% rated them as excellent. The course objectives were truly practical, applicable to real life, and described in sufficient detail, which indicates a high degree of elaboration and relevance for the participants.

Novices who formulated their goals using neural networks received positive feedback from respondents: 60% of participants noted that the objectives were presented well or very well. Many of them confidently assumed that only an experienced specialist in the field of educational program development could have completed such a work. The goals presented by the newcomers were distinguished by their practicality and meaningful detail.

The goals presented by the experienced instructional designer, who did not use neural networks, were rated by most subscribers as insufficiently clear. Firstly, there were only three, while the experienced colleague, using neural networks, formulated 12 goals, and the novice – 18. Secondly, the goals did not contain measurable or practical outcomes. They assumed that students would "learn" and "understand" the material, as well as "gain insights" from the course. Interestingly, Philippa's subscribers assumed these goals were formulated by artificial intelligence, not a professional. This underscores the importance of clearly and specifically formulating educational goals to improve their perception and effectiveness.

According to the study, 64% of respondents assumed the goals were formulated by a novice using artificial intelligence. This suggests that many instructional designers continue to believe that high-quality educational content is impossible without the involvement of an experienced professional. Philippa Hardman emphasizes the importance of the human factor in the development of educational materials.

Based on the results obtained, the author of the experiment draws the following conclusions.

• People find it difficult to distinguish between human and artificial intelligence, and their perceptions of the capabilities and limitations of the latter are often erroneous.
• Respondents had quite significant differences in their opinions on how to define and formulate learning objectives in general.

Another interesting finding concerns the differences in approaches to writing prompts for a neural network between a novice and an experienced instructional designer. A novice, as a rule, interacts with the neural network as with a human, while an experienced specialist formulates clear instructions based on their experience in prompting for professional tasks. This leads to the fact that the latter interaction often produces higher-quality results. This fact confirms that using neural networks is a specific skill, and results can vary depending on the user's level of training. If you've used a neural network once and weren't satisfied with the results, that doesn't mean the technology isn't effective. Perhaps you should work on your prompt writing skills. Improvements in this area can significantly improve the quality of output data and unlock the full potential of the neural network for your work.

Read also:

Examples of queries in ChatGPT for teachers can significantly simplify the process of teaching and communicating with students. Using artificial intelligence in education opens up new possibilities for educators. For example, teachers can ask questions about teaching methods, get tips on creating learning materials, or search for ideas for interactive activities.

Educators can use ChatGPT to prepare lessons, discuss topics students are struggling with, or develop individualized learning plans. Requests can range from book and resource recommendations to creating tests and assignments.

ChatGPT is also useful for receiving feedback on existing content or brainstorming project ideas. With ChatGPT, teachers can effectively tailor their teaching methods to student needs and increase engagement, which in turn leads to improved educational outcomes.

Thus, ChatGPT becomes a valuable tool for educators, allowing them to optimize their work and make learning more engaging and productive.

How was the teaching strategy assessed?

As part of the teaching strategy for the course, an experienced instructional designer proposed standard lectures and self-assessment tests. However, respondents did not appreciate this approach: many considered it outdated and did not contribute to student engagement in the learning process. As a result, 63% of respondents concluded that the proposed option is more suitable for a novice than an experienced instructional designer. This underscores the importance of modern teaching methods that activate student motivation and promote better assimilation of the material.

When developing a teaching strategy, we often overestimate the skills of experienced curriculum developers and underestimate the capabilities of artificial intelligence. AI has the potential to significantly improve the quality of work compared to traditional approaches focused solely on human resources. This reflects a bias similar to what has been observed previously: when faced with poor performance, we tend to blame AI or a lack of experience, while acknowledging that even experienced professionals can demonstrate poor results. Philippa Hardman emphasizes the importance of this aspect, calling for a more objective view of the interaction of humans and technology in the learning process. Novychok proposed a modern approach based on a problem-oriented method using case studies. This solution received a positive response from respondents. Interestingly, only 27% of respondents were able to guess the author of this approach, while 51% believed that it could have been proposed by an experienced instructional designer who does not use neural networks. The respondents' main arguments were related to the perception of experience: they believed that artificial intelligence lacks the life experience that only humans have. Moreover, the choice of strategy was perceived as justified, which, according to respondents, also indicated human involvement in the process, rather than automation.

An experienced instructional designer incorporated flipped classroom methods, case studies, and expert workshops into her teaching strategy, leveraging the power of neural networks. This approach proved successful: 62% of respondents rated it as good or very good, and 33% rated it excellent. Respondents correctly assumed that this method was developed by an experienced specialist who uses neural networks, as only a professional can effectively combine different methods and adapt them to each other. References to scientific research and a clear response structure also indicated a level of expertise.

Philippa Hardman drew the following conclusions from the second experiment.

• Neural networks can indeed be a valuable aid for instructional designers. Moreover, artificial intelligence partially equalizes the experience of a novice and a professional—because in the experiment, the instructional designer and a person unrelated to course development generally did an excellent job.
• At the same time, instructional designers themselves still don't believe in the capabilities of AI and think less of them than they actually are: after all, in this experiment, respondents attributed poor performance to neural networks, when in fact this was not the case.

Read also:

Learning Activities: 10 Common Formats

Learning activities play a key role in the educational process, offering a variety of formats for the transfer of knowledge and skills. There are many approaches to organizing learning activities, and each of them has its own characteristics and advantages. It is important to choose the appropriate format based on the objectives, audience, and content.

Classical lectures remain one of the most popular formats, allowing instructors to share knowledge with a large audience. Seminars offer the opportunity for more in-depth discussions of topics in small groups, which promotes active participant engagement. Masterclasses focus on hands-on learning, where participants can gain skills under the guidance of an experienced mentor.

Webinars and online courses are becoming increasingly relevant as they provide access to knowledge from anywhere in the world. This is especially important in a distance learning environment. Interactive trainings offer participants the opportunity to actively engage in the process, which increases the effectiveness of material absorption.

Symposia and conferences serve as a platform for exchanging experiences and discussing current issues in a particular field. They bring together experts and practitioners, which contributes to the development of a professional community. Interactive games and group exercises not only help teach but also develop team skills and creative thinking.

Round tables provide an opportunity to exchange opinions and jointly search for solutions, which is especially valuable in the decision-making process. Finally, internships and practical classes allow participants to gain real-world experience in their field, making learning more practical and tangible.

Each of these formats has its strengths and can be used effectively depending on the goals and objectives of the educational event. Choosing the right format maximizes learning outcomes and makes the process more engaging and productive.

How the course plan was rated

An experienced instructional designer who developed a course plan without using neural networks coped with this task better than with previous ones. Most respondents rated his plan as good. It included four modules, described concisely and without clearly formulated measurable goals. Half of the respondents were able to identify the author as someone who did not use neural networks. This may indicate that works created using artificial intelligence tend to be less detailed and voluminous.

The beginner's course using neural networks proved to be significantly more comprehensive and informative. Survey participants appreciated the high level of detail and the meticulously crafted outline, which created the impression of expert input in the course development. Ultimately, 52% of respondents were able to identify the author, which, according to Philippa Hardman, indicates that many still believe in the ability of artificial intelligence not only to create high-quality curricula but also to act as a virtual expert in a given field. This fact underscores the growing importance of AI in education and its potential role in developing courses for beginners.

The plan from an experienced instructional designer who used neural networks again ranked first. This development significantly surpasses the previous two, as it includes 12 modules containing both theoretical and practical materials. Each module is associated with clear objectives for students to achieve. Methodologists interviewed about this plan highly praised its quality, and most were able to correctly identify the author. They noted that the plan was likely created by a human, as it demonstrates a deep understanding of the subject matter and sophisticated content, which is consistent with clearly defined learning objectives. At the same time, the design of the work indicates the use of neural network technologies.

The third experiment demonstrated that methodologists recognize the usefulness of artificial intelligence.

• It can act as a subject matter expert;
• It serves as a good tool for working with text (which allows for the high-quality "packaging" of ideas and developments of an experienced instructional designer);
• It allows for the faster completion of certain parts of the work.

During the development of the finished product, the instructional designers paid special attention to how the human factor is reflected in the plan. They sought to see how clearly the human "footprint" is visible in the document compared to a machine-based approach. This was an important aspect, as human perception and creativity play a key role in instructional design.

Conclusions from the experiment's author

The future of instructional design will likely be determined by a symbiotic relationship between humans and artificial intelligence, emphasizes Philippa Hardman. In each specific case, the interaction will depend on the individual characteristics and needs of the user. This underscores the importance of adapting educational materials and methods to diverse user profiles, which in turn will maximize the potential of AI in the educational process.

Generative neural networks are becoming a useful tool for experienced instructional designers, allowing them to effectively implement ideas, provided that a professional clearly specifies what and how to do it. To achieve the best results, it is important to have a deep knowledge of instructional design and an understanding of how the neural network functions. The ability to formulate correct prompts and conduct a productive dialogue with such an "assistant" significantly affects the final result of the work.

A neural network can be a useful assistant for those new to instructional design, facilitating often difficult tasks. For example, it can help formulate learning objectives. Philippa Hardman notes that while a neural network cannot replace a true instructional design expert, it can still provide some support. Experiments show that a neural network can be an effective tool for facilitating faster and higher-quality completion of tasks in this field.

Artificial intelligence is a significant force in the professionalization of instructional design. By automating key aspects of our work, such as content creation, AI emphasizes the importance of deep domain knowledge. This allows us to approach the level of mastery we have strived for for many years, focusing our efforts on achieving high quality and, ultimately, making a significant impact on the educational process. Philippa Hardman emphasizes that this transformation opens new horizons for instructional design professionals.

It's interesting to interpret the situation when an experienced instructional designer, working on an experiment without the use of neural networks, found himself among the outsiders, losing out even to a novice. Philippa Hardman doesn't comment on this fact, but several conclusions can be drawn. First, experience doesn't always guarantee professionalism. Second, experienced professionals can have ingrained stereotypes and patterns that hinder them from finding optimal solutions. This experiment, which involved developing a teaching strategy, clearly confirmed this. Third, this case highlights the need to be wary of overestimating human professionalism, just as it underestimates the capabilities of neural networks. Neural networks can offer innovative approaches that sometimes outperform traditional methods used even by experienced teachers.