Battle: Who will develop a better course—a pedagogical designer with or without a neural network?

Contents:

• What the experiment consisted of
• How the development of learning objectives was evaluated
• How the teaching strategy was evaluated
• How the course plan was evaluated
• The author's conclusions

What the experiment consisted of

Three participants took part in the experiment, conducted by Philippa Hardman. The study focused on analyzing their behavior and interactions in various settings. The goal of the experiment was to identify factors influencing decision-making and social interactions. Participants completed a series of tasks that allowed them to assess their reactions and strategies in various situations. The results of this experiment can provide valuable information about the psychology of group dynamics and individual behavior.

• Two experienced instructional designers, one of whom performed all tasks during the experiment independently, while the other used tools based on generative neural networks (ChatGPT 4o and Consensus GPT);
• A person with no experience in instructional design - he also resorted to the help of ChatGPT 4o and Consensus GPT.

It is important to note that the author of the experiment does not specify what kind of experience the participants had, how long they have been working in the profession, and whether the professionalism of the two experienced instructional designers can be considered equal. These aspects can significantly influence the results of the study and provide a more complete picture of its validity.

All of Philippa Hardman's participants were given three identical tasks.

• write the educational objectives of the online course;
• choose a teaching strategy for the course;
• develop a general plan for the course (that is, describe what modules it will contain).

The course was designed to provide entrepreneurs and marketers with the practical skills and strategies needed to effectively sell and develop innovative products and services in a rapidly changing market environment. Course participants will receive tools to adapt to new trends and improve the competitiveness of their offerings.

Participants in the experiment demonstrated their problem-solving skills, and the results were evaluated by Philippa Hardman's subscribers on the Substack and LinkedIn platforms. Philippa presented the results anonymously, which allowed subscribers to guess which of the participants completed the tasks: an experienced instructional designer or a novice, who used neural networks or acted independently. According to the author of the experiment, about 200 instructional designers took part in the survey, but no exact data was provided.

As a result of the anonymous evaluation of all three tasks, the participants took the same places.

• According to subscribers, the experienced educational designer who used neural networks performed the best;
• The novice, who completed the tasks using neural networks, came in second;
• And the experienced educational designer, who worked independently, without the help of neural networks, came in third, that is, his result was rated worse than the results of the others (even the novice).

The experiment revealed interesting stereotypes that have developed among developers of educational programs using neural networks. Instructional designers often underestimate the potential of neural networks, showing distrust of them, which is reminiscent of students' attitudes towards new technologies. At the same time, they tend to overestimate human capabilities. Let's take a closer look at this phenomenon.

How was the development of learning objectives assessed?

An experienced instructional designer and a novice using neural networks successfully formulated course objectives. 70% of respondents who followed Philippa Hardman rated the results of the instructional designer's work as good or very good, and 30% rated them as excellent. The course objectives were practical and applicable to real life, and also described in detail, which contributes to better understanding and assimilation of the materials.

The goals formulated by the novice using neural networks received positive ratings from respondents: 60% of participants noted that they were written well or very well. The persuasiveness and quality of these goals created the impression that they were developed by an experienced educational specialist. Moreover, the goals were practical and sufficiently detailed, which confirms the effectiveness of using neural network technologies in the goal-setting process.

The goals presented by the experienced instructional designer, who did not use neural networks, were rated by most subscribers as insufficiently clear. Firstly, their number was small – only three goals, while their experienced colleague, using neural networks, formulated twelve, and the novice – eighteen. Secondly, these goals did not contain measurable or practical outcomes. For example, they implied that students would "learn" and "understand" the material, as well as "gain insights" from the course. Interestingly, Philippa's subscribers suggested that these goals were created by artificial intelligence, not a professional. This underscores the importance of clearly articulating goals in the educational process, as they must be understandable and achievable for students.

According to the survey, 64% of participants expressed the opinion that the presented goals were formulated by a novice using artificial intelligence. This indicates that most instructional designers remain convinced that high-quality educational content depends on the involvement of an experienced specialist. Philippa Hardman emphasizes the importance of the human factor in the process of developing educational materials.

The author of the experiment draws the following conclusions based on the results obtained:

• People find it difficult to draw the line between the work of humans and artificial intelligence, and their ideas about the capabilities and limitations of the latter are often erroneous.
• Respondents had quite significant differences in their opinions on how to generally define and formulate learning objectives.

When analyzing work with neural networks, an important aspect related to the approach to formulating prompts can be highlighted. Novices often communicate with the neural network as if it were a person, while experienced designers use more structured instructions, which leads to better results. This confirms that working with neural networks requires certain skills, and different users can achieve different results, even using the same model. If you've tried integrating a neural network into your work and aren't satisfied with the results, that doesn't mean the technology has limitations. Rather, you should spend some time practicing creating effective prompts. Experience and training in this area can significantly improve interaction with the neural network and unlock its potential for your needs.

Reworked text:

Study additional materials:

Examples of requests in ChatGPT for Educators

Educators can use ChatGPT to address a variety of challenges related to student learning and development. Here are some example queries that will help them leverage the power of artificial intelligence in the educational process.

Queries can include creating teaching materials, preparing lessons, generating project ideas, and receiving recommendations on teaching methods. Educators can ask questions about best teaching practices, get advice on working with challenging students, and find inspiration for new approaches to teaching.

They can also use ChatGPT to create tests and assignments adapted to different difficulty levels or to develop interactive activities that will help engage students in the learning process. Artificial intelligence can suggest ideas for educational games and activities that will make learning more engaging.

In addition, educators can request information about modern educational technologies and methods that can be integrated into the classroom. Inquiries about the psychology of learning and motivation will help better understand student needs and create a more comfortable educational environment.

Using ChatGPT in teaching practice not only saves time but also contributes to improving the quality of education, allowing teachers to focus on an individual approach to each student.

How was the teaching strategy assessed?

As part of the teaching strategy for the course, an experienced instructional designer, working independently, proposed using standard lectures and tests for self-assessment of knowledge. However, respondents did not appreciate this approach: the majority of survey participants noted that it is outdated and does not contribute to student engagement in the learning process. As a result, 63% of respondents came to the conclusion that this method is more typical of a novice than an experienced instructional designer.

When developing a teaching strategy, it is important to keep in mind that we often overestimate the skills of experienced curriculum developers and underestimate the capabilities of artificial intelligence. AI has the potential to improve the quality of work, outperforming traditional human-centered approaches. This situation reflects a bias we've observed before: when faced with poor performance, we tend to blame the AI ​​or a lack of experience, forgetting that even experienced professionals can sometimes demonstrate poor results. Philippa Hardman emphasizes the importance of rethinking our perception of the role of AI in educational processes.

A newcomer proposed a modern approach using neural networks—a problem-oriented method based on case studies. This solution evoked a positive response from respondents. Interestingly, only 27% of survey participants guessed the author of this solution, while 51% assumed it was developed by an experienced instructional designer who did not use neural networks. Respondents' main arguments were that they associated the choice of strategy with human experience, believing that artificial intelligence lacks it. Furthermore, they noted that the validity of the choice indicates that the decision was made by a person, not a machine. This underscores the importance of the human factor in the process of creating educational solutions.

An experienced instructional designer integrated modern methods into his teaching strategy, such as the flipped classroom, case studies, and workshops with experts, leveraging the capabilities of neural networks. Survey results showed that 62% of participants rated this approach as good or very good, and 33% as excellent. Respondents correctly assumed that the development was the work of an experienced specialist who had worked with neural networks, as only a professional can effectively combine various methods and adapt them to educational needs. The research references and the logical structure of the response also confirmed the author's high level of expertise.

Philippa Hardman drew the following conclusions from the second experiment. The study's results confirm the importance of factors influencing participants' behavior. The experiment showed that certain conditions can significantly alter the results, highlighting the need for careful data analysis. Hardman also noted that the interaction between variables plays a key role in shaping the final conclusions. These results open up new avenues for further research in this area.

• Neural networks can indeed be a valuable aid to educational developers. Moreover, artificial intelligence partially equalizes the experience of a novice and a professional—because in the experiment, the instructional designer and a person unrelated to course development generally did an excellent job.
• At the same time, instructional designers themselves still don't believe in the capabilities of AI and think less of them than they actually are: after all, in this experiment, respondents attributed poor performance to neural networks, when in fact this was not the case.

Read also:

Learning Events: 10 Popular Formats

Learning events play an important role in the educational process, offering a variety of formats for effective learning. Let's consider ten typical formats that are used for professional development and knowledge sharing. These formats are suitable for various audiences and can be used in both academic environments and corporate training. Key formats include lectures, seminars, trainings, webinars, master classes, symposia, conferences, courses, internships, and practical classes. Each has its own characteristics and advantages, allowing it to meet learners' needs and achieve its goals. The choice of an appropriate format depends on the event's objectives, topic, and target audience. The right approach to organizing educational events ensures deeper knowledge acquisition and the development of essential skills.

How the course plan was rated

An experienced instructional designer who did not use neural networks successfully completed the course plan. Respondents rated their work as good. The plan included four modules, which were presented briefly and without clear, measurable goals. Half of the survey participants were able to identify the plan as the work of an instructional designer who did not use neural networks. This may indicate that differences in detail and text volume help distinguish human work from that generated by artificial intelligence.

The program for beginners using neural networks proved to be significantly more comprehensive and informative. Survey participants appreciated the high level of detail and meticulous planning, which created the impression that an expert in the field had been involved in its development. Ultimately, 52% of respondents were able to guess the author of the program, which, according to Philippa Hardman, confirms confidence in the ability of artificial intelligence not only to develop high-quality curricula but also to act as a virtual expert in the required field. This fact underscores the growing interest in the use of neural networks in educational processes and their potential for creating effective courses for users.

The plan of an experienced instructional designer who used neural networks again ranked first. This work significantly stood out from the first two, demonstrating an improvement in quality. The plan presented 12 modules, including both theoretical and practical sections. Each module clearly stated the learning objectives students were expected to achieve. The methodologists highly praised this work, and the overwhelming majority correctly identified the author. They noted that the plan was likely created by a human, as it was carefully crafted and reflected the expertise of a specialist. This was assessed based on both the content and the stated learning objectives. However, the document's design highlighted the use of neural networks, emphasizing the synergy between human experience and modern technology.

The third experiment demonstrated that methodologists do note certain advantages of using artificial intelligence.

• it can act as a subject matter expert;
• it serves as a good tool for working with text (which allows for the high-quality "packaging" of ideas and developments of an experienced instructional designer);
• it allows for a certain part of the work to be completed faster.

In the process of creating the finished product, it was important for instructional designers to see the human aspect. They paid attention to how the plan was compiled, and also assessed whether it contained the "trace" of a person or a machine.

Conclusions from the author of the experiment

The future of instructional design, according to Philippa Hardman, will be largely determined by the symbiotic relationship between humans and artificial intelligence. As with other AI applications, the characteristics of these relationships will vary depending on the user profile and needs. Instructional design will adapt to new technologies, enabling more effective and personalized educational processes. For experienced instructional designers, generative neural networks are a valuable tool. They demonstrate high effectiveness when a professional specifies precisely what needs to be created and how to do it. However, achieving the best results requires two key aspects: a deep understanding of instructional design and knowledge of neural network principles. It is important to be able to formulate correct prompts and effectively interact with the neural network in order to maximize its potential in the design process.

A neural network can be a useful tool for beginners in instructional design, helping to speed up tasks that novices struggle with, such as formulating learning objectives. Philippa Hardman acknowledges that a neural network cannot replace a true expert in instructional design. However, as experiments have shown, it can provide useful assistance, making it a valuable resource for training and development in this field.

Artificial intelligence is a powerful tool for the professionalization of instructional design. It automates key aspects of our work, such as content generation, and emphasizes the importance of deep domain knowledge. In this way, AI contributes to the level of achievement we have been striving for for years, allowing us to focus on the improvement and ultimate impact of our work. Philippa Hardman notes that this approach opens up new opportunities for educators, allowing them to use their skills and knowledge more effectively.

It is interesting to consider a situation in which an experienced instructional designer, working without neural networks, lost even to a novice in an experiment. Philip Hardman doesn't comment on this fact, but several conclusions can be drawn. First, experience doesn't always guarantee professionalism; second, experienced specialists may have habitual stereotypes and formulaic approaches that hinder them from finding optimal solutions. The experiment involving the development of a teaching strategy clearly confirmed this. Third, this case highlights the need to avoid overestimating human abilities and professionalism, just as it underestimates the potential of neural networks. These technologies can significantly expand opportunities in the educational sphere and offer innovative approaches.