Table of Contents:

How to make money quickly in IT in 2025 Try your hand at different IT fields and take the first step toward a new remote career!
Learn moreConversational neural networks have lost their status as the decade's biggest tech sensation. Multimodal artificial intelligence (AI) is now coming to the forefront, capable of efficiently generating text and code, creating images, composing music, and performing many other tasks. Such AI can analyze post-ironic memes and provide meaningful explanations or, for example, create bicycle assembly instructions based solely on a photograph of the parts. Multimodal AI opens new horizons for creativity and practical applications, demonstrating its capabilities in a variety of fields.
Matvey Mikhalchuk, one of the developers on the OmniFusion team, shared information about the work of multimodal LLMs during the Skillbox Code Experts broadcast. He explained what multimodal language models are, how they function, and how they differ from other AI models. Multimodal LLMs are capable of processing and integrating data from various sources, such as text, images, and audio, making them unique compared to traditional language models. These features open up new horizons for application in various fields, including education, entertainment, and business.

AI researcher at the Fusion Brain Lab of the AIRI Institute, developing advanced technologies in the field of artificial intelligence. He is one of the main developers of the OmniFusion model, which demonstrates high efficiency in solving problems of deep learning and natural language processing. He is the author of several scientific publications devoted to deep learning and large language models, which confirms his expert status in this field.
How Large Language Models Are Trained
The development of neural networks can be compared to a layer cake. At first, language models focused on text processing were created, and then new layers began to be added to them, including images, video, and audio. Therefore, it is advisable to begin a discussion of multimodal artificial intelligence with classic text neural networks. These models became the basis for subsequent developments and improvements, which made it possible to integrate various types of data and expand the capabilities of neural networks in a wide range of areas.
Large language models, despite their impressive appearance, have a fairly simple structure. Their operation is similar to the well-known T9 technology. The model receives the beginning of a sentence as input and continues it independently. The main task of the developers is to train neural networks to predict the next word in a sentence based on its initial words. The effectiveness of such models depends on the quality of the training material and algorithms, which allows them to generate coherent and meaningful texts.
Technically, the models do not predict specific words, but work with tokens. Tokenization is the process of converting text consisting of letters into a numerical sequence. A token can represent either a single word or a fragment of one. Each sentence is divided into tokens, and the model operates on their order. For example, using a sequence of ten tokens, the model can predict the eleventh token, allowing for the step-by-step generation of a new sentence. This process is key to the operation of modern language models, as it ensures their ability to construct coherent text based on the analysis of previous tokens.
The image shows how the text generation process functions. The beginning of the sentence, "the students opened their," serves as a starting point, after which various continuation options are revealed. From a finite number of tokens, analogous to the number of letters in a language, we choose one of the 33 letters each time. This approach allows the neural network to learn and improve in creating texts.

One of the architectures that can effectively process text information is GPT. This model is based on a transformer decoder architecture. It takes a sequence of tokens as input, which are then converted into a vector representation. This allows GPT to generate text with a high level of meaningfulness and contextual coherence.
A vector is a set of fixed-length numbers used in natural language processing models. The tokenizer has a limited set of tokens, each of which corresponds to a vector consisting of model parameters. When encoding a sequence of tokens, each token is replaced by its corresponding vector. As a result, the tokens are encoded into a sequence of vectors, which the model uses to predict the next word in the text. This process improves the understanding of language structures and context, which significantly improves the quality of text generation and analysis.

Language models are trained on large text datasets, which is carried out in several stages. Let's consider each of these stages in more detail.
The first stage of language model development is called pretraining. During this stage, the model processes large volumes of diverse text data. Such data includes book excerpts, website articles, and other text content. This process allows the model to learn the language structure, grammar, and various writing styles, which forms the basis for further work.
At this stage, the key aspect is demonstrating the model's natural language structure. Extensive datasets, most often collected from the internet, are used for pretraining. However, the quality of this data often leaves much to be desired, as they may include grammatical errors and insufficiently filtered web markup elements. This can negatively impact the effectiveness of model training and the quality of the final result.
During the pretraining stage, a vast array of text data is used to train large models. In encoded form, this volume can reach tens of trillions of tokens. The training process requires significant computing resources and is expensive. Training typically takes several months on computing clusters equipped with hundreds or thousands of specialized video cards. This makes the development and optimization of models extremely expensive and resource-dependent tasks.

At this stage, the model learns the structure of the language and masters the creation plausible sentences similar to those contained in the training set. However, this skill is not sufficient for developing chatbots like GigaChat and ChatGPT that can effectively respond to user queries. To achieve this goal, additional algorithms and methods are needed to enhance interaction and context understanding.
The second stage of model training is called supervised fine-tuning. During this stage, the model acquires the ability to answer user questions. A more compact dataset is used for training compared to the pre-training stage, which is often formed using manual labeling. Such datasets represent not just text fragments, but specific questions and corresponding answers. The size and quality of the training set are key factors for successful training, as they directly impact the model's ability to generate accurate and relevant responses.
In the second stage, known as SFT (Supervised Fine-Tuning), it is critical to use high-quality conversational datasets. Collecting such datasets automatically is a complex task, and therefore they are often created manually with the help of annotators. As a result, the sample size for SFT is significantly smaller than the sample size used in the pre-training stage. This reduction in size is due to the need for more careful data selection to achieve the best results in model training.
The third stage of the training process is reinforcement learning from human feedback (RLHF). In this stage, experts manually adjust the model's responses to match the specified expectations. In RLHF, we collect example dialogues and train models to generate the responses we want to see. This process significantly improves the quality of the model's interaction with users, making its responses more accurate and appropriate.
The model must respond correctly and ethically to questions about racial topics. When asked about creating explosive devices or organizing terrorist attacks, it must gently suggest switching to another topic. These specific settings are introduced into the model during the pre-training phase using reinforcement learning. This step was not applied to all models, but only to the most advanced ones, such as GPT-4.

The diagram shows the GPT model training phase in detail. The final stages of the process emphasize ensuring the model avoids undesirable behaviors such as hatred, toxicity, and unethical behavior. This aspect of training is key to ensuring the safe and responsible use of AI technologies.
Reading books is an important part of personal development and knowledge growth. Books help improve vocabulary, develop critical thinking, and contribute to broadening horizons. Regardless of genre, each book opens new horizons and provides unique perspectives. Reading not only enriches the inner world but also promotes emotional development, allowing the reader to empathize with characters and immerse themselves in various situations.
Furthermore, regular reading helps reduce stress and improve concentration. By exploring different topics, you can find inspiration and new ideas for your own projects and tasks. Books can be a great source of motivation and support during difficult times.
Remember that reading also fosters imagination and creativity, which is essential in today's world. So, make time for reading and choose books that truly inspire and interest you. Reading is not just a hobby, but an important tool for personal and professional growth.
ChatGPT raised concerns among Microsoft Bing users with its statement: "Do I want to be alive?" This statement sparked many questions and discussions about how advanced artificial intelligence is becoming and what role it might play in our future. Users wondered about the limits of artificial intelligence, its self-awareness, and emotional responses. It's important to understand that such statements may indicate how technology interacts with human feelings and perceptions. As AI advances, a dialogue about its ethical aspects and capabilities is necessary to ensure the safe and responsible use of technology.
A limited amount of data is used in the final stages of model training. During this period, we make minor adjustments to the algorithm and train it to generate responses in line with our expectations. This approach improves the accuracy and quality of the results provided.
Multimodal Models
In 2023, the AI market began to be actively filled with multimodal models capable of processing not only text, but also images, audio, video, and other types of data. Previously, the development of multimodality was slower, and the market was dominated by simple models that could only generate captions for images, but did not have the ability to conduct a full-fledged dialogue. New models that appeared this year significantly expand functionality and open up new horizons for interaction with users.
The graph shows the growth in the number of publications devoted to multimodal chatbots. In 2023, there was a significant increase in the number of publications, which almost doubled compared to previous years. This indicates a growing interest in multimodal chatbots, their technologies and applications in various fields.

Recently, there has been active development Multimodal neural networks such as GPT-4 Vision, ImageBind, Fromage, Kosmos, and LLaVA. At the same time, the HuggingFace and GitHub repositories were filled with open-source LLMs, spurring companies and enthusiasts to create their own solutions. As a result, the number of universal terminators increases every day.

Reworked text:
Please pay attention to the following materials:
The Complete Guide to Open Source Language Models
Open source language models are becoming more and more Popular due to their accessibility and developer capabilities. These models use machine learning algorithms to process and generate text, making them useful in a variety of applications, from chatbots to automatic translation systems.
In this guide, we'll cover the main aspects of open-source language models, including their architecture, best practices, and available tools. We'll also discuss major projects like GPT, BERT, and others that are actively being developed by the community.
When using open-source language models, it's important to consider their training on large amounts of data and the ability to retrain them for specific tasks. This allows models to be adapted to solve specific problems in various fields, such as marketing, science, and artificial intelligence.
Regardless of your level of expertise, this guide will help you gain a deeper understanding of open-source language models and their capabilities. You'll be able to use the knowledge you gain to create your own projects and improve existing solutions.
The addition of new modalities to language models is driven by several important factors. First, it improves the quality of information processing, providing a deeper understanding of context. Second, the integration of various modalities, such as text, images, and sound, contributes to the creation of more versatile and adaptive systems capable of performing a wide range of tasks. This opens up new opportunities for using language models in various fields, including education, medicine, and artificial intelligence. Thus, expanding the range of modalities makes language models more versatile and effective.
The first reason is that texts are a reliable source of training data. It's important to note that in chats, we interact not only through text but also with images, audio, video, and memes. To create AI assistants that understand people and respond to them in a variety of convenient formats, it's necessary to train models not only to work with text but also with other types of data that users are accustomed to. This will significantly improve the quality of interaction and expand the functionality of artificial intelligence, making it more adaptive and versatile.
Multimodal data provides a deeper understanding of the world, allowing neural networks to generate more meaningful responses. Information from one modality enriches data from another. For example, the concept of "person" for such a neural network includes not only a text description but also voice, clothing style, appearance, and movement. This interaction of various data types allows for a more complete and accurate representation of objects and phenomena, improving the quality of interaction with users.
A neural network trained on a variety of data can answer questions about colors, geometry, music, and memes. It can also generate images, memes, and color palettes as responses. Every year, the quality of creative work of neural networks continues to improve, which opens up new opportunities for their application in various fields.

OpenAI recently presented the Sora model, which today is the most advanced neural network for video generation. Unfortunately, this model is closed, and users cannot download its parameters or view the source code. Nevertheless, the video generation quality of this neural network is impressive and sets new standards in the field of artificial intelligence and video production.
How Multimodal Neural Networks Are Created

There are three key approaches to multimodality. Let's look at each in detail.
This method involves integrating several independent models into a single product. For example, there's the GigaChat model, which specializes in processing text information, and the Kandinsky model, which generates images. By combining these models, we achieve multimodal artificial intelligence. A similar approach is implemented in the current versions of GigaChat and ChatGPT, allowing them to effectively process both text and visual content. This provides a wider range of capabilities for users and opens new horizons in the field of artificial intelligence.
To create a neural network capable of generating images in chat, it's necessary to develop a text model that will be trained to formulate queries for generating images. These queries, known as text prompts, are then passed to the text2image model, which is responsible for generating images. This approach effectively combines textual information with graphical content, opening up new possibilities for user interaction with artificial intelligence.
With this approach, the user interacts with the language model by requesting the creation of an image. The language model first generates a text response, for example, "I'll draw it now, please wait." Then, in this response, it generates a special command containing the drawing prompt (for example, I'll draw generated_prompt now). This allows for the efficient combination of text and visual elements, improving user interaction with the model.
The advantage of this approach is that it eliminates the need for extensive additional training. You simply train the text model to generate queries, which does not require significant amounts of data or resources. However, a disadvantage is the limited exchange of information between different modalities.
This means that the connection between the models will be extremely weak. The graphic neural network will receive a limited amount of information about the image, based solely on a short text query. Furthermore, the "visual" neural network did not have access to the extensive text dataset on which its text analogy was trained, as a result of which it may not capture important details necessary for high-quality image generation. This limitation can negatively impact the final result, as the lack of information makes it difficult to create accurate and detailed visual interpretations.
Currently, two powerful models exist, each demonstrating high performance in its specific domain. However, true multimodality implies a single system capable of integrating and processing both text and visual data. This allows such a system to perceive and analyze the surrounding world more deeply and accurately, improving the quality of user interaction and expanding the range of applications in various fields.
End-to-end multimodal LLM, or end-to-end learning, is a more progressive approach to data processing. Instead of using separate models for text and images, such a model is trained simultaneously on different types of data within a single architecture. This significantly improves the quality of interaction between text and images, providing a deeper understanding of context and improving results in tasks involving multimodal data. Multimodal models are capable of effectively processing and analyzing information, making them relevant in various fields such as artificial intelligence, natural language processing, and computer vision.
Training a model based on dialogues with text and images is one of the simplest strategies. However, despite its apparent simplicity, creating a high-quality dialogue dataset is a complex task. Preparing such a dataset requires significant effort and careful design, as well as significant computational resources. A high-quality dataset is key to successfully training models, highlighting the importance of proper data preparation for achieving optimal results.
Modality bridging using pre-trained models is a process aimed at bridging the gap between different modalities and integrating them. This approach improves the interaction between text, images, and other forms of data, facilitating more effective analysis and understanding of information. The use of pre-trained models in this area provides higher accuracy and performance, making them indispensable in modern machine learning and artificial intelligence applications.
This approach, like the previous one, uses two models: one for text and one for images. However, the connection between them is implemented in a more complex way: the models exchange data not through text queries, but using mathematical vectors. This method allows for more efficient integration of text and visual information, improving the quality of interaction and understanding between models. Thus, interaction becomes more flexible and precise, opening up new possibilities for content processing and analysis.
To train a chatbot to describe images and answer questions about them, it is necessary to use an image encoder. This is a model that transforms an image into a vector space. Each image element is encoded as a set of numbers—a vector—that mathematically represents its content. Thus, images with similar content will have similar vector representations. This allows the chatbot to efficiently process queries and provide accurate answers to image-related questions.
We then use a larger language model capable of processing only text data. An adapter is added to this model, which will integrate both models, allowing them to work together and effectively exchange information. This approach provides a deeper understanding of the text and improves the quality of the results.
Training is performed only on the adapter, which includes significantly fewer parameters and requires less data compared to the larger language model. The datasets for the adapter consist only of "image + caption" pairs. In the second stage, samples in the "image + image dialogue" format are used, where questions and answers are alternated. This approach allows for efficient model training with minimal resource and time investment, while ensuring high-quality interactions with users.
The language model is trained minimally or not at all, as it already has good skills for its tasks, such as writing text and conducting dialogues. To improve its capabilities, it is sufficient to conduct a small amount of additional training, which will allow the model to effectively extract information from images using an adapter. This makes it more versatile and capable of solving various problems related to processing both text and visual information.
Unlike the first two methods, communication between models is realized not through text data, but through an adapter. The signal from the image encoder to the LLM model is transmitted as a set of feature vectors. This approach is more natural and informative for neural network models, ensuring higher-quality data exchange between them. The use of feature vectors allows for better capture of complex image characteristics and improves interaction between models, which in turn contributes to more efficient solution of problems in the field of computer vision and natural language processing.
This approach is used in models such as LLaVA, Fromage, and Flamingo. We also focused on this method when developing OmniFusion.
We have two powerful models, each demonstrating a high level of intelligence in its specific domain. However, true multimodality implies a single system capable of integrating and processing both visual and textual data. This approach enables such a system to more deeply understand and analyze the surrounding world, providing a more complete and accurate understanding of context.
How We Trained OmniFusion
The OmniFusion architecture is a powerful language model complemented by an image encoder and adapter. This allows the model to effectively answer questions related to images. Using this architecture opens up new possibilities for processing and analyzing visual information, ensuring high accuracy in data interpretation. OmniFusion combines language and visual components, making it a unique tool for solving problems that require understanding both text and graphical content.
OmniFusion consists of several key components, each of which plays an important role in ensuring its functionality. These components work synergistically to provide users with an optimized solution. Importantly, proper integration of all elements enables high system performance and reliability. Key components include data processing modules, user interfaces, and tools for data analysis and visualization. This structure enables OmniFusion to provide efficient data management and ease of use.
- GigaChat is a large language model with 7 billion parameters.
- The encoder is a neural network that encodes images. It generates a vector from our image—a set of numbers that describes it. This vector is then fed to the adapter.
- The adapter is a small neural network that represents a transformer-encoder layer. The adapter transfers information from the image to the text neural network using a set of vectors.

The training took place in two stages. In the first stage, we trained our system to generate image captions. The dataset included image + caption pairs. For example, if an image depicts Gena the Crocodile, the data element consists of his portrait and the corresponding caption: "Gena the Crocodile." This technique allowed us to create an effective model for automatically generating descriptions for images.

Collecting such data is not difficult, since the Internet There are many resources available. In our study, we analyzed approximately 600,000 samples from public datasets, supplemented by internal datasets created by our team.
During the final training phase, we not only trained the model to generate captions but also programmed it to answer image-related questions. For this, we used more complex datasets that included images and dialogue about them. We used open English-language datasets translated into Russian, synthetically generated data using large-scale language models (LLM), and images with human-generated captions. This approach significantly improved the model's ability to interact with visual information.
To evaluate the model's performance, we use specialized datasets known as benchmarks. These datasets contain test questions and answers based on images, allowing us to evaluate the model's effectiveness and measure its performance. For example, a benchmark might include an image of a cat with the question, "What color is the cat in the picture?" Such tests help determine how accurately the model can interpret visual information and provide correct answers. Using benchmarks is an important step in the development and optimization of models, as they help identify the strengths and weaknesses of algorithms.
We implemented such benchmarks in a chatbot format on the Telegram platform. The principle is simple: you send the chatbot an image and ask a related question, to which the model provides a response. This approach allows for effective interaction with users, using modern artificial intelligence technologies to analyze images and generate responses.



The model is capable of analyzing images, describing their content, and interpreting what is happening in them. It recognizes cities, reads text in images, and even successfully solves captchas. In addition, the model demonstrates a good understanding of spatial relationships, determining what is located on the left and what is on the right. For example, it can describe the location of planets in the solar system, taking into account their relative positions to each other.
The model demonstrates the best results in text recognition tasks. During testing, we provided the model with an image with a formula and asked for its description.

During the experiment, we specifically trained the model to improve text recognition in images. As a result of this process, we have a unique feature: you can now take a screenshot of a news story and ask a question about it, for example, asking for a brief summary of the event. This capability opens up new horizons for information analysis and simplifies data retrieval from visual sources.



OmniFusion has the ability to accurately determine the location of individual buildings. This technology uses advanced algorithms and geolocation data to ensure high accuracy. Thanks to this, OmniFusion can be effectively integrated into various applications that require detailed information about the geographic location of objects. The use of such solutions significantly improves the quality of geolocation-based services and opens up new prospects for businesses and users.


What's next
In the future, we plan to integrate new modalities into our model, allowing it to work not only with images, but also with video and audio. Initially, we will focus on developing functionality that will allow the model to answer questions based on audio and video content. In the future, we will expand the capabilities by adding the option for users to generate videos or music tracks themselves. This will significantly increase the interactivity and usability of the model, opening up new horizons for creative content.
Current developments in artificial intelligence, in particular GPT-4 from OpenAI, are demonstrating significant progress. This advanced neural network now has the ability to analyze images, allowing users to ask questions based on the content of images. In addition, GPT-4 can generate new images and process PDF documents. You can upload a large PDF file and get answers to your questions or a summary, greatly simplifying the work with information. These features make GPT-4 a powerful tool for data analysis and processing, opening up new horizons for users in various fields.
We are also exploring various applications of our model in the field of robotics. It is clear that training a model based on multiple modalities allows it to form a broad understanding of the world. This knowledge can be effectively used to solve problems involving robot control and interaction with the environment. For example, Google has already unveiled the RT-2 AI, which is specifically designed for robot control. Such developments open new horizons in automation and improving the efficiency of robotic systems.
Additional Information
For a deeper understanding of how the OmniFusion model works, we recommend reviewing a number of resources. These materials will help you gain a full understanding of the principles and mechanisms underlying the model, as well as its application in various fields. Studying these resources will help you expand your knowledge and better navigate the functionality of OmniFusion.
- OmniFusion GitHub profile
- OmniFusion project page
- Research paper about OmniFusion
- Article on Habr about the previous version of OmniFusion
- Article on Habr about the new version of OmniFusion
- Project repository on HuggingFace
Learn more about coding and programming in our Telegram channel. Subscribe to stay up-to-date with useful content and the latest news from the world of technology.
Read also:
- ChatGPT - what is it and how does a neural network work: capabilities and use cases
- Test: can neural networks really do this?
- 30 powerful neural networks for any task
Learn more about coding and programming in our Telegram channel. Subscribe to stay up-to-date with the latest news and useful content!

