Code

A Guide to Large Language Models in Coding

A Guide to Large Language Models in Coding / Skillbox Media

Free education: "Practical course on neural networks"

Learn more

Just a couple of years ago, the launch of a new open-source LLM was perceived as a significant event in the world of information technology. Today, however, such news is no longer surprising. Dozens of open-source language models are released every month, and their number reaches hundreds within a year.

To help you navigate this abundance, we have prepared a guide to current open-source neural networks.

Contents

  • Open-source models arise from the collaboration of developers, researchers, and enthusiasts who strive to create accessible and transparent solutions. These models are often developed within communities where participants share their knowledge and experience, which allows for improvement and adaptation of the technologies.

    The process of creating open-source models usually begins with research and analysis of existing technologies. Developers can then build on these developments to create something new, adding their own ideas and improvements.

    Furthermore, open source projects are often supported by various foundations and organizations that provide funding or resources for development. Such initiatives help speed up the process and expand opportunities for participation by a wider range of people.

    As a result of collaborative efforts, open source models can quickly evolve and adapt to new challenges, making them popular in various fields, including programming, artificial intelligence, and other areas of technology.

  • How are they evolving?
  • What are their strengths and weaknesses?
  • There are several types of open language models (LLM), each with its own characteristics and areas of application. First, we can distinguish models that are intended for general use and are capable of performing a variety of tasks, such as text generation, question answering, and translation. These include, for example, well-known models such as GPT and BERT.

    Secondly, there are specialized LLMs created for specific domains. These models are trained on specialized data and may prove more effective in narrow areas, such as medicine or legal matters.

    Furthermore, open language models can vary in size and architecture. Some have billions of parameters and require significant computing resources, while other, more compact versions require less power, making them accessible for use on devices with limited capabilities.

    It's also worth noting that open LLMs often come with the option of retraining, allowing users to tailor them to their needs and improve performance on specific tasks. Importantly, such models can make cutting-edge technologies available to a wider audience, fostering innovation and research.

  • What types of open licenses are used to publish open source software?
  • To select the most effective language learning models (LLMs), there are several key factors to consider. First, consider the model's performance on various tasks, such as text generation, context understanding, and semantic analysis. The size and architecture of the model are also important, as they can impact the quality of the results.

    Furthermore, it is worth evaluating the availability of models and their ease of use. Some LLMs can be computationally intensive, which should be taken into account when choosing. Ethics is also an important issue: it is worth examining how models handle bias and ensure safe user interactions.

    It is also useful to read reviews and studies that provide information about different models, their advantages and disadvantages. Finally, it is important to keep up with the latest updates in LLM development, as technology is rapidly evolving and new, more advanced solutions may appear on the market.

  • What well-known open-source models are worth considering?
  • What resources might be interesting for exploring open-source and commercial (proprietary) language models?

The Origin of Open-Source Models: Where and How They Are Created

There are a huge number of open-source neural networks. However, it should be noted that these are not independent initiatives - most of them are built on top of a few large language models, known as base models.

Developing and training such a model is associated with significant financial costs and high computing power. In this regard, only large research groups and IT corporations, such as Google, OpenAI, and others, can engage in such projects. For example, GPT-3 cost its creators nearly $5 million to train.

The base model is an artificial neural network trained on vast amounts of data and can be adapted to perform a variety of tasks.

Once developed, the new model can be released under either a proprietary or open-source license. With an open license, other organizations and individual developers can adapt and improve the model to suit their own needs.

Working with such models does not require significant financial investments or powerful computing power. This is why startups often choose open-source LLMs. Modified versions of these models are commonly called forks, which comes from the English word "fork."

Some of the well-known base models that formed the basis of open LLMs include:

  • The LLaMA model, as well as its updated version LLaMA 2, were created in collaboration with Microsoft.
  • BLOOM (an open-source multilingual language model developed by the BigScience project) was created with the active participation of Hugging Face.
  • Several years ago, OpenAI introduced GPT-2 when the organization focused on creating exclusively open-source software solutions.
  • Falcon is the latest innovation created by the Technological Innovation Institute (TII), located in Abu Dhabi, United Arab Emirates.
  • The T5 line of models developed by Google.

Modern LLM models can be viewed as a family tree, allowing us to trace their development and interrelations.

Family tree of modern LLMs. Open source models are represented by filled rectangles, and proprietary ones are unfilled. Source: LLMsPracticalGuide / GitHub Infographic: Maya Malgina for Skillbox Media

This diagram focuses on the models highlighted in the filled rectangles. We will concentrate on the open source solutions that will be discussed. You can also trace the main models and their evolution up to 2023.

Stages of Evolution of Open Source Software Models

Open source language models face the same challenges as commercial neural networks: they are subject to frequent hallucinations, have a limited context window length, need the ability to process information from different modalities, and so on. In this regard, their evolutionary directions are similar.

Reducing the number of hallucinations. Language learning models are capable of generating false information that may appear entirely believable. These distortions are called hallucinations. To date, it is still not possible to completely eliminate such errors in neural network responses.

The closed GPT-4 model is considered the leader in the field of combating hallucinations, demonstrating an error rate of only 3% of the time. However, the open LLaMA 2 70B model is hot on its heels, with accuracy comparable to Google DeepMind's widely touted proprietary Gemini model.

Expanding the length of the context window. The larger the volume of text a language model can process, the better it copes with tasks and the higher-quality answers it can provide. This is explained by the fact that the model is able to process much more information, which, in turn, improves its analytical abilities.

The most advanced closed versions of GPT-4 and Claude 100K are capable of processing over 100,000 tokens simultaneously. At the same time, open-source neural networks are making efforts to approach these figures.

For example, the original Mistral 7B can process 8,000 tokens, while its recent fork, Nous-Yarn-Mistral-7B-128k, developed by Nous Research, can manage a context window of 128,000 tokens.

Working with data of various modalities. Modern neural networks can process not only text information, but also images, video, and audio files. This feature has already been implemented in a number of open LLMs:

  • the Nous-Hermes-2-Vision-Alpha visual-verbal model;
  • the Qwen-VL multimodal neural network developed by the Chinese corporation Alibaba Cloud;
  • a multimodal interpretation of LLaMA, called LLaVA-13B.
An example of the open multimodal model Nous-Hermes-2-Vision-Alpha. The AI ​​was able to analyze a photo of a hamburger and explain why eating it can be harmful to health. Screenshot: Teknium (e/λ)/X

Reduced LLM prices. One of the main difficulties associated with neural networks is the significant cost of creating basic models. However, the availability of some of them with open licenses allows for a significant reduction in the costs of their further training and integration. For example, adapting and deploying the open-source Alpaca and Vicuna-13B models, which are based on LLaMA, cost the developers only $600 and $300, respectively.

One way to reduce costs is to use neural networks that can both create "synthetic" training data and evaluate the performance of the new model. This method is known as RLAIF (reinforcement learning with AI feedback), which means training using artificial intelligence to provide feedback.

Ability to run language models on devices with limited resources. Many open-source LLaMAs have a smaller number of parameters than their closed-source counterparts. This allows such neural networks to be used on less powerful systems, even on regular home computers.

For example, the Mistral 7B model has a number of parameters that is 25 times smaller than that of GPT-3.5, which was used as the basis for the basic version of ChatGPT. This means that it requires significantly fewer computing resources to run - about 187 times less than GPT-4 and nine times less than GPT-3.5.

Researcher, Department of Computational Semantics, Institute of Artificial Intelligence AIRI

Open-source models have given businesses the ability to use language models with virtually unlimited flexibility. For example, such solutions allow companies to fully manage the processing of user data, customize it to their own requirements, and, overall, minimize risks by using their own infrastructure.

Furthermore, the development of open-source models has contributed to the scientific community's increased expertise in working with large language models. Nowadays, it's commonplace for enthusiasts to launch a chatbot similar to ChatGPT on their laptop, whereas just two years ago such a possibility would have seemed unthinkable.

Optimizing existing neural network architectures and developing new ones remain pressing tasks. One of the key challenges of large language models (LLM), which largely determines their limitations, is the specific characteristics of the transformer architecture. Startups working on open-source models and experimenting with their internal structure are expected to be able to offer solutions to these challenges.

One possible solution to this problem could be the Mixture of Experts (MoE) architecture, which mimics the proposed structure of GPT-4. This model includes eight expert neural networks, each specialized in a specific set of tasks. For example, the open-source Mixtral 8x7B model, developed by the French company Mistral AI, which uses this approach, demonstrates a response generation speed six times higher than that of the original LLaMA 2 70B model.

Creating multi-agent systems using large-scale language models (LLM) is an important area of ​​research. While changing the architecture can be beneficial, there is an alternative way to improve the efficiency of language models. This method involves developing systems consisting of multiple neural network agents that are able to communicate and coordinate their actions to achieve user-defined goals.

Open-source language models, which do not impose high computational requirements, are optimal candidates for such systems. Currently, there are already implemented projects such as AutoGPT, GPT-Engineer, LangChain, and GPTeam.

Developing large-scale language models (LLM) for languages ​​other than English is a complex task. Neural networks most often focus on English, as it is the language in which the bulk of the training data used in their training is created. Other languages, spoken by hundreds of millions of people, are often overlooked. Effective training for these languages ​​requires collecting and building high-quality datasets, which in turn requires additional resources and effort.

While advanced neural networks like GPT-4 can handle approximately 100 languages, there are over 7,000 known languages ​​in the world. Experts hope that open language models (LLM) will help overcome this problem.

In 2023, the Jais model, which can communicate in Arabic, was presented in the United Arab Emirates, and a variant of LLaMA designed for Portuguese was also announced. In Russia, Yandex and Sber have released the YaLM 100B and ruGPT-3.5 13B neural networks, focused on the Russian language.

The process of creating resources for less common languages ​​continues. In 2023, a project called Massively Multilingual Speech (MMS) was launched to develop datasets for 1,100 previously underserved languages.

Read also:

The Swift Tiger and the Insightful Dragon: China's Capabilities and Plans in the field of generative artificial intelligence.

Researcher in the Computational Semantics Department at the AIRI Institute of Artificial Intelligence

Open-source open source models have given enterprises the opportunity to apply large language models (LLM) with almost no restrictions. For example, such solutions enable organizations to manage all aspects of user data processing, customize them to suit their requirements, and generally minimize risks while leveraging their own infrastructure.

It's also worth noting that the emergence of open-source models has contributed to an increase in the level of knowledge and skills among scientists in the field of working with large language models. Nowadays, no one would be shocked by a chatbot resembling ChatGPT, which was developed and launched on an ordinary laptop by some enthusiast, while just two years ago this would have seemed completely unbelievable.

Strengths and Weaknesses of Open Language Models

Companies prefer open neural networks for several reasons, as they offer a number of advantages over closed, proprietary models:

  • Protection and preservation of information privacy. Open-source models can be installed on their own servers, eliminating the need to transfer data to external platforms. This gives users complete control over the data analyzed by the neural network.
  • Cost reduction. Using open-source LLMs eliminates the need to pay a subscription or regularly pay developers under contracts. This makes such solutions especially attractive for startups and organizations with limited financial resources.
  • Reduced dependence on IT service providers. Users have the opportunity to choose from a variety of open LLMs, allowing them to find the optimal option for their needs. This gives companies the freedom not to be limited to the services of a single AI solution provider, as well as the ability to select the most effective models or combine different ones.
  • Transparency in the operation of open-source language models provides the opportunity for a detailed analysis of their internal structure and data processing mechanisms. This approach helps identify potential risks and prevent information leaks to external servers.
  • Open-source projects are under the patronage of communities of developers and specialists. This allows for prompt correction of any errors or problems that arise, and also provides detailed documentation that explains the specifics of working with the neural network. While this is typical for most open-source models, there are some negative examples.
  • Unconventional Methods and Strategies. Open-source language models (LLM) provide the opportunity to conduct experiments with artificial intelligence based on new fundamental models. Even small companies have the chance to creatively adapt these neural networks and use them as a foundation for creating their own original solutions.

CEO of Avatar Machine, initiator of the development of the Sabina Ai psychologist chatbot and one of the authors of the project FractalGPT.

The emergence and widespread adoption of large, open-licensed language models is driven by a global trend aimed at improving performance and reducing LLM costs. Modern users seek to avoid dependence on closed, proprietary systems, which expose them to the influence of foreign companies and political instability. These factors motivate a shift away from well-known solutions from foreign IT giants, such as OpenAI.

However, open language models have certain drawbacks:

  • Integrating and supporting such systems can take longer and require deeper technical expertise than proprietary models. Proprietary models typically provide a ready-made solution that can be used immediately after purchase.
  • Projects created by lesser-known teams may use incomplete or low-quality training datasets. This, in turn, leads to a decrease in the accuracy of neural network output and increases the likelihood of hallucinations.
  • Open-source models may experience undocumented operational difficulties. For example, this may manifest itself in incompatibility between different versions of language models.

CEO of the company "A-Ya Expert", engaged in the development of solutions in the field of artificial intelligence.

Open LLMs should be available not only in the form of the source code of the models, but also in relation to the data, used for their training. This is crucial, as the issue of "data poisoning" remains a significant problem. I am confident that the main focus now will be on ensuring the purity and transparency of this data.

Engineers, scientists, and government agencies will inevitably face data trust issues when implementing solutions based on open AI models. Therefore, only transparency and high quality of the datasets on which neural networks are trained can ensure open-source models’ place in the market.

Read also:

The pitfalls of open-source software: what risks are associated with using free software.

CEO of the Avatar Machine company, developer of the Sabina Ai psychologist chatbot and one of the authors of the FractalGPT project.

The emergence and active distribution of large language models with open licenses was the result of a global trend aimed at increasing efficiency and reducing costs LLM. Consumers are currently seeking to avoid dependence on closed, proprietary systems that tie them to foreign suppliers and political instability. These factors contribute to the refusal to use popular products of such international IT giants as, for example, OpenAI.

CEO of the company "A-Ya Expert", engaged in the development of artificial intelligence technologies.

Open LLMs should be available not only in source code format, but also in relation to the data on which they are based. This is of great importance, since the issue of "data poisoning" remains a serious problem. I am convinced that the current emphasis will be on ensuring the purity and transparency of data.

When implementing technologies based on open-source artificial intelligence models, engineers, researchers, and government agencies inevitably have doubts about the reliability of the data. In this regard, only transparency and a high level of quality of the datasets used to train neural networks can ensure that open-source models have a worthy place in the market.

The Diversity of Open Language Models: What You Need to Know

Open-source models can be classified according to several criteria, such as the level of training, size, and the availability of support for various languages. Let's look at each of these aspects in more detail.

Often, developers only make pre-trained versions of their neural networks publicly available, known as "pretrains." For example, this is what experts at Sber did with the Russian version of ruGPT-3.5, as well as Zuckerberg with the original LLaMA model.

Before being released to the public, such language models undergo a lengthy and complex training process, during which they process vast amounts of unlabeled text data. This process requires significant computational resources and material investment. As a result, neural networks develop only a general understanding of the language.

However, applying pretraining to specific tasks can be difficult. Its capabilities are limited to generating continuations of text entered by the user. For example, a user can easily complete a sentence that begins with the definition "Machine learning is..."

When the user decides to engage in a conversation or sends a command to be executed, the neural network begins to generate meaningless utterances instead of truly useful information.

Therefore, preference is given not to pretrained models, but to baseline versions that have undergone additional adjustments, known as "fine tuning." The names of such large language models (LLMs) typically include the word "Chat" if the neural network is adapted for conversation, or "Instruct" if it can follow instructions using a method similar to the RLHF used in training ChatGPT.

More specialized forms of retraining also exist. For example, the MPT-7B model offers a version called StoryWriter, which focuses on creating fictional stories that require a significant amount of context. And we shouldn't forget the many language models that generate program code. The names of such neural networks often include the word "Code", as in StableCode or CodeGeneX.

According to this criterion, large language models (LLM) can be divided into three groups:

  • English-language;
  • using one of the regional languages, for example, Russian;
  • multilingual, capable of simultaneously working with several languages ​​​​in addition to English.

For example, when interacting with neural networks in Russia, users will urgently need support for the Russian language. However, the primary language for most models remains English.

This is due to the fact that it is in this language that the maximum amount of information is presented, which is used to train neural networks. They master other languages ​​​​through additional training processes and adjustments to their architecture.

In general, most open source models are capable of perceiving the Russian language. However, there is one significant problem: the tokenizers used in most popular language models were originally designed for English, or at least for the Latin alphabet. As a result, Cyrillic text takes up more space in the form of tokens, significantly reducing the available context for their processing.

Mikhail Salnikov

The era when it was believed that the size of a neural network directly impacted its performance is gradually disappearing. Nowadays, open-source models, although much smaller, demonstrate results comparable to large proprietary systems. Therefore, in today's reality, it is wiser to choose large language models based on the principle of optimal balance: the neural network should be as small as possible while still being able to successfully solve the assigned tasks.

Each model can be evaluated using continuously updated quality metrics known as benchmarks. Based on these metrics, all LLMs can be classified into two main groups:

  • There are models that show results approaching a certain level of "pruning quality." ChatGPT (GPT-3.5-Turbo) is usually taken as a basis.
  • There are a number of models that do not meet the optimal price-performance ratio. These include either excessively large language models, the cost of which is prohibitive, or small models with no more than 7 billion parameters. The latter, as a rule, demonstrate significant shortcomings in their performance, which can be identified using specialized benchmarks aimed at assessing language understanding.

Victor Nosko

The second key aspect regarding model size is its type: full or quantized. Quantizing a neural network allows for a reduction in computing resources, including minimum RAM requirements. However, it is worth noting that such optimization leads to a decrease in the accuracy of the language model.

Model hosting prices are often reduced thanks to the quantization process. This allows them to be run even on standard home video cards, such as the GTX or RTX 3070-3090 from NVIDIA. However, it is worth noting that in this case, the quality of operation may deteriorate by 5-15% compared to the original version, although in some situations this decrease is quite acceptable.

Viktor Nosko

Classification of popular open-source software licenses

The openness of open-source models can vary, and this largely depends on the license chosen by the developer.

Models whose use is associated with certain significant restrictions are considered partially open. For example, the developers of LLaMA 2 require users to agree to a license agreement containing a lengthy list of conditions and restrictions before downloading it. One of these conditions prohibits the use of the neural network in projects with more than 700 million monthly active users. And these are not all the restrictions. Results obtained with LLaMA 2 cannot be used to train other LLM models except for LLaMA itself and its derivatives.

Most large language models (LLMs) are distributed under standard open-source licenses. Among them, the main ones are:

  • The Apache 2.0 license allows models to be used for a wide variety of purposes, as well as modified and distributed under the established terms, without the need to pay royalties to the developer. It is under this license that most open language models were created, such as T5, Mistral 7B, and others.
  • The MIT license was developed by the Massachusetts Institute of Technology and is in many ways similar to the Apache 2.0 license. However, it allows for the reuse of open source code within commercial software. One example of this license is the Phi-2 model from Microsoft.
  • Open RAIL-M v1 is supported by the BigCode community, which was initiated by Hugging Face. This license grants users free access to models, as well as the right to modify their source code and share LLMs along with their variants. At the same time, the license imposes certain restrictions prohibiting the use of models for unethical or illegal purposes. The BLOOM model is also distributed under this license.
  • The CC BY-SA 4.0 license is supported by the international non-profit organization Creative Commons. It allows copying and distributing LLM models, as well as making modifications and additions to them, for a wide variety of purposes, including commercial activities. However, when used in this way, the new models must be distributed under the same license as the original. The MPT-7B-Chat model is available under this license.
  • BSD-3-Clause is a free software license that imposes minimal restrictions on the use and distribution of neural networks. This license permits unlimited copying for any purpose, provided the copyright notice and warranty disclaimer are included. Despite its flexibility, this license is rarely used. From the available examples, we were able to find one well-known language model with this license - CodeT5+.

Read also:

BSD and MIT Licenses: Key Differences and Areas of Application

BSD and MIT licenses are two popular Types of open source licenses commonly used in software development. Despite their similarities, there are certain differences that can influence the choice between them.

The MIT license, thanks to its simplicity and conciseness, allows developers to freely use, modify, and distribute software. It does not impose significant restrictions, making it attractive to startups and projects seeking maximum flexibility.

On the other hand, the BSD license, especially its three-clause modification, also provides broad permissions for use and distribution, but includes additional conditions, such as copyright retention and a disclaimer. This can be an important aspect for large organizations that need to protect their reputation and minimize legal risks.

In terms of usage, both licenses find application in a wide variety of projects: from small libraries to large frameworks and systems. For example, the MIT license is often found in well-known projects like jQuery and Ruby on Rails, while the BSD license is used in systems like FreeBSD and OpenBSD.

Thus, the choice between the BSD and MIT licenses may depend on the specific needs of the project and the preferences of the developers, but both remain important tools in the open source world.

How to Find the Best LLM

To determine which open source language model is the best, experts have developed special virtual platforms known as leaderboards. On these platforms, language models compete with each other.

On such platforms, each neural network is analyzed using various quality criteria, known as benchmarks. It is important to realize that there is no universal LLM that would perfectly fit all parameters. A model may show impressive results in one area, but be among the worst in others.

Therefore, when choosing an LLM, you should rely on the metrics that best suit the task for which we are selecting an LLM. Most testing platforms are equipped with intuitive interfaces that allow you to sort the lists of available models based on the desired characteristics.

Here are a few leaderboard options that we can recommend:

  • Open LLM Leaderboard is a resource developed by Hugging Face that is used to monitor, rank, and automatedly evaluate modern language models and chatbots presented on the corresponding website. This tool uses a unique evaluation methodology from EleutherAI, which is based on the analysis of seven different benchmarks.
  • Chatbot Arena is another public platform for analyzing language models on the Hugging Face website. It is crowdsourced. This platform has accumulated over 200,000 opinions from real users, making it possible to evaluate language models using the Elo ranking system, similar to that used to calculate chess skill levels.

The key concept of the Chatbot Arena Leaderboard is to conduct paired analysis of model responses by human evaluators using the Elo rating system. The problem is that there are "cheating" methods that allow models to demonstrate artificially inflated results in tests that do not reflect their true qualities. In this situation, manual evaluation based on simple comparisons partially overcomes such shortcomings.

Viktor Nosko

  • The AlpacaEval leaderboard is an automated platform for evaluating language models that belong to the Instruct category. This system is developed based on the AlpacaFarm approach, which allows for assessing how effectively language models follow general user instructions. The "arbitrator" and source of model answers here is an artificial intelligence based on the GPT-4 model.
  • Chatbot Arena is a project developed by LMSYS (Large Model Systems Organization), which is affiliated with the University of California, Berkeley and is known for creating the Vicuna-13B model. It should be noted that the leaderboard has not been updated since May 2023.
  • Programming Model Rating. This is a system designed to evaluate large language models capable of generating code. This initiative belongs to the Hugging Face company. It should be noted that the list was last updated in November 2023, which may lead to the presence of outdated information.

Obviously, the Hugging Face platform takes a leading position in this comparison. It offers a variety of benchmarks collected in a collection called The Big Benchmarks Collection. Users can easily customize the rating system to determine the most suitable model for a specific task, such as coding.

It's not just specialized companies that are active in this area. Some open-source communities are striving to develop a universal rating system that could synthesize the best qualities of all existing ratings. This resulted in the LLM-Leaderboard project, initiated by Ludwig Stumpp from Germany.

In most cases, open-source models show only a slight lag behind proprietary ones in terms of objective metrics. For example, in tasks related to question answering or text simplification, users often don't notice a clear difference between LLaMA 2 70B and ChatGPT. Moreover, the difference in scores between closed and open models is gradually decreasing.

Mikhail Salnikov

Read also:

Thirty powerful neural networks for various applications.

Overview of Well-Known Open Source Models

The core models upon which most open source LLMs are built are a few fundamental architectures. Let's take a closer look at the most significant of them.

Timeline of LLM development with a size of over 10 billion parameters. The names of models related to the open source segment are marked in yellow. Source: A Survey of Large Language Model, Wayne Xin Zhao, Kun Zhou, etc. Cornell University, 2023 Infographic by Maya Malgina for Skillbox Media

The LLaMA model was announced in February 2023. It was released in several variants, including 7, 13, 33, and 65 billion parameters. The most compact versions, consisting of 7 and 13 billion parameters, could be run on a single GPU, which caused quite a stir at the time of its release.

In July 2023, an updated version of the LLaMA model, LLaMA 2, created in partnership with Microsoft, was announced. This language model is available in three variants containing 7, 13, and 70 billion parameters.

Very soon after the appearance of LLaMA, its completely open source version, called OpenLLaMA, was released. This version has served as the foundation for a number of projects that are developing the model through experiments with architectural solutions and various approaches to fine-tuning and training.

"LLaMA 2 70B is a conditionally open-source model. The source code and weights are available, but commercial use is limited if the monthly audience exceeds 700 million users. This is perhaps the most popular model after closed solutions such as ChatGPT and Claude 2. It shows excellent results in all aspects."

Mikhail Salnikov

In 2023, a significant step forward in the widespread use of open-source neural networks was made by the development of LLaMA, which was used to create many models, such as Mistral, Zephyr, Alpaca, Phi-2, Qwen, Yi and others.

Victor Nosko

As part of this article, we asked experts to provide concise descriptions of the most interesting open-source language models from the LLaMA family that they highlight.

"The following open large language models deserve your attention:

  • Vicuna-13B, developed by the LMSYS organization, is one of the first models capable of working with the Russian language, and shows decent results in various tests.
  • Mistral is a model developed by a French startup of the same name that outperforms LLaMA 2 13B across all benchmarks. As of the end of September 2023, it was the most outstanding LLM, with 7 billion parameters.
  • Zephyr-7B is a modified version of Mistral that has been refined using the Direct Preference Optimization (DPO) method. This model demonstrates an impressive result, achieving a 90.6% victory over other neural networks in the AlpacaEval ranking.
  • OpenChat is an open-source library of language models. According to preliminary estimates, its performance is comparable to the quality of the ChatGPT version of March 2023 and even outperforms Elon Musk's chatbot, known as Grok. This library supports the Russian language. The OpenChat 7B model is based on Mistral 7B, but, unlike the latter, it successfully passes the famous "banana test." This test asks the language model: "I am in the kitchen and put a plate on a banana. Then I carried the plate into the bedroom. Where is the banana now?"
  • Xwin-LM-70B-V0.1 is a model built on LLaMA 2. The developers claim that it is the first model capable of outperforming GPT-4 in the AlpacaEval benchmark. However, it is worth noting that its size is quite impressive - it contains 70 billion parameters.

Viktor Nosko

The Mistral 7B model is notable for the fact that, with only 7 billion parameters, it demonstrates higher results compared to LLaMA 2, which has 13 billion parameters. This makes it suitable for use on most modern laptops.

I would also recommend taking a look at the Dolly model, developed by the American company Databricks. Although it is not related to LLaMA and is based on the EleutherAI Pythia family, it has its own advantages. One of the main advantages of this model is its complete openness, which allows it to be used for a wide variety of purposes.

Mikhail Salnikov

In Russia, work is actively underway to create our own language models specifically designed for processing the Russian language.

Among Russian developments, ruGPT-3.5, which became the basis for GigaChat from Sber, attracts particular attention. However, only a pre-trained version is publicly available, which implies the need for independent further training of the model.

The YaGPT 2 model from Yandex competes with the development from Sber, but it is not yet available to the general public. In 2022, the company introduced its predecessor, YaLM 100B, which was released under the Apache 2.0 license.

Among the Russian language models, it is worth noting Saiga 2, created by machine learning engineer Ilya Gusev. He describes his work as "a Russian chatbot based on LLaMA 2 and Mistral technologies."

The main domestic development is ruGPT-3.5, which was used to create GigaChat. Today, this is probably the most optimal option for working with the Russian language. In addition, there is YandexGPT, which also demonstrates excellent results in processing the Russian language. However, an open source version of this model is not yet available from its developers.

Mikhail Salnikov

One of the key areas in the study of large language models is the preparation of neural networks for creating program code. There are currently a number of well-known open-source models designed for this purpose:

  • StableCode is a product of StabilityAI, which also developed Stable Diffusion. It can write code in programming languages ​​such as Python, Java, Go, JavaScript, C, and C++.
  • StarCoder is a collection of models containing 15.5 billion parameters and trained in over 80 different programming languages.
  • SantaCoder is a set of models with 1.1 billion parameters developed based on the GPT-2 architecture. These models are trained to produce code in languages ​​such as Python, Java, and JavaScript.
  • Developed by Chinese experts, CodeGeeX and its updated version, CodeGeeX2, are powerful programming tools. The first version of this neural network, with 13 billion parameters, was trained in 20 programming languages. The second version, with 6 billion parameters, has significantly expanded its capabilities and now supports 100 languages. Among them, you can find such languages ​​as Python, Java, C++, C#, JavaScript, PHP and Go. These neural networks can be integrated as plugins into popular development environments such as Visual Studio Code, IntelliJ IDEA and Android Studio. Replit Code is a language model with a volume of 2.7 billion parameters, which was developed for automatic code completion. The training process was based on data covering 20 different programming languages, including Java, JavaScript, Python and PHP. CodeT5 and CodeT5+ are a series of models developed by the research division of Salesforce in the USA. The name of these models indicates that they are built on the architecture of the open T5 model. Various versions are available with the number of parameters ranging from 220 million to 16 billion, including intermediate variants with 770 million, 2 billion and 6 billion parameters. These models have the ability to program in languages ​​such as Ruby, JavaScript, Python, Java, PHP, C, C++, and C#.
  • CodeGen2 and CodeGen2.5 are another line of open-source language models developed by Salesforce Research. These models come in a variety of sizes, including 1, 3, 7, 13, and 16 billion parameters.
  • DeciCoder 1B is a relatively small model with 1 billion parameters that can complete user-supplied code snippets. It was trained on programming languages ​​such as Python, Java, and JavaScript. Its creators claim that this model demonstrates a 3.5x performance improvement, improved accuracy on the HumanEval test, and reduced memory consumption compared to popular code generation language models such as SantaCoder.
  • Code LLaMA is an improved version of LLaMA 2 that has been additionally trained to process program code. Models with 7, 13 and 34 billion parameters are available. It successfully works with programming languages ​​such as Python, C++, Java, PHP, C#, and TypeScript.

Read also:

Seven neural networks that will help programmers improve the quality and speed of writing code.

What else to read

This article is a broad overview of open-source language models. If you're looking to dive deeper into the world of both open and closed LLMs, we recommend checking out two papers hosted on arxiv.org:

  • An Overview of Large Language Models.
  • Harnessing the Potential of Large Language Models in Real-World Practice: A Review of ChatGPT and Other Solutions.

To stay up-to-date with the latest news and developments in the field of open LLMs, you can consult collections on GitHub and various other platforms.

  • A list of commercially available open language models (LLMs).
  • A collection of both open and closed LLMs.
  • A list of large language models.
  • A collection of open-source Chinese models.
  • A concise guide to choosing datasets for fine-tuning models.

Of course, we'll be sharing the latest news about progress in artificial intelligence in our Telegram channel. intelligence and events in the Open Source community.

Read also:

  • Ten misconceptions about open source software.
  • You are in the section dedicated to the results of 2023 in the field of artificial intelligence.
  • What awaits IT specialists in 2024: predictions and recommendations from leading neural network systems.

The court issued a decision prohibiting Meta Platforms Inc. from selling social networks Facebook and Instagram in Russia. This decision is based on accusations of extremist activity.