Virtual production: what it is, a simple explanation of the technology / ITech content

Contents:

• What is virtual production?
• What does virtual production really mean?
• Why is Unreal Engine used in virtual production?
• For which scenes is virtual production most suitable?
• Where else, besides film, can this technology be used?
• How to start a career in virtual production?
• What skills do you need to have to work with virtual production?
• What software do you need to know how to use?
• What future awaits virtual production?

In recent years, virtual production technology has become increasingly popular in content creation. It is gradually replacing the traditional green screen in filmmaking and other forms of media content. Virtual production allows for the integration of real and digital elements, significantly expanding the creative possibilities of creators. Using this technology provides a higher level of realism and flexibility during the filming process, allowing for the creation of impressive visual effects and improving the quality of the final product. Virtual production is becoming an essential tool for entertainment industry professionals every year. To gain a deeper understanding of virtual production technology and its potential applications, we spoke with Alexander Pasechny, Head of Development at eXtraOrdinary Virtual Production, a virtual production studio, XOVP, and Ekaterina Fakherdinova, Development Director at XOVP. Their experience and expertise allowed us to explore key aspects of virtual production, including its impact on the film and television industries and the opportunities it offers content creators. This article will provide you with up-to-date information and helpful tips on the topic. We'll cover key aspects in detail and provide recommendations to help you better understand and apply the knowledge gained. Read on to learn more.

• What is virtual production technology?
• Where, besides film, is it used?
• How to start a career in virtual production.

What is virtual production?

Virtual production is an innovative approach to filming where actors are in a studio surrounded by LED screens. These screens display the film's environment, creating the effect of presence in a realistic world. Using a tracking system, the environment dynamically changes depending on the actors' movements, giving scenes a natural feel and depth. This method significantly improves the production process, allowing for more efficient integration of visual effects and reducing post-production time. Virtual production is becoming increasingly popular in the film industry, opening up new possibilities for creative solutions and visual storytelling. On-screen visualization is achieved using the Unreal Engine, allowing for control over various parameters, such as lighting, time of day, and weather conditions. This is a key difference between virtual production and traditional green screen filming, where graphics are added during editing. As a result, the final image can only be imagined. In contrast, with virtual production, the result is visible directly on set in real time, significantly simplifying the creative process and allowing for immediate changes. Pre-production is a key stage in virtual production, no less important and labor-intensive than the main production and post-production. This stage requires not only script development and storyboard preparation, but also the creation of a virtual environment. Careful consideration of all aspects during pre-production ensures the high quality of the final product and minimizes risks during filming. By focusing on pre-production processes, you can significantly improve the efficiency of your entire team and achieve the desired results in creating captivating content.

The Star Wars series "The Mandalorian" represents one of the most significant applications of virtual production technology in recent years. This innovative technique allows for the creation of visually stunning scenes by combining live action with computer graphics, significantly increasing the quality and impact of the final product. "The Mandalorian" has become not only a landmark project within the franchise but also a significant milestone in the development of the film industry, demonstrating how virtual production is changing the approach to content creation.

The Mandalorian is not the first project to use this technology. However, it was in this series that it was implemented in its modern form, including the use of Unreal Engine, LED screens and camera tracking. This technology significantly improves visual effects and allows for the creation of more realistic backgrounds and environments, which makes the filming process more efficient and convenient.

And what exactly do we mean by virtual production?

Virtual production is the process of creating content that involves obtaining computer graphics in real time. This concept encompasses various aspects of production, including compositing, which is the process of combining footage, computer graphics, backgrounds, and other elements into a single, multi-layered edit. Virtual production significantly accelerates the visual effects process and improves the quality of the final product. Technologies used in virtual production are becoming increasingly popular in the film and game development industries, providing creators with new opportunities to realize their ideas.

The creators of the series "The Mandalorian" implemented a technology known as In-Camera VFX. This term means that special effects are created directly in the frame. Using Unreal Engine, the team generates images in real time, allowing the camera operator to see the final result during filming. This approach significantly improves the quality of visual effects and simplifies the on-set workflow.

Why use Unreal Engine in virtual production?

Unreal Engine is the leading game engine specifically designed for virtual production. To date, it remains the only tool that fully meets the requirements of this field. Attempts to implement virtual production on other platforms, such as Unity or Godot, are currently limited to enthusiasts with experience and in-depth knowledge. Unreal Engine offers the necessary tools and technologies that make the process of creating virtual content more efficient and accessible to industry professionals.

The main challenge lies not only in creating a single scene, but also in correctly rendering the image on screen. Achieving a specific distortion effect that will be perceived as believable by the viewer requires high-quality graphics and advanced technology. Currently, only Unreal Engine offers such capabilities.

While technologies continue to evolve thanks to Epic Games' own developments, the core toolset used in the XOVP project is the result of our own developments. This is why we do not impose strict requirements on new specialists regarding full knowledge of the Unreal Engine toolset, since it is largely proprietary. Our team is focused on implementing innovative solutions and adapting technologies to the unique needs of a project, which allows us to attract talented specialists with varying levels of experience.

Which scenes is best suited for virtual production?

Virtual production is ideal for scenes in which physical human presence is impossible, as well as for situations where the laws of physics do not matter. In a virtual environment broadcast on screens, gravity can be disabled in real time, allowing objects to float freely in the air, or the orientation of the space can be changed, for example, by turning the room upside down. This technology opens up new horizons for creative solutions in film and television, allowing directors and screenwriters to realize the most daring ideas without the limitations of the real world.

Last year, the film "We've Arrived!" was released, directed by Yuri Yarushnikov and created by XOVP studio. As part of the virtual production, the team had to film two scenes in one shift: one in an apartment and one on the roof of a house in wintry Karelia. With a traditional approach, the crew would have had to find a house with a suitable roof and a suitable apartment, which would have complicated the process, especially given that filming took place in September. Using modern virtual production technologies allowed us to significantly optimize the process and reduce the time spent searching for locations, which ultimately contributed to a high-quality result.

During one shift, we first shot the interior of the apartment, and then immediately moved on to filming on the roof, where the actor was suspended on cables. Thus, in one shift we were able to move from filming indoors to exterior scenes.

XOVP presented an interesting project – the children's popular science show "Gdektok", which was broadcast on the Okko platform. In this show, a researcher travels through time, demonstrating dinosaurs and extinct sea creatures to viewers. The project team was faced with the task of not only creating background animation with dinosaurs on LED screens behind the presenter, but also introducing elements of interaction with these creatures in the foreground, similar to augmented reality (AR) technology. The show's cameramen had the opportunity to observe the dinosaurs in preview, which allowed them to effectively compose shots and achieve high-quality visual results.

This year, the release of the large-scale war drama "Air" under the direction of Alexei German Jr. is expected. The film is being produced by Metrafilms and Amedia Production. In the film, air battles between aircraft were filmed in a documentary style, which adds realism. To achieve the desired effect, filming was carried out at a strictly defined time of year and in controlled conditions. This work promises to be a significant event in the world of cinema and attract the attention of viewers interested in historical events and military themes.

Creating realistic scenes for cinema is always a challenge, especially when it comes to shooting on location. Unfortunately, such scenes may not turn out to be as impressive as we would like. In our film, we developed a unique approach by integrating elements of real-world filming into traditional virtual production. We used filmed flyovers, displayed on a screen in real time, adding CG aircraft and visual effects created using Unreal Engine. For this shoot, we used a screen that is the largest in the CIS and previously considered the largest in Eurasia, significantly enhancing the quality and visual impact of our project.

The director of photography for this film is a professional pilot, which allowed him to immediately spot any inaccuracies in aircraft behavior. Therefore, we developed an algorithm that significantly simplified the process of creating plausible trajectories for the aircraft and their subsequent editing. All aircraft in the film accurately display the operation of flaps and ailerons, taking into account the laws of physics. It took us several months to implement these details.

We used similar technology for filming the combat scenes. For example, a real airplane is positioned on the screen in a tail-only position, creating the effect of viewing it from behind. A 3D model of the airplane is shown on the screen, firing virtual bullets toward the real airplane. At this moment, a pyrotechnic effect is activated on the real airplane's fuselage. This synthesis looks so realistic that sometimes even we have difficulty distinguishing the virtual airplane from the real one. This approach significantly enhances visual perception and creates a unique experience for viewers.

Where else, besides cinema, can this technology be applied?

Virtual production is widely used in areas requiring a high degree of immersion. It is ideal for educational projects, simulators for pilots and astronauts, online concerts, and various events with the participation of speakers. The use of virtual production allows speakers to immerse audiences in a unique atmosphere, and augmented reality clearly demonstrates key points and simplifies the perception of complex concepts through interactive elements. This approach significantly increases audience engagement and makes learning more effective and memorable.

Recently, the creation of cutscenes for video games, simulating traditional filming, has become increasingly popular. This process is part of virtual production. A camera operator can film an actor with a tennis ball, and the result will be a high-quality cutscene with detailed mocap and natural camera movements. This approach allows us to significantly increase the level of cinematography and realism of game stories, which, in turn, improves the player experience and makes games more attractive to a wider audience.

The fashion for cinematic cutscenes continues to develop and attracts the attention of video game fans. Modern technologies allow even a single developer to create content that can compete with AAA projects. This opens up new horizons for creativity and the quality of the presented material, and we can expect even more impressive achievements in this field in the future. Cinematic cutscenes are becoming an important element of the gameplay, enhancing immersion and the emotional component of games.

Mocap photography plays a key role in virtual production, especially when it comes to the implementation of virtual 3D characters. Digital extras are becoming a necessity for the modern film industry, as they provide significant budget savings. Numerous successful films have already featured stadiums with a capacity of 50,000–60,000 spectators. The number of such projects is expected to increase, underscoring the importance of using technology in film production.

Virtual production opens up new possibilities for reconstructing historical events and recreating buildings, providing a connection to the past. Not all black-and-white retro photographs and letters are of interest to young people, as their language is digital. The creation of digital museums and interactive exhibitions brings history to life, making it more accessible and appealing to a new generation. Such projects contribute to the preservation of cultural heritage and help understand historical events in a modern context.

Currently, we actively use neural networks, which allows us to create a set of tools in which modern technologies complement each other. This contributes to the development of new, progressive directions in filmmaking. The use of neural networks opens new horizons for creativity and improves the quality of produced content.

How to start a career in virtual production?

Working with virtual production is divided into two key areas: producing content for a virtual environment and filming. Most studios have two main specializations: virtual production operators and content developers. This division allows for efficient management of the processes of creating high-quality visual content and integrating it into the virtual environment, which in turn improves overall productivity and the quality of the final product.

Creating content in 3D is an excellent opportunity for artists and designers. 3D artists have the easiest time getting started, as their skills are ideally suited to developing visual elements. Level designers from the gaming industry can also find their niche, as they understand the principles of composition and visual perception. For example, in some cases, details that aren't in the line of sight can be ignored. For example, if a window high on a wall overlooks a distant landscape, the lack of a detailed mountain model can be disappointing, as the camera won't be able to position itself to view it. It's important to consider such aspects when creating content to ensure maximum engagement and realism in visual solutions. A virtual production operator should have a clear understanding of the Unreal Engine environment. Understanding the specifics of this platform will significantly facilitate their work. If a specialist already has experience developing virtual reality projects, they will be able to quickly adapt to the requirements of virtual production, as many principles and tools are similar. Understanding how Unreal Engine works opens up new possibilities for creating realistic virtual scenes and animations.

Classical camera training, especially in television, places a significant emphasis on learning how to use chroma key. Students will learn the basic techniques and methods necessary for effectively using this tool in video production. Tracking, an essential aspect of creating high-quality video content, will also be covered.

Unfortunately, in our experience, there are significantly fewer specialists from the gaming industry than film professionals. The XOVP team is dominated by retrained film professionals rather than game development experts. However, it's worth noting the positive aspects: television professionals are skilled in working under tight deadlines. In game development, creating a single location can take a month, while in our field this is impossible.

Currently, the industry is experiencing a significant personnel shortage, as most specialists come from related fields and require retraining. We are actively working to solve this problem by initiating educational programs for leading educational institutions that train filmmakers and media specialists. We recently began collaborating with schoolchildren, introducing them to new technologies that they quickly grasp. I'm confident that when this younger generation grows up and faces career choices, high-quality education in virtual manufacturing will be available in Russia. This will help create qualified personnel for the growing industry and improve its future development.

What skills do you need to work with virtual production?

A good specialist in the field of virtual production is a versatile professional who combines many skills. The ability to work with computer graphics and camera work are only part of the necessary competencies. It is important to deeply understand the cause-and-effect relationships in various aspects of the work in order to effectively solve emerging problems. In addition, the specialist must be proficient in PC use and have good knowledge of information technology. This allows him to adapt to rapidly changing conditions and implement innovative solutions in the virtual production process.

In our field, content managers undergo an internship in the control room as operators to better understand the processes and avoid errors in their work. Content optimization is a critical aspect, and we achieve a high level of its implementation. Our materials are optimized down to the last detail, achieving results comparable to AAA games. At the same time, the content must be aesthetically pleasing, as it is often presented in close-up. Stress resistance is a must for success in video production. It's also important to be able to quickly navigate situations and improvise. A filming shift involves at least 50 people, and no one cares why the content isn't ready yet. The shift is scheduled for a specific day and time, and being late is unacceptable. Postponing the shoot entails significant financial losses. Therefore, the ability to adapt to change and keep cool in stressful situations plays a key role in the successful implementation of projects in this area.

What software do I need to know how to use?

Extensive knowledge of software for editing 3D models, video editing, color correction, and animation significantly facilitates the process of mastering specialized skills. The more experience a person has with such tools, the faster they can adapt to new technologies and methods in the field of digital creativity. This creates a solid foundation for further development and enhancement of professional competencies in the field of graphics and animation.

To successfully master 3D space, it is necessary to develop spatial thinking skills and an understanding of its structure. An important aspect is practice, as well as experience in the development and use of virtual reality (VR) technologies, which significantly improves the perception of and interaction with three-dimensional objects. Developing in VR allows for a deeper understanding of the principles of 3D space and the creation of higher-quality and more effective solutions.

What does the future hold for virtual production?

Projects using virtual production technologies are already actively developing in Russia, and a significant increase in their number can be expected in the coming years. This area is becoming increasingly popular, providing new opportunities for content creation and improved visual effects. In the future, virtual production will play a key role in the development of the film industry and other media areas, which will certainly affect the quality and variety of produced content.

Virtual production, like any new direction, goes through several stages of development: first, overcoming, then acceptance, and ultimately - falling in love with the product. We are at the stage when professionals are beginning to experience a real attachment to this technology. Working with skeptics is especially revealing. Unlike young representatives of the film and media industry, they are experienced professionals. They often find it more difficult to embrace innovations. However, once immersed in virtual production, even they are amazed by the capabilities and potential of this technology.

Good virtual production is based on quality clients. At the beginning of our journey, we had to spend a lot of time explaining that the screens weren't just videos and how this format worked. Today, we're seeing more informed clients, indicating a growing interest in virtual production. We've also built a strong client base, and we speak the same language with them. This allows us to more effectively implement projects and achieve high results.

Recently, we've seen a growing interest in unique cases created specifically for virtual production. This is leading to a shift in the creative process: creators are now beginning to think in the context of special effects, viewing their ideas not just as individual stories or shots, but as holistic, comprehensive solutions. In this regard, there is a need for new visual approaches that will help implement ideas at a higher level.