Photogrammetry for 3D Environment Artists: Five Uses

Contents:

• Briefly about the speaker and photogrammetry in general
• Photogrammetry of monolithic locations (traditional method)
• Photogrammetry of monolithic locations using the latest technologies
• Creating modular assets
• Scanning people to obtain NPC models
• PBR materials
• Lighting
• Results

Photogrammetry has long established itself as an important tool in various industries, and 3D scanning of real objects has become commonplace. However, many are still not exploiting the full potential of this technology. At the Blender Conference 2023, Gleb Alexandrov presented five effective ways to use photogrammetry in creating 3D environments. Skillbox Media's Gamedev editorial team shares the highlights of his presentation.

Briefly about the speaker and photogrammetry in general

3D artist and video blogger from Belarus. On his YouTube channel, he publishes Blender tutorials, training colleagues and helping them develop their skills. Together with 3D modeler and environment artist Eydie Burrows, he founded the creative lab Creative Shrimp, offering training courses for anyone interested in mastering 3D design and modeling.

Photogrammetry is a 3D scanning method that can now be performed even with a smartphone. This process involves several key steps. First, the object is photographed from different angles, ensuring that each shot is clear and well-lit. After the photo shoot, the photos are uploaded to a specialized application, which analyzes the images and creates a point cloud. This point cloud is used to create a dense polygonal mesh, to which a texture is then applied. For more information on photogrammetry and its applications, please check out our material.

Optimizing text for SEO requires the use of keywords and phrases that will help improve the visibility of the page in search engines. At the same time, it is important to preserve the meaning and structure of the original text.

When you read, you not only gain information but also create a connection with the material. Try to pay attention to details that may be useful for your understanding of the topic. Reading also develops critical thinking and helps you form your own views.

We recommend taking the time to study a variety of sources of information to gain a more complete understanding of the subject at hand. This will help you make informed conclusions and broaden your horizons.

Reading also helps develop your vocabulary and improve your writing skills. The more you read, the easier it will be for you to express your thoughts and ideas. This is especially important in today's world, where information plays a key role in communication.

Dedicate time to reading to enrich your knowledge and skills. This will help you not only in your personal life but also in your professional work.

Photogrammetry is a method that allows you to create three-dimensional models of objects based on photographs. This process involves using multiple images taken from different angles to reconstruct the shape and texture of objects. In the gaming industry, photogrammetry is widely used to create high-quality game assets.

Photogrammetry allows developers to create realistic 3D models that significantly improve the visual quality of games. This method saves time and resources on modeling, as it uses data from photographs instead of manually creating objects.

It is important to remember that for the successful use of photogrammetry, it is necessary to consider lighting, shooting angle, and photo quality. Proper image preparation and processing help achieve accurate and detailed models. Using photogrammetry in game development not only speeds up the process but also increases player immersion in the virtual world thanks to realistic visuals.

Therefore, photogrammetry is becoming an essential tool in the developers' arsenal, allowing them to create high-quality and detailed game assets with minimal time.

The Creative Shrimp team regularly uses photogrammetry in their creative projects, especially when creating realistic 3D scenes. In his talk, Gleb goes into detail about the different types of data produced using this technology, including:

• monolithic scenes, either traditionally or cutting-edge scanned;
• asset sets;
• NPCs;
• PBR materials;
• lighting.

Let's take a closer look at each of these segments.

Photogrammetry of Monolithic Locations (Traditional Method)

Full scanning of a location or room allows you to create a monolithic 3D scene that represents a single, integral object. Although this approach may seem excessive, it is ideal for concepts with a seamless environment. Gleb shared an example from his practice: his team visited an abandoned warehouse to photograph objects that would later be used in a set of assets. Among the found items was an abandoned Volga, which had been in the same place for several decades. This scanning method not only preserves details of the environment, but also allows you to create more realistic and immersive 3D models for various projects.

The cameraman suggested launching a drone to film the entire location in isometric perspective. The proposal was approved, and the team used a series of aerial photographs to recreate the scene in 3D using RealityCapture software. The collage below shows that the final result, created using lighting sources in Blender, visually resembles the game's location design. This approach allows for high detail and realism, making the 3D model more appealing to users and enhancing the experience of the location during gameplay.

Of course, this approach is not used in game development due to optimization considerations. Nevertheless, monolithic locations look impressive, especially with the addition of atmospheric lighting and visual effects, as well as character assets. Such scenes can serve as an excellent basis for creating CG animations, which opens up new possibilities for visualization and storytelling in gaming projects.

3D scanning of large-scale scenes allows you to effectively recreate unique Historical sites in a single visit. The footage shown here shows a 3D reconstruction of the ruins of the Bernardine Monastery in Brest. This technology not only preserves cultural heritage but also provides the opportunity to explore long-lost architectural monuments in high detail. 3D modeling of such objects contributes to their preservation, and also makes it possible to create virtual tours for educational purposes.

This approach to photogrammetry is not universal due to the peculiarities of the original polygonal mesh. A large number of polygons negatively affects performance, which should be taken into account when working with 3D models. In addition, the model files of monolithic objects are significant in size, since each polygon and texture is unique. However, even low-resolution models can be useful for experiments in virtual cinematography. The example below shows how Blender rendering, with lighting and simulations, significantly transformed a 3D scan of the Tholos do Escoural from Sketchfab. Using such technologies allows you to create impressive visual effects and improve the quality of model presentation, which makes them valuable for various projects in the field of animation and visualization.

Ultra-realism and uniqueness are key advantages of modern content. Ultra-realism allows you to create images and objects that are as close to reality as possible, making them attractive and impressive to viewers. Uniqueness guarantees originality and distinguishes content from competitors, which is especially important in a saturated market. These qualities not only attract audiences but also increase brand trust. As a result, using ultra-realistic and unique content can significantly increase the visibility and success of your project online.

The disadvantages of using 3D scanning include limited flexibility in the technology's application, which can hinder adaptation to various industries. It is also worth noting that the 3D scanning process can reduce overall productivity, as it requires significant time and resources. In addition, finding interesting locations worthy of 3D scanning is difficult, which can limit the ability to create high-quality content.

Photogrammetry of Monolithic Locations Using the Latest Technologies

In recent years, there has been significant progress in the field of photogrammetry, due to the introduction of advanced technologies. Among them, it is worth mentioning Neural Radiance Fields (NeRF) and Gaussian Splatting, which offer new approaches to creating three-dimensional models. These methods provide greater accuracy and realism in image processing, which opens up new possibilities for the application of photogrammetry in various fields, such as virtual reality, architectural design, and the film industry. The development of technologies such as NeRF and Gaussian Splatting contributes to improved visualization quality and expands the boundaries of possibilities in the field of 3D modeling.

Materials obtained as a result of scanning have dynamic properties, including reflection and refraction of light on the surface. Unlike traditional photogrammetry methods, modern technologies can capture the smallest details, such as hair texture and leaves. This opens up new possibilities for creating high-quality 3D models and improves the accuracy of object visualization in various fields, including architecture, design, and virtual reality.

Today, modern technologies have become accessible to every user. In particular, the INRIA research institute has released the open-source 3D Gaussian Splatting application. Anyone can test it by following the instructions published on the GitHub platform. This application offers unique capabilities for working with 3D graphics and can be used for both scientific and educational purposes.

The visualization developed by Gleb and his team using this program demonstrates high quality and attention to detail. Only 30 shots were taken to create the boat scene, highlighting the team's efficiency. The second scene was completed just a few days before leaving for the conference, demonstrating a fast and professional approach to completing tasks. These examples illustrate not only the creative process, but also the team's ability to work under time constraints.

Gleb emphasizes that similar technologies would have been impossible to implement just a year ago, but they are rapidly developing. Today, apps like Luma AI and Polycam are already implementing these innovations. These apps are available for download on smartphones, making them accessible to a wide audience.

While NeRF and 3D Gaussian Splatting technologies represent a significant step forward, there are currently no clear methods for working with content created with them. The main problem is that these technologies do not support standard polygonal meshes, making them incompatible with the usual workflows of 3D artists. This creates challenges for integrating new methods into existing pipelines and requires the development of new approaches for the effective use of these technologies in 3D modeling.

The advantages of this technology include the ability to capture objects with fine structure, such as fur and branches, as well as the accurate reproduction of the properties of various materials. This enables a high level of detail and realism in modeling, which is especially important for creating high-quality content and visualizations.

The disadvantages of this solution include a lack of flexibility in application and limited editing capabilities. These aspects can significantly limit its use in various projects requiring adaptation and a customized approach. The lack of flexibility hinders integration with other tools, and limited editing capabilities can hinder the implementation of unique user requirements.

Creating Modular Assets

Photogrammetry is a versatile process that finds wide application in various fields, including video game development. Assets obtained through 3D scanning are actively used by developers to improve graphics and the realism of gameplay. Furthermore, ready-made prop sets created using photogrammetry are available on many marketplaces, opening up opportunities for additional income. Integrating such assets into projects not only saves development time but also improves the quality of the final product.

To create a high-quality set of 3D objects, first of all, you need to select a suitable location with a large number of objects corresponding to the given theme. Then, you need to allocate time for a separate scanning session for the most interesting objects. The resulting 3D source files should be processed using specialized 3D modeling applications. As a result, a full-fledged library of props is formed, ready for integration into any project, which significantly speeds up the development process and improves the quality of the final product.

With this approach, each scanned object should Several optimization steps are required, including polygon reduction, detail transfer from the high-poly mesh to the low-poly mesh, and texture baking. However, this process can be challenging for large-scale structures. The structure of the original mesh is often chaotic, making it difficult to extract individual elements from a monolithic piece. Optimization is a key step in improving performance and enhancing the visual quality of models, so it's important to consider these aspects when working with large-scale objects. If the subject is small, the artist may want to take it home or to the studio for scanning in more comfortable conditions. In this case, additional props, such as a rotating stand, can be helpful, allowing for efficient capture of the object from all angles. This approach ensures high-quality scanning and allows for greater attention to detail, which is especially important in artistic practice. Using specialized equipment helps create more accurate and attractive images, which improves the results of work and increases interest in the final product.

Creating modular assets requires significant effort, but this effort is sure to bear fruit. Once development is complete, all that remains is to place the finished objects in the scene and create a new environment. This new space may even be more organic than the original location used for scanning. Modular assets significantly speed up the process of creating game worlds, while ensuring a high level of detail and realism. Thus, by investing time in the creation of high-quality modular elements, developers gain the opportunity to create unique and diverse game locations.

The advantages of versatility and industry standards make this product indispensable in its category. Versatility allows for its application in various conditions and areas, significantly expanding its functionality. Industry standards guarantee high quality and reliability, meeting modern requirements and user expectations. Choosing this solution ensures stability and confidence in its use.

The disadvantages of virtual shooting include the need for careful planning, which includes choosing a suitable location and the time spent on the shooting process itself. In addition, virtual shooting requires post-processing, which includes such stages as mesh cleaning, retopology, and texture baking. These processes may require additional resources and skills, which should be taken into account when organizing the project.

Scanning People for NPC Models

In this category of assets, the main subject of the scan is a living person. The resulting 3D model can be used as an NPC extra in games and other visual projects. It is not necessary to scan a person in a standard T-pose or A-pose, which opens up more opportunities for creativity and variety in character creation. This approach allows you to create unique and realistic models that fit better into the context of a game or animation.

This technique was popularized by renowned VFX artist Jan Hubert. The method involves placing a person in a predetermined pose appropriate to the scene and then scanning them as a regular object. Once the 3D model is created, armature bones are added to provide head and torso movement. This approach allows for realistic animation and captures all the nuances of character behavior, significantly improving the quality of visual content.

It should be emphasized that this rig is designed exclusively for basic movements, such as head bobbing and torso rotation, and is not suitable for complex animation. The result is a character that can be effectively integrated into the environment. In terms of creating extras, such NPCs can look quite realistic, which helps improve the overall perception of the gaming environment.

Characters created using this method can be supplemented with baked animations, for example, from the Mixamo library, or animations recorded using Rokoko Vision. However, it is worth noting that this method is a more accessible option for photogrammetry, and the quality of the resulting models will be inferior to the results achieved using professional equipment. Therefore, for best results, it is recommended to minimize the movements of the created 3D characters.

It is important to consider that a person is a living being, and during photogrammetry, they can blink, breathe, and even fidget if they stand in one position for a long time. These natural movements can lead to visible defects in the final 3D scan, which may require re-shooting. To avoid such situations, it is recommended to inform models in advance of the need to remain still and use appropriate methods to minimize movement during the session. This will ensure higher quality and more accurate results during the 3D scanning process.

The benefits of quickly creating unique NPCs for crowd scenes include increased player interest and engagement. Unique, unwritten characters enrich the gameplay experience, creating a more vibrant and varied atmosphere. This allows developers to effectively populate game worlds, avoiding repetition and formulaic features. Rapid creation of such NPCs saves time and resources and simplifies the development process, allowing developers to focus on more meaningful aspects of the game. The integration of unique NPCs helps improve the user experience, which in turn can increase the popularity and success of the game.

The disadvantages of this method include the lack of flexibility in its application, as well as the fact that the process itself is more complex compared to scanning inanimate objects. This can make it difficult to use in a variety of situations that require adaptability and simplicity.

PBR Materials

PBR materials are created using a process similar to scanning asset sets. First, an expert selects a photographic surface that needs to be converted into a material. Then, they photograph section after section, carefully ensuring that the images in the pictures overlap each other and are not joined end to end. This approach ensures a more accurate capture of texture and surface detail, which in turn improves the quality of the final PBR material. This method allows for realistic material display in 3D graphics, which is especially important for the creation of high-quality visualizations and games.

The photographs are then loaded into a photogrammetry application such as Meshroom or RealityCapture, which generates a dense polygonal mesh based on the captured data. The resulting 3D surface is converted to a plane, creating a set of basic texture maps, including a height map. The content is then transferred to an image editor, where the textures are processed to achieve a seamless effect. The result is a realistic PBR material suitable for use in various 3D applications and games.

Height mapping obtained from a scanned surface can be combined with various PBR materials to achieve interesting visual effects. However, this method is less versatile than node manipulation, for example, in Substance 3D Designer. The limited resolution of the original texture and the fixed surface pattern can limit your creativity. However, using height maps is a great way to add uniqueness and depth to your content, enriching it with detail and texture.

The advantages of this technology include a realistic height map, which provides Accurate terrain mapping. Furthermore, the shooting process is simple and convenient, allowing for quick, high-quality results.

The disadvantages of using this system include the lack of procedural adjustments. This limitation can reduce the system's flexibility and adaptability, making it difficult to customize to meet specific user requirements.

Lighting

After a scanning session, the object's lighting data can be either deleted or retained. Deleting data is typically used in a standard pipeline when the artist intends to independently adjust the lighting in a future project. In such cases, it is recommended to photograph the object in cloudy weather or under indirect lighting. If shadows and reflections cannot be avoided, several methods can be used to minimize them.

• processing content in specialized programs, such as Texture De-Lighter from Agisoft;
• shooting with flash;
• applying cross-polarized filters to the original image.

In practice, there is sometimes a need to capture a high dynamic range of brightness, known as HDR. This is especially relevant when creating interactive scenes for Virtual Production. To obtain an HDRI image, you need to take each photo with different exposure options. After this, the resulting frames are combined into a single image using specialized utilities. Detailed information on this process can be found in the article on the Poly Haven portal.

Virtual The baked-light environment for creating cinematic scenes offers unique opportunities for achieving natural lighting. This approach ensures a harmonious perception of light, regardless of changes in exposure. This allows us to achieve high image quality and realism in visual effects, which is especially important for the professional film industry.

The benefits of HDR lighting include its ability to provide a higher dynamic range, which results in a more realistic image. With HDR lighting, you can enjoy vibrant colors and deep shadows, which is especially important for watching movies and video games. HDR technology improves contrast and color rendition, making the image richer and more detailed. This makes HDR lighting an ideal choice for home theaters and professional use in the visual media industry.

Disadvantages: the process requires significant labor costs; rarely implemented in standard pipelines.

Bottom Line

Photogrammetry finds wide application in personal, educational, and professional projects. Although the results obtained with its help may vary, the technology itself remains in demand across various industries. Professionals such as 3D artists, level designers, engineers, and scientists can benefit greatly from it. Photogrammetry allows us to digitize the surrounding world and transform it into 3D graphics, preserving the beauty and detail of reality. The use of this technology opens up new horizons in visualization and modeling, making it an essential tool in modern digital art and science.