So, What is Volumetric Capture?
Volumetric capture involves using multiple cameras and sensors to film a subject, creating a full volume recording of the subject, rather than a flat 2D image. Through postproduction, this captured volume data becomes a volumetric video, which is then viewable from any angle, with realistic depth, color and lighting, on any compatible platform, including mobile devices.The uses for volumetric video range from creating #realistic holograms to highlighting #3D environments to putting together an augmented reality (AR) project and more. It can be used for character creation in #VFX and #games, #immersive simulations, #360walkthroughs, its application extends in any atmosphere where viewers need to experience a product/ person or service in 360, e.g., potential shoppers can look at what clothes may look like in motion on a real person, from any angle.
It can put you directly on the stage with your favourite band and allow you to control the view during the biggest sporting events in the world.
And best of all, it can be accomplished with existing technology. Using dedicated streaming software with some of our own workflows at Xeno Reality, full volumetric videos without a cap on length can be adapted to work even under unstable bandwidths. while 5G is making it easier than ever to consume large files, the growth of internet and its ability to be accessed across the world will only make Volume Captures more inclusive in narrative storyboards than ever before. And while most volumetric videos are created on a specially configured capture stage, today a consumer’s #LiDAR camera can also do the same, albeit with lesser precision and reduced quality. In short, volumetric video is a new way for people to create and consume videos.
So Why Choose Volumetric Content?
At its core, volumetric video offers viewers a more immersive experience. Instead of going to a theatre for a concert or a play, volumetric video brings the performance to you, wherever you are, by turning the actors or musicians into photoreal holograms who you can sit next to or watch up-close and personal. The days of speaking to a bunch of talking heads on a video conference screen is over, you could stand in front of a full-body digital avatar of the person you're chatting with and walk around them, while they view a photoreal digital double of you. You could learn new dance moves or practice a golf swing, while a photorealistic instructor gives you feedback. The possibilities are endless, and there is serious money involved. By 2026, the volumetric video market is expected to approach the USD 5 Billion, and those are likely conservative estimates. A few of the higher profile uses so far have been:
Music: One common use for volumetric video in music is for promotional videos. Artists ranging from RadioHead to Eminem have used the tech for a variety of purposes to represent themselves in volumetric form. Imogen Heap created an immersive music performance for VR headsets. As part of a promotion for Samsung phones, members of South Korea's popular band BTS were recreated in photorealistic virtual form, allowing fans to pose for a selfie with their volumetric avatars. More recently, BTS also teamed up with Coldplay for a music video (and subsequent TV performances) of the latter's song "My Universe,” where volumetric avatars of both bands join in each other's performances.
Virtual Production and Sports: Entire landscapes/ sports arenas/ venues have been captured using volumetric methods, then added to a game engine. These digitized backgrounds are then be displayed on LED screens set against a stage, giving performers the ability to immerse themselves in the virtual worlds. Sporting events have also benefited from volumetric video. You see it each time a live NFL game shows a replay where the camera angle rotates around the action. Viewers now have the opportunity to choose the angle they want, all in real time and enjoy 360 immersive content in real time
Fashion: Several clothing brands are using volumetric capture to highlight collections. The clothes are worn and demonstrated by models in interactive volumetric videos accessible on browsers or via AR on mobile devices, allowing potential purchasers to truly see how the garments drape and move with the wearer — far more than a static, 2D photo ever could. Thus, creating a more likeness for buyers to be able to visualise clothing fits which are difficult on online mediums to decide purchases. Workflows involving Volumetric Capture are now being integrated with AI to create amazing narrative experiences that can be personalised for each user thereby allowing more users to use retail solutions globally from anywhere and taking brands beyond geography. E.g., Google Shopping’s new Generative AI tool.
Documentaries: Volumetric video is also being deployed for more serious purposes, documentary filmmakers are now creating immersive stories that capture the viewer’s attention, some of them highlighting role play and gamification to bring out immersive and interactive participation from viewers.
Content for these types of experiences is being created by production companies, artists, and developers all over the world in volumetric capture studios, and in more and more cases, using consumer-grade LiDAR cameras.
How to Create to Volumetric Videos
Volumetric video can be created using one of several volumetric capture techniques, including the combination of existing technologies. A few of the most common ways to engage in volumetric capture are:
Photogrammetry: Multiple photos of a subject are taken from various angles and stitched together by computational algorithms to generate 3D models.
LiDAR: A relatively new technique, LiDAR (an acronym for laser imaging, detection and ranging) is a method of laser scanning used for measuring and recording landscapes or buildings to create 3D scans and depth maps. LiDAR produces 3D point clouds, which account for every point in space, in a scene or object. Instead of pixels in 2D scenes, such points are represented by voxels, which don't just possess x and y coordinates, but also z (depth information).
Motion and Performance Capture: Markers are placed on the subject, and sensors (including precision cameras) record the movement of those markers in 3D. That data is then remapped into CG models.
360° Video Cameras: Multiple, synced arrays of cameras capture 360° video, for use with 3D glasses and VR headsets.
Light Field Cameras: Cameras that measure the intensity of light in a scene, along with the precise direction that the light is traveling. Using micro-lenses, users can capture an image, then adjust lighting, perspective, and focal points in post-production.
Now, point-accurate, photorealistic holograms — built up using depth, motion, and light measurements — can exist on almost any monitor, VR headset or mobile device in a live space or rendered environments (or through dedicated devices like the Looking Glass). Users can have an immersive experience through volumetric scenes, or encounter a volumetric hologram in real time. They can change the lighting or focus, or move around in any direction. Once the hologram has been created, exploration is limitless.
The Rise of Volumetric Entertainment
Volumetric video is helping to redefine several forms of entertainment and promises to speed up production schedules. One of the biggest proponents for using volumetric video in film is Diego Prilusky, the former head of Intel Studios. Prilusky recently oversaw the creation of a massive 10,000-square foot custom-built domed studio, created to explore the possibilities of volumetric capture. He even hosted a TED talk discussing the potential of the technology by describing a recent use of the studio.Intel Studios recruited movie set designers to create a desert town, complete with sand, trees, and appropriate scenery. For the performers on stage, it was the same as a traditional set, but instead of using a handful of large cameras, hundreds of specialized cameras recorded every point within the dome. With the filming complete and the enormous amount of data generated compressed, a volumetric video was produced of the sequence. Viewers could travel through the scene and completely explore the action from any direction and any viewpoint — even that of the horse — all in real time. The technology was put to the test in a partnership with Paramount Pictures to explore immersive media in a Hollywood movie production. Together with legendary director Randal Kleiser (The Blue Lagoon, Flight of the Navigator), Prilusky's Intel Studios team reimagined Kleiser’s iconic 1978 movie Grease as a multi-user, location-based experience, ultimately recording a short song and dance sequence using volumetric capture. During his TED presentation and using just an iPad with an AR view, Prilusky was able to bring the Grease XR performers onto the stage in hologram form, and walk among them as they sang and danced to "You're The One That I Want."Although the potential for immersive storytelling is clear, volumetric video is for more than just film or even entertainment. Along with film stages, sporting events and concerts, virtual sets are becoming common in live TV and weather studios as well. Real-time game engines from pioneers like Epic Games (Unreal Engine) and Unity Technologies have been critical to the rise of volumetric video, delivering interactivity and virtual studios to the public. Along with a democratization in game development, the rise of publicly-accessible game engines — along with volumetric video — have led to the rise of new techniques like virtual production which truly has changed workflows of production using gamified content creation and more and more studios and content creators are using volumetric capture for enhanced immersive experiences.
Streaming Volumetric Video
While technology around volumetric video has evolved, but eventually the conquest is to be able to stream photorealistic volumetric content to any device. Today, being able to watch a sporting event and have 360-degree replays of a goal or a touchdown (or a penalty if you want to get mad at the refs). You could watch a player from multiple perspectives, slow the action down and analyse any part of a replay, then return to the live game, or watch it all on a mobile device with the game on your TV. It’s all possible right now using existing technology, but the problem is the sheer amount of data involved. Raw volumetric video files can be massive. The processing power required to live stream an event like that and in-full to consumers is massive and requires dedicated hardware and this is still evolving. Also there’s the matter of streaming the data to users, with adaptive bitrate streaming solution, we are now able to broadcast volumetric video files and 360 video files. The content works on 4G without issue, but the new faster speeds of 5G are ideal for the larger files. Given the potential for more immersive experiences, and the ever-increasing competition in nearly every field that can benefit from multimedia, the growth of volumetric video is all but assured. All that’s needed now are the story tellers and experience creators that will help usher in the next generation of video. And they are already here.
If you wish to know how you can use Volume capture to create effective #intuitive, #immersive, #interactive #narrative stories around your brands and experiences, talk to us on info@xenoreality.me
