Volumetric Vol. 2: Keyword Explanations for Volumetric Videos

Free viewpoint video

3D data generated by volumetric capture is sometimes called free-viewpoint video because it can be rendered as 2D video from free viewpoints of up, down, left, right, front and back of the subject in post-processing after shooting. A long time ago, Professor Takeo Kanade of Carnegie Mellon University in the United States and others developed a technology called EyeVision, which became a hot topic when it was used in “Super Bowl 35” held in January 2001.

3D scan

Photogrammetry

Photogrammetry, also called photogrammetry, is a method of generating a 3D model from a large number of photographs (still images). Photographs are taken with a general digital camera, and 3D data can be obtained by processing the photographic data with software dedicated to photogrammetry. Since the hardware used is a commercially available camera or personal computer, it is also a method that allows you to easily create a 3D model. It is assumed that the subject is stationary.

Motion capture

While volumetric capture captures the subject’s movement and texture (actual) at the same time, the method of capturing only the movement of the subject, especially the human, is called motion capture. With a head-mounted display (HMD) for VR, the movement and tilt of the HMD and the controller held in the hand can be taken, so that the self (the head and hands of the avatar) reflected on the screen can be changed according to the physical movements of the HMD and the controller. Moving is an example of motion capture.

Light Field Camera

A light field camera is literally a camera that can record a light field. A normal camera sensor records the intensity and color of the light that enters the sensor, but it cannot record the angle at which the light enters. By recording that angle as well, it is possible to change the focus of the photographed photograph in post-processing, and it is possible to change the lighting and texture.

Depth camera

A device that can measure the distance from the camera to the subject is called a depth camera. The table below shows examples of depth cameras on the market as of 2020, and you can see that there are various methods for measuring depth. In the 3D scanning described above, not only dedicated 3D scanners but also these depth cameras are sometimes used.

■ Examples of depth cameras (depth sensors) used for 3D scanning

Visual Hull method

One of the basic data processing algorithms used in volumetric capture is the visual volume intersection method. In the visual volume intersection method, multiple 2D silhouettes are photographed by synchronized cameras, a visual volume is created with the silhouette as a cross section with the camera position as the vertex, and the silhouette is projected back into 3D space. This is a technique for restoring dimensional shapes. Also called Shape from Silhouette.

Point cloud

One of the data formats of volumetric video is point cloud. A point cloud is a point that is the minimum unit of data acquired by a camera or sensor. It gathers and forms a group. This 3D point cloud data continues in the time direction like a movie.

Mesh

Another data format for volumetric video is a point-to-point mesh. With point cloud data, there is space between points as you zoom in, but with mesh data, each point becomes a vertex and is connected by sides and faces. There are several methods of converting point cloud data to mesh data, and in the process noise removal, smoothing, interpolation, super-resolution creation, etc. are sometimes performed, and high image quality and high precision of data are realized. .

Relighting

The texture of volumetric video often reflects the lighting conditions at the time of shooting, and if the 3D data is used in a lighting environment completely different from that at the time of shooting, the brightness, color, shading, etc. of the texture will be reproduced. not. Rewriting technology solves this problem. In 3D data that can be relighted, reflection coefficients are added to the texture data, resulting in volumetric video data with higher reproducibility.

Originally Written in Japanese by Takayuki Aoki