JP7320400B2 - VIDEO PRODUCTION PROCESSING DEVICE AND PROGRAM THEREOF - Google Patents

Description

The present invention relates to a video presentation processing device and program for adding visual guidance effects to VR content.

Conventionally, two-dimensional displays have been the mainstream display devices for viewing video content. In recent years, the use of display devices such as head-mounted displays and smart glasses capable of viewing virtual reality (VR) and augmented reality (AR) content is also expanding (for example, Non-Patent Documents 1 and 2). . Hereinafter, VR and AR content will be referred to as VR content.

Masaru Yamaguchi, Significance of VR distribution of 360° video by public broadcasting - Broadcasting research and surveys for 2020 and beyond, October 2017, P. 90-97 Naosuke Kamiya, NHK Program Technology Exhibition, FDI, July 2018, P. 28-29

However, in many cases, two-dimensional video content and VR content are produced separately according to the type of display device, even though the contents are the same. For this reason, when creating VR content, the know-how of the creators of two-dimensional video content (for example, broadcasting stations and production companies) cannot be fully utilized. For example, it is difficult to perform visual storytelling by restricting the line of sight and field of view by camera work during shooting, which is a video production method used in the production of 2D video content, and by “guiding the line of sight” without telops, narration, or dialogue. be.

Accordingly, an object of the present invention is to provide a video presentation processing apparatus and a program thereof that can efficiently produce VR content having a visual guidance effect that guides the subject to the VR content based on the movement characteristics of the naked eye.

In view of the above-described problems, a video effect processing apparatus according to the present invention generates volumetric capture information consisting of the three-dimensional shape and surface pattern of a subject by volumetric capture that digitizes the movement of the subject into three-dimensional data. , a video production processing device that uses volumetric capture information and camera parameters of a camera that shoots a subject to add a visual guidance effect that guides the VR content of the subject according to the movement characteristics of the naked eye, The configuration includes volumetric capture means, visual volume calculation means, three-dimensional saliency map generation means, object recognition means, importance setting means, importance multiplication means, and visual guidance means.

According to such a configuration, the volumetric capture means generates VR content and the volumetric capture information by volumetric capture.
The visual volume calculation means calculates a quadrangular pyramid-shaped area represented by the lens principal point, which is the center coordinate of the lens of the imaging camera, and the imaging angle of view of the imaging camera, which is included in the camera parameters, as the imaging range of the imaging camera. Calculate as volume. The camera parameters reflect the camera work of the cameraman, which is one of the know-how of the creator of the two-dimensional video content. Therefore, the visual volume calculated from the camera parameters includes the subject that the creator of the two-dimensional video content wishes to guide the line of sight.

The 3D saliency map generating means generates a 3D saliency map of the 3D shape and surface pattern of the subject contained in the visual volume based on the volumetric capture information. This three-dimensional saliency map contains information obtained by converting the three-dimensional shape of the object into a depth image on a plane, and information obtained by quantifying the gradient and color of the surface pattern.

The subject recognition means recognizes the type of subject included in the visual volume by machine learning. The importance setting means associates and sets the type of the subject recognized by the subject recognition means and the importance. The importance multiplication means divides the visual volume into a plurality of divided areas, assigns the importance set for each type of subject to the divided areas in which the subject is divided, and assigns the importance of the divided areas to the photographing camera. A first coefficient that is preset to decrease as the distance from the focus position to the divided area increases, and a first coefficient that is preset to decrease as the divided area deviates from the focus position based on the depth of focus of the camera parameters. Multiply by the second coefficient. That is, the importance multiplication means multiplies the first coefficient and the second coefficient so that the importance of the subject to which the producer of the two-dimensional video content wants to guide the line of sight becomes higher.

The visual guidance means generates rendering parameters in which the importance multiplied by the importance multiplication means is reflected in the volumetric capture information, and adds the rendering parameters to the VR content. In other words, this VR content has a rendering parameter added so that a subject with a high degree of importance is gazed at, so the visual guidance effect is enhanced.

Here, the line-of-sight guidance effect is a video production effect that guides (attracts) the viewer's line of sight to the VR content of the subject intended by the creator of the video content due to the characteristics of the movement of the naked eye. For example, the visual guidance effect is to emphasize, brighten, or brighten a desired subject, or blur (defocus) a subject that is not desired to guide the viewer's gaze.

The present invention can also be implemented by a program that causes hardware resources such as a CPU, memory, and hard disk provided in a computer to operate as the above-described image presentation processing device.

According to the present invention, it is possible to efficiently create VR content with a high visual-guiding effect so that a producer of 2D video content can gaze at a subject to which the visual-sight is to be guided.

1 is a schematic configuration diagram of a VR content production system according to an embodiment; FIG. 1 is a block diagram showing the configuration of a video effect processing device according to an embodiment; FIG. FIG. 4 is an explanatory diagram illustrating calculation of a visual volume in the embodiment; FIG. 10 is an explanatory diagram for explaining an importance DB in the embodiment; FIG. 4 is an explanatory diagram illustrating estimation of a gaze parameter in an embodiment; 3 is a flow chart showing the operation of the image effect processing device of FIG. 2;

(embodiment)
[Configuration of VR content production system]
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.
A configuration of a VR content production system 1 according to an embodiment will be described with reference to FIG.
As shown in FIG. 1 , a VR content production system 1 produces VR content with a high visual guidance effect, and includes a fixed camera 2 , a photographing camera 3 , and a video presentation processing device 4 .

The fixed camera 2 is a camera for accurately tracking the movement of a subject 9 photographed by a photographing camera 3 to be described later and performing volumetric capture for digitizing it as three-dimensional data. The fixed camera 2 outputs a video image of the subject 9 to the video presentation processing device 4 . The image captured by the fixed camera 2 is subjected to volumetric capture by the image effect processing device 4. - 特許庁For example, the fixed camera 2 is fixed at a predetermined position in a photography studio (not shown). In FIG. 1, five fixed cameras 2 are illustrated for easy viewing of the drawing, but the number of fixed cameras 2 is not particularly limited.

The photographing camera 3 is a general live-action camera for photographing the subject 9 and outputs the photographed image of the subject 9 to the image effect processing device 4 . The video captured by this camera 3 is used for producing two-dimensional video content. For example, a cameraman (not shown) operates the photographing camera 3 to photograph the subject 9 . At this time, the fixed camera 2 and the photographing camera 3 may be photographed at the same time.
Note that the photographing camera 3 may be a virtual camera. In this case, the cameraman uses a manipulator capable of inputting camera parameters to operate the virtual camera and shoot the subject 9 .

The image effect processing device 4 uses the volumetric capture information and the camera parameters of the imaging camera 3 to add a visual guidance effect to the VR content of the subject 9 generated in advance by volumetric capture.

[Configuration of image production processing device]
The configuration of the image effect processing device 4 will be described with reference to FIG.
As shown in FIG. 2, the image effect processing device 4 includes volumetric capture means 40, camera parameter estimation means 41, visual volume calculation means 42, three-dimensional saliency map generation means 43, and subject recognition means 44. , importance labeling means (importance setting means) 45 , gaze parameter estimation means (importance multiplication means) 46 , and visual guidance means 47 .

The volumetric capture means 40 generates VR content and volumetric capture information by volumetric capture. In this embodiment, the volumetric capture means 40 generates VR content and volumetric capture information of the subject 9 by performing volumetric capture, which will be described later, on the video from each fixed camera 2 . Then, the volumetric capture means 40 outputs the generated volumetric capture information and VR contents to the camera parameter estimation means 41 and the line of sight guidance means 47 .

Note that volumetric capture is a method of acquiring the three-dimensional shape and surface pattern (for example, surface characteristics such as texture) of the subject 9 in time series. The volumetric capture information is information representing the three-dimensional shape and surface pattern of the subject 9 obtained by volumetric capture. For example, volumetric capture includes the method described in Reference 1.
Reference 1: “4D Views”, [online], Crescent Inc., [searched on May 17, 2019], Internet <URL: https://www.crescentinc.co.jp/company/>

The camera parameter estimation means 41 estimates the camera parameters of the photographing camera 3 by camera calibration. In this embodiment, the camera parameter estimating means 41 estimates the camera parameters of the imaging camera 3 by performing general camera calibration on the image captured by the imaging camera 3 . For example, as a method of camera calibration, there are methods described in JP-A-2011-118724 and JP-A-2014-127068, so detailed description thereof will be omitted. Then, the camera parameter estimation means 41 outputs the estimated camera parameters and the captured image to the visual volume calculation means 42 .

Note that the camera parameters represent the position and orientation of the photographing camera 3 operated by the cameraman and the photographing angle of view, and include, for example, pan, tilt, zoom, focus position, iris, and the position of the lens principal point. there is In other words, it is considered that the camera parameters reflect the camera work of the cameraman, which is one of the know-how of the creator of the two-dimensional video content.

Here, when the photographing camera 3 is a virtual camera, the camera parameter estimation means 41 may estimate camera parameters based on the operation result of a manipulator that operates the virtual camera. In this case, the camera parameter estimating means 41 may use the camera parameters and the volumetric capture information to render the video captured by the virtual camera and present it to the cameraman as the viewfinder video of the virtual camera.

The visual volume calculating means 42 calculates a quadrangular pyramid-shaped area represented by the lens principal point and the imaging angle of view of the camera parameters estimated by the camera parameter estimating means 41 as the visual volume, which is the photographing range of the photographing camera 3. is. This visual volume is volume information that indicates which area is the shooting target in the three-dimensional space. As shown in FIG. 3, the visual volume V has a vertex at the lens principal point _VT , and has a size corresponding to the horizontal photographing angle of view _θH and the vertical photographing angle of view _θV . Further, the bottom surface _VB is set near the focus position of the photographing camera 3 so that the back surface of the subject 9 can be accommodated in the visual volume V in the depth direction. Therefore, the visual volume V includes the subject 9 (for example, a person's face) to which the creator of the two-dimensional video content wants to guide the line of sight.
After that, the visual volume calculating means 42 outputs the calculated visual volume V to the three-dimensional saliency map generating means 43 and the object recognizing means 44 .

The 3D saliency map generation means 43 refers to the volumetric capture information from the volumetric capture means 40 and generates a 3D saliency map. As shown in FIG. 3 , the three-dimensional saliency map generating means 43 generates a three-dimensional saliency map for the subject 9 included in the visual volume V calculated by the visual volume calculating means 42 . The three-dimensional saliency map generating means 43 then outputs the generated three-dimensional saliency map to the gaze parameter estimating means 46 .
In FIG. 3, of the entire subject 9, the area of the subject 9 included in the visual volume V is illustrated with solid lines, and the area of the subject 9 not included in the visual volume V is illustrated with broken lines.

The three-dimensional saliency map represents information obtained by converting the three-dimensional shape of the subject 9 into a planar depth image, and information obtained by quantifying the conspicuity of the gradient and color of the surface pattern of the subject 9. there is That is, the three-dimensional saliency map represents the degree of attention calculated from each feature map by generating three feature maps of luminance value, color space, and gradient direction. Here, a three-dimensional saliency map is generated for each subject 9 included in the visual volume V. FIG.

An example of the 2D saliency map is described in detail in References 2 and 3, and the 3D saliency map can be generated in the same procedure, so further explanation is omitted.
Reference 2: Itti, Koch, "A saliency-based search mechanism for overt and covert shifts of visual attention", Vision Research, 40(2000), 1489-1506
Reference 3: Digital Image Processing [Revised New Edition], Association for the Promotion of Image Information Education, March 9, 2015, pp.244-246

The subject recognition means 44 recognizes the type of the subject 9 included in the visual volume V from the visual volume calculation means 42 by machine learning. The type of subject indicates the type of subject 9 included in the script of the two-dimensional video content, such as the main character, main character's face, supporting characters, extras, and the like. For example, the subject recognition means 44 refers to the volumetric capture information, targets the three-dimensional shape and surface pattern of the subject 9, and renders each subject 9 into a two-dimensional image from six directions, up, down, left, right, front and back. Then, the subject recognition means 44 recognizes the type of the subject 9 rendered as a two-dimensional image by machine learning. Note that the method described in reference 3 can be cited as a method of machine learning.
Reference 3: Joseph Redmon, Ali Farhadi,”YOLOv3: An Incremental Improvement“,2018.4.8

Here, when the type of the subject 9 is a face, the subject recognition means 44 may also recognize the direction of the face such as "forward" or "backward" by machine learning. For example, when the main character faces the front, the subject recognition means 44 recognizes the direction of the face as "forward".
After that, the subject recognizing means 44 outputs the type of the recognized subject 9 to the importance labeling means 45 .

The importance labeling means 45 labels (corresponds to) the type of the subject 9 from the subject recognition means 44 and the degree of importance set in advance for each type of the subject 9 . Furthermore, when the type of subject 9 is a face, the importance labeling means 45 labels the direction and importance of the face recognized by the subject recognizing means 44 because the viewer's gaze tends to focus on the face. That is, the importance labeling unit 45 generates an importance DB in which the type of the subject 9, the direction of the subject 9, and the importance are associated with each other, as shown in FIG. The importance labeling means 45 then outputs the generated importance DB to the gaze parameter estimation means 46 .

“Subject type” represents the type of the subject 9 recognized by the subject recognition means 44 .
"Direction of subject" indicates the direction of the face (for example, "forward" or "backward") when the type of subject 9 is a face. In the example of FIG. 4, the direction of the main character's face is "forward".
"Importance" represents the importance of the subject 9 in the two-dimensional video content. This degree of importance takes a large value if the object 9 is important, and a small value if the object 9 is not important. Here, the importance level is manually set based on the words (for example, lines) included in the script for producing the 2D video content. For example, the degree of importance is set based on the appearance frequency of words included in the script (eg, TF-IDF).

Based on the 3D saliency map from the 3D saliency map generation means 43 and the importance DB from the importance labeling means 45, the gaze parameter estimation means 46 calculates the gaze parameter of each voxel as described below. It estimates parameters (importance). Then, the gaze parameter estimation means 46 outputs the estimated gaze parameters to the gaze guidance means 47 .

<Estimation of gaze parameters>
Estimation of gaze parameters will be described with reference to FIG.
As shown in FIG. 5, the gaze parameter estimator 46 divides the visual volume V into a plurality of voxels (divided regions) B. FIG. As a result, the subject 9 included in the visual volume V is also divided into voxels B. FIG. The voxels B are rectangular parallelepipeds, and the number and size thereof are arbitrary based on the angle of view (aspect ratio) of the photographing camera 3 . In the example of FIG. 5, the angle of view of the imaging camera 3 is 4:3, and the visual volume V is divided into a total of 48 voxels B, 4 in the horizontal direction, 3 in the vertical direction, and 4 in the depth direction. there is Also, in the example of FIG. 5, the number of divisions in the depth direction is adjusted to the larger number of divisions in the horizontal or vertical direction. Further, the bottom surface _VB of the visual volume V is set so that when divided into voxels B, the focus position of the photographing camera 3 is the center of the voxel space in the depth direction. Note that the voxel space is a space composed of a set of voxels B. FIG.

Next, the gaze parameter estimating means 46 assigns the importance stored in the importance DB to the voxel B into which the subject 9 is divided. As a result, the importance is estimated for each voxel B, and the importance becomes the gaze parameter of each voxel B. FIG.

Next, the gaze parameter estimating means 46 multiplies the importance of each voxel B by a first coefficient and a second coefficient, which will be described later. That is, the gaze parameter estimating means 46 multiplies the importance of each voxel B by the first coefficient and the second coefficient so that the importance of the subject 9 to which the producer of the two-dimensional video content wants to guide the line of sight is high. This first coefficient is a coefficient set in advance so as to decrease as the distance from the focus position (the center of the voxel space) of the photographing camera 3 to each voxel B increases. Specifically, the first coefficient is inversely proportional to the square of the distance L as shown in the following equation (1), and the distance L is set to 1 on one side of the voxel B. Note that W represents a weight set in advance with an arbitrary value of 1 or more.
W/(L ² +1) Expression (1)

Here, when the type of the subject 9 corresponding to the voxel B is a face, the first coefficient is set according to the direction of the face as shown in Equations (1-2) and (1-3) below. may As a result, in the visual guidance means 47 described later, it is possible to realize the "front space", which is the visual presentation effect of the two-dimensional video content. Note that W1 and W2 represent weights set in advance with arbitrary values of 1 or more.
Forward: W1/(L ² +1) Equation (1-2)
Backward: W2/(L ² +1) Equation (1-3)

Also, the second coefficient is a coefficient corresponding to the depth of focus of the imaging camera 3 . Specifically, the second coefficient is a coefficient preset based on the depth of focus of the camera parameters so that the voxel B becomes smaller as it deviates from the focus position. That is, the second coefficient becomes smaller with distance from the in-focus position in both the direction toward and away from the photographing camera 3, with the in-focus position on the camera optical axis as a reference.

For example, the weights W, W1, and W2 of the first coefficients are set to "2.0" based on empirical rules. Further, for example, the second coefficient is set to "0" at the limit position where the resolution is 1/4 of the resolution at the time of focusing, and is set to "2.0" at the in-focus position. A value obtained by linearly interpolating between positions.

Returning to FIG. 2, the description of the configuration of the image effect processing device 4 is continued.
The gaze guidance means 47 generates rendering parameters in which the importance (gazing parameters) estimated by the gaze parameter estimating means 46 is reflected in the volumetric capture information from the volumetric capture means 40 .

First, the line-of-sight guidance means 47 normalizes the importance of each voxel B. As shown in FIG. For example, the line-of-sight guidance means 47 normalizes the importance of voxel B with a value from "0" to "1". Next, the visual guidance means 47 multiplies the surface pattern (color saturation) of the volumetric capture information by the normalized importance. As a result, the higher the importance of each voxel, the brighter the color. Further, the line-of-sight guidance means 47 generally obtains a coefficient indicating the relationship between the kernel size and the focal length of the bokeh filter to be reflected in the rendering from the focus position in the three-dimensional space, the size of the imaging device of the photographing camera 3, and the aperture of the lens. are calculated based on a specific lens model, and the calculated coefficients are reflected in the rendering parameters. Thus, each voxel B expresses defocus.

Then, the visual guidance means 47 adds rendering parameters reflecting the degree of importance and depth of focus to the VR content from the volumetric capture means 40, and outputs the VR content. In this way, in the VR content, the color of the subject 9 whose line of sight is to be guided becomes vivid, and defocus is expressed.

[Processing of image production processing device]
Processing of the image effect processing device 4 will be described with reference to FIG.
As shown in FIG. 6, in step S1, the volumetric capture means 40 generates VR content and volumetric capture information by volumetric capture.

In step S2, the camera parameter estimation means 41 estimates camera parameters of the photographing camera 3 by camera calibration.
In step S3, the visual volume calculation means 42 calculates a quadrangular pyramid-shaped area represented by the lens principal point and the imaging angle of view of the camera parameters estimated in step S2 as the visual volume V, which is the imaging range of the imaging camera 3. do.

In step S4, the three-dimensional saliency map generating means 43 refers to the volumetric capture information generated in step S1 and generates a three-dimensional saliency map.
In step S5, the subject recognition means 44 recognizes the type of subject 9 included in the visual volume V calculated in step S3 by machine learning.
In step S6, the importance labeling unit 45 labels the type of the subject 9 recognized in step S5 and the degree of importance set in advance for each type of the subject 9. FIG.

In step S7, the gaze parameter estimation means 46 estimates the gaze parameter of each voxel B based on the 3D saliency map and the importance DB.
In step S8, the visual guidance means 47 generates rendering parameters in which the gaze parameters are reflected in the volumetric capture information, and adds the generated rendering parameters to the VR content.

[Action/effect]
As described above, the image effect processing device 4 generates VR content in which the color of the subject 9 to which the line of sight is to be guided becomes vivid and the defocus is expressed. In this way, the image effect processing device 4 can make the creator of the two-dimensional image content gaze at the subject 9 to guide the line of sight, and can efficiently create VR content with a high line of sight guidance effect. That is, the video effect processing device 4 not only makes the production of VR content more efficient, but also enables the intention of the creator to be conveyed in the VR content at the same level as in the two-dimensional video.

Furthermore, the image effect processing device 4 reflects the direction of the face, which tends to attract the viewer's gaze, to the degree of importance. As a result, the image rendering processing device 4 can apply the "front blank", which is the image rendering method used in the production of two-dimensional image content, to VR content.

(Modification)
Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes and the like without departing from the gist of the present invention.

In the embodiments described above, lens distortions not included in the camera parameters may be taken into account. That is, for lens distortion, a lens distortion model is selected in advance, and a lens distortion coefficient is set from the lens distortion model using camera parameters related to lens distortion. Then, the visual volume calculation means calculates, as the visual volume, a quadrangular pyramid-shaped area represented by the photographing angle of view reflecting the set lens distortion coefficient and the lens principal point. As a result, even when shooting with an ultra-wide-angle lens that is susceptible to lens distortion, the visual volume can be obtained accurately, so the subject to which the line of sight is to be guided can be accurately gazed at.

In the embodiment described above, the user may manually adjust the weight of the first coefficient via a volume dial or GUI (Graphical User Interface). Alternatively, the subject type and the first coefficient may be learned by a neural network, and the first coefficient corresponding to the recognized subject type may be estimated using the learned discriminator.

The visual guidance effect is not limited to the embodiment described above. For example, a bounding box that includes voxels exceeding a predetermined reference value may be set based on the degree of importance, and the bounding box may be included in the VR content. In this case, by drawing a bounding box with a rectangular solid line corresponding to the viewer's viewpoint, the area desired to be focused on may be specified (emphasized).

In each of the above-described embodiments, the image effect processing device has been described as independent hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk provided in a computer to operate as the above-described image presentation processing device. These programs may be distributed via a communication line, or may be distributed after being written in a recording medium such as a CD-ROM or flash memory.

1 VR content production system 2 Fixed camera 3 Shooting camera 4 Video effect processing device 40 Volumetric capture means 41 Camera parameter estimation means 42 Visual volume calculation means 43 3D saliency map generation means 44 Object recognition means 45 Importance labeling means (important degree setting means)
46 gaze parameter estimation means (importance multiplication means)
47 Line-of-sight guidance means

Claims (5)

A camera of a photography camera that generates volumetric capture information consisting of a three-dimensional shape and surface pattern of a subject by volumetric capture that digitizes the movement of the subject into three-dimensional data, and photographs the volumetric capture information and the subject. A video effect processing device that uses a parameter to add a visual guidance effect that guides the VR content of the subject according to the movement characteristics of the naked eye,
volumetric capture means for generating the VR content and the volumetric capture information from the volumetric capture;
A quadrangular pyramid-shaped area represented by the lens principal point, which is the center coordinate of the lens of the imaging camera, and the imaging angle of view of the imaging camera, which is included in the camera parameters, is defined as a visual volume, which is the imaging range of the imaging camera. a visual volume calculating means for calculating;
3D saliency map generating means for generating a 3D saliency map of a 3D shape and surface texture of an object contained in the visual volume based on the volumetric capture information;
subject recognition means for recognizing the type of subject included in the visual volume by machine learning;
importance level setting means for setting the type of the subject recognized by the subject recognition means and the level of importance in association with each other;
dividing the visual volume into a plurality of divided areas, assigning the importance set for each type of the subject to the divided areas in which the subject is divided, and assigning the assigned importance to the divided areas of the photographing camera; a first coefficient set in advance so that the distance from the focus position to the divided area becomes smaller as the distance increases; importance multiplication means for multiplying the set second coefficient;
visual guidance means for generating rendering parameters in which the importance multiplied by the importance multiplication means is reflected in the volumetric capture information, and adding the rendering parameters to the VR content;
A video presentation processing device comprising: camera parameter estimation means for estimating the camera parameters by camera calibration ;
2. The video presentation processing device according to claim 1, further comprising: The subject recognition means recognizes a face as the subject and also recognizes the direction of the face by the machine learning,
3. The image effect processing apparatus according to claim 2, wherein said importance level setting means sets the direction of the face recognized by said subject recognition means in association with said importance level. The visual volume calculation means calculates, as the visual volume, a quadrangular pyramid-shaped region represented by the photographing angle of view and the lens principal point in which a preset lens distortion coefficient is reflected. The image effect processing device according to any one of claims 1 to 3. A program for causing a computer to function as the image effect processing device according to any one of claims 1 to 4.

Images