JP2023072709A - Three-dimensional vr imaging device, three-dimensional vr reproduction system, three-dimensional vr imaging method, program therefor, and recording medium in which program is recorded - Google Patents

Abstract

To provide a three-dimensional VR imaging device with which, when capturing a three-dimensional VR moving-image, it is possible to focus on a video other than the front (imaging object).SOLUTION: The present invention comprises rotation support means for supporting imaging means in a horizontal direction so as to be rotatable 360 degrees, and moving-image editing means for editing a moving-image. The imaging means captures an image of background material data which is a moving-image material for the background, while rotating the imaging means 360 degrees using the rotation support means and captures an image of main part material data which is a moving-image material in which imaging objects are captured while rotating the imaging means 360 degrees using the rotation support means, with the imaging objects being arranged in substantially circular form around the imaging means. The moving-image editing means includes a still image generation unit for generating a still image from the background material and main part material data, a position information detection unit for detecting information related to a rotation position of the rotation support means when imaging, with regard to a plurality of generated still images, and a moving-image synthesis unit for synthesizing the background material data and main part material data together and generating synthesized moving-image data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a 3D VR imaging apparatus and a 3D VR imaging method for capturing a 3D VR (Virtual Reality) moving image (including images). The present invention may also relate to a 3D VR playback system that includes a 3D VR shooting device that shoots a 3D VR movie and plays back the shot movie. The present invention may also relate to a program for a three-dimensional VR imaging method and a recording medium recording the program.

VR (virtual reality) is a technology that creates an environment that is not the actual thing, but has the same essence as a function, by stimulating the five senses. It is said. In shooting a three-dimensional VR moving image, an object to be shot is positioned at the center of the screen, and the outer shape of the central object is shot from 360-degree directions to generate stereoscopic image data. In Japanese Patent Laid-Open No. 2004-100000, a moving object moving distance and a moving object are calculated based on data on the movement of a moving object detected by motion detection means and data on the posture and position of the head detected by posture and position detection means. Based on the current position of the moving object obtained by correcting the estimated position of the moving object from the estimated position and the movement distance of the user's head, the movement of the moving object in the real world image data captured by the imaging means is corrected. A technology related to a VR spatial image correction system having a control means for generating display data, synthesizing the display data with an image of a virtual world to create a VR spatial image, and displaying the VR spatial image on a display means is disclosed. In Patent Document 2, in order to shoot a three-dimensional VR image with two cameras, a base having a horizontal portion and a vertical portion that are perpendicular to each other in an L shape, a first digital a first support supporting a camera; and a second support standing upright from a vertical portion parallel to the horizontal portion and supporting a second digital camera, the first support being slidable with the horizontal portion; There is disclosed a technique related to a three-dimensional VR image capturing device that enables adjustment of the distance between a first digital camera and a second digital camera. US Pat. No. 5,300,000 describes a camera-related method and apparatus well suited for use in capturing stereoscopic image data, such as left and right eye image pairs, for mounting multiple cameras simultaneously. Regarding the camera rig obtained, the camera rig includes three mounting positions corresponding to three different directions 120 degrees apart, one or more of the mounting positions can be used at a time, and if one camera pair is used: The rig can be rotated to capture images corresponding to positions unmounted by camera pairs, and still images from those positions are combined with corresponding images in the forward direction to generate a 360 degree field of view. Alternatively, camera pairs, or individual cameras, may be included in each of the mounted positions to capture images in multiple directions.

JP 2021-51512 A JP 2019-117288 A Japanese Patent Publication No. 2017-521897

In shooting a 3D VR moving image, images at positions other than the front (central object) are out of focus. When shooting a three-dimensional VR video, for example, the background image is out of focus and blurred. The techniques disclosed in Patent Literatures 1 to 3 do not describe a method for focusing an image other than the front (central object) in shooting a 3D VR moving image.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a 3D VR imaging apparatus and a 3D VR imaging method capable of focusing an image other than the front (object to be shot) in shooting a 3D VR moving image. Another object of the present invention is to provide a 3D VR playback system including a 3D VR shooting device capable of focusing an image other than the front (object to be shot) in shooting a 3D VR moving image. good. Further, the present invention provides a program for a three-dimensional VR shooting method capable of focusing an image other than the front (object to be shot) in shooting a three-dimensional VR moving image, and a recording medium storing the program. It can be a purpose.

In order to solve the above problems, one embodiment of the present invention provides a three-dimensional moving image that displays a first moving image and a second moving image simultaneously captured at a position with parallax from the first moving image. A three-dimensional VR imaging device for imaging, comprising imaging means for imaging the first moving image and the second moving image, rotation support means for supporting the imaging means so as to be rotatable by 360 degrees in a horizontal direction, and the imaging. a moving image editing means for editing the moving image captured by the means, wherein the shooting means captures background material data, which is a background moving image material, while rotating the shooting means 360 degrees using the rotation support means; Main material data, which is a moving image material obtained by photographing the object to be photographed while rotating the photographing means by 360 degrees using the rotation support means in a state in which the object to be photographed is arranged in a substantially circular shape around the photographing means. and the moving image editing means includes a still image generating unit that generates a still image from the background material data and the main material data, and the plurality of still images generated by the still image generating unit that are rotated at the time of shooting. A three-dimensional image comprising: a position information detection unit for detecting information about the rotational position of a support means; and a moving image synthesizing unit for synthesizing the background material data and the main material data to generate synthetic moving image data. To provide a VR imaging device. Further, another embodiment of the present invention is the three-dimensional VR imaging device of the above one embodiment, wherein the still image generation unit generates a background still image using the background material data, Main part material data is used to generate a main part still image, and the moving image synthesizing unit first generates a plurality of the background still images and a plurality of the main part based on the information about the rotational position detected by the position information detection unit. Then, the main still image is divided into a target portion where the object is photographed and a non-target portion where the object is not photographed, and then The three-dimensional VR imaging apparatus may be characterized in that the synthesized moving image data is generated by connecting a plurality of corrected images obtained by replacing the non-target portion of the main still image with the background still image. . Further, another embodiment of the present invention is the three-dimensional VR imaging device of the above one embodiment, wherein the still image generation unit generates a background still image using the background material data, Main part material data is used to generate a main part still image, and the moving picture synthesizing unit first identifies a plurality of background feature points on the plurality of background still images, and generates a plurality of the main part still images. Then, based on the plurality of background feature points and the plurality of main feature points, the plurality of background still images and the plurality of main feature points are specified. Then, the main still image is divided into a target portion in which the photographing object is photographed and a non-target portion in which the photographing object is not photographed, and then the main still image The three-dimensional VR imaging apparatus may be characterized in that the synthesized moving image data is generated by connecting a plurality of corrected images obtained by replacing the non-target portion of the image with the background still image.

Another embodiment of the present invention is a three-dimensional VR playback system including any one of the three-dimensional VR imaging devices described above, and display means for displaying the synthesized moving image data generated by the moving image synthesizing unit to a user. and reproduction condition detection means for detecting information about the display position of the display means that moves when the user changes the viewpoint, wherein the display means detects the display position detected by the reproduction condition detection means. The three-dimensional VR playback system may be characterized by displaying the synthetic moving image data corresponding to information relating to. In another embodiment of the present invention, the three-dimensional VR playback system further includes video communication means for transmitting/receiving data between the three-dimensional VR imaging device and the display means in a wired and/or wireless manner. The display means may receive and display the synthesized moving image data generated by the moving image synthesizing section using the moving image communication means.

Another embodiment of the present invention is a 3D VR imaging device for capturing a 3D moving image displaying a first moving image and a second moving image simultaneously captured at a position with parallax from the first moving image. A photographing means for photographing the first moving image and the second moving image, a rotating support means for supporting the photographing means so as to be rotatable by 360 degrees in a horizontal direction, and editing the moving images photographed by the photographing means. moving image editing means, wherein the photographing means photographs background material data, which is a background moving image material, while rotating the photographing means by 360 degrees using the rotation support means, and Main material data, which is a moving image material obtained by photographing the object to be photographed, while rotating the photographing means by 360 degrees using the rotating support means in a state in which the photographing means are arranged in a substantially circular shape around the means, and the moving picture editing means. a still image generation unit for generating a still image from the background material data and the main material data; A three-dimensional VR imaging device characterized by comprising a position information detection unit for detecting and a moving image synthesizing unit for synthesizing the background material data and the main material data to generate synthesized moving image data. A 3D VR shooting method for shooting a 3D moving image, wherein background material data, which is a background moving image material, is shot using the shooting means and rotating the shooting means by 360 degrees using the rotation support means. a step of photographing a background material; A plurality of background still images are generated from the background material data as the still images by using a main material shooting step of shooting main material data, which is a moving image material for the main part in which an object is shot, and the still image generating unit. a background still image generation step; a main story still image generation step of generating a plurality of main story still images from the main content material data as the still images using the still image generation unit; and a position information detection unit. , a position information detecting step of detecting information about the rotational position at the time of shooting for each of the plurality of background still images and the plurality of main still images generated by the still image generation unit; First, an image collation step of collating the plurality of background still images and the plurality of main part still images based on the information about the rotational position detected by the position information detection unit; Next, an image segmentation step of dividing the main still image into a target portion in which the shooting target is shot and a non-target portion in which the shooting target is not shot; and then combining a plurality of corrected images obtained by replacing the non-target portion of the main still image with the background still image to generate the synthetic moving image data. To provide a three-dimensional VR imaging method characterized by:

Further, another embodiment of the present invention may be a program for causing a computer to execute the above three-dimensional VR imaging method. Alternatively, a computer-readable recording medium recording the above program may be used.

According to the 3D VR imaging device, the 3D VR imaging method, the 3D VR playback system, the program for the 3D VR imaging method, and the recording medium recording the program, according to the present invention, when shooting a 3D VR moving image, Images other than (the central object) can be focused. Further, according to the 3D VR imaging device, the 3D VR imaging method, the 3D VR playback system, the program for the 3D VR imaging method, and the recording medium recording the program according to the present invention, the image quality of the 3D VR moving image You can reduce the size of the video without lowering the speed. Further, according to the 3D VR imaging device, the 3D VR imaging method, the 3D VR playback system, the 3D VR imaging method program, and the recording medium recording the program, additional devices, devices, and equipment according to the present invention etc., can be realized with a simple configuration, and the cost of creating a moving image can also be reduced.

1 is a schematic configuration diagram showing an example of a 3D VR imaging device and a 3D VR playback system according to an embodiment of the present invention; FIG. It is an explanatory view explaining an example of a photography means concerning an embodiment of the present invention, and a rotation support means. 1 is an explanatory diagram illustrating an example of VR goggles (display means, reproduction condition detection means, video storage means, video communication means, etc.) according to an embodiment of the present invention; FIG. FIG. 4 is an explanatory diagram illustrating an example of three-dimensional VR imaging according to the embodiment of the present invention; FIG. 4 is an explanatory diagram illustrating an example of playback of the VR goggles according to the embodiment of the present invention; 1 is a functional block diagram illustrating an example of functions of a 3D VR imaging device and a 3D VR playback system according to an embodiment of the present invention; FIG. FIG. 4 is a flow chart diagram illustrating an example of the operation of the 3D VR imaging method according to the embodiment of the present invention;

The present invention will be described using examples of a 3D VR imaging device, a 3D VR playback system, and a 3D VR imaging method according to embodiments. Note that the present invention can also be applied to a three-dimensional video that displays a first video and a second video shot simultaneously at a position with a parallax from the first video, in addition to the three-dimensional VR imaging device or the like that will be described later. Anything (equipment, machine, part, system, application, etc.) that shoots (still images, moving images, images including audio, etc.) or displays the captured 3D moving images. can be done. In addition, the present invention provides information terminals (mobile phones, smartphones, tablets, personal computers, televisions, projectors, glasses-type terminals, wristwatch-type terminals, other wearable terminals, etc.) capable of two-way data communication, for example. It may also be used when shooting or playing back moving images.

The present invention will be described in the order shown below.
1. Configuration of 3D VR imaging device and 3D VR playback system2. 3. Functions of 3D VR shooting and 3D VR playback; Three-dimensional VR shooting method, its program and its recording medium

(1. Configuration of 3D VR shooting device and 3D VR playback system)
A 3D VR shooting device captures a 3D video (still image, video, video including audio, etc.) having a first video and a second video simultaneously shot at a position with parallax from the first video. It is the thing to be photographed (equipment, machine, part, system, application, etc.). The 3D VR playback system includes the 3D VR imaging device described above, and displays a 3D video shot by the 3D VR shooting device or a video edited from the 3D video shot (equipment, machine, parts , systems, applications, etc.).

Configurations of a 3D VR imaging device and a 3D VR playback system according to an embodiment of the present invention will be described with reference to FIGS. 1 to 5. FIG. Here, FIG. 1 is a schematic configuration diagram showing an example of a 3D VR imaging device 100 and a 3D VR playback system 100S according to an embodiment of the present invention. FIG. 2(a) is a schematic side view of an example of the photographing means 11 and the rotation support means 12 according to the embodiment of the present invention. FIG. 2(b) is an explanatory diagram illustrating an example of the operation of rotating the photographing means 11 in the horizontal direction by 360 degrees according to the embodiment of the present invention. FIG. 3 is a schematic external view illustrating an example of VR goggles (display means 21, reproduction condition detection means 22, video storage means 23, video communication means 24, etc.) according to an embodiment of the present invention. FIG. 4 is an explanatory diagram illustrating an example of imaging of an object to be imaged (person to be imaged) ST in three-dimensional VR imaging according to the embodiment of the present invention. FIG. 5(a) is an explanatory diagram illustrating an example of playback (moving image display in the direction of -90 degrees in the horizontal direction) of the VR goggles according to the embodiment of the present invention. FIG. 5(b) is an explanatory diagram illustrating an example of playback (video display in the direction of 0 degrees in the horizontal direction) of the VR goggles according to the embodiment of the present invention. FIG. 5(c) is an explanatory diagram illustrating an example of reproduction (moving image display in the direction of +45 degrees in the horizontal direction) of the VR goggles according to the embodiment of the present invention. 1 to 5 are merely examples, and the present invention is not limited to the three-dimensional VR imaging apparatus and the like shown in FIG. 1 and the like.

As shown in FIG. 1, in this embodiment, the 3D VR imaging device 100 includes control means 10 for controlling the overall operation of the 3D VR imaging device 100, a first moving image and a second moving image (first moving image). (moving images taken at the same time at a position with parallax); rotating support means 12 for supporting the photographing means 11 so as to be able to rotate 360 degrees in the horizontal direction; and video editing means 13 for editing. Further, in this embodiment, the three-dimensional VR imaging device 100 is a storage means for storing moving images captured by the imaging means 11 and moving images edited by the moving image editing means 13, editing conditions, and editing results (for example, synthesized moving image data to be described later). 14, and transmitting/receiving means 15 for transmitting the moving image stored in the storage means 14 to a display.

The 3D VR imaging device 100 according to the present invention displays a 3D video (still image, video , video with sound, etc.). In this embodiment, the three-dimensional VR imaging device 100 (1) uses the rotation support means 12 to support the imaging means 11 so as to be able to rotate 360 degrees in the horizontal direction at the time of imaging, and (2) uses the imaging means 11 to (3) edit the moving image captured by the imaging means 11 using the moving image editing means 13; (4) use the storage means 14 to edit the moving image captured by the imaging means 11; The moving image editing means 13 stores the edited moving image, the editing conditions, and the editing result, and (5) the transmitting/receiving means 15 is used to transmit the moving image stored by the storage means 14 to a display device.

Each configuration will be specifically described below.

The control means 10 is a means for instructing each component of the 3D VR imaging device 100 to operate and controlling the operation of the 3D VR imaging device 100 . The control means 10 can use, for example, an arithmetic device composed of a CPU and a memory. Also, the control means 10 may be configured to use an arithmetic device, a controller, or other control means pre-installed in the three-dimensional VR imaging apparatus 100 .

In this embodiment, the control means 10 performs an operation of inputting information (hereinafter referred to as "input information") related to video playback input from outside the 3D VR imaging device 100, and outputting the information to the outside of the 3D VR imaging device 100. It controls the operation of outputting information (hereinafter referred to as “output information”) related to the moving image to be processed. Note that the control means 10 can use character data, image data, moving image data, audio data, scan data, etc. as the input information.

The photographing means 11 is means for photographing a first moving image and a second moving image (moving images simultaneously photographed at a position with parallax from the first moving image). The photographing means 11 can be a stereo camera (for example, a camera capable of recording information in the depth direction by simultaneously photographing an object from a plurality of different directions). In this embodiment, the photographing means 11 photographs background material data, which is a background moving image material, while rotating the photographing means 11 by 360 degrees using the rotation support means 12 (for example, Ma in FIG. 2(b)). Further, in the present embodiment, the photographing means 11 rotates the photographing means 360 degrees using the rotation support means so that the photographing object ST is arranged in a substantially circular shape around the photographing means 11. Main part material data, which is a moving image material obtained by shooting the ST, is shot (for example, FIG. 4).

The rotation support means 12 is means for supporting the photographing means 11 so as to be rotatable by 360 degrees in the horizontal direction. In the present embodiment, the rotary support means 12 supports the photographing means 11 with, for example, a tripod, fixes the standing position and height of the photographing means 11, and rotates 360 degrees in the horizontal direction while only panning (for example, FIG. 2). (a) and (b)). It should be noted that the positional relationship of the imaging means 11 (for example, left and right stereo cameras) does not change even while the rotation support means 12 rotates 360 degrees in the horizontal direction.

The moving image editing means 13 is a means for editing moving images shot by the shooting means 11 . In this embodiment, the moving picture editing means 13 includes a still image generating section 13a that generates a still image from the background material data and the main story material data, and a position information detecting section that detects information regarding the rotational position of the rotation support means 12 at the time of shooting. 13b, and a video synthesizing unit 13c for synthesizing background material data and main material data to generate synthetic video data. A configuration using a device may be used.

The still image generation unit 13a generates a background still image using the background material data, and generates a main story still image using the main story material data. Specifically, in the present embodiment, the still image generation unit 13a generates a plurality of background still images from the background material data and generates a plurality of main story still images from the main story material data as still images.

The position information detection section 13b detects information about the rotational position of the rotation support means 12 at the time of photographing for the plurality of still images generated by the still image generation section 13a. Specifically, in the present embodiment, the position information detection unit 13b detects information about the rotational position at the time of shooting for each of the plurality of background still images and the plurality of main part still images generated by the still image generation unit 13a. and adds information about the rotational position to the still image.

The moving image synthesizing unit 13c synthesizes the background material data and the main material data to generate synthetic moving image data. Specifically, in one embodiment, the moving image synthesizing unit 13c performs the following operations: (1) First, based on the information about the rotational position detected by the position information detecting unit 13b, a plurality of background still images and a plurality of main part still images are generated. (2) Next, the main still image is divided into a target portion in which the object to be photographed is photographed and a non-target portion in which the object to be photographed is not photographed, and (3) , generating a plurality of corrected images in which the non-target portion of the main still image is replaced with the collated background still image, and connecting the generated plurality of corrected images to generate synthesized moving image data.

Another embodiment in which the moving image synthesizing unit 13c generates synthesized moving image data is as follows: (1) first, a plurality of background feature points on a plurality of background still images are specified; (2) Next, based on the plurality of background feature points and the plurality of main feature points, a plurality of background still images and a plurality of main feature still images are identified. (3) Then divide the still image for the main part into a target part where the object is photographed and a non-target part where the object is not photographed, and (4) Next, the still image for the main part A plurality of corrected images are generated by replacing the non-target portion of the image with the collated background still image, and the generated plurality of corrected images are joined together to generate synthetic moving image data.

The storage means 14 is a means for storing a moving image photographed by the photographing means 11 and a moving image edited by the moving image editing means 13, editing conditions, and editing results (for example, synthesized moving image data to be described later). The storage means 14 can use a known technology (hard disk, memory, etc.) for storing electronic information as a database. Note that the storage means 14 may also store information other than the above. Also, a known technique can be used as a method for storing information by the storage means 14 . The 3D VR imaging device 100 may be configured to use a storage device mounted on, for example, VR goggles (eg, FIG. 3).

The transmitting/receiving means 15 is a means for transmitting/receiving a three-dimensional moving image to/from a device that displays the moving image stored in the storage means 14 (for example, FIG. 3). Specifically, in the present embodiment, the transmitting/receiving means 15 receives information regarding reproduction (position information, viewing angle direction, angle of view, reproduction range, etc.) transmitted from a device that displays moving images (for example, VR goggles). , transmits the 3D video corresponding to the received information about playback to the VR goggles. The transmitting/receiving means 15 may also transmit/receive information other than the above (for example, company code, medium type code, video classification code, serial number code, etc.). A known technique can be used for the method of transmitting and receiving by the transmitting/receiving means 15 . The transmitting/receiving means 15 can transmit/receive, for example, via network means (wired and/or wireless such as LAN, wireless LAN, Internet).

As a result, conventionally, in the case of VR shooting, focus was not achieved at positions other than the front (shooting object, central object), but according to the 3D VR shooting device according to the present invention, Generating multiple corrected images in which non-objective parts are replaced with background still images (images in focus), and combining the generated multiple corrected images to generate composite video data (three-dimensional video). Therefore, it is possible to focus on images other than the front (central object) when shooting 3D VR moving images. Further, according to the three-dimensional VR imaging device according to the present invention, a plurality of corrected images are generated by replacing the non-target portion of the main still image with the background still image (image in focus), and the generated plurality of corrected images are generated. Since it is possible to generate synthetic video data (3D video) by joining the corrected images, it is possible to focus images other than the front (central object) in shooting 3D VR videos. It is possible to reduce the size of the moving image without lowering the image quality of the original VR moving image. Further, according to the three-dimensional VR imaging apparatus of the present invention, (1) the rotation support means 12 is used to support the imaging means 11 so as to be able to rotate 360 degrees in the horizontal direction at the time of imaging, and (2) the imaging means 11 is used. (3) the video editing means 13 can be used to edit the video shot by the shooting means 11; It can be realized with a simple configuration, and the moving image creation cost can be reduced.

In the case of 4K 3D video, a resolution of 4000×2000px is required, and since the left and right images are independent, a 360-degree video doubles the video size, requiring 4000×4000px. For this reason, a huge amount of data is required for distribution, and in some cases it is very difficult to operate in distribution or live. According to the three-dimensional VR imaging device according to the present invention, a plurality of correction images are generated by replacing the non-target portion of the main still image with the background still image (image in focus), and the generated plurality of corrections are generated. Images can be spliced together to generate composite video data (three-dimensional video). It is possible to reduce the size of the video without lowering the image quality of the video, and it is possible to halve (or less) the size of the video without lowering the quality. It is possible to view (transmit and receive) 3D and high-resolution video (three-dimensional video).

As shown in FIG. 1, the 3D VR playback system 100S is a system including the 3D VR imaging device 100 described above, and displays (plays back) a 3D video captured and/or edited by the 3D VR imaging device 100. ). In this embodiment, the 3D VR playback system 100S includes a display means 21 for displaying synthetic moving image data (3D moving image) generated by the moving image synthesizing unit 13c to the user, and a display unit 21 that moves when the user changes the viewpoint. and reproduction condition detection means 22 for detecting information about the display position of the display means 21 . In addition, in the present embodiment, the 3D VR playback system 100S includes a moving image communication means 24 for transmitting and receiving data between the 3D VR imaging device 100 and the display means 21 by wire and/or wirelessly, and storing received 3D moving images. and a moving image storage means 23 for storing the moving image. In addition, the 3D VR playback system 100S (display means 21) uses a recording medium such as a videotape, DVD, or the like on which a 3D moving image (moving image captured or edited by the 3D VR imaging device 100) is recorded to reproduce the 3D moving image. may be displayed (reproduced). Also, the three-dimensional VR playback system 100S (display means 21) may be VR goggles as shown in FIG.

As shown in FIG. 1, in this embodiment, the display unit 21 includes a moving image reproducing unit 21a for displaying moving images and a moving image temporary recording unit 21b for temporarily recording moving images to be reproduced. The 3D VR playback system 100S (display means 21) uses the video communication means 24 to transmit information about the display position detected by the playback condition detection means 22 to the 3D VR imaging device 100, and corresponds to the information about the display position. The combined moving image data is received from the three-dimensional VR imaging device 100 and displayed by the moving image reproducing unit 21a. The 3D VR playback system 100S (display means 21), as shown in FIG. 5(a), corresponds to the direction of -90 degrees when the user changes the viewpoint in the horizontal direction to -90 degrees. 3D animation can be displayed. As shown in FIG. 5B, the 3D VR playback system 100S (display means 21) displays a 3D image corresponding to the direction of 0 degrees when the user changes the viewpoint in the horizontal direction to 0 degrees. You can display videos. As shown in FIG. 5(c), the 3D VR playback system 100S (display means 21) displays a 3D image corresponding to the direction of 45 degrees when the user changes the viewpoint in the horizontal direction to the direction of 45 degrees. You can display videos. Thus, according to the 3D VR playback system 100S according to the present invention, it is possible to display a video (3D video) according to the usage state of the user.

Thus, according to the 3D VR playback system 100S according to the present invention, the same effects as those of the 3D VR imaging device 100 can be obtained. Further, according to the 3D VR playback system 100S according to the present invention, a plurality of corrected images are generated by replacing the non-target portion of the main still image with the background still image (image in focus). Synthetic video data (three-dimensional video) can be generated by joining a plurality of corrected images. It is possible to reduce the size of the video without reducing the image quality of the 3D VR video. It is possible to view (transmit and receive) full 3D and high-resolution video (three-dimensional video) at high speed.

(2. Function of 3D VR shooting and 3D VR playback)
Functions of the 3D VR imaging device 100 and the 3D VR playback system 100S according to the embodiment of the present invention will be described with reference to FIG. Here, FIG. 6 is a functional block diagram illustrating an example of functions of the 3D VR imaging device and the 3D VR playback system according to the embodiment of the present invention. Note that the example of the functions shown in FIG. 6 is just an example, and the present invention is not limited to the functions of the 3D VR imaging device and the 3D VR playback system shown in FIG.

As shown in FIG. 6, in block B00, the I/F (interface means) performs input/output of information (signals) between the three-dimensional VR imaging apparatus 100 and the outside in accordance with operation instructions from the control means 10. FIG. Here, the I/F (interface means) may be one through which information is input by a user, user, viewer, or the like of the three-dimensional VR imaging device 100 .

In block B01, the control means 10 instructs each component of the 3D VR camera 100 and the 3D VR playback system 100S to operate, and controls the operations of the 3D VR camera 100 and the 3D VR playback system 100S. Here, the control means 10 can control the operation of the 3D VR imaging device 100 based on the information input using the I/F. In addition, the control means 10 may store information such as operating conditions, operating states, and operating results in a built-in storage unit.

In block B02, the photographing means 11 and the rotation support means 12 start the photographing operation in accordance with the photographing instruction (operation instruction of the control means 10). At this time, the photographing means 11 outputs the photographed moving image data (background material data, main material data, etc.) to the moving image editing means 13 .

In block B03, the moving image editing means 13 edits the moving image data captured by the imaging means 11 according to the editing instruction (operation instruction of the control means 10). Here, the moving image editing means 13 uses the still image generation unit 13a to generate a still image from the background material data and the main material data, and uses the position information detection unit 13b to generate a plurality of still images generated by the still image generation unit 13a. Information on the rotational position of the rotation support means 12 at the time of photographing is detected for the image, and the background material data and the main material data are synthesized using the moving image synthesizing section 13c to generate synthetic moving image data. After that, the moving image editing means 13 outputs the generated synthetic moving image data to the storage means 14 .

In block B04, the storage means 14 stores the moving image data shot by the shooting means 11 and the synthesized moving image data generated by the moving image editing means 13 in accordance with the storage instruction (operation instruction of the control means 10). Further, the transmitting/receiving means 15 transmits the data (synthesized moving image data, etc.) stored in the storing means 14 to the displaying means 21 (VR goggles, etc.) via the moving image storing means 23 and the moving image communication means 24 .

In block B<b>05 , the moving image communication means 24 receives the synthesized moving image data and the like transmitted from the 3D VR imaging device 100 . Also, the moving image storage means 23 stores the synthesized moving image data and the like transmitted from the 3D VR imaging device 100 .

In block B06, the reproduction condition detection means 22 detects information regarding reproduction, such as position information data, according to the display instruction (operation instruction of the control means 10). Further, the reproduction condition detection means 22 outputs the detection result to the moving image storage means 23 .

In block B07, the display means 21 displays a three-dimensional moving image (composite moving image data) corresponding to the result detected by the reproduction condition detecting means 22. FIG.

(3. Three-dimensional VR shooting method, its program and its recording medium)
The operation (3D VR imaging method) of the 3D VR imaging device 100 according to the embodiment of the present invention will be described. It should be noted that, in the following description, mainly different parts from the 3D VR imaging device 100 (3D VR playback system 100S) according to the above-described embodiment will be described.

The operation (three-dimensional VR shooting method) of shooting a three-dimensional video by the three-dimensional VR shooting device 100 (FIG. 1) will be described with reference to FIG. Here, FIG. 7 is a flow chart illustrating an example of the operation (3D VR imaging method) of the 3D VR imaging device 100 according to the embodiment of the present invention. The operation of the 3D VR imaging device shown in FIG. 7 is an example, and the present invention is not limited to the operation of the 3D VR imaging device shown in FIG.

As shown in FIG. 7, in step S701, the 3D VR imaging device 100 first uses the control means 10 (FIG. 1) to start the operation of reproducing video. The control means 10 can start the operation of reproducing, for example, according to information input from the outside of the 3D VR imaging device 100 (user or the like). After that, the 3D VR imaging device 100 proceeds to step S702. Note that the 3D VR imaging device 100 may use the storage means 14 (FIG. 1) to store imaging information and the like in advance before starting the operation of imaging a moving image (before step S701).

Next, in step S702, the 3D VR imaging device 100 uses the imaging means 11 (FIG. 1) to rotate the imaging means 360 degrees using the rotation support means 12 (FIG. 1) while capturing the moving image material for the background. A certain background material data is photographed (background material photographing step). After that, the 3D VR imaging device 100 proceeds to step S703.

Next, in step S703, the 3D VR imaging apparatus 100 uses the imaging means 11 (FIG. 1) to move the imaging target in a substantially circular shape around the imaging means 11 using the rotation support means 12. Main part material data, which is a main part moving image material obtained by photographing an object to be photographed, is photographed while the photographing means is rotated 360 degrees (main part material photographing step). After that, the 3D VR imaging device 100 proceeds to step S704.

In step S704, the 3D VR imaging device 100 uses the still image generation unit 13a (FIG. 1) to generate a plurality of background still images as still images from the background material data captured in step S702 (still background image generation step). After that, the 3D VR imaging device 100 proceeds to step S705.

In step S705, the 3D VR imaging device 100 uses the still image generation unit 13a (FIG. 1) to generate a plurality of main story still images as still images from the main story material data shot in step S703 (main story still images). image generation step). After that, the 3D VR imaging device 100 proceeds to step S706.

In step S706, the 3D VR imaging apparatus 100 uses the position information detection unit 13b to rotate the plurality of background still images generated in step S704 and the plurality of main feature still images generated in step S705. Information about the position is detected (position information detection step). After that, the 3D VR imaging device 100 proceeds to step S707.

In step S707, the 3D VR imaging device 100 uses the moving image synthesizing unit 13c to match the plurality of background still images with the plurality of main still images based on the information about the rotational position detected in step S706. (image matching step). After that, the 3D VR imaging device 100 proceeds to step S708.

In step S708, the 3D VR imaging device 100 uses the moving image synthesizing unit 13c to divide the main still image into a target portion where the shooting target is shot and a non-target portion where the shooting target is not shot. Divide. (image segmentation step). After that, the 3D VR imaging device 100 proceeds to step S709.

In step S709, the three-dimensional VR imaging device 100 uses the moving image synthesizing unit 13c to join together a plurality of corrected images obtained by replacing the non-target portions of the main still image divided in step S708 with the background still image, Synthetic moving image data is generated (synthetic moving image generation step). After that, when the 3D VR imaging device 100 determines to end the imaging operation, the 3D VR imaging device 100 proceeds to "END" in the figure and ends the operation.

As described above, the operation of shooting a 3D moving image (3D VR shooting method) according to the present invention provides the same effect as the 3D VR shooting device 100 (described above) according to the present invention.

A program for causing a computer to execute the 3D VR imaging method according to the present invention uses the 3D VR imaging apparatus 100 according to the above-described embodiment, (1) uses imaging means, uses rotation support means, (2) a background material photographing step of photographing background material data, which is a background moving image material, while rotating the photographing means by 360 degrees; (3) a main material shooting step of shooting main material data, which is a moving image material for a main part, obtained by shooting an object to be shot while rotating the shooting means by 360 degrees using the rotating support means; (4) a background still image generation step of generating a plurality of background still images from the background material data as still images; (5) a main still image generation step of generating an image; and (6) using a moving image synthesizing unit, a plurality of background still images and a plurality of main still images based on the information about the rotational positions detected by the position information detecting unit. and (7) using the moving image synthesizing unit, next, in the still image for the main part, a target part where the shooting target is shot and a non-shooting target part where the shooting target is not shot. (8) Using the moving image composition unit, a plurality of corrected images obtained by replacing the non-target portions of the still image for the main part with the still image for the background are joined together to obtain synthesized moving image data. and a synthetic moving image generating step of generating

According to the above program, an effect equivalent to that of the 3D VR imaging device 100 (described above) according to the present invention can be obtained.

The present invention may also be a computer-readable recording medium recording the above program. Computer-readable media such as flexible disks, CD-ROMs, DVDs, and memory cards can be used as recording media. Furthermore, the present invention may be a transferable medium capable of transmitting the program via a network such as the Internet.

As described above, the embodiments according to the present invention have been described, but the present invention is not limited to the above embodiments. That is, the present invention can be variously modified, changed, or arbitrarily modified based on the contents described in the claims.

100S: 3D VR playback system 100: 3D VR imaging device
10: Control Means 11: Photographing Means 12: Rotational Support Means 13: Moving Image Editing Means 13a: Still Image Generating Unit 13b: Position Information Detecting Unit 13c: Moving Image Synthesizing Unit 14: Storage Means 15: Transmission/Reception Means 21: Display Means (VR Goggles Such)
21a: moving image playback unit 21b: temporary moving image recording unit 22: playback condition detection means (VR goggles, etc.)
23: Video storage means (VR goggles, etc.)
24: Video communication means (VR goggles, etc.)
Ma: Movement direction (360 degree rotation)
ST: Object to be photographed

Claims (8)

A 3D VR shooting device for shooting a 3D video displaying a first video and a second video shot simultaneously at a position with parallax from the first video,
a photographing means for photographing the first moving image and the second moving image;
a rotation support means for supporting the photographing means so as to be rotatable by 360 degrees in a horizontal direction;
and video editing means for editing the video shot by the shooting means,
The photographing means photographs background material data, which is a background moving image material, while rotating the photographing means 360 degrees using the rotation support means, and the object to be photographed is arranged in a substantially circular shape around the photographing means. photographing main material data, which is a moving image material obtained by photographing the object to be photographed while rotating the photographing means by 360 degrees using the rotating support means in the arranged state;
The motion picture editing means includes a still image generation section for generating a still image from the background material data and the main material data, and rotation of the rotation support means during shooting of the plurality of still images generated by the still image generation section. A position information detection unit that detects information about a position, and a moving image synthesizing unit that synthesizes the background material data and the main material data to generate synthetic moving image data,
A three-dimensional VR imaging device characterized by: The still image generation unit generates a background still image using the background material data, generates a main story still image using the main story material data,
The moving image synthesis unit
First, each of the plurality of background still images and the plurality of main part still images are compared based on the information about the rotational position detected by the position information detection unit,
Next, dividing the main still image into a target portion in which the object to be photographed is photographed and a non-target portion in which the object to be photographed is not photographed,
Next, connecting a plurality of corrected images in which the non-target portion of the main still image is replaced with the background still image to generate the synthesized moving image data;
3. The three-dimensional VR imaging device according to claim 1, characterized by: The still image generation unit generates a background still image using the background material data, generates a main story still image using the main story material data,
The moving image synthesis unit
First, identifying a plurality of background feature points on the plurality of background still images, identifying a plurality of feature points on the plurality of main feature still images,
Next, the plurality of background still images and the plurality of main feature still images are matched based on the plurality of background feature points and the plurality of main feature points,
Next, dividing the main still image into a target portion in which the object to be photographed is photographed and a non-target portion in which the object to be photographed is not photographed,
Next, connecting a plurality of corrected images obtained by replacing the non-target portion of the main still image with the background still image to generate the composite moving image data;
3. The three-dimensional VR imaging device according to claim 1, characterized by: A three-dimensional VR playback system including the three-dimensional VR imaging device according to any one of claims 1 to 3,
display means for displaying the synthesized moving image data generated by the moving image synthesizing unit to a user;
reproduction condition detection means for detecting information about the display position of the display means that moves when the user changes the viewpoint;
The display means displays the synthetic moving image data corresponding to the information regarding the display position detected by the reproduction condition detection means.
A three-dimensional VR playback system characterized by: further comprising moving image communication means for transmitting and receiving data between the three-dimensional VR imaging device and the display means in a wired and/or wireless manner,
The display means uses the moving image communication means to receive and display the synthesized moving image data generated by the moving image synthesizing unit.
The three-dimensional VR playback system according to claim 4, characterized by: A 3D VR shooting device for shooting a 3D video displaying a first video and a second video shot simultaneously at a position with parallax from the first video,
a photographing means for photographing the first moving image and the second moving image;
a rotation support means for supporting the photographing means so as to be rotatable by 360 degrees in a horizontal direction;
and video editing means for editing the video shot by the shooting means,
The photographing means photographs background material data, which is a background moving image material, while rotating the photographing means 360 degrees using the rotation support means, and the object to be photographed is arranged in a substantially circular shape around the photographing means. photographing main material data, which is a moving image material obtained by photographing the object to be photographed while rotating the photographing means by 360 degrees using the rotating support means in the arranged state;
The motion picture editing means includes a still image generation section for generating a still image from the background material data and the main material data, and rotation of the rotation support means during shooting of the plurality of still images generated by the still image generation section. A position information detection unit that detects information about a position, and a moving image synthesizing unit that synthesizes the background material data and the main material data to generate synthetic moving image data,
Using a three-dimensional VR imaging device characterized by
A three-dimensional VR shooting method for shooting the three-dimensional video,
a background material photographing step of photographing background material data, which is a background moving image material, using the photographing means while rotating the photographing means 360 degrees using the rotation support means;
For the main part, in which the object to be photographed is photographed using the photographing means while rotating the photographing means 360 degrees using the rotation support means in a state in which the object to be photographed is arranged in a substantially circular shape around the photographing means. a main material shooting step for shooting main material data, which is a moving image material;
a background still image generating step of generating a plurality of background still images from the background material data as the still images using the still image generating unit;
a main story still image generating step of generating a plurality of main story still images from the main story material data as the still images, using the still image generating unit;
A position information detection step of detecting, by using the position information detection unit, information regarding the rotational positions at the time of photographing for each of the plurality of background still images and the plurality of main part still images generated by the still image generation unit. and,
An image collation step of collating the plurality of background still images and the plurality of main part still images, using the moving image synthesizing unit, based on the information about the rotational position detected by the position information detecting unit. and,
Next, using the moving image synthesizing unit, an image segmentation step of dividing the main still image into a target portion in which the shooting target is shot and a non-target portion in which the shooting target is not shot;
a synthetic moving image generation step of generating the synthetic moving image data by connecting a plurality of corrected images obtained by replacing the non-target portion of the main still image with the background still image, using the moving image synthesizing unit; A three-dimensional VR imaging method characterized by comprising: A program for causing a computer to execute the three-dimensional VR imaging method according to claim 6. A computer-readable recording medium recording the program according to claim 7 .

Images