JP2019009699A - Multi-viewpoint video acquisition device, multi-viewpoint video output device, and multi-viewpoint video system - Google Patents

Abstract

To provide a multi-viewpoint video acquisition device, a multi-viewpoint video output device, and a multi-viewpoint video system.SOLUTION: A multi-viewpoint video acquisition device (MAIA) includes: a mirror support body (SPT) that surrounds at least one part of the circumference of a subject; a plurality of mirrors (CM) supported by a subject side (inner side) of the mirror support body; and an imaging part (CAM) that is surrounded by the mirror support body, and photographs the subject viewed from a different view point of the plurality of mirrors by photographing the plurality of mirrors at the same time. The multi-viewpoint video acquisition device can easily acquire a video image of a multi-viewpoint. In the video image of the multi-viewpoint, the subject can be three-dimensionally captured by switching a view point of a user during reproduction. The multi-viewpoint video acquisition device enables an observer to three-dimensionally show by a naked eye as if the subject exists here.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a multi-view video acquisition device, a multi-view video output device, and a multi-view video system.

  A technique for stereoscopically presenting a video, that is, a 3D representation, has been developed in the form of outputting a 3D video using a two-lens stereo camera. However, such a conventional 3D expression can stereoscopically observe what is viewed from one viewpoint, but cannot cope with it when viewed from different viewpoints. For example, it is possible to deal only with a seat in a movie theater or when sitting on a chair at home, that is, viewing from a single viewpoint that does not move.

  Recently, when an object (subject) is observed from various viewpoints (multi-viewpoints) in a situation where people move or view from various angles at exhibitions or museums, There is an increasing need for three-dimensional presentation of target objects (video). Also in the field of games and CG, there is an increasing need for moving to an arbitrary viewpoint and presenting (representing) a target object (video) viewed from the moved viewpoint in user operations and game development. I am doing.

  Conventional techniques are being developed to generate multi-viewpoint images instead of such a single viewpoint. For example, “Imaging system, imaging method, image reproducing device, and program” (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2015-185965

This prior art
“The image data generation system 1 switches a plurality of cameras 10 installed at equal intervals on a concentric circle C with the subject O as a central point and a radius R, and output switching of images captured by the cameras 10. And an image data generation device 20 that generates image data by performing image analysis on the image output from the one camera 10 and subject O in the image. Are detected and switched from the specific camera 10 to the images output from the cameras 10 in order along a predetermined circumferential direction (for example, clockwise) in the concentric circle C.
It is a technique.

  In the above-described prior art, “multiple cameras” arranged around the subject are essential, and it is necessary to control and synchronize each camera, resulting in a complicated and expensive configuration. In order to solve the above-mentioned problems, the inventors of the present application easily obtain a multi-viewpoint video by using a single camera by arranging a plurality of mirrors in the surroundings, and further acquiring in this way. I came up with a technique to reproduce video from any viewpoint from the multi-view video.

  Accordingly, an object of the present invention is to provide a multi-view video acquisition device, a multi-view video output device, and a multi-view video system.

A multi-viewpoint video acquisition device according to a first invention is:
A mirror support surrounding at least a part of the periphery of the subject;
A plurality of mirrors supported on the subject side (inside) of the mirror support;
An imaging unit that is surrounded by the mirror support and images the subject viewed from different viewpoints of each of the plurality of mirrors by simultaneously imaging the plurality of mirrors;
Have

The multi-viewpoint video acquisition device according to the second invention is
Each of the plurality of mirrors is
A plane mirror,
In consideration of the relationship between the position of the imaging unit and the position of the subject, the imaging unit is arranged with an angle adjusted so that the subject reflected in each of the mirrors is reflected.
It is characterized by that.
According to this configuration, since the optical system is a plane mirror, angle adjustment is necessary, but conversion processing from a convex surface to a flat surface at the time of output (during reproduction) is unnecessary.

A multi-viewpoint video acquisition device according to a third invention is
The plurality of mirrors are
A convex mirror,
A multi-view video acquisition device characterized by that.
According to this configuration, since the optical system is a convex mirror, it is not necessary to adjust the angle so that the subject is reflected, and there is no need to adjust the placement of the subject. do it.

A multi-viewpoint video output device according to a fourth invention is
An acquisition unit that acquires videos (for example, videos shot by the multi-view video acquisition device MAIA) shot from a plurality of viewpoints (directions and angles);
A storage unit that stores the acquired video as original video data, and stores input system coordinate information including coordinates of the imaging device at the time of shooting, a plurality of viewpoint coordinates (mirror coordinates), subject coordinates, and distortion correction information;
Separate video data for each viewpoint that generates multiple separate video data for each viewpoint by referring to the input system coordinate information and separating the original video data into video data for each viewpoint and correcting the distortion of the video by the mirror and the imaging device lens. A generator,
A viewpoint position sensing unit for detecting the position of the observer;
A plurality of display units;
A playback unit that causes the display unit corresponding to the detected position of the observer to display the video of the separated video data for each viewpoint corresponding to the position;
Have
There may be one display unit. In that case, it is preferable to display an image in a corresponding partial area of the display unit.

A multi-viewpoint video output apparatus according to the fifth invention is
The plurality of display units are
It is either a special plate, mirror, or display (liquid crystal display, plasma display, etc.)
It is characterized by that.

The multi-viewpoint video system according to the sixth invention is:
A multi-view video system including a multi-view video acquisition device and a multi-view video output device,
The multi-viewpoint video acquisition device includes:
A mirror support surrounding at least a part of the periphery of the subject;
A plurality of mirrors supported on the subject side (inside) of the mirror support;
An imaging unit that is surrounded by the mirror support and images the subject viewed from different viewpoints of each of the plurality of mirrors by simultaneously imaging the plurality of mirrors;
Have
The multi-view video output device is:
An acquisition unit for acquiring video captured by the multi-viewpoint video acquisition device;
A storage unit that stores the acquired video as original video data, and stores input system coordinate information including coordinates of the imaging device at the time of shooting, a plurality of viewpoint coordinates (mirror coordinates), subject coordinates, and distortion correction information;
A separate viewpoint-specific video data generation unit that generates a plurality of viewpoint-specific separated video data by referring to input system coordinate information and separating original video data into viewpoint-specific video data and correcting distortion caused by a mirror and an imaging device lens When,
A viewpoint position sensing unit for detecting the position of the observer;
Multiple display units (projection units (mirrors, special plates, etc.) or display displays (liquid crystal displays, plasma displays, etc.)
A playback unit that causes the display unit corresponding to the detected position of the observer to display (project) the video of the separated video data for each viewpoint corresponding to the position;
Have

The multi-viewpoint video system according to the seventh invention is
The plurality of display units are
It is either a special plate, mirror, or display (liquid crystal display, plasma display, etc.)
It is characterized by that.

  As described above, the solution of the present invention has been described as an apparatus, but the present invention can be realized as a method substantially corresponding to these, and the scope of the present invention also includes these. I want you to understand.

  According to the present invention, it is possible to easily acquire multi-viewpoint images. In addition, a multi-viewpoint video can be captured in a stereoscopic manner by switching the user's viewpoint during playback. In addition, the observer can make the subject appear three-dimensional as if it is present with the naked eye.

FIG. 1 is a schematic diagram showing an outline of a multi-view video acquisition apparatus according to an embodiment of the present invention. FIG. 2 is a schematic diagram for explaining a video acquired by the multi-view video acquisition device shown in FIG. FIG. 3 is a flowchart illustrating an example of processing executed by the multi-view video acquisition apparatus illustrated in FIG. FIG. 4 is a block diagram showing an outline of a multi-view video acquisition apparatus according to an embodiment of the present invention. FIG. 5 is a block diagram showing an overview of a multi-view video acquisition device (input system) and a multi-view video acquisition device (output system) according to an embodiment of the present invention. FIG. 6 is a schematic diagram showing an input system and an output system for each viewpoint in the multi-view video acquisition apparatus and multi-view video acquisition apparatus shown in FIG. FIG. 7 is a flowchart showing an example of processing executed by the multi-view video output apparatus shown in FIG. FIG. 8 is a schematic diagram showing an input system and an output system for each viewpoint in the multi-view video acquisition apparatus and multi-view video acquisition apparatus shown in FIG. FIG. 9 is a flowchart showing an example of processing executed by the multi-view video output apparatus shown in FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a schematic diagram showing an outline of a multi-view video acquisition apparatus according to an embodiment of the present invention.
As shown in the figure, the multi-view video acquisition device MAIA
A mirror support SPT surrounding at least a part of the periphery of the subject SBJ;
A plurality of mirrors CM supported on the subject side (inside) of the mirror support SPT;
An imaging unit CAM that captures the subject viewed from different viewpoints of each of the plurality of mirrors by being simultaneously photographed by the plurality of mirrors surrounded by a mirror support SPT;
A subject support section SSPT that supports the subject SBJ (when the subject is a person, it is preferable to use a chair);
Have The plurality of mirrors CM are preferably convex mirrors. The imaging unit CAM preferably uses one or a plurality of fisheye lenses that can take a panoramic image of the surroundings.

  According to this configuration, the input device (video) for compact and simple 3D expression by capturing the video three-dimensionally by combining 2D video (planar video) with different angles, that is, viewpoints. Acquisition device) can be realized.

  FIG. 2 is a schematic diagram for explaining a video acquired by the multi-view video acquisition device shown in FIG. As shown in the drawing, a plurality of convex mirrors CM1, CM2, CM3, CM4 are arranged around the subject SBJ, and the subject SBJ reflected on the convex mirror can be photographed by the imaging unit CAM. As a result, it is possible to acquire videos of a plurality of subjects viewed from different angles (viewpoints). In addition, in order to acquire such images from a plurality of viewpoints, a large number of imaging units are required. However, in this configuration, the images from a plurality of mirrors are concentrated on a single imaging unit, and the images are collected. Acquisition is realized. In this example, the convex mirror CM1 inputs (acquires) an image of the subject SBJ viewed from the direction A1 (angle A1) to the imaging unit CAM. Similarly, the convex mirrors CM2, CM3, CM4 input (acquire) the image of the subject SBJ viewed from the directions A2 (angle A2), A3 (angle A3), and A4 (angle A4) to the imaging unit CAM. Of course, the convex mirror CM7 arranged behind the subject SBJ can input (acquire) the image of the subject SBJ viewed from the direction A7 (angle A7) to the imaging unit CAM. In this explanation, the case of acquiring the image in the horizontal direction is shown, but the image can be acquired in the same way in all directions (angles) where the mirror is arranged, such as upward and downward, or diagonally upward and diagonally downward. It is.

  FIG. 3 is a flowchart illustrating an example of processing executed by the multi-view video acquisition apparatus illustrated in FIG. As shown in the figure, in step S11, the photographic subject is placed at the center of the mirror support. Next, in step S12, the imaging unit shoots each mirror at the same time, thereby shooting a photographic subject viewed from a different viewpoint of each mirror. Finally, in step S13, the storage unit stores video information including video of the photographic subject viewed from multiple viewpoints or transmits the video information to the outside.

FIG. 4 is a block diagram showing an outline of a multi-view video acquisition apparatus according to an embodiment of the present invention. As shown in the figure, the multi-view video acquisition device MAIA
A mirror support SPT surrounding at least a part of the periphery of the subject SBJ;
A plurality of mirrors CM supported on the subject side (inside) of the mirror support SPT;
An imaging unit CAM that captures the subject viewed from different viewpoints of each of the plurality of mirrors by being simultaneously photographed by the plurality of mirrors surrounded by a mirror support SPT;
A subject support section SSPT that supports the subject SBJ (when the subject is a person, it is preferable to use a chair);
A storage unit MEM for storing captured video information;
A playback unit PL (output unit) that plays back or outputs video information;
Have The plurality of mirrors CM are preferably convex mirrors. The imaging unit CAM preferably uses one or a plurality of fisheye lenses that can take a panoramic image of the surroundings.

  In the above-described embodiments, the present invention has been described as an input multi-view video acquisition device, but the present invention can also be realized as an output (playback) multi-view video output device. Hereinafter, embodiments of the output system (reproduction system) will be described with reference to the drawings.

FIG. 5 is a block diagram showing an overview of a multi-view video acquisition device (input system) and a multi-view video acquisition device (output system) according to an embodiment of the present invention. As shown in the figure, the multi-view video output device MAIO
An acquisition unit ACQ that acquires videos (for example, videos shot by the multi-view video acquisition device MAI) shot from a plurality of viewpoints (directions and angles);
A storage unit MEM that stores the acquired video as original video data, and stores input system coordinate information including the coordinates of the photographing device at the time of photographing, a plurality of viewpoint coordinates (mirror coordinates), subject coordinates, curvature of each mirror, and the like. When,
A viewpoint-separated video data generation unit IMG that generates a plurality of viewpoint-separated video data, with reference to input system coordinate information, separated into video data for each viewpoint, and corrected for distortion caused by the convex mirror and the imaging device lens;
A viewpoint position sensing unit PS for detecting the position of the observer;
Projector PRJ (mirror, special plate, etc.)
A playback unit PL for projecting separated viewpoint-specific video data corresponding to the position onto a projection unit corresponding to the detected position of the observer;
Have

  In this embodiment, the display unit has been described as a projection unit that projects an image. However, a display display such as a liquid crystal display or a plasma display may be used. Further, there may be one or more projection units and display displays. In that case, an image is displayed in a corresponding partial area of the display unit.

  As shown in the figure, a multi-view video system MAIS can be configured by a multi-view video acquisition device (input system) and a multi-view video acquisition device (output system).

  FIG. 6 is a schematic diagram showing an input system and an output system for each viewpoint in the multi-view video acquisition apparatus and multi-view video acquisition apparatus shown in FIG. As shown in the figure, in the input system AQS of the multi-viewpoint video acquisition device, a subject SBJ is photographed in a pseudo direction A1 (angle A1) using a convex mirror disposed in the direction. Similarly, the subject SBJ is photographed in a pseudo direction A2 (angle A2) using a convex mirror disposed in the direction.

  Next, the output system will be described. In the output system VRS, the viewpoint position sensing unit PS1 detects the position of the observer OBS1 located in the direction A1 ′ corresponding to the direction A1. In response to this detection, the playback unit PL outputs the separated video data for each viewpoint corresponding to the position to the mirror display MD1 (projection unit) arranged in the direction A1 ′ of the projection unit corresponding to the detected position of the observer. A virtual image VI1 of the subject SBJ-G is projected. Again, in this embodiment, a “mirror display” that is a projection unit that projects an image has been described as an example of the display unit. However, any display device may be used, and a display display such as a liquid crystal display may be used. Or a plasma display may be used. Further, there may be one or more projection units and display displays. In that case, an image is displayed in a corresponding partial area of the display unit. The display device may be 2D or 3D.

  Similarly, the viewpoint position sensing unit PS2 detects the position of the observer OBS2 located in the direction A2 ′ corresponding to the direction A2. In response to this detection, the playback unit PL outputs the separated video data for each viewpoint corresponding to the position to the mirror display MD2 (projection unit) arranged in the direction A2 ′ of the projection unit corresponding to the detected position of the observer. A virtual image VI2 of the subject SBJ-G is projected.

  By changing the viewpoint from A1 ′ to A2 ′ while the observer is walking, the viewpoint of the subject is changed even with the naked eye, so that there is an effect that the subject exists there and looks three-dimensional. Although the viewpoint position sensing unit has been described in the form of two, the position may be sensed three-dimensionally by one sensor. Similarly, although two mirror displays are shown, one projection unit may be used.

  To reiterate, in this configuration, when an image from a monitor or a projector is displayed as a virtual image by a mirror or special plate, if the observer changes the viewpoint of viewing the monitor, the image to be output is corresponding to that. It also switches, making it appear three-dimensional as if the subject is there, even with the naked eye.

  FIG. 7 is a flowchart showing an example of processing executed by the multi-view video output apparatus shown in FIG. As shown in the figure, in step S21, the acquisition unit acquires videos shot from a plurality of viewpoints (directions and angles) (for example, videos shot by a multi-viewpoint video acquisition device). Next, in step S22, the storage unit stores the acquired video as original video data, and includes the coordinates of the photographing device at the time of photographing, a plurality of viewpoint coordinates (mirror coordinates), subject coordinates, and distortion correction information. Stores input system coordinate information. Next, in step S23, the viewpoint-specific separated video data generation unit refers to the input system coordinate information, separates the original video data into viewpoint-specific video data, and corrects distortion caused by the mirror and the imaging device lens. A plurality of viewpoint-separated video data is generated. Next, in step S24, the viewpoint position sensing unit detects the position of the observer. Finally, in step S25, the reproduction unit projects the separated video data for each viewpoint corresponding to the position onto the projection unit (mirror or the like) corresponding to the detected position of the observer.

  FIG. 8 is a schematic diagram showing an input system and an output system for each viewpoint in the multi-view video acquisition apparatus and multi-view video acquisition apparatus shown in FIG. As shown in the figure, the input system AQS and the output system VRS of the multi-view video acquisition device are the same as those in FIG. In this example, the output system VRS1 projects an interpolated video at an arbitrary virtual viewpoint between viewpoints. As shown in the figure, in the output system VRS1, the viewpoint position sensing unit PS12 detects the position of the observer OBS12 located in the direction A12 ′ corresponding to between the direction A1 and the direction A2. In response to this detection, the interpolated and generated viewpoint-specific separated video data corresponding to the position is reproduced by the playback unit PL, and the mirror display is arranged in the projection unit direction A12 ′ corresponding to the detected position of the observer. The virtual image VI2 of the subject SBJ-G is projected onto the MD 12 (projection unit). In the case of three or more viewpoints, an arbitrary virtual viewpoint may be a point positioned in a polygon connecting three or more points. It is also possible to generate an interpolated video by a so-called extrapolation method that makes a virtual viewpoint a point outside the straight line, not between two viewpoints. However, what can be easily applied to an interpolated image by extrapolation is limited to a subject whose shape is relatively simple, such as a rectangular parallelepiped, and whose shape can be easily predicted by changing the viewpoint by mathematical processing.

  As a result, the observer can observe the subject more realistically and more realistically by viewing the interpolated virtual video. Two or more interpolated images can be provided between two points. Moreover, it is possible to generate one or two or more interpolated videos for a virtual viewpoint between three points such as a triangle. The interpolated video is preferably generated by applying CG, CAD, or a mathematically known technique. As in the system of FIG. 5, it is possible to project in real time by constructing a system in which an input system and an output system are combined.

  FIG. 9 is a flowchart showing an example of processing executed by the multi-view video output apparatus shown in FIG. As shown in the figure, in step S31, the acquisition unit acquires videos shot from a plurality of viewpoints (directions and angles) (for example, videos shot by a multi-viewpoint video acquisition device). Next, in step S32, the storage unit stores the acquired video as original video data, and includes the coordinates of the photographing device at the time of photographing, a plurality of viewpoint coordinates (mirror coordinates), subject coordinates, and distortion correction information. Stores input system coordinate information. Next, in step S33, the viewpoint-specific separated video data generation unit refers to the input system coordinate information, separates the original video data into viewpoint-specific video data, and corrects image distortion caused by the mirror and the imaging device lens. A plurality of viewpoint-specific separated video data is generated, and based on the viewpoint-separated video data of two or more viewpoints, video data of an arbitrary virtual viewpoint between the two or more viewpoints is converted into viewpoint-separated video. Generate further as data. This step S33 is different from step S23 in FIG. 7, and in this step, an interpolated video is generated. Next, in step S34, the viewpoint position sensing unit detects the position of the observer. Finally, in step S35, the reproduction unit projects the separated video data for each viewpoint corresponding to the position onto the projection unit (such as a mirror) corresponding to the detected position of the observer. As described above, in this flow, the interpolated video of the virtual viewpoint is also projected, so that the observer who is the user can appreciate a more realistic and realistic stereoscopic video. Since an image corresponding to any viewpoint between the viewpoints can be generated as an interpolated image, the system can be realized more precisely according to the viewpoint movement of the observer, and the subject can be viewed more realistically in three dimensions. Become.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, the processes and functions included in each unit and each step can be rearranged so as not to be logically contradictory, and a plurality of means / units and steps can be combined or divided into one. Is possible. Alternatively, it should be noted that some components, functions, processes, steps, etc. of the apparatus, method, program, etc. according to the present invention can be located in a remote server or the like.

A1 ... Angle A2 ... Angle A3 ... Angle A4 ... Angle A7 ... Angle ACQ ... Acquisition unit AQS ... Input system CAM ... Imaging unit CM1 ... Convex mirror CM2 ... Convex mirror CM7 ... Convex mirror IMG ... Per-view separation video data generation unit MAIA ... Multi-view video acquisition device MAIO ... Multi-view video output device MAIS ... Multi-view video system MD1 ... Mirror display MD2 ... Mirror display MEM ... Storage unit OBS1 ... Observer OBS2 ... Observer PL ... Reproduction unit PRJ ... Projection unit PS ... Viewpoint position Sensing unit PS1 ... viewpoint position sensing unit PS2 ... viewpoint position sensing unit SBJ ... subject SPT ... mirror support SSPT ... subject support unit VI1 ... virtual image VI2 ... virtual image VRS ... output system

Claims (7)

There is a multi-viewpoint video acquisition device,
A mirror support surrounding at least a part of the periphery of the subject;
A plurality of mirrors supported on the subject side of the mirror support;
An imaging unit that is surrounded by the mirror support and images the subject viewed from different viewpoints of each of the plurality of mirrors by simultaneously imaging the plurality of mirrors;
A three-dimensional imaging device. The multi-view video acquisition device according to claim 1,
Each of the plurality of mirrors is
A plane mirror,
In consideration of the relationship between the position of the imaging unit and the position of the subject, the imaging unit is arranged with an angle adjusted so that the subject reflected in each of the mirrors is reflected.
A multi-view video acquisition device characterized by that. The multi-view video acquisition device according to claim 1,
The plurality of mirrors are
A convex mirror,
A multi-view video acquisition device characterized by that. A multi-view video output device,
An acquisition unit for acquiring images shot from a plurality of viewpoints;
A storage unit that stores the acquired video as original video data, and stores input system coordinate information including coordinates of the imaging device at the time of shooting, a plurality of viewpoint coordinates, subject coordinates, and distortion correction information;
By separating the original video data into video data for each viewpoint with reference to the input system coordinate information and correcting a distortion caused by a mirror and an imaging device lens, a plurality of video data for each viewpoint are generated. A generator,
A viewpoint position sensing unit for detecting the position of the observer;
A plurality of display units;
A playback unit that causes the display unit corresponding to the detected position of the observer to display the video of the separated video data for each viewpoint corresponding to the position;
A multi-viewpoint video output device. The multi-view video output device according to claim 4,
The plurality of display units are
Either a special plate, mirror, or display,
A multi-view video output device characterized by that. A multi-view video system including a multi-view video acquisition device and a multi-view video output device,
The multi-viewpoint video acquisition device includes:
A mirror support surrounding at least a part of the periphery of the subject;
A plurality of mirrors supported on the subject side of the mirror support;
An imaging unit that is surrounded by the mirror support and images the subject viewed from different viewpoints of each of the plurality of mirrors by simultaneously imaging the plurality of mirrors;
Have
The multi-view video output device is:
An acquisition unit for acquiring video captured by the multi-viewpoint video acquisition device;
A storage unit that stores the acquired video as original video data, and stores input system coordinate information including coordinates of the imaging device at the time of shooting, a plurality of viewpoint coordinates, subject coordinates, and distortion correction information;
By separating the original video data into video data for each viewpoint with reference to the input system coordinate information and generating a plurality of video data for each viewpoint by correcting the distortion of the video by the mirror and the imaging device lens A video data generation unit;
A viewpoint position sensing unit for detecting the position of the observer;
A plurality of display units;
A playback unit that causes the display unit corresponding to the detected position of the observer to display the video of the separated video data for each viewpoint corresponding to the position;
Having
Multi-view video system. The multi-view video system according to claim 6,
The plurality of display units are
Either a special plate, mirror, or display,
A multi-view video system characterized by this.

Images