JP7068906B2 - Multi-view camera controller and its program - Google Patents

Description

The present invention relates to a multi-view camera control device and a program thereof.

Integral Photography (IP) method is known as one of the stereoscopic image display methods capable of freely visually recognizing a stereoscopic image from an arbitrary viewpoint. In this IP method, a real subject is photographed by one camera, and an element image is generated by using a lens array arranged on a plane.

Further, in the IP method, an element image can be generated by using a three-dimensional model generated from an image taken by a multi-viewpoint robot camera (Non-Patent Documents 1 and 2). In these conventional techniques, the reproduction area of the IP display is set in the real space, and the multi-viewpoint image is captured at the minimum angle of view where the quadrangular pyramid-shaped reproduction area fits by the cooperative control of the multi-viewpoint robot camera, and the element image is generated. It is something to do.

Ikeya et al., "Development of Multiview Robot Camera for Three-Dimensional Restoration", 2017 IEICE General Conference Proceedings, Information and Systems Lectures 2, D-11-3, 2017, p.3 Ikeya et al., "Integral 3D Imaging Technology Using Multiview Robot Camera", Technical Report of the Institute of Image Information and Television Engineers, November 30, 2017

However, the above-mentioned conventional technique has a problem that when the viewer moves within the visual range of the IP display, the stereoscopic image displayed by the IP display is defective. Specifically, in the prior art, as shown in FIG. 8, the reproduction area _ARold of the IP display is set with reference to the position of the master camera _CM . Therefore, in the prior art, when the viewer views the IP display at the edge of the visual range, the reproduction area _ARold may not be sufficiently large and the stereoscopic image may not be displayed correctly.
Since FIG. 8 is a plan view, the quadrangular pyramid-shaped reproduction region AR _old is shown in a trapezoidal shape.

Therefore, it is an object of the present invention to provide a multi-view camera control device and a program thereof that can suppress the loss of the stereoscopic image displayed by the IP stereoscopic image display device.

In view of the above-mentioned problems, the multi-view camera control device according to the present invention is an IP stereoscopic image from a multi-view image taken by a multi-view camera consisting of a preset master camera and a reference camera other than the master camera. It is a multi-view camera control device that controls a multi-view camera to generate an IP stereoscopic image to be displayed by an image display device, and is a parameter input means, a reproduction area setting means, a reference camera control means, and a shooting command means. And, it was configured to include.

According to this configuration, in the multi-view camera control device, the depth direction reproduction range of the IP stereoscopic image display device and the viewing position of the IP stereoscopic image display device are input as parameters by the parameter input means.
The multi-view camera control device obtains a quadrangular pyramid-shaped spatial region that passes through the screen peripheral edge of the IP stereoscopic image display device with the viewing position as the apex at least at the viewing position at the end by the reproduction area setting means. Then, the reproduction area setting means obtains a quadrangular pyramid-shaped viewing area in which the spatial area and the reproduction range in the depth direction overlap. Further, the reproduction area setting means sets the sum of the viewing areas obtained at each viewing position as the reproduction area of the IP stereoscopic image display device.

The multi-view camera control device controls the posture and angle of view of the reference camera so that the reproduction area of the IP stereoscopic image display device fits in the angle of view of the reference camera by the reference camera control means.
The multi-view camera control device commands the reference camera to shoot in synchronization with the shooting of the master camera by the shooting command means.

As described above, since the reproduction area of the IP stereoscopic image display device is set by using the viewing position at the edge, it is larger than the case where it is set only by the position of the master camera. Therefore, even if the viewer views the IP stereoscopic image display device from the edge of the viewing area, it is possible to reduce the situation where the reproduction area of the IP stereoscopic image display device is insufficient.
The multi-view camera control device described above can also be realized by a multi-view camera control program in which a general computer is operated in cooperation as the above-mentioned means.

According to the present invention, even if the viewer views the IP stereoscopic image display device from the edge of the viewing area, the situation where the reproduction area of the IP stereoscopic image display device is insufficient is reduced, and the IP stereoscopic image display device displays. It is possible to suppress the loss of the stereoscopic image.

It is an overall block diagram of the multi-viewpoint image shooting system in embodiment. It is explanatory drawing explaining the arrangement of the multi-viewpoint camera of FIG. It is a block diagram which shows the structure of the multi-viewpoint camera control device of FIG. It is explanatory drawing of the parameter set in the parameter setting means of FIG. It is explanatory drawing of the reproduction area set by the reproduction area setting means of FIG. 3, and is the figure which viewed the reproduction area from the vertical direction. It is explanatory drawing which saw the reproduction area of FIG. 5 from the horizontal direction. It is a flowchart which shows the operation of the multi-viewpoint camera control device of FIG. It is explanatory drawing explaining the setting of the reproduction area in the prior art.

(Embodiment)
[Outline of multi-viewpoint video shooting system]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.
An outline of the multi-viewpoint video imaging system 1 according to the embodiment will be described with reference to FIG.
As shown in FIG. 1, the multi-viewpoint video shooting system 1 shoots a multi-viewpoint video with a multi-viewpoint robot camera (multi-viewpoint camera) C, and uses the shot multi-viewpoint video to produce an IP stereoscopic video (element image). It is what you generate. At this time, the multi-viewpoint video imaging system 1 sets the reproduction area of the IP stereoscopic image display device in consideration of the viewing position of the IP stereoscopic image display device (not shown) that displays the IP stereoscopic image.

First, in the multi-viewpoint video shooting system 1, the viewing position of the IP stereoscopic image display device is set in advance. Subsequently, a cameraman (not shown) operates the master camera _CM in posture and zoom, and sets the parameters at that time in the multi-viewpoint video shooting system 1. Subsequently, in the multi-viewpoint video shooting system 1, the reproduction area of the IP stereoscopic image display device is set in consideration of the viewing position of the IP stereoscopic image display device.
The details of the viewing position and the reproduction area of the IP stereoscopic image display device will be described later.

Subsequently, in the multi-viewpoint video shooting system 1, the reference cameras C _n (C ₁ to C ₆ ) are arranged in a regular hexagonal shape in order to make the baseline B (FIG. 2) between adjacent cameras evenly spaced. At this time, the baseline B between adjacent cameras is calculated based on the permissible parallax angle of the depth estimation process that generates the three-dimensional model.
The subscript n is an integer indicating the number of reference cameras C _n (however, 1 ≦ n ≦ 6).

Subsequently, the multi-viewpoint video imaging system 1 controls the posture and angle of view of the reference camera _Cn so that each vertex constituting the reproduction area of the IP stereoscopic image display device fits in the angle of view of the reference camera _Cn . Then, the multi-viewpoint video shooting system 1 _shoots the subject 90 with the master camera CM and the reference camera _Cn , and generates a multi-viewpoint video. Further, the multi-viewpoint video shooting system 1 generates a three-dimensional model of the subject 90 from the generated multi-viewpoint video, and generates an IP stereoscopic image of the three-dimensional model.

[Overall configuration of multi-view video shooting system]
Hereinafter, the overall configuration of the multi-viewpoint video shooting system 1 will be described.
As shown in FIG. 1, the multi-viewpoint video imaging system 1 includes a multi-viewpoint camera camera C, a multi-viewpoint camera control device 2, a three-dimensional model generation device 3, and an IP stereoscopic image generation device 4.

The multi-viewpoint robot camera C includes one preset master camera _CM and a reference camera _Cn other than the master camera _CM . Here, in the multi-viewpoint robot camera C, the reference camera _Cn is arranged not only in the horizontal direction but also in the vertical direction. In the present embodiment, the multi-viewpoint robot camera C includes a master camera CM and six reference cameras _{Cn arranged} in a regular _hexagonal shape around the master camera CM. At this time, the multi-viewpoint robot camera C is arranged so that the axis connecting the master camera _CM and the reference cameras C ₃ and C ₆ located on the left and right _sides of the master camera CM is horizontal.
Further, the multi-viewpoint robot camera C is connected to the multi-viewpoint camera control device 2 and the three-dimensional model generation device 3 via a cable.

The master camera CM is a shooting camera operated by a _cameraman . Here, the _cameraman operates the posture (pan, tilt), angle of view (zoom), and depth of the master camera CM. Then, the master camera CM outputs the posture, angle of view, and depth at that time to the multi- _view camera control device 2 as parameters. Further, when the _cameraman releases the shutter, the master camera CM captures the subject 90 and outputs the captured image to the three-dimensional model generation device 3. At this time, the master camera CM _notifies the multi-view camera control device 2 of the timing at which the captured image is captured (shooting notification).

The reference camera _Cn is a shooting camera that shoots a shot image used for stereo matching in the depth estimation process when generating a three-dimensional model. That is, the reference camera _Cn is automatically controlled to follow the master camera _CM . In the present embodiment, the reference camera _Cn is mounted on a pan head capable of controlling pan, tilt, and zoom (angle of view). Then, the reference camera _Cn drives the posture (pan, tilt) and the zoom (angle of view) in response to the control signal from the multi-view camera control device 2. Further, the reference camera _Cn takes a picture of the subject 90 from the multi-view camera control device 2 at the timing in which the picture is instructed, and outputs the photographed image to the three-dimensional model generation device 3.

As shown in FIG. 2, the multi-viewpoint robot camera C manually adjusts the baseline B between adjacent cameras. The baseline B is the distance between the master camera CM and each reference camera _Cn , and the distance between adjacent reference _{cameras Cn} . For example, in the multi-viewpoint robot camera C, the master camera CM and each reference camera _Cn are mounted on a _tripod (not shown). In this case, the cameraman or the like refers to the baseline information output by the multi-view camera control device 2, and manually moves the master camera _CM and each reference camera _Cn .

The multi-view camera control device 2 controls the multi-view robot camera C (reference camera _Cn ) when the multi-view robot camera C captures a multi-view image. Details of this multi-view camera control device 2 will be described later.

The three-dimensional model generation device 3 generates a three- _dimensional model of the subject 90 from the captured image of the master camera CM. In the present embodiment, in order to perform the depth estimation process, the three-dimensional model generation device 3 performs stereo matching using the images taken by the master camera _CM and each reference camera _Cn . Then, the 3D model generation device 3 outputs the generated 3D model to the IP stereoscopic image generation device 4.

The IP stereoscopic image generation device 4 generates an IP stereoscopic image from a three-dimensional model generated by the three-dimensional model generation device 3. In the present embodiment, the IP stereoscopic image generation device 4 reproduces the positional relationship between the multi-viewpoint robot camera C, the subject 90, and the IP stereoscopic image display device in a virtual space, and produces a multi-viewpoint image of a 3D point cloud model by oblique projection. Take a picture.

Since the details of the multi-viewpoint robot camera C, the three-dimensional model generation device 3, and the IP stereoscopic image generation device 4 are described in References 1 and 2 below, further description thereof will be omitted.
Reference 1: Iketani et al., "Development of Multiview Robot Camera for Three-Dimensional Restoration", 2017 IEICE General Conference Proceedings, Information and Systems Lectures 2, D-11-3, 2017, p.3
Reference 2: Iketani et al., "Integral Stereoscopic Imaging Technology Using Multiview Robot Camera", Technical Report of the Institute of Image Information and Television Engineers, November 30, 2017

[Configuration of multi-view camera control device]
The configuration of the multi-view camera control device 2 will be described with reference to FIG.
As shown in FIG. 3, the multi-view camera control device 2 includes a parameter setting means (parameter input means) 20, a reproduction area setting means 22, a baseline calculation means 24, a reference camera control means 26, and a shooting command means. 28 and.

The parameter setting means 20 sets (inputs) various parameters necessary for controlling the multi-viewpoint robot camera C. In the present embodiment, the parameter setting means 20 _acquires the posture and angle of view of the master camera CM and the depth from the multi-viewpoint robot camera C as parameters. Further, the cameraman inputs the parameters to the parameter setting means 20 by using an operation means such as a mouse and a keyboard (not shown). For example, the parameters input by the cameraman include the depth direction reproduction range of the IP stereoscopic image display device, the screen size, the pop-out amount, the viewing position, and the viewing range. Then, the parameter setting means 20 outputs various set parameters to the reproduction area setting means 22.

<Explanation of parameters>
Hereinafter, various parameters set in the parameter setting means 20 will be described.
As shown in FIG. 4, the _posture of the master camera CM represents pan / tilt when the _cameraman operates the master camera CM to take a picture of the subject 90.
The angle of view _θ of the master camera CM represents the horizontal and vertical angles of view when the _cameraman operates the master camera CM to shoot the subject 90.
Depth d represents the distance from the master camera CM to the _gazing point G.
The gazing point G is the position of the subject 90. That is, the gazing point G represents the position of the subject 90 that is mainly desired to be displayed as an IP stereoscopic image.
In FIG. 4, the X-axis represents the horizontal direction, the Y-axis represents the vertical direction, and the Z-axis represents the depth direction.

As shown in FIG. 5, the depth direction reproduction range D of the IP stereoscopic image display device represents a range in which the IP stereoscopic image display device can reproduce a stereoscopic image in the depth direction (Z-axis direction). As shown in FIG. 5, the depth direction reproduction range D of the IP stereoscopic image display device represents the period from the front reproduction range to the back reproduction range centered on the screen 5 (lens array).
The screen size of the IP stereoscopic image display device represents the width W and the height H (FIG. 6) of the screen 5.
The pop-out amount Δ of the IP stereoscopic image display device represents the distance at which the gazing point G is displayed away from the screen 5 of the IP stereoscopic image display device in the depth direction.
The viewing area Ω of the IP stereoscopic image display device represents an angle at which the IP stereoscopic image display device can properly display the stereoscopic image.

The viewing position V of the IP stereoscopic image display device represents a position where the viewer views the IP stereoscopic image when displaying the IP stereoscopic image generated by the multi-viewpoint image photographing system 1 on the IP stereoscopic image display device. For example, the viewing position V of the IP stereoscopic image display device can be arbitrarily set in consideration of the pitch and focal length of the element lenses of the lens array included in the IP stereoscopic image display device and the resolution of the IP stereoscopic image display device. For example, a plurality of viewing positions V can be set at equal intervals in the horizontal direction and the vertical direction with reference to the position of the master camera _CM .
In the parameter setting means 20 described above, only the viewing position V used for calculating the spatial area and the viewing area described later may be set.

Returning to FIG. 3, the description of the configuration of the multi-view camera control device 2 will be continued.
The reproduction area setting means 22 obtains a quadrangular pyramid-shaped spatial area that passes through the screen peripheral edge of the IP stereoscopic image display device with the viewing position as the apex at least at the viewing position at the end. Then, the reproduction area setting means 22 obtains a quadrangular pyramid-shaped viewing area in which the spatial area and the reproduction range in the depth direction overlap. Further, the reproduction area setting means 22 sets the sum of the viewing areas obtained at each viewing position as the reproduction area of the IP stereoscopic image display device.

<Setting of reproduction area>
Hereinafter, the setting of the reproduction area of the IP stereoscopic image display device will be described with reference to FIGS. 5 and 6.
Here, as shown in FIGS. 5 and 6, 11 viewing positions of the IP stereoscopic image display device are set in the horizontal direction and the vertical direction with the position of the master camera _CM as a reference (center). .. Further, the central viewing position _VC corresponds to the position of the master camera _{CM, and the viewing positions VL and VR} at both left and right ends are located at the ends of the viewing range Ω. Further, in the present embodiment, each viewing position V is located at the same distance in the depth direction (that is, on the same XY plane).

First, consider the horizontal direction. The reproduction area setting means 22 calculates the scale ratio k using the equation (1) including the depth d, the angle of view θ, the screen size (width W), and the pop-out amount Δ. Specifically, the reproduction area setting means 22 substitutes the depth d, the angle of view θ, the screen size W, and the pop-out amount Δ input from the parameter setting means 20 into the equation (1) to calculate the scale ratio k. This scale ratio k represents the ratio of the scale between the virtual space in which the three- _dimensional model is placed and the real space taken by the master camera CM.

As shown in FIG. 5, the width of the screen 5 is kW, and the left end _EL and the right end _ER are obtained from the width kW of the screen 5. Further, the amount of protrusion in the real space is kΔ, and the reproduction range in the depth direction is kD in the depth direction with the screen 5 as the center.

Next, the reproduction area setting means 22 sets the viewing position V when generating the element image in the real space to be photographed by the master camera _CM . Then, the reproduction area setting means 22 adapts the reproduction range of the IP stereoscopic image display device to the square pyramid-shaped viewing area formed by a straight line connecting each viewing position V and the shape of the screen 5. By synthesizing the viewing areas for each viewing position V in this way, the reproduction area AR of the IP stereoscopic image display device of FIG. 5 is set.

First, consider the leftmost viewing position _VL . In this case, the reproduction area setting means 22 is a quadrangular pyramid that passes through the peripheral edge _E (left end _EL , right end _ER , upper end _EU , lower end ED) of the screen 5 with the viewing position _VL as the apex in the real space. Find the spatial area of the shape. When viewed from the vertical direction as shown in FIG. 5, the spatial region of the viewing position _VL consists of a line segment extending from the viewing position _VL toward the left end _{EL of the screen 5 and a line segment extending from the viewing position VL toward} the screen 5. It is a triangular area surrounded by a line segment extending toward the right end _ER of (shown by a broken line). Then, the reproduction area setting means 22 obtains a viewing area in which the spatial area of the viewing position _VL and the reproduction range in the depth direction overlap (shown by a thick broken line). This viewing area is a quadrangular pyramid-shaped area surrounded by a side surface of a spatial area, a plane corresponding to the front reproduction range, and a plane corresponding to the back reproduction range.
Since FIG. 5 is a plan view, the spatial area that is actually a quadrangular pyramid is shown in a triangular shape, and the viewing area in the shape of a quadrangular pyramid is shown in a trapezoidal shape (the same applies to FIG. 6).

Next, consider the viewing position _VR at the right end. In this case, the reproduction area setting means 22 obtains a space area passing through the peripheral edge E of the screen 5 with the viewing position _VR as the apex in the real space (shown by a dotted chain line). Then, the reproduction area setting means 22 obtains a viewing area in which the spatial area of the viewing position _VR and the reproduction range in the depth direction overlap (shown by the alternate long and short dash line).

Here, the reproduction area setting means 22 obtains the viewing area not only in the horizontal direction but also in the vertical direction. As shown in FIG. 6, the reproduction area setting means 22 obtains a quadrangular pyramid-shaped space area passing through the peripheral edge E of the screen 5 with the viewing position V _U at the upper end as the apex in the real space. When viewed from the horizontal direction as shown in FIG. 6, the spatial area of the viewing position V _U is a line segment extending from the viewing position V _U toward the upper end _{EU of the screen 5 and a line segment extending from the viewing position V U toward} the screen 5. It is a triangular area surrounded by a line segment _extending toward the lower end ED of the above (shown by a broken line). Then, the reproduction area setting means 22 obtains a viewing area in which the spatial area of the viewing position _VD and the reproduction range in the depth direction overlap (shown by a thick broken line). Further, the reproduction area setting means 22 obtains the viewing area of the viewing position V _U at the lower end as well as the viewing position V _D at the upper end.
In FIG. 6, θ _V represents the angle of view in the vertical direction, H represents the height of the screen 5, and Ω _V represents the viewing range in the vertical direction.

In the present embodiment, the reproduction area setting means 22 obtains the spatial area and the visual area only at the viewing positions V ( _VL , _VR , V _U , V _D ) located at the edges in the horizontal direction and the vertical direction. However, another viewing position V may be used. For example, the reproduction area setting means 22 may obtain the space area and the viewing area from the central viewing position VC, or may obtain the space area and the viewing area for each viewing position _V.

Next, the reproduction area setting means 22 sets the sum of the viewing areas obtained at each viewing position V as the reproduction area AR. Specifically, the reproduction area setting means 22 obtains the reproduction area AR by ORing the viewing areas (OR calculation) of each viewing position V. That is, the reproduction area setting means 22 sets the maximum area in which the viewing area obtained at any of the viewing positions V exists as the reproduction area AR so that the reproduction area AR becomes the maximum. As shown in FIGS. 5 and 6, the reproduction region AR is a region having a shape in which the upper and lower surfaces of two quadrangular pyramids are combined at the position of the screen 5, and is composed of 12 vertices _Pi (dots). Illustrated). In order to make the drawings easier to see, only a part of the reference numeral _Pi is shown.

Next, the reproduction area setting means 22 makes the scale of the virtual space that generates the three-dimensional model equal to the real space, and sets the reproduction area AR set in the real space at the same position and scale in the virtual space.
After that, the reproduction area setting means 22 outputs the obtained reproduction area AR and the scale ratio k, and various parameters from the parameter setting means 20 to the baseline calculation means 24.

Returning to FIG. 3, the description of the configuration of the multi-view camera control device 2 will be continued.
The baseline calculation means 24 calculates the baseline B (distance between adjacent cameras) between adjacent cameras in the multi-viewpoint robot camera C from a preset allowable parallax angle.
The permissible parallax angle is the convergence angle of adjacent cameras allowed to generate a highly accurate 3D model, and is, for example, an appropriate value according to the depth estimation process (depth estimation algorithm). It is set in advance.

In the present embodiment, the baseline calculation means 24 calculates the baseline B between adjacent cameras by the equation (2). This equation (2) is expressed by the permissible parallax angle Φ, the depth d included in each parameter, the depth direction reproduction range D, and the pop-out amount Δ. Therefore, the baseline calculation means 24 can calculate the baseline B by substituting the allowable parallax angle Φ and various parameters into the equation (2).

Then, the baseline calculation means 24 outputs the calculated baseline B to the outside (for example, a display (not shown)). Then, the cameraman or the like manually _arranges the master camera CM and each reference camera _Cn according to the baseline B.

Further, the baseline calculation means 24 outputs the reproduction area AR from the reproduction area setting means 22, the scale ratio k, and various parameters to the reference camera control means 26.
In FIG. 3, since the baseline B is set manually, the output from the baseline calculation means 24 to the reference camera _Cn is shown by a broken line.

The reference camera control means 26 controls the posture and the angle of view of the reference camera _Cn so that the reproduction region AR from the baseline calculation means 24 fits in the angle of view of the reference camera _Cn .
First, the reference camera control means 26 acquires the world coordinates of each vertex _Pi of the reproduction region AR. The subscript i is an integer indicating which vertex it is (however, 1 ≦ i ≦ 12).

Next, the reference camera control means 26 has a pan value P _i pan and a pan value P i pan so that all the vertices P _i of the reproduction region AR fit in the angle of view O of each reference camera C _n in the camera coordinate system of each reference camera C _n . The tilt value _Pitilt is calculated. In the present embodiment, as shown in the equation (3), the pan value _O'pan is the pan maximum value Max ( _Pi pan ₎ and the pan when the reference camera Cn faces each vertex Pi of the reproduction region AR. It is obtained from the average with the minimum value Min (Pi pan) (same for tilt _O'tilt ).

Next, the reference camera control means 26 uses the equation (4) to fit the angle of view θ of each reference camera _Cn so that all the _vertices Pi of the reproduction region AR fit in the camera coordinate system after the camera attitude control. ´ is calculated.
In equation (4), W / H represents the aspect ratio of the IP stereoscopic image display device, and if represents a function that performs the operation of the previous stage when the conditional expression of the latter stage is satisfied.

After that, the reference camera control means 26 generates a control signal representing the calculated posture (pan value θ'pan, tilt value _θ'tilt ) and angle of view θ', and outputs the generated control signal to the reference camera Cn. .. In response to this control signal, each reference camera _Cn is driven to take a desired posture and zoomed to an angle of view θ'.

The shooting command means 28 commands the reference camera _Cn whose posture and angle of view are controlled to shoot in synchronization with the shooting of the master camera _CM . In the present embodiment, the shooting command means 28 commands all reference cameras _Cn to shoot the shot image at the timing when the shooting notification is input from the master camera _CM .
Since each means other than the reproduction area setting means 22 is described in the above-mentioned References 1 and 2, further detailed description thereof will be omitted.

[Operation of multi-view camera control device]
The operation of the multi-view camera control device 2 will be described with reference to FIG. 7 (see FIG. 3 as appropriate).
As shown in FIG. 7, the parameter setting means 20 sets various parameters. This parameter includes the _attitude of the master camera CM, the angle of view θ, and the depth d. Further, the parameters include the depth direction reproduction range D of the IP stereoscopic image display device, the screen size (width W, height H), the pop-out amount Δ, the viewing position V, and the viewing area Ω (step S1).

The reproduction area setting means 22 sets the reproduction area of the IP stereoscopic image display device. Specifically, the reproduction area setting means 22 obtains a spatial area passing through the peripheral edge E of the screen 5 with the viewing position V as the apex. Next, the reproduction area setting means 22 obtains a viewing area in which the spatial area of the viewing position V and the reproduction range in the depth direction overlap. Further, the reproduction area setting means 22 sets the sum of the obtained viewing areas as the reproduction area AR (step S2).

The baseline calculation means 24 calculates the baseline B between adjacent cameras in the multi-viewpoint robot camera C. Specifically, the baseline calculation means 24 calculates the baseline B using the above equation (2) based on the allowable parallax angle of the depth estimation process for generating the three-dimensional model (step S3).

The reference camera control means 26 controls the posture and the angle of view of the reference camera _Cn so that the reproduction area AR fits in the angle of view of the reference camera _Cn . Specifically, the reference camera control means 26 uses the above equations (3) and (4) to generate a control signal representing the posture and angle of view _θ'of the reference camera Cn, and the generated control signal. Is output to the reference camera _Cn (step S4).

The multi-view camera control device 2 commands all reference cameras _Cn to shoot a shot image at the timing when a shooting notification is input from the master camera CM by the _shooting command means 28 (step S5).

[Action / Effect]
As described above, the multi-view camera control device 2 according to the embodiment sets the reproduction region AR using the viewing position V at the end, as shown in FIGS. 5 and 6. As a result, the reproduction area AR has a shape in which the upper and lower surfaces of the two quadrangular pyramids are combined at the position of the screen 5, and the front side is larger than the back side. On the other hand, as shown in FIG. 8, the conventional reproduction area AR _old is set only at the position of the master camera CM, so that it has a _quadrangular pyramid-shaped shape, and the front side is narrower than the back side. As described above, the reproduction area AR set by the multi-view camera control device 2 is larger than that of the conventional reproduction area AR _old . Therefore, in the multi-view camera control device 2, even when the viewer views the IP stereoscopic image display device at the edge of the viewing area, it is possible to reduce the situation where the reproduction area AR is insufficient. As a result, the multi-view camera control device 2 can suppress the loss of the stereoscopic image and provide a high-quality IP stereoscopic image.

Further, when the multi-view camera control device 2 uses only the viewing positions _VL , _VR , V _U , and V _D at the edges, the calculation amount required for setting the reproduction area AR is suppressed, and the IP stereoscopic image is displayed. Can be provided quickly. On the other hand, when each viewing position V is used, the multi-view camera control device 2 can set a more accurate reproduction area AR and provide a higher quality IP stereoscopic image.

(Modification example)
Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design changes and the like within a range that does not deviate from the gist of the present invention.
In the above-described embodiment, 11 observation positions are set in the horizontal direction and the vertical direction, but the present invention is not limited to this. This observation position can be arbitrarily set, and a plurality of observation positions can be set in the two-dimensional direction with reference to the position of the master camera.

In the above-described embodiment, the multi-view camera is arranged in a regular hexagonal shape, but the present invention is not limited to this. For example, in the present invention, the multi-view camera may be arranged in a square shape, a circular shape, or a regular polygonal shape. Further, in the present invention, a stereo camera may be used as the multi-view camera.

Although described in the above-described embodiment as manually adjusting the baseline between adjacent cameras, the present invention is not limited thereto. For example, a multi-view camera supports each reference camera and mounts a baseline on an adjustable support mechanism (not shown). Then, the support mechanism refers to the baseline information from the multi-view camera control device 2 and automatically adjusts the baseline between adjacent cameras.

In the above-described embodiment, the multi-view camera control device has been described as independent hardware, but the present invention is not limited thereto. For example, the multi-view camera control device can also be realized by a multi-view camera control program in which hardware resources such as a CPU, a memory, and a hard disk included in a computer are cooperatively operated as the above-mentioned means. This program may be distributed via a communication line, or may be written and distributed on a recording medium such as a CD-ROM or a flash memory.

1 Multi-viewpoint video shooting system 2 Multi-viewpoint camera control device 3 3D model generation device 4 IP stereoscopic image generation device 5 Screen of IP stereoscopic image display device 20 Parameter setting means (parameter input means)
22 Reproduction area setting means 24 Baseline calculation means 26 Reference camera control means 28 Shooting command means 90 Subject AR, AR _old IP reproduction area of stereoscopic image display device C Multi-viewpoint robot camera (multi-viewpoint camera)
_CM Master camera C _n , C ₁ to C ₆ Reference camera E Peripheral E _L Left end E _R Right end E _U Upper end E _D Lower end G Gaze point P, P _i Vertex V, _{VC, V D , VL} , _VR , V _U viewing position

Claims (4)

In order to generate an IP stereoscopic image to be displayed by an IP stereoscopic image display device from a multi-viewpoint image taken by a multi-viewpoint camera consisting of one master camera set in advance and a reference camera other than the master camera. It is a multi-view camera control device that controls the viewpoint camera.
Parameter input means in which the depth direction reproduction range of the IP stereoscopic image display device and the viewing position of the IP stereoscopic image display device are input as parameters, and
At least at the viewing position at the end, a quadrangular pyramid-shaped spatial region that passes through the screen peripheral edge of the IP stereoscopic image display device with the viewing position as the apex is obtained, and the spatial region and the depth direction reproduction range overlap. A reproduction area setting means for obtaining a quadrangular pyramid-shaped viewing area and setting the sum of the viewing areas obtained at each of the viewing positions as a reproduction area of the IP stereoscopic image display device.
A reference camera control means for controlling the posture and angle of view of the reference camera so that the reproduction area of the IP stereoscopic image display device fits in the angle of view of the reference camera.
A shooting command means for instructing the reference camera to shoot in synchronization with the shooting of the master camera,
A multi-view camera control device characterized by being equipped with. In the parameter input means, a plurality of the viewing positions are input in a two-dimensional direction with reference to the position of the master camera.
The multi-view camera control device according to claim 1, wherein the reproduction area setting means calculates the space area and the viewing area for each viewing position. The parameter input means further inputs a depth d representing the distance from the gazing point, which is the position of the subject, to the master camera, an angle of view θ of the master camera, and a pop-out amount Δ of the IP stereoscopic image display device. Being done
The reproduction area setting means uses the equation (1) including the depth d, the angle of view θ, the screen size W of the IP stereoscopic image display device, and the pop-out amount Δ to display the viewing area and the above. Calculate the scale ratio k of the reproduction area,

The multi-view camera control device according to claim 1 or 2, wherein the reproduction region having dimensions corresponding to the scale ratio k is set. A multi-view camera control program for causing a computer to function as the multi-view camera control device according to any one of claims 1 to 3.

Images