JP6606397B2 - Multi-viewpoint robot camera control device and program thereof - Google Patents

Description

  The present invention relates to a multi-viewpoint robot camera control apparatus for controlling a multi-viewpoint robot camera and a program therefor, in order to generate a multi-viewpoint video in which the same subject is displayed from different viewpoints.

  Conventionally, by arranging a robot camera to surround the subject and switching the captured video along the sequence of the robot camera, there is a video representation called multi-view video representation that presents a moment with a moving subject from various viewpoints. Are known. For this multi-view video representation, for example, a multi-view video representation system described in Non-Patent Document 1 is used.

  This multi-view video presentation system is configured by placing a plurality of robot cameras on an electric pan head and controlling the panning and tilting mechanically, so that the multi-view video described above can be used even when an object such as a sports game moves around. The expression is realized in near real time. Specifically, the photographer operates one master camera and shoots so that the subject to be watched always appears in the center of the screen. At this time, this multi-view video expression system automatically controls the electric pan head so that the line of sight of the other slave cameras is congested at the gazing point according to the camera work of the cameraman.

Bullet Time Using Multi-Viewpoint Robotic Camera System, Kensuke Ikeya, CVMP'14 11th European Conference on Visual Media Production (CVMP 2014), 13-14 November 2014

  However, in the multi-view video expression system described above, an operator such as a cameraman has to manually set a gaze point while viewing the images of all the robot cameras divided and displayed on the finder. At this time, in the multi-viewpoint video expression system, there is a problem that the operator operates the master camera so that the subject appears in the center of the screen, and the setting operation of the gazing point places a heavy burden on the operator.

  Accordingly, it is an object of the present invention to provide a multi-viewpoint robot camera control device and a program thereof that can reduce the burden on the operator.

  In view of the above-described problems, the multi-viewpoint robot camera control device according to the present invention is configured by a plurality of robot cameras that photograph the subject in order to generate a multi-viewpoint image in which the same subject is displayed from different viewpoints. A multi-viewpoint robot camera control apparatus that controls a multi-viewpoint robot camera, comprising a camera parameter storage unit, a subject area extraction unit, a finder image generation unit, and a gaze point coordinate calculation unit.

According to this configuration, the multi-viewpoint robot camera control device stores in advance camera parameters representing the position and orientation of one master camera set in advance among the plurality of robot cameras by the camera parameter storage unit.
In the multi-viewpoint robot camera control device, the subject area extraction unit receives a conversion distance image of a photographed image obtained by photographing the subject by the master camera, and a predetermined position (for example, the center of the screen) in the input conversion distance image. A distance value within a range set in advance with reference to a distance value of a reference pixel or a reference distance value that is an average of a distance value of the reference pixel and a distance value of surrounding pixels , and the reference pixel is included A pixel area is extracted as a subject area.

The multi-viewpoint robot camera control device generates a finder image in which a marker indicating the contour of the extracted subject area is drawn on the captured image by the finder image generation unit, and the generated finder image is mounted on the operation interface unit. Output to the viewfinder.
The multi-viewpoint robot camera control device receives an operation signal of the master camera by an operator who views the viewfinder image by the gazing point coordinate calculation unit, and converts the input operation signal with the camera parameter. The world coordinate of the gazing point representing the position of the subject is calculated, and the calculated world coordinate of the gazing point is output to a slave camera which is a robot camera other than the master camera.

In other words, when an operator such as a cameraman captures a subject at a predetermined position (for example, the center of the screen), the multi-viewpoint robot camera control device displays the marker on the finder, so that the operator It becomes easy to recognize. Furthermore, the multi-viewpoint robot camera control device outputs the position of the subject to each slave camera as the world coordinates of the point of interest. Thereafter, each slave camera takes an attitude such that it faces the gazing point input from the multi-viewpoint robot camera control device.
As described above, the multi-viewpoint robot camera control apparatus can control the slave camera so that the operator faces the gazing point without the operator performing the gazing point setting operation.

According to the present invention, the following excellent effects can be obtained.
Since the multi-viewpoint robot camera control device according to the present invention can control the slave camera so that the operator faces the gazing point without the operator performing the gazing point setting operation, the burden on the operator can be reduced.

1 is a schematic diagram of a multi-view video shooting system according to a first embodiment of the present invention. It is explanatory drawing explaining the multiview robot camera of FIG. It is a block diagram which shows the structure of the multiview robot camera control apparatus of FIG. It is a figure which shows an example of a picked-up image. It is a figure which shows an example of a conversion distance image. FIG. 4 is an explanatory diagram for explaining extraction of a subject area by a subject area extraction unit in FIG. 3. It is a figure which shows an example of a finder image. It is a flowchart which shows operation | movement of the multiview video imaging system of FIG. It is the schematic of the multiview video imaging system which concerns on 2nd Embodiment of this invention. It is explanatory drawing explaining the operation interface part of FIG. It is a block diagram which shows the structure of the multiview robot camera control apparatus of FIG. It is explanatory drawing explaining tracking of a gaze point. It is explanatory drawing explaining the setting of the gaze point with respect to another to-be-photographed object. 10 is a flowchart showing the operation of the multi-view video shooting system of FIG. 9. It is a block diagram which shows the structure of the multiview robot camera control apparatus which concerns on 3rd Embodiment of this invention. It is explanatory drawing explaining tracking of a gaze point. 16 is a flowchart showing an operation of the multi-view video shooting system of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, the same means and the same process are denoted by the same reference numerals, and description thereof is omitted.

(First embodiment)
[Configuration of multi-viewpoint video shooting system]
With reference to FIG. 1 and FIG. 2, the configuration of the multi-view video shooting system 1 according to the first embodiment of the present invention will be described.
As shown in FIG. 1, the multi-view video shooting system 1 generates a multi-view video in which the same subject T is displayed from different viewpoints, and includes a multi-view robot camera 10, an operation interface unit 30, and a multi-view robot. A camera control device 40 and a recording device 50 are provided.

  The multi-viewpoint robot camera 10 includes a plurality of robot cameras 20. In the present embodiment, the multi-viewpoint robot camera 10 is composed of seven robot cameras 20.

  The robot camera 20 generates a captured image of the subject T, and includes a camera body 21, a tripod 23 (FIG. 2), and a pan head 25. In the present embodiment, the camera body 21 is a general color camera that captures color images. Of course, the camera body 21 is not limited to a color camera, and may be a monochrome camera, for example.

As shown in FIG. 2, the robot camera 20 is mounted on a tripod 23. The robot camera 20 is driven in a biaxial direction of a pan axis and a tilt axis by a pan head 25 provided above the tripod 23, and can zoom in and out.
The captured image captured by the robot camera 20 is output to the multi-viewpoint robot camera control device 40 and the recording device 50.

In the multi-viewpoint robot camera 10, a distance camera 27 is mounted on the camera platform 25 of any one robot camera 20 (not shown in FIG. 2). Here, the distance camera 27 and the camera body 21 are preferably arranged so that their optical axes are parallel to each other.
Note that the distance camera 27 and the camera body 21 can be corrected in the optical axis direction according to camera parameters, and thus do not necessarily have to be arranged in parallel.

The distance camera 27 captures a distance image (depth map) storing a distance value for each pixel as a moving image. For example, the distance camera 27 includes an active camera that captures a distance image by irradiation of light such as a TOF (Time Of Flight) method, a structured illumination method, or the like. The distance camera 27 may be a passive camera that captures a distance image without irradiating light. As this passive camera, two color cameras are arranged on the left and right, and by searching for the correspondence between each pixel of both color cameras (stereo matching method), a stereo camera that takes a distance image by the principle of triangulation is given. It is done.
A distance image captured by the distance camera 27 is output to the multi-viewpoint robot camera control device 40.

The robot camera 20 on which the distance camera 27 is mounted is called a master camera _20M . The master camera 20 _M, the attitude is controlled in conjunction with the operation of the operation interface 30 provided with a viewfinder 38 which will be described later.
On the other hand, the robot camera 20 the distance camera 27 is not mounted is referred to as the slave camera 20 _S. The posture of the slave camera 20 _S is controlled so as to face the gazing point based on information from the multi-viewpoint robot camera control device 40 (world coordinates of the gazing point).
Note multiview robot camera 10, among the plurality of robot camera 20, any one cars is preset as a master camera 20 _M, the rest is pre-set as a slave camera 20 _S.

As in Figure 2, the operation interface 30 is for operator cameraman or the like to operate the master camera 20 _M, a tripod 31, a camera platform 33, a pan rod 35, the encoder 37 (FIG. 3) And a finder 38.

Here, the operation interface unit 30 is mounted on a tripod 31 together with a finder 38, and a pan bar 35 is extended from a pan 33 provided above the tripod 31. When the operation interface unit 30 is rotated in the pan direction or the tilt direction using the pan bar 35, the master camera _20M is rotated in accordance with the operation. That is, the operator of the photographer or the like, the master camera C _M is to face the subject (gaze point), operating the operation interface unit 30 while looking through the viewfinder 38.

  Further, the operation interface unit 30 measures the current pan value and tilt value in real time by the built-in encoder 37. Then, the operation interface unit 30 outputs the measured pan value and tilt value to the multi-viewpoint robot camera control device 40.

The multi-view robot camera control device 40 controls the multi-view robot camera 10 in order to generate a multi-view video.
Details of the multi-viewpoint robot camera control device 40 will be described later.

The recording device 50 records a captured image input from the multi-viewpoint robot camera control device 40 and generates a multi-viewpoint video from the recorded captured image. For example, the recording device 50 generates a multi-view video using the method described in Reference Document 1 below.
Reference 1: Bullet Time Using Multi-Viewpoint Robotic Camera System, Kensuke Ikeya, CVMP'14 11th European Conference on Visual Media Production (CVMP 2014), 13-14 November 2014

[Configuration of multi-viewpoint robot camera control device]
The configuration of the multi-viewpoint robot camera control device 40 will be described with reference to FIG.
As shown in FIG. 3, the multi-viewpoint robot camera control device 40 includes a conversion parameter storage unit (camera parameter storage unit) 41, a distance image conversion unit 42, a subject area extraction unit 43, a finder image generation unit 44, and a note. And a viewpoint coordinate calculation unit 45.

Conversion parameter storage unit 41 is for previously storing camera parameters of all the robot camera 20, including the master camera 20 _M as transformation parameters. Examples of the conversion parameters include an internal parameter matrix, a rotation matrix, and a parallel progression sequence of each robot camera 20.

  Here, in the multi-viewpoint video shooting system 1, the conversion parameter of each robot camera 20 is obtained by camera calibration, and the result is stored in advance in the conversion parameter storage unit 41. For example, the camera calibration method is described in the above-mentioned reference 1.

The distance image conversion unit 42 converts the distance image input from the distance camera 27 into a converted distance image of a captured image of the master camera _20M . Specifically, the distance image conversion unit 42, a projection using the conversion parameters of the conversion parameter storing section 41 master camera 20 stored in _M, the three-dimensional point cloud of the distance image to the imaging surface of the master camera 20 _M As a result, the image is converted into a conversion distance image having the same number of pixels as the captured image and the same pixel position.

For example, when the distance image has a lower resolution than the captured image, the distance image conversion unit 42 may perform an interpolation process on the converted distance image. Examples of the interpolation processing method include linear interpolation or a method of projecting a polygon set on a distance image onto a captured image.
Thereafter, the distance image conversion unit 42 outputs the converted distance image to the subject region extraction unit 43.

As a result, the captured image and the converted distance image correspond to each other, and the image coordinates match. In other words, the converted distance image is a depth map that represents the distance value of each pixel of the captured image. Here, the distance value represents the distance from the master camera _20M to the subject T, that is, the depth value of the subject T.
The image coordinates are coordinates indicating a position in the image.

For example, it is assumed that the master camera _20M has captured the captured image of FIG. In this case, the conversion distance image is an image in which the distance value is represented by the pixel value (luminance value) of each pixel as shown in FIG. In the converted distance image of FIG. 5, the hatched portion represents the distance value from the master camera _20M to the subject T, and the white portion represents the distance value from the master camera _20M to the background.

  The subject region extraction unit 43 extracts a subject region including the reference pixel with reference to the reference distance value in the reference pixel at a predetermined position in the converted distance image input from the distance image conversion unit 42. In the present embodiment, the reference pixel is a pixel located at the center of the conversion distance image.

  Specifically, the subject area extraction unit 43 obtains a pixel area that has a distance value within a preset range with reference to the distance value of the reference pixel and includes the reference pixel. For example, consider a case where the conversion distance image of FIG. 5 is input. In this case, as shown in FIG. 6, each pixel in the area of the subject T has a distance value within a preset range, and the reference pixel S is included in the area of the subject T. Accordingly, the subject area extraction unit 43 obtains the image area of the subject T.

  At this time, the subject region extraction unit 43 may use an average of distance values between the reference pixel and surrounding pixels located around the reference pixel instead of the distance value of the reference pixel. As a result, the subject region extraction unit 43 can accurately determine the pixel region even when the distance value of the reference pixel is significantly different from the distance value of the surrounding pixels.

  Thereafter, the subject region extraction unit 43 extracts the obtained contour portion of the pixel region from the conversion distance image, and outputs the extracted subject region to the finder image generation unit 44. Further, the subject area extraction unit 43 outputs the reference distance value to the gaze point coordinate calculation unit 45.

The finder image generation unit 44 generates a finder image in which markers are drawn on a captured video. This marker indicates the outline of the subject area input from the subject area extraction unit 43. For example, the finder image generating unit 44, as shown in FIG. 7, the marker M in the captured image input from the master camera 20 _M to generate a viewfinder image combined. Then, the finder image generation unit 44 outputs the generated finder image to the finder 38.
Thereby, a marker indicating the outline of the subject T is displayed on the finder 38 (marker display).

Gazing point coordinate calculation unit 45, to convert the operation interface 30 master camera 20 _M operation signal input from the (pan value and the tilt value) in the conversion parameter, the world coordinates of the gaze point representing the position of the subject T Is calculated.
The world coordinates are three-dimensional coordinates common to the robot cameras 20.

For example, the gazing point coordinate calculation unit 45, a described in reference 1 the formula (2) to (5), a rotation matrix and translation matrix of the master camera 20 _M, pan value and the tilt value of the master camera 20 _M And the reference distance value input from the subject region extraction unit 43, and calculates the world coordinates of the point of interest. Then, the gaze point coordinate calculation unit 45 outputs the calculated world coordinates of the gaze point to the platform 25 of all the robot cameras 20.

  Here, each robot camera 20 calculates a pan value and a tilt value so as to face the world coordinates of the gazing point input from the gazing point coordinate calculation unit 45 using the respective camera parameters. For example, each robot camera 20 substitutes the rotation matrix and parallel progression sequence of each robot camera 20 and the world coordinates of the point of interest in Equations (7) to (10) described in Reference Document 1, and the pan value and Calculate the tilt value. Then, each robot camera 20 takes a posture facing the gazing point by driving the camera platform 25 according to the calculated pan value and tilt value.

[Operation of multi-viewpoint video shooting system]
The operation of the multi-view video shooting system 1 will be described with reference to FIG. 8 (see FIGS. 1 and 3 as appropriate).
Operator, through the operation interface unit 30, so as to capture the subject T in the center of the screen, to operate the master camera 20 _M (step S1).
Note that the processing in step S1 is illustrated by a broken line for manual operation by the operator.

Multiview robot camera controller 40, by the distance image conversion unit 42, using the conversion parameters of the master camera 20 _M, converts the distance image conversion distance image (step S2).
In the multi-viewpoint robot camera control device 40, the subject region extraction unit 43 extracts the subject region from the converted distance image input from the distance image conversion unit 42 (step S3).

Multiview robot camera controller 40, the gazing point coordinate calculation unit 45, a pan value, and the tilt value and the reference distance value of the master camera 20 _M, using the conversion parameters, to calculate the global coordinates of the gaze point (step S4 ).
Each robot camera 20 obtains a pan value and a tilt value from the world coordinates of the gazing point using the conversion parameter, and controls the camera platform (step S5).

In the multi-viewpoint robot camera control device 40, the finder image generation unit 44 generates a finder image in which a marker indicating the contour of the extracted subject area is drawn (step S6).
Master camera 20 _M displays a finder image in the finder 38 (step S7).

  Note that the multi-viewpoint video shooting system 1 repeats the processes of steps S1 to S7 and records the shot images necessary for multi-viewpoint video representation in the recording device 50. Then, the recording device 50 generates a multi-view video from the recorded captured image (not shown in FIG. 8).

[Action / Effect]
As described above, the multi-view video shooting system 1 controls the robot camera 20 so as to face the subject T when the operator operates to catch the subject T at the center of the screen. As described above, in the multi-viewpoint video shooting system 1, the operator T can shoot the subject T without performing the gazing point setting operation, so that the burden on the operator can be reduced.

(Second Embodiment)
[Configuration of multi-viewpoint video shooting system]
With reference to FIGS. 9 and 10, the configuration of the multi-view video shooting system 1B according to the second embodiment of the present invention will be described while referring to differences from the first embodiment.

The multi-viewpoint video shooting system 1B is different from the first embodiment in that the point of gaze can be tracked and the point of gaze can be set for other subjects located outside the center of the screen.
As shown in FIG. 9, the multi-view video shooting system 1B includes a multi-view robot camera 10, an operation interface unit 30B, a multi-view robot camera control device 40B, and a recording device 50.

As shown in FIG. 10, the operation interface unit 30 </ b> B includes a tripod 31, a pan head 33, a pan bar 35, an encoder 37 (FIG. 11), and a distance adjustment knob 39.
The distance adjustment knob 39 is a dial attached to substantially the center of the bread bar 35. The distance adjustment knob 39 outputs the distance value of another subject as a distance difference value with reference to the distance value of the gazing point reference pixel. That is, the distance difference value represents the difference between the distance value of the subject located at the center of the screen and the distance value of other subjects located outside the center of the screen.

  Here, when it is desired to set a gazing point on another subject located outside the center of the screen, the operator operates the distance adjustment knob 39. Then, the gazing point is set for another subject according to the operation amount of the operator.

[Configuration of multi-viewpoint robot camera control device]
The configuration of the multi-viewpoint robot camera control device 40B will be described with reference to FIG.
As shown in FIG. 11, the multi-viewpoint robot camera control device 40B includes a conversion parameter storage unit 41, a distance image conversion unit 42, a subject area extraction unit 43B, a finder image generation unit 44B, and a gaze point coordinate calculation unit 45B. , A gazing point image coordinate storage unit 46.

  The subject region extraction unit 43B divides the conversion distance image into candidate regions of the subject T by labeling processing, and includes candidate regions including the image coordinates of the gazing point in the captured image of the previous frame stored in the gazing point image coordinate storage unit 46. Are extracted as subject areas. Then, the subject area extraction unit 43B calculates the position of the center of gravity of the extracted subject area and outputs it to the gazing point coordinate calculation unit 45B (tracking the gazing point).

The subject area extraction unit 43B calculates an extraction target distance value obtained by adding the distance difference value input from the distance adjustment knob 39 and the reference distance value, and uses the calculated extraction target distance value as a reference for other subject areas. Is extracted (setting of the gazing point for other subjects). This other subject area is an image area representing another subject.
Details of the subject region extraction unit 43B will be described later.

  The finder image generation unit 44B generates a finder image in which a marker indicating the outline of the other subject region input from the subject region extraction unit 43B is drawn on the captured image. The finder image generation unit 44B has the same finder image generation method as that of the first embodiment, and thus further description thereof is omitted.

The gazing point coordinate calculation unit 45B causes the gazing point image coordinate storage unit 46 to store the gravity center position of the subject region input from the subject region extraction unit 43B as the image coordinates of the gazing point in the captured image of the previous frame.
Other points and the gazing point coordinate calculation unit 45B are the same as those in the first embodiment, and thus further description thereof is omitted.

  The gazing point image coordinate storage unit 46 stores the gazing point image coordinates in the captured image of the previous frame input from the gazing point coordinate calculation unit 45B. The image coordinates of the gazing point in the captured image of the previous frame are referred to when tracking the gazing point in the captured image of the current frame.

<Tracking the point of interest>
With reference to FIG. 12, the tracking of the gazing point will be described (see FIG. 11 as appropriate).
In FIG 12, illustrates the subject T ₁ contained in the captured image of the previous frame in broken lines, the object T ₂ included in the captured image of the current frame to be processed shown in solid lines.

First, the process for the captured image of the previous frame will be described.
Each unit of the multi-viewpoint robotic camera controller 40B, compared captured image of the previous frame, the same processing as in the first embodiment, generates a viewfinder image object T ₁ is is marker display.

At this time, the object area extracting unit 43B is extracted to calculate the gravity center position G ₁ of the object T _1, and outputs the gazing point coordinate calculation unit 45B. The gazing point coordinate calculation unit 45B has a center of gravity position G ₁ of the object T ₁ which is input from the object area extracting unit 43B, stored in the gazing point image coordinate storage section 46 as the image coordinates of the gazing point in the captured image of the previous frame Let

Next, processing for the captured image of the current frame will be described.
The distance image conversion unit 42 converts the distance image into a converted distance image corresponding to the captured image of the current frame by the same method as in the first embodiment.

The subject area extraction unit 43B divides the converted distance image into candidate areas by a labeling process described in Reference Document 2 below, for example. The labeling process is a process of dividing the converted distance image into candidate regions composed of pixels having distance values within a preset range.
Reference 2: Association for Promotion of Image Information Education, “Digital Image Processing”

Subsequently, the subject region extraction unit 43B reads the image coordinates of the gazing point in the captured image of the previous frame from the gazing point image coordinate storage unit 46. Here, as in FIG. 12, the image area of the object T _2, contains gaze point G ₁ in the photographed image of the previous frame. Therefore, the object area extracting unit 43B outputs the image region of the object T ₂ is extracted from the conversion range image as a subject area, the extracted subject region in the viewfinder image generating unit 44. Further, the finder image generator 44B generates a viewfinder image object T ₂ is marker display.

Then, the object area extracting unit 43B is extracted to calculate the gravity center position G ₂ of the object T _2, and outputs the gazing point coordinate calculation unit 45B. The gazing point coordinate calculation unit 45B is in described in reference 1 Equation (2) to (5), a rotation matrix and translation matrix of the master camera 20 _M, pan value and the tilt value of the master camera 20 _M And the distance value at the center of gravity G ₂ of the subject T ₂ are substituted to calculate the world coordinates of the gazing point. Then, gazing point coordinate calculation unit 45B has a center of gravity position G ₂ of the object T _2, is stored in the gazing point image coordinate storage section 46 as the image coordinates of the gazing point in the captured image of the previous frame.

Thus, the multi-viewpoint robotic camera controller 40B, even if the subject T is moving, if the center of gravity position of the object T ₁ of the previous movement if included in the area of the object T ₂ of the post-movement, the object T ₂ it sets the gaze point, the object T ₂ can marker display.

<Setting the point of interest for other subjects>
With reference to FIG. 13, setting of a gazing point for another subject will be described (see FIG. 11 as appropriate).
In this Figure 13, shown with the object T _A located in the center of the screen, and another object T _B is located in other than the center of the screen. Further, the distance value of the object _{T A} and _{D A,} the distance value of the object _{T B} and _{D B,} the distance difference value and [Delta] D.

Rather than the subject T _A located in the center of the screen, consider the case of setting the gaze point on the subject T _B is located in the back of the subject T _A. In this case, as the distance difference ΔD ₌ D A -D _B, the operator operates the distance adjusting knob 39. Then, the distance adjustment knob 39 outputs the distance difference value ΔD to the subject area extraction unit 43B.

Subject region extraction unit 43B, the reference distance value (distance value _{D A} of the subject _{T A),} adds the distance difference value [Delta] D, to calculate the extraction target distance value (distance value _{D B} of the subject _{T B).} Then, the object area extracting unit 43B is the same method as the first embodiment, obtaining the image region of the object T _B the calculated extraction target distance value as a reference. Furthermore, the object area extracting unit 43B extracts from the conversion range image pixel area of the subject T _B as the subject region, and outputs the extracted subject region in the viewfinder image generating unit 44.

Thus, the multi-viewpoint robotic camera controller 40B, even if the position is subject T _B in addition to the center of the screen, and sets the gaze point on the object T _B, the subject T can marker display.
Note that this method can be used even when the captured image includes two or more other subjects T. Also, the setting of the gazing point for other subjects can be used together with the tracking of the gazing point.

[Operation of multi-viewpoint video shooting system]
The operation of the multi-view video shooting system 1B will be described with reference to FIG. 14 (see FIG. 11 as appropriate).

The operator operates the distance adjustment knob 39 so that the subject whose gaze point is to be set is displayed as a marker (step S10).
Note that the processing in step S10 is illustrated by a broken line for manual operation by the operator.

The multi-viewpoint robot camera control device 40B calculates the extraction target distance value by adding the reference distance value and the distance difference value by the subject region extraction unit 43B (step S11).
The multi-viewpoint robot camera control device 40B divides the conversion distance image into candidate regions by the subject region extraction unit 43B (step S12).
In the multi-viewpoint robot camera control device 40B, the subject region extraction unit 43B extracts a candidate region including the image coordinates of the gazing point in the captured image of the previous frame as a subject region (step S13).

Multiview robot camera controller 40B is the gazing point coordinate calculating unit 45B, the distance value at the center of gravity of the subject T after the movement and the pan value and the tilt value of the master camera 20 _M, using the conversion parameters, the fixation point Is calculated (step S14).
The multi-viewpoint robot camera control device 40B causes the gazing point coordinate calculation unit 45B to store the position of the center of gravity of the moved subject in the gazing point image coordinate storage unit 46 as the image coordinates of the gazing point in the captured image of the previous frame (Step S40). S15).

  Note that the multi-view video shooting system 1B repeats the processing of steps S1 to S7 and records the shot images necessary for multi-view video representation in the recording device 50. Then, the recording device 50 generates a multi-view video from the recorded captured image (not shown in FIG. 14).

[Action / Effect]
As described above, the multi-view video shooting system 1B allows the robot camera 20 to track the gazing point of the subject T and face the gazing point that is being tracked without the operator continuing to capture the subject T at the center of the screen. The operator's burden can be reduced.
Furthermore, even when the subject T is located outside the center of the screen, the multi-view video shooting system 1B sets a gazing point on the subject T and controls the robot camera 20 so as to face the set gazing point. Can be reduced.

(Third embodiment)
With reference to FIG. 15, the configuration of a multi-view video shooting system 1 </ b> C according to the third embodiment of the present invention will be described while referring to differences from the second embodiment.
The multi-viewpoint video shooting system 1C is different from the second embodiment in that the gazing point can be tracked even when the gravity center position of the subject T before movement is not included in the image area of the subject T after movement.

[Configuration of multi-viewpoint robot camera control device]
The configuration of the multi-viewpoint robot camera control device 40C will be described with reference to FIG.
As shown in FIG. 15, the multi-viewpoint robot camera control device 40C includes a conversion parameter storage unit 41, a distance image conversion unit 42C, a subject region extraction unit 43C, a finder image generation unit 44B, and a gaze point coordinate calculation unit 45C. , A gazing point image coordinate storage unit 46, a captured image storage unit 47, and a gazing point tracking unit 48.

The distance image conversion unit 42C performs the same processing as in the first embodiment, and then outputs the converted conversion distance image to the gaze point tracking unit 48.
Other points and the distance image conversion unit 42C are the same as those in the first embodiment, and thus further description thereof is omitted.

Similar to the second embodiment, the subject region extraction unit 43C divides the conversion distance image into candidate regions, and calculates the gravity center position of the divided candidate regions. Then, the subject region extraction unit 43C extracts a candidate region having the center of gravity closest to the pixel coordinate of the post-movement gazing point input from the gazing point tracking unit 48 as the subject region.
Details of the subject area extraction unit 43C will be described later.

The gazing point coordinate calculation unit 45C calculates the world coordinate of the gazing point using the post-movement gazing point input from the gazing point tracking unit 48 described later. For example, the gazing point coordinate calculating unit 45C is the described in Reference 1 Equation (2) to (5), a rotation matrix and translation matrix of the master camera 20 _M, pan value and the tilt value of the master camera 20 _M And the distance value at the gazing point after the movement is substituted to calculate the world coordinates of the gazing point. Then, the gazing point coordinate calculation unit 45C outputs the calculated world coordinates of the gazing point to all the robot cameras 20.

  The captured image storage unit 47 stores the captured image input from the gazing point tracking unit 48. The captured image of the previous frame is referred to when tracking the gazing point in the captured image of the current frame.

Gazing point tracking unit 48, and requests the motion vector of the focus point by matching the captured image of the current frame input from the photographed image and the master camera 20 _M of the previous frame stored in the photographed image storage section 47 . The gazing point tracking unit 48 calculates the pixel coordinates indicated by the movement vector starting from the gazing point in the captured image of the previous frame as the pixel coordinates of the gazing point after movement, and uses the calculated pixel coordinates of the gazing point after movement as the subject. The data is output to the region extraction unit 43C and the gaze point coordinate calculation unit 45C.

Here, the gazing point tracking unit 48 stores the calculated pixel coordinates of the post-movement gazing point in the gazing point image coordinate storage unit 46 as image coordinates of the gazing point in the captured image of the previous frame. Further, the fixation point tracking unit 48, a captured image of the current frame input from the master camera 20 _M, is stored in the photographed image storage section 47 as a photographic image of the previous frame.

<Tracking the point of interest>
With reference to FIG. 16, tracking of the point of gaze will be described (see FIG. 15 as appropriate).
In FIG 16, illustrates the subject T ₁ contained in the captured image of the previous frame in broken lines, the object T ₂ included in the captured image of the current frame to be processed shown in solid lines.

The gazing point tracking unit 48 performs matching processing on the captured images of the previous frame and the current frame, and calculates a movement vector indicating where the gazing point of the previous frame has moved in the current frame. Specifically, the gazing point tracking unit 48 performs block matching on the captured images of the previous frame and the current frame, and obtains the image areas of the subjects T ₁ and T ₂ included in the captured images. The gazing point tracking unit 48 obtains the gravity center position _G 1, _{G 2} of the object _T 1, _{T 2,} calculating the moving vector connecting from the gravity center position _{G 1} of the object _{T 1} to the center of gravity position _{G 2} of the object _{T 2} .
Thereafter, the gazing point tracking unit 48, as a starting point the center of gravity position G ₁ of the object T _1, the pixel coordinates of the center of gravity position G ₂ of the object T ₂ showing movement vector is calculated as the pixel coordinates of the mobile after fixation point.

Subject region extraction unit 43C includes a fixation point tracking unit pixel coordinates G ₂ after movement fixation point input from 48, since the gravity center position G ₂ of the object T ₂ are identical, a subject image area of the object T ₂ Extract as a region.

Thus, the multi-viewpoint robotic camera controller 40C, whether or not the gravity center position G ₁ of the moving front of the object T ₁ is included in the image area of the object T ₂ of the after movement, the object T ₂ Note sets a viewpoint, the contour of the object T ₂ can marker display.

[Operation of multi-viewpoint video shooting system]
With reference to FIG. 17, the operation of the multi-viewpoint video shooting system 1C will be described (see FIG. 15 as appropriate).

In the multi-viewpoint robot camera control device 40C, the gazing point tracking unit 48 obtains a movement vector of the gazing point by matching the captured images of the previous frame and the current frame.
In the multi-viewpoint robot camera control device 40C, the gaze point tracking unit 48 calculates the pixel coordinates indicated by the movement vector from the gaze point in the captured image of the previous frame as the pixel coordinates of the post-movement gaze point (step S20).

  The multi-viewpoint robot camera control device 40C uses the subject region extraction unit 43C to extract, as a subject region, a candidate region that has the closest gravity center position to the pixel coordinates of the post-moving gazing point (step S21).

[Action / Effect]
As described above, the multi-view video shooting system 1C tracks the gazing point of the subject T regardless of whether or not the center of gravity of the subject T before movement is included in the image area of the subject T after movement. Since the robot camera 20 can be controlled to face the gazing point during tracking, the burden on the operator can be reduced.

  As mentioned above, although each embodiment of this invention was explained in full detail, this invention is not limited to above-described embodiment, The design change etc. of the range which does not deviate from the summary of this invention are also included.

In each of the above-described embodiments, the reference pixel is described as the pixel at the center of the screen, but the present invention is not limited to this. That is, a pixel at an arbitrary position in the conversion distance image can be set as the reference pixel.
In each of the above-described embodiments, the method described in Reference 1 is used to calculate the camera parameter and the gazing point, but the present invention is not limited to this.
In each of the above-described embodiments, the method described in Reference 1 is used to calculate the world coordinates of the gazing point, the pan value, and the tilt value, but the present invention is not limited to this.

  In each of the above-described embodiments, the multi-viewpoint robot camera control device has been described as independent hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a multi-viewpoint robot camera control program that causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to operate cooperatively as a multi-viewpoint robot camera control device. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

1, 1B, 1C Multi-viewpoint video shooting system 10 Multi-viewpoint robot camera 20 Robot camera 20 _M master camera 20 _S slave camera 21 Camera body 23 Tripod 25 Head 27 Distance camera 30, 30B, 30C Operation interface unit 31 Tripod 33 Head 35 Pan stick 37 Encoder 38 Finder 39 Distance adjustment knobs 40, 40B, 40C Multi-viewpoint robot camera control device 41 Conversion parameter storage unit (camera parameter storage unit)
42, 42C Distance image conversion units 43, 43B, 43C Subject area extraction units 44, 44B, 44C Viewfinder image generation units 45, 45B Gaze point coordinate calculation unit 46 Gaze point image coordinate storage unit 47 Captured image storage unit 48 Gaze point tracking unit 50 Recording device

Claims (6)

A multi-viewpoint robot camera control apparatus that controls a multi-viewpoint robot camera composed of a plurality of robot cameras that photograph the subject in order to generate a multi-viewpoint image in which the same subject is displayed from different viewpoints,
A camera parameter storage unit that stores in advance camera parameters representing the position and orientation of one master camera set in advance among the plurality of robot cameras;
A conversion distance image of a photographed image obtained by photographing the subject by the master camera is input, and a reference pixel distance value at a predetermined position or a distance value of the reference pixel and a distance value of peripheral pixels in the input conversion distance image . A subject area extraction unit that has a distance value within a preset range with reference to an average reference distance value , and that extracts a pixel area including the reference pixel as a subject area;
A finder image generation unit that generates a finder image in which a marker indicating the contour of the extracted subject area is drawn on the captured image, and outputs the generated finder image to the finder;
An operation signal of the master camera by an operator who has viewed the viewfinder image is input, and the input operation signal is converted by the camera parameter, thereby calculating world coordinates of a gazing point representing the position of the subject, A gazing point coordinate calculating unit that outputs the calculated world coordinates of the gazing point to a slave camera that is a robot camera other than the master camera;
A multi-viewpoint robot camera control device comprising: The subject area extraction unit receives a distance difference value, which is a difference in distance value between the subject and another subject other than the subject, and adds the input distance difference value and the reference distance value. Calculate a target distance value, extract other subject areas based on the calculated extraction target distance value,
The finder image generation unit generates a finder image in which a marker indicating an outline of the extracted other subject area is drawn on the photographed image, and outputs the generated finder image to the finder. Item 4. The multi-viewpoint robot camera control device according to Item 1. A gazing point image coordinate storage unit that stores image coordinates of the gazing point in the captured image of the previous frame;
The subject region extraction unit divides the converted distance image into candidate regions of the subject by labeling processing, and includes the image coordinates of the gazing point in the captured image of the previous frame stored in the gazing point image coordinate storage unit. A candidate area is extracted as the subject area, the center of gravity position of the extracted subject area is calculated,
The gazing point coordinate calculation unit stores the centroid position of the subject area calculated by the subject area extraction unit in the gazing point image coordinate storage unit as image coordinates of a gazing point in a captured image of the previous frame. The multi-viewpoint robot camera control device according to claim 1 or 2. A captured image storage unit for storing the captured image of the previous frame;
A gazing point image coordinate storage unit for storing image coordinates of a gazing point in the captured image of the previous frame;
The gazing point in the photographed image of the previous frame is obtained by obtaining a movement vector of the gazing point by matching the photographed image of the previous frame stored in the photographed image storage unit with the photographed image of the current frame input from the master camera. The pixel coordinate indicated by the movement vector starting from the starting point is calculated as the pixel coordinate of the post-movement gazing point, and the calculated pixel coordinate of the post-movement gazing point is used as the gazing point image coordinate in the captured image of the previous frame. A gazing point tracking unit for storing in the image coordinate storage unit and storing the captured image of the current frame in the captured image storage unit;
The subject area extraction unit divides the conversion distance image into candidate areas of the subject by a labeling process, calculates a centroid position of the divided candidate area, and the centroid position is the highest in the pixel coordinates of the post-movement gazing point. The multi-viewpoint robot camera control device according to claim 1, wherein the candidate area that is close is extracted as the subject area.   The distance image conversion part which converts into the said conversion distance image the distance image image | photographed with the distance camera is input further, The distance image conversion part characterized by the above-mentioned. The multi-viewpoint robot camera control device described in 1.   A multi-viewpoint robot camera control program for causing a computer to function as the multi-viewpoint robot camera control device according to any one of claims 1 to 5.

Images