JP2016042639A - Camera posture controller, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a controller for controlling the relative posture between cameras.SOLUTION: A controller operable to control, on the basis of the posture of a reference camera, the posture of at least one camera to be controlled comprises: posture-estimation means for estimating a posture parameter of the camera to be controlled with respect to the posture of the reference camera on the basis of a first image shot by the reference camera and a second image shot by the camera to be controlled; error-estimation means for estimating, on the basis of the posture parameter, an error from a target posture of the camera to be controlled with respect to the reference camera; and control means for controlling the posture of the camera to be controlled so as to reduce the error.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a camera attitude control technique.

  When shooting a stereoscopically viewable image using a plurality of cameras or when shooting an image as a material for generating a stereoscopic image, the relationship between the optical axes of the plurality of cameras is an appropriate relationship. There is a need to. Specifically, it is necessary that the optical axes of the cameras are parallel, the vertical positions thereof are the same, and the alignments of the pixels of the image to be captured are also parallel. It is very difficult to manually adjust the posture of each camera so as to satisfy these conditions.

  For this reason, Non-Patent Document 1 has a configuration in which the posture of each camera is roughly adjusted manually, and the image captured by each camera is processed to generate an image in which the deviation of the posture of each camera is corrected. Disclosure. Patent Document 1 and Non-Patent Document 2 disclose a configuration in which the posture is controlled so that each camera faces the same gazing point.

JP 2011-146762 A

Fukushima Yoshihisa, et al., “2D camera array rectification by parallelizing feature point trajectory”, Journal of the Institute of Image Information and Television Engineers, vol.61, no. 4, pp. 564-571, April 2008 Naoto Kawauchi, et al., "Development of camera-controlled pan-tilt for new video system EYE VISION", Robotics and Mechatronics Lecture, p. 59, 2002

  In a configuration in which the camera posture is corrected by image processing, a geometric operation is performed on a captured image, and thus the image is translated or converted from a rectangular image to a distorted square. For this reason, in the image after image processing, pixels that cannot be used for subsequent processing such as stereoscopic image generation occur. That is, only a part of the photographed range can be used for subsequent processing. In particular, in the configuration described in Non-Patent Document 1, since an image is captured by roughly adjusting manually, an unusable range becomes large. Further, the configurations described in Patent Document 1 and Non-Patent Document 2 adjust the optical axis of the camera so as to face a certain gazing point, and do not adjust the camera so that the optical axes are parallel.

  The present invention provides a control device, a control method, and a program for controlling a relative posture between a plurality of cameras.

According to an aspect of the present invention, the control device that controls the attitude of one or more control target cameras based on the attitude of the reference camera includes a first image captured by the reference camera and a first image captured by the control target camera. Attitude estimation means for estimating an attitude parameter of the control target camera based on the attitude of the reference camera based on two images; and a target of the control target camera based on the reference camera based on the attitude parameter An error estimation means for estimating an error from the posture;
Control means for controlling the attitude of the camera to be controlled so as to reduce the error.

  The relative posture between a plurality of cameras can be controlled.

The block diagram of the control apparatus by one Embodiment. The block diagram of the attitude | position estimation part by one Embodiment.

  Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. In addition, the following embodiment is an illustration and does not limit this invention to the content of embodiment. In the following drawings, components that are not necessary for the description of the embodiments are omitted from the drawings.

  FIG. 1 is a configuration diagram of a control device 2 that controls the postures of the cameras 1-1 to 1-3 according to an embodiment. In the following description, the number of cameras is three, but this is an example, and the present invention can be applied to attitude control of an arbitrary number of two or more cameras. Each of the cameras 1-1 to 1-3 is attached to a camera platform, and is configured so that its direction can be adjusted in so-called pan, tilt, and roll directions. Then, the attitude control unit 25 of the control device 2 outputs a control signal and adjusts the pan, tilt, and roll directions for each of the cameras 1-1 to 1-3. In this embodiment, one camera is used as a reference camera, and the postures of the other two cameras to be controlled are controlled based on the posture of the reference camera. In the following description, it is assumed that the camera 1-2 is a reference camera, the optical axes of the cameras are parallel, the vertical positions thereof are the same, and the pixels of the images captured by the cameras are parallel. Assume that the postures of the control target cameras 1-1 and 1-3 are controlled.

  Each camera 1-1 to 1-3 outputs a captured image. The input unit 21 receives images from the cameras, synchronizes the images taken by the cameras 1-1 to 1-3, and outputs the images to the posture estimation unit 22. The processing in the control device 2 is performed in units of images captured by the cameras 1-1 to 1-3 substantially simultaneously with the external synchronization signal. FIG. 2 is a configuration diagram of the posture estimation unit 22. A feature point is a pixel having a large change in pixel value, such as a corner portion of an object included in an image. The feature point detection unit 221 detects a plurality of feature points for each image, and outputs to the feature amount calculation unit 222 the position of the feature point in the image, a scale indicating the pixel region including the feature point, and the like. The feature amount calculation unit 222 calculates a feature amount related to the pixel value for the pixels in the region specified by the feature point and the scale, normalizes the direction, and calculates the feature amount as the feature point. The feature amount is a value based on a difference in pixel value between a position to be a feature point and its surrounding pixels, for example, a vector representing a difference between pixel values of a plurality of surrounding pixels and a feature point pixel. be able to. For detection of feature points and calculation of feature amounts, for example, known methods such as SIFT, ORB, and Ferns can be used.

  The corresponding point determination unit 223 determines whether the image captured by the reference camera 1-2 and the control target cameras 1-1 and 1-3 are based on the feature points of each image and the degree of coincidence or similarity of the feature amounts regarding the feature points. The correspondence between the feature points of the captured image is determined. For example, when the feature amount is a vector, a vector indicating the feature amount of each feature point of the image captured by the reference camera 1-2 and each feature point of the image captured by the control target cameras 1-1 and 1-3. The square of the difference between the corresponding elements of the vector indicating the feature amount is integrated, and the feature point pair having the lowest integrated value can be determined as the corresponding feature point. Corresponding point determination unit 223 is a posture parameter calculation unit 224 for a pair of feature points of reference camera 1-2 and control target camera 1-1 and a pair of feature points of reference camera 1-2 and control target camera 1-3. Output to.

  The posture parameter calculation unit 224 calculates the posture parameters of the control target cameras 1-1 and 1-3 when the reference camera 1-2 is used as a reference based on the feature point pair. In calculating the posture parameters, for example, publicly known as described in “Shizuo Shimada, Basic Mathematics for CAD / CG (Mathematics Series in the Internet Age (7)), Kyoritsu Publishing, July 2000”. Can be used. In general, the posture parameter is expressed by a translation vector t and a rotation matrix R. The posture parameter calculation unit 224 of the posture estimation unit 23 outputs the posture parameter to the error estimation unit 23.

The error estimation unit 23 obtains an attitude error of the control target camera 1-1 and the camera 1-3 with respect to the reference camera 1-2 based on the attitude parameter. In the present embodiment, this error can be obtained based on the inverse matrix R ⁻¹ of the rotation matrix R. Specifically, the inverse matrix R ⁻¹ is

, The tilt angle error θx, the pan angle error θy, and the roll angle error θz can be obtained by the following equations, respectively.
θx = arcsin (b ₃ )
θy = arctan (−b ₁ / b ₂ )
θz = arctan (−a ₃ / c ₃ )
The error estimation unit 23 outputs the tilt angle error θx, the pan angle error θy, and the roll angle error θz to the determination unit 24.

  The determination unit 24 compares each of the tilt angle error θx, the pan angle error θy, and the roll angle error θz with a corresponding threshold value. The threshold value in each direction can be set to a value that is half the control resolution in the corresponding direction of the camera. This is because even if there is an error that is less than half of the control resolution, the error cannot be made smaller. If any error exceeds the threshold value, the determination unit 24 outputs the amount of posture error of the camera to the posture control unit 25, assuming that the posture control has not been completed for the camera. The attitude control unit 25 controls the camera so that the error amount becomes small based on the received error amount of the camera. When the control of the camera is completed, the posture control unit 25 notifies the posture estimation unit 22 of the completion of the posture control. The posture control unit 22 estimates the posture of a camera for which posture control has not been completed as described above. In this manner, until the posture control of all the cameras is completed, the posture control of the camera by the posture control unit 25, the posture estimation by the posture estimation unit 22, the error estimation by the error estimation unit 23, and the determination unit 24 The determination by is repeated. When the posture control of all the cameras is completed, the determination unit 24 outputs the posture parameters of the control target cameras 1-1 and 1-3 to the correction unit 26.

  When the correction unit 6 receives the attitude parameters of the control target cameras 1-1 and 1-3, the correction unit 6 corrects the error for the images captured by the cameras 1-1 and 1-3 input from the input unit 21. Image processing is performed, and a corrected image is output. For example, when there are two cameras, an image of the reference camera 1-2 can be corrected.

  As described above, in the present embodiment, the control device 2 controls the postures of the control target cameras 1-1 and 1-3 based on the posture of the reference camera 1-2. Therefore, the area of the image that cannot be used in the image after the image processing in the correction unit 26 is reduced.

  In the present embodiment, the error is obtained based on the rotation matrix R, but the error amount can be evaluated based on the feature point pair. For example, the average value or the maximum value of the difference in the vertical direction between the feature point pairs of the reference camera 1-2 and the control target camera 1-1 can be used as an evaluation value for evaluating the error. In this case, the determination unit 24 determines whether or not the attitude control of the control target camera 1-1 is completed by comparing the evaluation value with a threshold value. Furthermore, only the pan direction and the tilt direction can be used to determine whether or not the attitude control has been completed. Further, as for the adjustment direction of each camera, it is possible to adjust the posture other than the pan direction, the tilt direction, and the roll direction, such as translation of the camera itself.

  The control device according to the present invention can be realized by a program that causes a computer to operate as the control device. These computer programs can be stored in a computer-readable storage medium or distributed via a network.

Claims (11)

A control device that controls the attitude of one or more controlled cameras based on the attitude of a reference camera,
Attitude estimation means for estimating an attitude parameter of the control target camera based on the attitude of the reference camera based on a first image taken by the reference camera and a second image taken by the control target camera;
Error estimation means for estimating an error from a target posture of the control target camera based on the reference camera based on the posture parameter;
Control means for controlling the attitude of the control target camera so as to reduce the error;
A control device comprising: The posture estimation means includes
Detecting means for detecting a plurality of feature points for each of the first image and the second image;
Calculating means for calculating the feature amount of each feature point for each of the first image and the second image;
Determination means for determining a correspondence relationship between a plurality of feature points of the first image and a plurality of feature points of the second image based on a feature amount of each feature point;
Calculation means for calculating the posture parameter based on corresponding feature points of the first image and the second image;
The control device according to claim 1, further comprising:   The said attitude | position parameter contains the information which shows the angle of the said control object camera on the basis of the attitude | position of the said reference camera about each of a pan direction, a tilt direction, and a roll direction. Control device.   The error estimated by the error estimation unit is an error of an angle of the control target camera in a pan direction, a tilt direction, and a roll direction with reference to an attitude of the reference camera. Control device. The posture estimation means outputs an evaluation value indicating a deviation of the posture of the control target camera based on the posture of the reference camera;
5. The control according to claim 2, wherein the control unit controls an attitude of the control target camera so as to reduce the error when the evaluation value exceeds a threshold value. 6. apparatus.   The control apparatus according to claim 5, wherein the evaluation value is an error in an angle of the control target camera in a pan direction, a tilt direction, and a roll direction with reference to an attitude of the reference camera.   7. The control device according to claim 6, wherein the threshold values in the pan direction, the tilt direction, and the roll direction are determined based on a minimum rotation angle in the pan direction, the tilt direction, and the roll direction of the control target camera, respectively.   The control device according to claim 5, wherein the evaluation value is a value based on a difference in position in a vertical direction between corresponding feature points of the first image and the second image.   The correction means for correcting at least one of the first image and the second image based on the posture parameter if the evaluation value does not exceed the threshold value. The control device according to any one of the above.   A program that causes a computer to function as the control device according to any one of claims 1 to 9. A control method in a control device for controlling the attitude of one or more control target cameras based on the attitude of a reference camera,
An acquisition step of acquiring a first image captured by the reference camera and a second image captured by the control target camera;
A posture estimation step of estimating a posture parameter of the control target camera based on the posture of the reference camera based on the first image and the second image;
An error estimation step for estimating an error from a target posture of the control target camera based on the reference camera based on the posture parameter;
A control step for controlling the posture of the controlled camera so as to reduce the error;
The control method characterized by including.