JP4906683B2 - Camera parameter estimation apparatus and camera parameter estimation program - Google Patents

Description

  The present invention relates to a camera parameter estimation device and a camera parameter estimation program for estimating camera parameters when a subject is imaged with a camera.

  In general, when an image obtained by capturing a subject with a camera and another image are combined, they are combined based on camera parameters such as the posture, position, and angle of view when the camera is picked up. In this case, it is common to use camera parameters calibrated in advance as cameras with known camera positions or the like, or use camera parameters estimated from images captured by cameras with unknown camera positions or the like.

  There are various methods for estimating camera parameters from an image. For example, by using a plurality of known objects (landmark groups) whose three-dimensional relative positional relationship does not change with time, the coordinates (image coordinate groups) of each object in the image are observed to capture the image. There is a method for estimating the camera parameters of the camera (see Patent Document 1). Further, there is a technique for estimating the relative position and orientation of a camera with respect to a landmark group by solving a perspective four-point problem (P4P problem: Perspective Four-Point Problem) (see Non-Patent Document 1). In addition, for example, there is a method for estimating the relative position and orientation of a camera from a temporal trajectory of a rectangular marker (see Patent Document 2).

  Also, the image coordinate group of the landmark group using a landmark group in which the relative positional relation of the three-dimensional relative position relationship does not change with time and the relative positional relationship of each other is unknown, and moves relative to the landmark group. There is a method of estimating the relative position and posture of the camera with respect to the landmark group and the relative positional relationship between the landmarks by observing the image over a plurality of time points (see Non-Patent Document 2).

Furthermore, an image of a linear landmark group with a known three-dimensional shape, such as a line drawn on a sports stadium, is detected by Hough transform or generalized Hough transform, and the landmark group is determined based on the Hough parameter. There is a method for estimating the relative position and orientation of the camera with respect to (see Patent Document 3).
JP 2006-33449 A JP 2003-196663 A JP 2004-234333 A H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, K. Tachibana. Virtual Object Manipulation on a Table-Top AR Environment.In Proceedings of ISAR 2000, Oct 5th-6th, 2000. J. Weng, TS Huang, N. Ahuja.Motion and Structure from Two Perspective Views: Algorithms, Error Analysis, and Error Estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 11, Issue 5, pp. 451-476, May 1989.

  However, the methods described in Patent Documents 1 and 2 and Non-Patent Document 1 described above are effective when the imaging range of the camera is narrow because the arrangement of landmark groups is fixed. In order to acquire camera parameters for posture, position, angle of view, etc., it is necessary to install a lot of landmarks over a wide area so that a predetermined number of landmarks are included in the presumed shooting area. There is.

  Further, the technique described in Non-Patent Document 2 has a problem that in principle, camera parameters cannot be acquired unless the camera (specifically, the first optical principal point) moves relative to the landmark group. is there. Also, use this method when the camera does not move sufficiently, such as when the amount of movement is small, when there is no camera operation, or when the camera operation is limited to pan, tilt, and zoom. There is a problem that can not be.

  Furthermore, in the method described in Patent Document 3, as in the methods described in Patent Documents 1 and 2 and Non-Patent Document 1, a predetermined number or more of landmarks are included in a presumed shooting area. In addition, the shape of the landmark is limited to a predetermined straight line or curve.

  The present invention has been made to solve the above-described problems, and it is possible to estimate camera parameters using an arbitrary fixed object or moving object as a landmark without fixing the position of the landmark. It is an object to provide a camera parameter estimation device and a camera parameter estimation program.

  The present invention was devised to achieve the above-mentioned object. First, the camera parameter estimation device according to claim 1 includes a first image captured by a first camera having a known camera parameter, a camera A camera parameter estimation device that estimates a camera parameter of the second camera based on a second image captured by a second camera whose parameter is unknown, the first landmark extraction means, a position estimation means, A second landmark extracting unit, a landmark associating unit, and a parameter estimating unit are provided, and the landmark associating unit further includes a projecting unit and a collating unit.

  In such a configuration, the camera parameter estimation device extracts the coordinates of the landmark in the image from the first image as the reference image coordinates based on the image characteristics of the landmark by the first landmark extraction unit. The image features of the landmarks can be predetermined features such as brightness, color, pattern, and the like. Then, the camera parameter estimation device uses the position estimation unit to backproject the reference image coordinates under a predetermined constraint based on the known camera parameters that are the camera parameters of the first camera, so that the three-dimensional landmarks are obtained. Estimate the coordinates. For example, the position estimation means can estimate a three-dimensional coordinate from a two-dimensional coordinate by giving a constraint condition that the landmark exists on the ground.

  In the camera parameter estimation device, the second landmark extracting unit extracts the coordinates of the landmark in the image from the second image as the observed image coordinates based on the image characteristics of the landmark. Note that the first landmark extracting unit and the second landmark extracting unit need only extract the position of the coordinates of the landmarks, and therefore, it is not always necessary to use the same image feature.

  Then, the camera parameter estimation device converts the three-dimensional coordinates estimated by the position estimation means into predicted image coordinates that are two-dimensional coordinates in the second image by the landmark association means, and collates with the observed image coordinates. The landmark is associated between the first image and the second image.

  The landmark associating means calculates the predicted image coordinates by projecting and transforming the three-dimensional coordinates with the camera parameters already estimated by the projecting means. Thereby, the landmark extracted on the first image is projected on the second image. The landmark associating means associates the landmark with the collating means based on the distance between the predicted image coordinates and the observed image coordinates. As a result, when a plurality of landmarks are extracted, the landmark on the first image and the landmark on the second image are associated one-to-one.

  Then, the camera parameter estimation device uses the parameter estimation unit based on the three-dimensional coordinates corresponding to the landmark and the observed image coordinates in the same landmark associated by the landmark association unit. Estimated camera parameters that are camera parameters of the two cameras are estimated. It should be noted that a known solution of the perspective n-point problem can be applied to the estimation of the estimated camera parameter in the parameter estimation means.

  According to a second aspect of the present invention, there is provided the camera parameter estimation device according to the first aspect, wherein the landmark association unit further includes a prediction unit.

  In this configuration, the camera parameter estimation device predicts the camera parameters at the current time based on the time-series estimated camera parameters estimated by the parameter estimation unit by the prediction unit. Thus, the camera parameter estimation device can calculate the predicted image coordinates in which the current landmark position is predicted by projecting and transforming the three-dimensional coordinates with the camera parameters predicted by the prediction means in the projection means. .

  Furthermore, the camera parameter estimation apparatus according to claim 3 is the camera parameter estimation apparatus according to claim 1 or 2, wherein the landmark association means includes a first feature extraction means, a second feature extraction means, , Further provided.

  In such a configuration, the camera parameter estimation device extracts, from the first image, the image feature in the vicinity region of the reference image coordinates as the first feature by the first feature extraction unit in the landmark association unit. In addition, the camera parameter estimation device extracts, as the second feature, the image feature in the vicinity region of the observed image coordinates from the second image by the second feature extraction unit. As a result, the image feature becomes a feature amount allowing a certain amount of error. Then, the camera parameter estimation device associates the landmark with the collating unit based on the weighted addition of the degree of similarity between the first feature and the second feature and the distance between the predicted image coordinate and the observed image coordinate.

  Further, the camera parameter estimation device according to claim 4 is the camera parameter estimation device according to any one of claims 1 to 3, wherein the parameter estimation means includes a coordinate association means, a parameter calculation means, It was set as the structure provided with.

  In such a configuration, the camera parameter estimation device uses the parameter estimation unit to calculate the observed image coordinates based on the three-dimensional coordinates estimated by the position estimation unit and the correspondence results by the landmark association unit. Map 3D coordinates. As a result, the two-dimensional observation image coordinates obtained from the second image captured by the second camera are converted into three-dimensional coordinates in the real space.

  And a camera parameter estimation apparatus calculates an estimated camera parameter from the three-dimensional coordinate of several observation image coordinate by a parameter calculation means. In this way, the parameter calculation means sequentially inputs the three-dimensional coordinates of the observed image coordinates associated by the coordinate association means, and at the stage when the coordinates for solving the perspective n-point problem are input, Estimated camera parameters are calculated using a solution for the n-point problem.

  Furthermore, the camera parameter estimation program according to claim 5 is based on a first image captured by a first camera whose camera parameters are known and a second image captured by a second camera whose camera parameters are unknown. A computer functioning as first landmark extracting means, position estimating means, second landmark extracting means, landmark associating means, parameter estimating means for estimating camera parameters of the second camera; did.

  In such a configuration, the camera parameter estimation program extracts, from the first image, the coordinates of the landmark in the image as the reference image coordinates by the first landmark extraction means based on the image characteristics of the landmark. Then, the camera parameter estimation program performs backprojection conversion of the reference image coordinates under a predetermined constraint condition based on the known camera parameter that is the camera parameter of the first camera by the position estimation unit, thereby obtaining the three-dimensional landmark. Estimate the coordinates.

  Further, the camera parameter estimation program extracts the coordinates of the landmark in the image from the second image as the observed image coordinates by the second landmark extraction unit based on the image characteristics of the landmark.

  Then, the camera parameter estimation program converts the three-dimensional coordinates estimated by the position estimation means into predicted image coordinates that are two-dimensional coordinates in the second image by the landmark association means, and collates with the observed image coordinates. The landmark is associated between the first image and the second image.

  The landmark associating means predicts camera parameters at the present time based on the time-series estimated camera parameters calculated by the parameter calculating means, and projects and converts the three-dimensional coordinates of the landmarks using the predicted camera parameters. To calculate predicted image coordinates, and further associate landmarks based on the distance between the predicted image coordinates and the observed image coordinates.

  According to the first, fourth, and fifth aspects of the present invention, it is possible to extract and associate the same landmark from an image captured by a camera with known camera parameters and an image captured by a camera with unknown camera parameters. Therefore, it is possible to estimate an unknown camera parameter using any fixed object or moving object as a landmark without fixing the position of the landmark. Thereby, since it is not necessary to install a unique landmark, unnecessary marks can be removed at the imaging site, and natural images can be captured. According to the first, fourth, and fifth aspects of the present invention, if a camera parameter is known and an unknown camera and each captured image includes at least one landmark, sequential imaging is performed. The positions of a plurality of landmarks can be detected from the captured image, and unknown camera parameters can be estimated. Thus, the number of landmarks when estimating camera parameters can be reduced.

  According to the second aspect of the present invention, since the camera parameters at the present time can be estimated from the already estimated camera parameters and the landmarks can be associated with each other, the coordinates of the landmarks on the image can be obtained with high accuracy. it can. As a result, landmarks can be associated with high accuracy, and unknown camera parameters can be estimated with high accuracy.

  According to the third aspect of the present invention, since the image feature of the reference image coordinates in the first image and the image feature of the observed image coordinates in the second image are respectively image features in the vicinity region of the coordinate position, And the landmarks can be associated with each other with high accuracy in the first image and the second image. Thereby, the estimation accuracy of the unknown camera parameter can be improved.

Embodiments of the present invention will be described below with reference to the drawings.
[Outline of the present invention]
First, the outline of the present invention will be described with reference to FIG. FIG. 1 is an explanatory diagram for explaining the outline of the present invention.

  The camera parameter estimation device 1 includes an image (first image) captured by a camera (first camera) C1 with known camera parameters and an image (first image) captured by a camera (second camera) C2 with unknown camera parameters. The camera parameters of the camera C2 are estimated based on the two images.

  Here, the camera parameter estimation device 1 extracts a plurality of moving objects and stationary objects as landmarks M (players in FIG. 1) from an image captured by the camera C1 and an image captured by the camera C1. The landmark is an area having a shading or color change that can be distinguished from other local areas on the image.

  Then, the camera parameter estimation device 1 calculates coordinates (three-dimensional coordinates) in the real space of the landmark M based on the landmark M captured by the camera C1 and the known camera parameters of the camera C1, and the camera C2 The camera parameters of the camera C2 are estimated by associating with the image coordinates of the landmark M captured in step S2.

  FIG. 1 shows an example in which the cameras C1 and C2 are installed in the soccer field and the player is a landmark. However, if a part of the landmark M can be observed simultaneously by the cameras C1 and C2, It is not limited to the installation location.

[About camera parameters]
Next, camera parameters in the camera parameter estimation apparatus according to the embodiment of the present invention will be described.

  The camera parameters are external parameters determined by the installation position and installation posture of the camera, the focal length and distortion of the lens, the shape of the image sensor, the pixel size and the number of pixels, and the relative positional relationship between the lens and the image sensor. It is an index that includes both internal parameters. In the present embodiment, the camera installation position and installation posture, and the lens focal length are used as the camera parameters. The installation position and installation posture of the camera can be defined with reference to an arbitrary coordinate system (hereinafter referred to as a world coordinate system) fixedly defined in the real space.

  In this world coordinate system, a representative arbitrary point in a space where the cameras C1 and C2 (see FIG. 1) are installed is set as an origin, and three representative and independent directions are set as axial directions. For example, when the cameras C1 and C2 are installed in a soccer stadium as shown in FIG. 1, the position of the center mark of the soccer court is the origin of the world coordinate system, the touch line direction (x axis), the half way line direction ( The y-axis) and the vertically upward direction (z-axis) are the three axes of the world coordinate system. For example, when the cameras C1 and C2 are installed in a room such as a studio, the center of the floor in the room is set as the origin of the world coordinate system, and the horizontal east direction, the horizontal north direction, and the vertical upward direction are set in the world coordinate system. Three axes are assumed.

The camera installation posture can be expressed by posture conversion between a camera coordinate system fixed to the camera and the above-described world coordinate system. For example, as shown in the following equation (1), the installation posture of the camera is a vector when a vector v ^(w) when a certain direction vector in real space is viewed from the world coordinate system is viewed in the camera coordinate system. v ^(c) can be expressed by a 3 × 3 rotation matrix R to be converted.

The camera installation position can be defined as a position vector when the origin position of the camera coordinate system is viewed in the world coordinate system. For example, the installation position of the camera uses the rotation matrix R and the position vector t of the installation position of the camera, so that the position vector p ^(w) in the world coordinate system as shown in the following equation (2 ^). Can be converted into a position vector q ^(c) in the camera coordinate system. In addition, although the unit of each component of a position vector is arbitrary, it can be made into a unit, for example.

The camera coordinate system used in the above-described equations (1) and (2) has, for example, the first optical principal point of the camera lens as the origin and two axes in two independent directions on the camera image plane. The remaining one axis can be taken in the optical axis direction. The two independent directions on the image plane of the camera can be defined by, for example, the horizontal direction and the vertical direction of the camera image sensor.
Hereinafter, the configuration and operation of the camera parameter estimation apparatus according to the embodiment of the present invention will be described in detail.

[First Embodiment: Configuration of Camera Parameter Estimation Device]
First, the configuration of the camera parameter estimation apparatus according to the first embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the camera parameter estimation apparatus according to the first embodiment of the present invention.

  Here, the camera parameter estimation device 1 includes a first landmark extraction unit 10, a second landmark extraction unit 11, a landmark association unit 12, a position estimation unit 13, and a parameter estimation unit 14. The camera parameter estimation device 1 connects the camera C1 and the camera C2 to the outside.

  The camera C1 is a camera whose camera parameters are known. The camera C2 is a camera whose camera parameters are unknown. An image captured by the camera C1 (first image) and an image captured by the camera C2 (second image) are input to the camera parameter estimation device 1.

  Here, the image coordinate system of the first image and the second image uses a two-dimensional coordinate system on the image plane, and here, two axes taken on the image plane among the three axes constituting the camera coordinate system. Is the image coordinate system. In the following description, the first image captured by the camera C1 is denoted by I, and the pixel value at the image coordinate r of the first image I is denoted by I (r). In addition, the second image captured by the camera C2 is denoted as J, and the pixel value at the image coordinate o of the second image J is denoted as J (o).

  Note that when the cameras C1 and C2 are monochrome cameras, the pixel values can be represented by, for example, discrete scalar values of two or more values (for example, 256 values). When the cameras C1 and C2 are color cameras or multi-band cameras, the pixel values are, for example, two or more dimensions (for example, three dimensions of red, green, and blue) and two or more dimensions (for example, 256 values). It can be represented by a discrete vector value. In addition, when the cameras C1 and C2 are color cameras or multiband cameras, a color table (color conversion table: a table associating discrete index values with pixel colors) is used, so that the pixel values are two or more (for example, 65536 values) may be represented by a color index of discrete scalar values.

  In addition, since the camera C1 and the camera C2 associate landmarks, it is necessary to capture at least one landmark in common. For example, when used in the soccer stadium shown in FIG. 1, the camera C1 captures the entire soccer field, and the camera C2 captures a part including the landmark in the soccer field. More than one landmark can be imaged in common.

The first landmark extracting means 10 determines the coordinates of a plurality of landmarks in the image (hereinafter referred to as reference) from the first image captured by the camera C1 whose camera parameters are known based on the predetermined image characteristics of the landmark. (Referred to as image coordinates). The first landmark extracting means 10 attaches an identifier (first identifier) to the extracted reference image coordinates and outputs it to the position estimating means 13. Further, the number of landmarks that have been extracted by the first landmark extracting unit 10 and the M, its m-th (m = 1, 2, ..., M) the reference image coordinates of landmarks and r _m. The m is used as a first identifier.

  Note that the first landmark extraction unit 10 uses, for example, a Moravec instant operator that uses a minimum value of directional dispersion in a local region on an image as an image feature and selects a point at which the image feature is maximized. Can be extracted. Further, when a marker such as an object painted in a specific color or a disk having a concentric circular pattern is artificially installed in the object scene, the first landmark extraction means 10 selects an operator for detecting a specific color or pattern. It may be applied to an image and a detected point or a representative point (for example, the center of gravity) of a detection area may be extracted as a landmark.

  Further, the first landmark extracting means 10 may extract a moving object such as an animal or a person (or a person's face) as a landmark. For example, when a person's face is used as a landmark, the first landmark extracting means 10 uses a general face detector (for example, P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proc. Of IEEE Conference on Computer Vision and Pattern Recognition, Kauai, HI, December 2001.), representative points of detected face regions can be used as landmarks.

The second landmark extracting means 11 is configured to determine the coordinates (hereinafter referred to as observation) of a plurality of landmarks from the second image captured by the camera C2 whose camera parameters are unknown based on predetermined image characteristics of the landmarks. (Referred to as image coordinates). The second landmark extraction unit 11 attaches an identifier (second identifier) to the extracted observation image coordinates and outputs the identifier to the landmark association unit 12 and the parameter estimation unit 14. Further, the number of landmarks that have been extracted by the second landmark extracting unit 11 and the N to the n-th (n = 1, 2, ..., N) the observed image coordinates of the landmarks in the o _n. The n is used as a second identifier.

  The landmark extraction process in the second landmark extraction unit 11 may be the same process as the first landmark extraction unit 10 or a different process. For example, when a red disk that is an artificial pattern is used as a landmark, the first landmark extraction unit 10 extracts a landmark by focusing on a circular pattern that is the shape of a red disk, and extracts a second landmark. The means 11 may extract landmarks by paying attention to the red color of the red disk.

  The landmark association unit 12 converts actual coordinates (three-dimensional coordinates) estimated by the position estimation unit 13 described later into predicted image coordinates that are two-dimensional coordinates in the second image, and collates with observed image coordinates. The landmarks are associated with each other between the first image and the second image. Here, the landmark association unit 12 includes a prediction unit 121, a projection unit 122, and a collation unit 123.

  The prediction unit 121 predicts the current camera parameter as the predicted camera parameter based on the time-series estimated camera parameter calculated (estimated) by the parameter estimation unit 14 described later. The predicted camera parameter predicted by the prediction unit 121 is output to the projection unit 122.

とし、現時点における予測カメラパラメータを

として、予測手段１２１が行う予測手法について説明する。また、ここで、Ｔ_ｋは、推定開始時点をｋ＝１として、ｋ回目の観測時点における時刻を表すものとする。この時刻の単位には、例えば秒を用いることができる。 Here, the estimated camera parameters estimated by the parameter estimation means 14 before the estimation time are calculated.

And the current predicted camera parameters

A prediction method performed by the prediction unit 121 will be described. Here, T _k represents the time at the k-th observation time point where the estimation start time point is k = 1. As the unit of time, for example, seconds can be used.

  For example, as shown in the following equation (3), the prediction unit 121 can obtain the current predicted camera parameter by linear prediction from the estimated camera parameter at the previous time point and the camera parameter at the previous time point.

Further, for example, as shown in the following equation (4), the prediction unit 121 obtains the current predicted camera parameter from the linear combination of the estimated camera parameters up to the previous time point or the linear combination of the predicted camera parameters up to the previous time point. Can be sought. In equation (4), a _i and b _i indicate real tap coefficients (weighting coefficients).

  Further, for example, the prediction unit 121 may obtain the prediction camera parameter and its error covariance matrix using a Kalman filter or an extended Kalman filter based on the estimated camera parameter at the previous time point and its error covariance matrix.

  Here, the landmark associating means 12 is provided with the predicting means 121. However, as shown in the following equation (5), the estimated camera parameter at the previous time point is used as it is as the current predictive camera parameter. Also good. In this case, the prediction unit 121 is omitted from the landmark association unit 12 and the time-series estimated camera parameters calculated (estimated) by the parameter estimation unit 14 described later are input to the projection unit 122 as they are. Thereby, when there is no big change in camera work, the camera parameter estimation apparatus 1 can be configured with a simple configuration.

  The projection unit 122 predicts the current coordinates of the reference image coordinates by projecting and converting actual coordinates (three-dimensional coordinates) estimated by the position estimation unit 13 described later, based on the prediction camera parameters predicted by the prediction unit 121. It is calculated as image coordinates.

For example, when the projection transformation (perspective projection transformation) h is expressed by the following equation (6) by the camera parameter θ from the real coordinate p to the image coordinate o, the projection unit 122 receives the first identifier input from the position estimation unit 13. the real coordinate p _m corresponding to m, is converted to the two-dimensional predicted image coordinates o _{m 'by} the following equation (7).

The projection means 122 outputs the predicted image coordinates obtained by the first identifier m (7) formula _{o m} 'and the pair _{(m, o} m') and the matching means 123 for all m. Note that the projection unit 122 operates by inputting the predicted camera parameter, but in the initial state, the predicted camera parameter serving as a reference for the operation is not input.

  Therefore, the projection unit 122 is input with the camera parameter of the initial value of the camera C2 as the initial value of the predicted camera parameter in advance, or stores the camera parameter of the initial value of the camera C2 in advance in a storage unit that is not illustrated. The operation is performed by reading at the initial operation. After the initial operation, predicted camera parameters are sequentially input to the projection unit 122, and predicted image coordinates can be calculated using the predicted camera parameters.

The collating unit 123 associates landmarks based on the distance between the predicted image coordinates calculated by the projecting unit 122 and the observed image coordinates extracted by the second landmark extracting unit 11. Here, the matching unit 123 is based on a set of pairs (m, o _m ′) of the first identifier and the predicted image coordinates and a set of pairs (n, o _n ) of the second identifier and the observed image coordinates. Then, a pair of the first identifier and the second identifier that are considered to have predicted and observed the same landmark is output to the parameter estimation unit 14 as a correspondence result.

For example, the matching means 123, a pair of the first identifier and the predicted image coordinates (m, o m _') For all m in, the following equation (8), the predicted image coordinates o _m' closest observed image coordinates It obtains a second identifier _{n m} of outputs (m, _{n m)} as pairs of the parameter estimation unit 14.

Incidentally, in equation (8), ‖o _n -o _m '‖ is, _{o n} a _{o m'} is a norm which represents the distance between the, for example, it can be used Euclidean norm.

The position estimation unit 13 performs backprojection conversion on the reference image coordinates extracted by the first landmark extraction unit 10 using known camera parameters that are camera parameters of the camera C1, thereby changing the actual coordinates (three-dimensional coordinates) of the landmarks. To be estimated. The position estimating means 13, the actual coordinates p _m Landmarks first identifier m the number of the first identifier (m, p _m) determined as the data of the pairs of landmarks correlating means 12 and parameter estimation means 14 for output. The position estimation unit 13 may sequentially input known camera parameters from the camera C1. When the camera C1 is a fixed camera, the position estimation unit 13 stores the known camera parameters in a storage unit (not shown) in advance. It is good also as reading.

Note that the reference image coordinate r _m whereas the two-dimensional coordinates, the actual coordinate p _m landmarks is a three-dimensional coordinates. For this reason, in order for the position estimation means 13 to perform the back projection, a planar constraint condition on the actual coordinates where the landmark exists is necessary. For example, by giving a constraint condition that the landmark is on a plane such as a floor or the ground, the position estimating unit 13 can back-project the reference image coordinates to the real coordinates.

Here, with reference to FIG. 3, a method of back projecting the reference image coordinates to the real coordinates in the position estimating means 13 will be described in detail. FIG. 3 is an explanatory diagram for explaining a back projection technique in the position estimating means. In FIG. 3, it is assumed that a landmark M exists on a curved surface K, and a landmark image m is captured on the first image I captured by the camera C1.
In the example of FIG. 3, the curved surface K where the landmark M (real coordinate p) exists is expressed by the following equation (9) using the function f.

  For example, when the three axes of the world coordinate system are the X axis, the Y axis, and the Z axis, in order to constrain the landmark M to the plane Z = 0, the curved surface K is expressed as a function f shown in the following equation (10). That's fine.

  Here, the origin of the camera coordinate system of the camera C1 is taken as the first optical principal point T, and the position vector in the world coordinate system is t. Also, let R be the rotation matrix for posture conversion from the world coordinate system to the camera coordinate system. The x and y axes of the camera coordinate system of the camera C1 are in a plane parallel to the image plane of the first image I, and the z axis is parallel to the optical axis and oriented in the field. It is assumed that the optical axis is perpendicular to the image plane.

  Further, the image coordinate system on the image plane of the first image I is a two-dimensional coordinate system stretched by only two axes of the x-axis and the y-axis in the camera coordinate system, and the origin is between the image plane and the optical axis. Take the intersection. The lens of the camera C1 follows the pinhole model, and its focal length is f.

  Further, it is assumed that the reference image coordinate in the first image I input to the position estimation unit 13 is r, and the camera parameter input as the known camera parameter includes the position vector t and the rotation matrix R. .

  As shown in FIG. 3, the landmark M starts from the first optical principal point T and is k times the vector reaching the landmark image m on the image plane of the first image I (k is a positive real coefficient). The real coordinate p can be expressed by the following equation (11).

  Here, the coefficient k (however, k> 0) can be obtained by combining the equation (9) and the equation (10). When there are two or more solutions of the simultaneous equations with respect to the coefficient k, the smallest value among them is used. The actual coordinate p is obtained by substituting the coefficient k thus obtained in the equation (11). For example, when the landmark is on the plane where Z = 0, the relationship of the following equation (12) is established by the equation (10). Further, from the equation (12), the coefficient k is obtained as shown in the following equation (13).

  Then, by substituting the coefficient k obtained by the equation (13) into the equation (11), the actual coordinate p is obtained as shown in the following equation (14).

  This equation (14) takes the X axis and Y axis of the world coordinate system on the floor surface, for example, when there is a landmark on the flat floor surface, and takes the Z axis upward, for example, orthogonal to them. It can be applied.

Here, the method of obtaining the real coordinates by giving the constraint condition of the plane Z = 0 has been described, but the constraint condition may be given for X and Y. Thereby, for example, the position estimating means 13 can obtain the actual coordinates of the landmark on the wall surface existing in the vertical direction from the floor surface.
Returning to FIG. 2, the description of the configuration of the camera parameter estimation apparatus 1 will be continued.

  The parameter estimation unit 14 includes the actual coordinates (three-dimensional coordinates) of the reference image coordinates estimated by the position estimation unit 13 corresponding to the landmark, in the same landmark associated by the landmark association unit 12. Based on the observed image coordinates extracted by the second landmark extracting means 11, the camera parameters (estimated camera parameters) of the camera C2 are estimated. Here, the parameter estimation unit 14 includes a coordinate association unit 141 and a parameter calculation unit 142.

  The coordinate association unit 141 is an observation extracted by the second landmark extraction unit 11 based on the actual coordinates (three-dimensional coordinates) estimated by the position estimation unit 13 and the correspondence result by the landmark association unit 12. The real coordinates are associated with the image coordinates.

Here, the coordinate association unit 141 is a set of pairs (m, p _m ) of the first identifier output from the position estimation unit 13 and the actual coordinates of the landmark, and the first output from the landmark association unit 12. From a set of a pair (m, n _m ) of one identifier and a second identifier, and a set of a pair (n, o _n ) of a second identifier and observation image coordinates output from the second landmark extraction means 11 Then, a set of pairs (p _m , o _nm ) of real coordinates and observed image coordinates regarding the same landmark is obtained. Incidentally, o _nm shows the observed image coordinates associated with the second identifier n _m to be the first identifier m a pair. Then, the coordinate association unit 141 performs parameter calculation on a set of pairs (p _m , o _nm ) of the actual coordinates and the observed image coordinates as many as the number of pairs (m, n _m ) of the first identifier and the second identifier. It outputs to the means 142.

  The parameter calculation unit 142 calculates estimated camera parameters, which are camera parameters of the camera C2, based on the actual coordinates (three-dimensional coordinates) of the plurality of observed image coordinates associated with the coordinate association unit 141. The camera parameters calculated by the parameter calculation unit 142 are sequentially output to the outside as the camera parameters of the camera C2.

Here, the parameter calculation unit 142 calculates the camera parameters of the camera C2 when a predetermined number of pairs (p _m , o _nm ) of landmark real coordinates and observed image coordinates are input. . For this calculation, when a three-dimensional coordinate (real coordinate) of n points on a two-dimensional plane (image plane) is specified, a general method for solving a perspective n-point problem for estimating camera parameters can be used. For example, when the landmarks exist on the same plane, the parameter calculation unit 142 receives four perspective points when four pairs (p _m , o _nm ) of the actual coordinates of the landmarks and the observed image coordinates are input. The camera parameters of the camera C2 are calculated by solving the problem.

  By configuring the camera parameter estimation device 1 in this way, the camera parameter estimation device 1 is configured so that the image captured by the camera C1 whose camera parameter is known (first image) and the image captured by the camera C2 whose camera parameter is unknown. From the (second image), the camera parameters of the camera C2 can be estimated.

  Further, the camera parameter estimation device 1 uses any object or pattern that can be commonly observed by the camera C1 and the camera C2 as a landmark without fixing the position of the landmark, and determines the camera parameter of the camera C2. Can be estimated. In addition, the camera parameter estimation device 1 is not limited to an object or pattern arbitrarily set in the scene, but a static or dynamic pattern projected on the scene, an object or pattern that originally exists in the scene, or Objects, people, etc. that move in the field can be used as landmarks.

Accordingly, the camera parameter estimation device 1 uses, for example, the camera C2 as a relay camera to estimate the camera parameters of a camera used for sports relay, and shoots a wide range of the stadium with the camera C1 whose camera parameters are known. The camera parameters of the camera C2 can be estimated without disturbing the competition by using the athlete, the ball, the line or mark drawn on the ground, the equipment placed in the field as a landmark. it can.
The camera parameter estimation apparatus 1 can be operated by a camera parameter estimation program that causes a general computer to function as each of the means described above.

[First Embodiment: Operation of Camera Parameter Estimation Device]
Next, the operation of the camera parameter estimation apparatus according to the first embodiment of the present invention will be described with reference to FIG. 4 (refer to FIG. 2 as appropriate). FIG. 4 is a flowchart showing the operation of the camera parameter estimation apparatus according to the first embodiment of the present invention.

  First, the camera parameter estimation device 1 uses the first landmark extraction means 10 to extract landmark coordinates (reference image coordinates) from a captured image (first image) captured by the camera C1 whose camera parameters are known (first image). Step S1). Then, the camera parameter estimation device 1 uses the position estimation unit 13 to backproject the reference image coordinates extracted in step S1 with the known camera parameters that are camera parameters of the camera C1, thereby converting the actual coordinates of the landmarks (three-dimensional coordinates). ) Is calculated (step S2).

  Further, the camera parameter estimation device 1 extracts the coordinates of the landmarks (observed image coordinates) from the captured image (second image) captured by the camera C2 whose camera parameters are unknown by the second landmark extraction unit 11 ( Step S3).

  Then, the camera parameter estimation device 1 uses the landmark association unit 12 to associate landmarks based on the actual coordinates calculated in step S2 and the observed image coordinates extracted in step S3.

  That is, the camera parameter estimation device 1 predicts the current camera parameter as the predicted camera parameter based on the time-series estimated camera parameter calculated in step S8 described later by the prediction unit 121 of the landmark association unit 12. (Step S4). In step S4, if the estimated camera parameter is not calculated in step S8, the operation is not performed and the process proceeds to step S5 as it is.

  Then, the camera parameter estimation device 1 projects and converts the actual coordinates (three-dimensional coordinates) estimated in step S2 by the projection unit 122 of the landmark association unit 12 using the predicted camera parameters predicted in step S4. Thus, the current coordinates of the reference image coordinates are calculated as predicted image coordinates (step S5). In step S5, when the predicted camera parameter is not predicted in step S4, that is, in the initial state, the predicted image coordinates are calculated using the camera parameter having a predetermined initial value in the projection unit 122.

  And the camera parameter estimation apparatus 1 is based on the distance between the coordinate of the prediction image coordinate calculated by step S5 by the collation means 123 of the landmark matching means 12, and the observation image coordinate extracted by step S3. A landmark is associated (step S6).

  Thereafter, the camera parameter estimation device 1 estimates the camera parameters of the camera C2 by the parameter estimation unit 14 based on the actual coordinates of the landmark calculated in step S2 and the observed image coordinates extracted in step S3.

  That is, the camera parameter estimation device 1 is extracted in step S3 based on the actual coordinates (three-dimensional coordinates) calculated in step S2 and the correspondence results in step S6 by the coordinate association unit 141 of the parameter estimation unit 14. The actual coordinates are associated with the observed image coordinates (step S7).

  Then, the camera parameter estimation apparatus 1 uses the parameter calculation unit 142 of the parameter estimation unit 14 to calculate the camera parameters of the camera C2 from the real coordinates of the plurality of observed image coordinates associated in step S7 by the method of solving the perspective n-point problem. (Estimated camera parameters) are calculated (step S8).

  Thereafter, when the camera parameter estimation device 1 is operating (Yes in step S9), the camera parameter estimation device 1 returns to step S1 and continues the operation. In step S4, the camera parameters of the camera C2 are sequentially predicted based on the estimated camera parameters calculated in step S8.

  Through the above operation, the camera parameter estimation device 1 uses the camera (first image) captured by the camera C1 with known camera parameters and the image (second image) captured by the camera C2 with unknown camera parameters. C2 camera parameters can be estimated.

[Second Embodiment: Configuration of Camera Parameter Estimation Device]
Next, the configuration of the camera parameter estimation device according to the second embodiment of the present invention will be described with reference to FIG. FIG. 5 is a block diagram showing a configuration of a camera parameter estimation apparatus according to the second embodiment of the present invention.

  The camera parameter estimation device 1B shown in FIG. 5 is similar to the camera parameter estimation device 1 shown in FIG. 2, and includes an image (first image) captured by a camera (first camera) C1 with known camera parameters, The camera parameters of the camera C2 are estimated based on the image (second image) captured by the camera (second camera) C2 whose parameter is unknown.

  Here, the camera parameter estimation device 1B includes a first landmark extraction unit 10, a second landmark extraction unit 11, a landmark association unit 12B, a position estimation unit 13, and a parameter estimation unit 14. The camera parameter estimation device 1B connects the camera C1 and the camera C2 to the outside. The configuration other than the landmark associating unit 12B is the same as that of the camera parameter estimation device 1 described in FIG.

  The landmark associating means 12B is for associating landmarks between the first image and the second image. . The landmark associating unit 12B adds a function of collating image features to the function of associating landmarks with the coordinate matching of the landmark associating unit 12 described in FIG. Here, the landmark association unit 12B includes a prediction unit 121, a projection unit 122, a collation unit 123B, a first feature extraction unit 124, and a second feature extraction unit 125. The prediction unit 121 and the projection unit 122 have the same configuration as that of the camera parameter estimation apparatus 1 described in FIG.

The first feature extraction unit 124 extracts each image feature corresponding to the reference image coordinate group {r _m } extracted from the first image I by the first landmark extraction unit 10 as a first feature group {F _m }. To do. Here, the first feature F _m is an image feature in the reference image coordinate r _m (or a region near the reference image coordinate r _m ) corresponding to the first identifier m of the first image I.
As the first feature F _m , for example, a pixel value I (r _m ) at the reference image coordinate r _m can be used as shown in the following equation (15).

At this time, the first feature F _m is a scalar value when the first image I is a monochrome image or represented by a color index, the first image I is a multiband image such as a color image, and the color is a vector. When expressed as a value, it is a vector value.
The first feature F _m, for example, a pixel value group of the neighboring region of the reference image coordinate r _m may be used. In this case, for example, as shown in the following equation (16), with a total of five pixels in the reference image coordinate r _m and its four neighboring pixels to the first feature F _m.

Here, Δx is a vector from the center of a certain pixel to the center of the pixel immediately adjacent to that one pixel, and Δy is a vector from the center of a certain pixel to the center of the next pixel below that one pixel. Incidentally, by expanding the range near, it may constitute a first feature F _m as a template pixel value pattern.
The first feature F _m may be, for example, be used following the as shown in equation (17), the reference image coordinate r color histogram in the vicinity of the _m h _{m (c).} Here, c represents a pixel value (that is, color). At this time, the first feature F _m is a table relating to the color c.

This set of case, the first feature extraction unit 124, first, the reference image coordinate r _m default radius of the circular area around the pixel positions that exist (Other, oval area, the default shape regions such as the rectangular region) Set _Dm . Subsequently, the first feature extraction unit 124 obtains a color histogram h _m (c) of the pixel value I (r) with respect to all the image coordinates r satisfying the image coordinates rεD _m (see formula (18)). In the equation (18), | X | represents the original number of the finite set X.

The second feature extraction unit 125 extracts each image feature corresponding to the observed image coordinate group {o _n } extracted from the second image J by the second landmark extraction unit 11 as a second feature group {G _n }. To do. The second feature is extracted by the same method as the method by which the first feature extraction unit 124 extracted the first feature.

For example, when the first feature extraction unit 124 extracts the first feature according to the equation (15), the second feature extraction unit 125 uses the observation as the second feature G _n as shown in the following equation (19). using the pixel value J _{(o n)} in the image coordinate _{o n.}

Also, if the first feature extraction means 124 for extracting the first feature by the formula (16), the second feature extraction unit 125, as shown in the following equation (20), the observed image coordinates o _n and 4 A total of five pixels in the vicinity is set as the second feature _Gn .

Further, when the first feature extraction unit 124 extracts the first feature according to the equation (17), the second feature extraction unit 125 performs the observation as the second feature G _n as shown in the following equation (21). using _g n (c) a color histogram in the vicinity of the image coordinates _{o n.} Here, c represents a pixel value (that is, color). At this time, the second feature G _n is a table relating to the color c.

This set of case, the second feature extraction unit 125, first, the observed image coordinates o _n default radius of the circular area around the pixel positions that exist (Other, oval area, the default shape regions such as the rectangular region) setting the E _n. Subsequently, the second feature extraction unit 125 obtains a color histogram g _n (c) of the pixel value J (o) with respect to all the image coordinates o satisfying the image coordinates oεE _n (see formula (22)).

  The collating unit 123B includes a degree of similarity between the first feature extracted by the first feature extracting unit 124 and the second feature extracted by the second feature extracting unit 125, and the predicted image coordinates calculated by the projecting unit 122. The landmarks are associated with each other on the basis of weighted addition with the distance between the coordinates of the observed image coordinates extracted by the second landmark extracting means 11.

For example, the matching unit 123B extracts the set of pairs (m, F _m ) of the first identifier m and the first feature F _m extracted by the first feature extraction unit 124 and the second feature extraction unit 125. A set of pairs (n, G _n ) of the second identifier n and the second feature G _n and a set of pairs (m, o _m ′) of the first identifier m calculated by the projection means 122 and the predicted image coordinates. If the second identifier n and the observed image coordinates o _n the pair extracted in the second landmark extracting unit 11 (n, o _n) based on a set of the image feature indicated by the following equation (23) Performs weighted addition with the distance between coordinates. Thereby, the image feature landmarks are similar, and the distance interval can be obtained second identifier n _m corresponding to the shorter first identifier m.

Here, α and β are positive constants. Further, (23) where, ‖G _n -F _m ‖ _Feature shows norm for the feature quantity of the _{G n} and _{F m.} For example, when both the first feature F _m and the second feature G _n are scalar values, the matching unit 123B can set the absolute value of the distance as a norm. When both the first feature F _m and the second feature G _n are vector values (for example, a pixel value of a plurality of pixels, a pixel value of a color image, etc.), the matching unit 123B uses the Euclidean distance and the Manhattan distance as norms. Etc. can be used. When both the first feature F _m and the second feature G _n are histograms (for example, a color histogram), the matching unit 123B can use the Bhattacharyya distance as the norm.

Further, (23) where, ‖o _n -o _m '‖ _Position is, _{o n} a _{o m'} indicating the norm for the position of the. As the norm for this position, for example, the Euclidean distance can be used.
Then, the matching unit 123B uses (1, n ₁ ), (2, n ₂ ),..., (M, n _M ), which is a pair of the corresponding first identifier m and second identifier _nm , as a correspondence result. It outputs to the parameter estimation means 14.

By configuring the camera parameter estimation device 1B in this way, the camera parameter estimation device 1B is configured so that the image captured by the camera C1 whose camera parameter is known (first image) and the image captured by the camera C2 whose camera parameter is unknown. From the (second image), it is possible to accurately estimate the camera parameters of the camera C2 by associating the landmarks with accuracy based on the landmark positions and the image characteristics.
The camera parameter estimation device 1B can be operated by a camera parameter estimation program that causes a general computer to function as each of the means described above.

[Second Embodiment: Operation of Camera Parameter Estimation Device]
Next, the operation of the camera parameter estimation apparatus according to the second embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the camera parameter estimation apparatus according to the second embodiment of the present invention.

  The operation of the camera parameter estimation device 1B shown in FIG. 6 is basically the same as the operation of the camera parameter estimation device 1 shown in FIG. 4, but step S1B is added after step S1, and step S3B is The difference is that it is added after step S3, and step S6B operates instead of step S6. The other steps are the same as the operation of the camera parameter estimation device 1 shown in FIG.

  After step S1, the camera parameter estimation device 1B uses the first feature extraction unit 124 to extract the feature (first feature) of the reference image coordinates extracted in step S1 from the first image (step S1B). In addition, after step S3, the camera parameter estimation device 1B extracts the feature (second feature) of the observed image coordinates extracted in step S3 from the second image by the second feature extraction unit 125 (step S3B). .

  Then, after step S5, the camera parameter estimation device 1B uses the matching unit 123B to determine the degree of similarity between the first feature extracted in step S1B and the second feature extracted in step S3B, and the predicted image calculated in step S5. Landmarks are associated with each other based on the weighted addition of the coordinates and the distance between the coordinates of the observation image coordinates extracted in step S3 (step S6B).

  The operation in step S1B may be performed at any timing after step S1 and before step S6B. The operation of step S3B may be performed at any timing after step S3 and before step S6B.

  With the above operation, the camera parameter estimation device 1B uses the image (first image) captured by the camera C1 whose camera parameter is known and the image (second image) captured by the camera C2 whose camera parameter is unknown. C2 camera parameters can be estimated.

It is explanatory drawing for demonstrating the outline | summary of this invention. It is a block diagram which shows the structure of the camera parameter estimation apparatus which concerns on 1st embodiment of this invention. It is explanatory drawing for demonstrating the method of the back projection in the position estimation means of the camera parameter estimation apparatus which concerns on 1st embodiment of this invention. It is a flowchart which shows operation | movement of the camera parameter estimation apparatus which concerns on 1st embodiment of this invention. It is a block diagram which shows the structure of the camera parameter estimation apparatus which concerns on 2nd embodiment of this invention. It is a flowchart which shows operation | movement of the camera parameter estimation apparatus which concerns on 2nd embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 1B Camera parameter estimation apparatus 10 1st landmark extraction means 11 2nd landmark extraction means 12, 12B Landmark matching means 121 Prediction means 122 Projection means 123 Collation means 124 1st feature extraction means 125 2nd feature extraction means 13 Position estimation means 14 Parameter estimation means 141 Coordinate association means 142 Parameter calculation means C1 Camera (first camera)
C2 camera (second camera)

Claims (5)

Camera parameter estimation for estimating camera parameters of the second camera based on a first image captured by a first camera with known camera parameters and a second image captured by a second camera with unknown camera parameters A device,
First landmark extraction means for extracting the coordinates of the landmark in the image from the first image as reference image coordinates based on predetermined image features of the landmark;
Position estimation means for estimating the three-dimensional coordinates of the landmarks by backprojecting the reference image coordinates extracted by the first landmark extraction means with known camera parameters that are camera parameters of the first camera;
Second landmark extraction means for extracting the coordinates of the landmark in the image from the second image as observation image coordinates based on the image characteristics of the landmark;
By converting the three-dimensional coordinates estimated by the position estimating means into predicted image coordinates that are two-dimensional coordinates in the second image, and collating with the observed image coordinates, the first image and the second image Landmark association means for associating the landmarks between,
Based on the three-dimensional coordinates and the observed image coordinates corresponding to the landmark, the estimated camera parameter that is the camera parameter of the second camera is set in the same landmark correlated by the landmark correlation unit. Parameter estimating means for estimating,
The landmark association means includes
Projection means for calculating the predicted image coordinates by projecting the three-dimensional coordinates by the estimated camera parameters;
Matching means for associating the landmarks based on a distance between coordinates between the predicted image coordinates and the observed image coordinates;
A camera parameter estimation device comprising: The landmark association unit further includes a prediction unit that predicts a camera parameter at the present time as a predicted camera parameter based on the time-series estimated camera parameter estimated by the parameter estimation unit,
2. The camera parameter estimation apparatus according to claim 1, wherein the projection unit calculates the predicted image coordinates by projecting and converting the three-dimensional coordinates using the predicted camera parameters predicted by the prediction unit. The landmark association means includes
A first feature extracting means for extracting, as a first feature, an image feature in the vicinity region of the predicted image coordinates from the first image;
A second feature extraction means for extracting, from the second image, an image feature in the vicinity region of the observation image coordinates as a second feature;
The collating means associates the landmarks based on a weighted addition of a degree of similarity between the first feature and the second feature and a distance between the predicted image coordinate and the observed image coordinate. The camera parameter estimation apparatus according to claim 1 or 2. The parameter estimation means includes
Based on the three-dimensional coordinates estimated by the position estimating means and the correspondence result by the landmark associating means, coordinate associating means for associating the three-dimensional coordinates with the observed image coordinates;
Parameter calculating means for calculating the estimated camera parameter based on the three-dimensional coordinates of the plurality of observed image coordinates associated by the coordinate associating means;
The camera parameter estimation apparatus according to any one of claims 1 to 3, further comprising: In order to estimate the camera parameters of the second camera based on the first image captured by the first camera whose camera parameters are known and the second image captured by the second camera whose camera parameters are unknown, Computer
First landmark extraction means for extracting, from the first image, a coordinate of the landmark in the image as a reference image coordinate based on a predetermined image feature of the landmark;
Position estimating means for estimating the three-dimensional coordinates of the landmarks by backprojecting the reference image coordinates extracted by the first landmark extracting means with known camera parameters that are camera parameters of the first camera;
Second landmark extraction means for extracting, from the second image, the coordinates of the landmark in the image as observation image coordinates based on the image characteristics of the landmark;
By converting the three-dimensional coordinates estimated by the position estimating means into predicted image coordinates that are two-dimensional coordinates in the second image, and collating with the observed image coordinates, the first image and the second image Landmark associating means for associating the landmarks between,
Based on the three-dimensional coordinates and the observed image coordinates corresponding to the landmark, the estimated camera parameter that is the camera parameter of the second camera is set in the same landmark correlated by the landmark correlation unit. Function as a parameter estimation means to estimate,
The landmark association means includes
Based on the time-series estimated camera parameters calculated by the parameter calculation means, the current camera parameters are predicted as predicted camera parameters, and the predicted image coordinates are calculated by projecting the three-dimensional coordinates using the predicted camera parameters. The camera parameter estimation program further associates the landmarks based on a distance between the predicted image coordinates and the observed image coordinates.

Images