JP2009124204A - Apparatus and program for estimating camera parameter - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus for estimating camera parameters capable of estimating the camera parameter while using an arbitrary stationary object or moving object as a landmark. <P>SOLUTION: The apparatus 1 for estimating the camera parameters includes: a first landmark extracting means 10 for extracting the coordinates of a landmark as coordinates of a reference image from a first image; a second landmark extracting means 11 for extracting the coordinates of a landmark as coordinates of an observation image from a second image; a landmark associating means 12 for associating the landmark on the basis of the similarity of image feature; a position estimating means 13 for estimating the three-dimensional coordinates of the landmark on the basis of a known camera parameter as a camera parameter of a first camera; and a parameter estimating means 14 for calculating a camera parameter of a second camera on the basis of the three-dimensional coordinates estimated by the position estimating means 13. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a camera parameter estimation device and a camera parameter estimation program for estimating camera parameters when a subject is imaged with a camera.

  In general, when an image obtained by capturing a subject with a camera and another image are combined, they are combined based on camera parameters such as the posture, position, and angle of view when the camera is picked up. In this case, it is common to use camera parameters calibrated in advance as cameras with known camera positions or the like, or use camera parameters estimated from images captured by cameras with unknown camera positions or the like.

  There are various methods for estimating camera parameters from an image. For example, by using a plurality of known objects (landmark groups) whose three-dimensional relative positional relationship does not change with time, the coordinates (image coordinate groups) of each object in the image are observed to capture the image. There is a method for estimating the camera parameters of the camera (see Patent Document 1). Further, there is a technique for estimating the relative position and orientation of a camera with respect to a landmark group by solving a perspective four-point problem (P4P problem: Perspective Four-Point Problem) (see Non-Patent Document 1). In addition, for example, there is a method for estimating the relative position and orientation of a camera from a temporal trajectory of a rectangular marker (see Patent Document 2).

  Also, the image coordinate group of the landmark group using a landmark group in which the relative positional relation of the three-dimensional relative position relationship does not change with time and the relative positional relationship of each other is unknown, and moves relative to the landmark group. There is a method of estimating the relative position and posture of the camera with respect to the landmark group and the relative positional relationship between the landmarks by observing the image over a plurality of time points (see Non-Patent Document 2).

Furthermore, an image of a linear landmark group with a known three-dimensional shape, such as a line drawn on a sports stadium, is detected by Hough transform or generalized Hough transform, and the landmark group is determined based on the Hough parameter. There is a method for estimating the relative position and orientation of the camera with respect to (see Patent Document 3).
JP 2006-33449 A JP 2003-196663 A JP 2004-234333 A H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, K. Tachibana. Virtual Object Manipulation on a Table-Top AR Environment.In Proceedings of ISAR 2000, Oct 5th-6th, 2000. J. Weng, TS Huang, N. Ahuja.Motion and Structure from Two Perspective Views: Algorithms, Error Analysis, and Error Estimation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 11, Issue 5, pp. 451-476, May 1989.

  However, the methods described in Patent Documents 1 and 2 and Non-Patent Document 1 described above are effective when the imaging range of the camera is narrow because the arrangement of landmark groups is fixed. In order to acquire camera parameters for posture, position, angle of view, etc., it is necessary to install a lot of landmarks over a wide area so that a predetermined number of landmarks are included in the presumed shooting area. There is.

  Further, the technique described in Non-Patent Document 2 has a problem that in principle, camera parameters cannot be acquired unless the camera (specifically, the first optical principal point) moves relative to the landmark group. is there. Also, use this method when the camera does not move sufficiently, such as when the amount of movement is small, when there is no camera operation, or when the camera operation is limited to pan, tilt, and zoom. There is a problem that can not be.

  Furthermore, in the method described in Patent Document 3, as in the methods described in Patent Documents 1 and 2 and Non-Patent Document 1, a predetermined number or more of landmarks are included in a presumed shooting area. In addition, the shape of the landmark is limited to a predetermined straight line or curve.

  The present invention has been made to solve the above-described problems, and it is possible to estimate camera parameters using an arbitrary fixed object or moving object as a landmark without fixing the position of the landmark. It is an object to provide a camera parameter estimation device and a camera parameter estimation program.

  The present invention was devised to achieve the above-mentioned object. First, the camera parameter estimation device according to claim 1 includes a first image captured by a first camera having a known camera parameter, a camera A camera parameter estimation device that estimates camera parameters of the second camera based on a second image captured by a second camera whose parameters are unknown, the first landmark extraction means, and a second landmark extraction Means, landmark association means, position estimation means, and parameter estimation means.

  In such a configuration, the camera parameter estimation device extracts the coordinates of the landmark in the image from the first image as the reference image coordinates based on the image characteristics of the landmark by the first landmark extraction unit. In the camera parameter estimation device, the second landmark extracting unit extracts the coordinates of the landmark in the image from the second image as the observed image coordinates based on the image characteristics of the landmark. The image features of the landmarks can be predetermined features such as brightness, color, pattern, and the like. Note that the first landmark extracting unit and the second landmark extracting unit need only extract the position of the coordinates of the landmarks, and therefore, it is not always necessary to use the same image feature.

  Then, the camera parameter estimation device is based on the degree of similarity between the image feature corresponding to the reference image coordinates on the first image and the image feature corresponding to the observed image coordinates on the second image by the landmark association unit. Thus, the landmark is associated between the first image and the second image. As a result, when a plurality of landmarks are extracted, the landmark on the first image and the landmark on the second image are associated one-to-one.

  Further, the camera parameter estimation device performs backprojection conversion of the reference image coordinates under a predetermined constraint condition based on the known camera parameter that is the camera parameter of the first camera by the position estimation unit, thereby obtaining a three-dimensional landmark. Estimate the coordinates. For example, the position estimation means can estimate a three-dimensional coordinate from a two-dimensional coordinate by giving a constraint condition that the landmark exists on the ground.

  Then, the camera parameter estimation device uses the second camera based on the three-dimensional coordinates corresponding to the landmark and the observed image coordinates in the same landmark correlated by the landmark correlation unit by the parameter estimation unit. Estimated camera parameters that are camera parameters are estimated. It should be noted that a known solution of the perspective n-point problem can be applied to the estimation of the estimated camera parameter in the parameter estimation means.

  The camera parameter estimation device according to claim 2 is the camera parameter estimation device according to claim 1, wherein the landmark association unit includes a first feature extraction unit, a second feature extraction unit, and a collation unit. It was set as the structure provided with.

  In such a configuration, the camera parameter estimation device extracts, from the first image, the image feature in the vicinity region of the reference image coordinates as the first feature by the first feature extraction unit in the landmark association unit. In addition, the camera parameter estimation device extracts, as the second feature, the image feature in the vicinity region of the observed image coordinates from the second image by the second feature extraction unit. As a result, the image feature becomes a feature amount allowing a certain amount of error. Then, the camera parameter estimation device associates the landmark with the matching unit based on the degree of similarity between the first feature and the second feature.

  Furthermore, the camera parameter estimation device according to claim 3 is the camera parameter estimation device according to claim 1 or 2, wherein the parameter estimation means includes a coordinate association means and a parameter calculation means. did.

  In such a configuration, the camera parameter estimation device uses the parameter estimation unit to calculate the observed image coordinates based on the three-dimensional coordinates estimated by the position estimation unit and the correspondence results by the landmark association unit. Map 3D coordinates. As a result, the two-dimensional observation image coordinates obtained from the second image captured by the second camera are converted into three-dimensional coordinates in the real space.

  And a camera parameter estimation apparatus calculates an estimated camera parameter from the three-dimensional coordinate of several observation image coordinate by a parameter calculation means. In this way, the parameter calculation means sequentially inputs the three-dimensional coordinates of the observed image coordinates associated by the coordinate association means, and at the stage when the coordinates for solving the perspective n-point problem are input, Estimated camera parameters are calculated using a solution for the n-point problem.

  According to a fourth aspect of the present invention, there is provided a camera parameter estimation program based on a first image captured by a first camera whose camera parameters are known and a second image captured by a second camera whose camera parameters are unknown. The computer functions as first landmark extraction means, second landmark extraction means, landmark association means, position estimation means, parameter estimation means in order to estimate camera parameters of the second camera; did.

  In such a configuration, the camera parameter estimation program extracts, from the first image, the coordinates of the landmark in the image as the reference image coordinates by the first landmark extraction means based on the image characteristics of the landmark. Further, the camera parameter estimation program extracts the coordinates of the landmark in the image from the second image as the observed image coordinates by the second landmark extraction unit based on the image characteristics of the landmark.

  The camera parameter estimation program is based on the degree of similarity between the image feature corresponding to the reference image coordinates on the first image and the image feature corresponding to the observed image coordinates on the second image by the landmark association unit. Thus, the landmark is associated between the first image and the second image.

  Further, the camera parameter estimation program performs backprojection conversion of the reference image coordinates under a predetermined constraint condition based on the known camera parameter that is the camera parameter of the first camera by the position estimation unit, thereby obtaining a three-dimensional landmark. Estimate the coordinates.

  Then, the camera parameter estimation program associates the three-dimensional coordinates with the observed image coordinates based on the three-dimensional coordinates estimated by the position estimation unit and the correspondence results by the landmark association unit by the coordinate association unit. Then, the camera parameter estimation program calculates estimated camera parameters from the three-dimensional coordinates of the plurality of observed image coordinates by the parameter calculation means.

  According to the first, third, and fourth aspects, it is possible to extract and associate the same landmark from an image captured by a camera with a known camera parameter and an image captured by a camera with an unknown camera parameter. Therefore, it is possible to estimate an unknown camera parameter using any fixed object or moving object as a landmark without fixing the position of the landmark. Thereby, since it is not necessary to install a unique landmark, unnecessary marks can be removed at the imaging site, and natural images can be captured. Further, according to the first, third, and fourth aspects of the invention, a camera with known camera parameters and an unknown camera can sequentially capture images if each captured image includes at least one landmark. The positions of a plurality of landmarks can be detected from the captured image, and unknown camera parameters can be estimated. Thus, the number of landmarks when estimating camera parameters can be reduced.

  According to the second aspect of the present invention, since the image feature of the reference image coordinates in the first image and the image feature of the observed image coordinates in the second image are respectively image features in the vicinity region of the coordinate position, And the landmarks can be associated with each other with high accuracy in the first image and the second image. Thereby, the estimation accuracy of the unknown camera parameter can be improved.

Embodiments of the present invention will be described below with reference to the drawings.
[Outline of the present invention]
First, the outline of the present invention will be described with reference to FIG. FIG. 1 is an explanatory diagram for explaining the outline of the present invention.

  The camera parameter estimation device 1 includes an image (first image) captured by a camera (first camera) C1 with known camera parameters and an image (first image) captured by a camera (second camera) C2 with unknown camera parameters. The camera parameters of the camera C2 are estimated based on the two images.

  Here, the camera parameter estimation device 1 extracts a plurality of moving objects and stationary objects as landmarks M (players in FIG. 1) from an image captured by the camera C1 and an image captured by the camera C1. The landmark is an area having a shading or color change that can be distinguished from other local areas on the image.

  Then, the camera parameter estimation device 1 calculates coordinates (three-dimensional coordinates) in the real space of the landmark M based on the landmark M captured by the camera C1 and the known camera parameters of the camera C1, and the camera C2 The camera parameters of the camera C2 are estimated by associating with the image coordinates of the landmark M captured in step S2.

  FIG. 1 shows an example in which the cameras C1 and C2 are installed in the soccer field and the player is a landmark. However, if a part of the landmark M can be observed simultaneously by the cameras C1 and C2, It is not limited to the installation location.

[About camera parameters]
Next, camera parameters in the camera parameter estimation apparatus according to the embodiment of the present invention will be described.

  The camera parameters are external parameters determined by the installation position and installation posture of the camera, the focal length and distortion of the lens, the shape of the image sensor, the pixel size and the number of pixels, and the relative positional relationship between the lens and the image sensor. It is an index that includes both internal parameters. In the present embodiment, the camera installation position and installation posture, and the lens focal length are used as the camera parameters. The installation position and installation posture of the camera can be defined with reference to an arbitrary coordinate system (hereinafter referred to as a world coordinate system) fixedly defined in the real space.

  In this world coordinate system, a representative arbitrary point in a space where the cameras C1 and C2 (see FIG. 1) are installed is set as an origin, and three representative and independent directions are set as axial directions. For example, when the cameras C1 and C2 are installed in a soccer stadium as shown in FIG. 1, the position of the center mark of the soccer court is the origin of the world coordinate system, the touch line direction (x axis), the half way line direction ( The y-axis) and the vertically upward direction (z-axis) are the three axes of the world coordinate system. For example, when the cameras C1 and C2 are installed in a room such as a studio, the center of the floor in the room is set as the origin of the world coordinate system, and the horizontal east direction, the horizontal north direction, and the vertical upward direction are set in the world coordinate system. Three axes are assumed.

The camera installation posture can be expressed by posture conversion between a camera coordinate system fixed to the camera and the above-described world coordinate system. For example, as shown in the following equation (1), the installation posture of the camera is a vector when a vector v ^(w) when a certain direction vector in real space is viewed from the world coordinate system is viewed in the camera coordinate system. v ^(c) can be expressed by a 3 × 3 rotation matrix R to be converted.

The camera installation position can be defined as a position vector when the origin position of the camera coordinate system is viewed in the world coordinate system. For example, the installation position of the camera uses the rotation matrix R and the position vector t of the installation position of the camera, so that the position vector p ^(w) in the world coordinate system as shown in the following equation (2 ^). Can be converted into a position vector q ^(c) in the camera coordinate system. In addition, although the unit of each component of a position vector is arbitrary, it can be made into a unit, for example.

The camera coordinate system used in the above-described equations (1) and (2) has, for example, the first optical principal point of the camera lens as the origin and two axes in two independent directions on the camera image plane. The remaining one axis can be taken in the optical axis direction. The two independent directions on the image plane of the camera can be defined by, for example, the horizontal direction and the vertical direction of the camera image sensor.
Hereinafter, the configuration and operation of the camera parameter estimation apparatus according to the embodiment of the present invention will be described in detail.

[Configuration of Camera Parameter Estimation Device]
First, the configuration of the camera parameter estimation apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration of the camera parameter estimation apparatus according to the embodiment of the present invention.

  Here, the camera parameter estimation device 1 includes a first landmark extraction unit 10, a second landmark extraction unit 11, a landmark association unit 12, a position estimation unit 13, and a parameter estimation unit 14. The camera parameter estimation device 1 connects the camera C1 and the camera C2 to the outside.

  The camera C1 is a camera whose camera parameters are known. The camera C2 is a camera whose camera parameters are unknown. An image captured by the camera C1 (first image) and an image captured by the camera C2 (second image) are input to the camera parameter estimation device 1.

  Here, the image coordinate system of the first image and the second image uses a two-dimensional coordinate system on the image plane, and here, two axes taken on the image plane among the three axes constituting the camera coordinate system. Is the image coordinate system. In the following description, the first image captured by the camera C1 is denoted by I, and the pixel value at the image coordinate r of the first image I is denoted by I (r). In addition, the second image captured by the camera C2 is denoted as J, and the pixel value at the image coordinate o of the second image J is denoted as J (o).

  Note that when the cameras C1 and C2 are monochrome cameras, the pixel values can be represented by, for example, discrete scalar values of two or more values (for example, 256 values). When the cameras C1 and C2 are color cameras or multi-band cameras, the pixel values are, for example, two or more dimensions (for example, three dimensions of red, green, and blue) and two or more dimensions (for example, 256 values). It can be represented by a discrete vector value. In addition, when the cameras C1 and C2 are color cameras or multiband cameras, a color table (color conversion table: a table associating discrete index values with pixel colors) is used, so that the pixel values are two or more (for example, 65536 values) may be represented by a color index of discrete scalar values.

  In addition, since the camera C1 and the camera C2 associate landmarks, it is necessary to capture at least one landmark in common. For example, when used in the soccer stadium shown in FIG. 1, the camera C1 captures the entire soccer field, and the camera C2 captures a part including the landmark in the soccer field. More than one landmark can be imaged in common.

The first landmark extracting means 10 determines the coordinates of a plurality of landmarks in the image (hereinafter referred to as reference) from the first image captured by the camera C1 whose camera parameters are known based on the predetermined image characteristics of the landmark. (Referred to as image coordinates). The first landmark extraction means 10 attaches an identifier (first identifier) to the extracted reference image coordinates and outputs the identifier to the landmark association means 12 and the position estimation means 13. Further, the number of landmarks that have been extracted by the first landmark extracting unit 10 and the M, its m-th (m = 1, 2, ..., M) the reference image coordinates of landmarks and r _m. The m is used as a first identifier.

  Note that the first landmark extraction unit 10 uses, for example, a Moravec instant operator that uses a minimum value of directional dispersion in a local region on an image as an image feature and selects a point at which the image feature is maximized. Can be extracted. Further, when a marker such as an object painted in a specific color or a disk having a concentric circular pattern is artificially installed in the object scene, the first landmark extraction means 10 selects an operator for detecting a specific color or pattern. It may be applied to an image and a detected point or a representative point (for example, the center of gravity) of a detection area may be extracted as a landmark.

  Further, the first landmark extracting means 10 may extract a moving object such as an animal or a person (or a person's face) as a landmark. For example, when a person's face is used as a landmark, the first landmark extracting means 10 uses a general face detector (for example, P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proc. Of IEEE Conference on Computer Vision and Pattern Recognition, Kauai, HI, December 2001.), representative points of detected face regions can be used as landmarks.

The second landmark extracting means 11 is configured to determine the coordinates (hereinafter referred to as observation) of a plurality of landmarks from the second image captured by the camera C2 whose camera parameters are unknown based on predetermined image characteristics of the landmarks. (Referred to as image coordinates). The second landmark extraction unit 11 attaches an identifier (second identifier) to the extracted observation image coordinates and outputs the identifier to the landmark association unit 12 and the parameter estimation unit 14. Further, the number of landmarks that have been extracted by the second landmark extracting unit 11 and the N to the n-th (n = 1, 2, ..., N) the observed image coordinates of the landmarks in the o _n. The n is used as a second identifier.

  The landmark extraction process in the second landmark extraction unit 11 may be the same process as the first landmark extraction unit 10 or a different process. For example, when a red disk that is an artificial pattern is used as a landmark, the first landmark extraction unit 10 extracts a landmark by focusing on a circular pattern that is the shape of a red disk, and extracts a second landmark. The means 11 may extract landmarks by paying attention to the red color of the red disk.

The landmark associating unit 12 determines whether the first image and the second image are based on the degree of similarity between the image feature of the first image corresponding to the reference image coordinates and the image feature of the second image corresponding to the observation image coordinates. Are associated with landmarks. The landmark correlation means 12 corresponds to the same point on the real space (location), or obtains all the set of the reference image coordinates r _m corresponding allowing certain errors the observed image coordinates o _n The landmarks are associated by outputting all pairs of the first identifier m and the second identifier n. Here, the landmark association unit 12 includes a first feature extraction unit 121, a second feature extraction unit 122, and a collation unit 123.

The first feature extraction unit 121 extracts each image feature corresponding to the reference image coordinate group {r _m } extracted from the first image I by the first landmark extraction unit 10 as a first feature group {F _m }. To do. Here, the first feature F _m is an image feature in the reference image coordinate r _m (or a region near the reference image coordinate r _m ) corresponding to the first identifier m of the first image I.
As the first feature F _m , for example, a pixel value I (r _m ) at the reference image coordinate r _m can be used as shown in the following equation (3).

At this time, the first feature F _m is a scalar value when the first image I is a monochrome image or represented by a color index, the first image I is a multiband image such as a color image, and the color is a vector. When expressed as a value, it is a vector value.
The first feature F _m, for example, a pixel value group of the neighboring region of the reference image coordinate r _m may be used. In this case, for example, as shown in the following equation (4), with the reference image coordinate r _m and a total of five pixels in the four neighboring pixels to the first feature F _m.

Here, Δx is a vector from the center of a certain pixel to the center of the pixel immediately adjacent to that one pixel, and Δy is a vector from the center of a certain pixel to the center of the next pixel below that one pixel. Incidentally, by expanding the range near, it may constitute a first feature F _m as a template pixel value pattern.
The first feature F _m may be, for example, it is used the following (5) As shown in equation, the reference image coordinate r color histogram in the vicinity of the _m h _{m (c).} Here, c represents a pixel value (that is, color). At this time, the first feature F _m is a table relating to the color c.

This set of case, the first feature extraction means 121, first, the reference image coordinate r _m default radius of the circular area around the pixel positions that exist (Other, oval area, the default shape regions such as the rectangular region) Set _Dm . Subsequently, the first feature extraction unit 121 obtains a color histogram h _m (c) of the pixel value I (r) with respect to all the image coordinates r satisfying the image coordinates rεD _m (see the equation (6)). In this equation (6), | X | represents the original number of the finite set X.

Second feature extraction means 122, extracts the respective image feature from the second image J corresponding to the second landmark extracting unit 11 extracts the observed image coordinate group {o _n} as the second feature groups {G _n} To do. The second feature is extracted by the same method as the method by which the first feature extraction unit 121 extracts the first feature.

For example, when the first feature extraction unit 121 extracts the first feature according to the equation (3), the second feature extraction unit 122 observes the second feature G _n as shown in the following equation (7). using the pixel value J _{(o n)} in the image coordinate _{o n.}

Also, if the first feature extraction means 121 for extracting the first feature by the equation (4), the second feature extraction means 122, as shown in the following equation (8), the observed image coordinates o _n and 4 A total of five pixels in the vicinity is set as the second feature _Gn .

Further, when the first feature extraction unit 121 extracts the first feature according to the equation (5), the second feature extraction unit 122 uses the observation as the second feature G _n as shown in the following equation (9). using _g n (c) a color histogram in the vicinity of the image coordinates _{o n.} Here, c represents a pixel value (that is, color). At this time, the second feature G _n is a table relating to the color c.

This set of case, the second feature extraction unit 122 first observed image coordinates o _n default radius of the circular area around the pixel positions that exist (Other, oval area, the default shape regions such as the rectangular region) setting the E _n. Subsequently, the second feature extraction unit 122 obtains a color histogram g _n (c) of the pixel value J (o) with respect to all the image coordinates o satisfying the image coordinates oεE _n (see formula (10)).

  The matching unit 123 associates landmarks based on the degree of similarity between the first feature extracted by the first feature extraction unit 121 and the second feature extracted by the second feature extraction unit 122. Here, the matching unit 123 searches for the second feature that is closest (similar) to the first feature based on the distance (dissimilarity) of the image feature, and sets the pair of the corresponding first identifier and second identifier. Is output to the parameter estimation means 14 as a correspondence result.

That is, the matching means 123 calculates the distance d between the features of the second feature _{G n} of the first feature _{F m} and the second identifier n of the first identifier m (m, n) (( 11) see formula), searching a second characteristic G _n that is closest to the first feature F _m of the first identifier m, the second identifier n corresponding to the second feature G _n, a second identifier corresponding to the first identifier m n _m (see equation (12)).

Then, the matching unit 123 uses (1, n ₁ ), (2, n ₂ ),..., (M, n _M ), which is a pair of the corresponding first identifier m and second identifier _nm , as the correspondence result. It outputs to the parameter estimation means 14. However, the distance d (m, n _m) between the features of the first identifier m and the second identifier n _m when not less than the threshold value in the (or when exceeding a certain threshold), a first identifier m the corresponding a pair of the second identifier n _{m (m,} n _m) may be not output. As a result, the accuracy of landmark association can be increased.
Note that ‖G _n −F _mで shown in the equation (11) represents the distance between the first feature F _m and the second feature G _n . For example, when both the first feature F _m and the second feature G _n are scalar values, the matching unit 123 can obtain the distance d by calculating the absolute value of the distance.

When both the first feature F _m and the second feature G _n are vector values (for example, pixel values of a plurality of pixels, pixel values of a color image, etc.), the matching unit 123 uses the Euclidean distance, Manhattan as the distance d. Distance or the like can be used. When both the first feature F _m and the second feature G _n are histograms (for example, a color histogram), the matching unit 123 can use the Bhattacharyya distance as the distance d.

The position estimation unit 13 performs backprojection conversion on the reference image coordinates extracted by the first landmark extraction unit 10 using known camera parameters that are camera parameters of the camera C1, thereby changing the actual coordinates (three-dimensional coordinates) of the landmarks. To be estimated. The position estimating means 13, the actual coordinates p _m Landmarks first identifier m the number of the first identifier (m, p _m) determined as the data pairs, and outputs to the parameter estimation unit 14. The position estimation unit 13 may sequentially input known camera parameters from the camera C1. When the camera C1 is a fixed camera, the position estimation unit 13 stores the known camera parameters in a storage unit (not shown) in advance. It is good also as reading.

Note that the reference image coordinate r _m whereas the two-dimensional coordinates, the actual coordinate p _m landmarks is a three-dimensional coordinates. For this reason, in order for the position estimation means 13 to perform the back projection, a planar constraint condition on the actual coordinates where the landmark exists is necessary. For example, by giving a constraint condition that the landmark is on a plane such as a floor or the ground, the position estimating unit 13 can back-project the reference image coordinates to the real coordinates.

Here, with reference to FIG. 3, a method of back projecting the reference image coordinates to the real coordinates in the position estimating means 13 will be described in detail. FIG. 3 is an explanatory diagram for explaining a back projection technique in the position estimating means. In FIG. 3, it is assumed that a landmark M exists on a curved surface K, and a landmark image m is captured on the first image I captured by the camera C1.
In the example of FIG. 3, the curved surface K where the landmark M (real coordinate p) exists is expressed by the following equation (13) using the function f.

  For example, when the three axes of the world coordinate system are the X-axis, the Y-axis, and the Z-axis, in order to constrain the landmark M to the plane Z = 0, the curved surface K is changed to a function f shown in the following equation (14). That's fine.

  Here, the origin of the camera coordinate system of the camera C1 is taken as the first optical principal point T, and the position vector in the world coordinate system is t. Also, let R be the rotation matrix for posture conversion from the world coordinate system to the camera coordinate system. The x and y axes of the camera coordinate system of the camera C1 are in a plane parallel to the image plane of the first image I, and the z axis is parallel to the optical axis and oriented in the field. It is assumed that the optical axis is perpendicular to the image plane.

  Further, the image coordinate system on the image plane of the first image I is a two-dimensional coordinate system stretched by only two axes of the x-axis and the y-axis in the camera coordinate system, and the origin is between the image plane and the optical axis. Take the intersection. The lens of the camera C1 follows the pinhole model, and its focal length is f.

  Further, it is assumed that the reference image coordinate in the first image I input to the position estimation unit 13 is r, and the camera parameter input as the known camera parameter includes the position vector t and the rotation matrix R. .

  As shown in FIG. 3, the landmark M starts from the first optical principal point T and is k times the vector reaching the landmark image m on the image plane of the first image I (k is a positive real coefficient). The real coordinate p can be expressed by the following equation (15).

  Here, the coefficient k (where k> 0) can be obtained by simultaneously combining the equation (13) and the equation (14). When there are two or more solutions of the simultaneous equations with respect to the coefficient k, the smallest value among them is used. The actual coordinate p is obtained by substituting the coefficient k thus obtained in the equation (15). For example, when the landmark is on a plane where Z = 0, the relationship of the following equation (16) is established by the equation (14). Further, from the equation (16), the coefficient k is obtained as shown in the following equation (17).

  Then, by substituting the coefficient k obtained in the equation (17) into the equation (15), the real coordinate p is obtained as shown in the following equation (18).

  This equation (18) takes the X-axis and Y-axis of the world coordinate system on the floor surface, for example, when there is a landmark on the flat floor surface, and takes the Z-axis upward, for example, perpendicular to them. It can be applied.

Here, the method of obtaining the real coordinates by giving the constraint condition of the plane Z = 0 has been described, but the constraint condition may be given for X and Y. Thereby, for example, the position estimating means 13 can obtain the actual coordinates of the landmark on the wall surface existing in the vertical direction from the floor surface.
Returning to FIG. 2, the description of the configuration of the camera parameter estimation apparatus 1 will be continued.

  The parameter estimation unit 14 includes the actual coordinates (three-dimensional coordinates) of the reference image coordinates estimated by the position estimation unit 13 corresponding to the landmark, in the same landmark associated by the landmark association unit 12. Based on the observed image coordinates extracted by the second landmark extracting means 11, the camera parameters (estimated camera parameters) of the camera C2 are estimated. Here, the parameter estimation unit 14 includes a coordinate association unit 141 and a parameter calculation unit 142.

  The coordinate association unit 141 is an observation extracted by the second landmark extraction unit 11 based on the actual coordinates (three-dimensional coordinates) estimated by the position estimation unit 13 and the correspondence result by the landmark association unit 12. The real coordinates are associated with the image coordinates.

Here, the coordinate association unit 141 is a set of pairs (m, p _m ) of the first identifier output from the position estimation unit 13 and the actual coordinates of the landmark, and the first output from the landmark association unit 12. From a set of a pair (m, n _m ) of one identifier and a second identifier, and a set of a pair (n, o _n ) of a second identifier and observation image coordinates output from the second landmark extraction means 11 Then, a set of pairs (p _m , o _nm ) of real coordinates and observed image coordinates regarding the same landmark is obtained. Incidentally, o _nm shows the observed image coordinates associated with the second identifier n _m comprising the first identifier m a pair. Then, the coordinate association unit 141 performs parameter calculation on a set of pairs (p _m , o _nm ) of the actual coordinates and the observed image coordinates as many as the number of pairs (m, n _m ) of the first identifier and the second identifier. It outputs to the means 142.

  The parameter calculation unit 142 calculates estimated camera parameters, which are camera parameters of the camera C2, based on the actual coordinates (three-dimensional coordinates) of the plurality of observed image coordinates associated with the coordinate association unit 141. The camera parameters calculated by the parameter calculation unit 142 are sequentially output to the outside as the camera parameters of the camera C2.

Here, the parameter calculation unit 142 calculates the camera parameters of the camera C2 when a predetermined number of pairs (p _m , o _nm ) of landmark real coordinates and observed image coordinates are input. . For this calculation, when a three-dimensional coordinate (real coordinate) of n points on a two-dimensional plane (image plane) is specified, a general method for solving a perspective n-point problem for estimating camera parameters can be used. For example, when the landmarks exist on the same plane, the parameter calculation unit 142 receives four perspective points when four pairs (p _m , o _nm ) of the actual coordinates of the landmarks and the observed image coordinates are input. The camera parameters of the camera C2 are calculated by solving the problem.

  By configuring the camera parameter estimation device 1 in this way, the camera parameter estimation device 1 is configured so that the image captured by the camera C1 whose camera parameter is known (first image) and the image captured by the camera C2 whose camera parameter is unknown. From the (second image), the camera parameters of the camera C2 can be estimated.

  Further, the camera parameter estimation device 1 uses any object or pattern that can be commonly observed by the camera C1 and the camera C2 as a landmark without fixing the position of the landmark, and determines the camera parameter of the camera C2. Can be estimated. In addition, the camera parameter estimation device 1 is not limited to an object or pattern arbitrarily set in the scene, but a static or dynamic pattern projected on the scene, an object or pattern that originally exists in the scene, or Objects, people, etc. that move in the field can be used as landmarks.

Accordingly, the camera parameter estimation device 1 uses, for example, the camera C2 as a relay camera to estimate the camera parameters of a camera used for sports relay, and shoots a wide range of the stadium with the camera C1 whose camera parameters are known. The camera parameters of the camera C2 can be estimated without disturbing the competition by using the athlete, the ball, the line or mark drawn on the ground, the equipment placed in the field as a landmark. it can.
The camera parameter estimation apparatus 1 can be operated by a camera parameter estimation program that causes a general computer to function as each of the means described above.

[Operation of camera parameter estimation device]
Next, the operation of the camera parameter estimation apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flowchart showing the operation of the camera parameter estimation apparatus according to the embodiment of the present invention.

  First, the camera parameter estimation device 1 uses the first landmark extraction means 10 to extract landmark coordinates (reference image coordinates) from a captured image (first image) captured by the camera C1 whose camera parameters are known (first image). Step S1). Further, the camera parameter estimation device 1 extracts the coordinates of the landmarks (observed image coordinates) from the captured image (second image) captured by the camera C2 whose camera parameters are unknown by the second landmark extraction unit 11 ( Step S2).

  Then, the camera parameter estimation device 1 uses the landmark association unit 12 to associate landmarks based on the reference image coordinates extracted in step S1 and the observed image coordinates extracted in step S2.

  That is, the camera parameter estimation device 1 uses the first feature extraction unit 121 of the landmark association unit 12 to convert the image feature in the first image corresponding to the reference image coordinates (reference image coordinate group) to the first feature (first feature). Group) (step S3). Also, the camera parameter estimation device 1 uses the second feature extraction unit 122 of the landmark association unit 12 to convert the image feature in the second image corresponding to the observed image coordinates (observed image coordinate group) to the second feature (second feature). Group) (step S4).

  Then, the camera parameter estimation apparatus 1 causes the image feature to be most similar between the first feature group extracted in step S3 and the second feature group extracted in step S4 by the matching unit 123 of the landmark association unit 12. Is associated with the landmark (step S5).

  Further, the camera parameter estimation device 1 uses the position estimation unit 13 to backproject the reference image coordinates extracted in step S1 with known camera parameters that are camera parameters of the camera C1, thereby to convert the actual coordinates of the landmarks (three-dimensional coordinates). ) Is calculated (step S6).

  Then, the camera parameter estimation device 1 uses the parameter estimation unit 14 to observe the observation image coordinates extracted in step S2, the corresponding result of the landmarks associated in step S5, and the landmarks (reference image coordinates calculated in step S6). ) To estimate camera parameters of the camera C2.

That is, the camera parameter estimation device 1 is extracted in step S2 based on the actual coordinates (three-dimensional coordinates) calculated in step S6 and the correspondence result in step S5 by the coordinate association unit 141 of the parameter estimation unit 14. The actual coordinates are associated with the observed image coordinates (step S7).
Then, the camera parameter estimation apparatus 1 uses the parameter calculation unit 142 of the parameter estimation unit 14 to calculate the camera parameters of the camera C2 from the real coordinates of the plurality of observed image coordinates associated in step S7 by the method of solving the perspective n-point problem. (Estimated camera parameters) are calculated (step S8).

  Through the above operation, the camera parameter estimation device 1 uses the camera (first image) captured by the camera C1 with known camera parameters and the image (second image) captured by the camera C2 with unknown camera parameters. C2 camera parameters can be estimated.

It is explanatory drawing for demonstrating the outline | summary of this invention. It is a block diagram which shows the structure of the camera parameter estimation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the method of the back projection in the position estimation means of the camera parameter estimation apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the camera parameter estimation apparatus which concerns on embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera parameter estimation apparatus 10 1st landmark extraction means 11 2nd landmark extraction means 12 Landmark matching means 121 1st feature extraction means 122 2nd feature extraction means 123 Verification means 13 Position estimation means 14 Parameter estimation means 141 Coordinates Corresponding means 142 Parameter calculating means C1 Camera (first camera)
C2 camera (second camera)

Claims (4)

Camera parameter estimation for estimating camera parameters of the second camera based on a first image captured by a first camera with known camera parameters and a second image captured by a second camera with unknown camera parameters A device,
First landmark extraction means for extracting the coordinates of the landmark in the image from the first image as reference image coordinates based on predetermined image features of the landmark;
Second landmark extraction means for extracting the coordinates of the landmark in the image from the second image as observation image coordinates based on the image characteristics of the landmark;
Based on the degree of similarity between the image feature corresponding to the reference image coordinates on the first image and the image feature corresponding to the observed image coordinates on the second image, the first image and the second image Landmark association means for associating the landmarks with each other,
Based on a known camera parameter that is a camera parameter of the first camera, a position estimation unit that estimates the three-dimensional coordinates of the landmark by backprojecting the reference image coordinates under a predetermined constraint condition;
Based on the three-dimensional coordinates corresponding to the landmarks and the observed image coordinates in the same landmarks correlated by the landmark correlation unit, estimated camera parameters that are camera parameters of the second camera are Parameter estimating means for estimating;
A camera parameter estimation device comprising: The landmark association means includes
A first feature extracting means for extracting, from the first image, an image feature in the vicinity region of the reference image coordinates as a first feature;
Second feature extraction means for extracting, from the second image, an image feature in the vicinity region of the observed image coordinates as a second feature;
Matching means for associating the landmarks based on the degree of similarity between the first feature and the second feature;
The camera parameter estimation apparatus according to claim 1, further comprising: The parameter estimation means includes
Based on the three-dimensional coordinates estimated by the position estimating means and the correspondence result by the landmark associating means, coordinate associating means for associating the three-dimensional coordinates with the observed image coordinates;
Parameter calculating means for calculating the estimated camera parameter based on the three-dimensional coordinates of the plurality of observed image coordinates associated by the coordinate associating means;
The camera parameter estimation apparatus according to claim 1, further comprising: In order to estimate the camera parameters of the second camera based on the first image captured by the first camera whose camera parameters are known and the second image captured by the second camera whose camera parameters are unknown, Computer
First landmark extraction means for extracting, from the first image, a coordinate of the landmark in the image as a reference image coordinate based on a predetermined image feature of the landmark;
Second landmark extraction means for extracting the coordinates of the landmark in the image from the second image as observation image coordinates based on the image characteristics of the landmark;
Based on the degree of similarity between the image feature corresponding to the reference image coordinates on the first image and the image feature corresponding to the observed image coordinates on the second image, the first image and the second image Landmark association means for associating the landmarks with each other,
Based on a known camera parameter that is a camera parameter of the first camera, a position estimation unit that estimates the three-dimensional coordinates of the landmark by backprojecting the reference image coordinates under a predetermined constraint condition;
Coordinate associating means for associating the three-dimensional coordinates with the observed image coordinates based on the three-dimensional coordinates estimated by the position estimating means and the correspondence result in the landmark associating means,
Parameter calculating means for calculating an estimated camera parameter that is a camera parameter of the second camera based on the three-dimensional coordinates of the plurality of observed image coordinates associated by the coordinate associating means;
A camera parameter estimation program that functions as a computer program.

Images