JP2010261785A - Camera calibrating device and camera calibration program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a camera calibration device for estimating a camera parameter, even when a visually characteristic point and a known control point in a world coordinate are given separately. <P>SOLUTION: This camera calibration device 1 includes: a landmark designation means 12 for designating an image coordinate of a control point and an image domain from an input image; a characteristic extraction means 14 for extracting characteristic point arrangement information and an image characteristic from the image domain; a world coordinate input means 16 for inputting a control point world coordinate; a landmark storage means 18 for storing the characteristic point arrangement information, the image characteristic and the control point world coordinate; a characteristic extraction means 22 for extracting a characteristic point arrangement information and an image characteristic from a new input image; a collation means 24 for correlating each image characteristic having high similarity; a control point estimation means 26 for estimating a control point image coordinate of the new input image from the characteristic point arrangement information, the arrangement information and a collation result; and a camera parameter estimation means 28 for estimating the camera parameter from the control point world coordinate and the estimated control point image coordinate. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a camera calibration apparatus and a camera calibration program for estimating a camera parameter including one or more of the position, posture, or focal length of a camera based on an image photographed by a camera to be calibrated.

  Conventionally, in camera calibration, a predetermined or measured pattern is photographed by a camera to be calibrated, and camera parameters are obtained from the shape of the pattern image analytically or by repeated convergence calculation. ing.

  As a typical method of recursive convergence calculation, while evaluating the degree of coincidence between the actually observed pattern image and the observed pattern image estimated from the camera parameters and pattern model, There is a method of sequentially correcting camera parameters so that the degree is maximized.

  As a pattern, a method using correspondence between a plurality of observation points and points on the model (see Non-Patent Document 1), a plurality of observed lines (straight lines, line segments, curves, etc.), and a model There is a method using a correspondence relationship with a line.

  In the method using point correspondence, points that intersect each other, vertices of broken lines, or representative points (such as the center of gravity) of a specific pattern such as a circle or rectangle are extracted as feature points, and the correspondence with the model is determined. The method we seek is the mainstream.

  On the other hand, in the method using the correspondence between lines, for example, parameters of straight lines and curves are calculated by Hough transform or generalized Hough transform, and correspondence between the parameters and the model is performed (Patent Literature). 1).

  Also, a method of obtaining the relative position, relative orientation, and focal length information of the camera between the image pairs by extracting a plurality of visually characteristic points in the image pairs and associating the feature points with each other. (See Non-Patent Document 2).

JP 2004-234333 A

R. Y. Tsai, An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision.Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp. 364-374, 1986. C. Poelman and T. Kanade, A paraperspective factorization method for shape and motion recovery.IEEE Trans. On Pattern Analysis and Machine Intelligence, 19 (3): 206-218, March 1997.

  In the conventional camera calibration method, feature points, straight lines, and curves in an image are extracted and their coordinates and parameters are obtained as the first step, and then the coordinates and parameters of points and lines obtained by projecting the model are converted. The camera parameters are estimated by comparing the distances with the coordinates and parameters obtained from the observed image.

  However, in these methods, it is necessary to model in advance where a visual pattern such as a feature point, a straight line, or a curve is present in a real space having world coordinates. For this reason, there is a problem that the camera calibration cannot be performed when a known point of the world coordinates is not visually characteristic or when the world coordinates of the visually characteristic point is not known.

  The present invention has been made to solve such a problem, and it is possible to estimate camera parameters even when visually distinctive points and known points of world coordinates are given separately. An object of the present invention is to provide a camera calibration apparatus and a camera calibration program.

  The present invention was devised to achieve the above object, and first, the camera calibration device according to claim 1 is characterized by extracting the image coordinates of the known control points of the world coordinates designated in the input image. A camera calibration device that calculates image parameters of a control point by estimating image coordinates of the control point from a feature point extracted from a new input image using a point, the landmark designating means, and a first feature extraction Means, world coordinate input means, landmark storage means, second feature extraction means, collation means, control point estimation means, and camera parameter estimation means.

  In such a configuration, the camera calibration apparatus designates the image coordinates of the control point, which is a point whose world coordinates are known, and the image area to be associated with the control point in the input image by the landmark designation unit. Here, world coordinates are coordinates for representing a position in real space, and image coordinates are coordinates for representing a position in an input image.

  Then, the camera calibration device extracts one or more feature points that are visually characteristic points from the image region designated by the landmark designation unit by the first feature extraction unit, and arranges the feature points First arrangement information on the image and a first image feature obtained by quantifying the visual feature of the feature point are extracted. That is, there may be a plurality of feature points extracted from one image region. The arrangement information and the image feature are generally vector quantities. Also, the camera calibration device inputs the world coordinates of the control point designated by the landmark designation means by the world coordinate input means.

  Then, the camera calibration apparatus uses the landmark storage unit to control the first arrangement information and the first image feature extracted by the feature extraction unit from the image area associated with the control point, and the control input from the world coordinate input unit. A plurality of landmarks are stored with a set of points and the world coordinates as one landmark. As a result, a plurality of pieces of first arrangement information and first image information and the world coordinates of the control points are stored in association with each other.

  Furthermore, the camera calibration device extracts one or more feature points which are visually characteristic points from the new input image by the second feature extraction unit, and second arrangement information regarding the arrangement of the feature points. And a second image feature obtained by quantifying the visual feature of the feature point.

  Then, the camera calibration device selects the first image feature of the landmark read from the landmark storage unit by the collating unit from among the second image features extracted by the second feature extracting unit. A second image feature having a high degree of similarity is searched, and the matching result is output as matching result information for each landmark. Here, as the similarity, for example, cosine similarity can be used.

  Thereafter, the camera calibration device uses the control point estimating means to read the first arrangement information of the landmarks read from the landmark storage means, the second arrangement information inputted from the second feature extraction means, and the collating means. The image coordinates of the control points corresponding to the landmarks in the new input image are estimated on the basis of the collation result information input from, and output as control point estimated image coordinates.

  Then, the camera calibration apparatus estimates the camera parameters by the camera parameter estimation means based on the world coordinates of the control points read from the landmark storage means and the control point estimation image coordinates input from the control point estimation means. For estimating the camera parameters, a known perspective N-point problem solution method or a non-linear least square method can be used.

  The camera calibration device according to claim 2 is the camera calibration device according to claim 1, wherein the first arrangement information includes a position vector from the control point to the feature point in the input image and a rotation in the vicinity of the feature point. Information and scale information, and the second arrangement information includes image coordinates of feature points in the new input image, rotation information and scale information in the vicinity of the feature points, and the control point estimation means stores these information. It was set as the structure which estimates the control point estimation image coordinate.

  In such a configuration, the camera calibration device uses the control point estimation unit to rotate information and scale information included in the first arrangement information with respect to the position vector included in each first arrangement information of the landmark, Using the rotation information and scale information included in the second arrangement information extracted together with the first image feature extracted together with the first arrangement information and the corresponding second image feature via the matching result, By converting the position vector in the new input image and subtracting the converted position vector from the image coordinates of the feature points included in the second arrangement information, a temporary control point image coordinate estimated value is obtained, and the landmark The temporary control point image coordinate estimated value obtained from each of the first arrangement information is averaged by a predetermined method to obtain the land in the new input image. And the control point estimation image coordinates of the control points corresponding to the over-click.

The camera calibration device according to claim 3 is the camera calibration device according to claim 2, wherein the control point estimation means divides the whole of the new input image into a plurality of partial areas, and the provisional control point image is obtained. Depending on which partial region the coordinate estimation value belongs to, the values related to the partial region are accumulated one by one, and an average is obtained only for the temporary control point image coordinate estimation value related to the partial region having the largest total value. It is characterized by.
With this configuration, the camera calibration apparatus can estimate the image coordinates of the control point corresponding to the landmark from the temporary control point image coordinate estimated value related to the partial region having the largest total value by the control point estimating unit. .

The camera calibration apparatus according to claim 4 is the camera calibration apparatus according to claim 2, wherein the collating means outputs the similarity together with the collation result as collation result information for each landmark, and estimates the control point. The means divides the entire new input image into a plurality of partial areas, and relates to the calculation of the temporary control point image coordinate estimated value according to which partial area the temporary control point image coordinate estimated value belongs to A value corresponding to the degree of similarity corresponding to the matching result is accumulated in a value related to the partial area, and an average is obtained only for the temporary control point image coordinate estimated value related to the partial area having the largest total value. To do.
With this configuration, the camera calibration apparatus determines the partial area for which the average is calculated in consideration of the similarity by the control point estimation means, so that the image coordinates of the control point in consideration of the similarity can be estimated. .

  Further, the camera calibration program according to claim 5 uses the image coordinates of the known control points of the world coordinates specified in the input image and the extracted feature points to perform the control from the feature points extracted in the new input image. In order to estimate the image coordinates of the points and calculate the camera parameters, the computer is used for landmark designation means, first feature extraction means, world coordinate input means, second feature extraction means, collation means, control point estimation. And function as camera parameter estimation means.

  In this configuration, the camera calibration program designates the image coordinates and image area of the control point in the input image by the landmark designation means. Further, the camera calibration program extracts feature points from the image area by the first feature extraction means, extracts first arrangement information and first image features, and uses the control points related to the image area. It is stored in the landmark storage means as a constituent element of the associated landmark. Then, the camera calibration program inputs the world coordinates of the control point by the world coordinate input means, and stores it in the landmark storage means as a constituent element of the landmark associated with the control point.

  Further, the camera calibration program extracts feature points from the new input image by the second feature extraction unit, and extracts second arrangement information and second image features. Then, the camera calibration program searches the first image feature of the landmark for the first image feature of the landmark with the second image feature having the highest similarity and uses the matching result for each landmark. Is output as verification result information. Further, the camera calibration program calculates the image coordinates of the control point in the new input image based on the first arrangement information of the landmark, the second arrangement information, and the matching result information by the control point estimation means. Estimate and output as control point estimation image coordinates. Then, the camera calibration program estimates the camera parameters based on the world coordinates of the control points and the control point estimated image coordinates by the camera parameter estimation means.

  According to the first and fifth aspects of the invention, the camera can be calibrated even when visually distinctive points and world coordinate known points are given separately. That is, in the landmark used for calibration, it is not always necessary to match a visually characteristic point with a known control point of world coordinates. This makes it possible to estimate camera parameters and calibrate the camera in a more general scene as compared to the conventional method that requires the use of landmarks with known world coordinates and visually characteristic features.

  According to the second aspect of the present invention, when the control point estimation means estimates the image coordinates of the control point, the position vector of the input image is obtained using the rotation information and scale information of the input image and the new input image. Is converted into a new input image position vector, which is used to determine a provisional control point image coordinate estimate and average the provisional control point image coordinate estimate over each landmark. Since the image coordinates are obtained, the control points can be estimated more accurately in the new input image, and therefore the camera parameters can be estimated and the camera calibration can be performed more accurately.

  According to the third aspect of the present invention, the control point estimating means divides the entire new input image into a plurality of partial areas, and only the partial area having the largest temporary control point image coordinate estimation value to which it belongs most. Since the average value of the estimated control point image coordinates is obtained, the control point can be estimated more accurately. Therefore, the camera parameter can be estimated and the camera calibration can be performed more accurately.

  According to the fourth aspect of the present invention, the control point estimation means divides the entire new input image into a plurality of partial areas, and determines the partial area having the largest estimated temporary control point image coordinate value belonging to Since the average value of the control point image coordinate estimated value is obtained only for the partial area, it is possible to estimate the accurate control point according to the similarity of the image features, and thus more accurately the camera. Parameter estimation and camera calibration are possible.

It is a block diagram which shows the structure of the camera calibration apparatus which concerns on embodiment of this invention. It is a figure which shows an example which designated the pattern image area | region and the control point image coordinate in the input image by the landmark designation | designated means. It is a figure which shows an example of the relationship of the pattern image area | region regarding a nth landmark, a feature point, an image feature, feature point arrangement | positioning information, a control point, and a control point image coordinate. It is a figure which shows an example of the memory | storage system of the feature point arrangement | positioning information by the landmark memory | storage means, an image feature, and a control point world coordinate. It is a figure which shows the calculation method of the temporary control point image coordinate estimated value in a control point estimation means. It is a flowchart which shows operation | movement of the landmark designation | designated function part of the camera calibration apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the camera parameter estimation function part of the camera calibration apparatus which concerns on embodiment of this invention.

Embodiments of the present invention will be described below with reference to the drawings.
In the present embodiment, a coordinate system for representing a position in real space is referred to as a world coordinate system, and coordinates in the world coordinate system are referred to as world coordinates. Here, the world coordinate system is represented by a Cartesian coordinate system, and the respective axes are the X axis, the Y axis, and the Z axis.
In addition, a coordinate system for representing a position in the input image is referred to as an image coordinate system, and coordinates in the image coordinate system are referred to as image coordinates. Here, it is assumed that the x-axis is taken rightward in the horizontal direction of the pixel arrangement and the y-axis is taken downward in the vertical direction of the pixel arrangement.

  A reference for measurement related to an input image used for camera calibration is referred to as a landmark. There may be one landmark or a plurality of landmarks. In the present invention, one landmark includes a control point that is a known point in the world coordinates and a feature point that is a visually characteristic point, and there are one or more feature points. To do.

[Configuration of camera calibration device]
Below, with reference to FIG. 1, the structure of the camera calibration apparatus 1 which concerns on embodiment of this invention is demonstrated.

The camera calibration apparatus 1 according to the present embodiment estimates camera parameters by estimating a control point of a new input image landmark from a landmark of the input image.
Specifically, the camera calibration device 1 extracts feature point arrangement information and image features relating to feature points from an image region specified in the input image that has been input, together with the world coordinates of control points associated with the image region. Stored as a landmark, in the newly input image, the image coordinates of the control point are estimated from the arrangement information and image feature regarding the extracted feature point and the information regarding the read landmark, and the control point The camera parameters are estimated and output from the image coordinates and the world coordinates of the control points.

  The camera calibration apparatus 1 includes a landmark designation function unit 10 and a camera parameter estimation function unit 20 depending on the function. The landmark designation function unit 10 includes landmark designation means 12, feature extraction means 14, world coordinate input means 16, and landmark storage means 18. The camera parameter estimation function unit 20 includes a feature extraction unit 22, a collation unit 24, a control point estimation unit 26, and a camera parameter estimation unit 28.

First, the configuration of the landmark designation function unit 10 of the camera calibration apparatus 1 will be described.
[[Configuration of Landmark Specification Function]]
The landmark designation means 12 designates an image area or the like corresponding to the landmark.
Specifically, the landmark designation means 12 has an image area for extracting feature points (hereinafter referred to as a pattern image area) and a known world coordinate in the input image I (x, y) inputted. An image coordinate of a control point that is a point (hereinafter referred to as a control point image coordinate) is designated, and the designated result is output to the feature extraction means 14.

In the landmark designation means 12, a plurality of pattern image areas and control point image coordinates can be designated, and one designation corresponds to one landmark. When the landmark designation means 12 designates N (N is a natural number), the pattern image area corresponding to the nth (1 ≦ n ≦ N) landmark L ⁽ⁿ⁾ is controlled by D ⁽ⁿ⁾ . The point is ^denoted by C ⁽ⁿ⁾ and the image coordinate of the control point C ⁽ⁿ⁾ is ^denoted as c ⁽ⁿ⁾ = [c _x ⁽ⁿ⁾ , _cy ⁽ⁿ⁾ ] ^T. Here, the superscript T represents a transposed matrix. Note that I (x, y) means that a value (luminance value, etc.) I (x, y) is defined at each point (x, y) of the image coordinates.

  For example, an input image I (x, y) is presented on a display device such as a liquid crystal display or a CRT, a cursor is superimposed on the input image I, and a rectangular region is designated as a pattern image region by a mouse drag operation or the like. . In addition, the control point image coordinates are designated by clicking the mouse button.

FIG. 2 is a diagram showing an example in which a pattern image region and control point image coordinates are designated in the input image by the landmark designation means 12. Shows a pattern image region D ⁽ⁿ⁾ rectangular region is specified as shown by a broken line in FIG. 2, the image coordinates c of the control point position indicated by × marks specified ^{C (n) (n) =} [c x ( ⁿ⁾ , _cy ⁽ⁿ⁾ ] ^T.

The feature extraction means (first feature extraction means) 14 extracts information on feature points constituting the landmark from the input image.
Specifically, the feature extraction unit 14 extracts a feature point that is a visually characteristic point from the pattern image area of the input image I (x, y) designated by the landmark designation unit 12, The feature point arrangement information regarding the arrangement of the feature points and the image features obtained by quantifying the visual features of the feature points are extracted and output to the landmark storage unit 18.
The feature extraction unit 14 extracts M ⁽ⁿ⁾ feature points (M ⁽ⁿ⁾ is a natural number), which is extracted from the pattern image region D ⁽ⁿ⁾ corresponding to the ^nth landmark L ^(n). The m th (1 ≦ m ≦ M ⁽ⁿ⁾ ) feature point is defined as E _m ⁽ⁿ⁾ .

The feature extraction unit 14 uses at least position information in the input image as the feature point arrangement information g _m ⁽ⁿ⁾ of the feature point E _m ⁽ⁿ⁾ , that is, the feature point E from the control point C ⁽ⁿ⁾ in the image coordinate system. _m reaches the ⁽ⁿ⁾ vector to extract information including a ^{_{[p m (n), q}} m (n)] T. For the calculation of this vector, the control point image coordinates c ⁽ⁿ⁾ designated by the landmark designation means 12 are used.

In the present embodiment, the feature point arrangement _information, ^g m _{⁽ⁿ⁾ =} a ^{_{[p m (n), q}} m (n), r m (n), s m (n)] T. _Here, ^{r m (n)} denotes the rotation information, the input image I (x, y) is visually distinctive orientation with the _E ^{m (n)} near the feature point. For example, as r _m ⁽ⁿ⁾ , a representative edge orientation (for example, high contrast) included in a partial image near the feature point E _m ⁽ⁿ⁾ can be used. Further, s _m ⁽ⁿ⁾ represents scale information, and is visually characteristic size information that the input image I (x, y) has in the vicinity of the feature point E _m ⁽ⁿ⁾ . For example, as s _m ⁽ⁿ⁾ , a representative texture size (that is, a representative spatial frequency component of the texture ⁾ included in the partial image in the vicinity of the feature point E _m ⁽ⁿ⁾ can be used.

Feature extracting means 14, these feature point arrangement information _g ^{m _(n)} = a ^{_{[p m (n), q}} m (n), r m (n), s m (n)] T, for example, SIFT ( Scale Invariant Feature Transform (DG Lowe, “Object recognition from local scale-invariant features” Proc. Of IEEE International Conference on Computer Vision (ICCV), pp. 1150-1157, 1999) Obtainable. The key point, orientation, and scale correspond to the feature point, rotation information, and scale information of this embodiment, respectively.

  Specifically, SIFT processing is as follows. In other words, feature points (key points) and scale information thereof are determined by detecting extreme values from a Difference-of-Gaussian (DoG) image. Further, a histogram in the gradient direction is obtained from the peripheral area of the feature point, and a peak is assigned as the orientation. Then, the surrounding area determined by the scale of the feature points is rotated according to the orientation, and the surrounding area is weighted by a Gaussian window according to the distance from the feature points, and then divided into 16 4 × 4 small areas, and each of the small areas has 8 directions. By obtaining the gradient direction histogram, a 128-dimensional SIFT feature description that is invariant to rotation and scale is obtained.

Note that the feature extraction unit 14 collects the feature point arrangement information g _m ⁽ⁿ⁾ of all M ⁽ⁿ⁾ feature points extracted from the pattern image region D ⁽ⁿ⁾ into a matrix G ^(n), and G ^{( n)} may be output (see equation (1)).

Furthermore, the feature extraction unit 14 uses the spatial distribution of the lightness and color near E _m ⁽ⁿ⁾ in the input image I (x, y) as the image feature f _m ⁽ⁿ⁾ of the feature point E _m ^(n). Information (quantity and vector quantity) characterizing the pattern is extracted. It is preferable that the image feature f _m ⁽ⁿ⁾ has a value that hardly changes with rotation and enlargement / reduction. Further, the image feature f _m ⁽ⁿ⁾ is less likely to change even with a uniform change in luminance value (in a region larger than the size of the region near the feature point E _m ^{(n) )} of the image. Preferably there is.

The feature extraction unit 14 can use, for example, the SIFT description (SIFT descriptor) in the SIFT as the image feature f _m ⁽ⁿ⁾ .

Note that the feature extraction unit 14 collects the image features f _m ⁽ⁿ⁾ of all M ⁽ⁿ⁾ feature points extracted from the pattern image region D ⁽ⁿ⁾ into a matrix F ^(n), and F ⁽ⁿ⁾ May be output (see equation (2)).

FIG. 3 shows a pattern image region D ⁽ⁿ⁾ , a feature point E _m ⁽ⁿ⁾ , an image feature f _m ⁽ⁿ⁾ , feature point arrangement information g _m ⁽ⁿ⁾ , control points regarding the ^nth landmark L ^(n). C ^(n), and the control point image coordinates is a diagram showing an example of the relationship between _{^{[c x (n), c}} y (n)] T. In FIG. 3, three feature points are shown. In the example of FIG. 3, the control point C ⁽ⁿ⁾ exists outside the pattern image area D ⁽ⁿ⁾ , but the control point C ⁽ⁿ⁾ exists inside or on the boundary of the pattern image area D ^(n). May be.

The world coordinate input means 16 inputs the world coordinates of the control point.
Specifically, the world coordinate input means 16 is an interface that inputs world coordinates for each control point for which image coordinates are designated by the landmark designation means 12 and outputs the world coordinates to the landmark storage means 18.
Entered the world coordinate input unit 16, the world coordinates of the control points ^{C (n)} corresponding to the landmark ^L of the _{^{n (n) w (n)}} = [w x (n), w y (n), w _z ⁽ⁿ⁾ ] ^T.

The world coordinate input means 16 may be a method of inputting the control point world coordinates w ⁽ⁿ⁾ numerically by operating a man-machine interface such as a keyboard, knob, or mouse. Further, for example, it may be an interface that receives the world coordinates w ⁽ⁿ⁾ of the control point measured by another surveying device connected to the outside of the present device.

The landmark storage means 18 stores information relating to landmarks.
Specifically, the landmark storage unit 18 sets a set of feature point arrangement information and image features input from the feature extraction unit 14 and control point world coordinates input from the world coordinate input unit 16 for each landmark ( Storage means for storing a plurality of sets (for each control point). The landmark storage means 18 outputs the stored data when there is a read request from the collation means 24, the control point estimation means 26, and the camera parameter estimation means 28 of the camera parameter estimation function unit 20 described later. To do. The landmark storage means 18 may be a general storage medium such as a semiconductor memory or a hard disk, and the type thereof is not limited.

  For example, as shown in FIG. 4, the landmark storage unit 18 can store a set of feature point arrangement information, image features, and control point world coordinates in a tabular format for each landmark number.

Next, the configuration of the camera parameter estimation function unit 20 of the camera calibration apparatus 1 will be described.
[[Configuration of camera parameter estimation function]]
The feature extraction means (second feature extraction means) 22 extracts feature point information from a new input image.
Specifically, the feature extraction unit 22 extracts feature points that are visually characteristic points from a newly input image (hereinafter referred to as a new input image) I (x, y). Then, the arrangement information relating to the arrangement of the feature points and the image features obtained by quantifying the visual features of the feature points are extracted, the image features are sent to the matching means 24, and the placement information is sent to the control point estimation means 26. Each is output.
The feature extraction unit 22 performs substantially the same processing as the feature extraction unit 14 of the landmark designation function unit 10 except for the area for extracting feature points and the like.

The feature extraction unit 22 sets the number of feature points extracted from the new input image I (x, y) to K (K is a natural number), and the extracted k-th (1 ≦ k ≦ K) -th feature point. Let _Bk . The feature extraction means 22 extracts the image coordinate [x _k , y _k ] ^T , the rotation information ψ _k and the scale information σ _k which are the position information of the feature point as the arrangement information γ _k of the feature point B _k (( 3) Refer to equation). The rotation information [psi _k and scale information sigma _k is the same as the rotation information feature extraction means 14 Landmarks designation unit 10 extracts r _{m ⁽ⁿ⁾} and scale information s _{m ^(n).}

Note that the feature extraction means 22 may collect the arrangement information γ _{k of} all K feature points extracted from the new input image I (x, y) into a matrix Γ and output Γ ((4 ) See formula).

Further, the feature extraction means 22 characterizes the spatial distribution pattern such as the density and color in the vicinity of B _k in the new input image I (x, y) as the image feature φ _k of the feature point B _k ( (Quantity and vector quantity). This image feature φ _k is the same as the image feature f _m ⁽ⁿ⁾ extracted by the feature extraction unit 14 of the landmark designation function unit 10.

Note that the feature extraction unit 22 may collect the image features φ _{k of} all K feature points extracted from the new input image I (x, y) into a matrix Φ and output Φ ((5 ) See formula).

The matching unit 24 associates the feature points of the input image with the feature points of the new input image.
Specifically, the collation unit 24 performs feature extraction unit for each image feature f _m ⁽ⁿ⁾ constituting the image feature F ⁽ⁿ⁾ of each landmark L ⁽ⁿ⁾ read from the landmark storage unit 18. from the image feature phi _k constituting the image feature Φ input from 22, and searches the most similar image features phi _k, the number k as the collation result h _{m ^(n),} to the control point estimating means 26 (1 ≦ n ≦ N, 1 ≦ m ≦ M ⁽ⁿ⁾ , 1 ≦ k ≦ K).

Collation means 24, an image feature _f ^{m (n)} constituting an image feature ^{F (n),} the similarity between the image feature phi _k constituting the image feature ^{_{Φ, λ (f m (n}} ), φ _k ), and those having high similarity are associated with each other. For example, the cosine similarity can be used as the similarity λ (f, φ) (see equation (6)).

Then, the matching unit 24 searches the image feature φ _k that maximizes the similarity λ (f _m ⁽ⁿ⁾ , φ _k ) for each image feature f _m ⁽ⁿ⁾ , and uses the number k as a matching result h. _m ⁽ⁿ⁾ is output (see equation (7)).

Note that the collating unit 24 collects collation results h _m ⁽ⁿ⁾ for all M ⁽ⁿ⁾ image features of the landmark L ⁽ⁿ⁾ as collation result information H ^(n), and outputs a matrix H ⁽ⁿ⁾ . (Refer to equation (8)).

Further, the matching unit 24 may output each similarity λ _m ⁽ⁿ⁾ in addition to each matching result h _m ^{(n) (} see equation (9)). When outputting a matrix (8) in place of the matrix ^{H (n)} shown in the expression, and outputs a similarity lambda _m ⁽ⁿ⁾ obtained by adding (10) shown in the expression matrix ^{H (n).}

The control point estimation means 26 estimates the coordinates of the control point in the new input image.
Specifically, the control point estimation unit 26 is input from the feature point arrangement information (G ⁽¹⁾ ,..., G ^(N) ) of each landmark read from the landmark storage unit 18 and the feature extraction unit 22. Each of the positions in the new input image I (x, y) based on the arrangement information Γ of the feature points and the matching result information (H ⁽¹⁾ ,..., H ^(N) ) input from the matching means 24. It is estimated whether there is a control point (C ⁽¹⁾ ,..., C ^(N) ) of the landmark (L ⁽¹⁾ ,..., L ^(N) ), and the estimated control point estimated image coordinates (κ ⁽¹⁾ ,..., Κ ^(N) ) are output to the camera parameter estimating means 28.

The control point estimation means 26 estimates the control point estimated image coordinates κ ⁽ⁿ⁾ of the ^nth landmark L ⁽ⁿ⁾ by the following estimation procedure (1 ≦ n ≦ N). This control point estimation image coordinate κ ⁽ⁿ⁾ is estimated by voting (Masari Takagi, Hironobu Fujiyoshi, “Traffic road sign recognition using SIFT features”, 13th Image Sensing Symposium SSII07, LD2- 06, 2007).

First, the control point estimating means 26, the feature point of the m included in the landmark of the n ^{L (n)} _E ^{m (n),} determined the control point image coordinate estimated values of the provisional tau _m to ⁽ⁿ⁾ . That is, the control point estimation unit 26 uses the feature point arrangement information G _m ⁽ⁿ⁾ = [p _m ^{(n )} regarding the feature point E _m ^{(n) based} on the feature point arrangement information G ⁽ⁿ⁾ read from the landmark storage unit 18. _{^{), q m (n),}} r m (n), taken out ^{s m (n)] T.} Further, from the verification result information ^H received from the collating unit 24 ^(n), the verification result taken out _h ^{m (n)} relating to the feature point _E ^{m (n).} Further, the feature point arrangement information with the number h _m ⁽ⁿ⁾ is extracted from the feature point arrangement information Γ input from the feature extraction means 22 (see equation (11)).

Here, the control point estimating means _26, ^g m _{⁽ⁿ⁾ =} a ^{_{[p m (n), q}} m (n), r m (n), s m (n)] T and (11), the control estimating the image coordinates of the point _C ^{m (n),} the control point image coordinate estimated values of the provisional tau _m ⁽ⁿ⁾ to ((12) see formula) is obtained by (13).

FIG. 5 is a diagram for explaining the meaning of the equation (13). In FIG. 5, subscripts are omitted for convenience of explanation. The left diagram in FIG. 5 represents a part of an input image, the right diagram in FIG. 5 represents a part of a new input image, and the left diagram and the right diagram have the same number of pixels. In the left figure, it is assumed that the feature point has rotation information r and scale information s, and in the right figure, the corresponding feature point having the maximum similarity has rotation information ψ and scale information σ. The expression (13) uses these pieces of information to estimate a temporary control point image coordinate to the position vector of the right figure obtained by rotating and enlarging (reducing) the position vector [p, q] ^T related to the feature point of the left figure. plus the value is intended to mean that the degree of similarity is the image coordinates [x, y] ^T of the maximum feature points.

In this way, the control point estimation means 26 uses the temporary control point image coordinates for all the feature points E _m ⁽ⁿ⁾ (1 ≦ m ≦ M ⁽ⁿ⁾ ) included in the landmark L ^(n). estimate tau _m ⁽ⁿ⁾ of determined. Then, in the image coordinates, a place where τ _m ⁽ⁿ⁾ is spatially dense is searched for, an average value of the temporary control point image coordinate estimated values τ _m ^{(n) in} the vicinity of the dense image coordinates is obtained, and the average The value is output as the control point estimated image coordinate κ ⁽ⁿ⁾ .

As a method for calculating the average value, for example, the control point estimation unit 26 divides the entire new input image I (x, y) into a plurality of partial regions (for example, may be divided into a grid pattern, Adjacent partial areas may be divided so as to overlap each other), and 1 depending on which partial area the temporary control point image coordinate estimated value τ _m ⁽ⁿ⁾ (1 ≦ m ≦ M ⁽ⁿ⁾ ) belongs to Voting is performed for each vote, and the partial region R ⁽ⁿ⁾ with the largest number of votes is obtained. Subsequently, the control point estimating means 26 obtains an average value of only the temporary control point image coordinate estimated value τ _m ⁽ⁿ⁾ cast in the partial area R ⁽ⁿ⁾ having the largest number of votes. The average value may be output as the control point estimated image coordinates κ ⁽ⁿ⁾ .

When the matching result information H ⁽ⁿ⁾ shown in the equation (10) is used instead of the equation (8), the control point estimation unit 26 does not use one vote for voting in the average value calculation method. Alternatively, a variable number of votes corresponding to the similarity λ _m ⁽ⁿ⁾ may be cast. For example, the number of votes a _m ⁽ⁿ⁾ to be cast can be set as shown in the equation (14) using an increase function α in a broad sense. Here, a polygonal line function can be used as the increase function α in a broad sense (see equation (15)).

The control point estimating means 26, in addition to the control point estimation image coordinates kappa ^(n), the control point estimation reliability v a ^{(n) (1 ≦ n ≦} N) may be output. For example, as the control point estimation reliability v ⁽ⁿ⁾ , the number of votes of the partial region R ⁽ⁿ⁾ having the largest number of votes can be used. Alternatively, the control point estimation reliability v ⁽ⁿ⁾ may be obtained by dividing the number of votes of the partial region R ⁽ⁿ⁾ having the largest number of votes by the total number of votes.

In this way, the control point estimation means 26 determines the control point estimation image coordinates κ ⁽ⁿ⁾ (and the control point estimation reliability v ⁽ⁿ⁾ ) of the ^nth landmark L ⁽ⁿ⁾ as necessary. obtain. The estimation procedure, all landmark ^{(L (1), ...,} L (N)) was performed for the control point estimation image coordinates ^{(κ (1), ...,} κ (N)) ( also, if necessary Then, the control point estimation reliability (v ⁽¹⁾ ,..., V ^(N) )) is obtained.

The camera parameter estimation means 28 estimates camera parameters.
Specifically, the camera parameter estimation unit 28, N pieces of control point world coordinates read from the landmark storage unit ^{18 (w (1), ...} , w (N)) and are input from the control point estimating means 26 The camera parameter θ is calculated based on the ^N control point estimation image coordinates (κ ⁽¹⁾ ,..., Κ ^(N) ), and is output as a result of the camera parameter estimation function unit 20.

For example, the camera parameter estimation means 28 projects the N control point world coordinates (w ⁽¹⁾ ,..., W ^(N) ) onto the image coordinates by the camera parameter θ ′, and the result and the control point estimated image coordinates ( θ ′ that minimizes an error from κ ⁽¹⁾ ,..., κ ^(N) ) can be obtained by the nonlinear least square method, and the result can be used as the camera parameter θ (see equation (16)). Here, the function β (w; θ) represents the projection from the world coordinate w to the image coordinate by the camera parameter θ.

Further, the camera parameter estimation means 28 may consider the control point estimation reliability (v ⁽¹⁾ ,..., V ^(N) ) input from the control point estimation means 26 when estimating the camera parameter θ. For example, equation (17) can be used instead of equation (16).

  Furthermore, for example, the solution of the perspective N-point problem by RANSAC (MA Fischler, RC Bolles, “Random sample consensus, a paradigm for model fitting with applications to image analysis and automated cartography”, Comm. Of the ACM, Vol. 24, pp 381-395, 1981.) can be used.

  In this way, by configuring the camera calibration device 1, even if a visually characteristic point and a known point in the world coordinates are given separately, the world coordinates specified in the input image can be obtained. Using the image coordinates of the known control points and the extracted feature points, the camera parameters can be calculated by estimating the image coordinates of the control points from the feature points extracted in the new input image.

  In addition, the position of the origin of the world coordinate system and how to take each axis are arbitrary. In the present embodiment, the world coordinate system is represented by a Cartesian coordinate system, and the axes thereof are the X axis, the Y axis, and the Z axis. In this case, for example, the origin is placed on a survey reference such as a triangular point, The two axes Y and Y can be in the horizontal plane, the X axis can be north, the Y axis can be east, and the Z axis can be vertically upward. Alternatively, each axis may be taken in accordance with a facility such as a sports stadium or a competition field. For example, with the center mark center of the soccer field as the origin, the X-axis is parallel to the touch line and the right side when viewed from the coach area side, the Y-axis is the half-way line direction and away from the coach area, and the Z-axis is It can be taken vertically and upward with respect to the ground of the field.

  Note that the feature extraction unit 22 of the camera parameter estimation function unit 20 may extract feature points or the like not from the entire new input image I (x, y) but from a fixed region defined in the input image.

  The camera calibration apparatus 1 can be applied to a video camera such as a broadcast video camera and a consumer video camera in addition to a general camera (digital camera). The camera calibration apparatus 1 can also realize each unit as a function program in a computer, and can also operate the camera calibration program by combining the function programs.

[Operation of camera calibration device]
Next, the operation of the camera calibration apparatus 1 in the embodiment of the present invention will be described. As described above, the camera calibration apparatus 1 is roughly classified into the landmark designation function unit 10 and the camera parameter estimation function unit 20 according to the function. Below, operation | movement is demonstrated about each.

[[Operation of Landmark Specification Function]]
First, the operation of the landmark designation function unit 10 of the camera calibration apparatus 1 according to the embodiment of the present invention will be described with reference to FIG.

  First, the camera calibration apparatus 1 designates the pattern image area D corresponding to one landmark L and the image coordinates c of the control point C in the input image by the landmark designation means 12 (S11).

  Then, the camera calibration device 1 extracts some feature points from the designated pattern image region D by the feature extraction unit 14 (S12). Thereafter, the camera calibration apparatus 1 extracts the feature point arrangement information g of one feature point E by the feature extraction unit 14 (S13), and further extracts the image feature f of the feature point E (S14). Thereafter, the camera calibration apparatus 1 determines whether or not there is a feature point from which the feature point arrangement information and the image feature have not yet been extracted by the feature extraction unit 14 (S15), and if there is (Yes in S15). Returns to S13, and if not (No in S15), proceeds to S16.

  And the camera calibration apparatus 1 inputs the world coordinate w of the control point C of the landmark L by the world coordinate input means 16 (S16). This corresponds to the control point C to which the image coordinate c is input by the landmark specifying means 12.

  Then, the camera calibration apparatus 1 stores the set of the feature point arrangement information G and the image feature F for all the feature points of the landmark L and the world coordinates w of the control point C by the landmark storage unit 18 ( S17).

  Then, the camera calibration device 1 determines whether or not to process another landmark by the landmark designating unit 12 (S18). If yes (Yes in S18), the process returns to S11. In (No in S18), the operation of the landmark designation function unit 10 of the camera calibration device 1 is terminated.

  The operation of the landmark designation function unit of the camera calibration device 1 is not limited to this, and can be changed without departing from the spirit of the present embodiment. For example, S16 may be performed prior to S11-S15. Further, for example, the pattern image area D corresponding to all landmarks L to be processed first and the image coordinates c of the control point C may be designated by S11.

[[Operation of camera parameter estimation function]]
Next, the operation of the camera parameter estimation function unit 20 of the camera calibration apparatus 1 according to the embodiment of the present invention will be described with reference to FIG.

First, the camera calibration device 1 includes several feature points B _k , arrangement information γ _{k of the} feature points, and image features φ from an input image (or a fixed region thereof) newly input by the feature extraction unit 22. _k is extracted (S21). This is the same as the operation of the feature extraction unit 14 of the landmark designation function unit 10 except for the difference between extraction from the entire input image or extraction from the designated pattern image region, S12, S13, S14, The operation is almost the same as S15 (see FIG. 6).

Then, the camera calibration apparatus 1 reads the image feature F of one landmark L from the landmark storage unit 18 by the collating unit 24 (S22). Then, the camera calibration device 1, the collation means 24, the image feature f included in the image feature F, the similarity is the collation result h the number of the image feature phi _k that maximizes (S23). Thereafter, the camera calibration apparatus 1 determines whether or not there is an image feature for which the collation result has not been obtained by the collation unit 24 (S24), and if there is (Yes in S24), returns to S23, and if not (S24) No), the process proceeds to S25.

  Thereafter, the camera calibration device 1 collects the collation results h by the collating unit 24 and sets the collation result information H of the landmark L (S25). Thereafter, the camera calibration device 1 determines whether or not there is a landmark for which collation result information is not obtained by the collating unit 24 (S26). If there is a landmark (Yes in S26), the process returns to S22. If NO in S26, the process proceeds to S27.

Then, the camera calibration apparatus 1 reads the feature point arrangement information G of one landmark L from the landmark storage unit 18 by the control point estimation unit 26 (S27). Thereafter, the camera calibration device 1 uses the control point estimation unit 26 to set the feature point E of the landmark L to L and E obtained from the feature point arrangement information g of G corresponding to E and the matching result information H. A temporary control point image coordinate estimated value τ is calculated from the feature point arrangement information γ _h with the corresponding matching result h as a number, using equation (13) (S28). Thereafter, the camera calibration apparatus 1 determines whether or not there is a feature point for which a temporary control point image coordinate estimated value has not been obtained by the control point estimating means 26 (S29). If not (No in S29), the process proceeds to S30.

  Thereafter, the camera calibration device 1 obtains the average value of the temporary control point image coordinate estimated value τ by the control point estimating means 26 and sets it as the control point estimated image coordinate κ of the landmark L (S30). Thereafter, the camera calibration device 1 determines whether or not there is a landmark for which the control point estimation image coordinates are not obtained by the control point estimation means 26 (S31), and if there is (Yes in S31), returns to S27. If not (No in S31), the process proceeds to S32.

  Then, the camera calibration apparatus 1 uses the camera parameter estimation means 28 to determine the perspective N-point problem based on the control point estimated image coordinates κ over all landmarks and the control point world coordinates w read from the landmark storage means. The camera parameter θ is calculated using a solution or a predetermined nonlinear least square method (S32). Thereafter, the operation of the camera parameter estimation function unit 20 of the camera calibration apparatus 1 is terminated.

  The operation of the camera parameter estimation function unit 20 of the camera calibration device 1 is not limited to this, and can be changed without departing from the spirit of the present embodiment.

  With the above operation, even if a visually characteristic point and a known point in the world coordinates are given separately, the image coordinates and extraction of the known control point in the world coordinates specified in the input image are extracted. Using the feature points, the camera parameters can be calculated by estimating the image coordinates of the control points from the feature points extracted in the new input image.

DESCRIPTION OF SYMBOLS 1 Camera calibration apparatus 10 Landmark designation | designated function part 12 Landmark designation | designated means 14 Feature extraction means (1st feature extraction means)
16 World Coordinate Input Unit 18 Landmark Storage Unit 20 Camera Parameter Estimation Function Unit 22 Feature Extraction Unit (Second Feature Extraction Unit)
24 collating means 26 control point estimating means 28 camera parameter estimating means

Claims (5)

Using the image coordinates of the known control points of the world coordinates specified in the input image and the extracted feature points, the image coordinates of the control points are estimated from the feature points extracted in the new input image, and camera parameters are calculated. A camera calibration device that
In the input image, landmark designation means for designating the image coordinates of the control point, which is a point whose world coordinates are known, and an image region to be associated with the control point;
One or more feature points that are visually characteristic points are extracted from the image area designated by the landmark designation means, and first arrangement information relating to the arrangement of the feature points and the visual representation of the feature points are extracted. First feature extraction means for extracting a first image feature obtained by quantifying the feature;
World coordinate input means for inputting world coordinates of the control point designated by the landmark designation means;
A set of the first arrangement information and the first image feature extracted by the feature extraction unit from the image region associated with the control point, and the world coordinates of the control point input from the world coordinate input unit. Landmark storage means for storing a plurality of landmarks as one landmark;
One or more feature points that are visually characteristic points are extracted from the new input image, and second arrangement information related to the arrangement of the feature points and the visual features of the feature points are quantified. Second feature extraction means for extracting two image features;
The second image having the highest degree of similarity among the second image features extracted by the second feature extraction unit for each first image feature of the landmark read from the landmark storage unit. A matching means for searching for a feature and outputting the matching result as matching result information for each landmark;
Based on each first arrangement information of landmarks read from the landmark storage means, second arrangement information inputted from the second feature extraction means, and matching result information inputted from the matching means, Control point estimation means for estimating an image coordinate of the control point corresponding to the landmark in a new input image, and outputting it as a control point estimation image coordinate;
Camera parameter estimation means for estimating camera parameters based on the world coordinates of the control points read from the landmark storage means and the control point estimation image coordinates input from the control point estimation means;
A camera calibration device comprising: The first arrangement information includes a position vector from the control point to the feature point in the input image, rotation information and scale information in the vicinity of the feature point,
The second arrangement information includes image coordinates of the feature points in the new input image, rotation information and scale information in the vicinity of the feature points,
The control point estimation means includes
The first information extracted together with the rotation information and scale information included in the first arrangement information, and the first arrangement information, with respect to the position vector included in the first arrangement information of the landmark. A position vector in a new input image using the rotation information and scale information included in the second arrangement information extracted together with the second image feature corresponding to the image feature and the corresponding matching result. Converting, and subtracting the converted position vector from the image coordinates of the feature points included in the second arrangement information to obtain a temporary control point image coordinate estimated value;
Control of the control points corresponding to the landmarks in the new input image by averaging the temporary control point image coordinate estimated values obtained from the first arrangement information of the landmarks by a predetermined method The camera calibration apparatus according to claim 1, wherein the coordinates are point estimated image coordinates. The control point estimation means includes
The whole of the new input image is divided into a plurality of partial areas, and according to which partial area the temporary control point image coordinate estimated value belongs to, the values relating to the partial area are accumulated one by one, and the total The camera calibration apparatus according to claim 2, wherein an average is obtained only for a temporary control point image coordinate estimated value relating to the partial region having the largest value. The verification means includes
Output the similarity with the matching result as matching result information for each landmark,
The control point estimation means includes
The entire new input image is divided into a plurality of partial areas, and the temporary control point image coordinate estimated value is calculated according to which partial area the temporary control point image coordinate estimated value belongs to. The value corresponding to the similarity corresponding to the collation result is accumulated in the value related to the partial area, and an average is obtained only for the temporary control point image coordinate estimated value related to the partial area having the largest total value. The camera calibration device according to claim 2, characterized in that: Using the image coordinates of the known control points of the world coordinates specified in the input image and the extracted feature points, the image coordinates of the control points are estimated from the feature points extracted in the new input image, and camera parameters are calculated. Computer to
Landmark designation means for designating image coordinates of the control point, which is a point whose world coordinates are known, and an image area to be associated with the control point in the input image;
One or more feature points that are visually characteristic points are extracted from the image area designated by the landmark designation means, and first arrangement information relating to the arrangement of the feature points and the visual representation of the feature points are extracted. A first feature extraction unit that extracts a first image feature obtained by quantifying the feature, and stores the first image feature in a landmark storage unit as a constituent element of the landmark associated with the control point related to the image region;
World coordinate input means for inputting the world coordinates of the control point designated by the landmark designation means, and storing it in the landmark storage means as a component of the landmark associated with the control point;
One or more feature points that are visually characteristic points are extracted from the new input image, and second arrangement information related to the arrangement of the feature points and the visual features of the feature points are quantified. Second feature extraction means for extracting two image features;
The second image having the highest degree of similarity among the second image features extracted by the second feature extraction unit for each first image feature of the landmark read from the landmark storage unit. A matching means for searching for features and outputting the matching result as matching result information for each landmark,
Based on each first arrangement information of landmarks read from the landmark storage means, second arrangement information inputted from the second feature extraction means, and matching result information inputted from the matching means, Control point estimation means for estimating an image coordinate of the control point corresponding to the landmark in a new input image, and outputting it as a control point estimated image coordinate;
Camera parameter estimation means for estimating camera parameters based on the world coordinates of the control points read from the landmark storage means and the control point estimation image coordinates input from the control point estimation means;
A camera calibration program characterized by functioning as

Images