JP2006033449A - Device and program for calculating/calibrating camera parameter - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device and a program for calculating/calibrating camera parameter capable of calibrating camera parameter, by efficiently corresponding a reference point on the model of an existing pattern or structure to a live-action image. <P>SOLUTION: A camera parameter calculating/calibrating device 1 calculates at least one of camera parameters including a position coordinate, photographing direction, and focal length of a camera; and calibrates the camera parameter according to the calculated result. It comprises a camera parameter storage means 7, image input means 3, point correspondence input means 5, image generating means 15, image composite means 17, image display means 19, inverse projection means 9, point correspondence storage means 11, and a camera parameter calculation means 13. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a camera parameter calculation calibration apparatus and a camera parameter calculation calibration program for calculating and sequentially calibrating camera parameters representing characteristics such as a camera installation position and orientation and a lens focal length.

  Conventionally, as a technique for calculating and calibrating (calibrating) camera parameters representing characteristics such as camera installation position (coordinates), posture (photographing direction), and focal length of a lens, the “camera calibration apparatus and A method, an image processing apparatus and method, a program providing medium, and a camera ”(see Patent Document 1) are disclosed.

  In this method, the camera parameters are automatically calibrated by executing the correspondence between the captured image captured by the camera and the definition of the geometric shape and the reference image having a unique pattern by a method such as a luminance error minimization method. It is something to do.

JP 2000-350239 A (paragraphs 0060-0074, FIG. 1)

  However, in the conventional method of automatically calibrating camera parameters, such as the method described in “Camera Calibration Device and Method, Image Processing Device and Method, Program Provision Medium, and Camera”, a stable calibration result ( In order to obtain a calibration result, it is necessary to use an artificial pattern such as a reference image or project the artificial pattern using a projector or the like. For this reason, if you attempt to calibrate camera parameters based on a model of an existing pattern or structure, for example, a white line drawn on a sports stadium or a model of a building being photographed, the existing pattern In addition, there is a problem that it is difficult to obtain a stable operation when the reference point on the model of the structure and the actual image captured by the camera are automatically associated with each other. On the other hand, there is a problem that there is no efficient user interface for performing correspondence manually, and data input is difficult.

  Therefore, in the present invention, it is possible to perform the calibration of the camera parameters by solving the above-described problems and efficiently associating the reference points on the existing pattern or model of the structure with the photographed image. An object of the present invention is to provide a camera parameter calculation calibration device and a camera parameter calculation calibration program.

  In order to solve the above-mentioned problem, the camera parameter calculation calibration device according to claim 1 calculates at least one of camera parameters including a camera position coordinate, a shooting direction, and a focal length, and sets the camera parameter according to the calculated calculation result. A camera parameter data calculating and calibrating apparatus for calibrating, comprising a camera parameter storage means, a photographed image input means, a point correspondence input means, a reference image generation means, a composite image output means, a composite image display means, and a back projection Means, point correspondence information storage means, and camera parameter calculation means.

  According to such a configuration, the camera parameter calculation calibration apparatus stores in advance the initial camera parameter which is at least one initial value of the camera parameter in the camera parameter storage unit, and the calculation result calculated by the camera parameter calculation unit Store (update) new camera parameters configured according to The camera here refers to a video camera (photographing camera), a digital camera, an image scanner, or the like, which acquires an image (image acquisition device). The camera parameters are the position coordinates on the plane of the installation position where the camera is installed, the angle indicating the shooting direction (posture) that is the direction in which the camera is shooting the subject, and the focal length from the camera to the subject. It contains at least one. First, the camera parameter calculation / calibration apparatus inputs a photographed image captured by the camera by a photographed image input unit. This live-action image includes a subject (existing pattern or structure) having some geometric shape (three-dimensional geometric shape).

  Subsequently, the camera parameter calculation / calibration apparatus uses the point correspondence input unit to set a model reference point that serves as a reference on a model related to a known structure that matches the geometric shape included in the actual image input by the actual image input unit. And the actual image corresponding point on the actual image are input as image coordinates. The model is included in the photographed image, that is, matches the geometric shape existing in the subject space, and is, for example, a planar figure such as a triangle or a quadrangle, or a three-dimensional figure such as a triangular pyramid or a quadrangular prism. The model reference point is a reference point on the model, for example, each vertex of the triangle when the model is a triangle. The live action corresponding point indicates a place where the subject is present on the live action image. The model reference point and the live-action corresponding point are given as image coordinates (x, y).

  Then, the camera parameter calculation calibration device performs projection conversion of the data related to the input model by the reference image generation unit according to the initial camera parameter or the camera parameter stored in the camera parameter storage unit, so that the point correspondence input unit A graphic representing the correlation between the input model reference point and the actual shooting corresponding point on the live-action image is generated as a reference image. Projection conversion is to perform perspective projection and form a figure by substituting the coordinates of the model reference point into a preset mathematical expression. The drawing representing the correlation is an arrow-like shape (see FIG. 4 to be described later) connecting the model reference point and the actual shooting corresponding point.

  Then, the camera parameter calculation calibration device superimposes the reference image generated by the reference image generation means and the actual image by the composite image output means and outputs the composite image, and displays the output composite image as a composite image display Display by means. In addition, the camera parameter calculation calibration device calculates the three-dimensional coordinates of the model reference point based on the image coordinates input by the point correspondence input means by the back projection means, and the calculated three-dimensional coordinates and the point correspondence input. It stores in the point correspondence information storage means as point correspondence information that associates the image coordinates of the corresponding point of the live-action input by the means.

  Thereafter, the camera parameter calculation calibration device calculates camera parameters based on the point correspondence information stored in the point correspondence information storage means by the camera parameter calculation means, and calibrates the camera parameters stored in the camera parameter storage means. (Update.

  The camera parameter calculation calibration program according to claim 2, wherein the camera parameter calculation program calculates at least one of camera parameters including a camera position coordinate, a shooting direction, and a focal length, and calibrates the camera parameter according to the calculated calculation result. An actual camera image input means that stores an initial camera parameter that is at least one initial value of the parameter, and that includes a camera parameter storage means for storing a camera parameter calibrated according to the calculation result and a point correspondence information storage means. , Point correspondence input means, reference image generation means, composite image output means, composite image display means, back projection means, point correspondence information storage control means, and camera parameter calculation means.

  According to this configuration, the camera parameter calculation / calibration program is input to the photographed image input unit by the photographed image input unit, and the point correspondence input unit inputs the photographed image captured by the photographed image input unit. Correlation between a model reference point serving as a reference on a model related to a known structure that matches the geometric shape and a corresponding point on a live-action image is input as image coordinates. Subsequently, the camera parameter calculation calibration program performs projection conversion of the data related to the input model by the reference image generation unit according to the initial camera parameter or the camera parameter stored in the camera parameter storage unit, so that the point correspondence input unit A figure representing the correlation between the model reference point input in step 1 and the corresponding point on the live-action image is generated as a reference image. Then, the camera parameter calculation calibration program superimposes the reference image generated by the reference image generation unit by the composite image output unit and the actual image and outputs the composite image, and outputs the composite image by the back projection unit by the point correspondence input unit. Based on the input image coordinates, the three-dimensional coordinates of the model reference point are calculated. The camera parameter calculation calibration program associates the three-dimensional coordinates of the model reference point calculated by the back projection unit with the image coordinates of the actual shooting corresponding point input by the point correspondence input unit by the point correspondence information storage control unit. Stored as point correspondence information in the point correspondence information storage means, the camera parameter calculation means calculates camera parameters based on the point correspondence information stored in the point correspondence information storage means, and stores them in the camera parameter storage means. The camera parameters (initial camera parameters) are calibrated (updated).

  According to the first and second aspects of the present invention, the correlation between the model reference point of the existing pattern or structure and the corresponding point of the live-action image is input as the image coordinates, the reference image is generated, By displaying the composite image with the image, it is possible to efficiently associate the model reference point with the live-action corresponding point, and by adjusting these positions, it is possible to perform camera parameter calibration. it can.

Next, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.
<Configuration of camera parameter calculation calibration device>
FIG. 1 is a block diagram of a camera parameter calculation calibration apparatus. As shown in FIG. 1, the camera parameter calculation / calibration apparatus 1 calculates camera parameters such as the installation position (position coordinates), posture (photographing direction), and lens focal length of an image acquisition device (not shown). The camera parameters are calibrated, and image input means (actual image input means) 3, point correspondence input means 5, camera parameter storage means 7, back projection means 9, and point correspondence storage means. (Point correspondence information storage control means) 11, camera parameter calculation means 13, image generation means (reference image generation means) 15, image composition means (composite image output means) 17, image display means (composite image display means) 19).

  The image input means 3 is connected to an image acquisition device 2 such as a video camera, a digital camera, or an image scanner, and captures a camera real image (real image signal) acquired (captured and input) by the image acquisition device. The data is stored in a frame memory (not shown) provided inside.

  The point correspondence input unit 5 is configured such that the operator of the apparatus 1 applies a pair of coordinates between the model reference point and the live-action corresponding point on the display screen (image coordinate system) to the composite image displayed by the image display unit 19. Is input as image coordinates. This point correspondence input means 5 comprises a measuring means for measuring coordinates such as a rotary encoder and an input means for inputting contact information such as a tact switch, and is connected to, for example, a computer (such as a personal computer). It can be realized with a mouse, digitizer, tablet, touch panel, or the like.

In addition, for example, in the case of a mouse, the pair of coordinates by the point correspondence input means 5 can be input by coordinate information when the mouse button is pressed and coordinate information when the mouse button is released. One of the coordinate pairs is represented as a start point P (p _x , p _y ), and the other is represented as an end point Q (q _x , q _y ). Note that these are part of a graphic representing the correlation between the model and the photographed image, and are called image coordinates P and image coordinates Q.

  The camera parameter storage means 7 is constituted by a memory or the like, and stores the input camera parameter initial value (initial camera parameter) and also stores the calculated camera parameter. The camera parameter initial value (initial camera parameter) may be input (given) from the outside, or may be a value stored in advance.

  The camera parameters are the three axes of the camera with respect to the installation position (three-dimensional position coordinates of the camera) of the image acquisition device 2 (for example, camera) and the photographing direction (reference pattern [existing pattern or model of structure in the subject space]). Relative posture) and at least one of the focal length of the lens. Here, a total of seven parameters (three parameters for three-dimensional position coordinates, three parameters for three-axis relative posture and one parameter for lens focal length) It is expressed. It should be noted that a lens distortion parameter may be included in the camera parameters including these seven parameters.

Pan angle (azimuth angle) α, tilt angle (elevation angle) δ, and roll angle (around the optical axis) relative to the coordinate system (hereinafter referred to as the world coordinate system) fixed to the reference pattern (model) ), The three-dimensional coordinates (T _X , T _Y , T _Z ) in the world coordinate system as the three-dimensional position coordinates (three-dimensional position) of the camera, and the focal length as the focal length of the lens. The length f is used.

Backprojection means 9, image coordinates P (p _x, p _y) of the start point input by the point corresponding input means 5, to determine corresponding to any point on the model, the image coordinates of the starting point P (p _x , P _y ) for outputting world coordinates W (W _X , W _Y , W _Z ). For example, first, the z-axis having the same x-axis and y-axis as the x-axis and y-axis of the image coordinate system displayed on the image display means 19 and orthogonal to both the x-axis and y-axis is selected. An orthogonal coordinate system xyz is defined (this orthogonal coordinate system xyz is referred to as a camera coordinate system).

  The origin of the camera coordinate system is set at a point separated in the (−z) direction by the focal length f from the center of the image plane displayed on the image display means 19. This camera coordinate system is relative to the model (ie, the world coordinate system) with the posture (shooting direction) and position (camera direction) described by the camera parameters stored in the camera parameter storage means 7 or the camera parameter initial values (initial camera parameters). It is assumed that the position coordinates are taken.

Here, the method of back projection from the image coordinates to the world coordinates in the back projection means 9 will be described with reference to FIG. FIG. 2 is a diagram schematically illustrating back projection from image coordinates to world coordinates.
As shown in FIG. 2, a half line h starting from the origin (projection center) of the camera coordinate system and passing through the camera coordinates of the start point P input by the point correspondence input means 5 is assumed. Subsequently, intersections or neighborhood points of the half line h with vertices, ridges, surfaces, and other characteristic geometric features inside and outside the model are obtained. FIG. 2 shows an example in which a point on the model having the closest Euclidean distance to the half line h is obtained. The world coordinates of the point thus obtained are set as W (W _X , W _Y , W _Z ).
Here, the seven camera parameters are defined (represented) by the following formula 1.

As shown in Equation (1), the camera parameter is defined (represented) by a column vector π (hereinafter referred to as camera parameter π). The image coordinates P ^ (p ^{^} _x , p ^{^} _y ) obtained by conversion using the camera parameter π are used.

Next, the relationship between the world coordinate system and the camera coordinate system will be exemplified and described with reference to FIG. FIGS. 3A to 3E are schematic diagrams illustrating the relationship between the world coordinate system and the camera coordinate system using vectors.
As shown in FIG. 3A, the origin in the world coordinate system (X, Y, Z) and the origin in the camera coordinate system (x, y, z) are vectors T (T _X , T _Y , T _Z ). That is, the position of the origin of the camera coordinate system in the world coordinate system (X, Y, Z) is represented by T (T _X , T _Y , T _Z ).

Also, as shown in FIGS. 3B to 3E, the orientation of the camera coordinate system (camera shooting direction) is the pan angle α with respect to the reference coordinate system x ₀ -y ₀ -z ₀ (FIG. 3C). ), A tilt angle δ (FIG. 3D) and a roll angle φ (FIG. 3E). The axes of the world coordinate system, the reference coordinate system, and the camera coordinate system are all set to the same scaling.

An example of the definition of the camera coordinate system based on the pan angle α, the tilt angle δ, and the roll angle φ will be described with reference to FIGS. First, as shown in FIG. 3B, the axes of x ₀ , y ₀ and z ₀ of the reference coordinate system x ₀ -y ₀ -z ₀ are the X, (-Z) and Y axes of the world coordinate system. It shall be in the same direction as the axis. Subsequently, as shown in FIG. 3C, the coordinate system x ₁ -y ₀ -z ₁ is changed by rotating the reference coordinate system x ₀ -y ₀ -z ₀ by an angle α around the y ₀ axis. obtain. Then, as shown in FIG. 3D, the coordinate system x ₁ -y ₀ -z ₁ is rotated about the x ₁ axis by an angle δ to obtain the coordinate system x ₁ -y ₂ -z ₂ . . Finally, as shown in FIG. 3E, the coordinate system x ₁ -y ₂ -z ₂ is rotated about the z ₂ axis by an angle φ to obtain the camera coordinate system xyz.

Next, an example of the definition of the image coordinate system xy on the display screen displayed on the image display means 19 (see FIG. 1) will be described with reference to mathematical expressions. The x-axis and y-axis of the image coordinate system are the same as the x-axis and y-axis of the camera coordinate system. That is, both direction and scaling are the same. With the above definition, the relationship between the world coordinates W (W _X , W _Y , W _Z ) and the image coordinates P ^ (p ^{^} _x , p ^{^} _y ) is given by the following equation (2).

In Equation (2), f is a focal length (focal length).
Here, when J is a Jacobian matrix obtained by partial differentiation of the image coordinates P ^ (p ^{^} _x , p ^{^} _y ) with the camera parameter π, the following equation (3) is obtained.

Returning to FIG. 1, the description of the configuration of the camera parameter calculation calibration apparatus 1 will be continued.
The point correspondence storage unit 11 includes a memory or the like, and includes the world coordinates W (W _X , W _Y , W _Z ) obtained by the back projection unit 9 and the end point image input by the point correspondence input unit 5. A pair with the coordinates Q (q _x , q _y ) is stored (added or updated) as one record. Assuming that the total number of records stored in the point correspondence storage means 11 is N, the nth record is represented by [W ⁽ⁿ⁾ (W _X ⁽ⁿ⁾ , W _Y ⁽ⁿ⁾ , W _Z ⁽ⁿ⁾ ), Q ^{( n)} (q _x ⁽ⁿ⁾ , q _y ⁽ⁿ⁾ )] is represented by a superscript with parentheses.

  The camera parameter calculation unit 13 newly calculates a camera parameter based on the record information stored in the point correspondence storage unit 11 in accordance with the trigger input. The trigger input is performed by the user of the camera parameter calculation / calibration apparatus 1 using an operation unit (not shown). For example, the trigger input is interlocked with a physical switch and a virtual button displayed on the image display means 19. In this embodiment, camera parameters are calculated by the Levenberg-Marquardt method, which is one algorithm of the nonlinear least square method.

In the Levenberg-Marquardt method, first, for the world coordinates W ⁽ⁿ⁾ (W _X ⁽ⁿ⁾ , W _Y ⁽ⁿ⁾ , W _Z ⁽ⁿ⁾ ) of each record stored in the point correspondence storage unit 11, Using the conversion formula from the world coordinates to the image coordinates shown in Equation (2), the world coordinates W ⁽ⁿ⁾ (W _X ⁽ⁿ⁾ , W _Y ⁽ⁿ⁾ , W _Z ⁽ⁿ⁾ ) are converted into the image coordinates P. ^ ^(N) (p ^{^} _x ⁽ⁿ⁾ , p ^{^} _y ⁽ⁿ⁾ ).

And in this Levenberg-Marquardt method, the image coordinates P ^ ⁽ⁿ⁾ for each of the N records _{^{(p ^ x (n),}} p ^ y (n)) and the image coordinates _{^{Q (n) (q x (}} n) , Q _y ⁽ⁿ⁾ ) with respect to a vector e in which the errors are arranged in the column direction and the respective image coordinates P ^ ⁽ⁿ⁾ (p ^{^} _x ⁽ⁿ⁾ , p ^{^} _y ⁽ⁿ⁾ ) A matrix K obtained by arranging the Jacobian matrix J ⁽ⁿ⁾ obtained in 3) in the column direction is defined by the following equation (4).

  Subsequently, the camera parameter correction vector Δπ is defined by the following equation (5).

In Equation (5), λ is a positive parameter.
In the case of the Levenberg-Marquardt method, for example, the camera parameter π is estimated by the following steps.
First, a positive constant set in advance is substituted for λ (step a1). For example, λ = 0.001. Subsequently, a camera parameter correction vector Δπ is obtained from Equation (4) and Equation (5) (Step a2). Then, it is determined whether or not the number of executions for obtaining the camera parameter correction vector Δπ exceeds a specified threshold value (step a3), and when it exceeds (step a3, Yes), the process ends, and when it does not exceed (step a3, No). to the [pi _tmp is a temporary camera parameter correction vector Derutapai, sets a π _tmp = π + Δπ (step a4).

Subsequently, in the Levenberg-Marquardt method, each world coordinate W ⁽ⁿ⁾ (W _X ⁽ⁿ⁾ , W _Y ⁽ⁿ⁾ , W _Z ⁽ⁿ⁾ ) is ^expressed by Equation (2) using π _tmp. The transformation is executed, and the resulting image coordinates are set to (p _tmpx ⁽ⁿ⁾ , p _tmpy ⁽ⁿ⁾ ) (step a5). Then, the calculation shown in the following equation (6) is executed to obtain a vector e _tmp (step a6).

If ‖e _tmp ‖ ≧ ‖e‖ with respect to the vector e _tmp obtained by the equation (6), a constant (for example, 10) greater than 1 is applied (step a7), and the process returns to step a2. Then, the camera parameter π is updated by π = π _tmp (step a8). Incidentally, ‖e _tmp ‖ indicates that the vector e _tmp is returned to a scalar.

Furthermore, when the value of ‖e‖-‖e _tmp ‖ falls below the specified threshold value, the process returns to step a2 (step a9). Then, λ is multiplied by a constant larger than 0 and smaller than 1 (for example, multiplied by 0.1) (step a10). Then, the process returns to step a2 (step a11) and ends (step a12).

  In the Levenberg-Marquardt method, the camera parameters (camera parameter initial values) stored in the camera parameter storage means 7 are rewritten by the camera parameters π obtained in steps a1 to a12 as described above.

  The image generation unit 15 is configured to generate a reference image based on the input model (model-related data, world coordinate system) and the camera parameter π or the camera parameter initial value (initial camera parameter) stored in the camera parameter storage unit 7. Is generated. For example, the image generation unit 15 first performs perspective projection (projection conversion) by substituting the world coordinates of the vertex group existing on the model into the mathematical formula (2) to obtain the image coordinate group.

  Subsequently, the image generation means 15 changes the pixel value of the pixel at the position of the image coordinate group stored in the frame memory (not shown) based on the obtained image coordinate group, or changes the pixel value to the area on the model. The reference image can be generated by changing the pixel value on the frame memory in the polygon determined by the corresponding plurality of pixel coordinates.

Further, the image generation means 15 is a point correspondence (world coordinates W (W _X , W _Y , W _Z ) stored in the point correspondence storage means 11 and the image coordinates Q (q _x , q _y ). Two records) may be made into a figure and generated as a reference image.

Here, an example of a method of making the point correspondence into a graphic in the image generation means 15 will be described below. First, the world coordinates W ⁽ⁿ⁾ (W _X ⁽ⁿ⁾ , W _Y ⁽ⁿ⁾ , W _Z ⁽ⁿ⁾ ) are converted into image coordinates using the conversion formula from world coordinates to image coordinates shown in Equation (2). P ^ ⁽ⁿ⁾ (p ^{^} _x ⁽ⁿ⁾ , p ^{^} _y ⁽ⁿ⁾ ). Then, on the frame memory which is not shown, the image coordinates ^{P ^ (n) (p ^} x (n), p ^ y (n)) from the image coordinates _{^{Q (n) (q x (}} n), q y ⁽ⁿ⁾ Draw an arrow leading to (a figure representing a correlation). The drawing of the arrow is executed for the point-corresponding set (record) stored in the point-corresponding storage unit 11.

  The image generation unit 15 is set to newly generate a reference image every time the camera parameter π stored in the camera parameter storage unit 7 is updated. The image generation means 15 is set so as to newly generate a reference image every time the point correspondence set (record) stored in the point correspondence storage means 11 is updated.

  The image synthesizing unit 17 superimposes the camera real image input by the image input unit 3 and the reference image generated by the image generating unit 15 to generate a new image (synthesized image). An example of a composite image by the image composition means 17 is shown in FIG. FIG. 4 is a diagram showing an example of a composite image (before calibration [before calibration]).

  As shown in FIG. 4, the result of perspective projection of the model is expressed by a thick line, so that a reference image is generated. This reference image is superimposed on the camera actual image and output to the image display means 19 ( Displayed). In addition, the point correspondence sets (records) stored in the point correspondence storage unit 11 are superimposed as arrows.

Here, FIG. 5 shows a composite image after the operation of the camera parameter calculation unit 13, that is, when the camera parameter π stored in the camera parameter storage unit 7 is updated. FIG. 5 is a diagram illustrating an example of a composite image (after calibration [after calibration]).
As shown in FIG. 5, each time the camera parameter π is updated (every time it is updated), the composite image is updated and fed back to the user of the device 1 via the image display means 19, that is, can be visually recognized. The

  Similarly, when a point correspondence set (record) stored in the point correspondence storage means 11 is updated by an operation by an operation means (not shown) of the apparatus 1, the composite image is immediately updated, Feedback to the user of the device 1, i.e. visible.

Returning to FIG. 1, the description of the configuration of the camera parameter calculation calibration apparatus 1 will be continued.
The image display means 19 displays a composite image obtained by combining the camera real image and the reference image by the image composition means 17. The image display means 19 is a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, or the like.

  According to the camera parameter calculation / calibration apparatus 1, an actual camera image captured by a camera (not shown) is input by the image input unit 3, and an actual image input by the image input unit 3 by the point correspondence input unit 5. The correlation between the image coordinates of the starting point (model reference point) that is the reference on the model of the known structure that matches the geometric shape contained in the image and the image coordinates of the end point (actual shooting corresponding point) on the live-action image Are input as image coordinates. Subsequently, the input model (data relating to the model) is projected and converted by the image generation unit 15 according to the initial camera parameter or the camera parameter stored in the camera parameter storage unit, so that the input is performed by the point correspondence input unit 5. A figure (arrow) representing the correlation between the image coordinates of the start point (model reference point) and the image coordinates of the end point (actual shooting corresponding point) on the live-action image is generated as a reference image. Then, the image synthesizing unit 17 superimposes the reference image generated by the image generating unit 15 and the photographed image and outputs it as a synthesized image.

  At this time, the back projection means 9 calculates the world coordinates (three-dimensional coordinates) of the image coordinates (model reference point) of the starting point based on the image coordinates input by the point correspondence input means 5, and the point correspondence storage means. 11, the world coordinates (three-dimensional coordinates) of the start point image coordinates (model reference point) calculated by the back projection means 9 and the end point image coordinates (actual shooting corresponding points) input by the point correspondence input means 5 are obtained. Based on the point-corresponding set (record) stored in the point-corresponding storage means 11 by the camera parameter calculation means 13 as a correlated point-corresponding pair (record, point correspondence information). The camera parameters are calculated, and the camera parameters (initial camera parameters) stored in the camera parameter storage means 7 are calibrated (updated). For this reason, the model and the live-action image are efficiently associated with each other using the reference point (start image coordinate) on the model of the existing pattern or structure and the image coordinate of the end-point on the live-action image. Parameter calibration can be performed.

<Operation of camera parameter calculation calibration device>
Next, the operation of the camera parameter calculation calibration apparatus 1 will be described with reference to the flowchart shown in FIG. 6 (see FIG. 1 as appropriate).
First, the camera parameter calculation / calibration apparatus 1 sets the camera parameter storage unit 7 from the outside or sets preset camera parameter initial values (initial camera parameters) to store all of the point correspondence storage unit 11. Clear (erase) (step S1).

  Subsequently, the camera parameter calculation / calibration apparatus 1 inputs the camera real image by the image input unit 3 (step S2), and the image generation unit 15 based on the camera parameter initial value stored in the camera parameter storage unit 7. Then, the points on the inputted model are projected, and the synthesized image is obtained by superimposing and synthesizing on the camera real image by the image synthesizing means 17. The camera parameter calculation / calibration apparatus 1 uses the image generation means 15 to make this point when the point correspondence storage means 11 has a point correspondence group that is a set (record) of a plurality of points. The correspondence group is converted into a figure (reference image) and further superimposed on the synthesized image (step S3). The synthesized image thus synthesized is displayed on the image display means 19.

Then, the camera parameter calculation / calibration apparatus 1 determines whether or not point correspondence (image coordinates of the start point and image coordinates of the end point) is input to the point correspondence input unit 5 (step S4), and is not determined to be input. In the case (step S4, No), the process proceeds to step S8. When it is determined that the input has been made (step S4, Yes), the point correspondence input by the point correspondence input means 5 (the start point image coordinates and the end point image coordinates). ) Is substituted into P (p _x , p _y ) and Q (q _x , q _y ) (step S5).

Then, the camera parameter calculation / calibration apparatus 1 performs back projection by the back projection means 9, and world coordinates W (W _X , W _Y , on the model corresponding to the image coordinates P (p _x , p _y ) of the starting point. W _Z ) is calculated (step S6). Subsequently, the camera parameter calculation / calibration apparatus 1 stores a set of world coordinates W (W _X , W _Y , W _Z ) and end point image coordinates Q (q _x , q _y ) in the point correspondence storage unit 11, that is, The point-corresponding set (record) is stored (step S7).

Then, the camera parameter calculation / calibration apparatus 1 determines whether or not the camera parameter calculation means 13 has received a trigger input (step S8). If it has not been determined that there has been a trigger input (step S8, No), step S11 is performed. When it is determined that the input has been made (step S8, Yes), the camera parameter calculation means 13 calculates the camera parameter π (step S9). That is, the camera parameter calculation / calibration apparatus 1 includes a point correspondence group stored in the point correspondence storage unit 11, that is, a plurality of records (world coordinates W (W _X , W _Y , W _Z ) on the model, and the end point. A new camera parameter π is calculated on the basis of the image coordinates Q (a combination with the image coordinates Q (q _x , q _y )) and stored (overwritten) in the camera parameter storage unit 7 (step S10).

  Thereafter, the camera parameter calculation calibration device 1 determines whether or not to end the process (operation) (step S11), and when it is determined to be ended by an external end signal or the like (step S11, Yes). If the process is terminated and it is not determined that the process is terminated (No in step S11), the process returns to step S2 and the process is continued.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment. For example, in the present embodiment, the camera parameter calculation calibration apparatus 1 has been described. However, the processing of each component of the apparatus 1 can be a camera parameter calculation calibration program described in a general or special computer language. It is also possible to use a camera parameter calculation calibration method in which the processing of the configuration is regarded as a process of processing information such as a camera actual image. In these cases, the same effect as the camera parameter calculation calibration apparatus 1 can be obtained.

  The camera parameter calculation / calibration apparatus 1 also includes a model based on the shape of a goal area (goal post) or penalty area (a model that matches the geometric shape) on a soccer field, a goal area (goal post) or penalty that is actually captured. It is also possible to calibrate the camera parameters with the real image of the area. Furthermore, if a model that matches the geometric shape, such as a rugby field, a tennis court, or a basketball court, is obtained, the camera parameter calculation calibration device 1 can be applied to the calibration of camera parameters.

It is a block diagram of the camera parameter calculation calibration apparatus which concerns on embodiment of this invention. It is a figure explaining operation | movement of the back projection means of the camera parameter calculation calibration apparatus shown in FIG. It is a figure showing the positional relationship and attitude | position relationship of a world coordinate system and a camera coordinate system. It is the figure which showed an example of the synthesized image (Before calibration [Before calibration]). It is the figure which showed an example of the synthesized image (after calibration [after calibration]). It is the flowchart explaining operation | movement of the camera parameter calculation calibration apparatus shown in FIG.

Explanation of symbols

1 Camera parameter calculation calibration device 3 Image input means (actual image input means)
5 Image generation means 7 Camera parameter storage means 9 Back projection means 11 Point correspondence storage means (point correspondence information storage means)
13 camera parameter calculation means 15 image generation means (reference image generation means)
17 Image composition means (composition image output means)
19 Image display means (composite image display means)

Claims (2)

A camera parameter data calculation calibration device that calculates at least one of camera parameters including a camera position coordinate, a shooting direction, and a focal length, and calibrates the camera parameter according to the calculated result,
Camera parameter storage means for storing an initial camera parameter, which is at least one initial value of the camera parameter, and storing a camera parameter calibrated according to the calculation result;
A photographed image input means for inputting a photographed image captured by the camera;
The correlation between the model reference point serving as a reference on the model that matches the geometric shape included in the photographed image input means by the photographed image input means and the photographed corresponding point on the photographed image is set as an image coordinate. Point corresponding input means to input,
Data relating to the model is input, and the model reference point input by the point-corresponding input means is obtained by projecting transformation of the data relating to the model according to the initial camera parameters or camera parameters stored in the camera parameter storage means. A reference image generating means for generating a figure representing a correlation with a corresponding point on the live-action image as a reference image;
A composite image output unit that superimposes the reference image generated by the reference image generation unit and the photographed image and outputs it as a composite image;
Composite image display means for displaying the composite image output by the composite image output means;
Back projection means for calculating the three-dimensional coordinates of the model reference point based on the image coordinates input by the point correspondence input means with reference to the composite image displayed by the composite image display means;
Point correspondence information storage means for storing point correspondence information associating the three-dimensional coordinates of the model reference point calculated by the back projection means and the image coordinates of the actual shooting corresponding point input by the point correspondence input means;
Camera parameter calculation means for calculating the camera parameters based on the point correspondence information stored in the point correspondence information storage means and calibrating the camera parameters stored in the camera parameter storage means;
A camera parameter calculation calibration apparatus comprising: An initial camera parameter which is at least one initial value of the camera parameter in order to calculate at least one of the camera parameters including the camera position coordinates, the shooting direction, and the focal length, and to calibrate the camera parameters according to the calculated calculation result And a computer comprising a camera parameter storage means for storing camera parameters calibrated according to the calculation result, and a point correspondence information storage means,
A photographed image input means for inputting a photographed image captured by the camera;
The correlation between the model reference point serving as a reference on the model that matches the geometric shape included in the photographed image input means by the photographed image input means and the photographed corresponding point on the photographed image is set as an image coordinate. Point corresponding input means,
Data relating to the model is input, and the model reference point input by the point-corresponding input means is obtained by projecting transformation of the data relating to the model according to the initial camera parameters or camera parameters stored in the camera parameter storage means. , A reference image generating means for generating a figure representing a correlation with a corresponding point on the live-action image as a reference image;
A composite image output unit that superimposes the reference image generated by the reference image generation unit and the photographed image and outputs it as a composite image;
Composite image display means for displaying the composite image output by the composite image output means;
Back projection means for calculating the three-dimensional coordinates of the model reference point based on the image coordinates input by the point correspondence input means with reference to the composite image displayed by the composite image display means;
Point correspondence information that associates the three-dimensional coordinates of the model reference point calculated by the back projection unit with the image coordinates of the actual shooting corresponding point input by the point correspondence input unit is stored in the point correspondence information storage unit. Point correspondence information storage control means,
Camera parameter calculation means for calculating the camera parameters based on the point correspondence information stored in the point correspondence information storage means and calibrating the camera parameters stored in the camera parameter storage means;
A camera parameter calculation calibration program characterized by being made to function as:

Images