JP4533090B2 - Index calibration apparatus and information processing method - Google Patents

Description

  The present invention calibrates arrangement information of an index attached to an imaging apparatus with respect to the imaging apparatus.

  In recent years, research on mixed reality has been actively conducted for the purpose of seamless connection between the real space and the virtual space. An image display device that presents a mixed reality is drawn on a virtual space (for example, computer graphics) generated in accordance with the position and orientation of an imaging device on an image of a real space that is captured by an imaging device such as a video camera. This is realized by a video see-through method in which an image of a virtual object, character information, etc.) is overlaid and displayed.

  Applications of such image display devices include virtual reality, such as surgical support that superimposes the state of the body on the patient's body surface, and mixed reality games that fight against virtual enemies floating in real space. Different new fields are expected.

  A common requirement for these applications is how to accurately align the real space and the virtual space, and many efforts have been made in the past. The alignment problem in mixed reality results in the problem of determining the position and orientation of the imaging device in the scene (ie, in the world coordinate system).

  As a method of solving this problem, by arranging or setting a plurality of indicators in the scene, using the coordinates in the world coordinate system of the indicators and the coordinates of the projected image of the indicators in the image captured by the imaging device, Generally, the position and orientation of the imaging device in a scene are obtained.

  A method for calculating the position and orientation of an imaging device from a set of coordinates in a world coordinate system of an index in a scene and image coordinates of a projected image has been proposed for a long time in the field of photogrammetry (for example, non-patent literature). 1, see Non-Patent Document 2).

  In addition, by setting a plurality of indices on the measurement target object, photographing the target object with an objective viewpoint camera installed outside, and detecting the image coordinates of the projected image of the index in the captured objective viewpoint image, The position and orientation are also obtained (see, for example, Non-Patent Document 2 and Patent Document 1).

Further, the inventors of the present invention also calculate the position and orientation of the imaging device by detecting the projected image of the index in the scene from the image captured by the imaging device itself that is the measurement target object in Japanese Patent Application No. 2003-323101. The method and the method for calculating the position and orientation of the imaging device by detecting the projected image of the index set on the imaging device itself from the objective viewpoint image obtained by imaging the imaging device that is the measurement target object from the objective viewpoint position Thus, a method for measuring the position and orientation of the imaging apparatus has been proposed.
JP 09-323280 A R. M.M. Haralick, C.I. Lee, K.M. Ottenberg, and M.M. Nole: Review and analysis of solutions of the three point perspective positive estimation lab, International Journal of Computer Vision, vol. 13, no. 3, pp. 331-356, 1994. D. G. Low: Fitting parametrized three-dimensional models to images, IEEE Transactions on PAMI, vol. 13, no. 5, pp. 441-450, 1991. Uchiyama, Yamamoto, Tamura, Hybrid Registration Method for Mixed Reality -Combination of 6 DOF Sensor and Vision Method-, Transactions of the Virtual Reality Society of Japan, vol. 8, no. 1, pp. 119-125, 2003. Fujii, Kanbara, Iwasa, Takemura, Yokoya, Positioning with stereo camera combined with gyro sensor for augmented reality, IEICE Technical Report PRMU99-192 (Science Tech. Vol.99, no. 574, pp. .1-8), 1999. A. I. Comfort, E.M. Marchand, F.M. Chaumette, A real-time tracker for markerless augmented reality, Proc. Int'l Symp. on Mixed and Augmented Reality 2004, pp. 36-45, 2004.

  By the way, in the method of calculating the position and orientation of the target object by detecting the image coordinates of the projected image of the index in the objective viewpoint image, the relative of the plurality of indices set to the measurement target object and the measurement target object The correct positional relationship must be known. When the relative positional relationship of each index is unknown, for example, when using the colored spherical index or the circular index and using the center of gravity of the projected image of the index on the image as a feature In addition, in the case where an index whose feature is described by one point is individually installed on the measurement target object, the three-dimensional position of each index in the measurement target object coordinate system is not measured in advance. Must not. On the other hand, when the relative positional relationship between the indices is known, for example, even when the indices similar to those described above are used, these indices are mounted on a base jig in advance. The position of each index has already been measured in this coordinate system (the coordinate system for describing the relative position of the index is referred to as the index coordinate system in the following), and this jig is attached to the object to be measured In such a case, the position and orientation of the jig in the measurement target object coordinate system need only be known (in addition, an index whose feature is described by a plurality of points, for example, a known shape If each vertex of the projected image of the index on the image is used as a feature using a square or triangle index, the index itself is interpreted as a collection of a plurality of indices, "When the positional relationship is known" To be considered as). However, a method for accurately calibrating them is not generally known, and until now there has been no measure other than using an inaccurate position or position and orientation obtained by manual measurement as a known value. As a result, in the method for calculating the position and orientation of the target object by detecting the image coordinates of the projected image of the index in the objective viewpoint image, the position and orientation of the target object can be measured only with practically low accuracy. There was room for improvement.

  The present invention has been made in view of the above problems, and an object of the present invention is to enable easy and accurate acquisition of index placement information with respect to an imaging apparatus.

To achieve the above object, the invention according to claim 1 is an index calibration device for correcting the location information for the main view view imaging apparatus of the plurality of objective viewpoint index attached to the subjective-view imaging apparatus, the subjective-view Objective viewpoint imaging means for imaging an objective viewpoint image including a plurality of objective viewpoint indices attached to the imaging apparatus from a known objective viewpoint position and orientation fixed in the world coordinate system, and from the subjective viewpoint imaging apparatus to the real space a subjective-view image input means for inputting the person view image position in arranged the world coordinate system comprises a plurality of subjective-view-indices are known, the objective viewpoint image input for inputting the objective viewpoint image from the objective viewpoint image pickup means It means a person view marker detecting means for detecting each of the image coordinates of the plurality of subjective-view-indices from the subjective-view image, the image coordinates of the plurality of objective-view-index before And an objective-view-index detection means for detecting from an objective viewpoint image, and initial value input means for inputting an initial value of the allocation information of each of the plurality of objective viewpoint index to the subjective-view imaging apparatus, detected by the subjective-view-index detection unit Position and orientation calculation means for calculating the subjective viewpoint position and orientation of the subjective viewpoint imaging device based on the image coordinates of each of the plurality of subjective viewpoint indicators and the position of the subjective viewpoint indicator in the world coordinate system; Based on the subjective viewpoint position / posture, the objective viewpoint position / posture, the initial value, and camera parameters specific to the objective viewpoint imaging means stored in advance, each of the plurality of objective viewpoint indices in the objective viewpoint image A theoretical value calculating means for calculating a theoretical value of image coordinates, image coordinates of each of the plurality of objective viewpoint indices detected from the objective viewpoint image, and the objective viewpoint finger On the basis of the error between the theoretical value of the image coordinates, the correction value calculating means for calculating a correction value of the placement information for the main view view imaging apparatus of a plurality of calibrated objective viewpoint index, based on the correction value, a plurality of Correction means for correcting arrangement information of the objective viewpoint index with respect to the subjective viewpoint imaging apparatus .
In order to achieve the above object, the invention according to claim 2 is an index calibration apparatus that corrects arrangement information of a plurality of objective viewpoint indices attached to a subjective viewpoint imaging apparatus with respect to the subjective viewpoint imaging apparatus, Objective viewpoint imaging means for imaging an objective viewpoint image including a plurality of objective viewpoint indices attached to the subjective viewpoint imaging apparatus from a known objective viewpoint position and orientation fixed in the world coordinate system, and realization from the subjective viewpoint imaging apparatus Subjective viewpoint image input means for inputting a subjective viewpoint image including a plurality of subjective viewpoint indices whose positions in the world coordinate system arranged in space are known; and an objective viewpoint for inputting the objective viewpoint image from the objective viewpoint imaging means Image input means, subjective viewpoint index detection means for detecting image coordinates of each of the plurality of subjective viewpoint indices from the subjective viewpoint image, and image locations of the plurality of objective viewpoint indices Objective viewpoint index detecting means for detecting the objective viewpoint index from the objective viewpoint image, initial value input means for inputting initial values of arrangement information of the plurality of objective viewpoint indices for the subjective viewpoint imaging device, and the objective viewpoint index detecting means Position / orientation calculation means for calculating a relative position / orientation between the objective viewpoint imaging device and the objective viewpoint index based on the image coordinates of each of the plurality of objective viewpoint indices detected in step, and the objective viewpoint position / orientation Relative position and orientation of the objective viewpoint imaging device and the objective viewpoint index, the initial value, camera parameters specific to the objective viewpoint imaging means stored in advance, and the world coordinates of the subjective viewpoint index A theoretical value calculating means for calculating a theoretical value of image coordinates of each of the plurality of subjective viewpoint indices in the subjective viewpoint image based on a position in the system; Based on an error between each detected image coordinate of the plurality of subjective viewpoint indices and a theoretical value of the image coordinate of the subjective viewpoint index, a correction value of the arrangement information of the plurality of calibration target objective viewpoint indices with respect to the subjective viewpoint imaging device And a correction means for correcting the arrangement information of a plurality of objective viewpoint indices with respect to the subjective viewpoint imaging device based on the correction values.

  According to the present invention, it is possible to easily and accurately acquire index placement information with respect to an imaging apparatus.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
The index calibration apparatus according to the present embodiment acquires the position of the index attached to the imaging apparatus with respect to the imaging apparatus for each index. Hereinafter, the index calibration apparatus and index calibration direction according to the present embodiment will be described.

  FIG. 1 is a diagram illustrating a schematic configuration of an index calibration apparatus 100 according to the present embodiment. As shown in FIG. 1, the index calibration apparatus 100 according to the present embodiment includes a subjective viewpoint index detection unit 110, a position / orientation calculation unit 120, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, data management. The unit 170 and the calibration information calculation unit 180 are connected to the imaging device 130 that is a calibration target.

On the imaging device 130, an index P ^k (k = 1,, K ₂ ) (hereinafter referred to as an objective viewpoint) to be calibrated by a user who uses the index calibration apparatus according to the present embodiment. It is assumed that at least one (referred to as an index) is set. The position of the objective viewpoint index in the subjective viewpoint camera coordinate system (a coordinate system in which one point on the imaging device 130 is defined as the origin and three orthogonal axes are defined as the X axis, the Y axis, and the Z axis, respectively) is unknown. The position of the objective viewpoint index in the unknown subjective viewpoint camera coordinate system is information that is calibrated by the index calibration apparatus according to the present embodiment, that is, the output of the index calibration apparatus according to the present embodiment.

At a plurality of positions in the real space, the world coordinate system (one point in the real space is defined as the origin as an index for photographing with the imaging device 130 (hereinafter referred to as a subjective viewpoint index (reference index)) and orthogonal to each other. A plurality of subjective viewpoint indices Q ^k (k = 1,, K ₁ ) whose positions in the three axes are defined in the coordinate system defined as the X axis, the Y axis, and the Z axis, respectively, are arranged. It is desirable that the subjective viewpoint index Q ^k is installed so that at least three or more indices are observed by the imaging device 130 when acquiring the data for index calibration. In the example of FIG. 1, four subjective viewpoint indices Q ¹ , Q ² , Q ³ , and Q ⁴ are arranged, and three of them, Q ¹ , Q ³ , and Q ^4, are included in the field of view of the imaging device 130. Shows the situation.

The subjective viewpoint index Q ^k may be constituted by, for example, circular markers each having a different color, or may be constituted by feature points such as natural features each having a different texture feature. It is also possible to use a square index formed by a square area having a certain area. Any form may be used as long as the image coordinates of the projected image on the photographed image can be detected and the index can be identified.

  An image output from the imaging device 130 (hereinafter referred to as a subjective viewpoint image) is input to the subjective viewpoint index detection unit 110.

Person view marker detecting unit 110 inputs the subjective-view image from the imaging device 130, detects the image coordinates of the subjective-view-indices Q ^k has been taken in the input image. For example, when each of the subjective viewpoint indices Q ^k is composed of markers having different colors, an area corresponding to each marker color is detected from the subjective viewpoint image, and the center of gravity position is detected as the detected coordinate of the index. To do. Further, when each of the subjective viewpoint indices Q ^k is composed of feature points having different texture characteristics, template matching based on the template image of each index held in advance as known information is performed on the subjective viewpoint image. By applying, the position of the index is detected. When a quadratic index is used as the reference index, labeling is performed after binarizing the image, and an area formed by four straight lines is detected as an index candidate from a region having a certain area or more. Further, by detecting whether or not there is a specific pattern in the candidate area, false detection is eliminated, and the direction and identifier of the index are acquired. In the present specification, the quadrangular indices detected in this way are considered to be four indices formed by four vertices.

Furthermore, subjective-view-index detection unit 110 outputs the image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} each detected and the identifier _{k n} to the position and orientation calculation unit 120. Here, n (n = 1,..., N) is an index for each detected index, and N represents the total number of detected indices. For example, in the case of FIG. 1, N = 3, identifiers k ₁ = 1, k ₂ = 3, k ₃ = 4 and the corresponding image coordinates u ^Qk1 , u ^Qk2 , u ^Qk3 are output. The subjective viewpoint index detection unit 110 according to the present embodiment sequentially executes the index detection process triggered by the input of an image from the imaging device 130, but in response to a request from the position / orientation calculation unit 120 (at that time) The process may be executed using the subjective viewpoint image input in (1).

The position / orientation calculation unit 120 is based on the correspondence relationship between the detected image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn and the world coordinates x _W ^Qkn of the indices previously stored as known information. 130 positions and orientations are calculated. A method for calculating the position and orientation of an imaging device from a set of world coordinates and image coordinates of a subjective viewpoint index has been proposed for a long time in the field of photogrammetry and the like (for example, see Non-Patent Document 1 and Non-Patent Document 2). ). For example, when the subjective viewpoint indices are arranged on the same plane, the position and orientation of the imaging device can be obtained based on the calculation of two-dimensional homography by detecting four or more indices. In addition, a method of using the position and orientation of an imaging device using six or more indices that are not on the same plane is also well known. In addition, the error between the theoretical value of the index image coordinates calculated from the estimated position and orientation of the imaging device and the actual measurement value of the index image coordinates is calculated by repeated calculation such as the Gauss-Newton method using the image Jacobian. By minimizing, a method for optimizing the estimated values of the position and orientation can be used. The calculated position and orientation of the imaging device 130 in the world coordinate system are output to the data management unit 170 in accordance with a request from the data management unit 170. Note that the position / orientation calculation unit 120 in this embodiment sequentially executes the above-described position and orientation calculation processing triggered by the input of data from the subjective viewpoint index detection unit 110, but the request from the data management unit 170 It is also possible to adopt a configuration in which processing is executed in accordance with (using input data at that time). In the following, the position and orientation of the imaging device 130 are held by a 4 × 4 homogeneous coordinate matrix M _WC (modeling conversion matrix; a matrix that converts coordinates in the subjective viewpoint camera coordinate system into coordinates in the world coordinate system). Shall.

  The objective viewpoint camera 140 is fixedly arranged at a position where the imaging device 130 for acquiring the data for index calibration can be imaged. It is assumed that the position and orientation of the objective viewpoint camera 140 in the world coordinate system are held in advance in the calibration information calculation unit 180 as known values.

The objective viewpoint index detection unit 150 inputs an image (objective viewpoint image) captured by the objective viewpoint camera 140, and an image of the objective viewpoint index ^Pk captured in the image by the same processing as the subjective viewpoint index detection unit 110. detecting the coordinates, and outputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected according to the requirements of the data management unit 170 and the identifier _{k m} to the data management unit 170. Here, m (m = 1,..., M) is an index attached to each detected index, and M represents the total number of detected indices. For example, in the case of FIG. 1, M = 2, identifiers k ₁ = 1, k ₂ = 2 and image coordinates u ^Pk1 and u ^Pk2 corresponding to these are output. The objective viewpoint index detection unit 150 in the present embodiment sequentially executes the above-described index detection processing triggered by the input of the objective viewpoint image, but in response to a request from the data management unit 170 (input at that time) The processing may be performed using an objective viewpoint image.

  The instruction unit 160 sends a “data acquisition” instruction to the data management unit 170 when a data acquisition command is input from an operator (not shown) and a “calibration information calculation” instruction when the calibration information calculation command is input. Transmit to the calculation unit 180. The command input to the instruction unit 160 can be performed by pressing a key to which a specific command is assigned, for example, using a keyboard. In addition, the command may be input by any method such as a GUI displayed on the display.

  When receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 170 inputs the position and orientation of the imaging apparatus 130 in the world coordinate system from the position / orientation calculation unit 120, and receives an objective viewpoint from the objective viewpoint index detection unit 150. The image coordinate of the index and its identifier are input, and the set of [position and orientation of the imaging device 130-image coordinate of the objective viewpoint index] is added to the data list prepared for each identifier of the objective viewpoint index and is retained. Here, the position and orientation of the imaging device 130 input from the position / orientation calculation unit 120 are the same time as the image capturing time when the image coordinates of the objective viewpoint index input from the objective viewpoint index detection unit 150 are detected. It is. Further, the data management unit 170 outputs a data list for each identifier of the generated objective viewpoint index to the calibration information calculation unit 180 in accordance with a request from the calibration information calculation unit 180.

  Upon receiving an instruction “calculation information calculation” from the instruction unit 160, the calibration information calculation unit 180 inputs a data list from the data management unit 170, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position of each objective viewpoint index in the subjective viewpoint camera coordinate system) is output.

  FIG. 2 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  In step S2010, the instruction unit 160 determines whether or not a data acquisition command has been input from the operator. The operator inputs a data acquisition command when the imaging device 130 is placed at a position where data for index calibration is acquired. If the data acquisition command is input, the instruction unit 160 shifts the process to step S2020.

In step S2020, the data management unit 170 inputs the position and orientation M _WC of the imaging apparatus 130 in the world coordinate system from the position / orientation calculation unit 120.

In step S2030, the data management unit 170, the objective-view-index detection unit 150, and inputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected by an objective-view-index detection unit 150 and the identifier _{k m.} By the above processing of steps S2020, S2030, the position and orientation of the imaging device can obtain an image coordinate of the objective-view-index when an _{M WC.} Note that the information input from the objective viewpoint index detection unit 150 is not necessarily related to all objective viewpoint indices, and may be information regarding the index detected on the objective viewpoint image at that time.

Next, in step S2040, the data management unit 170 adds the input data set to the data list L ^Pk for each detected objective viewpoint index P ^km . Specifically, the M _WC input from the position / orientation calculation unit 120 is M _WCi , the u ^Pkm input from the objective viewpoint index detection unit 150 is u _i ^Pk , and the set of [M _WCi , u _i ^Pk ] is objective. while sorting by referring to the identifiers k _m input from the view-index detection unit 150, and registers in the data list L ^Pk on objective view-index P ^k as i-th data about P ^k. Here, i (i = 1,..., I ^Pk ) is an index for each set of data registered in the data list L ^Pk , and I ^Pk represents the total number of sets of data registered for the objective viewpoint index P ^k. ing.

  Data is acquired by the above processing.

In step S2050, the data list related to all objective viewpoint indexes or the data list related to at least one objective viewpoint index acquired so far by the data management unit 170 is sufficient to calculate calibration information. A determination is made as to whether it has. If at least one data list or all data lists do not satisfy the condition, the process returns to step S2010 and waits for the input of a data acquisition command. On the other hand, if all the data lists or at least one data list satisfies the condition for calculating the calibration information, the process proceeds to step S2060. As a condition that a data list related to an objective viewpoint index P ^k has enough information to calculate calibration information, for example, two or more sets of data [M _WCi , u _i ^Pk ] having different data lists L ^Pk are obtained. It can be mentioned. As will be described later, two equations are obtained from one set of data (see Equation 3). If two or more sets of data are obtained, the subjective viewpoint camera coordinates of the objective viewpoint index, which is three parameters from the four equations, are obtained. The position in the system can be determined. However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S2060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S2070. If not input, the process returns to step S2010 and waits for the input of the data acquisition command.

The calibration information calculation unit 180 handles the calibration information to be obtained, that is, the position of the objective viewpoint index in the subjective viewpoint camera coordinate system as a ternary vector [x _C y _C z _C ] ^T. Hereinafter, this unknown parameter is described as a state vector s ^Pk = [x _C ^Pky y _C ^Pk z _C ^Pk ] ^T. Hereinafter, the description will be continued assuming a process for a certain objective viewpoint index ^Pk, but each process is performed in common for all objective viewpoint indices corresponding to the data list that satisfies the conditions for calculating the calibration information. .

In step S2070, the calibration information calculation unit 180 gives an appropriate initial value (for example, [000] ^T ) to the state vector s ^Pk .

In step S2080, the calibration information calculation unit 180 calculates all _{i values from} each data [M _WCi , u _i ^Pk ] (i = 1, 2, _... , I ^Pk ) and the state vector s ^{Pk in} the data list L ^Pk. against an objective-view-index ^{P k} of the objective viewpoint image coordinates of theory ^{_{u i Pk '= [u xi}} Pk', u yi Pk '] is calculated. Here, the theoretical value of the objective viewpoint image coordinates of the objective viewpoint index refers to data of the position (coordinates) that should be visible in the objective viewpoint image of the objective viewpoint index P ^k given the position in the subjective viewpoint camera coordinate system. . The calculation of u _i ^Pk ′ is a function of the state vector s ^Pk representing the position of the objective viewpoint index P ^{k in} the subjective viewpoint camera coordinate system.

に基づいて行われる。

Based on.

Specifically, the function F _i () is obtained from the objective viewpoint camera coordinates of the objective viewpoint index P ^k when the i-th data is acquired (that is, when the position and orientation of the imaging device 130 are M _WCi ). The following equation for ^{obtaining the} position vector x _Bi ^Pk from s ^Pk :

And the following equation for ^obtaining the coordinates u _i ^Pk ′ on the objective viewpoint image of the objective viewpoint index P ^k from x _Bi ^Pk ,

It is constituted by. Here, f ^B _x and f ^B _y are focal lengths of the objective viewpoint camera 140 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values. _MWB is a transformation matrix for converting coordinates in the objective viewpoint camera coordinate system into world coordinates, and is calculated in advance based on the position and orientation of the objective viewpoint camera 140 in the world coordinate system that is held in advance as a known value. It is assumed that

In step S2090, the calibration information calculating unit 180, for all i, the actual image coordinates _u ^{i Pk} objective-view-index ^{P k} included in each data in the data list ^{L Pk,} the image coordinates of the corresponding an error △ _u ^{i Pk} of the theoretical value _u ^{i Pk} 'is calculated by the following equation.

In step S2100, the calibration information calculating unit 180, for all i, the image Jacobian about the state vector ^{s Pk} (i.e., solutions obtained by partially differentiating the function of Equation 1 _F i () for each element of the state vector ^{s Pk} Jacobian matrix of 2 rows × 3 columns having, in each _{^{element) J uis Pk (= ∂u i}} Pk / ∂s Pk) is calculated. Specifically, a 2-row × 3-column Jacobian matrix J _uxBi ^Pk (= ∂u) having a solution obtained by partial differentiation of the right side of Expression 3 with each element of the position vector x _Bi ^Pk on the objective viewpoint camera coordinate system. _i ^Pk / ∂x _Bi ^Pk ) and a 3 × 3 Jacobian matrix J _xBis ^Pk (= ∂x _Bi ^Pk /) having a solution obtained by partial differentiation of the right side of Equation 2 with each element of the state vector s ^Pk. ∂s ^Pk), and calculates the _J ^{uis Pk} by the following equation.

In step S2110, the calibration information calculating unit 180 is calculated in the above step, based on the error △ _u ^{i Pk} and the Jacobian matrix _J ^{uis Pk} for all ^i, and calculates the correction value △ ^{s Pk} of ^{s Pk.} Specifically, an error vector of a 2I ^Pk dimensional vector in which errors Δu _i ^Pk for all i are arranged vertically

And a Jacobian matrix _Juis ^Pk vertically arranged in 2I ^Pk rows × 3 columns matrix

And ^{Δs Pk} is calculated from the following equation using the pseudo inverse matrix Φ ⁺ of Φ.

Here, since Δs ^Pk is a three-dimensional vector, if 2I ^Pk is 3 or more, that is, I ^Pk is 2 or more, ^{Δs Pk} can be obtained. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

In step S2120, the calibration information calculation unit 180 uses the correction value Δs ^Pk calculated in step S2110 to correct the position vector s ^Pk in the subjective viewpoint camera coordinate system of the objective viewpoint index P _k according to Equation 11 to obtain The new value is set as a new ^sPk .

In step S2130, the calibration information calculation unit 180 uses some criteria such as whether the error vector U is smaller than a predetermined threshold or whether the correction value Δs ^Pk is smaller than a predetermined threshold. It is determined whether or not it has converged. If not converged, the processing after step S2080 is performed again using the corrected state vector s ^Pk .

The calculation is determined to have converged in step S2130, in step S2140, the calibration information calculating unit 180, the state vector s ^Pk obtained, as a parameter indicating the position in the subjective-view camera coordinate system of the objective-view-index P ^k Output.

  Finally, in step S2150, it is determined whether or not to end the calibration process. When the operator instructs the index calibration apparatus 100 to end the calibration process, the process is terminated. When the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S2010 again, and the data Wait for input of get command.

  Through the above processing, the position of the index attached to the imaging apparatus with respect to the imaging apparatus can be acquired easily and accurately.

<Modification 1-1>
In the present embodiment, the steepest descent method expressed by Equation 10 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Needless to say, the essence of the present invention is not impaired by any numerical calculation method. In this embodiment, in the processing from step S2070 to step S2130, an initial value is given to the position of the index to be obtained, and the optimum value is obtained by iterative calculation using the image Jacobian. However, the index is calculated by a simpler calculation method. It is also possible to obtain the position of. For example, if Equation 3 is expanded,

The relationship is obtained. Using this relationship,

Thus, s ^Pk can be obtained directly from the data list L ^Pk = [M _WCi , u _i ^Pk ] (i = 1,, I ^Pk ). here,

Represents.

<Modification 1-2>
Further, in the present embodiment, the position / orientation calculation unit 120 includes the image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn detected from the captured image of the imaging device 130 and the world of indices previously stored as known information. The position and orientation of the imaging device 130 are calculated based on the correspondence with the coordinates x _W ^Qkn . However, the image features used to obtain the position and orientation of the imaging device need to be represented by dots. There is no. For example, a method for obtaining the position and orientation of an imaging apparatus using line features as shown in Non-Patent Document 2, and an imaging apparatus using geometric features such as an ellipse as shown in Non-Patent Document 5 The subjective viewpoint index detection unit 110 and the position / orientation calculation unit 120 may be realized by a method for obtaining the position and orientation of the camera, or may be a combination of these, or a camera using any other image feature A position and orientation estimation method may be used. In addition, a 6-DOF position and orientation sensor such as a magnetic sensor and a 3-DOF orientation sensor such as a gyro sensor are attached to the imaging device 130, and image characteristics detected from the captured image of the imaging device 130 and known information The position and orientation of the imaging device 130 may be calculated by a hybrid method based on both the correspondence relationship between the index held in advance and the world coordinates and the measurement value of the sensor attached to the imaging value 130. (For specific calculation methods, see, for example, Non-Patent Document 3 and Non-Patent Document 4).

[Second Embodiment]
In the index calibration apparatus according to the present embodiment, the index coordinates for the imaging apparatus when a plurality of indices mounted on the imaging apparatus have an index coordinate system and the coordinates of each index in the index coordinate system are known. The position and orientation of the system, or the position of each index with respect to the imaging device is acquired. Hereinafter, the index calibration apparatus and index calibration direction according to the present embodiment will be described.

  FIG. 3 is a diagram illustrating a schematic configuration of the index calibration apparatus according to the present embodiment. The same parts as those in FIG. 1 are given the same numbers and symbols, and detailed description thereof will be omitted. As shown in FIG. 3, the index calibration apparatus 300 according to the present embodiment includes a subjective viewpoint index detection unit 110, a position / orientation calculation unit 120, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, data management. The unit 370 and the calibration information calculation unit 380 are connected to the imaging device 130 that is a calibration target.

On the imaging device 130, a plurality of indices P ^k (k = 1,, K ₂ ) (first implementation) to be calibrated by a user who uses the index calibration apparatus according to this embodiment. In the following, it is assumed that, similarly to the form, this is referred to as an objective viewpoint index). As in the first embodiment, the position of the objective viewpoint index in the subjective viewpoint camera coordinate system is unknown, but in this embodiment, the relative positional relationship between the indices is known. For example, a case where a set of four objective viewpoint indices P ^k (k = 1, 2, 3, 4) forms one square index R ¹ having a known size is equivalent. FIG. 3 shows the situation. At this time, one point on the square indices R ¹ is defined as an origin, further triaxial respective X-axis orthogonal to each other, Y-axis and a coordinate system that is defined as Z-axis called an index coordinate system, index coordinate system The coordinates of each objective viewpoint index P ^k in are known. Therefore, if the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system (= conversion matrix for converting the coordinates in the index coordinate system to the subjective viewpoint camera coordinates) is found, the subjective viewpoint of each objective viewpoint index P ^k that is unknown is known. The position in the viewpoint camera coordinate system can be calculated. From this, it is assumed that the information calibrated by the index calibration device according to the present embodiment is the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system.

The plurality of positions in real space, as in the first embodiment, as an index for taking by the imaging apparatus 130, the position is the subjective-view-indices Q ^k is known is set in the world coordinate system.

Person view marker detecting unit 110, like the first embodiment, enter the subjective-view image capturing apparatus 130 is captured, it detects the image coordinates of the subjective-view-indices Q ^k has been taken in an image, detected image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} to output the identifier _{k n} to the position and orientation calculation unit 120.

Similar to the first embodiment, the position / orientation calculation unit 120 includes the detected image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn , and the world coordinates x _W ^{Qkn of the} indices previously stored as known information. The position and orientation of the imaging device 130 are calculated based on the corresponding relationship. This calculation method is as described in the portion explaining the first embodiment. The calculated position and orientation of the imaging device 130 in the world coordinate system are output to the data management unit 370 in accordance with a request from the data management unit 370.

  Similar to the first embodiment, the objective viewpoint camera 140 is fixedly disposed at a position where the image capturing apparatus 130 when capturing the data for index calibration can be imaged. It is assumed that the position and orientation of the objective viewpoint camera 140 in the world coordinate system are held in advance in the calibration information calculation unit 380 as known values.

The objective viewpoint index detection unit 150 receives the objective viewpoint image captured by the objective viewpoint camera 140, as in the first embodiment, and is objectively captured in the image by the same processing as the subjective viewpoint index detection unit 110. detecting the image coordinates of the viewpoint index ^{P k,} and outputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected according to the requirements of the data management unit 370 and the identifier _{k m} to the data management unit 370. In the example of FIG. 3, since a square index is used as the index, the square index detection processing described in the first embodiment is executed.

  In the same way as in the first embodiment, the instruction unit 160 gives a “data acquisition” instruction to the data management unit 370 when a data acquisition command is input from an operator (not shown), and “ An instruction “calibration information calculation” is transmitted to the calibration information calculation unit 380.

  Upon receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 370 inputs the position and orientation in the world coordinate system of the imaging device 130 from the position / orientation calculation unit 120, and receives an objective viewpoint from the objective viewpoint index detection unit 150. The image coordinate of the index and its identifier are input, and a set of [position and orientation of the imaging device 130 -image coordinate of the objective viewpoint index -identifier of the objective viewpoint index] is added to one data list and held. Here, the position and orientation of the imaging device 130 input from the position / orientation calculation unit 120 are the same time as the image capturing time when the image coordinates of the objective viewpoint index input from the objective viewpoint index detection unit 150 are detected. It is. In addition, the data management unit 370 outputs the generated data list to the calibration information calculation unit 380 in accordance with a request from the calibration information calculation unit 380.

  When the calibration information calculation unit 380 receives an instruction of “calibration information calculation” from the instruction unit 160, the calibration information calculation unit 380 inputs a data list from the data management unit 370, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system) is output.

  FIG. 4 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The same parts as those in FIG. 2 are given the same numbers and symbols, and detailed description thereof is omitted. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  From step S2010 to step S2030, processing similar to that of the first embodiment is performed. When step S2030 is completed, the process proceeds to step S4040.

Next, in step S4040, the data management unit 370 adds the input data set as data D _j to the data list L for all the detected objective viewpoint indices P ^km . Specifically, the _{M WC} input from the position and orientation calculation unit 120 as _{M WCJ,} the identifier _{k m} input from the objective-view-index detection unit 150 and _{k j,} the ^{u Pkm} similarly input from an objective-view-index detection unit 150 As u _j ^Pkj , a set of D _j = [M _WCj , u _j ^Pkj , k _j ] is registered in the data list L as the j-th data. Here, j (j = 1,..., J) is an index for each data set registered in the data list L, and J represents the total number of registered data sets.

  Data is acquired by the above processing.

In step S4050, the data management unit 370 determines whether the data list acquired so far has enough information to calculate calibration information. If the condition is not satisfied, the process returns to step S2010 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the conditions for calculating calibration information, the process proceeds to step S2060. The condition that the data list has sufficient information to calculate the calibration information includes, for example, that the data list L includes data relating to at least three different objective viewpoint indices P ^k. . However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S2060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S4070. If not input, the process returns to step S2010 and waits for the input of the data acquisition command.

The calibration information calculation unit 380 handles the calibration information to be obtained, that is, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system as a 6-value vector [xyzξψζ] ^T. Hereinafter, this unknown parameter is described as a state vector s = [xyzξψζ] ^T.

In step S4070, the calibration information calculation unit 380 gives an appropriate initial value to the state vector s. As an initial value, an operator may manually input an approximate value via the instruction unit 160, or a plurality of objective viewpoint indices input at a certain same time (that is, detected from the same objective viewpoint image). Are extracted from the list L, and using this data, the position and orientation of the index coordinate system in the objective viewpoint coordinate system at the same time are calculated by a known method, and the obtained position and orientation are used in the index coordinate system. A transformation matrix M _BM for transforming the coordinates into the objective viewpoint camera coordinate system is obtained, and further, using the position and orientation M _WC of the imaging device input at the same time held in the list L, to obtain a conversion matrix M _CM representing the position and orientation of the index-based coordinates in the subjective-view camera coordinate system, it may be using the position and orientation represented by this matrix as an initial value of s.

Here, _MWB is a transformation matrix that converts coordinates in the objective viewpoint camera coordinate system into world coordinates, and is based on the position and orientation of the objective viewpoint camera 140 in the world coordinate system that is held in advance as a known value. It is assumed that it has been calculated in advance.

  As a known method for calculating the position and orientation of the index coordinate system in the objective viewpoint coordinate system, for example, when the indices are arranged on the same plane, two-dimensional homology is used by using four or more indices. It is possible to use a technique for obtaining the position and orientation of the index coordinate system based on the calculation of the graphic. It is also possible to use a technique that uses six or more indices that are not on the same plane, or a technique that uses these solutions as initial values and obtains an optimal solution by iterative calculations such as the Gauss-Newton method.

In step S4080, the calibration information calculation unit 380 determines that each data D _j = [M _WCj , u _j ^Pkj , k _j ] (j = 1, 2, ^... J) in the data list L and the state vector s are The theoretical value u _j ^Pkj ′ = [u _xj ^Pkj ′, u _yj ^Pkj ′] of the objective viewpoint image coordinates of each objective viewpoint index P ^{kj is} calculated for all j. Here, the theoretical value of the objective viewpoint image coordinate of the objective viewpoint index ^means that the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system of the objective viewpoint index P ^kj whose position vector x _M ^Pkj in the index coordinate system is known is s. Is the data of the position (coordinates) that should be visible in the objective viewpoint image. The calculation of u _j ^Pkj ′ is a function of the state vector s representing the position of the index coordinate system in the subjective viewpoint camera coordinate system.

Based on.

Specifically, the function F _j () is obtained from the objective viewpoint camera coordinates of the objective viewpoint index P ^kj when the j-th data is acquired (that is, when the position and orientation of the imaging device 130 are M _WCj ). A position vector x _Bj ^Pkj is obtained from s as follows:

And the following equation for ^obtaining the coordinates u _j ^Pkj ′ on the objective viewpoint image of the objective viewpoint index P ^kj from x _Bj ^Pkj :

It is constituted by. Here, f ^B _x and f ^B _y are focal lengths of the objective viewpoint camera 140 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values. M _CM (s) is a modeling conversion matrix (matrix for converting coordinates in the index coordinate system to coordinates in the subjective viewpoint camera coordinate system) determined by s, and is defined by the following equation.

However,

It is.

In step S4090, the calibration information calculation unit 380 calculates the actual image coordinates u _j ^{Pkj of the} objective viewpoint index P ^kj included in each data in the data list L and the corresponding image coordinate theory for all j. error △ _u ^{j PKJ} the value _u ^{j PKJ} 'is calculated by the following equation.

In step S4100, the calibration information calculation unit 380 performs, for each _j, a solution obtained by partially differentiating the image Jacobian relating to the state vector s (that is, the function F _j () of Expression 17 by each element of the state vector s). 2 rows × 6 columns Jacobian matrix) J _uj ^Pkj _s (= ∂u _j ^Pkj / ∂s). Specifically, a 2-row × 3-column Jacobian matrix J _ujxBj ^Pkj (= 持_つ ^u) having a solution obtained by partial differentiation of the right side of Expression 19 with each element of the position vector x _Bj ^Pkj on the objective viewpoint camera coordinate system. _j ^Pkj / ∂x _Bj ^Pkj ) and a 3 × 6 Jacobian matrix J _xBj ^Pkj _s (= ∂x _Bj ^Pkj /) having a solution obtained by partial differentiation of the right side of Equation 18 with each element of the state vector s. ∂s) is calculated, and J _uj ^Pkj _s is calculated by the following equation.

In step S4110, the calibration information calculation unit 380 calculates the correction value ^{Δs of} _s based on the error ^Δu _j ^Pkj and the Jacobian matrix J _uj ^Pkj _s for all j calculated in the above steps. Specifically, an error vector of a 2J-dimensional vector in which errors ^Δu _j ^Pkj for all j are arranged vertically

And Jacobi matrix J _uj ^Pkj _s vertically arranged in 2J rows × 6 columns matrix

And Δs is calculated from the following equation using the pseudo inverse matrix Φ ⁺ of Φ.

Here, since Δs is a six-dimensional vector, Δs can be obtained when 2J is 6 or more, that is, J is 3 or more. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

  In step S4120, the calibration information calculation unit 380 uses the correction value Δs calculated in step S4110 to correct the state vector s representing the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system according to Equation 27, and obtain Let the obtained value be a new s.

  In step S4130, calibration information calculation section 380 converges the calculation using some criterion such as whether error vector U is smaller than a predetermined threshold or whether correction value Δs is smaller than a predetermined threshold. It is determined whether or not. If not converged, the processing after step S4080 is performed again using the corrected state vector s.

If it is determined in step S4130 that the calculation has converged, in step S4140, the calibration information calculation unit 380 uses the obtained state vector s as calibration information, that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system. Output as. The form of output at this time may be s itself, or the position component of s may be represented by a ternary vector, and the posture component may be represented by an Euler angle or a 3 × 3 rotation matrix. then, it may be a coordinate transformation matrix M _CM generated from s.

  Finally, in step S2150, it is determined whether or not to end the calibration process. If the operator instructs the index calibration device 300 to end the calibration process, the process is terminated. If the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S2010 again to acquire data. Wait for command input.

  Through the above processing, the position or position / orientation of the index attached to the imaging apparatus with respect to the imaging apparatus can be acquired easily and accurately.

<Modification 2-1>
In this embodiment, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system are output as the calibration information, but the objective viewpoint index P in the subjective viewpoint camera coordinate system is obtained from the obtained state vector s. The positions of ^k may be calculated individually and output as calibration information. In this case, the position of the objective viewpoint index P ^k in the subjective viewpoint camera coordinate system is obtained by the product of the coordinate transformation matrix M _CM obtained from s and the known position x _M ^Pk of the objective viewpoint index P ^k in the index coordinate system. be able to.

<Modification 2-2>
Further, in the present embodiment, the position / orientation calculation unit 120 includes the image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn detected from the captured image of the imaging device 130 and the world of indices previously stored as known information. The position and orientation of the imaging device 130 are calculated based on the correspondence relationship with the coordinate x _W ^Qkn , but the position and orientation using other image information is the same as <Modification 1-2> of the first embodiment. Calculation and position / orientation calculation using other sensors may also be performed.

<Modification 2-3>
In the present embodiment, the steepest descent method expressed by Expression 26 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Needless to say, the essence of the present invention is not impaired by any numerical calculation method. Further, in the present embodiment, a method for obtaining an optimal value of s for all input data by giving appropriate initial values to the position and orientation s of the index coordinate system in the subjective viewpoint camera coordinate system, and by iterative calculation using the image Jacobian. However, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system can also be obtained by using another simple calculation method. For example, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system are determined by using only the detection coordinates of a plurality of objective viewpoint indices on one objective viewpoint image by the processing procedure described in the description of step S4070. It may be obtained and output as calibration information.

<Modification 2-4>
In the present embodiment, a square index is used as the objective viewpoint index, but any index type may be used as long as the index group has a known position of each index in the index coordinate system. For example, it may be a set of a plurality of circular indicators as used in the first embodiment, or a plurality of types of indicators may be mixed. Further, even when a plurality of index coordinate systems are provided, the above processing is performed individually for each index coordinate system, or the above processing is performed in parallel for each index coordinate system. Can be calibrated as well.

[Third Embodiment]
In the index calibration device according to the present embodiment, as in the second embodiment, a plurality of indexes mounted on the imaging apparatus have an index coordinate system, and the coordinates of each index in the index coordinate system are known. In this case, the position and orientation of the index coordinate system with respect to the imaging apparatus or the position of each index with respect to the imaging apparatus is acquired. Hereinafter, the index calibration apparatus and index calibration direction according to the present embodiment will be described.

FIG. 5 is a diagram showing a schematic configuration of the index calibration device according to the present embodiment. The same parts as those in FIG. 3 are given the same numbers and symbols, and detailed description thereof is omitted. As shown in FIG. 5, the index calibration apparatus 500 according to the present embodiment includes a subjective viewpoint index detection unit 110, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, a position and orientation calculation unit 520, and data management. 570 and a calibration information calculation unit 580, which are connected to the imaging device 130 to be calibrated. It is assumed that a plurality of indices P ^k (k = 1,, K ₂ ) to be calibrated are set on the imaging device 130 as in the second embodiment.
The subjective viewpoint index Q ^k is set at a plurality of positions in the real space, as in the second embodiment, and the subjective viewpoint index detection unit 110, the objective viewpoint camera 140, the objective viewpoint index detection unit 150, Since the operation of the instruction unit 160 is the same as that of the second embodiment, detailed description thereof is omitted. However, the subjective viewpoint index detection unit 110 outputs the detection result to the data management unit 570 in response to a request from the data management unit 570, and the objective viewpoint index detection unit 150 outputs the detection result to the position / orientation calculation unit 520. However, this is different from the second embodiment. In the second embodiment, since the position and orientation of the imaging device 130 are obtained using the detection result of the subjective viewpoint index, the number of indexes necessary for calculating the position and orientation of the imaging device 130 is 1. Although it has been necessary to detect from one subjective viewpoint image, in the present embodiment, the number of indices detected from one subjective viewpoint image may be one or more.

The position / orientation calculation unit 520 receives the image coordinates ^uPkm of the objective viewpoint index P ^km and its identifier from the objective viewpoint index detection unit 150, and the three-dimensional coordinates of each index in the index coordinate system held in advance as known information. Based on the correspondence with x _M ^Pkm , the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system are calculated. Since this calculation method is the same as the processing described in step S4070 of the second embodiment, detailed description thereof is omitted. The calculated position and orientation are output to the data management unit 370 in accordance with a request from the data management unit 370. In the following, the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system are held by a 4 × 4 homogeneous coordinate matrix M _BM (matrix that converts coordinates in the index coordinate system into coordinates in the objective viewpoint camera coordinate system). It shall be.

  Upon receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 570 inputs the image coordinates of the subjective viewpoint index and its identifier from the subjective viewpoint index detection unit 110, and the objective viewpoint camera coordinates from the position / orientation calculation unit 520. The position and orientation of the index coordinate system in the system is input, and the pair of [position and orientation of the index coordinate system in the objective viewpoint camera coordinate system−image coordinates of the subjective viewpoint index−identifier of the subjective viewpoint index] is input. Created for each viewpoint index, added to one data list and retained. Further, the data management unit 570 outputs the generated data list to the calibration information calculation unit 580 in accordance with a request from the calibration information calculation unit 580.

  When the calibration information calculation unit 580 receives an instruction of “calibration information calculation” from the instruction unit 160, the calibration information calculation unit 580 inputs a data list from the data management unit 570, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system) is output.

  FIG. 6 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  In step S6010, the instruction unit 160 determines whether a data acquisition command has been input from the operator. The operator inputs a data acquisition command when the imaging device 130 is placed at a position where data for index calibration is acquired. If the data acquisition command is input, the instruction unit 160 shifts the process to step S6020.

In step S6020, the data management unit 570, from the position and orientation calculation unit 520 inputs the position and orientation _{M BM} index coordinate system in the objective view camera coordinate system.

In step S6030, the data management unit 570, from the subjective-view-index detection unit 110, and inputs the image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} of people each detected by the subjective-view-index detection unit 110 and the identifier _{k n.}

In step S6040, the data management unit 570 adds each input subjective viewpoint index Q ^kn to the data list L as data D _j . Specifically, the objective viewpoint camera coordinate position and orientation M _BM index coordinate system in system input from the position and orientation calculating section 520 and M _BMj, the identifier k _n input from the subjective-view-index detection unit 110 and k _j, Similarly, a group of D _j = [M _BMj , u _j ^Qkj , k _j ] is registered in the data list L as j-th data, where u ^Qkn inputted from the subjective viewpoint index detection unit 110 is u _j ^Qkj . Here, j (j = 1,..., J) is an index for each data set registered in the data list L, and J represents the total number of registered data sets.

  Data is acquired by the above processing.

In step S6050, the data management unit 570 determines whether the data list acquired so far has enough information to calculate calibration information. If the condition is not satisfied, the process returns to step S6010 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the conditions for calculating calibration information, the process proceeds to step S6060. The condition that the data list has sufficient information to calculate the calibration information includes, for example, that the data list L includes data relating to at least three different subjective viewpoint indices Q ^k. . However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S6060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S6070. If not, the process returns to step S6010 again to wait for the input of the data acquisition command.

As in the second embodiment, the calibration information calculation unit 580 handles the calibration information to be obtained, that is, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system as a six-value vector [xyzξψζ] ^T. Hereinafter, this unknown parameter is described as a state vector s = [xyzξψζ] ^T.

  In step S6070, the calibration information calculation unit 580 gives an appropriate initial value to the state vector s. As the initial value, for example, the operator manually inputs an approximate value via the instruction unit 160.

In step S6080, the calibration information calculation unit 580 determines each subjective viewpoint index Q for all j from the data D _j (j = 1, 2,..., J) and the state vector s in the data list L. The theoretical value u _j ^Qkj ′ = [u _xj ^Qkj ′, u _yj ^Qkj ′] of the subjective viewpoint image coordinates of ^kj is calculated. Here, the theoretical values of the subjective viewpoint image coordinates of the subjective viewpoint index are M _{BMj for} the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system, and s for the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system. Sometimes, the position x _W ^Qkj in the world coordinate system refers to the data of the position (coordinates) that should be visible in the subjective viewpoint image of the subjective viewpoint index Q ^kj . The calculation of u _j ^Qkj 'is a function of s

Based on.

Specifically, the function F _j () is the subjective viewpoint index Q ^kj when the j-th data is acquired (that is, when the position and orientation of the index coordinate system in the objective viewpoint camera coordinate system is M _BMj ). The position x _Cj ^Qkj in the subjective viewpoint camera coordinates of

And the following equation for ^obtaining the coordinates u _j ^Qkj ′ on the subjective viewpoint image of the subjective viewpoint index Q ^kj from x _Cj ^Qkj :

It is constituted by. Here, f ^C _x and f ^C _y are the focal lengths of the imaging device 130 in the x-axis direction and the y-axis direction, respectively, and M _WB is a transformation matrix that converts coordinates in the objective viewpoint camera coordinate system into world coordinates, respectively. It is assumed that this value is held in advance. M _CM (s) is a modeling conversion matrix (matrix for converting coordinates in the index coordinate system to coordinates in the subjective viewpoint camera coordinate system) determined by s, and is defined by Expression 20. Note that the product (M _CM (s) · M _BMj ⁻¹ · M _WB ⁻¹ ) of the right-hand side matrix in Expression 29 is the position and orientation (M _WC ⁻ ) of the imaging device 130 in the world coordinate system determined by s and M _BMj . ¹ ).

In step S6090, the calibration information calculation unit 580 calculates the actual image coordinates u _j ^{Qkj of the} subjective viewpoint index Q ^kj included in each data in the data list L and the corresponding image coordinate theory for all j. error △ _u ^{j Qkj} the value _u ^{j Qkj} 'is calculated by the following equation.

In step S6100, the calibration information calculation unit 580 performs, for each j, an element obtained by partially differentiating the image Jacobian relating to the state vector s (that is, the function F _j () of Equation 28 with respect to each element of the state vector s. (2 rows × 6 columns Jacobian matrix) J _uj ^Qkj _s (= ∂u _j ^Qkj / ∂s). Specifically, a 2-row × 3-column Jacobian matrix J _ujxCj ^Qkj (= ∂u) having a solution obtained by partial differentiation of the right side of Expression 30 with each element of the position vector x _Cj ^Qkj on the subjective viewpoint camera coordinate system. _j ^Qkj / ∂x _Cj ^Qkj ) and a 3 × 6 Jacobian matrix J _xCj ^Qkj _s (= ∂x _Cj ^Qkj /) each having a solution obtained by partial differentiation of the right side of 29 with each element of the state vector s s) and J _uj ^Qkj _s is calculated by the following equation.

In step S6110, the calibration information calculation unit 580 calculates the correction value ^{Δs of} _s based on the error ^Δu _j ^Qkj and the Jacobian matrix J _uj ^Qkj _s for all j calculated in the above steps. Specifically, an error vector U of 2J-dimensional vectors in which errors ^Δu _j ^Qkj for all j are arranged vertically and a Jacobian matrix J _uj ^Qkj _s are arranged in a matrix of 2J rows × 6 columns, Δs is calculated from Equation 26 using the pseudo inverse matrix Φ ⁺ of Φ. Here, since Δs is a six-dimensional vector, Δs can be obtained when 2J is 6 or more, that is, J is 3 or more. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

  In step S6120, calibration information calculation section 580 corrects state vector s according to equation 27 using correction value Δs calculated in step S6110, and sets the obtained value as new s.

  In step S6130, calibration information calculation section 580 converges the calculation using some criterion such as whether error vector U is smaller than a predetermined threshold or whether correction value Δs is smaller than a predetermined threshold. It is determined whether or not. If not converged, the processing after step S6080 is performed again using the corrected state vector s.

If it is determined in step S6130 that the calculation has converged, in step S6140, the calibration information calculation unit 580 uses the obtained state vector s as calibration information, that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system. Output as. The form of output at this time may be s itself, or the position component of s may be represented by a ternary vector, and the posture component may be represented by an Euler angle or a 3 × 3 rotation matrix. then, it may be a coordinate transformation matrix M _CM generated from s.

  Finally, in step S6150, it is determined whether or not to end the calibration process. When the operator instructs the index calibration device 500 to end the calibration process, the process is ended. When the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S6010 again to acquire data. Wait for command input.

  Through the above processing, the position or position / orientation of the index attached to the imaging apparatus with respect to the imaging apparatus can be acquired easily and accurately.

<Modification 3-1>
In this embodiment, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system are output as the calibration information, but the objective viewpoint index P in the subjective viewpoint camera coordinate system is obtained from the obtained state vector s. The positions of ^k may be calculated individually and output as calibration information.

<Modification 3-2>
In the present embodiment, the steepest descent method expressed by Expression 26 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Regardless of which numerical calculation method is applied, the essence of the present invention is not impaired.

<Modification 3-3>
In the present embodiment, a square index is used as the objective viewpoint index, but any index type may be used as long as the index group has a known position of each index in the index coordinate system. For example, it may be a set of a plurality of circular indicators as used in the first embodiment, or a plurality of types of indicators may be mixed. Further, even when a plurality of index coordinate systems are provided, the above processing is performed individually for each index coordinate system, or the above processing is performed in parallel for each index coordinate system. Can be calibrated as well.

<Modification 3-4>
In this embodiment, an index that gives a two-dimensional coordinate on an image is used as a subjective viewpoint index. However, line features and geometric features as disclosed in Non-Patent Document 2 and Non-Patent Document 5 are used. May be used as a reference index for evaluation. For example, when a line feature is used, the error vector U is constituted by an error Δd calculated from the detected value d from the image and the estimated value d ′ from the state vector s using the distance from the origin of the straight line as an error evaluation criterion. Then, the matrix Φ is constituted by a 1 row × 4 column Jacobian matrix J _ds (= ∂d / ∂s) having a solution obtained by partial differentiation of the calculation formula of d ′ with each element of the state vector s. The correction value can be calculated by the same framework as in the above embodiment. Note that the calculation formula of d ′ is disclosed as a function of the position and orientation of the imaging apparatus in Non-Patent Document 2, Non-Patent Document 5, and the like, and the position and orientation of the imaging apparatus is obtained as a function of s (Formula 29) Therefore, it can be easily obtained from these. In addition, it is possible to use these features together by stacking line features and point features, errors obtained from other indices, and image Jacobians.

[Other Embodiments]
An object of the present invention is to supply a storage medium (or recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus. Needless to say, this can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above (shown in FIGS. 2, 4, and / or 6).

It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 1st Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 1st Embodiment. It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 2nd Embodiment. It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 3rd Embodiment.

Claims (9)

An index calibration device for correcting the location information for the main view view imaging apparatus of the plurality of objective viewpoint index attached to the subjective-view imaging apparatus,
Objective viewpoint imaging means for capturing an objective viewpoint image including a plurality of objective viewpoint indices attached to the subjective viewpoint imaging device from an objective viewpoint position and orientation that is fixed and known in the world coordinate system ;
Subjective viewpoint image input means for inputting a subjective viewpoint image including a plurality of subjective viewpoint indices whose positions in the world coordinate system arranged in the real space are known from the subjective viewpoint imaging device;
And objective viewpoint image input means for inputting the objective viewpoint image from the objective-view imaging means,
Subjective viewpoint index detection means for detecting the image coordinates of each of the plurality of subjective viewpoint indices from the subjective viewpoint image;
Objective viewpoint index detection means for detecting image coordinates of the plurality of objective viewpoint indices from the objective viewpoint image;
An initial value input means for inputting an initial value of each arrangement information of the plurality of objective viewpoint indices with respect to the subjective viewpoint imaging device;
Based on the image coordinates of each of the plurality of subjective viewpoint indices detected by the subjective viewpoint index detecting means, and the position of the subjective viewpoint index in the world coordinate system, the subjective viewpoint position and orientation of the subjective viewpoint imaging device is calculated. Position and orientation calculation means for
Each of the plurality of objective viewpoint indices in the objective viewpoint image based on the subjective viewpoint position / orientation, the objective viewpoint position / orientation, the initial value, and camera parameters specific to the objective viewpoint imaging means stored in advance. A theoretical value calculating means for calculating a theoretical value of the image coordinates of
Based on the error between the image coordinates of each of the plurality of objective viewpoint indices detected from the objective viewpoint image and the theoretical value of the image coordinates of the objective viewpoint index , a plurality of calibration target objective viewpoint indices for the subjective viewpoint imaging device Correction value calculating means for calculating the correction value of the arrangement information ;
Correction means for correcting the arrangement information of the plurality of objective viewpoint indices with respect to the subjective viewpoint imaging device based on the correction value;
An index calibration device characterized by comprising: An index calibration apparatus that corrects arrangement information of a plurality of objective viewpoint indices attached to a subjective viewpoint imaging apparatus with respect to the subjective viewpoint imaging apparatus,
Objective viewpoint imaging means for capturing an objective viewpoint image including a plurality of objective viewpoint indices attached to the subjective viewpoint imaging device from an objective viewpoint position and orientation that is fixed and known in the world coordinate system;
Subjective viewpoint image input means for inputting a subjective viewpoint image including a plurality of subjective viewpoint indices whose positions in the world coordinate system arranged in the real space are known from the subjective viewpoint imaging device;
Objective viewpoint image input means for inputting the objective viewpoint image from the objective viewpoint imaging means;
Subjective viewpoint index detection means for detecting the image coordinates of each of the plurality of subjective viewpoint indices from the subjective viewpoint image;
Objective viewpoint index detection means for detecting image coordinates of the plurality of objective viewpoint indices from the objective viewpoint image;
An initial value input means for inputting an initial value of each arrangement information of the plurality of objective viewpoint indices with respect to the subjective viewpoint imaging device;
Position and orientation calculation means for calculating a relative position and orientation between the objective viewpoint imaging device and the objective viewpoint index based on the image coordinates of each of the plurality of objective viewpoint indices detected by the objective viewpoint index detection means; ,
The objective viewpoint position and orientation, the relative position and orientation of the objective viewpoint imaging device and the objective viewpoint index, the initial value, the camera parameters specific to the objective viewpoint imaging means stored in advance, and the subjective viewpoint A theoretical value calculating means for calculating a theoretical value of each image coordinate of the plurality of subjective viewpoint indices in the subjective viewpoint image based on a position of the index in the world coordinate system;
Based on the error between the image coordinates of each of the plurality of subjective viewpoint indices detected from the subjective viewpoint image and the theoretical value of the image coordinates of the subjective viewpoint index, a plurality of calibration target objective viewpoint indices for the subjective viewpoint imaging device Correction value calculating means for calculating the correction value of the arrangement information;
Correction means for correcting the arrangement information of the plurality of objective viewpoint indices with respect to the subjective viewpoint imaging device based on the correction value;
An index calibration device characterized by comprising: The index calibration apparatus according to claim 1, wherein the camera parameter is a focal length. Determining means for determining whether or not the error is smaller than a preset threshold;
Further comprising
The initial value input means includes
4. The method according to claim 1, wherein when the error is determined not to be smaller than a preset threshold value, the corrected arrangement information is re-input as the initial value. 5. The indicator calibration device described. The image coordinates of the objective viewpoint index, the index calibration apparatus according to any one of claims 1 to 4, characterized in that an image coordinates of one point or more points defined by 該客view view-indices. An index calibration method indicator calibration device for correcting the location information for the main view view imaging apparatus of the plurality of objective viewpoint index attached to the subjective-view imaging apparatus performs,
The objective viewpoint imaging means possessed by the index calibration apparatus captures an objective viewpoint image including a plurality of objective viewpoint indices attached to the subjective viewpoint imaging apparatus from a known objective viewpoint position and orientation fixed in the world coordinate system. Objective viewpoint imaging process,
Subjective viewpoint image input means possessed by the index calibration apparatus inputs a subjective viewpoint image including a plurality of subjective viewpoint indices whose positions in the world coordinate system are known that are arranged in the real space from the subjective viewpoint imaging apparatus. An image input process;
An objective viewpoint image input unit that the index calibration apparatus has, an objective viewpoint image input step of inputting the objective viewpoint image from the objective viewpoint imaging unit;
Subjective viewpoint index detection means included in the index calibration apparatus, a subjective viewpoint index detection step of detecting image coordinates of each of the plurality of subjective viewpoint indices from the subjective viewpoint image;
An objective viewpoint index detection unit that the index calibration device has, an objective viewpoint index detection step of detecting image coordinates of the plurality of objective viewpoint indices from the objective viewpoint image;
An initial value input unit that the index calibration device has, an initial value input step of inputting an initial value of arrangement information of each of the plurality of objective viewpoint indexes with respect to the subjective viewpoint imaging device;
The position and orientation calculation means of the index calibration device is based on the image coordinates of each of the plurality of subjective viewpoint indices detected in the subjective viewpoint index detection step, and the position of the subjective viewpoint index in the world coordinate system. A position and orientation calculation step of calculating a subjective viewpoint position and orientation of the subjective viewpoint imaging device;
The theoretical value calculation means possessed by the index calibration device is based on the subjective viewpoint position and orientation, the objective viewpoint position and orientation, the initial value, and camera parameters specific to the objective viewpoint imaging means stored in advance. A theoretical value calculating step of calculating a theoretical value of each image coordinate of the plurality of objective viewpoint indices in the objective viewpoint image;
The correction value calculation means included in the index calibration device has a plurality of values based on errors between the image coordinates of each of the plurality of objective viewpoint indices detected from the objective viewpoint image and the theoretical value of the image coordinates of the objective viewpoint index. A correction value calculating step of calculating a correction value of the arrangement information of the objective viewpoint index to be calibrated with respect to the subjective viewpoint imaging device;
A correcting step in which the correction means included in the index calibration device corrects arrangement information of a plurality of objective viewpoint indexes with respect to the subjective viewpoint imaging device based on the correction value;
An index calibration method characterized by comprising: An index calibration method performed by an index calibration apparatus that corrects arrangement information of a plurality of objective viewpoint indices mounted on a subjective viewpoint imaging apparatus with respect to the subjective viewpoint imaging apparatus,
The objective viewpoint imaging means possessed by the index calibration apparatus captures an objective viewpoint image including a plurality of objective viewpoint indices attached to the subjective viewpoint imaging apparatus from a known objective viewpoint position and orientation fixed in the world coordinate system. Objective viewpoint imaging process,
Subjective viewpoint image input means possessed by the index calibration apparatus inputs a subjective viewpoint image including a plurality of subjective viewpoint indices whose positions in the world coordinate system are known that are arranged in the real space from the subjective viewpoint imaging apparatus. An image input process;
An objective viewpoint image input unit that the index calibration apparatus has, an objective viewpoint image input step of inputting the objective viewpoint image from the objective viewpoint imaging unit;
Subjective viewpoint index detection means included in the index calibration apparatus, a subjective viewpoint index detection step of detecting image coordinates of each of the plurality of subjective viewpoint indices from the subjective viewpoint image;
An objective viewpoint index detection unit that the index calibration device has, an objective viewpoint index detection step of detecting image coordinates of the plurality of objective viewpoint indices from the objective viewpoint image;
An initial value input unit that the index calibration device has, an initial value input step of inputting an initial value of arrangement information of each of the plurality of objective viewpoint indexes with respect to the subjective viewpoint imaging device;
The position / orientation calculation means of the index calibration device is configured to determine whether the objective viewpoint imaging device and the objective viewpoint index are relative to each other based on the image coordinates of each of the plurality of objective viewpoint indices detected in the objective viewpoint index detection step. Position and orientation calculation step for calculating a general position and orientation;
The theoretical value calculation means included in the index calibration device includes the objective viewpoint position and orientation, the relative position and orientation of the objective viewpoint imaging device and the objective viewpoint index, the initial value, and the objective viewpoint stored in advance. A theoretical value calculation step of calculating a theoretical value of each image coordinate of the plurality of subjective viewpoint indices in the subjective viewpoint image based on a camera parameter unique to the imaging means and a position of the subjective viewpoint index in the world coordinate system. When,
The correction value calculation means included in the index calibration device has a plurality of errors based on the error between the image coordinates of each of the plurality of subjective viewpoint indices detected from the subjective viewpoint image and the theoretical value of the image coordinates of the subjective viewpoint index. A correction value calculating step of calculating a correction value of the arrangement information of the objective viewpoint index to be calibrated with respect to the subjective viewpoint imaging device;
A correcting step in which the correction means included in the index calibration device corrects arrangement information of a plurality of objective viewpoint indexes with respect to the subjective viewpoint imaging device based on the correction value;
An index calibration method characterized by comprising: The program for making a computer perform each process which the parameter | index calibration method of Claim 6 or 7 has. A computer-readable storage medium storing the program according to claim 8 .

Images