JP4533193B2 - Information processing apparatus and information processing method - Google Patents

Description

  The present invention relates to a method for obtaining a position or position / orientation of an index placed on an object in an object coordinate system.

  In recent years, research on mixed reality has been actively conducted for the purpose of seamless connection between the real space and the virtual space. An image display device that presents a mixed reality is drawn on a virtual space (for example, computer graphics) generated in accordance with the position and orientation of an imaging device on an image of a real space that is captured by an imaging device such as a video camera. This is realized by a video see-through method in which an image of a virtual object, character information, etc.) is overlaid and displayed.

  Applications of such image display devices include virtual reality, such as surgical support that superimposes the state of the body on the patient's body surface, and mixed reality games that fight against virtual enemies floating in real space. Different new fields are expected.

  A common requirement for these applications is how to accurately align the real space and the virtual space, and many efforts have been made in the past. The problem of alignment in mixed reality is that the relative position and orientation relationship between the target object on which virtual information is to be superimposed and the imaging device (hereinafter referred to as the position of the target object relative to the imaging device and Resulting in the problem of seeking a posture.

  As a method for solving this problem, on the target object, a plurality of indicators whose arrangement in the coordinate system of the target object is known is set or set, and the coordinates of the indicator in the object coordinate system, which is known information, In general, the position and orientation of a target object with respect to an imaging device are obtained by using the coordinates of a projected image of an index in an image captured by the imaging device.

  A method for calculating the position and orientation of a target object with respect to an imaging apparatus from a set of index coordinates in an object coordinate system and image coordinates of a projected image has been proposed for a long time in the field of photogrammetry (for example, non-patent literature). 1, see Non-Patent Document 2).

  In addition, a 6-DOF position and orientation sensor such as a magnetic sensor is attached to both the imaging device and the target object, and is measured by the sensor in addition to the coordinates of the projected image of the index in the image captured by the imaging device (the sensor is the reference A hybrid position / orientation measurement method for obtaining the position and orientation of the target object with respect to the imaging device by further using the measured values of the position and orientation of each of the imaging device and the target object (in the reference coordinate system) is also proposed ( For example, in Patent Document 1, an error of a sensor measurement value is corrected using information on a projected image of an index detected on an image).

In addition, in the Japanese Patent Application No. 2003-323101, the present applicant sets an index on the target object for the purpose of measuring the position and orientation of the target object in a certain reference coordinate system, and uses a camera fixed to the reference coordinate system. A method has been proposed in which the position and orientation of the target object with respect to the fixed camera are obtained by photographing, and thereby the position and orientation of the target object in the reference coordinate system are obtained.
R. M.M. Haralick, C.I. Lee, K.M. Ottenberg, and M.M. Nole: Review and analysis of solutions of the three point perspective positive estimation lab, International Journal of Computer Vision, vol. 13, no. 3, pp. 331-356, 1994. D. G. Low: Fitting parametrized three-dimensional models to images, IEEE Transactions on PAMI, vol. 13, no. 5, pp. 441-450, 1991. Uchiyama, Yamamoto, Tamura: “Hybrid registration method for mixed reality: combined use of 6-DOF sensor and vision method”, Transactions of the Virtual Reality Society of Japan, vol. 8, no. 1, pp. 119-125, 2003. Hirofumi Fujii, Noriyuki Kanbara, Hidehiko Iwasa, Haruo Takemura, Naokazu Yokoya, alignment with stereo camera combined with gyro sensor for augmented reality, IEICE technical report PRMU99-192 (IEICE technical report vol.99) No. 574, pp. 1-8, 1999. A. I. Comfort, E.M. Marchand, F.M. Chaumette, A real-time tracker for markerless augmented reality, Proc. Int'l Symp. on Mixed and Augmented Reality 2004, pp. 36-45, 2004.

  By the way, in the above method for calculating the position and orientation of the target object by detecting the image coordinates of the projected image of the index in the image captured by the imaging device, a plurality of indices set for the target object, the target object, The relative positional relationship of must be known. When the relative positional relationship of each index is unknown, for example, when using the colored spherical index or the circular index and using the center of gravity of the projected image of the index on the image as a feature In addition, in the case where an index whose feature is described by one point is individually installed on the target object, the three-dimensional position of each index in the target object coordinate system must be measured in advance. . On the other hand, when the relative positional relationship between the indices is known, for example, even when the indices similar to those described above are used, these indices are mounted on a base jig in advance. (The coordinate system for describing the relative position of the index is called the index coordinate system in the following.) The position of each index has already been measured, and this jig is attached to the target object. In such a case, the position and orientation of the jig in the target object coordinate system need only be known (note that an index whose features are described by a plurality of points, for example, a square having a known shape, When using each index of the projected image on the image as a feature using a triangular index, the index itself is interpreted as a collection of multiple indices, and the `` relative positional relationship between the indices As a kind of `` if known '' To be obtained). However, a method for accurately calibrating them is not generally known, and until now there has been no measure other than using an inaccurate position or position and orientation obtained by manual measurement as a known value. As a result, in the method of calculating the position and orientation of the target object by detecting the image coordinates of the projected image of the index in the image captured by the imaging device, the position and orientation of the target object are practically low in accuracy. There was room for improvement that only measurement was possible.

  The present invention has been made in view of the above problems, and an object of the present invention is to easily and accurately acquire the position or position and orientation of an index attached to an object with respect to the target object (that is, in the target object coordinate system). And

In order to achieve the above object, the present invention has the following configuration .

An information processing apparatus for correcting the position in the object coordinate system with reference to the each of the objects of the plurality of indices allocated on the object, and the object position and orientation measurement means for measuring the position and orientation in the world coordinate system of the object A picked-up image input unit that inputs a picked-up image obtained by picking up a plurality of indices arranged on the object with a pick-up unit, a pick-up position and orientation measuring unit that measures a position and orientation of the image pick-up unit in the world coordinate system, and the picked-up image Index detecting means for detecting image coordinates of each of the plurality of indices, setting value input means for inputting position setting values of each of the plurality of indices in the object coordinate system, and positions of the objects in the world coordinate system orientation, and on the basis of the position in the world coordinate system of the imaging means, and the relative position and orientation calculation means for calculating the relative relative position and orientation between the object and the imaging means, before Index calculation means for calculating a calculated value of coordinates on the surface of the captured image of each of the plurality of indices based on a set value, the relative position and orientation, and camera parameters specific to the imaging means held in advance; An error calculating means for calculating an error between each of the detected image coordinates of the plurality of indices and a calculated value of the coordinates of each of the calculated plurality of indices on the surface of the captured image; and Correction value calculating means for calculating a correction value of the position of each of the plurality of indices in the object coordinate system, and correcting a set value of the position of each of the plurality of indices in the object coordinate system based on the correction value Correction means .

  According to the present invention, the position or position and orientation of an index with respect to an object can be acquired simply and accurately.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
The index calibration apparatus according to this embodiment is for calibrating an index mounted on a target object, and uses a 6-DOF position and orientation sensor mounted on the target object and an objective viewpoint camera installed at a fixed position. Thus, the position of each index with respect to the target object (that is, in the target object coordinate system) is calculated for each index. Hereinafter, an index calibration device and an index calibration method according to the present embodiment will be described. In the present embodiment, it is assumed that a 6-DOF position / orientation sensor is configured by “an imaging device mounted on a target object and a reference index whose position in the world coordinate system is known”. Further, it is assumed that the target object to which the 6-DOF position / orientation sensor is attached is the “imaging device” itself. That is, the index calibration apparatus according to the present embodiment is for calibrating an index (to be observed by another imaging apparatus) mounted on the imaging apparatus, and the coordinate system of the imaging apparatus (hereinafter referred to as this). It can be said that the position of each index in the subjective viewpoint camera coordinate system is calculated for each index.

  FIG. 1 is a diagram illustrating a schematic configuration of an index calibration apparatus 100 according to the present embodiment. As shown in FIG. 1, the index calibration apparatus 100 according to the present embodiment includes a subjective viewpoint index detection unit 110, a position / orientation calculation unit 120, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, data management. The unit 170 and the calibration information calculation unit 180 are connected to the imaging device 130 to which the index to be calibrated is attached.

It should be noted that one or more indices P ^k (k = 1,, K ₂ ) (hereinafter referred to as objective viewpoint indices) to be calibrated are set on the imaging device 130. . Objective viewpoint in the subjective viewpoint camera coordinate system (= target object coordinate system: a coordinate system in which one point on the imaging device 130 is defined as an origin and three axes orthogonal to each other are defined as an X axis, a Y axis, and a Z axis, respectively) The position of the index is unknown, and the position of the unknown objective viewpoint index is information that is calibrated by the index calibration apparatus according to the present embodiment.

At a plurality of positions in the real space, the world coordinate system (one point in the real space is defined as the origin as an index for photographing with the imaging device 130 (hereinafter referred to as a subjective viewpoint index (reference index)) and orthogonal to each other. A plurality of subjective viewpoint indices Q ^k (k = 1,, K ₁ ) whose positions in the three axes are defined in the coordinate system defined as the X axis, the Y axis, and the Z axis, respectively, are arranged. It is desirable that the subjective viewpoint index Q ^k is installed so that at least three or more indices are observed by the imaging device 130 when acquiring the data for index calibration. In the example of FIG. 1, four subjective viewpoint indices Q ¹ , Q ² , Q ³ , and Q ⁴ are arranged, and three of them, Q ¹ , Q ³ , and Q ^4, are included in the field of view of the imaging device 130. Shows the situation.

The subjective viewpoint index Q ^k may be constituted by, for example, circular markers each having a different color, or may be constituted by feature points such as natural features each having a different texture feature. It is also possible to use a square index formed by a square area having a certain area. Any form may be used as long as the image coordinates of the projected image on the photographed image can be detected and the index can be identified.

  An image output from the imaging device 130 (hereinafter referred to as a subjective viewpoint image) is input to the subjective viewpoint index detection unit 110.

Person view marker detecting unit 110 inputs the subjective-view image from the imaging device 130, detects the image coordinates of the subjective-view-indices Q ^k has been taken in the input image. For example, when each of the subjective viewpoint indices Q ^k is composed of markers having different colors, an area corresponding to each marker color is detected from the subjective viewpoint image, and the center of gravity position is detected as the detected coordinate of the index. To do. Further, when each of the subjective viewpoint indices Q ^k is composed of feature points having different texture characteristics, template matching based on the template image of each index held in advance as known information is performed on the subjective viewpoint image. By applying, the position of the index is detected. When a quadratic index is used as the reference index, labeling is performed after binarizing the image, and an area formed by four straight lines is detected as an index candidate from a region having a certain area or more. Further, by detecting whether or not there is a specific pattern in the candidate area, false detection is eliminated, and the direction and identifier of the index are acquired. In the present specification, the quadrangular indices detected in this way are considered to be four indices formed by four vertices.

Furthermore, subjective-view-index detection unit 110 outputs the image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} each detected and the identifier _{k n} to the position and orientation calculation unit 120. Here, n (n = 1,..., N) is an index for each detected index, and N represents the total number of detected indices. For example, in the case of FIG. 1, N = 3, identifiers k ₁ = 1, k ₂ = 3, k ₃ = 4 and the corresponding image coordinates u ^Qk1 , u ^Qk2 , u ^Qk3 are output. The subjective viewpoint index detection unit 110 according to the present embodiment sequentially executes the index detection process triggered by the input of an image from the imaging device 130, but in response to a request from the position / orientation calculation unit 120 (at that time) The process may be executed using the subjective viewpoint image input in (1).

The position / orientation calculation unit 120 is based on the correspondence relationship between the detected image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn and the world coordinates x _W ^Qkn of the indices previously stored as known information. 130 positions and orientations are calculated. A method for calculating the position and orientation of an imaging device from a set of world coordinates and image coordinates of a subjective viewpoint index has been proposed for a long time in the field of photogrammetry and the like (for example, see Non-Patent Document 1 and Non-Patent Document 2). ). For example, when the subjective viewpoint indices are arranged on the same plane, the position and orientation of the imaging device can be obtained based on the calculation of two-dimensional homography by detecting four or more indices. In addition, a method of using the position and orientation of an imaging device using six or more indices that are not on the same plane is also well known. In addition, the error between the theoretical value of the index image coordinates calculated from the estimated position and orientation of the imaging device and the actual measurement value of the index image coordinates is calculated by repeated calculation such as the Gauss-Newton method using the image Jacobian. By minimizing, a method for optimizing the estimated values of the position and orientation can be used. The calculated position and orientation of the imaging device 130 in the world coordinate system are output to the data management unit 170 in accordance with a request from the data management unit 170. Note that the position / orientation calculation unit 120 in this embodiment sequentially executes the above-described position and orientation calculation processing triggered by the input of data from the subjective viewpoint index detection unit 110, but the request from the data management unit 170 It is also possible to adopt a configuration in which processing is executed in accordance with (using input data at that time). In the following, it is assumed that the position and orientation of the imaging device 130 are held by ternary vectors [x yz] ^T and [ξ ψ ζ] ^T , respectively. There are various methods for expressing the posture by ternary values, but here, the rotation angle is expressed by the magnitude of the vector, and the ternary vector is defined by defining the rotation axis direction by the vector direction. It shall be. At this time, the position and orientation are represented by a 6-dimensional vector [x yz ξ ψ ζ] ^T.

  The objective viewpoint camera 140 is fixedly arranged at a position where the imaging device 130 for acquiring the data for index calibration can be imaged. It is assumed that the position and orientation of the objective viewpoint camera 140 in the world coordinate system are held in advance in the calibration information calculation unit 180 as known values.

The objective viewpoint index detection unit 150 inputs an image (objective viewpoint image) captured by the objective viewpoint camera 140, and an image of the objective viewpoint index ^Pk captured in the image by the same processing as the subjective viewpoint index detection unit 110. detecting the coordinates, and outputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected according to the requirements of the data management unit 170 and the identifier _{k m} to the data management unit 170. Here, m (m = 1,..., M) is an index attached to each detected index, and M represents the total number of detected indices. For example, in the case of FIG. 1, M = 2, identifiers k ₁ = 1, k ₂ = 2 and image coordinates u ^Pk1 and u ^Pk2 corresponding to these are output. The objective viewpoint index detection unit 150 in the present embodiment sequentially executes the above-described index detection processing triggered by the input of the objective viewpoint image, but in response to a request from the data management unit 170 (input at that time) The processing may be performed using an objective viewpoint image.

  The instruction unit 160 sends a “data acquisition” instruction to the data management unit 170 when a data acquisition command is input from an operator (not shown) and a “calibration information calculation” instruction when the calibration information calculation command is input. Transmit to the calculation unit 180. The command input to the instruction unit 160 can be performed by pressing a key to which a specific command is assigned, for example, using a keyboard. In addition, the command may be input by any method such as a GUI displayed on the display.

  When receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 170 inputs the position and orientation of the imaging apparatus 130 in the world coordinate system from the position / orientation calculation unit 120, and receives an objective viewpoint from the objective viewpoint index detection unit 150. The image coordinate of the index and its identifier are input, and the set of [position and orientation of the imaging device 130-image coordinate of the objective viewpoint index] is added to the data list prepared for each identifier of the objective viewpoint index and is retained. Here, the position and orientation of the imaging device 130 input from the position / orientation calculation unit 120 are the same time as the image capturing time when the image coordinates of the objective viewpoint index input from the objective viewpoint index detection unit 150 are detected. It is. Further, the data management unit 170 outputs a data list for each identifier of the generated objective viewpoint index to the calibration information calculation unit 180 in accordance with a request from the calibration information calculation unit 180.

  Upon receiving an instruction “calculation information calculation” from the instruction unit 160, the calibration information calculation unit 180 inputs a data list from the data management unit 170, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position of each objective viewpoint index in the subjective viewpoint camera coordinate system) is output.

  FIG. 2 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  In step S2010, the instruction unit 160 determines whether or not a data acquisition command has been input from the operator. The operator inputs a data acquisition command when the imaging device 130 is placed at a position where data for index calibration is acquired. If the data acquisition command is input, the instruction unit 160 shifts the process to step S2020.

In step S2020, the data management unit 170 inputs the position and orientation t = [x yz ξ ψ ζ] ^T of the imaging device 130 in the world coordinate system from the position / orientation calculation unit 120.

In step S2030, the data management unit 170, the objective-view-index detection unit 150, and inputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected by an objective-view-index detection unit 150 and the identifier _{k m.} The objective viewpoint index detection unit 150 always performs index detection processing on the input objective viewpoint image, and objective processing when the position and orientation of the imaging apparatus is t is performed by the processing in steps S2020 and S2030 described above. The image coordinates of the viewpoint index can be obtained. Note that the information input from the objective viewpoint index detection unit 150 is not necessarily related to all objective viewpoint indices, and may be information regarding the index detected on the objective viewpoint image at that time.

Next, in step S2040, the data management unit 170 adds the input data set to the data list L ^Pk for each detected objective viewpoint index P ^km . Specifically, t input from the position / orientation calculation unit 120 is set as t _i = [x _i y _i z _i ξ _i ψ _i ζ _i ] ^T, and u ^Pkm input from the objective viewpoint index detection unit 150 is u _i ^Pk. _as, [t _i, ^{u i Pk]} a set of an objective-view-index while sorting by referring to the identifiers _{k m} input from the detection unit 150, a data list related to the objective-view-index ^{P k} as i-th data about ^{P k} Register with ^LPk . Here, i (i = 1,..., I ^Pk ) is an index for each set of data registered in the data list L ^Pk , and I ^Pk represents the total number of sets of data registered for the objective viewpoint index P ^k. ing.

  Data is acquired by the above processing.

In step S2050, the data list related to all objective viewpoint indexes or the data list related to at least one objective viewpoint index acquired so far by the data management unit 170 is sufficient to calculate calibration information. A determination is made as to whether it has. If at least one data list or all data lists do not satisfy the condition, the process returns to step S2010 and waits for the input of a data acquisition command. On the other hand, if all the data lists or at least one data list satisfies the condition for calculating the calibration information, the process proceeds to step S2060. As a condition that the data list related to a certain objective viewpoint index P ^k has enough information to calculate calibration information, for example, two or more sets of data [t _i , u _i ^Pk ] having different data lists L ^Pk are obtained. It can be mentioned. As will be described later, two equations are obtained from one set of data (see Equation 3). If two or more sets of data are obtained, the subjective viewpoint camera coordinates of the objective viewpoint index, which is three parameters from the four equations, are obtained. The position in the system can be determined. However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S2060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S2070. If not input, the process returns to step S2010 and waits for the input of the data acquisition command.

The calibration information calculation unit 180 handles the calibration information to be obtained, that is, the position of the objective viewpoint index in the subjective viewpoint camera coordinate system as a ternary vector [x y z] ^T. Hereinafter, this unknown parameter is described as a state vector s ^Pk = [x ^Pky y ^Pk z ^Pk ] ^T. Hereinafter, the description will be continued assuming a process for a certain objective viewpoint index ^Pk, but each process is performed in common for all objective viewpoint indices corresponding to the data list that satisfies the conditions for calculating the calibration information. .

In step S2070, the calibration information calculation unit 180 gives an appropriate initial value (for example, [0 0 0] ^T ) to the state vector s ^Pk .

In step S2080, the calibration information calculation unit 180 calculates all i from the data [t _i , u _i ^Pk ] (i = 1, 2,..., I ^Pk ) and the state vector s ^{Pk in} the data list L ^Pk. against an objective-view-index ^{P k} of the objective viewpoint image coordinates of theory ^{_{u i Pk '= [u xi}} Pk', u yi Pk '] is calculated. Here, the theoretical value of the objective viewpoint image coordinates of the objective viewpoint index refers to data of the position (coordinates) that should be visible in the objective viewpoint image of the objective viewpoint index P ^k given the position in the subjective viewpoint camera coordinate system. . The calculation of u _i ^Pk ′ is a function of the state vector s ^Pk representing the position of the objective viewpoint index P ^{k in} the subjective viewpoint camera coordinate system.

Based on.

Specifically, the function F _i () is obtained from the objective viewpoint camera coordinates of the objective viewpoint index P ^k when the i-th data is acquired (that is, when the position and orientation of the imaging device 130 are t _i ). The following equation for ^{obtaining the} position vector b _i ^Pk from s ^Pk :

_And, following expression from ^{b i Pk} obtaining an objective-view-index ^{P k} coordinates _u ^{i Pk} on objective viewpoint images'

It is constituted by. Here, f ^B _x and f ^B _y are focal lengths of the objective viewpoint camera 140 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values. _MWB is a transformation matrix for converting coordinates in the objective viewpoint camera coordinate system into world coordinates, and is calculated in advance based on the position and orientation of the objective viewpoint camera 140 in the world coordinate system that is held in advance as a known value. It is assumed that M _WCi is a modeling conversion matrix (matrix for converting coordinates in the subjective viewpoint camera coordinate system to coordinates in the world coordinate system) determined by t _i , and is defined by the following equation.

  However,

である。

It is.

In step S2090, the calibration information calculating unit 180, for all i, the actual image coordinates _u ^{i Pk} objective-view-index ^{P k} included in each data in the data list ^{L Pk,} the image coordinates of the corresponding an error △ _u ^{i Pk} of the theoretical value _u ^{i Pk} 'is calculated by the following equation.

In step S2100, the calibration information calculating unit 180, for all i, the image Jacobian about the state vector ^{s Pk} (i.e., solutions obtained by partially differentiating the function of Equation 1 _F i () for each element of the state vector ^{s Pk} Jacobian matrix of 2 rows × 3 columns having, in each _{^{element) J uis Pk (= ∂u i}} Pk / ∂s Pk) is calculated. Specifically, a 2 × 3 Jacobian matrix J _ubi ^Pk (= （ _u) having a solution obtained by partial differentiation of the right side of Equation 3 with each element of the position vector b _i ^Pk on the objective viewpoint camera coordinate system. _i ^Pk / ∂b _i ^Pk ) and a 3 × 3 Jacobian matrix J _bis ^Pk (= ∂b _i ^Pk /) having a solution obtained by partial differentiation of the right side of Equation 2 with each element of the state vector s ^Pk. ∂s ^Pk), and calculates the _J ^{uis Pk} by the following equation.

In step S2110, the calibration information calculating unit 180 is calculated in the above step, based on the error △ _u ^{i Pk} and the Jacobian matrix _J ^{uis Pk} for all ^i, and calculates the correction value △ ^{s Pk} of ^{s Pk.} Specifically, an error vector of a 2I ^Pk dimensional vector in which errors Δu _i ^Pk for all i are arranged vertically

And a Jacobian matrix _Juis ^Pk vertically arranged in 2I ^Pk rows × 3 columns matrix

And ^{Δs Pk} is calculated from the following equation using the pseudo inverse matrix Φ ⁺ of Φ.

Here, since Δs ^Pk is a three-dimensional vector, if 2I ^Pk is 3 or more, that is, I ^Pk is 2 or more, ^{Δs Pk} can be obtained. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

In step S2120, the calibration information calculation unit 180 uses the correction value Δs ^Pk calculated in step S2110 to correct the position vector s ^Pk in the subjective viewpoint camera coordinate system of the objective viewpoint index P _k according to Equation 11 to obtain The new value is set as a new ^sPk .

In step S2130, the calibration information calculation unit 180 uses some criteria such as whether the error vector U is smaller than a predetermined threshold or whether the correction value Δs ^Pk is smaller than a predetermined threshold. It is determined whether or not it has converged. If not converged, the processing after step S2080 is performed again using the corrected state vector s ^Pk .

The calculation is determined to have converged in step S2130, in step S2140, the calibration information calculating unit 180, the state vector s ^Pk obtained, as a parameter indicating the position in the subjective-view camera coordinate system of the objective-view-index P ^k Output.

  Finally, in step S2150, it is determined whether or not to end the calibration process. When the operator instructs the index calibration apparatus 100 to end the calibration process, the process is terminated. When the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S2010 again, and the data Wait for input of get command.

  With the above processing, the position of the index attached to the target object (imaging device) with respect to the target object (that is, in the target object coordinate system) can be acquired easily and accurately.

<Modification 1-1>
In the present embodiment, the Newton method expressed by Equation 10 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the Newton method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Needless to say, the essence of the present invention is not impaired by any numerical calculation method. In this embodiment, in the processing from step S2070 to step S2130, an initial value is given to the position of the index to be obtained, and the optimum value is obtained by iterative calculation using the image Jacobian. However, the index is calculated by a simpler calculation method. It is also possible to obtain the position of. For example, if Equation 3 is expanded,

The relationship is obtained. Using this relationship,

Thus, s can be obtained directly. here,

Represents.

<Modification 1-2>
In the present embodiment, a combination of “an imaging device mounted on a target object and a reference index whose position in the world coordinate system is known” is used as a 6-DOF position and orientation sensor. If the position and orientation of the target object in the coordinate system can be measured, the 6-DOF position and orientation sensor may be configured by other methods. For example, a six-degree-of-freedom position and orientation sensor such as a magnetic sensor or a three-degree-of-freedom posture sensor such as a gyro sensor is attached to the imaging device 130, and each subjective viewpoint index Q ^kn detected from a captured image of the imaging device 130 is detected. In a hybrid method based on both the correspondence between the image coordinates u ^Qkn and the world coordinates x _W ^Qkn of the index held in advance as known information, and the measured value of the sensor attached to the imaging value 130, The position and orientation of the imaging device 130 may be calculated (refer to, for example, [Non-Patent Document 3] and [Non-Patent Document 4] for specific calculation methods).

<Modification 1-3>
In the present embodiment, since the target object is an imaging device, video information captured by the target object itself can be used to measure the position and orientation of the target object in the world coordinate system. When the target object is an arbitrary object, a 6-DOF position and orientation sensor such as a magnetic sensor can be used to measure the position and orientation. The configuration in that case is shown in FIG.

  An index calibration apparatus 500 according to a modification of the present embodiment shown in FIG. 5 is for calibrating an index mounted on an arbitrary target object 510, and is a 6-DOF position / orientation sensor mounted on the target object 510. The position of each index with respect to the target object 510 (that is, in the target object coordinate system) is calculated for each index using 520 and the objective viewpoint camera 140 installed at a fixed position. As the six-degree-of-freedom position / orientation sensor 520, a magnetic position / orientation sensor (FASTRAK of Polhemus, USA, Flock of Birds, USA) can be used, or an ultrasonic sensor can be used. A 6-DOF position / orientation sensor may be used. The index calibration apparatus 500 includes a position / orientation calculation unit 530 as a component that replaces the subjective viewpoint index detection unit 110 and the position / orientation calculation unit 120 in FIG. The position / orientation calculation unit 530 inputs a position / orientation measurement value (the position and orientation of the sensor itself in the sensor coordinate system) from the six-degree-of-freedom position / orientation sensor 520, and uses the known calibration information from the position / orientation measurement value. The position and orientation of the target object 510 in the coordinate system are calculated and output to the data management unit 170 in accordance with a request from the data management unit 170. The other parts are exactly the same as those of the index calibration apparatus 100.

<Modification 1-4>
In this embodiment, an objective viewpoint camera installed at a fixed position is used as a camera for photographing the target object. However, if the position and orientation of the camera in the world coordinate system can be measured, the objective viewpoint camera is not fixed. May be.

For example, a 6 DOF position and orientation sensor such as a magnetic sensor (Folstrak from Polhemus, USA, Flock of Birds from Ascension Tech.), An optical sensor (Optontrak from Canada, VICON from UK VICON), or an ultrasonic sensor. May be attached to the objective viewpoint camera, and the position and orientation of the objective viewpoint camera with respect to the world coordinate system may be obtained from the sensor measurement values at the time of image capturing. Further, the position and orientation of the objective viewpoint camera may be obtained from the image coordinates u ^Qkn of the subjective viewpoint index Q ^kn photographed by the objective viewpoint camera and the world coordinates Xw ^{Qkn of the} index previously held as known information. In addition, the above-described 6-DOF position and orientation sensor and 3-DOF orientation sensor are attached to the objective viewpoint camera, and a hybrid method based on both the objective viewpoint image and the measured value of the sensor ([Non-patent Document 3] [Non-patent Document] 4]), the position and orientation of the objective viewpoint camera may be obtained.

In this case, the position and orientation of the objective viewpoint camera measured by the above-described means are acquired simultaneously with the position and orientation of the imaging device 130 in step S2020, and stored in the data list in step S2040. In step 2080, M _WB in Expression 2 (representing the position and orientation of the objective viewpoint camera in the world coordinate system) is stored in the data list, not a fixed value previously stored as a known value. A value calculated based on the measured position and orientation values may be used.

[Second Embodiment]
In the index calibration device according to the present embodiment, when a plurality of indices attached to the target object has an index coordinate system, and the coordinates of each index in the index coordinate system are known, the indices attached to the target object An index coordinate system for the target object (that is, in the target object coordinate system) that uses a 6-DOF position and orientation sensor attached to the target object and an objective viewpoint camera installed at a fixed position. And the position of each index with respect to the target object. Hereinafter, an index calibration device and an index calibration method according to the present embodiment will be described. In this embodiment as well, as in the first embodiment, a 6-DOF position / orientation sensor based on “an imaging device mounted on a target object and a reference index whose position in the world coordinate system is known”. Is configured. Further, it is assumed that the target object to which the 6-DOF position / orientation sensor is attached is the “imaging device” itself.

  FIG. 3 is a diagram illustrating a schematic configuration of the index calibration apparatus according to the present embodiment. The same parts as those in FIG. 1 are given the same numbers and symbols, and detailed description thereof will be omitted. As shown in FIG. 3, the index calibration apparatus 300 according to the present embodiment includes a subjective viewpoint index detection unit 110, a position / orientation calculation unit 120, an objective viewpoint camera 140, an objective viewpoint index detection unit 150, an instruction unit 160, data management. The unit 370 and the calibration information calculation unit 380 are connected to the imaging device 130 on which an index to be calibrated is attached.

It is assumed that objective viewpoint indices P ^k (k = 1,..., K ₂ ) that are objects to be calibrated are set at a plurality of positions on the imaging device 130. It is assumed that the position of each index on the subjective viewpoint camera coordinate system (= target object coordinate system) is unknown, but the relative positional relationship of each index is known. For example, a case where a set of four objective viewpoint indices P ^k (k = 1, 2, 3, 4) forms one square index R ¹ having a known size is equivalent. FIG. 3 shows the situation. At this time, one point on the square indices R ¹ is defined as an origin, further triaxial respective X-axis orthogonal to each other, Y-axis and a coordinate system that is defined as Z-axis called an index coordinate system, index coordinate system The coordinates of each objective viewpoint index P ^k in are known. Therefore, if the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system (= conversion matrix for converting the coordinates in the index coordinate system to the subjective viewpoint camera coordinates) is found, the subjective viewpoint camera of the unknown objective viewpoint index P ^k is known. The position in the coordinate system can be calculated. From this, it is assumed that the information calibrated by the index calibration device according to the present embodiment is the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system.

The plurality of positions in real space, as in the first embodiment, as an index for taking by the imaging apparatus 130, the position is the subjective-view-indices Q ^k is known is set in the world coordinate system.

Person view marker detecting unit 110, like the first embodiment, enter the subjective-view image capturing apparatus 130 is captured, it detects the image coordinates of the subjective-view-indices Q ^k has been taken in an image, detected image coordinates ^{u Qkn} of the subjective-view-indices ^{Q kn} to output the identifier _{k n} to the position and orientation calculation unit 120.

Similar to the first embodiment, the position / orientation calculation unit 120 includes the detected image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn , and the world coordinates x _W ^{Qkn of the} indices previously stored as known information. The position and orientation of the imaging device 130 are calculated based on the corresponding relationship. The calculated position and orientation of the imaging device 130 in the world coordinate system are output to the data management unit 370 in accordance with a request from the data management unit 370.

  Similar to the first embodiment, the objective viewpoint camera 140 is fixedly disposed at a position where the image capturing apparatus 130 when capturing the data for index calibration can be imaged. It is assumed that the position and orientation of the objective viewpoint camera 140 in the world coordinate system are held in advance in the calibration information calculation unit 380 as known values.

The objective viewpoint index detection unit 150 receives the objective viewpoint image captured by the objective viewpoint camera 140, as in the first embodiment, and is objectively captured in the image by the same processing as the subjective viewpoint index detection unit 110. detecting the image coordinates of the viewpoint index ^{P k,} and outputs the image coordinates ^{u Pkm} of the objective-view-index ^{P miles} detected according to the requirements of the data management unit 370 and the identifier _{k m} to the data management unit 370. However, in the example of FIG. 3, since the square index is used as the index, the detection process of the square index described in the first embodiment is executed.

  In the same way as in the first embodiment, the instruction unit 160 gives a “data acquisition” instruction to the data management unit 370 when a data acquisition command is input from an operator (not shown), and “ An instruction “calibration information calculation” is transmitted to the calibration information calculation unit 380.

  Upon receiving the “data acquisition” instruction from the instruction unit 160, the data management unit 370 inputs the position and orientation in the world coordinate system of the imaging device 130 from the position / orientation calculation unit 120, and receives an objective viewpoint from the objective viewpoint index detection unit 150. The image coordinate of the index and its identifier are input, and a set of [position and orientation of the imaging device 130 -image coordinate of the objective viewpoint index -identifier of the objective viewpoint index] is added to one data list and held. Here, the position and orientation of the imaging device 130 input from the position / orientation calculation unit 120 are the same time as the image capturing time when the image coordinates of the objective viewpoint index input from the objective viewpoint index detection unit 150 are detected. It is. In addition, the data management unit 370 outputs the generated data list to the calibration information calculation unit 380 in accordance with a request from the calibration information calculation unit 380.

  When the calibration information calculation unit 380 receives an instruction of “calibration information calculation” from the instruction unit 160, the calibration information calculation unit 380 inputs a data list from the data management unit 370, performs a calibration process based on the data list, and obtains the calibration obtained as a result. Information (that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system) is output.

  FIG. 4 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. The same parts as those in FIG. 2 are given the same numbers and symbols, and detailed description thereof is omitted. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  From step S2010 to step S2030, processing similar to that of the first embodiment is performed. When step S2030 is completed, the process proceeds to step S4040.

Next, in step S4040, the data management unit 370 adds the input data set as data D _j to the data list L for all the detected objective viewpoint indices P ^km . Specifically, the t input from the position and orientation calculation unit _{120 t j = [x j y} j z j ξ j ψ j ζ j] is ^T, the identifier _{k m} input from the objective-view-index detection unit 0.99 _{k j} Similarly, u ^Pkm inputted from the objective viewpoint index detection unit 150 is ^set as u _j ^Pkj , and a set of D _j = [t _j , u _j ^Pkj , k _j ] is registered in the data list L as the j-th data. Here, j (j = 1,..., N ^Pk ) is an index for each data set registered in the data list L, and N ^Pk represents the total number of registered data sets.

  Data is acquired by the above processing.

In step S4050, the data management unit 370 determines whether the data list acquired so far has enough information to calculate calibration information. If the condition is not satisfied, the process returns to step S2010 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the conditions for calculating calibration information, the process proceeds to step S2060. The condition that the data list has sufficient information to calculate the calibration information includes, for example, that the data list L includes data relating to at least three different objective viewpoint indices P ^k. . However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S2060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S4070. If not input, the process returns to step S2010 and waits for the input of the data acquisition command.

The calibration information calculation unit 380 handles the calibration information to be obtained, that is, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system as a 6-value vector [x yz ξ ψ ζ] ^T. Hereinafter, this unknown parameter is described as a state vector s = [x yz ξ ψ ζ] ^T.

In step S4070, the calibration information calculation unit 380 gives an appropriate initial value to the state vector s. As an initial value, an operator may manually input an approximate value via the instruction unit 160, or a plurality of objective viewpoint indices input at a certain same time (that is, detected from the same objective viewpoint image). Are extracted from the list L, and using this data, the position and orientation of the index coordinate system in the objective viewpoint coordinate system at the same time are calculated by a known method, and the obtained position and orientation are used in the index coordinate system. A transformation matrix M _BM for transforming the coordinates into the objective viewpoint camera coordinate system is obtained, and further, a transformation determined by Equation 4 based on the position and orientation t of the imaging device input at the same time held in the list L calculated matrix M _WC, by the following equation, to obtain a transformation matrix M _CM representing the position and orientation of the index-based coordinates in the subjective-view camera coordinate system, the position represented by this matrix Attitude may be used as the initial value of s a.

Here, _MWB is a transformation matrix that converts coordinates in the objective viewpoint camera coordinate system into world coordinates, and is based on the position and orientation of the objective viewpoint camera 140 in the world coordinate system that is held in advance as a known value. It is assumed that it has been calculated in advance.

  As a known method for calculating the position and orientation of the index coordinate system in the objective viewpoint coordinate system, for example, when the indices are arranged on the same plane, two-dimensional homology is used by using four or more indices. It is possible to use a technique for obtaining the position and orientation of the index coordinate system based on the calculation of the graphic. It is also possible to use a technique that uses six or more indices that are not on the same plane, or a technique that uses these solutions as initial values and obtains an optimal solution by iterative calculations such as the Gauss-Newton method.

In step S4080, the calibration information calculation unit 380 determines from the data D _j = [t _j , u _j ^Pkj , k _j ] (j = 1, 2, ^... , N ^Pk ) and the state vector s in the data list L. The theoretical value u _j ^Pkj ′ = [u _xj ^Pkj ′, u _yj ^Pkj ′] of the objective viewpoint image coordinates of each objective viewpoint index P ^{kj is} calculated for all j. Here, the theoretical value of the objective viewpoint image coordinate of the objective viewpoint index is the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system of the objective viewpoint index P ^kj whose position vector h _M ^Pkj in the index coordinate system is known s Is the data of the position (coordinates) that should be visible in the objective viewpoint image. The calculation of u _j ^Pkj ′ is a function of the state vector s representing the position of the index coordinate system in the subjective viewpoint camera coordinate system.

Based on.

Specifically, the function F _j () is obtained from the objective viewpoint camera coordinates of the objective viewpoint index P ^kj when the j-th data is acquired (that is, when the position and orientation of the imaging device 130 are t _j ). A position vector b _j ^Pkj is obtained from s as follows:

And the following equation for ^obtaining the coordinates u _j ^Pkj ′ on the objective viewpoint image of the objective viewpoint index P ^kj from b _j ^Pkj :

It is constituted by. Here, f ^B _x and f ^B _y are focal lengths of the objective viewpoint camera 140 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values. M _WCj is a modeling transformation matrix determined by t _{j and} is defined by an expression in which i in Expression 4 is replaced with j. M _CM (s) is a modeling conversion matrix (matrix for converting coordinates in the index coordinate system to coordinates in the subjective viewpoint camera coordinate system) determined by s, and is defined by the following equation.

  However,

It is.

In step S4090, the calibration information calculation unit 380 calculates the actual image coordinates u _j ^{Pkj of the} objective viewpoint index P ^kj included in each data in the data list L and the corresponding image coordinate theory for all j. error △ _u ^{j PKJ} the value _u ^{j PKJ} 'is calculated by the following equation.

In step S4100, the calibration information calculation unit 380 performs, for each _j, a solution obtained by partially differentiating the image Jacobian relating to the state vector s (that is, the function F _j () of Expression 17 by each element of the state vector s). 2 rows × 6 columns Jacobian matrix) J _uj ^Pkj _s (= ∂u _j ^Pkj / ∂s). Specifically, a 2 × 3 Jacobian matrix J _ujbj ^Pkj (= （ _u ⁾ having a solution obtained by partial differentiation of the right side of Expression 19 with each element of the position vector b _j ^Pkj on the objective viewpoint camera coordinate system. _j ^Pkj / ∂b _j ^Pkj ) and a 3 × 6 Jacobian matrix J _bj ^Pkj _s (= ∂b _j ^Pkj /) having a solution obtained by partial differentiation of the right side of ^Equation 18 with each element of the state vector s. ∂s) is calculated, and J _uj ^Pkj _s is calculated by the following equation.

In step S4110, the calibration information calculation unit 380 calculates the correction value ^{Δs of} _s based on the error ^Δu _j ^Pkj and the Jacobian matrix J _uj ^Pkj _s for all j calculated in the above steps. Specifically, an error vector of a 2J-dimensional vector in which errors ^Δu _j ^Pkj for all j are arranged vertically

And Jacobi matrix J _uj ^Pkj _s vertically arranged in 2J rows × 6 columns matrix

And Δs is calculated from the following equation using the pseudo inverse matrix Φ ⁺ of Φ.

Here, since Δs is a six-dimensional vector, Δs can be obtained when 2J is 6 or more, that is, J is 3 or more. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

  In step S4120, the calibration information calculation unit 380 uses the correction value Δs calculated in step S4110 to correct the state vector s representing the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system according to Equation 27, and obtain Let the obtained value be a new s.

  In step S4130, calibration information calculation section 380 converges the calculation using some criterion such as whether error vector U is smaller than a predetermined threshold or whether correction value Δs is smaller than a predetermined threshold. It is determined whether or not. If not converged, the processing after step S4080 is performed again using the corrected state vector s.

If it is determined in step S4130 that the calculation has converged, in step S4140, the calibration information calculation unit 380 uses the obtained state vector s as calibration information, that is, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system. Output as. The form of output at this time may be s itself, or the position component of s may be represented by a ternary vector, and the posture component may be represented by an Euler angle or a 3 × 3 rotation matrix. then, it may be a coordinate transformation matrix M _CM generated from s.

  Finally, in step S2150, it is determined whether or not to end the calibration process. When the operator instructs the index calibration device 300 to end the calibration process, the process is ended. When the operator instructs the continuation of calibration process (recalibration), the process returns to step S2010 again, and the data Wait for input of get command.

  With the above processing, the position or position and orientation of the index attached to the target object (imaging device) with respect to the target object (that is, in the target object coordinate system) can be acquired easily and accurately.

<Modification 2-1>
In this embodiment, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system are output as the calibration information, but the objective viewpoint index P in the subjective viewpoint camera coordinate system is obtained from the obtained state vector s. The positions of ^k may be calculated individually and output as calibration information. In this case, the position of the objective viewpoint index P ^k in the subjective viewpoint camera coordinate system is obtained by the product of the coordinate transformation matrix M _CM obtained from s and the known position h _M ^Pk of the objective viewpoint index P ^k in the index coordinate system. be able to.

<Modification 2-2>
In the present embodiment, a combination of “an imaging device mounted on a target object and a reference index whose position in the world coordinate system is known” is used as a 6-DOF position and orientation sensor. If the position and orientation of the target object in the coordinate system can be measured, the 6-DOF position and orientation sensor may be configured by other methods. For example, as in the first embodiment, a six-degree-of-freedom position and orientation sensor such as a magnetic sensor and a three-degree-of-freedom posture sensor such as a gyro sensor are attached to the imaging device 130 and detected from the captured image of the imaging device 130. ^Further , the correspondence between the image coordinates u ^{Qkn of} each subjective viewpoint index Q ^kn and the world coordinates x _W ^Qkn of the index held in advance as known information, and the measured value of the sensor attached to the imaging value 130 The position and orientation of the imaging device 130 may be calculated by a hybrid method based on both (see, for example, Non-Patent Document 3 and Non-Patent Document 4 for specific calculation methods).

<Modification 2-3>
In the present embodiment, the steepest descent method expressed by Expression 26 is used to calculate the correction value of the state vector. However, the correction value need not necessarily be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be used, or a statistical method such as M estimation which is a known robust estimation method may be combined. Needless to say, the essence of the present invention is not impaired by any numerical calculation method. Further, in the present embodiment, a method for obtaining an optimal value of s for all input data by giving appropriate initial values to the position and orientation s of the index coordinate system in the subjective viewpoint camera coordinate system, and by iterative calculation using the image Jacobian. However, the position and orientation of the index coordinate system in the subjective viewpoint camera coordinate system can also be obtained by using another simple calculation method. For example, the position and orientation of the index system coordinates in the subjective viewpoint camera coordinate system are determined by using only the detection coordinates of a plurality of objective viewpoint indices on one objective viewpoint image by the processing procedure described in the description of step S4070. It may be obtained and output as calibration information.

<Modification 2-4>
In the present embodiment, a square index is used as the objective viewpoint index, but any index type may be used as long as the index group has a known position of each index in the index coordinate system. For example, it may be a set of a plurality of circular indicators as used in the first embodiment, or a plurality of types of indicators may be mixed. Further, even when a plurality of index coordinate systems are provided, the above processing is performed individually for each index coordinate system, or the above processing is performed in parallel for each index coordinate system. Can be calibrated as well.

<Modification 2-5>
In the present embodiment, since the target object is an imaging device, video information captured by the target object itself can be used to measure the position and orientation of the target object in the world coordinate system. When the target object is an arbitrary object, a 6-DOF position and orientation sensor such as a magnetic sensor can be used to measure the position and orientation. The configuration in that case is the same as <Modification 1-3> of the first embodiment shown in FIG. As a six-degree-of-freedom position and orientation sensor, a magnetic position and orientation sensor (Folstral of Polhemus, USA, Flock of Birds of Ascension Tech.) And an optical sensor (Optontrak of Canada, VICON of UK VICON), An ultrasonic sensor or the like can be used, and any other 6-degree-of-freedom position / orientation sensor may be used.

<Modification 2-6>
In this embodiment, an objective viewpoint camera installed at a fixed position is used as a camera for photographing a target object. However, as in <Modification 1-4> of the first embodiment, a camera in the world coordinate system is used. If the position and orientation of the camera can be measured, the objective viewpoint camera need not be fixed.

In this case, the measured position and orientation of the objective viewpoint camera are acquired at the same time as the position and orientation of the imaging device 130 in step S2020, and stored in the data list in step S4040. In step 4080, M _WB in Expression 18 (representing the position and orientation of the objective viewpoint camera in the world coordinate system) is stored in the data list, not a fixed value previously stored as a known value. A value calculated based on the measured position and orientation values may be used.

[Third Embodiment]
In the index calibration apparatus according to the present embodiment, calibration of a plurality of indices mounted on a target object is performed by photographing the target object with a movable imaging apparatus having a 6-DOF position and orientation sensor. In the first or second embodiment, only the position or position and orientation of the index are handled as unknown parameters, but in this embodiment, not only the position or position and orientation of the index but also the position and orientation of the camera relative to the object are handled as unknown parameters. . Hereinafter, an index calibration device and an index calibration method according to the present embodiment will be described.

  The index of the present embodiment is composed of three or more point groups that are not on the same straight line. Assume that the relative positions of a plurality of points constituting the index are known, that is, the positions of the points constituting the index in the index coordinate system defined by each index are known. In the present embodiment, when a plurality of the above-described indices are arranged on the object, the position and orientation of the index coordinate system defined by each index with respect to the object (that is, with respect to the object coordinate system) is obtained as calibration information. .

  FIG. 6 is a diagram showing a schematic configuration of an index calibration apparatus 1100 according to this embodiment. As shown in FIG. 6, the index calibration apparatus 1100 according to the present embodiment includes an index detection unit 1020, a position / orientation calculation unit 1030, an imaging unit 1010, an instruction unit 1050, a data management unit 1040, and a calibration information calculation unit 1060. And connected to the object 1000 on which the index is mounted.

  The object 1000 is equipped with a 6-DOF position / orientation sensor A2, and can measure the position and orientation of the object with respect to the world coordinate system. The 6-DOF position and orientation sensor A2 is connected to the position and orientation calculation unit 1030. The imaging unit 1010 is equipped with a 6-degree-of-freedom position / orientation sensor A1, and can measure the position and orientation of the camera constituting the imaging unit with respect to the world coordinate system. The 6-DOF position and orientation sensor A1 is connected to the position and orientation calculation unit 1030.

On the object 1000, a plurality of indices based on the above definition are arranged. Here, the index arranged on the object is represented by P ^k (k = 1,, K _o ). Here, _Ko is the number of indices to be calibrated arranged on the object. The index P ^k is composed of points p ^ki ((k = 1,, K _o ), (i = 1,, N _k )). Here, N _k represents the total number of points constituting the index P ^k .

As described above, the indices arranged on the object have known positions in the index coordinate system of the points constituting each of the indices, but the object coordinate system (one axis on the object is defined as an origin and three axes orthogonal to each other is defined. In the coordinate system defined as X axis, Y axis and Z axis, respectively, is unknown. On the object, as a reference index for defining the object coordinate system, three or more indices whose positions in the object coordinate system which are not on the same straight line are known are arranged. All the reference indices need to be observed in the same image as other calibration target markers by the imaging unit 1010 that acquires index calibration data. In addition, the index P ^k is an index that can detect the screen coordinates of the projected image on the captured image, and is an index that can identify each point that constitutes the index. Any form may be adopted.

  The imaging unit 1010 captures an index placed on the object from various positions and directions. The captured image is input to the index detection unit 1020.

The index detection unit 1020 receives an image from the imaging unit 1010, and detects image coordinates of each point constituting the index ^Pk photographed in the input image.

Furthermore, the index detector 1020 outputs the image coordinates ^{u Pkni} of ^{p kni} points constituting the index ^{P kn} each detected and the identifier _{k n} to the data management unit 1040. Here, n (n = 1,..., M) is an index for each detected index, and M represents the total number of detected indices. For example, the case of FIG. 6 shows a case where a square-shaped index having identifiers 1, 2, and 3 is captured, M = 3, identifiers k ₁ = 1, k ₂ = 2 and k ₃ = 3 and corresponding image coordinates u ^Pk1i , u ^Pk2i , u ^Pk3i , (i = 1, 2, 3, 4) are output.

  The position / orientation calculation unit 1030 calculates the object coordinates from the position / orientation in the world coordinate system of the camera and the object obtained from the 6-DOF position / orientation sensor A1 attached to the camera and the 6-DOF position / orientation sensor A2 attached to the object. Calculate the position and orientation of the camera in the system. The position and orientation of the camera in the object coordinate system are output to the data management unit 1040 in accordance with a request from the data management unit 1040.

In addition, the position / orientation calculation unit 1030 internally represents the position and orientation of the camera in the object coordinate system by a ternary vector [x y z] ^T and [ξ ψ ζ] ^T , respectively. There are various methods for expressing the posture by ternary values, but here, the rotation angle is expressed by the magnitude of the vector, and the ternary vector is defined by defining the rotation axis direction by the vector direction. It shall be. At this time, the position and orientation are each represented by a 6-dimensional vector [x y z ξ ψ ζ] ^T.

  The instruction unit 1050 calibrates a “data acquisition” instruction to the data management unit 1040 when a data acquisition command is input from an operator (not shown), and a “calibration information calculation” instruction when a calibration information calculation command is input. It transmits to the information calculation part 1060.

  Upon receiving the “data acquisition” instruction from the instruction unit 1050, the data management unit 1040 inputs the position and orientation of the camera in the object coordinate system from the position / orientation calculation unit 1030, and the index image coordinates and the image coordinates thereof from the index detection unit 1020. An identifier is input, and a pair of [position and orientation in the object coordinate system of the camera−index image coordinate−index identifier] is added to one data list and held. Here, the position and orientation of the camera in the object coordinate system input from the position and orientation calculation unit is the same time as the image capturing time when the image coordinates of the index input from the index detection unit 1020 are detected. Further, the data management unit 1040 outputs the generated data list to the calibration information calculation unit 1060 in accordance with a request from the calibration information calculation unit 1060.

  When the calibration information calculation unit 1060 receives an instruction “calculation information calculation” from the instruction unit 1050, the calibration information calculation unit 1060 inputs a data list from the data management unit 1040, performs a calibration process based on the data list, and obtains a calibration obtained as a result. Information (that is, the position and orientation of the index coordinates in the object coordinate system) is output.

  FIG. 7 is a flowchart of processing performed when the calibration apparatus according to the present embodiment obtains calibration information. Note that the program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) of the present embodiment, and is read and executed by a CPU (not shown).

  In step S6010, the instruction unit 1050 determines whether a data acquisition command is input from the operator. The operator inputs a data acquisition command when the object 1000 is placed at a position where data for index calibration is acquired. When the data acquisition command is input, the instruction unit 1050 shifts the process to step S6020.

In step S6020, the data management unit 1040 inputs the camera position and orientation in the object coordinate system t _OC = [x _OC y _OC z _OC ξ _OC ψ _OC ζ _OC ] ^T from the position and orientation calculation unit 1030. The position / orientation calculation unit 1030 always calculates t _OC . A measurement _matrix s _A1 = [x _A1C y _A1C z _A1C ξ _A1C ψ _A1C ζ _A1C ] ^T obtained from the six-degree-of-freedom position and orientation sensor A1 attached to the camera represents a transformation matrix representing the position and orientation of the camera in the world coordinate system M _{wc (s A1),} representing the measured value _{s A2 = [x A2O y A2O} z A2O ξ A2O ψ A2O ζ A2O] position and orientation of an object in the world coordinate system obtained from ^T in and attached to the object 6 DOF position and orientation sensor A2 When the transformation matrix is represented as M _wo , the transformation matrix M _oc representing the position and orientation of the camera in the object coordinate system is represented as Expression (33).

By separating the translation component and the posture component from the matrix M _oc obtained by Expression 28, the position and posture of the imaging device in the object coordinate system t _OC = [x _OC y _OC z _OC ξ _OC ψ _OC ζ _OC ] ^T is obtained. be able to.

In step S6030, the data management unit 1040, the index detection unit 1020, and inputs the image coordinates group ^{u Pkni} of points constituting the index ^{P kn} detected by the index detector 1020 and the identifier _{k n.} The index detection unit 1020 always performs index detection processing on the input image, and the processing of this step can obtain the image coordinates of the index when the position and orientation of the imaging device are t _OC. it can. Note that the information input from the index detection unit 1020 is not necessarily related to all the indices, and may be information regarding the indices detected on the image at that time.

Next, in step S6040, the data management unit 1040, the indices ^{P kn} all detection is added as data _{D L} sets of input data in the data list DL. Specifically, a _{t OC} input from the position and orientation calculation unit 1030 and _{t OCj = [x OCj y OCj} z OCj ξ OCj ψ OCj ζ OCj] T, the identifier _{k n} input from the index detector 1020 and _{k nj} Similarly, u ^Pkni input from the index detection unit 1020 is ^set as u _L ^Pknji , and a set of D _L = [t _OCj , u _L ^Pknji , k _nj ] is registered in the data list DL as the Lth data. Here, j (j = 1,..., N _J ) is an index for the captured image, and L (L = 1,, N _L ) is an index for each set of data registered in the data list DL. , N _J represents the total number of captured images, and N _L represents the total number of registered data sets.

  Data is acquired by the above processing.

  In step S6050, the data management unit 1040 determines whether the data list acquired so far has enough information to calculate calibration information. If the condition is not satisfied, the process returns to step S6010 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the conditions for calculating calibration information, the process proceeds to step S6060. The condition that the data list has information sufficient to calculate the calibration information includes, for example, that the data list DL includes data on at least three different reference indices. However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S6060, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S6070. If not, the process returns to step S6010 again to wait for the input of the data acquisition command.

The calibration information calculation unit 1060 handles the calibration information to be obtained as in the second embodiment, that is, the position and orientation of the index in the object coordinate system as a six-value vector. Hereinafter, this unknown parameter is described as a state vector s _m = [x _m y _m z _m ξ _m ψ _m ζ _m ] ^T. In this embodiment, the camera position and orientation t _{OCj with} respect to the object is also handled as an unknown parameter. t _OCj is _described as a state vector s _cj = [x _ocj y _ocj z _ocj ξ _ocj ψ _ocj ζ _ocj ] ^T

In step S6070, the calibration information calculation unit 1060 gives appropriate initial values to the respective state vectors s _m and s _cj . As the initial value of s _cj , t _OCj obtained from the sensor output is used. The initial value of s _m, the operator may manually input the approximate values via the instruction unit 1050. If the transformation matrix from the camera coordinate system to the object coordinate system is M _oc and the transformation matrix from the object coordinate system to the index coordinate system attached to the object is M _MO , the transformation matrix M _MC from the camera coordinate system to the index coordinates is obtain.

In step S4080, the calibration information calculation unit 1060 obtains each data D _L = [t _OCj , u _L ^Pknji , k _j ], ((j = 1, 2, ^... , N _J ) (L = 1,2 ,,,, _{N L)} and the state vector _{s m, s cj} from, for all L, calculated values _u ^{lL Pknji} of the image coordinates of the index ^{_{P knj '= [u xL Pknji}} ', u yL Request ^Pknji ']. here, the calculated value of the image coordinates of the indices, the position vector _h ^{M Pknj} known point ^{P Knj} in index coordinate system of the marker position and orientation are _{s m} in the object coordinate system, image The data of the position (coordinates) that should be seen in. The calculation of u _L ^Pknji 'is a function of the state vector s = [s _m s _Cj ] representing the position of the index coordinate system in the object coordinate system

Based on.

Specifically, the function F _L () is the camera coordinates of the index P ^knj when the _Lth data is acquired (that is, when the position and orientation of the imaging device 1010 in the data set is t _OCj ). A position vector c _L ^Pknji at is obtained from s _m , s _{cj as} follows:

^And the following equation for ^obtaining the coordinates u _L ^Pknji ′ on the image of the index P ^knj from c _L ^Pknji :

It is constituted by. Here, f ^B _x and f ^B _y are the focal lengths of the camera 1010 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values. Also,

Is a transformation matrix (matrix that transforms from the index coordinate system to the object coordinate system),

Is a modeling transformation matrix (matrix for transformation from the object coordinate system to the camera coordinate system) determined by s _cj .

Is defined by:

  However,

It is. Also,

  However,

It is.

In step S6090, the calibration information calculation unit 1060 for all L, the actual image coordinates u _L ^Pknji of the index P ^knj included in each data in the data list DL and the calculated values u of the image coordinates corresponding thereto. error △ _u ^{L Pknji} of the _L ^Pknji 'is calculated by the following equation.

In step S6100, the calibration information calculation unit 1060 _{applies the} image Jacobian (that is, the function F _j () of Equation 30 to the state vectors s _m and s _cj for the state vector s = [s _m s _cj ] for all L. (2 × N _L ) rows × (6 × K _o + 6 × K _NJ ) Jacobian matrix) J _uL ^Pknji _s (= ∂u _L ^Pknji / ∂s) Is calculated. Here, N _L is the total number of all detected indices, and K _o is the total number of indices to be calibrated. K _NJ is the number of images taken.

In step S6110, the calibration information calculation unit 1060 calculates the correction value ^{Δs of} _s based on the error ^Δu _L ^Pknji and the Jacobian matrix J _uL ^Pknji _s for all L calculated in the above steps. Specifically, an error vector of 2N _L- dimensional vectors in which errors ^Δu _L ^Pkndji for all ^Ls are arranged vertically

And Jacobian matrix _J ^{uL Pknji} _s aligned vertically (2N _L) matrix of rows _{× (6 × K o + 6} × K NJ) column

And Δs is calculated from the following equation using the pseudo inverse matrix Φ ⁺ of Φ.

Here, since Δs is a (6 × K _o + 6 × K _NJ ) dimensional vector, if 2N _L is (6 × K _o + 6 × K _NJ ) or more, _Δs can be obtained. Note that Φ ⁺ can be determined by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T , but may be determined by other methods.

In step S6120, calibration information calculation unit 1060 corrects s according to equation 41 using correction value Δs calculated in step S6110, and sets the obtained value as new s. Here, the state vector s = [s _m s _cj ] is the index position / posture state vector s _{m in} the object coordinate system and the camera position / posture state vector s _Cj in the object coordinate system.

  In step S6130, the calibration information calculation unit 1060 converges the calculation by using some criterion such as whether the error vector U is smaller than a predetermined threshold or whether the correction value Δs is smaller than a predetermined threshold. It is determined whether or not. If not converged, the processing after step S6080 is performed again using the corrected state vector s.

If it is determined in step S6130 that the calculation has converged, in step S6140, the calibration information calculation unit 1060 uses s _m in the obtained state vector s as calibration information, that is, the position of the index coordinate system in the object coordinate system. And output as a posture. The output form at this time may be s _m itself, the position component of s _m is represented by a ternary vector, and the posture component is represented by an Euler angle or a 3 × 3 rotation matrix. it also may be a coordinate conversion matrix M _OM generated from s _m.

  Finally, in step S6150, it is determined whether or not to end the calibration process. When the operator instructs the index calibration apparatus 1100 to end the calibration process, the process is ended. When the operator instructs the continuation (recalibration) of the calibration process, the process returns to step S6010 again, and the data Wait for input of get command.

  Through the above processing, the position or position and orientation of the index attached to the target object with respect to the target object (that is, in the target object coordinate system) can be acquired easily and accurately.

<Modification 3-1>
In the present embodiment, the imaging unit 1010 is configured to be freely movable, but the imaging unit 1010 may be installed at a fixed position as in the objective viewpoint cameras in the first and second embodiments. . In this case, the 6-DOF position / orientation sensor A1 is unnecessary, and the position / orientation calculation unit 1030 includes the position / orientation of the imaging unit 1010 in the world coordinate system held in advance as a known value and the 6-DOF position / orientation sensor A1. Based on the position and orientation of the object in the world coordinate system obtained from the above, the position and orientation of the imaging unit 1010 in the object coordinate system are calculated. In the first embodiment and the second embodiment, the position / orientation of the objective viewpoint camera needs to be accurately obtained as a known value, but in this embodiment, the position / orientation calculation unit 1030 outputs. Since correction processing is performed on the position and orientation of the camera in the object coordinate system, there is an advantage that only a rough position and orientation need be input as known values.

<Modification 3-2>
In this embodiment, a 6-DOF position / orientation sensor is used to obtain the relative positional relationship between the camera and the object. However, any sensor can be used as long as the position and orientation of the camera in the object coordinate system can be measured. Good.

  For example, a 6-DOF position and orientation sensor such as a magnetic sensor (Folstrak of Polhemus, USA, Flock of Birds of Ascension Tech.), An optical sensor (Optotrak of NDI, VICON of UK), or an ultrasonic sensor. It may be attached to the camera and the object, and the position and orientation with respect to the object coordinate system of the camera at the time of image capturing may be obtained from the sensor measurement value. Similarly to the first and second embodiments, the object and the object are photographed from the measurement values of the six-degree-of-freedom position and orientation sensor configured by the camera mounted on the object and the reference index whose position in the world coordinate system is known. You may obtain | require the relative position and orientation of the camera to perform. Further, the above-described 6-degree-of-freedom position and orientation sensor and 3-degree-of-freedom orientation sensor are attached to the object and the camera that photographs the object, and the object and the camera that photographs the object by a hybrid method based on both the photographed image and the sensor measurement The relative position and orientation may be obtained.

<Modification 3-3>
Note that the position and orientation of the 6-DOF position and orientation sensor A1 attached to the camera can be obtained after the position and orientation of the indicator attached to the object is calibrated by the indicator calibration apparatus according to the present embodiment. is there. Here, the position and orientation are represented by a 4 × 4 matrix M as shown in Expression 42 using a three-dimensional translation vector t representing the position and a 3 × 3 rotation matrix R representing the orientation.

As described in the third embodiment, when obtaining the position and orientation relative to the object of index attached to the object, the position and orientation M _OC with respect to the object of the camera at the time of photographing the image using the calibration time I want. The relative position and orientation M _A2A1 of the 6- _DOF position / orientation sensor attached to the camera and the 6-DOF position / orientation sensor attached to the object can be calculated from the sensor output values. When the position and orientation _MOA2 with respect to the object of the six-degree-of-freedom position / orientation sensor attached to the object are obtained by some method, Expression 43 is established.

Here summarized _{M OA2} and _{M A2A1} is known information as follows.

  Expression 43 is expressed as Expression 45 using Expression 44.

When N _J images are captured, M _OC and M _OA1 that exist for each captured image, and the position and orientation M _A1C of the six-degree-of-freedom position / orientation sensor attached to the camera can be expressed as in Expression 46.

  here,

Then, Expression 46 can be expressed as Expression 49.

When there are a plurality of captured images, it is possible to calculate as shown in Equation 50 using a generalized inverse matrix of B to obtain M _A1C .

In addition, in order to obtain M _A1C using A and B, another method may be used.

From the 4 × 4 matrix M _A1C representing the calculated position and orientation, a 3 × 3 rotation matrix R representing the orientation and a three-dimensional translation vector t representing the position are obtained by decomposition as shown in Equation 42. As described above, the position and orientation of the 6-DOF position and orientation sensor attached to the camera with respect to the camera can be obtained.

[Other Embodiments]
An object of the present invention is to supply a storage medium (or recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus. Needless to say, this can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above (shown in FIGS. 2 and / or 4).

It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 1st Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 1st Embodiment. It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 2nd Embodiment. It is a figure which shows schematic structure of the parameter | index calibration apparatus which concerns on <modification 1-3> of 1st Embodiment, and <modification 2-5> of 2nd Embodiment. It is a figure which shows schematic structure of the parameter | index structure apparatus which concerns on 3rd Embodiment. It is a flowchart which shows the schematic process of the parameter | index calibration method which concerns on 3rd Embodiment.

Claims (6)

An information processing apparatus for correcting the position in the object coordinate system with reference to the each of the objects of the plurality of indices allocated on the object,
An object position and orientation measuring means for measuring the position and orientation of the object in the world coordinate system;
Captured image input means for inputting a captured image obtained by capturing a plurality of indexes arranged on the object by an imaging means;
Imaging position and orientation measurement means for measuring the position and orientation of the imaging means in the world coordinate system;
A marker detecting means for detecting the image coordinates of each of said plurality of indices from the captured image,
A set value input means for inputting a set value of the position of each of the plurality of indices in the object coordinate system;
A relative position and orientation calculating means for calculating a relative relative position and orientation between the object and the imaging means based on the position and orientation of the object in the world coordinate system and the position of the imaging means in the world coordinate system ;
Index calculating means for calculating a calculated value of coordinates on the surface of the captured image of each of the plurality of indices based on the set value, the relative position and orientation, and camera parameters specific to the imaging means held in advance; ,
An error calculating means for calculating an error between each of the detected image coordinates of the plurality of indices and a calculated value of coordinates on the surface of the captured image of each of the calculated indices;
Correction value calculation means for calculating a correction value of the position of each of the plurality of indices in the object coordinate system based on the error;
Correction means for correcting each of the set values of the positions of each of the plurality of indices in the object coordinate system based on the correction values;
An information processing apparatus comprising: The information processing apparatus according to claim 1, wherein the camera parameter is a focal length of the imaging apparatus. The position / orientation measurement apparatus according to claim 1, wherein the relative position / orientation is a position / orientation of the imaging unit in the object coordinate system. Convergence determining means for determining whether the error or the correction value is smaller than a preset threshold value;
When it is determined that the error or the correction value is not smaller than a preset threshold value, the setting value input unit inputs the corrected setting value, and the index calculation unit sets the corrected setting. Based on the value, a calculated value of coordinates on the surface of the captured image of each of the plurality of indices is calculated, the error calculating means calculates the error, and the correction value calculating means is based on the error. The position / orientation measurement apparatus according to claim 1, wherein the correction value is calculated. An information processing method for an information processing apparatus performs to correct the position in the object coordinate system with reference to the each of the objects of the plurality of indices allocated on the object,
An object position / orientation measurement step in which the object position / orientation means of the information processing apparatus measures the position and orientation of the object in the world coordinate system;
A captured image input step of inputting a captured image obtained by capturing an image of a plurality of indices arranged on the object with an imaging unit ;
An imaging position / orientation measurement step in which an imaging position / orientation measurement unit included in the information processing apparatus measures a position / orientation of the imaging unit in a world coordinate system;
A marker detecting step index detecting means for the information processing apparatus has found to detect the image coordinates of each of said plurality of indices from the captured image,
A setting value input unit that the information processing apparatus has, a setting value input step of inputting a setting value of a position of each of the plurality of indices in the object coordinate system;
The relative position / orientation calculation means of the information processing apparatus is configured to determine a relative position between the object and the imaging unit based on the position / orientation of the object in the world coordinate system and the position of the imaging unit in the world coordinate system. the relative position and orientation calculation step of calculating the relative position and orientation,
An index calculation unit included in the information processing apparatus is configured to determine whether each of the plurality of indexes on the surface of the captured image is based on the setting value, the relative position and orientation, and a camera parameter specific to the imaging unit that is held in advance. An index calculation process for calculating a calculated value of coordinates;
The error calculation means included in the information processing apparatus calculates an error between the image coordinates of each of the detected plurality of indices and the calculated value of the coordinates of the calculated plurality of indices on the surface of the captured image. An error calculating step,
A correction value calculating unit that the information processing apparatus has, based on the error, a correction value calculating step of calculating a correction value of a position of each of the plurality of indices in the object coordinate system;
A correction step in which the correction means included in the information processing apparatus corrects a set value of the position of each of the plurality of indices in the object coordinate system based on the correction value;
An information processing method characterized by comprising: A program for causing a computer to execute each step of the information processing method according to claim 5 .

Images