JP4991395B2 - Information processing method and information processing apparatus - Google Patents

Description

  The present invention relates to an information processing method and an information processing apparatus, and more particularly, to an information processing method and an information processing apparatus for calibrating position / orientation conversion information between image capturing apparatuses in order to acquire the position and orientation of a plurality of image capturing apparatuses.

  In recent years, research on mixed reality has been actively conducted for the purpose of seamless connection between the real space and the virtual space. An image display device that presents a mixed reality presents an image of a virtual space (for example, a virtual object or character information drawn by computer graphics) on a real space image captured by an imaging device such as a video camera. This is realized by superimposing display.

  Applications of such image display devices include virtual reality such as surgical support that superimposes the state of the body on the patient's body surface and mixed reality games that fight virtual enemies floating in real space. Different new fields are expected.

  A common requirement for these applications is how to accurately align the real space and the virtual space, and many efforts have been made in the past.

  In addition, in order to present a stereoscopic image, efforts are being made to take a real space image corresponding to each of the left and right viewpoints with a stereo camera and to superimpose and display the corresponding virtual space image.

  In this case, since each image viewed from the left and right viewpoints is generated, efforts are made to make each processing operation unit independent in order to prevent a heavy load and a slow processing time. In that case, an independent general-purpose computer is often used as each processing operation unit.

  Also, in such a system, a head-mounted device that incorporates an imaging device called a HMD and a display device is often used.

  The problem of alignment in mixed reality results in the problem of obtaining the three-dimensional position and orientation of the imaging device in the world coordinate system (hereinafter simply referred to as the world coordinate system) set in the real space. As a method for solving these problems, a three-dimensional position and orientation sensor such as a magnetic sensor or an ultrasonic sensor is generally used.

  The output value output from the three-dimensional position / orientation sensor includes coordinate conversion for converting the position / orientation of the measurement point in the sensor coordinate system to the position / orientation of the imaging apparatus, and the converted position / orientation in the sensor coordinate system in the world coordinate system. It is used after being converted by coordinate conversion to be converted into.

  The position and orientation of one imaging device in the left and right imaging devices are obtained as described above, but the position and orientation of the other imaging device are obtained by converting the position and orientation according to the conversion information of the positional relationship between the two imaging devices. .

  In order to maintain a normal stereoscopic effect, it is not desirable to separately obtain and use the positions and orientations of the two imaging devices, and this conversion must always be performed using constant conversion information. Used.

  FIG. 4 is a diagram illustrating a configuration of a general image display device that presents mixed reality to the user.

  A rough block includes a position / orientation sensor 130, a master processing unit 170, a slave processing unit 1170, and the head mounting device 100. The position / orientation sensor 130 is configured by, for example, Folstrak of Polhemus, which is a magnetic sensor. Each of the master processing unit 170 and the slave processing unit 1170 is constituted by a general-purpose computer. The head-mounted device 100 is a stereo video see-through HMD, and a FASTRAK receiver that is a magnetic sensor is fixed.

  In the head-mounted device 100, a master display unit 110, a master video camera 120, a slave display unit 1110, and a slave video camera 1120 are fixed. When the user wears the head-mounted device 100 so that the master display unit 110 and the slave display unit 1110 are positioned in front of the user's eyes, the scene in front of the user's eyes is displayed by the master video camera 120 and the slave video camera 1120. Imaged.

  Accordingly, it is assumed that an image captured by the master video camera 120 is displayed on the master display unit 110. Then, the user observes the scene in front of the eyes that would be observed with the naked eye if the head-mounted device 100 is not worn, via the master video camera 120 and the master display unit 110.

  The position and orientation sensor 130 is a device that measures the position and orientation of a measurement point fixed to the head-mounted device 100 in the sensor coordinate system.

  The position / orientation sensor 30 is configured by, for example, Polhemus FASTRAK, which is a magnetic sensor including a receiver 131, a sensor control unit 132, and a transmitter 133. The receiver 131 is fixed to the head-mounted device 100 as a measurement point, and the sensor control unit 132 measures the position and orientation of the receiver 131 in the sensor coordinate system with the position and orientation of the transmitter 133 as a reference, and outputs a measurement value. .

  On the other hand, the master processing unit 170 includes a master position / orientation information conversion unit 140, a memory 150, an image generation unit 160, and a position / orientation information transmission unit 180, and can be constituted by, for example, one general-purpose computer.

  The master position / orientation information conversion unit 140 converts the measurement value input from the position / orientation sensor 130 in accordance with the calibration information stored in the memory 150, calculates the position / orientation of the video camera 120 in the world coordinate system, and calculates the position / orientation information. Output as. The position / orientation transmission unit 180 transmits the position / orientation information input from the position / orientation information conversion unit 140 to the position / orientation information reception unit 1180 of the slave processing unit 1170.

  The image generation unit 160 generates a virtual image based on the position / orientation information input from the position / orientation information conversion unit 140, and superimposes the virtual image on a real image captured by the video camera 120 and outputs the virtual image. The display unit 110 displays the image input from the image generation unit 160 on the display screen.

  On the other hand, the slave master processing unit 1170 includes a slave position / orientation information conversion unit 1140, a memory 1150, an image generation unit 1160, and a position / orientation information reception unit 1180, and can be configured by, for example, one general-purpose computer.

  The position / orientation information receiving unit 1180 receives the position / orientation information transmitted from the position / orientation information transmitting unit 180 of the master processor 170 and outputs the position / orientation information to the slave position / orientation information conversion unit 1140.

  The slave position / orientation information conversion unit 1140 converts the position / orientation information of the master input from the position / orientation transmission unit 180 according to the calibration information stored in the memory 1150, and calculates the position / orientation of the video camera 1120 in the world coordinate system. . And this is output as position and orientation information.

  The image generation unit 1160 generates a virtual image based on the position / orientation information input from the slave position / orientation information conversion unit 1140, and superimposes the virtual image on the real image captured by the video camera 1120 and outputs the virtual image. The display unit 1110 displays the image input from the image generation unit 1160 on the display screen.

  With the above configuration, the user can feel as if the virtual object is three-dimensionally present in the real space in front of him.

  Next, a method for calculating the position and orientation of the slave video camera 1120 in the world coordinate system in the slave position and orientation information conversion unit 1140 will be described with reference to FIG.

5, the position and orientation of the master video camera 120 and _{M MW} in the world coordinate system 200, the position and orientation of the slave video camera 1120 and _{M SW,} the position and orientation of the slave video camera 1120 viewed from the coordinate system of the master video camera 120 Let _MSM . Note that the position and orientation of the object B in the coordinate system A is expressed by a viewing transformation matrix M _BA (4 × 4) from the coordinate system A to the coordinate system B.

At this time, _MSW is
M _SW = M _SM · M _MW (Formula A)
Can be indicated by

Among them, M _MW is calculated by the master position / orientation information conversion unit 140 and input to the slave position / orientation information conversion unit 1140 via the position / orientation information transmission unit 180 and the position / orientation reception unit 1180. Further, the _MSW is output from the slave position / orientation information conversion unit 1140 to the image generation unit 1160. M _{SM corresponds} to the calibration information required to convert the _{M MW} to _{M SW.} The slave position / orientation information conversion unit 1140 uses the M _MW input from the position / orientation information reception unit 1180 and the M _SM stored in the memory 1150 to calculate _MSW based on (Equation A), This is output to the image generation unit 1160.

  In order for the user to view a video with a natural three-dimensional feeling, it is necessary to perform accurate alignment between the master video camera 120 and the slave video camera 1120, and accurate calibration information is stored in the memory 1150 by some means. Need to be set. Only when accurate calibration information is given, a stereoscopic image can be seen.

  Note that the calibration information in the memory 1150 is not necessarily a viewing transformation matrix, and may take any form as long as the information can define the position and orientation of the other coordinate system as viewed from one coordinate system.

  For example, the position and orientation may be expressed by a total of six parameters including three parameters that describe the position and three parameters that express the posture by Euler angles. The posture may be expressed by four parameters, a ternary vector defining the rotation axis and a rotation angle around the axis, or the rotation angle may be expressed by the magnitude of the vector defining the rotation axis. It may be expressed by three parameters. Moreover, you may express by the parameter showing those inverse transformations. However, in any case, the position and orientation of the object in the three-dimensional space have only a total of 6 degrees of freedom, with 3 degrees of freedom in position and 3 degrees of freedom in orientation. Therefore, the number of unknown parameters necessary for calibration of the image display apparatus is six parameters necessary for conversion from the coordinate system 120 of the master video camera to the coordinate system 1120 of the slave camera.

One known method for setting calibration information is as follows. That is, the user or operator, via the input unit (not shown), and change interactively the 6 parameters defining M _SM stored in the memory 1150 (or equivalent 6 or more parameters to it), the exact position There is a method in which adjustment is performed by trial and error until alignment is achieved.

Further, if the master arithmetic processing unit 170 and the slave arithmetic processing unit 1170 are built in one general-purpose computer, calibration is performed using the images of both the master video camera 120 and the slave video camera 1120 as inputs. It is also possible to do it. In this case, there is a method in which the master video camera 120 is connected to the slave processing unit 1170 only when calibration is performed, and after the calibration is completed, the original connection state in FIG. 4 is restored.
Sato, Uchiyama, Yamamoto: UG + B method: Registration method using subjective and objective viewpoint cameras and posture sensors, Transactions of the Virtual Reality Society of Japan, vol. 10, no. 3, pp. 391-400, 2005. Sato, Uchiyama, Tamura: Registration method in mixed reality, Transactions of the Virtual Reality Society of Japan, vol. 8, no. 2, pp. 171-180, 2003. M.M. A. Fischler and R.M. C. Bolles (June 1981): Random Sample Consensus: A paradigm for model fitting with application to imaging analysis, Comm. of the ACM, vol. 24, no. 6, pp. 381-395, 1981.

  However, in the former method, since it is necessary to adjust six unknown parameters at the same time, the adjustment takes a considerable time, and there is a problem that accurate calibration information is not always obtained. In the latter method, there is a problem that the connection state is changed in order to perform calibration, and the operation of returning it to the original state is necessary, which is confusing.

  The present invention has been made in view of the above problems, and calibration information for converting the position and orientation of a master video camera connected to separate devices into the position and orientation of a slave video camera in the world coordinate system can be simplified. The purpose is to get to.

  A further object of the present invention is to obtain the calibration information described above without changing the physical connection.

In order to achieve the object of the present invention, the information processing method of the present invention comprises:
The first imaging device and the second imaging device whose mutual positional relationship is fixed are connected to the first processing unit and the second processing unit, respectively, and from the position and orientation of the first imaging device, An information processing method for calculating calibration information between the first imaging device and the second imaging device, which is necessary for obtaining the position and orientation of the imaging device,
A step of acquiring an image coordinate of a plurality of feature points on a captured image captured by the first imaging device by an acquisition unit of the first processing unit;
A calculation step of calculating the position and orientation of the first imaging device in the world coordinate system based on the acquired value of the image coordinates of the feature point and the coordinates of the feature point in the world coordinate system stored in advance;
A transmission step of transmitting the calculated position and orientation from the first processing unit to the second processing unit;
An acquisition step in which an acquisition unit of the second processing unit acquires image coordinates of the plurality of feature points on a captured image captured by the second imaging device;
Based on the communicated position and orientation, the acquired value of the image coordinate acquired by the acquisition unit of the second processing unit, and the coordinates of the feature point in the world coordinate system stored in advance, the calibration information is A step of calculating by the calculating means of the second processing unit;
Is provided.

The information processing apparatus of the present invention
The first imaging means and the second imaging means whose positional relationship is fixed to each other are connected to the first processing unit and the second processing unit, respectively, and from the position and orientation of the first imaging unit, An information processing apparatus for calculating calibration information between the first imaging device and the second imaging device, which is necessary for obtaining the position and orientation of the second imaging means,
The first processing unit is
Acquisition means for acquiring image coordinates of a plurality of feature points on a captured image captured by the first imaging means;
Calculation means for calculating the position and orientation of the first imaging device in the world coordinate system based on the acquired value of the image coordinates of the feature point and the coordinates of the feature point in the world coordinate system stored in advance;
Transmission means for transmitting the calculated position and orientation to the second processing unit,
The second processing unit is
Acquisition means for acquiring image coordinates of the plurality of feature points on a captured image captured by the second imaging device;
Based on the position and orientation transmitted by the transmission unit, the acquired value of the image coordinate acquired by the acquisition unit of the second processing unit, and the position of the feature point in the world coordinate system stored in advance, the calibration Computing means for computing information.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to acquire easily the calibration information for converting the acquisition value of the position and orientation information of a 1st imaging device into the position and orientation of a 2nd imaging device.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

  In the following embodiments, a case where the calibration apparatus and the method of the present invention are applied to the calibration of the image display apparatus of FIG. 1 will be described.

  FIG. 3 is a conceptual diagram for explaining the arrangement of landmarks in a real space and a set of cameras when calibration is performed according to the present embodiment. In the present embodiment, markers 4001 to 4004 of different colors arranged on real objects are shown as landmarks detected as feature points. In FIG. 3, a texture pattern is used instead of a color.

  As shown in FIG. 3, while the same landmark is photographed from each camera whose positional relationship is fixed, a plurality of images are photographed by changing the position of the camera as will be described later, and calibration is performed.

  In order to calibrate the imaging apparatus using the calibration apparatus of this embodiment, at least three or more landmarks whose world coordinates are known are arranged in the real space to be imaged by the imaging apparatus. There is a need. Here, for example, each landmark has a different color so that the image coordinates of the projected image on the captured image can be detected and which landmark can be identified. Suppose that

  FIG. 1 shows the configuration of an information processing apparatus that is a sensor calibration apparatus according to the present embodiment. As shown in FIG. 1, the calibration device according to the present embodiment includes a master arithmetic processing device 300, a slave arithmetic processing device 1300, a head mounting device 100, and a position and orientation sensor 130.

  The position and orientation sensor 130, the display unit 110, and the display unit 1110 perform the same operations as those described with reference to FIG.

  The master processing unit 300 includes a world coordinate holding unit 310, an image coordinate acquisition unit 320, a data management unit 330, a position and orientation calculation unit 340, a position and orientation information transmission unit 380, and others. The master arithmetic processing device 300 is connected to the head mounting device 100 and the slave arithmetic processing device 1300 that are calibration targets. For connection to the slave arithmetic processing unit 1300, it is possible to use communication means such as Ethernet (registered trademark). In the present embodiment, the master processing unit 300 functions as a first processing unit.

  The world coordinate holding unit 310 holds the coordinates of each landmark in the world coordinate system, and outputs them according to a request from the data management unit 330.

  The image coordinate acquisition unit 320 inputs an image captured by the video camera 120, detects the image coordinate of the landmark captured in the image and its identification number, and in accordance with a request from the data management unit 330, Information is output to the data management unit 330. A landmark is detected from an image by using a landmark having color as identification information, extracting a specific color area from the image by threshold processing, and detecting the position of the center of gravity as an image coordinate of the landmark having the color. Do that. In the present embodiment, the video camera 120 functions as the first imaging device.

  When the image coordinates and identification number of the landmark are input from the image coordinate acquisition unit 320, the data management unit 330 acquires the world coordinates of the corresponding landmark from the world coordinate holding unit 310, and positions the world coordinates and the image coordinates. The information is output to the posture information calculation unit 340.

  The position / orientation information calculation unit 340 calculates the position / orientation of the video camera 120 from the world coordinates and the image coordinates, and outputs them to the position / orientation information transmission unit 380 and the image generation unit 160.

As a method for calculating the position and orientation of the video camera 120, it is generally performed to sequentially measure the position and orientation of the camera using time-series images continuously captured by the camera (non-patent document). 1, non-patent document 2, non-patent document 3). For example, the position and orientation of the camera in the reference coordinate system can be obtained by the following procedure.
(1) A plurality of landmarks whose positions in the reference coordinate system (reference coordinates) are known are arranged or set on an indoor floor, wall, table surface, or the like.
(2) The coordinates of the projected image of the landmark in the captured image taken by the camera are detected.
(3) The position and orientation of the camera are obtained based on the correspondence between the detected image coordinates of the landmark and the reference coordinates of the landmark.

  The position and orientation information transmission unit 380 outputs the position and orientation information received from the position and orientation information calculation unit 340 to the position and orientation information reception unit 1380 of the slave arithmetic processing device 1300.

  The slave processing unit 1300 includes a world coordinate storage unit 1310, an image coordinate acquisition unit 1320, a data management unit 1330, a calibration information calculation unit 1340, an instruction unit 1350, and others. The slave arithmetic processing device 1300 is connected to the head mounting device 100 and the master arithmetic processing device 300 that are calibration targets. In the present embodiment, the slave arithmetic processing device 1300 functions as a second processing unit.

  The world coordinate holding unit 1310 holds the coordinates of each landmark in the world coordinate system, and outputs them according to a request from the data management unit 1330.

  The information held in the world coordinate holding unit 1310 needs to be the same as the information in the world coordinate holding unit 310 of the master processing unit 300 in advance. It is also possible to use the calibration device more easily by providing communication means and synchronizing.

  The image coordinate acquisition unit 1320 receives an image captured by the video camera 1120, detects the image coordinate of the landmark captured in the image and its identification number, and in accordance with a request from the data management unit 1330, Information is output to the data management unit 1330. A landmark is detected from an image by using a landmark having color as identification information, extracting a specific color area from the image by threshold processing, and detecting the position of the center of gravity as an image coordinate of the landmark having the color. Do that. In the present embodiment, the video camera 1120 functions as a second imaging device.

  When the data management unit 1330 receives an instruction of “data acquisition” from the instruction unit 1350, the image coordinate and identification number of the landmark are input from the image coordinate acquisition unit 1320, and the world of the corresponding landmark is received from the world coordinate holding unit 1310. Coordinates are entered. Further, sensor measurement values at the same time as the captured image are input from the master position / orientation information receiving unit 1380, and a set of [world coordinates-master position / orientation measurement values-image coordinates] is added to the data list and held. In addition, the generated data list is output to the calibration information calculation unit 1340 in accordance with a request from the calibration information calculation unit 1340.

  Upon receiving an instruction “calculation information calculation” from the instruction unit 1350, the calibration information calculation unit 1340 receives a data list from the data management unit 1330, calculates calibration information based on the data list, and outputs the calculated calibration information. To do.

  The instruction unit 1350 gives a “data acquisition” instruction to the data management unit 1330 when a data acquisition command is input from the user, and a “calibration information calculation” instruction to the calibration information calculation unit 1340 when a calibration information calculation command is input. Send to. A command can be input to the instruction unit 1350 by pressing a key to which a specific command is assigned, for example, using a keyboard. In addition, the command may be input by any method such as a GUI displayed on a display (not shown).

  FIG. 2 is a flowchart showing a processing procedure of the calibration apparatus according to the present embodiment. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus according to the present embodiment, and is read and executed by a CPU (not shown).

  In step S4010, instruction unit 1350 determines whether or not a data acquisition command has been input from the operator. If a data acquisition command has been input, the process proceeds to step S4020. In the following, it is assumed that the viewpoint position where the video camera 1120 is located when the data acquisition command is input is Vj.

  In step S4020, the data management unit 1330 receives a measurement value when the video camera 1120 is located at the viewpoint position Vj from the position / orientation information reception unit 1380.

を入力する。

Enter.

  In step S4030, the data management unit 1330 identifies the landmark Qk identification number k and image coordinates on the captured image captured by the video camera 1120 located at the viewpoint position Vj.

を、画像座標取得部１３２０から入力する。この入力は、当該撮像画像上に複数のランドマークが撮像されている場合には、それら各々のランドマークに対して行われる。

Is input from the image coordinate acquisition unit 1320. This input is performed for each landmark when a plurality of landmarks are captured on the captured image.

  In step S4040, the data management unit 1330 receives the world coordinates (corresponding to each identification number k) of each landmark Qk input from the image coordinate acquisition unit 1320.

を、世界座標管理部１３３０から入力する。

Is input from the world coordinate management unit 1330.

  In step S4050, the data management unit 1330 adds the input data to the data list Li for each detected landmark. Specifically, the image coordinates for the landmark Qk are

，その世界座標を

, Its world coordinates

，その際のマスタカメラの位置姿勢を

, The position and orientation of the master camera at that time

として、

As

の組を、ｉ番目のデータとしてリストに登録する。ここでｉは現在リストに登録されているデータ総数から１を引いた値を示す。
以上によって、データの取得が行われる。

Are registered in the list as the i-th data. Here, i indicates a value obtained by subtracting 1 from the total number of data currently registered in the list.
Data acquisition is performed as described above.

  In step S4060, the instruction unit 1350 determines whether the data list acquired up to the present has enough information to calculate the calibration information. If the data list does not satisfy the condition, the process returns to step S4010 again and waits for the input of a data acquisition command. On the other hand, if the data list satisfies the conditions for calculating calibration information, the process proceeds to step S4070. As conditions for calculating calibration information, for example, data on four or more different landmarks is obtained, data is acquired at a plurality of viewpoint positions, and the total number of data is three or more. This is a condition. However, since the accuracy of the calibration information derived as the diversity of input data increases, the condition may be set so as to request more data.

  In step S4070, it is determined whether a calibration information calculation command has been input from the operator. If the calibration information calculation command has been input, the process proceeds to step S4080. If not, the process returns to step S4010 again to wait for the input of the data acquisition command.

The calibration information calculation unit 1340 internally uses the ternary vector that defines the orientation information in the calibration information to be obtained by defining the rotation angle by the vector size and the rotation axis direction by the vector direction. Express. _Then, the _{M SM,} the position of the master video camera 1120 in the coordinate system slave video camera 1120 is defined _{(x CS, y CS, z} CS) and orientation

によって表現し、６の未知パラメータは、ベクトル

The six unknown parameters are represented by the vector

によって表現される。

Is represented by

In step S4080, the calibration information calculation unit 1340 gives an appropriate initial value (for example, s = [000000] ^T ) to the vector s.

  In step S4090, the calibration information calculation unit 1340, for each data Li (i = 1, 2,..., N) in the list, the theoretical value of the image coordinates of the landmark.

を、センサ出力

The sensor output

、世界座標

, World coordinates

および現在のｓに基づいて算出する。

And based on the current s.

は、ｓを変数として、センサ出力

Is the sensor output with s as a variable

および世界座標

And world coordinates

によって定義される関数

Functions defined by

によって定めることができる。

Can be determined by.

  Specifically, first,

に基づいて、ｓの構成要素である（ｘ_ＣＳ，ｙ_ＣＳ，ｚ_ＣＳ）及び

(X _CS , y _CS , z _CS ) and the components of s based on

からビューイング変換行列Ｍ_ＳＭを算出する。次に、当該ランドマークのカメラ座標

To calculate the viewing transformation matrix _MSM . Next, the camera coordinates of the landmark

を、

The

によって算出する。最後に、画像座標の理論値を、

Calculated by Finally, the theoretical value of the image coordinates

によって算出する。ただし、ｆはビデオカメラ１１２０の焦点距離である。

Calculated by Here, f is the focal length of the video camera 1120.

  In step S4100, the calibration information calculation unit 1340 calculates the theoretical value of the image coordinates of the landmark for each data Li in the list.

と実測値ｕ^ｉとの誤差△ｕ^ｉを、

An error △ ^{u i} of the measured value ^{u i} and,

によって算出する。

Calculated by

  In step S4110, the calibration information calculation unit 1340, with respect to each data Li in the list, a Jacobian of 2 rows × 6 columns having a solution obtained by partial differentiation of the right side of (Formula B) with each element of the vector s. line; queue; procession; parade

を算出する。具体的には、（式Ｅ）の右辺をカメラ座標

Is calculated. Specifically, the right side of (Equation E) is the camera coordinates

の各要素で偏微分した解を各要素に持つ２行×３列のヤコビ行列

2-row x 3-column Jacobian matrix with partial differential solutions for each element

と、（式Ｄ）の右辺をベクトルｓの各要素で偏微分した解を各要素に持つ３行×６列のヤコビ行列

And a 3 × 6 Jacobian matrix having a solution obtained by partial differentiation of the right side of (Equation D) with each element of the vector s.

をそれぞれ、（式Ｅ）および（式Ｄ）に基づいて算出し、

Are calculated based on (Equation E) and (Equation D), respectively.

として算出する。

Calculate as

In step S4120, the calibration information calculation unit 1340 calculates the error Δu ⁱ and the Jacobian matrix for all data Li in the list calculated in steps S4100 and S4110.

に基づいて、ｓの補正値△ｓを算出する。具体的には、全てのデータに対する誤差△ｕ^ｉおよびヤコビ行列

Based on the above, a correction value Δs of s is calculated. Specifically, error Δu ⁱ and Jacobian matrix for all data

をそれぞれ垂直に並べたベクトルＵ＝［△ｕ^１ △ｕ^２ ・・・ △ｕ^Ｎ］^Ｔおよび

U = [Δu ¹ Δu ² ... Δu ^N ] ^T

And using the generalized inverse matrix of Φ,
^{△ s = (Φ T Φ)} -1 Φ T U ( Equation H)
Calculate as

In step S4130, calibration information calculation unit 1340 corrects s using correction value Δs calculated in step S4120.
s + Δs → s (Formula I)

  In step S4140, the calibration information calculation unit 1340 determines whether the calculation has converged by using some criterion such as whether U is sufficiently small or whether Δs is sufficiently small. If not converged, the processing from step S4090 is performed again using the corrected s.

  In step S4150, the calibration information calculation unit 1350 outputs the obtained s as calibration information. The calibration information is output, for example, in the form of a viewing transformation matrix calculated from s. The output form may be s itself, or any other position and orientation description method.

  Based on this calibration information, a virtual image is generated and combined with a real image.

<Modification 1>
In the above-described embodiment, this is realized by using two independent general-purpose computers. However, in recent computers where the multiplexing of CPUs and GPUs has progressed, it is also effective to make the master and slave processes independent in the same general-purpose computer. Even in this case, the calibration method according to the present invention makes it easy and accurate. Calibration can be performed.

<Modification 2>
In the above embodiment, the acquisition of the image coordinates of the landmark and its identification number is performed by extracting a specific color area from the image by threshold processing using the landmark having the color as identification information, This is done by detecting the image coordinates of the landmarks it has.

  However, any other method may be used as long as the projected coordinates of the landmarks on the image and the landmark identification number can be specified. For example, using a landmark having a specific pattern as identification information, a region of the specific pattern may be extracted from the image by pattern matching, and the detection position may be output as the image coordinates of the landmark having the pattern.

  The image processing is not necessarily performed, and the landmark image coordinates and the identification number thereof may be input manually by an operator. In this case, the image coordinate acquisition unit 320 uses some GUI that allows the operator to easily input the landmark position by, for example, specifying the landmark position on the captured image displayed on the work display by clicking the mouse. It is desirable to have the configuration.

  Also, when using a plurality of landmarks having the same characteristics that cannot be identified by image processing, the image coordinates of the landmarks are acquired by image processing, and the identification number is input manually. You can also. In addition, any other method may be used for landmark identification. For example, when rough calibration information is obtained from the beginning, the landmark is calculated by comparing the theoretical value of the image coordinate of each landmark calculated by (Equation B) with the image coordinate of the detected landmark. May be identified.

  Further, in either case of acquisition by image processing or acquisition by manual input, the landmark does not necessarily have to be artificial (artificial), and natural features may be used. Natural features can be realized by registering in advance in association with the outline of the building and its coordinate position.

<Modification 3>
In the above embodiment, an appropriate value is set as the initial value of the calibration information in step S4080. However, when the initial value and the actual value are far from each other, the solution does not converge in the above embodiment, and calibration information cannot be obtained.

  In order to deal with such a situation, combinations of different positions and orientations are set in advance, and these are sequentially used as initial values to perform the processing from step S4090 to step S4140, and s when the solution converges is selected. You may do it. Further, an initial value input unit for inputting an initial value (or position and orientation information necessary for generating the initial value) by an operator is further prepared, and calibration information calculation processing is performed using the input initial value. It is also possible.

<Modification 4>
In a more preferred modification of the present invention, the calibration device further includes an image generation unit. The image generation unit superimposes the image coordinates of the landmark detected by the image coordinate acquisition unit 1320 on the captured image and outputs the superimposed image to the display screen. According to this modification, the operator can input a data acquisition instruction while confirming the detection status of the landmark.

  Further, the image generation unit calculates the theoretical value of the image coordinates of each landmark based on the calibration information calculated by the calibration information calculation unit 1340, and superimposes this on the captured image and outputs it to the display screen. According to this modification, the operator can verify the calibration work by comparing the actual landmark and the calculation position displayed there.

<Modification 5>
In the embodiment described above, the instruction unit 1350 for inputting a control command from the operator is provided. However, this input is not always necessary. For example, each time the image coordinate acquisition unit 1320 detects a landmark, it is added to the data list, and when the data list satisfies the condition, the calibration information calculation unit 1340 calculates the calibration information. You may make it the structure which performs.

<Modification 6>
In the above embodiment, the position and orientation between a plurality of cameras of a stereo video see-through HMD that presents mixed reality is calibrated. However, the scope of application of the present invention is not limited to this, and the present invention can be applied to any other application that measures the position and orientation of two or more fixed imaging devices. In that case, a master imaging device can be set, and the position and orientation from the master imaging device to another imaging device can be calibrated.

[Other Embodiments]
An object of the present invention is to supply a storage medium (or recording medium), which is a computer-readable memory storing a program code constituting a software control program for realizing the functions of the above-described embodiments, to a system or apparatus, and the system. It goes without saying that this can also be achieved by the computer (or CPU or MPU) of the apparatus reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowchart described above (shown in FIG. 2).

It is a figure which shows the structure of the calibration apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the process sequence of the calibration apparatus which concerns on one Embodiment of this invention. It is a figure which shows the concept of the calibration in one Embodiment of this invention. It is a figure which shows the structure of the general image display apparatus which presents mixed reality. It is a figure explaining the conversion for calculating | requiring the position and orientation of a slave camera from the position and orientation measurement value of a master video camera.

Claims (8)

The first imaging device and the second imaging device whose mutual positional relationship is fixed are connected to the first processing unit and the second processing unit, respectively, and from the position and orientation of the first imaging device, An information processing method for calculating calibration information between the first imaging device and the second imaging device, which is necessary for obtaining the position and orientation of the imaging device,
A step of acquiring an image coordinate of a plurality of feature points on a captured image captured by the first imaging device by an acquisition unit of the first processing unit;
A calculation step of calculating the position and orientation of the first imaging device in the world coordinate system based on the acquired value of the image coordinates of the feature point and the coordinates of the feature point in the world coordinate system stored in advance;
A transmission step of transmitting the calculated position and orientation from the first processing unit to the second processing unit;
An acquisition step in which an acquisition unit of the second processing unit acquires image coordinates of the plurality of feature points on a captured image captured by the second imaging device;
Based on the communicated position and orientation, the acquired value of the image coordinate acquired by the acquisition unit of the second processing unit, and the coordinates of the feature point in the world coordinate system stored in advance, the calibration information is A step of calculating by the calculating means of the second processing unit;
An information processing method comprising:   The information processing method according to claim 1, wherein the feature point is a marker arranged in a real space.   The information processing method according to claim 1, wherein the feature point is a natural feature existing in a real space.   In the step of acquiring by the acquisition unit of the first processing unit and the acquisition unit of the second processing unit, the feature point is detected by performing image processing on the captured image, and the image coordinates are calculated. The information processing method according to claim 1.   The step of acquiring by the acquisition unit of the first processing unit and the acquisition unit of the second processing unit acquires the image coordinates based on an instruction from a user. Information processing method.   A control program for realizing the information processing method according to any one of claims 1 to 5 by a computer.   A computer readable memory storing the control program according to claim 6. The first imaging means and the second imaging means whose positional relationship is fixed to each other are connected to the first processing unit and the second processing unit, respectively, and from the position and orientation of the first imaging unit, An information processing apparatus for calculating calibration information between the first imaging device and the second imaging device, which is necessary for obtaining the position and orientation of the second imaging means,
The first processing unit is
Acquisition means for acquiring image coordinates of a plurality of feature points on a captured image captured by the first imaging means;
Calculation means for calculating the position and orientation of the first imaging device in the world coordinate system based on the acquired value of the image coordinates of the feature point and the coordinates of the feature point in the world coordinate system stored in advance;
Transmission means for transmitting the calculated position and orientation to the second processing unit,
The second processing unit is
Acquisition means for acquiring image coordinates of the plurality of feature points on a captured image captured by the second imaging device;
Based on the position and orientation transmitted by the transmission unit, the acquired value of the image coordinate acquired by the acquisition unit of the second processing unit, and the position of the feature point in the world coordinate system stored in advance, the calibration An information processing apparatus comprising an operation means for calculating information.

Images