JP2013101128A - Information processing apparatus, and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for easily and accurately performing calibration of an attitude sensor mounted on an image capturing apparatus or a physical object.SOLUTION: An attitude measurement value acquisition unit 145 acquires an attitude measurement value measured by an attitude sensor 110 mounted on an image capturing apparatus 120, and an attitude measurement value measured by a sensor 115 mounted on a physical object 125. An image processing unit 135 acquires image coordinates, on a captured image, of marks arranged on the physical object 125. A calibration unit 160 determines at least one of location information of the sensor 110 with respect to the image capturing apparatus 120 and location information of the sensor 115 with respect to the physical object 125 by using the two attitude measurement values and image coordinates.

Description

  The present invention relates to a calibration technique for an attitude sensor mounted on an imaging device or a real object.

  Research on mixed reality that superimposes and presents characters and CG images on real objects has been actively conducted. A presentation device that presents mixed reality superimposes and draws an image generated according to the position and orientation relationship between the imaging device and the real object on a captured image of the real object captured by an imaging device such as a video camera. It can be realized as a device for displaying this. In order to realize such a presentation device, it is necessary to measure the position and orientation of a real object with respect to the imaging device in real time.

  In general, the position and orientation of a real object with respect to an imaging apparatus are measured using an image captured by the imaging apparatus. In this method, a feature such as a marker arranged on a real object is detected on a captured image, and the position and orientation of the real object are calculated based on the detected feature.

  Further, a method has been proposed in which a posture sensor is attached to each of the imaging device and the real object, and a posture measurement value obtained by the posture sensor and a captured image obtained by the imaging device are used in combination (see Non-Patent Document 1). According to this method, the position and orientation of a real object can be stably measured as compared with the case where only a captured image is used.

  By the way, in the method disclosed in Non-Patent Document 1, it is assumed that the postures of the imaging device and the real object can be measured by the posture sensor. However, the posture measurement value actually output by the posture sensor represents the posture of the posture sensor itself, not the posture itself of the measurement target object (the imaging device and the real object). That is, the posture measurement value obtained by the posture sensor cannot be used as it is as the posture of the measurement target object, and it is necessary to perform some coordinate conversion. Specifically, coordinate conversion is required to convert the posture of the posture sensor itself into the posture of the measurement target object. This coordinate transformation can be defined by the attitude of the attitude sensor in a coordinate system based on the measurement target object (hereinafter referred to as a measurement target object coordinate system). Hereinafter, the posture of the posture sensor in the measurement target object coordinate system is referred to as “position information” of the posture sensor as necessary. Such arrangement information may be referred to as “Local Transform” of the attitude sensor. Further, the work and processing for obtaining the placement information of the posture sensor is called “calibration” of the posture sensor.

  One of the calibration methods that have been performed so far is a trial and error method in which an operator interactively increases or decreases a parameter representing a local transform.

  Patent Document 1 discloses a method for automatically calculating the placement information of a posture sensor attached to an imaging device using images obtained by imaging markers placed in real space from a plurality of viewpoints. According to this method, the posture sensor mounted on the imaging apparatus can be calibrated by a very simple operation.

JP 2005-326275 A

Kotake, Sato, Uchiyama, Yamamoto: “Hybrid registration using tilt angle constraint”, Image Recognition and Understanding Symposium (MIRU2006), pp.241-248, July 2006. K. Satoh, S. Uchiyama, H. Yamamoto, and H. Tamura: "Robust vision-based registration utilizing bird's-eye view with user's view," Proc. 2nd IEEE / ACM Int'l Symp. On Mixed and Augmented Reality ( ISMAR 2003), pp.46-55, October 2003. AI Comport, E. Marchand, and F. Chaumette, “A real-time tracker for markerless augmented reality,” Proc. 2nd IEEE / ACM Int'l Symp. On Mixed and Augmented Reality (ISMAR 2003), pp.36-45 , October 2003.

  With the technique disclosed in Patent Document 1, it is not possible to calibrate a posture sensor attached to any real object other than the imaging device (in other words, a real object that does not have the imaging device). Therefore, conventionally, when an attitude sensor is attached to an arbitrary real object other than the image pickup apparatus, the operator has to adjust the local transform of the attitude sensor by trial and error. However, this operation is not easy, and there is a problem of requiring high skill from the operator.

  In addition, the method disclosed in Patent Document 1 has a problem that a parameter used as an initial value must be estimated and input in advance by some method.

  Further, in the method disclosed in Patent Document 1, it is not necessary to input in advance which direction the gravity axis is facing in the world coordinate system that defines the three-dimensional position of the marker arranged in the real space. There was a problem that it should not be.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for easily and accurately calibrating an orientation sensor attached to an imaging device or a real object.

  In order to achieve the object of the present invention, for example, an information processing apparatus of the present invention comprises the following arrangement.

That is, the posture measurement value for acquiring the posture measurement value by the posture sensor for the imaging device attached to the imaging device and the posture measurement value by the posture sensor for the real object attached to the real object imaged by the imaging device. Acquisition means;
Image acquisition means for acquiring an image obtained by the imaging device imaging the real object;
Image coordinate acquisition means for acquiring image coordinates on the image of an index placed on the real object;
At least one of the arrangement information of the posture sensor for the imaging device with respect to the imaging device and the arrangement information of the posture sensor for the real object with respect to the real object is a posture measurement value by the posture sensor for the imaging device, for the real object And a calculation means for obtaining using a posture measurement value obtained by a posture sensor and the image coordinates.

  With the configuration of the present invention, the posture of the orientation sensor attached to the imaging device or the real object with respect to the imaging device or the real object can be acquired easily and accurately. Moreover, the trouble of inputting the initial value and the direction of the gravity axis in advance can be omitted.

It is a block diagram which shows the function structure of the system which has a sensor calibration apparatus which concerns on the 1st Embodiment of this invention. 2 is a block diagram showing a hardware configuration of a computer applicable to the sensor calibration apparatuses 100 and 500. FIG. It is a flowchart of the main process which the sensor calibration apparatus 100 performs. It is a flowchart which shows the detail of the process in step S3060. It is a block diagram which shows the function structure of the system which has a sensor calibration apparatus which concerns on the 2nd Embodiment of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
The information processing apparatus according to the present embodiment simultaneously calibrates a posture sensor (image sensor posture sensor) attached to an imaging device such as a video camera and a posture sensor (real object posture sensor) attached to a real object. Therefore, hereinafter, the information processing apparatus according to the present embodiment is referred to as a sensor calibration apparatus. Below, this sensor calibration apparatus and the sensor calibration method which this sensor calibration apparatus performs are demonstrated.

  FIG. 1 is a block diagram showing a functional configuration of a system having a sensor calibration apparatus according to the present embodiment. As shown in FIG. 1, the sensor calibration device 100 is connected to a posture sensor 110 and a posture sensor 115 that are calibration targets, and an imaging device 120 that can capture a moving image represented by a video camera.

  The sensor calibration apparatus 100 includes an image acquisition unit 130, an image processing unit 135, a three-dimensional coordinate holding unit 140, an orientation measurement value acquisition unit 145, an instruction unit 150, a data management unit 155, a calibration unit 160, a position and orientation calculation unit 170, and an image. The generation unit 180 and the display unit 190 are included.

  The posture sensor 110 is attached to the imaging device 120, and the imaging device 120 is a posture measurement object for the posture sensor 110. The posture sensor 115 is attached to the real object 125, and the real object 125 is a posture measurement target for the posture sensor 115. The purpose of the sensor calibration apparatus 100 according to the present embodiment is to obtain the arrangement information of the attitude sensor 110 with respect to the imaging apparatus 120 and the arrangement information of the attitude sensor 115 with respect to the real object 125.

  Each attitude sensor 110, 115 measures its own attitude in the sensor coordinate system. The posture sensor is a sensor unit based on, for example, a gyro sensor and an acceleration sensor. Specifically, it consists of TISS-5-40 from Tokimec, Inc., InertiaCube3 from InterSense, Inc., etc. The posture measurement value measured by these sensors is calculated as a value obtained by integrating the inertia measurement value with time, and thus includes an accumulation error. However, these attitude sensors have a function of canceling accumulation of errors in the tilt angle direction (pitch angle and roll angle) by observing the direction of gravity of the earth with a built-in acceleration sensor. Therefore, there is a property that no accumulation error occurs in the tilt angle direction. In other words, the azimuth angle direction (yaw angle direction) has an error (azimuth error) accumulated with the passage of time. Further, since the sensor coordinate system of the posture sensor is set with reference to the gravity axis, the sensor coordinate systems of the plurality of posture sensors are defined in common, leaving an azimuth component that is one degree of freedom around the gravity axis.

Hereinafter, a posture measurement value of three degrees of freedom measured by the posture sensor 110 is expressed as a 3 × 3 rotation matrix R _WSaτ . Similarly, a posture measurement value of three degrees of freedom measured by the posture sensor 115 is expressed as a 3 × 3 rotation matrix R _WSbτ . The posture measurement values measured by the posture sensors 110 and 115 are input to the posture measurement value acquisition unit 145 as data.

  When the posture measurement value acquisition unit 145 receives a request from the data management unit 155, the posture measurement value data received from each of the posture sensors 110 and 115 is output to the data management unit 155. In addition, the posture measurement value acquisition unit 145 receives the arrangement information (Local Transform) of each of the posture sensors 110 and 115 calculated by the calibration unit 160 described later from the calibration unit 160, and holds this. Further, the posture measurement value acquisition unit 145 converts the posture measurement values of the posture sensors 110 and 115 into information indicating the postures of the imaging device 120 and the real object 125, respectively, and outputs the converted information to the position and posture calculation unit 170. To do. Such conversion is performed by multiplying the posture measurement values of the posture sensors 110 and 115 by the corresponding Local Transform.

  Note that the position and orientation of the imaging device 120 are defined by the position and orientation of the camera coordinate system set on the imaging device 120. The camera coordinate system is a coordinate system in which the projection center of the imaging system of the imaging device 120 is the origin and the optical axis direction is the -Z axis, and the X axis and the Y axis are mutually independent axes orthogonal to the Z axis. Defined. Further, the position and orientation of the real object 125 are defined by the position and orientation of the object coordinate system set on the real object 125. The object coordinate system is a coordinate system in which one point in the real object 125 is defined as an origin and three axes orthogonal to each other are defined as an X axis, a Y axis, and a Z axis, respectively.

  The imaging device 120 captures a moving image of a real space including the real object 125, and the captured image (captured image) of each frame is input to the image acquisition unit 130 as data. Upon receiving the captured image data from the imaging device 120, the image acquisition unit 130 stores the data as necessary, and then outputs the data to the image processing unit 135 and the image generation unit 180.

  Here, a plurality of indices Qk (k = 1,,, K) whose positions (object coordinates) in the object coordinate system are known are arranged on the real object 125 as indices for imaging by the imaging device 120. Has been. Here, k represents the identification number of the index, and K represents the total number of indices. It is desirable that the index Qk is arranged so that at least four or more indexes can be simultaneously observed by the imaging device 120 when acquiring sensor calibration data. In the example of FIG. 1, four indexes Q1, Q2, Q3, and Q4 are arranged on the same plane of the real object 125.

  The indicator Qk is constituted by, for example, a color marker composed of small areas each having a different color. Alternatively, each feature point is a feature point such as a natural feature having different texture features. As described above, as an index that can be used in the present embodiment, an index that can detect the image coordinates of a projected image on a captured image and that can be identified by any method. As long as it is anything. The indicator may be set intentionally (artificially). In addition, for example, features that originally exist on the real object such as corners or patterns of the real object may be used as indices. Further, a polygon marker formed by a monochrome polygonal area having a certain area may be used. In this case, each vertex of the polygon is treated as an index.

The three-dimensional coordinate holding unit 140 holds a set of identification numbers and object coordinates for each index (Q1 to Q4 in the case of FIG. 1) arranged in the real space. That is, in the case of FIG. 1, the set of the identification number of the index Q1 and the object coordinates, the set of the identification number of the index Q2 and the object coordinates, the set of the identification number of the index Q3 and the object coordinates, and the identification number of the index Q4 A set with the object coordinates is held in the three-dimensional coordinate holding unit 140 as data. When receiving the identification number from the image processing unit 135, the three-dimensional coordinate holding unit 140 returns the object coordinates held as a set with this identification number to the image processing unit 135. In the following, the object coordinates of the index Qk with the identification number k are denoted as xO ^Qk .

  When the image processing unit 135 receives a captured image from the image acquisition unit 130, the image processing unit 135 detects an index included in the captured image and obtains the image coordinates. Furthermore, the image processing unit 135 identifies the detected index and identifies the identification number of the index.

  Here, the detection of the index is performed by a method according to the type of the index to be used. For example, when each of the indices is composed of color markers having different colors, an area corresponding to each marker color is detected from the captured image. Then, the position of the center of gravity is set as the detection coordinate of the index. Further, when each of the indices is composed of feature points having different texture features, the position of the index is detected by template matching using a template image. In this case, the template image of each index is held in advance as known information. When a square marker is used, the captured image is subjected to binarization processing and labeling processing, and a label area formed by four straight lines is detected as a marker candidate. Furthermore, erroneous detection is eliminated by determining whether or not there is a specific pattern in the rectangular area of each marker candidate. Further, the identification number of the rectangular marker is acquired by the pattern in the rectangular area. Finally, the coordinates of the four vertices of the rectangular area are output as the positions of the four indices.

  As described above, the technique for extracting the identification number of the index in the captured image and the image coordinate is a well-known technique in the relevant field, and therefore, a more detailed description of the process for acquiring the image coordinate and the identification number is provided. Omitted.

  When the identification number of the detected index is specified, the image processing unit 135 sends the specified identification number to the three-dimensional coordinate holding unit 140 in order to obtain the object coordinates of the detected index. As a result, the object coordinates set as a set with the identified identification number are returned from the three-dimensional coordinate holding unit 140, and are acquired. That is, the image processing unit 135 can obtain the object coordinates of the detected index from the three-dimensional coordinate holding unit 140 by sending the identification number of the detected index to the three-dimensional coordinate holding unit 140. In this way, the image processing unit 135 obtains identification numbers, image coordinates, and object coordinates for all indices detected from one captured image.

Then, the image processing unit 135 ^outputs a set (j = 1,,, Jτ) of the image coordinates u ^Qkjτ and the object coordinates ^{xO Qkj for} each detected index to the position and orientation calculation unit 170. Hereinafter, each detected set is referred to as correspondence information, and a set of all detected indices (a set of image coordinates and object coordinates) is collectively referred to as index information Mτ. Therefore, the index information Mτ includes correspondence information for each index.

  Here, the symbol τ is an identifier representing the time (timing) when the image acquisition unit 130 acquires the captured image from the imaging device 120. Therefore, Mτ represents index information obtained from the captured image acquired by the image acquisition unit 130 at the time represented by the identifier τ (hereinafter simply referred to as time τ). The time τ is measured by a timer (not shown), and each part of the sensor calibration apparatus 100 can appropriately acquire the time measured by the timer. Of course, anything other than the time can be used as long as it can represent “timing”.

  A symbol j indicates a serial number assigned to each index included in one captured image. Such a symbol j appears for explanation. The symbol Jτ represents the total number of indices detected by the image processing unit 135 from the captured image acquired by the image acquisition unit 130 at time τ. The symbol kj represents the identification number of the jth index.

The image processing unit 135 further calculates the estimated value t ′ _COτ of the position of the real object 125 relative to the imaging device 120 and the estimated value R ′ _COτ of the posture using the index information Mτ. Here, t ′ _COτ represents a three-dimensional vector representing the position. R ′ _COτ represents a 3 × 3 matrix representing the posture.

  Since a method for obtaining the position and orientation of a real object from the correspondence between the three-dimensional coordinates of a plurality of points on the real object and the image coordinates is widely known in the field of photogrammetry and computer vision, a detailed description of the method is provided. Is omitted. Note that the calculation of the position and orientation of the real object 125 with respect to the imaging device 120 is different from the processing in the position / orientation calculation unit 170 described later, and the orientation measurement values obtained by the orientation sensors 110 and 115 are not used.

When receiving a request from the data management unit 155, the image processing unit 135 outputs the index information Mτ, the estimated value t ′ _COτ of the position of the real object 125, and the estimated value R ′ _COτ of the _posture to the data management unit 155.

  The instruction unit 150 transmits a “data acquisition” instruction to the data management unit 155 when a “data acquisition” command is input by the operator. Further, when the “calculation of arrangement information” command is input, an instruction of “calculation of arrangement information” is transmitted to the calibration unit 160. The command input to the instruction unit 150 is performed by pressing a key to which a specific command is assigned, for example, using a keyboard. Further, the command may be input by any method such as a GUI displayed on the display.

When the data management unit 155 receives an instruction of “data acquisition” from the instruction unit 150, the data management unit 155 performs an image processing unit based on a captured image acquired by the image acquisition unit 130 at a time (time τ) closest to the time when the instruction is received. The information group acquired by 135 is acquired. That is, the index information Mτ, the estimated value t ′ _COτ of the position of the real object 125, and the estimated value R ′ _COτ of the posture are acquired from the image processing unit 135. In addition, when receiving a “data acquisition” instruction from the instruction unit 150, the data management unit 155 acquires data of posture measurement values obtained by the posture sensors 110 and 115 at time τ.

  Then, the data management unit 155 sets each piece of information acquired in this way as a set and additionally registers it in the data list managed by itself.

The posture measurement value R _WSaτ of the posture sensor 110 acquired by the posture measurement value acquisition unit 145 at time τ.
The posture measurement value R _WSbτ of the posture sensor 115 acquired by the posture measurement value acquisition unit 145 at time τ.
The estimated value t ′ _COτ of the position of the real object 125 obtained based on the captured image acquired by the image acquisition unit 130 at time τ, and the estimated value R ′ _{COτ of the} posture
Index information Mτ obtained based on the captured image acquired by the image acquisition unit 130 at time τ
・ Set of image coordinates u ^Qk1τ and object coordinates ^{xO Qk1} of the first index ・ Set of image coordinates u ^Qk2τ and object coordinates ^{xO Qk2} of the second index ・…
When the data management unit 155 receives a request from the calibration unit 160, the data management unit 155 sends the data list to the calibration unit 160.

  When the calibration unit 160 receives an instruction of “placement information calculation” from the instruction unit 150, the calibration unit 160 requests the data management unit 155 to acquire the data list. The data management unit 155 acquires the data list sent in response to this acquisition request. The calibration unit 160 uses the acquired data list to perform numerical calculation processing described later, thereby obtaining the placement information of the orientation sensor 110 (that is, the local transform) and the placement information of the orientation sensor 115. The obtained arrangement information is sent as data to the posture measurement value acquisition unit 145, held there, and output to the outside as needed.

  The above is a configuration for simultaneously executing the calibration of the attitude sensor 110 arranged on the imaging device 120 and the attitude sensor 115 arranged on the real object 125. The configuration described next is a part for presenting mixed reality for confirming the result obtained by the configuration.

  The position / orientation calculation unit 170 calculates the position and orientation of the real object 125 with respect to the imaging device 120 by the method disclosed in Non-Patent Document 1.

  That is, as described above, the image processing unit 135 obtains the index information Mτ based on the captured image every time a captured image for one frame is input from the imaging device 120, and calculates the position and orientation of the acquired index information Mτ. To the unit 170. Therefore, the position / orientation calculation unit 170 receives the index information Mτ.

  In addition, the position / orientation calculation unit 170 acquires the posture measurement value acquisition unit 145 obtained as the posture of the imaging device 120 and the posture of the real object 125 at the time closest to the time τ (which may be regarded as almost the time τ). To do.

  Here, “the posture of the imaging device 120 at the time closest to the time τ” is obtained as follows. That is, the posture measurement value of the posture sensor 110 acquired by the posture measurement value acquisition unit 145 at the time closest to the time τ is used by using the arrangement information of the posture sensor 110 held by the posture measurement value acquisition unit 145 at this time. It is obtained as described above. The “posture of the real object 125 at the time closest to the time τ” is obtained as follows. That is, the posture measurement value of the posture sensor 115 acquired by the posture measurement value acquisition unit 145 at the time closest to the time τ is used by using the arrangement information of the posture sensor 115 held by the posture measurement value acquisition unit 145 at this time. It is obtained as described above.

  Then, the position / orientation calculation unit 170 uses the index information Mτ, the orientation of the imaging device 120, and the orientation of the real object 125, and uses the technique disclosed in Non-Patent Document 1 to determine the real object 125 with respect to the imaging device 120. Calculate position and orientation. Since this calculation is performed based on the assumption that the given Local Transform is correct, the obtained position and orientation reflect the accuracy of the Local Transform.

  The image generation unit 180 acquires a captured image from the image acquisition unit 130. In addition, the position and orientation of the real object 125 with respect to the imaging device 120 are acquired from the position / orientation calculation unit 170, and a virtual object image drawn based on the acquired position and orientation is superimposed on the captured image, so that mixed reality is obtained. Is generated (superimposed image). Then, the image generation unit 180 outputs the generated image to the display unit 190.

  The display unit 190 acquires an image that presents mixed reality from the image generation unit 180, and displays this to present the mixed reality to the operator. The operator can visually check the image and judge the accuracy of the calibration result with the accuracy of the alignment between the real space and the virtual space in the presented mixed reality. Note that the display unit 190 may be integrated with the imaging device 120. In that case, the imaging device 120 and the display unit 190 can constitute a so-called head-mounted display.

  Next, main processing performed by the sensor calibration apparatus 100 will be described below with reference to FIG. 3 showing a flowchart of the processing. 3, the operator inputs a “data acquisition” command several times while changing the postures of the imaging device 120 and the real object 125 in various ways. Then, an operation of inputting a “placement information calculation” command is performed. In the following, the symbol τ is used as the time when the “data acquisition” command is input. When the time τ is different, the postures of the imaging device 120 and the real object 125 are different.

  In step S3010, the instruction unit 150 determines whether or not a “data acquisition” command is input from the operator. When the “data acquisition” command is input, the instruction unit 150 transmits a “data acquisition” instruction to the data management unit 155, and shifts the processing to step S3020. On the other hand, if the “data acquisition” command has not been input, the process proceeds to step S3050.

In step S3020, the data management unit 155 acquires the image processing result acquired by the image processing unit 135 from the image processing unit 135 based on the captured image acquired by the image acquisition unit 130 at time τ. That is, as described above, the image processing results are the index information Mτ, the estimated value t ′ _COτ of the position of the real object 125, and the estimated value R ′ _COτ of the posture.

In step S3030, the data management unit 155 _acquires the posture measurement values R _WSaτ and R _WSbτ measured by the posture sensor 110 and the posture sensor 115 at the time τ from the posture measurement value acquisition unit 145.

In step S3040, the data management unit 155 additionally registers in the data list L a set [R _WSaτ , R _WSbτ , t ′ _COτ , R ′ _COτ , M _τ ] that is a combination of the information groups acquired in steps S3020 and S3030. .

  A set acquired by one “data acquisition” instruction is registered in the data list L by the processing from step S3010 to step S3040 described above. A plurality of sets can be registered in the data list L by executing the processing from step S3010 to step S3040 a plurality of times. At this time, the time τ (τ = 1,, T) can be interpreted as an identifier of each set registered in the data list L. Here, T represents the number of different sets (different time and different postures) included in the data list L.

  In step S3050, the instruction unit 150 determines whether or not the “placement information calculation” command is input by the operator. When the “placement information calculation” command is input, the instruction unit 150 transmits the “placement information calculation” instruction to the calibration unit 160 and shifts the processing to step S3060. On the other hand, if the “placement information calculation” command has not been input, the process proceeds to step S3080.

  In step S <b> 3060, the calibration unit 160 acquires the data list L from the data management unit 155, and performs processing for calculating the arrangement information of each of the attitude sensors 110 and 115 using the acquired data list L. Details of this processing will be described later with reference to the flowchart of FIG.

  Next, in step S3070, the calibration unit 160 outputs the arrangement information of the posture sensors 110 and 115 calculated in step S3060 to the posture measurement value acquisition unit 145. Further, the arrangement information is output to the outside of the sensor calibration apparatus 100 as necessary.

  In step S3080, the position / orientation calculation unit 170 acquires the index information Mτ acquired by the image processing unit 135 from the image processing unit 135 based on the captured image acquired by the image acquisition unit 130 at time τ.

  In step S <b> 3090, the position / orientation calculation unit 170 acquires what the orientation measurement value acquisition unit 145 has obtained as the orientation of the imaging device 120 and the orientation of the real object 125 at time τ. Each acquired posture represents the posture after coordinate conversion is performed on the posture measurement value based on the local transform of the posture sensor held by the posture measurement value acquisition unit 145. Therefore, if the arrangement information calculation process in step S3060 is correctly performed, the posture obtained in this step is correct between the posture of the imaging device 120 and the posture of the real object 125 (of course, sensor measurement errors are included). To represent.

  In step S3100, the position / orientation calculation unit 170 determines the position of the real object 125 relative to the imaging device 120 based on the index information Mτ acquired in step S3080, the orientation of the imaging device 120 acquired in step S3090, and the orientation of the real object 125. Calculate the posture. Such calculation processing can be performed by, for example, a known method disclosed in Non-Patent Document 1, and thus detailed description thereof is omitted. Then, the position / orientation calculation unit 170 outputs the calculated position and orientation data to the image generation unit 180. Moreover, the calculated position and orientation data are output to the outside as necessary.

  In step S <b> 3110, the image generation unit 180 generates an image that presents mixed reality and outputs the generated image to the display unit 190. An image that presents mixed reality is generated by the following process. First, the image generation unit 180 acquires the position and orientation of the real object 125 with respect to the imaging device 120 from the position / orientation calculation unit 170. Further, the captured image acquired by the image acquisition unit 130 at time τ is acquired from the image acquisition unit 130. Then, a virtual object image drawn based on the position and orientation of the real object 125 with respect to the imaging device 120 is superimposed on the captured image. As a virtual object, for example, arrows representing three axes of the object coordinate system may be displayed as information independent of the type of the real object 125. Further, the X = 0 plane, the Y = 0 plane, the Z = 0 plane, and the like may be displayed in a wire frame. Alternatively, a wire frame model of the real object 125 may be held in advance, and this may be superimposed and displayed on the real object 125. Any virtual object may be used as long as it can be confirmed that the position and orientation of the real object 125 are correctly obtained.

  Next, in step S3120, the display unit 190 obtains an image that presents mixed reality from the image generation unit 180, and displays this to present the mixed reality to the operator.

  Finally, in step S3130, it is determined whether or not to end the process. If the operator inputs an instruction to end the process via the instruction unit 150, the process ends. On the other hand, if there is no instruction to end the process, the process returns to step S3010 again.

  Next, details of the arrangement information calculation processing in step S3060 will be described using the flowchart shown in FIG. FIG. 4 is a flowchart showing details of the processing in step S3060.

The calibration unit 160 treats the orientation information of the attitude sensor to be obtained, that is, the local transform, as a ternary vector ω = [ξ ψ ζ] ^T. There are various methods for expressing the posture by ternary values, but here, the rotation angle is expressed by the magnitude of the vector, and the ternary vector is defined by defining the rotation axis direction by the vector direction. It shall be. In the following description, each Local Transform is represented by a 3 × 3 rotation matrix R (ω) as necessary. Conversion between the rotation matrix and the ternary vector can be easily performed by a known conversion formula. Hereinafter, the arrangement information of the attitude sensor 110 is expressed as ω _SC = [ξ _SC ψ _SC ζ _SC ] ^T (unknown), and the arrangement information of the attitude sensor 115 is expressed as ω _SO = [ξ _SO ψ _SO ζ _SO ] ^T (unknown). To do.

In the process of calculating Local Transform, the calibration unit 160 estimates the position t _COτ = [x _COτ y _COτ z _COτ ] ^T (unknown) of the real object 125 at each time τ registered in the data list L. Further, a correction value φτ (unknown) of the relative azimuth error at each time τ is estimated. Here, the relative azimuth error represents the relative value of the azimuth error included in the posture measurement value of the posture sensor 110 and the azimuth error included in the posture measurement value of the posture sensor 115. The relative azimuth error includes a difference in azimuth component in the sensor coordinate system of each posture sensor.

Hereinafter, describing the unknown parameters of these calculation target in the (6 + 4T) dimensional state vector _{s = [ω SC ω SO t} CO1 φ 1 ··· t COτ φ τ ··· t COT φ T] T. In the following calculation, the value of the state vector s is optimized by repetitive calculation using the index information and posture measurement values at a plurality of times held in the data list L. Specifically, the value of s so that the error between the actual value of the image coordinate of each detection index and the theoretical value of the image coordinate of this index when a certain state vector s is assumed is minimized as a whole. Update.

  In step S4010, the calibration unit 160 acquires the data list L from the data management unit 155.

In step S4020, the calibration unit 160 gives an appropriate initial value to the state vector s. First, as the initial value of t _COτ at each τ, the estimated position t ′ _COτ of the real object 125 at the corresponding time τ in the data list L is set. In addition, as an initial value common to all _φτ , a value that has not yet been selected is selected from values obtained by equally dividing the entire circumferential direction. For example, when the entire circumferential direction is divided into six, one of 0 degrees, 60 degrees, 120 degrees, 180 degrees, 240 degrees, and 300 degrees is set. In addition, as an initial value of ω _SC, a posture that has not yet been selected as a combination with the initial value of φ _τ is selected and set from several candidates prepared in advance. Further, using the initial values of φ _τ and ω _SC set above as the initial value of ω _SO , the posture calculated by the following equation is set.

Here, each of R _WSaτ , R _WSbτ and R ′ _COτ represents a value obtained from one set (at a certain time τ) selected from the data list L. ΔR (φ _τ ) represents a 3 × 3 rotation matrix that adds rotation by φ _{τ in the} azimuth direction (correction of relative azimuth error), and is defined by the following equation.

  On the other hand, R (ω) is a rotation matrix determined by ω and is defined by the following equation.

  However, (theta) is defined by following Formula.

In step S4030, the calibration unit 160 calculates the theoretical value of the index image coordinates for each piece of correspondence information registered in the data list L. The theoretical value of the image coordinate of the index indicates the coordinate at which this index should be visible in the captured image when a hypothesis is given to the position and orientation of the real object 125 with respect to the imaging device 120. Hereinafter, the theoretical value of the image coordinate regarding the set of the object coordinate xO _i of the index and the image coordinate (actually measured value) u _i is expressed as u _i ′. Here, the symbol i (i = 1,,, I) is a serial number assigned to each of the correspondence information of the index in the data list L, and I represents the total number of correspondence information of the index. That is, a relationship of I = Σ τ J _τ.

The theoretical value u _i ′ is calculated by the following equation based on the observation equation F _τi () with the state vector s as a variable.

Here, τ _i represents the time of the set to which the i-th correspondence information belongs. The observation equation F _τi () is configured by conversion from object coordinates _{xO i} to camera coordinates x _Ci and conversion from camera coordinates x _Ci to image coordinates u _i ′. The former is model view transformation, and is defined by the following equation using ω _SC , ω _SO , t _COτi , and φ _τi that are elements of the state vector s.

  On the other hand, the latter is perspective transformation and is defined by the following equation.

  Here, fx and fy are the focal lengths of the imaging device 120 in the x-axis direction and the y-axis direction, respectively, and are assumed to be held in advance in the calibration unit 160 as known values.

In step S4040, the calibration unit 160, for each of the index correspondence information (that is, for all i), the error between the image coordinates u _i obtained as the actual measurement values and the theoretical values u _i ′ of the corresponding image coordinates. Δu _i is calculated by the following equation.

In step S4050, the calibration unit 160 calculates an image Jacobian J _i (= ∂u _i / ∂s) related to the state vector s for all i. The image Jacobian is a matrix (Jacobi matrix) of 2 rows × (6 + 4T) columns each having a solution obtained by partial differentiation of the observation equation F _τi () of (Expression 5) with respect to each element of the state vector s. Specifically, first, a 2-row × 3-column Jacobian matrix J _u — xCi (= ∂u _i) having a solution obtained by partial differentiation of the right side of (Expression 7) with each element of the position vector x _Ci on the camera coordinate system. / ∂x _Ci ) is calculated. Further, a Jacobian matrix J _{xCi_s} (= ∂x _Ci / ∂s) of 3 rows × (6 + 4T) columns having a solution obtained by partial differentiation of the right side of (Expression 6) with each element of the state vector s is calculated. Then, the image Jacobian J _i is calculated by the following equation.

Since the method for deriving the Jacobian matrix _{Ju_xCi} is known, its description is omitted. For example, Non-Patent Document 2 and the like disclose its derivation method. On the other hand, J _{xCi_s} is calculated as follows. In order to explain the following calculation, (Expression 6) representing the model view conversion is considered as the following four expressions.

Following these equations, J _{xCi_s} can be decomposed into two matrices for ω _SC and other elements ( _denoting this as s ′) as follows:

Here, J _{xCi_ωSC} is a partial differentiation of (Expression 10) by ω _SC and can be derived from (Expression 10) and (Expression 11) as follows.

Here, J _{xCi_rWCτ} represents the partial differentiation of (Equation 10) by each element of R _WCτi . J _{rWCτ_rSC} represents the partial differentiation of (Equation 11) by each element of R (ω _SC ). _J rSC_ωSC represents the partial derivative of R (ω _SC) due to the ω _SC. J _{xCi_xWi} represents the partial differentiation of (Equation 10) by x _Wi . J _{xWi —} s ′ represents the partial differentiation of (Expression 12) by s ′, and can be obtained by decomposing into each element of s ′ as in the following expression.

Here, J _{XWi_omegaSO} are partial derivatives by omega _SO (12), (12), and can be derived from equation (13).

Here, J _{xWi_rWOτ} represents the partial differentiation of (Equation 12) by each element of _RWOτi . J _{rWOτ_rSO} represents the partial differentiation of (Equation 13) by each element of R (ω _SO ). J _{RSO_omegaSO} represents the partial derivative of R (ω _SO) by omega _SO. J _{xWi_tτ} represents the partial differentiation of (Expression 12) by t _COτi and can be derived from (Expression 12) as follows.

J _{xWi — φτ} represents the partial differentiation of (Expression 12) by _φτi , and can be derived from (Expression 12) and (Expression 13) as follows.

Here, J _{rWOτ_Δrτ} represents the partial differentiation of (Equation 13) by each element of R (φ _τi ). J _{Derutaarutau_faitau} represents the partial derivative of R _(φ τi) by phi _.tau.i.

J _{xCi_s} can be obtained by sequentially calculating these Jacobian matrices. It should be noted that derivation of further equations is possible because it is possible by general partial differential calculations.

In step S4060, the calibration unit 160 calculates the correction value Δs of s based on the error Δu _i and the Jacobian matrix J _i for all i calculated in the above steps. Specifically, an error vector U in which errors Δu _i for all _i are vertically arranged and a matrix Φ in which a Jacobian matrix J _i for all _i is vertically arranged are created, and Δs is calculated by the following equation.

Here, Φ ⁺ is a pseudo inverse matrix Φ ⁺ of Φ, and can be obtained by, for example, Φ ⁺ = (Φ ^T Φ) ⁻¹ Φ ^T. The derivation of Φ ⁺ may be performed by any other method.

  Note that calculating the correction value by the processing in step S4060 described above corresponds to applying the Gauss-Newton method, which is a kind of nonlinear optimization method. A correction value calculation method using the error vector U and the Jacobian matrix Φ has been proposed in addition to the Gauss-Newton method. For example, the Levenberg-Marquardt method, the steepest descent method, or the like may be used. In addition, a robust estimation method such as M estimation that excludes outliers may be combined, or any other solution may be used.

  In step S4070, the calibration unit 160 corrects the state vector s according to the following equation using the correction value Δs calculated in step S4060, and sets the obtained value as a new s.

  In step S4080, the calibration unit 160 uses the updated state vector s to calculate the theoretical value of the image coordinate for each of the index correspondence information by the same processing as in step S4030.

In step S4090, using the updated state vector s, the calibration unit 160 calculates an error Δu _i between the measured value and the theoretical value of the image coordinate for each of the index correspondence information by the same processing as in step S4040. To do.

  In step S4100, the calibration unit 160 determines whether to continue the iterative process using some criterion. If it is determined that no further iterative processing is necessary, the process proceeds to step S4110. On the other hand, if it is determined that the iterative process is to be performed, the processes from step S4050 to step S4090 are executed again using the updated s.

  The determination of whether or not to perform the iterative process is performed by determining whether or not the calculation has converged or diverged, for example. If it is determined that the calculation has neither converged nor diverged, the iterative process is continued. On the other hand, if it is determined that the calculation has converged or diverged, the iterative process is terminated. Further, it may be possible to always perform a certain number of iterations without determining convergence or divergence. The determination of convergence or divergence can be made, for example, based on the magnitude of the error vector U updated in step S4090. For example, the determination of convergence is performed based on a criterion such as whether or not the magnitude of the error vector U is smaller than a predetermined threshold, or whether or not the amount of change in the magnitude of the error vector U is smaller than a predetermined threshold. If both are smaller than the threshold value, it is determined that the process is to be terminated.

  Further, the divergence is determined based on whether or not the magnitude of the error vector U tends to increase or whether or not the magnitude of the error vector U is larger than a predetermined threshold. Further, the convergence determination can be performed based on the magnitude of the correction value Δs.

  Since it has been conventionally performed to perform the iterative calculation a finite number of times using the termination condition of such an iterative calculation, further description thereof will be omitted.

  In step S4110, the calibration unit 160 determines whether the optimization calculation (steps S4030 to S4100) performed on the initial value set in step S4020 has succeeded. Specifically, if the magnitude of the error vector U updated in step S4090 is equal to or smaller than a predetermined threshold value, it is determined that the optimization calculation has succeeded. If it is determined that the optimization calculation has failed, the process proceeds to step S4020. And another candidate is reset as an initial value of a state vector, and subsequent optimization calculation (Step S4030-Step S4100) is performed again. On the other hand, if it is determined that the optimization calculation is successful, the process is terminated.

  Through the above processing, the posture sensor mounted on the imaging device 120 or the real object 125 can be calibrated easily and accurately. Further, there is no need to input the initial value and the direction of the gravity axis in advance.

[Second Embodiment]
In the first embodiment, the orientation sensor 110 is attached to the imaging device 120. However, considering only the purpose of measuring the placement information of the orientation sensor 115 attached to the real object 125, it is not essential that the orientation sensor 110 is attached to the imaging device 120. In the present embodiment, a system in a case where no posture sensor is provided in the imaging apparatus 120 will be described.

  FIG. 5 is a block diagram showing a functional configuration of a system having the sensor calibration apparatus according to the present embodiment. In FIG. 5, the same components as those shown in FIG. The configuration shown in FIG. 5 is different from the configuration shown in FIG. 1 in the following three points.

The image sensor 120 is not provided with a posture sensor. The image sensor 120 is fixed in its posture with a tripod 500. The sensor calibration device 100 is newly provided with a posture acquisition unit 165. The explanation will focus on three points. Of course, the points other than those described below are the same as those in the first embodiment.

The imaging device 120 is installed on a tripod 500. The orientation R _WC of the imaging device 120 is measured in advance and is stored in the external memory 599 as data. In order to obtain the posture R _WC of the imaging device 120, several indexes whose known three-dimensional positions in the world coordinate system are arranged in the real space, and some of the indexes are imaged by the imaging device 120. Then, the image coordinates of the index are detected, and the orientation of the imaging device 120 in the world coordinate system may be calculated from the correspondence between the world coordinates of the index and the image coordinates. Note that as the posture R _WC of the imaging device 120, it is only necessary to obtain the inclination of the camera coordinate system with respect to the gravity axis (the axis indicating the opposite direction). That is, the direction in the world coordinate system may be an arbitrary direction, and the Y axis only needs to coincide with the opposite direction of the gravity axis.

The attitude acquisition unit 165 reads data of the attitude R _WC of the imaging device 120 held in the external memory 599. In the following processing, the posture measurement value R _WSaτ of the posture sensor 110 in the first embodiment may be replaced with the posture R _WC . In the first embodiment, the arrangement information ω _SC of the posture sensor 110 that was one of the derivation targets may be considered as a known unit matrix.

The subsequent processing is the same as that in the first embodiment. As a change from the first embodiment, components related to ω _SC are excluded from the state vector s and the Jacobian matrix J _i . Then, a process similar to the optimization calculation of the first embodiment is executed for this new s. Since ω _SC is known, the combination of initial values set in step S4020 is determined only by φ. That is, by selecting an angle with the initial value of phi, the initial value of omega _SO is uniquely determined by equation (1).

Note that the imaging device 120 is not necessarily fixed as long as the posture _RWC can be measured by some means. For example, the measurement may be continued by observing an indicator in the world coordinate system. Further, by attaching a sensor such as a magnetic sensor or an optical sensor to the imaging device 120, the position and orientation of the imaging device 120 may be measured by this sensor.

<Modification 1>
In the above embodiment, an index that represents one coordinate (hereinafter referred to as a point index) is used as an index. However, an index other than the point index can be used. For example, an index composed of line features such as edges (hereinafter referred to as a line index) may be used.

  An outline of processing when using a line index is described below. Even when the line feature is used, the framework for optimizing the state vector s using the index information at a plurality of times and the respective posture measurement values as input and using the projection error of the index as an evaluation criterion is the same. When the line index is used, the position and orientation of the real object 125 in the image processing unit 135 are estimated by a method disclosed in Non-Patent Document 3, for example.

  First, the calibration unit 160 calculates a theoretical value of image coordinates at which several points (referred to as division points) on the line index will be observed when the state vector is s. Then, the distance in the normal direction to the edge (actually measured value) actually detected in the vicinity of the image coordinates is defined as an error of the dividing point. Then, an error vector U is constituted by errors relating to all division points. On the other hand, the error derivation formula defined as a function of the state vector s is partially differentiated with respect to each element of s, and a 1 row × (6 + 4T) column Jacobian matrix Φ having the solution as each element is constructed.

  Then, the placement information can be calculated by performing the same optimization calculation as in the above embodiment using the error vector U and the Jacobian matrix Φ obtained above. Of course, it is also possible to combine these features by stacking line indexes, point indexes, errors obtained from other indexes, and image Jacobians.

<Modification 2>
In the above embodiment, the position t _COτ of the real object 125 at time τ is unknown in the optimization calculation in the calibration unit 160. However, the estimated value t ′ _COτ of the real object 125 at each time τ held in the data list L as a result of image processing is _set as t _COτ, and t _COτ is set as a known value in the optimization calculation step. May be handled. In this case, as a change from the above embodiment, a component related to t _COτ may be excluded from the state vector s and the Jacobian matrix J _i . For example, in the case of the first embodiment, the state vector s, is defined as _{s = [ω SC ω SO φ} 1 ... φ τ ... φ T] T. Then, the same processing as the optimization calculation in the above embodiment is executed for this s.

Further, in the above embodiment, in the optimization calculation of the calibration unit 160, it is assumed that the relative azimuth error of the posture sensor varies from time to time. However, the relative azimuth error may be assumed to be constant in a short time during which data for calibration is acquired. For example, in the case of the first embodiment, the state vector s can be defined as s = [ω _SC ω _SO φ t _CO1 ... t _COτ ... t _COT ] ^T. Then, a process similar to the optimization calculation of the above embodiment is executed for this s. Here, φ represents a relative azimuth error correction value that does not depend on time τ.

Of course, it is also possible to perform an optimization calculation using s = [ω _SC ω _SO φ] ^T as an unknown parameter under both assumptions described in this modification.

<Modification 3>
In the first embodiment, the arrangement information of the attitude sensor 110 and the arrangement information of the attitude sensor 115 are obtained simultaneously. However, when any one of the posture sensors has already been calibrated, the arrangement information of the calibrated posture sensor may be known. In this case, only the processing of the calibration unit 160 (step S3060) may be changed.

First, consider a case where the attitude sensor 115 has been calibrated and only the attitude sensor 110 has not been calibrated. In this case, since the arrangement information ω _SO of the attitude sensor 115 is known, a component related to ω _SO may be excluded from the state vector s and the Jacobian matrix J _i as a change from the first embodiment. Then, a process similar to the optimization calculation of the first embodiment is executed for this s. If ω _SO is known, the calculation of ω _SC in step S4020 may be performed according to the following equation.

At this time, when an angle is selected as the initial value of φ, the initial value of ω _SC is uniquely determined by (Equation 24).

Next, consider a case where the attitude sensor 110 has been calibrated and only the attitude sensor 115 has not been calibrated. In this case, since the arrangement information ω _SC of the posture sensor 110 is known, if the component related to ω _SC is excluded from the state vector s and the Jacobian matrix J _{i of} the first embodiment, as in the second embodiment. Good.

<Modification 4>
In the above-described embodiment, the posture sensor 110 and the posture sensor 115 are sensors that measure a triaxial posture. However, even if these posture sensors are sensors that measure only two tilt angles with respect to the gravity axis (generally called tilt sensors), it is not possible to calculate posture sensor arrangement information by the same method. it can. In this case, the parameter phi _tau in the optimization process, rather than a correction value of the relative azimuth error, would represent the difference between the relative azimuth angle between the imaging device 120 and physical object 125.

<Modification 5>
In the above embodiment, detection and identification of the index are performed by image processing. However, depending on the shape and situation of the real object 125, it may be difficult to prepare an index whose position on the object coordinate system is known and can be detected and identified by image processing. In such a case, index information may be manually input. In other words, the operator may input the index image coordinates using a pointing device such as a mouse on the captured image displayed on the monitor, and further input the index identification number. This is effective when the position on the object coordinate system is known and there is a point where the position can be visually identified on the image.

  Further, when only the detection of the index is possible by image processing, only the identification processing may be performed manually by the operator. In addition, an error in an index detected and / or identified by image processing may be corrected by manual input. Moreover, you may use combining the parameter | index detected and / or identified by image processing, and the parameter | index detected and / or identified by manual input.

[Third Embodiment]
The description has been given assuming that all the parts constituting the sensor calibration apparatuses 100 and 500 shown in FIGS. 1 and 5 are configured by hardware. However, some or all of the other units except for the configuration for holding data may be configured by software (computer program). In this case, the software is loaded onto a memory of a computer such as a PC (personal computer), and the CPU of the computer executes the software. Thereby, the computer can perform the same processing as the sensor calibration devices 100 and 500 shown in FIGS.

  FIG. 2 is a block diagram showing a hardware configuration of a computer applicable to the sensor calibration apparatuses 100 and 500.

  The CPU 1001 controls the entire computer using programs and data stored in the RAM 1002 and the ROM 1003, and executes the above-described processes performed by the sensor calibration apparatuses 100 and 500 to which the present computer is applied.

  The RAM 1002 has an area for temporarily storing programs and data loaded from the external storage device 1007 and the storage medium drive 1008 and programs and data received from the outside via the I / F (interface) 1009. The RAM 1002 also has a work area used when the CPU 1001 executes various processes. That is, the RAM 1002 can provide various areas as appropriate.

  The ROM 1003 stores setting data and a boot program for the computer.

  The keyboard 1004 and the mouse 1005 are examples of input devices for an operator of the computer to input various instructions to the CPU 1001, and other input devices may be used. For example, a command input to the instruction unit 150 is performed using the keyboard 1004 and the mouse 1005.

  The display unit 1006 corresponds to the display unit 190 and includes a CRT, a liquid crystal screen, or the like. The display unit 1006 can display the result of processing by the CPU 1001 using images, characters, and the like. For example, as described above, an image in which a captured image and a virtual object image are combined can be displayed.

  The external storage device 1007 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 1007 stores an OS (operating system) and programs and data for causing the CPU 1001 to execute the above-described processes performed by the sensor calibration devices 100 and 500. The external storage device 1007 also stores information described as known information in the above description.

  Programs and data stored in the external storage device 1007 are appropriately loaded into the RAM 1002 under the control of the CPU 1001. The CPU 1001 executes processes using the loaded program and data, so that the computer executes the above-described processes performed by the sensor calibration apparatuses 100 and 500.

  The storage medium drive 1008 reads programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and sends the read programs or data to the RAM 1002 or the external storage device 1007. A part of the information described as being stored in the external storage device 1007 may be recorded in the storage medium. The storage medium may function as the external memory 599.

  The attitude sensors 110 and 115 and the imaging device 120 are connected to the I / F 1009. The I / F 1009 includes the following multiple types of ports. A part of the function of the image acquisition unit 130 for acquiring a captured image is realized by an analog video port or a digital input / output port such as IEEE1394. In addition, a part of the function of the posture measurement value acquisition unit 145 that acquires the posture measurement value is realized by a USB port or an RS-232C port. Input data is taken into the RAM 1002 or the external storage device 1007 via the I / F 1009. Further, the arrangement information of each posture sensor calculated by the calibration unit 160 and the position and posture of the object calculated by the position / orientation calculation unit 170 may be an Ethernet (registered trademark) port or the like (function of the I / F 1009) as necessary. Is output to the outside.

  A bus 1010 connects the above-described units.

  Note that the hardware configuration of the computer applicable to the sensor calibration apparatuses 100 and 500 is not limited to the configuration shown in FIG. 2 and can be modified as appropriate by those skilled in the art.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

That is, at a plurality of time points, the posture measurement value obtained by the posture sensor for the imaging device attached to the imaging device and the posture measurement value obtained by the posture sensor for the real object attached to the real object imaged by the imaging device are acquired. Means for acquiring posture measurement values to be performed;
An image acquisition means for acquiring an image obtained by the imaging device imaging the real object at the plurality of time points ;
Image coordinate acquisition means for acquiring image coordinates on a plurality of images captured at the plurality of time points of a plurality of indices arranged on the real object and having known object coordinates representing positions on the real object When,
A position and orientation that calculates an estimated value of the position and orientation of the real object with respect to the imaging device at each of the plurality of time points based on the object coordinates for the plurality of indices and the image coordinates at each of the plurality of time points. A calculation means;
And layout information of the imaging device for the orientation sensor for the image capturing apparatus, and allocation information of the physical object for orientation sensor for the physical object, and the estimated value of the position of the physical object in each of said plurality of time points, the object the coordinates, in the plurality of time points, and the orientation measurement value measured by the orientation sensor for the image capturing apparatus, the orientation measurement value by the physical object for orientation sensor, and the image coordinates, the estimation of the position and orientation of the physical object relative to the imaging device And arrangement information calculation means for obtaining using the value .

Claims (9)

Posture measurement value acquisition means for acquiring a posture measurement value by a posture sensor for an imaging device attached to an imaging device and a posture measurement value by a posture sensor for a real object attached to a real object imaged by the imaging device When,
Image acquisition means for acquiring an image obtained by the imaging device imaging the real object;
Image coordinate acquisition means for acquiring image coordinates on the image of an index placed on the real object;
At least one of the arrangement information of the posture sensor for the imaging device with respect to the imaging device and the arrangement information of the posture sensor for the real object with respect to the real object is a posture measurement value by the posture sensor for the imaging device, for the real object An information processing apparatus comprising: a calculation unit that uses a posture measurement value obtained by a posture sensor and the image coordinates. The posture measurement value acquisition means acquires a posture measurement value obtained by the posture sensor for the imaging device and a posture measurement value obtained by the posture sensor for the real object each time the imaging device and the real object take different postures,
The acquisition means of the image, the image pickup device, the physical object, each taking a different posture, acquires an image in which the imaging device imaged,
The image coordinate acquisition means acquires the image coordinates of the index from each image acquired by the image acquisition means,
The calculation means includes an orientation measurement value obtained by the orientation sensor for the imaging device, an orientation measurement value obtained by the orientation sensor for the real object, acquired every time the imaging device and the real object have different orientations, and the imaging device Using the image coordinates of the index acquired from the captured image, at least one of the placement information of the orientation sensor for the imaging device with respect to the imaging device and the placement information of the orientation sensor for the real object with respect to the real object is calculated. The information processing apparatus according to claim 1. The calculating means includes
The coordinates of the index on the image determined on the basis of the unknown parameter with the calculation target of the arrangement information of the posture sensor for the imaging device with respect to the imaging device and the arrangement information of the posture sensor for the real object with respect to the real object The information processing apparatus according to claim 2, wherein the unknown parameter is obtained so as to reduce an error between the index and the image coordinate acquired by the image coordinate acquisition unit for the index. Attitude measurement value acquisition means for acquiring an attitude measurement value by an attitude sensor attached to a real object that does not have an imaging device;
An image acquisition means for acquiring an image obtained by an imaging device imaging the real object;
Image coordinate acquisition means for acquiring image coordinates on the image of an index placed on the real object;
Calculating means for obtaining the position information of the posture sensor with respect to the real object using the posture of the imaging device given as known information, the posture measurement value by the posture sensor, and the image coordinates. Information processing device. Means for generating an image of the virtual object based on the arrangement information calculated by the calculating means;
Means for generating a superimposed image in which the image of the virtual object is superimposed on the image captured by the imaging device;
The information processing apparatus according to claim 1, further comprising: means for displaying the superimposed image. An information processing method performed by an information processing apparatus,
The posture measurement value acquisition means of the information processing device is based on a posture measurement value obtained by a posture sensor for an imaging device attached to an imaging device and a posture sensor for a real object attached to a real object imaged by the imaging device. A posture measurement value acquisition step for acquiring posture measurement values;
An image acquisition step in which the image acquisition unit included in the information processing apparatus acquires an image obtained by the imaging apparatus imaging the real object;
An image coordinate acquisition unit in which the image coordinate acquisition unit of the information processing apparatus acquires image coordinates on the image of an index placed on the real object;
The calculation unit included in the information processing apparatus uses at least one of the arrangement information of the posture sensor for the imaging device with respect to the imaging device and the arrangement information of the posture sensor for the real object with respect to the real object as the posture for the imaging device. An information processing method comprising: a calculation step that uses a posture measurement value by a sensor, a posture measurement value by the physical object posture sensor, and the image coordinates. An information processing method performed by an information processing apparatus,
A posture measurement value acquisition step in which the posture measurement value acquisition means of the information processing apparatus acquires a posture measurement value by a posture sensor attached to a real object that does not have an imaging device;
An image acquisition step in which the image acquisition unit included in the information processing apparatus acquires an image obtained by the imaging apparatus imaging the real object;
An image coordinate acquisition unit that the image coordinate acquisition means of the information processing apparatus acquires the image coordinates on the image of an index placed on a real object;
The calculation unit included in the information processing apparatus obtains the position information of the posture sensor with respect to the real object using the posture of the imaging device given as known information, the posture measurement value by the posture sensor, and the image coordinates. An information processing method comprising: a calculation step. The computer program for functioning a computer as each means which the information processing apparatus of any one of Claims 1 thru | or 5 has .   A computer-readable storage medium storing the computer program according to claim 8.

Images