JP4726191B2 - Position / orientation correction apparatus, position / orientation correction method, program thereof, and storage medium - Google Patents

Description

  The present invention relates to a method and apparatus for measuring the position and orientation of an object, particularly the position and orientation of an imaging apparatus.

  In recent years, research on mixed reality has been actively conducted for the purpose of seamless connection between the real space and the virtual space. An image display device that presents a mixed reality is a virtual space image (for example, by computer graphics) that is generated according to the position and orientation of an imaging device on a real space image captured by an imaging device such as a video camera. It can be realized as a device that displays an image in which a drawn virtual object, character information, etc.) are superimposed and drawn.

  In order to realize such an image display device, a reference coordinate system defined in the real space (a coordinate system in the real space serving as a reference for determining the position and orientation of the virtual object to be superimposed) It is essential to measure the relative position and orientation with respect to the coordinate system (camera coordinate system). In order to draw a virtual object (virtual space image) according to a position in the real space, an image of the virtual object is generated using the same camera parameters as the real camera parameters of the imaging device with respect to the reference coordinate system. Because it must be. For example, when a virtual object is superimposed and displayed at a position in an actual room, a reference coordinate system is defined in the room, and the position and orientation of the imaging device in the reference coordinate system may be obtained. In addition, when a virtual pattern or label is superimposed and displayed on an actual box held by an observer, the object coordinate system of the box itself is considered as a reference coordinate system, and the box (reference coordinate system) of the image pickup apparatus is considered. What is necessary is just to obtain | require a position and attitude | position.

  As a method for measuring the position and orientation of the imaging device, a plurality of indices (artificial markers, natural features, etc.) are arranged or set in the real space, and the coordinates of the projected image of the indices in the image captured by the imaging device In general, the position and orientation of the imaging device are obtained based on the relationship with the coordinate information of the index (for example, Non-Patent Document 1). However, when using this approach, there is a restriction that the index must always be imaged.

  On the other hand, a 6-DOF position / orientation sensor using a magnetic sensor, an ultrasonic sensor, or the like is attached to the image pickup device, and the information obtained by using the image obtained by imaging the index of the position and orientation error of the image pickup device measured thereby. Attempts to correct by (image information) have been made (for example, Patent Document 1 and Patent Document 2). In the method disclosed in Patent Document 2, if an index is detected in the captured image, the error of the sensor measurement value is corrected based on the information, and if the index is not detected, the 6-free-position / posture sensor is detected. The measured value is used as it is as the position and orientation of the imaging apparatus. Since the position and orientation of the imaging device can be obtained regardless of whether or not the index is detected, it is possible to stably present the mixed reality.

  In the method of Patent Document 2, when the number of detected indices is three or more, six degrees of freedom of the position and orientation of the imaging device are calculated based on the image information, and the number of detected indices is In the case of two points and one point, a process for correcting only one of the position and orientation of the imaging apparatus measured by the sensor (2 or 3 degrees of freedom) is applied. That is, the algorithm used for calculating the position and orientation of the imaging apparatus is switched using the number of detected indices as a criterion. As a result, even when the position and orientation of the imaging device cannot be obtained from only the image information (when the number of captured images is less than 3 points), the error is reduced based on the sensor measurement value. It is possible to acquire a position and orientation that have been corrected to cancel as much as possible.

  On the other hand, in the method of Patent Document 1, a process of correcting only one of the position and orientation of the imaging device measured by the sensor based on the image information is applied regardless of the number of detected indices. In this correction method, when correcting the posture, a correction value for the posture measurement value is calculated by individually obtaining a rotation correction value that cancels the error on the indicator for each detected indicator and averaging them. Is done. In addition, when correcting the position, a parallel movement correction value that cancels the error on the index is obtained for each detected index, and the correction value for the position measurement value is calculated by averaging them. . Since the degree of freedom of correction is limited to 2 or 3 regardless of the number of indices, a stable solution can be obtained even when the amount of information is insufficient.

Kato et al: “Augmented Reality System Based on Marker Tracking and Its Calibration”, Transactions of the Virtual Reality Society of Japan, vol.4, no.4, pp.607-616, 1999. J. Park, B. Jiang, and U. Neumann: "Vision-based pose computation: robust and accurate augmented reality tracking," Proc. 2nd International Workshop on Augmented Reality (IWAR'99), pp.3-12, 1999. D. G. Lowe: "Fitting parameterized three-dimensional models to images," IEEE Transactions on PAMI, vol.13, no.5, pp.441-450, 1991. JP 2003-222509 A JP 2003-279310 A JP2003-344018

  In the method of the above-mentioned Patent Document 2, whenever three or more indices are observed, an algorithm for obtaining 6 degrees of freedom of position and orientation from image information is selected. However, in reality, even if the number of indices is 3 or more, as in the case where the indices are unevenly distributed in a part of the image, the 6 degrees of freedom of the position and orientation can be stabilized. There are situations where the input image does not contain enough image information to find. In such a situation, the method of Patent Document 2 sometimes has insufficient accuracy and stability of the obtained solution.

  On the other hand, the method of Patent Document 1 is a method in which stability is more important than accuracy, and there is insufficient image information as in the case where an index is unevenly observed in a part of an image. Even if it is below, it can measure stably compared with the method of patent document 2. FIG. However, even when sufficient image information is obtained, only correction of some parameters is performed, and thus it is impossible to obtain the accuracy that makes the best use of the image information.

  Further, since the method of Patent Document 1 only calculates the average of two-dimensional correction values for each index, optimal correction is performed so as to minimize the sum of errors on all indices. I couldn't.

  The present invention has been made in view of such problems of the prior art, and a position and orientation measurement method capable of measuring the position and orientation of a measurement target object while achieving both stability and accuracy, and An object is to provide an apparatus.

The above-described object is a position / orientation correction apparatus that corrects one or more of a plurality of parameters constituting the position / orientation of the imaging apparatus, and a plurality of position / orientation input means for inputting a plurality of parameters and a plurality of parameters are arranged in the real space. An image input unit that inputs an image obtained by imaging an index with an imaging device, a detection unit that detects an actual measurement value of the image coordinates of the index from the image, and which of a plurality of parameters is to be corrected based on a detection result by the detection unit Based on the plurality of parameters and the actual measurement values, the parameter to be corrected is determined as the parameter to be corrected without correcting the parameter that has not been determined as the parameter to be corrected by the determining unit. Correction means for correcting the parameters, the correction means based on the index coordinates and a plurality of parameters, the theoretical value of the index image coordinates A theoretical value calculating means for calculating, a correction value calculating means for calculating a correction value of a parameter determined as a correction target so as to reduce an error between the actual measurement value and the theoretical value, and a plurality of parameters based on the correction value This is achieved by a position / orientation correction apparatus comprising: a parameter to be corrected and a parameter correction unit that corrects the determined parameter without correcting the parameter to be corrected and the parameter that has not been determined .

The above-described object is a position / orientation correction apparatus that corrects a plurality of parameters constituting the position / orientation of a measurement target object, and is provided with a plurality of position / orientation input means for inputting a plurality of parameters and a plurality of measurement target objects. An image input unit that inputs an image obtained by imaging an index with an imaging device, a detection unit that detects an actual measurement value of the index image from the captured image, and which of a plurality of parameters is corrected based on a detection result by the detection unit Based on a plurality of parameters and actual measurement values, a determination unit that determines whether to be a target, a parameter that has not been determined as a correction target parameter by the determination unit, is determined as a correction target parameter without correcting the parameter. Correction means for correcting the parameter, and the correction means calculates a theoretical value of the image coordinate of the index based on the plurality of parameters. A theoretical value calculating means, a correction value calculating means for calculating a correction value of the parameter to be corrected and the determined parameter so as to reduce an error between the actually measured value and the theoretical value, and a plurality of parameters based on the correction value And a parameter correction unit that corrects the parameter to be corrected and the determined parameter without correcting the parameter to be corrected and the parameter that has not been determined. .

The above-described object is a position / orientation correction apparatus that corrects a plurality of parameters constituting the position / orientation of the imaging apparatus, and a plurality of position / orientation input means for inputting a plurality of parameters are arranged in the real space, and the positions are Based on the detection result of the detection means by the image input means for inputting the image captured by the imaging device, the image input means for inputting a known index, the actual value of the image coordinates of the index from the image, and a plurality of parameters Based on a plurality of parameters and actual measurement values, a determination unit that determines which one is to be corrected, and a parameter that is not determined as a correction target parameter by the determination unit among the plurality of parameters is not corrected. A correction unit that corrects the determined parameter, and the correction unit is configured to display the index image based on the position of the index and the plurality of parameters. A theoretical value calculating means for calculating the theoretical value of the standard, a correction value calculating means for calculating the correction value of the parameter to be corrected and the determined parameter so as to reduce an error between the actually measured value and the theoretical value, and a correction value A correction target parameter and a parameter correction unit that corrects the determined parameter without correcting the correction target parameter among the plurality of parameters, and the plurality of parameters, Parameters corresponding to six components of three position components indicating positions and three posture components indicating postures, and when the determining means has one index detected by the detecting means, among the plurality of parameters When the parameters corresponding to two or less components are determined as parameters to be corrected, and there are two indices detected by the detection means, the image coordinates of the detected indices Based on the distance, a parameter corresponding to three or less components among a plurality of parameters is determined as a correction target parameter, and when there are three or more indices detected by the detection means, the image coordinates of the detected indices The position / orientation correction apparatus is characterized in that a parameter corresponding to six or less components among a plurality of parameters is determined as a correction target parameter based on the area of the region obtained from the above.

The above-described object is a position / orientation correction apparatus that corrects a plurality of parameters constituting the position / orientation of a measurement target object, and is provided with a plurality of position / orientation input means for inputting a plurality of parameters and a plurality of measurement target objects. An image input unit that inputs an image obtained by imaging an index with an imaging device, a detection unit that detects an actual measurement value of the index image from the captured image, and which of a plurality of parameters is corrected based on a detection result by the detection unit Based on a plurality of parameters and actual measurement values, a determination unit that determines whether to be a target, a parameter that has not been determined as a correction target parameter by the determination unit, is determined as a correction target parameter without correcting the parameter. Correction means for correcting the parameter, and the correction means calculates a theoretical value of the image coordinate of the index based on the plurality of parameters. A theoretical value calculating means, a correction value calculating means for calculating a correction value of the parameter to be corrected and the determined parameter so as to reduce an error between the actually measured value and the theoretical value, and a plurality of parameters based on the correction value A correction target parameter and parameter correction means for correcting the determined parameter without correcting the correction target parameter and the parameter that has not been determined, and the plurality of parameters include three position components indicating the position. And a parameter corresponding to six components of the three posture components indicating the posture, and when the determination unit has one index detected by the detection unit, a parameter corresponding to two or less components among the plurality of parameters Is determined as a parameter to be corrected, and when there are two indices detected by the detection means, based on the distance between the image coordinates of the detected indices, When a parameter corresponding to three or less components among the number of parameters is determined as a parameter to be corrected, and there are three or more indices detected by the detection means, the area of the region obtained from the image coordinates of the detected indices This is achieved by a position and orientation correction apparatus characterized in that a parameter corresponding to six or less components among a plurality of parameters is determined as a correction target parameter .

  According to the position and orientation measurement apparatus according to the present invention, a correction value that minimizes the error between the theoretical value and the actual measurement value of the image coordinate of the index is obtained, and the measurement value of the sensor is corrected by this correction value. Compared to the method, measurement with superior stability and accuracy can be realized.

Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.
[First Embodiment]
The position / orientation measurement apparatus according to the present embodiment measures the positions and orientations of the imaging apparatus and the measurement target object. Hereinafter, the position and orientation measurement apparatus and the position and orientation measurement method according to the present embodiment will be described.

  FIG. 1 shows a configuration of a position / orientation measurement apparatus according to the present embodiment. As shown in the figure, the position / orientation measurement apparatus 100 according to this embodiment includes an image input unit 160, an index detection unit 110, a sensor measurement value input unit 150, and a position / orientation calculation unit 120. And it is connected to the imaging device 130 and the position and orientation sensor 140 (140a, 140b) attached to the imaging device 130 and the measurement target object 170.

The plurality of positions on the measurement target object 170 include a plurality of indices Q _k (k = 1, 2,...) Whose positions in the object coordinate system (the coordinate system defined on the measurement target object 170) are known. , K). The example of FIG. 1 shows a situation in which K = 9, that is, nine indices Q _{1 to} Q ₉ are arranged. The index Q _k may be constituted by, for example, markers of the same shape (circular in the figure) each having a different color, or may be constituted by feature points such as natural features each having a different texture feature. It is also possible to use a square index formed by a rectangular single color area having a certain area. Any form may be used as long as the image coordinates of the projected image on the photographed image can be detected and the index can be identified by any method. The indicator may be set intentionally, or may be a natural shape that is not set intentionally.

  For example, an image output from the imaging device 130 that is a video camera (hereinafter referred to as a captured image) is input to the position and orientation measurement device 100.

  Position and orientation sensors 140a and 140b, which are six-degree-of-freedom sensors, are attached to the imaging device 130 and the measurement target object 170, respectively, and measure the positions and orientations of the imaging device 130 and the measurement target object 170 in the reference coordinate system. The measurement value output by the position / orientation sensor 140 is input to the position / orientation measurement apparatus 100. The position / orientation sensor 140 is constituted by, for example, FASTRAK of Polhemus, USA. Note that the position and orientation measured by the position and orientation sensor 140 include errors due to the influence of magnetic field distortion and the like.

  The image input unit 160 converts a captured image input to the position / orientation apparatus 100 into digital data and outputs the digital data to the index detection unit 110.

  The sensor measurement value input unit 150 inputs each measurement value from the position / orientation sensor 140 (140a, 140b), and outputs it to the position / orientation calculation unit 120.

Index detecting section 110 receives the captured image from the image input unit 160, detects the image coordinates of the indices Q _k has been taken in the input image. For example, when each of the indices Q _k is composed of markers having different colors, an area corresponding to each marker color is detected from the photographed image, and the barycentric position is set as the detected coordinates of the index. Further, when each of the indices Q _k is composed of feature points having different texture characteristics, template matching is performed on the captured image by using a template image of each index that is held in advance as known information. , Detect the position of the indicator. In addition, when a quadratic index is used, labeling is performed after binarizing the image, and an area formed by four straight lines is detected as an index candidate. Further, by detecting whether or not there is a specific pattern in the candidate area, false detection is eliminated, and the identifier of the index is acquired. In the present specification, the quadrangular indices detected in this way are considered to be four indices formed by four individual vertices.

Index detecting section 110 further outputs the image coordinates ^{u Qkn} of the index _{Q kn} each detected and the identifier _{k n} to the position and orientation calculation unit 120. Here, n (n = 1,..., N) is an index for each detected index, and N represents the total number of detected indices.

The position / orientation calculation unit 120 measures the position and orientation measurement values of the imaging device 130 and the measurement target object 170, which are outputs from the sensor measurement value input unit 150, and the image coordinates of each index _Qkn , which is an output from the index detection unit 110. u ^Qkn is input, based on the input information, the error of the measurement value of the position and orientation of the measurement target object 170 or the imaging device 130 is corrected, and the corrected position and orientation data is output.

  Note that at least a part of the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 120 illustrated in FIG. 1 may be realized as independent devices, or one each. Alternatively, it may be implemented as software that implements its function by being installed in a plurality of computers and executed by the CPU of the computer. In the present embodiment, each unit (image input unit 160, index detection unit 110, sensor measurement value input unit 150, and position / orientation calculation unit 120) is realized by software and installed in the same computer. .

  FIG. 2 is a diagram illustrating a basic configuration of a computer for realizing the functions of the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 120 by executing software. It is.

  The CPU 1001 controls the entire computer using programs and data stored in the RAM 1002 and the ROM 1003, and controls the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 120. The execution of software is controlled to realize the functions of each unit.

  The RAM 1002 includes an area for temporarily storing programs and data loaded from the external storage device 1007 and the storage medium drive 1008, and a work area required for the CPU 1001 to perform various processes.

  The ROM 1003 generally stores computer programs and setting data. A keyboard 1004 and a mouse 1005 are input devices, and an operator can input various instructions to the CPU 1001 using these devices.

  The display unit 1006 includes a CRT, a liquid crystal display, or the like, and can display, for example, a message to be displayed for position and orientation measurement.

  The external storage device 1007 is a device that functions as a large-capacity information storage device such as a hard disk drive, and stores an OS (operating system), a program executed by the CPU 1001, and the like. In the description of the present embodiment, information that is described as being known is stored here, and loaded into the RAM 1002 as necessary.

  The storage medium drive 1008 reads a program or data stored in a storage medium such as a CD-ROM or DVD-ROM in accordance with an instruction from the CPU 1001 and outputs it to the RAM 1002 or the external storage device 1007.

The I / F 1009 is an analog video port for connecting the imaging device 130 or a digital input / output port such as IEEE1394, a serial port such as RS232C or USB for connecting the position / orientation sensor 140, and the calculated position and orientation. It is composed of an Ethernet (registered trademark) port for output to the outside. The data input by each is taken into the RAM 1002 via the I / F 1009. Part of the functions of the image input unit 160 and the sensor measurement value input unit 150 are realized by the I / F 1009.
The above-described components are connected to each other by a bus 1010.

  FIG. 3 is a flowchart showing a processing procedure of the position / orientation calculation unit 120. In the present embodiment, the CPU 1001 is implemented by executing a program that realizes the function of the position / orientation calculation unit 120. It is assumed that the program code according to the flowchart is already loaded from, for example, the external storage device 1007 to the RAM 1002 in the previous stage of performing the following processing.

In step S3000, the position and orientation calculation unit 120 inputs the image coordinates ^{u Qkn} of the index _{Q kn} each detected and the identifier _{k n} from the index extraction unit 110. It is ^assumed that the three-dimensional coordinates x _O ^Qkn in the object coordinate system of each index are loaded in advance in the RAM 1002 as known values.

  In step S <b> 3005, the position / orientation calculation unit 120 inputs, from the sensor measurement value input unit 150, measurement values obtained by the position and orientation sensors of the imaging device 130 and the measurement target object 170.

  In step S3010, the position / orientation calculation unit 120 determines whether an index is detected. If no index is detected, the process proceeds to step S3080, and otherwise, the process proceeds to step S3020.

  In step S3020, the position / orientation calculation unit 120 determines whether or not the total number of detected indices is one. If the total number of indices is one, the process proceeds to step S3025. Otherwise, the process proceeds to step S3030.

In step S3025, the position / orientation calculation unit 120 determines an error (an index in the object coordinate system) on the detected index with respect to the pan and tilt angles in the orientation measurement value of the imaging device 130 obtained in step S3005. The ^error between the theoretical value u ^{Qkn *} of the projected coordinates derived from the measured values of the three-dimensional coordinates x _O ^Qkn of the imaging device 130 and the measurement object 170 and the actual detected coordinates u ^Qkn ). Correction is added, and the process proceeds to step S3080. As the correction process in step S3025, since a known process can be applied, no further explanation will be given. Specifically, for example, Patent Document 1 (a correction method by rotation of a camera using one landmark (paragraphs “0019” to “0029”) and a correction method by rotation of a camera using a plurality of landmarks (paragraph “ 0039 "-" 0050 "))) can be used.

  In step S3030, the position / orientation calculation unit 120 determines whether the total number of detected indices is two. If the total number of indices is two, the process proceeds to step S3033. Otherwise, the process proceeds to step S3040.

In step S <b> 3033, the position / orientation calculation unit 120 calculates the distance between the two detected indices on the image, and the threshold value T ₁ (for example, ₁ of the diagonal length of the image) defined as a predetermined value. Comparison with / 4). Distance advances to step S3035 in the case of thresholds T ₁ or more, the flow advances to step S3025 otherwise, performing the processing described above.

  In step S3035, the position / orientation calculation unit 120 uses the measurement values of the position and orientation of the measurement target object 170 obtained in step S3005 and the measurement values of the position of the imaging device 130 as fixed values, and the measurement values of the orientation of the imaging device 130. Is corrected so as to minimize the sum of errors on all detected indices, and the process proceeds to step S3080. Details of this processing step will be described later.

  In step S3040, the position / orientation calculation unit 120 determines that the convex hull includes the image coordinates of all detected indices (this step is executed only when the total number of detected indices is three or more). Is calculated. Since the convex hull calculation method for the point cloud on the image is widely known as a basic matter of image processing, detailed description thereof is omitted.

  6, 7, and 8 illustrate an example of a captured image acquired by capturing the measurement target object 170 with the imaging device 130. Indices Q1, Q5, Q6, and Q9 are shown on the photographed image 600 shown in FIG. 6, indicators Q1, Q2, Q6, and Q7 are on the photographed image 700 in FIG. 7, and indicators Q2, Q6, and Q6 are on the photographed image 800 in FIG. It is assumed that Q7 is observed and other indices are hidden by another object (in this embodiment, the user's hand 610).

When indices are detected for these captured images and the processing steps of the position / orientation calculation unit 120 shown in the flowchart of FIG. 3 are executed, the number of indices detected in each image is 3 or more, so step S3040. Subsequent processing is executed. 9, FIG. 10 and FIG. 11 show the convex hull (900, 1000, 1100) formed by the detection index, which is obtained when the processing of step S3040 is performed on the captured images 600, 700, 800, respectively. . In the following, when the area of each convex hull is expressed as A ₉₀₀ , A ₁₀₀₀ , and A ₁₁₀₀ , A ₁₁₀₀ <T ₂ <A ₁₀₀₀ <with respect to threshold T ₂ and threshold T ₃ described below. It is assumed that the relationship T ₃ <A ₉₀₀ is satisfied.

In step S3050, the position / orientation calculation unit 120 compares the area of the convex hull calculated in step S3040 with a threshold T ₂ (for example, 1/16 of the total area of the captured image) set as a predetermined value. Do. The process advances to step S3060 when the area is the threshold _{T 2} or more, the process proceeds to step S3025 otherwise. 6 to 11, the process proceeds to step S3025 for the captured image 800, and to step S3060 for the captured images 700 and 600.

In step S <b> 3060, the position / orientation calculation unit 120 compares the area of the convex hull calculated in step S <b> 3040 with a threshold T ₃ (for example, 1/9 of the area of the entire captured image) set as a predetermined value. Do. The process advances to step S3070 If the area is not less than the threshold value _{T 3,} the process proceeds to step S3035 otherwise. 6 to 11, the process proceeds to step S3035 for the captured image 700 and to step S3070 for the captured image 600.

  In step S3070, the position / orientation calculation unit 120 calculates the position and orientation of the imaging apparatus 130 that minimizes the sum of errors on all the detected indices. Details of this processing step will be described later.

  In step S3080, the position / orientation calculation unit 120 outputs data representing the positions and orientations of the imaging device 130 and the measurement target object 170 obtained in steps S3025, S3035, and S3070 to the outside via the I / F 1009. Alternatively, these data are stored on the RAM 1002 in a state where they can be used by other applications.

  In each of the steps S3025, S3035, and S3070, only the measurement value related to the imaging device 130 is corrected. Therefore, the position and orientation data output in step S3080 is related to the measurement target object 170 in step S3005. It becomes the measurement value itself by the sensor input in. However, the output form of the position and orientation is not limited to this, and conversely, the measurement value by the sensor is output as it is without correcting the measurement value of the position and orientation of the imaging device 130, and the measurement target object 170 is output. The measured values of the position and orientation may be corrected and output. In this case, in step S3080, the coordinate conversion process described below is performed, and then the converted position and orientation are output.

When the position and orientation are expressed by a 4 × 4 coordinate transformation matrix using a homogeneous coordinate system, the corrected position and orientation M _WO of the measurement target object 170 are the position of the imaging device 130 obtained as a sensor measurement value, and Posture M ^# _WC (hereinafter, ^# is a symbol representing a measurement value by the sensor), position and posture M ^# _WO of the measurement target object 170 obtained as the sensor measurement value, and imaging obtained as a result of processing up to step S3070 Based on the corrected position and orientation M _WC of the device 130, the following calculation is performed.

  At this time, the position and orientation of the measurement target object 170 in the camera coordinate system defined by the imaging device 130 are:

It becomes. That is, the corrected position and orientation (M _WC ) of the imaging device 130 obtained as a result of the processing up to step S3070 and the position and orientation (M ^# _WO ) of the measurement target object 170 obtained as sensor measurement values are obtained. When outputting, the position and orientation (M ^# _WC ) of the imaging device 130 obtained as sensor measurement values, and the corrected position and orientation (M _WO ) of the measurement target object 170 obtained by Equation 1 are output. In this case, the relative positions and orientations of the measurement target object 170 and the imaging device 130 are equivalent. Therefore, it is preferable that the output form can be selected according to the form required by the application using the output of the apparatus.

Further, the position and orientation (M _CO ) of the measurement target object 170 in the camera coordinate system may be calculated from Equation 2 and output, or the position and orientation (M _CO ^-1) of the imaging device 130 in the object coordinate system. ) May be obtained and output.

In step S3090, the position / orientation calculation unit 120 determines whether or not to end the process. If the process is not ended, the process proceeds to step S3000.
As described above, the position and orientation of the imaging apparatus are measured.

Next, the processing steps for calculating the posture of the image capturing apparatus 130 in step S3035 will be described with reference to the flowchart of FIG. In the following, the attitude of the imaging device 130, which is an unknown parameter to be calculated, is expressed as a ternary vector.
s = ω _WC = [ξ _WC ψ _WC ζ _WC ] ^T
Is expressed internally.

  There are various methods for expressing the posture by ternary values, but here, the rotation angle is expressed by the magnitude of the vector, and the ternary vector is defined by defining the rotation axis direction by the vector direction. It shall be. The posture ω is expressed by the following equation:

Can also be represented by a 3 × 3 rotation matrix R, and ω and R can be uniquely converted from each other. Since the conversion method from R to ω is known, the detailed description thereof is omitted.

  In step S4000, the position / orientation calculation unit 120 sets the orientation of the imaging device 130 obtained as the sensor measurement value as an initial value of s.

In step S4010, the position and orientation calculation unit 120, for each of the indices _{Q kn,} we calculate an estimated value ^{u Qkn *} of the image coordinates. The calculation of u ^{Qkn *} is an index observation equation defined by s, that is, a function for calculating image coordinates from coordinates x _O ^Qkn (previously held as known information) of each index Q _{kn in} the object coordinate system. :

Based on. Specifically, the observation equation Fc () is ^defined from x _O ^Qkn and s as camera coordinates of the index (the viewpoint position of the imaging device 130 is defined as the origin, and three axes orthogonal to each other are defined as the X axis, the Y axis, (Coordinate system defined as Z axis) x _C ^Qkn

And the following equation for ^{obtaining the} image coordinates u ^{Qkn *} from the camera coordinates x _C ^Qkn :

It is constituted by. Here, t _WC and t _WO are three-dimensional vectors representing the position measurement values (parallel movement components) of the imaging device 130 and the measurement target object 170, and are treated as fixed values here. Also, R _WO is a 3 × 3 rotation matrix representing the orientation measurement value (rotation component) of the measurement target object 170, and is similarly handled as a fixed value. Further, f ^C _x and f ^C _y are focal lengths of the imaging device 130 in the x-axis direction and the y-axis direction, respectively, and are held in advance as known values.

In step S4020, the position / orientation calculation unit 120 calculates an error Δu ^Qkn between the estimated value u ^{Qkn *} of the image coordinates and the actual measurement value u ^Qkn for each index Q _kn ^according to the following equation.

In step S4030, the position / orientation calculation unit 120 has, for each index Q _kn , each element having a solution obtained by partially differentiating the image Jacobian relating to s (that is, the observation equation Fc () of Equation 4 by each element of s). 2 rows × 3 columns Jacobian matrix) J _us ^Qkn (= ∂u / ∂s) is calculated. Specifically, a Jacobian matrix J _ux ^Qkn (= ∂u / ∂x) of 2 rows × 3 columns having a solution obtained by partial differentiation of the right side of Equation 6 with each element of camera coordinates x _C ^Qkn , and an equation 5) A 3 row × 3 column Jacobian matrix J _xs ^Qkn (= ∂x / ∂s) having a solution obtained by partial differentiation of the right side of 5 with each element of the vector s is calculated, and J _us ^Qkn is calculated by the following equation: To do.

In step S4040, the position / orientation calculation unit 120 calculates a correction value Δs of s based on the error Δu ^Qkn and the image Jacobian J _us ^Qkn calculated in steps S4020 and S4030. Specifically, a 2N-dimensional error vector in which errors Δu ^Qkn are arranged vertically

And a matrix of 2N rows × 3 columns in which the image Jacobian J _us ^Qkn are arranged vertically

And using the pseudo inverse matrix Θ ⁺ of Θ,

Calculate as Thus, in this embodiment, the correction value Δs is calculated using the steepest descent method.

  In step S4050, the position / orientation calculation unit 120 corrects s according to Equation 12 using the correction value Δs calculated in step S4040, and sets the obtained value as a new estimated value of s.

  In step S4060, the position / orientation calculation unit 120 converges the calculation by using some criterion such as whether the error vector U is smaller than a predetermined threshold or whether the correction value Δs is smaller than a predetermined threshold. It is determined whether or not. If not converged, the processing from step S4010 is performed again using the corrected s. On the other hand, if it is determined that it has converged, the process proceeds to step S4070.

In step S4070, the position / orientation calculation unit 120 sets s obtained through the processing up to step S4060 as an estimated value ω _WC (after correction) of the attitude of the imaging device 130.

  As described above, the position parameter obtained as the sensor measurement value is fixed, and the posture parameter that minimizes the sum of errors on all the indices can be obtained.

Next, the processing steps for calculating the position and orientation of the imaging device 130 in step S3070 will be described with reference to the flowchart of FIG. In the following, the position and orientation of the imaging device 130, which are unknown parameters to be calculated, are expressed by a six-value vector s = [ _tWC ^T ω _WC ^T ] ^T = [x _WC y _WC z _WC ξ _WC ψ ψ _WC ζ _WC ] ^T Expressed internally.

  In step S5000, the position / orientation calculation unit 120 sets the position and orientation of the imaging device 130 obtained as the sensor measurement values as the initial value of s.

In step S5010, the position and orientation calculation unit 120, for each of the indices _{Q kn,} we calculate an estimated value ^{u Qkn *} of the image coordinates. The calculation of u ^{Qkn *} is an observation equation of an index defined by s, that is, a function for calculating image coordinates from coordinates x _O ^Qkn in the object coordinate system of each index Q _kn :

Based on. Specifically, the observation equation F′c () is an expression 5 for ^obtaining the camera coordinates x _C ^Qkn of the index from x _O ^Qkn and s, and an expression 6 for ^{obtaining the} image coordinates u ^{Qkn *} from the camera coordinates x _C ^Qkn. It is constituted by. However, t _WC (representing the position of the imaging device 130) in Equation 5 is treated as a part of parameters constituting s, not as a fixed value. On the other hand, with respect to the position t _WO and orientation R _WO of the measurement object 170, used as it is as a fixed value the measured value by the sensor.

In step S5020, the position / orientation calculation unit 120 calculates an error Δu ^Qkn between the estimated value u ^{Qkn *} of the image coordinates and the actual measurement value u ^Qkn for each index Q _kn using Equation 7.

In step S5030, the position / orientation calculation unit 120 obtains, for each index Q _kn , an image Jacobian relating to s (that is, a solution obtained by partially differentiating the observation equation F′c () of Equation 13 with each element of s. 2 rows × 6 columns Jacobian matrix) J _us ^Qkn (= ∂u / ∂s). Specifically, a Jacobian matrix J _ux ^Qkn (= ∂u / ∂x) of 2 rows × 3 columns having a solution obtained by partial differentiation of the right side of Equation 6 with each element of camera coordinates x _C ^Qkn , and an equation 5. ^Calculate a Jacobian matrix J _xs ^Qkn (= ∂x / ∂s) of 3 rows × 6 columns having a solution obtained by partial differentiation of the right side of 5 with each element of the vector s, and calculate J _us ^Qkn by Equation 8. To do.

In step S5040, the position / orientation calculation unit 120 calculates a correction value Δs of s based on the error Δu ^Qkn and the image Jacobian J _us ^Qkn calculated in steps S5020 and S5030. Specifically, a 2N-dimensional error vector U in which errors Δu ^Qkn are arranged vertically and a matrix Θ of 2N rows × 6 columns in which image Jacobians J _us ^Qkn are arranged vertically are created, and a pseudo inverse matrix Θ ⁺ And is calculated by Equation 11.

  In step S5050, the position / orientation calculation unit 120 corrects s according to Equation 12 using the correction value Δs calculated in step S5040, and sets the obtained value as a new estimated value of s.

  In step S5060, the position / orientation calculation unit 120 converges the calculation by using some criterion such as whether the error vector U is smaller than a predetermined threshold or whether the correction value Δs is smaller than the predetermined threshold. It is determined whether or not. If not converged, the processing from step S5010 is performed again using the corrected s. On the other hand, if it is determined that it has converged, the process proceeds to step S5070.

  In step S5070, the position / orientation calculation unit 120 sets s obtained by the processing up to step S5060 as an estimated value of the corrected position and orientation of the imaging apparatus 130.

In the present embodiment, in the process of step S3070, the position and orientation of the imaging device 130 are targeted for correction. However, the process may be performed using the position and orientation of the measurement target object 170 as the subject of correction. In this case, the unknown parameter to be calculated is a six-value vector s = [t _WO ^T ω _WO ^T ] ^T = [x _WO y _WO z _WO ξ _WO ψ _WO ζ _WO ] representing the position and orientation of the measurement target object 170. The position and orientation of the measurement target object 170 expressed internally by ^T and obtained as a sensor measurement value are used as the initial value of s. Further, Equation 5 constituting the observation equation F′c () is transformed as the following equation.

At this time, with respect to the position t _WC and the posture R _WC of the imaging device 130, the measured value by the sensor is used as a fixed value as it is.

  Further, the position and orientation of the measurement target object 170 in the camera coordinate system and the position and orientation of the imaging device 130 in the object coordinate system may be obtained as unknown parameters.

In the former case, the unknown parameters to be calculated, 6 values representing the position and orientation of the measurement object 170 in the camera coordinate system vector ^{_{s = [t CO T ω CO}} T] T = [x CO y CO z CO ξ CO ψ _CO ζ _CO ] ^T expresses internally. In addition, the position and orientation (M ^# _CO ) of the measurement target object 170 in the camera coordinate system are expressed by the following equation:

Based on the sensor measurement values (M ^# _WC and M ^# _WO ), this is used as the initial value of s. Further, Equation 5 constituting the observation equation F′c () is transformed as the following equation.

On the other hand, in the latter case, an unknown parameter to be calculated is a six-value vector s = [t _OC ^T ω _OC ^T ] ^T = [x _OC y _OC z _OC ξ] representing the position and orientation of the imaging device 130 in the object coordinate system. _OC ψ _OC ζ _OC ] ^T Further, the position and orientation (M ^# _CO ) of the imaging device 130 in the object coordinate system are expressed by the following equation:

Based on the sensor measurement values (M ^# _WC and M ^# _WO ), this is used as the initial value of s. Further, Equation 5 constituting the observation equation F′c () is transformed as the following equation.

  As described above, it is possible to correct the error of the position and orientation measurement values obtained by the sensor using the relative position and orientation relationship between the measurement target object 170 and the imaging device 130 as an unknown parameter.

  In the conventional position and orientation measurement apparatus, since the distribution of the index is not taken into consideration, the solution becomes unstable when the index is unevenly observed in a part of the area on the image. On the other hand, according to the position / orientation measurement apparatus according to the present embodiment, since the method considering the size of the index distribution range is selected in step S3033 or steps S3040, S3050, and S3060, for example, the detected index Even if the number is large, when they are unevenly distributed in a small area on the image, it can be determined that it is insufficient to stably obtain the 6 degrees of freedom of the position and orientation. Therefore, even if the index is unevenly distributed in some areas on the image, the appropriate position and orientation estimation method is selected, and the possibility of falling into an unstable solution situation is reduced, A stable solution can be obtained as compared with the conventional method.

  Further, according to the position / orientation measurement apparatus according to the present embodiment, when only some parameters constituting the position and orientation of the imaging apparatus are selected to correct the sensor measurement value, It is possible to perform correction so as to minimize the sum of errors. Therefore, even when it is intended to stabilize the measurement of the position and orientation of the imaging apparatus by reducing the unknown parameters, it is possible to perform measurement with higher accuracy than in the past.

[Second Embodiment]
The position and orientation measurement apparatus according to the present embodiment measures the position and orientation of an arbitrary measurement target object in a reference coordinate system set in a room or the like.

  The position / orientation measurement apparatus according to the present embodiment has a position / orientation sensor attached to the imaging apparatus, and is fixedly installed at a known position and attitude using a tripod or the like. Is different. Only the differences from the first embodiment will be described below.

  FIG. 12 shows the configuration of the position / orientation measurement apparatus according to this embodiment. As shown in the figure, the position / orientation measurement apparatus 1200 according to this embodiment includes an image input unit 160, an index detection unit 110, a sensor measurement value input unit 150, and a position / orientation calculation unit 1220. 130 and the position / orientation sensor 140.

  The imaging device 130 is fixedly installed at a known position and posture in the space by a tripod 1280. The captured image output by the imaging device 130 is input to the image input unit 160 of the position / orientation measurement apparatus 1200. It is assumed that the position and orientation of the imaging device 130 in the reference coordinate system are held in advance as known values.

  The sensor measurement value input unit 150 receives the measurement values of the position and orientation of the measurement target object 170 from the position and orientation sensor 140 and outputs them to the position and orientation calculation unit 1220.

The position / orientation calculation unit 1220 receives the measurement value of the position and orientation of the measurement target object 170 that is output from the sensor measurement value input unit 150 and the image coordinates u ^Qkn of each index Q _kn that is output from the index detection unit 110. Then, based on the input information, the error of the measurement value of the position and orientation of the measurement target object 170 is corrected, and the corrected position and orientation data is output.

  Note that at least a part of the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 1220 illustrated in FIG. 12 may be realized as independent devices, or one each. Alternatively, it may be implemented as software that implements its function by being installed in a plurality of computers and executed by the CPU of the computer. In this embodiment, each unit (the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 1220) is realized by software and installed in the same computer. . Since the basic configuration of the computer for realizing the functions of the respective units by executing the software is the same as that of the first embodiment, the description thereof is omitted.

  The processing procedure of the position / orientation calculation unit 1220 in this embodiment is almost the same as the processing procedure (corresponding to the flowchart of FIG. 3) of the position / orientation calculation unit 120 in the first embodiment. The only difference is that the position and orientation of the imaging device 130 are not given as measurement values but are held in advance as known values. The position / orientation calculation unit 1220 puts known values of the position and orientation of the image capturing apparatus 130 as provisional measurement values of the position and orientation of the image capturing apparatus 130, and is the same as steps S3000 to S3070 in the first embodiment. The measurement value of the position and orientation of the imaging device 130 is corrected according to the process, and based on the correction result, the measurement value of the position and orientation of the measurement target object 170 is corrected in step S3080 as in the first embodiment. (Formula 1). In the present embodiment, the position / orientation calculation unit 1220 outputs only the position and orientation of the measurement target object 170 in step S3080.

  As described above, the position and orientation of the measurement object are measured. As described above, the position / orientation measurement apparatus according to the present embodiment also selects a method in consideration of the size of the distribution range of the index, and therefore, the index is unevenly observed in a partial region on the image. Even in such a case, an appropriate position and orientation estimation method is selected, and a stable solution can be obtained as compared with the conventional method.

[Third Embodiment]
The position / orientation measurement apparatus according to the present embodiment measures the position and orientation of an imaging apparatus in a space such as a room. The position / orientation measurement apparatus according to this embodiment will be described below.

  Unlike the first embodiment, there is no measurement target object other than the imaging device in the present embodiment, and the position / orientation measurement device according to the present embodiment only measures the position and orientation of the imaging device. ing. Only the differences from the first embodiment will be described below.

  FIG. 13 shows the configuration of the position / orientation measurement apparatus according to this embodiment. As shown in the figure, the position / orientation measurement apparatus 1300 according to this embodiment includes an image input unit 160, an index detection unit 110, a sensor measurement value input unit 150, and a position / orientation calculation unit 1320. 130 and the position / orientation sensor 140.

In addition, a plurality of indices Q _k whose positions in the reference coordinate system are known are disposed at a plurality of positions in the real space as indices for photographing by the imaging device 130. As in the first embodiment, the index Q _k is an index that can detect the image coordinates of the projected image on the captured image and can identify which index is somehow. Any form may be used.

  The sensor measurement value input unit 150 receives the position and orientation measurement values of the imaging device 130 from the position and orientation sensor 140 and outputs them to the position and orientation calculation unit 1320.

The position / orientation calculation unit 1320 receives the measurement values of the position and orientation of the imaging device 130 that are outputs of the sensor measurement value input unit 150 and the image coordinates u ^Qkn of each index Q _kn that are outputs of the index detection unit 110. Based on the input information, the error of the measurement value of the position and orientation of the imaging device 130 is corrected, and the corrected position and orientation data is output.

  Note that at least a part of the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 1320 illustrated in FIG. 13 may be realized as independent devices, or one each. Alternatively, it may be implemented as software that implements its function by being installed in a plurality of computers and executed by the CPU of the computer. In the present embodiment, each unit (the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 1320) is realized by software and installed in the same computer. . Since the basic configuration of the computer for realizing the functions of the respective units by executing the software is the same as that of the first embodiment, the description thereof is omitted.

  The processing procedure of the position / orientation calculation unit 1320 in this embodiment is similar to the processing procedure (corresponding to the flowchart of FIG. 3) of the position / orientation calculation unit 120 in the first embodiment. Only differences from the first embodiment will be described.

  In step S3005, the position / orientation calculation unit 1320 inputs the position and orientation of the imaging apparatus 130 measured by the sensor from the sensor measurement value input unit 150.

In step S3025, the position / orientation calculation unit 1320 determines an error on the detected index (the index in the reference coordinate system) with respect to the pan and tilt angles of the orientation measurement values of the imaging device 130 obtained in step S3005. A correction is made so as to cancel the error between the three-dimensional coordinate x _W ^Qkn and the theoretical value u ^{Qkn *} of the projected coordinate derived from the measured values of the position and orientation of the imaging device 130 and the actual detected coordinate u ^Qkn ), step S3080 Proceed with the process.

  As such correction processing, since a known method can be used, no further explanation will be given. Specifically, for example, Patent Document 1 (a correction method by rotation of a camera using one landmark (paragraphs “0019” to “0029”) and a correction method by rotation of a camera using a plurality of landmarks (paragraph “ 0039 "-" 0050 "))) can be used.

In step S3035, the position / orientation calculation unit 1320 sets the position measurement value of the imaging device 130 obtained in step S3005 as a fixed value, and performs three parameters (pan, tilt, and roll angle) representing the orientation measurement value of the imaging device 130. Then, correction is performed so as to minimize the sum of errors on all the detected indices, and the process proceeds to step S3080. This correction processing step is similar to the correction processing step (corresponding to the flowchart of FIG. 4) in step S3035 in the first embodiment. However, in the case of the present embodiment, the observation equation F ″ c () of the index is a function for calculating the image coordinates from the coordinates x _W ^Qkn (preliminarily stored as known information) of the index Q _kn in the reference coordinate system. It becomes. That is, the observation equation F ″ c () is expressed by the following equation.

Specifically, the observation equation F ″ c () is ^expressed by the following equation for ^obtaining the camera coordinates x _C ^Qkn of the index from x _W ^Qkn and s:

And is constituted by a camera coordinate _x ^{C Qkn} by Equation 6 for obtaining the image coordinates ^{u Qkn *,} only this point is different. Here, t _WC is a three-dimensional vector representing the position measurement value of the imaging device 130, and is treated as a fixed value here.

  In step S3070, the position / orientation calculation unit 1320 calculates the position and orientation of the imaging apparatus 130 that minimizes the sum of errors on all detected indices. The specific processing in this processing step can be realized by using a well-known technique as disclosed in, for example, Patent Document 2, and thus the description thereof is omitted here.

  In step S3080, the position / orientation calculation unit 1320 outputs the position and orientation of the imaging device 130 obtained as a result up to step S3070 to the outside via the I / F 1009. Alternatively, this data is stored on the RAM 1002 in a state where it can be used by other applications.

  As described above, the position and orientation of the imaging apparatus are measured. As described above, the position / orientation measurement apparatus according to the present embodiment also selects a method in consideration of the size of the distribution range of the index, and therefore, the index is unevenly observed in a partial region on the image. Even in such a case, an appropriate position and orientation estimation method is selected, and a stable solution can be obtained as compared with the conventional method.

[Fourth Embodiment]
The position and orientation measurement apparatus according to the present embodiment measures the position and orientation of an arbitrary measurement target object in a space such as a room. The position / orientation measurement apparatus according to this embodiment will be described below. In the present embodiment, it is different from the third embodiment in that the position and orientation of an arbitrary measurement target object, not an imaging device, are measured. Only the differences from the third embodiment will be described below.

  FIG. 14 shows the configuration of the position / orientation measurement apparatus according to this embodiment. As shown in the figure, the position / orientation measurement apparatus 1400 in this embodiment includes an image input unit 160, an index detection unit 110, a sensor measurement value input unit 150, an imaging unit 1430, a position / orientation measurement unit 1440, and a position / orientation calculation. The unit 1420 is connected to a measurement target object 1470.

  The imaging unit 1430 is fixedly attached to a measurement target object 1470 that is a target whose position and orientation are to be measured, and captures an image of a real space including an index.

  The position / orientation measurement unit 1440 is fixedly attached to the imaging unit 1430, measures the position and orientation of the imaging unit 1430 in the reference coordinate system, and outputs the measured value to the sensor measurement value input unit 150. The position / orientation measurement unit 1440 is configured by, for example, FASTRAK from Polhemus, USA.

  The image input unit 160 converts the captured image captured by the imaging unit 1430 into digital data and outputs the digital data to the index detection unit 110.

  The sensor measurement value input unit 150 inputs measurement values of the position and orientation of the imaging unit 1430 from the position / orientation measurement unit 1440 and outputs them to the position / orientation calculation unit 1420.

The position / orientation calculation unit 1420 receives the measurement values of the position and orientation of the imaging unit 1430 that are outputs of the sensor measurement value input unit 150 and the image coordinates u ^Qkn of each index Q _kn that are outputs of the index detection unit 110. Based on the input information, the error in the measured values of the position and orientation of the imaging unit 1430 is corrected.
The position / orientation calculation unit 1420 further includes information on the obtained position and orientation of the imaging unit 1430 and the relative position / orientation relationship between the imaging unit 1430 and the measurement target object 1470 held in advance as known values ( Specifically, information on which position on the measurement target object 1470 the imaging unit 1430 is installed (expressed by the position and orientation of the imaging unit 1430 in the object coordinate system defined by the measurement target object 1470) From this, the position and orientation of the measurement target object 1470 in the reference coordinate system are calculated, and this data is output to the outside via the I / F 1009. Alternatively, this data is stored on the RAM 1002 in a state where it can be used by other applications. Note that this conversion step is not always necessary, and the position and orientation of the imaging unit 1430 can be output as they are.

  In the present embodiment, the position / orientation measurement unit 1440 measures the position and orientation of the imaging unit 1430, but the position / orientation measurement unit 1440 is configured to measure the position and orientation of the measurement target object 1470. Also good. In this case, the position / orientation calculation unit 1420 uses a relative position / orientation relationship between the imaging unit 1430 and the measurement target object 1470 held in advance as a known value, and obtains a measurement target. The position and orientation of the imaging unit 1430 are calculated from the position and orientation of the object 1470, and the same processing as described above is executed using this as the initial value.

  Further, the position and orientation calculation unit 1420 corrects the orientation or position and orientation of the imaging unit 1430 in any of the steps S3025, S3035, and S3070, and then obtains the position and orientation of the measurement target object 1470. However, the steps S3035 and S3070 may be configured to directly obtain the position and orientation of the measurement target object 1470.

  When the position / orientation measurement unit 1440 measures the position and orientation of the imaging unit 1430, the position / orientation calculation unit 1420 compares the imaging unit 1430 and the measurement target object 1470 that are held in advance as known values. The position and orientation of the measurement target object 1470 are calculated from the position and orientation of the imaging unit 1430 obtained as the measurement value using the position and orientation relationship, and this is used as the initial value.

In step S3035, a ternary vector s = ω _WO = [ξ _WO ψ _WO ζ _WO ] ^T representing the posture of the measurement target object 1470 is set as an unknown parameter. In step S3070, a six-value vector s = [t _WO ^T ω _WO ^T ] ^T = [x _WO y _WO z _WO ξ _WO ψ _WO ζ _WO ] ^T representing the position and orientation of the measurement target object 1470 is set as an unknown parameter. And And the equation 20 necessary to define the observation equation F ″ c () is

Change to Here, t _OC and R _OC are a 3 × 3 rotation matrix and a three-dimensional vector representing the orientation of the imaging unit 1430 in the object coordinate system defined by the measurement target object 170, and are measured with the imaging unit 1430. It is assumed that it is held in advance as a known value representing the relative position and orientation relationship with the target object 1470. In step S3035, t _WO is set as a fixed value.

  As described above, the position and orientation of an arbitrary object are measured. As described above, the position / orientation measurement apparatus according to the present embodiment also selects a method in consideration of the size of the distribution range of the index, and therefore, the index is unevenly observed in a partial region on the image. Even in such a case, an appropriate position and orientation estimation method is selected, and a stable solution can be obtained as compared with the conventional method.

[Fifth Embodiment]
The position / orientation measurement apparatus according to the present embodiment measures the position and orientation of an imaging apparatus in a space such as a room. The position / orientation measurement apparatus according to this embodiment will be described below.

  FIG. 15 shows the configuration of the position / orientation measurement apparatus according to this embodiment. As shown in the figure, the position / orientation measurement apparatus 1500 according to the present embodiment includes an image input unit 160, an index detection unit 110, a sensor measurement value input unit 1550, and a position / orientation calculation unit 1520. 130 and the attitude sensor 1540.

  The present embodiment is different from the third embodiment in that an image sensor 130 is equipped with a posture sensor instead of a position and posture sensor. By using the posture sensor, it is possible to avoid the restriction that a region other than the measurement range of the sensor, which is present when the position and posture sensor is used, cannot be measured. Only the differences from the third embodiment will be described below.

  The posture sensor 1540 is attached to the imaging device 130, measures the current posture of the posture sensor 1540 itself, and outputs it to the sensor measurement value input unit 1550. The attitude sensor 1540 is a sensor unit based on, for example, a gyro sensor, and specifically includes a TISS-5-40 manufactured by Tokimec Corporation, InertiaCube2 manufactured by InterSense, Inc., and the like. The posture measurement values measured by these sensors are postures having errors that are different from the true posture. However, these attitude sensors have a tilt angle sensor that observes the direction of gravity of the earth as a constituent element, and have a function of canceling accumulation of drift errors in the tilt angle direction (pitch angle and roll angle). Therefore, there is a property that no drift error occurs in the tilt angle direction. In other words, the azimuth angle direction (yaw angle direction) has a drift error that accumulates with time.

  The sensor measurement value input unit 1550 receives an updated value of the azimuth drift error correction value from the position / orientation calculation unit 1520, and updates and holds the current azimuth drift error correction value of the orientation sensor 1540. Further, a posture measurement value is input from the posture sensor 1540, corrected by the current azimuth drift error correction value, and output to the position / orientation calculation unit 1520 as a predicted value of the posture of the imaging device 130.

The position / orientation calculation unit 1520 receives, as input data, a predicted value of the orientation of the imaging device 130 that is an output of the sensor measurement value input unit 1550 and an image coordinate u ^Qkn of each index Q _kn that is an output of the index detection unit 110. The position and orientation of the imaging device 130 are calculated and output. Also, the updated value of the orientation drift error correction value of the orientation sensor 1540 derived in the position and orientation calculation step is output to the sensor measurement value input unit 1550.

  Note that at least a part of the image input unit 160, the index detection unit 110, the sensor measurement value input unit 150, and the position / orientation calculation unit 1520 illustrated in FIG. 15 may be realized as independent devices, or one each. Alternatively, it may be implemented as software that implements its function by being installed in a plurality of computers and executed by the CPU of the computer. In the present embodiment, each unit (image input unit 160, index detection unit 110, sensor measurement value input unit 150, and position / orientation calculation unit 1520) is realized by software and installed in the same computer. . Since the basic configuration of the computer for realizing the functions of the respective units by executing the software is the same as that of the first embodiment, the description thereof is omitted.

  FIG. 16 is a flowchart of processing for calculating parameters indicating the position and orientation of the imaging apparatus 130, and is performed by the CPU 1001 executing a software program of the position / orientation calculation unit 1520.

In step S15000, the position and orientation calculation unit 1520 inputs the image coordinates ^{u Qkn} of the index _{Q kn} each detected and the identifier _{k n} from the index extraction unit 110. It is ^assumed that the three-dimensional coordinate x ^Qkn in the reference coordinate system of each index is ^preloaded in the ^{RAM 1002} as a known value.

In step S15005, the position / orientation calculation unit 1520 inputs the predicted value R ^* of the orientation of the imaging device 130 from the sensor measurement value input unit 1550, and sets this as an initial value of the orientation of the imaging device 130. In the second and subsequent processing, the position of the imaging device 130 calculated in the processing one loop before is set as the initial value of the orientation of the imaging device 130.

  In step S15010, the position / orientation calculation unit 1520 determines whether an index has been detected. If no index is detected, the process proceeds to step S15090; otherwise, the process proceeds to step S15020.

  In step S15020, the position / orientation calculation unit 1520 determines whether the total number of detected indices is one. If the total number of indices is one, the process proceeds to step S15025; otherwise, the process proceeds to step S15030.

  In step S15025, the position / orientation calculation unit 1520 corrects an error on the detected index using a value in the biaxial direction orthogonal to the visual axis in the parallel translation component of the initial value set in step S15005 as a correction target. Correction for canceling is added, and the process proceeds to step S15090. As a processing process in step S15025, since a known process can be provided, no further description will be given. Specifically, for example, Patent Document 1 (Correction method by translation of camera using one landmark (paragraphs “0030” to “0036”) and Correction method by translation of camera using a plurality of landmarks ( The method disclosed in paragraphs “0051” to “0056”) can be used, although Patent Document 1 describes the case where the measurement value of the 6-DOF position / orientation sensor is the target of error correction. The present embodiment is different in that a combination of the position obtained in the previous frame and the measured value of the orientation sensor is a correction target.

  In step S15030, the position / orientation calculation unit 1520 determines whether the total number of detected indices is two. If the total number of indices is two, the process proceeds to step S15032. Otherwise, the process proceeds to step S15040.

In step S15032, the position / orientation calculation unit 1520 calculates the distance between the two detected indices on the image, and the threshold value T ₁ (for example, ₁ of the diagonal length of the image) defined as the predetermined value. Comparison with / 8). Distance the flow advances to step S15034 when the thresholds _{T 1} or more, otherwise the process advances to step S15025.

In step S15034, the position / orientation calculation unit 1520 compares the detected distance between the two indices on the image and a threshold T ₂ (for example, ¼ of the diagonal length of the image) defined as a predetermined value. I do. Distance the flow advances to step S15038 when the threshold _{T 2} or higher, otherwise the process advances to step S15036.

  In step S15036, the position / orientation calculation unit 1520 sets three parameters (parallel movement components) representing the position among the initial values set in step S15005 as update targets (that is, the attitude measurement value does not include an error). Assuming that the correction is made to minimize the error on the detected index, only the position parameter is added, and the process proceeds to step S15090. The details of this processing step are disclosed in, for example, Patent Document 3 (sixth embodiment), and a description thereof will be omitted here. It is also possible to correct the position so that the error on the image is minimized by constructing an observation equation with the three parameters representing the position as unknown variables and performing error minimization calculation using the image Jacobian. Can do.

  In step S15038, the position / orientation calculation unit 1520 has a total of four parameters including three parameters (translation component) representing the position among the initial values set in step S15005 and the updated value φ of the azimuth drift error correction value of the attitude sensor 1340. Are treated as unknown parameters to be calculated, and these are calculated using information of detected indices. Details of this processing step are disclosed in, for example, Patent Document 3 (seventh embodiment), and thus description thereof is omitted here. Also, by constructing an observation equation with these four parameters as unknown variables and performing error minimization calculation using the image Jacobian, the position and orientation drift error correction values that minimize the error on the image can be obtained. An update value can be calculated. Further, the process proceeds to step S15080.

  In step S15040, the position / orientation calculation unit 1520 calculates a convex hull including the image coordinates of all detected indices (this step is executed only when the total number of indices is three or more). .

In step S15050, the position / orientation calculation unit 1520 compares the area of the convex hull calculated in S15040 with a threshold T ₃ (for example, 1/16 of the area of the image) set as a predetermined value. The process advances to step S15060 when the area is not less than the threshold value _{T 3,} the process advances to step S15036. Otherwise, perform the above-described processing.

In step S15060, the position and orientation calculation unit 1520 performs the comparison with the threshold value _{T 4} which is defined as an area with a predetermined value of the convex hull calculated in S15040. If the area is greater than or equal to a threshold T ₄ (for example, 1/9 of the area of the image), the process proceeds to step S15070. Otherwise, the process proceeds to step S15038, and the above-described process is performed.

In step S15070, the position / orientation calculation unit 1520 calculates the position and orientation of the imaging apparatus 130 that minimizes the sum of errors on all detected indices. This calculation is realized, for example, by obtaining a parameter that minimizes an error by an iterative solution of a nonlinear equation using the six parameters representing the position and orientation of the imaging device 130 as variables and using the initial value set in step S15005. The That is, an observation equation having six parameters representing position and orientation as unknown variables is constructed, and the correction value of the parameter is obtained using the error of the index projection position obtained based on this and the image Jacobian related to each parameter, and the correction is performed. By repeatedly performing the process, the position and orientation that minimize the error on the image are calculated.
The position / orientation calculation unit 1520 further calculates a change between the initial value of the attitude set in step S15005 and the obtained attitude, and sets the azimuth component as the updated value φ of the azimuth drift error correction value.

  In step S15080, the position / orientation calculation unit 1520 outputs the updated value φ of the azimuth drift error correction value obtained in step S15038 or step S15070 to the sensor measurement value input unit processing 1550.

  In step S15090, the position / orientation calculation unit 1520 outputs the position and orientation of the imaging device 130 obtained as a result up to step S15080 to the outside via the I / F 1009. Alternatively, these data are stored on the RAM 1002 in a state where they can be used by other applications.

  In step S15100, the position / orientation calculation unit 1520 determines whether or not to end the process. If the process is not ended, the process proceeds to step S15000.

As described above, the position and orientation of the imaging apparatus are measured.
According to the position / orientation measurement apparatus according to the present embodiment, the selection of a method in consideration of the size of the index distribution range (determination as to whether or not to update the azimuth drift error correction value) is performed in steps S15034 and S15050. Therefore, it is possible to avoid a situation in which the azimuth drift error correction value is incorrectly updated when the index is unevenly observed in a partial region on the image.

  Further, according to the position / orientation measurement apparatus according to the present embodiment, in addition to the effects of the third embodiment, the inclination angle information obtained by the attitude sensor can be obtained even when image information is insufficient due to the selection of the technique in step S15060. Advantages of the position and orientation estimation method based on normal image information that the position and orientation can be stably derived by reliability and the position and orientation can be estimated with high accuracy if sufficient image information is available Can be compatible.

[Other Embodiments]
(Modification 1)
In each of the above embodiments, the correction algorithm to be applied is selected based on the distribution of the indices on the image. These correction algorithms (steps S3025, S3035, S3070 in the first embodiment, these modifications in the second to fourth embodiments, and steps S15025, S15036, S15038, S15070 in the fifth embodiment) are: It does not necessarily have to be the one shown in the above embodiment, and can be changed to another correction algorithm as long as it is appropriately set according to the information amount of image information expected according to the distribution of indices. is there.

  For example, in steps S3025 and S3035 in the first to fourth embodiments, the position measurement value may be corrected instead of correcting the posture measurement value. In this case, in step S3025, for example, parallel translation of the camera using one or a plurality of indices as disclosed in paragraphs “0030” to “0036” and “0051” to “0056” of Patent Document 1. Based on the correction method according to (1), a correction that cancels the error on the detected index can be applied with the value in the biaxial direction orthogonal to the visual axis of the camera among the position measurement values as a correction target.

In step S3035, the unknown parameter to be calculated is a ternary vector s = t _WC = [x _WC y _WC z _WC ] ^T representing the position of the imaging device 130, and the position is corrected by the same processing procedure as in FIG. I do. The difference from the first embodiment is that Equation 5 constituting the observation equation is changed to the following equation.

Here, R _WC represents a posture measurement value by the sensor of the imaging device 130 used as a fixed value. Further, R _WO and t _WO represent measurement values obtained by the sensor of the position and orientation of the measurement target object 170, and are treated as fixed values, as in Equation 5 in the first embodiment.

  Since Equation 22 is a linear equation, a correction value can be obtained without using iterative calculation by creating a simultaneous equation from the observation equation combined with Equation 6 and solving it.

  Note that the user may freely select which of the posture correction and the position correction is prioritized via a UI (not shown). In this case, at the time of executing steps S3025 and S3035, an attitude or position correction algorithm is executed in accordance with the user's selection. Thereby, even when a position and orientation sensor having different characteristics is used, more preferable measurement can be performed.

  In step S3025, any two parameters out of the six parameters obtained as measurement values may be selected as correction targets, and these may be obtained as unknowns in the same manner as described above. Similarly, in step S3035, any three parameters out of six parameters obtained as measurement values may be selected as correction targets, and these may be determined as unknowns using the same framework as described above.

(Modification 2)
In the first to fourth embodiments, the processing is branched into three types of algorithms based on the distribution of indices on the image (steps S3033, S3050, and S3060). However, the number of branches performed based on the index distribution is not limited to three. For example, by omitting the processing in step S3050, it may be simply the process branches to step S3035 or step S3070 by the magnitude relation of the area of the threshold value _{T 3} and the convex hull. Further, more than four thresholds may be provided to perform four or more branches. For example, the number of parameters to be corrected (2 to 6) may be changed according to the area of the convex hull, and the position and orientation may be calculated by a similar method using this as an unknown.

(Modification 3)
In each of the above-described embodiments, the position and orientation of the measurement target (measurement target object or imaging device) are obtained by using the measurement values obtained by the position and orientation sensor or the orientation sensor and the image information obtained by the imaging device together. . However, the basic technical idea of selecting an algorithm to be used for obtaining the position and orientation based on the distribution of the indices on the image is that the measurement target is determined from only the image information without using the measurement values of these sensors. Even when the position and orientation are obtained, the present invention can be applied.
For example, in the configuration of the third embodiment illustrated in FIG. 13, the position / orientation measurement apparatus 1300 does not include the sensor measurement value input unit 150 and the position / orientation sensor 140 is not installed in the imaging apparatus 130. Think. In this case, the position / orientation calculation unit 1320 does not use the measured values of the position and orientation of the imaging device 130 by the position / orientation sensor 140 as initial values, but calculates the calculated values of the position and orientation of the imaging device 130 obtained in the previous frame ( Alternatively, if the position and orientation in the current frame predicted from the past calculated values are used, this is used as a temporary measurement value and the same processing as in FIG. And an attitude calculation algorithm can be selected.

(Modification 4)
In each of the above embodiments, when three or more indices are detected as a scale for measuring the distribution of the indices, the convex hull formed by the indices is obtained and the area is used. However, the scale for measuring the distribution of the index is not limited to this, and other scales may be used. For example, the index farthest from the image center is selected as the first index, the index farthest from the first index is selected as the second index, and the first and second indices are further selected. The index that is farthest from the connecting line is selected as the third index, and the area of the triangle formed by these first, second, and third indices can be used as a distribution scale instead of the convex hull area. good. For simplicity, the distance between the first and second indices may be used as a scale of the distribution, or any other arbitrary value that can represent the distribution numerically, such as dispersion of image coordinates or standard deviation of the indices. Information may be used as a measure of distribution.

(Modification 5)
In step S3035 in the first embodiment, as described in detail in step S4040 of FIG. 4, the steepest descent method expressed by equation 11 is used to calculate the correction value Δs based on the error vector U and the matrix Θ. ing. However, the correction value Δs need not always be calculated by the steepest descent method. For example, the LM method (Levenberg-Marquardt method) which is an iterative solution of a known nonlinear equation may be obtained, or a statistical method such as M estimation which is a known robust estimation method may be combined. It goes without saying that the essence of the invention is not impaired by any numerical calculation method. Further, by using s as a state vector and Equation 4 as an observation equation in the above-described embodiment, there is an effect that only some of the parameters measured by the position and orientation sensor are corrected using the image Jacobian. An extended Kalman filter and an iterative extended Kalman filter can be configured. Note that the extended Kalman filter and the iterative extended Kalman filter are well known as described in Non-Patent Document 2, and thus further description regarding the details is omitted.

(Modification 6)
In each of the above embodiments, an index (hereinafter referred to as a point index) that represents one coordinate is used. However, an index other than the point index can be used. For example, you may use the parameter | index (henceforth a line parameter | index) comprised by a line feature like what is used for a well-known position and orientation measurement apparatus (for example, refer nonpatent literature 3). For example, an error vector U is constituted by an error calculated from a detected value from an image and an estimated value by s using a distance from the origin to a line index as a reference for evaluation, and a solution obtained by partial differentiation of the observation equation by each element of s By configuring the matrix Θ by a 1 × 6 Jacobian matrix having as elements, position and orientation can be measured (corrected) in the same manner as in the above embodiment. In addition, it is also possible to use these features together by stacking error and image Jacobian obtained from line indexes and point indexes, and other indexes.

(Modification 7)
In each of the above embodiments, the subsequent processing is selected according to the distribution of the index with respect to the image area. However, a relative ratio to the measurement target object may be used as a scale. For example, in the first embodiment, in step S3040, in addition to the area S _H of the convex hull formed by the total detected indices, the measurement object 170 onto the image (three-dimensional data representing the general shape is stored The area S _O of the projected image is obtained, and these ratios (S _H / S _O ) are obtained. Then, conditional branching in steps S3050 and S3060 may be performed using a threshold that is appropriately set for the ratio.

It is also possible to use the distribution of the index in the physical space as a scale, not in the image. For example, in the first embodiment, in step S3040, a three-dimensional convex hull composed of the three-dimensional coordinates of all detection indices in the object coordinate system is obtained, the volume V _H is calculated, and the volume V of the measurement target object 170 is calculated. By obtaining the ratio (V _H / V _O ) with _O , the ratio of the distribution of the detection index on the measurement target object 170 is derived. Then, conditional branching in steps S3050 and S3060 may be performed using a threshold that is appropriately set for the ratio. Further, the area of a triangle formed by the detection indices of the three points farthest from each other in the object coordinate system is obtained, and the ratio to the size of the measurement target object 170 (for example, the rectangular area formed by the two longest sides of the circumscribed cuboid) Can be obtained and used as a scale.

  An object of the present invention is to supply a storage medium (or recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus. Is also achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the above-mentioned storage medium, the program code corresponding to the flowchart described above is stored in the storage medium.

It is a figure which shows the structure of the position and orientation measurement apparatus in 1st Embodiment. It is a figure which shows the basic composition of the computer which can implement | achieve each part of a position and orientation measurement apparatus with software. 6 is a flowchart illustrating processing for calculating the positions and orientations of the imaging device 130 and the measurement target object 170 in the embodiment. It is a flowchart explaining the detail of the position and attitude | position calculation process in step S3035 of FIG. It is a flowchart explaining the detail of the position and attitude | position calculation process in step S3070 of FIG. 6 is a diagram illustrating an example of a captured image acquired by capturing a measurement target object 170 with the imaging device 130. FIG. 6 is a diagram illustrating an example of a captured image acquired by capturing a measurement target object 170 with the imaging device 130. FIG. 6 is a diagram illustrating an example of a captured image acquired by capturing a measurement target object 170 with the imaging device 130. FIG. It is a figure which shows the convex hull obtained when the process of step S3040 is performed with respect to the picked-up image (captured image 600) of FIG. It is a figure which shows the convex hull obtained when the process of step S3040 is performed with respect to the picked-up image (photographed image 700) of FIG. It is a figure which shows the convex hull obtained when the process of step S3040 is performed with respect to the picked-up image (photographed image 800) of FIG. It is a figure which shows the structure of the position and orientation measurement apparatus in 2nd Embodiment. It is a figure which shows the structure of the position and orientation measurement apparatus in 3rd Embodiment. It is a figure which shows the structure of the position and orientation measuring apparatus in 4th Embodiment. It is a figure which shows the structure of the position and orientation measurement apparatus in 5th Embodiment. 14 is a flowchart for describing processing for calculating the position and orientation of an imaging apparatus according to a fifth embodiment.

Claims (26)

A position and orientation correction device that corrects one or more of a plurality of parameters constituting the position and orientation of an imaging device,
  Position and orientation input means for inputting the plurality of parameters;
  Image input means for inputting an image captured by the imaging device with a plurality of indicators arranged in real space;
  Detecting means for detecting an actual value of image coordinates of the index from the image;
  A determination unit for determining which of the plurality of parameters is to be corrected based on a detection result by the detection unit;
  Based on the plurality of parameters and the actually measured value, the parameter that has not been determined as the correction target parameter by the determining unit is not corrected, and the parameter determined as the correction target parameter is not corrected. Correction means for correcting;
Have
  The correction means is
    A theoretical value calculating means for calculating a theoretical value of the image coordinates of the index based on the coordinates of the index and the plurality of parameters;
    Correction value calculation means for calculating a correction value of the parameter determined as the correction target so as to reduce an error between the actual measurement value and the theoretical value;
    Parameter correcting means for correcting the parameter to be corrected and the determined parameter without correcting the parameter to be corrected among the plurality of parameters based on the correction value;
A position / orientation correction apparatus comprising: The position and orientation correction apparatus according to claim 1, wherein the index is fixed in the real space and has a known position. The index is attached to an object to be imaged that exists in real space,
  The position / orientation correction apparatus includes:
  Imaging target position and orientation input means for inputting a plurality of parameters constituting the position and orientation of the imaging target object
The position / orientation correction apparatus according to claim 1, further comprising: A position and orientation correction apparatus that corrects a plurality of parameters constituting the position and orientation of a measurement target object,
  Position and orientation input means for inputting the plurality of parameters;
  An image input means for inputting an image obtained by imaging an index placed on the measurement target object with an imaging device;
  Detecting means for detecting an actual measurement value of the image coordinates of the index from the captured image;
  A determination unit for determining which of the plurality of parameters is to be corrected based on a detection result by the detection unit;
  Based on the plurality of parameters and the actually measured value, the parameter that has not been determined as the correction target parameter by the determination unit is corrected without correcting the parameter that has been determined as the correction target parameter. Correction means for
Have
  The correction means is
    Based on the plurality of parameters, a theoretical value calculation means for calculating a theoretical value of the image coordinates of the index;
    Correction value calculating means for calculating a correction value of the parameter to be corrected and the determined parameter so as to reduce an error between the actual measurement value and the theoretical value;
    Parameter correcting means for correcting the parameter to be corrected and the determined parameter without correcting the parameter to be corrected among the plurality of parameters based on the correction value;
A position / orientation correction apparatus comprising: 5. The position / orientation correction apparatus according to claim 1, wherein the plurality of parameters input by the position / orientation input unit are measurement values measured by a position / orientation sensor. 6. Of the plurality of parameters input by the position and orientation input means, the orientation component is a measurement value measured by an orientation sensor,
  The determining means determines all the posture components of the plurality of parameters as parameters to be corrected in preference to the position components of the plurality of parameters based on the detection result by the detecting means. The position and orientation correction apparatus according to any one of claims 1 to 4. The position / orientation correction apparatus according to claim 4, wherein the position / orientation input unit is fixed in a real space and inputs a plurality of parameters constituting a known position / orientation. The correction value calculation unit calculates the correction value based on an error between the actual measurement value and the theoretical value and an image Jacobian of the index, according to any one of claims 1 to 7. The position and orientation measurement apparatus described. The position / orientation correction apparatus according to claim 1, wherein the determination unit determines a number of parameters based on the detection result as a parameter to be corrected. 10. The position according to claim 1, wherein the plurality of parameters are parameters corresponding to a total of six components including three position components indicating a position and three posture components indicating a position. Attitude correction device. The determination unit, when there is one index detected by the detection unit, determines a parameter corresponding to two or less components among the plurality of parameters as the parameter to be corrected. The position and orientation correction apparatus according to 10. The determination unit, when there are two indexes detected by the detection unit, determines a parameter corresponding to three or less components among the plurality of parameters as the parameter to be corrected. The position and orientation correction apparatus according to 10 or 11. When the determination means has two indices detected by the detection means, based on the distance between the image coordinates of the indices, parameters corresponding to three or less components among the plurality of parameters are corrected. The position / orientation correction apparatus according to claim 12, wherein the position / orientation correction apparatus is determined as a parameter. When the determination means has two indices detected by the detection means, and the distance between the image coordinates of the indices is equal to or greater than a preset distance threshold, three components of the plurality of parameters The position / orientation correction apparatus according to claim 13, wherein a parameter corresponding to less than three components is determined as the parameter to be corrected if the parameter corresponding to is less than the distance threshold. The determining unit determines a parameter corresponding to six or less components among the plurality of parameters as the correction target parameter when there are three or more indices detected by the detecting unit. Item 15. The position and orientation correction apparatus according to any one of Items 11 to 14. When the determination means has three or more indices detected by the detection means, it corresponds to 6 components or less of the plurality of parameters based on the area of the area obtained from the image coordinates of the detected indices. The position / orientation correction apparatus according to claim 15, wherein a parameter to be corrected is determined as a parameter to be corrected. When the determination means has three or more indices detected by the detection means, and the area of the region obtained from the image coordinates of the detected indices is equal to or more than a preset area threshold, the plurality 17. The position according to claim 16, wherein a parameter corresponding to six components is selected as a correction target, and a parameter corresponding to less than six components is determined as a correction target parameter if the parameter is less than a threshold of the distance. Attitude correction device. The position / orientation correction apparatus according to claim 16, wherein the area is an area of a convex hull obtained from image coordinates of the detected index. A position and orientation correction device that corrects a plurality of parameters constituting the position and orientation of an imaging device,
  Position and orientation input means for inputting the plurality of parameters;
  An image input means for inputting an image captured by the imaging device, with a plurality of indicators arranged in real space and having known positions;
  Detecting means for detecting an actual value of image coordinates of the index from the image;
  A determination unit for determining which of the plurality of parameters is to be corrected based on a detection result by the detection unit;
  Based on the plurality of parameters and the actually measured value, the parameter that has not been determined as the correction target parameter by the determination unit is corrected without correcting the parameter that has been determined as the correction target parameter. Correction means for
Have
  The correction means is
    Based on the position of the index and the plurality of parameters, theoretical value calculation means for calculating a theoretical value of the image coordinates of the index;
    Correction value calculating means for calculating a correction value of the parameter to be corrected and the determined parameter so as to reduce an error between the actual measurement value and the theoretical value;
    Parameter correcting means for correcting the parameter to be corrected and the determined parameter without correcting the parameter to be corrected among the plurality of parameters based on the correction value;
Have
  The plurality of parameters are parameters corresponding to six components of three position components indicating positions and three posture components indicating postures,
  The determining means is
    When the index detected by the detecting means is one, a parameter corresponding to two or less components among the plurality of parameters is determined as the parameter to be corrected,
    When there are two indices detected by the detection means, a parameter corresponding to three or less components among the plurality of parameters is selected based on the distance between the image coordinates of the detected indices. Determined as
    When there are three or more indices detected by the detection means, parameters corresponding to six or less components among the plurality of parameters are selected based on the area of the area obtained from the image coordinates of the detected indices. A position / orientation correction apparatus characterized by being determined as a parameter to be corrected. A position and orientation correction apparatus that corrects a plurality of parameters constituting the position and orientation of a measurement target object,
  Position and orientation input means for inputting the plurality of parameters;
  An image input means for inputting an image obtained by imaging an index placed on the measurement target object with an imaging device;
  Detecting means for detecting an actual measurement value of the image coordinates of the index from the captured image;
  A determination unit for determining which of the plurality of parameters is to be corrected based on a detection result by the detection unit;
  Based on the plurality of parameters and the actually measured value, the parameter that has not been determined as the correction target parameter by the determination unit is corrected without correcting the parameter that has been determined as the correction target parameter. Correction means for
Have
  The correction means is
    Based on the plurality of parameters, a theoretical value calculation means for calculating a theoretical value of the image coordinates of the index;
    Correction value calculating means for calculating a correction value of the parameter to be corrected and the determined parameter so as to reduce an error between the actual measurement value and the theoretical value;
    Parameter correcting means for correcting the parameter to be corrected and the determined parameter without correcting the parameter to be corrected among the plurality of parameters based on the correction value;
Have
  The plurality of parameters are parameters corresponding to six components of three position components indicating positions and three posture components indicating postures,
  The determining means is
    When the index detected by the detecting means is one, a parameter corresponding to two or less components among the plurality of parameters is determined as the parameter to be corrected,
    When there are two indices detected by the detection means, a parameter corresponding to three or less components among the plurality of parameters is selected based on the distance between the image coordinates of the detected indices. Determined as
    When there are three or more indices detected by the detection means, parameters corresponding to six or less components among the plurality of parameters are selected based on the area of the area obtained from the image coordinates of the detected indices. A position / orientation correction apparatus characterized by being determined as a parameter to be corrected. A position and orientation correction method performed by a position and orientation correction apparatus that corrects a plurality of parameters constituting the position and orientation of an imaging apparatus,
  A position and orientation input step in which the position and orientation input means of the position and orientation correction apparatus inputs the plurality of parameters;
  An image input step of inputting an image in which a plurality of image input means of the position and orientation correction device are arranged in the real space and the position is known, and is captured by the imaging device;
  A detection step in which the detection means of the position and orientation correction apparatus detects an actual measurement value of the image coordinates of the index from the image;
  A determination step in which the determination unit of the position and orientation correction device determines which of the plurality of parameters is to be corrected based on the detection result of the detection step;
  The correction means of the position / orientation correction apparatus does not correct the parameter that has not been determined as the correction target parameter by the determination step, based on the plurality of parameters and the actually measured value, and the correction A correction process for correcting the target parameter and the determined parameter,
  The correction step includes
    A theoretical value calculating step of calculating a theoretical value of image coordinates of the index based on the position of the index and the plurality of parameters;
    A correction value calculation step of calculating a correction value of the parameter to be corrected and the parameter determined so that the correction value calculation unit of the correction unit reduces an error between the actual measurement value and the theoretical value;
    Based on the correction value, the parameter correction unit of the correction unit does not correct the parameter that is not determined as the correction target parameter among the plurality of parameters, and corrects the parameter that is determined as the correction target parameter. Parameter correction process to
A position and orientation correction method characterized by comprising: A position and orientation correction method performed by a position and orientation correction apparatus that corrects a plurality of parameters constituting the position and orientation of a measurement target object,
  A position and orientation input step in which the position and orientation input means of the position and orientation correction apparatus inputs the plurality of parameters;
  An image input step in which the image input means of the position / orientation correction apparatus inputs an image obtained by imaging an index placed on the measurement target object by an imaging apparatus;
  A detection step in which the detection means of the position and orientation correction device detects an actual measurement value of the image coordinates of the index from the captured image;
  A determination step in which the determination unit of the position and orientation correction device determines which of the plurality of parameters is to be corrected based on the detection result of the detection step;
  The correction means of the position / orientation correction apparatus does not correct the parameter that has not been determined as the correction target parameter by the determination step, based on the plurality of parameters and the actually measured value, and the correction A correction process for correcting the target parameter and the determined parameter,
  The correction step includes
    A theoretical value calculating step in which the theoretical value calculating means of the correcting means calculates the theoretical value of the image coordinates of the index based on the plurality of parameters;
    A correction value calculation step of calculating a correction value of the parameter to be corrected and the parameter determined so that the correction value calculation unit of the correction unit reduces an error between the actual measurement value and the theoretical value;
    Based on the correction value, the parameter correction unit of the correction unit does not correct the parameter that is not determined as the correction target parameter among the plurality of parameters, and corrects the parameter that is determined as the correction target parameter. Parameter correction process to
A position and orientation correction method characterized by comprising: A position and orientation correction method performed by a position and orientation correction apparatus that corrects a plurality of parameters constituting the position and orientation of an imaging apparatus,
  A position and orientation input step in which the position and orientation input means of the position and orientation correction apparatus inputs the plurality of parameters;
  An image input step of inputting an image in which a plurality of image input means of the position and orientation correction device are arranged in the real space and the position is known, and is captured by the imaging device;
  A detection step in which the detection means of the position and orientation correction apparatus detects an actual measurement value of the image coordinates of the index from the image;
  A determination step in which the determination unit of the position and orientation correction device determines which of the plurality of parameters is to be corrected based on the detection result of the detection step;
  The correction means of the position / orientation correction apparatus does not correct the parameter that has not been determined as the correction target parameter by the determination step, based on the plurality of parameters and the actually measured value, and the correction A correction process for correcting the target parameter and the determined parameter;
Have
  The correction step includes
    A theoretical value calculating step of calculating a theoretical value of image coordinates of the index based on the position of the index and the plurality of parameters;
    A correction value calculation step of calculating a correction value of the parameter to be corrected and the parameter determined so that the correction value calculation unit of the correction unit reduces an error between the actual measurement value and the theoretical value;
    Based on the correction value, the parameter correction unit of the correction unit does not correct the parameter that is not determined as the correction target parameter among the plurality of parameters, and corrects the parameter that is determined as the correction target parameter. Parameter correction process to
Have
  The plurality of parameters are parameters corresponding to six components of three position components indicating positions and three posture components indicating postures,
  In the determination step,
    When the number of indices detected in the detection step is one, a parameter corresponding to two or less components among the plurality of parameters is determined as the parameter to be corrected,
    When there are two indices detected in the detection step, parameters corresponding to three or less components among the plurality of parameters based on the distance between the image coordinates of the detected indices are parameters to be corrected. Determined as
    When there are three or more indices detected in the detection step, parameters corresponding to 6 components or less of the plurality of parameters are selected based on the area of the area obtained from the image coordinates of the detected indices. A position / orientation correction method, wherein the position / orientation correction method is determined as a correction target parameter. A position and orientation correction method performed by a position and orientation correction apparatus that corrects a plurality of parameters constituting the position and orientation of a measurement target object,
  A position and orientation input step in which the position and orientation input means of the position and orientation correction apparatus inputs the plurality of parameters;
  An image input step in which the image input means of the position and orientation correction apparatus inputs an image obtained by imaging a plurality of indices arranged on the measurement target object;
  A detection step in which the detection means of the position and orientation correction apparatus detects an actual measurement value of the image coordinates of the index from the image;
  A determination step in which the determination unit of the position and orientation correction device determines which of the plurality of parameters is to be corrected based on the detection result of the detection step;
  The correction means of the position / orientation correction apparatus does not correct the parameter that has not been determined as the correction target parameter by the determination step, based on the plurality of parameters and the actually measured value, and the correction A correction process for correcting the target parameter and the determined parameter;
Have
  The correction step includes
    A theoretical value calculating step in which the theoretical value calculating means of the correcting means calculates the theoretical value of the image coordinates of the index based on the plurality of parameters;
    A correction value calculation step of calculating a correction value of the parameter to be corrected and the parameter determined so that the correction value calculation unit of the correction unit reduces an error between the actual measurement value and the theoretical value;
    Based on the correction value, the parameter correction unit of the correction unit does not correct the parameter that is not determined as the correction target parameter among the plurality of parameters, and corrects the parameter that is determined as the correction target parameter. Parameter correction process to
Have
  The plurality of parameters are parameters corresponding to six components of three position components indicating positions and three posture components indicating postures,
  In the determination step,
    When the number of indices detected in the detection step is one, a parameter corresponding to two or less components among the plurality of parameters is determined as the parameter to be corrected,
    When there are two indices detected in the detection step, parameters corresponding to three or less components among the plurality of parameters based on the distance between the image coordinates of the detected indices are parameters to be corrected. Determined as
    When there are three or more indices detected in the detection step, parameters corresponding to 6 components or less of the plurality of parameters are selected based on the area of the area obtained from the image coordinates of the detected indices. A position / orientation correction method, wherein the position / orientation correction method is determined as a correction target parameter. The program for making a computer perform each process of the position and orientation correction | amendment method of any one of Claim 21 thru | or 24. A computer-readable recording medium storing the program according to claim 25.

Images