JP4810403B2 - Information processing apparatus and information processing method - Google Patents

Description

  The present invention relates to a technique for estimating the placement information of indices placed in a three-dimensional space.

  In recent years, research on mixed reality (MR) technology has been active. MR technology is a technology that seamlessly fuses real space and virtual space created by computers. Among MR technologies, Augmented Reality (also called AR, augmented reality, augmented reality) technology that displays virtual space superimposed on real space is attracting particular attention.

  An AR image display device is realized by either a video see-through method or an optical see-through method.

  The former is a virtual space (virtual object or character information drawn by computer graphics (CG), etc.) generated in accordance with the position and orientation of the imaging device in an image of the real space taken by the imaging device such as a video camera. This is a method for displaying a composite image obtained by superimposing and rendering the above image.

  The latter is a method of displaying an image of a virtual space generated according to the position and orientation of the viewer's viewpoint on an optical see-through display mounted on the viewer's head.

  AR technology includes various operations such as surgery support that superimposes the state of the body on the patient's body surface, architectural simulation that superimposes the virtual building on the vacant land, and assembly support that superimposes the work procedure and wiring during assembly work. Application to the field is expected.

  One of the most important issues in AR technology is how to accurately align the real space and the virtual space, and many efforts have been made. In the case of the video see-through method, the alignment problem in the AR results in a problem of obtaining the position and orientation of the imaging device in the scene (that is, in the reference coordinate system). On the other hand, the optical see-through method results in a problem of obtaining the observer's viewpoint or display position and orientation in the scene.

  As a method for solving the former problem, the following method is generally used. First, a plurality of indicators are arranged or set in the scene. Then, the position and orientation of the imaging apparatus in the reference coordinate system are obtained from the correspondence between the projected position of the index in the image captured by the imaging apparatus and the position of the index in the reference coordinate system.

  As a method for solving the latter problem, the following method is generally used. An imaging device is attached to a measurement object (that is, an observer's head or display), the position and orientation of the imaging device are obtained by the same method as the former, and the position and orientation of the measurement object are obtained based thereon.

  FIG. 1 is a block diagram showing a functional configuration of a conventional position and orientation measurement apparatus. The position and orientation of the imaging apparatus are measured by associating the two-dimensional coordinates of the index detected from the image captured by the imaging apparatus with the three-dimensional position of the index in the reference coordinate system.

  As shown in FIG. 1, the conventional position / orientation measurement apparatus 100 includes an index detection unit 110 and a position / orientation calculation unit 120, and an imaging apparatus 130 that captures a real space is connected to the index detection unit 110. ing.

  In the real space, k indices Qk (k = 1, 2,..., K) whose positions in the reference coordinate system are known are arranged as indices for obtaining the position and orientation of the imaging device 130. Has been. In the example of FIG. 1, four indexes Q1, Q2, Q3, and Q4 are arranged, and three of them, Q1, Q3, and Q4, are included in the field of view of the imaging device 130.

  The index Qk is an index that can detect the projected position of the index in the captured image and can identify the index, such as a circular marker having a different color. Any form may be sufficient. For example, a natural feature point in a three-dimensional space may be used as an index and detected from the captured image by template matching.

  The real space image captured by the imaging device 130 is input to the index detection unit 110. The index detection unit 110 detects the image coordinates of the index Qk photographed on the image input from the imaging device 130. For example, when each of the indices Qk is composed of circular markers having different colors, an area corresponding to each marker color is detected from the input image, and the barycentric position of the detected area is determined as the area. The detected coordinates of the index corresponding to.

Then, the index detection unit 110 outputs the detected image coordinates u ^Mkn and the identifier kn of each index Qkn to the position / orientation calculation unit 120. Here, n (= 1, 2,..., P) is a variable representing the serial number of the detected index, and P represents the total number of detected indices. For example, in the case of FIG. 1, P = 3, and identifiers k1 = 1, k2 = 3, k3 = 4 and the corresponding image coordinates u ^Mk1 , u ^Mk2 , u ^Mk3 are output from the index detection unit 110. The

The position and orientation calculation unit 120, based on the correspondence between the image coordinates u ^Mkn indices Qkn of each detected, and the position in the reference coordinate system of the index Qkn held beforehand as known information, the imaging device 130 Is calculated. A method for calculating the position and orientation of an imaging device from a set of three-dimensional coordinates and image coordinates of an index has been proposed for a long time in the fields of computer vision and photogrammetry (see Non-Patent Document 1 and Non-Patent Document 2). ). The position / orientation calculation unit 120 calculates the position and orientation of the imaging device 130 by the method described in Non-Patent Document 1, for example.

  In addition, although the case where a plurality of point indexes existing in the three-dimensional space are used has been described here, for example, as disclosed in Non-Patent Documents 3 and 4, a square-shaped index (hereinafter, referred to as a square shape index) is known. A position / orientation calculation method of an imaging apparatus using a square index) has been proposed.

  In addition, as disclosed in Non-Patent Document 7, for example, a position / orientation calculation method for an imaging apparatus that combines a square index and a point index has also been proposed. The point index has the merit that it can be set even in a narrow place. In addition, the square index is easy to identify based on the result of recognizing the pattern on the index. Furthermore, since the amount of information of one index is large, the position and orientation of the imaging device can be obtained from only one index. There is a merit. As described above, since the point index and the square index have merits, they can be used complementarily.

  Conventionally, the position and orientation of the imaging device have been acquired based on the image captured by the imaging device by the above method.

  On the other hand, there are also methods disclosed in Patent Documents 1 and 2 and Non-Patent Document 5, for example. In other words, a 6-DOF position / orientation sensor such as a magnetic sensor or an ultrasonic sensor is attached to the imaging device to be measured, and the position and orientation are measured by using the detection of the index by image processing as described above. ing. Since the output value of the sensor can be obtained stably although the accuracy varies depending on the measurement range, the method using both the sensor and the image processing can improve the robustness as compared with the method using only the image processing.

  In the registration method using an index, in order to obtain the position and orientation in the reference coordinate system of the imaging device to be measured, the position in the reference coordinate system in the case of a point index, the position in the reference coordinate system in the case of a square index, and The posture and size need to be known. In the case of a square index, the square index itself is often used as a reference for the coordinate system without providing a reference coordinate system separately. However, when a plurality of square markers are used, the relationship between the relative position, posture, and size of each other needs to be known, so that the reference coordinate system is still necessary.

  The position, orientation and size of the index can be measured manually using a tape measure or a protractor or by a surveying instrument. However, measurement of the position and orientation of an index using an image has been conventionally performed due to problems of accuracy and labor. The position measurement of the point index can be performed by a method called a bundle adjustment method.

  The bundle adjustment method is based on a constraint condition that a point index is photographed from multiple directions by an imaging device, and that the point index, the projection point on the image of the point index, and the viewpoint of the imaging device exist on the same straight line. Is. Under such constraint conditions, an error (projection error) between the projection position where the index is actually observed on the image, the position and orientation of the imaging apparatus, and the projection position calculated from the position of the index is minimized. Then, the position and orientation of the imaging device that has captured the image by repeated calculation and the position of the point index are obtained.

Non-Patent Document 6 discloses a method for measuring the positions and orientations of a large number of square markers arranged in a three-dimensional space. In Non-Patent Document 6, the position and orientation of an imaging device that takes a large number of images of a large number of square markers arranged in a three-dimensional space and captures each image by iterative calculation so as to minimize the projection error, and a square The position and orientation of the marker are obtained.
Japanese Patent Laid-Open No. 11-084307 JP 2000-041173 A RM Haralick, C. Lee, K. Ottenberg, and M. Nolle: “Review and analysis of solutions of the three point perspective pose estimation problem”, Int'l. J. Computer Vision, vol.13, no.3, pp .331-356, 1994. MA Fischler and RC Bolles: “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography”, Comm. ACM, vol.24, no.6, pp.381-395, 1981. Junichi Kokimoto: “Augmented Reality Construction Method Using Two-Dimensional Matrix Code”, Interactive System and Software IV, Modern Science, 1996. Kato, M. Billinghurst, Asano, Tachibana: “Augmented reality system based on marker tracking and its calibration”, Transactions of the Virtual Reality Society of Japan, vol.4, no.4, pp.607-616, 1999. A. State, G. Hirota, DT Chen, WF Garrett and MA Livingston: “Superior augmented reality registration by integrating landmark tracking and magnetic tracking”, Proc. SIGGRAPH '96, pp. 429-438, 1996. G. Baratoff, A. Neubeck and H. Regenbrecht: “Interactive multi-marker calibration for augmented reality applications”, Proc. ISMAR2002, pp.107-116, 2002. H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto and K. Tachibana: “Virtual object manipulation on a table-top AR environment”, Proc. ISAR2000, pp.111-119, 2000.

  In the conventional method for measuring the position and orientation of an index using an image, the size of the index is obtained in advance by a manual operation using a tape measure or a protractor or a surveying instrument prior to the measurement of the position and orientation of the index. However, when many indicators of various sizes are used, a measurement technique using an image has been required due to the problem of labor.

  However, even if the size of the index is obtained using an image, a measurement error may occur. That is, the size of the index obtained using the image may be different from the size of the index actually designated when the user prints the index using a printer or the like. At this time, an error occurred in the measurement result of the position and orientation of the finally obtained index due to the size error. Therefore, it has been required that the size of the index used when finally obtaining the position and orientation of the index is the same as the size of the index actually created by the user.

  The present invention has been made in view of the above problems, and an object of the present invention is to obtain index placement information with higher accuracy.

  In order to achieve the object of the present invention, for example, an information processing method of the present invention comprises the following arrangement.

That is, an information processing method for estimating index placement information,
An image acquisition step in which the image acquisition means acquires an image in which a scene including the index is captured;
An index detecting step in which the index detecting means detects the index from the image;
A first index arrangement information estimation unit, wherein the first index arrangement information estimation unit estimates the arrangement information in the scene of the index detected in the index detection process;
An index information correcting step in which the index information correcting means corrects at least one of the arrangement information estimated in the first index arrangement information estimating step to a known value;
A second index arrangement information estimating unit, wherein the second index arrangement information estimating means estimates the arrangement information whose value is not corrected in the index information correcting process, using the arrangement information corrected in the index information correcting process as a constant; It is characterized by providing.

  According to the first aspect of the present invention, the index placement information can be obtained with higher accuracy. Further, according to the invention of claim 2 of the present application, the position, posture or size of the index can be obtained with higher accuracy.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
FIG. 2 is a block diagram showing a functional configuration of the system according to the present embodiment. As shown in the figure, the system according to the present embodiment includes an imaging unit 210 and an index position / posture estimation apparatus 2.

  First, the imaging unit 210 will be described. The imaging unit 210 is a camera that captures a moving image / still image in the real space, and captures a scene in the real space in which an index is arranged. One or more captured images are input to the image capturing unit 220 included in the index position / posture estimation apparatus 2.

  Next, the index position / posture estimation apparatus 2 will be described. As shown in the figure, the index position / posture estimation apparatus 2 includes an image capturing unit 220, an index detection / identification unit 230, an index position / posture size calculation unit 235, an index arrangement condition setting unit 240, and an index position / posture calculation unit 250. ing.

  The image capturing unit 220 receives the image input from the imaging unit 210 and sends it to the index detection / identification unit 230 at the subsequent stage. The index detection / identification unit 230 detects the index from the image, and identifies each detected index. The index position / posture size calculation unit 235 obtains the position, posture, and size of the index based on the index detection / identification result by the index detection / identification unit 230. The index placement condition setting unit 240 sets a constraint condition related to the placement of the index. In the present embodiment, the size of the index obtained by the index position / posture size calculation unit 235 is corrected to the size of the index actually created by the user. The index position / orientation calculation unit 250 calculates the position and orientation of the index so as to satisfy the constraint condition regarding the index arrangement set by the index arrangement condition setting unit 240 based on the index detection / identification result by the index detection / identification unit 230.

  FIG. 3 is a block diagram showing a hardware configuration of a computer 300 applicable to the index position / posture estimation apparatus 2. Reference numeral 301 denotes a CPU that controls the entire computer 300 using programs and data stored in the ROM 306 and the RAM 307 and executes processing described later that is performed by the index position / posture estimation apparatus 2. That is, it functions as the image capturing unit 220, the index detection / identification unit 230, the index position / posture size calculation unit 235, the index arrangement condition setting unit 240, and the index position / posture calculation unit 250 shown in FIG. Therefore, in the following description, when each of these units is described as an operation subject, it should be interpreted that the CPU 301 functions as each of these units.

  A graphic accelerator 302 performs partial processing related to image processing, such as generating an image of a virtual space. The processed image is output to a display unit (not shown) included in the HMD (head mounted display) 304.

  An image capturing device 310 sends an image input from the camera 311 functioning as the imaging unit 210 to the hard disk 305 or the RAM 307. The image capturing device 310 performs A / D conversion, appropriate signal processing, and the like on the image signal input from the camera 311 although it is not the purpose of the description here and will not be described in detail. As a result, the image captured by the camera 311 is sent to the hard disk 305 and the RAM 307 as image data having an appropriate format.

  The hard disk 305 stores an OS (operating system) and programs and data for causing the CPU 301 to execute each process described later that is performed by the index position / posture estimation apparatus 2. The program includes a program for causing the CPU 301 to execute the functions of the image capturing unit 220, the index detection / identification unit 230, the index position / posture size calculation unit 235, the index placement condition setting unit 240, and the index position / posture calculation unit 250. include. The hard disk 305 also stores what is described later as known information. As described above, the programs and data stored in the hard disk 305 are appropriately loaded into the RAM 307 according to the control by the CPU 301. Since the CPU 301 executes processing using the loaded program and data, the computer 300 executes each processing described later as what the index position / orientation estimation apparatus 2 performs.

  The ROM 306 stores setting data and a boot program for the computer 300.

  The RAM 307 has an area for temporarily storing programs and data loaded from the hard disk 305, image data input from the image capturing device 310, and the like. Further, the RAM 307 also has a work area used when the CPU 301 executes various processes. That is, the RAM 307 can provide various areas as appropriate.

  Reference numerals 308 and 309 are a mouse and a keyboard, respectively, and various instructions can be input to the CPU 301 by an operator of the computer 300.

  Reference numeral 380 denotes an I / F (interface) for connecting the printer 312 to the computer 300. Print data to the printer 312 is sent from the computer 300 to the printer 312 via the I / F 380. The

  A bus 399 connects the CPU 301, graphic accelerator 302, image capture device 310, hard disk 305, ROM 306, RAM 307, mouse 308, keyboard 309, and I / F 380.

  As is well known, the printer 312 forms images, characters, and the like on a recording medium such as paper according to print data sent from the computer 300. The printer 312 may be configured by any device as long as it has a printing function.

  Next, an outline of the operation of the index position / posture estimation apparatus 2 having the above configuration will be described.

  First, indices used in the present embodiment will be described. FIG. 4 is a diagram for explaining indices used in the present embodiment. The index shown in FIG. 4 is a plane index in which the arrangement information is expressed by the position and orientation in the three-dimensional space. In the present embodiment, a square index is used as a plane index, but the plane index is not limited to a square index, and any form may be used as long as it is a plane graphic index. The position of the square index in the three-dimensional space is assumed to be the center of the square, that is, the intersection of two diagonal lines of the square. In addition, as shown in FIG. 4, a square index coordinate system is taken so that the normal direction of the index is the z-axis, and the x-axis and y-axis are parallel to two adjacent sides of the index. The attitude of the coordinate system of the square index that has been viewed is set as the attitude of the square index. The square index in the present embodiment is surrounded by a black frame so that it can be easily detected on the image, and the square inside the black frame is used as the square index.

  The size of the square index is assumed to be the length in the x-axis or y-axis direction shown in FIG. That is, the length of one side of the square is set as the index size. Although only one type of index size is shown in FIG. 4, indexes of different sizes are actually used. It is assumed that the user prints one or more indicators in advance and applies them to a wall, floor, ceiling, object in the scene, or the like before executing the method of the present embodiment.

<1. Shooting indicators>
The image capturing unit 210 captures a large number of scene images in which indices are arranged, and each captured image is output to the image capturing unit 220. The image capturing unit 220 sends each received image to the subsequent index detection and identification unit 230.

<2. Index detection and identification>
The index detection / identification unit 230 detects the index from the image received from the image capturing unit 220 and performs identification. Here, the detection of the index is to obtain the image coordinates of the index on the two-dimensional image, and in the case of a square index, to obtain the image coordinates of each vertex. The image coordinates of the index may be obtained automatically, or may be manually designated by the user clicking on the image with a mouse. When automatically obtaining the image coordinates of each vertex of the square index, for example, the image is binarized and the black frame area of the square index is detected. Then, a rectangle inscribed in the black frame region is obtained, and the image coordinates of each vertex of the obtained rectangle are obtained as the image coordinates of each vertex of the square index. On the other hand, when obtaining manually, the image coordinate instruct | indicated by the user is calculated | required as an image coordinate of a parameter | index.

  The identification of the index is to uniquely identify the index detected in the image. The identification of the index may be performed automatically or manually by the user. In the case of automatically identifying a square index, for example, a unique pattern is provided for each index, and the identification is performed by recognizing a pattern possessed by each index from an image obtained by capturing the index. In the latter case, identification is performed in response to user input.

  The index image coordinate detection process and the index identification process performed by the index detection / identification unit 230 are well-known techniques, and thus detailed description thereof is omitted.

<3. Index position / posture size calculation>
The index position / orientation size calculation unit 235 obtains the position, orientation, and size of the index present in the scene.

  More specifically, the index position / posture size calculation unit 235 calculates the position, posture, and size of the index in the reference coordinate system based on the detection / identification result of the index by the index detection / identification unit 230. Hereinafter, details of the processing will be described.

First, perspective projection conversion will be described. FIG. 6 is a diagram illustrating the camera (imaging unit 210) coordinate system and the screen coordinate system. The origin o _i of the screen coordinate system is the intersection of the visual axis and the image plane, and the horizontal direction of the image plane is the _xi axis and the vertical direction is the y _i axis. The distance between the origin o _c and the image plane of the camera coordinate system (focal length) and f. The z _c axis of the camera coordinate system is the direction opposite to the viewing direction along the viewing axis of the image pickup unit 210, the x _c axis is parallel to the horizontal direction of the image, and the y _c axis is parallel to the vertical direction of the image.

  Point on the camera coordinate system by perspective projection transformation

As shown in the following formula (1), the screen coordinates are

Is projected to a point.

  In the present embodiment, it is assumed that the lens distortion in the imaging unit 210 does not exist or has been corrected, and the imaging unit 210 is assumed to be a pinhole camera. In Expression (1), as shown in FIG. 7, a point 701 in the space, a projection point 702 on the image of this point 701, and the imaging unit 210 (camera) position (viewpoint) 703 are collinear (collinear). ) And is also called a collinear conditional expression. FIG. 7 is a diagram for explaining the collinear conditional expression.

  The position of the imaging unit 210 in the reference coordinate system

, The orientation of the imaging unit 210 (actually, the orientation of the reference coordinate system with respect to the camera coordinate system)

Suppose that

Is a posture expression method with three degrees of freedom, and the posture is expressed by a rotation axis vector and a rotation angle. When the rotation angle and r _a, r _a is

Is represented by the following formula (2).

  Also, the rotation axis vector

Then,

When

The relationship is expressed as shown in Equation (3).

(Rotation angle r _a , rotation axis vector

) And 3 × 3 rotation transformation matrix

Is expressed as the following equation (4).

  A point on the reference coordinate system

Camera coordinates

Is

,as well as

Is expressed by the following equation (5).

  From equations (1) and (5), a point on the reference coordinates is obtained by perspective projection transformation.

Is a point on the image as shown in equation (6) below

Projected on.

  Ideally

,

,

The projection position (calculation position) calculated from the equation (6) based on the above is coincident with the position actually observed on the image (observation position). Therefore, the horizontal deviation between the calculated position and the observation position is F, the vertical deviation is G, and the observation position is

Then, F and G are 0 as shown in the following formula (7).

  F and G are the positions of the imaging unit 210

, Imager 210 posture

, The position of the observation point on the reference coordinates

Is a function related to The square index is the position in the reference coordinate system

And attitude to the reference coordinate system

(

3 × 3 rotation transformation matrix corresponding to

And the scale in the reference coordinate system

(Since the scale is based on the center of the index shown in FIG. 4, the index size is twice the index scale). The position of the vertex of the square index in the square index coordinate system

And The position of the vertex of the square index in the reference coordinate system

Is the following equation (8):

When

When

Function.

  Therefore, as shown in the following equation (9), F and G are the positions of the imaging unit 210.

, Imager 210 posture

, Square indicator position

And the attitude of square indicators

And square indicator scale

Is a function of

  Expression (9) is a nonlinear equation regarding the position and orientation of the imaging unit 210, the position and orientation of the index, and the scale. Therefore, using Taylor expansion up to the first order term, linearization is performed in the vicinity of the approximate value of the position and orientation of the imaging unit 210, the position and orientation of the index, and the scale. Find position, posture and scale.

  Equation (10) is a linearization of Equation (9). Δtx, Δty, and Δtz are the positions of the imaging unit 210, and Δωx, Δωy, and Δωz are the postures of the imaging unit 210. Further, Δtsx, Δtsy, and Δtsz represent the position of the square marker, Δωsx, Δωsy, and Δωsz represent the posture of the square marker, and ΔL represents the correction amount for the approximate value of the square marker scale.

Here, F ⁰ and G ⁰ in Expression (10) are constants, and the projected position (calculated position) and observation position of the index obtained from the image capturing unit 210 position and orientation, the square index position and orientation, and the approximate value of the scale.

Is the difference.

  Expression (10) is an observation equation for one vertex of a square index observed on a certain image. Actually, since each vertex of the square index is observed on a plurality of images, a plurality of equations (10) are obtained. Therefore, a plurality of equations (10) are solved as simultaneous equations regarding correction values for the approximate values of the position and orientation of the imaging unit 210, the position and orientation of the square index, and the scale, and the approximation is corrected. As a result, the position and orientation of the imaging unit 210, the position and orientation of the square index, and the scale are obtained.

  FIG. 8 is a flowchart of processing for obtaining the position, posture, and size (twice the index scale) of the square index by the index position / posture size calculation unit 235. As will be described in detail later, when the index position / orientation calculation unit 250 obtains the position and orientation of the square index, substantially the same processing as the processing according to the flowchart of FIG. 8 is performed.

  Note that, in the previous stage of performing the processing according to the flowchart of the same figure, it is assumed that the shooting of the scene including the index and the index extraction and identification process from the captured image have been completed.

  First, in step S100, the index position / orientation size calculation unit 235 sets the position and orientation of the imaging unit 210 that captured each image, the position and orientation of the square index, and the approximate value of the scale.

  In step S110 and subsequent steps, a correction value for correcting the approximate value set in step S100 is obtained.

  In step S110, the index position / orientation size calculation unit 235 simultaneously calculates correction values for the approximate values that minimize the projection error from the approximate values of the position and orientation of the imaging unit 210, the position and orientation of the square index, and the scale. A process for determining simultaneous equations to be obtained is performed.

  Next, in step S120, a numerical calculation process for solving the simultaneous equations determined in step S110 is performed to obtain a correction value for each approximate value.

  In step S130, the approximate value is corrected using the correction value obtained in step S120 to obtain a new approximate value.

  Next, in step S140, it is determined whether or not the position and orientation of the imaging unit 210 corrected in step S130, the position and orientation of the square index, and the scale have converged to optimum values. If the result of such determination is that it has not converged, the processing is returned to step S110, and the subsequent processing is repeated. On the other hand, if it has converged, the processing by the index position and orientation size calculation unit 235 according to the flowchart of FIG.

  Next, the process in each step shown in FIG. 8 will be described in more detail.

  First, step S100 will be described.

  In step S100, the approximate values of the position and orientation of the imaging unit 210 that captured each image, the position and orientation of the square index, and the scale are set. Here, the number of captured images is N, and the position of the imaging unit 210 that captured each image is

, Posture

And

In addition, the number of square indices for obtaining the position, orientation and scale is K _s1 , and the positions of the square indices are

And the posture

And the scale is Li (i = 1,..., K _s1 ).

  The approximate value of the position and orientation of the imaging unit 210 may be obtained from the output value of this sensor by attaching a 6-DOF position and orientation sensor such as a magnetic sensor to the imaging unit 210, or the position in the reference coordinate system is known. You may obtain | require from the correspondence of the point of this, and the projection position on the image of this point. Alternatively, the position and orientation of the imaging unit 210 may be mechanically obtained as an approximate value like a motion control camera, or an image is taken using the imaging unit 210 whose position and orientation have been calibrated in advance, and the calibration result is obtained. The approximate value may be used. That is, acquisition and setting of approximate values of the position and orientation of the imaging unit 210 are not limited to a specific method.

  On the other hand, approximate values of the index position, orientation, and scale are taken from a plurality of positions using an imaging unit 210 equipped with a six-degree-of-freedom position orientation sensor such as a magnetic sensor, and stereo based on the plurality of images. You may measure and use the value calculated | required roughly. In addition, the image capturing unit 210 that can measure the position and orientation mechanically like a motion control camera and the image capturing unit 210 whose position and orientation have been calibrated in advance are photographed from a plurality of positions, and based on the plurality of images. A stereo measurement may be performed and a value roughly determined may be used. Moreover, you may use the value estimated once by the method of this embodiment.

  As described above, the acquisition form of any approximate value is not particularly limited.

  In step S100, one or more indices with a known size pasted in the scene are photographed so as to serve as a reference for the index scale in the reference coordinate system. By doing so, the relationship between the index scale obtained in steps S110 to S140 and the scale of the reference coordinate system (that is, the relationship between the index scale obtained in steps S110 to S140 and the size of the index existing in the scene). Can be associated.

Next, in step S110, the observation equation of Expression (10) is established for the number of indices observed on the image. Expression (10) is an observation equation that includes the correction values of the position and orientation of one imaging unit 210 and the correction values of the position, orientation, and scale of one square index as unknown parameters. Here, as shown in the following equation (11), observation equations are established for the positions and postures of N cameras and the positions, postures, and scales of K _s1 square markers. In this case, the number of correction values that are unknown is (6 × N + 7 × K _s1 ).

When the number of square indices detected from the image i (i = 1,..., N) is d _si , the number D _{s of} square indices detected from the N images is expressed by the following equation (12). become that way.

When the number of square indexes detected from N images is D _s , (4 × D _s ) observation equations (11), that is, (2 × 4 × D _s ) observation equations are established. When the constant terms F ⁰ and G ⁰ on the left side of the observation equation of Equation (11) are transferred to the right side and a simultaneous equation is established, this simultaneous equation can be written in matrix form as shown in Equation (13) below. .

Is called a Jacobian matrix, and is an array of partial differential coefficients for the positions and orientations of the F and G imaging units 210, the position and orientation of the square index, and the scale. Jacobian procession

Is the number of observation equations (2 × 4 × D _s ), and the number of columns is the number of unknowns (6 × N + 7 × K _s1 ).

Is a correction value vector. The number of elements of the correction value vector is the number of unknowns (6 × N + 7 × K _s1 ).

Is an error vector, the calculated position of the projection position by the approximate value and the observation position

Difference -F0, -G0 as elements.

Is the number of observation equations (2 × 4 × D _s ).

  Note that the origin, scale, and orientation of the reference coordinate system can be explicitly specified by simultaneously capturing a point index with a known position in the reference coordinate system or a square index with a known position, orientation, and scale. . In the equation (11) for these indices, the values of partial differential coefficients for the position and orientation of the index and the scale are all zero. In order to explicitly specify the origin, scale, and orientation of the reference coordinate system, it is sufficient to use three point indicators that are not in the same straight line with known positions if they are point indicators, and if they are square indicators, the position and orientation And one square index with a known scale may be used.

  Next, in step S120, correction values for the approximate values of the position and orientation of the imaging unit 210, the position and orientation of the square index, and the scale are obtained. The correction value is obtained based on the above equation (13). Jacobian procession

Is a square matrix, both sides of equation (13)

The correction value vector by multiplying the inverse matrix of

Ask for.

Is not a square matrix, the correction value vector is obtained by the least square method as shown in the following equation (14).

Ask for.

  Where Jacobian matrix

How to find each element of will be described.

  First

far. From the above equations (6) and (7), F and G can be written as the following equation (15). F and G are functions of x _c , y _c and z _c .

Jacobian matrix with partial differential coefficients of F and G by x _c , y _c and z _c

Can be written as the following equation (16).

Here, if x _c , y _c and z _c are functions of variables s1, s2,..., Sm, F and G are also functions of s1, s2,. Jacobian matrix with F, G s1, s2,.

Can be decomposed as shown in the following equation (17).

With camera position, camera posture, index position, index posture, index scale respectively

Can be used to obtain partial differential coefficients for F and G camera positions, camera orientations, index positions, index orientations, and index scales.

Jacobian matrix with partial derivatives of F, G camera positions t _x , t _y , t _z as elements

Can be written from the equation (5) as the following equation (18).

  However, posture

And 3 × 3 rotation transformation matrix

The relationship is expressed by equation (4).

  Jacobian matrix with partial differential coefficients for F and G camera orientations ωx, ωy, and ωz in each element

Can be decomposed and written as in the following equation (19).

  here,

Is a Jacobian matrix as shown in the following equation (20).

  Also,

Is a Jacobian matrix as shown in the following equation (21).

  Also,

As

Is a Jacobian matrix as shown in the following equation (22).

  further,

Is a Jacobian matrix as shown in the following equation (23).

  Jacobian matrix with partial differential coefficients in each element based on positions tsx, tsy, and tsz of square indices of F and G

Can be written as the following equation (24) from the equations (5) and (8).

  F, G square indicator posture

Jacobian matrix with partial differential coefficients by

Can be decomposed as shown in the following equation (25).

Can be obtained in the same manner as equations (21) to (23).

  Jacobian matrix with partial differential coefficients of each element based on scale L of square indices of F and G

Can be decomposed as shown in the following equation (26).

  In step S130, the approximate value is updated using the correction value thus obtained.

  Next, in step S140, the above-described convergence determination is performed. The convergence determination is performed by determining a condition such that the absolute value of the correction value falls below a specified threshold value, or the difference between the calculation position of the projection position and the observation position falls below a specified threshold value. In this example, when “below”, it is determined that “convergence” has occurred.

  As a result of the convergence determination process, when it is determined that the image has converged, the position and orientation of the index and the scale estimation are terminated, and an index size obtained by doubling the index scale value is output. If not converged, the process returns to step S110, and the simultaneous equations are established again using the corrected position and orientation of the imaging unit 210, the approximate position and orientation of the square index, and the scale, and the correction is performed. Repeat the process of finding the value.

<4. Index placement condition setting>
The index placement condition setting unit 240 sets a constraint condition related to the placement of the index. In the present embodiment, as a constraint condition related to the marker placement, a constraint condition is set such that the index size when the index position and orientation are finally obtained is the same as the index size actually created by the user.

  The size of the square index obtained in Steps S100 to S140 (twice the scale) is a size obtained by calculation to the last, and does not match the size of the index actually created by the user, resulting in an error. There is a case. Therefore, each of the sizes of at least one square index obtained in steps S100 to S140 is corrected to the size of the index actually created by the user.

  The following two cases can be considered as the size of the index actually created by the user.

  (Case 1) Before executing the method of the present embodiment, the index is printed using the printer 312. It is assumed that the size of the index to be printed is an instruction given by the user among several predetermined sizes.

  For example, several index sizes are determined in advance, and a table in which the determined several sizes are registered is created. The created table data is registered in the hard disk 305. FIG. 5A is a diagram illustrating a configuration example of a table in which a plurality of types of index sizes are registered.

  As shown in the figure, an index size is registered in the table every 10 mm. In other words, the size of the index used here is one of predetermined discrete values (10 mm increments). Then, the user selects any one size registered in the table. As for the selection method, for example, a display device (not shown) is connected to the computer 300, all sizes registered in this table are displayed on the display screen of this display device, and one of them is displayed as a mouse 308 or Instructions are given using the keyboard 309. That is, the selection method is not particularly limited. Then, the index of the selected size is printed by the printer 312.

  (Case 2) Similar to Case 1, before executing the method of the present embodiment, the index is printed using the printer 312. However, the difference from Case 1 is that the size of each index is not determined in advance, and the size printed by the user specifying an arbitrary size is stored in the hard disk 305 or the like as a new table value.

  FIG. 5B is a diagram illustrating a configuration example of a table in which an arbitrary size designated by the user is registered. As shown in the figure, index sizes of (arbitrary) sizes such as 23 mm, 12 mm, 45 mm, and 73 mm actually printed by the user are registered in the table.

  The index arrangement condition setting unit 240 is shown in FIG. 5A or FIG. 5B based on each of the sizes (twice the scale) of one or more square indices obtained in steps S100 to S140. Correct to the closest size registered in the table.

  For example, when the size of one side of the square index obtained in steps S100 to S140 is 32 mm, in the example of FIG. 5A, it is corrected to the closest size of 30 mm. Moreover, in the example of FIG.5 (b), it corrects to 23 mm which is the nearest size.

<5. Index Position / Orientation Calculation The index position / orientation calculation unit 250 obtains the position and orientation of each of one or more indices existing in the scene. At this time, in the present embodiment, the scale of the square index after the size is corrected by the index arrangement condition setting unit 240 is used. That is, in the above description, when the table in FIG. 5A is used, the square index size obtained in steps S100 to S140 is not 32 mm, but the closest size in the table in FIG. Use as a “modified indicator scale”. When the table of FIG. 5B is used, 23 mm which is the closest size in the table of FIG. 5B is used as the “corrected index scale”.

  The index position / orientation calculation unit 250 obtains the position and orientation of the index present in the scene. Basically, the index position / orientation size calculation unit 235 obtains the position, orientation, and size of the index (formula (1) ) To (26)). The only difference is that the position and orientation of the imaging unit 210 and the position and orientation of the index are obtained by using an index scale converted based on the size of the index corrected by the index arrangement condition setting unit 240 as a constant.

  As described above, according to the present embodiment, a large number of scene images in which indices are arranged are photographed, and indices are detected and identified from the captured images. First, the position, orientation, and size of the indices are obtained. Then, the obtained index size is corrected to the index size actually designated when the user prints the index with the printer, and the position and orientation of the index are obtained again using the corrected size.

  By doing so, the size of the index can be obtained using the image. In addition, by making the size of the index actually created by the user the size of the index when the position and orientation of the index are finally obtained, the position and orientation of the index can be obtained with higher accuracy. .

[Second Embodiment]
In the first embodiment, a large number of images of a scene where an index is arranged are captured, the index is detected and identified from the captured image, and the position, orientation, and size of the index are first obtained. Then, the size of the obtained index is corrected to the size of the index actually designated when the user prints the index with the printer, and the position and orientation of the index are obtained again using the corrected size.

  In this embodiment, a large number of images of a scene in which an index is arranged are captured, the index is detected and identified from the captured image, and first the position, orientation, and size of the index are obtained, and then the same as in the first embodiment. The subsequent processing is different. Next, in the present embodiment, the position of the obtained index is corrected to the position where the user has actually placed the index, and the attitude and size of the index are obtained again using the corrected position. Here, “the position where the user has actually placed the index” is assumed to be, for example, a real desk with a grid as an index placement target, and “when the user places the index according to the desk grid. Position.

  FIG. 9 is a block diagram showing a functional configuration of the system according to the present embodiment. In the figure, the same parts as those in FIG.

  As shown in the figure, the system according to this embodiment includes an imaging unit 210 and an index posture size estimation apparatus 900. The index posture size estimation apparatus 900 includes an image capturing unit 220, an index detection / identification unit 230, an index position / posture size calculation unit 235, an index arrangement condition setting unit 241, and an index posture size calculation unit 251.

  Note that the computer 300 shown in FIG. 3 can be similarly applied to the index posture size estimation apparatus 900. In that case, a program and data for causing the CPU 301 to execute the functions of the index arrangement condition setting unit 241 and the index posture size calculation unit 251 are stored in the hard disk 305. Then, when this program and data are loaded into the RAM 307 and the CPU 301 executes processing using the loaded program and data, the CPU 301 executes each processing described later as what the index posture size estimation apparatus 900 performs. become.

  Hereinafter, the index arrangement condition setting unit 241 and the index posture size calculation unit 251 in FIG. 9 will be described.

<4. Index placement condition setting>
The index arrangement condition setting unit 241 sets a constraint condition regarding the arrangement of the index. In the present embodiment, as a constraint condition regarding the placement of the index, a constraint condition that the position of the index when the posture and size of the index are finally obtained is the same as the position where the user actually placed the index is given.

  The position of the square index obtained by the index position / posture size calculation unit 235 (in other words, the position of the square index obtained in steps S100 to S140) is a calculated position to the last, and the user actually places the index. There is a case where an error occurs without matching with the position. Therefore, each of the positions of the one or more square indices obtained in steps S100 to S140 is corrected to the position where the user has actually placed the indices.

  The index arrangement condition setting unit 241 corrects each of the positions of the one or more square indices obtained in steps S100 to S140 to the closest position on the actual desk grid shown in FIG. For example, it is assumed that a discrete grid is written in 50 mm increments on the actual desk shown in FIG.

  FIG. 10 is a diagram illustrating a grid provided on a desk as a real object. If the position of the square index obtained in steps S100 to S140 is (X, Y) = (141 mm, 203 mm) with the origin of desk coordinates as a reference, it is the closest grid position (X, Y) = ( 150mm, 200mm). The desk coordinates are such that the normal direction of the desk is the Z axis, and the X and Y axes are parallel to the two sides of the desk. Information about each grid position is created in advance and registered in the hard disk 305.

<5. Index posture size calculation>
The index posture size calculation unit 251 obtains the posture and size of each of one or more indexes existing in the scene. At this time, in the present embodiment, the position of the square marker after being corrected by the marker arrangement condition setting unit 241 is used. In other words, in the above description, the position (X, Y) = (141 mm, 203 mm) of the square index obtained in steps S100 to S140 is not the position (X, Y) = (X, Y) = ( 150 mm, 200 mm) is used as the corrected position.

  The index posture size calculation unit 251 calculates the posture and size of an index existing in the scene. Basically, the index position / posture size calculation unit 235 calculates the index position, posture, and size (formula (1) ) To (26)). The only difference is that the position and orientation of the imaging unit 210 and the orientation and size of the index are obtained with the index position corrected by the index arrangement condition setting unit 241 as a constant.

  As described above, according to the present embodiment, a large number of scene images in which indices are arranged are photographed, indices are detected and identified from the captured images, and the position, orientation, and size of the indices are first obtained. Then, the obtained index position is corrected to the position where the user actually arranged the index, and the index posture and size are obtained again using the corrected position.

  By doing so, the size of the index can be obtained using the image. In addition, by setting the position where the index is actually placed by the user as the position of the index when the attitude and size of the index are finally obtained, the attitude and size of the index can be obtained with higher accuracy. .

[Third Embodiment]
In the first embodiment, a large number of images of a scene where an index is arranged are captured, the index is detected and identified from the captured image, and the position, orientation, and size of the index are first obtained. Then, the size of the obtained index is corrected to the size of the index actually designated when the user prints the index with the printer, and the position and orientation of the index are obtained again using the corrected size.

  In this embodiment, a large number of images of a scene in which an index is arranged are captured, the index is detected and identified from the captured image, and first the position, orientation, and size of the index are obtained, and then the same as in the first embodiment. The subsequent processing is different. Next, in the present embodiment, the posture of the obtained index is corrected to the posture in which the user has actually placed the index, and the position and size of the index are obtained again using the corrected posture. Here, “the posture in which the user has actually placed the index” means, for example, an actual floor, wall, or ceiling as the placement target of the index, and “attitude when the user places the indicator on the floor, wall, or ceiling” Is.

  FIG. 11 is a block diagram showing a functional configuration of the system according to the present embodiment. In the figure, the same parts as those in FIG.

  As shown in the figure, the system according to this embodiment includes an imaging unit 210 and an index position size estimation apparatus 1100. The index position size estimation apparatus 1100 includes an image capturing unit 220, an index detection and identification unit 230, an index position and orientation size calculation unit 235, an index arrangement condition setting unit 242, and an index position size calculation unit 252.

  Note that the computer 300 shown in FIG. 3 can be similarly applied to the index posture size estimation apparatus 1100. In that case, a program and data for causing the CPU 301 to execute the functions of the index arrangement condition setting unit 242 and the index position size calculation unit 252 are stored in the hard disk 305. Then, when this program and data are loaded into the RAM 307 and the CPU 301 executes processing using the loaded program and data, the CPU 301 executes each processing described later as what the index position size estimation device 1100 performs. become.

  Hereinafter, the index arrangement condition setting unit 242 and the index position size calculation unit 252 in FIG. 11 will be described.

<4. Index placement condition setting>
The index arrangement condition setting unit 242 sets a constraint condition regarding the arrangement of the index. In the present embodiment, as a constraint condition related to the placement of the index, a constraint condition is set such that the orientation of the index when the position and size of the index are finally obtained is the same as the orientation in which the user actually placed the index.

  The posture of the square index obtained by the index position / posture size calculation unit 235 (in other words, the posture of the square index obtained in steps S100 to S140) is a posture obtained by calculation to the last, and the user actually places the indicator. There may be an error that does not match the posture. Therefore, each of the postures of the one or more square markers obtained in steps S100 to S140 is corrected to the posture in which the user has actually placed the markers.

  The index arrangement condition setting unit 242 corrects each of the attitudes of the one or more square indices obtained in steps S100 to S140 to the closest attitude of the actual floor or wall shown in FIG. For example, the floor and wall shown in FIG. 12 have a discrete posture such as horizontal or vertical with respect to each axis of the reference coordinate system. If the posture of the square index obtained in steps S100 to S140 is deviated from the horizontal or vertical direction with respect to the reference coordinate system, the closest horizontal posture or vertical posture with respect to the reference coordinate system is obtained. Modify it to be the same as the posture of the nearby floor or wall. If there is a rotation error about the z-axis of the index, the x-axis and y-axis of the index shown in FIG. 4 may be corrected so that they are closest to the reference coordinate system in the horizontal or vertical direction. . FIG. 12 is a diagram illustrating walls and floors on which indicators are arranged. In addition, as a reference for correcting the posture of the index, a ceiling may be used in addition to the floor and the wall.

<5. Calculation of index position size>
The index position size calculation unit 252 determines the position and size of each of one or more indices existing in the scene. At this time, in the present embodiment, the posture of the square index after being corrected by the index arrangement condition setting unit 242 is used. That is, in the above description, the posture closest to the floor or wall in FIG. 12 is used as the corrected posture, not the posture of the square index obtained in steps S100 to S140.

  The index position size calculation unit 252 calculates the position and size of the index present in the scene. Basically, the index position / posture size calculation unit 235 calculates the position, posture, and size of the index (formula (1 ) To (26)). The only difference is that the position and orientation of the imaging unit 210 and the position and size of the index are obtained with the index orientation corrected by the index arrangement condition setting unit 242 as a constant.

  As described above, according to the present embodiment, a large number of scene images in which indices are arranged are photographed, indices are detected and identified from the captured images, and the position, orientation, and size of the indices are first obtained. Then, the posture of the obtained index is corrected to the posture in which the user has actually placed the index, and the position and size of the index are obtained again using the corrected posture.

  By doing so, the size of the index can be obtained using the image. Further, the position of the index and the size can be obtained with higher accuracy by making the attitude of the index where the user has actually placed the index into the attitude of the index when finally obtaining the position and size of the index. .

<Modification 1>
In the first to third embodiments, one parameter of the index position, posture, and size is fixed, and the remaining two parameters are obtained again. However, it goes without saying that two parameters of the index position, orientation, and size may be fixed and the remaining one parameter may be obtained again.

<Modification 2>
In the first to third embodiments, when one parameter among the position, posture, and size of the index is fixed, the parameter is corrected to the parameter that is closest to the parameter physically set by the user.

  However, in a special case such as when the light source environment of the scene where the index is photographed is poor, the closest parameter may not always be the correct value. In order to deal with such a case, a viewpoint for observing the virtual space is set to the position and orientation of the imaging unit 210 finally obtained, and an image of the virtual space that can be seen from the set viewpoint is generated and synthesized with the real space image. To do. If the user determines that the images are not properly combined, a message to that effect is input using the mouse 308 or the keyboard 309. When this input is made, the parameter value to be fixed can be selected from the three closest parameters (the number represented by an integer of 2 or more) and selected until CG is properly synthesized. You may recalculate based on the parameter.

<Modification 4>
In the second embodiment, the index position is corrected to the closest position in the grid. However, instead, a constraint condition that the distance between a plurality of indices is constant may be introduced based on the estimated index position, and the index placement information may be obtained with higher accuracy. That is, by using the constraint condition that the distance between the plurality of indices is constant, the plurality of index positions may be set as constants, and the attitude and size of the indices may be obtained again.

<Modification 5>
In the first to third embodiments, as a method for correcting the index arrangement information, a method is used in which the observation equation is Taylor-developed and corrected repeatedly using the first order term. This is a method of linearly approximating the observation equation locally in one iteration and estimating the correction amount of the unknown parameter when it is assumed that the error is zero. It is an equivalent method. The Newton method is a typical method for solving a nonlinear equation by numerical calculation, but the iterative calculation method used in the above embodiment is not limited to this method.

  For example, like the Newton method, the Levenberg-Marquardt method, which is known as a well-known technique, can be used to dynamically change the correction amount based on the variation in the correction amount of the unknown parameter estimated in one process. Good. Further, it may be a method in which the result of Taylor expansion is taken into consideration up to higher order terms and repeated calculation is performed.

  In other words, the essence of each of the embodiments described above is to find an optimal solution by effectively using each type of index even when a plurality of types of indexes having different arrangement information amounts exist. It is in. Therefore, the essence is not lost by a specific method of iterative calculation as a numerical solution.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structure of the conventional position and orientation measuring apparatus. It is a block diagram which shows the function structure of the system which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the computer 300 applicable to the parameter | index position and orientation estimation apparatus 2. FIG. It is a figure explaining the parameter | index used in the 1st Embodiment of this invention. (A) is a figure which shows the structural example of the table in which several types of parameter | index size is registered, (b) is a figure which shows the structural example of the table in which the arbitrary sizes which the user designated are registered. It is a figure explaining a camera (imaging part 210) coordinate system and a screen coordinate system. It is a figure explaining a collinear conditional expression. 14 is a flowchart of processing for obtaining the position, orientation, and size (twice the index scale) of a square index by the index position / posture size calculation unit 235. It is a block diagram which shows the function structure of the system which concerns on the 2nd Embodiment of this invention. It is a figure explaining the grid provided on the desk as a real object. It is a block diagram which shows the function structure of the system which concerns on the 3rd Embodiment of this invention. It is a figure which shows the wall and floor which arrange | position an parameter | index.

Claims (12)

An information processing method for estimating the placement information of an index,
An image acquisition step in which the image acquisition means acquires an image in which a scene including the index is captured;
An index detecting step in which the index detecting means detects the index from the image;
A first index arrangement information estimation unit, wherein the first index arrangement information estimation unit estimates the arrangement information in the scene of the index detected in the index detection process;
An index information correcting step in which the index information correcting means corrects at least one of the arrangement information estimated in the first index arrangement information estimating step to a known value;
A second index arrangement information estimating unit, wherein the second index arrangement information estimating means estimates the arrangement information whose value is not corrected in the index information correcting process, using the arrangement information corrected in the index information correcting process as a constant; An information processing method comprising: Arrangement information of the index, the information processing method according to claim 1, wherein the position in a scene of the index, or position, or the size.   The information processing method according to claim 1, wherein there are a plurality of the indexes in a scene.   The information processing method according to claim 1, wherein the known value is a value selected from predetermined discrete values. The information processing method according to any one of claims 1 to 3, wherein, in the index information correction step, the size of the index is corrected to a predetermined size for printing the index. 4. The information processing method according to claim 1, wherein in the index information correcting step, the size of the index is corrected to a size stored when the index is printed. 5. The information processing method according to any one of claims 1 to 3, wherein, in the index information correcting step, the position of the index is corrected to a position on a grid arranged in a scene. The information according to any one of claims 1 to 3, wherein in the index information correction step, the attitude of the index is corrected to a vertical or horizontal attitude with respect to a floor, wall, or ceiling in the scene. Processing method. An information processing apparatus for estimating index placement information,
Image acquisition means for acquiring an image in which a scene including an index is captured;
Index detecting means for detecting an index from the image;
First index arrangement information estimation means for estimating arrangement information in the scene of the index detected by the index detection means;
Index information correcting means for correcting at least one of the arrangement information estimated by the first index arrangement information estimating means to a known value;
An information processing apparatus comprising: second index arrangement information estimation means for estimating arrangement information whose value is not corrected by the index information correction means using the arrangement information corrected by the index information correction means as a constant . Arrangement information of the index, the information processing apparatus according to claim 9, wherein the position in a scene of the index, or position, or the size. The computer, computer program to function as each unit included in the information processing apparatus according to claim 9 or 10. A computer-readable storage medium storing the computer program according to claim 11.

Images