JP6581348B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to a technique for setting position and orientation information of a virtual object in a three-dimensional space in a mixed reality presentation system.

  Consider a method of arranging a virtual three-dimensional model such as CAD data (hereinafter referred to as a three-dimensional virtual object) in a three-dimensional space. In the conventional method, the observer first projects a three-dimensional virtual object as a two-dimensional image on a two-dimensional screen such as a display, and operates the position and orientation of the object using a keyboard or the like in a pseudo three-dimensional space. The arrangement of the three-dimensional virtual object was set. However, in this method, information on the three-dimensional virtual object is lost during projection onto the two-dimensional image, and it may be difficult to intuitively operate the three-dimensional virtual object and set the arrangement. In order to solve this problem, a method of manipulating and arranging a three-dimensional virtual object in a three-dimensional space using mixed reality (MR) technology (for example, Patent Document 1) has been attempted. .

  As an example of a method for operating a three-dimensional virtual object in a three-dimensional real space, there is a method using a six-degree-of-freedom position / orientation measurement apparatus (for example, a stylus of a FASTRAK sensor manufactured by Polhemus). In this method, the observer takes out the position and orientation change output from the 6-DOF position and orientation measurement apparatus, and sets the arrangement of the three-dimensional virtual object using the change as the moving rotation amount of the three-dimensional virtual object (patent) Reference 1). FIG. 4 is a schematic diagram showing how an observer performs a conventional operation method. In FIG. 4, when an observer holds a 6-DOF position / orientation measurement device in his / her hand and rotates and moves it as shown by the arrow in the figure, a state in which this rotation and movement is reflected in the three-dimensional virtual object is shown. ing.

  As a means for presenting to the observer, an HMD (Head Mounted Display) or the like is used. With this HMD, the observer can easily grasp the positional relationship between the real space and the three-dimensional virtual object and set the arrangement of the three-dimensional virtual object.

JP 2004-62758 A

Kenichi Hayashi, Hirokazu Kato, Masami Nishida, “Judgment of Real and Three-Dimensional Virtual Objects Using Boundary-Based Stereo Matching”, Transactions of the Virtual Reality Society of Japan, Vol. 10, no. 3, pp. 371-380, 2005

  Consider a case where the observer wants to set the position and orientation of a three-dimensional virtual object so that the arrangement of the three-dimensional virtual object imitating the shape of the real object and the real object is the same. At this time, since the intersection of the 6-DOF position and orientation measurement device and the real object is not a plane, it is difficult to set the position and orientation of the 3D virtual object accurately in a short time because the orientation is not uniquely determined. .

  The present invention has been made in view of such problems, and an object thereof is to accurately set the position and orientation of a three-dimensional virtual object in a short time.

As a means for achieving the above object, an information processing apparatus according to the present invention has the following configuration. That is, an acquisition unit that acquires a reference plane that is a plane corresponding to the shape of the real object, and a selection unit that selects one of the planes of the three-dimensional virtual object as a corresponding plane based on the distance from the reference plane Setting means for setting the position and orientation of the three-dimensional virtual object so that the reference plane and the corresponding plane have a predetermined positional relationship, and the position of the three-dimensional virtual object according to the movement of the reference plane And a determining means for determining the position of the three-dimensional virtual object based on the result of movement by the moving means.

  According to the present invention, it is possible to accurately set the position and orientation of a three-dimensional virtual object in a short time.

The block diagram which shows the function structure of information processing apparatus. FIG. 2 is a schematic diagram showing a functional configuration of the information processing apparatus according to the first embodiment. 3 is a flowchart showing a procedure of processing according to the first embodiment. The schematic diagram which shows a mode that the conventional operating method is performed. The schematic diagram which shows a mode that the reference plane and the reference plane coordinate system are defined with respect to the hand. The flowchart which shows the procedure of the detailed process of estimation of a reference plane position and orientation. The schematic diagram which shows the sampling point of the outline of a hand. The flowchart which showed the procedure of the detailed process which determines whether a reference plane and the three-dimensional virtual object cross | intersect. The figure which showed the example of the bounding box 901 of the three-dimensional virtual object 205. FIG. The flowchart which shows the procedure of the detailed process which cross | intersects the bounding box surface which cross | intersected in parallel with the reference plane. The schematic diagram which shows a mode that it cross | intersects the bounding box 901 and the reference plane 501 in parallel. The flowchart which shows the procedure of the detailed process of estimation of the relative position and orientation of a reference surface and a virtual object surface. The figure showing a mode that the three-dimensional virtual object 2050 is drawn according to the change of the reference plane coordinate system 502. FIG. 3 is a block diagram showing a functional configuration of an information processing apparatus according to Embodiment 2. FIG. FIG. 3 is a schematic diagram illustrating a functional configuration of an information processing apparatus according to a second embodiment. 9 is a flowchart showing a processing procedure according to the second embodiment. The block diagram which shows the hardware constitutions of the computer applicable to information processing apparatus.

  Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. An information processing apparatus according to the following embodiment sets position and orientation information of a virtual object in a three-dimensional space in a mixed reality presentation system that presents an observer with a virtual image by computer graphics or the like fused to a real space It is an information processing apparatus that can do this.

[Embodiment 1]
<Configuration>
FIG. 1 is a block diagram illustrating a functional configuration of the information processing apparatus according to the present embodiment. FIG. 2 is a schematic diagram of the information processing apparatus in the present embodiment, and FIG. 17 is a block diagram showing a hardware configuration of a computer applicable to the information processing apparatus. In the present embodiment, as shown in FIG. 2, the imaging unit 201, the display unit 202, and the imaging unit position and orientation acquisition unit 211 are attached to an HMD (Head Mounted Display) 200, and each is fixed to the main body of the HMD 200. . Two imaging units 201 and two display units 202 are provided in the main body of the HMD 200, the imaging unit 201R and the display unit 202R are for the right eye, and the imaging unit 201L and the display unit 202L are for the left eye. Hereinafter, R and L will be omitted unless specifically limited to the right eye and the left eye (ie, the imaging unit 201 and the display unit are 202 hereinafter). An observer wears the HMD 200 on the head to view an image (hereinafter, referred to as an MR image) in which a three-dimensional virtual object 205 input from the storage unit 104 is superimposed on a real video image displayed on the display unit 202. I can do it.

  The captured image acquisition unit 101 acquires the captured image captured by the imaging unit 201 as image data. The imaging unit position and orientation acquisition unit 211 estimates the viewpoint position and orientation of the observer from the result of measuring the position and orientation of the imaging unit 201. In the present embodiment, a magnetic sensor is used as the imaging unit position / orientation acquisition unit 211. The imaging unit position / orientation acquisition unit 211 receives a magnetic field generated from the magnetic field generator 210 arranged in the real space, and sends a change in the received magnetic field to the sensor controller 209.

The sensor controller 209 obtains the position and orientation of the imaging unit position and orientation acquisition unit 211 in the reference coordinate system 206 with the magnetic field generator 210 as the origin. Here, the “determining position and orientation” processing is as follows. That is, first, the sensor controller 209 generates a position / orientation conversion matrix ΔM _R using the change in position and orientation measured by the imaging unit position and orientation acquisition unit 211. Thereafter, the sensor controller 209 adds ΔM _R to the matrix M _R indicating the position and orientation (three-dimensional arrangement information) of the imaging unit position and orientation acquisition unit 211 of the previous frame, so that the imaging unit position and orientation acquisition unit in the next frame A matrix M _R ′ indicating the position and orientation of 211 is obtained as in the following (formula 1).
(Formula 1)

（式3）

（式4）

（式2）は行列W_t（基準座標系206に対する姿勢を表す3×3回転行列を含む行列）、（式3）は行列W_r（基準座標系206に対する位置を表す3次元ベクトルを含む行列）を示す式である。行列Wtと行列Wrにおける各成分は周知の通り、撮像部位置姿勢取得部211による位置姿勢の計測値に基づいて決まるものである。 Note that the matrix M _R ′ indicating the position and orientation of the imaging unit position and orientation acquisition unit 211, the matrix W _t indicating the position change component, and the matrix W _r indicating the orientation change component are each a 4 × 4 matrix. Sensor controller 209 as the initial value of the matrix M _R, stores in advance the initial position and orientation of the imaging unit position and orientation acquisition unit 211. The position / orientation conversion matrix ΔM _R is a matrix calculated using the matrices W _t and W _r . The matrix W _t and the matrix W _r are obtained as outputs from the imaging unit position and orientation acquisition unit 211.
(Formula 2)

(Formula 3)

(Formula 4)

(Expression 2) is a matrix W _t (a matrix including a 3 × 3 rotation matrix representing an attitude relative to the reference coordinate system 206), and (Expression 3) is a matrix W _r (a matrix including a three-dimensional vector indicating a position relative to the reference coordinate system 206). ). As is well known, each component in the matrix Wt and the matrix Wr is determined based on the position and orientation measurement values obtained by the imaging unit position and orientation acquisition unit 211.

The sensor controller 209 outputs the position and orientation M _R ′ of the imaging unit position and orientation acquisition unit 211 with respect to the reference coordinate system 206 to the workstation 203. Workstation 203, as shown in (Equation 5), performs coordinate conversion using the position and orientation matrix M _L between the image pickup unit position and orientation acquisition unit 211 and the imaging unit 201 is known, the position and orientation relative to the imaging unit coordinate system 208 Find a matrix M _C representing
(Formula 5)

  The sensor applicable to the imaging unit position / orientation acquisition unit 211 is not limited to a magnetic sensor, but may be other types of sensors. For example, an optical sensor, an ultrasonic sensor, or an inertial sensor may be used. Further, any device that can measure the position and orientation of the imaging unit, such as measurement of the imaging unit position and orientation by image processing, can be applied to the imaging unit position and orientation acquisition unit 211.

  The reference plane position / orientation estimation unit 103 obtains the shape of the space including the hand 212 from one or more stereo images acquired by the captured image acquisition unit 101. A technique for detecting a hand region and estimating a three-dimensional shape of a contour line is well known. For example, a technique disclosed in Non-Patent Document 1 is used. According to this method, the reference plane position / orientation estimation unit 103 detects the object region based on the difference from the key frame, and performs stereo matching of the boundary line of the object region. The reference plane position / orientation estimation unit 103 obtains a distance from the imaging unit 201 (camera) to the boundary line by triangulation with respect to the matched boundary line. Thereafter, the reference plane position and orientation estimation unit 103 estimates the position and orientation of the imaging unit 201 in the reference coordinate system 206, and obtains the position and orientation of the boundary line in the reference coordinate system 206. The reference plane position / orientation estimation unit 103 further obtains a reference plane 501 that is an approximate plane based on the position / orientation of the boundary line. FIG. 5 shows an example of the reference plane 501 obtained by the reference plane position / orientation estimation unit 103. The three-dimensional virtual object selection unit 102 determines whether an input for selecting the target three-dimensional virtual object 205 has been performed from the input unit 109.

The storage unit 104 stores data (arrangement information, shape information, etc.) of the three-dimensional virtual object 205 obtained by imitating the shape of the mockup 204 measured by the workstation 203. Data storage to the three-dimensional virtual object 205, the vertex three-dimensional position information, (referred to as the position and orientation matrix M _CG from the reference coordinate system 206) 4 × 4 matrix for coordinate transformation and the like. The storage unit 104 also stores initial values of the position and orientation of the three-dimensional virtual object coordinate system 207 in the reference coordinate system 206. The storage unit 104 inputs the stored initial value to the intersection determination unit 105 as position information for placing the three-dimensional virtual object 205 in the reference coordinate system 206. The storage unit 104 also stores a determination result as to whether the target three-dimensional virtual object 205 is selected in the three-dimensional virtual object selection unit 102, and inputs the determination result to the three-dimensional virtual object arrangement fixing unit 110. The storage unit 104 is not limited to the external storage device 17070 shown in FIG. 17 as a hardware configuration, and can store data in any location.

  Based on the reference plane 501 obtained from the reference plane position / orientation estimation unit 103 and the data of the three-dimensional virtual object 205 stored in the storage unit 104, the intersection plane determination unit 105 performs the reference plane 501 and the three-dimensional virtual object 205. However, as an example of the predetermined positional relationship, it is determined whether or not they intersect. The relative position / orientation estimation unit 106 obtains the position / orientation of the reference plane coordinate system 502 of the reference plane 501 and the three-dimensional virtual object coordinate system 207 of the three-dimensional virtual object 205, and calculates the relative position of the two coordinate systems. Find the position and orientation.

The three-dimensional virtual object position / orientation estimation unit 107 estimates the position / orientation of the three-dimensional virtual object 205 in the reference coordinate system 206. Using the position / orientation matrix M _SP and the relative position / orientation transformation matrix ΔM _re of the reference plane coordinate system 502, the position / orientation M _CG ′ of the three-dimensional virtual object 205 in the reference coordinate system 206 is obtained by the following (formula 6). .
(Formula 6)

  The image composition unit 108 superimposes the three-dimensional virtual object 205 on the captured image acquired by the captured image acquisition unit 101. At this time, the image synthesis unit 108 synthesizes the three-dimensional virtual object 205 at the viewpoint from the imaging unit 201 using the position and orientation information of the imaging unit 201 and the position and orientation information of the three-dimensional virtual object 205. When there is an input to the workstation 203 from an input device such as a keyboard 17040, a mouse 17050, or a game pad, the input unit 109 informs the three-dimensional virtual object selection unit 102 that there is an input.

In response to an input from the input unit 109, the three-dimensional virtual object arrangement fixing unit 110 receives a captured image from the captured image acquisition unit 101. The three-dimensional virtual object arrangement fixing unit 110 projects the captured image and the three-dimensional virtual object 205 on a two-dimensional plane, and changes the position and orientation of the three-dimensional virtual object 205 by fitting with the mockup 204 captured in the captured image. To do. The fitting is performed by a known method. For example, the feature extracted from the virtual object may be matched with the mock-up feature included in the captured image. After the fitting is completed, the 3D virtual object arrangement fixing unit 110 calculates the position and orientation M _CG ′ of the 3D virtual object 205. The three-dimensional virtual object arrangement fixing unit 110 stores the calculated position and orientation M _CG ′ in the external storage device 17070, and calculates the calculated position and orientation M in the transformation matrix M _CG of the three-dimensional virtual object coordinate system 207 in the reference coordinate system 206. Substitute _CG '.

<Processing procedure>
A processing procedure in this embodiment will be described with reference to a flowchart of FIG. In step S301, the captured image acquisition unit 101 acquires an image captured by the imaging unit 201. In step S302, the imaging unit position / orientation acquisition unit 211 obtains a matrix M _R ′ representing the position and orientation of the imaging unit position / orientation acquisition unit 211 in the reference coordinate system 206. Thereafter, the workstation 203 calculates a matrix M _C representing the position and orientation of the imaging unit coordinate system 208 using (Equation 5).

In step S303, the three-dimensional virtual object arrangement fixing unit 110 converts the three-dimensional virtual object 205 into the reference coordinate system 206 using the position / orientation matrix M _CG stored in the storage unit 104 in response to an input from the input unit 109. To place. In step S304, the storage unit 104 checks whether or not an operation (input) for selecting the target 3D virtual object 205 for setting the position and orientation has been performed from the 3D virtual object selection unit 102. When an operation for selecting the target three-dimensional virtual object 205 is performed, the process proceeds to step S305. On the other hand, if an operation for selection has not been performed, the process proceeds to step S301.

  In step S305, the captured image acquisition unit 101 inputs the captured image acquired from the imaging unit 201 to the reference plane position and orientation estimation unit 103, and the reference plane position and orientation estimation unit 103 estimates the position and orientation of the reference plane 501. FIG. 6 is a flowchart showing a detailed processing procedure of the position / orientation estimation step S305 of the reference surface 501. The processing procedure shown in FIG. 6 is based on the method described in Non-Patent Document 1. FIG. 7 is a diagram simulating the sampling point 701 of the contour line extracted from the hand 212 region. The sampling point 701 has information on a three-dimensional position in the imaging unit coordinate system 208.

  In step S601 of FIG. 6, the reference plane position / orientation estimation unit 103 estimates the position / orientation of the contour line in the three-dimensional space of the hand 212, and estimates the three-dimensional position of the sampling point 701.

求める近似平面上に存在する点は（式7）が成り立つ。（式7）に基づき、入力されたn個の点群がすべて近似平面上にあると仮定すると、（式8）が成り立つ。
（式8）

ここで、この式を、
（式9）

とおく。この（式9）のXの一般化逆行列を求めることで、近似平面の数式モデルのパラメータであるp(a, b, c)が求められる。ここで、（式9）のXの一般化逆行列をX⁺とすると
（式10）

となり、
（式11）

となる。 In step S602, the reference plane position / orientation estimation unit 103 calculates an approximate plane including the sampling point 701 using a mathematical model of an infinite plane by least square approximation. If the plane expression is ax + by + cz = 1, and this is expressed by a vector operation, it is expressed as (Expression 7).
(Formula 7)

The point existing on the approximate plane to be obtained is established by (Equation 7). Based on (Equation 7), assuming that all the input n point groups are on the approximate plane, (Equation 8) holds.
(Formula 8)

Where this equation is
(Formula 9)

far. By obtaining the generalized inverse matrix of X in (Equation 9), p (a, b, c) that is a parameter of the mathematical model of the approximate plane is obtained. Here, if the generalized inverse matrix of X in (Equation 9) is X ⁺ (Equation 10)

And
(Formula 11)

It becomes.

  By this (Expression 11), the parameter p (a, b, c) of the mathematical model of the approximate plane is obtained from the input point group. Sampling points 701 are set in the X matrix of (Expression 11), and parameters p (a, b, c) of the mathematical model of the approximate plane are obtained from the calculation result of (Expression 11). Here, by substituting the obtained parameters p (a, b, c) into (Equation 7), the mathematical expression of the approximate plane can be obtained. The method for obtaining the approximate plane is not limited to the method using the least square approximation, and other methods can be applied.

In step S603, the reference plane position / orientation estimation unit 103 obtains a rectangular parallelepiped that circumscribes the three-dimensional shape of the outline of the hand 212, obtains a line of intersection with the approximate plane obtained in step S602, and determines a plane area of the hand 212. This plane area is defined as a reference plane 501. First, the reference plane position / orientation estimation unit 103, as a cuboid circumscribing the three-dimensional shape of the contour line, the maximum X coordinate value, Y coordinate value, and Z coordinate value that can be taken by the sampling point 701 constituting the three-dimensional shape, Calculate the 6-plane formula using the minimum value. The reference plane position / orientation estimation unit 103 can determine the line of intersection between the calculated six planes and the approximate plane obtained in step S602, and can record a line segment on the surface of the rectangular parallelepiped. Furthermore, the reference plane position / orientation estimation unit 103 sets the intersections of the obtained plurality of line segments as reference plane vertices 504A to 504D in the adjacent plane area, and configures the reference plane 501 from these reference plane vertices. At this time, a reference plane coordinate system 502 is defined with an arbitrary vertex of the reference plane 501 as the origin. Also, placing the M _CSP coordinate transformation matrix to the reference plane coordinate system 502 from the imaging unit coordinate system 208.

  In this embodiment, in order to set a plane area, the reference plane position / orientation estimation unit 103 obtains a circumscribed rectangular parallelepiped and obtains a plane area (reference plane 501) from an intersection line between the rectangular parallelepiped and the approximate plane. It was. However, the present invention is not limited to this method, and the reference plane position / orientation estimation unit 103 may calculate the plane area of the palm using the three-dimensional shape of the outline of the hand 212.

  Returning to FIG. 3, in step S306, the relative position / orientation estimation unit 106 estimates the relative position / orientation between the reference plane 501 and the three-dimensional virtual object 205 estimated in step S305. In step S307, the intersection surface determination unit 105 determines whether the reference surface 501 estimated in step S305 intersects the three-dimensional virtual object 205. FIG. 8 is a flowchart illustrating a detailed processing procedure of step S307 in which the intersection surface determination unit 105 determines whether the reference surface 501 and the three-dimensional virtual object intersect.

  In step S801 of FIG. 8, the intersection determination unit 105 creates a bounding box of the three-dimensional virtual object 205. As a method of creating the bounding box of the three-dimensional virtual object 205, for example, there is a method of creating a rectangular parallelepiped that includes all the three-dimensional points constituting the three-dimensional virtual object 205. FIG. 9 shows an example of the bounding box 901 of the three-dimensional virtual object 205.

  In step S802, the intersection plane determination unit 105 determines whether the bounding box 901 and the reference plane 501 intersect. Specifically, each point of the bounding box 901 is compared with the position of the rectangular corner point constituting the reference plane 501 in the reference coordinate system 206. First, the intersection surface determination unit 105 obtains the maximum value and the minimum value of the X axis, the Y axis, and the Z axis in the reference coordinate system 206 of the bounding box 901. The intersection plane determination unit 105 also determines the maximum value and the minimum value of the X axis, the Y axis, and the Z axis in the reference coordinate system 206 for the reference plane 501. The intersection plane determination unit 105 calculates whether the maximum value and the minimum value of the reference plane 501 are included in the range from the maximum value of the bounding box 901 to the minimum value for each axis. If it is included, the intersection determination unit 105 determines that it intersects. If it is determined that they intersect, the process proceeds to step S803. If it is determined that they do not intersect, the processing flow ends. However, the algorithm for determining the intersection between the plane of the bounding box 901 and the reference plane 501 is not limited to this method, and the intersection may be determined by another method.

  In step S803, the intersection surface determination unit 105 fits the surface of the bounding box 901 determined to intersect in S802 to the reference surface 501. FIG. 10 is a flowchart showing a detailed processing procedure of step S803 in which the intersecting surface determination unit 105 fits the surface of the bounding box 901 to the reference surface 501 (intersects in parallel).

  In step S1001 of FIG. 10, the intersection surface determination unit 105 calculates the distance between each surface of the bounding box 901 and the reference surface vertices 504A to 504D, and takes the average value of the distances from the reference surface vertices 504A to 504D to the surface. . In step S1002, the intersection surface determination unit 105 obtains the average value of the distances from the reference surface vertices 504A to 504D for each surface of the bounding box 901. The surface with the shortest distance is the corresponding surface 901A. In step S1003, the intersection surface determination unit 105 rotates and moves the three-dimensional virtual object 205 so as to intersect the corresponding surface 901A and the reference surface 501 in parallel. FIG. 11 shows a state in which the bounding box 901 and the reference plane 501 intersect in parallel as a result of the movement.

  Returning to FIG. 3, when the reference plane 501 and the three-dimensional virtual object 205 intersect as a result of the processing in step S307, the processing proceeds to step S308. On the other hand, if it does not intersect, the process returns to step S301. In step S308, the three-dimensional virtual object position / orientation estimation unit 107 calculates the relative position / orientation between the reference plane 501 and the three-dimensional virtual object 205 estimated in step S306, and the position of the reference plane 501 estimated in step S305. The position and orientation of the three-dimensional virtual object 205 are estimated from the orientation. FIG. 12 is a flowchart showing a detailed processing procedure of step S308 in which the three-dimensional virtual object position / orientation estimation unit 107 estimates the relative position / orientation between the reference plane 501 and the three-dimensional virtual object 205.

In step S1201 of FIG. 12, the three-dimensional virtual object position / orientation estimation unit 107 estimates the position / orientation of the reference plane coordinate system 502 in the reference coordinate system 206. Specifically, the three-dimensional virtual object position and orientation estimation unit 107, the following equation (12), to estimate the position and orientation matrix M _SP reference plane coordinate system 502 in the reference coordinate system 206. The matrix M _CSP is a coordinate transformation matrix from the imaging unit coordinate system 208 to the reference plane coordinate system 502.
(Formula 12)

In step S1202, the three-dimensional virtual object position / orientation estimation unit 107 calculates the coordinate transformation matrix M _SP of the reference plane coordinate system 502 obtained in step S1201 and the coordinate transformation matrix M _CG of the three-dimensional virtual object coordinate system 207 in the reference coordinate system 206. As shown in (Equation 13), the relative position and orientation matrix M _re is calculated as follows.
(Formula 13)

  FIG. 13 is a diagram in which the three-dimensional virtual object 205 is drawn using the estimated position and orientation. FIG. 13 shows a state in which the three-dimensional virtual object 205 is drawn in accordance with the change of the reference plane 501. From FIG. 13, it can be seen that the three-dimensional virtual object 205 is moving in a state where the position and orientation of the three-dimensional virtual object 205 and the reference plane 501 are maintained.

  Returning to FIG. 3, in step S309, when the determination of the arrangement of the three-dimensional virtual object 205 is input by the three-dimensional virtual object arrangement fixing unit 110 referring to the input information from the input unit 109, the process proceeds to step S310. Move. Here, the arrangement is performed so that the contour line of the three-dimensional virtual object 205 matches the contour line of the mockup 204, and the position of the three-dimensional virtual object 205 is determined. On the other hand, if the placement determination has not been input, the process proceeds to step S301. In step S310, the three-dimensional virtual object arrangement fixing unit 110 stores the calculated position and orientation of the three-dimensional virtual object coordinate system 207 in the storage unit 104, and further substitutes them into the position and orientation of the three-dimensional virtual object coordinate system 207.

  In step S311, when an instruction to end the process is input or when a condition for ending the process is satisfied, the process ends. On the other hand, if the end instruction of this process has not been input and the condition for ending this process is not satisfied, the process proceeds to step S301.

  As described above, the information processing apparatus according to the present embodiment sets the position and orientation of a three-dimensional virtual object so that the arrangement of the real object and the three-dimensional virtual object simulating the shape of the real object are the same. At this time, the information processing apparatus operates the three-dimensional virtual object using another real object that can measure the position and orientation in the three-dimensional space. Since both the real object and the other real object are real objects, the posture of the three-dimensional virtual object can be uniquely determined by crossing the real objects with each other on the plane. This makes it possible to set the position and orientation of the three-dimensional virtual object accurately and in a short time.

[Embodiment 2]
In the first embodiment, the position of the mockup 204 in the reference coordinate system 206 is not measured. For this reason, if the observer moves the mockup 204 after fixing the position and orientation of the three-dimensional virtual object 205 in step S310, the three-dimensional virtual object 205 is displaced. Therefore, as a second embodiment, an embodiment in which a mockup coordinate system 1501 is defined for mockup will be described. In the present embodiment, the three-dimensional virtual object 205 may rotate and move in accordance with the rotation and movement of the mockup 204 in the three-dimensional space.

  FIG. 14 is a block diagram showing a functional configuration in the present embodiment. The same parts as those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted. FIG. 15 is a schematic diagram of the configuration of the information processing apparatus in the present embodiment. The same parts as those in FIG. 2 are given the same numbers, and the description thereof is omitted.

  The mockup position and orientation estimation unit 1401 estimates the position and orientation of the mockup coordinate system 1501 in the reference coordinate system 206 from the position and orientation acquired by the mockup position and orientation acquisition unit 1502. The relative position and orientation fixing unit 1402 calculates the relative position and orientation of the three-dimensional virtual object coordinate system 207 and the mockup coordinate system 1501, and stores the relative position and orientation in the storage unit 104.

  FIG. 16 is a flowchart showing a processing procedure in the present embodiment. The same parts as those in FIG. 3 are denoted by the same reference numerals, and the description thereof is omitted. In step S1601, the 3D virtual object arrangement fixing unit 110 estimates the position and orientation of the 3D virtual object coordinate system 207. In step S1602, the mockup position / orientation acquisition unit 1502 acquires the position and orientation of the mockup coordinate system 1501 in the reference coordinate system 206. In this embodiment, to simplify the description, it is assumed that the position and orientation of the mockup coordinate system 1501 is the same as the position and orientation obtained by the mockup position and orientation acquisition unit 1502. The mockup position / orientation acquisition unit 1502 calculates the relative position / orientation between the acquired mockup coordinate system 1501 and the three-dimensional virtual object coordinate system 207.

  In step S1603, the relative position / orientation fixing unit 1402 substitutes the result obtained by integrating the relative position / orientation obtained in S1602 with respect to the mockup coordinate system 1501 into the position / orientation of the three-dimensional virtual object coordinate system 207. Thereby, the relative position and orientation of the mockup 204 and the three-dimensional virtual object 205 are fixed. In step S1604, the storage unit 104 refers to the input information from the input unit 109 and checks whether or not an operation (input) to be selected when the relative position and orientation is fixed is performed. If an operation of selecting to fix the relative position and orientation is performed, the process proceeds to step S1603. On the other hand, if the operation of selecting to fix the relative position and orientation is not performed, the process proceeds to step S301.

  As described above, the information processing apparatus according to the present embodiment integrates the relative position and orientation calculated in step S1604 with respect to the mockup coordinate system 1501 when the mockup 204 rotates and moves in the three-dimensional space. Thereby, the information processing apparatus can obtain the position and orientation of the three-dimensional virtual object coordinate system 207.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Captured image acquisition unit, 102 3D virtual object selection unit, 104 Storage unit, 105 Cross plane determination unit, 106 Relative position and orientation estimation unit, 107 3D virtual object position and orientation estimation unit, 108 Image composition unit, 109 Input unit, 110 3D virtual object arrangement fixing unit, 200 HMD, 201 imaging unit, 202 display unit, 203 workstation, 204 mockup, 205 3D virtual object, 206 reference coordinate system, 207 3D virtual object coordinate system, 208 imaging unit Coordinate system, 209 sensor controller, 210 magnetic field generator, 211 imaging unit position and orientation acquisition unit, 212 hands

Claims (14)

An acquisition means for acquiring a reference plane which is a plane corresponding to the shape of the real object;
Selection means for selecting one of the surfaces of the three-dimensional virtual object as a corresponding surface based on the distance from the reference surface ;
Setting means for setting the position and orientation of the three-dimensional virtual object so that the reference plane and the corresponding plane have a predetermined positional relationship;
Moving means for moving the position of the three-dimensional virtual object in accordance with the movement of the reference plane;
Determining means for determining the position of the three-dimensional virtual object based on the result of movement by the moving means;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the acquisition unit acquires a surface that approximates the shape of the real object as the reference surface.   The information processing apparatus according to claim 1, wherein the acquisition unit calculates a plane including a sampling point of a contour line of the real object as the reference plane.   The said selection means selects the surface which is the surface whose distance with the said reference surface is the shortest among the surfaces of the said three-dimensional virtual object as said corresponding surface, The any one of Claim 1 to 3 characterized by the above-mentioned. The information processing apparatus described in 1.   The selection unit creates a bounding box of the three-dimensional virtual object, and selects, as the corresponding surface, a surface having the shortest distance from each vertex of the reference surface among the surfaces of the bounding box. The information processing apparatus according to any one of claims 1 to 4. The setting means, information according to any one of claims 1 5, characterized in that said reference surface and said corresponding surface to set the position and orientation of the three-dimensional virtual object so as to intersect in parallel Processing equipment.   The information processing apparatus according to claim 1, wherein the setting unit sets the position and orientation by rotating and moving the three-dimensional virtual object.   The three-dimensional virtual object imitates another real object, and the moving means moves the position of the three-dimensional virtual object so as to coincide with the contour line of the other real object. Item 8. The information processing apparatus according to any one of Items 1 to 7.   The information processing apparatus according to claim 8, further comprising a fixing unit that fixes a relative position between the other real object and the three-dimensional virtual object.   The information processing apparatus according to claim 1, further comprising display means for displaying an image based on the three-dimensional virtual object based on a position and orientation of an observer's viewpoint.   The display unit includes an imaging unit that captures an image of a real space, and displays the image of the three-dimensional virtual object superimposed on the image of the real space captured by the imaging unit. Item 11. The information processing apparatus according to Item 10.   The information processing apparatus according to claim 11, wherein the acquisition unit generates and acquires the reference plane from a plurality of images captured by the imaging unit. An acquisition step of acquiring a reference surface that is a surface corresponding to the shape of the real object;
A selection step of selecting one of the surfaces of the three-dimensional virtual object as a corresponding surface based on the distance from the reference surface ;
A setting step for setting the position and orientation of the three-dimensional virtual object so that the reference plane and the corresponding plane have a predetermined positional relationship;
A moving step of moving the position of the three-dimensional virtual object according to the movement of the reference plane;
A determination step of determining a position of the three-dimensional virtual object based on the result of movement in the movement step;
An information processing method characterized by comprising:   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 12.

Images