JP4630647B2 - Image presentation method and information processing apparatus - Google Patents

Description

  The present invention relates to virtual object presentation control in a mixed reality system.

  The mixed reality system provides a user with a synthesized video obtained by synthesizing a real space video and a virtual space video generated according to the user's viewpoint position, line-of-sight direction, and the like. In the mixed reality system, it is possible to present to the observer a feeling as if a virtual object actually exists in the real space, and more realistically than the conventional virtual reality system (VR system), Observation with an actual sense is possible.

  As a method of operating a CG (computer graphics) object in the mixed reality space, a system that measures or estimates six degrees of freedom of the position and orientation in the real space (herein called a six degree of freedom sensor) is used. There is a method of associating the measured or estimated six degrees of freedom with the CG object coordinate system. As such a six-degree-of-freedom sensor that realizes measurement, for example, the Fastrak system of Polhemus can be used. The Fastrak system can measure 6 degrees of freedom in the position and orientation of a receiver with a size of several centimeters. By associating this measurement value with the object coordinate system of the CG object, the CG object moves and rotates as the receiver moves and rotates.

By attaching the receiver to the operation device, an operation such as moving a CG object by the operation device becomes possible. For example, the CG object can be moved by associating the CG object with an operation device such as a pointer.
JP 2004-62758 A

  However, conventionally, when associating the CG object with the operation device, the posture of the CG object is maintained at the time when the association is performed. For this reason, for example, when an attempt is made to move a CG object on a tray-like operation device, the posture of the CG object at the time of association is maintained even on the tray. Therefore, for example, even if the pen that has been put in the pen is placed on the tray, the pen remains standing on the tray. For this reason, it is difficult to manipulate and observe a CG object like a real object.

  The present invention has been made in view of the above-described problems. When an operation device and a virtual object are associated with each other, the virtual object can be arranged so that the relationship between the positions and orientations thereof is appropriate, and a more natural virtual The object is to realize manipulation of objects.

In order to achieve the above object, an image presentation method according to the present invention includes:
An image presentation method in an information processing apparatus for drawing a virtual object in a real space observed by an observer,
A detecting step in which the detecting means detects the position and orientation of the operating device that actually exists in the real space; and
Whether the determination unit associates the virtual object with the operation device based on whether the relationship between the position and orientation of the virtual object and the position and orientation of the operation device is a preset relationship. A determination step for determining;
The position and orientation of the virtual object so that the relationship between the position and orientation of the virtual object and the position and orientation of the operating device is a preset relationship when it is determined that the changing unit associates in the determination step. and a change step of changing,
The drawing means includes a drawing step of drawing the virtual object with the position and orientation changed in the changing step.

In order to achieve the above object, an information processing apparatus according to the present invention comprises the following arrangement. That is,
An information processing apparatus for drawing a virtual object in a real space observed by an observer,
Detection means for detecting the position and orientation of the operating device that actually exists in the real space;
Determination means for determining whether or not to associate the virtual object with the operation device based on whether or not the relationship between the position and orientation of the virtual object and the position and orientation of the operation device is a preset relationship When,
Change that changes the position and orientation of the virtual object so that the relationship between the position and orientation of the virtual object and the position and orientation of the operating device is a preset relationship when it is determined by the determination means Means,
Drawing means for drawing the virtual object with the position and orientation changed by the changing means.

  According to the present invention, when the operation device and the virtual object are associated with each other, the virtual object is arranged so that the relationship between the positions and orientations thereof is appropriate. Objective.

  In the following embodiment, an example will be described in which a 6-degree-of-freedom sensor is attached to an underlay-like operation device, so that a CG object can be operated and observed as if it is placed on the underlay. An example will be described in which a CG screw can be operated by attaching a 6-degree-of-freedom sensor to a screwdriver-like operation device.

  Note that the operation device itself is not necessarily in the form of an underlay or a screwdriver. For example, if a simple receiver is used as an operation device as it is and an underlay-like device is displayed in CG, the operator can use it as an underlay-like operation device. Further, if the shaft portion is displayed in CG on the operation device having only the grip portion of the screwdriver, the operator can use it as a screwdriver-like operation device (described later with reference to FIG. 5A). Hereinafter, what the operator thinks is operating—an underlay-like operation device or a screwdriver-like operation device—is referred to as an operation item. That is, the operation item may be configured to include an operation device as an entity and a CG that supplements a visually lacking portion.

  Further, in the following embodiments, it is possible to associate a CG object with an operation item at a desired timing, or to solve the association, and further, it is possible to realize a realistic experience. For example, by associating an apple CG on a shelf with an underlay-like operation item, the apple can be moved as if it is on the underlay, and the apple CG and underlay on the desk. By solving the relationship with the operation item, it is possible to place the apple on the desk-that is, to move the apple from the shelf to the desk using the underlay. Also, by associating the CG of the screw on the component tray with the screwdriver-like operation item, the screw can be moved to the screw hole, and further, the screw on the component tray is The operation of inserting into the hole (screw tightening) can be realized.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating a configuration example of a virtual reality presentation system according to the present embodiment. Reference numeral 100 denotes a head-mounted video input / output device (called a head-mounted display, HMD: Head Mounted Display, etc., hereinafter referred to as HMD). An observer can observe an image obtained by combining the real space and the virtual space by wearing the HMD 100 on the head. The magnetic transmitter 200 generates a magnetic field, and the magnetic receivers 201 and 202 measure changes in the magnetic field generated by the magnetic transmitter 200. The position / orientation measuring apparatus 205 controls the magnetic transmitter 200 and measures the position / orientation of each magnetic receiver based on the measurement results from the magnetic receivers 201 and 202. A magnetic receiver 201 is attached to the HMD 100, and the viewpoint position and line-of-sight direction of the observer are calculated.

  The operation device 300 is used to form an operation item that can be operated by an observer, and a magnetic receiver 202 is incorporated therein. Based on the measurement result of the magnetic receiver 202, the position / posture measurement apparatus 205 can calculate the position / posture of the operation device 300, that is, the position / posture of the operation item. Here, it is assumed that the operation device 300 is provided with an underlay-like portion, and an operation item is formed without replenishment with CG. The HMD 100 incorporates two sets of video display devices 101 and video input devices 102, one set for each of the right eye and the left eye.

  The information processing apparatus 400 generates a CG image based on the position / orientation information calculated by the position / orientation measurement apparatus 205 and superimposes the CG image on the image input from the image input apparatus 102 of the HMD 100 to generate a composite image. The obtained synthesized video is output to the video display device 101.

  In the present embodiment, the six-degree-of-freedom sensor refers to the magnetic receivers 201 and 202, and the six-degree-of-freedom sensor system refers to the magnetic transmitter 200, the magnetic receivers 201 and 202, the position / posture measuring device 205, and the information processing device. It refers to a set of software (not shown) that operates within the 400 and obtains the position and orientation of the magnetic receivers 201 and 202. In order to distinguish the six-degree-of-freedom sensors, the magnetic receiver 201 is called a viewpoint detection six-degree-of-freedom sensor, and the magnetic receiver 202 is called an object manipulation six-degree-of-freedom sensor.

  Next, a specific configuration of the HMD 100 of the present embodiment will be described with reference to FIG. The video display device 101 also shown in FIG. 1 is configured by a small liquid crystal display device of about 0.5 to several inches. The free-form surface prism 103 serves as a lens for enlarging the image of the image display device 101. With these configurations, the image displayed on the image display device 101 is presented to the observer as an image equivalent to 90 inches, for example, 2 meters ahead.

  The video input device 102 also shown in FIG. 1 is configured by an imaging device such as a CCD camera or a CMOS camera. The imaging system prism 104 serves as a lens for converging light in the real space to the video input device 102. By disposing the imaging system prism 104 outside the free-form surface prism 103 so that the optical axes coincide with each other, the parallax between the video input by the video input device 102 and the video displayed on the video display device 101 is eliminated. It is possible to reproduce the image of the space without a sense of incongruity.

  Next, a specific configuration of the information processing apparatus 400 in FIG. 1 will be described with reference to FIG. The information processing apparatus 400 can be realized by a general personal computer or workstation, and the functions of each unit shown in FIG. 3 are realized by a CPU (not shown) executing a predetermined control program. .

  The video capture units 401R and 401L capture video data input from the video input device 102 as digital signals in the processing device. The position / posture information input unit 404 captures the position / posture data of the HMD 100 and the object operation device 300 sent from the position / posture measurement apparatus 205 into the information processing apparatus 400. The position / posture calculation unit 405 calculates the positions of the HMD 100 and the object operation device 300 based on the data from the position / posture input unit 404. This calculation result is reflected in the 3DCG data 406. Note that the 3DCG data 406 is stored in, for example, a hard disk or a RAM included in the information processing apparatus 400. The 3DCG data 406 includes 3DCG data of an operation item directly operated by the object operation device 300 and 3DCG data of a CG object to be operated by the operation item. Further, the 3DCG data 406 includes data necessary for CG drawing, such as the position / posture of the operation item in the world coordinate system and the position / posture of the HMD 100. Based on the 3DCG data 406, the CG rendering unit 407 calculates the position, size, angle (perspective), and the like at which CG data should be drawn, and renders 3DCG data such as a CG object.

  The video synthesizing units 402R, L synthesize the video data from the video capturing units 401R, 401 and the CG rendered by the CG rendering unit 407, and generate a composite image for the right eye and the left eye. The video generation units 403R and L send the right-eye and left-eye composite images generated by the video synthesis units 402R and L to the video display device 101 as video signals.

  The arrangement rule 408 is information that defines “at what timing” and “how” to execute the fixation / release of the operation item of the CG object. The arrangement rule 408 is stored in, for example, a hard disk or RAM in the information processing apparatus 400. The arrangement determination unit 409 determines the arrangement of the CG object according to the arrangement rule 408. The determination result by the arrangement determination unit 409 is passed to the coordinate conversion unit 413. The coordinate conversion unit 413 changes the 3DCG data 406 based on the determination result of the arrangement determination unit 409, that is, the arrangement rule 408.

  Various conditions are described in the conditions of the placement rule. The arrangement condition calculation unit 412 calculates whether or not these conditions are satisfied. One or more arrangement condition calculation units 412 exist in the system according to the conditions. The result of the arrangement condition calculation unit 412 is passed to the arrangement determination unit 409. The input of the arrangement condition calculation unit 412 includes a case where the feature of the CG object is used and a case other than that. When the features of the CG object are used, the CG arrangement feature extraction unit 410 is used. In other cases, the arrangement feature extraction unit 411 is used.

  The CG arrangement feature extraction unit 410 extracts specific features from the 3DCG data. In the case of a complicated feature, the output of another CG arrangement feature extraction unit 410 may be used as an input. For example, when handling a part having a predetermined shape, the size of the part may be required. Therefore, a CG arrangement feature extraction unit for calculating “the size of the part” is required. If the size of the part is a property, it is only necessary to read it, but if it is not, it is necessary to obtain the size from the surface information. The feature extraction unit will be used. One or more CG placement feature extraction units 410 exist in the system depending on the features declared in the placement rules 408 (for example, the positions and orientations of a plurality of surfaces set for the CG object). In the present embodiment, the CG arrangement feature extraction unit 410 calculates the position and orientation of each surface of the bounding box of the CG object, and also calculates the position and orientation of the underlying surface of the operation device 300. Of course, the underlying surface of the operation item may be supplemented with CG. In this case, the position and orientation of the surface are calculated from the 3DCG data and the position and orientation information of the operation device 300. The arrangement feature extraction unit 411 indicates an internal state of the information processing apparatus 400 such as an input from an external device such as whether or not a push button has been pressed, and a result of image processing performed on a video acquired by the video capture unit 401. Used for input. Depending on the conditions of the placement rule 408, the placement feature extraction unit 411 may exist in the system, may exist, or may exist. Note that the case where the internal state of the information processing apparatus 400 such as the result of performing image processing on the video acquired by the video capture unit 401 is input is, for example, the following case. ZIF socket (Zero Insertion Force Socket; a type of IC socket that handles devices that are frequently inserted and removed, such as ROM writers, and those that require a lot of force to insert and remove the latest multi-pin CPU / MPU, etc. If the lever is in the released state, the IC cannot be inserted or removed. Therefore, when the IC is a CG and the ZIF socket is a real object, whether the lever of the ZIF socket is in the released state or the locked state is obtained by image processing, and whether or not the CG IC can be inserted and removed is determined using this result.

  The process flow of the system configured as described above will be described with reference to FIG. Using the data of the magnetic transmitter 200 and the magnetic receiver 202, the position / orientation measuring apparatus 205 measures the position / orientation of the object operating device 300 in the real space (step S2010). Similarly, using the data of the magnetic transmitter 200 and the magnetic receiver 201, the position / posture measuring apparatus 205 measures the position / posture in the real space of the HMD 100 worn by the observer (S2020). Data obtained by the measurement is taken into the information processing apparatus 400 through the position / posture information input unit 404. The position / orientation calculation unit 405 calculates the position and orientation at which each CG object is drawn, and the result is reflected in the 3DCG data 406 (S2030).

In parallel with these processes, a process according to the arrangement rule is performed and reflected in the 3DCG data. Here, as an example, by using an underlay-like operation item (operation device 300) and pressing this button close to the CG object, a push button (not shown) provided on the operation device 300 is pressed, thereby the CG object Think of making it fit exactly on the surface of the underlay. First, the surface used for the process demonstrated below is demonstrated.
・ Under surface 1: Front surface of underlay operation item ・ Under surface 2: Back surface of underlay operation item ・ Object surfaces 1 to 6: Each surface constituting a hexahedral bounding box in the object coordinate system of a CG object .

In the present embodiment, it is assumed that the arrangement rule 408 is set as follows, for example (∧ represents a logical product, and ¬ represents negative).
Condition 1: The push button was pressed.
Condition 2: The object coordinate system of the CG object is stationary in the world coordinate system.
Condition 3: The degree of identity of the underlying surface 1 and the object surface 1 is higher than the degree of identity of other combinations.
Condition 4: The degree of identity of the underlying surface 1 and the object surface 2 is higher than the degree of identity of other combinations.
Condition 5: The degree of identity of the underlying surface 1 and the object surface 3 is higher than the degree of identity of other combinations.
Condition 6: The degree of identity of the underlying surface 1 and the object surface 4 is higher than the degree of identity of other combinations.
Condition 7: The degree of identity of the underlying surface 1 and the object surface 5 is higher than the degree of identity of other combinations.
Condition 8: The degree of identity of the underlying surface 1 and the object surface 6 is higher than the degree of identity of other combinations.
Condition 9: The degree of identity of the underlying surface 2 and the object surface 1 is higher than the degree of identity of the other combinations.
Condition 10: The degree of identity of the underlying surface 2 and the object surface 2 is higher than the degree of identity of other combinations.
Condition 11: The degree of identity of the underlying surface 2 and the object surface 3 is higher than the degree of identity of other combinations.
Condition 12: The degree of identity of the underlying surface 2 and the object surface 4 is higher than the degree of identity of other combinations.
Condition 13: The degree of identity of the underlay surface 2 and the object surface 5 is higher than the degree of identity of other combinations.
Condition 14: The degree of identity of the underlying surface 2 and the object surface 6 is higher than the degree of identity of other combinations.
Instruction 1: The object coordinate system of the CG object is stationary within the object coordinate system of the operation item.
Instruction 2: The object coordinate system of the CG object is stationary in the world coordinate system.
Instruction 3: Move / rotate the object surface 1 on the underlying surface 1.
Instruction 4: Move / rotate the object surface 2 on the underlying surface 1.
Instruction 5: Move / rotate the object surface 3 on the underlying surface 1.
Instruction 6: Move / rotate the object surface 4 onto the underlay surface 1.
Instruction 7: Move / rotate the object surface 5 on the underlying surface 1.
Instruction 8: Move / rotate the object surface 6 on the underlying surface 1.
Instruction 9: Move / rotate the object surface 1 on the underlying surface 2.
Instruction 10: Move / rotate the object surface 2 onto the underlay surface 2.
Instruction 11: Move / rotate the object surface 3 on the underlying surface 2.
Instruction 12: Move / rotate the object surface 4 onto the underlay surface 2.
Instruction 13: Move / rotate the object surface 5 on the underlying surface 2.
Instruction 14: Move / rotate the object surface 6 onto the underlying surface 2.
(Condition 1) ∧ (Condition 2) ∧ (Condition 3) ⇒ (Instruction 1) ∧ (Instruction 3).
(Condition 1) ∧ (Condition 2) ∧ (Condition 4) ⇒ (Instruction 1) ∧ (Instruction 4).
(Condition 1) ∧ (Condition 2) ∧ (Condition 5) ⇒ (Instruction 1) ∧ (Instruction 5).
(Condition 1) ∧ (Condition 2) ∧ (Condition 6) ⇒ (Instruction 1) ∧ (Instruction 6).
(Condition 1) ∧ (Condition 2) ∧ (Condition 7) ⇒ (Instruction 1) ∧ (Instruction 7).
(Condition 1) ∧ (Condition 2) ∧ (Condition 8) ⇒ (Instruction 1) ∧ (Instruction 8).
(Condition 1) ∧ (Condition 2) ∧ (Condition 9) ⇒ (Instruction 1) ∧ (Instruction 9).
(Condition 1) ∧ (Condition 2) ∧ (Condition 10) ⇒ (Instruction 1) ∧ (Instruction 10).
(Condition 1) ∧ (Condition 2) ∧ (Condition 11) ⇒ (Instruction 1) ∧ (Instruction 11).
(Condition 1) ∧ (Condition 2) ∧ (Condition 12) ⇒ (Instruction 1) ∧ (Instruction 12).
(Condition 1) ∧ (Condition 2) ∧ (Condition 13) ⇒ (Instruction 1) ∧ (Instruction 13).
(Condition 1) ∧ (Condition 2) ∧ (Condition 14) ⇒ (Instruction 1) ∧ (Instruction 14).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 3) ⇒ (Instruction 2) ∧ (Instruction 3).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 4) ⇒ (Instruction 2) ∧ (Instruction 4).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 5) ⇒ (Instruction 2) ∧ (Instruction 5).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 6) ⇒ (Instruction 2) ∧ (Instruction 6).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 7) ⇒ (Instruction 2) ∧ (Instruction 7).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 8) ⇒ (Instruction 2) ∧ (Instruction 8).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 9) ⇒ (Instruction 2) ∧ (Instruction 9).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 10) ⇒ (Instruction 2) ∧ (Instruction 10).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 11) ⇒ (Instruction 2) ∧ (Instruction 11).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 12) ⇒ (Instruction 2) ∧ (Instruction 12).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 13) ⇒ (Instruction 2) ∧ (Instruction 13).
(Condition 1) ∧¬ (Condition 2) ∧ (Condition 14) ⇒ (Instruction 2) ∧ (Instruction 14).

Here, the degree of identity between the surface A and the surface B is defined as follows in the present embodiment, for example.
-If the inner product of the normal vector of surface A and the normal vector of surface B ≥ 0 ⇒ the degree of identity is 0
・ Other than above ⇒ Make the reciprocal of the distance between surface A and surface B equal.
As the distance between the surface A and the surface B, for example, the distance between the centers of gravity of the surface A and the surface B may be used.

  In the following, a case will be described as an example where Condition 2 is satisfied as a precondition, that is, the object coordinate system of the CG object is stationary in the world coordinate system.

  The CG arrangement feature extraction unit extracts CG arrangement features from the 3DCG data 406 (S3010). Specifically, the relationship between the object coordinate system of the CG object and the world coordinate system, the underlying surface 1, the underlying surface 2, the object surface 1, the object surface 2, the object surface 3, the object surface 4, the object surface 5, and the object surface 6 Extraction. In addition, the arrangement feature extraction unit 411 extracts arrangement characteristics indicating whether or not the operation item push button has been pressed (S3020). The features extracted by the CG placement feature extraction unit 410 and the placement feature extraction unit 411 are arranged by the placement condition calculation unit 412 with values of placement conditions (success / failure) such as success / failure of condition 1, success / failure of condition 2,. Is expressed (S3030). Based on the values of these arrangement conditions, the arrangement determination unit 409 makes an arrangement determination (S3040). As a result, for example, when (Condition 1) ∧ (Condition 2) 成立 (Condition 3) is satisfied, the instruction (Instruction 1) ∧ (Instruction 3) is sent from the arrangement determination unit 409 to the coordinate conversion unit 413 as described above. Sent. Then, instructions 1 and 3 are executed for the 3DCG data 406 (S3050). Thus, when the push button is pressed in a state where the object surface 1 set as the CG object and the underlay surface 1 set as the operation item are brought close to each other, the coordinate conversion unit 413 causes the positions of the object surface 1 and the underlay surface 1 and Move and rotate the CG object so that their normal directions match (more precisely, their normals are in opposite directions (inner product of normal vectors is -1)) By associating the object coordinate system of the CG object within the object coordinate system of the operation item, the two are associated with each other.

  Further, in parallel with the above operation, an image in the real space is taken into the information processing apparatus 400 through the video capture unit 401 from the video input device 101 of the HMD 100 (S4010). The CG rendering unit 407 renders a CG using the 3DCG data 406 and develops it in a memory such as a video buffer (S2040). As a result, the CG object is drawn so that the object surface 1 of the CG object and the underlay surface 1 (surface) of the operation item coincide. On the other hand, the real space video data captured in S4010 is also developed in a memory such as a video buffer (S4020), and the video synthesis unit 402 superimposes the CG video generated in S2040 (S5010). The synthesized video is displayed on the video display device 101 of the HMD 100 via the video generation unit 403 (S5020).

  Information is presented in real time by repeatedly performing the processes of S2010, S2020, S2030, S2040, S4010, S4020, S5010, and S5020 at the video update interval in the video display device 101 or the update interval in the CG rendering 2040.

  By changing the arrangement rule, for example, it is possible to make a desired setting such that the association is not executed no matter how close the surface of the tray and the upper surface of the bounding box of the CG object are. In addition, since it is possible to describe as an arrangement rule, such setting and changing can be easily performed. Moreover, although the bounding box comprised by a hexahedron was used, it is possible to use not only a hexahedron but arbitrary polyhedrons. In the above description, in the association between the surface of the operation item and the surface of the CG object, the relationship between the postures of the two is determined by matching the selected surface with the surface. Here, at which position on the surface of the operation item the CG object is arranged is determined by the arrangement rule. In the above-described embodiment (when a CG object is placed on the underlay), when the CG object is arranged on the underlay surface 1 side, for example, (a) the perpendicular foot that is lowered from the center of gravity of the CG object to the underlay surface 1 A method is conceivable in which the position of the leg of the perpendicular line that is lowered from the center of gravity of the CG object to the underlay surface 1 does not change even after the arrangement, so as to coincide with the center point of the surface 1. In addition, in the case of a screwdriver and screw described below, one location is designated.

  Next, presentation of MR that enables a virtual experience of screw tightening by using a screwdriver as an operation item and a screw-like CG as a CG object will be described. In the following, there will be described a case of a virtual experience in which there are three screw CGs, and one of them is used for screw tightening.

  In FIG. 5A, the object operating device 501 is only the “nigiri” portion of the screwdriver, and the magnetic sensor 202 is incorporated inside. The object operation device 501 is equipped with a push button 502. When viewed from an observer, “axis 503” is displayed in CG on the object operation device 501 and can be handled as an operation item for screwdriver.

  First, terms in this example are defined. As shown in FIG. 5B, it is assumed that there are three CGs of “screws”, and these are distinguished as “screw 1”, “screw 2”, and “screw 3”. “Screwdriver shaft” is the CG “axis 503” of the operation item, “Screwdriver tip” is the tip 504 of the screwdriver shaft 503, and “Screw head” is the head 511 of the screw CG 510. “The distance between the screwdriver and the screw” is a distance 523 between the tip of the screwdriver and the screw head. Furthermore, “the direction of the screwdriver” means the direction 521 from the operation device side toward the tip of the screwdriver along the screwdriver axis 503, and “the direction of the screw” means that from the screw head along the axis of the CG of the screw. The direction 522 is toward the opposite side.

Assume that the arrangement rules of this example are set as follows, for example.
Condition 1: The push button is pressed.
Condition 2: The object coordinate system of the screw 1 is stationary in the world coordinate system. ,
Condition 3: The object coordinate system of the screw 2 is stationary in the world coordinate system.
Condition 4: The object coordinate system of the screw 3 is stationary in the world coordinate system.
Condition 5: (Condition 2) ∧ (Condition 3) ∧ (Condition 4).
Condition 6: Distance between screwdriver and screw 1 <constant value.
Condition 7: Distance between screwdriver and screw 2 <constant value.
Condition 8: Distance between screwdriver and screw 3 <constant value.
Condition 9: Distance between screwdriver and screw 1 <distance between screwdriver and screw 2
Condition 10: Distance between screwdriver and screw 2 <distance between screwdriver and screw 3
Condition 11: Distance between screwdriver and screw 3 <distance between screwdriver and screw 1
Condition 12: (Condition 6) ∧ (Condition 9) ∧¬ (Condition 11).
Condition 13: (Condition 7) ∧¬ (Condition 9) ∧ (Condition 10).
Condition 14: (Condition 8) ∧¬ (Condition 10) ∧ (Condition 11).
Instruction 1: The object coordinate system of the screw 1 is stationary within the object coordinate system of the operation item.
Instruction 2: The object coordinate system of screw 1 is stationary in the world coordinate system.
Instruction 3: The object coordinate system of the screw 2 is stationary in the object coordinate system of the operation item.
Instruction 4: The object coordinate system of the screw 2 is stationary in the world coordinate system.
Instruction 5: The object coordinate system of the screw 3 is stationary within the object coordinate system of the operation item.
Instruction 6: The object coordinate system of the screw 3 is stationary in the world coordinate system.
Instruction 7: Rotate the direction of the screw 1 to coincide with the direction of the screwdriver.
Instruction 8: Rotate the direction of the screw 2 to coincide with the direction of the screwdriver.
Instruction 9: Rotate the direction of the screw 3 to coincide with the direction of the screwdriver.
Instruction 10: The screw head of the screw 1 is moved to coincide with the tip of the screwdriver.
Instruction 11: Move the screw head of the screw 2 so as to coincide with the tip of the screwdriver.
Instruction 12: Move the screw head of the screw 3 so as to coincide with the tip of the screwdriver.
(Condition 1) ∧ (Condition 5) ∧ (Condition 12) ⇒ (Instruction 1) ∧ (Instruction 7) ∧ (Instruction 10).
(Condition 1) ∧ (Condition 5) ∧ (Condition 13) ⇒ (Instruction 3) ∧ (Instruction 8) ∧ (Instruction 11).
(Condition 1) ∧ (Condition 5) ∧ (Condition 14) ⇒ (Instruction 5) ∧ (Instruction 9) ∧ (Instruction 12).
¬ (Condition 1) ⇒ (Instruction 2) ∧ (Instruction 4) ∧ (Instruction 6).

  When the arrangement rule as described above is used, as shown in FIG. 6A, the operator brings the tip 504 of the screwdriver close to the screw and presses the push button 502, so that FIG. As shown, the screw with the screw head 511 closest to the tip 504 sticks to the screwdriver tip 504 upon execution of instructions 10-12. At this time, the execution of any of the instructions 7 to 9 causes the CG to rotate so that the direction 522 of the screw that is attached matches the direction 521 of the screwdriver. After that, while holding down the push button 502, bring the screw to a place to be tightened, and release the push button, and a virtual experience of a series of screw tightening operations can be performed ((c) and (d) in FIG. 6).

  As described above, according to the above embodiment, a CG object is placed on an underlay-like operation item as if a real object is placed, or a screwdriver-like operation item is screwed like a real screw. CG can be arranged, and in the real world, it is possible to experience in advance what can happen (virtual experience).

  Although the embodiments have been described in detail above, the present invention can take an embodiment as, for example, a system, an apparatus, a method, a program, or a storage medium, and specifically includes a plurality of devices. The present invention may be applied to a system that is configured, or may be applied to an apparatus that includes a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowchart shown in the figure) that realizes the functions of the above-described embodiment is directly or remotely supplied to the system or apparatus, and the computer of the system or apparatus Is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

  As a recording medium for supplying the program, for example, floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, a client computer browser is used to connect to an Internet homepage, and the computer program of the present invention itself or a compressed file including an automatic installation function is downloaded from the homepage to a recording medium such as a hard disk. Can also be supplied. It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.

  Furthermore, after the program read from the recording medium is written to a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU or the like provided in the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It is a figure which shows the structural example of the system by embodiment. It is a figure which shows the specific structure of a head mounting | wearing type | mold video input / output device. It is a figure which shows the specific structure of the information processing apparatus by embodiment. It is a figure which shows the flow of the image presentation process by embodiment. It is a figure explaining the term regarding the operation item and operation object by embodiment. It is a figure explaining the virtual experience by embodiment.

Claims (11)

An image presentation method in an information processing apparatus for drawing a virtual object in a real space observed by an observer,
A detecting step in which the detecting means detects the position and orientation of the operating device that actually exists in the real space; and
Whether the determination unit associates the virtual object with the operation device based on whether the relationship between the position and orientation of the virtual object and the position and orientation of the operation device is a preset relationship. A determination step for determining;
The position and orientation of the virtual object so that the relationship between the position and orientation of the virtual object and the position and orientation of the operating device is a preset relationship when it is determined that the changing unit associates in the determination step. Change process to change,
An image presentation method, comprising: a drawing unit that draws the virtual object with the position and orientation changed in the changing step.   2. The changing step according to claim 1, wherein the changing unit changes the position and orientation of the virtual object so that a plane set for the virtual object matches a plane set for the operation device. 2. The image presentation method according to 1.   In the changing step, the changing means is set in the operation device and a plurality of surfaces set in the virtual object based on the relationship between the position and orientation of the virtual object and the position and orientation of the operation device. 3. The image presentation method according to claim 2, wherein which of the plurality of faces is determined to be coincident.   In the determination step, whether the determination unit has a preset relationship between the position and orientation of the plurality of surfaces set in the virtual object and the position and orientation of the plurality of surfaces set in the operation device. The image presentation method according to claim 1, wherein it is determined whether to associate the virtual object with the operation device based on whether or not the virtual object is associated.   In the changing step, the changing means changes the position and orientation of the virtual object so that the direction of the axis set for the virtual object matches the direction of the axis set for the operating device. The image presentation method according to claim 1.   6. The changing step according to claim 5, wherein the changing unit changes the position and orientation of the virtual object so that a predetermined part of the virtual object and a predetermined part of the operation device coincide with each other. The image presentation method described.   7. The image presentation method according to claim 1, wherein a part of the operation device is configured by a real object and the remaining part is configured by CG. A storage unit for storing a description of a rule that defines a relationship between a position and orientation when the virtual object and the operation device are associated;
In the changing step, the changing means causes the virtual object so that the relationship between the position and orientation of the virtual object and the position and orientation of the operating device is defined by a rule description stored in the storage means. The image presentation method according to claim 1, wherein the position and orientation of the object is changed. An information processing apparatus for drawing a virtual object in a real space observed by an observer,
Detection means for detecting the position and orientation of the operating device that actually exists in the real space;
Determination means for determining whether or not to associate the virtual object with the operation device based on whether or not the relationship between the position and orientation of the virtual object and the position and orientation of the operation device is a preset relationship When,
Change that changes the position and orientation of the virtual object so that the relationship between the position and orientation of the virtual object and the position and orientation of the operating device is a preset relationship when it is determined by the determination means Means,
An information processing apparatus comprising: a drawing unit that draws the virtual object with the position and orientation changed by the changing unit.   The control program for making a computer perform each process of the image presentation method of any one of Claims 1 thru | or 8.   A computer-readable storage medium storing a control program for causing a computer to execute each step of the image presentation method according to claim 1.

Images