JP2000250689A - Virtual object presentation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual object presentation system capable of feeding back the inner force sense of a virtual object with a high degree of freedom without restraining a body in complicated mechanical configuration. SOLUTION: A reaction force presenting member 6 can be mounted on a prescribed part of a human body and is composed of a magnetic substance. A marker tracing device 4 detects the position of the reaction force presenting member 6. A CPU 1 calculates a positional relation between the virtual object and the prescribed part of the human body in a virtual reality space from the position of the reaction force presenting member 6 detected by the marker tracing device 4 and calculates a reaction force to be applied to the reaction force presenting member 6 on the basis of that positional relation. A magnetic field generator 5 generates a magnetic field and applies the reaction force corresponding to the calculated result of the CPU 1 to the reaction force presenting member 6.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a virtual object presentation system, and more particularly, to a virtual object presentation system for presenting a force sense of a virtual object in a virtual reality space.

[0002]

2. Description of the Related Art In a virtual reality space created by virtual reality technology, CG (computer gr
aphics) The video and the video from the remote controlled camera are H
This is indicated to the operator by a display device such as an MD (head mounted display). At the same time, the body (finger, hand, arm, foot, etc.) of the operator in the virtual reality space is also shown as a CG image based on data obtained by an input device such as a data glove. Therefore, if the operation of grasping the virtual object with the real hand is performed, the virtual hand in the video grasps the virtual object in accordance with the movement of the real hand.

However, it is not possible to feed back the presence of a virtual object to a real hand as a force sensation (force sense) (so-called force feedback), so that a reaction force from the virtual object cannot be obtained. Therefore, the positional relationship between the virtual object and the virtual hand can be recognized only through the video,
It is difficult to get a high degree of reality. In order to solve this, an exoskeleton type force sense presentation device (so-called force display) that enables force feedback is disclosed in JP-A-6-324622 and JP-A-5-232859.
JP, JP-A-6-102980, JP-A-8-3
No. 14626, JP-A-9-330016, JP-A-8-314626, and JP-A-9-503603
It is proposed in Japanese Patent Publication No.

[0004]

However, the conventionally proposed exoskeleton type force sensation display device includes a part of a human body (finger, hand, arm, foot, etc.) to be subjected to force feedback.
Must be restrained by a complicated mechanical configuration, and only the specific operation of the operator can be dealt with.

The present invention has been made in view of such a situation, and it is possible to feed back the sense of force of a virtual object with a high degree of freedom without restricting the operator's body with a complicated mechanical configuration. It is an object to provide a virtual object presentation system.

[0006]

In order to achieve the above object, a virtual object presentation system according to the present invention is configured to be attachable to a predetermined part of a human body and to present a reaction force composed of a magnetic material. A member, a tracking device that detects the position of the reaction force presenting member, and a positional relationship between the virtual object and the predetermined portion in the virtual reality space is calculated from the position of the reaction force presenting member detected by the tracking device. A calculating device that calculates a reaction force applied to the reaction force presenting member based on the positional relationship, and a reaction force corresponding to a calculation result of the calculation device is applied to the reaction force presenting member by generating a magnetic field. And a magnetic field generator.

[0007]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a virtual object presentation system embodying the present invention will be described with reference to the drawings. Note that the same or corresponding portions in the embodiments are denoted by the same reference numerals, and redundant description will be appropriately omitted.

First, FIG. 1 shows a schematic configuration of a virtual object presentation system according to the present invention. This virtual object presentation system is an exoskeleton type force display system that provides force feedback corresponding to a virtual object in a virtual reality space to a predetermined part of a human body, and includes a CPU (centr
al processing unit) 1, a viewpoint detecting device 2, an image display device 3, a marker tracking device 4, a magnetic field generating device 5, and a reaction force presenting member 6.

The viewpoint detecting device 2 is a device for detecting a position where an operator is looking in a virtual reality space. The viewpoint detection device 2 includes, for example, a line-of-sight input sensor that detects a line of sight from the rotation angle of the operator's eyes, and a head tracker that detects a line of sight from the inclination angle of the operator's head.
(For example, incorporated in the HMD). The video display device 3 is a display device that presents a virtual reality space as visual information to an operator based on data from the CPU 1.
(For example, HMD). The displayed image is a CG image, an image from a remotely operated camera, or the like. An image centered on a virtual object viewed by the operator in the virtual reality space is displayed based on data from the viewpoint detection device 2. . At this time, the operator's body (for example, finger, hand, arm, foot, etc.) in the virtual reality space is also changed by the marker tracking device 4.
Is shown as a CG image based on the data obtained by. Therefore, if the operation of grasping the virtual object with the real hand is performed, the virtual hand in the video grasps the virtual object in accordance with the movement of the real hand.

The reaction force presenting member 6 has an attractive force in a magnetic field,
It is made of a magnetic material that receives at least one of the repulsive forces as a magnetic force. For example, metal materials such as iron, nickel, and cobalt, and alloys thereof, magnetic materials such as iron oxide and metal oxide (ferrite); permanent magnets made of such magnetic materials. The reaction force presenting member 6 is provided at a predetermined portion of the human body (for example, a finger, a hand,
(Arms, feet, etc.). Examples of such a reaction force presenting member 6 include a magnetic ring,
Examples include a magnetic tube, a glove having a magnetic pattern having such a shape, and the like. Further, a marker is provided on the reaction force presenting member 6, and the position of the reaction force presenting member 6 can be detected by the marker tracking device 4 (for example, a thumb tracker). Note that a general motion capture system may be used as the marker tracking device 4.

The CPU 1 calculates a positional relationship between a virtual object and a predetermined part of a human body in a virtual reality space from a position of the reaction force presenting member 6 detected by the marker tracking device 4, and based on the positional relationship, calculates a reaction. This is an arithmetic unit that calculates a reaction force applied to the force presenting member 6. The magnetic field generation device 5 is a device that generates a magnetic field to apply a reaction force corresponding to the calculation result of the CPU 1 to the reaction force presenting member 6, and is configured by, for example, an electromagnet formed of a coil.

Next, an embodiment of a virtual object presentation system according to the present invention will be described with reference to FIGS. This system provides force feedback corresponding to the virtual object V1 (FIG. 3) to the arm P1 of the operator P, and FIGS. 2 and 3 show the state of use. When using this system, the operator P places the HMD1 on the head.
3 (FIG. 2), and a magnetic tube 16 (FIG. 3) corresponding to the reaction force presenting member 6 is mounted on the arm P1. And Figure 2,
The arm P1 is inserted into the box 17 as shown in FIG.
The HMD 13 (FIG. 2) contains the viewpoint detection device 2, and the control unit 11 (FIG. 2) contains the CPU 1. Inside the box 17, a CCD (Charge Coupled Device) camera 14 (FIG. 3) constituting the marker tracking device 4 and an electromagnet 15 constituting the magnetic field generator 5 are provided.
a, 15b (FIG. 3).

FIG. 4A shows the inside of the box 17 as viewed from the side, and FIG. 4B shows the state of the box 17 as viewed from the front.
The state inside is shown. The magnetic tube 16 is made of a permanent magnet. Here, as shown in FIG.
The S pole 16a is located on the a side, and the N pole 16b is located on the electromagnet 15b side. In addition, when there is a concern about the influence on the human body, a magnetic shield may be provided on the inner wall of the tube as necessary. Also, the magnetic tube 1
A total of six (three on each side) markers 16M are provided on both sides of the outer peripheral surface of the six. The CCD camera 14 captures video data of the entire space between the electromagnets 15a and 15b,
It has a function of detecting the position of the magnetic tube 16 from the position of each marker 16M. The magnetic tube 1
The inclination of 6 is detected from the distance difference from the CCD camera 14 to each marker 16M.

The arm P is moved from the state shown in FIG.
When the arm P1 moves in the virtual reality space and reaches the contact position with the virtual object V1, the electromagnet 15b is excited so that the magnetic tube 16 side becomes the N pole. FIGS. 5A and 5B show the state inside the box 17 at this time in association with FIGS. 4A and 4B. Since the N pole portion 16b is located on the electromagnet 15b side of the magnetic tube 16, a magnetic force is generated in the magnetic field generated by the electromagnet 15b,
The magnetic tube 16 receives a reaction force due to repulsion between the pole portion 16b and the electromagnet 15b. As a result, the reaction force received by the magnetic tube 16 is transmitted to the arm P1,
The arm P1 is stopped. Thus, the virtual object V
The force sense of 1 is fed back to the arm P1 via the magnetic tube 16. Since the arm P1 is not restrained by a complicated mechanical configuration, force feedback corresponding to the virtual object V1 is achieved with a high degree of freedom.

The electromagnet 15b is connected to the magnetic tube 16 by N.
Instead of being excited to be a pole, the electromagnet 15a may be excited so that the magnetic tube 16 side has an N pole.
In this case, the magnetic tube 16 receives a reaction force due to the attractive force between the S pole portion 16a and the electromagnet 15a.
When the magnetic tube 16 is made of a metal material such as iron that does not have magnetism, the attractive force between the magnetic tube 16 and the electromagnet 15a causes the magnetic tube 16 to be attached to the magnetic tube 16 regardless of the magnetic pole direction of the electromagnet 15a. A reaction force can be given.

The reaction force applied to the magnetic tube 16 as described above is determined by the CPU 1 in the control unit 11 by C
From the position of the magnetic tube 16 detected by the CD camera 14, the virtual object V1 and the arm P in the virtual reality space are determined.
1 and is calculated based on the positional relationship. For example, the magnitude of the reaction force applied to the magnetic tube 16 can be adjusted by controlling the exciting current value, intermittent drive control by time division, or the like. And the electromagnet 15
a or 15b generates a magnetic field, so that the CPU
The reaction force corresponding to the calculation result of 1 is applied to the magnetic tube 16. Furthermore, if data on the characteristics (for example, hardness, softness, elasticity, brittleness, etc.) of the virtual object V1 is used, the characteristics of the virtual object V1 can be reflected on the reaction force applied to the magnetic tube 16. Also, the virtual object V1
When the arm P1 moves in a state of contact with the arm, the arm P1 is vibrated by repeating the reversal of the magnetic field, whereby it is also possible to present a tactile sensation (smoothness, roughness, etc.).

Next, another embodiment of the virtual object presentation system according to the present invention will be described with reference to FIGS. FIG. 6 shows an example of an image in the virtual reality space, and shows a state in which the virtual finger P3 is touching the virtual object V2,
This is indicated by a G image. The present system performs force feedback corresponding to the virtual object V2 on the hand P2 and the finger P3 of the operator P, and FIGS. 7 and 8 show the use state thereof. When using this system, the operator P wears the HMD 13 (FIG. 7) on the head and attaches the magnetic ring 26 (FIG. 8) corresponding to the reaction force presenting member 6 to one finger P.
Attach to 3. The magnetic ring 26 is a non-magnetic metal ring, and a marker 26M is provided on the outer periphery thereof as shown in FIG. As in the above-described embodiment (FIG. 2), the HMD 13 (FIG. 7) includes the viewpoint detecting device 2 and the control unit 11 (FIG. 7) includes the CPU 1.

As shown in FIG. 9, a multi-joint drive mechanism 29 having a sensor unit 28 at its tip is provided in the box 27.
Is provided. The sensor unit 28 includes a CCD camera 24 forming the marker tracking device 4 and two electromagnets 25 forming the magnetic field generator 5. The CCD camera 24 captures the video data of the space in the box 27. The video data is the magnetic ring 2
6 is used to detect the position of the magnetic ring 26 from the position of the marker 26M provided on the 6.

When the hand P2 on which the magnetic ring 26 is mounted is put into the box 27, the CCD camera 24 detects the marker 26M of the magnetic ring 26, and based on the detection result, the magnetic ring 26 and the sensor unit are detected. The multi-joint drive mechanism 29 performs tracking so that the distance from the multi-joint 28 is kept constant. Then, as shown in FIG. 8, when the finger P3 reaches the contact position with the virtual object V2, the electromagnet 25 of the sensor unit 28 is excited to generate a magnetic field, and the magnetic force is generated by the attractive force between the magnetic ring 26 and the electromagnet 25. The magnetic ring 26 receives the reaction force. As a result, the reaction force received by the magnetic ring 26 is transmitted to the finger P3, and the finger P3
Etc. are stopped. In this way, the force sense of the virtual object V2 is fed back to the finger P3 via the magnetic ring 26. Since the finger P3 is not restrained by a complicated mechanical configuration, force feedback corresponding to the virtual object V2 is achieved with a high degree of freedom.

Instead of using a magnetic ring 26 made of a non-magnetic metal, a magnetic ring 26 made of a permanent magnet may be used. In this case, the positional relationship between the marker 26M and the magnetic pole needs to be set appropriately in advance,
A reaction force can be applied to the magnetic ring 26 by appropriately selecting the attraction and repulsion between the electromagnet 25 and the magnetic ring 26. For example, if a magnetic field having the same polarity as the virtual object V2 side of the magnetic substance ring 26 is given from the virtual object V2 side by the electromagnet 25,
A repulsive force between the magnetic ring 26 and the electromagnet 25 can apply a reaction force to the magnetic ring 26.

The reaction force applied to the magnetic ring 26 as described above is determined by the CPU 1 in the control unit 11
The positional relationship between the virtual object V2 and the finger P3 or the like in the virtual reality space is calculated from the position of the magnetic ring 26 detected by the D camera 24, and is calculated based on the positional relationship. For example, the magnitude of the reaction force applied to the magnetic ring 26 can be adjusted by controlling the exciting current value, intermittent drive control by time division, or the like. When the electromagnet 25 generates a magnetic field, a reaction force corresponding to the calculation result of the CPU 1 is applied to the magnetic ring 26. Furthermore, by using data on characteristics (for example, hardness, softness, elasticity, brittleness, and the like) of the virtual object V2, the characteristics of the virtual object V2 can be reflected in the reaction force applied to the magnetic ring 26. Further, the finger P is in contact with the virtual object V2.
When 3 moves, the finger P3 or the like is vibrated by repeating the reversal of the magnetic field, whereby a tactile sensation (smoothness, roughness, etc.) can be presented.

FIG. 10 shows a state in which three magnetic rings 26 are attached to one finger P3, and a sensor unit 28 and an articulated drive mechanism 29 which are in charge of each of the magnetic rings 26 are opposite to each magnetic ring 26. 4 shows an embodiment for applying force. FIG. 12 shows a state in which a plurality of (15) magnetic substance rings 26 are attached to each finger P3 and wrist P4 as shown in FIG. At 29, an embodiment in which a reaction force is applied to each magnetic ring 26 is shown.

According to these embodiments (FIGS. 10 to 12), the sense of force of the virtual object V2 is fed back to a plurality of points of the hand P2, so that a higher sense of reality can be obtained. Note that the magnetic ring 26 is not necessarily the finger P3.
Since it does not have to be directly attached to
The relative position of each magnetic ring 26 may be fixed to one member in advance. For example, a thin glove to which each magnetic ring 26 is fixed so that the marker 26M is properly positioned may be used, or a thin glove having a magnetic pattern formed on the surface may be used.

In the case of the embodiment shown in FIG. 12, a magnetic field having the same polarity as the virtual object V2 side of the magnetic ring 26 is applied to the virtual object V2.
When applied by the electromagnet 25 from the side, a repulsive force between the magnetic ring 26 and the electromagnet 25 applies a reaction force to the magnetic ring 26. As a result, the magnetic ring 26
Is transmitted to the finger P3 and the wrist P4, and the force sense of the virtual object V2 is fed back to the entire hand P2 via the magnetic ring 26. In order to perform such force feedback satisfactorily, it is necessary to appropriately attach each magnetic ring 26 to the finger P3 or the like. The pre-search box 31 shown in FIG. 13 checks the mounting state of each magnetic ring 26.

The pre-search box 31 contains all markers 2
This is a device for checking whether or not 6M (FIG. 11) faces the same direction (here, the back of the hand).
When the hand P2 on which the magnetic ring 26 is mounted is placed at a predetermined position 32 in the pre-search box 31, the image data of the hand P2 and the magnetic ring 26 is captured by the upper CCD camera unit 33. The verticality and the like of the magnetic ring 26 are checked from the arrangement of the markers 26M, and if there is a magnetic ring 26 that is not properly mounted, a warning is displayed on the display unit 34 as to which one is.

FIG. 14 shows an embodiment for performing scissor-type tracking. The magnetic ring 26 is provided with two markers 26M that are respectively detected and tracked by the two CCD cameras 24. Then, marker tracking and magnetic field generation for one magnetic ring 26 are performed between the two sensor units 28, and the magnetic ring 26
Is given a reaction force. FIG. 15 shows an embodiment in which enclosing type (ring zone) tracking is performed. The magnetic ring 26 is provided with four markers 26M that are respectively detected and tracked by the four CCD cameras 24. Then, within the enclosed sensor unit 28A, marker tracking and magnetic field generation are performed on one magnetic ring 26 from above, below, left and right, and a reaction force is applied to the magnetic ring 26. Here, the square-shaped sensor unit 28 is used.
Although A is used, other polygonal or annular sensor units may be used according to the number of markers 26M and the like.

As shown in FIGS. 14 and 15, a plurality of markers 26M are provided on one magnetic ring 26, a CCD camera 24 for detecting each marker 26M, and an electromagnet 25 for generating a magnetic field on each marker 26M side. If the configuration is made such that marker tracking and magnetic field generation are performed, more accurate tracking can be performed, and the reaction force applied to the magnetic ring 26 can be more finely adjusted and controlled. The reaction force to the magnetic ring 26 can be obtained by appropriately selecting the attraction or repulsion between the electromagnet 25 and the magnetic ring 26 as described above.

Each system described above is not applied only to the virtual reality space constituted by CG. The present invention is also applicable to a system for remotely controlling an actual object through a display device, for example, a micromanipulator installed and used on a microscope, a manipulator of a remotely controlled robot, and the like.

[0029]

As described above, according to the present invention, the operator only needs to attach the reaction force presenting member to a predetermined portion, so that the body is not restricted by a complicated mechanical configuration. In addition, since force feedback corresponding to the virtual object is performed in the magnetic field, it is possible to present a force sense of the virtual object with a high degree of freedom. Therefore, it is possible to achieve a higher sense of reality with a simple configuration.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a schematic configuration of a virtual object presentation system according to the present invention.

FIG. 2 is a diagram showing a use state of the embodiment of the present invention.

FIG. 3 is an exemplary sectional view showing a configuration of a main part of the embodiment of FIG. 2;

FIG. 4 is a sectional view showing the position of the magnetic tube in a state where the arm is separated from the virtual object in the embodiment of FIG. 2;

FIG. 5 is a sectional view showing the position of the magnetic tube when the arm is in contact with the virtual object in the embodiment of FIG. 2;

FIG. 6 is a diagram showing an example of a video in a virtual reality space.

FIG. 7 is a diagram showing a use state of another embodiment of the present invention.

FIG. 8 is a perspective view showing an external configuration of a main part of the embodiment of FIG. 7;

FIG. 9 is a perspective view showing the appearance of a magnetic ring, a sensor unit, and an articulated drive mechanism which constitute the embodiment of FIG. 7;

FIG. 10 is a diagram showing a use state of an embodiment using three magnetic rings.

FIG. 11 is a perspective view showing a state where 15 magnetic rings are mounted on a hand.

FIG. 12 is a diagram showing a use state of an embodiment using 15 magnetic rings.

FIG. 13 is a perspective view showing a pre-search box for checking a mounting state of a magnetic ring.

FIG. 14 is a diagram showing a use state of the embodiment for performing scissor-type tracking.

FIG. 15 is a diagram showing a use state of the embodiment in which the enclosure type tracking is performed.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... CPU (computing device) 2 ... Viewpoint detecting device 3 ... Video display device 4 ... Marker tracking device (tracking device) 5 ... Magnetic field generating device 6 ... Reaction force presenting member 11 ... Control unit (computing device) 13 ... HMD (video) Display device) 14: CCD camera (tracking device) 15a: Electromagnet (magnetic field generating device) 15b: Electromagnet (magnetic field generating device) 16: Magnetic tube (reaction presenting member) 16M: Marker 24: CCD camera (tracking device) 25 ... Electromagnet (magnetic field generator) 26 ... Magnetic ring (reaction presenting member) 26M ... Marker 28 ... Sensor unit 28A ... Sensor unit 29 ... Multi-joint drive mechanism V1 ... Virtual object V2 ... Virtual object P1 ... Arm P2 ... Hand P3 … Finger P4… wrist

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Takao Kobayashi Osaka International Building Minolta Co., Ltd. 2-3-13 Azuchicho, Chuo-ku, Osaka-shi

Claims (1)

[Claims] 1. A reaction force presenting member configured to be attachable to a predetermined part of a human body and made of a magnetic material, a tracking device for detecting a position of the reaction force presenting member, and the tracking device An operation for calculating a positional relationship between the virtual object and the predetermined part in the virtual reality space from the detected position of the reaction force presenting member, and calculating a reaction force applied to the reaction force presenting member based on the positional relationship. A virtual object presentation system, comprising: a device; and a magnetic field generation device that generates a magnetic field to apply a reaction force according to a calculation result of the arithmetic device to the reaction force presentation member.