JP2004054694A - Sense presentation device and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a simple device capable of presenting a feeling of presence by arranging actuators and a display so that the presentation of vision and hearing and the presentation of force/touch are integrally presented. <P>SOLUTION: A sense presentation device having a video display 12 for presenting a picture of an object and capable of presenting senses for virtually operating the object displayed on the video display 12 to an operator comprises a presentation part 11 to be driven on an approximate plane by two or more actuators (3, 5, 7, 9) and a touch presentation part 20 arranged on the presentation part 11 and capable of presenting multi-dimensional touch to the operator and the video display 12 for displaying the object is arranged on the lower face of the presentation part 11. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
[Field of the Invention]
The present invention relates to a human interface device in the field of virtual reality, and more particularly to a sensation presentation device and a sensation presentation method for virtually presenting a sensation and operating a virtual object.
[0002]
[Prior art]
In recent years, various human interface devices using virtual reality technology have been developed. These allow machines to be handled in a more natural way for humans, whereas humans had to adjust to machines in the past. These devices virtually create a sense of reality with respect to the five senses of human beings, so that information can be exchanged interactively with machines, so that more complex and sophisticated operations can be easily performed. It is also expected to be an operating device that replaces a keyboard and a mouse because it makes it easier for children, the elderly and the physically handicapped to operate a computer. Such a human interface device is considered to be used as an input device for CAD or modeling, as an operation device of a remote work robot, or as a medical / welfare device.
As the visual presenting device, there are an immersive display and an open display, and as the immersive display, many head mounted displays (HMDs) have been developed. In addition, as the open type display, there are a normal CRT monitor, a stereoscopic monitor, and the like. As a force sense and tactile sense presentation device, there is a “force sense presentation device” (FIG. 7) disclosed in Japanese Patent Application Laid-Open No. Hei 6-324622, which presents a force sense to a fingertip using an actuator and a wire. The two-dimensional force sense presentation device (FIG. 9) disclosed in WO98 / 08159 has a drive system in the X direction connected to a drive system in the Y direction, and a mouse-like drive system in the X direction. Are connected. When presenting a force sensation, a reaction force is presented to the grip portion by the driving force of the rotary motor.
[0003]
[Problems to be solved by the invention]
In such a human interface device, it is important that the human interface device can be operated with a more realistic sense of human sensation such as sight, hearing, force, and touch. At this time, different types of sensations such as sight and force sense or sight and tactile sense must be fused and presented without contradiction. However, in the devices shown in FIGS. 7 and 9, if the visual sense is a visual only device, and if the haptic force is a haptic only device, the sensation is divided into a sight and a haptic sense or a sight and a haptic sense, or a haptic and a haptic sense. There was no sensation presentation device. In the apparatus disclosed in Japanese Patent Application Laid-Open No. Hei 9-114360 (FIG. 8), by attaching a small display to the tip of the manipulator, it is possible to obtain a feeling as if the user were actually operating an object projected on the display. However, there is a problem that the user only feels the weight of the object and lacks a sense of reality because he has the display itself.
In some cases, disparate sensations can be integrated by software even with devices having different sights and haptics. When an immersive display (HMD) is used as a visual presenting device, a sense of reality is realized by matching the position of a pointer representing a displayed operating point with the position of an actual hand using a position sensor. There is a problem that the system is complicated because it is worn on the head and the system becomes complicated.
[0004]
On the other hand, when an open display (monitor) as shown in FIG. 9 is used, the system is simple. However, since the visual and haptic devices are physically separated from each other, the displayed pointer and the real hand are used. Positions could not be matched, and as described above, it was impossible to achieve a more realistic fusion of vision, force, and touch. In view of the above problems, the present invention provides a simple and highly realistic device by arranging an actuator and a display so as to present visual and auditory presentation and force / tactile presentation in an integrated manner. is there.
[0005]
[Means for Solving the Problems]
The sensation presentation device according to claim 1, further comprising a video display for presenting an image of the object, and a sensation presentation device for presenting to the operator a sensation of virtually operating the object displayed on the video display. A presentation unit driven on a substantially flat surface by two or more actuators, and a tactile presentation unit installed on the presentation unit to present a multidimensional tactile sense to the operator, and a lower surface of the presentation unit The image display for displaying an object is arranged.
[0006]
A sensation presentation device according to a second aspect of the present invention is characterized in that when the operator operates the object, a reaction force corresponding to the operation is presented by the presentation unit.
[0007]
A sensation presentation device according to a third aspect of the present invention is characterized in that the presentation unit and the actuator are connected by a wire or a belt.
[0008]
A sensation presentation device according to a fourth aspect of the present invention is characterized in that a hearing presentation means for presenting a hearing to the operator is provided.
[0009]
According to the sensation presentation device according to any one of claims 1 to 4, visual and auditory senses, force senses, and tactile senses are integrated, and a virtual object projected on a display can be operated with a higher sense of reality. Further, according to the sensation presentation device of the third aspect, the device can be made thin and compact, and the maximum reaction force can be increased, so that a smooth operation can be realized.
[0010]
According to a fifth aspect of the present invention, in the sensation presentation apparatus, the reaction force generation unit that generates a reaction force in the presentation unit includes an amplifier that generates a thrust command to the actuator and an I / O unit that inputs and outputs a signal. And a communication unit that performs data communication between the control unit and the object processing unit, the object processing unit performing simulation and drawing with respect to the deformation and movement of the virtual object. It is. According to the fifth aspect, the sensation presentation device according to the first to fourth aspects can be operated while presenting an effective presence.
[0011]
The sensation presentation method according to claim 6 of the present invention is a sensation having a video display for presenting an image of an object and a presentation unit for presenting to the operator a sensation of virtually operating the object displayed on the video display. In the presentation method, image processing is performed on a target object using image data of a real video taken by a camera, virtual model data is created based on the image processing, and a real image is displayed on the video display. A reaction force is presented based on the virtual model data.
[0012]
The sensation presentation method according to claim 7 of the present invention is characterized in that the virtual model data is physical properties such as shape, material, and hardness of one or more objects to be operated. .
[0013]
According to claims 6 and 7, even a real image such as a photograph can be virtually touched.
[0014]
The sensation presentation method according to claim 8 of the present invention is such that sound effect data is associated in advance in accordance with the physical characteristics of the virtual model data, contact is determined from the position of the presentation unit and the position of the object, and contact is determined. When the sound is detected, the sound effect is reproduced or generated, and is presented by the auditory presentation means, whereby the sense of presence can be further enhanced.
[0015]
The sensation presentation method according to claim 9 of the present invention is characterized in that the virtual model data is downloaded through a network. According to this, it is possible to virtually touch an image on a device connected via a network, such as on a homepage of the Internet.
[0016]
BEST MODE FOR CARRYING OUT THE INVENTION
First Embodiment A sensation presentation apparatus according to a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a schematic view of the sensory presentation device of the present invention as viewed from above. In this sensation presentation device, the presentation unit 11 is arranged on a two-dimensional plane by the linear motor stators 2 and 4 and the movers 3 and 5 in the X-axis direction and by the linear motor stators 6 and 8 and the movers 7 and 9 in the Y-axis direction. Drive with However, the linear motor is actually concealed by the cover 1 and cannot be seen from the outside. The operation in the Y-axis direction is performed along the linear guide 10, but is wire-driven in the Y-axis direction as described later. The reason why two linear motors are provided for each of the X axis and the Y axis is to obtain a larger thrust. Furthermore, by using a linear motor and wires, the Y-axis stators 6, 8 and the movers 7, 9 do not need to be arranged on the X-axis linear motor, so that the X-axis motor capacity can be reduced, and The inertia of the presentation section 11 in the X-axis direction can be reduced. This also makes the device itself thin and compact. Furthermore, since the linear motor is a direct drive that does not use gears, a smooth operation can be obtained at the same time as a large thrust and compactness as described above.
[0017]
Further, the liquid crystal display 12 is disposed on a plane below the linear guide 10. With such an arrangement, vision and force sense or tactile sense are naturally fused. In other words, the operation point of the virtual object seen by the eye matches the position of the actual hand of the user, and it is possible to obtain a feeling of touching or tracing the virtual object projected on the screen as it is. By matching the sense of sight and the somatic sense (sense of the position of one's hand) in this way, the operability and the sense of reality are significantly improved. Although not shown, one linear encoder is attached to one linear motor, and the linear motor is controlled using information of the encoder. With the linear motor and the encoder, it is possible to operate the virtual object projected on the liquid crystal display 12 by pressing or deforming the virtual object, and to obtain a texture such as resistance when pressed and softness / hardness of the virtual object. . Here, the actuator is described as a single-axis linear motor, but is not limited to this, and may be, for example, a linear motor such as a so-called surface motor that can operate in two axial directions with one mover.
[0018]
The presentation unit 11 is a part where the operator feels a force sense or a tactile sense by putting a fingertip. The presenting unit 11 presents not only a force sense presented by a biaxial linear motor, but also a vertical reaction force, and a fine texture or the like. It presents a tactile sensation. The presenting unit 11 is provided with a tactile presenting unit 20, and can present a tactile sensation. This part can take various forms depending on the application. For example, as a tactile sensation, a vibrating contact as described in Japanese Patent Application Laid-Open No. H7-77944 is driven by a piezoelectric actuator or the like, or a liquid as disclosed in Japanese Patent Application Laid-Open No. H12-112329 is divided. Alternatively, those injected into vesicles may be used. Further, instead of actively providing a tactile sensation to a fingertip using an actuator, a tactile sensation may be provided passively by a mechanism such as a spring. In this way, adding the tactile presentation mechanism on top of the haptic presentation mechanism not only pushes and deforms the virtual object projected on the liquid crystal display, but also combines the haptic and tactile sense by reproducing the tactile sensation at the fingertip. In addition, the method of presenting and expressing senses becomes richer, and the sense of presence is improved.
[0019]
FIG. 2 is a cross-sectional view of the sensation presentation device of the present invention as viewed in the X-axis direction. Here, reference numeral 13 denotes a support portion that supports the presentation portion 11 and moves on the linear guide 10. The support 13 is driven by a wire 14 supported by a pulley 15. Reference numeral 16 denotes a guide directly connected to the linear motors 2, 3 and 4, 5 in the X-axis direction. The running unit 17 is moved on the guide 16 by the linear motors 6, 7, 8, and 9, and the presentation unit 11 is moved through a wire. Is driven. The liquid crystal display 12 is disposed between the support unit 13 and the traveling unit 17. Reference numerals 18 and 19 denote guides for a linear motor in the X-axis direction. Although not shown, an encoder used to measure the position of the motor and control the motor is arranged beside the guides 18 and 19. . By performing the wire driving in this manner, the motor is not disposed on the liquid crystal display 12, so that the concealment of the liquid crystal display 12 is minimized. Further, since the load at the time of the X-axis operation is small, the operation can be performed lightly. Further, since there are no motors above and below the liquid crystal display 12, there is no influence from the electromagnetic force from the motor.
[0020]
FIG. 3 is a system block diagram of this embodiment. FIG. 3 is a diagram showing a configuration of a reaction force generating means for actually presenting a sensation in the sensation presentation device described with reference to FIGS. Become. In the computer 200, the position information of the presentation unit 11 of the sensation presentation device is taken into the object processing unit 201 through the communication unit 202. The position information is directly converted into the position of the operation pointer, and a contact determination is made as to whether or not the operation pointer and the virtual object drawn in advance on the video display are interfering. Further, it is assumed that physical properties of the virtual object, such as mass, spring characteristics, viscous friction characteristics, and surface roughness of the virtual object, are determined in advance. Next, when the virtual object comes into contact with the operation pointer, a simulation calculation is performed on how the virtual object deforms and moves according to the physical characteristics of the virtual object and the position and speed of the operation pointer. Further, the result is subjected to drawing calculation as computer graphics by the object processing unit 201 and displayed on the liquid crystal display 12 through the video card 203.
[0021]
Here, the operation pointer always coincides with the position of the presented presentation unit 11. In this way, the point touching the projected virtual object always coincides with the position of the actual fingertip, so that the visual and somatic sensations match, and it is as if the virtual object is actually directly operated. A very high sense of reality can be obtained. Further, since the finger is placed on the video display, it is not necessary to display the operation pointer representing the operation point on the display.
The communication unit 202 communicates with the control unit 100. The communication is, for example, high-speed serial communication such as USB. Since real-time properties are required to give a sense of reality, it is desirable that communication between the computer 200 and the control unit 100 be performed at high speed. Through this high-speed communication unit, the computer 200 transmits the motion information based on the calculation result of the deformation / motion simulation to the control unit 100, and fetches the position information of the sense presenting unit from the control unit 100 as described above.
[0022]
In the control unit 100, encoder position information is taken into the CPU 101 via an I / O 102 from an encoder of the apparatus main body. The CPU 101 controls the linear motors 3, 5, 7, and 9 based on the motion information from the computer 200, and outputs a thrust command to the amplifiers 103 and 104. The tactile actuator 30 of the presentation unit 11 is also processed by the control unit 100 in the same manner as described above, and outputs a thrust command to the amplifier 105.
Further, by attaching the speaker 31 or the like to the presentation device and generating an auditory signal in response to a command from the object processing unit 201 and generating a sound using the sound card 204, it is possible to improve the sense of presence.
[0023]
Here, an example in which the control unit 100 and the computer 200 operate with separate CPUs has been described. However, the present invention is not limited to this configuration. For example, a configuration in which one CPU performs all of the above processing may be used.
[0024]
FIG. 4 shows a software functional block diagram of the above processing contents. The model data calling unit of the object processing unit 201 performs a process of reading the shape and physical characteristics of the virtual object from the storage or the like into the internal memory. The contact determination unit determines whether interference has occurred based on a comparison between the current position of the presentation unit and the shape data in the model data, or calculates the amount of sinking into the virtual object. The deformation / motion simulation unit calculates the amount of deformation and the amount of motion of the object according to the physical characteristics and the degree of interference set for the virtual object, and simultaneously calculates the reaction force generated at that time. For example, when the hardness of the surface of the virtual object is considered by a spring / damper model, the reaction force is as follows.
F = KΔX + DdΔX / dt
Here, F is the reaction force, K is the spring constant, D is the viscosity coefficient, and ΔX is the amount of intrusion from the virtual object surface into the object calculated by the contact determination unit. The reaction force data calculated here is sent to the presentation unit drive calculation unit of the CPU 101. The presenting unit drive calculating unit generates a thrust command to a linear motor amplifier for driving the presenting unit based on the reaction force data from the object processing unit (including the CPU) 201 and outputs the thrust instruction to the amplifier. In this way, force sense presentation based on model data can be realized.
[0025]
Further, the teaching data reproducing unit calls an operation pattern stored in advance from the memory, and sends the operation pattern to the presentation unit drive calculation unit. When reproducing the teaching data, for example, if the operation pattern is a position command, the presenting unit drive calculation unit calculates a thrust command obtained by multiplying the position deviation and the speed deviation by a gain.
The auditory data generation unit of the object processing unit 201 selects or generates a sound effect according to the amount of deformation and movement calculated by the deformation and movement simulation unit and the set material, and outputs the sound effect to the speaker. And perform a drawing operation and output it to the display.
[0026]
A second embodiment will be described with reference to FIG. By using a pen-shaped object as shown in FIG. 5 as the presentation unit, it is possible to realize a feeling of operating a virtual object with a pen, a brush, or the like. The pen-shaped grip portion 20 is fixed to the presentation portion 11 'by a joint 21. The presenting unit 11 ′ is fixed to the presenting unit support base 13 ′ by a spring 22, and the presenting unit 11 ′ moves up and down when the pen-shaped gripping unit 20 is pressed down. The pressure at the time of pressing is measured by the pressure sensor 23. This can be used as an educational device for calligraphy and hard writing. The following example illustrates the case of calligraphy.
[0027]
First, the relationship between the pressure detected by the pressure sensor 21 and the thickness of the line drawn on the video display 12 is determined in advance for each selectable brush thickness. Then, the teacher grips the pen-shaped gripping portion 20 to perform a model brush stroke, and draws a character on the video display 12 with a line thickness corresponding to the pressure of the brush. The locus of the brush and the thickness of the line at that time are stored in a storage unit (not shown) of the computer 200 in FIG. When the students practice, the brush stroke is performed in the same manner as described above, but the teacher's characters stored at that time are projected on the video display 12, and the stroke is performed by tracing the teacher's characters. . At this time, it is also possible to assist the student's writing by playing back the trajectory of the teacher stored above using the linear motor shown in FIG. 1 as if the teacher wrote with the student's hand. . As described above, the reproducing means reads the teaching data by the teaching data reproducing unit of the CPU 101, and reproduces the locus of the teaching data by the presenting unit drive calculating unit.
[0028]
At this time, since the user traces right above the liquid crystal display 12 shown in FIG. 1, there is a sense of presence as if he were practicing calligraphy or hard brush on paper, and effective learning is possible. Here, a passive spring has been described as an example that presents a tactile sensation, but by actively controlling this using an actuator, it is possible to sense the pen pressure of the teacher finely.
Such a learning method is effective when the teacher cannot provide sufficient guidance to each student or when the student learns by himself when there is no teacher at the place.
[0029]
A third embodiment will be described with reference to FIG. First, an image photographed by a digital camera or digital still image data taken into a personal computer by a scanner is prepared. Thus, the image processing unit 301 performs image processing such as binarization and edge detection. The image processing result is used by the model generation unit 302 to generate contour shape data of the target object. This is the shape model of the virtual object. The physical characteristic setting means sets the physical characteristics of the entire contour shape or a certain range. The physical characteristics referred to here include a spring characteristic, a viscous friction characteristic of the virtual object surface, a roughness of the virtual object surface, and the like. As described above, the shape model and the physical characteristics corresponding to the shape model are stored in the storage unit as model data of the virtual object. Then, force sense presentation is performed based on the model data by the method described with reference to FIG.
A photograph of a human face will be described as an example of a digital image. As described above, after binarizing a photograph of a person, performing edge detection and noise removal, data interpolation of discontinuous points is performed, and face contour shape data is generated. At the same time, physical characteristics such as the softness of the face surface are set and used as model data of the virtual object. Then, the real image of the photograph of the face of the person is displayed on the display of the sensation presentation device as shown in FIG. 1, and the presentation of force sense based on the model data is performed by the method of FIG. This makes it possible to present a feeling as if the person's face is being touched with a fingertip. In addition, since there is a presenting portion for presenting a reaction force on the display, the viewpoint and the fingertip position match, and it is possible to enjoy an extremely high sense of presence. In addition, by providing a tactile sensation presentation mechanism that presents irregularities on the presenting unit, it is possible to feel irregularities on the face.
[0030]
Further, the same can be performed for an image on a homepage in an Internet environment. The image on the homepage is linked to the model data created in advance by the method of FIG. 6 so that the user can download the model data. Therefore, force sense presentation is performed on the downloaded model data in the same manner as described above. This means that, for example, it is possible not only to see the image of a product in Internet shopping or the like, but also to virtually touch it. Further, by associating one piece of sound effect data with the material information of the model data and reproducing the sound effect when a contact is detected, and presenting the sound effect through a speaker, the sense of presence is further improved. By checking the product by touching it, it is possible for the customer to purchase with more confidence, and for the shop, it is possible to excite the customer with the willingness to purchase.
[0031]
【The invention's effect】
According to the first to fifth aspects of the present invention, visual and auditory senses, force senses, and tactile senses are combined, and a virtual object projected on a display can be operated with a higher sense of reality. In particular, by simultaneously generating a force sensation and a tactile sensation, various expression methods of the virtual object become possible, and the sense of presence is improved. Further, by disposing the display at the lower part of the presentation unit and matching the position of the drawn operation pointer with the presentation unit, it is possible to match the sense of sight and the somatic sensation, thereby improving the sense of reality. Furthermore, even if a liquid crystal display is arranged by a linear motor and a wire mechanism, a thin, compact and simple device can be realized.
According to the sixth and seventh aspects of the present invention, even a real image such as a photograph can be virtually touched. According to the eighth aspect, the sense of presence is further improved by the sound effect when touched. Further, according to the ninth aspect, it is possible to virtually touch an image on a homepage on the Internet.
[Brief description of the drawings]
1 is a schematic view of a first embodiment of the present invention; FIG. 2 is a cross-sectional view of a mechanism of the first embodiment of the present invention; FIG. 3 is a system block diagram of the first embodiment of the present invention; FIG. 5 is a functional block diagram of the first embodiment of the present invention. FIG. 5 is a diagram showing a presentation section of the second embodiment of the present invention. FIG. 6 is a diagram showing a model generation process of the third embodiment of the present invention. 7 is a diagram illustrating an example of a conventional force sense presentation device. FIG. 8 is a diagram illustrating an example of a conventional force sense and vision presenting device. FIG. 9 is an example of a conventional force sense and vision presenting device. [Explanation of symbols]
2, 4: X-axis direction motor stator 3, 5: X-axis direction motor mover 6, 8: Y-axis direction motor stator 7, 9: Y-axis direction motor mover 11: presenting unit 12: Display 14: Wire 20: Tactile presentation unit 31: Auditory presentation unit 103, 104: Amplifier 100: Control unit 102: I / O unit 201: Object processing unit 202: Communication unit

Claims (9)

A sensation presentation device having a video display for presenting an image of an object, and presenting to the operator a feeling of virtually operating the object displayed on the video display,
A presentation unit driven on a substantially plane by two or more actuators;
A sensation presentation device, comprising: a tactile presentation section provided on the presentation section for presenting a multidimensional tactile sensation to the operator, wherein the video display for displaying the object is arranged on a lower surface of the presentation section. 2. The sensation presentation device according to claim 1, wherein when the operator operates the object, a reaction force corresponding to the operation is presented by the presentation unit. 3. The sensation presentation device according to claim 1, wherein the presentation unit and the actuator are connected by a wire or a belt. 4. The sensation presentation device according to claim 1, further comprising a hearing presentation means for presenting a hearing to the operator. A control unit including an amplifier for generating a thrust command to the actuator and an I / O unit for inputting and outputting a signal;
For the deformation and movement of the virtual object, comprising an object processing unit that performs simulation and drawing,
The sensory presentation device according to claim 1, further comprising a communication unit configured to perform data communication between the control unit and the object processing unit. A video display for presenting an image of an object, and a sensation presentation method having a presentation unit for presenting to the operator a sensation of virtually operating the object displayed on the video display,
Perform image processing on the target object with the image data of the actual video taken by the camera,
Create virtual model data based on the image processing,
The video display displays a real image,
A sensation presentation method for presenting a reaction force based on the virtual model data. 7. The sensation presentation method according to claim 6, wherein the virtual model data is physical properties such as a shape, a material, and a hardness of one or a plurality of objects to be operated. In advance, sound effect data is associated in accordance with the physical characteristics of the virtual model data,
Determine contact from the position of the presentation unit and the position of the object,
Play or generate the sound effect when contact is detected,
8. The sensation presentation method according to claim 6, wherein the presentation is performed by an auditory presentation unit. 9. The sensation presentation method according to claim 6, wherein the virtual model data is downloaded through a network.

Images