WO1997019398A1

WO1997019398A1 - A method for presenting virtual reality enhanced with tactile stimuli, and a system for executing the method

Info

Publication number: WO1997019398A1
Application number: PCT/IB1996/001231
Authority: WO
Inventors: David Victor Keyson
Original assignee: Philips Electronics N.V.; Philips Norden Ab
Priority date: 1995-11-24
Filing date: 1996-11-15
Publication date: 1997-05-29
Also published as: JPH10513593A; EP0806002A1

Abstract

A method for enabling a user of a data processing system to navigate through a virtual reality environment, comprises generating a sequence of visual representations of the virtual reality environment on a visual display in response to the user manipulating a control device. Furthermore, the method provides via a feedback mechanism appertaining to the manipulated control device, a preprogrammed dynamic tactile stimulus from an available database with tactile stimuli to the user. Such is under selective and combined control by an actual interval of the sequence of generated visual representations and furthermore instantaneous mapping of a pointing direction of the control device on a predetermined image subfield.

Description

A method for presenting virtual reality enhanced with tactile stimuli, and a system for executing the method.

FIELD OF THE INVENTION

The invention relates to a method for enabling a user of a data processing system to navigate through a virtual reality environment, wherein the method comprises generating a sequence of visual representations of the virtual reality environment on a visual display in response to the user manipulating a control device. The invention also relates to a data processing system for executing the method. Further advantageous aspects are recited in dependent Claims.

BACKGROUND ART For some time, film makers have recognized the fact that the impression of a visual representation is enhanced when combined with stimuli for other sensory perceptions such as sound. Certain films have been presented in an entourage wherein the audience is seated on a platform that is made to move in synchronism with actions displayed on the screen. For example, presenting a film of a roller coaster ride is accompanied by the platform being tilted and shaken so as to give an audience a sensation of genuinely being on the ride, or presenting a film about an earthquake uses infra-sound to physically shake the audience. The audience has however no influence over the video information being shown nor over the tilting of the platform or over the shaking.

State of the art flight simulators use video information in combination with tilting of a platform accommodating a mock-up cock-pit with a flight deck. Here, the person in the cockpit guides through the flight deck's controls both the movement of the plat¬ form and the video information being displayed. The chosen path of flight determines the video information shown and the tilting angle determines the direction of gravity that in turn is felt as inertia by the person flying with the seat off his pants. It is clear that use, development and maintenance of these systems are very expensive and that they can be exploited commercially only for mass entertainment or for professional use.

US Patent 5,459,382 to Jacobus et al describes a method for providing tactile feedback in a virtual reality environment with six degrees of freedom. Dependent on the actual motion control effected by the user person on an interface device, first a visual re- ality force field generator is activated, which in its turn causes a force signal to be generated back to the interface device. Through the definition of a conservative force field, various ty¬ pes of force feedback can be emulated to the user, for so allowing a user to exercise operati¬ ons that later will have to be executed in real world. However, the reference does not allow exercising with unexpected, and in particular, dynamic disturbances that would make real-life operating so much more complicated. Moreover, the reference relates principally to simulating real-world, rather than a fully virtual reality such as is proper to entertainment systems.

SUMMARY OF THE INVENTION

Therefore, amongst other things it is an object of the invention to provide a low-cost method for enabling a user to experience an enhanced virtual reality, in particular whilst having to cope with such unexpected disturbances. It is a further object to provide such a method in a multimedia environment. Now, according to one of its aspects, the method according to the invention is characterized by providing via a feedback mechanism appertaining to the manipulated control device, a preprogrammed dynamic tactile stimulus from an available database with tactile stimuli, to the user under selective and combined control by an actual interval of the sequence of generated visual representations. The inventor has recognized that tactile stimuli, in particular dynamic stimuli, when added to visual, and possibly auditory, stimuli, can contribute significantly to the user's impression of being actually involved in a virtual reality scenery such as in video games. Video and/or graphics are used to create visual representations of a specific scenery from a variety of viewpoints. The visual representation is altered through the control device as if the user were moving through the scenery. The visual representation shown on the display is a mapping of the virtual scenery onto a plane selected through the control device. A tactile texture is joined to the video information. Similar to the visual representations, tactile representations are mappings of attributes, pre-assigned to events in the virtual reality scenery, onto the control device. When moving through the virtual environment shown on the display the user feels through the con- trol device a texture associated or other stimulus from a particular surface area or object in the virtual environment so as to intensify the impression of actually being there.

Control devices that provide tactile feedback have been disclosed in European Patent Application 489 469, corresponding US Patent Application serial No. 08/678, 115 (PHN 13,522) and non-prepublished European Patent Application 95200599.9, corresponding US Patent Application Serial No. 08/615,559 (PHN 15,232), all assigned to the present assignee. These applications disclose a trackball and a computer mouse whose revolving members are manipulated by a user, but are also driven by electric motors under software control so as to provide positive and negative torques. These devices are used to provide tactile stimuli for guiding a cursor through a labyrinth shown on the display. The positive and negative torques are experienced by the user as positive and negative reaction forces as if the revolving member's movements were constrained to the fixed path shown. The tactile feedback is related to specific absolute coordinates of the cursor on the screen.

The tactile stimulus in the present invention is generated under control of the occurrence of at least a specific one of the visual representations of the virtual scenery, in combination with an actual pointing direction of the control device on a predetermined image subfield. The occurrence of a specific pixel pattern or texture, a particular color, or a particular level of greyness is accompanied by one or more particular tactile stimuli. This integration of visual and tactile is utilized to let the user associate a tactile stimulus with an object, an area or an event in the virtual reality scenery. Alternatively or supplementarily, the tactile stimulus in the invention is generated depending on a rate at which certain visual representations succeed each other in the sequence. This combination is employed to let the user associate the tactile stimulus with a speed at which the user moves through the virtual reality scenery. For example, a video recording is made of a real scenery from a plurality of positions and from a plurality of viewing angles. The video data are used to provide the user with particular ones of a variety of visual representations of a virtual reality on a display when moving through the scenery. That is, the displayed image as a whole changes when the user moves, about as if perceived through a moving car's windscreen. The user is enabled to navigate through this virtual reality by controlling, for example, a mouse, trackball or joystick provided with features that provide force feedback. Tactile textures are associated with the visual representation. The visual representation of grass is combined with a slight amount of drag felt through the mouse or trackball when moving across a lawn. The impression of a path of cobblestones is evoked by a sequence of alternating pulling and pushing forces that change more rapidly the quicker one moves through the virtual reality scenery. A representation of a tree is felt as an impenetrable object. A lake is represented by a pit and a dynamic wave resistance pattern when moving. A resistance with directional characteristics gives the impression of going uphill or going in the teeth of a gale, etcetera. These tactile stimuli associated with objects or events in the virtual reality scenery are used in a video game, e.g. , wherein the user has to ride a (virtual) motorbike cross-country through a virtual landscape by controlling one or more control devices of the kind specified above. The landscape has different types of surface regarding texture (hardness, flatness) and extent, such as mud, asphalt, grass, loose sand, cobblestones, brushwood, frozen underground, and all kinds of obstacles such as trees, stone walls, hills, rivers, ditches, flock of sheep, turnpikes, etcetera. The relevant area of the landscape being displayed at a particular moment depends on the bike's actual direction and location with respect to the scenery. The surface of the area is tactilely represented by stimuli experienced by the user through the control device. The above tactile stimuli correspond to spatial characteristics, such as locations or orientations in the virtual scenery. One could also provide tactile stimuli with temporal characteristics or conditional tactile stimuli. In the example of the virtual motorbike, incidental gusts of wind, trying to throw the motorbike out of control, are represented by a short-lasting sudden sideways reaction force exerted by the control device on the user. A conditional tactile stimulus could be the occurrence of irregular and tiny shocks such as simulating tomatoes thrown by a (virtual) angry farmer when being passed by too close.

It is clear that tactile stimuli give a new dimension to virtual reality and considerably increase the sense of the user of actually being involved. The synergetic combination of video, audio and tactile stimuli in a software application considerably extends the concept of multimedia applications. Not only video games but also (semi-) professional application may benefit from tactile enhancement. For example, a computer model of a novel suspension for a motorcycle or a computer model of a novel steering geometry for a car may use the invention to provide a first impression of the behaviour of the suspension or steering in all kinds of terrain and at all kinds of speed.

BRIEF DESCRIPTION OF THE DRAWING

The invention is explained in further detail and by way of example with reference to the accompanying drawing wherein: Figure 1 is a block diagram of a system in the invention;

Figures 2-4 illustrate various events in an example of a software application that combines video and tactile stimuli;

Figure 5 is a first geometry diagram of the invention;

Figure 6 is a second geometry diagram of the invention. Throughout the drawing, same reference numerals indicate similar or corresponding features.

DETAILED EMBODIMENTS Figure 1 is a diagram of a data processing system 100 according to the invention. System 100 comprises a display 102, a control device 104 and a data processing apparatus 106. Control device 104 comprises, for example, one or more of the devices dis¬ closed in published European patent application 0 489 469. Manipulation of device 104 controls, via apparatus 106, an image or a sequence of images that are shown on display 102. Apparatus 106 supplies to device 104 control signals that generate stimuli for being tactilely sensed by the user. The signals have a predetermined relationship with the images actually shown on display 102.

Apparatus 106 supplies to display 102 video and/or graphics data that represents a virtual reality scenery. The visual representation of the scenery changes in response to manipulation of control device 104. Manipulation of device 104 gives the visual impression of travelling through the scenery. For example, successive images are interrelated through a perspective transformation or through a displacement transformation.

Apparatus 106 also supplies the control signals to evoke tactile feedback to the user through control device 104. The control signals are generated in synchronism with the movement through the virtual reality scenery or with the occurrence of some event in the virtual reality scenery, i.e. , the control signals are related to a succession of images. Apparatus 106 therefore is operative to combine video or graphics data with tactile data, the latter determining the generation of the control signals. The parallel supply of tactile data and video data is pre-determined in the sense that the generation of a particular image or of a sequence of particular images is accompanied by a particular tactile feedback. This is realized, for example, much in the same way as video or graphics data are accompanied by audio data in conventional video games. The user of the system in the invention is enabled to select the video data interactively and the corresponding tactile data are then generated automatically. For example, a more rapid sequence of images then also brings about a more rapid sequence of the associated tactile stimuli.

More specifically, system 100 comprises a first storage means 108 to store video data for generation of the visual representations under control of control device 104, and second storage means 110 to store tactile data for control of the generation of the tactile stimulus in a predetermined relationship with the video data. First and second storage means 108 and 110 are preferably physically integrated with one another, and the tactile data is preferably logically combined with the video data. In order to create smoothly moving pictures, the video data throughput must be correspondingly high, while the throughput of tactile data is lower than that of video data as tactile stimuli are typically low-frequency signals. The integration allows reading of the video data and the tactile data, which is logically combined with the video data, without substantially hampering the flow of video data by undesired, time-consuming context switching. One way to achieve this is, for example, merging tactile data with some video data records, identifying the tactile data as such upon reading, and supplying the tactile data to a look-up table 1 12. The tactile data are then just signals to control the generation of the tactile stimuli at control device 104. Preferably, look-up table 112 is user-programmable to enable selection or tuning of the tactile feedback. Preferably, storage means 108 and 1 10 are integrated with each other in a single information carrier 1 14 or cartridge such as a diskette or a CD. System 100 can then be used with different software applications by simply exchanging the carrier 1 14. Figures 2-4 give some examples of snap-shots of a video game of the kind mentioned above, wherein the user is supposed to play the role of a police officer on a motorcycle 200. The user interacts with the game through control device 104 that controls speed and direction of motorcycle 200 through the virtual reality scenery. Control device comprises, for example, a first trackball or joystick for directional control and a second trackball or joystick for speed control, both the speed controller and the directional controller being provided with force-feedback means. The officer is to chase a vehicle 202 driven by poachers 204, 206, 208 and 210. Poachers 204-210 are desperately trying to stay beyond reach of the strong arm of the law and drive their truck 202 wildly through the scenery. The scenery changes as the hunt continues. End of the game is when, for example, either the poachers escape or the user has come so close that the truck's license plate 212 can be read, i.e. , plate 212 is being displayed in legible characters. Preferably, the video representation of motorcycle 200 is made to lean with respect to the vertical and horizontal axes in the scenery in synchronism with a change of direction brought about via control device 104 to intensify the impression of the user riding motorcycle 200 himself. In Figure 2 motorcycle 200 is trying to catch truck 202 speeding along a bumpy path 204 of loose sand. It is hard to ride a motorcycle along a predetermined line through loose sand as the rear wheel tends to start wobbling, trying to throw the motorcycle off course, and as steering requires a considerable torque to be applied to the handlebars when changing directions. This is made to be felt at control device 104 by system 100 exerting alternating transversal forces on the user's hand depending on, e.g., speed and leaning angle. The user therefore has to concentrate very hard in order not to loose control over motorcycle 200 and, at the same time, not to loose sight of truck 202.

In Figure 3 motorcycle 200 is ridden on a grey autumn day, in a drizzle at dusk (dinner time), along a road 300 with bumps and potholes that are made to be felt as sudden shocks at control device 104, and fallen leaves that tend to gather where the road makes a bend, and that are made to be felt as a sudden loss of resistance at control device 104.

In Figure 4 motorcycle 200 is chasing truck 202 along a street. Poacher 210 is trying to hit the officer by throwing objects at him, such as, say, tomatoes, spare wheels and other things that are carried in the trunk of a truck for this purpose. So, in addi¬ tion to keep motorcycle 200 on course, the user has to avoid projectiles 402, 404 and 406 that otherwise may hit the windscreen of the motorcycle and block the user's vision, or, worse, throw motorcycle 200 off the track altogether. Note that the tactile stimuli introduced under Figures 2 and 3 relate to a spatial characteristic: the virtual texture of some areas in the scenery. The tactile stimulus in Figure 4 is made conditional in that it depends on the occurrence of the situation wherein certain events coincide: both the projectile 400 and motorcycle 200 are to occupy the same space in the virtual reality scenery at the same time in order to generate the tactile stimuli. In addition to spatial and conditional tactile stimuli, temporal tactile stimuli may be introduced. For example, the front tyre of motorcycle 200 may sustain damage as a result of which it is loosing pressure. Steering motorcycle 200 then becomes increasingly difficult, which is felt at control device 104 as wobbling reaction forces of increasing amplitude. Further, the pave¬ ment of the route through the virtual reality scenery can be made to feel increasingly slippery with time, e.g. , through reduction of reaction forces on the user manipulating control device 104.

Variations on this theme and refinements are manifold. Software applications integrating visual and tactile stimuli extend well beyond the above good-guy/bad- guy scenario. As alternatives to the latter scenario, consider the following: tactile stimuli can be used in a video game dealing with how to negotiate obstacles on different types of terrain in a cross-country run, steeple chase or voyage of discovery; tactile stimuli can be used to si¬ mulate the water waves in a video game relating to canoeing in wild water or to the mooring of a vessel to an oil-rig in a storm; in a video game concerning a low-level flight of a heli¬ copter over broken ground and among obstacles tactile cues may be used to transmit the o vibration, that in reality is inherent in this type of aircraft.

Alternatively, the tactile data to accompany the video data may be calculated in real-time when moving through the virtual reality scenery. This approach can be used, for example in a professional or semi-professional environment, when a computer model for a suspension system or a steering geometry of a vehicle is tested. Using the invention provides a first impression of the behaviour of the suspension in all kinds of terrain and at all kinds of speed. The developer is thus enabled to make a purposive selection of parameter values describing the best performing computer trials for real testing later on.

Above examples merely illustrate an enhanced form of virtual reality by adding tactile stimuli to video and audio information. Above examples are by no means intended to limit the scope of the present invention.

FURTHER IMPLEMENTATIONS OF THE TACTILE STIMULI

In creating and enhancing experiences within pseudo-worlds such as those which may appear in a video film, a number of tactual fields can be used in combination with auditory and visual images. For example, one could spatially navigate through linked vi¬ deo information, similar to moving through a virtual reality environment, and encounter various heard, seen and felt objects. Tactual representations for specific objects may extend beyond a representation of the object's physical surface plane. Dynamic fields can be created using tactual information to suggest a certain feeling or experience. They can also be used to guide the user in a suggested direction of movement. The fields may be displayed using force feedforward (active hand-independent device movement) and feedback through a device such as a 3-D trackball with force feedback. Furthermore two or more dynamic tactual fields can be combined to create a new experience. Sound effects (e.g. , from a video source) may also accompany tactual effects. Several examples of dynamic tactual fields are described below.

The "conveyor belt"

The conveyor belt acts like a walking platform that moves. When the user enters the range of the conveyor belt, the input device (e.g. trackball) begins to move and leads the user to a predesignated position. This movement can be based on using a constant force, which causes the input device to accelerate. Alternatively, the latter motion can be time-based. In the time-based mode, the time in which the user should be moved from point a to point b is specified. The amount of force on the device is adjusted in real-time to maintain the average velocity at a required value, given time and distance. A "bumpy road"

The bumpy road consists of two combined tactual fields, being a "road field" and a "texture field", respectively. The road guides the user along a predesignated course by providing force feedback walls. The user can also feel the width of the path. Unlike the conveyor belt, the device does not move on it's own, rather the user can passively follow a felt path. The texture can be changed at any point along the path by superposing a texture field on the road field. The texture field can be adjusted to provide "bump" sensations at a given intensity. The control-pixel distance between "bumps" can also be specified in the x and y directions. By using more than one texture field on a path, the texture can change as the user moves along the road. Sounds associated with moving over bumps have also been used in synchronization with the felt bumps to enhance the experience.

A "swirling effect"

A cyclone or swirling effect can be created to enhance the experience of falling into a "hole" . The user can see and hear effects and feel the device (e.g., ball) enter a circling-like mode. The effect can also be used to create vibrations using very tight swirls.

Conveyor belt - force based

ConvF.F.x = F * cos(alpha) ConvF.F.y = F * sin(alpha)

Here ConvF.F is the force vector, composed of two elements x and y, applied on the conveyor belt (force based) object, f (user-variable) is the force magnitude, and alpha (user-variable) is the angle over which the conveyor is rotated.

Conveyor belt - time based

ConvT.F.x = F * cos(alpha) ConvT.F.y = F * sin(alpha)

Here ConvT.F Is the force vector, composed of two elements x and y, applied in the conveyor belt (time based) object, alpha (user variable) is the angle over which the conveyor is rotated, and F = kp * errorP + kd * delta errorP + kl * integral errorP

Here kp, kd and ki are constants with typical values of 48, 16 and 0.01 respectively, and

errorP = length (DC - CP)

Here errorP is the difference between the desired cursor position DP at a particular moment and the current cursor position CP, and

DP = a - (elapsed_time)/(total_time) * (b-a)

Here a is the cursor position at the moment the conveyor belt started, b is the position the cursor should be moved towards, and total_time (user variable) is the amount of time the cursor movement should take, and

delta errorP ^*= errorP - (previous value of errorP)

Here integral errorP = sum of all calculated errorP values so far, and with errorP and integral_errorP held in the ranges [-25,25] and [-5000, 5000] respectively.

Road object

road.F.x = F * s * sin(alpha) road.F.y = F * s * (-cos(alpha))

Here road.F is the force vector, composed of two elements x and y, applied to the road object, F is the force magnitude, and alpha is the angle over which the road is rotated, and

1 , if the cursor is in section A s = 0, if the cursor is in section B

-1, if the cursor is in section C Texture object

texture.F.x = - w.x * F * k texture.F.y = - w.y * F * k

Here texture.F is the force vector applied on the texture object, composed of two elements x and y, and F is the force magnitude, and

- unit vector of v

Here v = (current cursor position) - (cursor position the last time k was equal to 1), and

k = 1, if abs(v.x) > Gx, or abs(v.y) > Gy, k = O, otherwise.

Here, Gx is the granularity of the texture in the X direction and Gy Is the granularity of the texture in the y direction.

Swirl

swirl.F.x = F * Cx * sin(2*pi * f/t) swirl.F.y = F * Cy * cos(2*pf * f/t)

Here swirl. F is the force vector, composed of two elements x and y, applied during the Swirl effect, Cl and C2 are constants representing the force magnitude in the x and y directions, respectively, f is the frequency of the swirl, t (O ≤t ≤ d) is the current time, d is the duration of the effect, and

F = ml + t/d * (m2-ml)

Here ml and m2 are contants (O ≤ml ≤ l , 0 < m2 ≤ l) representing the start and end points, respectively, of the slope function affecting the amplitude.

Claims

CLAIMS:

1. A method for enabling a user of a data processing system to navigate through a virtual reality environment, wherein the method comprises generating a sequence of visual representations of the virtual reality environment on a visual display in response to the user manipulating a control device, characterized by providing via a feedback mechanism appertaining to the manipulated control device, a preprogrammed dynamic tactile stimulus from an available database with tactile stimuli, to the user under selective and combined control by an actual interval of the sequence of generated visual representations.

2. A method as claimed in Claim 1 , furthermore comprising a step of instantaneous mapping of a pointing direction of the control device on a predetermined image subfield, and furthermore mapping said dynamic tactile stimulus on said subfield.

3. A method as claimed in Claims 1 or 2, wherein the dynamic tactile stimulus is selected from the database under control of at least one of the following: an occurrence of a specific visual representation of at least one specific object actually dis- played, or a rate of change of the visual representation of such at least one specific object.

4. A method as claimed in Claims 1 , 2, or 3, wherein said dynamic tactile stimulus emulates a physical interference with said user moving through said virtual reality environment.

5. A method as claimed in any of Claims 1 to 4, wherein said tactile stimulus emulates moving in a topographical or meteorological disturbance associated to said virtual reality environment.

6. A method as claimed in any of Claim s 1 to 5, wherein said tactile stimulus emulates an impact with an external virtual object in said virtual reality environment.

7. A method as claimed in any of Claims 1 to 6, wherein encounter with said tactile stimulus is mapped on a subfield of the visual display in congruence with a particular visual area representation of said subfield.

8. A method as claimed in Claims 7, wherein said image subfield is displayed with a particular texture, colour, or greyness level that is correlated to said tactile stimulus provided.

9. A method as claimed in any of Claims 1 to 8, wherein a selection of said tactile is accompanied by associated sound.

10. A method as claimed in any of Claims 1 to 9, and allowing to superpose two or more individual tactile stimuli.

11. A data processing system for enabling a user to navigate through a virtual reality environment scenery and comprising a display, and furthermore a control device for user-interactive generation of a sequence of visual representations of the scenery on the display, characterized in that the control device is operative to provide through at least one feedback mechanism of the control device to the user a preprogrammed dynamic tactile stimulus from a database repertoire of tactile stimuli under selective and combined control of an actual interval of the sequence of generated visual representations, and furthermore instantaneous mapping of a pointing direction of the control device on a predetermined image subfield.

12. A system as claimed in Claim 11, comprising first storage means to store image data for the generation of the visual representations under control of the control device, and second storage means to store tactile data for control of the generation of the tactile stimulus in a predetermined relationship with the video data.

13. A system as claimed in Claim 12, wherein the first and second storage means are physically integrated with one another, and wherein the tactile data is logically combined with the image data.

14. A system as claimed in Claims 12 or 13, and comprising a look-up table for control of the generation of the tactile stimulus in response to the tactile data.

15. A system as claimed in Claim 14, wherein the look-up table is user- programmable.

16. An information carrier comprising the first and second storage means for use with the system as claimed in Claims 13, 14 or 15.