EP3595787A1

EP3595787A1 - Method for generating multimedia data associated with a system for practicing sports

Info

Publication number: EP3595787A1
Application number: EP17717240.0A
Authority: EP
Inventors: Olivier ZITVOGEL; Konstantin PRINZ
Original assignee: Holodia AG
Current assignee: Holodia Inc
Priority date: 2017-03-13
Filing date: 2017-03-13
Publication date: 2020-01-22
Also published as: US11511176B2; US20230077369A1; US20200269122A1; WO2018167532A1

Abstract

The invention proposes reducing motion sickness for users of mechanical systems used to practice sports that comprise an immersive virtual reality device. In particular, the invention is based on the use of prediction and interpolation algorithms that enable fluid movements to be displayed within the virtual environment.

Description

METHOD FOR GENERATING MULTIMEDIA DATA ASSOCIATED WITH A SPORT SYSTEM

The present invention relates to the field of mechanical sports systems comprising a virtual reality immersion device and more particularly methods for reducing motion sickness during the use of such a system. Mechanical sports systems are systems that allow a user to practice an indoor sports activity by generally simulating real sports activity on a controlled device. Examples of such systems include indoor bicycles, elliptical bicycles, indoor rowers, treadmills, etc. These systems typically consist of the sports device itself which allows the user to produce an effort. This is typically the bicycle, the rower etc. This sporting device is coupled to an adjustable resistance that allows to control the level of effort to be provided by the user during the use of the sports device. This resistance can take a multitude of forms, it may be for example an inertial wheel slowed by friction or an adjustable magnetic field or blades immersed in water whose movement slows the movement of the sports device as in some indoor rowers.

Today, the resistance setting is usually controlled by a computer controlling the activity. This control computer allows the display of different activity programs and the display of a man-machine interface to choose his program and thus to act on the level of resistance. The functions of the control computer are supplemented by functions for monitoring and establishing statistics on the activity sessions, the user's cardiac monitoring, etc.

Virtual Reality is an immersive, visual, sound interactive computer simulation of real or imaginary environments. The purpose of virtual reality is to allow a person (or several) sensory-motor and cognitive activity in an artificial world, created numerically, which can be imaginary, symbolic or a simulation of certain aspects of the real world. Coupled with a mechanical sports system, virtual reality can simulate sport in a virtual environment that can, for example, simulate outdoor practice. Thus, the practice is made more fun and enjoyable than a practice in static indoor environment.

Despite the many possible applications, the use of virtual reality is not yet widespread and attempts to commercialize a product based on virtual reality have encountered a recurring problem: The motion sickness or disease of virtual reality.

Motion sickness (cybersickness) occurs when exposure to a virtual environment causes symptoms that are similar to the symptoms of motion sickness. This syndrome is the source of dizziness, headache, loss of orientation, nausea, etc. It can occur during the virtual experience, but also during the return to reality and can persist for hours.

Motion sickness is different from motion sickness in that it can be caused by the visual perception of movement without the need for a real movement of the user. Motion sickness is the critical point that limits the launch of commercial applications based on virtual reality.

The present invention aims to solve the aforementioned drawbacks and more particularly to reduce motion sickness in the user of a mechanical sports system comprising a virtual reality immersion device. The invention is based on the use of prediction and interpolation algorithms for displaying a fluid movement within the virtual environment.

The invention relates to a method for generating multimedia data by a virtual environment generator associated with a mechanical sports system and intended for a virtual reality headset, characterized in that it comprises: a step of receiving kinetic samples at a first regular or irregular frequency, these kinetic samples being representative of a position of the user in an estimated displacement resulting from the use of the mechanical sports system during a session of 'exercise ;

a step of determining a camera position within a virtual environment, said position representing the position of said user in the virtual environment, said determination being made at a second regular frequency, called the display frequency, greater than the first frequency and interpolation of the position of the user received;

a step of generating the multimedia data corresponding to the position of the camera at the display frequency.

According to a particular embodiment, the step of determining the camera position comprises:

a calculation step at the first frequency of an instantaneous speed corresponding to the speed of the user between the last received sample and the previous one.

According to a particular embodiment, the method furthermore comprises:

a step of determining an instantaneous speed adjusted from the instantaneous speed, the position of the simulated user in the virtual environment and the position received in the last sample, this step being done at the first frequency.

According to a particular embodiment, the method furthermore comprises:

a step of determining the position of the camera at the display frequency according to the previous position of the camera and the instantaneous speed adjusted.

According to a particular embodiment, the method further comprises: a step of determining the stopping of the displacement when a time elapsed without receiving a new kinetic sample is greater than a threshold depending on the average time interval between two kinetic samples.

According to a particular embodiment, the method further comprises:

a step of retranscribing the movements of the body made by the user to the movements of the camera in the virtual environment.

According to a particular embodiment, the mechanical sports system being an indoor rower, the method further comprises:

a step of inversion of the direction of movement.

The invention also relates to a sports system comprising:

- a mechanical sports system;

a virtual environment generator associated with the mechanical sports system; and

a virtual reality headset connected to the virtual environment generator;

characterized in that the virtual environment generator comprises:

means for receiving kinetic samples at a first regular or irregular frequency, these kinetic samples being representative of a position of the user in an estimated displacement resulting from the use of the mechanical sports system during a session of 'exercise ;

means for determining a camera position within a virtual environment, said position representing the position of said user in the virtual environment, said determination being made at a second regular frequency, called the display frequency, greater than the first frequency and interpolation of the position of the user received; means for generating the multimedia data corresponding to the position of the camera at the display frequency.

According to a particular embodiment, the virtual reality headset has an OLED display.

According to a particular embodiment, the virtual reality headset has a field of view of less than 180 degrees. According to a particular embodiment, the system comprises a hardware setting for adjusting the interpupillary distance.

The invention also relates to a computer program comprising instructions adapted to the implementation of each of the steps of the method according to the invention when said program is executed on a computer.

The invention also relates to an information storage means, removable or not, partially or completely readable by a computer or a microprocessor comprising code instructions of a computer program for executing each of the steps of the method according to the invention. the invention.

In a particular embodiment, steps of the above method are determined by computer program instructions. Consequently, the invention also relates to a computer program on an information medium, this program being capable of being implemented by a microprocessor, this program comprising instructions adapted to the implementation of the steps of the method such as than mentioned above. This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

The invention also relates to a microprocessor-readable information medium, comprising instructions of a computer program as mentioned above.

The information carrier may be any entity or device capable of storing the program. For example, the medium may comprise storage means, such as a ROM, for example a microcircuit ROM, or a magnetic recording means, for example a hard disk, or a flash memory.

On the other hand, the information medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means. The program according to the invention may in particular be downloaded to a storage platform of an Internet type network.

Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question.

The aforementioned information carrier and computer program have characteristics and advantages similar to the method they implement. Other features and advantages of the invention will become apparent in the following description in relation to the accompanying drawings, given as non-limiting examples:

FIG. 1 illustrates an example of a sports system according to the invention;

FIG. 2 illustrates the functional relationships between the various components of the system according to an exemplary embodiment of the invention;

FIG. 3 illustrates the method of generating the multimedia data in an exemplary embodiment of the invention; FIG. 4 illustrates the interpolation method according to an exemplary embodiment of the invention;

Figure 5 is a schematic block diagram of an information processing device for implementing one or more embodiments of the invention.

Motion sickness is triggered in virtual reality users for several different reasons, some techniques, others due to the functioning of the human body. Among the predominant causes that generate motion sickness when using immersive virtual reality we can cite the following causes:

The functional discord between visual experience and motor activity. This first cause is one of the main causes and occurs when the virtual environment does not react in a manner consistent with the actual movements of the user of the mechanical sports system. This occurs, for example, when there is too much latency between a user's movement and the display of the consequence of that movement in the virtual environment.

Another cause of motion sickness is a blur or saccade that results in an inconsistency between the virtual experience and the vision in the real world.

Motion sickness also occurs when the displayed virtual image does not respect the field of view of the human eye. The use of wider or smaller angles of view leads to an inconsistency for the brain.

Postural instability during the virtual reality experience can occur when the position of the body that naturally tends to readjust its position to become upright is not consistent with the virtual environment. In this case, there is an inconsistency between the perceived visual information and the information of the user's inner ear. This postural instability is generating motion sickness. Individuals have a variable interpupillary distance around 62mm. This distance may vary slightly among individuals. The result of this variation is a feeling of vagueness with respect to the perceived image and a rejection of the virtual experience by the user generating motion sickness.

Finally, when the transition is too fast between the virtual world and the real world, users report experiencing the effects of nausea and dizziness typical of motion sickness. Figure 1 illustrates an example of a sports system according to the invention. A user 1.1 uses an indoor rowing machine 1.2. This rower 1.2 comprises a kinetic actuator 1.3 which allows it to print a movement typically by pulling on actuator connected by a ribbon to a kinetic sensor 1.4. The sensor opposes resistance to movement controlled by a control computer 1.5. This control computer besides the control of the resistance of the apparatus records the parameters of the movement, tensile force, number of strokes per minute, extension of the draft and others transmitted by the kinetic sensor 1.4. From these parameters, it calculates statistical information to the user and displays them on the control screen included in the control computer 1.5. In the context of the invention, the mechanical sports system also comprises a virtual environment generator 1.6. Since the user is equipped with a vision helmet 1.7, the virtual environment generator 1.6 obtains from the control computer 1.5 a set of parameters on the displacement generated by the action of the user. For example, it is possible to obtain the position of the user on a regular basis as well as the time elapsed since the beginning of the session. A session being defined as the period of time between user activation of the system and shutdown. These displacement parameters can be obtained either at the request of the virtual environment generator or issued more or less regularly by the control computer according to the models of mechanical sports systems.

The control computer and the virtual environment generator are two specifically programmed information processing devices for the function to perform. They are typically devices such as that illustrated in FIG. 5 and described later. Those skilled in the art also understand that in some embodiments, the control computer and virtual environment generator functions can be performed by a single information processing device without changing their mode of operation.

This is just one example, an apartment bike, an elliptical trainer or even a treadmill could be used in the same way. Figure 2 illustrates the functional relationships between the various components of the system according to an exemplary embodiment of the invention.

The kinetic actuator 1.3 allows the user to print the motion to the mechanical sports system. This movement is transmitted, arrow 2.1, to the inertia sensor 1.4. This transmission is typically mechanical, for example via a traction tape in the case of a rower. It may be a pedal assembly chain transmission in the case of an indoor bicycle.

The inertia sensor 1.4 captures mechanical motion and transforms it into a transmitted digital data item, arrow 2.2, to the control computer 1.5. The transmission can take any form known to those skilled in the art, wired or wireless, allowing the transmission of a digital signal between the sensor and the control computer 1.5, for example a serial bus, a simple cable transmitting information modulated digital or Bluetooth link etc.

The control computer 1.5 then uses this motion information to perform, among other things, a displacement estimate. This displacement estimate uses the movement as well as a model to estimate the displacement that would be a real boat undergoing the same effort from a rower. The way this estimate is done is not the subject of this document and will not be described here further. This estimate allows the control computer 1.5 to transmit this information, arrow 2.3 to the virtual environment generator 1.6. This transmission may be periodic according to a frequency specific to the control computer or generated by an event. For example some indoor bicycles transmit displacement information at each wheel revolution performed by the user. It is understandable that in this situation, the transmission is irregular and unpredictable. In still other cases, the transmission can be carried out on request of the virtual environment generator. In this case, the latter is master of the transmission frequency within the capabilities of the control computer. The control computer usually imposes a maximum frequency that can not be exceeded. Here again the transmission can be carried out by any means. It can be a wired transmission, for example USB, Ethernet or serial link. It can be a wireless transmission, for example Bluetooth or Wifi. It may also be any inter-process communication means when the two control computer and virtual environment generator functions are executed by the same information processing device.

Typically, the virtual reality headset 1.7 also transmits motion information to the virtual environment generator 1.6. This information concerns the head movements made by the user and is integrated with the movement information received from the control computer 1.5 for the generation of the virtual environment.

Other sensors can be used to retrieve various information about the movements of the user and / or his environment:

- Sensors installed on the user: accelerometer, magnetometer, gyroscope, physiological constants sensors (eg heart rate, respiratory rate, body temperature, blood pressure, oxygen saturation ...)

- Remote sensors such as camera (whatever the spectrum), sonar, electromagnetic sensors, etc. With the help of this movement information, the virtual environment generator 1.6 updates the user's position in the environment. 'environment. It is essentially about situating the camera in a three-dimensional environment. Then, the position of the camera being determined, the images seen by this camera are generated to be returned to the virtual reality headset 1.7 with the sound accompanying the scene. The virtual reality headset is typically a three-dimensional vision headset equipped with headphones. The generated images are therefore typically three-dimensional images including an image for each of the user's eyes.

2.4 transmission between the headset and the virtual environment generator can be performed by any means. It can be a wired transmission, for example USB, Ethernet or serial link. It can be a wireless transmission, for example Bluetooth or Wifi.

Figure 3 illustrates the method of generating the multimedia data in an exemplary embodiment of the invention. This data is generated for the virtual reality headset from the motion data transmitted by the control computer.

The kinetic samples 3.1 consist of the data transmitted by the control computer. A sample corresponds to a data transmission. The particular content of the transmitted data may vary between different mechanical sports systems. In all cases this data is representative of a user's position in an estimated displacement from the use of the mechanical sport system during an exercise session. In the indoor rower-based embodiment, the transmitted data includes a distance traveled and an associated time stamp (timestamp) which gives the relative date at the beginning of the session at which this distance was traveled virtually by the user using the mechanical sports system.

The accumulation step 3.2 allows the virtual environment generator to store the received samples. These samples are transmitted at a time frequency that may be regular or not. The accumulation makes it possible to memorize these samples when they arrive for their later use. The kinetic interpolation step 3.3 makes it possible to interpolate between the samples to generate a sufficient sample frequency to obtain a smooth simulation of the movement in the virtual environment. The details of this interpolation will be described later in connection with FIG. 4. Indeed, the transmission frequency of the samples is typically insufficient and would lead without interpolation to a jerky motion of the camera in the virtual environment.

Step 3.4 determines the movement of the camera in the virtual environment based on the interpolated samples. The position of the camera represents the position of the user in the virtual environment corresponding to his activity on the mechanical sports system. The movement of the camera also integrates the head movement data transmitted by the helmet and / or the body movement data transmitted by other position sensors. This aspect is not developed in this document but is essential to maintain the coherence between the real movements of the user and the virtual movements.

Step 3.5 then consists in generating the images and possibly the sound corresponding to the new position of the camera in the virtual environment. These images and sound represent the virtual environment as seen by the camera. These data constitute the multimedia data 3.6 which will be transmitted to the user to be returned by the helmet thus realizing the immersion of the user in the virtual world. FIG. 4 illustrates the interpolation method according to an exemplary embodiment of the invention.

One of the important causes of motion sickness is the non-fluidity of camera movements in the virtual environment. If the image is jerky, the perceived movement is no longer consistent with the movements of the user and disturbs his inner ear. But the samples transmitted by the control computer of the mechanical sports system are transmitted with a frequency which is generally not important, typically of the order of 50 Hz for a rower or even between 0.5 Hz and 2 Hz for a bicycle transmitting a sample at each pedal revolution. Experience shows that refreshing the images by the virtual reality helmet greater than or equal to 60 Hz is desirable to reduce the risk of motion sickness. In the exemplary embodiment, the refresh frequency is 75 Hz. In order to obtain fluid movements at the display refresh rate, it is therefore necessary to interpolate the positions of the camera and not to be content with update this position when transmitting a new sample. The interpolation method takes as input the movement data 4.1 transmitted by the control computer. These data comprise in the exemplary embodiment a current distance and the time tag of the current time.

The interpolation process is divided into two distinct modules that do not execute at the same frequency. A first module, symbolized by the arrow 4.8, runs at the transmission frequency of the samples, this transmission frequency being able to be irregular and unpredictable. It consists in calculating an adjusted instantaneous speed 4.5. Then a second module symbolized by the arrow 4.9 uses this instantaneous speed adjusted to produce a position of the camera 4.7 at the display frequency, 75 Hz in the exemplary embodiment.

Upon receiving a sample, a step 4.2 allows the computation of the instantaneous speed from the motion data of the received sample and the movement data of the previous sample. The instantaneous speed is the speed of the user between the last received sample and the previous one. First of all we calculate the difference of the distances traveled between the two samples, delta_distance = current_distance - distance_previous. The time difference separating the two samples is then calculated, delta_time = current_time - previous_time. The instantaneous speed 4.3 then corresponds to the ratio of the two, see instantaneous = delta_distance / delta_time. The instantaneous speed is used to update the user's position at the display frequency, pending receipt of a new sample. When this happens, it is likely that the position of the simulated user in the virtual environment is late or early compared to the position received in the last sample. To avoid the accumulation of these estimation errors, it is necessary to adjust the estimated speed of movement when receiving a new sample, that is to say, to moderate it if the estimate is in advance, or to accentuate it if the estimate is late. The adjustment step 4.4 makes it possible to calculate an adjusted instantaneous speed 4.5. This adjustment avoids a drift of the calculated position in the virtual environment with respect to the position transmitted by the samples without generating saccades as would a sudden registration on receipt of the sample. Typically, the adjustment consists in applying an adjustment coefficient to the instantaneous speed as a function of the difference between the new sample received and the estimated position. This adjustment factor is less than 1 when the new sample is in advance and greater than 1 when the new sample is late. In the exemplary embodiment, the adjustment coefficient is between 0.8 and 1.2. The adjustment coefficient takes the maximum values, 0.8 or 1.2 when the difference between the received sample and the estimated position is greater than 1 meter. It evolves linearly between the two terminals as a function of the difference for deviations less than or equal to 1 meter. Of course, these numerical values are an example in the context of a rower and may be different depending on the type of sports system and the frequency of the samples.

It is then this instantaneous speed 4.5 which is used by step 4.6 to determine at the display frequency the current position of the camera in the virtual environment. The position of the camera is then updated to the refresh rate according to the following formula: position = previous position + instantaneous speed adjusted * dt. Where dt represents the duration between two refreshments is 13.3 ms for a frequency of 75 Hz.

Some mechanical sports systems transmit samples only when the user is in motion, for example a bicycle that transmits a sample every turn of the bottom bracket. For these systems, only the detection of the non-reception of sample makes it possible to detect the stop of the user. Advantageously, a stopping determination step is therefore added to this method. For example, stopping is determined when a time elapsed without receiving a new kinetic sample is greater than a threshold depending on the average time interval between two kinetic samples. For example, the threshold may be defined as three times the average time interval between two kinetic samples.

Thus, a fluid movement at the sufficient display frequency is generated which reduces the risk of triggering a motion sickness in the user. This risk is further reduced by the use of a headset using an OLED display with low retinal persistence, although any other display technology can be used. Advantageously, the helmet used has a field of vision less than 180 degrees, thus respecting the fields of vision to which the brain is accustomed, which also reduces the risk of onset of motion sickness.

Postural instability during the virtual reality experience is countered by the transcription of body movements made by the user to the movements of the camera in the virtual environment. Thus the visual information and those of the inner ear are concordant at all times.

Advantageously, the system is provided with a hardware setting, for example at the helmet, to adjust the inter-pupillary distance and thus perfectly adapt the experience in three dimensions to the morphology of the user.

To cushion transitions between the real world and the virtual world, the system has transition spaces. This area allows the user to get used to the virtual experience and to make their previous settings before starting the simulation. In the case of an indoor rower, the logic is that the user makes the progress of the canoe he moves in the virtual environment. It turns out that this mode of progression in the virtual environment is not comfortable and can contribute to the onset of motion sickness. The inversion of the direction of the virtual boat and therefore a progression where the user faces the direction of movement does not cause discomfort to the user and in fact reduces the risk of occurrence of motion sickness. Advantageously, the virtual environment generator thus integrates a reversal of the direction of movement in the case of an indoor rower. Figure 5 is a schematic block diagram of an information processing device 500 for implementing one or more embodiments of the invention. The information processing device 500 may be a peripheral such as a microcomputer, a workstation or a mobile telecommunication terminal. The device 500 comprises a communication bus connected to:

a central processing unit 501, such as a microprocessor, denoted CPU;

a random access memory 502, denoted RAM, for storing the executable code of the method of realization of the invention as well as the registers adapted to record variables and parameters necessary for the implementation of the method according to embodiments of the invention, the memory capacity thereof may be supplemented by an optional RAM connected to an extension port, for example;

a ROM 503, ROM, for storing computer programs for implementing the embodiments of the invention;

a network interface 504 is normally connected to a communication network on which digital data to be processed are transmitted or received.

The network interface 504 may be a single network interface, or composed of a set of different network interfaces (for example wired and wireless, interfaces or different types of wired or wireless interfaces). Data packets are sent on the network interface for transmission or are read from the network interface for reception under the control of the software application running in the processor 501;

a user interface 505 for receiving user input or for displaying information to a user; an optional storage medium 506 denoted HD;

an input / output module 507 for receiving / sending data from / to external peripherals such as hard disk, removable storage medium or others.

The executable code may be stored in a read-only memory 503, on the storage medium 506 or on a digital removable medium such as for example a disk. According to one variant, the executable code of the programs can be received by means of a communication network, via the network interface 504, in order to be stored in one of the storage means of the communication device 500, such as the storage medium 506, before being executed.

The central processing unit 501 is adapted to control and direct the execution of the instructions or software code portions of the program or programs according to one of the embodiments of the invention, which instructions are stored in one of the aforementioned storage means. After turning on the power, the CPU 501 is able to execute instructions stored in the main RAM 502, relating to a software application, after these instructions have been loaded from the ROM for example. Such software, when executed by the processor 501, causes the steps of the flowcharts shown in Figures X to Y to be executed.

In this embodiment, the apparatus is a programmable apparatus that uses software to implement the invention. However, in the alternative, the present invention may be implemented in the hardware (for example, in the form of a specific integrated circuit or ASIC).

Naturally, to meet specific needs, a person skilled in the field of the invention may apply modifications in the foregoing description. Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to specific embodiments, and modifications that are within the scope of the present invention will be obvious to someone skilled in the art.

Claims

A method of generating multimedia data by a virtual environment generator associated with a mechanical sports system and intended for a virtual reality headset, characterized in that it comprises:

a step of receiving kinetic samples at a first regular or irregular frequency, these kinetic samples being representative of a position of the user in an estimated displacement resulting from the use of the mechanical sports system during a session of 'exercise ;

2. Method according to claim 1, characterized in that the step of determining the camera position comprises:

3. Method according to claim 2, characterized in that it further comprises:

4. Method according to claim 3, characterized in that it further comprises: a step of determining the position of the camera at the display frequency according to the previous position of the camera and the instantaneous speed adjusted.

Method according to one of claims 1 to 4, characterized in that it further comprises:

a step of determining the stopping of the displacement when a time elapsed without receiving a new kinetic sample is greater than a threshold depending on the average time interval between two kinetic samples.

Method according to one of claims 1 to 5, characterized in that it further comprises:

7. Method according to one of claims 1 to 6, characterized in that, the mechanical sports system being an indoor rower, the method further comprises:

a step of inversion of the direction of movement.

8. Sport system comprising:

- a mechanical sports system;

a virtual reality headset connected to the virtual environment generator;

characterized in that the virtual environment generator comprises:

means for receiving kinetic samples at a first regular or irregular frequency, these kinetic samples being representative of a position of the user in an estimated displacement from the use of the mechanical sports system during an exercise session;

means for determining a camera position within a virtual environment, said position representing the position of said user in the virtual environment, said determination being made at a second regular frequency, called the display frequency, greater than the first frequency and interpolation of the position of the user received;

means for generating the multimedia data corresponding to the position of the camera at the display frequency.

9. System according to claim 8 characterized in that the virtual reality headset has an OLED display.

10. System according to claim 8 or 9 characterized in that the virtual reality headset has a field of view less than 180 degrees.

11. System according to one of claims 8 to 10, characterized in that it comprises a hardware setting for adjusting the interpupillary distance.

12. A computer program comprising instructions adapted to the implementation of each of the steps of the method according to any one of claims 1 to 7 when said program is run on a computer.

13. An information storage medium, removable or not, partially or completely readable by a computer or a microprocessor comprising code instructions of a computer program for executing each of the steps of the method according to any one Claims 1 to 7.