GB2616266A

GB2616266A - Apparatus and method of nausea reduction for in-car entertainment

Info

Publication number: GB2616266A
Application number: GB2202808.8A
Authority: GB
Inventors: Erwan Damien Uberti David
Original assignee: Sony Interactive Entertainment Inc
Current assignee: Sony Interactive Entertainment Inc
Priority date: 2022-03-01
Filing date: 2022-03-01
Publication date: 2023-09-06
Also published as: GB202202808D0

Abstract

A method or system of nausea reduction comprising detecting vehicle motion; estimating a component of the motion that is parallel to a display; shifting output content in a direction opposing the estimated component; and outputting the shifted content on the display. The vehicle motion may be detected within a nauseogenic frequency range that may be one of: a range with an upper bound below the ride frequency of the car, or a range within 0.5Hz to 0.8 Hz. The display may comprise an orientation sensor such that the estimated component that is parallel to the display may be responsive to the detected orientation. When the display is substantially vertical the vertical and lateral motion of the car may be detected. When the display is substantially horizontal the fore-aft and lateral movement of the car may be detected. The display may be mounted within the vehicle or integral with an entertainment device.

Description

Intellectual Property Office Application No G132202808.8 RUM Date:30 August 2022 The following terms are registered trade marks and should be read as such wherever they occur in this document: USB PS5 micro SD Intellectual Property Office is an operating name of the Patent Office www.gov.uk/ipo

APPARATUS AND METHOD OF NAUSEA REDUCTION FOR IN-CAR ENTERTAINMENT

The present invention relates to an apparatus and method of nausea reduction for in-car entertainment.

Activities such as reading, watching TV, and playing videogames in a moving vehicle can cause nausea, because of a perceived disconnect between what the user is looking at and the motion of the vehicle.

It is thought that the mechanism behind such nausea is that in response to the motion reported by the inner ear being different to the motion reported by the eyes, the brain has evolved a nausea response because the most likely cause of such an effect was having eaten something poisonous that caused either hallucination or a blurring of vision.

Whilst the effect can occur when looking at a static book in a moving vehicle, the effect can be more acute when watching a TV programme or movie where the displayed images can move and pan in a manner directly contradictory to actual physical motion, and can be more acute still for some videogames where the user themselves is directing the viewpoint potentially contrary to the motion they feel, and often at a speed and to a degree that is greater than in most TV shows and movies.

As a result, susceptible individuals can feel ill quickly.

The present invention seeks to alleviate or mitigate this problem.

SUMMARY OF THE INVENTION

Various aspects and features of the present invention are defined in the appended claims and within the text of the accompanying description.

In a first aspect, a nausea reduction system for in-vehicle use is provided in accordance with claim 1.

In another aspect, an entertainment device comprising such a system is provided in accordance with claim 10.

In another aspect, a vehicle comprising such a system is provided in accordance with claim 11.

In another aspect, a method of nausea reduction for in-vehicle use is provided in accordance with claim 12.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: - Figure 1 is a schematic diagram of an entertainment device in accordance with embodiments of

the description;

Figure 2 is a schematic diagram of an entertainment device in accordance with embodiments of the description; - Figure 3 is a schematic diagram of a vehicle in accordance with embodiments of the description; and - Figure 4 is a flow diagram of method of nausea reduction for in-vehicle use in accordance with

embodiments of the description.

DESCRIPTION OF THE EMBODIMENTS

An apparatus and method of nausea reduction for in-car entertainment are disclosed. In the following description, a number of specific details are presented in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to a person skilled in the art that these specific details need not be employed to practice the present invention. Conversely, specific details known to the person skilled in the art are omitted for the purposes of clarity where appropriate.

Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, Figure 1 shows a typical entertainment device 400 that may be used for in-vehicle entertainment.

The entertainment device is typically a smartphone, tablet, laptop, or handheld videogame console that plays media either stored (and for a game, run) locally, and/or streamed (and for a game, run) from a remote server.

A typical portable entertainment device 400 comprises a central processing unit (CPU) (410). The CPU may communicate with components of the smart phone either through direct connections or via an I/O bridge 414 and/or a bus 430 as applicable.

In the example shown in Figure 2, the CPU communicates directly with a memory 412, which may comprise a persistent memory such as for example Flash ® memory for storing an operating system and applications (apps), and volatile memory such as RAM for holding data currently in use by the CPU. Typically persistent and volatile memories are formed by physically distinct units (not shown). In addition, the memory may separately comprise plug-in memory such as a microSD card, and also subscriber information data on a subscriber information module (SIM) (not shown).

The entertainment device may also comprise a graphics processing unit (GPU) 416. The GPU may communicate directly with the CPU or via the I/O bridge, or may be part of the CPU. The GPU may share RAM with the CPU or may have its own dedicated RAM (not shown) and is connected to the display 418 of the mobile phone. The display is typically a liquid crystal (LCD) or organic light-emitting diode (OLED) display, but may be any suitable display technology, such as e-ink. Optionally the GPU may also be used to drive one or more loudspeakers 420 of the entertainment device.

Alternatively, the speaker may be connected to the CPU via the I/O bridge and the bus. Other optional components of the entertainment device may be similarly connected via the bus, including a touch surface 432 such as a capacitive touch surface overlaid on the screen for the purposes of providing a touch input to the device, a microphone 434 for receiving speech from the user, one or more cameras 436 for capturing images, a global positioning system (GPS) unit 438 for obtaining an estimate of the entertainment device geographical position, and wireless communication means 440.

Where included, the wireless communication means 440 may in turn comprise several separate wireless communication systems adhering to different standards and/or protocols, such as Bluetooth® (standard or low-energy variants), near field communication and Wi-Fi 8, and also phone based communication such as 2G, 3G, 4G, and/or 5G.

The entertainment device is typically powered by a battery (not shown) that may be chargeable via a power input (not shown), that in turn may be part of a data link such as USB (not shown).

In general a suitable entertainment device (400) will comprise a CPU and a memory for storing and running one or more apps, and optionally wireless communication means operable to instigate and typically maintain wireless communication, for example when streaming content. It will be appreciated however that the entertainment device may be any device that has these capabilities.

For example, referring now to Figure 2 the entertainment device may be a system embedded in the vehicle itself, supplying content to one or more displays provided in the vehicle (or linked to it via a wired or wireless link).

A typical embedded entertainment device 10 may be a conventional videogame console 10 such as the Sony ® PlayStation S® (PS5), optionally adapted to be embedded in a vehicle.

The videogame console 10 comprises a central processor 20. This may be a single or multi core processor, for example comprising eight cores as in the P55. The videogame console also comprises a graphical processing unit or GPU 30. The GPU can be physically separate to the CPU, or integrated with the CPU as a system on a chip (SoC) as in the P55.

The videogame console also comprises RAM 40, and may either have separate RAM for each of the CPU and GPU, or shared RAM as in the P55. The or each RAM can be physically separate, or integrated as part of an SoC as in the PS5. Further storage is provided by a disk 50, either as an external or internal hard drive, or as an external solid state drive, or an internal solid state drive as in the P55.

The videogame console may transmit or receive data via one or more data ports 60, such as a USB port, Ethernet ® port, WiFi ® port, Bluetooth ® port or similar, as appropriate. It may also optionally receive data via an optical drive 70.

Interaction with the system is typically provided using one or more handheld controllers 80, such as the DualSense ® controller in the case of the P55.

Audio/visual outputs from the videogame console are typically provided through one or more A/V ports 90, or through one or more of the wired or wireless data ports 60.

Where components are not integrated, they may be connected as appropriate either by a dedicated data link or via a bus 100.

Vehicles where such entertainment devices are built-in, removably mounted, or simply used (for example in a hand-held configuration) include but are not limited to ships, planes, trains, coaches, busses, and cars. Whilst users can be susceptible to nausea in all these vehicles, road vehicles in particular are an issue due to the comparative prevalence of use with such devices, and so purely for illustrative purposes embodiments of the description refer to cars, but it will be appreciated that the approaches disclosed herein are similarly applicable to other vehicles.

As noted elsewhere herein, nausea is primarily caused by a mismatch between felt and seen motion. It is therefore helpful to characterise the motion of a vehicle.

Referring now to figure 3, a typical travelling car comprises a superposition of intentional and unintentional motions in three axes; fore-aft (x-axis), lateral (y-axis), and vertical (z-axis). There are also motion components from roll, pitch and yaw, but typically these translate into axial motion in the various parts of the vehicle.

Both intentional and unintentional motions can occur in frequency ranges that tend to cause nausea. For shorter journeys or in-town car journeys, intentional motions (e.g. acceleration and steering) may be the main source of such frequencies, whilst for long distance journeys, unintentional motions (e.g. resonant rocking or environmental effects) may be the main source of such frequencies.

To a first approximation, intentional motions have a frequency close to 0 Hz; the vehicle typically travels at a net or mean velocity Vc primarily in a forwards direction, and in normal use accelerates or decelerates on the x-axis over a matter of several seconds. Meanwhile road inclines on the z-axis typically take in the order of tens of seconds to minutes to traverse, and the act of completing a left or right turn on the x/y axis typically takes several seconds. These periods tend to be even longer for larger vehicles such as coaches, trains, and planes.

Hence depending in part on the vehicle size, suspension stiffness (where relevant) and driving style (where relevant), intentional motions can thus be assumed to have a frequency below one of 0.5, 0.4. 0.3, 0.2, 0.1, and 0.05 Hz in each direction. Typically, x-axis and z-axis intentional motions can be assumed to have a lower frequency than y-axis motions.

In any event, intentional motion frequencies can typically characterised as very low frequency, below 0.5, 0.4. 0.3, 0.2, 0.1, or 0.05 Hz.

Meanwhile unintentional motion frequencies typically occur in an adjacent but higher range.

Most of these motions occur due to the vehicle suspension and the road (or sea) surface, but also wind and other environmental factors can contribute.

Vehicle suspension typically has a damped resonance characteristic, and for vertical motion the resonance is typically in the order of 1-2Hz. Some cars with 'soft' or 'boat' suspension may be in the range 0.5-1.5 Hz, whilst some with sports suspension may be in the range 1.5-2.5 Hz. This is also referred to as the ride frequency of the vehicle.

Other motions may come from bumps and pot-holes that the suspension cannot fully absorb, and also buffeting from wind -both natural wind but also air displaced by other passing vehicles. In addition, a vehicle may have a fault such as a problem with wheel alignment or damage to the suspension that can cause it to shake. Typically however these are higher frequency vibrations than the ride frequency.

Meanwhile, motions that cause nausea are typically centred around 0.2 Hz. In one study, around 2/3 of subjects became nauseous after 22 minutes exposure at 0.1 Hz; all subjects became nauseous after 11 minutes exposure at 0.2 Hz, and around 1/2 of subjects became nauseous after 28 minutes exposure at 0.4 Hz.

Hence there is a marked peak in nauseogenic effect at around 0.2 Hz, dropping off on either side (but still notable in ranges similar to very soft vehicle suspensions).

Notably however, as noted above these frequencies are commonly encountered in intentional vehicle driving, particularly in-town were speed is less constant and there is more quick turning.

Hence when a passenger is being moved in one or more axes between around 0.05 to to 0.5 Hz, and particularly around 0.2 Hz, they can rapidly start to feel sick.

One well-known way to reduce nausea in these circumstances is to look at the horizon, for example through the front windscreen of the car. This provides a fixed visual point that helps the passenger's brain to predict the movement experienced by the ears' vestibular system.

In embodiments of the present description, this approach to nausea reduction is incorporated into the display of (and optionally the generation of) content viewed by a passenger.

In particular, vehicle movements within a nauseogenic frequency range (for example between around 0.05 to 0.5 Hz) are visually compensated for by panning the displayed image in an opposing direction, thereby creating a fixed point within the display (or at least motion in the displayed content that is consistent relative to an underlying fixed point).

The nauseogenic frequency range can be taken to be 0.1 to 0.4 Hz, or more generally 0.05 to 0.5 Hz, or still more generally below 0.8 Hz, or still more generally below the ride frequency of the vehicle, optionally by a predetermined amount (for example between 0.1 and 1 Hz below the ride frequency, depending on how low the ride frequency is) The ride frequency may be known for a particular vehicle, particularly in the case of an entertainment device that is built into the vehicle by the manufacturer, or may be input via a user interface, or may be measured directly during ongoing travel.

Vehicle movements within the nauseogenic frequency range in one or more directions (and typically in each of the X, V. and Z directions) can be measured by one or more motion sensors such as accelerometers and/or gyroscopes (442, 65) as required, the motion sensors optionally being integral to the entertainment device (for example in the case of a smartphone or similar, or a suitably adapted videogame console) or separate (for example connected to a data port 60, or included in a separate peripheral such as controller 80).

As noted elsewhere herein, the measured vehicle movements can be visually compensated for by panning the displayed image in an opposing direction.

Hence for example pre-recorded content such as a television show or film may be shaken in opposition to the motion of the vehicle to make the content visually more stable. Optionally the television show or film may be shown with some of the peripheral image of the frames outside the bounds of the display by default, so that when frames are perturbed by panning, this pulls the peripheral image into view. This disguises the panning effect as the image appears to remain full screen, rather than one or more edges of the image shifting inwards as the image is panned.

Interactive generated material such as for a video game may be similarly panned and may be similarly displayed with some of the peripheral image of each frame outside the bounds of the display by default so as to disguise the panning effect. Alternatively or in addition, the virtual camera used to generate each frame may be moved in opposition to the vehicle motion so that the rendered view itself pans instead of the displayed image. In this latter instance, optionally the displayed image may also still be panned, albeit by a smaller amount, to take account of vehicle motion that has occurred during the rendering process, i.e. during the time between the selection of the virtual camera position and the output of the rendered image.

In any event, by shifting the displayed image and/or the virtual camera used to generate the displayed image, the viewpoint presented by the displayed content becomes fixed (or less variable), which can assist with nausea reduction for the user.

It will be appreciated that whilst movements can occur in each of the X, Y, and Z directions, typically panning of a displayed image will be in two directions. Consequently information regarding the orientation of the display in the vehicle is of use in responding to the correct components of the vehicle motion. In particular, if the display is mounted vertically (for example on the back of a seat in front of the user), then panning or rendering may be performed in opposition to Y and Z direction vehicle motion. Meanwhile if the display is mounted horizontally (for example for a portable device being held in the lap of the user), then panning or rendering may be performed in opposition to X and Y direction vehicle motion.

More generally, the panning or rendering may be performed in opposition to the components of the X, I, and Z motion parallel with the display used by the entertainment device. Optionally if one component is small (e.g. the display is within a threshold angle of horizontal or vertical) then that substantially orthogonal motion component may be ignored.

Hence in summary, frequency components of vehicle motion within a neurogenic range and substantially parallel with a display being used by entertainment device are used to drive opposing motion of the images on the display, either by panning the images and/or changing the viewpoint at which the images are rendered (i.e. collectively 'shift' the image), so as to create a fixed (or less variable) notional horizon within or behind the displayed images (even if the content displayed comprises its own motion relative to this notional horizon). This helps to reduce the chances of nausea by someone watching the display within the moving vehicle.

Optionally, alternatively or in addition to correcting for vehicle motion within a neurogenic frequency range, images may be similarly panned or re-composed in response to absolute motion; hence (excluding gross forward motion), small X, Y, or Z movements are detected using the motion sensors and the components of these movements parallel to the display are used to drive an opposing corrective pan and/or re-composition.

Hence for example any movement within and up to the corrective range available by panning the image and/or recomposing the image may be corrected for, including higher frequency movements if they occur, such as at the ride frequency or higher (typically limited by the refresh frequency of the display).

Optionally however this vehicle movement may again be limited to movement within the neurogenic frequency range discussed elsewhere herein, for example by low-pass or band-pass filtering the detected motion signals.

Referring again to Figures 1 to 3, in a summary embodiment of the present description a nausea reduction system for in-vehicle use comprises the following.

Firstly, a graphics processor (for example CPU 410 or 20, and/or GPU 416 or 30) operable (for example due to suitable software instruction) to output content images for display on a display (which may or may not be integral with the device comprising the nausea reduction system), as described elsewhere herein.

Secondly, at least a first motion sensor operable to detect vehicle motion (for example gyroscope and/or accelerometer 442 or 65), as described elsewhere herein.

Thirdly, a motion compensation processor (for example CPU 410 or 20, and/or GPU 416 or 30), operable (for example due to suitable software instruction) to estimate at least a first component of the detected vehicle motion that is parallel to the display, as described elsewhere herein.

Wherein, the graphics processor is operable (again for example due to suitable software instruction) to shift the output content images in a direction opposing the estimated at least a first component of the detected vehicle motion that is parallel to the display, thereby creating a fixed (or less variable) notional horizon within or behind the displayed images, as described elsewhere herein.

- In an instance of this summary embodiment, which the vehicle motion is detected within a nauseogenic frequency range; for example by low pass or band pass filtering the motions signals, or limiting processing of the signals to the frequency range, as described elsewhere herein.

- In this instance, optionally the vehicle motion is detected within one selected from the list consisting of a frequency range with an upper bound at a predetermined threshold frequency below the ride frequency of the vehicle (for example between 0.1 and 1 Hz below, depending on the ride frequency), and a frequency range with an upper bound in turn within the range 0.8 Hz to 0.5Hz, as described elsewhere herein.

- In an instance of this summary embodiment, the system comprises an orientation sensor configured to measure the orientation of the display displaying the output content images, and the motion compensation processor estimates the at least a first component of the detected vehicle motion that is parallel to the display responsive to the measured orientation of the display, as described elsewhere herein. Alternatively (for example in the case of a pre-installed display), the orientation can be assumed (typically to be vertical).

- In this instance, optionally when the display is oriented substantially vertically, the motion compensation processor estimates the vertical and lateral vehicle motion, as described elsewhere herein.

- Similarly in this instance, optionally when the display is oriented substantially horizontally, the motion compensation processor estimates the fore-aft and lateral vehicle motion, excluding the main directional motion of the vehicle (e.g. small variations in fore-aft vehicle motion, but not the near-DC gross motion of the vehicle in its direction of travel), as described elsewhere herein.

- In an instance of this summary embodiment, the graphics processor shifts the output content images by panning them, as described elsewhere herein. This is of particular use for pre-recorded material but can also be used for videogames, for example to correct for motion that has occurred since the start of the rendering pipeline.

- In an instance of this summary embodiment, for video games the graphics processor shifts the output content images by moving the viewpoint of a virtual camera used to compose the image, as described elsewhere herein.

- In an instance of this summary embodiment, the display is one or more selected from the list consisting of mounted within the vehicle, and integral with an entertainment device, as described elsewhere herein. Hence the display may or may not be mounted in the vehicle, and may or may not be integral with the entertainment device.

The nausea reduction system of this summary embodiment may be comprised within an entertainment device (for example a video playback system, videogame console, or general handheld device such as a phone or tablet), as described elsewhere herein.

Similarly the nausea reduction system of this summary embodiment may be comprised within a vehicle, as described elsewhere herein. In this case the nausea reduction system may be part of an entertainment device and/or a display that are built into the vehicle, and/or may provide the image shifting function for a separate device (e.g. a device that plugs into the vehicle to output content to a vehicle display).

Turning now to Figure 4, a corresponding method of nausea reduction for in-vehicle use comprises the following steps.

In a first step 410, detecting vehicle motion, as described elsewhere herein.

In a second step 420, estimating at least a first component of the detected vehicle motion that is parallel to a display, as described elsewhere herein.

In a third step 430, shifting output content images in a direction opposing the estimated at least a first component of the detected vehicle motion that is parallel to the display, as described elsewhere herein.

And in a fourth step 440, outputting the shifted content images for display to the display, as described elsewhere herein.

It will be apparent to a person skilled in the art that variations in the above method corresponding to operation of the various embodiments of the apparatus as described and claimed herein are considered within the scope of the present invention, including but not limited to that: - the vehicle motion is detected within a nauseogenic frequency range being one selected from the list consisting of a frequency range with an upper bound at a predetermined threshold frequency below the ride frequency of the vehicle, and a frequency range with an upper bound in turn within the range 0.8 Hz to 0.5Hz, as described elsewhere herein.

-The method comprises the step of measuring the orientation of the display displaying the output content images, and the motion compensation processor estimates the at least a first component of the detected vehicle motion that is parallel to the display responsive to the measured orientation of the display, as described elsewhere herein.

It will be appreciated that the above methods may be carried out on conventional hardware, such as the example entertainment devices (10, 400) described elsewhere herein, suitably adapted as applicable by software instruction or by the inclusion or substitution of dedicated hardware (for example where necessary at least a first motion detector).

Thus the required adaptation to existing parts of a conventional equivalent device may be implemented in the form of a computer program product comprising processor implementable instructions stored on a non-transitory machine-readable medium such as a floppy disk, optical disk, hard disk, solid state disk, PROM, RAM, flash memory or any combination of these or other storage media, or realised in hardware as an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array) or other configurable circuit suitable to use in adapting the conventional equivalent device. Separately, such a computer program may be transmitted via data signals on a network such as an Ethernet, a wireless network, the Internet, or any combination of these or other networks.

The foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.

Claims

CLAIMS1. A nausea reduction system for in-vehicle use, comprising: a graphics processor operable to output content images for display to a display; at least a first motion sensor operable to detect vehicle motion; a motion compensation processor, operable to estimate at least a first component of the detected vehicle motion that is parallel to the display; and the graphics processor being operable to shift the output content images in a direction opposing the estimated at least a first component of the detected vehicle motion that is parallel to the display.
2. The nausea reduction system of claim 1, in which the vehicle motion is detected within a nauseogenic frequency range.
3. The nausea reduction system of claim 2, in which the vehicle motion is detected within one selected from the list consisting of: a frequency range with an upper bound at a predetermined threshold frequency below the ride frequency of the vehicle; and 15ii. a frequency range with an upper bound in turn within the range 0.8 Hz to 0.5Hz.
4. The nausea reduction system of any preceding claim, comprising: an orientation sensor configured to measure the orientation of the display displaying the output content images; and the motion compensation processor estimates the at least a first component of the detected vehicle motion that is parallel to the display responsive to the measured orientation of the display.
5. The nausea reduction system of claim 4, in which when the display is oriented substantially vertically, the motion compensation processor estimates the vertical and lateral vehicle motion.
6. The nausea reduction system of claim 4, in which when the display is oriented substantially horizontally, the motion compensation processor estimates the fore-aft and lateral vehicle motion, excluding the main directional motion of the vehicle.
7. The nausea reduction system of any preceding claim, in which the graphics processor shifts the output content images by panning them.
8. The nausea reduction system of any preceding claim, in which for video games the graphics processor shifts the output content images by moving the viewpoint of a virtual camera used to compose the image.
9. A nausea reduction system according to any preceding claim, in which the display is one or more selected from the list consisting of: mounted within the vehicle; and integral with the entertainment device.
10. An entertainment device comprising a nausea reduction system according to any preceding claim.
11. A vehicle comprising a nausea reduction system according to any preceding claim.
12. A method of nausea reduction for in-vehicle use, comprising the steps of: detecting vehicle motion; estimating at least a first component of the detected vehicle motion that is parallel to a display; shifting output content images in a direction opposing the estimated at least a first component of the detected vehicle motion that is parallel to the display; and outputting the shifted content images for display on the display.
13. A method of nausea reduction according to claim 12, in which the vehicle motion is detected within a nauseogenic frequency range being one selected from the list consisting of: a frequency range with an upper bound at a predetermined threshold frequency below the ride frequency of the vehicle; and ii. a frequency range with an upper bound in turn within the range 0.8 Hz to 0.5Hz.
14. A method of nausea reduction according to claim 12 or claim 13, comprising the step of: measuring the orientation of the display displaying the output content images; and in which the motion compensation processor estimates the at least a first component of the detected vehicle motion that is parallel to the display responsive to the measured orientation of the display.
15. A computer program comprising computer executable instructions adapted to cause a computer system to perform the method of any one of claims 12 to 14.