DE202023107282U1 - Interactive vehicle entertainment system using vehicle interior and vehicle exterior sensors - Google Patents

Abstract

An interactive entertainment system for a vehicle, comprising a control unit configured to perform the following steps:
- Receiving vehicle surroundings data which was recorded by at least one vehicle exterior sensor of the vehicle;
- Determine, based on the vehicle surroundings data, at least one object property of at least one object which is located in the vehicle surroundings;
- Obtaining a user input from at least one vehicle occupant for the at least one object, which determines an estimated object property of the at least one object;
- comparing the determined object property and the estimated object property to determine user feedback;
- Outputting the user feedback to the at least one vehicle occupant.

Description

Technical area

Various embodiments of the present invention relate to entertainment systems in vehicles, in particular interactive vehicle entertainment systems that use vehicle interior and vehicle exterior sensors of the vehicle in real time. Furthermore, corresponding vehicles, methods, computer programs and computer-readable storage media are provided.

background

Traditional in-vehicle entertainment systems are typically limited to passive experiences such as playing videos or simple on-screen interactivity without incorporating the real vehicle environment. However, with the emergence of increasingly automated and autonomous vehicles, the demand for more interactive and immersive entertainment options while driving is growing. Previous entertainment systems have clear limitations here - they neither use interior sensors to track and interpret the movements or interactions of the vehicle occupants, nor do they incorporate real-time data from the various vehicle-exterior sensors into the entertainment experience. In addition, existing systems do not enable enhanced, cross-vehicle multiplayer experiences by incorporating vehicle-to-vehicle (V2V) communication.

Summary

Therefore, there is a need for improved vehicle entertainment systems that overcome or mitigate at least some of the limitations and disadvantages mentioned. This task is solved with the features of the independent claims. Further advantageous embodiments are described in the dependent claims.

The solution according to the invention is described below in relation to the claimed entertainment systems, in other words entertainment systems, which can be used in vehicles. Corresponding vehicles, procedures, computer programs and electronically readable storage media are also described. It is to be understood that features, advantages and alternative embodiments can be assigned to the other categories, and vice versa. In other words, the entertainment systems can be improved by features described in the context of the vehicles, methods, computer programs and storage media, and vice versa.

A vehicle entertainment system is configured to provide an entertainment application to at least one occupant of the vehicle. An occupant can thus be a user of the entertainment system.

The interactive entertainment system can be arranged in the vehicle or implemented as a remote or cloud system via a wireless data connection. The entertainment system includes a control unit configured to perform the following steps.

In one step, vehicle environment data is received that was recorded by at least one exterior sensor of the vehicle.

This can include receiving or recording data about the environment around the vehicle using sensors, e.g. image data from a camera or distance measurements from a lidar system. Such an exterior sensor can be, for example, a camera, a radar, a lidar or an ultrasonic sensor that is installed on parts of the vehicle's exterior, such as the bumper, fender, roof, and monitors the exterior of the vehicle.

In one step, at least one object property of at least one object, which is located in the vehicle environment based on the vehicle environment data, is determined by the control unit. The at least one object can include several objects, for example a grouping of objects, in particular two or more objects that are related to one another. An object property can therefore also be an object attribute that is assigned to the at least one object, in particular a group of objects.

An object that is located in the vehicle environment, i.e. the vehicle exterior, can include other vehicles, infrastructure such as traffic signs or zebra crossings but also animals, people or objects in the area around the vehicle that can be detected by the sensor. The object property can relate to an attributive feature of an object such as shape, color, size or type/class of the object, for example another vehicle. This can typically be achieved by processing sensor data through machine learning models trained end-to-end.

In some examples, machine learning models may be trained end-to-end, i.e. they go through a complete learning process without intermediate steps or manual features. The algorithms are optimized solely based on the sensor and other input data to produce the desired results or predictions. For this purpose, extensive training input data sets are used, which were initially prepared manually. Specifically, the sensor data was enriched with ground truth information by human annotators marking features such as object classes, positions or movements in the data sets. The resulting example sets from original sensor inputs plus the addition of the “ground truth” data then serve as input for the learning algorithms. The models go through multiple training iterations and learning cycles on this data to develop an understanding of the features to be identified and the required prediction results. After training, the machine learning (ML) models can be applied to real-time input data of the vehicle environment and interior. They classify objects, interpret activities or estimate attribute values without further manual specifications by transferring the trained understanding of the ground truth-marked example situations to new input data. Powerful and accurate recognition of complex situations can be achieved through fully data-driven optimization of the models on extensive training data sets with annotated sensor data.

In one step, a user input from at least one vehicle occupant, which includes an estimated object property of the object, is received by the control device. The user input is a user input regarding an object. The user input may select, or determine, or identify an object in the vehicle environment. User input for an object determines or specifies an estimated object property. Entering or receiving the user input can be based on a previously provided predetermined (game) rule. Receiving user input may include the vehicle occupant specifying an object or their own assessment of the object property of an object via an interface/interface/HMI, for example naming the color they believe to be the object or entering this on a display/touchscreen. The user input can thus identify an object that corresponds to a predetermined game rule, or a predetermined object property, and/or specify an object property for the object.

In some examples, the user input in the interactive entertainment system may include a selection, determination or identification of an object in the vehicle environment by the vehicle occupant. For example, an object may be another vehicle that is located in the area around the ego vehicle detected by the vehicle exterior sensors. However, other objects, such as traffic-relevant objects or people in traffic, or objects of interest are also possible.

In some examples, the user's selection or designation of the object may take various forms. In one embodiment, the vehicle occupant directly indicates the object that he would like to select or determine via the user interface, for example the touchscreen or by voice command. This can be done by tapping on the display, naming the object type, for example “blue BMW”, or by making a gesture towards the object.

For the vehicle occupant to select or determine an object, the interactive entertainment system can optionally specify a user task or (game) rule, in other words a predetermined object property. One or more objects in the vehicle environment may satisfy this predetermined object property, and thus the user input, when identifying an object, may indicate that the object satisfies the predetermined object property. This can also be communicated via the interface between the system and the occupant, for example in the form of a text, a spoken announcement or a pictorial representation.

The user task or game rule defines a specific condition or situation that is to be fulfilled by the vehicle occupant by selecting a suitable object.

For example, the rule can specify that the user should identify objects with a certain object property.

Example tasks or rules would be the entertainment system telling the occupant to select or identify a red car in the area. The (game) rule can thus specify an object property, which can include a predetermined object property. The system specifies the rule that the user should identify a vehicle from a specific manufacturer, such as a BMW. The rule of the game states that a truck must be marked in the vehicle's surroundings. The user is given the task of selecting the vehicle furthest to the left as the object.

However, more complex tasks or rules are also conceivable: For example, the rule could specify a predetermined grouping, spatial or temporal sequence in the vehicle environment or interaction between one or more objects, spatial or temporal sequence of an object within the vehicle environment, associations or temporal sequences of several objects in the vehicle environment.

For example, the entertainment system could specify the following more complex tasks or rules. The user should mark a certain constellation or situation with several vehicles, for example two red cars driving next to each other or three motorcycles in a row. This could include, for example, an overtaking maneuver by a car or a motorcycle and a truck. The player should select a grouping of objects of the same or different types, for example at least a specified number of trucks or a series of vehicles with a specified object property.

The task is to create a thematic association between different elements and to mark them, e.g. a fire engine and a fire hydrant, a garbage truck and a garbage can, a person at a zebra crossing, a bicycle on a bicycle path at a specified distance or closer to an intersection. Or parents with a child in a group on a sidewalk or in a vehicle. The occupant is asked to follow and select a temporal sequence or interaction between objects, for example first a car turning and then a car braking. For example, an object could also be a person. And the object property could be an intention of the person.

As a more complex scenario, it is also possible to identify a traffic situation such as a lane change or a parking maneuver by selecting the vehicles involved.

In principle, in addition to individual objects (people or inanimate objects or vehicles), relationships, spatial groupings, intentions and connections between several objects and/or elements in the vehicle environment can also be defined as tasks or game rules. This requires the user to interpret the overall situation in order to make an appropriate selection of objects that meet the required condition. The difficulty and cognitive demands of such interactions can be gradually adjusted.

This creates immersive, sophisticated mental connections between the real environment and the virtual experience. The entertainment experience achieves a high standard for the user by taking into account more complex scenarios, relationships and dynamics between a variety of real objects. Depending on the user's abilities, a flow experience can be created for the user.

Thus, certain characteristics or values of object properties are defined by task or rule specifications, while other specified object properties, for example color, brand or position, are defined. Based on these criteria, the player should then interactively select a suitable, real object in the vehicle's surroundings that fulfills the specified object properties.

The user task or game rule defines and specifies the framework for correct object selection. Further evaluation by the control unit then checks whether the criterion specified by the system was met by the user input or object selection. This results in the corresponding points being awarded or feedback for the interactive gameplay.

The combination of task/rule specifications and user-selected objects creates a meaningful connection between the system and the occupant. The immersive gameplay, linked to the real vehicle environment, is understandable and interactive for the passenger.

In some examples, the user selects an object indirectly by specifying a suspected or estimated property of the object. This can again be done via input on the touch display or other modalities such as voice input. For example, if the user enters the color of one of the vehicles in the area, he implicitly selects the vehicle that he thinks corresponds to this color.

Regardless of the type of input, the user selection, after being received by the control unit, is displayed on the display device for the user. This takes the form of a highlight, a tag or a highlighting of the corresponding object. This means that the vehicle occupant can see on the display which object was actually selected or determined by their input.

The marking of the object selected by the user can take various forms. Color markings are conceivable, for example framing the object in a signal color. Other options are changes to the object contours such as thickening or hatching. Partial markings can also be implemented, such as inserting a symbol or label. These visual highlights make the actual The object recognized by the system as selected by the user is made clear to the user.

In addition to purely visual marking, other modalities are also possible. Acoustic marking would also be possible, for example in the form of a confirmation tone or a spoken announcement after the object has been selected. Haptic signaling via vibrations of the display or controller is also conceivable.

In general, the marking of the object after user input serves to give the occupant clear and unambiguous feedback about which real object in the vehicle environment he has actually selected. This forms the basis for the subsequent evaluation and playful feedback as to whether the user's assessment/estimation of an object property corresponds to the assessment of the evaluation by a machine learning model/artificial intelligence, i.e. lies within the limits of the machine-determined object property. Linking the virtual object with the real object also creates an immersive experience.

In one step, the determined object property and the estimated object property are compared to determine a user feedback, in other words a user response, i.e. a feedback from the entertainment system to the user regarding his user input.

In a further step, the user feedback is output for the at least one vehicle occupant.

The comparison and the output of user feedback can be implemented in such a way that the control unit checks whether the occupant's input matches the automatically determined object property and then outputs feedback as to whether the estimate was correct or incorrect.

The entertainment system according to the invention therefore uses exterior sensors on the vehicle to detect objects in the environment and determine their properties. Occupants can provide an estimated object property as input. The control unit compares the actual detected property of the object with the occupant's input. Based on this comparison, feedback is generated and output to the occupant, e.g. whether the estimate was correct. This creates an interactive entertainment system based on environmental detection and occupant input.

The at least one vehicle exterior sensor can be a component of an autonomous driving system of the vehicle. Autonomous vehicles have systems that make it possible to drive partially or completely without the intervention of a human driver. The systems then take over control of acceleration, braking and steering. Central sensors for the system's environment detection can include radar sensors for distance and speed measurement, cameras for visual recognition of other road users, lidar for high-precision 3D imaging of the environment, ultrasound for close-range detection around the vehicle, and GPS for position determination. Data streams of at least one or a combination of these sensors can be made available to the entertainment system via an interface as vehicle exterior sensors.

The at least one vehicle exterior sensor can be a component of an Advanced Driver Assistance System (ADAS) of the vehicle. The at least one vehicle exterior sensor can include one or more of a camera, a radar sensor, a lidar sensor and an ultrasonic sensor.

In preferred examples, the at least one exterior sensor can include several exterior cameras of the vehicle. In preferred examples, the display device can be a touchpad integrated into a vehicle seat.

The exterior sensor that provides the environmental data for the entertainment system can be a sensor that is already present in the vehicle for driver assistance systems (ADAS). These can be systems installed in current vehicles, such as adaptive cruise control, lane departure warning or emergency braking. The sensors used for this, namely camera(s), radar, lidar and/or ultrasonic sensor, can also be used to provide the vehicle's environmental data for the entertainment system. This creates a synergy effect and enables the reuse and integration of existing ADAS sensors.

Specific examples of ADAS systems are the adaptive cruise control (ACC), the automatic emergency braking system (AEB), lane departure warning (LDA) and blind spot, traffic sign or night vision assistants. To detect the surroundings, these systems rely on cameras that record visual data and images from the vehicle's surroundings. On the other hand, imaging systems are also used. Sensors such as radar systems, laser scanners (Lidar) or ultrasonic sensors are used to precisely measure distances, positions or speeds of other objects.

By integrating the entertainment system with ADAS components, this sensor technology can now be used multiple times. The existing cameras, radar, lidar and ultrasonic sensors that are installed for the driver assistance systems can also record data for the interactive entertainment system in parallel. This enables cost-effective technical integration while reusing existing hardware. At the same time, the entertainment system benefits from the high-quality sensors, which are already state-of-the-art in vehicle electronics. Modern vehicles already have a large number of cameras, e.g. front cameras, rear cameras, cameras in the exterior mirrors and radar, lidar and ultrasonic sensors around the vehicle body. These enable 360-degree recording of the entire vehicle environment and therefore provide data for the interactive entertainment system. This results in the ADAS sensor technology being used for two different, complementary applications - driver assistance and passenger entertainment.

The vehicle environment data may include a real-time data stream from the at least one vehicle exterior sensor.

The data about the vehicle surroundings provided by the outdoor sensor can be continuously recorded in real time and transmitted to the entertainment system. It is a continuous, or sampled, data flow of the current sensor readings perceived by humans, not snapshots transmitted with measurement interruptions or with a time delay. This enables interactive use by the vehicle occupants, which is based on a current environmental situation. Changes in the vehicle's surroundings, such as the movements of other road users, are visible in the entertainment system without any noticeable latency. This creates an immersive gaming or entertainment experience for passengers that is linked to the real, dynamic traffic situation.

Receiving user input may include the following steps. In one step, the object and the object property are displayed for the occupant on a display device in the interior of the vehicle that is assigned to the occupant. In a further step, the user input is entered on the display device by the user and is therefore received on the display device, for example a touchscreen.

The interaction with the occupant can be designed in such a way that the object identified by the system in the vehicle environment is first displayed with its automatically determined properties on a screen or display, which can be viewed by the occupant in the vehicle interior. In some examples, the display device may also not be associated with the vehicle, but may include a hand-held or worn user terminal. The user input with the estimated object property can then take place via the same display device, for example by the occupant making a selection on the touchscreen or typing something. Input and feedback via a special device such as VR glasses or a head-up display or gesture recognition is also conceivable. This makes it possible to display the surrounding object as well as record the occupant input via the interface in the vehicle.

The entertainment system can continuously record and evaluate the sensor and camera data from the vehicle's surroundings in real time. If an object is now detected in the real outside world, it can be displayed in real time on the display device in the interior.

Real-time processing allows the user to associate the real object and the displayed object with each other on the display. He then submits his input regarding the property of this object. This achieves a time-synchronous comparison between the real vehicle environment and the displayed representation for the interactive game.

The displayed object can be marked, in other words tagged, based on the user input in one step. The mark or tag can be displayed to the user.

This can happen, for example, in such a way that a live image of the vehicle's surroundings with a superimposed marking of the selected object is displayed on the occupant's display or VR glasses. The marking can be a color or shape highlight, a label, a frame or similar. At the same time as the display, the occupant sees the real object through the windshield, e.g. another car outside in traffic. This exact object appears on the display in a live view with the corresponding label.

Receiving user input may include receiving vehicle interior data, wel che are detected by at least one vehicle interior sensor, and determining the user input from the vehicle interior data using a machine learning model.

Not only the classic variant via touchscreen, buttons or voice can be used to record occupant input. Alternatively, it is also possible to use special interior sensors that monitor and interpret gestures and/or poses of people in the vehicle.

These can be cameras, microphones, radar sensors or sensors in seats/doors. The recorded data is then evaluated using machine learning models. Using adaptive algorithms and neutral networks, even complex gestures, facial expressions or statements can be analyzed and user input regarding the specific object can be derived from them.

As an example, a camera records how the occupant points at a car in the real environment. The pointing gesture is recognized by the algorithm and interpreted as a selection of this environmental object. The occupant himself does not have to actively enter anything; this happens automatically through the interpretation of his activities inside the vehicle.

The at least one vehicle interior sensor may be configured to monitor occupant activities in the interior of the vehicle. The at least one vehicle interior sensor may include one or more of, for example, a camera, a microphone, and a touch sensor.

The interior sensor, which records the occupant activities to determine user inputs, can include different types of sensors to record various interactions of the people in the vehicle. A camera records visual user behavior, gestures, facial expressions, and direction of gaze. A microphone records acoustic utterances and voice commands. A touch sensor registers direct inputs via touchpads or touch-sensitive surfaces.

This allows both specific control inputs to be measured via the sensors and more complex user behavior such as pointing gestures or voice commands to be analyzed. Several sensor types can be combined to increase robustness and comprehensively interpret occupant input. This allows for comfortable, natural interaction with the entertainment system through normal vehicle interior behavior.

In some examples, the controller may be configured to receive and fuse data from multiple types of indoor sensors to determine user input. Sensor data from different modalities - such as camera images for recording gestures and facial expressions, microphone signals for recording voice input and input via touchpads - can be combined. By fusing these indoor sensor values, the detection accuracy of the actual intent of a user input can be increased. An example would be the combination of the visual identification of a target object outside the vehicle using a camera recording of the occupant's head movement and arm gestures with a simultaneous verbal statement of this target object. Taken together, this can be seen as a selection of the real object by the vehicle occupant with higher probability and less risk of misinterpretation.

Examples of vehicle interior sensors for combination and secondary use are interior camera systems, which enable detection of occupant activities such as gestures or viewing directions by recording visual data. There are also usually microphones for voice control in the vehicle, which can also be used to analyze passengers' verbal statements. Other sensors include pressure sensors in the seats, which primarily serve to detect weight, but could also secondarily measure people's interactions when touching or gripping. Active sensors in the area of seat belts or handles or side windows, which record contact points or acceleration values, may also be included. Thermal imaging systems or radar systems for occupant detection or heartbeat monitors that provide user-related vital data are also conceivable. With the appropriate combination of several such sensor sources, particularly robust and reliable inferences about user input are possible through data fusion and interpretation.

The user feedback for the at least one occupant can be output optically and/or acoustically on a display device inside the vehicle. The user feedback for the at least one occupant can be output via a virtual reality (VR) or augmented reality (AR) interface.

The system's feedback or output to the vehicle occupant, reflecting the user input and/or the quality of his inputs and/or interactions with the entertainment system, can be realized in different ways. Via visual display and/or Loudspeakers. Graphic or textual outputs or spoken texts/sounds are output here. Via interfaces for virtual and augmented reality. This includes head-up displays, data glasses and augmented reality systems in which digitally generated feedback elements are displayed in the user's actual field of vision. This creates a particularly immersive, playful experience and enables contextual graphic overlays that can be linked to real objects in real time. The user feedback, i.e. the feedback from the entertainment system to the user/occupant, is thus perceived as particularly realistic.

The controller may be further configured to receive a second user input from a second vehicle occupant, the second user input comprising a second estimated object characteristic of the object. Comparing the determined object characteristic and the second estimated object characteristic to determine second user feedback for the second occupant. And maintaining individual scores for the first and second occupants based on the accuracy of their respective estimated object characteristics compared to the determined object characteristic.

The system not only supports individual passengers, but can also manage multiple participants within the interactive game. A second or further input from other passengers in the vehicle can be received via the corresponding interfaces, creating a competition.

For each occupant, an individual comparison is made between self-assessed and actual property characteristics. Accordingly, each player receives personalized feedback and points based on correct guesses.

This also enables competition between different passengers in the vehicle, in which points can be iteratively collected from each player in order to create high scores and a ranking list.

The vehicle or entertainment system may further include a vehicle-to-vehicle communication interface, wherein the control unit may be further configured to send score data to another vehicle via the vehicle-to-vehicle communication interface. Receiving score data from the other vehicle via the vehicle-to-vehicle communication interface. And providing a competitive interactive gaming experience between occupants of the vehicle and occupants of the other vehicle using the sent and received score data.

The occupants' scores can be exchanged in real time with other road users via a special communication interface between vehicles (V2V). This allows for interactions and competitions between people in different vehicles.

Participants not only see the scores of the passengers in their own car, but also the scores of the passengers in other nearby vehicles. This creates a shared gaming experience across vehicle boundaries. Groups in different cars can compete against each other.

Thus, the techniques according to the present disclosure may enable a more interactive and connected entertainment experience for occupants of a vehicle.

A vehicle includes an entertainment system in accordance with the present disclosure.

A corresponding method for providing or executing an entertainment application is provided, which may include at least some of the steps described in the context of the disclosed entertainment systems.

The method may also include one or more or any combination of features or steps described in connection with the entertainment systems disclosed herein.

The entertainment method may be a computer-implemented method executed, for example, by at least one shared or dedicated computing device of the vehicle and/or the entertainment system.

Such a computing device, for example a computing device or computer, which can be arranged on board a vehicle, includes a computing unit or processor, a storage unit or memory, an interface for receiving and providing sensor signals and user inputs, the storage unit containing commands that can be executed by the computing unit which, when executed by the computing unit, cause it to carry out the steps of any method or combination of methods in accordance with the present disclosure.

A computer program, or computer program product, comprises instructions that, when executed by a processor, cause the processor to perform the steps of any method or combination of methods according to the present disclosure.

For example, the computer program can include software that can be loaded into a memory of a programmable control device or a computing unit of a medical imaging system. With this computer program product, the methods described can be carried out when the computer program product is running in the control device. The computer program product may require program resources, e.g. libraries and auxiliary functions, in order to implement the methods described. In other words, the claim directed to the computer program product is intended to protect, in particular, software with which the methods described can be carried out. The software can be a source code (e.g. C++) that still needs to be compiled and bound or that only needs to be interpreted, or it can be an executable software code that only needs to be loaded into the corresponding computing unit or control device for execution.

An electronically readable storage medium includes instructions that, when executed by a processor, cause the processor to perform the steps of any method or combination of methods in accordance with the present disclosure. For example, the data and commands for carrying out the method according to the invention can be stored in a distributed database, in particular a cloud.

For example, the storage medium can comprise a DVD, a magnetic tape, an HDD or SSD hard drive or a USB stick on which electronically readable control information, in particular the software described, is stored. If this control information (software) is read from the storage medium and stored in a control device or computing unit of a vehicle, the methods described can be carried out.

The entertainment system, the computing device, the computer program or computer program product, and the computer-readable data storage are configured to perform any method or combination of steps according to the present disclosure.

For such vehicles, computing devices, computer programs and electronically readable storage media or cloud solutions, technical effects can be achieved that correspond to the technical effects for the entertainment system according to the present disclosure.

Although the features described in the summary above and the detailed description below are described in the context of specific examples, it is to be understood that the features may not only be used in the respective combinations, but may also be used in isolation or in any combination, and features from different examples may be combined and correlated with each other, unless expressly stated otherwise.

The above summary is therefore intended to provide only a brief overview of some features of some embodiments and implementations and is not intended to be limiting. Other embodiments may include features other than those described above.

Short description of the characters

The present invention is explained in more detail below using preferred embodiments with reference to the accompanying drawings.

• 1 zeigt eine schematische Darstellung Fahrzeugs mit einem Unterhaltungssystem, gemäß Ausführungsbeispielen der Erfindung.
• 2 zeigt eine schematische Darstellung der Komponenten des Fahrzeugunterhaltungssystems der 1, gemäß Ausführungsbeispielen der Erfindung.
• 3 zeigt eine schematische Illustration eines Fahrzeug-Unterhaltungssystems, gemäß Ausführungsbeispielen der Erfindung.
• 4 zeigt eine weitere schematische Illustration eines Fahrzeug-Unterhaltungssystems, gemäß Ausführungsbeispielen der Erfindung.

In the figures, the same reference numbers designate the same or similar elements. The figures are schematic representations of various embodiments of the invention. Elements shown in the figures are not necessarily drawn to scale. Rather, the various elements shown in the figures are reproduced in such a way that their function and general purpose is understandable to those skilled in the art.

• 1 shows a schematic representation of a vehicle with an entertainment system, according to exemplary embodiments of the invention.
• 2 shows a schematic representation of the components of the vehicle entertainment system 1 , according to embodiments of the invention.
• 3 shows a schematic illustration of a vehicle entertainment system, according to embodiments of the invention.
• 4 shows another schematic illustration of a vehicle entertainment system, according to embodiments of the invention.

Detailed description of implementation examples

In the following, concepts of exemplary embodiments of the invention are explained in more detail with reference to the accompanying drawings.

The above-described properties, features and advantages of this invention, as well as the manner in which they are achieved, will become clearer and more readily understood in connection with the following description of embodiments. It should be noted that the description of the embodiments is not to be understood in a limiting sense. The scope of the claimed solution is not intended to be limited by the embodiments described below or by the figures, which are for illustrative purposes only.

The drawings are to be considered schematic representations and the elements depicted in the drawings are not necessarily drawn to scale. Rather, the various elements are presented in such a way that their function and general purpose are apparent to a person skilled in the art. Any connection or coupling between functional blocks, devices, components, modules or other physical or functional units shown in the drawings or described herein may also be through a direct or indirect connection or coupling. A coupling between the components can also be established via a wireless connection. Function blocks can be implemented in hardware, firmware, software, or a combination thereof.

Various technologies for in-vehicle entertainment systems are described below, particularly with reference to a car. However, it should be understood that these technologies can also be used in other vehicle types and scenarios, such as: Similar systems could be installed in buses to provide entertainment to passengers, especially on longer journeys, or to provide contextualized, game-based information. In autonomous shuttles that operate on fixed routes, the technology could be used to provide information and locally adapted entertainment to waiting or driving passengers. Such a system can also be used for private entertainment in a motorhome or caravan. In rail transport, games and experiences adapted to the landscape and tailored to the travel route could make the time more enjoyable. The position sensors enable contextualization. Flying taxis with multiple passengers could use the technology to virtually complement the flight views and make them explorable for the occupant. Cruise ships are another possible application.

In some examples, the position and motion sensors also enable personalized, context-sensitive content and tasks, for example in a driving training vehicle to learn traffic situations in a playful way. By incorporating the real environment and movement, immersive, previously unknown experiences and interaction/learning opportunities are created. The methods described can therefore be transferred to many transport areas and learning/teaching areas.

1 shows a schematic representation of vehicle 1 with an entertainment system, according to exemplary embodiments of the invention.

As in 1 shown, the vehicle 1 includes several exterior sensors 3 that are configured to record vehicle surroundings data. Furthermore, a display device 4 is shown, which is assigned to a rear seat passenger, for example, but in other examples can also be assigned to the front passenger. The vehicle and/or the entertainment system contains a control unit 5. The vehicle also contains several vehicle interior sensors 2 for detecting and monitoring occupants or occupant activities. The control unit 5 is coupled to the exterior sensors 3 and interior sensors 2 and configured to receive and process the recorded sensor data to provide an interactive gaming experience for the occupants.

The exterior sensors 3 of the vehicle 1 1 may include, for example, cameras, radar sensors, lidar sensors or ultrasonic sensors. These are arranged at various locations around the vehicle, such as in bumpers, fenders or on the vehicle roof, and monitor the exterior of the vehicle. The exterior sensors thus generate a data stream of vehicle surroundings data that continuously monitor and characterize or represent the exterior of the vehicle.

The vehicle interior sensors 2 may include sensor types such as cameras, microphones, radar sensors and touch sensors and may be positioned inside the vehicle, such as in seats, doors or other surfaces. They are configured to detect occupant activities such as physiological characteristics, gestures, facial expressions or haptic interactions.

The display device 4, which can be assigned to occupants, can include a touch-sensitive display, loudspeakers and interfaces for virtual or augmented reality, such as data glasses. An occupant interacts with the entertainment system via the display device 4 and the entertainment system 4 interacts with the occupant of the vehicle 1.

The interactive vehicle entertainment system can have different architectures. In one possible design, the functions are distributed across several control units in the vehicle.

For example, a central infotainment module can provide the user interface via display and audio. Occupant detection is carried out by a separate unit that is linked to interior cameras and microphones. The processing of the vehicle surroundings data from the ADAS sensors is implemented on an assistance control unit. By merging this sensor data from different modules, an improved entertainment application can be provided.

An alternative is to bundle essential software components on a powerful system-on-a-chip of a central processing unit. This controls touch displays, processes camera and microphone streams using neural networks and analyzes radar/lidar object detection. External interfaces enable connection to the ADAS sensors. In a cloud-based version, sensor streams, user inputs and system states can be continuously transferred to servers in a data center. Scalable AI (artificial intelligence) modules run there for real-time analysis.

In general, modern computing platforms and high-performance chipsets of autonomous vehicles can execute complex algorithms such as deep learning in real time. By connecting to ADAS, positioning and communication systems, immersive, vehicle and user-adaptive entertainment concepts can be implemented.

2 shows a schematic representation of the components of the vehicle entertainment system of the 1 , according to embodiments of the invention.

As in 2 shown, the control unit 5 comprises a processor 6 and a memory 7, which can carry out the steps of the entertainment system according to the invention. The control unit 5 can be integrated, for example, in a control unit of the vehicle 1 or in or as a dedicated entertainment module of the vehicle.

The control unit 5 is coupled to the outside sensors 3 and the inside sensors 2 via a data connection 11. The outside sensors 3, as shown in 1 shown, can be part of a driver assistance system (ADAS) or an autonomous driving system of the vehicle 1 and data streams from these systems can be further used in the entertainment system.

A user interface of the display device 4 makes it possible to record user inputs from an occupant, e.g. via touchscreen, voice or gestures. The display device 4 also serves to output user feedback. The data streams of the interior sensors 2 can also be used to record and determine user feedback by the entertainment system.

The memory 7 contains instructions, in particular a computer program with instructions, which, when executed by the processor 6, provide functions of the entertainment system according to the invention.

3 shows a further schematic representation of a vehicle entertainment system according to embodiments of the invention.

Shown on the right is a vehicle interior 10, in particular a vehicle rear seat. A user makes an entry on a display device 4, in this case a tablet with a touchscreen, which is attached to the back of the driver's seat. Vehicles 8 can be seen on the tablet, which are located outside vehicle 1 in real time. The vehicles 8 have different vehicle types and colors.

For example, the task of the entertainment system for the user may include marking a vehicle of a certain type or color on the display. Since the vehicles 8 are shown synchronously in real time in the outside area and on the display 4, the user can make his selection as a touch input on the touchscreen, and also decide based on a view from the vehicle, for example. Based on the input, the selected vehicle, such as the left vehicle on the display device 4 here, can be provided with a label or a marking.

As a result, an evaluation of the user input is possible according to the techniques described in order to provide the user with a playful experience and provide user feedback about the accuracy of its selections.

3 shows a further schematic illustration of a vehicle entertainment system according to embodiments of the invention.

A vehicle interior 10 is shown on the right-hand side, in particular a vehicle rear seat. A user makes an input on a display device 4, in this case a tablet with a touchscreen, which is attached to the back of the driver's seat. The display device 4 can also be a head-up display, VR glasses or a device with gesture recognition.

The display shows several vehicles 8 in real time, which are located outside the vehicle 1 and are detected by the outside sensors 3, e.g. cameras, radar or lidar. The vehicles 8 have different vehicle types, e.g. different manufacturers or types, as well as different colors.

Possible tasks of the entertainment system for the user can include marking a vehicle of a certain type or color on the display. Since the vehicles 8 are shown synchronously in real time in the outside space 9 and on the display device 4, the user can make his selection as a touch input on the touchscreen of the display device 4.

Based on the input, the selected vehicle, such as the left vehicle here, can be provided with a label or a color marking. The marking can also be done based on the user's head movement, arm gesture or speech, which are recorded by interior sensors 2 and evaluated by a machine learning model.

As a result, an evaluation of the user input is possible according to the techniques described in order to provide the user with a playful experience and feedback on the accuracy of their selection. The control unit 5 compares the estimated with the actual object properties (determined by the entertainment system) and assigns a point value according to the match. Competition between several inmates is therefore possible.

4 shows a schematic illustration of a vehicle entertainment system according to embodiments of the invention.

As in 4 shown, there are two vehicles 8 in the outside area 9 of an autonomous minibus 1. In the interior 10 of the minibus 1 there is an occupant and user 12 of the entertainment system. The entertainment system is integrated in vehicle 1.

If the occupant 12 interacts with the system by pointing at one of the vehicles 8, this is detected by interior sensors 2, in particular cameras. The sensor data is evaluated by a machine learning model to determine the vehicle 8 selected by the user.

If, for example, the right vehicle 8 has been selected by the occupant 12, this can be visualized for the user via an augmented reality display using a colored or graphic marking. The marked vehicle appears visually highlighted for the user on a display or via AR glasses.

Furthermore, according to the invention, the user input, i.e. the selection of the vehicle, can be compared with the attributes or properties of the vehicle determined by the system. Based on the agreement or non-agreement between the estimated and the actual object property, feedback can be given to the user 12, for example via augmented reality or to a user terminal. This enables an interactive, playful concept in a multi-person environment, such as public transport.

• Objekte, wie andere Fahrzeuge in der Umgebung des Egofahrzeugs, und Personen in der Fahrzeugumgebung werden durch Aussenraumsensoren (Kamera, Radar, Lidar) erfasst. Durch Auswertung der Sensordaten mit Hilfe von maschinellem Lernen lassen sich Objekteigenschaften bzw. -attribute für die Objekte bestimmen, beispielsweise Fahrzeugtyp (Limousine, LKW etc.), Farbe, Grösse oder Position, oder mit einem Fahrzeug assoziierte Verkehrssituation.

From the above, some general conclusions can be drawn:

• Objects, such as other vehicles in the vicinity of the ego vehicle, and people in the vehicle's surroundings are detected by external sensors (camera, radar, lidar). By evaluating the sensor data with the help of machine learning, object properties or attributes can be determined for the objects, for example vehicle type (limousine, truck, etc.), color, size or position, or traffic situation associated with a vehicle.

In the game context, the task can now be that an occupant of a specific detected vehicle selects, thus determines or specifies the vehicle in the real environment by typing, voice command or gesture on the display. The user input may thus include an object identification to select an object with a predetermined object property.

This selection means that the vehicle with a specific property specified by the system as a task (eg blue BMW) has been determined and selected by the user. Through his interaction, ie user input, with the entertainment system, the vehicle occupant selects an object or a group of objects in the real vehicle environment. This is done as previously described by tapping, gesture or voice command.

This selection made thus represents a determination of an object and/or an associated predetermined object property by the user. By marking a blue car, for example, the occupant simultaneously estimates and determines that the corresponding vehicle actually has the color property “blue”.

In other words, the user selects a real object and thereby tags it with a suspected object property. Through his interaction, he tells the system which element he believes has a certain, predetermined or self-selected characteristic - in this example: blue color.

This assignment of an estimated property to a real object by the vehicle occupant is recorded and evaluated by the entertainment system. In a subsequent processing step, it is analyzed whether the user's assumption about the object property is correct.

For this purpose, the actual properties of the object selected by the user, determined by sensors, are used. The automatically extracted attributes are compared with the properties estimated by the occupant, i.e. the properties implicitly determined by the object selection.

This comparison between estimated and real object characteristics then forms the basis for appropriate scoring and feedback in the interactive game. This makes it possible to assess whether the user was correct in his selection and assessment of the object's properties.

The software compares the estimated properties of the selected object assumed by the user with the actual properties determined by the sensor processing.

If these match (the user has correctly selected the blue BMW), points are awarded or feedback is given accordingly. Otherwise (user has chosen the wrong car) this will be reflected in the scoring.

The interactive vehicle entertainment system includes a display device (e.g. display, AR glasses) for displaying a game using real-time use of the analyzed environmental data.

There are different user interfaces for receiving input: touchscreen, voice input, gesture recognition via indoor sensors.

Object properties of the recorded vehicles, such as color, license plate and vehicle type, can be determined and used for the game.

Real-time processing and synchronization of the sensor streams from outdoor and indoor sensors is essential. This enables interactive game concepts linked to the real scenery.

The user input in the game may include selecting a specific object (e.g., vehicle) if the game task includes selecting an object with a specific object property determined by the system (e.g., blue Tesla).

By typing, gesturing or speaking, the user marks the object that he or she assumes corresponds to the required property.

In the subsequent evaluation of the software, it is checked whether the user input represents the correct object selection according to the game specifications. Points are awarded or feedback is provided accordingly.

This clarification makes it clear that user input as part of the game principle can include object selection based on a given attribute value.

By comparing the estimated properties with the actual object properties, playful feedback or scoring can be realized for the users.

A competition between several drivers with points being awarded is possible. Cross-vehicle game variants using V2V communication are also conceivable.

In summary, the techniques described enable an immersive, interactive entertainment experience in the vehicle context that integrates exterior sensors, interior sensors and user input in real time into an entertainment application so that a user can interact with the real-time vehicle environment. The game concepts described and the interaction of the rear occupants with the entertainment system can minimize driver distraction. Children in particular, who often distract the driver due to boredom, can be kept busy and entertained by the interactive system. This increases safety in road traffic because the driver can concentrate better on the traffic and there are fewer distractions in the interior. Another option for greater safety is to link the entertainment system to driver assistance systems. For example, warnings or sounds in the game could be weakened as soon as critical driving situations are detected by the vehicle system. The entertainment system therefore also contributes to greater attention and safety for the driver by reducing distractions in the interior. Complex situational tasks can be trained by machine, so that the system can be used, for example, as a (vehicle-to-vehicle) driver training system.

Claims (10)

An interactive entertainment system for a vehicle, comprising a control unit configured to perform the following steps: - Receiving vehicle surroundings data which was recorded by at least one vehicle exterior sensor of the vehicle; - Determine, based on the vehicle surroundings data, at least one object property of at least one object which is located in the vehicle surroundings; - Obtaining a user input from at least one vehicle occupant for the at least one object, which determines an estimated object property of the at least one object; - comparing the determined object property and the estimated object property to determine user feedback; - Outputting the user feedback to the at least one vehicle occupant. entertainment system Claim 1 , wherein the at least one vehicle exterior sensor is a component of an autonomous vehicle system or an Advanced Driver Assistance System (ADAS), and comprises one or more of a camera, a radar sensor, a lidar sensor and an ultrasonic sensor . entertainment system Claim 1 or 2 , wherein the vehicle environment data includes a real-time data stream from the at least one vehicle exterior sensor. An entertainment system according to any preceding claim, wherein receiving user input comprises: - Displaying the at least one object and the object property for the vehicle occupant on a display device in the interior of the vehicle, which is assigned to the vehicle occupant; and - Receiving the user input on the display device based on the at least one object displayed thereon. An entertainment system according to any preceding claim, wherein receiving user input comprises: - Receiving vehicle interior data which is detected by at least one vehicle interior sensor; and - Determine user input using vehicle interior data through a machine learning model. Entertainment system according to Claim 5 , wherein the at least one vehicle interior sensor is configured to monitor occupant activities within the interior of the vehicle, and comprises one or more of a camera, a radar sensor, a microphone, and a touch sensor disposed within the interior of the vehicle. Entertainment system according to one of the preceding claims, wherein the user feedback for the at least one vehicle occupant - on the display device according to Claim 4 is output optically and/or acoustically, or - is output via a virtual reality (VR) or augmented reality (AR) interface of the vehicle, or - is output via a wireless data connection to a user terminal assigned to at least one vehicle occupant. Entertainment system according to one of the preceding claims, wherein the control unit is further configured to: - Receiving a second user input from a second vehicle occupant, the second user input comprising a second estimated object property of the at least one object; - comparing the determined object property and the second estimated object property to determine second user feedback for the second vehicle occupant; and - Maintaining individual scores for the first and second vehicle occupants based on the accuracy of their respective estimated object characteristics compared to the determined object characteristic. entertainment system Claim 8 , further comprising vehicle-to-vehicle communication cation interface, wherein the control unit is further configured to: - send score data to another vehicle via the vehicle-to-vehicle communication interface; - receiving score data from the other vehicle via the vehicle-to-vehicle communication interface; and - enabling a competitive interactive gaming experience between vehicle occupants of the vehicle and vehicle occupants of the other vehicle using the sent and received score data. A vehicle comprising an entertainment system according to any preceding claim.

Images