DE102022108774A1 - User interface, means of transport and method for operating a heating/air conditioning system of a means of transport - Google Patents

Abstract

A user interface, a means of transport (10) and a method for operating a heating/air conditioning system of a means of transport (10) by an occupant (4) of the means of transport (10) are proposed. The method comprises the steps: determining a representation of a surface of an interior of the means of transport (10), loading a virtual control element (20) for the heating/air conditioning system, displaying the virtual control element (20) in a predefined spatial relationship to the surface using an augmented Reality (AR) display device (2), determining a predefined interaction of the occupant (4) with the virtual control element (20) by means of a sensor (3) assigned to the augmented reality display device (2), and in response thereto sending a control command by means of the augmented reality display device (2) to the means of transport to influence the heating/air conditioning system in accordance with the predefined interaction.

Description

The present invention relates to a user interface, a means of transport and a method for operating a heating/air conditioning system of a means of transport by an occupant of the means of transport. In particular, the present invention relates to an ergonomic and intuitive way to adjust a setting of a heating/air conditioning system.

In vehicles, heating/air conditioning systems are usually adjusted using controls located in a center console or dashboard. Voice commands are also increasingly being used to adjust fan speed and target temperature.

Furthermore, augmented reality (AR) display devices are known, by means of which additional display/control elements calibrated to the viewer's surroundings can be displayed.

Based on the aforementioned prior art, it is an object of the present invention to make the operation of a heating/air conditioning system in a means of transport ergonomic and intuitively understandable.

Disclosure of the invention

The aforementioned object is achieved according to the invention by a method for operating a heating/air conditioning system of a means of transport by an occupant of the means of transport. The means of transport can be, for example, a car, van, truck, motorcycle, watercraft and/or aircraft. The occupant can be, for example, a driver and/or a front passenger, in particular a rear passenger or another passenger. In a first step, a data record for the heating/air conditioning system representing a virtual control element is loaded. For this purpose, an optical representation or a related data set is loaded from a data memory and the optical representation is preferably adapted to a current setting of the heating/air conditioning system. For this purpose, the optical representation can be defined in terms of shape, color, content, etc., in particular also in terms of its size. In addition, the virtual control element is displayed in a predefined spatial relationship to a surface of the means of transport, in particular to a dashboard and/or an air vent of the heating/air conditioning system, using the augmented reality display device. A virtual control element that is fundamentally stationary with respect to the means of transport and not necessarily with respect to its surroundings is therefore displayed. If the spatial position of the virtual control element represents, for example, a target value of the heating/air conditioning system, the virtual representation can take place directly at the corresponding spatial position. The virtual representation can, so to speak, “stick” to an vent of the heating/air conditioning system and extend into the room. In particular, the virtual representation can correspond spatially with air emerging directly from the vent, so that a user who interacts with the virtual representation using his hand also has the haptic experience of the air flow. If a visual appearance of the virtual control element also varies depending on a current target value, the visual appearance can be adapted in a corresponding manner to the currently set target value. In addition, an interaction of the occupant with the virtual control element is then determined using a sensor (in particular an optical sensor) assigned to the augmented reality display device. This determination is carried out automatically and/or using sensors. In particular, it can be determined that the occupant's hand penetrates into a spatial area of the means of transport in which the virtual display of the virtual control element takes place. It is determined, so to speak, that the user wants to interact with or manipulate the virtual control element. In response, a control command is sent to the means of transport by means of the augmented reality display device to influence the heating/air conditioning system in accordance with the predefined interaction. In other words, a corresponding control command is generated depending on the manner in which the occupant interacts with the virtual control element. For example, interacting may include moving the virtual control element, rotating, pinching, pulling apart, or the like. Depending on the design, target values of the heating/air conditioning system can be adjusted conveniently and intuitively.

The subclaims show preferred developments of the invention.

Optionally, in a previous step, the surface in the interior of the means of transport can be optically scanned. This step can also be understood as scanning or measuring/locating the surface of the interior. It can take place (automatically) in the respective vehicle, for example after a first pairing of an augmented reality display device with the means of transport. Alternatively or additionally, this step may have been carried out at the factory to provide an initial model for the surface ab initio. This model can be used, for example, in augmented reality Display device stored, transmitted to it by the means of transport or kept available for download from a server. For example, the scanning process can also be carried out using a smartphone carried by the user. In particular, an optical sensor of an augmented reality display device worn by the user is used for this purpose. The scanning step can be carried out every time the means of transport and/or the augmented reality display device is put into operation in order to be able to correspond to any changes in function, geometry and color of the surface that may have been made in the meantime.

The virtual control element can be adjusted in the vertical direction, for example, to increase a target value. The target value can be, for example, a target temperature.

Alternatively or additionally, the target value can be an air flow speed. In particular, the target value can be a seating position-specific or even vent-specific target size. For this purpose, several virtual control elements, one at a sitting position or at an air vent, can be provided in the means of transport and used in the manner according to the invention.

The virtual control element can also be operated rotationally, for example by the user gripping the control element with two fingers and allowing his fingers to rotate around a virtual center position of the control element. In this way, too, the representation of the virtual control element can be adjusted and, in addition to the rotation position, given a new color and/or an otherwise qualified optical appearance.

The representation of the virtual control element can also be reduced to a minimum, so that when the virtual control element is operated, the impression is created that one is interacting with the real control element of the vehicle to which the virtual control element is assigned.

The virtual control element can, for example, be designed as an animation. In particular, an air flow can illustrate air emerging as streamlines from the vent being served. When the occupant pulls this animation towards them, they increase the airflow, while when they push the airflow away (into the air vent), they decrease the airflow strength. The visual appearance of the streamlines can be lengthened/enlarged or shortened/reduced accordingly.

If motorized and/or automatically adjustable air guidance means are provided, the virtual control element can be used to generate control commands for the air guidance means (slats, duct alignment, etc.). As described above, operation can take place through direct haptic interaction of the user with the virtual streamlined air flow control element. If the occupant raises their hand, the airflow is directed upwards; if they lower their hand, the airflow is directed downwards. The same applies to left and right. In other words, the airflow “follows” the user’s limb, as does the virtual control element that illustrates it. As long as the pinching gesture lasts and has sufficient spatial reference to the virtual control element, the user essentially “shifts” the main direction or center of gravity of the air flow.

The operation and illustration using the virtual control element can relate to the spatially (directly) addressed air vents or to several air vents selected or assigned to the user's sitting position or to all air vents of the means of transport. This may depend on the parameter. While the temperature often relates to several sitting positions, the air flow direction has so far usually only been operated by the person affected by the changed air flow. However, by illustrating the streamlines according to the invention, it also makes sense to adjust the air flow direction for other occupants (e.g. children) by a user (e.g. a parent) equipped with the augmented reality display device, who can see the effect of his setting through the augmented reality -Display device can be detected visually immediately.

To start the method according to the invention, the user can interact with the heating/air conditioning system in a conventional manner. For example, he can operate a button (button) intended for this purpose or a button on a touchscreen, which relates to the setting of the heating/air conditioning system. In response to this, the virtual control element can be displayed as an additional option using the augmented reality display device in order to be able to carry out the continuous operation of the heating/air conditioning system in a particularly convenient and intuitive manner according to the invention. Alternatively, simply focusing (possibly supplemented by an additional confirmation input, such as blinking, nodding, clicking in the air, etc.) can also be used as an opportunity to start the heating/air conditioning control using the augmented reality display device. In other words, by focusing or When the occupant interacts with the conventional control element, the virtual control element is displayed using the augmented reality display device. This does not preclude the user from being able to use a voice command or other input to start the method according to the invention.

The method according to the invention can preferably be further developed by automatically/sensor-technically determining a limb of the occupant in a display area of the augmented reality display device. In other words, a user's limb is recognized and acknowledged in the detection range of the optical sensor of the augmented reality display device. Acknowledgment can be done, for example, by displaying a cursor in the area of the limb, in particular in the area of the fingertips of one of the occupant's hands. In principle, however, a sensor of the means of transport can also detect the limb in the detection area or in the area of the occupant's line of sight and send a corresponding signal to the AR display device. Alternatively or additionally, the occupant's limb can be determined at a predefined spatial distance from the virtual control element. In other words, depending on a current position of the virtual control element, the user can be given feedback that this hand is in the vicinity of an operable, virtual control element and the occupant can thus be encouraged to interact with the virtual control element. The displayed cursor can, for example, be designed visually depending on a recognized finger position of the user's hand. For example, fingers that are spread apart can be acknowledged by a first optical appearance of the cursor, while fingers that are brought together (pinched) can be acknowledged by a second optical appearance that differs from the first optical appearance. This means the user knows how the hand was currently detected by the optical sensor and can adjust the finger position if necessary or interact directly with the virtual control element.

Preferably, the interface according to the invention can use the optical sensor to recognize a predefined gripping gesture of the occupant in a predefined, spatial relationship to the virtual control element. For example, the gripping gesture can always refer to the closest virtual control element or to the virtual control element to which it at least does not exceed a predefined distance. In response to determining the grasping gesture, the virtual control element and/or the cursor can be optically modified. In this way, the user is given feedback that his hand is now assigned to the virtual control element in question and that future movements/gestures using the limb will relate to the selected virtual control element.

According to a second aspect of the present invention, a user interface is proposed for operating a heating/air conditioning system of a means of transport by an occupant of the means of transport. The user interface is designed, for example, as an electronic control device or can be implemented in an electronic control device (in particular in addition to other functions). The user interface has a data input, an evaluation unit and a data output. In this way, the user interface is set up to use the data input to optically scan a surface of an interior of the means of transport or to receive corresponding data from the means of transport, a smartphone or another data storage device, in particular a server. This process could also be understood as a scan, modeling of a component of the interior surface of the means of transport or as a calibration of the user interface. In addition, a virtual control element for the heating/air conditioning system is loaded via the data input. In particular, a design corresponding to a current target setting of the heating/air conditioning system and in particular also a room position corresponding to the target value for displaying the virtual control element are predefined. Using the data output, the virtual control element can be displayed in a predefined, spatial relationship to the surface using an augmented reality display device.

The augmented reality display device is in particular a device fixed to the user's head, which has a transparent display device or a transparent screen. Such augmented reality display devices can be referred to as augmented reality glasses or smart glasses. These often have an optical sensor in their housing, so that a predefined spatial relationship is always guaranteed between the display device, a user's head position and the optical sensor. Using the data input, a predefined interaction or manipulation of the virtual control element by the occupant can also be determined. The assigned sensor can optically detect and locate a limb of the user. The gesture/manipulation determined by the sensor now refers to the virtual control element that is virtually closest to the limb during the determined gesture/manipulation. Subsequently, in Depends The ability of the gesture/manipulation generates predefined control commands using the augmented reality display device in order to influence the mode of operation, in particular target parameters of the heating/air conditioning system. In particular, the means of transport can receive these control commands from the augmented reality display device wirelessly or by wire and, in turn, can adapt the heating/air conditioning system and its operating parameters. In this way, the features, combinations of features and the resulting advantages of the method according to the invention are clearly implemented in a corresponding manner, so that reference is made to the above statements to avoid repetition.

According to a third aspect of the present invention, a means of transportation is proposed which has a user interface according to the second-mentioned aspect of the invention. The means of transport can be designed as a car, van, truck, motorcycle, aircraft and/or watercraft. Through the user interface according to the invention, the means of transport is also able to realize the features, combinations of features and advantages of the above aspects.

• 1 eine schematische Darstellung eines erfindungsgemäß ausgestalteten Fortbewegungsmittels, an dessen Fahrerarbeitsplatz ein Anwender ein Ausführungsbeispiel einer erfindungsgemäßen Anwenderschnittstelle zur Ausführung eines erfindungsgemäß ausgestalteten Verfahrens verwendet;
• 2 eine schematische perspektivische Ansicht eines Armaturenbretts mit virtuellem Bedienelement in einem ersten Betriebszustand;
• 3 eine schematische perspektivische Ansicht eines Armaturenbretts mit virtuellem Bedienelement in einem zweiten Betriebszustand;
• 4 eine schematische perspektivische Ansicht eines Armaturenbretts mit virtuellem Bedienelement in einem dritten Betriebszustand; und
• 5 ein Flussdiagramm veranschaulichend Schritte eines Ausführungsbeispiels eines erfindungsgemäßen Verfahrens zur Bedienung einer Heiz-/Klimaanlage eines Fortbewegungsmittels durch einen Insassen.

The invention is explained in more detail below using exemplary embodiments. Show this:

• 1 a schematic representation of a means of transportation designed according to the invention, at whose driver's workplace a user uses an exemplary embodiment of a user interface according to the invention to carry out a method designed according to the invention;
• 2 a schematic perspective view of a dashboard with a virtual control element in a first operating state;
• 3 a schematic perspective view of a dashboard with a virtual control element in a second operating state;
• 4 a schematic perspective view of a dashboard with a virtual control element in a third operating state; and
• 5 a flowchart illustrating steps of an exemplary embodiment of a method according to the invention for operating a heating/air conditioning system of a means of transport by an occupant.

Embodiments of the invention

1 shows a perspective view of a passenger workplace of a means of transport 10 designed according to the invention, in which an occupant 4 has taken a seat as a passenger. The occupant 4 wears an augmented reality (AR) display device 2 in the form of AR glasses, through whose optical sensor 3 in the form of a 2D camera the field of vision of the occupant 4 is optically recorded. The signals from the optical sensor 3 are fed to an electronic control unit 9 via a data input 8. This also receives any necessary data and instructions from a local data memory 5 via the data input 8. Data processed by the electronic control unit 9 can be sent via the data output 11 to a transparent display of the AR display device 2. In addition, an information technology link to the means of transport 10 can be carried out via an antenna (not shown). For this purpose, the means of transport 10 also has an antenna 13 and an electronic control device 9 linked to it. A data memory 5 holds target parameters of the heating/air conditioning system.

An air vent 22 is arranged in the dashboard 1 of the means of transport 10. To illustrate the current air flow, a virtual control element 23 is shown in the form of streamlines. The streamlines are displayed at the position that was recognized by the optical sensor 3 as the area in front of the air vent 22. In addition to the air vent 22, another virtual control element 24 is shown in the form of a ball, which floats above the dashboard 1. A plus sign is visualized at the top of the ball, while a minus sign is visualized below the ball. The occupant 4 can now raise or lower the target temperature by raising or lowering the virtual control element. In the illustration shown, a limb 12 in the form of a hand of the occupant 4 is determined in the area of the streamlines by means of the optical sensor 3. In response, a cursor 25 is virtually illustrated in the area of the fingertip, which informs the occupant 4 that he can use the virtual control element 23 to adjust a target air flow strength of the air vent 22. If the occupant 4 pulls the cursor 25 towards him (double arrow P), the target air flow intensity is increased; if he pushes the cursor 25 in the direction of the air vent 22, the target air flow intensity is reduced. The occupant 4 can start the display of the virtual controls 23, 24 by focusing on conventional controls 26, 27 for the target temperature and the target air flow intensity. Either after a short timer (in the range 0.5 to 2 seconds) has elapsed after the start of the Focusing the control elements 26, 27, the display of the virtual control elements 23, 24 is started or by confirming the focusing via blinking, nodding the head, acoustic input or similar.

2 shows the view of the in 1 occupants shown on a dashboard 1 of a means of transport 10 designed according to the invention, into which an air vent 22 is embedded. After the occupant has initiated the interaction according to the invention, a virtual control element 23 is shown in the form of streamlines in front of the air vent 22, while on the left in the picture a virtual control element 24 is shown with a pictogram symbolizing average temperatures. Above the pictogram, the current target temperature (22.5 °C) and indicators “Plus” for increasing the target temperature by raising the virtual control element 24 and “Minus” for lowering the target temperature by lowering the virtual control element 24 are shown .

3 shows the in 2 presented view after the occupant has performed a pinching gesture in the spatial area of the virtual control element 24 with a limb 12 in the form of his hand. A cursor 25 is displayed in the area of the fingertips of the limb 12 to identify the position which is understood by the user interface to be manipulated. By lifting in the direction of an arrow P, the user has increased the target temperature to 23.0 °C. To maintain this temperature, he must stop the pinching gesture by moving his index finger and thumb apart.

4 shows the in 2 presented situation after the user has lowered the target temperature to 16 ° C by lowering the virtual control element 24 and his right hand as a limb 12 performs a pinching gesture in the area of the streamlines in front of the air vent 22. Again, the cursor 25 shows the user the position that is affected by a horizontal movement carried out while maintaining the pinch gesture. According to the double arrow P, the user can increase the amount of air flow by pulling his limb 12 and reduce it by moving closer to the air vent 22. If the occupant raises their hand, the airflow is directed upwards; if they lower their hand, the airflow is directed downwards. The same applies to left and right. In other words, the user “shifts” the main direction or “focus” of the airflow during the pinch gesture. If he releases his pinching gesture by moving his index finger and thumb apart, the currently set amount of air flow is retained and he can remove his limb 12 from the detection range of the optical sensor.

5 shows steps of an exemplary embodiment of a method according to the invention for operating a heating/air conditioning system of a means of transport by an occupant. In step 100, a surface of an interior of the means of transport is scanned or scanned using an optical sensor. In step 200, a virtual control element or a data record representing such is loaded for the heating/air conditioning system. In step 300 it is now determined that the occupant focuses on a conventional control element for influencing the operation of the heating/air conditioning system and performs a confirmation gesture by tapping in the air at a position in front of an air vent. In response, in step 400, a virtual control element is displayed in a predefined, spatial relationship to the scanned surface using an augmented reality display device. In particular, the scanned surface can be characterized by a grill of an air vent or a position of a button/control element for influencing the operation of the heating/air conditioning system. In step 500, an interaction of the user by gripping and rotating or moving the virtual control element is then determined using sensors. This is done using an optical sensor of the augmented reality display device. Last but not least, in step 600, a control command is generated which corresponds to the occupant's interaction with the virtual control element and is intended to cause the parameters of the heating/air conditioning system to be adjusted. For this purpose, the augmented reality display device establishes a wireless connection to the means of transport, via which the control command is sent to the means of transport and the heating/air conditioning system is influenced in the desired manner.

Reference symbol list

1

dashboard

2

Augmented reality display device

3

Optical sensor

4

inmate

5

Data storage

8th

Data input

9

Electronic control unit

10

Means of transportation

11

Data output

12

Inmate's limb

13

antenna

22

Air vents

23, 24

Virtual control element

25

cursor

26, 27

Conventional controls

100 to 600

Procedural steps

P

Double arrow

Claims (11)

Method for operating a heating/air conditioning system of a means of transport (10) by an occupant (4) of the means of transport (10) comprises the steps: • Loading (200) of a virtual control element (20) for the heating/air conditioning system, • Representing (400) the virtual control element (20) in a predefined spatial relationship to a surface using an augmented reality display device (2), • Determining (500) a predefined interaction of the occupant (4) with the virtual control element (20) by means of a sensor (3) assigned to the augmented reality display device (2), and in response thereto • Sending (600) a control command using the augmented reality display device (2) to the means of transport to influence the heating/air conditioning system in accordance with the predefined interaction. Procedure according to Claim 1 , wherein the virtual control element (20) is designed to be displaceable in the vertical, in particular exclusively in the vertical direction. Procedure according to Claim 1 or 2 , wherein the virtual control element (20) is designed to be displaceable in the horizontal, in particular exclusively in the horizontal, direction. Method according to one of the preceding claims, wherein the virtual control element (20) represents a target temperature. Method according to one of the preceding claims, wherein the virtual control element (20) represents an air flow speed and/or an air flow direction. Method according to one of the preceding claims further comprising - Determining (300) whether the occupant (4) is focusing and/or interacting with respect to a predefined area in the interior of the means of transport (10) and in response thereto - Displaying the virtual control element (20). Method according to one of the preceding claims further comprising - Determining a limb (12) of the occupant (4) - in a display area of the augmented reality display device (2) and/or - at a predefined spatial distance from the virtual control element and in response to it - Automatic display of a cursor (21) in the area of the limb (12). Procedure according to Claim 7 further comprising - determining a predefined gripping gesture of the occupant (4) in a predefined spatial relationship to the virtual control element (20) and in response thereto - optically modifying the virtual control element (20) and / or the cursor (21). Method according to one of the preceding claims further comprising - Optical sensor-based scanning (100) of the surface of an interior of the means of transport (10) and/or - Loading a data representation of the surface. User interface for operating a heating/air conditioning system of a means of transport (10) by an occupant (4) of the means of transport (10), comprising: - a data input (8), - an evaluation unit (9) and - a data output (11), whereby the user interface is set up, • use the data input to determine a representation of a surface of an interior of the means of transport (10) and • to load a virtual control element (20) for the heating/air conditioning system, • using the data output (11) to display the virtual control element (20) in a predefined spatial relationship to the surface using an augmented reality display device (2), • using the data input (8) to determine a predefined interaction of the occupant (4) with the virtual control element (20) using a sensor (3) assigned to the augmented reality display device (2), and • using the data output (11) to send a control command using the augmented reality display device (2) to the means of transport to influence the heating/air conditioning system in accordance with the predefined interaction. Means of transport comprising a user interface Claim 10 .

Images