DE102020106021A1 - METHOD AND SYSTEM FOR OPERATING A SELECTION MENU OF A GRAPHIC USER INTERFACE BASED ON THE CAPTURE OF A ROTATING CLEAR GESTURE - Google Patents

Abstract

The invention is directed to a method and a corresponding system for operating a rotation menu of a GUI which is based on the detection of a free space gesture. The method comprises: (i) detecting a rotating free space gesture with the aid of a free space gesture detection sensor, which rotating free space gesture is performed by a user; (ii) displaying on a display device at least a portion of a rotation menu, the rotation menu having a plurality of individually selectable menu items arranged in a sequential order along an arc having a specific predefined selection position along the arc; (iii) the synchronous and uniform scrolling of the sequence of menu items in the GUI along the arc in response to the detected gesture, so that the scrolling speed is aligned with a detected rotational speed and direction of rotation of the detected gesture according to a predefined transformation ratio; (iv) determining a start and an end of the detected gesture and an accumulated rotating scrolling angle or a distance between a starting point and an end point of the scrolling movement; (v) if the accumulated scroll angle or the accumulated scroll distance reaches or exceeds a respective predefined threshold value, the automatic scrolling of the sequence of menu items by a remaining angle or distance, so that the next menu item along the direction of rotation to the Selection position is moved, and otherwise automatically reversing the scrolling movement of the sequence of menu items in order to bring the last menu item which previously occupied the selection position back to the selection position; and (vi) determining this particular menu item as a currently selected menu item or a menu item selected in the sense of a preselection, which is located at the selection position after the automatic scrolling has ended.

Description

The present invention relates to the field of human-machine interfaces (HMI), for example for vehicles such as automobiles. In particular, the invention is directed to a method and a system for operating a selection menu of a graphical user interface (GUI) that is based on the detection of a rotating free space gesture carried out by a human user, for example by a driver or another occupant of the vehicle .

Modern HMIs, including those for vehicles, are no longer limited to providing various pushbuttons or switches or other conventional input devices for receiving user input. In particular, in addition to touch-sensitive surfaces, particularly touchscreens, some automakers have recently added the ability to capture free space gestures performed by a human user to the range of available input methods for automotive HMIs. Typically, such gestures are captured by a single 2D or 3D camera in the vehicle when they are performed in the camera's field of view and are used to control a functionality, such as the volume, of an entertainment system, for example an infotainment system Vehicle or to switch a light in the vehicle on or off. Non-automotive applications in which HMIs based on the detection of free space gestures can be used include, in particular, HMIs for medical devices or devices for entertainment electronics.

Another trend that has recently become more prominent is the taking of so-called “selfies”. H. of digital snapshots, i.e. photos or videos, of yourself and optionally also of others, often for use in social networks. Electronic hand-held devices such as smartphones or tablet computers are mostly used for this purpose. While capturing and browsing such images has become ubiquitous in stationary environments such as restaurants, offices, or homes due to the use of such handheld electronic devices, capturing and browsing images could be the same in a moving vehicle such as in a car, for example, are in conflict with the safety requirements and the given spatial restrictions, especially if the driver is supposed to take the pictures or browse the pictures himself or to be in the pictures himself. Correspondingly, fixed image sensors of various types can be integrated at one or more suitable locations in the vehicle itself together with the corresponding HMIs for their operation and for the handling of images recorded with them. Browsing or selecting images or other objects in an HMI, however, remains a challenge in such an environment, including a security challenge.

The present invention is based on the object of providing an easy-to-use HMI for receiving user inputs that can be reliably and safely operated by a human user even in a safety-relevant environment, such as a moving vehicle.

A solution to this problem is provided by the teaching of the independent claims. Various preferred embodiments of the present invention are provided by the teaching of the dependent claims.

A first aspect of the invention is directed to a method for operating a rotating menu of a graphical user interface, GUI, which is based on the detection of a free space gesture carried out by a human user. The method comprises: (i) detecting a rotating free space gesture with the aid of a free space gesture detection sensor, for example a continuous arm, forearm or finger rotation movement carried out by a human user in a field of view of the free space gesture detection sensor; (ii) displaying on a display device at least a portion of a rotation menu, the rotation menu having a plurality of individually selectable menu items arranged in a sequential order along an arc having a specific predefined selection position along the arc; (iii) synchronously and uniformly scrolling the sequence of menu items in the GUI along the arc in response to the detected rotating free space gesture, so that the scrolling speed is aligned according to a predefined transformation ratio to a detected rotational speed and direction of rotation of the detected rotating free space gesture; (iv) determining a start and an end of the detected rotating free space gesture and an accumulated rotating scroll angle or a distance between a starting point of the scrolling movement of the menu items, which corresponds to the start of the gesture, and an end point of the scrolling movement of the menu items, which corresponds to the end of the gesture; (v) if the accumulated scrolling angle or the accumulated scrolling distance reaches or exceeds a respective predefined threshold value, automatic scrolling of the sequence of menu items by a remaining angle or distance, so that the next Menu item is moved along the direction of rotation to the selection position, and otherwise automatically reversing the scrolling movement of the sequence of menu items in order to bring the last menu item which previously occupied the selection position back to the selection position; and (vi) determining this specific menu item as a currently selected menu item or a menu item selected in the sense of a preselection, which is located at the selection position after the automatic scrolling has ended.

The term “free space gesture” as used here relates in particular to a physical movement or a state of a human user, i. H. a gesture performed in ordinary three-dimensional space to control or otherwise interact with one or more devices, such as one or more screens that display a GUI, without having to physically touch them.

Accordingly, the term “free space gesture detection sensor” as used here relates to a sensor that is configured to detect a free space gesture. Such a sensor can in particular, and without this should be viewed as a restriction, have a camera of the “Time of Flight” (TOF) type.

The term “field of view” (FOV) of a sensor, as it is used here, refers to the spatial extent of the observable world that can be recorded by the sensor at a given point in time. In the case of optical instruments or sensors, for example cameras, it is usually a solid angle over which a detector is sensitive to electromagnetic radiation.

As used herein, the term “arc” refers to a curve or line that is not straight, at least not everywhere. In particular, an arc can be defined on a two-dimensional surface and bent in only one direction. In particular, an arc, without this should be viewed as a restriction, can be an arc of a circle. An arc can also be a closed loop of any shape, for example a complete circle or an ellipse.

The terms “first”, “second”, “third” and the like in the description and in the claims are used to distinguish between similar elements and not necessarily to describe a sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein may be carried out in sequences other than those described or illustrated herein.

If the term “have” or “include” is used in the present description and claims, this does not exclude further elements or steps. When an indefinite or a definite article is used where it refers to a singular noun, for example “a” or “an”, “der / die / das”, this includes a plural of that noun unless specifically stated otherwise other is indicated.

Furthermore, unless expressly stated otherwise, “or” refers to an inclusive or rather than an exclusive or. For example, a condition A or B is met by any of the following: A is true (or present) and B is not true (or not present), A is not true (or not present), and B is true (or present) , and both A and B are true (or present).

The above-mentioned method offers an easy-to-use and easy-to-learn contactless HMI in the form of a GUI for receiving user inputs in the form of free space gestures, more precisely rotating free space gestures, with which a selection or pre-selection of a menu item from a menu of several on a display of the HMI displayed elements can be performed reliably and safely by a human user even in a safety-related environment, such as a moving vehicle. This is particularly crucial for achieving a high level of user satisfaction for such an input method.

In addition, the use of rotating free space gestures as the basis for the (pre) selection process has the advantage that a human user can typically perform such a rotating gesture in an “endless” manner, in contrast to most other gestures (e.g. linear gestures ) who suffer from limitations in the mobility of the respective body parts of a human user. For example, a swiping, panning or “pinch” gesture (for example from left to right or from top to bottom) ends naturally when the possible range of motion of the finger or hand performing the gesture is exhausted is, while this does not apply, for example, to a rotating free space gesture such as turning, in circular movements, of a finger, preferably an index finger, around its first (main) knuckle or an entire arm around the shoulder joint.

Carrying out such a free space gesture is generally easier than operating a specific button, switch or other classic HMI element, since it is sufficient for the gesture to be carried out within the field of view of the free space gesture detection sensor without the need to activate a To position fingers exactly on a typically specific small point of a classic HMI element, for example a specific surface of a pushbutton or a toggle switch. In addition, it is not necessary for the user to observe the execution of the gesture in order to be able to execute it correctly. This allows the user, in particular a driver of a vehicle, to continue to observe his surroundings, in particular the traffic around him, while he is performing the gesture.

Therefore, defining the operation of the HMI according to this method, namely in such a way that it is based on the detection of rotating free space gestures and on a plurality of menu items arranged one after the other along an arc, is not just an arbitrary choice, but rather a choice very specific choice. It enables a particularly efficient, reliable and easy-to-use way of performing and recording user inputs for the purpose of making a selection or preselection from a given limited or even unlimited amount of available selection options (menu items) provided by the HMI.

In addition, the user can conveniently set the speed of rotation of the rotating free space gesture and thus indirectly also the scrolling movement of the sequence of menu items at any time, without leaving an ongoing input process or the GUI mode and without having to change any other parameters .

Preferred embodiments of the method are described below, which can be combined with one another or with other aspects of the present invention as desired, unless such a combination is expressly excluded or technically impossible.

In some embodiments, the menu items of the rotating menu are evenly distributed along the arc. In addition, the threshold value is set such that it corresponds to at least 30%, preferably at least 50%, of the scrolling angle or the distance that is required to scroll through a first menu item located at the selection position to replace an immediately following menu item in the sequence of menu items at this selection position. Such a choice of the threshold value has proven to be advantageous because it defines a sufficiently high threshold value for the extent of the rotary movement by executing the rotating free space gesture, which is necessary to move the sequence of menu items by one position (or, accordingly, to the next menu item in each case the selection position). In addition, the threshold value is preferably defined in such a way that it corresponds to no more than 80%, preferably no more than 60% of the scrolling angle or the distance that is required to scroll to a first menu item that is at the selection position is to be replaced by an immediately following menu item in the sequence of menu items at this selection position. This has also proven to be advantageous, since it limits the extent of the rotary movement by executing the rotating free space gesture, which is necessary to move the sequence of menu items by one position or to move a respective next menu item to the selection position, and thus helps to increase the user comfort when performing the gesture.

In some embodiments, detecting the rotating free space gesture performed by a human user in a field of view of the free space gesture detection sensor includes detecting a rotational movement of one or more of an arm, a forearm, or at least one finger, for example an index finger, a hand of the human user. All of these rotary movements with one or more of these body parts have the advantage that they can in principle be carried out as continuous free space gestures with a freely selectable starting point, end point and number of rotations (or accumulated rotation angle) and thus a very wide range of different input options for the user enable. This in turn enables the use of a menu with a large number of different menu items from which the user can choose.

In some embodiments, the arc of the rotation menu is designed as an endless closed loop, preferably a regular closed loop, such as a circle or an ellipse. Such a design in the form of a closed loop could in particular be combined either with a defined limited number of menu items, such as a defined set of selectable functionalities, or even with an initially unknown number of menu items, such as a possibly variable number of photos or covers of music albums to be searched may be a preferred choice. Designing the arch as a regular shape has the advantage of doing this helps to increase the simplicity of a corresponding rotating clearance gesture to be performed. In particular, such an approach could be advantageous if the rotating free space gesture is to be adapted to a high degree to the shape of the arc (loop), as a result of which learning and the actual execution of such a gesture is simplified.

In some embodiments, the method further comprises triggering a predefined event in response to determining the selected menu item based on the detected rotating free space gesture, the event being associated with the selected menu item. Such an event could in particular be a certain functionality of the HMI itself or one or more other functionalities or devices that can be controlled via the HMI. For example, such functionalities could relate to selecting an operating mode of a particular device, for example an entertainment or navigation system of a vehicle. According to these embodiments, the method can therefore not only be used to select a specific menu item, but also to trigger a corresponding desired event that is linked to this specific menu item.

As an alternative to this, in some embodiments the method further comprises the triggering of a predefined event in response to a combination of both the determination of the menu item selected in the sense of a preselection on the basis of the detected rotating free space gesture and an additional user input which the preselection of the in the sense of the Preselection of selected menu item confirmed as an intended selection, the event being linked to the thus selected menu item. Such additional user input could, for example, take place in that the user executes a further predefined free space gesture, for example a non-rotating free space gesture, such as, for example, a linear movement of a body part within the field of view of the free space gesture detection sensor. These embodiments enable a two-step selection process, wherein the first (rotating) free space gesture is used to make a preselection from a number of available options (menu items), while a second, different free space gesture (or other predetermined user input) is used to: to confirm the selection and to trigger the event only after this confirmation has taken place. This can be particularly useful in applications, for example safety-related applications, where avoiding unintentional inputs could be crucial.

In some embodiments, the transformation ratio is defined as a configurable parameter, and the method further comprises: (i) receiving user input to select a desired value for that parameter; and (ii) using a transformation ratio corresponding to the selected value of the parameter for subsequent operation of the rotation menu. In particular, in some of these embodiments, the user input can even be made by a free space gesture of a rotating or non-rotating type. The possibility of configuring the transformation ratio, in particular by the user himself, makes it possible to achieve an optimal personal adaptation of the behavior of the HMI to the preferences of the user, which can differ from user to user or from application to application. For example, when scrolling through a large list of menu items, such as a list with many photos, the use of a larger transformation ratio, e.g. H. a higher scroll rate per angular velocity of the rotating gesture, while in a situation where there are relatively few menu items to choose from, a lower transformation ratio might be preferable. The latter can in particular also apply to safety-relevant applications in which the effective avoidance of an unintentional selection of a menu item that is adjacent to the actually desired menu item could be a critical requirement.

In some embodiments, the transformation ratio is defined such that the scroll direction corresponds to the direction of rotation of the detected gesture. This can be considered a preferred choice in many situations because it helps to simplify the user's interaction with the HMI and to reduce his or her cognitive load on input. On the other hand, in some selected situations the use of different directions of rotation for the scrolling movement on the one hand and for the rotating gesture on the other hand can be useful, for example in game applications, where an increase in the cognitive load on the user might even be desired.

In some embodiments, the transformation ratio is defined such that the rotational speed of the scrolling movement is equal to or less than the rotational speed of the detected rotating free space gesture. As already explained above, this can be particularly useful in those cases in which there is a relatively limited number of available menu items in the rotation menu and when the avoidance of errors in the selection process is of great importance is. This can also be useful in situations where the user in question may have difficulty following a rapid scrolling movement, which may be especially the case with some elderly or disabled users.

In some embodiments, the menu item that was identified as a selected or preselected menu item is temporarily provided with an animation effect in the GUI in order to indicate to the user that it has been identified as a selected or preselected menu item.

In some embodiments, the method further comprises outputting an individual characteristic sound associated with a particular menu item if that particular menu item is currently at the selection position during the movement of the sequence of menu items in response to the clearance gesture.

These two specific groups of embodiments can in particular also be used to enable safe operation of the HMI / GUI by the user, in particular to avoid unintentional selections.

In some embodiments, the method further comprises (i) receiving a plurality of digital images or capturing a plurality of digital images with an image sensor; and (ii) associating each of at least two of the digital images with one of the menu items of the rotation menu so that a respective one of these images is selected or pre-selected based on the detected rotating free space gesture when its associated menu item is selected . Accordingly, the method can thus be used to scroll through a set of images that have been recorded and to select a desired image, for example to trigger a display thereof or to enlarge its display in the GUI.

In some embodiments, the method further comprises displaying the selected or preselected image in a predefined format, size, or orientation that differs from that of the associated menu item in response to its selection or preselection.

One possible application of these image-related embodiments is browsing and the (pre-) selection of so-called selfies, for example in a vehicle-based environment, in which a vehicle could be equipped with a photo camera in order to record such images. In particular, the recording of such images could be triggered by a further specific, predetermined free space gesture, which can be detected by the above-mentioned free space gesture detection sensor.

In particular, the German patent application discloses DE 102020106003.3 , the entire content of which is incorporated herein by reference, methods and systems for triggering image recordings based on free space gestures to be performed by a human user. In particular, the methods and systems of the present invention can be combined with the methods and systems disclosed in DE 102020106003.3 are disclosed. In particular, one and the same system can be used to carry out both the method of the present invention and one or more of the methods described in DE 102020106003.3 are disclosed to implement (cf. 1 ).

In some embodiments, the free space gesture detection sensor is or includes a time of flight (TOF) camera that is used to detect the specific, predetermined free space gesture. A "TOF camera" is a 3D camera system that measures distances based on a time of flight (TOF) method. The measuring principle is based on the fact that the scene to be recorded is illuminated by a light pulse and the camera measures the time it takes for the light to reach the object and return for each pixel. Due to the constancy of the speed of light, this time is directly proportional to the distance. The camera thus provides the distance of the object imaged on each pixel. The use of a TOF camera system represents a particularly effective and high-resolution implementation option for a 3D image sensor and can in particular be used for the reliable and efficient detection of free space gestures.

In some embodiments, the recognition of said rotating free space gesture comprises at least one of the following: (i) processing sensor data generated by the free space gesture detection sensor to check whether the sensor data represents a situation in which a user has this specific predetermined Perform free space gesture or not; and (ii) transmitting sensor data generated by the clearance gesture detection sensor to a remote (e.g., off-vehicle) processing platform, e.g., to a server that can be reached via the Internet or other communication link, and as In response, receiving result data which are generated by the processing platform and which indicate whether the sensor data represents a situation in which a user makes this specific predetermined free space gesture or not. While option (i) is particularly useful for an autonomous solution, the can be fully implemented by a system to be integrated into a vehicle, option (ii) has the advantage that external processing power, for example that of a powerful server, can be used instead, which might otherwise not be available for the HMI (for example in a Vehicle) would be available or would increase its complexity and cost. This applies in particular to the generally computationally intensive processing that is required in connection with the correct recognition of a specific gesture on the basis of the sensor data supplied by the free space gesture detection sensor.

In particular, in some related embodiments, the processing of the sensor data is performed separately from a method for controlling the free space gesture detection sensor. In particular, the processing of the sensor data in connection with the recognition of said specific predetermined rotating free space gesture can be carried out by a remote processing platform, such as for example a server in a backend, which can be reached via the Internet or another communication connection, during an application which defines the GUI and which is used to control the operation of the clearance gesture detection sensor and the display device, can be executed on a local (e.g. vehicle-based) processor platform (e.g. in an infotainment system or other control unit of the vehicle). In particular, this enables, on the one hand, an optimal division of the processing required to carry out the method and, on the other hand, the possibility of configuring, optimizing, scaling, expanding, maintaining or replacing the respective processing means separately, be it on the hardware or on the the software level or both.

In some embodiments, each of a predetermined limited set of a plurality of different predetermined free space gestures that may possibly be performed by the user is defined as representing that specific predetermined rotating free space gesture. This means that there can be more than one specific, predetermined free space gesture that is sufficient to control the scrolling movement of the sequence of menu items in the GUI. This can be advantageous in at least two ways: Firstly, it can be used to increase the user-friendliness and thus also the reliability of the method by also defining other similar, but not identical, gestures as qualifying gestures for controlling scrolling, as long as they are safely distinguishable from other gestures that are not intended to similarly control scrolling. Second, this can be used to enable an implementation that allows the user to choose between different scrolling options, such as, without this being seen as a limitation, different options for scrolling speed (amounts of the transformation ratio) or between positive and negative transformation ratios (ie the same or opposite directions of rotation).

A second aspect of the present invention is directed to a system for operating a rotating menu of a graphical user interface, GUI, based on the detection of a free space gesture performed by a human user, the system configured to perform the method of the first Aspect of the present invention, as described in particular above, performs.

A third aspect of the present invention is directed to a computer program or a computer program product containing instructions for causing the system according to the second aspect to carry out the method according to the first aspect of the present invention.

The computer program (or computer program product) can in particular be implemented in the form of a data carrier on which one or more programs for carrying out the method are stored. This is preferably a data carrier, such as a CD, a DVD or a flash memory module. This can be advantageous if it is intended that the computer program product is to be traded as an individual product independently of the processor platform on which the one or more programs are to be executed. In another implementation, the computer program (or computer program product) is provided as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network.

The system according to the second aspect can accordingly have a program memory in which the computer program according to the third aspect is stored. Alternatively, the system can also be set up in such a way that it accesses a computer program via a communication link that is available externally, for example on one or more servers or other data processing units, in particular to exchange data with the latter that is made during the execution of the computer program are used or represent outputs of the computer program.

The various embodiments and advantages described above in connection with the first aspect of the present invention also apply in a similar manner to the other aspects of the invention.

• 1 zeigt schematisch ein beispielhaftes Fahrzeug mit einem System nach einer Ausführungsform der vorliegenden Erfindung;
• 2 zeigt schematisch eine beispielhafte GUI, die gemäß einem Verfahren der vorliegenden Erfindung, wie zum Beispiel gemäß dem Verfahren nach der in 4 dargestellten Ausführungsform, betrieben werden kann;
• 3 zeigt eine weitere Veranschaulichung einer beispielhaften GUI, wie zum Beispiel der GUI von 2, die gemäß einem Verfahren der vorliegenden Erfindung, wie zum Beispiel gemäß dem Verfahren nach der in 4 dargestellten Ausführungsform, betrieben werden kann; und
• 4 zeigt ein Flussdiagramm, das eine beispielhafte Ausführungsform des Verfahrens nach der vorliegenden Erfindung veranschaulicht.

Further advantages, features and applications of the present invention are shown in the following detailed description and the accompanying figures, wherein:

• 1 Figure 3 schematically shows an exemplary vehicle having a system in accordance with an embodiment of the present invention;
• 2 FIG. 11 schematically shows an exemplary GUI that is generated according to a method of the present invention, such as, for example, according to the method according to FIG 4th illustrated embodiment, can be operated;
• 3 FIG. 13 shows another illustration of an exemplary GUI, such as the GUI of FIG 2 which, according to a method of the present invention, such as, for example, according to the method according to the in 4th illustrated embodiment, can be operated; and
• 4th Figure 3 shows a flow diagram illustrating an exemplary embodiment of the method according to the present invention.

In the figures, the same reference symbols are used for the same or corresponding elements of the systems described here.

According to 1 exhibits an exemplary vehicle 100 according to one embodiment of the present invention, a system 105 for controlling, in particular for triggering, image recordings of at least part of the interior of the vehicle 100 on. To this end, the system instructs 105 a free space gesture detection sensor 110 on, in the passenger compartment on an inner surface of the roof of the vehicle 100 is arranged. The free space gesture detection sensor 110 is designed as a 3D image sensor of the "Time of Flight" (TOF camera) type. He has a field of view (FOV) 115 which is essentially in the shape of a cone that has its tip on the clearance gesture detection sensor 100 and extends mainly downwards, but optionally also with a horizontal directional component to one next to the driver's seat of the vehicle 100 arranged center console. The free space gesture detection sensor 110 is configured to detect free space gestures G that are within its field of view 115 from a human user U, such as a driver of the vehicle 100 to be executed. In the in 1 The specific example shown, the user U performs a specific predetermined free space gesture G with two fingers of her right hand, these two fingers forming a “V” shape within a virtual plane that is essentially horizontal and thus predominantly perpendicular to the predominantly vertical central direction of the Cone that the FOV 115 of the gesture detection sensor 110 Are defined. Accordingly, the "V" shape is for the gesture detection sensor 110 "Visible" and thus recognizable as such. Similarly, user U can perform a rotating gesture by rotating the index finger of her hand around its main knuckle, for example to “draw” a virtual circle, and such a rotating gesture can be detected in a similar manner, particularly when the gesture detection sensor 210 is a 3D sensor.

The system 105 further comprises an image sensor 120 in the form of a 2D photo camera, which is sensitive at least in parts of both the visible and the infrared range of the electromagnetic spectrum, so that it cannot only take pictures when the interior of the vehicle 100 - whether artificially or by daylight - is illuminated, but also when it is relatively dark, especially at night. The image sensor 120 is located on the rearview mirror of the vehicle in the passenger compartment and has a field of view 125 , which extends predominantly in the direction of the position of the driver's seat and at least partially covers it and optionally also the position of one or more other passenger seats of the vehicle 100 . The image sensor is accordingly 120 set up to take photos or videos of one or more occupants of the vehicle 100 can record while they are sitting in their respective passenger seats, ie in particular "selfies". While the field of view 115 of the gesture detection sensor 110 and the field of view 125 of the image sensor 120 Although they partially overlap, they do not completely match, so that in particular a spatial area above the center console of the vehicle 100 only within the FOV 115 of the gesture detection sensor 110 but not within the FOV 125 of the image sensor 120 lies. If a free space gesture G is carried out by the user U within this spatial region, body parts or other objects that the user U may use to carry out the gesture G are accordingly not shown in photos or videos that are captured by the image sensor 120 be included.

Each of the sensors 110 and 120 can have its own lighting unit 111 respectively. 121 assigned to the associated field of view 115 respectively. 125 to illuminate with electromagnetic radiation to which the respective sensor is sensitive, for example in the visible or infrared part of the electromagnetic spectrum. Any of these lighting units 111 and 121 can be activated individually and deactivated. In particular, they can be controlled in such a way that at any given point in time only one or none of the two lighting units 111 and 121 is activated.

In addition, the system 105 a processing unit 130 on the signaling with both the gesture detection sensor 110 as well as with the image sensor 120 is coupled to control them and to receive their respective sensor data for further processing. In particular, the processing unit 130 contain one or more computer programs in a corresponding memory which are configured as an application for triggering image recordings on the basis of free space gestures, so that a user U can trigger the recording of images by being in the FOV 115 of the image capture sensor 110 performs a specific, predetermined gesture that corresponds to such an image recording.

Furthermore, the processing unit 130 be configured so that it can recognize one or more different predetermined gestures when these are represented by the sensor data received from the gesture detection sensor 110 to be delivered. The gesture detection sensor is accordingly 110 in combination with the processing unit 130 then able to capture one or more different, predetermined free space gestures, including rotating free space gestures, made by the user U within the FOV 115 of the sensor 110 are executed. In particular, one or more of these gestures can be specified as gestures which, if they are correctly recognized, the image recording by the image sensor 120 trigger.

Optionally, the system 105 furthermore a communication unit 135 configured to receive data over a communication link with an off-vehicle processing platform 145 , such as a backend server. As in 1 shown, the communication link can be, for example, a wireless link and accordingly the communication unit 135 signaling with an antenna 140 of the vehicle 102 so as to transmit and receive RF signals for communication over the wireless link.

In particular, the structure can be used to collect sensor data from the gesture detection sensor 110 are generated to the processing platform 145 for processing similar to the above with respect to the processing unit 130 in connection with gesture recognition. This can be particularly advantageous when a considerable number of different and sometimes complex gestures must be reliably recognized, so that a large amount of processing power is required in order to carry out the recognition and differentiation of these various gestures. Typically, it could be simpler and more efficient to have such a large amount of computing power outside the vehicle on a dedicated processing platform 145 instead of the processing unit 130 to be designed as a corresponding high-performance processing unit. The latter approach could have all sorts of negative effects, including higher average gesture recognition costs per vehicle per gesture, or higher space, power, or cooling requirements, etc.

Furthermore, the system 105 an HMI with a display 150 to display information for the user U on. In particular, the HMI is configurable to display a rotation menu, as described below with reference to FIG 2 and 3 will be described in detail.

The processing unit 130 can in particular be configurable, for example by means of a corresponding computer program, in order to also control the display 150 to control and especially in connection with the display 150 and the gesture detection sensor 110 to carry out the method according to the present invention, such as the method 300 according to the following with reference to the 4th illustrated embodiment.

It is now on the 2 Referred to in an exemplary GUI 200 which, according to a method of the present invention, such as, for example, the method according to the in 4th illustrated embodiment, can be operated, is designed on a screen 150 (see. 1 ), which can in particular be a rectangular screen, to be displayed. The GUI 200 is defined on a virtual plane that has a first section 205 that is not currently on the screen 150 and a second section 210 which is essentially the same shape as the screen 150 and which is currently represented on it. Inside the GUI 200 are a plurality of menu items 215 , for example photos or other graphic objects, evenly distributed one after the other along an arc, which in particular can be a circular arc or a section thereof, as in 2 shown. The distribution of the menu objects along the arc can in particular be referred to as a menu wheel, in particular if the arc is defined as a closed (ie endless) loop, such as a circle or an ellipse.

When a corresponding free space rotating gesture is detected, the sequence of menu items 215 (for example menu wheel) corresponding to a transformation ratio T along a direction of rotation 220 moved (rotated) as follows with reference to FIG 3 is explained in more detail. Accordingly, during this movement of the sequence of menu items and at any given point in time, only a corresponding section of that sequence is displayed in the section 210 the GUI 200 displayed while the remaining menu items 215 currently in the section not shown 205 the GUI 200 until finally, when the detected gesture is continued, they are in the displayed section 210 be moved into or even completely through it.

At a position on the arc and within the currently displayed section 210 the GUI 200 becomes a specific position, which can be occupied by a menu item 215a, as a selection position 225 Are defined. This position can be for example a position in the middle of the visible part of the arch within the section 210 the GUI 200 be defined. As below with reference to 4th is explained in more detail, a menu item 215a, which is at the selection position 225 is located when an end of a predetermined rotating free space gesture is detected, thereby selected or pre-selected in the sense of a selection by the user.

It is now on 3 Reference is made to the a detailed view of a section of the sequence of menu items 215 shows, which for the purpose of a simplified illustration has only three exemplary menu items 215-1, 215-2 and 215-3. When a corresponding free space rotating gesture by the gesture detection sensor 120 (see. 1 ) is detected, an accumulated rotation angle θ corresponding to the accumulated rotation angle of the movement of a certain body part such as an index finger of the user performing the gesture is continuously obtained, and the sequence becomes accordingly 215 of menu items within the GUI 200 rotated in the same direction of rotation as that of the gesture. The angular speed of rotation of the sequence of menu items 215 however, may differ from the gesture, for example the rotational movement of the user's index finger, in a predetermined manner. For example, if an angle ϕ is the accumulated degree of rotation of the sequence of menu items 215 and another angle θ indicates the accumulated degree of rotation of the finger of the user performing the gesture, then these two angles may be different. In particular, a transformation ratio T can be defined as the derivative T = dϕ / dθ, which in the case of a constant speed corresponds to the ratio ϕ / θ of the angle itself. In particular, extensive tests have shown that values for T in the range from 50: 1 to 90: 1 represent a selection that is particularly well received, a good balance between efficiency and selection speed on the one hand and a reliable selection process, the unintended selections, on the other largely avoids bidding. Accordingly, with such values for T, a high degree of user satisfaction can be achieved. For example, a value for T = 72: 1 has proven to be particularly satisfactory. In this case, the menu wheel rotates by ϕ = 5 ° if the rotating gesture corresponds to an angle of θ = 360 °.

Furthermore, a further angle alpha α can be defined in order to specify the angle between the respective positions of two adjacent menu items within the sequence along the arc. As below with reference to 4th is explained, this angle α can be used to define a criterion to determine whether the menu item after completion of the rotating gesture (i) 215 that is just leaving the selection position is selected as the selected or pre-selected menu item, or (ii) instead the next immediately following menu item that is currently in the process of reaching the selection position, but which may not yet be completely in the Moved the selection position into it.

It is now on the 4th Referred to that a procedure 300 according to an embodiment of the present invention. In addition, for a better explanation and without restriction, reference is again made to the 1 until 3 Referenced. The procedure 300 has a step 305 on which a free space gesture detection sensor 110 of the TOF camera type continuously “looks out”, ie its field of view 115 to free space gestures made by a user U, for example a driver or another occupant of the vehicle 100 executed, monitored. When the gesture detection sensor 110 is activated, is also its lighting unit 111 activated to the field of view 115 to illuminate while the lighting unit 121 of the image sensor 120 is disabled at this time. Accordingly, the detection activity of the gesture detection sensor becomes 110 not by lighting provided by the lighting unit 121 of the image sensor 120 is sent out, negatively influenced (desynchronization).

When the gesture detection sensor detects the start of a predetermined rotating free space gesture which is carried out by a user U and which is deemed to be necessary for operating the in 2 and 3 is viewed as a qualifying GUI, it solves the step 310 in which the sequence of menu items 215 (Menu wheel) according to a pre-configured parameter defining the transformation ratio T is scrolled as above with reference to FIG 3 was explained. If T is a positive value, then the direction of rotation of the menu wheel corresponds to the direction of rotation of the rotating gesture, otherwise it is opposite to it. The higher the value of T, the faster the scrolling movement of the menu wheel will be relative to the movement of the gesture, ie to the rotational movement of the finger of the user performing the gesture.

If the user ends the clearance gesture, this is done in a further step 320 of the method 300 detected, and the total accumulated scroll angle ϕ of the menu wheel, which has accumulated between the beginning and the end of the rotating gesture, is determined.

Then in one step 325 this total accumulated scroll angle ϕ of the menu wheel is compared with a predefined threshold angle ϕ _T , which can in particular be defined as a value between 40% and 60%, for example 50% of the angle α between the respective positions of two adjacent menu items within the sequence along the arch, as in 3 is shown.

If in a subsequent step 325 it is determined that the angle ϕ has not yet reached the threshold angle ϕ _T (325 - no; ie ϕ <ϕ _T ), then a step is made 330 triggered in that the menu wheel is automatically rotated back to an initial position, which can be defined in particular as the position of the menu wheel in which the last menu item 215a, which was there during the scrolling movement, the selection position 225 to leave still on the selection position 225 was centered. The procedure 300 then returns to step for one more pass 305 return.

Otherwise ( 325 - Yes; ie ϕ> ϕ _T ) becomes a step 335 triggered, in which the menu wheel is automatically turned so far forwards that a next menu item, which immediately follows the last menu item 215a that was there during the scrolling movement, is the selection position 225 to leave, in the middle of the selection position 225 emotional.

A confirmation step then follows after a first implementation option (pre-selection option, shown in dashed lines) 340 , in which the preselection of the menu item 215a currently present at the selection position after the automatic rotation can be confirmed or discarded by an additional user input as an intended selection.

If the preselection is discarded ( 345 - no), the procedure reverses 300 for one more run through to the step 305 in which the user can try again to make a correct entry using the rotating gesture. Otherwise, if the preselection is confirmed (345 - yes), the menu item that is currently at the selection position becomes 225 is located than the currently selected menu item 115a is determined.

As a result of the confirmed selection, in one step 350 a certain event is triggered which is related to the new position of the menu wheel, ie with the selected menu item. Such an event could, for example, consist in the fact that an enlarged version of the selected menu item, for example a photo, within the GUI 200 is shown.

According to an alternative implementation option (immediate selection option), the steps 340 and 345 left out so the step 350 immediately on the step 335 follows.

While at least one exemplary embodiment of the present invention has been described above, it should be understood that a large number of variations thereon exist. It should also be noted that the exemplary embodiments described only represent non-limiting examples of how the present invention can be implemented and that it is not intended to limit the scope, applicability or configuration of the devices and methods described here. Rather, the preceding description will provide a person skilled in the art with instructions for implementing at least one exemplary embodiment of the invention, it being understood that various changes in the mode of operation and the arrangement of the elements of the exemplary embodiment can be made without the dependency on the in the subject matter specified in each case and its legal equivalents are deviated from the attached claims.

List of reference symbols

100

vehicle

105

System for triggering image recordings based on free space gestures

110

TOF type free space gesture detection sensor

111

Lighting unit for free space gesture detection sensor 110

115

Field of view of the clearance gesture detection sensor 110

120

Image sensor, photo camera

121

Illumination unit for image sensor 120

125

Field of view of the image sensor 120

130

Processing unit

135

Communication unit

140

Wireless communication link antenna

145

Processing platform external to the vehicle, for example server

150

advertisement

200

exemplary view of a graphical user interface on which a rotation menu can be seen

205

virtual level of the rotation menu, in particular a section that is not displayed therefrom

210

displayed part of the rotation menu

215

individual images arranged in an evenly spaced sequence along an arc

215

single image that is at the selection position

220

Scroll direction

225

Selection position

300

exemplary embodiment of a method for operating a rotating menu of a GUI

305-350

Method steps of method 300

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102020106003 [0034]

Claims (16)

A method (300) for operating a rotating menu of a graphical user interface, GUI, (200) based on the detection of a free space gesture carried out by a human user (U), the method comprising: detecting (305) a rotating free space gesture ( G) with the aid of a free space gesture detection sensor (120) carried out by a human user (U) in a field of view (115) of the free space gesture detection sensor; Displaying at least a portion (210) of a rotation menu on a display device (150), the rotation menu having a plurality of individually selectable menu items (215) which are arranged in a sequential order along an arc which has a specific predefined selection position (225) along the Has bow; synchronous and uniform scrolling (310) of the sequence of menu items in the GUI along the arc in response to the detected rotating free space gesture, so that the scrolling speed is aligned according to a predefined transformation ratio (T) to a detected rotational speed and direction of rotation of the detected rotating free space gesture ; Determining (305; 320) a beginning and an end of the detected rotating free space gesture and an accumulated rotating scroll angle (ϕ) or a distance between a starting point of the scrolling movement of the menu items, which corresponds to the beginning of the gesture, and an end point of the scrolling -Movement of the menu items corresponding to the end of the gesture; if the accumulated scroll angle or the accumulated scroll distance _{reaches or exceeds a respective predefined threshold value (ϕ T} ), the sequence of menu items is automatically scrolled further by a remaining angle or distance, so that the next menu item in each case along the direction of rotation to the Selection position is moved, and otherwise automatically reversing the scrolling movement of the sequence of menu items in order to bring the last menu item which previously occupied the selection position back to the selection position; and determining this specific menu item as a currently selected menu item or a menu item selected in the sense of a preselection, which is located at the selection position after the automatic scrolling has ended. Method (300) according to Claim 1 wherein: the menu items (215) of the rotation menu are evenly distributed along the arc; and the threshold value (ϕ _T ) is set in such a way that it corresponds to at least 30% of the scrolling angle (a) or the distance that is required to scroll through a first menu item that is located at the selection position (225) is to be replaced by an immediately following menu item in the sequence of menu items at this selection position. Method (300) according to Claim 2 , wherein the threshold value (ϕ _T ) is set in such a way that it corresponds to at least 50% of the scrolling angle (a) or the distance that is required to scroll through a first menu item that is located at the selection position (225 ) is to be replaced by an immediately following menu item (215) in the sequence of menu items at this selection position. The method (300) according to any one of the preceding claims, wherein the detection of the rotating free space gesture, which is carried out by a human user in a field of view of the free space gesture detection sensor, the detection of a rotational movement of one or more of an arm, a forearm or at least one finger Hand of the human user. The method (300) according to any one of the preceding claims, wherein the arc of the rotating menu is designed as an endless closed loop. The method of any preceding claim, further comprising triggering (350) a predefined event in response to determining the selected menu item (115a) based on the detected rotating free space gesture, the event being associated with the selected menu item. Method (300) according to one of the Claims 1 until 5 , the method further comprising triggering (350) a predefined event in response to a combination of both the determination of the menu item selected in the sense of a preselection on the basis of the detected rotating free space gesture and an additional user input (340) which the preselection of the in terms of of the preselection selected menu item is confirmed as an intended selection, the event being linked to the menu item thus selected. The method (300) according to any one of the preceding claims, wherein the transformation ratio (T) is defined as a configurable parameter and the method further comprises: Receiving user input to select a desired value for that parameter; and Use a transformation ratio that corresponds to the selected value of the parameter for the subsequent operation of the rotation menu. The method (300) according to any one of the preceding claims, wherein the transformation ratio (T) is defined such that the scroll direction (220) coincides with the direction of rotation of the detected gesture. The method (300) according to any one of the preceding claims, wherein the transformation ratio is defined such that the rotational speed of the scrolling movement is equal to or less than the rotational speed of the detected rotating free space gesture. Method (300) according to one of the preceding claims, wherein the menu item that was determined as a selected or preselected menu item is temporarily provided with an animation effect in the GUI in order to indicate to the user that it is determined as a selected or preselected menu item became. The method (300) according to any one of the preceding claims, wherein the method further comprises outputting an individual characteristic sound associated with a particular menu item when that particular menu item is at that particular moment during the movement of the sequence of menu items in response to the clearance gesture Selection position. The method (300) according to any one of the preceding claims, wherein the method further comprises: Receiving a plurality of digital images or taking a plurality of digital images with an image sensor; and Assigning each of at least two of the digital images to one of the menu items of the rotation menu, so that a respective one of these images is selected or preselected on the basis of the detected rotating free space gesture when its assigned menu item is selected. Method (300) according to Claim 13 wherein the method further comprises displaying the selected or preselected image in a predefined format, size, or orientation that is different from that of the associated menu item in response to its selection or preselection. A system (105, 150) for operating a rotating menu of a graphical user interface, GUI, (200) based on sensing a free space gesture (G) performed by a human user (U), the system being configured such that it carries out the method (300) according to one of the preceding claims. Computer program that contains instructions for the system (105, 150) according to Claim 15 to cause the procedure according to one of the Claims 1 until 14th to execute.

Images