DE102019118952A1 - Method, device and computer program for controlling a user interface - Google Patents

Abstract

Exemplary embodiments deal with methods, a device and a computer program for controlling a user interface. One method of controlling a user interface includes projecting the user interface onto a hand of a user. The method further comprises determining three-dimensional distance information of a distance between the hand and a distance sensor. The method further comprises determining a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information. The method further comprises controlling the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor.

Description

Technical area

Exemplary embodiments deal with methods, a device and a computer program for controlling a user interface, more precisely, but not exclusively, with concepts for controlling a user interface in which the user interface is projected onto a hand.

background

There are a number of approaches to controlling user interfaces: For example, haptic buttons or levers, such as simple push buttons, have been used to operate machines since the beginning of electrical systems. In order to be able to operate more complex systems, these buttons were gradually combined in so-called keyboards, as they are also used today to operate computers. But other input devices, such as mice, are also used to operate user interfaces.

Another way of operating user interfaces that has been frequently used since the early 2000s is touchscreens (touch-sensitive display devices), in which the user can manipulate the user interface directly with a finger or a so-called stylus (pen for operating touchscreens). However, these input options all have the disadvantage that their implementations require a certain amount of space or have poor ergonomics: Keyboards and mice need a large area on which they can be accommodated, while touchscreens are ergonomically questionable in many cases, for example in screens that are arranged approximately vertically in front of the user.

There is a need for an improved input concept for controlling a user interface.

Summary

This requirement is met by the method, the device and the computer program according to the independent claims.

Embodiments are based on the insight that a hand can be used to both display the user interface and to control the user interface by detecting a movement of the hand. For this purpose, according to the invention, the user interface is projected onto the hand of a user. The movement of the hand along a depth axis is recorded via a distance sensor, i.e. the movement of the hand is not only recorded as a lateral movement, for example, but also in the depth axis, which enables the use of more intuitive gestures in the control of the user interface. For example, a rotation of the hand can be recognized, which leads to the user interface being activated or deactivated, or a curvature of the hand can be recognized and used to select options in the user interface.

Exemplary embodiments create a method for controlling a user interface. The method includes projecting the user interface onto a hand of a user. The method further comprises determining three-dimensional distance information of a distance between the hand and a distance sensor. The method further comprises determining a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information. The method further comprises controlling the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor. By projecting the user interface onto the hand, detecting the movement of the hand in the depth axis and controlling the user interface based on the detected movement, the user interface can be operated without an area having to be provided for operating the user interface, so that at least some embodiments solve the problem according to the invention.

In at least some exemplary embodiments, determining the movement of at least parts of the hand includes determining a rotational movement of the hand. The user interface can be controlled based on the rotational movement. The detection of the rotational movement is made possible or improved by determining the movement in the depth axis, so that corresponding movements can be used to operate the user interface.

For example, the method can include recognizing whether an inside or outside of the hand is facing the distance sensor, and activating the user interface if the inside of the hand is turned towards the distance sensor by a rotational movement. The rotation of the hand can be used to activate the user interface by switching intuitively between a side of the hand that is only used passively to an actively used side of the hand.

Additionally or alternatively, determining the movement of at least parts of the hand can include determining a rotational or curving movement of at least parts of the hand. The user interface can be controlled based on the rotation or curvature movement. The detection of the rotational or curvature movement is also made possible or improved by determining the movement in the depth axis, so that corresponding movements can be used to operate the user interface.

A selection option can be selected from a plurality of selection options by curving or rotating the hand. The selection is made possible by the curvature or rotation, since both a degree of rotation of the rotation and an angle of curvature or center of curvature allow a fine-grained selection of options.

For example, the selection of the selection option can be visualized by a selection field in the user interface. The selection field can be controlled by the curvature or rotation of the hand in such a way that the selection field moves between the selection options based on a position of the hand in accordance with the force of gravity. By making a selection based on gravity, the user can be given an intuitive way of selecting the options.

Alternatively or additionally, the user interface can also be operated using the curvature of the fingers. For example, at least two selection options of the user interface can be projected onto at least two fingers. One of the at least two options can be selected if the corresponding finger is curved. This enables a direct relationship between the selection option and the part of the hand that is to be actuated in order to select the selection option.

For example, one of the at least two selection options can be selected if, due to the curvature of the finger, the corresponding finger touches another finger onto which no selection option is projected. For example, the selection can be confirmed by bringing a finger, projection and thumb together.

In some exemplary embodiments, determining the movement of at least parts of the hand can include determining a movement of the entire hand along the depth axis. The user interface can be controlled based on the movement of the entire hand along the depth axis. This can be used, for example, to confirm a selection or to change values gradually.

For example, an element of the user interface can be changed or activated by moving the entire hand along the depth axis in the direction of the distance sensor. In the case of activation, this can correspond to "clicking" a mouse.

In at least some exemplary embodiments, the three-dimensional distance information of the distance between the hand and a distance sensor comprises a plurality of distance values for different sections of the hand. The movement of at least parts of the hand can be determined based on the plurality of distance values. This enables or improves recognition of the rotation and / or the curvature of the hand.

The method can further include providing an acoustic feedback based on the control of the user interface. This allows the user improved feedback on the abstract operation of the user interface by hand.

In at least some exemplary embodiments, the control of the user interface is independent of a movement of the hand orthogonal to the depth axis. This avoids situations in which the user has to maintain an uncomfortable hand position while operating the user interface.

The method can include providing a control signal for the user interface in order to effect the control of the user interface. Controlling the user interface can be visualized through the user interface. The control signal is generated in order to be able to control the user interface accordingly after processing the movement of the hand. Because the control signal contains corresponding instructions for controlling the user interface, these can also be used to visualize the control.

In some exemplary embodiments, the method includes recognizing whether the hand is being opened or closed based on the movement of at least parts of the hand along the depth axis. The method may include activating the user interface if it is detected that the hand is being opened and deactivating the user interface if it is detected that the hand is being closed. By "opening" and "closing" the hand, the user can intuitively signal that he wishes to use the user interface or that he wishes to stop using it.

Embodiments also provide a further method for controlling the user interface. Where the previous procedure related to the The overall system comprising distance sensor, projection device, processing and user interface relates, the further method relates to the corresponding features in relation to a processing module, for example a processor which is designed to carry out the processing steps of the above-mentioned method. Consequently, the further procedure corresponds to a large extent to the procedure presented above. The further method detects providing a display signal for a projection device in order to cause the user interface to be projected onto a hand of a user. The further method further comprises receiving a distance measurement signal from a distance sensor. The further method further comprises determining three-dimensional distance information of a distance between the hand and the distance sensor based on the distance measurement signal. The further method further comprises determining a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information. The further method further comprises providing a control signal for the user interface in order to effect a control of the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor

Embodiments also create a program with a program code for performing the further method when the program code is executed on a computer, a processor, a control module or a programmable hardware component.

Embodiments also create a device for controlling a user interface. The device comprises at least one interface designed for communication with a projection device and with a distance sensor. The device further comprises a processing module designed to provide a display signal for the projection device in order to project the user interface onto a hand of a user. The processing module is also designed to receive a distance measurement signal from the distance sensor. The processing module is designed to determine three-dimensional distance information of a distance between the hand and the distance sensor based on the distance measurement signal. The processing module is designed to determine a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information. The processing module is further designed to provide a control signal for the user interface in order to effect a control of the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor.

Figure list

• 1a bis 1c zeigen Flussdiagramme von Ausführungsbeispielen von Verfahren zum Steuern einer Benutzeroberfläche;
• 1d zeigt ein Blockdiagramm eines Ausführungsbeispiels einer Vorrichtung zum Steuern einer Benutzeroberfläche.
• 2a bis 2d zeigen schematische Zeichnungen eines Ausführungsbeispiels, in dem ein Telefonanruf angenommen wird;
• 3a und 3b zeigen schematische Zeichnungen eines weiteren Ausführungsbeispiels, in dem ein Auswahlmenü in 3 Dimensionen bedient wird; und
• 4a und 4b zeigen schematische Zeichnungen eines weiteren Ausführungsbeispiels, in dem eine Kategorie durch das Antippen von Fingerspitzen ausgewählt wird.

Some examples of devices and / or methods are explained in more detail below with reference to the accompanying figures, merely by way of example. Show it:

• 1a to 1c show flow charts of exemplary embodiments of methods for controlling a user interface;
• 1d FIG. 11 shows a block diagram of an embodiment of a device for controlling a user interface.
• 2a to 2d show schematic drawings of an embodiment in which a telephone call is accepted;
• 3a and 3b show schematic drawings of a further embodiment in which a selection menu is operated in 3 dimensions; and
• 4a and 4b show schematic drawings of a further embodiment in which a category is selected by tapping fingertips.

description

Various examples will now be described in more detail with reference to the accompanying figures, in which some examples are shown. In the figures, the strengths of lines, layers and / or areas may be exaggerated for clarity.

Accordingly, while other examples are susceptible of various modifications and alternative forms, some specific examples thereof are shown in the figures and will be described in detail below. However, this detailed description does not limit other examples to the particular forms described. Other examples may cover all modifications, equivalents, and alternatives that come within the scope of the disclosure. Throughout the description of the figures, the same or similar reference symbols refer to the same or similar elements which, when compared with one another, may be identical or implemented in a modified form while they provide the same or a similar function.

It will be understood that when an element is referred to as being “connected” or “coupled” to another element, the elements may be connected or coupled directly, or through one or more intermediate elements. If two Elements A and B are combined using an “or”, this is to be understood in such a way that all possible combinations are disclosed, ie only A, only B and A and B, unless otherwise explicitly or implicitly defined. An alternative phrase for the same combinations is “at least one of A and B” or “A and / or B”. The same applies, mutatis mutandis, to combinations of more than two elements.

The terminology used herein to describe specific examples is not intended to be limiting of additional examples. When a singular form, e.g. B. "ein, an" and "der, die, das" are used and the use of only a single element is neither explicitly nor implicitly defined as mandatory, further examples can also use plural elements to implement the same function. When a function is described below as being implemented using multiple elements, other examples may implement the same function using a single element or a single processing entity. It is further understood that the terms “comprises”, “comprising”, “having” and / or “having” the presence of the specified features, integers, steps, operations, processes, elements, components and / or a group thereof when used Specify but not exclude the presence or addition of one or more other characteristics, integers, steps, operations, processes, elements, components and / or a group thereof.

Unless otherwise defined, all terms (including technical and scientific terms) are used herein with their normal meaning in the field to which examples belong.

The 1a to 1c show flowcharts of exemplary embodiments of methods for controlling a user interface. The procedure of 1a and 1b its coverage area relates to an overall system in which a projection device, a distance sensor and a processing module are combined in order to carry out the method. The method includes projecting 110 the user interface on a user's hand. The method further includes determining 120 of three-dimensional distance information of a distance between the hand and a distance sensor. The method further includes determining 130 a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information. The method further includes controlling 140 the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor. The method can be carried out by a system comprising the projection device, the distance sensor and the processing module.

1c In contrast, it is directed towards the corresponding method from the point of view of the processing module. The method includes providing 110a a display signal to a projection device to cause the user interface to project onto a hand of a user 110 becomes. The method further includes obtaining 120a a distance measurement signal from a distance sensor and determining 120 of three-dimensional distance information of a distance between the hand and the distance sensor based on the distance measurement signal. The method further includes determining 130 a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information. The method further comprises providing 140a a control signal for the user interface to control 140 the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor. As can be seen, the method of 1c an implementation of the method of 1a and 1b represents, in which the processing steps are carried out by the processing module, for example in connection with at least one interface. The features and developments of the method are correspondingly 1a and 1b also on the procedure of 1c applicable.

1d shows a block diagram of an embodiment of a corresponding device 10 to control a user interface. The device 10 includes at least one interface 12 designed for communication with a projection device 16 and with a distance sensor 18th , ie the at least one interface is with the projection device 16 and with the distance sensor 18th connectable. The device 10 further comprises a processing module 14th , the one with the at least one interface 12 is coupled. The processing module 14th is designed, for example, to the method of 1c , and thus indirectly also the 1a and 1b to perform, for example in connection with the at least one interface 12 . For example is the processing module 14th designed to provide a display signal for the projection device 16 (via the at least one interface 12 ) to project the user interface onto a user's hand (ie, to cause the user interface to be projected onto the user's hand). The processing module 14th is formed to receive a distance measurement signal from the distance sensor 18th (via the at least one interface 12 ). The processing module 14th is designed to determine three-dimensional distance information of a distance between the hand and the distance sensor based on the distance measurement signal. The processing module 14th is designed to determine a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information. The processing module 14th is designed to provide a control signal for the user interface in order to effect a control of the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor. 1d also shows a system comprising the projection device, the distance sensor and the device 10 .

The following description applies to both the methods of 1a to 1c as well as the device of the 1d .

Exemplary embodiments deal with methods, computer programs and devices for controlling a user interface. In this case, the user interface is a visual user interface, ie it is perceived visually by the user, and it can be changed, ie the user interface changes, for example based on an input by the user or based on events of an operating system. In exemplary embodiments, the user interface is projected onto a (single) hand of the user 110 . To make this possible, in at least some exemplary embodiments a display signal for a projection device is provided 110a to cause the user interface to project onto a user's hand 110 becomes. This display signal can include, for example, video and / or control information that is designed to cause the display signal to be projected onto the user interface.

For example, the projection device can be arranged above the user's hand, for example in a floor lamp, in a ceiling lamp, on the ceiling of a room or a vehicle or on a wall or an electronic device such as a monitor. The user interface can be projected (from above) onto a surface of the hand of the user by the projection device. The distance sensor can optionally also be arranged above the hand of the user, for example like the projection device / parallel to the projection device. Alternatively, the distance sensor can be arranged below the user's hand.

The projection of the user interface onto the hand of the user can depend on the distance between the hand of the user and the projection device. In general, the user interface has a fixed target size that can be adjusted based on the distance between the user's hand and the projection device, e.g. by a predetermined absolute size (about 8x10 cm) or a predetermined size relative to the hand (about 50% / 60 % / 70% / 80% of an area on the back of the hand or the palm of the hand) of the user interface. Furthermore, the projection of the user interface onto the hand of the user can be adjusted based on the distance in order to achieve that the user interface is displayed “sharply” on the hand, for example by adjusting the focal distance. In other words, the method can include an adaptation of the projection of the user interface based on the three-dimensional distance information, for example in order to achieve a predefined absolute size or a predefined size relative to the hand, or in order to focus the projection. The projection can be brought into focus by changing the focal distance based on the three-dimensional distance information. The absolute or relative size of the user interface can be done by changing the angle of view based on three-dimensional distance information. Furthermore, the projection can be adjusted based on a movement of the hand orthogonally to the depth axis, i.e. a target direction of the projection can be adjusted as a function of the movement of the hand orthogonally to the depth axis, for example based on the three-dimensional distance information.

The method includes determining 120 the three-dimensional distance information of the distance between the hand and the distance sensor. The procedure can be a receiving 120a of the distance measurement signal from the distance sensor. The three-dimensional distance information can be determined based on the distance measurement signal. The distance sensor can be a so-called transit time-based distance sensor (also Time of Flight, ToF), for example a transit-time-based distance sensor based on modulated light or a transit-time-based distance sensor based on ultrasound. The distance sensor can be an optical distance sensor, for example a travel time-based distance sensor based on modulated light, a distance sensor based on structured light, or a LIDAR-based distance sensor (light detection and ranging, detection and distance measurement based on light). The distance measurement signal can include, for example, raw data or preprocessed data from the corresponding distance sensors, and the determination of the three-dimensional distance information can comprise processing the raw data or the preprocessed data.

The distance information is three-dimensional, ie the distance information distinguishes not only the distances between the distance sensor and the hand in the depth axis, but also orthogonally to the depth axis. If, for example, the depth axis (also the z-axis) is defined by the axis between the hand and the distance sensor, a lateral plane is spanned orthogonally to it, which is defined by an x-axis and a y-axis. Within the lateral plane, which is spanned orthogonally to the depth axis, the distance information includes distinctions, ie the movement of the hand is not only determined relative to the depth axis, but also orthogonally to it. This can be used not to perceive the hand as a whole, but to determine and track a plurality (for example at least two, at least four, at least 10) distance values for different sections of the hand. In other words, the three-dimensional distance information of the distance between the hand and a distance sensor can comprise a plurality of distance values for different sections of the hand. The movement of at least parts of the hand (which may correspond to the different sections) can be determined based on the plurality of distance values 130 will. The majority of distance values can be used not only to record the movement of the hand "as a whole" relative to the depth axis, but also to individually determine the movement of the sections / parts of the hand.

The method includes determining 130 the movement of at least parts of the hand along the depth axis between the hand and the distance sensor based on the three-dimensional distance information. The movement of at least parts of the hand can be caused by a rotation of the hand, by a curvature (of parts) of the hand, or by a movement of the (entire) hand along the depth axis. In other words, the movement of at least parts of the hand along the depth axis does not only include a movement that is carried out parallel to the depth axis, but (any) movements of the hand that include at least one component along the depth axis. For example, if the hand is bent, parts of the hand are moved along the depth axis (as well as orthogonally to it), while other parts remain rigid. When rotating, the entire hand (apart from the axis of rotation) is moved along the depth axis (moving around the axis of rotation). The axis of rotation can correspond to the natural axis of rotation of the hand (ie the hand is rotated by rotating the wrist and / or the upper arm). Furthermore, movements can be recorded in which the entire hand moves along the depth axis.

For example, the determining 130 determining the movement of at least parts of the hand 132 include a rotational movement of the hand. Different approaches can be chosen. For example, determining the rotational movement can include tracking a distance of a plurality of sections of the hand in order to detect a rotational movement of the plurality of sections about an axis of rotation. For example, a three-dimensional distance value can be determined for each section. The respective three-dimensional distance value can be compared with the corresponding distance value of a previous point in time over several points in time in order to determine a difference between the distance values. Based on this, it can now be determined whether the three-dimensional distance values change in a way that is caused by a rotation of the hand. Alternatively, a plane can be formed from the plurality of distance values (for example by interpolation), and the rotation of the hand can be determined by recognizing a change in the position of the plane. Alternatively, the plurality of distance values can be used as input values for a neural network, the neural Netz is trained to determine the rotational movement of the hand based on distance values.

In some exemplary embodiments, determining the movement of at least parts of the hand can be determining 134 a rotational or curving movement of at least parts of the hand. Analogous to determining 132 Various approaches can be used for the rotational movement. For example, determining the rotational or curving movement can include tracking a distance of a plurality of sections of the hand in order to detect a rotational or curving movement of the plurality of sections (for example about an axis of rotation or relative to a base of the hand). For example, a three-dimensional distance value can be determined for each section. The respective three-dimensional distance value can be compared with the corresponding distance value of a previous point in time over several points in time in order to determine a difference between the distance values. Based on this, it can now be determined whether the three-dimensional distance values change in a way that is caused by a rotation or curvature of the hand (or else is caused by a movement of the entire hand along the axis). Alternatively, a plane can be formed from the plurality of distance values (for example by interpolation), and the rotation of the hand can be determined by recognizing a change in the position of the plane. The level can be divided into several sub-levels. If the position of the sub-planes changes with respect to one another, a curvature movement can be determined. Alternatively, the plurality of distance values can be used as input values for a neural network, the neural network being trained to determine the rotational movement and / or curvature movement of the hand based on distance values.

In addition or as an alternative to the rotation and / or curvature, a movement of the entire hand along the depth axis can also be determined, for example an upward or downward movement of the hand (if the distance sensor is arranged below or above the hand). For example, an average value of the plurality of distance values can be formed, i.e. the movement of the entire hand along the depth axis can be determined based on the average value of the plurality of distance values. The movement of the entire hand along the depth axis along the depth axis can be taken into account, for example, if it exceeds a distance threshold value within a predetermined period of time (for example more than 1 cm within 0.5 s). In addition, the movement of the entire hand along the depth axis along the depth axis can be taken into account, for example, if there is no (or only a slight) rotational movement and / or no (or only a slight) curvature movement.

The method further includes controlling 140 the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor. The user interface can be controlled based on the rotational movement, based on the rotational or curving movement and / or based on the movement of the entire hand along the depth axis. For example, the method can provide 140a a control signal for the user interface to effect control of the user interface. The user interface can be generated, for example, by an operating system of the processing module. The control signal can be provided to the operating system and interpreted by it. The user interface can be controlled based on the control signal. In order to visualize this control, an output of the user interface can be generated, which is projected onto the hand of the user via the projection device. In order to achieve this, the display signal can be generated based on the control of the user interface and the display signal can be provided to the projection device. The projection device can be, for example, a projector, for example a video projector such as an LCD-based projector (liquid crystal display) or a DLP-based projector (digital light processing). The projection device can be made compact. Correspondingly, the projection device can be a mini projector or a pico projector. Controlling the user interface can be visualized through the user interface. The user interface then includes one or more descriptions, one or more graphic elements, one or more selection options, one or more selection fields and / or one or more visual indicators. For example, selection options can be visualized as text or in the form of pictograms. Descriptions can be visualized as text. Visual indicators, such as selection fields, can be visualized using frames highlighted in color or frames highlighted by their light intensity. The selection of a selection option can be visualized by a selection field in the user interface. The visualizations are shown on the hand via the projection of the user interface.

The following shows how the particular movement can be used to control the user interface. In the following, it is assumed that the movement is converted into a control instruction (of the control signal) for the user interface, i.e. an abstract movement is processed and the control instruction is generated based on the movement (possibly context-dependent). Both the movements and the effects on the user interface are described in abstract terms, but it should always be noted that the respective movement is detected, a corresponding control instruction is generated based on the detected movement and this control instruction is suitable for controlling the user interface accordingly .

For example, the curving or rotating movement of the hand can be used to control the user interface. A selection option can be selected from a plurality of selection options by curving or rotating the hand. Thus, in a simple exemplary embodiment, the user interface can include a selection field, the plurality of selection options being able to be displayed sequentially in the selection field. Based on the rotational movement, the user interface can be controlled in such a way that the selection options are sequentially shifted (ie "scrolled") through the selection field. This can either happen as long as the rotational movement is being carried out (with a distinction being made between the two directions of rotation), or the displacement can be carried out as long as a rotational position of the hand is different from a rest position of the hand. The curvature of the hand can be used similarly. If the curvature of the hand creates a tilt, the selection options can be sequential through the selection field shifted, roughly along the inclination. This is a simple form of gravity control of the user interface. When the user interface is operated according to gravity, a position, curvature and / or rotation of the hand is interpreted in such a way that a selection field (similar to a “cursor”) or the selection options is / are moved along the force of gravity. If the hand is held at an angle (by rotation or curvature), the selection field or the selection options move downwards according to gravity. In other words, a selection field can be controlled by the curvature or rotation of the hand in such a way that the selection field moves between the selection options based on a position of the hand in accordance with the force of gravity. The selection field (the cursor) can be balanced in the hand. The selection field can, for example, be displaceable in one dimension (along an axis) or displaceable in two dimensions (along two axes that run orthogonally to one another). According to gravity means that, when the hand is tilted, the selection field moves to where a surface of the hand has a lower height above the ground (the user interface being projected onto the surface).

Alternatively or additionally, for example, a curvature of a finger can be used to control the user interface. For example, as in 4a and 4b shown, at least two selection options of the user interface projected onto at least two fingers 110 will. For example, the at least two selection options can be projected onto at least two fingers of the group of an index finger, a middle finger, a cow finger and a little finger. One of the at least two selection options can be selected if the corresponding finger is curved (ie if a curvature of the corresponding finger was determined via the three-dimensional three-dimensional distance information). The selection option can be selected, for example, if the curvature of the corresponding finger exceeds a predefined degree of curvature. Alternatively (or additionally) one of the at least two selection options can be selected if the corresponding finger touches another finger on which no selection option is projected (for example the thumb) due to the curvature of the finger. If the selection option is considered to be selected, this can be displayed via a visual indication, for example by lighting up the corresponding selection options in the projection. This visual indication can also be used in other exemplary embodiments, for example when selecting a selection option by moving the selection field according to the force of gravity.

In some embodiments, the user interface can be controlled based on the movement of the entire hand along the depth axis. For example, by moving the entire hand along the depth axis in the direction of the distance sensor, an element of the user interface can be activated, for example a selection option can be confirmed. For example, a selection option can be confirmed by jerking the hand upwards along the depth axis (e.g. in the direction of the distance sensor). Alternatively or additionally, an element of the user interface can be changed by moving the entire hand along the depth axis in the direction of the distance sensor, for example one of the plurality of options can be selected or a numerical value can be changed gradually or according to a predetermined interval (depending on the distance the entire hand from the distance sensor).

In addition to the position of the hand for selecting options, fixed gestures can also be used to carry out defined actions, for example to activate the user interface 154 or to deactivate. One possibility is based on recognizing whether the inside of the hand or the outside of the hand is turned upwards. In other words, the method can be recognition 150 whether the inside of the hand or the outside of the hand is facing the distance sensor. The method can also include activation 154 the user interface, if the palm of the hand is turned towards the distance sensor by a rotational movement. The method can also include deactivation 156 the user interface if the outside of the hand is turned towards the distance sensor by a rotational movement. Alternatively or additionally, the activation / deactivation can be based on whether the hand is opened or closed. The method can also include recognition 152 whether the hand is opened or closed based on the movement of at least parts of the hand along the depth axis. The method can activate 154 the user interface if it is detected that the hand is being opened and / or a deactivation 156 the user interface in the event that the hand is detected to be closed.

In some exemplary embodiments, a movement of the hand that is purely orthogonal to the depth axis can be disregarded. In other words, the control of the user interface can be orthogonal to the depth axis independently of a movement of the hand.

The method can also provide 160 an acoustic feedback based on the control of the user interface (e.g. via an acoustic output device), for example by providing an audio signal for the acoustic output device. The at least one interface can accordingly 12 be coupled to an acoustic output device, ie the system can comprise the acoustic output device. The acoustic feedback can take place, for example, each time the user interface is controlled, or (only) when a selection option is selected or confirmed. For example, a (subtle) warning tone can be played.

The at least one interface 12 can for example correspond to one or more inputs and / or one or more outputs for receiving and / or transmitting information, for example in digital bit values based on a code, within a module, between modules, or between modules of different entities. In exemplary embodiments, the processing module 14th correspond to any controller or processor or a programmable hardware component. For example, the processing module 14th can also be implemented as software that is programmed for a corresponding hardware component. In this respect, the processing module 14th be implemented as programmable hardware with appropriately adapted software. Any processors such as digital signal processors (DSPs) can be used. Embodiments are not restricted to a specific type of processor. There are arbitrary processors or also several processors for the implementation of the processing module 14th conceivable.

More details and aspects of the method, the device, the computer program and the vehicle are mentioned in connection with the concept or examples which are described before or after. The method, the device, the computer program or the vehicle may include one or more additional optional features that correspond to one or more aspects of the proposed concept or the examples described, as described before or after.

Embodiments generally deal with object control in body-projected user interfaces. In general, in body-projected user interfaces, a cursor on one hand can be controlled via lateral x / y movements.

In addition, a more intuitive way to control a cursor on a hand is explored. The object can be controlled by rotating the hand, for example, by balancing the cursor in the hand.

Such control of the cursor is described with the aid of the following figures.

2a to 2d show schematic drawings of an embodiment in which a telephone call is accepted. The phone call can be an example of notifications triggered by the system. In 2a becomes a hand 200 shown, the outside of which faces the distance sensor and the projection device. The user interface is projected onto the hand as it relates to the 1a to 1d was introduced. When a call comes in, an icon for the call appears, in this case a telephone receiver 210 , projected onto the hand. If the user turns the hand so that the inside of the hand is facing the distance sensor, the user interface is activated, as in FIG 2 B is shown. Then can a name of the caller 220 , an icon for answering the call (telephone receiver 230 ) and an icon for rejecting the phone call (telephone receiver 240 ) are displayed). By bending the hand (orthogonal to the direction of rotation) you can choose between the two options (see 2c ), with a selection box 250 is shown to illustrate which option is currently selected. By moving the hand along the depth axis in the direction of the distance sensor, the selection can be confirmed and the call accepted (this can be further clarified by an acoustic impulse), whereupon another menu of the user interface can be shown ( 2d ), which also contains information about the call 270 , such as the call time, and another telephone receiver icon 260 includes. If the user wants to end the call, he can, by moving his hand along the depth axis in the direction of the distance sensor, the telephone receiver symbol 260 activate and the call will be ended. The selection can again be made by illuminating the symbol 260 can be visualized.

3a and 3b show schematic drawings of a further exemplary embodiment in which a selection menu is operated in 3 dimensions. This is, for example, a user-initiated interaction. The user interface can be opened by opening the hand 300 activated and, as in 3a shown. To make the appropriate choices 310 of the selection menu can be selected on the one hand the selection field / cursor 320 be balanced in the X / Y plane on the hand. By rotating or curving like a rotation, the cursor can be moved along a first axis 330 be moved, by bending the hand, as in 3b can be seen, orthogonal to the first axis, the cursor can be along a second axis 340 be moved. The cursor illustrates which Option is selected. If the user wants to confirm a selection option, he can, similar to the example of 2a to 2d to do this by moving the hand along the depth axis in the direction of the distance sensor (by lifting the hand). Alternatively or additionally, values can be adjusted by raising or lowering the hand along the depth axis when the cursor is resting on the corresponding value. If the user closes the hand (by bending and / or rotating the hand so that the outside of the hand is facing the distance sensor), the user interface can be "closed" (deactivated).

4a and 4b show schematic drawings of a further embodiment in which a category is selected by tapping fingertips. Doing this is done on the fingertips of the hand 400 each symbols 410 - 440 projected that match the corresponding categories. If a finger is now bent, a selection of a category is made, which is indicated by a selection field / cursor 450 is shown. If the selection is to be confirmed, as in 4b shown, the thumb is guided to the corresponding finger and the category is selected by tapping.

The aspects and features that are described together with one or more of the previously detailed examples and figures can also be combined with one or more of the other examples in order to replace an identical feature of the other example or to add the feature in the other example to introduce.

Examples can furthermore be or relate to a computer program with a program code for executing one or more of the above methods when the computer program is executed on a computer or processor. Steps, operations or processes of various methods described above can be carried out by programmed computers or processors. Examples can also include program storage devices, e.g. Digital data storage media that are machine, processor, or computer readable and encode machine, processor, or computer executable programs of instructions. The instructions perform or cause some or all of the steps in the procedures described above. The program storage devices may e.g. B. digital storage, magnetic storage media such as magnetic disks and tapes, hard disk drives or optically readable digital data storage media or be. Further examples can also include computers, processors or control units which are programmed to carry out the steps of the method described above, or (field) programmable logic arrays ((F) PLAs = (field) programmable logic arrays) or (field) programmable ones Gate Arrays ((F) PGA = (Field) Programmable Gate Arrays) that are programmed to perform the steps of the procedures described above.

The description and drawings only represent the principles of the disclosure. Furthermore, all examples listed here are expressly intended to serve only illustrative purposes in order to support the reader in understanding the principles of the disclosure and the concepts contributed by the inventor (s) for the further development of the technology. All statements herein about principles, aspects, and examples of the disclosure as well as specific examples of the same include their equivalents.

A function block referred to as a “means for ...” performing a specific function can refer to a circuit that is designed to perform a specific function. Thus, a “means for something” can be implemented as a “means designed for or suitable for something”, e.g. B. a component or a circuit designed for or suitable for the respective task.

Functions of various elements shown in the figures, including each function block referred to as "means", "means for providing a signal", "means for generating a signal", etc. may be in the form of dedicated hardware, e.g. B “a signal provider”, “a signal processing unit”, “a processor”, “a controller” etc. as well as being implemented as hardware capable of executing software in conjunction with the associated software. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some or all of which can be shared. However, the term "processor" or "controller" is by no means limited to hardware that is exclusively capable of executing software, but can include digital signal processor hardware (DSP hardware; DSP = digital signal processor), network processor, application-specific integrated circuit (ASIC = application Specific Integrated Circuit), Field Programmable Gate Array (FPGA), Read Only Memory (ROM) for storing software, Random Access Memory (RAM), and non-volatile storage device (storage). Other hardware, conventional and / or custom, can also be included.

For example, a block diagram may represent a high level circuit diagram that implements the principles of the disclosure. Similarly, a flowchart, sequence diagram, state transition diagram, pseudocode, and the like may represent various processes, operations, or steps, for example, essentially represented in computer readable medium and so performed by a computer or processor, whether or not such Computer or processor is shown explicitly. Methods disclosed in the description or in the claims can be implemented by a device having means for performing each of the respective steps of these methods.

It is to be understood that the disclosure of multiple steps, processes, operations, or functions disclosed in the specification or claims should not be construed as being in order, unless explicitly or implicitly otherwise, e.g. B. for technical reasons is given. Therefore, the disclosure of several steps or functions does not limit them to a specific order, unless these steps or functions are not interchangeable for technical reasons.

Further, in some examples, a single step, function, process, or operation may include and / or be broken down into multiple sub-steps, functions, processes, or operations. Such sub-steps can be included and part of the disclosure of this single step, unless they are explicitly excluded.

Furthermore, the following claims are hereby incorporated into the detailed description, where each claim can stand on its own as a separate example. While each claim may stand on its own as a separate example, it should be noted that although a dependent claim in the claims may refer to a particular combination with one or more other claims, other examples also combine the dependent claim with the subject matter of each other dependent or independent claims. Such combinations are explicitly suggested here unless it is indicated that a particular combination is not intended. Furthermore, it is intended to include features of a claim for any other independent claim, even if that claim is not made directly dependent on the independent claim.

Claims (18)

A method of controlling a user interface, the method comprising: Projecting (110) the user interface onto a hand of a user; Determining (120) three-dimensional distance information of a distance between the hand and a distance sensor; Determining (130) a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information; and Controlling (140) the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor. The procedure according to Claim 1 wherein determining (130) the movement of at least parts of the hand comprises determining (132) a rotational movement of the hand, wherein the user interface is controlled based on the rotational movement. The procedure according to Claim 2 , further comprising recognizing (150) whether the inside of the hand or the outside of the hand is facing the distance sensor, and activating (154) the user interface if the inside of the hand is facing the distance sensor by a rotational movement. The method according to one of the Claims 1 to 3 wherein determining (130) the movement of at least parts of the hand comprises determining (134) a rotational or curving movement of at least parts of the hand, wherein the user interface is controlled based on the rotational or curving movement. The procedure according to Claim 4 , wherein a selection from a plurality of options is selected by a curvature or rotation of the hand. The procedure according to Claim 5 , the selection of the selection option being visualized by a selection field in the user interface, the selection field being controlled by the curvature or rotation of the hand such that the selection field moves between the selection options based on a position of the hand in accordance with the force of gravity. The procedure according to Claim 4 wherein at least two selection options of the user interface are projected (110) onto at least two fingers, one of the at least two selection options being selected if the corresponding finger is curved. The procedure according to Claim 7 , one of the at least two selection options being selected if the corresponding finger touches another finger on which no selection option is projected due to the curvature of the finger. The method according to one of the Claims 1 to 8th wherein determining (130) movement of at least parts of the hand comprises determining (136) movement of the entire hand along the depth axis, wherein the user interface is controlled based on movement of the entire hand along the depth axis. The procedure according to Claim 9 , an element of the user interface being changed or activated by moving the entire hand along the depth axis in the direction of the distance sensor. The method according to one of the Claims 1 to 10 wherein the three-dimensional distance information of the distance between the hand and a distance sensor comprises a plurality of distance values for different sections of the hand, the movement of at least parts of the hand being determined (130) based on the plurality of distance values. The method according to one of the Claims 1 to 11 , further comprising providing (160) acoustic feedback based on the control of the user interface. The method according to one of the Claims 1 to 12 wherein control of the user interface is orthogonal to the depth axis independent of movement of the hand. The method according to one of the Claims 1 to 13 wherein the method comprises providing (140a) a control signal for the user interface in order to effect the control of the user interface, the control of the user interface being visualized by the user interface. The method according to one of the Claims 1 to 14th , the method comprising recognizing (152) whether the hand is being opened or closed based on the movement of at least parts of the hand along the depth axis, the method activating (154) the user interface if it is recognized that the Hand is opened, and disabling (156) the user interface if the hand is detected to be closed. A method of controlling a user interface, the method comprising: Providing (110a) a display signal to a projection device to cause the user interface to be projected onto a hand of a user; Receiving (120a) a distance measurement signal from a distance sensor; Determining (120) three-dimensional distance information of a distance between the hand and the distance sensor based on the distance measurement signal; Determining (130) a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information; and Providing (140a) a control signal for the user interface in order to effect a control (140) of the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor. Program with a program code for performing the method according to Claim 16 if the program code is executed on a computer, a processor, a control module or a programmable hardware component. A device (10) for controlling a user interface (20), the device (10) comprising: at least one interface (12) designed for communication with a projection device (16) and with a distance sensor (18); and a processing module (14) designed to: Providing a display signal for the projection device (16) in order to project the user interface onto a hand of a user, Receiving a distance measurement signal from the distance sensor (18), Determination of three-dimensional distance information of a distance between the hand and the distance sensor based on the distance measurement signal, Determining a movement of at least parts of the hand along a depth axis between the hand and the distance sensor based on the three-dimensional distance information, and Providing a control signal for the user interface (20) to effect a control of the user interface based on the movement of at least parts of the hand along the depth axis between the hand and the distance sensor.

Images