WO2024126342A1

WO2024126342A1 - Method and system for capturing user inputs

Info

Publication number: WO2024126342A1
Application number: PCT/EP2023/085065
Authority: WO
Inventors: Finn Jacobsen; Sina BUCH
Original assignee: Gestigon Gmbh
Priority date: 2022-12-15
Filing date: 2023-12-11
Publication date: 2024-06-20
Also published as: DE102022133384A1

Abstract

The invention relates to a method for capturing user inputs in an input device, the input device having: an input surface (1) on which a graphical user interface (4) having at least one control element (13) is displayed; and a capturing device (3), the control element (13) being configured such that it can be moved within a movement range (15) of the user interface (4). In the method, a hand (7) of a user is captured by means of the capturing device (3), and the position (8) of the hand (7) in a capture region (10) is determined, the capture region (10) being a three-dimensional spatial region which is assigned to the input surface (1). It is determined whether the hand (7) of the user is located in an activation region (6), the activation region (6) being a part of the capture region (10) which is located at a distance (11) from the input surface (1), and it is determined whether the hand (7) of the user is located in a control region (5), the control region (5) being a part of the capture region (10) which is located between the input surface (1) and the activation region (6). Furthermore, a position of a pointer (9) on the input surface (1) is determined, the position of the pointer (9) being determined by projecting the position of the hand (7) onto the input surface (1) if the position of the hand (7) is detected in the capture region (10). If it is determined that the hand (7) of the user is located in the control region (5), (i) the control element (13) moves with the pointer (9) if it is determined that the pointer (9) is within the movement range (15) on the control element (13), and (ii) the control element (13) is positioned at an edge (15c) of the movement range (15) if it is determined that the pointer (9) is outside the movement range (15).

Description

METHOD AND SYSTEM FOR COLLECTING USER INPUT

The present invention relates to a method and a system for detecting user inputs in an input device, in particular in a vehicle. In particular, user inputs in a vehicle can be detected without contact.

Modern vehicles usually have at least one control unit installed, which the driver of the vehicle or other passengers can use to control various functions. These can be various vehicle or comfort functions, such as settings for the navigation system, the air conditioning, seat settings, lighting settings and the like. Various functions of an infotainment system can also be operated, such as playing music, making phone calls and the like.

Conventional control units can be controlled using corresponding control buttons. Modern systems usually have at least one display, which can be located centrally in the dashboard, for example. The individual functions, menus and the like can be shown here. These displays are often touch-sensitive, i.e. they are designed as touchscreens so that the desired function can be controlled by touching the screen. For this purpose, the control elements are designed in such a way that they can be reached and operated with a finger. A function can be called up when the corresponding area of the touch-sensitive screen is touched. Other control options can also be provided, such as holding and dragging a control element, swiping gestures and the like.

However, such control units require a touch-sensitive display and are therefore limited in their design. Modern vehicles often have multiple displays and increasingly larger displays. However, space is limited and the shape of the displays is usually essentially flat and rectangular and they cannot be distributed arbitrarily in a vehicle. They must be accessible to a user with their hand in order to be able to control a function by touch. Other surfaces in the vehicle without a display are not available for user input. In vehicles, it is also known to control certain functions contactlessly using gestures without reference to a display. To do this, a user performs certain predefined gestures in free space, for example in an area of the vehicle cabin above the center console or in front of the dashboard. Gestures in free space can be used to control functions of the infotainment system, for example, or other vehicle functions such as opening and closing a sunroof. While these gestures can be learned, they are performed in free space and a user does not receive immediate feedback, for example where or how exactly a certain gesture should be performed so that it successfully and reliably controls the desired function. Users are therefore often unsure and it can be difficult to perform the gestures correctly until they are recognized by the system.

In general, appropriate feedback is very helpful for usability. In particular, with touchless interaction, one problem is giving the user useful, supportive feedback. The user often feels lost, does not know how to interact and what the system actually recognizes.

The present invention is based on the object of improving the detection of user inputs in an input device, in particular in a vehicle. In particular, contactless detection of user inputs is to be improved.

This object is achieved according to the teaching of the independent claims. Various embodiments and developments of the invention are the subject of the subclaims.

A first aspect of the invention relates to a method, in particular a computer-implemented method, for detecting user inputs in an input device, wherein the input device has an input surface on which a graphical user interface with at least one control element is displayed, and a detection device, wherein the control element is configured such that it is movable within a movement range of the user interface. In the method, a user's hand is detected by means of the detection device and a position of the hand in a detection area is determined, wherein the detection area is a three-dimensional spatial area which is assigned to the input surface. It is determined whether the user's hand is in an activation area, wherein the activation region is a part of the detection region which is arranged at a distance from the input surface, and it is determined whether the user's hand is located in a control region, wherein the control region is a part of the detection region which is arranged between the input surface and the activation region. A position of a pointer on the input surface is further determined, wherein the position of the pointer is determined by projecting the position of the hand onto the input surface when the position of the hand is detected in the detection region. If it is determined that the user's hand is in the control region, (i) the control element moves with the pointer if it is determined that the pointer is within the movement region on the control element, and (ii) the control element is positioned at an edge of the movement region if it is determined that the pointer is outside the movement region.

The aforementioned method according to the first aspect is therefore based in particular on the fact that user inputs are recorded without contact. The position of the hand is determined in a three-dimensional spatial area which is assigned to the input surface, for example a spatial area which is located in front of the input surface. A detection area is divided into two areas. An activation area is arranged at a distance from the input surface.

If the hand is then in the control area, which is arranged between the activation area and the input surface, i.e. closer to the input surface, a control takes place, more precisely a movement of the control element within the movement area. The user does not have to touch the input surface for this, but it is sufficient if the hand is in the control area to control a corresponding function. A special touch-sensitive display is therefore not necessary, which allows a flexible design of the input surface. The possibility of contactless interaction means that surfaces can also be provided as input surfaces which are not suitable for operation that would require touching corresponding sensors. It goes without saying, however, that the user can still touch the input surface with his hand. However, this has no influence on the control within the meaning of the invention. Such an input surface can be located in a vehicle in particular. According to the method according to the first aspect, it is specified how a control can be moved within a defined area on the user interface, ie in a movement area. In this way, different types of controls or buttons can be defined, such as sliders or freely movable design elements.

In this context, particular attention is paid to edge cases. Interaction problems can arise when the user's finger (i.e. the pointer) exceeds the outer limits of the movement area, such as a visible area of the application. It should be ensured that the control cannot leave the movement area. If the finger (pointer) is outside the movement area, the control is positioned at the edge of the movement area instead of leaving the area and thus possibly no longer being visible to the user. This also allows for unusual shapes of the user interface, which may go beyond rectangular screens. It also takes into account the independent design of the input interface and the capture device.

The term "user interface" or "graphical user interface" used here refers in particular to a graphical representation of control elements which are linked to a specific function and allow a user to control the function. The user interface ("UI" or "graphical user interface" - GUI) can contain "control elements" ("UI elements") such as input areas, buttons, symbols, buttons, icons, sliders, toolbars, selection menus and the like which a user can operate, in particular in the sense of the present invention, without touching them. The (graphical) user interface can also be referred to as a (graphical) user interface.

The term "motion area" used here refers in particular to an area of the user interface, which can be a part of the user interface or the entire user interface. The motion area is thus a two-dimensional area that defines the movement limits of a movable control element.

The term “input surface” as used here refers in particular to any surface, in particular in a vehicle, on which the user interface can be represented graphically. It can be a display or a projection surface as described in more detail below. The input surface can have any design. It can be flat, level and rectangular, or have any shape. In particular, a point on the input surface can be described with two-dimensional coordinates.

The term "detection device" used here refers in particular to a device that can detect objects in three-dimensional space without contact and determine their position. In particular, the detection device can detect and localize a user's hand. For example, optical methods can be used to detect a user's hand in space. The detection device can consist of one part or several parts, which can depend on which detection area is to be covered. For example, one camera can be provided, or several cameras.

The term "three-dimensional spatial area" used here refers in particular to an area that can be described by three-dimensional coordinates. A position in the three-dimensional spatial area has unique three-dimensional coordinates. The coordinate system can be chosen arbitrarily. For example, a coordinate system of the detection device can be selected, or a coordinate system relating to the input surface. In particular, there is a relationship between the detection area and the input surface in order to be able to assign the position of the hand to corresponding locations on the input surface.

The term “vehicle” as used herein refers in particular to a passenger car, including all types of motor vehicles, hybrid and battery-powered electric vehicles, as well as vehicles such as sedans, vans, buses, trucks, delivery vans and the like.

The term "function" used here refers in particular to technical features which can be present in a vehicle, for example in the interior, in order to be controlled by a corresponding control system. In particular, these can be functions of the vehicle and/or an infotainment system, such as lighting, audio output (e.g. volume), air conditioning, telephone, etc. The terms "comprises,""includes,""has,""includes,""has,""with," or any other variation thereof, as used herein, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent in such method or apparatus.

Furthermore, unless explicitly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or". For example, a condition A or B is satisfied by one of the following conditions: A is true (or exists) and B is false (or absent), A is false (or absent) and B is true (or exists), and both A and B are true (or exists).

As used herein, the terms "a" or "an" are defined to mean "one or more". The terms "another" and "another" and any other variation thereof are defined to mean "at least one other".

The term “plurality” as used here should be understood to mean “two or more”.

The term "configured" or "set up" to perform a specific function (and respective modifications thereof) is to be understood in the sense of the invention that the corresponding device is already in a design or setting in which it can perform the function or that it is at least adjustable - i.e. configurable - so that it can perform the function after the corresponding setting. The configuration can be carried out, for example, by setting parameters of a process sequence or switches or the like to activate or deactivate functionalities or settings. In particular, the device can have several predetermined configurations or operating modes, so that configuration can be carried out by selecting one of these configurations or operating modes.

Preferred embodiments of the method are described below, which can be combined with each other as well as with the other aspects of the invention described, unless this is expressly excluded or is technically impossible. In some embodiments, the control element is positioned at a location on the edge of the movement area that is the shortest distance from the pointer. This maintains a relationship between the position of the pointer and the position of the control element even if the control element is not moved beyond the edge of the movement area. This location can also typically correspond to the location on the edge of the movement area at which the pointer left the movement area, or at least be very close to this location.

In some embodiments, the control is positioned at the edge of the movement area such that it remains completely visible within the movement area. This can prevent the control from disappearing even partially when the pointer is moved out of the movement area, so that operation is made easier at a later time when the control is to be grasped again. This can be solved, for example, by using an anchor point, such as a center of the control, as the position of the control, which must always be spaced from the edge of the movement area by at least half the size of the control (in particular if the anchor point is the center of the control).

In some embodiments, the control element moves along the edge of the movement area as long as it is determined that the user's hand is in the control area and that the pointer is outside the movement area. It can thus be provided that the control element is not automatically released when the pointer is moved out of the movement area, at least as long as the user's hand is in the control area, i.e. close enough to the input surface. The control element can then, for example, move along the edge of the movement area and be located at a point on the edge that is the smallest distance from the pointer. In this way, the user can still recognize control of the control element, even if the pointer is no longer in the movement area.

In some embodiments, the control element is positioned at the position of the pointer, ie in particular centered with respect to the pointer, when it is determined that the user's hand enters the control area and when it is determined that the pointer is within the movement area and preferably also on the Control is located. In other words, the control can jump to the position of the pointer. If the control is located exactly at the position of the pointer, rather than slightly offset, for example, it can be easier to use. An anchor point, such as a center point of the control, can be positioned at the position of the pointer as soon as the user's hand (or fingertip) enters the control area.

In some embodiments, the position of the pointer on the input surface is determined by orthogonal projection of the position of the hand onto the input surface when the position of the hand is detected in the detection area. The position of the hand in three-dimensional space is mapped to two-dimensional coordinates of the input surface during the (orthogonal) projection. As mentioned, the position of the pointer can be determined by an orthogonal projection of the position of the hand from three-dimensional space onto the two-dimensional input surface. The position of the hand can be defined by a point on the hand that is closest to the input surface, as explained below. This can also be a fingertip that points in the direction of the input surface. Instead of an orthogonal projection, it can also be provided to determine a pointing direction of a finger, for example an outstretched index finger, and to determine the position of the pointer on the input surface by extension in the pointing direction.

In some embodiments, the control area has a base area which contains at least the input surface and the control area extends from the base area to a cover surface, wherein a distance between the base area and the cover surface of the control area corresponds to the distance between the input surface and the activation area. The base area can also correspond substantially to the input surface. The base area and cover surface can be the same size. For example, the control area can be cuboid-shaped with the input surface as the base area. The control area thus directly adjoins the input surface and occupies the space in front of the input surface, in particular on the entire surface of the input surface. The distance can be selected according to the requirements, for example about 2 cm to about 5 cm, such as about 3 cm. In other words, the control area is the area in the immediate vicinity of the input surface. So if a user moves his hand close to the input surface (or touches it), i.e. into the control area, a function can be controlled. If the hand remains outside the control area, with other In other words, further away than the distance (or “depth” of the control area), there will be no control or “release” of the user interface.

In some embodiments, the activation area borders the control area, the activation area having a base area that contains at least the top surface of the control area. The base area of the activation area can correspond to the top surface of the control area. The activation area can be cuboid-shaped, for example. The activation area thus forms an area that is further away from the input surface than the control area. If a user approaches the input surface with his hand, his hand will typically first enter the activation area and only then the control area. If the hand is in the activation area, this can be used, for example, to "wake up" the user interface, which can be visually displayed. The user thus receives early feedback before the actual execution of the control if his hand is in the activation area. This can facilitate the subsequent control, since the user can see that his hand has been properly detected, which can facilitate navigation. This can be helpful, especially since the user moves his hand freely in the space in front of the input surface.

In some embodiments, a distance between the base and the top surface of the activation area is greater than the distance between the input surface and the activation area. The activation area is thus larger than the control area. This means that feedback can be given to the user in a larger area, while the actual control or operation of the user interface takes place in a smaller area near the input surface. Depending on the application, the dimensions can be adjusted accordingly. The activation area can, for example, have a depth of 10 cm or more.

In some embodiments, the representation of the user interface indicates the position of the pointer when the position of the hand is detected in the detection area. The representation can show the pointer itself, e.g. as a point on the input surface. However, it can also be provided to display the pointer in another way, for example by highlighting an area in which the pointer is located, such as a button which is located in the area of the pointer. This serves to give the user appropriate feedback as to where the pointer is located on the input surface in order to simplify operation. With other In other words, the area where the pointer is located becomes the focus for the user, so the activation area can also be called the “focus area”.

In some embodiments, the position of the user's hand is determined as a position of a point on the user's hand that is closest to the input surface. In order to allow a clear determination of the position of the user's hand, a point on the hand is advantageously selected that represents the position of the hand. In principle, any point on the hand can be selected as the position of the hand, such as a center of the hand. However, it is advantageous to use the point on the hand that is closest to the input surface for the position determination. This makes it easier to calculate the position of the hand. In addition, the frontmost point of the hand can be considered particularly relevant from the user's perspective for the input on the input surface.

In some embodiments, detecting the user's hand includes determining whether at least one finger of the hand is pointing in the direction of the input surface. In principle, the position of the hand can be arbitrary and the position of the hand can be determined in the detection area as described above. For a more precise determination, however, it can be determined whether the user is extending a finger in the direction of the input surface, for example the index finger. A user can use this hand position intuitively for operation. It can then be provided to control a function only when the user makes a pointing movement, e.g. by extending his index finger. If the user does not make a pointing movement, the control of a function can be suppressed, since in such a case the user may not intend to control a function. Determining the hand position can therefore improve the accuracy of the operation. In addition, the position of a fingertip can be determined with greater accuracy. This position can then be assumed to be the relevant position of the hand, in particular if the tip of the extended finger is the point on the hand that is closest to the input surface, as described above.

In some embodiments, the user interface function is activated when it is determined that the hand enters the control area and deactivated when the hand leaves the control area. For example, a control surface of the graphical user interface can be pressed when the user enters the control area with his hand (or fingertip), that is, comes very close to the input surface. (or touches it). As long as the hand is in the control area, the control surface can then be held, for example. It can then be released when the user withdraws his hand, ie when the hand (or fingertip) leaves the control area. The entry and exit of the hand into the control area can thus be used advantageously for interaction with the user interface. Further possibilities for how different events can be triggered by the position of the hand, or more precisely the distance of the hand from the input surface, are explained below.

In some embodiments, at least one transition event is further determined. A transition event can be determined when the hand passes from one of the areas to another, i.e. leaves one area and enters another. This can be the case in particular when it is determined that: the hand enters the control area, the hand leaves the control area, the hand enters the activation area, the hand leaves the activation area, the hand leaves the detection area, or the hand enters the detection area. Together with the position on the input surface, i.e. the pointer position as explained above, complete control of the user interface can be implemented in this way.

A second aspect of the invention relates to a system for data processing, comprising at least one processor which is configured to carry out the method according to the first aspect of the invention. The system also has at least one display device which is configured to display a graphical user interface on an input surface, and a detection device which is configured to detect a hand of a user in a three-dimensional spatial area and to determine a position of the hand in the three-dimensional spatial area.

In some embodiments of the system, the display device comprises a projection device which is set up to project a representation of the graphical user interface onto a surface. This allows a free design of the display of the user interface on any surface onto which the user interface can be projected. For example, areas of the dashboard of a vehicle can be used to display the user interface without having to be equipped with a display or the like. Other areas, such as the center console or the A-pillar of a vehicle, can also be used flexibly to display the user interface. A projector can be located, for example, in the roof lining, e.g. in the area of the interior mirror. and project the user interface onto any surface, which can then be used as an input surface for contactless control as described above. It goes without saying that the display device can also be designed as a display, which, however, does not have to be touch-sensitive in the sense of the invention.

In some embodiments of the system, the detection device comprises at least one image detection device, in particular a camera. The position of a user's hand in three-dimensional space can be determined in a simple manner using a camera. One camera or several cameras can be provided. The camera can be an infrared camera. The at least one camera is advantageously a time-of-flight camera (ToF camera). By using such a 3D sensor device, the position of the hand in three-dimensional space and its movement can be detected directly. 2D sensors can also be combined to detect the position of the hand in three-dimensional space.

A third aspect of the invention relates to a computer program with instructions which, when executed on a system according to the second aspect, cause the system to carry out the method according to the first aspect.

The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules.

The system according to the second aspect can accordingly have a program memory in which the computer program is stored. Alternatively, the system can also be set up to access a computer program that is available externally, for example on one or more servers or other data processing units, via a communication connection, in particular in order to exchange data with this program, which are used during the course of the process or computer program or represent outputs of the computer program.

The features and advantages explained with respect to the first aspect of the invention also apply accordingly to the other aspects of the invention.

Further advantages, features and possible applications of the present invention will become apparent from the following detailed description taken in conjunction with the drawings.

It shows:

Fig. 1 schematically shows an input device according to an embodiment with an input surface, a display device and a detection device;

Fig. 2 schematically shows a three-dimensional view of a detection area in front of an input surface;

Fig. 3 schematically shows a side view of a detection area in front of a

Input interface (hand in focus area);

Fig. 4 schematically shows a side view of a detection area in front of a

Input interface (hand in control area);

Fig. 5 shows schematically a control element with a pointer in different positions;

Fig. 6 schematically shows a range of motion of a movable control element;

Fig. 7 schematically shows possible directions of movement of a movable control element;

Fig. 8a, b, c schematically show the behavior of a movable control element at an edge; and

Fig. 9a, b schematically show the behavior of a movable control element with respect to a pointer. Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention.

Fig. 1 shows an input device for detecting a user input. A graphical user interface (GUI) is shown on an input surface 1. This can be projected onto the input surface 1 by means of a projector 2. Alternatively, the input surface 1 can also be formed by a screen, such as a conventional display, which does not have to be a touch-sensitive display. Thus, any surface can be used to detect proximity, for example any surface in a vehicle. As explained below, user inputs are detected without contact, and the user can receive appropriate feedback from the system to simplify the operation of the user interface. For this purpose, a detection device 3 is provided as a sensor, which detects a user's hand and determines its position in three-dimensional space. The detection device 3 can, for example, be a 3D sensor, such as a time-of-flight camera, or can comprise optical 2D sensors. The system enables the user to touch the input surface 1 (or possibly a screen) if he or she wishes to do so. In this way, a similar experience to conventional touchscreens can be achieved, but with certain differences in behavior. However, control is basically possible without touching.

The detection device 3 forms a hand recognition system that detects a person's hand in real time in a suitable field of view (the detection area) with a suitable level of detail. This means that the detection device 3 can be able to detect fingertip positions in a defined 3D world coordinate system (which may or may not coincide with the sensor coordinate system). In addition, it can be provided that a classification is made as to whether or not an extended (pointing) finger is present, i.e. whether a detected hand is currently making a pointing gesture. A control unit ("controller"; not shown) can also be provided that calibrates the system. In particular, the controller knows the physical position and size of the input surface 1 (in world coordinates), so that it can geometrically relate the detection data of a hand (e.g. finger positions) to the surface positions. The controller receives the inputs from the detection system and converts them into suitable inputs for the GUI application. As will be explained below, particularly with reference to Fig. 2, 3 and 4, the system enables the user to control a pointer 9 which represents a logical position on the input surface 1 or the user interface 4. The pointer 9 may or may not have a visual appearance. It may, for example, be represented as a point, or it may be invisible per se. In addition, the pointer 9 can logically be activated or deactivated at any time. The pointer 9 is deactivated when the hand is not currently in the vicinity of the input surface 1, i.e. outside the detection area 10, is not in a pointing position (so that no pointing finger can be detected) or for other reasons which are not currently detected by the hand detection.

It is advantageous if the user is provided to control the pointer 9 with a hand 7 that is in a pointing position, i.e. with an extended index finger 8 that is located near the input surface 1. In addition, the user can trigger a control (analogous to a mouse click) that is determined by the distance of the hand from the input surface 1. The various distance ranges in front of the input surface 1 are explained below.

The position of the index finger 8, as detected by the detection device 3, determines the position of the pointer 9 on the input surface 1. There are several variants of how an assignment can be realized. In particular, it should be noted that the finger position is a position in three-dimensional space (detection area 10), while the pointer position is a value in two-dimensional space (input surface 1 or screen).

A simple and effective variant of the assignment is an orthogonal projection of the position of the hand 7 (i.e. the tip of the index finger 8) onto the input surface 1. The detection device 3 detects a pointing finger 8 and the position of its tip. The position of the pointer 9 can then simply be the orthogonal projection of the position of the fingertip onto the input surface 1. If no pointing finger is detected, the pointer can be deactivated. A further simplified variant of this solution can consist in simply detecting the position of a hand tip, i.e. the frontmost point of the hand in a certain forward direction, and interpreting this as the finger pointing position. If the forward direction is suitably selected (for example orthogonal to the screen or to the input surface 1), then this detected position is actually the index finger position when the user points at the screen. This way, no classification of finger or pointing position is required.

It can also be provided that the detection device 3 recognizes a pointing finger 8 including its pointing direction. In this case, an index finger beam is determined that begins at the lower end of the finger and runs along the finger direction. The pointer position 9 is then the intersection point of the index beam with the input surface 1. However, this requires that an index finger is reasonably straight and a pointing direction can be assigned. If this is not the case, the recognition system would reject the input and the pointer would be deactivated.

The 3D finger position not only determines the pointer position, but also (when interpreted in a temporal context) a control or a trigger that leads to the control of a function. For this purpose, the detection area 10 in front of the input surface 1 can be divided into two three-dimensional spatial areas, which are referred to here as the control area 5 and the activation area 6. In the simplest case, these areas are cuboid-shaped boxes, as shown in Fig. 2. The control area 5 can also be referred to as the “trigger box” and the activation area 6 as the “focus box”. However, these areas can also take on other shapes depending on the shape of the input surface 1. The sizes given are only examples and can vary depending on the circumstances or the application, for example depending on the position and size of the input surface 1, content of the GUI, etc.

The base area of the control area 5 corresponds to the input surface 1. The depth of the control area 5 can be chosen appropriately, for example about 2 to 3 cm. The activation area 6 is another area that is located "in front of" or "above" the control area 5 and whose depth can also be chosen appropriately, for example about 10 cm. Now the hand 7 has a certain approximation state with respect to the input surface 1 and thus the GUI at any time, which can be expressed by the presence or absence of the hand 7 in the detection area 10 or, more precisely, the activation area 6 and the control area 5. This can be one of three defined states, as explained below, and is determined by the position of the index finger 8 (if a pointing hand is detected). If, as shown in Fig. 3, the position of the index finger 8 is in the activation area 6 (the “focus box”), this state can be referred to as “focus”. If, as shown in Fig. 4, the position of the index finger 8 is in the control area 5 (the “trigger box”), this state can be referred to as “trigger”. The “idle” state can be a situation in which the position of the index finger 8 is not in any of the fields, i.e. outside the detection area 10. This is especially true if no hand or index finger is detected at all. It is useful to apply a filter to avoid flickering between the states at times when the detected finger position is close to the box boundaries. A hysteresis, for example, is a suitable filter.

At any point in time (or frame), the hand input relevant to the system is completely described by the current pointer position (x and y coordinates) and the current approach state (z coordinate). This data, a pair of pointer position and approach state, can be referred to as the control state. Note that the information within the control state is conceptually the same as the (relevant) information about the detected hand, but is fully expressed in screen coordinates (or logical, non-geometric information) and no longer depends on the geometric setup and calibration of the system (position of the sensor, etc.). Therefore, the control state is a self-sufficient input to a graphical user interface (GUI) that provides complete information about a pointing device, similar to a mouse or touch input.

Particularly relevant for control are the moments in which the state of the approach changes, i.e. when the position of the hand leaves one of the areas and/or enters one of the areas. The (temporal) detection of the hand position can be based on frames, so that a state change of the approach occurs when the control state determined in frame n+1 results in a different state in the approach than the previous frame n. Therefore, the following transition events can be defined, which correspond to state changes and can be relevant for controlling the functions of the user interface.

The transition event "TriggerStateEnterEvent" is triggered when the state changes to "Trigger". The transition event "TriggerStateLeaveEvent" is triggered when the state "Trigger" changes to any other state. These two Transition events can be used to select functions, similar to a mouse click or touching a touchscreen. In this way, drag-and-drop functionality can also be implemented. As long as the hand (or the tip of the index finger 8) remains in the control area 5, an object of the user interface is held and then released when the hand 7 leaves the control area 5.

In addition, the following events can be defined. The transition event "FocusStateEnterEvent" is triggered when the state changes to "Focus". The transition event "FocusStateLeaveEvent" is triggered when the state changes from "Focus" to any other state. The transition event "IdleStateEnterEvent" is triggered when the state changes to "Idle". The transition event "IdleStateLeaveEvent" is triggered when the state changes from "Idle" to any other state. These events enable the implementation of additional functions. For example, the presentation of the user interface can react to the fact that the user is not near the input surface 1 with the controlling hand, but approaches the input surface 1 by entering the activation area 6 with his hand. The user thus receives feedback that the system has successfully responded to his hand 7 and he can continue operating the system.

In general, adequate feedback is very helpful for usability. In touchless interaction in particular, one problem is giving the user useful, supportive feedback. The user often feels lost, does not know how to interact and what the system actually recognizes. The system described here can provide direct feedback through spatial information about the finger position, so that the user understands what the system recognizes in order to react and adjust the finger position and movement. Different states for user interface controls ("UI elements") are described, which, depending on user interaction, return real-time information and thus feedback to the user. The system shows what it recognizes and which UI elements can be operated.

When interacting with Ul elements in a 3D space, the system must react sensitively to changes in the user's finger position. More feedback gives the user more security. The combination of horizontal and vertical movement of the Fingers together with the depth data (distance to the screen or input surface) leads to complex state changes, as described above.

Such feedback is particularly beneficial in a situation where a user is sitting in a vehicle and wants to interact with virtually displayed objects on a dashboard. As soon as they lift their finger to select an element of the user interface, clickable elements can appear larger and be easily highlighted. If they then move their finger over one of these elements, it can be additionally highlighted by a "hover effect". If they now click on the element, it can visually appear like a real button for clicking down. Over time, the user learns to interact better and better so that their "click" is recognized by the system. In addition to visual feedback, acoustic feedback can also be provided, for example various clicking noises.

Fig. 5 shows a schematic of a control element 13, which is shown in the form of a round button and is thus visible to a user as part of the user interface 4. For operation or control, however, it is advantageous if a user does not have to hit the control element exactly with the pointer 9. For this purpose, the visible area of the control element 13 is surrounded by an area 14. For the sake of simplicity, this is square along the x and y coordinates. On the left, a situation is shown in which the pointer 9 is outside the control element 13 and in particular also outside the area 14. The control element 13 is not addressed. However, if the pointer 9 is within the area 14, as shown on the right in Fig. 5, the control element 13 is addressed. The pointer 9 is regarded as “on the control element 13” if its position is at least in the area 14.

In the following, with reference to Fig. 6 to Fig. 9, it is described how a control element 13 can be moved, for example moved via drag & drop, by a user pressing with a finger 8 on the input surface 1 in the area of the control element 13 (see Fig. 5 and explanations therefor) and executing a swiping movement on the input surface 1. The control element 13 can also be moved if the user has not yet touched the input surface 1. For this purpose, the distance 1 1 to the input surface is defined, as explained in detail above. If the finger 8 (or the fingertip) falls below this distance 1 1 , i.e. if the finger moves into the control area 5, and is at that moment within the boundaries of the control element 13 (or the area 14 surrounding the control element 13), the control element 13 can be moved. Fig. 6 shows a movable control element 13 which can be moved on the user interface 4. The movement of the control element 13 is on the user interface

4 is limited vertically and horizontally to a movement area 15. The limits 15a, 15b in the vertical and horizontal direction can be defined individually and freely. The area 15 of the possible movement of the control element 13 defined in this way can correspond to the edges of the user interface 4, extend beyond them or only cover a certain (partial) area within the user interface 4.

Movable controls (“Ul elements”) can follow the user’s finger as long as the user is near the input surface 1. A movement of the control 13 is triggered when the finger enters the control area 5 (TriggerStateEnterEvent). Depending on the implementation, the finger must remain there for a short time or the movement can start immediately.

The finger 8 does not have to touch the input surface 1. In this way, Ul elements can be manipulated and moved with a finger that is close to, but not directly on, the input surface 1. By leaving the control area 5 (TriggerStateLeaveEvent), the control 13 is released so that it can no longer be moved and remains in the last position it had at the time of the transition event "TriggerStateLeaveEvent", i.e. when the finger 8 left the control area 5.

Fig. 7 shows a variant in which the permissible range of movement of the control element 13 is completely limited horizontally or vertically. The control element 13 can therefore only be moved in a straight line on one axis. This can simplify the operation of a slider, for example, since the user does not have to carry out the movement exactly horizontally or vertically, since the control element 13 remains on one axis. It goes without saying that the user must hold his finger 8 and thus the pointer 9 at least within the area 14 around the control element 13 in order to cause the control element 13 to be moved.

The situation may now arise that the control element 13 has reached the limits of the area 15, but the user has placed his finger within the control area

5 continues to move in the x and/or y direction. However, the control element 13 cannot or should not move out of the area 15. It should be noted in particular that the input surface 1 (or a display) and the detection device 3 are independent components. If the finger leaves the boundaries of the area 15 (or even the input surface 1) not by pulling the hand back but by crossing the edges, the movable control element 13 must also react accordingly to this situation and should not leave the area 15 in the x or y direction. As explained below, this is solved by positioning the control element 13 at the edge 15c of the area 15.

As shown in Fig. 8a, the control element 13 is positioned at the left edge 15c because the finger and thus the pointer 9 has left the defined area 15 (in the x direction). The distance between the control element 13 and the finger or pointer 9 is minimal without the control element 13 leaving the area 15. A similar situation is shown in Fig. 8b. Here the pointer 9 has left the area 15 in the x direction and y direction, so that the control element is positioned in the corner.

Even if the pointer 9 is outside the movement area 15, the user retains control of the control element 13 as long as his finger is close enough to the input surface 1 (i.e. within the control area 5). As soon as the pointer 9 returns to the area 15, the operation (i.e. moving) of the control element 13 can continue. The control element 13 is only released and remains in its last position when the user removes his finger from the input surface 1 (in the z direction). As long as the pointer 9 is outside the movement area 15, the control element 13 can then move along the edge 15c of the movement area 15, whereby it is always located at a point on the edge 15c that is the smallest distance from the pointer 9.

The control element 13 is thus placed on the edge 15c of the area 15 as follows. If the finger position (or pointer position) exceeds a limit value in the vertical or horizontal direction, the control element 13 is placed on this edge 16c (or in a corner). Both directions are set independently of one another. The control element 13 is placed such that an anchor point 16, which can in particular be the center or center of gravity, is spaced from the corresponding edge 15c so that the control element 13 remains completely visible. If the anchor point 16 is the center of the control element 13 (see also Fig. 8c), it remains spaced from the edge 15c by half the size of the control element 13 so that the control element 13 remains completely within the area 15. A possible behavior of a control element when the fingertip 8 enters the control area 5 (TriggerStateEnterEvent) is illustrated in Fig. 9a and Fig. 9b. Fig. 9a shows that the pointer 9 does not hit the movable control element 17 exactly. This is in principle sufficient for an actuation, i.e. moving the control element 17 by a corresponding hand movement. However, it can be advantageous for operation if the control element 17 lies exactly on the finger, i.e. the pointer 9. For this purpose, it can be provided that the center point 18 of the control element 17 can snap into the position of the pointer 9, as shown in Fig. 9b. This is also helpful, for example, when moving and rearranging objects on the user interface 4, such as icons, images and the like.

A movable control element on a user interface 4 in the system for contactless interaction described above can be used in particular in a vehicle, where the user interface 4 is projected, for example, onto the passenger's dashboard. The passenger can turn the music system up or down. With a movable control element, such as a horizontal slider, the user can, for example, adjust the volume with a touch gesture on the dashboard. The slider moves as soon as the user comes close to the input surface 1 with his finger or touches it and pulls his finger in one direction. Even if the finger then leaves the slider in the vertical direction, the user still has control over the horizontal movement. The slider is therefore not automatically released as soon as the finger is no longer directly above it. Only when the finger is removed (in the z direction) does the slider remain in its last position.

It can also be intended that a user can freely place different images on an area in a view. To do this, he can tap on an image and drag it with his finger to another position within the displayed boundaries of the user interface. Here, the image is thus designed as a movable control element.

While at least one exemplary embodiment has been described above, it should be noted that a large number of variations thereon exist. It should also be noted that the exemplary embodiments described are only non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description is intended to provide a person skilled in the art with guidance for implementing at least one exemplary embodiment, it being understood that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter defined in the appended claims and its legal

equivalents are deviated from.

LIST OF REFERENCE SYMBOLS

1 Input interface

2 Projector

3 Detection device (camera)

4 User interface

5 Control area

6 Activation area

7 Hand

8 Hand position / index finger

9 hands

10 Detection range

11 Depth of the control area

12 Depth of the activation area

13 Control (visible)

14 active area of the control

15 Range of motion

15a horizontal limitation of the range of motion

15b vertical limitation of the range of motion

15c Edge of the movement area

16 Anchor point

17 Control

18 Center of the control

Claims

EXPECTATIONS

1. A method for detecting user inputs in an input device, wherein the input device has an input surface (1) on which a graphical user interface (4) with at least one control element (13) is displayed, and a detection device (3), wherein the control element (13) is configured such that it is movable within a movement range (15) of the user interface (4), the method comprising:

Detecting a hand (7) of a user by means of the detection device (3) and determining a position of the hand in a detection area (10), wherein the detection area (10) is a three-dimensional spatial area which is assigned to the input surface (1);

Determining whether the user's hand (7) is located in an activation area (6), wherein the activation area (6) is a part of the detection area (10) which is arranged at a distance (11) from the input surface (1);

Determining whether the user's hand (7) is located in a control area (5), wherein the control area (5) is part of the detection area (10) which is arranged between the input surface (1) and the activation area (6); and

Determining a position of a pointer (9) on the input surface (1 ), wherein the position of the pointer (9) is determined by projecting the position of the hand (7) onto the input surface (1 ) when the position of the hand (7) is detected in the detection area (10); wherein, if it is determined that the user's hand (7) is in the control area (5),

(i) the control element (13) moves with the pointer (9) when it is determined that the pointer (9) is within the movement range (15) on the control element (13), and

(ii) the control element (13) is positioned at an edge (15c) of the movement range (15) when it is determined that the pointer (9) is outside the movement range (15).

2. Method according to claim 1, wherein the control element (13) is positioned at a location on the edge (15c) of the movement region (15) which has the shortest distance to the pointer (9).

3. Method according to claim 1 or 2, wherein the control element (13) is positioned at the edge (15c) of the movement area (15) such that it remains completely visible within the movement area (15).

4. Method according to one of the preceding claims, wherein the control element (13) moves along the edge (15c) of the movement area (15) as long as it is determined that the hand (7) of the user is in the control area (5) and that the pointer (9) is outside the movement area (15).

5. Method according to one of the preceding claims, wherein, when it is determined that the hand (7) of the user enters the control area (5) and when it is determined that the pointer (9) is within the movement range (15), the control element (17) is positioned at the position of the pointer (9).

6. Method according to one of the preceding claims, wherein the control region (5) has a base area which contains at least the input surface (1 ) and the control region (5) extends from the base area to a cover surface, wherein a distance (11 ) between the base area and the cover surface of the control region (5) corresponds to the distance (1 1 ) between the input surface (1 ) and the activation region (6).

7. The method according to claim 6, wherein the activation region (6) adjoins the control region (5), wherein the activation region (6) has a base area which contains at least the cover surface of the control region (5).

8. The method according to claim 7, wherein a distance (12) between the base surface and the top surface of the activation region (6) is greater than the distance (11) between the input surface (1) and the activation region (6).

9. Method according to one of the preceding claims, wherein the position of the hand (7) of the user is determined as a position of a point of the hand (7) of the user which is closest to the input surface (1). Method according to one of the preceding claims, wherein detecting the hand (7) of the user comprises determining whether at least one finger (8) of the hand (7) points in the direction of the input surface (1). Method according to one of the preceding claims, further comprising: determining a transition event when it is determined that: the hand enters the control area (5); the hand leaves the control area (5); the hand enters the activation area (6); the hand leaves the activation area (6); the hand leaves the detection area (10); or the hand enters the detection area (10). System for data processing, comprising at least one processor configured to carry out the method according to one of the preceding claims, and at least one display device (2) configured to display a graphical user interface (4) on an input surface (1), and a detection device (3) configured to detect a hand (7) of a user in a three-dimensional spatial area and to determine a position of the hand in the three-dimensional spatial area. System according to claim 12, wherein the display device comprises a projection device (2) which is set up to project a representation of the graphical user interface (4) onto a surface (1). System according to claim 12 or 13, wherein the detection device (3) comprises at least one camera. Computer program with instructions which, when executed on a system according to one of claims 12 to 14, cause the system to carry out the method according to one of claims 1 to 11.