WO2020043440A1

WO2020043440A1 - Directional estimation of an open space gesture

Info

Publication number: WO2020043440A1
Application number: PCT/EP2019/071079
Authority: WO
Inventors: Niklas HERMES; Sascha KLEMENT
Original assignee: Gestigon Gmbh
Priority date: 2018-08-31
Filing date: 2019-08-06
Publication date: 2020-03-05
Also published as: DE102018121317A1

Abstract

The invention relates to a method, an apparatus and a computer program for estimating a direction of an open space gesture at a man-machine interface. The method comprises: receiving (SO) measurement data that have been sensed by sensors and that represent the open space gesture as a point cloud in a multidimensional space; determining (S4) density data that represent a density distribution corresponding to the point cloud, for which density distribution each individual point in the point cloud is assigned information relating to density which characterises a local point density of the point cloud in the environment of each point; applying (S6) a dimension-reducing approximation method to the density data in order to determine estimation data that represent a one-dimensional direction curve which characterises a directional aspect of the density distribution; and generating (S8) and preparing directional data representing directional information that is determined by the spatial plot of the direction curve represented by the estimation data, in the superordinate multidimensional space.

Description

DIRECTIONAL ESTIMATION OF A CLEAR SPACE GESTURE

The present invention relates to a method and a device for estimating a directional information conveyed by a free space gesture for determining a user input on a human-machine interface, in particular a vehicle. Furthermore, the invention relates to a computer program that is configured to execute the method.

While classic human-machine interfaces (abbreviated "MMS" or more frequently "MMI") are regularly based on a user's contact-based interaction with a corresponding control element, for example a physical switch or a touch-sensitive surface or display device, etc., MMI are now known , in which a static or dynamic gesture of a human user, performed in free space without physical contact with an MMI device, is sensed. This sensory detection of such gestures, also known as “free space gestures”, can be carried out, for example, using a suitable camera. Then, by means of mathematical methods, in particular image recognition methods, conclusions are drawn from the sensor-acquired data on a user input intended by means of the recorded gesture.

In particular in connection with vehicle-related applications, such mils capable of recognizing free-space gestures can be used to make inputs for operating the vehicle or a subsystem thereof. Furthermore, it is also possible for the free space gestures to relate to the surroundings of the vehicle, such as pointing to an interesting object that is visible from the vehicle and external to the vehicle, and the recognition of such a free space gesture serves to provide an input for a system , which is to provide information relating to the object or causes the vehicle to react to it.

For the implementation of such contactless MMI, gesture recognition systems are known in which calibrated 2D sensors are used in order to detect and track points associated with certain features along the contour of a user's hand, in particular from above and from the side. On the basis of the detected points, a pointing direction associated with the gesture carried out in three-dimensional space can then be estimated. the. Such systems are dependent on the recognition of clearly distinguishable features (points) of the human hand and require fixed and calibrated hardware with at least two 2D sensors.

In addition, methods are known in which body-related information is used to infer a pointing direction expressed by a gesture. In most cases, a head position and a hand position are detected and a display direction is estimated based on this. Alignments or even movements of the fingers of the hand are not detected or taken into account.

The present invention has for its object to further improve the detection of free space gestures and in particular the determination of a mediating directional information.

This object is achieved in accordance with the teaching of the independent claims. Various embodiments and developments of the invention are the subject of the subclaims.

A first aspect of the invention relates to a, in particular computer-implemented, method for estimating direction information conveyed by a free space gesture for determining a user input on a human-machine interface, in particular a vehicle. The method comprises: (i) receiving sensor-acquired measurement data, which a free space gesture carried out by a user with respect to the human-machine cut parts, in particular by means of arm, hand and / or fingers, as spatially multidimensional, ie 2-dimensional or 3-dimensional represent dimensional, inhomogeneous point cloud in a correspondingly multidimensional space; (ii) determining, on the basis of the measurement data, density data which represent a density distribution corresponding to the point cloud, in which case density information is assigned to the individual points of the point cloud, which identifies a local point density of the point cloud in the vicinity of the respective point; (iii) applying a dimension-reducing approximation method to the density data to determine estimated data which is an estimate of a one-dimensional direction curve characterizing at least one direction aspect of the density distribution; represent; and (iv) generation and provision of directional data which represent directional information which is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space. “Direction information” in the sense of the invention is to be understood in particular to mean information that indicates at least one specific, selected direction in space. Optionally, it can also specify its temporal dependence, if necessary.

A “point cloud” in the sense of the invention is to be understood in particular as a set of points of a vector space, in particular of the three-dimensional space or of a two-dimensional subspace thereof, which has an unorganized spatial structure (“cloud”). A point cloud is described by the points contained therein, each of which can be detected in particular by their spatial coordinates in a coordinate system assigned to the vector space.

A “direction curve” in the sense of the invention is to be understood in particular as a one-dimensional curve or in particular a one-dimensional direction curve in a higher-dimensional space, which has at least one directional aspect characterized by it. In the simplest case, the direction curve can represent a straight line as a one-dimensional direction curve, the direction of which at the same time indicates the direction aspect. In the general case, however, it can also have any curvature, including multiple curvatures. The directional curve is preferably at least for the most part, ideally everywhere, continuous and differentiable.

In the solution according to the invention, the desired directional information is consequently not determined directly from the point distribution of the recorded point cloud, but instead a density distribution representing a probability distribution is first derived from the point distribution, on the basis of which a one-dimensional directional curve is determined only by means of the dimension-reducing approximation method mentioned, which characterizes the desired direction information. In this way, the achievable reliability and accuracy in the estimation-based determination of directional information from point clouds can be increased. In addition, the method requires only relatively low computing resources, so that it is particularly suitable for real-time capable applications. In addition, the method is independent of a specific orientation of the sensor used to record the measurement data, for example a camera, which makes it particularly suitable for determining, in particular also finger-based, gestures of free space within a vehicle, since the sensor is provided in a wide variety of positions there can be. Preferred embodiments of the method are described below, each of which, unless expressly excluded or technically impossible, can be combined with one another as well as with the other described other aspects of the invention or used as corresponding embodiments of the latter.

In some embodiments, the sensory multidimensional detection of the free space gesture carried out by a user with respect to the human-machine cut parts is carried out using a TOF camera system. A “TOF camera system” in the sense of the invention is to be understood as a 3D camera system that measures distances on the basis of a time of flight (“TOF”) method. The measuring principle is based on the fact that the scene to be recorded is illuminated by means of a light pulse, and the camera measures for each pixel the time it takes for the light to reach the object and back again. Due to the constant speed of light, this time is directly proportional to the distance. The camera thus supplies the distance of the object depicted on it for each pixel. The use of a TOF camera system represents a particularly effective and high-resolution implementation possibility for the at least one sensor required for sensory detection of the free space gesture.

In some embodiments, the multi-dimensional space is three-dimensional, and the measurement data represent a point cloud that depicts the detected free space gesture in three-dimensional, 3D.

In a first variant, a density distribution corresponding to this is determined directly for the 3D point cloud by assigning density information to the individual points of the 3D point cloud, which identifies a local density of the 3D point cloud in the area surrounding the respective point, wherein associated density data representing the resulting 3D density distribution are generated for the 3D point clouds. A dimension-reducing image of the 3D point cloud on one or in particular several point clouds of lower dimensions is not necessary in this variant, so that corresponding procedural steps can be saved.

In a second, alternative variant, the determination of the density data comprises: (i) dimension-reducing mapping of the 3D point cloud onto a plurality N, with N> 1, each spatially two-dimensional, 2D, point clouds, each in a 2D belonging to N - Subspaces of the multidimensional space are defined, at least two of which are not parallel to one another in the superordinate 3D space; and (ii) determining, for each of the N 2D point clouds, a density distribution corresponding thereto, in which the individual points of the respective 2D point cloud are each assigned density information which is a local density of the respective 2D point cloud in the vicinity of the respective one Dotts characterizes, whereby for each of the 2D point clouds the associated resulting 2D density distribution representing assigned density data are generated. Furthermore, the approximation method is used separately for each of the 2D density distributions represented by the respective density data to estimate a respective one-dimensional, 1 D, direction curve characterized by them, whereby for each of the 2D density distributions the assigned 1 D direction curve is assigned Estimated data are generated. In addition, the directional information is determined on the basis of a, in particular weighted or unweighted, averaging of at least two, preferably all, of the individual 1-D directional curves represented by the respective estimation data in the superordinate 3D space, by way of directional information related to the 3D space to obtain.

The latter can be done in particular by means of gradient determination with respect to a 1-D direction curve resulting from the averaging. These, as well as any further embodiments based on them, have the advantage that the complexity of the subsequent calculations can be significantly reduced due to the dimensional reduction that has been carried out, since overall less and only two-dimensional data have to be processed. In particular, the performance can be increased and thus the real-time capability of the method can be achieved or improved.

In some of these embodiments according to the second variant, the dimension-reducing imaging has a projection image along one of the three dimensions of the 3D point cloud represented by the measurement data onto a corresponding 2D point cloud in a corresponding one of the N 2D subspaces. The dimension reduction to two dimensions is thus carried out by means of projection. For example, when using a Cartesian coordinate system, the projection can take place in particular along one of the axes of this coordinate system, so that, for example, all points of the point cloud in the three-dimensional space spanned by the coordinate axes X, Y and Z along the Z axis onto the X / Y Plane are projected. Of course, this is only an example and should not be understood as a limitation. In particular, the projection can take place along any projection direction and, instead of a Cartesian coordinate system, another coordinate system of the same dimension can be used to represent the coordinates of the points.

In some embodiments, determining the density data further comprises: (i) spatial quantizing the multidimensional point cloud represented by the measurement data

or optionally each of the 2D point clouds obtained therefrom by means of a respective dimension-reducing image, and (ii) determining the density data on the basis of a resultant density distribution of the points of the corresponding quantized point cloud. Instead of a continuous or fixed image depending on the discrete resolution of the corresponding sensor system generated the measurement data, the point cloud is now mapped onto a predefined spatial grid and thus quantized. The individual cells of the grid can be referred to in particular when using a rectangular or even cube grid in the three-dimensional case as "voxel" and in the two-dimensional case as "pixel". The quantization in turn simplifies the complexity of the subsequent calculations and in particular enables simple indexing for the voxels or the pixels instead of coordinate information. In particular, the complexity reduction can include that instead of floating point arithmetic, the simpler and less complex integer arithmetic can now be used. This simplifies and speeds up use, particularly in embedded systems. In addition, the type of quantization allows the accuracy of the system to be adjusted, in particular even increased, since the type of quantization enables an adjustable trade-off between accuracy and speed (coarse grid = fast but inaccurate, fine grid = slower but more precise) .

In some developments of these embodiments, the quantization can additionally relate to a value assigned to the respective voxel or pixel, which is determined on the basis of the points of the point cloud present therein. The quantization can in particular be carried out in a binary manner, so that the value of a voxel or pixel, if its number of points lies below a predetermined threshold, is set to a first of the two binary values, for example to “0”, during the quantization, while at Number of points above the threshold this value is set to the second of the two binary values, for example "1". If the number of points is equal to the threshold, the value can be set so that it is set to one of the binary values in a predefined manner. In this way, the number of points to be processed can be reduced even further by means of these are taken into account by only considering those voxels or pixels with the second value in the further process, which further increases the performance of the process.

In some of these embodiments, in the case of spatial quantization of a 2D point cloud obtained by means of a respective dimension-reducing image, the density distribution for each point of the resulting 2D point cloud is determined on the basis of the number of those points of the multidimensional point cloud represented by the measurement data, which according to the Projection mapping can be mapped onto the same quantized point of the 2D point cloud. If, in the case of dimension-reducing imaging, many points are mapped onto the same pixel of the resulting 2D point cloud, this pixel receives a high value corresponding to the number of these points or derived therefrom, whereas in the opposite case if only a few points are on the Pixels are mapped, this value is correspondingly lower. In this way, the original point cloud can be converted into a two-dimensional density distribution, which characterizes the resulting respective 2D point cloud, in an effective and easy-to-execute and thus high-performance manner.

In some embodiments, the determination of the density data further comprises filtering the respective point cloud, on the basis of which the respective density distribution represented by the respective density data, by means of, preferably multiple, folding of the density distribution with a respective filter core, which in particular can have a low-pass property and in particular can be a smoothing kernel is determined. In one variant, the filter core can in particular be a Gaussian filter core. With the aid of this filtering, the original density distribution, as it results in particular from the quantization, is smoothed, which in particular can improve the applicability of the subsequent method steps and the quality of the resulting estimation results for the directional information. Without the filtering, the information between the points would not be interpolated, with the result that significantly less information would be available for the subsequent method steps, in particular for a PCA (described below), which in some cases could lead to less reliable results. Another advantage of filtering is that individual "wrong" points (eg those generated by measurement noise or incorrect measurements) are smoothed and thus have less influence on the further process, in particular, for example, a PCA. The noise usually associated with TOF sensors is thus suppressed by means of the filtering and the quality of the result can be increased in this way. In some embodiments, a main component analysis, PCA, or a regression method is used as the dimension-reducing approximation method for determining the respective estimation data.

The PCA is a particularly suitable and performant approximation method within the scope of the method according to the invention. The PCA can in particular be weighted so that smaller main components resulting therefrom are neglected for the resulting determination of the directional data. In particular, in the simplest case, the PCA can be terminated immediately after the determination of the first main component, so that a determination of the further main components is no longer necessary. The directional information represented by the directional data is then determined on the basis of only the largest main component ascertained by means of the PCA (i.e. those components with the greatest intrinsic value determined as part of the PCA). This enables a particularly performant and simple implementation.

In some embodiments in which a PCA with respect to a quantized and filtered two-dimensional density distribution F (i, j) the covariance matrix, M, the PCA receives the following initial values:

With

and

in which

and where (i, j) are index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values P (i, j) or density values F (i, j) filtered by means of convolution with the core K, and the kernel is indexed with the index pairs (n, m).

In some embodiments, the method further comprises: (i) segmenting the respective point cloud provided for determining the associated density data therefrom into a plurality of different segments on the basis of a recognition of different body parts of the user represented by the point cloud; (ii) selection of a real subset of the segments by means of a selection criterion which is defined as a function of the one or more body parts intended for performing the free space gesture; and (iii) determining the associated density data only based on the portion of the point cloud represented by the selected subset of the segments. In this way it is possible to determine the desired directional information solely on the basis of a section of the sensory point cloud. This is particularly advantageous if the point cloud additionally presents one or more other body parts or sections thereof beyond a body part of the user that is relevant for determining the directional information. The segmentation also allows, on the one hand, to avoid unnecessary processing of the portions of the point cloud that are not relevant in this sense, and on the other hand increases the required reliability and accuracy of the results, since these cannot be falsified by irrelevant portions of the point cloud.

In some embodiments, the point cloud represented in the measurement data can correspond in particular to a free space gesture performed with at least part of the upper limbs of a user. The segmentation contains at least one selection segment that only corresponds to one or more fingers of the user, and the selection criterion is defined in such a way that the subset of the segments determined thereby contains at least, preferably exclusively, this selection segment. In spite of sensory detection of a larger part of the upper limbs, focusing can be achieved by means of the segmentation solely on aspects of the open space gesture carried out by one or more fingers of the user. In particular, a pointing direction expressed with the finger or fingers can be recognized reliably, with high performance and / or with high accuracy. In some embodiments, the measurement data represent a temporal development of the free space gesture as a spatially multidimensional and at the same time correspondingly time-dependent point cloud. In addition, the procedural determination of a directional information provided by the free space gestures is carried out several times, in each case for a different point in time with respect to the time-dependent point cloud. The directional information is generated and made available in such a way that it represents the temporal development of the directional information resulting from this multiple determination. In this way, a temporal development of the detected free space gesture can also be tracked, for example a temporal change in direction information, such as a pointing direction, expressed by the free space gesture.

In some embodiments (i), the measurement data in M + 1 dimensions represent a temporal development of the free space gesture as a spatially M-dimensional and at the same time correspondingly time-dependent point cloud, with M> 1, (ii) the determination of the density data is a dimension-reducing mapping of the the measurement data in M spatial and an additional temporal dimension represented point cloud with the temporal dimension omitting an at least two-dimensional space in order to obtain a representation of a 2D or 3D space area corresponding to the free space; and (iii) the directional data are generated and provided in such a way that the directional information represented by them, which is determined by the spatial profile of the directional curve represented by the estimated data in the superordinate multidimensional space, is defined by the directional curve and thereby by it characterizes at least partially enclosed 2D or 3D spatial areas. The dimension-reducing mapping of the point cloud represented by the measurement data in M spatial and an additional temporal dimension can be carried out in particular by means of projection along the temporal dimension and, in the case of M = 3, optionally along one of the spatial dimensions. In this way, the method can be expanded or configured in such a way that it can be used to estimate a directional information conveyed by a free space gesture for determining a user input on a human-machine interface, the directional information being replaced by or in addition to a pointing direction Free space gesture defines the defined area, and thus in particular a region, ie describes a 2D or 3D spatial area.

This should be explained using a short example: Suppose the user executes a three-dimensional pointing gesture (M = 3), in which he moves the hand or arm in a circular motion. leads so that the pointing direction has a time course in which it describes a cone. The tip of the cone is (i) the finger root (in the case of a pure finger movement) or (ii) the elbow in the case of a forearm movement emanating from it) or (iii) the shoulder joint (in the case of a movement of the entire arm emanating from it), and the region desired by the user is located inside the cone. The point cloud can then first be described in M + 1 = 4 dimensions with three space and one time dimension and from this a dimension-reducing projection along the time dimension and optionally one of the space dimensions onto a three- or two-dimensional space can follow. The resulting point cloud in this reduced space could then (for example for the above gesture) have a shape or behavior that at least partially encloses a spatial area, for example in which it has an at least largely closed ring shape. Proceeding from this, corresponding density data can then be generated in accordance with the method and, in turn, directional information can be generated from this, which describes the region outlined by the free space gesture. The applicability of the method to the estimation of pure pointing gestures to the estimation of spatial gestures can thus be expanded.

In some embodiments, determining a direction information conveyed by the free space gesture on the basis of the estimated data comprises determining at least one direction vector lying tangential to the direction curve represented by the estimated data. This can be done in particular by means of a derivation in the sense of a mathematical differentiation or gradient formation with respect to the direction curve. In this way, directional information characterizing it can be obtained in a simple manner from the directional curve, which corresponds to an estimate of the directional information conveyed by the free space gesture.

A second aspect of the invention relates to a device for estimating a direction information conveyed by a free space gesture for determining a user input on a human-machine interface, in particular a vehicle, the device being set up to carry out the method according to the first aspect of the invention. The device can be, in particular, a control device for a vehicle, in particular a motor vehicle, where the control device itself has or is set up to have the above-mentioned man-machine interface parts, with which it is connected to receive the measurement data. A third aspect of the invention relates to a computer program configured to carry out the method of the first aspect of the invention. The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be traded independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloadable via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules.

The device according to the second aspect of the invention can accordingly have a program memory in which the computer program is stored. Alternatively, the device can also be set up to access a computer program which is available externally, for example on one or more servers or other data processing units, in particular in order to exchange data with it, which are used during the course of the method or computer program or Represent editions of the computer program.

The features and advantages explained above in relation to the first aspect of the invention also apply accordingly to the other aspects of the invention.

Further advantages, features and possible uses of the present invention result from the following detailed description in connection with the figures.

It shows

1 schematically shows a system for estimating direction information, including an estimation device, according to various embodiments of the invention;

FIG. 2 shows a flow diagram to illustrate a preferred embodiment of the method according to the invention; FIG. 3 for the purpose of illustrating the segmentation process S1 from FIG. 2, a two-dimensional representation of an exemplary point cloud represented by corresponding measurement data obtained by recording a human arm, wherein a segment corresponding to the fingers of the arm is identified;

FIG. 4 shows graphical representations to illustrate data occurring during the method according to FIG. 2 and derived from the measurement data; FIG. and

5 shows an exemplary illustration of an interpretation of a detected free space gesture as a spatial gesture for estimating directional information, which defines a 2D area region selected by the free space gesture, according to embodiments of the invention.

Throughout the figures, the same reference numerals are used for the same or mutually corresponding elements of the invention.

Fig. 1 shows schematically a system 1 for estimating direction information according to various embodiments of the invention. The system 1 has an estimation device 2 and a sensor 3 for three-dimensional detection of a free space gesture carried out by a user of the system. The estimation device 2 can in particular be a computer, which can be equipped with a processor platform 2a and a program and data memory 2b and a data output 2c for outputting output data determined by the computer, in particular in the form of directional information, the sensor 3 in particular have a TOF (f / me of /// g / 7f) camera system. Other types of sensors, in particular sensors based on ultrasound or radar measurement, can also be used instead or in combination therewith. A computer program can be stored in the program memory 2b, which can consist of one or more program modules and which is configured to cause the estimating device 2 when it runs on the processor platform 2a to perform the method according to the invention, for example as follows based on FIGS. 2 to 5 described.

The flowchart shown in FIG. 2 serves to illustrate a preferred embodiment of the method according to the invention for estimating direction information conveyed by a free space gesture for determining a user input on a human-machine interface, which in particular corresponds to system 1 or can have one . The actual estimation process, which processes S1 to S8 1 is preceded by a measurement process SO, in which measurement data MD are generated by sensors, in particular with a measurement sensor 3 according to FIG. 1, which represent a spatially three-dimensional point cloud, which in turn represents a free space gesture detected by the user during the measurement. FIG. 3 shows a two-dimensional representation of an exemplary 3D point cloud obtained in this way, which represents parts of the upper arm, the lower arm and the hand of an arm of a user performing the free space gesture.

In the embodiment of the estimation method described here, the three-dimensional point cloud represented by the measurement data MD is first segmented as part of a preprocessing as part of a segmentation process S1 using one or more known methods, for example using a "random forest" classification method, to generate segment data SD to be obtained which correspond to the section of the measurement data MD relevant for the further course of the process. In Fig. 3, such a section, which corresponds to the fin like the hand, is marked by a black box.

The point cloud represented by the segment data SD is then imaged in a projection process S2 in N = 2 two-dimensional point clouds. This is advantageously done by means of projection along one of the three spatial dimensions, here by way of example - without this being understood as a restriction - when using a Cartesian coordinate system, on the one hand along the Z direction, so that a two-dimensional projection of the point cloud onto the XY plane results, and on the other hand along the Y direction, so that there is accordingly a second projection on the XZ plane. FIGS. 4 (a) and 4 (e) graphically show these two-dimensional projections resulting from the segment data SD, which are each represented by corresponding projection data BD _xy or BD _XZ .

A quantization process S3 follows, in which each of the 2D point clouds represented by the projection data BD _xy or BD _{XZ is} quantized according to a predefined spatial grid, so that now only predefined spatial positions are defined for the points (pixels). The two quantized 2D point clouds, which are shown graphically by way of example in FIGS. 4 (b) and 4 (f), are represented by corresponding image data ID _xy and ID _XZ . The image data can thus be represented by corresponding data fields as follows:

I _xy (i, j) and I _xz (i, j) with the 2 D pixel indexing i = 1, ..., k and j = l, ..., l The quantization process S3 concludes a preprocessing section of the method which precedes the actual estimation of the direction information.

This estimation takes place in an estimation section of the method in which corresponding density data PD _xy or PD _XZ are first obtained from the quantized image data ID _xy or ID _XZ in a probability estimation process S4. For this purpose, a density value is assigned to each pixel of the two quantized 2D point clouds, which is equal to the number of points that were mapped onto the corresponding pixel according to the image data using the preceding processes S2 and S3. The density data can thus be represented as follows by corresponding data fields:

P _xy (i, j) and P _xz (i, j) with i = l, k and j = 1, ..., I

FIGS. 4 (c) and 4 (g) graphically depict such quantized 2D data fields in which the value of a pixel is represented by a corresponding gray value for the purpose of illustration, a darker color corresponding to a higher probability. Instead of the number of points, it is also conceivable to choose a dependent or derived other value that is suitable for at least indirectly characterizing this number of points.

A filter process S5 now follows, in which the density data PD _xy or PD _{XZ is} folded by means of a core K _; which in particular is a Gaussian 2D core, for example of the form:

can be filtered as follows:

The core K is indexed with the index pairs (n, m). The kernel K has the variance s which can be selected as a parameter and a normalization factor a which is selected such that it normalizes the sum of all core elements to 1. Corresponding filter data FD _xy and FD _XZ, for example, result from the filtering, which in particular preferably represents smoothing Figures 4 (d) and 4 (h) are graphically represented as corresponding filtered pixel clouds F _xy (i, j) and F _xz (i, j).

In a subsequent approximation process S6, which is carried out with the aid of a dimensionally reducing approximation method, which in the present example is chosen as the main component analysis (PCA), corresponding estimation data ED are then obtained from the filter data, each representing a directional curve which is shown in FIGS. d) or 4 (h) is shown as a corresponding white arrow. In the simplest case, only the largest main component determined by means of the PCA is taken into account, which here in each case defines the direction and position of the corresponding white arrow.

If a PCA is used with regard to the quantized and filtered, two-dimensional density distribution F (i, j), the covariance matrix, M, of the PCA can in particular receive the following initial values:

rn it

and

in which

and where (i, j) each index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values P (i, j) or by means of folding with the core K ge filtered density values F (i, j), and the indexing of the kernel with the index pairs (n, m) follows.

Finally, the desired direction information can be derived from the estimated data ED in a direction determination process S8 and represented by means of corresponding direction data RD. This can be done in particular in such a way that the one-dimensional directional curves (white arrows) represented by the estimation data ED, which are defined in the respective 2D space, are transferred to a common three-dimensional space in which the two 2D spaces are each missing the third Dimension are expanded, and the estimated data ED are then averaged in the common 3D space (this corresponds to a message of the two white arrows) in order to obtain a resulting direction curve. In the more general case, the estimation data ED each represent a curved directional curve, so that after averaging the two directional curves, a generally likewise curved resulting directional curve is created in 3D space, from which a desired directional information is obtained, for example by means of gradient formation at a desired point of the result directional curve or linear regression.

The method previously described by way of example with reference to FIGS. 2 to 4 can in particular also be carried out for different points in time of the execution of the free space gesture, so that time-dependent directional information is obtained and the time dynamics of a free space gesture can thus also be recorded.

5 finally shows an exemplary illustration of an interpretation of a detected free space gesture as a spatial gesture for estimating directional information, which defines a 2D surface area selected by the free space gesture, according to embodiments of the invention.

Suppose the user carries out a three-dimensional pointing gesture (M = 3), in which he moves the hand or arm in a circular motion, so that the pointing direction has a temporal course (here marked by the five times to to t ₄ ) in which it describes a cone C. The cone tip T is (i) the finger root (in the case of a pure finger movement) or (ii) the elbow in the case of a forearm movement emanating from it) or (iii) the shoulder joint (in the case of a movement of the entire arm emanating from it) , and the region desired by the user is located inside the cone, optionally also in its projection along the cone axis towards even greater distances. The point cloud can then first be described in M + 1 = 4 dimensions with three space and a time dimension and from this a dimension-reducing projection along the time dimension and optionally one of the space dimensions into a three- or two-dimensional space. The resulting point cloud in this reduced space could then (for example for the above gesture) have a shape or behavior that at least partially encloses a spatial area, for example the hatched two-dimensional surface or region A, for example in which it, like 5 has an at least largely closed ring shape. Proceeding from this, method-specific density data FD and, in turn, directional information or corresponding directional data RD can be generated which describe or represent the region A outlined by the free space gesture. In particular, this can also be used to select an object X, which is located in region A, using the free space gesture. The object can in particular correspond to an operating element of a human-machine interface or another element of vehicle equipment. Objects outside the vehicle are also conceivable. Thus, the applicability of the method can be extended to the estimation of pure pointing gestures to the estimation of spatial gestures.

While at least one exemplary embodiment has been described above, it should be noted that a large number of variations exist. It should also be noted that the exemplary embodiments described are only non-limiting examples, and it is not intended to thereby limit the scope, applicability, or configuration of the devices and methods described here. Rather, the foregoing description will provide those skilled in the art with a guide to implementing at least one example embodiment, it being understood that various changes in the operation and arrangement of the elements described in an example embodiment may be made without departing from the attached subject matter as well as its legal equivalents are deviated from. Reference list

1 system for estimating directional information

2 estimation device

2a processor platform

2b program and data memory

2c data output

3 TOF camera system

MD measurement data

SD segment or segment data representing it

BD projection data

ID image data

PD density data

FD filter data

ED estimation data

RD directional data

A 2D area selected using free space gesture

C cone

T cone tip

X object enclosed by the selected 2D surface

to-t ₄ sequence of different times corresponding to the execution of the free space gesture

Claims

Expectations

1. A method for estimating a directional information conveyed by a free space gesture for determining a user input on a human-machine interface (1), in particular a vehicle, the method comprising:

Receiving sensor-acquired measurement data (MD), which represent a free space gesture carried out by a user with regard to the human-machine cut parts as a spatially multidimensional inhomogeneous point cloud in a correspondingly multidimensional space;

Determining, on the basis of the measurement data, density data (PD), which represent a density distribution corresponding to the point cloud, in which density information is assigned to the individual points of the point cloud, which identifies a local point density of the point cloud in the vicinity of the respective point;

Applying a dimension-reducing approximation method to the density data to determine estimated data (ED), which estimate an estimate of a one-dimensional direction curve that characterizes at least one direction aspect of the density distribution; represent; and

Generation and provision of directional data (RD), which represent a directional information that is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space.

2. The method of claim 1, wherein:

the multidimensional space is three-dimensional, and the measurement data represent a spatially three-dimensional, 3D, point cloud representing the detected free space gesture; determining the density data comprises:

- Dimension-reducing mapping of the 3D point cloud to a plurality N, with N> 1, each spatially two-dimensional, 2D, point clouds (BD), each of which is defined in one of N associated 2D subspaces of the multidimensional space, at least two of which are in the superordinate 3D space not parallel to each other; and

- Determine for each of the N 2D point clouds a corresponding density distribution (PD), in which the individual points of the respective 2D point cloud are each assigned ne density information, which is a local density of the respective 2D point cloud in the vicinity of the respective point indicates, for each of the 2D point clouds representing the respective resulting 2D density distribution and associated density data are generated;

the approximation method is used separately for each of the 2D density distributions represented by the respective density data to estimate a respective one-dimensional, 1 D, direction curve which is characterized by them, whereby for each of the 2D density distributions the associated 1 D direction curve representing associated estimated data is generated become; and

the direction information (RD) is determined on the basis of averaging at least two of the individual 1-D direction curves represented by the respective estimation data (ED) in the superordinate 3D space in order to obtain direction information relating to the 3D space.

3. The method according to claim 2, wherein the dimension-reducing mapping comprises a projection mapping along one of the three dimensions of the 3D point cloud represented by the measurement data onto a corresponding 2D point cloud (BD) in a corresponding one of the N 2D subspaces.

4. The method according to any one of the preceding claims, wherein the determination of the density data further comprises:

Spatial quantization of the multidimensional point cloud represented by the measurement data or, if applicable, each of the 2D point clouds obtained therefrom by means of a respective dimension-reducing image, and determination of the density data (PD) on the basis of a resulting density distribution of the points of the corresponding quantized point cloud (ID).

5. The method according to claim 4, wherein in the case of spatial quantization of a 2D point cloud obtained by means of a respective dimension-reducing image, the density distribution for each point of the resulting 2D point cloud is determined on the basis of the number of those points of the multidimensional point cloud represented by the measurement data , which are mapped to the same quantized point of the 2D point cloud according to the projection image.

6. The method according to any one of the preceding claims, wherein the determination of the density data (PD) further comprises: Filtering the respective point cloud, from which the respective density distribution represented by the respective density data is determined, by folding the density distribution with a respective filter core that has a low-pass property.

7. The method according to 6, wherein the filter core is a Gaussian filter core.

8. The method according to any one of the preceding claims, wherein a main component analysis, PCA, or a regression method is used as the dimension-reducing approximation method for determining the respective estimation data.

9. The method according to claim 8, wherein in the case of using a PCA, the direction information represented by the direction data (RD) is determined on the basis of only the largest main component determined by means of the PCA.

10. The method according to claim 8 or 9, wherein if a PCA is used with respect to a quantized and filtered, two-dimensional density distribution F (i, j), the covariance matrix, M, the PCA receives the following initial values:

With

and

in which

and where (i, j) are index pairs for indexing the cells of the quantized 2D subspace or their associated unfiltered density values P (i, j) or density values filtered by convolution with the core F (i, j), and the indexing of the core K with the index pairs (n, m).

1 1. The method according to any one of the preceding claims, further comprising:

Segmenting the respective point cloud provided for determining the associated density data therefrom into several different segments on the basis of a recognition of different body parts of the user depicted by the point cloud;

Selection of a real subset of the segments by means of a selection criterion which is defined as a function of the one or more body parts intended for the execution of the free space gestures; and

Determine the associated density data only on the basis of the part (SD) of the point cloud represented by the selected subset of the segments.

12. The method according to claim 1 1, wherein the point cloud represented in the measurement data (MD) corresponds to a free space gesture performed with at least part of the upper limbs of a user; the segmentation contains at least one selection segment (SD), which only corresponds to one or more fingers of the user, and the selection criterion is defined in such a way that the subset of the segments determined thereby contains at least this selection segment.

13. The method according to any one of the preceding claims, wherein the measurement data represent a temporal development of the free space gesture as a spatially multidimensional and at the same time time-dependent point cloud; and

the procedural determination of a directional information conveyed by the free space gesture takes place several times, in each case for a different point in time with respect to the time-dependent point cloud; and

the direction data (RD) are generated and provided in such a way that they represent the temporal development of the direction information resulting from this multiple determination.

14. The method according to any one of the preceding claims, wherein:

the measurement data in M + 1 dimensions represent a temporal development (to-t ₄ ) of the free space gesture as spatially M-dimensional and at the same time correspondingly time-dependent point cloud, with M>1;

The determination of the density data is preceded by a dimension-reducing mapping of the point cloud represented by the measurement data in M spatial and an additional temporal dimension while the temporal dimension is omitted to an at least two-dimensional space in order to represent a 2D, 3D or 3D To obtain space area (A); and

the directional data (RD) are generated and provided in such a way that the directional information they represent, which is determined by the spatial course of the directional curve represented by the estimated data in the superordinate multidimensional space, is defined by the directional curve and at least partially by it includes enclosed space.

15. The method as claimed in one of the preceding claims, wherein determining a direction information conveyed by the free space gesture on the basis of the estimated data comprises determining at least one direction vector lying tangentially to the direction curve represented by the estimated data.

16. The device (1) for estimating a directional information conveyed by a free space gesture for determining a user input on a man-machine interface, in particular a vehicle, the device being set up to carry out the method according to one of the preceding claims.

17. A computer program configured to carry out the method according to one of claims 1 to 15.