WO2024115386A1

WO2024115386A1 - Method for operating a vehicle function of a motor vehicle by means of ultrasound-based gesture control

Info

Publication number: WO2024115386A1
Application number: PCT/EP2023/083166
Authority: WO
Inventors: Maximilian Poepperl; Niko Moritz Scholz
Original assignee: Valeo Schalter Und Sensoren Gmbh
Priority date: 2022-11-29
Filing date: 2023-11-27
Publication date: 2024-06-06
Also published as: DE102022131463A1

Abstract

The invention relates to a method for operating a vehicle function (5) of a motor vehicle (1) by means of ultrasound-based gesture control. The method comprises: providing (S1) ultrasound information (10) which describes at least one object in an environment of at least one ultrasound sensor (2); determining (S2) movement information (12) which describes a movement sequence of the at least one object by applying a movement determination criterion (11) to the provided ultrasound information (10); checking (S4) whether the object is a body part intended for gesture control of the vehicle function (5) by applying an object checking criterion (15) to the provided ultrasound information (10) and the determined movement information (12); and, if the object is the body part intended for gesture control, operating (S6) the vehicle function (5).

Description

Method for operating a vehicle function of a motor vehicle by means of an ultrasound-based gesture control

The invention relates to a method for operating a vehicle function of a motor vehicle by means of an ultrasound-based gesture control. The invention also relates to a motor vehicle, a control device and a computer program product for carrying out such a method.

A motor vehicle can have a vehicle function that can be controlled by means of a detected gesture that is performed by a user. Such a vehicle function can be provided, for example, for gesture-controlled opening or closing of a trunk of the motor vehicle. The vehicle function operated by means of gesture control can be based on data from a capacitive system. Alternatively, it is possible for the gesture to be detected by means of at least one ultrasonic sensor of the motor vehicle.

For this purpose, the motor vehicle has at least one ultrasonic sensor that is designed to detect at least one object in the surroundings of the motor vehicle. By evaluating ultrasonic information provided by the ultrasonic sensor, a distance between the ultrasonic sensor and the object in the surroundings can be determined. If ultrasonic information is provided by several ultrasonic sensors, the object in the surroundings can also be localized. Typically, the motor vehicle has several ultrasonic sensors that are installed, for example, in a rear area of the motor vehicle, wherein these are intended in particular so that a parking assistance system can be provided for the motor vehicle.

DE 10 2020 116 716 A1 shows a method for activating an ultrasound-based gesture control when a person approaches a vehicle. An ultrasound signal is emitted by a subset of ultrasound sensors and at least one echo signal is received by a subset of the ultrasound sensors. An object classification is then carried out in the echo signal to detect a person. If the person is positively detected, the ultrasound-based gesture control is activated.

Radar-based gesture-controlled vehicle functions are also known. For example, in this context, US 2019/0302253 A1 describes a method in which radar data is received and a trajectory of a movement is determined from this radar data. It is then determined whether the trajectory of the movement corresponds to a human signature, whereby the human signature is associated with a specific control of the vehicle.

In connection with ultrasound-based gesture control, it is known that these have a relatively high false trigger rate compared to, for example, radar-based solutions. The false trigger rate can be referred to as the false positive rate. It is at least the object of the invention to provide a solution by means of which a vehicle function for a motor vehicle, which is operated by means of ultrasound-based gesture control, is provided with a low false triggering rate.

The problem is solved by the subject matter of the independent patent claims.

One aspect of the invention relates to a method for operating a vehicle function of a motor vehicle by means of an ultrasound-based gesture control. The vehicle function is designed, for example, for contactless and thus gesture-controlled opening or closing of a trunk of the motor vehicle. Alternatively or additionally, the vehicle function can be designed for contactless opening or closing of a door of the motor vehicle. Alternatively or additionally, the function that is operated by means of ultrasound-based gesture control can be located in an interior of the motor vehicle, i.e. it can be controlled by means of a gesture performed in the interior of the motor vehicle.

The method includes providing ultrasonic information. The ultrasonic information describes at least one object in an environment of at least one ultrasonic sensor. The at least one ultrasonic sensor is preferably arranged in the motor vehicle, for example behind and/or in a bumper of the motor vehicle. The motor vehicle preferably has several ultrasonic sensors, for example six ultrasonic sensors in a rear area of the motor vehicle and/or six ultrasonic sensors in a front area of the motor vehicle. Alternatively or additionally, the at least one ultrasonic sensor can be arranged in the interior of the motor vehicle, for example in a roof area of the motor vehicle.

The ultrasound information can be provided, for example, by the at least one ultrasound sensor first emitting an ultrasound signal that is reflected by the object in the environment, whereupon the at least one ultrasound sensor receives the reflected ultrasound signal as an echo. The ultrasound information provided describes the object in the environment in such a way that it describes, for example, at least a distance to it. The ultrasound information can therefore be detected by the at least one ultrasound sensor of the motor vehicle. It can then be provided, for example, to a control device of the motor vehicle. The ultrasound information is preferably described by ultrasound data or comprises the ultrasound data.

The at least one object is, for example, a user of the vehicle function. The object is in particular a body part of the user, such as a foot, a leg, an arm and/or a hand. The object can thus perform the gesture by means of which the function is to be operated. Alternatively or additionally, the object can be a pedestrian, an animal, an object and/or another road user in the area. The area is spatially limited, for example, by a detection range of the at least one ultrasonic sensor. A range of the ultrasonic sensor can, for example, be limited to 5 meters, so that in this example a boundary of the detection range runs 5 meters from the ultrasonic sensor. The method includes determining movement information that describes a movement sequence, in particular a movement, of the at least one object. The movement information is determined by applying a movement determination criterion to the ultrasound information provided. The ultrasound information is therefore evaluated in such a way that information about the movement sequence performed by the object is available. The movement information can, for example, describe the length of time for which the object is moved, whether there is at least a waiting time without the object moving and/or the distance the object is from the ultrasound sensor during, before and/or after the movement. The movement sequence can consist exclusively of the movement of the object. The movement information is ultimately evaluated ultrasound information. The ultrasound information can therefore alternatively be referred to as raw data and the movement information alternatively as processed data of the respective ultrasound sensor.

The movement determination criterion is an algorithm and/or at least a rule, when applied, the ultrasound information provided is evaluated in such a way that the movement sequence of the object is determined and described. The movement determination criterion is based, for example, on machine learning methods, such as an artificial neural network. The artificial neural network has been trained, for example, to be able to use the ultrasound information to deduce whether and how the object is currently moving in the vicinity of the ultrasound sensor. It can therefore, for example, at least distinguish between approaching, waiting and/or performing a specific movement, such as a predetermined gesture. When training the object verification criterion, various types of moving objects in the vicinity of the ultrasound sensors are preferably taken into account, such as a pedestrian walking past the motor vehicle or stopping in front of it, an animal, a ball rolling past and/or another road user, such as a cyclist or another motor vehicle.

The method includes checking whether the at least one object is a body part intended for gesture control of the vehicle function. This is done by applying an object verification criterion to the ultrasound information provided and the determined movement information. The object verification criterion is an algorithm and/or at least one rule that can be used to at least determine whether the object is to be assigned to a specific body part of a person or not. The intended body part is, for example, the user's foot, leg, arm and/or hand. It may already be sufficient that the check determines that the object is assigned to the person and not, for example, to the animal or the object.

If the object is the body part intended for gesture control, the vehicle function is operated. For example, the vehicle function can then be activated or deactivated. For example, if it is determined that a body part intended for contactless opening of the trunk, such as the foot of a user, is the object for which the ultrasound information is to be provided and the movement information has been determined, the trunk can be opened or closed automatically without contact, for example. Ultimately, the vehicle function is operated using ultrasound-based gesture control if it has at least been determined that the body part intended for gesture control has been detected by at least one ultrasound sensor and that a movement has been made by this body part in the detection range of the ultrasound sensor.

To check whether the object is the intended body part, both the raw data, i.e. the ultrasound information provided, and the processed data, i.e. the movement information determined, are evaluated. This means that a false trigger rate, i.e. a false positive rate, can be kept low, since the object is checked based on both the raw data and the processed data. A vehicle function for a motor vehicle that is operated using ultrasound-based gesture control is therefore provided with a low false trigger rate.

An advantageous embodiment provides that the object verification criterion is based on machine learning methods. It is based in particular on an artificial neural network and/or a support vector machine. Methods of artificial intelligence are therefore used. The artificial neural network is, for example, a flat neural network, in particular a feedforward neural network. It is possible for the artificial neural network to have several concealed or hidden neurons and/or layers of such neurons. The support vector machine is an automatic method for pattern recognition that is provided in the form of software. The object verification criterion was previously trained at least to recognize, based on ultrasound information and movement information, that there is an object in the vicinity of the ultrasound sensor and that this object is a specific part of a person's body. Ultimately, the object verification criterion is used to reliably determine whether the recognized object can be assigned to a predetermined object class, whereby this object class describes the part of the body that is intended for gesture control of the vehicle function.

An additional embodiment provides that user movement information is determined which describes a movement sequence of the intended body part carried out by the user, in particular a movement of the intended body part carried out by the user. The object verification criterion is retrained taking into account the determined user movement information. It is therefore possible for an actual user of the motor vehicle to perform the gesture with his body part and thereby provide information on which body part is intended for which movement in order to operate the vehicle function. The already pre-trained object verification criterion, in particular the artificial neural network, can then be trained by means of a further learning process to recognize the user-specific gesture, for example, with the possibly user-specific selected body part. By taking the user's movement sequence into account, approaches and waiting times typical for this user before or after the movement of the body part can also be taken into account. A fine adjustment of a Detection threshold of the object verification criterion. It is assumed here that the object verification criterion was previously trained in general to recognize a body part as an object and is now only refined in such a way that, using the user movement information, details on, for example, a movement pattern of the user's body part are available, which are used for further training.

This allows the object verification criterion to be adapted to the user using the optimization step described. This means that it is always possible to reliably detect whether the object is the body part intended for the vehicle function or not, since the object verification criterion has been specifically trained to detect the body part of the user of the vehicle function. This can further improve the false trigger rate, such as the false opening rate of the trunk.

Furthermore, one embodiment provides that the user movement information is determined by applying the movement determination criterion to further ultrasound information. The further ultrasound information is recorded when the user carries out the movement sequence, in particular the movement of the body part. For this purpose, for example, the same or even the identical at least one ultrasound sensor is used, by means of which the ultrasound information is provided for operating the vehicle function. The evaluation of the further ultrasound information to determine the movement information can be carried out analogously to the procedure described above, i.e. by applying the movement determination criterion to the further ultrasound information provided. Ultimately, analogous data is provided to the object verification criterion, which describe the movement sequence, in particular at least the movement, of the at least one object that was recorded for the user's body part, so that actual information about the user movement is available. This makes the retraining described particularly useful.

Furthermore, in one embodiment, it can be provided that the additional ultrasound information is also taken into account during retraining. When retraining the object verification criterion, the determined user movement information and the additional ultrasound information on the basis of which the user movement information was determined can be provided to it. The retraining can therefore also be carried out on the basis of the raw data and the processed data. By evaluating the additional ultrasound information, for example, a shape and/or size of the user's body part can be determined and taken into account. This ensures that various possible information, which can be taken from the echo signals detected by the ultrasound sensor, for example, is taken into account during retraining.

A further embodiment provides that feature information describing at least one feature of the echo signal received by the at least one ultrasonic sensor is determined by applying a feature recognition criterion to the provided ultrasonic information. The object verification criterion is then applied to the determined feature information. It can therefore be provided that the object verification criterion is applied in addition to the provided Ultrasound information, in particular as an alternative to this, is applied to the feature information. The feature information describes already evaluated features that can be taken from the ultrasound information. The feature information is thus, for example, a pre-evaluated version of the raw data. The feature information describes, for example, an amplitude of the echo signal, in particular in the area of a maximum of the echo signal, and/or a width of the maximum. The feature recognition criterion is an algorithm and/or a rule, when applied, for example based on mathematical evaluation methods and/or machine learning methods, at least one feature can be read out and quantified from the raw data provided by the ultrasound sensor. The feature of the echo signal can alternatively be referred to as a feature of the echo signal. This can simplify the object verification criterion in such a way that it no longer has to evaluate, for example, a curve that is provided as ultrasound information, but can access the features of the ultrasound information directly because the feature information is provided to it. This can, for example, reduce the computational effort required to apply the object verification criterion.

According to an additional embodiment, it is provided that the feature information describes at least one of the following features: One of the possible features is an amplitude of a global maximum and/or at least one local maximum. The amplitude is specified, for example, in the form of a voltage that is detected by the ultrasonic sensor. The amplitude can alternatively be referred to as a maximum value. It is assumed here that the echo signal has, for example, several maxima. A main maximum and side maxima surrounding it can therefore be distinguished, for example. The main maximum can be the global or the local maximum. Furthermore or alternatively, the feature information can describe a width of the respective maximum, a slope in the area of the respective maximum and/or a distribution of the local maxima, for example around the global and/or another local maximum and/or adjacent to it. Alternatively or additionally, a shape of the respective maximum can be described as a feature. The features mentioned can, for example, at least partially be used to determine what type of object is located in front of the ultrasonic sensor and in particular is moving in front of it. In other words, a reflection strength of the object, i.e. for example the amplitude, and/or characteristics of the reflection pattern, such as the width and/or the shape of the maximum, can be provided and taken into account. This information can be taken from the ultrasound information as its features. If the feature information is not provided, but the object verification criterion is only applied to the ultrasound information and the movement information, this information is also available, but must first be evaluated, for example as part of applying the object verification criterion. Ultimately, various features of the echo signal can be evaluated, provided they can be taken from the ultrasound information provided.

In addition, an embodiment provides that the determined movement information describes at least one of the following movement characteristics: An approach of the object to the at least one ultrasonic sensor, a first waiting time after the approach of the object, a movement of the approached object, a second waiting time after the movement of the approached object and/or a removal of the object after the movement and/or after the second waiting time. A distinction can therefore be made between several phases or stages described by the movement information. These phases are included in the movement sequence. The movement of the approached object is, for example, the execution of a certain gesture and thus a certain movement pattern by the object. The movement of the approached object can, for example, be a kicking movement of the user's foot. Alternatively or additionally, the movement can, for example, be a circular movement, a semi-circular movement, an angular movement, a raising and subsequent lowering and/or a different movement pattern.

The movement feature can also or alternatively be a change in the distance of the object during the approach. It can therefore be recorded, for example, which distance the object travels during the approach to the ultrasonic sensor. This can be used, for example, to determine whether a person is approaching the ultrasonic sensor, i.e., for example, walking towards it, or not. Alternatively or additionally, the movement feature can be a duration of the respective waiting time. If, for example, before or after the movement is carried out, the object waits in front of the ultrasonic sensor without further change in distance to the ultrasonic sensor, this can indicate that gesture control is desired, for which the user first prepares and/or after which he waits until the vehicle function has been carried out. In this context, a maximum waiting time can be specified, with which the recorded duration of the respective waiting time is compared. If, for example, the maximum waiting time is not met, it can be concluded that the object is permanently in the vicinity of the ultrasonic sensor and, for example, no gesture control is taking place. Alternatively or additionally, the movement feature can describe a duration of the movement of the approached object. For example, a maximum time period can be specified that the gesture control can last. If, for example, the movement lasts longer than the specified movement duration, it can be concluded that no specific gesture was made for gesture control, but that the object is being moved in the detection range of the ultrasonic sensor for another reason. The maximum time period and/or the maximum waiting time can be stored in the object verification criterion, for example.

Alternatively or additionally, the movement feature can describe a position of the object at the beginning of the movement and/or at the end of the movement. The position of the object at the beginning of the movement can be referred to as the start position and the position at the end of the movement as the end position, for example. In particular, a change in position between the position at the beginning and the position at the end can be regarded as a movement feature, i.e. the change in position between the start position and the end position. This can be used, for example, to determine whether the object is back at its starting point of the movement after the movement, which could lead to the conclusion that a movement intended for gesture control, such as a kicking movement, has actually taken place and not the object has simply changed its position, for example by walking past the motor vehicle. For this, it is necessary, for example, that the position of the object at the beginning of the movement and the position at the end of the movement are each stored so that the described change in position can be determined.

Alternatively or additionally, the movement feature can describe a direction of movement of the approaching object, i.e., for example, whether it continues to move toward the ultrasonic sensor or not. In particular, for some of the movement features described, it is necessary to evaluate data from several ultrasonic sensors and thus, for example, to evaluate several pieces of ultrasonic information provided, such as for evaluating the direction of movement and the position of the object at the start of the movement and/or at the end of the movement. Preferably, data from several ultrasonic sensors are always compared with one another, as this makes it possible not only to determine the distance or the distance of the object from the respective ultrasonic sensor, but also to localize the object in the vicinity of the sensors. Ultimately, a detailed analysis of the movement of the object in the detection range of the at least one ultrasonic sensor can be carried out, so that the object verification criterion has detailed information about the movement of the object. This can therefore reliably determine whether the moving object can actually be the body part intended for gesture control or not.

In addition, one embodiment provides that the ultrasonic information provided is always recorded by several spatially separated ultrasonic sensors. These are preferably located in the bumper in the rear area of the motor vehicle and thus in the area around the trunk. For example, two, four, six or eight individual ultrasonic sensors can be arranged in the rear area so that the entire rear area of the motor vehicle can be monitored using the ultrasonic sensors. Analogously, ultrasonic sensors can be provided in the bumper in the front area of the motor vehicle. Alternatively or additionally, at least one ultrasonic sensor can be arranged in a respective door of the motor vehicle. This ensures that the environment of the motor vehicle relevant to the respective function can be reliably monitored.

In addition, one embodiment provides that the object is a user's foot. The method described is therefore specifically designed for, for example, a kick or other type of movement of the foot in the detection range of the ultrasonic sensors of a motor vehicle, for example for contactless opening or closing of a trunk. Alternatively or additionally, the object can be the user's leg, arm and/or hand. Depending on the desired object, a different arrangement of the ultrasonic sensors can be provided so that they are located in a typical area in which the respective body part of the user is typically moved.

In a further embodiment, it is provided that if the object is the body part intended for gesture control, a movement gesture of the object is determined. The vehicle function is operated according to the determined movement gesture. The movement gesture is determined at least by evaluating the ultrasound information and/or the movement information. Preferably, when determining the movement gesture in addition to or as an alternative to the ultrasound information, the feature information is taken into account. The movement gesture can be described, for example, by movement gesture information that is determined when evaluating the ultrasound information, the movement information and/or the feature information. The determined movement gesture can describe a specific movement, for example of the user's foot as an object, such as the kicking movement or one of the other movement patterns described above. The movement gesture is provided, for example, in the form of a determined movement trajectory of the object. It is therefore not only possible to record whether the body part intended for gesture control is being moved, but also to record and evaluate a specific movement gesture that is carried out by this body part.

In this context, it can be recognized, for example, that part of an object arranged in front of the motor vehicle, such as the user's leg or foot, is being moved relative to the rest of the body. This can be recognized, for example, by the fact that a location of a maximum in the echo signal changes over time towards shorter distances to the ultrasonic sensor, whereas remaining maxima remain at a constant distance as a stationary part of the echo signal, so that it can be concluded that only part of the object and thus the body part is being moved specifically towards the motor vehicle. Ultimately, this allows the vehicle function to be operated reliably.

The method steps are carried out in particular by means of the control device of the motor vehicle. The ultrasonic information is provided to this device and it can operate the vehicle function by controlling a trunk opening device, for example.

A distance and a spacing between the object and the at least one ultrasonic sensor are synonymous. The change in distance described above can therefore alternatively be referred to as a change in distance.

A further aspect of the invention relates to a motor vehicle. The motor vehicle is designed to carry out the method described above. The motor vehicle carries out the method. The motor vehicle is thus designed to provide the ultrasound information, to determine the movement information by applying the movement determination criterion to the ultrasound information provided, to check whether the object is the body part intended for gesture control of the vehicle function by applying the object verification criterion to the ultrasound information provided and the determined movement information and, if the object is the body part intended for gesture control, to operate the vehicle function. The motor vehicle is, for example, a passenger car, a truck, a bus and/or a motorcycle.

An advantageous embodiment of the motor vehicle according to the invention provides that the motor vehicle comprises at least one ultrasonic sensor in a rear area of the motor vehicle. The vehicle function is designed to automatically open and/or close a trunk of the motor vehicle using the ultrasound-based gesture control. In other words, the vehicle function is designed for contactless opening or closing of the trunk of the motor vehicle. The motor vehicle preferably has several ultrasonic sensors in the rear area, which are arranged there, for example, in the bumper. Preferably, at least six ultrasonic sensors are arranged spatially separated from one another.

A further aspect of the invention relates to a control device for a motor vehicle. The control device is designed to carry out the described method. The control device carries out the described method, in particular an embodiment or a combination of embodiments of the method. The control device has a processor device. The processor device can have at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (Field Programmable Gate Array) and/or at least one DSP (Digital Signal Processor). Furthermore, the processor device can have program code, which can alternatively be referred to as a computer program product. The program code can be stored in a data memory of the processor device. The motor vehicle according to the invention can have the described control device.

A further aspect of the invention relates to a computer program product. The computer program product is a computer program. The computer program product comprises instructions which, when the program is executed by a computer, such as by the control device, cause the computer to carry out the steps of the method according to the invention.

The embodiments described in connection with the method according to the invention, each individually and in combination with one another, apply accordingly, where applicable, to the motor vehicle according to the invention, the control device according to the invention and the computer program product according to the invention. The invention includes combinations of the described embodiments.

Showing:

Fig. 1 is a schematic representation of a motor vehicle with several ultrasonic sensors;

Fig. 2 shows a schematic representation of a signal flow graph of a method for operating a vehicle function of a motor vehicle by means of an ultrasound-based gesture control;

Fig. 3 shows a schematic representation of a signal flow graph of further method steps for a method according to Fig. 2;

Fig. 4 shows a schematic representation of an echo signal waveform generated by a

ultrasonic sensor is detected; and Fig. 5 is a schematic representation of a motion curve of an object in the vicinity of an ultrasonic sensor.

Fig. 1 shows a motor vehicle 1 that has several ultrasonic sensors 2. Here, six spatially separated ultrasonic sensors 2 are positioned in a rear area 3 of the motor vehicle 1. These can monitor the surroundings of the motor vehicle 1 in the rear area 3. The surroundings of the motor vehicle 1 are spatially limited by a boundary of the respective detection range of the respective ultrasonic sensor 2, which each have a range of up to 5 meters, for example.

The motor vehicle 1 has a control device 4 in which a vehicle function 5 is stored. The vehicle function 5 can be carried out by the control device 4. It is designed, for example, to control a trunk 6 of the motor vehicle

1 to open or close without contact, i.e. to open and close the trunk 6 automatically. This vehicle function 5 can be operated using an ultrasound-based gesture control. For this purpose, measurement data from the ultrasonic sensors

2 in the rear area 3 in order to recognize a gesture, based on the recognition of which, for example, the trunk 6 is automatically opened. The evaluation of the measurement data of the individual ultrasonic sensors 2 as well as further steps for providing the vehicle function 5 are carried out by means of the control device 4 of the motor vehicle 1.

In the vicinity of the motor vehicle 1 there is a user 7 who performs a gesture with a foot 8 in the detection range of the ultrasonic sensors 2 or at least some of the ultrasonic sensors 2 in the rear area 3. This gesture can be, for example, a kicking movement of the foot 8 of the user 7.

In Fig. 1, ultrasonic sensors 2 are also outlined in a front area 9 of the motor vehicle 1. Ultrasonic sensors 2 (not outlined here) can also be arranged in doors of the motor vehicle 1 in order to monitor respective side areas of the motor vehicle 1. It can thus be possible to provide further vehicle functions 5 by means of ultrasound-based gesture control, such as an automatic opening or closing of at least one door of the motor vehicle 1.

Fig. 2 shows method steps of a method for operating the vehicle function 5 of the motor vehicle 1 by means of ultrasound-based gesture control. In a method step S1, ultrasound information 10 is provided which describes at least one object in the environment of at least one of the ultrasound sensors 2. The ultrasound information 10 preferably describes the object that is located in the environment of the multiple ultrasound sensors 2 in the rear area 3 of the motor vehicle 1. The object can be, for example, the user 7, in particular the foot 8 of the user 7. The ultrasound information 10 can, before it is provided, be detected by means of at least one of the ultrasound sensors 2, preferably by means of several of the ultrasound sensors 2. It is then provided, for example, to the control device 4 of the motor vehicle 1. In a method step S2, a movement determination criterion 11 is applied to the provided ultrasound information 10 in order to determine movement information 12. The movement information 12 describes a movement sequence of the at least one object, such as the user 7 and his foot 8.

It can also be provided that in a method step S3 a feature recognition criterion 13 is applied to the provided ultrasound information 10 in order to determine feature information 14. The feature information 14 describes at least one feature of an echo signal that is received by the at least one ultrasound sensor 2. It is provided here that the at least one ultrasound sensor 2 emits an ultrasound pulse that is reflected from the object in the environment, wherein the reflected ultrasound pulse is received as an echo signal by the ultrasound sensor 2, in particular by several of the ultrasound sensors 2.

In a method step S4, it is checked whether the object is a body part intended for gesture control of the vehicle function 5. This is done by applying an object verification criterion 15 to the provided ultrasound information 10 and the determined movement information 12. It can be applied to the determined feature information 14 as an alternative to the ultrasound information 10 or in addition to it. By applying the object verification criterion 15, it can be determined whether the object is the foot 8 of the user 7, which is intended for gesture control of the vehicle function 5. The foot 8 is then the intended body part.

The object verification criterion 15 is based, for example, on machine learning methods, in particular on an artificial neural network and/or a support vector machine. The object verification criterion 15 was previously trained to at least recognize whether the object is a specific body part intended for the vehicle function 5 or not. This means that the object verification criterion 15 can, for example, recognize that the object in the detection area is the user 7 who moves his foot 8 for gesture control.

In a method step S5, it can be provided that, if the object is the body part intended for gesture control, a movement gesture 16 of the object is determined. The movement gesture 16 is determined by evaluating the ultrasound information 10, the movement information 12 and/or the feature information 14. The movement gesture 16 is then provided so that in the method step S6 the vehicle function 5 is operated according to the determined movement gesture 16. Independently of determining the movement gesture 16, it is provided in the method step S5 that, if the object is the body part intended for gesture control, the vehicle function 5 is operated.

If it is determined in method step S4 that the object is not the body part intended for gesture control of vehicle function 5, method steps S1 and, if applicable, S2 and S3 can be repeated.

Fig. 3 shows the process steps for retraining the object verification criterion

15. In a method step S7, the user 7 can select at least one Gesture control of the vehicle function 5 is used to carry out the movement 29 (see reference number 29 in Fig. 5) of the body part provided for by the gesture control of the vehicle function 5 and this is recorded by means of the at least one ultrasonic sensor 2. A movement sequence of the user 7 is preferably recorded, which includes the movement 29 but also phases without movement of the body part. In this way, further ultrasonic information 17 is recorded and provided. This describes the execution of the movement 29 of the body part by the user 7, for example the foot 8.

In a method step S8, the movement determination criterion 11 can be applied to the additional ultrasound information 17. This determines user movement information 18. The user movement information 18 describes the movement 29 performed by the user 7 of the body part intended for gesture control of the vehicle function 5.

In a method step S9, the object verification criterion 15 can then be retrained taking into account the determined user movement information 18. The additional ultrasound information 17 can also be taken into account here.

An example of an echo signal curve 19 is sketched in Fig. 4. This can be described by the ultrasound information 10. On the two sketched axes, on the one hand, an amplitude 20, for example as a voltage specification, and on the other hand a time 21, for example specified in seconds, are plotted. The feature information 14 can describe individual features of the echo signal curve 19, such as the amplitude 20 of a global maximum 22 and/or at least one local maximum 23 of the echo signal curve 19. Furthermore, the feature can describe a width 24 of a respective maximum, here sketched for the global maximum 22. Alternatively or additionally, the feature information 14 can describe a slope in the region of the respective maximum 22, 23, a distribution of the local maxima 23, in particular around the global maximum 22 or another local maximum 23, and/or a shape of the respective maximum 22, 23. Ultimately, the feature information 14 describes a concrete detail that can be taken from the echo signal curve 19.

An exemplary movement curve 25 for the object is outlined in Fig. 5. The movement curve 25 can be described by the movement information 12. A distance 26 between the object and the respective ultrasonic sensor 2 and the time 21 are plotted on the axes. It is clear that the movement curve 25 and thus the movement information 12 can describe different phases. These are an approach 27 of the object to the at least one ultrasonic sensor 2, a first waiting time 28 after the approach 27 of the object, a movement 29 of the approached object, for example a kicking movement, a second waiting time 30 after the movement 29 of the approached object and/or a distance 31 of the object away from the ultrasonic sensor 2 after the movement 29 and/or the second waiting time 30. The typical distance changes in these individual phases are outlined. The determined movement information 12 can describe one of these phases as a respective movement feature, that is to say the course of the approach 27, the first waiting time 28, the movement 29, the second waiting time 30 and/or the distance 31. Alternatively or additionally, the movement information 12 can describe a change in distance and thus the change in distance of the object during the approach 27, a duration of the respective waiting time 28, 30, a duration of the movement 29 of the approached object, a position and/or a direction of the movement 29 of the approached object. Alternatively or additionally, a position 32 of the object at the start of the movement 29 and/or a position 33 at the end of the movement can be taken into account as a movement feature. In particular, a change in position between the position 32 at the start of the movement 29 and the position 33 at the end of the movement 29 is considered here.

Overall, the examples show feature-based foot recognition for ultrasonic sensor systems. It is important not only to recognize the foot 8 and determine a movement pattern purely based on the foot distance, but also to specifically extract various features from the ultrasound data, i.e. the ultrasound information 10. These features can be those from the movement pattern, for example, such as changes in distance during the foot movement or a movement time (duration of the movement 29). These movement features can be included in the movement information 12. In addition, features from the ultrasound signal itself can be taken into account. Typical features are then a reflection value of the foot (for example amplitude 20 of a main maximum) or a characteristic value of the reflection pattern (for example the width 24 of the echo and thus of the respective maximum 22, 23). This means that the ultrasound information 10 itself can be evaluated and taken into account. Furthermore, features can be calculated from several ultrasonic sensors 2, such as the direction of the foot movement or the position 32, 33 at the start and end time, for example as a projection in two dimensions. This information has previously been referred to as possible movement features.

The features mentioned can be evaluated using common machine learning methods, i.e. by applying the object verification criterion 15. Possible ways of doing this are flat neural networks or conventional approaches such as the support vector machine. Ultimately, a binary approach that determines whether the foot 8, i.e. the specified body part, is present or not, is sufficient. By evaluating features at different data levels and classifying the data using machine learning, the performance of foot detection can be significantly improved. The false positive rate, i.e. accidental opening of the trunk, is particularly important here. It is precisely the features at the raw data level, i.e. by taking into account the ultrasound information 10, that can significantly improve detection if the appropriate hardware is available and, ideally, even classify the object, i.e. object verification in process step S4 becomes possible.

Another advantage is that the system can also be adapted to the user 7 himself using the machine learning approach described. With a pre-trained network, the detection thresholds can be fine-tuned again after a short training process. To do this, the end customer, i.e. user 7, only has to make a few foot movements. These can then recorded and retrained in an optimization step as described in process steps S7 to S9.

An example of a typical process when opening a trunk with the foot 8 can be seen in Fig. 5. This process shows at least four steps that can be referred to as approaching, waiting, kicking and waiting (see reference numbers 27 to 30). Features from this sequence are, for example, the duration of the kick, i.e. the movement 29, but also the amplitude 20 and thus the change in distance of the kick, such as the change in position described. In addition, there can be features such as waiting times, i.e. the duration of the respective waiting time 28, 30 or the maximum change in distance in a waiting process. In addition, the features can be extracted from the ultrasound signal itself, i.e. the ultrasound information 10 can be taken into account. For this purpose, a maximum 22, 23 can be clearly seen in Fig. 4, which could come from a foot reflection. Various smaller echoes can be seen before and after the actual reflection. The distribution of the echoes, but also the shape of the main reflection can be evaluated and taken into account as features.

The features from this data can be used to classify the foot movement using the support vector machine or a simple feedforward neural network with a few layers. For example, a hidden layer with several hidden neurons can be provided. Typically, numerous classifications are used, which means it makes sense to use several hidden neurons and, if necessary, to insert several hidden layers with complete connections. With the approach described, it is possible to significantly reduce the false positive rate. Using the method, it is therefore possible to use ultrasonic sensors 2 already included in the motor vehicle 1 for the contactless opening of the trunk 6.

Claims

Claims Method for operating a vehicle function (5) of a motor vehicle (1) by means of an ultrasound-based gesture control, comprising:

- providing (S1) ultrasonic information (10) which describes at least one object in an environment of at least one ultrasonic sensor (2);

- determining (S2) movement information (12) which describes a movement sequence of the at least one object by applying a movement determination criterion (11) to the provided ultrasound information (10);

- checking (S4) whether the object is a body part intended for gesture control of the vehicle function (5) by applying an object verification criterion (15) to the provided ultrasound information (10) and the determined movement information (12); and

- if the object is the body part intended for gesture control, operating (S6) the vehicle function (5). Method according to claim 1, wherein the object verification criterion (15) is based on machine learning methods, in particular on an artificial neural network and/or a support vector machine. Method according to claim 2, wherein user movement information (18) is determined which describes a movement sequence of the intended body part carried out by the user (7), and the object verification criterion (15) is retrained taking into account the determined user movement information (18) (S9). Method according to claim 3, wherein the user movement information (18) is determined by applying the movement determination criterion (11) to further ultrasound information (17) which is recorded when the user (7) carries out the movement sequence (S7). Method according to claim 4, wherein the further ultrasound information (17) is also taken into account during retraining. Method according to one of the preceding claims, wherein feature information (14) describing at least one feature of an echo signal received by the at least one ultrasonic sensor (2) is determined (S3) by applying a feature recognition criterion (13) to the provided ultrasonic information (10) and the object verification criterion (15) is applied to the determined feature information (14). Method according to claim 6, wherein the feature information (14) describes at least one of the following features:

- an amplitude (20) of a global maximum (22) and/or at least one local maximum (22);

- a width (24) of the respective maximum (22, 23);

-a slope in the region of the respective maximum (22, 23);

- a distribution of the local maxima (23), in particular around the global maximum (22) and/or another local maximum (23); and/or

- a shape of the respective maximum (22, 23). Method according to one of the preceding claims, wherein the determined movement information (12) describes at least one of the following movement characteristics:

- an approach (27) of the object to the at least one ultrasonic sensor (2);

- a first waiting time (28) after the approach of the object;

- a movement (29) of the approached object;

- a second waiting time (30) after the movement (29) of the approached object;

- a distance (31) of the object after the movement (29) and/or the second waiting time (30);

- a change in distance of the object during the approach (27);

- a duration of the respective waiting period (28, 30);

- a duration of movement (29) of the approached object;

- a position (32, 33) of the object at the beginning of the movement (29) and/or at the end of the movement (29), in particular a change in position between the position (32) at the beginning and the position (33) at the end; and/or

- a direction of movement (29) of the approaching object. Method according to one of the preceding claims, wherein the provided ultrasound information (10) is detected by several spatially separated ultrasound sensors (2). Method according to one of the preceding claims, wherein the object is a foot (8) of a user (7). Method according to one of the preceding claims, wherein if the object is the body part provided for gesture control, a movement gesture (16) of the object is determined (S5) and the vehicle function (5) is operated according to the determined movement gesture (16), wherein the movement gesture (16) is determined at least by evaluating the ultrasound information (10) and/or the movement information (12). Motor vehicle (1) which is designed to carry out a method according to one of the preceding claims. Motor vehicle (1) according to claim 12, wherein the at least one ultrasound sensor (2) is arranged in a rear area (3) of the motor vehicle (1) and the vehicle function (5) is designed to automatically open and/or close a trunk (6) of the motor vehicle (1) by means of the ultrasound-based gesture control. Control device (4) for a motor vehicle (1), wherein the control device (4) is designed to carry out a method according to one of claims 1 to 11. Computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to one of claims 1 to 11.