DE102019210144A1 - Combination of sensor systems in the vehicle to improve the recognition of user commands - Google Patents

Abstract

A method for controlling a vehicle system (12) comprises: receiving audio data (30) in an evaluation unit (18) of the vehicle (10) which were recorded with an audio sensor (14a, 14b) of the vehicle (10); Receipt of optical and / or spatial data (36, 38) in the evaluation unit (18), which were recorded with an optical and / or spatial sensor (16a, 16b, 16c), the optical and / or spatial data (36, 38) encode an image and / or a spatial position of at least one user (26a, 26b) in the vehicle (10); Evaluation of the audio data (30) together with the optical and / or spatial data (36, 38) with the evaluation unit (18) and recognition of a user command (44); and controlling the vehicle system (12) based on the user command (44).

Description

The invention relates to a method, a computer program and a computer-readable medium for controlling a vehicle system. The invention also relates to a vehicle system.

Speech recognition, lip reading by camera system and gesture control or recognition of body language are known technologies. Gesture control systems and voice control systems have already been implemented in vehicles in order to control the vehicle or one of its subsystems. A disadvantage when evaluating audio or video data in the vehicle can often be that the systems cannot clearly interpret or misunderstand the user commands. This can be due, among other things, to disturbance variables such as noise, darkness / ambient light, etc., which can falsify and / or cover up the user commands.

The object of the invention is to improve the input of user commands in a vehicle and their evaluation.

This object is achieved by the subject matter of the independent claims. Further embodiments of the invention emerge from the dependent claims and from the following description.

One aspect of the invention relates to a method for controlling a vehicle system. The method can be carried out automatically by a component of the vehicle, such as an on-board computer, an evaluation unit and / or the vehicle system itself. The vehicle system can be, for example, an on-board computer, navigation system, an infotainment system of the vehicle. It is possible for the vehicle system to be designed to control the vehicle, such as, for example, to steer, accelerate, brake, navigate. The vehicle system may include an actuator for controlling the vehicle.

Examples of possible vehicle types in which the system can be implemented are cars, trucks, buses, motorcycles, etc.

According to one embodiment of the invention, the method comprises: receiving in an evaluation unit of the vehicle audio data that were recorded with an audio sensor of the vehicle; Receiving optical and / or spatial data in the evaluation unit, which were recorded with an optical and / or spatial sensor, the optical and / or spatial data encoding an image and / or a spatial position of at least one user in the vehicle; Evaluation of the audio data together with the optical and / or spatial data with the evaluation unit and recognition of a user command; and controlling the vehicle system based on the user command.

In summary, audio data, for example from an indoor microphone, are evaluated in combination with, for example, video data in order to determine a possible user command from the combination of the data. By combining the data, the probability of recognition of the user command can be increased. The combination of different sensor systems can serve to interpret the user input more clearly or to select the most likely interpretation. The influences of disturbance variables can be minimized by the fact that different sensor technologies and possibly algorithms that use different input variables that provide clues for the meaning of the inputs complement each other.

Audio data can be digital data picked up by a microphone such as an indoor microphone. The audio data can encode speech and / or sounds.

Optical and / or spatial data can be data recorded by a camera, radar, lidar, etc. The optical and / or spatial data can encode an image or video of a user, such as the driver, in the vehicle. The optical and / or spatial data can also encode a location and / or positions of the user in space.

The evaluation unit can be part of the vehicle system or can be designed to control the vehicle system. The evaluation unit can be part of the vehicle's on-board computer.

The (desired) user command can be distinguished from a user input. The user input can be the direct user action, such as the utterance of a word command, a gesture, etc. The user command can be a number and / or a probability for a specific desired action of the vehicle system, i.e. the action that the vehicle system is to perform. An example of this can be that the radio should be quieter or that the vehicle should brake.

It may be that the optical and / or spatial data and the audio data were recorded in the same time period. Simultaneously recorded time segments of the two types of data can be evaluated by the evaluation unit in order to determine from a combined user input and / or that of different sensors (ie the audio sensor and the optical and / or spatial sensor) recorded user input to determine a user command. It is possible that both data are first converted separately from one another into possible user commands, which are then checked for plausibility with one another. It is also possible that both data are combined and converted into a user command, for example as inputs into an artificial neural network.

According to one embodiment of the invention, the method further comprises: evaluating the audio data and recognizing at least one possible voice command. A speech recognition algorithm can be used for this purpose. A voice command or the probability for several voice commands can be determined from the audio data.

According to one embodiment of the invention, the method further comprises: evaluating the optical and / or spatial data and recognizing at least one possible movement command. A movement command can for example be based on a gesture and / or a movement of the user. A movement command can also be based on a static position and / or a static posture of the user (for example a pose). A movement command or the probability for several movement commands can be determined from the optical and / or spatial data.

According to one embodiment of the invention, the method further comprises: Recognizing the user command by checking the plausibility of the at least one possible voice command and the at least one possible movement command. After evaluating the two types of data, a most likely user command can be determined based on the possible voice commands and the possible movement commands.

A table of plausible combinations of voice commands and movement commands can be stored for this purpose. The table can contain pairs of plausible voice commands and movement commands (such as “radio down” and “pointing to radio”).

It is also possible for the voice commands and the movement commands to encode the same commands that were determined with the different data types. For example, the movement commands can include commands determined by lip reading. In this case, the plausibility check can take place in that a user command is plausible that is present in both the possible voice commands and the possible movement commands.

According to one embodiment of the invention, probabilities are determined for the at least one possible voice command and the at least one possible movement command. A most likely user command can be determined from the probabilities. For example, the possible voice commands and the possible movement commands can be a selection of possible user commands that have been weighted with probabilities. In this case, the plausibility check can take place in that the user command which, according to the combined probabilities of the possible voice commands and the possible movement commands, has the highest probability is selected as the user command.

According to one embodiment of the invention, the optical and / or spatial data are evaluated in relation to a lip movement of the user. In this case, the possible movement commands can also be voice commands. An algorithm for lip reading speech can be used to evaluate image data, for example. The lip-read words can, for example, be compared and / or checked for plausibility with words originating from speech recognition.

According to one embodiment of the invention, the optical and / or spatial data are evaluated in relation to a gesture by the user. A gesture command can be determined from the optical and / or spatial data. An algorithm for recognizing body language such as facial expressions, gestures and posture can, for example, use image data to determine which vehicle system the user wants to operate with. This can also be used to check whether the voice command matches the gesture command.

According to one embodiment of the invention, a voice command is determined from the audio data, with the optical and / or spatial data being used to determine which user in the vehicle generated the voice command. For example, when using several microphones, additional sensors such as camera systems can also be used to weight or select the voice commands assigned to the microphones and / or to assign voice commands to individual users based on the direction of gaze and language.

According to one embodiment of the invention, the optical and / or spatial sensor comprises a video sensor. The optical and / or spatial data can include image data. For example, with a system based on artificial intelligence, lip movements and / or gestures of the user can be derived from image data.

According to one embodiment of the invention, the video sensor is designed to detect light in a non-visual wavelength range, the video sensor comprising an active light source for generating illumination in the non-visual wavelength range. When using cameras with active lighting in a non-visual wavelength range, such as infrared, the lighting can remain invisible to the user. The user is not disturbed, but the system can also work in the dark. Furthermore, the sources of interference can be smaller in the case of light in a non-visual wavelength range. Devices for suppressing extraneous light, such as optical filters, can also be used. Ambient light as a disturbance variable can also be reduced with a TOF (time of flight) camera with a photonic mixer.

It is also possible for the audio data and the optical and / or spatial data to be evaluated together in order to generate a user command directly. This can be seen as a sensor fusion approach. The audio data and the optical and / or spatial data can, for example, be processed jointly by a machine learning algorithm which directly outputs a user command or a probability for a user command. It is not necessary to first evaluate the data from different sensors separately from one another in order to determine a voice command and a movement command.

According to one embodiment of the invention, the optical and / or spatial sensor is designed to generate position and / or location data. Such position and / or location data can contain three-dimensional points. A position and / or position of the user, such as his arm, his lips, etc., can be coded with the three-dimensional points. The optical and / or spatial data can contain position and / or location data such as three-dimensional points. For example, the optical and / or spatial sensor can be or comprise a 3D camera, such as a TOF (time of flight) camera. 3D cameras can be suitable for recognizing body language such as gestures and / or postures.

According to one embodiment of the invention, the optical and / or spatial sensor comprises a sensor that scans (with beams). The sensor can comprise an ultrasonic sensor, a lidar sensor and / or a radar sensor. With ultrasound, laser beams and / or radar beams, the user can be scanned without interference from other light sources and / or spatial data can be generated that can make possible voice commands plausible.

Further aspects of the invention relate to a computer program which, when it is executed on a processor, carries out the method as described herein, as well as a computer-readable medium on which such a computer program is stored. A computer-readable medium can be a hard disk, a USB storage device, a RAM, a ROM, an EPROM or a FLASH memory. A computer-readable medium can also be a data communication network, such as the Internet, for example, which enables program code to be downloaded. The computer readable medium can be transitory or non-transitory. It is to be understood that features of the method, as described above and below, can also be features of the computer program and / or the computer-readable medium.

Another aspect of the invention relates to a vehicle system. The vehicle system comprises an audio sensor, an optical and / or spatial sensor and an evaluation unit which is designed to carry out the method as described above and below. The spatial sensor can also include an audio sensor. The vehicle system can furthermore comprise an actuator for controlling the vehicle. The vehicle system can be particularly suitable for HMI interaction in vehicles, since there are often corresponding disturbance variables (such as driving noises, conversations, changing lighting conditions, etc.) and at the same time uncomplicated and precise communication with the vehicle is desirable in order not to distract the driver, for example . It is to be understood that features of the method, as described above and below, can also be features of the vehicle system and vice versa.

• 1 zeigt schematisch ein Fahrzeug mit einem Fahrzeugsystem gemäß einer Ausführungsform der Erfindung.
• 2 zeigt schematisch ein Blockdiagramm für ein Fahrzeugsystem gemäß einer Ausführungsform der Erfindung.
• 3 zeigt ein Flussdiagramm für ein Verfahren zum Steuern eines Fahrzeugsystems gemäß einer Ausführungsform der Erfindung.

In the following, exemplary embodiments of the invention are described in detail with reference to the accompanying figures.

• 1 shows schematically a vehicle with a vehicle system according to an embodiment of the invention.
• 2 schematically shows a block diagram for a vehicle system according to an embodiment of the invention.
• 3 FIG. 10 shows a flow diagram for a method for controlling a vehicle system according to an embodiment of the invention.

The reference symbols used in the figures and their meaning are given in summarized form in the list of reference symbols. In principle, identical or similar parts are provided with the same reference symbols.

1 shows a vehicle 10 with a vehicle system 12 who have favourited audio sensors 14a , 14b (such as microphones), video sensors 16a , 16b (such as cameras) and an ultrasonic sensor 16c includes. There is also an evaluation unit 18th , for example the vehicle's on-board computer 10 , the drive system 20th and an infotainment system 22nd shown. Both the drive system 20th as well as the infotainment system 22nd can use an actuator 24a , 24b , such as a brake, a motor, a volume control, etc., have, which by the evaluation unit 18th can be controlled.

Next shows 1 two users 26a , 26b in the vehicle 10 such as a driver 26a and a person 26b in the back seat. From the users 26a , 26b caused user inputs, such as pronounced words, gestures, movements, etc., can from the sensors 14a , 14b , 16a , 16b , 16c recorded and converted into corresponding data. The evaluation unit 18th can from the sensors 14a , 14b , 16a , 16b , 16c Receive and evaluate generated sensor data and determine user commands from it. Based on these user commands, the evaluation unit 18th Control commands for the actuators 24a , 24b generate and thus control them.

One or both of the video sensors 16a , 16b can detect light in a non-visual wavelength range. One or both of the video sensors can do this 16a , 16b an active light source 17th for generating lighting in the non-visual wavelength range.

One or both of the video sensors 16a , 16b can also be TOF cameras, which can generate depth information in addition to image data. Also the ultrasonic sensor 16c , which can be a scanning sensor, can generate spatial data.

2 Figure 3 shows a block diagram of the vehicle system 12 , in which the evaluation unit 18th in several modules 28 , 34 , 42 , 46 is divided. The functionality of the individual blocks or modules is described below with reference to the 3 described in more detail.

An audio data evaluation module 28 receives audio data 30th from one or more audio sensors 14a , 14b and uses this to determine possible voice commands 32 . The audio data 30th can digitize data from one or more microphones 14a , 14b be. Any voice input from the user 26a , 26b are analyzed and translated into voice commands 32 converted. A voice command 32 can be a number or a character string (for example corresponding to the spoken user input). Any of the possible voice commands 32 may have a probability of how likely the voice command matches the user input.

A movement data evaluation module also receives 34 optical data 36 and / or spatial data 38 from one or more sensors 14a , 14b , 14c, such as one or more cameras, ultrasonic sensors, lidar sensors, etc. The optical data 36 can include image data. The spatial data can contain position and / or location data such as three-dimensional points. It is also possible, for example in the case of a TOF camera as a sensor, that the optical data 36 and / or spatial data 38 are mixed and, for example, pixels are provided with depth information.

The movement data evaluation module 34 analyzes any movement input from the user 26a , 26b and converts these into movement commands 40 around. A movement input can be a gesture, a facial expression, a posture, or a head position of one of the users 26a , 26b be. It should be understood that a movement can also be derived from a single image. A movement command 40 can be a number that codes a corresponding movement. Any of the possible movement commands 40 may have a probability of how likely the movement command will match the user input.

The plausibility check module 42 receives the possible voice commands 32 and the possible movement commands 40 , checks these with one another, for example using their probabilities, and uses them to generate a user command 44 . A user command can be a number that contains the user input of the user 26a , 26b coded.

The user command 44 is controlled by an optionally available control module 46 in control commands 48 converted and attached to the corresponding actuator 24a , 24b Posted.

The 3 Figure 11 shows a method for controlling the vehicle system 12 that with the one in the 2 evaluation module shown 18th can be performed automatically. The modules 28 , 34 , 42 , 46 from the 2 For example, software components can be installed in an on-board computer of the vehicle 10 are executed.

In step S10, the audio data 30th as well as the optical and / or spatial data 36 , 38 in the evaluation unit 18th receive. These dates 30th , 36 , 38 are then evaluated in steps S12, S14, S16 and a user command therefrom 44 recognized.

In step S12, the audio data evaluation module evaluates 28 the audio data 30th and recognizes possible voice commands 32 . For example, the audio data evaluation module 28 execute a speech recognition algorithm that uses the audio data 30th into written words, ie symbol strings made up of characters. Word commands or voice commands can be recognized in the words. Furthermore, based on the speech recognition algorithm, probabilities can be determined as to how likely a speech command is from the user 26a , 26b was pronounced.

In step S14, the movement data evaluation module evaluates 34 the optical data 36 and / or the spatial data and recognizes at least one possible movement command 40 . For example, the optical data 36 and / or the spatial data 38 with a lip reading algorithm in relation to a lip movement of the user 26a , 26b be evaluated. The audio data 30th in written words, ie symbol strings made up of characters, ie the possible movement commands 40 can be voice commands.

But it is also possible that the optical data 36 and / or the spatial data 38 in relation to a gesture by the user 26a , 26b be evaluated. For example, a spatial position of an arm of the user or users 26a , 26b in space in relation to the vehicle 10 or components thereof are determined. A movement command can then be issued from this position (approximately a vector) 42 such as “the driver has a 50% probability of pointing to the radio”.

The plausibility check module detects in step S16 42 then a user command 44 by giving the at least one possible voice command 32 and the at least one possible movement command 40 plausibility checked. As already described above, "plausibility check" can mean that voice commands 32 and movement commands 40 that have a high correlation to a user command 44 have to be selected. This correlation can be found in the plausibility check module 42 stored table. It is also possible that the correlation is determined with a machine learning algorithm.

The table can be two-dimensional or multi-dimensional and store possible voice commands, such as "radio louder", "radio quieter", "light on", "light off", etc., in one dimension. In the other, second dimension, possible movement commands, such as “pointing to radio”, “pointing to lighting”, etc., can be stored. Correlations between entries from both dimensions can then be stored in the table, such as 1 for "radio quieter" and "indicate radio".

Another possibility is that for at least one possible voice command 32 and the at least one possible movement command 40 Probabilities are determined and a most likely user command from the probabilities 44 is determined. For example, the movement commands 40 Be voice commands that have been determined by lip reading. Then the probabilities for the voice commands 32 and the movement commands 40 are compared or correlated and a probability for a possible user command can then be determined therefrom. The most likely user command and / or a user command with a probability greater than a threshold value can then be used as a user command 44 to get voted.

It is to be understood that probabilities related to voice commands 32 and the movement commands 40 can be combined with a correlation table.

It is also possible that from the optical data 36 and / or the spatial data 38 it is determined which user 26a , 26b in the vehicle 10 the voice command 32 generated. For example, voice commands 32 of passengers 26b who are controlling the vehicle 10 concern, be sorted out. Voice commands can also be used 32 of a user 26b on the back seat can be interpreted in such a way that vehicle components located at the rear, such as lighting and rear speakers, are to be controlled.

In step S18, the user command 44 from the control module 46 in control commands 48 converted and attached to the corresponding actuator 24a , 24b Posted.

In addition, it should be noted that “comprehensive” does not exclude any other elements or steps and “one” or “one” does not exclude a large number. It should also be pointed out that features or steps that have been described with reference to one of the above exemplary embodiments can also be used in combination with other features or steps of other exemplary embodiments described above. Reference signs in the claims are not to be regarded as a restriction.

List of reference symbols

10

vehicle

12

Vehicle system

14a

Audio sensor / microphone

14b

Audio sensor / microphone

16a

Video sensor / camera

16b

Video sensor / camera

16c

Ultrasonic sensor

17th

active light source

18th

Evaluation unit

20th

Drive system

22nd

Infotainment system

24a

Actuator

24b

Actuator

26a

User / driver

26b

User / co-driver

28

Audio data evaluation module

30th

Audio data

32

possible voice commands

34

Movement data evaluation module

36

optical data

38

spatial data

40

possible movement commands

42

Plausibility module

44

User command

46

Control module

48

Control command

Claims (13)

A method of controlling a vehicle system (12), the method comprising: Receiving audio data (30) in an evaluation unit (18) of the vehicle (10), which were recorded with an audio sensor (14a, 14b) of the vehicle (10); Receipt of optical and / or spatial data (36, 38) in the evaluation unit (18), which were recorded with an optical and / or spatial sensor (16a, 16b, 16c), the optical and / or spatial data (36, 38) encode an image and / or a spatial position of at least one user (26a, 26b) in the vehicle (10); Evaluation of the audio data (30) together with the optical and / or spatial data (36, 38) with the evaluation unit (18) and recognition of a user command (44); Controlling the vehicle system (12) based on the user command (44). Procedure according to Claim 1 , further comprising: evaluating the audio data (30) and recognizing at least one possible voice command (32); Evaluating the optical and / or spatial data (36, 38) and recognizing at least one possible movement command (40); Recognition of the user command (44) by checking the plausibility of the at least one possible voice command (32) and the at least one possible movement command (40). Procedure according to Claim 2 , with probabilities being determined for the at least one possible voice command (32) and the at least one possible movement command (40) and a most likely user command (44) being determined from the probabilities. Procedure according to Claim 2 or 3 , the optical and / or spatial data (36, 38) being evaluated in relation to a lip movement of the user (26a, 26b) and the possible movement commands (40) being voice commands. Method according to one of the preceding claims, wherein the optical and / or spatial data (36, 38) are evaluated in relation to a gesture by the user (26a, 26b); wherein a gesture command is determined from the optical and / or spatial data (36, 38). Method according to one of the preceding claims, wherein a voice command (32) is determined from the audio data (30); whereby from the optical and / or spatial data (36, 38) it is determined which user (26a, 26b) in the vehicle (10) generated the voice command (32). Method according to one of the preceding claims, wherein the optical and / or spatial sensor comprises a video sensor (16a, 16b) and wherein the optical and / or spatial data comprise image data (36, 38). Procedure according to Claim 7 wherein the video sensor (16a, 16b) is designed to detect light in a non-visual wavelength range; wherein the video sensor (16a, 16b) comprises an active light source (17) for generating an illumination in the non-visual wavelength range. Method according to one of the preceding claims, wherein the optical and / or spatial sensor (16a, 16b, 16c) is designed to generate position and / or location data; and / or wherein the optical and / or spatial data (36, 38) contain position and / or location data. Method according to one of the preceding claims, wherein the optical and / or spatial sensor comprises a scanning sensor (16c). Computer program which, when executed on a processor, carries out the method according to one of the preceding claims. Computer-readable medium on which a computer program is based Claim 11 is stored. A vehicle system (12) comprising: an audio sensor (14a, 14b); an optical and / or spatial sensor (16a, 16b, 16c); an evaluation unit (18) which is designed to perform the method according to one of the Claims 1 to 10 perform.

Images