DE102013016196B4 - Motor vehicle operation using combined input modalities - Google Patents

Abstract

Motor vehicle (10) with a first operating interface (38), through which voice control of at least one device (28) of the motor vehicle (10) is provided, and with a further operating interface (44), through which one on at least one other than voice Input modality-based operation of the at least one device (28) is provided, wherein the first operating interface (38) is designed to recognize at least one reference term (52) in a voice input (26) from a user (20), which refers to at least one object ( 18) indicates that the user (20) describes using the at least one other input modality, a control device (54) being designed to, depending on the at least one recognized reference term (52) and depending on one via the further operating interface (38 ) received user input (56) to determine the at least one object (18) and to generate at least one control command depending on the voice input (26) and the at least one determined object (18) and to send the at least one control command to the at least one device (28 ), characterized in that at least one environment sensor is provided, via which the control device (54) is designed to determine at least one actual state of the at least one object (18), wherein the control device (54) is additionally designed to Recognizing a relative information in the voice input (26) to generate the control command depending on the actual state and the relative information.

Description

The invention relates to a motor vehicle with two operating interfaces for entering operating inputs. The first operating interface provides voice control for at least one device of the motor vehicle. The second operating interface provides an operation for the at least one device that is based on an input modality other than speech, for example gestures or keyboard input.

From the DE 10 2006 055 252 A1 A telecommunications application is known for this purpose that enables a multimodal selection of objects. It is intended to control navigation through menus by moving a device in space and selecting a menu item using at least one voice command. According to a selection click or selection key press, at least one linguistic utterance can be used in combination with at least one short movement. For example, the commands “Stop”, “Stop” and “Activate” are provided as control instructions that can be specified via voice. Scrolling through menu options can be stopped and functionality associated with the option can be activated by a spoken execution command.

From the DE 10 2006 009 291 A1 Gesture recognition is known as an operating interface for a motor vehicle. A functional component to be operated can first be specified using a specific spoken word. For example, the driver can say the word “audio” or “climate”. Gestures executed by the driver are then interpreted as control instructions for the selected functional component.

The DE 10 2010 017 931 A1 describes an information system for providing information to a user in a vehicle. The information system includes gesture detection, wherein respective information is retrieved based on the direction of a user gesture. The information system can be operated using a voice command.

The US 2002/0135618 A1 describes how multi-modal input data can be used in programming environments.

The US 2011/0115702 A1 describes how language and gesturing can be used for computer programming.

In operating interfaces from the prior art, a sharp separation is always provided in that a first part of an operating input is specified with a first input modality, for example gestures, and a supplementary command, such as "Activate", is given with a second input modality, for example speech becomes.

In contrast, it is not known in the prior art that the user can freely choose which input modality he uses to enter individual elements of a control command.

The invention is based on the object of giving the user in a motor vehicle the opportunity to choose which input modality he would like to use to specify individual elements of a control command.

The task is solved by the subject matter of the independent patent claims. Advantageous developments of the invention result from the dependent patent claims.

The invention is based on the motor vehicle described above with the first operating interface for voice control and the further operating interface for operation by at least one other input modality, for example gesture recognition or keyboard input. According to the invention, the first operating interface (voice control) in the motor vehicle is designed to recognize at least one reference term in a user's voice input, which refers to at least one object that the user describes using the at least one other input modality. For example, the user can voice the following: “Show the time there!” to determine which screen a digital clock is displayed on. In particular, the reference term represents only a part of the voice input, i.e. in addition to the reference term, the voice utterance or voice input also includes other terms which, in particular, belong to the same operating error as the reference term.

The first operating interface recognizes the operating command that the time should be displayed. However, the voice input contains the term “there”, which refers to a place that the user did not pronounce. The term “there” forms a reference term that indicates that the user is apparently describing the location using a different input modality.

Accordingly, in the invention, a control device is designed to determine the at least one object depending on the at least one recognized reference term and depending on a user input received via a further operating interface. By For example, a gesture recognition system can determine where the user is currently pointing, namely, for example, to a specific screen. Depending on the voice input and the at least one determined location, the control device then generates at least one control command and sends the at least one control command to the at least one device, which is operated via the two operating interfaces. In the example, an infotainment system can be caused by the at least one control command to display the current time on the screen determined by pointing.

The invention further provides that in the motor vehicle according to the invention at least one environment sensor is provided, via which the control device is designed to determine at least one actual state of the at least one object. Such an environmental sensor can be, for example, a temperature sensor, a camera, an ultrasonic sensor, a radar sensor. The user can then say, for example, the following voice input: “Make it a little warmer in here!”. This language information contains two reference terms, namely “here” and “warmer.” The first reference term can be interpreted by the control device as the location of the user, which can be, for example, a specific seat in the motor vehicle when it comes to zone-specific air conditioning. The second reference term “warmer” indicates that the current temperature should be increased.

This makes it possible for the control device to detect the current temperature and make a relative change, for example increasing the temperature by 10%. For this purpose, the control device is designed to generate the control command when a relative information is detected in the voice input depending on the actual state and the relative information. In the example, for an actual state of "20°C room temperature", a control command can be generated that increases the temperature in the area of the user's seat to, for example, 22°C. Another example is the seat control “Adjust the backrest flatter!”

The invention has the advantage that different input modalities are not processed sequentially one after the other, i.e. a category does not have to be preselected using language in order to then select a menu entry using a pointer gesture, for example. Instead, the user can specifically supplement the voice input using the other input modality by using reference terms.

Operating errors are avoided if typical reference terms that are also used in natural language are used as reference terms. For this purpose, a further development of the invention provides that the first operating interface is designed to use at least one adverb of the location (for example “here”, “there”) and/or a demonstrative pronoun (for example “this”) as the at least one reference term. , “those”) and/or a comparative (e.g. “the larger”, “the smaller”) and/or a superlative (e.g. “the largest”, “the smallest”). An example speech input with a superlative can be, for example: “Take the top three songs from this list,” in which the two reference terms “this” and “the top three” can be recognized.

The invention makes it possible to easily combine speech with a variety of different input modalities. Accordingly, in the motor vehicle according to the invention, the further operating interface has at least one of the following devices: a gesture recognition, a handwriting recognition, a touchpad, a touchscreen, a rotary-push control, a keyboard, a viewing direction recognition.

A further embodiment provides that the control device controls route planning in a navigation device using the at least one control command. The user can, for example, voice input to a digital navigation map that is displayed on a screen: “I want to go there!” With the reference term “there”, the user can again, for example, refer to the current position of a mouse cursor, which then the navigation destination can be set.

Another further development envisages controlling a database query to determine location descriptions. The user can then, for example, point from a vehicle window to a building in the vehicle's surroundings and voice the input: "What do you see here?" The reference term "here" then refers again to the user's pointing gesture. In this way, the location for which a location description should be found in the database can then be specified for the database query.

Another development provides that a description of a vehicle function is determined in a digital operating manual of the motor vehicle. The user can therefore use the voice input: “What does that mean?” to determine that a warning symbol that is currently displayed and that the user points to is determined by the corresponding descriptive texts from the digital operating instructions.

A further advantage arises if the motor vehicle is designed to display on a display device an output of the at least one device triggered by the at least one control command at a location, so that from the user's perspective, the at least one object that he or she can use by means of the has referenced at least one reference term, and this output is together in the user's field of vision. Preferably, the object and the output lie within a solid angle of 45°, in particular 30°, measured from one eye of the user. For example, if the user points to a building in front of the vehicle and asks: “What do I see here?”, he can, for example, see a window root display that extends across the vehicle on a lower edge of the windshield at least over half the width of the vehicle The answer can be displayed directly below the building in the user's field of vision. For example, a head-up display can also be used to display information about the building directly next to the building in the user's field of vision.

The invention also provides a method which is characterized by the steps that have already been described in connection with the motor vehicle according to the invention. According to the method, at least one reference term is recognized by the first operating interface in a user's voice input, which refers to at least one object that the user describes using the at least one other input modality. A control device determines the at least one object depending on the at least one recognized reference term and depending on a user input received via the at least one further operating interface. Depending on the voice input and the at least one determined object, at least one control command is generated by the control input and sent to the at least one device. The invention also includes further developments of the method according to the invention, which have features that have already been explained in connection with the further developments of the motor vehicle according to the invention. For this reason, the corresponding developments of the method according to the invention will not be explained again here.

• 1 eine schematische Darstellung einer Ausführungsform des erfindungsgemäßen Kraftfahrzeugs in einer Umgebung;
• 2 eine schematische Darstellung von zwei Kamerabildern, die von dem Kraftfahrzeug von 1 erzeugt worden sind; und
• 3 eine schematische Darstellung eines Fahrzeuginnenraums des Kraftfahrzeugs von 1.

An exemplary embodiment of the invention is explained below with reference to the figures. It shows:

• 1 a schematic representation of an embodiment of the motor vehicle according to the invention in an environment;
• 2 a schematic representation of two camera images taken from the motor vehicle 1 have been generated; and
• 3 a schematic representation of a vehicle interior of the motor vehicle 1 .

The exemplary embodiment explained below is a preferred embodiment of the invention. In the exemplary embodiment, however, the described components of the embodiment each represent individual features of the invention that can be viewed independently of one another, which also develop the invention independently of one another and are therefore also to be viewed as part of the invention individually or in a combination other than that shown. Furthermore, the described embodiment can also be supplemented by further features of the invention that have already been described.

In 1 a motor vehicle 10 is shown in a bird's eye view, which can be, for example, a motor vehicle, in particular a passenger car. In the example shown, the motor vehicle 10 is driving towards a shopping center 12, for example. A parking garage 14 and several shopping buildings 16 are shown from the shopping center 12. Several signs 18 point to the parking garage 14. A driver 20 (see 2 ) sees the shopping center 12 in his field of vision 22. He wants to know whether there is a restaurant there.

As in 2 shown, the driver 20 points with one hand 24, for example, at the information signs 18 lying in his field of vision 22. He speaks or expresses the voice input “Are there any good restaurants?”

In the motor vehicle 10, the driver 20 is able to control, for example, a search in a database 28 by showing the hand 24 and the voice input 26 in combination, in which, for example, information regarding possible destinations can also be stored.

Based on 3 is explained below how in the motor vehicle 10 the driver 20 is able to control the database query using the combination of the voice input 26 and the pointing gesture with the hand 24. In 3 is a motor vehicle interior 30 from the perspective of the driver 20 shown. Shown are a steering wheel 32, the front windshield 34 and a disk root display 36 provided on the underside of the windshield 34.

The motor vehicle 10 has a first operating interface 38, which has a microphone Order 40 can have at least one microphone and a speech recognition device 42, which can be designed in a manner known from the prior art. A further operating interface 44 of the motor vehicle 10 can, for example, have an optical sensor 46, such as a single image camera, a video camera, a time-of-flight camera and/or an infrared camera. The optical sensor 46 generates, for example, video data V which can be received, for example, by a gesture recognition device 48. Gesture recognition of a gesture carried out by the driver 20 can be carried out by the gesture recognition device 48 based on the video data V. An environmental sensor 50, for example also a camera, can have its detection range directed into the driver's field of vision 22 and generate environmental data U of the motor vehicle surroundings. In 2 is illustrated, for example, by the video data V and the environment data U.

The voice recognition device 42 receives the voice input 26 via the microphone device 40. In the voice input 26, the voice recognition device 42 recognizes that there is a search command, here “restaurants”, and a reference term 52, which cannot be understood on the basis of the voice input 26 alone can. The reference term “there” indicates an object that the user does not describe in the voice input 26 itself, but using another input modality, here the hand gesture 24.

The speech recognition device 42 transmits the recognized speech input 26 to a control device 54, which can generate a control command, for example a search command for the database 28, from the speech input 26. The control device 24 can be provided, for example, by a program module of a control device or a central processing unit of the motor vehicle 10.

The reference term 52 contained in the voice input 26 must first be translated in terms of content, that is, converted into a concrete object information. The control device 24 receives coordinate data 56 from the gesture recognition device 48, which were determined based on the gesture carried out by the driver 20 with the hand 24.

The gesture recognition device 48 has recognized in the video data V by means of gesture recognition that the driver 20 has held the hand 24 straight forward with an extended index finger and has thereby aligned a pointing line 58 to an object outside the motor vehicle 10. Gesture recognition can be carried out in a manner known per se from the prior art. For example, Microsoft's “Kinect” function library can be used for this purpose.

The gesture recognition device 48 can extrapolate the pointing line 58 and, based on the surroundings data U, determine that the pointing line 58, for example, intersects the information signs 18 depicted in the surroundings data U. For the cutting area, i.e. the information signs 18 shown in the surrounding data U in the example, their geometric position XYZ can be determined, for example, from the vehicle's own position and a distance measurement. The position XYZ is transmitted from the gesture recognition device 48 to the control device 54 by coordinates 56. The position XYZ stands in the database 28 for an entire area, here for example the entire shopping center 12.

According to the invention, the control device 54 can now generally replace the reference term 52 with the user input received via the other input modality, here the coordinates 56, and overall construct a control command with the following slots: an operating command (here search query), a command object (here the location XYZ ) and parameters for the operating command (here restaurants). Specifically, this means that a search for location XYZ should be used to find information about restaurants. The information about the shopping center 12 is stored in the database 28 for the location XYZ (so-called point of interest).

The information about the restaurants in the shopping center 12 subsequently received from the database 28 by the control device 24 can be displayed to the driver 20 as information fields 60, for example in his field of vision 22 using the disk root display 26.

A second possible reference term 52 ', the term "good" can be recognized by the speech recognition device 52. The control device 54 can be designed to see in the term “good” a reference to a user profile of the driver 20 and to determine his preference there in relation to restaurants or a reference to a ranking and in the database 28 also a ranking of Determine user recommendations for the restaurants located in shopping center 12.

It can now be provided that the driver 20 utters the voice input “Which restaurant is this?” to one of the information fields 60 and in this case to one of the information fields 60 shows. The reference term “that” again signals to the control device 54 that the driver 20 is pointing to a specific one of the information fields 60 by means of a gesture.

In the same way, the driver can, for example, select additional control panels 62 on the window root display by pointing and give the voice input: “Activate this” or “What function does this control panel have?”

Overall, through the combination of speech and another input modality, such as a gesture, a cross-modality or cross-modality is realized by combining speech and gesture operation by detecting both inputs, i.e. the speech input 26 and an operating gesture with one hand 24, and depending on each other be interpreted. The user can therefore use a reference term in his voice input to indicate that the voice input needs to be supplemented with information from the other input modality. This advantageously results in natural communication using language and gestures, for example. In addition to gestures, a parallel evaluation of different interaction modalities (speech, displays, buttons/rotary/push controls, touchpad/touchscreen, gestures, looks) must be related to one another in order to understand the meaning of the input modalities and derive a single crossmodal one Combine utterance and call to action.

Overall, the invention in the automotive context enables the connection between pointing to a real-world object, e.g. B. a building, and triggering a function via voice.

Claims (6)

Motor vehicle (10) with a first operating interface (38), through which voice control of at least one device (28) of the motor vehicle (10) is provided, and with a further operating interface (44), through which one on at least one other than voice Input modality-based operation of the at least one device (28) is provided, wherein the first operating interface (38) is designed to recognize at least one reference term (52) in a voice input (26) from a user (20), which refers to at least one object ( 18) indicates that the user (20) describes using the at least one other input modality, a control device (54) being designed to, depending on the at least one recognized reference term (52) and depending on one via the further operating interface (38 ) received user input (56) to determine the at least one object (18) and to generate at least one control command depending on the voice input (26) and the at least one determined object (18) and to send the at least one control command to the at least one device (28 ), characterized in that at least one environment sensor is provided, via which the control device (54) is designed to determine at least one actual state of the at least one object (18), wherein the control device (54) is additionally designed to Recognizing a relative information in the voice input (26) to generate the control command depending on the actual state and the relative information. Motor vehicle (10). Claim 1 , wherein the first operating interface (38) is designed to recognize at least one adverb of the location and/or a demonstrative pronoun (52) and/or a comparative and/or a superlative as the at least one reference term (52). Motor vehicle (10) according to one of the preceding claims, wherein the further operating interface (44) comprises at least one of the following devices: a gesture recognition (48), a handwriting recognition, a touchpad, a touchscreen, a rotary-push control, a keyboard, a View direction detection. Motor vehicle (10) according to one of the preceding claims, wherein the control device (54) is designed to use the at least one control command to control route planning in a navigation device and/or a database query for determining location descriptions (60) and/or a description of a To determine vehicle function in a digital operating manual of the motor vehicle (10). Motor vehicle (10) according to one of the preceding claims, wherein the motor vehicle (10) is designed to display on a display device (36) an output (60) of the at least one device (28) in the motor vehicle (10) triggered by the at least one control command. arranged in one place in such a way that, from the perspective of the user (20), the at least one object (18) and the output (60) are located together in a field of vision (22) of the user (20). Method for operating a motor vehicle (10), wherein voice control of at least one device (28) of the motor vehicle (10) is provided by a first operating interface (38) and a different input modality other than voice is provided by a further operating interface (44). based operation of the at least one device (28) is provided, wherein the first operating interface (38) is in a language given (26) by a user (20) recognizes at least one reference term (52), which refers to at least one object (18) that the user (20) describes using the at least one other input modality, and a control device (54) depending on the at least one recognized reference term (52) and depending on a user input (56) received via the at least one further operating interface (44), the at least one object (18) is determined and depending on the voice input (26) and the at least one determined object (18) generates at least one control command and sends the at least one control command to the at least one device (28), characterized in that at least one environment sensor is provided, via which the control device (54) each provides at least one actual state of the at least one object ( 18), the control device (54) additionally generating the control command when a relative information is detected in the voice input (26) depending on the actual state and the relative information.

Images