DE102019132054A1 - Method for operating a virtual assistant - Google Patents

Abstract

In at least one embodiment of the method for operating a virtual assistant, wherein
- the virtual assistant includes several conversation modes (M1, M2, M3),
- each conversation mode is assigned to a conversation situation,
- the virtual assistant takes on different functions in a conversation in the different conversation modes,
- the following steps are carried out:
A) providing speech information (I1) which is representative of a speech input by one or more people;
C) determining a conversation situation as a function of the language information;
D) Setting the conversation mode assigned to the conversation situation and operating the virtual assistant in this conversation mode.

Description

A method for operating a virtual assistant is specified. In addition, a device, a computer program, a computer-readable storage medium and a vehicle are specified.

A task to be solved consists in specifying a method for operating a virtual assistant with which a conversation between several people can be efficiently controlled. Further objects to be solved consist in specifying a device, a computer program and a computer-readable storage medium for carrying out this method. Yet another object to be achieved consists in specifying a vehicle with such a device.

First, the method for operating a virtual assistant is given. A virtual assistant, also called intelligent virtual assistant, English Virtual Intelligent Assistant, IVA for short, or digital assistant, is software that, for example, performs a speech analysis based on speech recognition, interprets it semantically, processes it logically and as a result formulates an answer through speech synthesis or takes an action. For example, the architecture of the virtual assistant includes several components. For example, the architecture includes components for STT (Speechto-Text), NLU (Natural Language Understanding), DM (Dialog Management), NLG (Natural Language Generation) and TTS (Text-to-Speech, in German text-to-speech). In current architectures, language analysis (STT) and semantic interpretation (NLU) take place in one step.

Alternatively, the software can comprise a neural network with an end-to-end neural architecture. The neural network is then, for example, a component that receives a voice input as input and outputs a voice output as output.

According to at least one embodiment, the virtual assistant comprises several conversation modes. Each conversation mode is assigned to a conversation situation. The virtual assistant takes on different functions in a conversation in the different conversation modes. “Function in a conversation” means in particular the role in the conversation.

For example, a different algorithm for the logical processing of the speech information obtained through speech recognition is assigned to each conversation mode. Depending on the conversation mode, the virtual assistant can be set up to formulate certain answers by means of speech synthesis and / or to carry out certain actions.

According to at least one embodiment, the method comprises a step A) in which speech information is provided which is representative of a speech input or utterance by one or more people. The language information is generated by the virtual assistant through speech recognition, for example through natural language recognition, English Natural Language Understanding, or NLU for short. The language information is in particular computer data. The language information contains information about what was said by the at least one person.

According to at least one embodiment, a conversation situation is determined as a function of the language information in a step C). This is to be understood in particular as the fact that conversation information is ascertained that is representative of the conversation situation. “Conversation situation” is understood to mean the type of conversation. Types of conversation can be: meeting, debate, dialogue, monologue, argument, service conversation, discussion, chat, conversation, telephone conversation, negotiation. For example, depending on the language information, it is determined whether a conversation is taking place between several people or whether one or more people are talking to the virtual assistant. For example, the number of people involved in the conversation is also determined from the voice information.

The conversation situation can be determined again after each utterance or the conversation situation is initially determined after a first utterance or the conversation situation is determined again and again, at intervals after an utterance.

According to at least one embodiment, the method comprises a step D) in which the conversation mode assigned to the determined conversation situation is set and the virtual assistant is operated in this conversation mode. In other words, the assigned conversation mode is determined and then set as a function of the conversation information ascertained, which is representative of the conversation situation. In the set conversation mode, the virtual assistant can, for example, intervene in the conversation or not take part in a conversation at all.

For example, a parameter suitable for the conversation situation (parameterization, configuration) is set first. This parameter can then be used in individual components of the Architecture of the virtual assistant. For example, depending on this parameter, the dialog management (DM) is parameterized and a suitable dialog status, dialog status transitions, actions to be carried out are selected as a result. Furthermore, depending on the parameter, the NLG is parameterized, for example, and suitable text modules, text templates, etc. are selected as a result. Furthermore, the speech synthesis (TTS) can be parameterized as a function of the parameter, for example, and suitable voice styles, speaking speeds, pitches, intonations, etc. can be selected as a result.

The parameter can also be used in combined components of a newer architecture of a virtual assistant. For example, depending on the parameter, the component is parameterized, which corresponds to the function of the combination of NLG and TTS. This means that speech is synthesized directly from the dialog manager's output commands and suitable text modules, text templates, voice styles, speaking speeds, pitches, intonations, etc. are synthesized.

The parameter can also be used in an end-to-end neural architecture of a virtual assistant. The parameter is transferred to the end-to-end neural architecture, for example as a vector, together with the utterances / speech inputs.

Steps A), C) and D) are preferably carried out in alphabetical order. The method described here is, in particular, a method that can be carried out on a computer.

• - der virtuelle Assistent mehrere Gesprächsmodi umfasst,
• - jeder Gesprächsmodus einer Gesprächssituation zugeordnet ist,
• - der virtuelle Assistent in den unterschiedlichen Gesprächsmodi unterschiedliche Funktionen in einem Gespräch übernimmt,

werden folgende Schritte ausgeführt:

• A) Bereitstellen von Sprachinformationen, die repräsentativ für eine Spracheingabe von einer oder mehreren Personen sind;
• C) Ermitteln einer Gesprächssituation in Abhängigkeit von den Sprachinformationen;
• D) Einstellen des der ermittelten Gesprächssituation zugeordneten Gesprächsmodus und Betreiben des virtuellen Assistenten in diesem Gesprächsmodus.

In at least one embodiment of the method for operating a virtual assistant, wherein

• - the virtual assistant includes several conversation modes,
• - each conversation mode is assigned to a conversation situation,
• - the virtual assistant takes on different functions in a conversation in the different conversation modes,

the following steps are carried out:

• A) providing speech information representative of speech input from one or more people;
• C) determining a conversation situation as a function of the language information;
• D) Setting the conversation mode assigned to the determined conversation situation and operating the virtual assistant in this conversation mode.

Virtual assistants are often set up to communicate with one person only. If several people then speak to the virtual assistant, it cannot distinguish between the people. The conversation between the people and the virtual assistant would then be as if only one person were talking to the virtual assistant.

However, it is often desired that several people can talk to the virtual assistant, for example when several people are sitting in a car. There are virtual assistants who recognize that several people are talking to them and who can also differentiate between people.

The present invention is based on the knowledge that different conversation situations can arise, in particular when several people are talking to one another. One idea of the present invention is to provide a virtual assistant that adapts to the conversation situation, in particular to changing conversation situations. In the present case, this is achieved by assigning several conversation modes to the virtual assistant, in which the virtual assistant can be operated depending on the conversation situation. The virtual assistant automatically recognizes the conversation situation.

Such a virtual assistant can direct a conversation between people and adapt it to specific situations. For example, when several people are talking in a vehicle, the virtual assistant can calm down a conversation and thus increase security.

According to at least one embodiment, the method comprises a step B) in which a number of people is determined who speak to one another or to the virtual assistant. Alternatively, the number of people in a room or a vehicle cabin can be determined.

The number of people is determined, for example, as a function of the language information. Different people can be recognized by voice recognition, for example. Alternatively or additionally, the number of people is determined as a function of additional information. The additional information is, for example, information that cannot be traced back to a voice input. For example, the additional information is provided by sensors provided that detect the presence of people. For example, these are pressure sensors that are assigned to the seats of a vehicle and with which it is recognized whether a person is sitting on the seat. One or more camera sensors are also conceivable for determining the number of people.

According to at least one embodiment, the conversation situation is determined in step C) as an additional function of the determined number of people. For example, the virtual assistant includes different conversation modes for different conversation situations in conversations between at least two people.

According to at least one embodiment, one or more call features are determined in step C) and / or in step D) as a function of the voice information. Each conversation characteristic is an element from the group: conversation flow, context variables, emotions of the individual persons, social relationships between the persons, speaking times of the individual persons, speaking volume of the individual persons, speaking speed of the individual persons, speech interruptions by individual persons, repetition of terms by individual persons , Identities of the people. The determination of a conversation feature means in particular the determination of information that is representative of the conversation feature.

According to at least one embodiment, in step C) the conversation situation is determined as a function of the conversation feature (s) or as a function of the information that is representative of the conversation features. For example, the conversation features are compared with a stored communication pattern in order to determine the conversation situation. The conversation features can also be entered into a trained neural network and the conversation situation can be determined in this way. The communication pattern or the neural network can be generated by machine learning.

Some conversation features can additionally or alternatively be determined as a function of additional information that is independent of the speech information. Information from sensors, for example seat sensors or camera sensors, can be used as additional information. In particular, conversation features that are representative of the context variables mentioned above, social relationships between the people and identities of the people can also or only be determined as a function of the additional information that is independent of the language information. Context variables are understood to mean, for example, the time of day, schedules of the individual persons, weather conditions, traffic situation and the like.

According to at least one embodiment, the plurality of conversation modes comprise a first conversation mode in which the virtual assistant assumes the function of a mediator between a plurality of people.

According to at least one embodiment, the multiple conversation modes include a second conversation mode in which the virtual assistant summarizes a conversation between multiple people.

According to at least one embodiment, the multiple conversation modes include a third conversation mode in which the virtual assistant moderates a conversation between multiple people

The first, second and third conversation modes are therefore in particular conversation modes that are set if several people speak to one another or to the virtual assistant. In this respect, the first, second or third conversation mode is preferably switched on if the number of people determined is at least two.

According to at least one embodiment, when the virtual assistant is operated in the first conversation mode, a step is carried out in which the intentions of the persons are determined from the language information. In a further step, an output signal for asking a question about a person's intention is then generated. Alternatively, an output signal for outputting a compromise proposal for the different intentions is generated. The output signal can be output in the form of an audio signal. Alternatively or additionally, a video signal or lighting signal can be output.

For example, a first person in a vehicle wants to turn the radio up and a second person in the vehicle wants to turn the radio down. The virtual assistant would then, for example, generate an output signal that is representative of the compromise suggestion “Let's leave the volume as it is”.

According to at least one embodiment, when the virtual assistant is operated in the second conversation mode, a step is carried out in which the intentions of the persons are determined from the language information. In a subsequent step, an output signal is generated that is representative of a summary of the intentions.

For example, people who are sitting in a vehicle want to go out to eat, a first person communicating that they are available from 8 a.m. and a second person communicating that they are available from 9 a.m. The virtual assistant would then, for example, generate an output signal with the following content: “Person 1 can start at 8 o'clock, person 2 from 9 o'clock. Should I reserve a table for 9 o'clock? ”.

According to at least one embodiment, when the virtual assistant is operated in the third conversation mode, a step is carried out in which the contents of the conversation are determined as a function of the language information. In a further step, the speaking times of the people for the content are then determined as a function of the language information. In a subsequent step, the person with the shortest speaking time for the content is determined. In a further step, an output signal is generated to motivate the people with the shortest speaking time to comment on the content.

When determining the speaking times of the individual persons, it is determined, for example, when an identified person begins to speak and when the identified person stops speaking. The time difference in between is speaking time. If the same person starts speaking again in the same conversation, the new speaking time will be added to the old speaking time. The speaking time of a person is the sum of all speaking times of that person.

In particular when operating the virtual assistant in the first operating mode or in the third operating mode, repetitions of terms by individual persons can be recorded. For example, every time a person uses a certain term or every time a person repeats their statement in terms of content, a counter is incremented. Using the counter, the virtual assistant determines how often a certain person has already commented on a topic or how dominant the person appears in the conversation. Accordingly, the virtual assistant can ask the other people to comment on the content again. Pre-trained language models at word, sentence or paragraph level, such as BERT, UMLfit et cetera, can be used to identify statements that are identical or similar in terms of content or semantics.

Furthermore, when the virtual assistant is operating in the first and / or third operating mode, interruptions in speech by individual persons can be recorded. A speech interruption by a person is detected, for example, when a person starts to talk before a person who previously started talking has finished speaking. For example, the virtual assistant counts for each person how often they interrupted another person. Accordingly, the virtual assistant can withdraw speaking time from the person who has interrupted the most frequently or point out the interruptions or give the other person more speaking time.

The virtual assistant can, for example, recognize people's emotions during operation, for example based on the speaking speeds of the individual people. In particular in the first operating mode, the virtual assistant can then, depending on the emotions of the person, ask them to speak more or to calm them down or the like.

Depending on the operating mode, the virtual assistant can be set up to remove a microphone from a person. This measure is conceivable in particular in the event that each person is assigned a personal listening zone, for example in a vehicle or in different rooms of a house or in different rooms of different houses.

According to at least one embodiment, the method comprises a step E) in which image information is provided that is representative of an image or video of the one or more people.

According to at least one embodiment, one or more conversation features are determined in step C) and / or D) as a function of the image information. The conversation features are in particular the above-mentioned conversation features. In particular, emotions of the individual persons can be determined as a function of the image information.

So that the virtual assistant determines the correct conversation situation, sets the correct conversation mode and acts correctly in the set conversation mode, the virtual assistant is trained using machine learning, for example. For this purpose, information is provided on the basis of video recordings, audio recordings, emotion clustering, for example, to which a communication pattern is assigned. The virtual assistant can then be trained as a function of this communication pattern. For example, mediator training materials, video recordings and audio recordings can be used to train the behavior of the virtual assistant in the first operating mode.

The training can only relate to the classification / recognition of individual conversation features (feature learning) or the entire conversation situation. This classification can then take the form of the described parameter can be used both in a traditional architecture (STT-NLU-DM-NLG-TTS) or in an end-to-end neural architecture. Alternatively, the determination of the conversation situation can also be part of the end-to-end neural architecture.

Next, the computer program is given. The computer program comprises commands which, when the computer program is executed by a computer, cause the computer to carry out the method described here. In particular, the computer program includes the virtual assistant.

The computer program is stored, for example, on a computer-readable storage medium.

Next the device is given. The device comprises a processor which is set up to carry out a method described here.

According to at least one embodiment, the device comprises a microphone for receiving a voice input and a loudspeaker for outputting acoustic signals. The microphone and the loudspeaker form communication interfaces between the virtual assistant and the one or more people. The device preferably comprises a plurality of microphones and / or loudspeakers.

According to at least one embodiment, the device comprises a camera for creating an image or a video. The camera forms another communication interface between the people and the virtual assistant.

Next, the vehicle is specified. The vehicle includes a device as described herein. The vehicle is, in particular, a motor vehicle, for example a car. For each seat in the car, the device can include its own microphone for communication with the virtual assistant. Furthermore, the vehicle can include sensors for identifying the individual persons, such as sensors in the seats of the vehicle.

Further advantages and advantageous configurations and developments of the method, the device and the vehicle emerge from the following exemplary embodiments illustrated in connection with the figures. Identical, identical or identically acting elements are provided with the same reference symbols in the figures. The figures and the proportions of the elements shown in the figures are not to be regarded as being to scale. Rather, individual elements can be shown exaggeratedly large for better displayability and / or for better understanding.

• 1 und 2 ein Ausführungsbeispiel des Fahrzeugs in verschiedenen Ansichten,
• 3 bis 7 Ausführungsbeispiele des Verfahrens anhand von Ablaufdiagrammen.

Show it:

• 1 and 2 an embodiment of the vehicle in different views,
• 3rd to 7th Embodiments of the method based on flowcharts.

In the 1 an embodiment of a vehicle is shown in plan view. The vehicle includes a device 10 which is set up to carry out a method described here for operating a virtual assistant. The device 10 includes a microphone 11 , a loudspeaker 12th and a camera 13th .

In the 1 it can be seen that a large number of people are in the vehicle P who are talking to each other or who are talking to the virtual assistant.

In the 2 a side view of the vehicle is shown.

3rd shows a first embodiment of the method for operating a virtual assistant on the basis of a flowchart. The following is an example of the situation of 1 Referenced. In one step S1 the procedure will be language information I1 provided that are representative of speech input by the people P are in the vehicle. In one step S2 it determines how many people P sit in the vehicle. Then depending on the language information I1 determined a conversation situation. For example, this is initially done in one step S3 depending on the language information I1 one or more conversation characteristics, such as speaking speeds, speaking volumes and speaking times of the individual persons. These call characteristics can then be used in one step S4 can be compared with a conversation pattern stored in the virtual assistant in order to determine the conversation situation. Alternatively, the conversation features can also be entered into a neural network in order to determine the conversation situation. In the present case it is determined, for example, that the conversation situation is a discussion or a dispute. In one step S10 then becomes a first conversation mode M1 set and the virtual assistant will be in this conversation mode M1 operated. In the first conversation mode M1 the virtual assistant takes on the function of a mediator between people.

In the 4th a second embodiment of the method is shown on the basis of a flowchart. Here, too, reference is made to the situation of the 1 taken. The steps S1 , S2 , S3 and S4 are like related to the 3rd described. Thereupon the step S10 the first conversation mode M1 set, in which the virtual assistant takes on the role of a mediator. In a subsequent step S11 become the intentions of the people P from the language information I1 determined. In a subsequent step S12 an output signal is generated which is used to ask a question about a person's intention or to output a compromise proposal for the different intentions.

In the 5 a third embodiment of the method is shown on the basis of a flowchart, again with reference to FIG 1 is taken. The steps S1 to S4 are like in the 3rd described. Depending on the determined conversation situation, there is now a second conversation mode M2 set (step S20 ), in which the virtual assistant starts a conversation between people P summarizes. When performing the second conversation mode M2 be in one step S21 first of all, intentions of the people from the language information I1 determined and in one step S22 becomes an output signal AS generated that is representative of a summary of intentions. The output signal AS is for example via the microphone 12th issued as an acoustic signal.

In the 6th a fourth embodiment of the method is shown on the basis of a flowchart. Again be the step S1 to S4 run and then it is done in one step S30 a third conversation mode M3 set. When performing the third conversation mode M3 is in one step S31 First, the content of the conversation between the people depending on the language information I1 determined. Then in one step S32 The speaking times of the people on the content are determined as a function of the language information. This is done in one step S33 identifies the person who had the shortest speaking time on the content. Finally, in a step S34, an output signal AS generated that motivates the person with the shortest speaking time to comment on the content.

In the 7th a fifth embodiment of the method is shown on the basis of a flowchart. Here is language information I1 (Step S1 ) and image information I2 (Step S2 ), for example through the camera 13th the 1 are used. From the language information I1 and / or the image information I2 will be in crotch S3 first the number of people in the vehicle is determined. Depending on the language information I1 and the image information I2 conversation characteristics are determined (step S3 ) and depending on the conversation characteristics, the conversation situation is determined (step S4 ). In one step S10 the conversation mode that matches the conversation situation is then set, which in turn is the first conversation mode M1 is.

The number of people could also be queried from other sources, for example from an occupant model.

The invention is not restricted to the exemplary embodiments by the description thereof. Rather, the invention encompasses every new feature and every combination of features, which in particular includes every combination of features in the patent claims, even if these features or this combination itself is not explicitly specified in the patent claims or exemplary embodiments.

List of reference symbols

10

contraption

11

microphone

12th

speaker

13th

camera

P

person

S1-S34

Procedural steps

I1

Language information

I2

Image information

M1

first operating mode

M2

second operating mode

M3

third mode of operation

AS

Output signal

Claims (13)

Method for operating a virtual assistant, wherein - the virtual assistant comprises several conversation modes (M1, M2, M3), - each conversation mode (M1, M2, M3) is assigned to a conversation situation, - the virtual assistant in the different conversation modes has different functions in one Conversation takes over, - the method comprises the steps: A) providing voice information (I1) which is representative of a voice input from one or more people (P); C) determining a conversation situation as a function of the language information (I1); D) Setting the conversation mode assigned to the determined conversation situation (M1, M2, M3) and operating the virtual assistant in this conversation mode. Procedure according to Claim 1 , further comprising the step: B) determining a number of people (P) who speak to one another or to the virtual assistant; - wherein in step C) the conversation situation is determined as an additional function of the determined number of people (P). Procedure according to Claim 1 or 2 , whereby - in step C) and / or in step D) one or more conversation features are determined as a function of the language information, each conversation feature being an element from the group: conversation flow, context variables, emotions of the individual persons, social relationships between the persons, Speaking times of the individual persons, speaking volume of the individual persons, speaking speed of the individual persons, speech interruptions by individual persons, repetition of terms by individual persons, identities of the persons, - in step C) the conversation situation is determined depending on the conversation characteristic (s). Method according to one of the preceding claims, wherein the plurality of conversation modes (M1, M2, M3) comprise: - a first conversation mode (M1), in which the virtual assistant assumes the function of a mediator between several people (P), - a second conversation mode (M2) in which the virtual assistant summarizes a conversation between several people (P), - A third conversation mode (M3) in which the virtual assistant moderates a conversation between several people (P). Procedure according to Claim 4 , wherein - when operating the virtual assistant in the first conversation mode (M1), the following steps are carried out: - Determining the intentions of the persons (P) from the voice information (I1), - Generating an output signal (AS) for making a query about an intention of a Person or generation of an output signal (AS) to output a compromise proposal for the different intentions. Procedure according to Claim 4 wherein - when operating the virtual assistant in the second conversation mode (M2), the following steps are carried out: - Determining the intentions of the persons from the voice information (I1); - Generating an output signal (AS) which is representative of a summary of the intentions. Procedure according to Claim 4 wherein - when operating the virtual assistant in the third conversation mode (M3), the following steps are carried out: - determining a content of the conversation as a function of the language information (I1), - determining speaking times of the people for the content as a function of the language information (I1); - Identify the person with the shortest speaking time on the content; - Generating an output signal (AS) to motivate the person with the shortest speaking time to comment on the content. A method according to any one of the preceding claims, further comprising: E) providing image information (I2) which is representative of an image or video of the one or more people (P), wherein - In step C) and / or D) one or more call features are determined as a function of the image information (12). Computer program comprising instructions which cause the computer program to be executed by a computer, the method according to one of the Claims 1 to 7th to execute. Computer-readable storage medium on which the computer program is based Claim 8 is stored. Device (10) comprising a processor which is set up so that it uses a method according to one of the Claims 1 to 7th executes. Device (10) according to Claim 11 , further comprising: - a microphone (11) for receiving a voice input, - a loudspeaker (12) for outputting acoustic signals, - a camera (13) for creating an image or a video. Vehicle comprising a device (10) according to Claim 11 or 12th .

Images