JP2023048466A - Conversation management system and conversation management method - Google Patents

Abstract

To determine a speech production destination in conversation among a plurality of participants.SOLUTION: A conversation management system manages conversation in which a plurality of participants are present. The conversation management system stores past speech production history information of each of the participants in the conversation. The conversation management system acquires a text feature amount of first current speech production of a speaker. The conversation management system acquires a text feature amount of past speech production of each of a plurality of listeners. The conversation management system calculates a speech production destination probability of each listener on the basis of the text feature amount of the first current speech production and the text feature amount of the past speech production of each listener. The conversation management system determines a speech production destination of the first current speech production on the basis of the speech production probabilities of the listeners.SELECTED DRAWING: Figure 7

Description

The present disclosure relates to techniques for managing conversations of multiple participants.

In a conversation in which multiple people participate, there are technologies related to agents that support the conversation. In daily life, it is common for a plurality of people to have a conversation. Here, a person who participates in the conversation is called a user. In a conversation using a system in which an agent supporting conversation exists, a plurality of users not only converse with each other but also converse with an agent of the system. That is, each user conducts a conversation with one participant or with multiple participants. A participant here is a human user or an agent. Users coordinate conversations among themselves so that the purpose of the conversation can be achieved efficiently and effectively.

The agent may respond to previous user utterances, may spontaneously speak new ideas, or may interrupt the user when necessary. In order to achieve the purpose of conversation, it is important to be able to seamlessly continue conversations between users. For this reason, it is conceivable as a means of realizing this that an agent can speak appropriate content at an appropriate timing, thereby supporting the seamless continuation of conversations between users. In order for an agent to support conversations efficiently, it is essential to have the ability to identify which participant the speaker is talking to. However, it is not obvious how to detect who the speaker is speaking to in a conversation involving multiple users.

For example, US Pat. No. 9,761,247 discloses a system for supporting conversations involving multiple users. Multiple users converse with each other and with agents of the system. The system uses prosodic and lexical features in multi-user conversations to identify whether a speech is spoken to an agent or not.

U.S. Patent No. 9761247

Previous speech target detection techniques focus on choosing between two options: whether a user's speech is directed to an agent or to a human user. However, for true communication, conventional technology alone is insufficient. The agent of the system, for example, guesses which user in the conversation is most likely to speak next, and depending on the result of the guess, decides whether the agent should speak next, wait, etc. There is a need. For this purpose, it is important to appropriately estimate which of the users the speaker is speaking to.

There are two types of speech target detection: agent-oriented and human-oriented. Agent-directed means that the talkee belongs to an agent in the system. However, just estimating utterances for agents is not enough. In a conversation within a group, a speaker can speak to any of the other participants (including human users and agents) or to the entire group. Therefore, in detecting who the speaker is speaking to, identifying both those speaking to some specific user and those speaking to a group seamlessly continues the conversation. important to.

One aspect of the present disclosure is a conversation management system including one or more storage devices and one or more computing devices. The one or more storage devices store past speech history information of each of the plurality of participants in the conversation. The one or more computing devices obtain text features of a first current utterance of a speaker. The one or more computing devices acquire text feature amounts of past utterances of each of a plurality of listeners from the past utterance history information. The one or more computing devices calculate the utterance destination probability of each listener based on the text feature amount of the first current utterance and the text feature amount of the past utterance of each listener. The one or more computing devices determine the destination of the first current utterance based on the destination probabilities of the plurality of listeners.

According to one aspect of the present disclosure, it is possible to estimate the utterance destination of a speaker in a conversation in which a plurality of participants are present.

1 shows an example of a usage form of a conversation management system according to an embodiment of the present specification; Fig. 3 shows another form of use of the conversation management system of the present specification; A hardware configuration example of a conversation management system is shown. 1 shows an example of the hardware configuration of a user terminal; 1 shows a functional configuration example of a conversation management system according to an embodiment of the present specification; 4 shows a configuration example of participant-specific feature quantity management information; 4 shows a configuration example of utterance history management information; 4 is a flow chart of an example of operation by the conversation management system; Figure 2 shows a sample of utterances collected in a conversation. An example of input to and output from a recurrent neural network that can be used by the speech destination determination unit to determine the speech destination will be shown. Schematically illustrates an example of semantic relevance of words. An example of speech destination determination processing by multitasking is shown. Shows one of a series of utterances that make up a conversation. Shows one of a series of utterances that make up a conversation. Shows one of a series of utterances that make up a conversation. Shows one of a series of utterances that make up a conversation. Shows one of a series of utterances that make up a conversation. Shows one of a series of utterances that make up a conversation. Shows one of a series of utterances that make up a conversation. Shows one of a series of utterances that make up a conversation. It indicates an RNN that identifies whether a speaker's utterance destination is a group or an individual. Details of processing in LSTM are shown. The details of inputting words into the word embedding layer and LSTM are shown. If the addressee is an individual, indicate the RNN that determines which listener is the addressee. 1 shows the structure of MaLSTM.

The following description is divided into multiple sections or embodiments when necessary for convenience, but unless otherwise specified, they are not independent of each other, and one part or all of the other Modifications, details, supplementary explanations, etc. In addition, hereinafter, when referring to the number of elements (including number, numerical value, amount, range, etc.), unless otherwise specified or clearly limited to a specific number in principle, is not limited to the number of , and may be greater than or less than a specific number.

This system may be a physical computer system (one or more physical computers), or a system constructed on a group of computational resources (a plurality of computational resources) such as a cloud platform. A computer system or a group of computing resources includes one or more interface devices (for example, including communication devices and input/output devices), one or more storage devices (for example, including memory (main storage) and auxiliary storage devices), and one Including the above arithmetic unit.

When a function is realized by executing a program by an arithmetic device, the specified processing is performed using a storage device and/or an interface device as appropriate, so the function is at least part of the arithmetic device. may be The processing described with the function as the subject may be processing performed by a system having an arithmetic device. Programs may be installed from program sources. The program source may be, for example, a program distribution computer or a computer-readable storage medium (eg, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

One embodiment herein provides a mechanism for estimating who the speaker is speaking to (speaker target) when multiple users are having a conversation. One embodiment herein is a system for supporting conversations involving multiple participants. "Participant" means a participant in a conversation. A "participant" can be a "user" or an "agent". A "user" is a human being. An "agent" is, for example, a robot equipped with a speech recognition function, a computer system executing an application program, or its program. The agent may also have a voice synthesis function and be able to respond by voice.

In one embodiment herein, a group participating in a conversation may consist of three or more participants and may include one or more agents. However, the configuration may be such that the agent does not exist, and the conversation management system described in this embodiment simply estimates the next speaker in a conversation between human users. A conversation management system listens to utterances by participants in a conversation in which a plurality of participants are participating, and estimates which participant is the utterance destination.

In one embodiment herein, a conversation management system performs N-free classification. Here, N is a symbol representing the number of participants, which means that the system for detecting speech destinations can operate independently of the number of participants. The conversation management system is operable in conversations with any number of participants. In one embodiment herein, the conversation management system calculates a destination probability score for each participant based on calculations using the input utterances and each listener's utterance history and other information. The participant showing the highest probability is presumed to be the target of the current utterance.

A conversation management system according to an embodiment of the present specification estimates a speech destination by executing the processing described below. FIG. 1A illustrates an example usage of a conversation management system according to one embodiment herein. The system includes a conversation management system 101 and multiple user terminals 104 . In the example of FIG. 1A, conversation management system 101 consists of one server, and four users are participating in the conversation.

User terminal 104 accesses access point 103 either wirelessly as shown in FIG. 1A or via a cable. Conversation management system 101 and user terminal 104 communicate with each other via network 102 and access point 103 . The configuration of the communication network between conversation management system 101 and user terminal 104 is arbitrary.

The conversation management system 101 provides the user terminal 104 with services such as conversations via the network, such as conferences and meetings. In addition, conversation management system 101 executes agent programs to facilitate conversations between users. The conversation management system 101 provides a conversation service in which multiple users can participate and supports the conversation.

The conversation management system 101 can be composed of one or more computers, or can be composed of computer resources on the cloud. As such, conversation management system 101 may include one or more computing devices and one or more storage devices.

A plurality of users can log in to the conversation management system 101 using the user terminals 104 respectively, and have conversations, including large and small meetings, on the platform provided by the conversation management system 101 . A user can transmit his or her own video to other participants through a camera mounted on the user terminal 104, and can transmit his or her speech to other participants through a microphone. The user can visually recognize images of other users on the display device mounted on the user terminal 104, and can hear speech of other participants through the speaker.

FIG. 1B illustrates another use of the conversation management system 101 herein. Conversation management system 101 can be implemented, for example, in a desktop or laptop computer system. A plurality of users have a direct face-to-face conversation, and a conversation management system 101 present at the place of conversation supports the conversation between the users. In the example of FIG. 1B, four users 105 are participating in a conversation.

Each participant can converse with other participants without using the device. The conversation management system 101 can acquire voice data of the user's 105 utterances through the installed microphone, and speak to the participants through the speaker. Conversation management system 101 may be equipped with a camera so that an image of each user 105 can be obtained.

FIG. 2 shows a hardware configuration example of the conversation management system 101. As shown in FIG. The conversation management system 101 includes a main memory device 202 composed of volatile memory elements such as RAM, and an auxiliary memory device 203 composed of appropriate non-volatile memory elements such as SSD (Solid State Drive) and hard disk drive.

Further, the conversation management system 101 reads out a program 206 stored in the auxiliary storage device 203 or the like into the main storage device 202 and executes it, performs overall control of the system itself, and performs various determinations, calculations and control processes. It includes an arithmetic device 201 such as a CPU that performs the processing. Conversation management system 101 includes communication interface 204 for connecting to and exchanging data with a network.

The conversation management system 101 includes an input device 208 such as a keyboard, mouse, touch panel, microphone, and camera for receiving input operations, and an output device 305 such as a display for displaying processing results and a speaker for outputting voice. The components of conversation management system 101 can communicate with each other by internal bus 205 . Some of the components of conversation management system 101 may be omitted, such that output device 209 and input device 208 can be omitted in the configuration example shown in FIG. 1A.

The auxiliary storage device 203 stores a program 206 for implementing the functions necessary for the conversation management system 101 of this embodiment, as well as an information database (DB) 207 storing data necessary for various processes. . FIG. 2 illustrates program 206 loaded from secondary storage device 203 into main storage device 202 .

FIG. 3 shows an example hardware configuration of the user terminal 104 . The user terminal 104 includes a main memory device 302 composed of volatile memory elements such as RAM, and an auxiliary memory device 303 composed of appropriate non-volatile memory elements such as SSD and hard disk drive. The user terminal 104 further reads a program 308 stored in the auxiliary storage device 303 or the like into the main storage device 302 and executes it, performs overall control of the system itself, and performs various determinations, calculations, and control processes. It includes an arithmetic unit 301 such as a CPU. Programs 308 include programs that enable user terminal 104 to participate in conversations over a network.

The user terminal 104 includes a communication interface 306 for connecting to a network and exchanging data, an input device 304 such as a keyboard, mouse, touch panel, microphone, and camera for receiving input operations from the user, and processing results to the user. It includes an output device 305 such as a display for displaying and a speaker for outputting sound. These user terminal 104 components can communicate with each other via an internal bus 307 . The auxiliary storage device 303 stores programs 308 for implementing necessary functions and necessary information (data).

Next, the functions of conversation management system 101 will be described. FIG. 4 shows a functional configuration example of the conversation management system 101 according to one embodiment of the present specification. The functions described below are implemented by the arithmetic unit 201 of the conversation management system 101 executing the program 206, for example.

Program 206 of conversation management system 101 includes speech recognition unit 401 , participant identification unit 403 , audio feature amount extraction unit 405 , video feature amount extraction unit 407 , speech destination determination unit 411 , and agent 413 . The database 207 includes participant-specific feature quantity management information 421 and speech history management information 423 . Details of the participant-specific feature amount management information 421 and the utterance history management information 423 will be described later with reference to FIGS. 5 and 6, respectively.

A speech recognition unit 401 recognizes an utterance by a user. Speech is received via the microphone of conversation management system 101 (example of FIG. 1B) or over the network from user terminal 104 (example of FIG. 1A). For example, the speech recognition unit 401 can extract the speech as an utterance when the speech is followed by silence for a predefined silence period, for example, 200 milliseconds or longer.

The speech recognition unit 401 further converts the user's speech data into text. The result transformed from the utterance is the text feature of the utterance. The speech recognition unit 401, when given as an input in speech, converts it into vocabulary or words and outputs it. The utterance text is stored in the utterance history management information 423 .

The participant identification unit 403 determines the utterance position of the participant in the situation shown in FIG. 1B, and associates the determined utterance position of the participant with the participant ID. As shown in FIG. 1A, in the case of a conversation over a network using the user terminal 104, the participant identification unit 403 determines the participant ID based on the participant's login ID without determining the participant's position. be able to.

The voice feature quantity extraction unit 405 extracts a voice feature quantity from the voice data of the utterance by a predetermined processing method. Examples of speech features include MFCC (Mel-Frequency Cepstrum Coefficients), PLP (perceptual linear prediction), pitch, and the like. The voice feature amount extraction unit 405 stores the extracted voice feature amount in the utterance history management information 423 .

A video feature amount extraction unit 407 extracts a video feature amount from the speaker's video data by a predetermined processing method. The video feature amount is generated from pixel values of video data using, for example, a convolutional neural network. The video data is received from the camera of conversation management system 101 (example in FIG. 1B) or from user terminal 104 via the network (example in FIG. 1A). The video feature quantity extraction unit 407 stores the extracted video feature quantity in the speech history management information 423 . The video feature amount extraction unit 407 may extract video feature amounts of other users in addition to the video feature amount of the speaker and store them in the speech history management information 423 .

The speech destination determination unit 411 estimates the speech destination of the current speech based on the speaker's current speech and the listener's past speech history. As will be described later, the estimation of the utterance destination may be performed based on the unique feature amount of the listener, and the feature amounts of the voice data, text data, and video data at the time of utterance.

The agent 413 encourages the progress of the conversation by working with the participants in the conversation using voice or voice and video. The agent 413 can determine whether or not to permit an utterance based on the recognized utterance destination, and can determine the utterance content and utterance destination to be output based on the recognized utterance and the utterance destination.

FIG. 5 shows a configuration example of the participant-specific feature quantity management information 421. As shown in FIG. The participant-specific feature quantity management information 421 manages feature quantities specific to each participant participating in the conversation. The participant-specific feature amount management information 421 is set and registered in advance before the conversation is started. In the configuration example shown in FIG. 5, the participant-specific feature amount management information 421 includes a participant ID column 451, a name column 452, a role column 453, a soft skill column 454, and a hard skill column 455.

Participant ID column 451 indicates the ID of each participant in the conversation. A name column 452 indicates the name of each participant. A role column 453 indicates the role assigned to each participant. Role column 453 may indicate the participant's role in the organization or conversation. In the example of FIG. 5, the role column 453 indicates positions within the company. A soft skills column 454 indicates general (non-professional) skills that are not based on the participant's expertise. The hard skill column 455 indicates skills (professional skills) based on the participant's specialized knowledge. Note that the participant-specific feature amounts shown in FIG. 5 are merely examples, and feature amounts different from these may be registered.

FIG. 6 shows a configuration example of the speech history management information 423. As shown in FIG. The utterance history management information 423 manages the history of utterances in conversation. The speech history management information 423 is updated as the conversation progresses. In the configuration example shown in FIG. 6, the utterance history management information 423 includes an utterance ID column 471, a speaker ID column 472, an utterance text column 473, an utterance time column 474, an audio feature column 475, and a video feature column 476. .

The utterance ID column 471 indicates the ID of the utterance. The speaker ID column 472 indicates the ID of the participant (speaker) who made the speech. The utterance text column 473 indicates the text of the utterance (text feature amount). As described above, the speech text is generated by the speech recognition unit 401 and stored in the speech history management information 423. FIG. The utterance time column 474 indicates the time of utterance, for example, the start time.

The voice feature quantity column 475 indicates the voice feature quantity of the utterance. As described above, the speech feature quantity is extracted by the speech feature quantity extraction unit 405 and stored in the utterance history management information 423 . The video feature quantity column 476 indicates the video feature quantity of the speaker who made the utterance. As described above, the video feature quantity is extracted by the video feature quantity extraction unit 407 and stored in the speech history management information 423 . Video feature amounts of other users different from the speaker may also be stored together.

The utterance history management information 423 may store information on all utterances in a conversation, or may store only information on utterances of a specified number or less for each speaker ID. The utterance history management information 423 may manage paths indicating these files instead of storing the utterance texts and the feature values themselves. Note that the utterance history management information 423 may manage only part of the information shown in FIG.

The operation of conversation management system 101 will be described below. FIG. 7 is a flow chart of an operation example by the conversation management system 101. As shown in FIG. Conversation management system 101 estimates a speaker's speaking destination in a conversation involving three or more participants. Participants can be joined by agents 413 of conversation management system 101 in addition to human users.

First, the conversation management system 101 receives conversation audio data and video data (S11). Audio data can be received from the microphone of conversation management system 101 or user terminal 104 over the network, and video data can be received from the camera of conversation management system 101 or user terminal 104 over the network.

Next, the speech recognition unit 401 analyzes the speech data, recognizes one utterance, and extracts the speech data (S12). Specifically, speech is converted into text, and text features are extracted. The speech recognition unit 401 can extract the speech as an utterance, convert it into text, and output it when silence continues for a predetermined silence time or longer after the speech.

Next, the participant identification unit 403 analyzes the voice data of the utterance and determines the corresponding speaker (S13). The participant identification unit 403 can determine the speaker's ID from the voice data of the utterance using machine learning (for example, diarization technology). Diarization takes speech as input and outputs who the speaker is.

In addition, as a method that does not use machine learning, the participant identification unit 403 may use a specific microphone device including a multi-channel microphone array to calculate and obtain the relative positional relationship between the speaker and the microphone. It is possible. Then, the speaker can be identified by the participant's ID associated with the speaker's location information. If video information is also available, person identification based on facial images may be used. Furthermore, speaker estimation based on facial lip movements or speaker estimation using both image information and audio information may be performed.

Next, a feature amount is extracted from the speech data of the speech and the video data of the speaker at the speech time (S14). Specifically, the speech recognition unit 401 extracts the text feature quantity, the speech feature quantity extraction unit 405 extracts the feature quantity of the speech data, and the video feature quantity extraction unit 407 extracts the video feature quantity from the video data of the speaker. do. The speech recognition unit 401 can use known automatic speech recognition technology. The voice recognition unit 401, the voice feature amount extraction unit 405, and the video feature amount extraction unit 407 can be configured using, for example, a neural network. The speech recognition unit 401, the speech feature quantity extraction unit 405, and the video feature quantity extraction unit 407 each store the extracted feature quantity in the record of the utterance history management information 423 (S15).

Next, the utterance destination determining unit 411 determines the utterance destination of the current target utterance (S16). The details of the determination method of the speech destination will be described later. In one embodiment of the present specification, the utterance destination determination unit 411 acquires current utterance information and past utterance information from the utterance history management information 423, furthermore, from the participant-specific feature amount management information 421, Acquire unique features of each participant. The speech destination determination unit 411 determines the speech destination based on the acquired information.

In one embodiment of the present specification, the speech destination determination unit 411 determines the type of the speech destination (S17). Specifically, it is determined whether the speech destination is all listeners of the participants (the whole group), a specific single listener user, or an agent that the conversation management system 101 is executing. When a specific user is the speech destination, the speech destination determination unit 411 determines the user ID of the user.

The output of the speech destination determination unit 411 is one of the following. It can be "user ID", "agent" or "group". If the utterance destination is "agent", the agent 413 generates a response (S18) and outputs it (S19). If the speech destination is a specific "user ID" or "group", the agent 413 determines whether feedback is permitted (S20). If permitted (S20: YES), the agent 413 generates a response (S18) and outputs it (S19). If not, flow returns to step S11.

The agent 413 of the conversation management system 101 may be given feedback even if the output of the speech destination determination unit 411 is not the speech directed to the agent 413 . Examples of the conditions are shown below.

In some cases, the speech destination determination unit 411 returns the "user ID" of the speech destination, and the agent 413 cannot detect the feedback of a specific participant for a predetermined time. In this case, agent 413 may, for example, call the corresponding user ID by voice to prompt that person to answer.

When speech destination determination unit 411 returns that the speech destination is “group”, this group includes agent 413 of conversation management system 101 . Agent 413 may therefore provide feedback based on the knowledge it possesses.

Note that the conversation management system 101 may detect the emotions of the participants as additional information using the feature amount (visual feature amount) of the video data, the audio feature amount, or the text feature amount. This emotion estimation can use neural networks to estimate the emotions of the participants, such as whether the listener understands the speech. Agent 413 may give feedback according to the participant's emotions. For example, if it is determined that the listener does not understand the story, the speaker may be prompted to repeat the explanation.

Details of the processing by the speech destination determination unit 411 will be described below. The utterance destination determination unit 411 determines the utterance destination of the current utterance based on the past utterance history of the participant. This makes it possible to determine the utterance destination more appropriately. As described above, the utterance history management information 423 stores the feature amount information of each speaker's past utterances. The utterance history management information 423 accumulates not only the immediately preceding utterance, but also the history of a plurality of past utterances.

Each participant's utterance history stores, for example, up to k latest input feature amounts. k is a natural number, for example, a natural number of 2 or more. The past feature amount information means k utterance histories, that is, k past input feature amounts collected during conversation from each participant. The extracted speech history is regarded as past feature amount information.

For example, if only text features are considered and there are four participants in the conversation, the collected utterances will look like the sample shown in FIG. 8, for example. The current utterance is the utterance by JAMES. FIG. 8 shows the current spoken text of JAMES. As examples of past spoken texts, the spoken texts of ELENA, ROBY, and agent 413 are shown. Spoken text is a text feature of an utterance. The speech destination determination unit 411 determines the speech destination based on the relationship between the text feature amount of the current speaker's utterance and the text feature amount of the past utterances of other participants.

The speech destination determination unit 411 can generate, for example, the probability that each listener is the speech destination using a neural network-based architecture. FIG. 9 shows an example of inputs and outputs to a neural network that can be used by the speech destination determination unit 411 to determine the speech destination.

Neural network 501 outputs a probability 521 that one listener can be the speaker. The speech destination determination unit 411 can calculate the probability that each listener is the speech destination by executing neural network inference calculation for each listener. Although the configuration of this neural network is not limited, for example, a Transformer, a recurrent neural network, or a convolutional neural network can be used.

As shown in FIG. 9, the inputs of the neural network 501 are an input feature quantity 511 which is utterance information of the speaker, a text feature quantity (Listener Utterance) 512 of the listener's past utterance, and a unique feature quantity 513 of the listener. be. In the configuration example shown in FIG. 9, the utterance destination determination unit 411 determines the utterance destination based on the relationship between the listener's unique feature amount and the speaker's current utterance text feature amount, in addition to the listener's past utterance history. do. The input feature quantity 511, which is the utterance information of the speaker, may include a text feature quantity obtained from the text uttered by the speaker, a voice feature quantity obtained from the speech of the speaker, and a specific feature quantity of the speaker.

Another form for carrying out the present embodiment will be described. FIGS. 13 to 16 show the configuration of a neural network when implementing this embodiment with a recurrent neural network (RNN).

FIG. 13 is an RNN that identifies whether a speaker's utterance destination is a group or an individual. Generic features 1310 consist of information that helps identify a speaker. For example, information indicating who the addressee was in the previous utterance, information indicating whether the uttered text started with a conjunction, etc. can be used. These pieces of information are organized in the form of vectors and used as general features.

The RNN 1320 receives the speaker's speech feature obtained from the speech feature extraction unit 405 . Here, an LSTM (Long short-term memory) is used as the RNN cell.

A detailed description of the processing in LSTM 1320 is shown in FIG. 14A. Acoustic features 1410 of the speaker's speech for each time are input to forward network cell 1430 and backward network cell 1440 of the LSTM. Concatenate the vector of hidden layer values of the final cell of the forward computation and the final cell of the backward computation to obtain the final LSTM output vector 1450 .

Returning to FIG. 13, a layer 1330 performing Logistic Regression calculations reduces the dimensionality of the LSTM 1320 output vector. This logistic regression 1330 may be omitted.

The word embedding layer 1340 converts words included in features expressed in natural language into vectors. The output of word embedding layer 1340 is input to LSTM 1350 . Features expressed in natural language include, first, the text uttered by the speaker. In addition, there are speaker soft skills, hard skills, roles, and names. These are speaker's characteristic features. Note that features other than these may be input as long as they are features expressed in natural language.

The details of inputting words into word embedding layer 1340 and LSTM 1350 are described with reference to FIG. 14B. Each word 1461 of the natural language features is input to the word embedding layer 1462 and converted to a fixed-length vector. The output of word embedding layer 1462 is input to forward network cell 1463 and backward network cell 1464 of the LSTM. Concatenate the vectors of the hidden layer values of the last cell of the forward calculation and the last cell of the backward calculation to obtain the final LSTM output vector 1465 .

Returning to FIG. 13, layer 1360 constructs one multidimensional vector by concatenating the input vectors. The concatenated and generated multidimensional vectors are input to a multi-layer perceptron 1370, which outputs a probability 1380 that the utterance destination is a group.

FIG. 15 is an RNN that determines which listener is the speaking party when the speaking party is an individual. In this embodiment, for each listener, a neural network is configured that calculates the probability that the listener is the speaker. Neural networks 1570 and 1571 calculate the probability that the 1st and i-th listeners are the speaker, respectively. Since the configurations and parameters of the networks inside neural networks 1570 and 1571 are identical, only neural network 1570 will be described here.

The word embedding layer 1510 transforms each word in the natural language features into a finite length vector. In this word embedding layer, the text uttered by the speaker, the text uttered by the listener in the past, and the text representing the unique feature amount of the listener are input. By inputting the words contained in each text into the word embedding layer, we obtain a fixed-length vector for each word.

MaLSTM 1520 is a neural network that uses LSTM to calculate the magnitude of the relationship between two texts based on the Manhattan distance. One MaLSTM takes as input one of the text uttered by the speaker and one of the features that can be expressed by the text about the listener.

FIG. 16 shows the configuration of MaLSTM1520. MaLSTM accepts textual information 1610 about the speaker and textual information 1620 about the listener. The information represented by the text here may be the text of the utterance, or the text of the unique feature amount of the listener and the speaker.

Information on the speaker is represented by a word string 1611 . By inputting this into the word embedding layer, it is converted into a vector 1612 for each word. The vector of words is then input to forward network 1613 and backward network 1614 of the LSTM. The vector of information about the speaker 1615 is then obtained by concatenating the vectors of hidden layer values of the final cells of the forward and backward networks.

Listener information is also processed in the same manner as the speaker information. That is, through the concatenation of the word string 1621, the vector 1622 of words that are word embedding expressions, the LSTM forward network 1623, the LSTM backward network 1624, and the value of the hidden layer of the final cell, a vector 1625 of information about the listener is obtained. .

Information about the speaker and information about the listener are used to calculate the relationship between the two. Here, in layer 1630, we use the following equation to calculate the magnitude of the relationship based on the Manhattan distance.
Ds-l=exp(-||hs-hl||)
Ds-l represents the magnitude of the relationship between the information 1615 of speaker s and the information 1625 of listener l. The result 1640 is output.

Returning to FIG. 15, there are multiple MaLSTM 1520 . All accept the speaker's uttered text, but the other input is different in each MaLSTM. This other input is, for example, the text of the listener's past utterances or the text of the listener's unique feature amount. Layer 1540 concatenates these MaLSTM 1520 outputs to form a single vector.

Note that other feature amounts may be used, for example, focus information 1530 indicating whether the speaker was looking at the corresponding listener may be linked. This focus information can be detected from the speaker's face image or eye image. The vector output by layer 1540 is input to multi-layer perceptron 1550, which outputs probability 1560 that the listener in question is the speaker.

Through the processing of the neural network described above, it is possible to calculate the probability of whether or not the utterance destination is a group, and the probability that the utterance destination is each listener.

This makes it possible to more accurately estimate the listener of the utterance destination. Note that part of the input of the listener's information shown in FIG. 9 may be omitted. For example, the listener's unique feature amount and the speaker's audio feature amount and video feature amount may be omitted.

Speaker input features may include, for example, current text features, audio features, and video features uttered by the speaker. The speaker's input feature amount may include a speaker's unique feature amount. The text feature amount can be obtained from the speech text column 473, the voice feature amount column 475, and the video feature amount column 476 in the current utterance record of the utterance history management information 423, respectively.

The text feature quantity 512 of the listener's past utterance can be acquired from the utterance text column 473 of the utterance history management information 423 . A preset number of listeners' text features, eg, k past utterances, are selected. For example, text features of multiple past utterances are selected. When k past utterance texts are used, a special word (such as <SEP>) is interposed between each utterance text, and a concatenated word string can be used as an input. This makes it possible to more appropriately calculate the destination probability.

The listener's unique feature amount 513 is acquired from the listener's record in the participant's unique feature amount management information 421 . In one embodiment of the present specification, the unique feature amount 513 of the listener referred to for calculating the listener's speech destination probability includes βsoft skill, βhard skill, βrole, and βname.

In one embodiment of the present specification, βsoft skill, βhard skill, βrole, and βname are the words of the speaker's feature quantity input to the neural networks 501, 1340, and 1510, respectively, in each item of the listener's unique feature quantity. , for example, input speech text features or speaker-specific features words.

Furthermore, instead of using the word as it is, the word with the closest relationship to the word can be used. Specifically, βsoft skill, βhard skill, βrole, and βname have the closest relationship to the input word in the listener's "soft skill," "hard skill," "role," and "name," respectively. language.

In one embodiment of the present specification, unique features are selected that have a high semantic relationship to the input word for the speaker. A semantic relationship can be represented by a distance in the semantic space (semantic distance). There are several possible ways to compute the semantic distance.

For example, if we use the Manhattan distance to calculate the semantic relationship, we get the following formula:
Ds-t=exp(-||hs-ht||)
Ds-t is the semantic relationship between speaker s (e.g., the word "machine" in the speaker's text features) and the listener (speaker) t (e.g., the word "machine learning" in the listener's hard skill features). distance (distance score). The speaker and listener feature vectors are passed, for example, to an LSTM network to update hidden states (context features) hs and ht, respectively. The output of the LSTM network indicates the semantic relationship values between the input features.

The output relationships can be schematically shown as in FIG. For example, the word "machine" is related to "AI" in <name>, "machine learning" in <hard skills>, "teamwork" in <soft skills>, and "designer" in <role>. Among the types of features unique to each participant, words closest to the word "machine" are βsoft skill, βhard skill, βrole, and βname.

As described above, the speech destination determination unit 411 calculates the speech destination probability of each listener. In this way, since the probability of each listener is calculated independently of each other, it is possible to calculate the probability of any number of listeners and determine the utterance destination. The speech destination determination unit 411 can determine the listener with the highest probability as the speech destination.

The speech destination determination unit 411 compares the probabilities of all listeners, and determines that the speech destination is a group (all listeners) when a predetermined condition is satisfied. For example, when the difference between the minimum value and the maximum value of the speech destination probability is smaller than a preset threshold, the speech destination determination unit 411 may determine that the speech destination is a group, and the variance is less than the threshold. In some cases, it may be determined that the utterance destination is a group.

It is possible to calculate a semantic relationship score between the speaker's utterance feature amount and the listener's feature amount, and determine the utterance destination based on the score. A semantic relationship score can be represented, for example, by the Manhattan distance. As mentioned above, in conversations with three or more participants, speech may be directed to groups.

The speech destination determination unit 411 can use a single-task or multi-task neural network to estimate whether the speech is "directed to a group" or "directed to an individual." FIG. 11 shows an example of speech destination determination processing by multitasking.

In FIG. 11, first, the speaker's input feature amount (uttered text and unique feature amount) is calculated, and after further calculating the feature amount of each listener (past utterance text and unique feature amount), the speaker's utterance is Determine whether the utterance is for a group or for an individual (first task), and if it is for an individual, determine which listener the utterance was directed to (second task).

In step 601, features of the current speaker are calculated based on the spoken text and unique features. In step 605, by inputting the speaker's features into the neural network, the probability of whether the speaker is speaking to a group or to an individual is calculated. In step 606, if the probability that the addressee is a group is greater than 0.5, proceed to step 610, save the information that the addressee is a group, and terminate the process.

If it is not a group, go to step 611 and loop. Here, the target listener number is represented as i. In the process of step 604, the feature quantity of the i-th listener of interest is calculated. Here, for example, the text uttered by the listener i in the past and the unique feature amount are used.

In the process of step 607, the utterance destination determination unit 411 inputs the feature amount of the speaker and the feature amount of the listener i to the neural network trained for the second task. output the probability that This value is typically in the range 0-1.

Steps 612 and 613 are processing for repeating a loop. That is, the probability that the speaker spoke to all listeners is calculated. Finally, in the process of step 608, the listener with the maximum probability output in step 607 is found, the listener is saved as the utterance destination, and the process ends.

The utterance destination determination unit 411 starts determination of "group" or "individual" as the first task in the multitask neural network (605). A neural network estimates whether the speaker is a group or an individual from vectorized speaker input features. For example, if the neural network output probability (group probability) is higher than 0.5 (606: YES), the speech destination belongs to the group.

If the group probability is 0.5 or less (606: NO), the speech destination is determined to be an individual. Speech target determination unit 411 performs additional estimation by a neural network to obtain probability scores for each listener (607, 608). FIG. 11 shows only the calculation of scores for two listeners as an example.

Vectorized speaker input features obtained for group/individual use are used. The vectorized input features of the speaker are shared by the group/individual decision task and the listener's destination probability calculation task.

The utterance destination determination unit 411 can calculate the probability based on the semantic relationship between the vectorized feature amount of the speaker and the feature amount of the listener. The semantic relationship can be computed using, for example, the Manhattan distance and is output between 0 and 1.

The speech destination determination unit 411 uses the “shared information” of the two tasks. Unlike single-tasking, multi-tasking neural networks perform both a primary task (the task of determining group orientation or individual orientation) and one or more secondary tasks that may be related to the primary task (calculating probability scores for each listener). can be optimized at the same time.

Here, we will use a specific example to explain how to "share" two tasks. In one embodiment herein, a word embedding layer that converts words into word vectors is used in several places (1340 in FIG. 13, 1510 in FIG. 15). Although these are used in different tasks, they can all be learned as word embedding layers with common parameters. This is the first example of sharing.

A second sharing example relates to how to define a loss function used in neural network training. In one embodiment herein, the neural network loss is calculated by the following equation.
Llot = λ1 * LG + λ2 * LID
LG is the loss function for the group/individual decision task, LID is the loss function for the listener probability score calculation task, and Llot is the overall loss function. That is, the neural network is learned while simultaneously considering the properties of both group judgment and individual judgment. λ1 and λ2 are weights chosen to control the influence of each loss function in the neural network.

The speech destination determining unit 411 compares the “probability of being the speech destination” of all the listeners, and regards the speech destination candidate with the highest probability as the true speech destination. The speech destination determination unit 411 determines the participant ID corresponding to the listener as the speech destination ID.

Examples of conversations managed by the conversation management system 101 will be described below. Figures 12A through 12H show an example of a series of utterances that make up a conversation. Conversation management system 101 executes agent 413 and controls agent 413 according to the determined speech destination. In the example of FIGS. 12A-12H, three users 105A-105C are participating in a conversation with agent 413. In the example of FIGS.

Referring to FIG. 12A, user 105A is making an utterance 651 . The utterance destination determination unit 411 determines the utterance destination as “group” and returns it to the agent 413 . This means that anyone, including agent 413, can respond. Agent 413 may wait a predetermined amount of time for the user's utterance before beginning to respond.

Next, referring to FIG. 12B, user 105B utters a response 652 to the immediately preceding utterance 651 of user 105A. The speech destination determination unit 411 determines that the user 105A is the speech destination, and returns the determination result to the agent 413 .

Next, referring to FIG. 12C, user 105A utters a response 653 to the immediately preceding utterance 652 of user 105B. The utterance destination determination unit 411 determines the utterance destination as “group” and returns it to the agent 413 .

Next, referring to FIG. 12D, user 105B utters a response 654 to the immediately preceding utterance 653 of user 105A. The utterance destination determination unit 411 determines the utterance destination as “agent” and returns it to the agent 413 . Agent 413 generates a response based on what is said and what it knows.

Next, referring to FIG. 12E, agent 413 is issuing response 655 to previous utterance 654 of user 105B. The utterance destination determination unit 411 determines the utterance destination as “group” and returns it to the agent 413 .

Next, referring to FIG. 12F, user 105C utters response 656 to agent 413's utterance 655 immediately before. The utterance destination determination unit 411 determines the utterance destination as “group” and returns it to the agent 413 .

Next, referring to FIG. 12G, user 105A utters a response 657 to the previous utterance 656 of user 105C. The utterance destination determination unit 411 determines the utterance destination as “group” and returns it to the agent 413 .

Next, referring to FIG. 12H, agent 413 has uttered response 658 to previous utterance 657 of user 105A. Agent 413 can respond because the last utterance destination is a group. The utterance destination determination unit 411 determines the utterance destination as “group” and returns it to the agent 413 .

As described above, in a multi-user conversation, it is possible to more appropriately support the progress of the multi-user conversation by estimating the utterance destination and controlling the agent based on the estimation result. It should be noted that the speech destination determination result of the present disclosure can be applied to uses other than agent control. For example, the conversation history may include the utterance destination or transmit information indicating it to the user terminal 104 of the utterance destination.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

Further, each of the configurations, functions, processing units, etc. described above may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), or recording media such as IC cards and SD cards.

In addition, the control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In fact, it may be considered that almost all configurations are interconnected.

101 Conversation management system 104 User terminal 105 User 401 Voice recognition unit 403 Participant identification unit 405 Voice feature amount extraction unit 407 Video feature amount extraction unit 411 Speech destination determination unit 413 Agent 421 Participant-specific feature amount management information 423 Speech history management information

Claims (11)

A conversation management system,
one or more storage devices;
one or more computing devices;
The one or more storage devices store past utterance history information of each of a plurality of participants in the conversation,
The one or more computing devices,
obtaining the text features of the first current utterance of the speaker;
Obtaining a text feature amount of past utterances of each listener in a plurality of listeners from the past utterance history information,
calculating the utterance destination probability of each listener based on the text feature of the first current utterance and the text feature of the past utterance of each listener;
A conversation management system, wherein the destination of the first current utterance is determined based on the destination probabilities of the plurality of listeners. The conversation management system of claim 1, comprising:
The one or more storage devices store feature quantities associated with each of the plurality of participants,
The one or more computing devices,
Obtaining from the one or more storage devices a feature associated with each listener among a plurality of listeners;
A conversation management system that calculates the destination probability of each listener based on the feature associated with each listener. The conversation management system of claim 1, comprising:
The conversation management system, wherein the one or more computing devices calculate the utterance destination probability of each listener based on text features of a plurality of past utterances of each listener. The conversation management system of claim 1, comprising:
The conversation management system, wherein the one or more computing devices determine whether the utterance destination is all listeners based on the text feature amount of the first current utterance. The conversation management system of claim 1, comprising:
The one or more computing devices,
calculating a group probability indicating the probability that the utterance destination of the second current utterance is all listeners based on the text feature amount of the second current utterance;
A conversation management system, wherein if the group probability exceeds a threshold, the speech destination of the second current speech is determined to be all listeners. A conversation management system according to claim 5, wherein
The one or more computing devices,
performing a first task and a second task after the first task with a neural network;
the loss function of the neural network includes a loss in the first task and a loss in the second task;
The first task calculates the group probability based on the text features of the second current utterance,
The second task is to calculate a destination probability of each listener based on the text feature amount of the second current utterance used in the first task and the text feature amount of the past utterance of each listener. management system. The conversation management system of claim 1, comprising:
The one or more computing devices,
extracting a speech feature of the first current utterance;
A conversation management system that calculates the speech destination probability of each listener based on the speech feature amount. The conversation management system of claim 1, comprising:
The one or more computing devices,
run the programs of the agents who are participants in the conversation,
A conversation management system that controls the agent based on the determined speech destination. A conversation management system according to claim 4, wherein
The one or more computing devices,
run the programs of the agents who are participants in the conversation,
A conversation management system that permits the agent to respond to a current speech when determining that the speech destination is all of the plurality of listeners or the agent. The conversation management system of claim 1, comprising:
The one or more computing devices,
A conversation management system that determines the speaker based on an utterance position identified based on the audio of the first current utterance. A method for a conversation management system to manage a conversation having multiple participants, comprising:
The conversation management system stores past utterance history information of each of a plurality of participants in the conversation,
The method includes:
wherein the conversation management system obtains text features of a first current utterance of a speaker;
The conversation management system acquires text feature amounts of past utterances of each listener among a plurality of listeners from the past utterance history information,
The conversation management system calculates the utterance destination probability of each listener based on the text feature of the first current utterance and the text feature of the past utterance of each listener,
The method, wherein the conversation management system determines a target of the first current utterance based on the target probabilities of the plurality of listeners.

Images