JP7377900B2 - Dialogue text generation device, dialogue text generation method, and program - Google Patents

Description

The present invention relates to a dialogue text generation device, a dialogue text generation method, and a program.

In the world of natural language processing, which is one of the application fields of AI, automatic text generation is emerging. The quality of text automatically generated by AI in recent years is such that it is difficult to distinguish it from text written by humans. As an application of this technology, progress is being made in the generation of speech that allows dialogue with humans. Conventional models are capable of general conversation, but in order to apply them to business or services, it is necessary to customize (re-learn) the model to suit each area.

Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf, "Large-Scale Transfer Learning for Natural Language Generation", In ACL, 2019, 6053-6058 Yu Cao, Wei Bi, Meng Fang, and Dacheng Tao, D. "Pretrained Language Models for Dialogue Generation with Multiple Input Sources", In EMNLP, 2020, 909-917

However, there are problems with relearning, such as lowering the accuracy of responses to general utterances, the generated utterances sometimes breaking down as a language, and the difficulty of achieving a balance with the model before relearning. In addition, there is a problem in that it is difficult to reflect the context of the utterance (the attributes of the speaker and the content of the utterances so far) in relearning.

The present invention has been made in view of the above, and aims to provide a dialogue text generation technology that combines a model before relearning and a model after relearning in a well-balanced manner to generate utterances that reflect the context of the utterance. purpose.

A dialogue text generation device according to one aspect of the present invention is a dialogue text generation device that generates a response text to an utterance, and inputs a utterance text to which an attribute of a speaker is attached, and generates a previous utterance text and a next utterance text. The word of the next uttered text is extracted from the Transformer, which introduces a multi-view attention mechanism that controls access to other tokens for each type of token, using the word of the previous uttered text and the attribute. Input as an output result, and minimize an objective function representing the prediction accuracy of the attribute when some of the attributes are excluded, and an objective function representing the degree of congruence between the previous utterance text and the next utterance text. A learning section that learns a Transformer, inputs the previous utterance text and the speaker's attributes into the learned Transformer, connects the words recursively output from the Transformer, and generates a response text that becomes the speaker's next utterance. The multi-view attention mechanism includes a generation unit that generates a generator that allows access to all tokens of the attribute, and accesses all tokens of the attribute for the tokens of words of the next utterance text. With a self-attention mask that allows access to tokens of words that occur before this word, when the Transformer seeks attention, it refers to tokens of subsequent words for tokens of words in the next spoken text. Set attention to an infinitely small value to avoid this .

According to the present invention, it is possible to provide a dialogue text generation technique that combines a model before relearning and a model after relearning in a well-balanced manner and generates an utterance that reflects the context of the utterance.

FIG. 1 is a diagram showing an example of the configuration of a dialogue system according to this embodiment. FIG. 2 is a diagram showing an example of a deep learning model proposed in this embodiment. FIG. 3 is a diagram showing an example of a self-attention mask. FIG. 4 is a flowchart showing an example of the flow of learning processing. FIG. 5 is a diagram showing an example of learning data. FIG. 6 is a flowchart showing an example of the flow of dialogue generation processing.

[System configuration]
Embodiments of the present invention will be described below with reference to the drawings.

The dialogue system of this embodiment interprets the generation of an utterance as conditional text generation based on the context of the utterance (utterance history and attributes of the speaker), and uses the utterance text including the previous utterance and the attributes of the speaker ( This system generates the next uttered text according to the conditions when given as a context (attributes of the speaker of the uttered text to be generated). For example, if you input "May I help you?" as the previous utterance text and "Speaker A" as the speaker attribute, the next utterance text by speaker A, such as "Yes, please." generate. Even if the same utterance text is given, if the attribute of the speaker is changed to "Speaker B" and input, different utterance texts will be generated depending on the attributes of the speaker, such as "No, thank you." The attributes of the speaker include, for example, the speaker's identifier, age, occupation, or location. A plurality of attributes of the speaker may be input.

FIG. 1 is a diagram showing an example of the configuration of a dialogue system according to this embodiment. The dialogue system 1 shown in FIG. 1 includes a learning section 10, a generation section 20, a data storage section 30, a calculation result storage section 40, and an input/output section 50. Each part of the dialogue system 1 may be configured by a computer equipped with an arithmetic processing unit, a storage device, etc., and the processing of each part may be executed by a program. This program is stored in a storage device included in the dialog system 1, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be provided through a network.

The learning unit 10 inputs as learning data a series of uttered texts in which each utterance is given an attribute of the speaker, divides two consecutive uttered texts into words, and divides the words of the previous uttered text and the attributes of the speaker into words. is input as a condition, and the model is trained so that attributes can be placed in the same semantic space as words, and the model is trained to generate the next uttered text that corresponds to the previous uttered text.

The generation unit 20 inputs the immediately preceding utterance text and the attributes of the speaker into the trained model, and generates the utterance text that will be the next utterance of the speaker.

The data storage unit 30 holds learning data including a series of spoken texts. Each utterance is given an attribute of the speaker.

The calculation result storage unit 40 holds calculation results such as a deep learning model that generates text, parameters of the deep learning model obtained through learning, and distributed vectors (distributed embedding representations) of attributes and words.

The input/output unit 50 inputs attributes and utterances from the user terminal 5 and transmits them to the generation unit 20, receives the utterance text generated from the generation unit 20, and returns it to the user terminal 5 as a response sentence.

[Proposed model]
The model proposed in this embodiment will be described with reference to FIGS. 2 and 3. The proposed model shown in FIG. 2 is a deep learning model in which a multi-view attention mechanism is introduced into Transformer. The proposed model uses the uttered text that includes the previous utterance as a condition, and when the uttered text (HISTORY) and the speaker's attributes (ATTRIBUTE) are input to the proposed model, the next uttered text (UTTERANCE) based on the input speaker's attributes is calculated. Generate autoregressively. Transformer is a deep learning model mainly used in the field of natural language processing. When Transformer is used as a left-to-right language model, it is possible to probabilistically predict the next word from an input word, and to generate a dext by recursively repeating this process. The multi-view attention mechanism calculates the degree of association (Attention) between words, between attributes, or between words and attributes. Attributes can be referenced in both directions, and words can only be referenced in both the attribute and the previously occurring words in the text. This is a mechanism that allows you to refer to. The multi-view attention mechanism allows the Transformer to share parameters between the attribute side and the word side by having a different self-attention mask for controlling access to the context for each token (attribute and word).

Figure 3 shows an example of a self-attention mask used in the multi-view attention mechanism. In the example of FIG. 3, the self-attention mask is shown with reference source word h, attribute a, and word u arranged in the vertical direction, and reference destination word h, attribute a, and word u arranged in the horizontal direction. h corresponds to a word token in the utterance history, a corresponds to an attribute token, and u corresponds to a word token in the utterance to be generated (next utterance text). Black circles indicate referenceable tokens. The word h is a word that constitutes the immediately preceding utterance text. Word u is a word that constitutes the next utterance text. Regarding word h, since there is only input and no output, words h, u and attribute a are not referenced. Attribute a can refer to all words h and u and attribute a. The word u can refer to all of the word h, all of the attribute a, and only the word u that has appeared up to that point in the next uttered text. The Transformer that introduces the multi-view attention mechanism will be described below.

Attention is a score representing the degree of association between tokens (between attributes, between words, between attributes and words). Each token has a vector of Q (query), K (key), and V (value). Attention is a weighted sum of V, and the weight is calculated using Q and K, as shown in the following equation. In this embodiment, a multi-view attention mechanism is introduced in the attention calculation to control access to different tokens.

W _l ^Q , W _l ^K , W _l ^V ∈R ^d _h ^×d _k are learnable weights for calculating each of Q, K, V∈R ^x×d _k . d is the number of shared dimensions between the query and the key. M∈R ^xxx is the self-attention mask. When obtaining attention, a word token is given an infinitely small value so as not to refer to subsequent (i<j) words. H is a parameter constituting the Transformer, and is expressed by the following equation.

H _a ⁰ is an input to the Transformer, and for each token, a distributed embedding of the attribute or word (Token Embedding), a distributed embedding of the position (Positional Embedding), and a distributed embedding of the data format (Segment Embedding) are input. It is a combination. H _a ^l is the output of the lth layer and is the input to the next layer. Note that [CLS] in FIG. 2 is a token indicating the beginning. [SEP] is a token indicating a break. [SOA] is a token that indicates the beginning of the attribute, and [EOA] is a token that indicates the end of the attribute. [EOT] is a token indicating the end of text.

The proposed model introduced MAN and NUM as learning tasks. A model is trained using MAN so that attributes can be placed in the same semantic space as words. MAM is defined by the following equation.

Here, ζ represents the parameter to be learned. Let the attribute group in the j-th text be a _j = {a _j,1 , ..., a _j,i }, and the word group be w _j = {w _j,1 , ..., w _j,i }. . The word group includes words from the immediately preceding utterance text (HISTORY) and the next utterance text (UTTERANCE). m with a backslash indicates that the m-th attribute is masked. MAM represents the prediction accuracy of an attribute when some of the attributes are excluded, and a model can be learned to correctly estimate the masked attributes using MAM.

NUM is defined by the following equation.

Let the scoring function of consecutive utterances be _sζ (h _t , h _t+1 ). NUM evaluates how closely the previous utterance text and the next utterance text fit together. With NUM, a model can be trained to improve the accuracy of predicting the next uttered text that matches the context.

Model learning is performed by minimizing the following objective function.

_LLM is an objective function for learning the Transformer decoder, and by minimizing _LLM , the prediction accuracy of words generated autoregressively can be improved.

[motion]
Next, the learning process will be explained with reference to the flowchart in FIG.

In step S11, the learning unit 10 reads out a series of utterance texts and attributes from the data storage unit 30. FIG. 5 shows an example of a series of spoken texts. Attributes are assigned to each utterance in a series of utterance texts. In the example of FIG. 5, an identifier identifying the speaker is given. A plurality of attributes may be assigned as attributes of the speaker.

In step S12, the learning unit 10 acquires two consecutive uttered texts and divides each of the two uttered texts into words by morphological analysis. In the example of FIG. 5, the learning unit 10 first obtains "Hi!" from speaker A and "May I help you?" from speaker B, and divides each into words.

In step S13, the learning unit 10 inputs the words of the immediately preceding utterance text, the attributes of the speaker, and the words of the next utterance text into the model, and sets the model parameters so as to minimize the objective function shown above. Update. Specifically, the learning unit 10 inputs into the model the word of the immediately preceding utterance text as HISTORY, the attribute of the next speaker as ATTRIBUTE, and the word of the next utterance text as UTTERANCE. Multiple attributes of the speaker may be input.

In step S14, the learning unit 10 determines whether all spoken texts in the series of spoken texts have been processed. If there is unprocessed utterance text, the learning unit 10 returns to step S12, shifts the utterance text by one, obtains two consecutive utterance texts including the next utterance text, and continues processing. Specifically, in the example of FIG. 5, speaker B's "May I help you?" and speaker A's "Yes, please." are obtained and the process continues.

In this embodiment, by using only the immediately preceding utterance text for the target UTTERANCE, calculation costs can be reduced compared to the conventional technique that uses all the previous utterance texts.

Next, the dialog generation process will be explained with reference to the flowchart in FIG.

In step S21, the generation unit 20 inputs the immediately preceding utterance text and attributes received from the user terminal 5 into the model. The previous utterance text is divided into words and input into the model.

In step S22, the generation unit 20 connects the words recursively output by the model to generate the next uttered text.

In step S23, the generation unit 20 determines whether to end the generation of the spoken text. For example, when generating a continuous series of conversations, the generation unit 20 returns to step S21 and continues processing using the next utterance text generated in step 22 as the immediately preceding utterance text.

When the utterance text generation process by the generation unit 20 is completed, the generated utterance text is returned from the input/output unit 50 to the user terminal 5.

As described above, the dialogue system 1 of this embodiment is a system that includes the learning section 10 and the generation section 20 and generates response text to utterances. The learning unit 10 is a multi-viewer that inputs the utterance text to which the attribute of the speaker is attached, extracts words from the previous utterance text and the next utterance text, and controls access to other tokens for each token type. Input the words and attributes of the previous uttered text as conditions to the Transformer that introduces the tension mechanism, input the words of the next uttered text as the output result, and create an objective function that represents the prediction accuracy of the attributes when some of the attributes are excluded. The Transformer is trained to minimize the objective function representing the degree of congruence between the utterance text of 1 and the next utterance text. The generation unit 20 inputs the immediately preceding utterance text and the attributes of the speaker into a trained Transformer, connects the words recursively output from the Transformer, and generates a response text that will be the next utterance of the speaker. Thereby, it is possible to provide a dialogue system 1 that combines the model before relearning and the model after relearning in a well-balanced manner and generates an utterance that reflects the context of the utterance.

1 Dialogue system 10 Learning unit 20 Generation unit 30 Data storage unit 40 Calculation result storage unit 50 Input/output unit 5 User terminal

Claims (5)

A dialogue text generation device that generates a response text to an utterance,
We introduced a multi-view attention mechanism that inputs uttered text with speaker attributes, extracts words from the previous uttered text and next uttered text, and controls access to other tokens for each token type. Inputting the words of the previous utterance text and the attribute as conditions to the Transformer, inputting the words of the next utterance text as an output result, and excluding a part of the attributes, an objective function representing the prediction accuracy of the attribute, and the a learning unit that learns the Transformer to minimize an objective function representing the degree of engagement between the previous utterance text and the next utterance text;
A generation unit that inputs the immediately preceding utterance text and attributes of the speaker into a trained Transformer, connects words recursively output from the Transformer, and generates a response text that becomes the next utterance of the speaker,
The multi-view attention mechanism allows access to all tokens of the attribute, and provides access to all tokens of the attribute and words that appear before the word in the next spoken text. When the Transformer seeks attention, it sets the attention to an infinitely small value so that it does not refer to subsequent word tokens for the next word token in the spoken text. to be
Dialogue text generator. The dialogue text generation device according to claim 1 ,
The learning unit inputs utterance texts in a series of dialogues and learns by shifting the previous utterance text and the next utterance text one by one. A dialogue text generation method for generating a response text to an utterance, the method comprising:
The computer is
We introduced a multi-view attention mechanism that inputs uttered text with speaker attributes, extracts words from the previous uttered text and next uttered text, and controls access to other tokens for each token type. Inputting the words of the previous utterance text and the attribute as conditions to the Transformer, inputting the words of the next utterance text as an output result, and excluding a part of the attributes, an objective function representing the prediction accuracy of the attribute, and the Learning a Transformer to minimize an objective function representing the degree of engagement between the previous utterance text and the next utterance text,
Input the previous utterance text and the speaker's attributes into a trained Transformer, connect the words recursively output from the Transformer to generate a response text that will be the speaker's next utterance ,
The multi-view attention mechanism allows access to all tokens of the attribute, and provides access to all tokens of the attribute and words that appear before the word in the next spoken text. When the Transformer seeks attention, it sets the attention to an infinitely small value so that it does not refer to subsequent word tokens for the next word token in the spoken text. to be
Dialogue text generation method. The dialog text generation method according to claim 3 ,
A dialogue text generation method that inputs utterance texts in a series of dialogues and learns by shifting the previous utterance text and the next utterance text one by one. A program that causes a computer to operate as each part of the dialog text generation device according to claim 1 or 2 .

Images