CN113704443B - Dialog generation method integrating explicit personalized information and implicit personalized information - Google Patents
Dialog generation method integrating explicit personalized information and implicit personalized information Download PDFInfo
- Publication number
- CN113704443B CN113704443B CN202111051850.0A CN202111051850A CN113704443B CN 113704443 B CN113704443 B CN 113704443B CN 202111051850 A CN202111051850 A CN 202111051850A CN 113704443 B CN113704443 B CN 113704443B
- Authority
- CN
- China
- Prior art keywords
- information
- context
- explicit
- personalized information
- implicit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000009826 distribution Methods 0.000 claims abstract description 30
- 230000007246 mechanism Effects 0.000 claims abstract description 23
- 238000005070 sampling Methods 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 17
- 238000013528 artificial neural network Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 13
- 238000012549 training Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 7
- 230000001419 dependent effect Effects 0.000 claims description 7
- 239000013598 vector Substances 0.000 claims description 7
- 239000000284 extract Substances 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 238000012886 linear function Methods 0.000 claims description 2
- 238000005457 optimization Methods 0.000 claims description 2
- 238000002474 experimental method Methods 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 102100033814 Alanine aminotransferase 2 Human genes 0.000 description 1
- 241000238558 Eucarida Species 0.000 description 1
- 101000779415 Homo sapiens Alanine aminotransferase 2 Proteins 0.000 description 1
- 238000002679 ablation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Probability & Statistics with Applications (AREA)
- Databases & Information Systems (AREA)
- Machine Translation (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a dialogue generating method integrating explicit personalized information and implicit personalized information, which comprises the following steps: 1) Constructing an explicit personalized information extractor, using an encoder in a transformer as a context encoder to encode a context, and then using a personalized-context attention mechanism to encode given personalized information to obtain personalized information related to the context; 2) Constructing an implicit personalized information generator, and abstracting and sampling by using vMF distribution to obtain implicit personalized information; 3) Constructing a personalized information generator, generating by utilizing implicit personalized information, and supervising the implicit personalized information by using given personalized information to ensure that the implicit personalized information is related to the context and the explicit personalized information; 4) And constructing a reply generator, taking the above mentioned explicit personalized information, implicit personalized information and context as input of a decoder, and finally obtaining a corresponding reply. The consistency of individuation in the reply is improved, and the diversity of the reply is improved.
Description
Technical Field
The invention relates to the technical field of natural language processing and dialogue systems, in particular to a dialogue generation method integrating explicit and implicit personalized information.
Background
The artificial intelligence technology has been developed in recent years, and the appearance of many artificial intelligence products brings great convenience to the life of people. Intelligent dialog systems have also begun to penetrate into people's lives, and the advent of intelligent customer service, siri, cortana, google Now, etc. has enabled users to assist in accomplishing a variety of tasks. Such dialog systems belong to the task-type dialog system whose main objective is to solve certain problems within a limited dialog turn, but are mainly suited to fulfill specific tasks and are hardly adaptable to new tasks.
Thus, open domain dialog system [1] And started to draw attention of many researchers. Open domain dialog systems are used primarily for natural and consistent communication with humans under a wide range of chat topics. The dialogue system of the open domain is mainly divided into a search-based method and a generation-based method:
(1) The retrieval-based method mainly comprises the steps of selecting a reply which is most relevant to a context from candidate replies on the premise of giving the context; the method can obtain the reply which accords with grammar and is more standard, but the method can only select the content in the given candidate reply, the obtained reply is more single, and flexible and various replies are difficult to generate;
(2) Based on the method of generation, the replies related to the given context information are generated by the decoder mainly depending on the given context information, and the main framework is a Seq2Seq model [2] Generating model based on nerve variational coder [3] ，transformer [4] Model, GPT2 [5] Pre-training language models, and so forth. The replies generated by the method are more flexible and various.
The above approach mainly considers the diversity of replies, but rarely considers the consistency of replies with speaker personalized information. The personalized information is divided into explicit personalized information and implicit personalized information, wherein the explicit personalized information refers to given description information related to a speaker, and the implicit personalized information refers to information related to the explicit personalized information and inferred from the context.
Existing methods do not consider how to effectively combine these two types of information to promote consistency and diversity of replies.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provide a dialogue generation method integrating explicit and implicit personalized information. The method utilizes an explicit personalized information extractor, an implicit personalized information reasoner, a personalized information generator and a reply generator to finally generate replies fusing the explicit personalized information and the implicit personalized information, and compared with the existing model, the generated results obtain better results on indexes of Distinct-1, BLEU-2 and F1.
The invention aims at realizing the following technical scheme: a dialogue generation method integrating explicit and implicit personalized information comprises the following steps:
(1) Building an explicit personalized information extractor:
an explicit personalization information extractor for extracting context-dependent explicit personalization information O in each dialog of a training corpus cp The method comprises the steps of carrying out a first treatment on the surface of the It obtains the explicit personalized information O related to the context by the attention mechanism aiming at each dialog context and given explicit personalized information cp ；
(2) Constructing an implicit personalized information reasoner:
an implicit personalized information reasoner is used for reasoning the implicit personalized information p related to the context and the explicit personalized information imp The method comprises the steps of carrying out a first treatment on the surface of the It extracts explicit personalized information O cp And context representationAs the input of the reasoner, and distributed by vMF, sampling operation is carried out to obtain implicit personalized information p imp ；
(3) Constructing a personalized information generator:
the personalization generator is used for obtaining the implicit personalization information p imp Regenerated personalized information P; it will reason implicit personalisation information p imp And sending the personalized information to a decoder to obtain regenerated personalized information P.
(4) Building a reply generator:
the reply generator is used for obtaining the fused explicit personalized information O cp Implicit expressionIndividualization information p imp Y is returned to the original state; it extracts explicit personalization information O cp Implicit personalization information p of sum reasoning imp Context representationAnd sending the data to a decoder to obtain a reply y.
Further, in step (1), each dialogue in the training corpus is composed of context, replies and given explicit personalized information; where the context in each session is denoted as x= { X 1 ，x 2 ，...，x m },x i Representing the ith sentence in the context, m representing that the dialog context contains m sentences, wherein each sentence is represented in the form of Represents the jth word, M, in the ith sentence in the context i Representing sentence x i The number of words contained in the list; the replies in each session are denoted +.> Represents the j-th word in the reply, k represents the number of words contained in reply y; given explicit personalized information is denoted as P exp ＝{p 1 ，p 2 ，...，p n }，p i Representing an ith sentence in the given explicit personalization information, n representing that n sentences are contained in the given explicit personalization information, wherein each sentence is represented in the form +.> Representing a given explicit numberThe j-th word, N in the ith sentence in the sexualization information i Representing sentence p i The number of words contained in the word.
Further, in step (1), in order to obtain the context-related explicit personalized information, specific steps are as follows:
the explicit personalized information extractor is composed of a context encoder, a personalized information encoder.
First, a context encoder encodes context information using an encoder in a transform, the encoder of the transform being composed of N c A layer multi-head attention mechanism and a feedforward neural network. It splices m sentences in X to obtain complete context content, and uses them as input of context encoder to obtain representation about contextThe specific calculation formula is as follows:
FFN(x)＝max(O，xW 1 +b 1 )W 2 +b 2
wherein n is E [2, N c ]. MultiHead (·) represents the multi-headed attentiveness mechanism, FFN (·) represents the feedforward neural network. Wherein the method comprises the steps of and />Is the output of the n-layer multi-headed attention mechanism and the feedforward neural network. W (W) 1 ，W 2 ，b 1 ，b 2 Representing parameters in the feed forward neural network. Wherein the first layer->Representing the sum of the word embedding and the position embedding of the input; output of last layerFinally, it is marked->As a contextual representation. A representation O of the reply is similarly available y 。
Second, the personalized information encoder uses the encoder in the transducer to represent the contextExtracting the given explicit personalization information P to obtain the context-dependent explicit personalization information O cp . The encoder herein mainly includes a multi-head attention mechanism and a personalized-contextual attention mechanism, and its specific calculation formula is as follows:
O exp ＝MultiHead(O p ,O p ,O p )
wherein MultiHead (·) represents a multi-headed attentional mechanism, O exp Is the output of the multi-headed attention mechanism. O (O) p Word embedding and location embedding information representing given personalized information; o (O) exp A representation of a given personalized information is represented.
O cp ＝FFN(H cp )
FFN(x)＝max(0,xW 3 +b 3 )W 4 +b 4
Where PerConAtt (·) represents the personalization-contextual awareness mechanism and FFN (·) represents the feedforward neural network.Is a context obtained by a context encoderHereinafter, means O exp Is a representation of a given personalized information; h cp Output representing personalization-contextual attention mechanism, O cp Representing the output of the feedforward neural network, we use it to represent the context-dependent personalization information. W (W) 3 ，W 4 ，b 3 ，b 4 Representing parameters in the feed forward neural network.
Further, in step (2), the context representation is utilizedAnd explicit personalization information O related to context cp Implicit personalized information p through a vMF distribution-based a priori/posterior network imp The method comprises the steps of carrying out a first treatment on the surface of the The method specifically comprises the following steps:
first, vMF distribution, i.e., von Mises-Fisher distribution, is used to represent probability distribution on a unit sphere, and the probability density function is as follows:
wherein ,m represents->Dimension of space, p imp A unit random vector representing m dimensions; />Represents a direction vector on the unit sphere, |μ|=1; kappa.gtoreq.0 represents a concentration parameter; i ρ/2-1 Representing a modified Bessel function; />Finally expressed as a constant; the above distribution indicates the distribution of unit vectors on the sphere;
implicit personalization information p imp Sampling is performed according to the following formula:
wherein ω ε [ -1,1];
the KL loss function of the a priori/a posteriori networks distributed using vMF is as follows:
from the above loss function, two vMF distributions, q φ (p imp |X,Y,P)＝vMF(μ pos ,k pos ) Representing a posterior network for posterior distribution, p θ (p imp |X,P)＝vMF(μ prior ,κ prior ) For the a priori distribution to represent a priori network, KL (vMF (μ) pos ,k pos )||vMF(μ prior ,κ prior ) For calculating a KL divergence between two networks; wherein k is pos ,k prior Is constant, mu pos Is the parameter of posterior distribution, mu prior Is a priori distributed parameter, and the specific calculation process is as follows:
wherein fpos(·) and fprior (. Cndot.) is two linear functions, |cndot| is used for L2 regularization;representation context, O cp Representing explicit personalization information related to context, O y Representing a reply.
Further, in step (3), the personalization generator is configured to sample the obtained implicit personalization information P imp As input, the RNN decoder is used to generate personalization information, whereby the implicit personalization information generated by the supervision is to be consistent with the given personalization information. The specific formula is as follows:
where n represents the number of sentences of a given explicit personalization information; n (N) i Is the length, w, of the ith sentence in a given explicit personalization information P i,j Is the ith sentence in PA representation of the j-th word of (a); />Representing a probability of generating a word; finally, the part is subjected to model optimization by using the following loss function:
the loss function represents a reconstruction error of the personalized information.
Further, in step (4), the reply generator is configured to use the obtained contextContext-dependent explicit personalization information O cp Implicit personalization information p imp The final reply y is sent to the decoder. The decoder is shared with the decoder parameters of the personalization generator so that both can learn additional information during the generation process. The specific formula is as follows:
wherein pvocab The generation probability of the word list is represented; p is p vocab (w y,i ) Representing the generated word w y,i Probability of (2); k represents the length of response Y; x, P, P imp The context, explicit personalization information and implicit personalization information are respectively corresponding. The process is affected by implicit personalization, so the final loss function is in the form of:
wherein ,reconstruction error representing a reply +_>Representing a loss of implicit personalization information.
The loss of the whole process is as follows:
where λ represents a parameter.
Compared with the prior art, the invention has the beneficial effects that:
1. in order to improve consistency of personalization in replies, the present invention utilizes an explicit personalization information extractor in step (1) by combining dialog context information with a given explicitThe personalized information obtains the explicit personalized information O related to the context through a personalized-context attention mechanism cp The method is applied to final reply generation, so that the final reply contains explicit personalized information of a speaker; experiments show that the introduction of the information quantity can effectively improve BLEU-1 and BLEU-2 indexes, and the explicit personalized information extractor can obviously improve the consistency of personalized information about a speaker in reply.
2. In order to promote reply diversity, the invention utilizes an implicit personalized information reasoner in the step (2), and after obtaining the context and the related explicit personalized information, the implicit personalized information is modeled and sampled through vMF distribution, and finally the implicit personalized information p is obtained imp The information is applied to final reply generation, so that the final reply contains relevant implicit personalized information; experiments show that the introduction of the information quantity can effectively improve Distinct-1 index, and the implicit personalized information reasoner can improve reply diversity while utilizing implicit information.
3. To enhance the personalization of information p for implicit imp In step (3), the present invention uses a personalized information generator to control p by a decoder imp Further generation is carried out, so that the generated implicit personalized information is supervised to be related to the given explicit personalized information and has richer content; experiments have shown that the introduction of this item is beneficial for enhancing the control of implicit personalization information.
Drawings
Fig. 1 is a frame diagram of a dialog generating method for fusing explicit and implicit personalized information provided by the present invention:
(a) A model frame; (b) displaying the personalized information extractor; (c) an implicit personalized information reasoner; (d) a personalization-reply generator.
Detailed Description
The invention is described in further detail below with reference to the drawings and the specific examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Personalizing a data set with a personalized dialog data set ConvAI2 [6] The implementation of the invention is given as an example. The overall framework of the method is shown in fig. 1. The whole system algorithm flow comprises 5 steps of ConvAI2 data set processing, explicit personalized information extraction, implicit personalized information reasoning, personalized information generation and reply generation.
The method comprises the following specific steps:
(1) Processing of ConvAI2 personalized data set:
ConvAI2 personalized data set is PERSONA-CHAT [6] An extension of the dataset comprising 10981 dialogs for a total of 164356 sentences; wherein the respondents of each conversation contain a description of at least 4 given personalized information. And correspondingly processing the dialogue data set to obtain the context, reply and personalized information. And the whole data set is divided to obtain a training set containing 67112 dialogs, a verification set 8395 dialogs and a test set 4478 dialogs respectively.
(2) Explicit personalized information extraction:
to obtain explicit personalisation information O in each dialogue relating to the replier cp We use an explicit personalized information extractor to get the corresponding information, first calculate the context representation:
FFN(x)＝max(0 ， xW 1 +b 1 )W 2 +b 2
O exp ＝MultiHead(O p ，O p ，O p )
the meaning of the symbols in the formula is as described above.For output of the nth layer, the process includes N c A layer, wherein the last layer is marked +.>As a contextual representation.
O cp ＝FFN(H cp )
FFN(x)＝max(0，xW 3 +b 3 )W 4 +b 4
The meaning of the symbols in the formula is as described above. O (O) cp For the resulting explicit personalization information related to the context.
(3) Implicit personalized information reasoning:
o obtained according to the previous step cp Andimplicit personalized information reasoning is carried out, and O is distributed by vMF cp and />As input, implicit personalization information p imp As a potential variable, training is performed using the objective function:
wherein ,qφ (p imp The |X, Y, P) is a posterior distribution used to represent a posterior network, specifically denoted as q φ (p imp |X，Y，P)＝vMF(μ pos ，k pos )；p θ (p imp The i X, P) is a priori distribution used to represent a priori network, specifically P θ (p imp |X，P)＝vMF(μ prior ，κ prior )；KL(q φ (p imp |X，Y，P)||p θ (p imp I X, P)) is used to calculate KL divergence between the identification network and the a priori network; wherein k is pos ，k prior Constant, mu pos Is the parameter of posterior distribution, mu prior Is a parameter of the a priori distribution.
Finally, implicit personalized information p is obtained imp 。
(4) Personalized information generation:
in order to play a role in supervising the implicit personalized information, the personalized information generator is utilized to generate the description information for the replier, and the specific generation process is as follows:
the process is trained using the following objective function:
the meaning of the symbols in the formula is as described above.
(5) Reply generation:
in order to obtain the final output we require the use of the implicit personalization information p obtained above imp Context X, explicit personalization information P related to the context is taken as input, and is generated by the decoder as follows:
and training the generation process by using the following objective function:
representing reconstruction errors, p θ (Y|p imp X, P) represents the probability of generation of reply Y, P vocab (w y,i ) Representing the vocabulary distribution in the response. The meaning of the symbols in the formula is as described above.
The above equation represents the training goal of the entire model.
In a specific implementation, we set various parameters in advance for the model, for the encoder structure in the transducer, the dimension of the word vector is 512 and initialized randomly, the context encoder layer number N c Set to 3, the number of layers of the personalized encoder is 1. Wherein the head number of the multi-head attention mechanism is set to 8, and the size of the feedforward neural network is set to 2048. The dimension of the implicit personalization information is set to 180. The Batch size is set to 32. We updated the parameters using Adam's algorithm at an initial learning rate of 0.0001, in training we employed early-stop strategy, and the experiment was performed with one GPU.
Table 2 shows the present model (EIPD), a simplified version of the present model (D, IPD, EPD, EIPD Gau 、EIPD pd ) Results of other models (S2 SAP, CVAE, trans, perCVAE, transferTransfo) on ConvAI2 dataset and four evaluation indicators (Dist-1, BLEU-2, F1).
TABLE 2-1 automatic evaluation results of ConvAI2 dataset
Table 2-2 model ablation performance on ConvAI2 dialogue dataset
The comparative experimental algorithm in the table is described below:
s2SA: a standard seq2seq model with attention mechanisms;
CVAE: an RNN model that promotes reply diversity using latent variables;
trans: the converter model takes the dialogue context and the explicit personalized information as input;
PerCVAE: a memory-enhanced CVAE model that utilizes personalized information using a multi-hop attention mechanism;
TransferTransfo: a codec network comprising a hierarchy and a variable memory;
D、IPD、EPD、EIPD Gau 、EIPD pd : is 5 degradation models which we propose;
remarks: the method provided by the invention is EIPD, gau shows Gaussian distribution, vMF shows vMF distribution, which are all distribution representations in potential space; thereby generating a series of degradation models for EIPD. As can be seen from the experimental results in table 2, the effect of the dialogue generating method on diversity and consistency can be improved by extracting explicit personalized information, reasoning implicit personalized information and combining the implicit personalized information into the decoding process.
Table 3 results of manual evaluation of ConvAI2 dialogue dataset
Table 3 shows the results of the present model (EIPD), as well as other models (S2 SAP, CVAE, trans, perCVAE, transferTransfo), manually evaluating the results on the ConvAI2 dataset.
The present implementation invites 3 human annotators to determine the quality of the generated replies. We present them with 200 contexts, each model generating a corresponding reply. Each reply is rated according to the following criteria: G1. the generated replies are grammatically incorrect, independent of the context semantics, or inconsistent with the given personalized information; G2. the generated replies are smooth, but have weak relevance to the context, such as some general replies; G3. the generated replies are smooth, with contextSemantic correlation and slight agreement with personalized information; G4. the generated replies are not only smooth and semantically related, but also consistent with a given role. Consistency between annotators with Fleiss' kappa [7] The result of the kappa calculation was calculated to be greater than 0.4.
Table 4 shows examples of the methods described above:
table 4 example of generation on ConvAI2 dialogue dataset
As seen in the specific example of Table 4, the Baseline model often produces some fluent but irrelevant, or inadequately personalized replies. For comparison, we generated 2 replies by implicit personalized information reasoning using EIPD, the first reply being related to the personalized information "My favorite hobby is garding"; the second reply represents an extension to the personalized information, exploring the cause. Compared with the dialog generation method proposed before, the reply generated by the method is closer to the personal characteristics of the replier in the result, and the generated reply is more diversified and natural.
The invention is not limited to the embodiments described above. The above description of specific embodiments is intended to describe and illustrate the technical aspects of the present invention, and is intended to be illustrative only and not limiting. Numerous specific modifications can be made by those skilled in the art without departing from the spirit of the invention and scope of the claims, which are within the scope of the invention.
Reference is made to:
[1]Hongshen Chen,Xiaorui Liu,Dawei Yin,and Jiliang Tang.2017.A survey on dialogue systems:Recent advances and new frontiers.Acm Sigkdd Explorations Newsletter,19(2):25–35.
[2]Ilya Sutskever,Oriol Vinyals,and Quoc V Le.2014.Sequence to sequence learning with neural networks.In Advances in Neural Information Processing Systems 27(NIPS),pages 3104–3112.
[3]Tiancheng Zhao,Ran Zhao,and Maxine Eskenazi.2017.Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(ACL),pages 654–664.
[4]Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N.Gomez,Lukasz Kaiser,and Illia Polosukhin.2017.Attention is all you need.In Advances in Neural Information Processing Systems 30(NIPS),pages 5998–6008.
[5]Thomas Wolf,Victor Sanh,Julien Chaumond,and Clement Delangue.2019.Transfertransfo:A transfer learning approach for neural network based conversational agents.CoRR,abs/1901.08149.
[6]Saizheng Zhang,Emily Dinan,Jack Urbanek,Arthur Szlam,Douwe Kiela,and Jason Weston.2018.Personalizing dialogue agents:I have a dog,do you have pets tooIn ACL.
[7]Fleiss,J.L.,Cohen,J.:The equivalence of weighted kappa and the intra class correlation coecient as measures of reliability.Educational and psychological mea-surement.33(3),613-619(1973).
Claims (6)
1. a dialog generation method incorporating explicit and implicit personalized information, comprising the steps of:
(1) Building an explicit personalized information extractor:
an explicit personalization information extractor for extracting context-dependent explicit personalization information O in each dialog of a training corpus cp The method comprises the steps of carrying out a first treatment on the surface of the For each dialog context and given explicit personalization information, explicit personalization information O relevant to the context is derived by means of an attention mechanism cp ；
(2) Constructing an implicit personalized information reasoner:
implicit personalized information reasoner for reasoning about implicit personalizations related to context and explicit personalized informationInformation p imp The method comprises the steps of carrying out a first treatment on the surface of the It extracts explicit personalized information O cp And context representationAs the input of the reasoner, and distributed by vMF, sampling operation is carried out to obtain implicit personalized information p imp ；
(3) Constructing a personalized information generator:
the personalization generator is used for obtaining the implicit personalization information p imp Regenerated personalized information P; it will reason implicit personalisation information p imp Sending the information to a decoder to obtain regenerated personalized information P, and supervising the generated implicit personalized information;
(4) Building a reply generator:
the reply generator is used for obtaining the fused explicit personalized information O cp And implicit personalization information p imp Y is returned to the original state; it extracts explicit personalization information O cp Implicit personalization information p of sum reasoning imp Context representationAnd sending the data to a decoder to obtain a reply y.
2. The method of claim 1, wherein in step (1), each dialogue in the training corpus consists of context, replies and given explicit personalized information;
where the context in each session is denoted as x= { X 1 ，x 2 ，...，x m }，x i Representing the ith sentence in the context, m representing that the dialog context contains m sentences, wherein each sentence is represented in the form of Represents the jth word, M, in the ith sentence in the context i Representing sentence x i The number of words contained in the list;
the replies in each session are expressed as Represents the j-th word in the reply, k represents the number of words contained in reply y;
given explicit personalized information is denoted as P exp ＝{p 1 ，p 2 ，...，p n }，p i Representing an ith sentence in the given explicit personalization information, n representing that the given explicit personalization information contains n sentences, wherein each sentence is represented in the form of Representing the jth word, N, in the ith sentence in a given explicit personalized information i Representing sentence p i The number of words contained in the word.
3. The dialog generation method of claim 2, wherein in step (1), the specific steps are as follows:
the explicit personalized information extractor is composed of a context encoder and a personalized information encoder;
first, a context encoder encodes context information using an encoder in a transform, the encoder of the transform being composed of N c A layer multi-head attention mechanism and a feedforward neural network;
the context encoder splices m sentences in X to obtain complete context content, and takes the complete context content as context encodingThe input of the encoder gets a representation about the contextThe specific calculation formula is as follows:
FFN(x)＝max(0，xW 1 +b 1 )W 2 +b 2
wherein n is E [2, N c ]MultiHead (·) represents a multi-headed attentiveness mechanism, FFN (·) represents a feedforward neural network; wherein the method comprises the steps of and />Is the output of the n-layer multi-head attention mechanism and the feedforward neural network; w (W) 1 ，W 2 ，b 1 ，b 2 Representing parameters in the feed-forward neural network; wherein the first layer->Representing the sum of the word embedding and the position embedding of the input; output of last layer->Finally, it is marked->As a contextual representation;
a representation O of the reply is similarly available y ；
Second, the personalized information encoder uses the encoder in the transducer to represent the contextAnd given explicit personalization information P exp Extraction to obtain explicit personalized information O related to context cp The method comprises the steps of carrying out a first treatment on the surface of the The encoder herein mainly includes a multi-head attention mechanism and a personalized-contextual attention mechanism, and its specific calculation formula is as follows:
O exp ＝MultiHead(O p ，O p ，O p )
O p word embedding and location embedding information, O, representing given display personalization information exp Representing a representation of a given display personalization;
O cp ＝FFN(H cp )
FFN(x)＝max(0，xW 3 +b 3 )W 4 +b 4
wherein PerConAtt (·) represents the personalization-contextual awareness mechanism, FFN (·) represents the feedforward neural network; h cp Output representing personalization-contextual attention mechanism, O cp Representing the output of the feedforward neural network, which is used to represent the context-dependent personalization information; w (W) 3 ，W 4 ，b 3 ，b 4 Representing parameters in the feed forward neural network.
4. The dialog generation method of claim 1 wherein in step (2) context representation is usedAnd explicit personalization information O related to context cp Implicit individuals are obtained through a vMF distribution-based a priori/posterior networkSex information P imp ；
The method specifically comprises the following steps:
first, vMF distribution, i.e., von Mises-Fisher distribution, is used to represent probability distribution on a unit sphere, and the probability density function is as follows:
wherein ,m represents->Dimension of space, p imp A unit random vector representing m dimensions; />Mu represents a direction vector on the unit sphere, mu |=1; kappa.gtoreq.0 represents a concentration parameter; i m/2-1 Representing a modified Bessel function; />Finally expressed as a constant; the above distribution indicates the distribution of unit vectors on the sphere;
implicit personalization information p imp Sampling is performed according to the following formula:
wherein ω is [ -1,1]
The KL loss function of the a priori/a posteriori networks distributed using vMF is as follows:
from the above loss function, two vMF distributions, q φ (p imp |X，Y，P)＝vMF(μ pos ，k pos ) Representing a posterior network for posterior distribution, p θ (p imp |X，P)＝vMF(μ prior ，κ prior ) For the a priori distribution to represent a priori network, KL (vMF (μ) pos ，k pos )||vMF(μ prior ，κ prior ) For calculating a KL divergence between two networks; wherein k is pos ，k prior Is constant, mu pos Is the parameter of posterior distribution, mu prior Is a priori distributed parameter, and the specific calculation process is as follows:
wherein fpos(·) and fprior (. Cndot.) is two linear functions, |cndot| is used for L2 regularization, O y Representing a reply.
5. The dialog generation method of claim 1, wherein in step (3), the personalizing is performed by fusing explicit and implicit personalizing informationThe generator is used for obtaining implicit personalized information P obtained by sampling imp As input, generating personalization information using the RNN decoder, whereby the supervision generated implicit personalization information is to be in line with the given personalization information;
the specific formula is as follows:
n represents the number of sentences of a given explicit personalization information; n (N) i Is the length of the ith sentence in the given explicit personalization information P; w (w) i，j Is the ith sentence in PA representation of the j-th word of (a); />Representing a probability of generating a word; the final personalization generator performs model optimization using the following loss functions:
the loss function represents a reconstruction error of the personalized information.
6. The dialog generation method of claim 1, wherein in step (4), the reply generator is configured to use the obtained contextContext-dependent explicit personalization information O cp Implicit personalization information p imp Sending the final reply y into a decoder;
the decoder is shared with the decoder parameters of the personalization generator so that both can learn additional information during the generation process;
the specific formula is as follows:
wherein pvocab The generation probability of the word list is represented; p is p vocab (w y，i ) Representing the generated word w y，i Probability of (2); k represents the length of reply y;
the process is affected by implicit personalization, so the final loss function is in the form of:
wherein ,representing a recovered reconstruction error; the loss of the whole process is as follows:
where λ represents a parameter.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111051850.0A CN113704443B (en) | 2021-09-08 | 2021-09-08 | Dialog generation method integrating explicit personalized information and implicit personalized information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111051850.0A CN113704443B (en) | 2021-09-08 | 2021-09-08 | Dialog generation method integrating explicit personalized information and implicit personalized information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113704443A CN113704443A (en) | 2021-11-26 |
| CN113704443B true CN113704443B (en) | 2023-10-13 |
Family
ID=78659387
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111051850.0A Active CN113704443B (en) | 2021-09-08 | 2021-09-08 | Dialog generation method integrating explicit personalized information and implicit personalized information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113704443B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108763504A (en) * | 2018-05-30 | 2018-11-06 | 浙江大学 | It is a kind of that generation method and system are replied based on the dialogue for strengthening binary channels Sequence Learning |
| CN111209366A (en) * | 2019-10-10 | 2020-05-29 | 天津大学 | Implicit discourse relation identification method based on TransS-driven mutual excitation neural network |
| WO2020143137A1 (en) * | 2019-01-07 | 2020-07-16 | 北京大学深圳研究生院 | Multi-step self-attention cross-media retrieval method based on restricted text space and system |
| CN112100350A (en) * | 2020-09-11 | 2020-12-18 | 湖南大学 | Open domain dialogue model and method for enhancing reply personalized expression |
| WO2021139297A1 (en) * | 2020-07-28 | 2021-07-15 | 平安科技（深圳）有限公司 | Question-answer method and question-answer apparatus based on transformer model, and storage apparatus |
-
2021
- 2021-09-08 CN CN202111051850.0A patent/CN113704443B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108763504A (en) * | 2018-05-30 | 2018-11-06 | 浙江大学 | It is a kind of that generation method and system are replied based on the dialogue for strengthening binary channels Sequence Learning |
| WO2020143137A1 (en) * | 2019-01-07 | 2020-07-16 | 北京大学深圳研究生院 | Multi-step self-attention cross-media retrieval method based on restricted text space and system |
| CN111209366A (en) * | 2019-10-10 | 2020-05-29 | 天津大学 | Implicit discourse relation identification method based on TransS-driven mutual excitation neural network |
| WO2021139297A1 (en) * | 2020-07-28 | 2021-07-15 | 平安科技（深圳）有限公司 | Question-answer method and question-answer apparatus based on transformer model, and storage apparatus |
| CN112100350A (en) * | 2020-09-11 | 2020-12-18 | 湖南大学 | Open domain dialogue model and method for enhancing reply personalized expression |
Non-Patent Citations (2)
| Title |
|---|
| 基于多任务学习的中文事件抽取联合模型;贺瑞芳;段绍杨;;软件学报(第04期);全文 * |
| 基于语境交互感知和模式筛选的隐式篇章关系识别;郭凤羽;贺瑞芳;党建武;;计算机学报(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113704443A (en) | 2021-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111444709B (en) | Text classification method, device, storage medium and equipment | |
| Shou et al. | Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis | |
| CN110297887B (en) | Service robot personalized dialogue system and method based on cloud platform | |
| Bilquise et al. | Emotionally intelligent chatbots: A systematic literature review | |
| Wen et al. | Dynamic interactive multiview memory network for emotion recognition in conversation | |
| CN111159368A (en) | Reply generation method for personalized dialogue | |
| CN113435211B (en) | Text implicit emotion analysis method combined with external knowledge | |
| Hung et al. | Towards a method for evaluating naturalness in conversational dialog systems | |
| Wilks et al. | A prototype for a conversational companion for reminiscing about images | |
| Needle et al. | Gendered associations of English morphology | |
| Wang et al. | Information-enhanced hierarchical self-attention network for multiturn dialog generation | |
| Aattouri et al. | Modeling of an artificial intelligence based enterprise callbot with natural language processing and machine learning algorithms | |
| Wu et al. | Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis | |
| Mathur et al. | A scaled‐down neural conversational model for chatbots | |
| CN116955562A (en) | Intelligent customer service system based on artificial intelligence technology | |
| CN113704443B (en) | Dialog generation method integrating explicit personalized information and implicit personalized information | |
| Chopde et al. | Chatbot using deep learning | |
| Bai et al. | Low-rank multimodal fusion algorithm based on context modeling | |
| CN112052320B (en) | Information processing method, device and computer readable storage medium | |
| Bhagchandani et al. | Deep learning based chatbot framework for mental health therapy | |
| Kumar et al. | Pesubot: An empathetic goal oriented chatbot | |
| Ai et al. | A Two-Stage Multimodal Emotion Recognition Model Based on Graph Contrastive Learning | |
| Tracey et al. | Applying NLP to build a cold reading chatbot | |
| YIN | A compression-based BiLSTM for treating teenagers’ depression chatbot | |
| Thomas et al. | Seq2seq and Legacy techniques enabled Chatbot with Voice assistance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |