CN116432663B - Controllable diversity professional text generation method and system based on element diagram - Google Patents

Controllable diversity professional text generation method and system based on element diagram Download PDF

Info

Publication number
CN116432663B
CN116432663B CN202310685875.9A CN202310685875A CN116432663B CN 116432663 B CN116432663 B CN 116432663B CN 202310685875 A CN202310685875 A CN 202310685875A CN 116432663 B CN116432663 B CN 116432663B
Authority
CN
China
Prior art keywords
text
encoder
professional
category
semantic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310685875.9A
Other languages
Chinese (zh)
Other versions
CN116432663A (en
Inventor
孙宇清
王舰
马磊
袁峰
邢金宝
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SHANDONG SHANDA OUMA SOFTWARE CO Ltd
Original Assignee
SHANDONG SHANDA OUMA SOFTWARE CO Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SHANDONG SHANDA OUMA SOFTWARE CO Ltd filed Critical SHANDONG SHANDA OUMA SOFTWARE CO Ltd
Priority to CN202310685875.9A priority Critical patent/CN116432663B/en
Publication of CN116432663A publication Critical patent/CN116432663A/en
Application granted granted Critical
Publication of CN116432663B publication Critical patent/CN116432663B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/205—Parsing
    • G06F40/216—Parsing using statistical methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Machine Translation (AREA)

Abstract

The invention discloses a controllable diversity professional text generation method and system based on an element diagram, and belongs to the technical field of natural language processing. According to the invention, the relation between the views and the semantic features is established by constructing a condition encoder, so that the control of the views on the text semantics is realized, and the diversity of text expression is realized by randomly sampling different expression features. The introduction of the element sketch enables the invention to directly and definitely model the association relation between the viewpoint and the text semantic, and as one element sketch can represent the semantic of a plurality of professional texts of the same viewpoint, the text semantic can be strongly controlled by the viewpoint by utilizing a small amount of data training model; the text generation process integrating the semantic features and the expression features realizes the generation of diversified texts under semantic constraint, and prevents the generated texts from only pursuing diversity and neglecting semantics; the invention generates the dependent element sketch while generating the professional text, thereby realizing the interpretability of the generation process.

Description

Controllable diversity professional text generation method and system based on element diagram
Technical Field
The invention discloses a controllable diversity professional text generation method and system based on an element diagram, and belongs to the technical field of natural language processing.
Background
Controllable text generation is to generate text meeting the constraint of a control variable under the condition of the given control variable, and is a leading edge problem in the field of natural language processing. The professional text refers to a text for expressing individual views according to knowledge points and background descriptions in a specific field, and due to cognitive differences, views in the professional text are different, and expression forms are also greatly different:
for example, "the nominal stakeholder Li Mou does not agree to the mortgage equity by the actual sponsor Zhang Mou to cause loss," Li Mou claims Zhang Mou "or" Li Mou is the nominal stakeholder, has the right to mortgage equity to the first company, does not require compensation, "and so on. For specific knowledge points and background descriptions, professional texts can be divided into a limited number of categories according to the viewpoint difference, in the above example, the corresponding knowledge points are "the loss of actual sponsor caused by the equity of the nominal stakeholder, the actual sponsor requests the nominal stakeholder to undertake compensation responsibility, the people's court shall be supported", the background descriptions are "Li Mou is the nominal stakeholder, zhang Mou actually pays and enjoys investment rights, the loss is caused by the mortgage of the equity of the first company without Zhang Mou agreeing Li Mou", the viewpoints are classified as "compensation" or "compensation not needed", and the like.
From the above, the semantics of professional text are determined by a series of concepts or entities and their associations, which we define as viewpoint elements. The method is characterized in that the viewpoint-controllable diversified professional texts are generated by taking the viewpoints as control variables, the generated professional texts with semantics conforming to a given viewpoint and various expressions are generated, the generated texts not only relate to the logic relations of a plurality of viewpoint elements, but also reflect different changes and slight differences of the viewpoint element complex relations corresponding to different viewpoints, and meanwhile, the various language expression modes are reflected, and the factors make the viewpoint-controllable professional texts have very difficult task generation and important theoretical value. Meanwhile, in the intelligent era of wide application of the deep learning method, the problem of data scarcity is outstanding, so that the diversity text generation technology with controllable views has more application value.
For this reason, the following patent documents are disclosed in the prior art:
chinese patent document CN114297382A proposes a controllable text generation method based on the fine adjustment of parameters of a generated pre-training model, firstly, the word embedded layer of the generated pre-training model is subjected to self-defined condition coding, then, a control text is used as a prompt to carry out fine adjustment of parameters, and the parameters of the control text are updated.
Chinese patent document CN114510924a proposes a text generation method based on a pre-trained language model, in which a controller is introduced in a model decoding stage to perform vocabulary level control, so as to control a text generation result.
The two methods can control some simple attributes such as emotion and theme, but are difficult to perceive and control fine granularity difference among viewpoint elements, and the generated results lack diversity.
Chinese patent document CN113254604a discloses a professional text generation method based on a reference specification, the controllable method adopts a plurality of generator structures, and trains one generator for one control variable individually, and each generator generates a text conforming to one type of control variable. The model structure of the method is complex, and great calculation and time cost exists in the training stage.
In addition, the diversity of languages makes the text space huge, the methods respectively described in the three documents are generally difficult to realize effective training of the model under smaller labeling data, and the text generation process lacks interpretability, so that the use scene is limited.
Another requirement for view-controllable professional text generation tasks: the diversity of text narrative forms is also referred to in the following documents:
Chinese patent document CN111339749a discloses an unconditional text generation method, which is implemented by filtering sentences with large differences between true diversity text through a pre-trained filter.
Chinese patent document CN111597779a discloses a text generation method, apparatus, device and storage medium, which enhance text diversity by increasing the randomness of words.
The generation process of the professional text is a process of jointly participating in the expression modes of the internal semantics and the external manifestations determined by the viewpoint category, and the method does not consider the influence of the semantics on the generation result and can not simultaneously meet the generation targets of viewpoint controllability and expression diversity.
Disclosure of Invention
Aiming at the defects of the prior art, the invention discloses a controllable diversity professional text generation method based on an element diagram.
The invention also discloses a system for realizing the generation method.
Summary of The Invention
The invention introduces the element sketch as abstract description of viewpoint semantics in the professional text, thereby being capable of expressing long-distance dependency relationship among viewpoint elements. Based on the ideas that the viewpoint category decides the text semantic and the language expression decides the text narration mode, the invention provides a model based on a variational self-encoder structure, and the text semantic and the language expression are decoupled in an implicit space, which comprises two stages of model training and model use:
In the training stage, a graph variation self-encoder is used for encoding and reconstructing element diagrams to obtain a semantic hidden variable, and in the process, the control of the viewpoint category on the semantic hidden variable is realized through technologies such as reconstruction loss, contrast learning and the like; encoding and reconstructing the professional text by using a text variation self-encoder to obtain an expression hidden variable so as to reflect the distribution rule of text expression characteristics in the data set, and generating a final professional text by combining the expression hidden variable with a semantic hidden variable; a multi-angle discriminator is introduced to evaluate and generate text language fluency, field correlation and consistency with viewpoint categories, and feedback information is formed to guide update of two kinds of variation from the encoder.
In the using stage, the semantic hidden variables are controlled by view categories, the semantic hidden variables are combined with randomly sampled expression hidden variables, the controllability and the expression diversity of the text view are realized, and meanwhile, an element diagram is generated by the semantic hidden variables and is used as an interpretable basis of a generation result. The introduction of the element sketch promotes the constraint capability of the viewpoint category on the text semantics. Meanwhile, element sketches are generated in the text generation process, so that the model interpretability is improved.
Technical term interpretation
Background text: in this disclosure, a section of the text describing a fact event and its resulting impact is referred to.
Professional text: in the present invention, text of individual cognition and opinion of facts is expressed with respect to background text.
View category: in the present invention, background text and views are included for controlling variables that generate text.
Elements: the invention refers to a core concept or entity expressing semantics in professional text.
Element diagram: in the invention, a directed graph describing elements and association relations thereof in a professional text is defined, wherein vertexes are elements, and edges represent the association between the elements. For example, in one element diagram describing the relationship of a judicial person and a equity, the vertices include legal holders, actual holders, stakeholders, etc., and the relationships include investment, equity substitution, loss-causing relationships, etc.
Expressing hidden variables: the present invention refers to a variable that can determine the text expression.
Semantic hidden variables: the invention refers to a variable capable of determining text semantics.
The technical scheme of the invention is as follows:
the controllable diversity professional text generation method based on the element diagram is characterized by comprising the following steps of:
creating a variational self-encoder framework model, as in fig. 1, includes: a graph variant self-encoder, a text variant self-encoder and a plurality of discriminators; wherein the graph variation self-encoder comprises a graph encoder, a graph encoder and a condition encoder; the text variation self-encoder comprises a text encoder and a text decoder; the discriminant comprises a category consistency discriminant, a field relevance discriminant and a language model;
Processing specialized text for a given point of view category using a variational self-encoder framework model:
firstly, obtaining an element diagram corresponding to a professional text;
secondly, encoding the element sketch and the viewpoint category by using a graph encoder to obtain posterior distribution of semantic hidden variables; the method comprises the steps of using a conditional coder to code viewpoint categories to obtain semantic hidden variable prior distribution, using a text coder to obtain expression hidden variable posterior distribution, and simultaneously assuming the expression hidden variable prior distribution to be standard normal distribution;
training phase: sampling semantic hidden variables and expression hidden variables from posterior distribution of the semantic hidden variables and posterior distribution of the expression hidden variables obtained by encoding, splicing the semantic hidden variables and the expression hidden variables, and inputting the semantic hidden variables and the expression hidden variables into a text decoder to reconstruct an input text; simultaneously, inputting the semantic hidden variable into a graphic code reconstruction element diagram;
the training targets of the training phase include: on one hand, the KL divergence (Kullback-Leibler Divergence) of the prior distribution and the posterior distribution corresponding to semantic and expression hidden variables is minimized, and on the other hand, the probability of reconstructing input text and element sketches is maximized; in order to enhance the control of the generated result, introducing the discriminator to evaluate the smoothness, the field correlation and the consistency of the same-view category of the generated text;
The application stage comprises the following steps: given a viewpoint category, respectively sampling a semantic hidden variable and an expression hidden variable from the semantic hidden variable prior distribution and the expression hidden variable prior distribution, and inputting the semantic hidden variable and the expression hidden variable into a text of which the text decoder accords with the viewpoint category; meanwhile, the sampled semantic hidden variable input diagram decoder generates an element diagram which is used as an explanatory basis for generating a text.
According to the invention, the controllable diversity professional text generation method based on the element diagram comprises the following steps:
step S1: obtaining an element diagram corresponding to the professional text:
constructing an element set and a relation set aiming at all obtained professional text corpus, and acquiring an initial feature matrix of the element set and the relation set;
the method for constructing the element set and the relation set comprises the following steps: extracting elements existing in the corpus by combining word frequency statistical features in the professional text corpus and word segmentation tools such as 'bargain word segmentation', and forming element setsWherein said->Is an element, said->The number of elements in the element set; expert-defined relation set->Wherein, said->Is an element in a relational set, the KIs the number of relationships in the set of relationships;
the method for acquiring the initial feature matrix of the element set and the relation set comprises the following steps:
feature vectors of elements and relations are obtained through a Chinese pre-training word vector tool package ngram2vec, and then feature matrixes of an initial element set and a relation set are formed:
，/>
wherein ,is an element feature matrix; />Is a relation characteristic matrix; />For the number of elements in the set of elements,the dimension of the element feature vector; />Is the number of relationships in the set of relationships; />The dimension of the relation feature vector;
the method for acquiring the element sketch corresponding to the professional text comprises the following steps:
for all professional texts in the corpus, matching element setsObtaining the elements contained in the professional text;
manually labeling relations among elements in a small amount of professional texts in a corpus to form labeling data in a format of 'professional texts, elements-relations'; training a relation prediction model based on the labeling data, wherein the relation prediction model takes a professional text as input, and outputs one of any two elements contained in the professional textKA dimension vector representing the relationship between the two elements of the prediction KProbability on the individual relationship type; the relation prediction model preferably uses a pre-trained BERT (Bidirectional Encoder Representation from Transformers) model as a basic framework, and the specific process is to encode a professional text by using BERT (binary image analysis), so as to obtain a feature vector corresponding to each vocabulary in the professional text, wherein the feature vector of each element is obtained by taking the average value of the feature vectors of the vocabularies contained in the feature vector; to further predict the relationship type between two elements in a professional text, feature vectors of the two elements are splicedInput a MLP (Multilayer Perceptron) network, output aKVector of dimensions representing the relationship between two elements of the predictionKProbability on the individual relationship type;
aiming at the residual professional texts without manually labeling element relations in the corpus, predicting the association between elements in the professional texts by utilizing the trained relation prediction model, thereby forming an element sketch aiming at each professional text:
（1）
in the formula (1), theIs->Adjacent tensors of dimensions express +.>Relationship types between every two elements; />Is>And (4) the sum of->Wei->Is +.>Vector of dimensions, representing->Personal element and->The relation between the individual elements is- >Probability on the individual relationship type;
step S2: coding the element sketch and the viewpoint category corresponding to the professional text to obtain semantic hidden variable posterior distribution, and sampling the semantic hidden variable reconstruction element sketch in the distribution; coding the viewpoint category by using a condition coder to obtain semantic hidden variable prior distribution;
the goal of this step is to establish the association of the view category with the element diagram, the whole process is completed by the graph-variant self-encoder, and the input of the graph-variant self-encoder comprises the view categorySchematic of elements>Output of the element diagram after reconstruction ++>, wherein ,/>Representing background description->Representing a perspective; the graph variation self-encoder comprises a graph encoder, a graph encoder and a condition encoder:
diagram encoder pair element diagramAnd category of views->Coding to obtain semantic hidden variable +.>Posterior distribution of (2)，/>Representing the identity matrix, mean->Sum of variances->Obtained by the following formula:
（2）
in the formula (2) of the present invention,representing a diagram encoder; subsequently, from->Mid-sampling semantic hidden variable +.>；
The graph decoder depends on semantic hidden variablesGenerating a reconstructed element diagram>；/>Schematic of the elements after reconstitution->A corresponding adjacency tensor; />Is>And (4) the sum of->Wei->Is +.>Dimension vector, represent- >Personal element->And->Personal element->The relation between->Probability on the individual relationship type;by graphic encoder->According to the content of the element and the semantic hidden variable +.>And (3) predicting to obtain:
（3）
condition encoderFor obtaining category of views->Lower semantic hidden variable prior distributionThe method comprises the steps of carrying out a first treatment on the surface of the Let->Obeying mean->Sum of variances->Gaussian distribution>；Representing the identity matrix; /> and />Obtained by the following formula:
（4）
step S3: coding the professional text to obtain posterior distribution of the expression hidden variables, and sampling the expression hidden variables in the distribution; expressing hidden variables and reconstructing professional texts by using semantic hidden variables;
the goal of this step is to build text semantics and associations expressed to the generated text, done by the text variational self-encoder, which is entered as professional textSemantic hidden variable obtained by the last step of sampling +.>The output is reconstructed professional text +.>The method comprises the steps of carrying out a first treatment on the surface of the The text variation self-encoder comprises a text encoder and a text decoder;
the method for obtaining the posterior distribution of the expression hidden variables by the coded professional text comprises the following steps:
text encoder encodes input professional textObtaining expression hidden variable->Posterior distribution of (2)，/>Is an identity matrix, the mean value thereof->Sum of variances->Calculated by the following formula:
（5）
in the formula (5) of the present invention, Representing a text encoder, expressing the hidden variable +.>By from->Sampling to obtain;
the method for reconstructing the professional text by expressing the hidden variable and the semantic hidden variable sampled in the last step comprises the following steps:
text decoder and />For input, a reconstructed professional text is generated +.>, wherein ,/>Representing the text decoder:
（6）
step S4: inputting the reconstructed professional text into a discriminator, and providing multi-angle feedback information by the discriminator:
the discriminant comprises a category consistency discriminant, a field correlation discriminant and a language model, wherein the feedback of the discriminant guides the training diagram to be divided into a self-encoder and a text to be divided into a self-encoder;
the class consistency discriminator judges the reconstructed professional textClass of same views->Whether or not the two are consistent;
the domain correlation discriminator judges the reconstructed professional textWhether related to the current corpus field;
the language model is used for judging the textWhether the language expression is smooth;
the category consistency discriminator inputs a reconstructed professional textOutputting a probability vector +.>，/>Containing class consistencyProbability of each viewpoint category predicted by the sex discriminator +.>To->Representation category->At->The corresponding probability of (a) then category consistency feedback +. >The values of (2) are:
（7）
domain relevance discriminator input as reconstructed professional textOutput text +.>Probability of being related to the current corpus domain>Domain relevance feedback->The value of (2) is probability->：
（8）
Input of language model and reconstructed professional textThe output is reconstructed professional text +.>The probability of each word predicted by the language model forms the feedback of the fluency of the language +.>The value of which is reconstructed professional text +.>Probability product of each word:
（9）
in the formula (9) of the present invention,representation by language model according to ∈>Word predictive->The word->Probability of->Is->Is a length of (2);
for category of viewsComprehensive category consistency discriminant, domain correlation discriminant and language model to textFeedback information of (2) to get +.>Final feedback of->：
（10）
In the formula (10) of the present invention,respectively representing weights of the category consistency feedback and the language fluency feedback in the final feedback, and +.>；
Step S5: based on the discriminant feedback, the training graph variation is from the encoder and the text variation is from the encoder:
based on feedback information provided by the discriminator to the professional text, forming basic loss functions of the graph variation self-encoder and the text variation self-encoder:
（11）
in the formula (11) of the present invention, Representing a reconstruction probability of the text; />Representing a reconstruction probability of the elemental schematic; />Is a priori distribution of expression hidden variables, +.>Representing model parameters; />For +.>Divergence;
in addition to the underlying loss function, the present invention introduces additional loss functionsFor->Applying constraints: to prevent that the generation of text depends only on the expression hidden variables, minimizing the dependence on +.>Restructuring text +.>As in the first term in equation (12), at the same time, in order to attenuate +.>Semantic information contained in the content, minimizing reliance on +.>Reconstruction element sketch->As in the second term in equation (12):
（12）
in the formula (12) of the present invention,representing dependency->The probability of reconstructing the text; />Representation dependencyReconstructing the probability of the elemental schematic;
in order to make different views of the categoryCorresponding->Has large difference and increases contrast loss>Promote the same viewpoint category->Every professional text corresponding +.>Reducing the distribution similarity of different view categories +.>Lower part(s)Distribution similarity of (c):
（13）
in the case of the formula (13),similarity function representing distribution, ++>Representation and->From the same viewpoint categorySimple figure, ->Representation and->Element diagrams of different categories; />Representing dependency- > and />The posterior distribution of semantic hidden variables obtained by encoding; />Representing dependency-> and />The posterior distribution of semantic hidden variables obtained by encoding;
preferably, the JS distance (Jensen-Shannon Divergence) is introduced to measure the similarity of two distributions, i.e，/>The JS distance represents two distributions;
to sum up, the graph and text variations are derived from the final training loss function of the encoder：
（14）
In the case of the formula (14),a weight representing each loss function;
step S6: given the viewpoint category, generating diversified professional texts conforming to the viewpoint category by using a trained condition encoder, a graph decoder and a text decoder;
given a category of viewsObtaining semantic hidden variable prior distribution by a conditional coder>Is then sampled from this distribution the semantic hidden variable +.>From expressing the a priori distribution of hidden variables +.>Medium sampling expression hidden variable->The text decoder in the two kinds of hidden variables input step S3 generates professional text, and simultaneously, the semantic hidden variables are input into the graphic encoder in the step S2 to output element diagrams.
In the process, the hidden variables can be fixed, the viewpoint types are changed to obtain different semantic hidden variables, texts with different viewpoint types and similar language expressions are generated, the semantic hidden variables can also be fixed, and texts with consistent viewpoint types and different language expressions are generated by changing the hidden variables.
According to the present invention, before the step S6, a countermeasure training method for fusion contrast learning of a controllable diversity professional text generation method is further included, which is characterized by comprising:
step SS1: pre-training each component part of the variable self-encoder frame model independently;
the process aims at enabling each component of the model to have higher performance before joint training; the pre-training stage trains the multiple constituent parts of the discriminant and the graph variation self-encoder respectively.
The language model uses a model which is pre-trained on a large amount of general corpus, and in order to enable the language model to learn the expression mode which is commonly used in the field of professional texts, a plurality of professional texts are additionally added to train the language model so as to generate cross entropy of the texts and the real texts as a loss function;
the input of the category consistency discriminator is professional text generated by the text variation self-encoder, and a probability vector representing the category of the viewpoint is output, and the cross entropy of the predicted category and the real category is taken as a loss function;
the input data of the domain correlation discriminator comprises two types, namely a positive sample from the current corpus and a negative sample from other domain corpuses, the discriminator aims at giving high probability to the positive sample and giving low probability to the negative sample, and a specific loss function The method comprises the following steps:
（15）
in the case of the formula (15),representing the distribution of positive samples; />Representing the distribution of negative samples; />Representing decision->Probability associated with the current corpus domain;
pre-training loss function of graph variation self-encoderOn the one hand by contrast loss->Reducing the category->Every professional text corresponding +.>Distribution differences of (a) increase of different viewpoint categories +.>Lower part (C)Distribution differences of (2); on the other hand is to ensure->And distance from the prior distribution:
（16）
step SS2: model global countermeasure training
After the pre-training process of step SS1 is completed, an iterative countermeasure training is performed: the fixed text variation self-encoder and the graph variation self-encoder take the generated text as a negative sample, the real professional text as a positive sample, and train a class consistency discriminator; the text is generated by the text variation self-encoder and the graph variation self-encoder, the input discriminator obtains feedback, the loss update parameter is calculated, and the language model and the domain correlation discriminator are kept unchanged in the whole iteration process by continuously iterating until the model is integrally converged.
A system for realizing a controllable diversity professional text generation method based on an element diagram is characterized in that:
The system comprises a text variation self-encoder, a graph variation self-encoder and a plurality of discriminators;
the text variation self-encoder is configured to: in the training stage, the input text and the viewpoint category are encoded, the reconstructed text is output, and in the using stage, the text decoder contained in the text decoder generates the text conforming to the viewpoint category;
the graph variation self-encoder is configured to: in the training stage, coding the element sketch and the viewpoint category, outputting a reconstructed element sketch, in the using stage, coding the viewpoint category by a condition encoder contained in the element sketch, acquiring a semantic hidden variable, and generating the element sketch conforming to the semantic hidden variable by an graphic encoder;
the discriminant comprises a language model, a category consistency discriminant and a field relevance discriminant which are only applied to training optimization of auxiliary graph variation self-encoder and text variation self-encoding; the language model is input into a professional text generated by a text variation self-encoder and used for evaluating the language fluency of the professional text; the category consistency discriminator inputs the professional text generated by the text variation self-encoder and is used for judging consistency of the professional text and the viewpoint category; the domain relevance discriminator is used for inputting the professional text generated by the text variation self-encoder and judging the degree of the domain relevance between the text and the text corpus.
The invention has the advantages that:
the element diagram is a directed graph expressing the association relationship of viewpoint elements in professional text, wherein vertexes are viewpoint elements, and edges represent the association between vertexes. According to the method, an element sketch is introduced as a structured abstract description of professional text viewpoint semantics, semantic features and expression features of the text are decoupled in an implicit space by utilizing a variational self-encoder, and then the two types of features are combined to generate the professional text. In the process, the invention establishes the association of the views and the semantic features by constructing a condition encoder, realizes the control of the views on the text semantics, and realizes the diversity of text expression by randomly sampling different expression features. The introduction of the element sketch enables the invention to directly and definitely model the association relation between the viewpoint and the text semantic, and as one element sketch can represent the semantic of a plurality of professional texts of the same viewpoint, the text semantic can be strongly controlled by the viewpoint by utilizing a small amount of data training model; the text generation process integrating the semantic features and the expression features realizes the generation of diversified texts under semantic constraint, and prevents the generated texts from only pursuing diversity and neglecting semantics; the invention generates the dependent element sketch while generating the professional text, thereby realizing the interpretability of the generation process.
1. The invention provides a controllable diversity text generation model based on an element diagram, which is oriented to the professional field. Aiming at the defects that the existing model lacks the interpretability and only can control some simple attributes such as emotion, difficulty in perception and control of fine granularity difference among view elements and the like, the invention constructs a structured element diagram to express long-distance dependency relationship among concepts, establishes the association of control variables and the element diagram through a variation method, further depends on the view of the element diagram to control text, realizes the control of the internal logic relationship of the text, and provides the interpretability for the generating process when the model provided by the invention generates the text.
2. The invention realizes the generation of the diversified text under the semantic constraint, and is a variety in a controllable range. Compared with the existing diversity text generation method based on result filtering or vocabulary random sampling, the method lacks the defect of semantic constraint when diversity is improved, the technology combines semantics and expressions to jointly generate the text, so that the diversity displayed by the text is limited by the semantics of a core viewpoint on one hand, depends on the difference of the expressions on the other hand, and is more close to the real-world human writing form and process.
3. According to the invention, the multi-angle discriminator is introduced, the text generation is guided from the same viewpoint type consistency of the generated result, the field correlation on the content, the language fluency and the like, and a plurality of components of the discriminator can be pre-trained on a large amount of general field data, so that additional auxiliary information is introduced for the model, the type accuracy and the language fluency of the generated result are improved, and the dependence on the labeling data in the field is reduced.
Drawings
FIG. 1 is a block diagram of a controlled diversity professional text generation method and system based on element diagrams of the present invention;
FIG. 2 is a schematic diagram of a professional text with similar language expression and different viewpoint categories generated by sampling semantic hidden variables for multiple times while maintaining the expression hidden variables unchanged in the application stage of the method;
FIG. 3 is a schematic diagram of a professional text with different language expressions, wherein semantic hidden variables are kept unchanged in an application stage, and the hidden variables are expressed through multiple sampling to generate consistent viewpoint types;
in fig. 1, 2, 3:
expressed as a category of views; />Professional text conforming to the category of views; />Is->Corresponding element diagrams;to be from->Sampling the obtained semantic hidden variables; / >For semantic hidden variables +.>Is a priori distribution under the condition of (2); KL is a difference metric function between two distributions; />For training phase, for +.>A reconstructed element diagram; />To express hidden variables; />A priori distribution of hidden variables for expression; />For training phase, for +.>Reconstructed professional text.
Detailed Description
The invention will be described in more detail below with reference to examples and figures of the description, it being apparent that the invention can be embodied in many forms and is not limited to the examples set forth.
Example 1,
A controllable diversity professional text generation method based on element sketches, comprising:
creating a variational self-encoder framework model, as in fig. 1, includes: the graph variation self-encoder, the text variation self-encoder and the discriminator; wherein the graph variation self-encoder comprises a graph encoder, a graph encoder and a condition encoder; the text variation self-encoder comprises a text encoder and a text decoder; the discriminant comprises a category consistency discriminant, a field relevance discriminant and a language model;
processing specialized text for a given point of view category using a variational self-encoder framework model:
firstly, obtaining an element diagram corresponding to a professional text;
Secondly, under the constraint of viewpoint category, using a graph encoder to encode the element sketch to obtain posterior distribution of semantic hidden variables; acquiring semantic hidden variable prior distribution by using a condition encoder, acquiring expression hidden variable posterior distribution related to text expression by using a text encoder, and simultaneously assuming the expression hidden variable prior distribution to be standard normal distribution;
training phase: sampling semantic hidden variables and expression hidden variables from posterior distribution of the semantic hidden variables and posterior distribution of the expression hidden variables obtained by encoding, splicing the semantic hidden variables and the expression hidden variables, and inputting the spliced semantic hidden variables and the expression hidden variables into a text decoder to generate a text; simultaneously, inputting the semantic hidden variable into a graphic code reconstruction element diagram;
the training targets of the training phase include: on one hand, the KL divergence (Kullback-Leibler Divergence) of the prior distribution and the posterior distribution corresponding to semantic and expression hidden variables is minimized, and on the other hand, the probability of reconstructing input text and element sketches is maximized; in order to enhance the control of the generated result, introducing the discriminator to evaluate the smoothness, the field correlation and the consistency of the same-view category of the generated text;
the application stage comprises the following steps: given a viewpoint category, respectively sampling a semantic hidden variable and an expression hidden variable from the semantic hidden variable prior distribution and the expression hidden variable prior distribution, and inputting the semantic hidden variable and the expression hidden variable into a text of which the text decoder accords with the viewpoint category; meanwhile, the sampled semantic hidden variable input diagram decoder generates an element diagram which is used as an explanatory basis for generating a text.
The controllable diversity professional text generation method based on the element diagram specifically comprises the following steps:
step S1: obtaining an element diagram corresponding to the professional text:
constructing an element set and a relation set aiming at a corpus formed by all the obtained professional texts, and acquiring initial feature matrixes of the element set and the relation set;
the method for constructing the element set and the relation set comprises the following steps: extracting elements existing in the corpus by combining word frequency statistical features in the professional text corpus and word segmentation tools such as 'bargain word segmentation', and forming element setsWherein said->Is an element, said->The number of elements in the element set; expert-defined relation set->Wherein, said->Is an element in the set of relationships, said +.>Is the number of relationships in the set of relationships;
the method for acquiring the initial feature matrix of the element set and the relation set comprises the following steps:
feature vectors of elements and relations are obtained through a Chinese pre-training word vector tool package ngram2vec, and then feature matrixes of an initial element set and a relation set are formed:
，/>
wherein ,is an element feature matrix; />Is a relation characteristic matrix; />For the number of elements in the set of elements, The dimension of the element feature vector; />Is the number of relationships in the set of relationships; />The dimension of the relation feature vector; />
The method for acquiring the element sketch corresponding to the professional text comprises the following steps:
for all professional texts in the corpus, matching element setsObtaining the elements contained in the professional text;
manually labeling relations among elements in a small amount of professional texts in a corpus to form labeling data in a format of 'professional texts, elements-relations'; training a relation prediction model based on the labeling data, wherein the relation prediction model takes a professional text as input, and outputs one of any two elements contained in the professional textKA dimension vector representing the relationship between the two elements of the predictionKProbability on the individual relationship type; the relation prediction model preferably uses a pre-trained BERT (Bidirectional Encoder Representation from Transformers) model as a basic framework, and the specific process is to encode a professional text by using BERT (binary image analysis), so as to obtain a feature vector corresponding to each vocabulary in the professional text, wherein the feature vector of each element is obtained by taking the average value of the feature vectors of the vocabularies contained in the feature vector; in order to further predict the relationship type between two elements in the professional text, the feature vectors of the two elements are spliced and input into a MLP (Multilayer Perceptron) network to output a KVector of dimensions representing the relationship between two elements of the predictionKProbability on the individual relationship type;
aiming at the residual professional texts without manually labeling element relations in the corpus, predicting the association between elements in the professional texts by utilizing the trained relation prediction model, thereby forming an element sketch aiming at each professional text:
（1）
in the formula (1), theIs->Adjacent tensors of dimensions express +.>Relationship types between every two elements; />Is>And (4) the sum of->Wei->Is +.>Vector of dimensions, representing->Personal element and->The relation between the individual elements is->Probability on the individual relationship type;
step S2: coding the element sketch and the viewpoint category to obtain semantic hidden variable posterior distribution, and sampling the semantic hidden variable in the distribution to reconstruct the element sketch; coding the viewpoint category by using a condition coder to obtain semantic hidden variable prior distribution;
the goal of this step is to establish the association of the view category with the element diagram, the whole process is completed by the graph-variant self-encoder, and the input of the graph-variant self-encoder comprises the view categoryTo be processedSimple figure->Output of the element diagram after reconstruction ++>, wherein ,/>Representing background description- >Representing a perspective; the graph variation self-encoder comprises a graph encoder, a graph encoder and a condition encoder:
in a specific implementation, the graph encoder first uses a Bi-directional long and short Term Memory network BiLSTM (Bi-directional Long Short-Term Memory) for view categoriesEncoding to obtain a representation thereof:
(17)
wherein , and />And respectively representing hidden states corresponding to the last word of the text obtained by forward and backward LSTM network coding. />
Then, the diagram encoder encodes the element diagram Obtaining a simplified diagram of elements>Semantic representation vector +.>In this process, a matrix of fusion relationship type information is first acquired>:
(18)
wherein ,representing real space +.>Is->And (5) maintaining the parameter vector. By->The information transmission between the elements of the wheel updates the feature vectors of all the elements, and each information transmission process comprises two stages, namely, the fusion of the associated relation information and the element information of each element, and the updating of the feature expression vector of each element. Specifically, the firstThe fusion process of the relationship information and the element information of the wheel is expressed as:
wherein ,is->Characteristic representation matrix of all elements of the wheel, +.>First->The updating method of the feature expression matrix of all elements of the wheel comprises the following steps:
(20)
wherein ,parameter matrix representing real space, < > >Representing a nonlinear activation function. To->Representation->Characteristic representation matrix of all elements after transmission of wheel information, < >>Representation->Middle->Feature vectors of the individual elements. Element diagram->Semantic representation vector +.>Accumulating all element feature vectors, namely +.>；
Diagram encoder pair element diagramAnd category of views->Coding to obtain semantic hidden variable +.>Posterior distribution of (2)，/>Representing the identity matrix, mean->Sum of variances->Obtained by the following formula:
(2)
in the formula (2) of the present invention,representing a diagram encoder; semantic hidden variable +.>From->Sampling to obtain;
in a specific implementation, a graph encoderAccording to-> and />Obtaining semantic hidden variables through a first MLP networkPosterior distribution of->Mean>Sum of variances->，/>The unit matrix is as follows:
(21)
picture decoderAccording to semantic hidden variable +.>Generating a reconstructed element diagram>；/>Schematic of the elements after reconstitution->A corresponding adjacency tensor; />Is>And (4) the sum of->Wei->Is +.>Dimension vector, represent->Personal element->And->Personal element->The relation between->Probability on the individual relationship type; />By graphic encoder->According to the content of the element and the semantic hidden variable +.>And (3) predicting to obtain:
(3)
in particular implementation, forMiddle->Personal element and->Element feature vector +. > and />Splicing and inputting a second MLP network to predict the type of the relationship between two elements:
(22)
wherein ,is +.>Dimension vector, represent->Personal element->And->Individual elementsThe relation between->Probability on the individual type, thus, all +.>Constitutes +.>Adjacent tensor of dimension->Obtaining a schematic of the elements after reconstitution->。
Condition encoderFor obtaining category of views->Lower semantic hidden variable prior distributionThe method comprises the steps of carrying out a first treatment on the surface of the Let->Obeying mean->Sum of variances->Gaussian distribution>；/>Representing the identity matrix; /> and />Obtained by the following formula:
(4)
in a specific implementation, a condition encoderCoding vectors from view categories by a graph encoderCalculating to obtain conditional prior distribution of semantic hidden variables by using a third MLP network>Mean>Sum of variances：
(23)
Step S3: coding the professional text to obtain posterior distribution of the expression hidden variables, and sampling the expression hidden variables in the distribution; expressing hidden variables and reconstructing professional texts by using semantic hidden variables;
the goal of this step is to build text semantics and associations expressed to the generated text, done by the text variational self-encoder, which is entered as professional textSemantic hidden variable obtained by the last step of sampling +.>The output is reconstructed professional text +.>The method comprises the steps of carrying out a first treatment on the surface of the The text variation self-encoder comprises a text encoder and a text decoder;
The method for obtaining the posterior distribution of the expression hidden variables by the coded professional text, and sampling the expression hidden variables in the distribution comprises the following steps:
text encoder encodes input professional textObtaining expression hidden variable->Posterior distribution of (2)，/>Is an identity matrix, the mean value thereof->Sum of variances->Calculated by the following formula:
(5)
in the formula (5) of the present invention,representing a text encoder, expressing the hidden variable +.>By from->Sampling to obtain;
the method for reconstructing the professional text by expressing the hidden variable and the semantic hidden variable sampled in the last step comprises the following steps:
text decoder and />For input, a reconstructed professional text is generated +.>, wherein ,T_DECrepresenting the text decoder:
(6)
in a specific implementation, text isInput text encoder, text encoder uses BiLSTM network, output text +.>Semantic table of (a)Show->Obtain->The view type is encoded in the same manner as the view encoder, but the network parameters are different. Subsequently, based on->By a +.>The network gets the expression hidden variable->Posterior distributionMean>Sum of variances->，/>The unit matrix is as follows:
(24)
text decoder uses LSTM network structure, first fromMiddle sampling->,/>And +.>Splicing to obtain hidden variable->，/>As an initial hidden state of LSTM network, i.e Subsequently, the reconstructed professional text is output via the LSTM network>The generation process is expressed as follows:
(25)
(26)/>
wherein ,is->Probability distribution of each word in the dictionary obtained in the step +.>Is->Is>Personal word，By->Sampling to get->Indicating LSTM->Hidden state of step.
Step S4: inputting the reconstructed professional text into a discriminator, and providing multi-angle feedback information by the discriminator:
the discriminant comprises a category consistency discriminant, a field correlation discriminant and a language model, wherein the feedback of the discriminant guides the training diagram to be divided into a self-encoder and a text to be divided into a self-encoder;
the class consistency discriminator judges the reconstructed professional textClass of same views->Whether or not the two are consistent;
the domain correlation discriminator judges the reconstructed professional textWhether related to the current corpus field;
the language model is used for judging the textWhether the language expression is smooth;
the category consistency discriminator inputs a reconstructed professional textOutputting a probability vector +.>，/>Probability of each viewpoint category predicted by category consistency discriminator is included +.>To->Representation category->At->The corresponding probability of (a) then category consistency feedback +. >The values of (2) are:
(7)
domain relevance discriminator input as reconstructed professional textOutput text +.>Probability of being related to the current corpus domain>Domain relevance feedback->The value of (2) is probability->：
(8)
Input of language model and reconstructed professional textThe output is reconstructed professional text +.>The probability of each word predicted by the language model forms the feedback of the fluency of the language +.>The value of which is reconstructed professional text +.>Probability product of each word:
(9)
in the formula (9) of the present invention,representation by language model according to ∈>Word predictive->The word->Probability of->Is->Is a length of (2);
for category of viewsComprehensive category consistency discriminant, domain correlation discriminant and language model to textFeedback information of (2) to get +.>Final feedback of->：/>
(10)
In the formula (10) of the present invention,respectively representing weights of the category consistency feedback and the language fluency feedback in the final feedback, and +.>；
In a specific implementation, the class consistency discriminator uses special characters and />Splicing the input BERT to obtainCorresponding feature vector>The feature vector is then input into an MLP network to obtain a probability vector +.>：
(27)
Representing the probability that the text predicted by the model belongs to each viewpoint category. For a given +. >To->Representation category->At->The corresponding probability of (a) then category consistency feedback +.>The values of (2) are:
(28)
the domain relevance discriminator uses special characters，/>Splicing the input BERT to obtain ∈>Corresponding feature vector>Subsequently, the +.>Probability associated with a specialized text corpus domain:
(29)
domain relevance feedbackFor probability->：
(30)
The domain relevance discriminator and the class consistency discriminator may also share the same BERT, but do not share the MLP network.
Language model obtains text using GPT modelThe probability of each word on the language model. Language fluency feedback->Probability product for each word:
(31)
wherein ,representation by language model according to ∈>Word predictive->The word->Probability of->Is->Is a length of (c).
Combining the above-mentioned discriminant pairsIs +.>Obtaining pair->Is->, wherein />Respectively representing weights of the category consistency feedback and the language fluency feedback in the final feedback, and +.>。
(31.)
In the above description, BERT, MLP networks are independent networks in that parameters are not shared among different encoders and decoders.
Step S5: based on the discriminant feedback, the training graph variation is from the encoder and the text variation is from the encoder:
Based on feedback information provided by the discriminator to the professional text, basic objective functions of the graph variation self-encoder and the text variation self-encoder are formed:
(11)
in the formula (11) of the present invention,representing a reconstruction probability of the text; />Representing a reconstruction probability of the elemental schematic; />Is a priori distribution of expression hidden variables, +.>Representing model parameters; />KL divergence between the two distributions;
in addition to the underlying loss function, the present invention introduces additional loss functionsFor->Applying constraints: to prevent that the generation of text depends only on the expression hidden variables, minimizing the dependence on +.>Restructuring text +.>As in the first term in equation (12), at the same time, in order to attenuate +.>Semantic information contained in the content, minimizing reliance on +.>Reconstruction element sketch->As in the second term in equation (12):
(12)
in the formula (12) of the present invention,representing dependency->The probability of reconstructing the text; />Representation dependencyReconstructing the probability of the elemental schematic;
in order to make different views of the categoryCorresponding->Has a large difference, thus increasing contrast loss +.>Promote the same viewpoint category->Every professional text corresponding +.>Reducing the distribution similarity of different view categories +.>Down->Distribution similarity of (c):
(13)/>
in the case of the formula (13), Similarity function representing distribution, ++>Representation and->Element sketches of the same category, +.>Representation and->Element diagrams of different categories; />Representing dependency-> and />The posterior distribution of semantic hidden variables obtained by encoding; />Representing dependency-> and />The posterior distribution of semantic hidden variables obtained by encoding;
preferably, the JS distance (Jensen-Shannon Divergence) is introduced to measure the similarity of two distributions, i.e，/>The JS distance represents two distributions;
to sum up, the graph and text variations are derived from the final training loss function of the encoder：
(14)
In the case of the formula (14),a weight representing each loss function;
step S6: given the viewpoint category, generating diversified professional texts conforming to the viewpoint category by using a trained condition encoder, a graph decoder and a text decoder;
given a category of viewsObtaining semantic hidden variable prior distribution by a conditional coder>Is then sampled from this distribution the semantic hidden variable +.>From expressing the a priori distribution of hidden variables +.>Medium sampling expression hidden variable->Two types of hidden variables are input into a text decoder to generate professional text, and simultaneously, the semantic hidden variables are input into a graph decoder to output element diagrams.
In the process, the hidden expression variable can be fixed, the viewpoint type is changed to generate texts with different viewpoint types and similar language expressions, the semantic hidden variable can also be fixed, and the hidden expression variable is changed to generate texts with consistent viewpoint types and different language expressions.
EXAMPLE 2,
The method for generating a controllable diversity professional text based on an element diagram according to embodiment 1, before the step S6, further includes a countermeasure training method for fusion contrast learning of the controllable diversity professional text generating method, and is characterized by comprising:
step SS1: pre-training each component part of the variable self-encoder frame model independently;
the process aims at enabling each component of the model to have higher performance before joint training; the pre-training stage trains the multiple constituent parts of the discriminant and the graph variation self-encoder respectively.
The language model uses a model which is pre-trained on a large amount of general corpus, and in order to enable the language model to learn the expression mode which is commonly used in the field of professional texts, a plurality of professional texts are additionally added to train the language model so as to generate cross entropy of the texts and the real texts as a loss function;
the input of the category consistency discriminator is professional text, a probability vector representing the category of the viewpoint is output, and the cross entropy of the predicted category and the real category is taken as a loss function;
the input data of the domain correlation discriminator comprises two types, namely a positive sample from the current corpus and a negative sample from other domain corpuses, the discriminator aims at giving high probability to the positive sample and giving low probability to the negative sample, and a specific loss function The method comprises the following steps:
(15)
in the case of the formula (15),representing the distribution of positive samples; />Representing the distribution of negative samples; />Representing decision->Probability associated with the current corpus domain;
pre-training loss function of graph variation self-encoderOn the one hand by contrast loss->Reducing the category->Every professional text corresponding +.>Distribution differences of (a) increase of different viewpoint categories +.>Lower part (C)Distribution differences of (2); on the other hand is to ensure->And distance from the prior distribution:
(16）
step SS2: model global countermeasure training
After the pre-training process of step SS1 is completed, an iterative countermeasure training is performed: the fixed text variation self-encoder and the graph variation self-encoder take the generated text as a negative sample, the real professional text as a positive sample, and train a class consistency discriminator; the text is generated by the text variation self-encoder and the graph variation self-encoder, the input discriminator obtains feedback, the loss update parameter is calculated, and the language model and the domain correlation discriminator are kept unchanged in the whole iteration process by continuously iterating until the model is integrally converged.
EXAMPLE 3,
A system for implementing a controlled diversity professional text generation method based on element diagrams, the system comprising a text variation self-encoder, a diagram variation self-encoder and a discriminator;
The text variation self-encoder is configured to: in the training stage, the input text and the viewpoint category are encoded, the reconstructed text is output, and in the using stage, the text decoder contained in the text decoder generates the text conforming to the viewpoint category;
the graph variation self-encoder is configured to: in the training stage, coding the element sketch and the viewpoint category, outputting a reconstructed element sketch, in the using stage, coding the viewpoint category by a condition encoder contained in the element sketch, acquiring a semantic hidden variable, and generating the element sketch conforming to the semantic hidden variable by an graphic encoder;
the discriminant comprises a language model, a category consistency discriminant and a field relevance discriminant which are only applied to training optimization of auxiliary graph variation self-encoder and text variation self-encoding; the language model is input into a professional text generated by a text variation self-encoder and used for evaluating the language fluency of the professional text; the category consistency discriminator inputs the professional text generated by the text variation self-encoder and is used for judging consistency of the professional text and the viewpoint category; the domain relevance discriminator is used for inputting the professional text generated by the text variation self-encoder and judging the degree of the domain relevance between the text and the text corpus.
The specific application scenarios of fig. 1, fig. 2 and fig. 3 in combination with the above embodiments and the description are as follows:
in an automatic review task aiming at legal examination, the performance of an automatic review model is severely limited due to smaller supervision data of manual review, and in the scene, a large number of new data complement training data sets of specified types are generated through a controllable diversity professional text generation method based on element diagrams, and the formed new data sets can be used for training the automatic review model to improve the accuracy of the review model.
In the above embodiment, the background text is a section of fact description text and questions based on the text, the professional text is an answer of the examinee, and the questions are classified into correct and incorrect categories, and the category of the views includes the background text and the views to be controlled. By using the method provided by the invention, the text which corresponds to different viewpoint categories and is similar to the expression characteristics of the answer language of the whole examinee is generated.
The training process of the above embodiment includes two stages of pre-training and countermeasure training. In the pre-training stage, the components and graph variations of the main pre-training discriminant are separated from the encoder.
The language model is input as answer text of the examinee during pre-training, a subsequent text sequence is generated in an autoregressive mode, and the loss function used by the language model is already given in the description above; the category consistency discriminator inputs the category of the viewpoint and the answer of the examinee, outputs the probability of predicting the answer of the examinee to belong to a certain category, and the loss function used by the answer is given in the description above; the input of the domain correlation discriminator is a positive sample, the negative sample is from the Baidu encyclopedia and news website, the positive sample is an answer of a test taker, the discriminator predicts the probability that the input text belongs to the legal domain, and the used loss function is given in the description above; in the pre-training process of the graph variation self-encoder, the inputs are the viewpoint category and the element diagram, and a reconstructed element diagram is generated, and the objective function used by the reconstructed element diagram is already given in the description above;
After the pre-training process is finished, iterative countermeasure training is performed: and (3) fixing the discriminator, generating professional text by the text variation self-encoder and the graph variation self-encoder, inputting the professional text into the discriminator to obtain feedback, calculating loss and updating parameters. The fixed text variation self-encoder and the graph variation self-encoder take the generated text as a negative sample, the real answer text of the examinee as a positive sample, and train the class consistency discriminator. The above process is iterated until the model converges. The language model and domain relevance discriminant remain unchanged throughout the iterative process.
After the whole training process is finished, the view category is given, an expression hidden variable is sampled from standard normal distribution according to the condition encoder, and the two types of hidden variables are input into a text decoder to generate texts. Or fixing the semantic hidden variables, expressing the hidden variables through random sampling, and generating texts with consistent viewpoint types and different language expressions, as shown in figure 3. Through the above process, the expansion of the original data set is realized.

Claims (5)

1. The controllable diversity professional text generation method based on the element diagram is characterized by comprising the following steps of:
establishing a variational self-encoder framework model, comprising: the graph variation self-encoder, the text variation self-encoder and the discriminator; wherein the graph variation self-encoder comprises a graph encoder, a graph encoder and a condition encoder; the text variation self-encoder comprises a text encoder and a text decoder; the discriminant comprises a category consistency discriminant, a field relevance discriminant and a language model;
processing specialized text for a given point of view category using a variational self-encoder framework model:
firstly, obtaining an element diagram corresponding to a professional text;
secondly, encoding the element sketch and the viewpoint category by using a graph encoder to obtain posterior distribution of semantic hidden variables; the method comprises the steps of using a conditional coder to code viewpoint categories to obtain semantic hidden variable prior distribution, using a text coder to obtain expression hidden variable posterior distribution, and simultaneously assuming the expression hidden variable prior distribution to be standard normal distribution;
training phase: sampling semantic hidden variables and expression hidden variables from posterior distribution of the semantic hidden variables and posterior distribution of the expression hidden variables obtained by encoding, splicing the semantic hidden variables and the expression hidden variables, and inputting the spliced semantic hidden variables and the expression hidden variables into a text decoder to generate a text; simultaneously, inputting the semantic hidden variable into a graphic code reconstruction element diagram;
The application stage comprises the following steps: given a viewpoint category, respectively sampling a semantic hidden variable and an expression hidden variable from the semantic hidden variable prior distribution and the expression hidden variable prior distribution, and inputting the semantic hidden variable and the expression hidden variable into a text of which the text decoder accords with the viewpoint category; meanwhile, the sampled semantic hidden variable input diagram decoder generates an element diagram which is used as an explanatory basis for generating a text;
the professional text generation method specifically comprises the following steps:
step S1: obtaining an element diagram corresponding to the professional text:
constructing an element set and a relation set aiming at a corpus formed by all the obtained professional texts, and acquiring initial feature matrixes of the element set and the relation set;
the method for constructing the element set and the relation set comprises the following steps: combining word frequency statistical features and word segmentation tools in professional text corpus to form element setWherein said->Is an element, said->The number of elements in the element set; expert-defined relation set->Wherein, said->Is an element in the set of relationships, said +.>Is the number of relationships in the set of relationships;
the method for acquiring the initial feature matrix of the element set and the relation set comprises the following steps:
Feature vectors of elements and relations are obtained through a Chinese pre-training word vector tool package ngram2vec, and then feature matrixes of an initial element set and a relation set are formed:
，/>
wherein ,is an element feature matrix; />Is a relation characteristic matrix; />For the number of elements in the element set, < > for>The dimension of the element feature vector; />Is the number of relationships in the set of relationships; />The dimension of the relation feature vector;
the method for acquiring the element sketch corresponding to the professional text comprises the following steps:
manually labeling relations among elements in a small amount of professional texts in a corpus to form labeling data in a format of 'professional texts, elements-relations'; training a relation prediction model based on the labeling data, wherein the relation prediction model takes a professional text as input, and outputs one of any two elements contained in the professional textA dimension vector representing the relation between these two elements of the prediction in +.>Probability on the individual relationship type; the feature vector of the element is obtained by measuring the average value of the included vocabulary feature vector; in order to further predict the relationship type between two elements in the professional text, the feature vectors of the two elements are spliced and input into a MLP (Multilayer Perceptron) network to output a +. >Vector of dimensions representing the relationship between two elements of the prediction in +.>Probability on the individual relationship type;
aiming at the residual professional texts without manually labeling element relations in the corpus, predicting the association between elements in the professional texts by utilizing the trained relation prediction model, thereby forming an element sketch aiming at each professional text:
（1）
in the formula (1), theIs->Adjacent tensors of dimensions express +.>Relationship types between every two elements; />Is>And (4) the sum of->Wei->Is +.>Vector of dimensions, representing->Personal element and->The relation between the individual elements is->Probability on the individual relationship type;
step S2: coding the element sketch and the viewpoint category to obtain semantic hidden variable posterior distribution, and sampling the semantic hidden variable in the distribution to reconstruct the element sketch; coding the viewpoint category by using a condition coder to obtain semantic hidden variable prior distribution;
the input to the graph variant self-encoder includes a view categorySchematic of elements>Output of the element diagram after reconstruction ++>, wherein ,/>Representing background description->Representing a perspective; the graph variation self-encoder comprises a graph encoder, a graph encoder and a condition encoder:
diagram encoder pair element diagram And category of views->Coding to obtain semantic hidden variable +.>Posterior distribution of (2)，/>Representing the identity matrix, mean->Sum of variances->Obtained by the following formula:
（2）
in the formula (2) of the present invention,representing a diagram encoder; subsequently, from->Mid-sampling semantic hidden variables；
The graph decoder depends on semantic hidden variablesGenerating a reconstructed element diagram>；/>Schematic of the elements after reconstitution->A corresponding adjacency tensor; />Is>And (4) the sum of->Wei->Is +.>Dimension vector, represent->Personal element->And->Personal element->The relation between->Probability on the individual relationship type;by graphic encoder->According to the content of the element and the semantic hidden variable +.>And (3) predicting to obtain:
（3）
condition encoderFor obtaining category of views->Lower semantic hidden variable prior distributionThe method comprises the steps of carrying out a first treatment on the surface of the Let->Obeying mean->Sum of variances->Gaussian distribution>；Representing the identity matrix; /> and />Obtained by the following formula:
（4）
step S3: coding the professional text to obtain posterior distribution of the expression hidden variables, and sampling the expression hidden variables in the distribution; expressing hidden variables and reconstructing professional texts by using semantic hidden variables;
the goal of this step is to build text semantics and associations expressed to the generated text, which are separated by text variationsCompleted from encoder, its input being professional textSemantic hidden variable obtained by the last step of sampling +. >The output is reconstructed professional text +.>The method comprises the steps of carrying out a first treatment on the surface of the The text variation self-encoder comprises a text encoder and a text decoder;
the method for obtaining the posterior distribution of the expression hidden variables by the coded professional text comprises the following steps:
text encoder encodes input professional textObtaining expression hidden variable->Posterior distribution of (2)，/>Is an identity matrix, the mean value thereof->Sum of variances->Calculated by the following formula:
（5）
in the formula (5) of the present invention,representing a text encoder, expressing the hidden variable +.>By from->Sampling to obtain;
the method for reconstructing the professional text by expressing the hidden variable and the semantic hidden variable sampled in the last step comprises the following steps:
text decoder and />For input, a reconstructed professional text is generated +.>, wherein ,T_DECrepresenting the text decoder:
（6）
step S4: inputting the reconstructed professional text into a discriminator, and providing multi-angle feedback information by the discriminator:
the discriminant comprises a category consistency discriminant, a field correlation discriminant and a language model, wherein the feedback of the discriminant guides the training diagram to be divided into a self-encoder and a text to be divided into a self-encoder;
the class consistency discriminator judges the reconstructed professional textClass of same views->Whether or not the two are consistent;
the domain correlation discriminator judges the reconstructed professional text Whether related to the current corpus field;
the language model is used for judging the textWhether the language expression is smooth;
the category consistency discriminator inputs a reconstructed professional textOutputting a probability vector +.>，/>Probability of each viewpoint category predicted by category consistency discriminator is included +.>To->Representation category->At->The corresponding probability of (a) then category consistency feedback +.>The values of (2) are:
（7）
domain relevance discriminator input as reconstructed professional textOutput text +.>Probability of being related to the current corpus domain>Domain relevance feedback->The value of (2) is probability->：
（8）
Input of language model and reconstructed professional textThe output is reconstructed professional text +.>The probability of each word predicted by the language model forms the feedback of the fluency of the language +.>The value of which is reconstructed professional text +.>Probability product of each word:
（9）
in the formula (9) of the present invention,representation by language model according to ∈>Word predictive->The words areProbability of->Is->Is a length of (2);
for category of viewsComprehensive category consistency discriminant, domain correlation discriminant and language model versus text +.>Feedback information of (2) to get +.>Final feedback of- >：
（10）
In the formula (10) of the present invention,respectively representing weights of the category consistency feedback and the language fluency feedback in the final feedback, and +.>；
Step S5: based on the discriminant feedback, the training graph variation is from the encoder and the text variation is from the encoder:
based on feedback information provided by the discriminator to the professional text, forming basic loss functions of the graph variation self-encoder and the text variation self-encoder:
（11）
in the formula (11) of the present invention,representing a reconstruction probability of the text; />Representing a reconstruction probability of the elemental schematic; />Is a priori distribution of expression hidden variables, +.>Representing model parameters;for +.>Divergence;
（12）
at the publicIn the formula (12), the amino acid sequence of the compound,representing dependency->The probability of reconstructing the text; />Representing dependency->Reconstructing the probability of the elemental schematic; the first term in equation (12) is: minimizing reliance on +.>Is>The second term in equation (12) is: minimizing reliance on +.>Is>Probability of (2);
increase contrast loss：
（13）
In the case of the formula (13),similarity function representing distribution, ++>Representation and->Element sketches of the same category, +.>Representation and->Element diagrams of different categories; />Representing dependency-> and />The posterior distribution of semantic hidden variables obtained by encoding; / >Representing dependency-> and />The posterior distribution of semantic hidden variables obtained by encoding;
step S6: given the viewpoint category, generating diversified professional texts conforming to the viewpoint category by using a trained condition encoder, a graph decoder and a text decoder;
given a category of viewsBy semantic hidden variable a priori distribution ++>Sampling semanticsHidden variableFrom expressing the a priori distribution of hidden variables +.>Medium sampling expression hidden variable->Two kinds of hidden variables are input into a text decoder in the step S3 to generate professional text, and simultaneously, the semantic hidden variables are input into a graph decoder to output an element diagram.
2. The method for generating the controllable diversity professional text based on the element sketch according to claim 1, wherein the expression hidden variable is a variable capable of determining a text expression mode in a vector space and is used for generating texts with different viewpoint types and similar language expressions; the semantic hidden variable is a variable capable of determining text semantics in a vector space and is used for generating texts with consistent viewpoint types and different language expressions.
3. The method for generating a controlled diversity professional text based on an element diagram according to claim 1, wherein, in step S5, a JS distance metric is introduced to measure the similarity of two distributions, namely ，/>The JS distance represents two distributions;
to sum up, the graph and text variations are derived from the final training loss function of the encoder：
（14）
In the case of the formula (14),representing the weight of each loss function.
4. The method for generating a controlled diversity professional text based on an element diagram according to claim 1, further comprising a countermeasure training method for fusion contrast learning of the controlled diversity professional text generating method, before the step S6, comprising:
step SS1: pre-training each component part of the variable self-encoder frame model independently;
the process aims at enabling each component of the model to have higher performance before joint training; the pre-training stage respectively trains a plurality of component parts of the discriminator and a graph variation self-encoder;
the language model is pre-trained on a large number of general corpus, and in order to enable the language model to learn the expression mode commonly used in the professional text field, a plurality of professional text training language models are additionally added to generate cross entropy of the text and the real text as a loss function;
the input of the category consistency discriminator is professional text, a probability vector representing the category of the viewpoint is output, and the cross entropy of the predicted category and the real category is taken as a loss function;
The input data of the domain correlation discriminator comprises two types, namely a positive sample consistent with the professional text domain and a negative sample sampled from other domains, the discriminator aims at giving high probability to the positive sample and giving low probability to the negative sample, and a specific loss functionThe method comprises the following steps:
（15）
in the case of the formula (15),representing the distribution of positive samples; />Representing the distribution of negative samples; />Representing decision->Probability associated with the current corpus domain; pre-training loss function of graph variation self-encoder>The expression is as follows:
（16）
in the formula (16) of the present invention,is a contrast loss;
step SS2: model global countermeasure training
After the pre-training process of step SS1 is completed, iterative countermeasure training is performed.
5. A system for implementing the controlled diversity professional text generation method based on element diagram according to any of claims 1-4, characterized in that:
the system comprises a text variation self-encoder, a graph variation self-encoder and a plurality of discriminators;
the text variation self-encoder is configured to: in the training stage, the input text and the viewpoint category are encoded, the reconstructed text is output, and in the using stage, the text decoder contained in the text decoder generates the text conforming to the viewpoint category;
The graph variation self-encoder is configured to: in the training stage, coding the element sketch and the viewpoint category, outputting a reconstructed element sketch, in the using stage, coding the viewpoint category by a condition encoder contained in the element sketch, acquiring a semantic hidden variable, and generating the element sketch conforming to the semantic hidden variable by an graphic encoder;
the discriminant comprises a language model, a category consistency discriminant and a field relevance discriminant which are only applied to training optimization of auxiliary graph variation self-encoder and text variation self-encoding; the language model is input into a professional text generated by a text variation self-encoder and used for evaluating the language fluency of the professional text; the category consistency discriminator inputs the professional text generated by the text variation self-encoder and is used for judging consistency of the professional text and the viewpoint category; the domain relevance discriminator is used for inputting the text variation to generate professional text from the encoder and judging the degree of the text related to the domain of the text corpus.
CN202310685875.9A 2023-06-12 2023-06-12 Controllable diversity professional text generation method and system based on element diagram Active CN116432663B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310685875.9A CN116432663B (en) 2023-06-12 2023-06-12 Controllable diversity professional text generation method and system based on element diagram

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310685875.9A CN116432663B (en) 2023-06-12 2023-06-12 Controllable diversity professional text generation method and system based on element diagram

Publications (2)

Publication Number Publication Date
CN116432663A CN116432663A (en) 2023-07-14
CN116432663B true CN116432663B (en) 2023-10-13

Family

ID=87085827

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310685875.9A Active CN116432663B (en) 2023-06-12 2023-06-12 Controllable diversity professional text generation method and system based on element diagram

Country Status (1)

Country Link
CN (1) CN116432663B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117934229B (en) * 2024-03-18 2024-06-21 新励成教育科技股份有限公司 Originality excitation-based talent training guiding method, system, equipment and medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111597779A (en) * 2020-04-23 2020-08-28 腾讯科技（深圳）有限公司 Text generation method, device, equipment and storage medium
CN112699688A (en) * 2021-01-08 2021-04-23 北京理工大学 Text generation method and system with controllable discourse relation
WO2021143022A1 (en) * 2020-01-14 2021-07-22 平安科技（深圳）有限公司 Text generation method and apparatus
CN113254604A (en) * 2021-07-15 2021-08-13 山东大学 Reference specification-based professional text generation method and device
CN113990520A (en) * 2021-11-05 2022-01-28 天津工业大学 Traditional Chinese medicine prescription generation method based on controllable generation countermeasure network
CN115828142A (en) * 2022-12-17 2023-03-21 福州大学 Interpretable text classification system based on double-path encoder

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11586829B2 (en) * 2020-05-01 2023-02-21 International Business Machines Corporation Natural language text generation from a set of keywords using machine learning and templates
US11481552B2 (en) * 2020-06-01 2022-10-25 Salesforce.Com, Inc. Generative-discriminative language modeling for controllable text generation

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021143022A1 (en) * 2020-01-14 2021-07-22 平安科技（深圳）有限公司 Text generation method and apparatus
CN111597779A (en) * 2020-04-23 2020-08-28 腾讯科技（深圳）有限公司 Text generation method, device, equipment and storage medium
CN112699688A (en) * 2021-01-08 2021-04-23 北京理工大学 Text generation method and system with controllable discourse relation
CN113254604A (en) * 2021-07-15 2021-08-13 山东大学 Reference specification-based professional text generation method and device
CN113990520A (en) * 2021-11-05 2022-01-28 天津工业大学 Traditional Chinese medicine prescription generation method based on controllable generation countermeasure network
CN115828142A (en) * 2022-12-17 2023-03-21 福州大学 Interpretable text classification system based on double-path encoder

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Zhiting Hu,et al.Controllable Text Generation.《cs.LG》.2017,1-10页. *
一种基于参考规范的专业文本生成方法;胡宇，王舰，孙宇清;《中文信息学报》;第37卷(第3期);152-163页 *
基于深度学习的可控文本生成方案的设计与应用;张文静;《中国优秀硕士学位论文全文数据库 信息科技辑》;第2021年卷(第05期);I138-1463 *
基于深度学习的校园网络舆情研究;尚煜;《中国优秀硕士学位论文全文数据库 社会科学Ⅱ辑》;第2023年卷(第01期);H127-159 *

Also Published As

Publication number Publication date
CN116432663A (en) 2023-07-14

Similar Documents

Publication Publication Date Title
CN111897933B (en) Emotion dialogue generation method and device and emotion dialogue model training method and device
CN111966800B (en) Emotion dialogue generation method and device and emotion dialogue model training method and device
CN112214604A (en) Training method of text classification model, text classification method, device and equipment
CN111161740A (en) Intention recognition model training method, intention recognition method and related device
CN111930914B (en) Problem generation method and device, electronic equipment and computer readable storage medium
CN112528637B (en) Text processing model training method, device, computer equipment and storage medium
CN112307168A (en) Artificial intelligence-based inquiry session processing method and device and computer equipment
CN116432663B (en) Controllable diversity professional text generation method and system based on element diagram
CN110705273B (en) Information processing method and device based on neural network, medium and electronic equipment
CN112364148B (en) Deep learning method-based generative chat robot
CN113779310A (en) Video understanding text generation method based on hierarchical representation network
CN114880444A (en) Dialog recommendation system based on prompt learning
CN116975288A (en) Text processing method and text processing model training method
CN114048301B (en) Satisfaction-based user simulation method and system
CN115858756A (en) Shared emotion man-machine conversation system based on perception emotional tendency
CN114416948A (en) One-to-many dialog generation method and device based on semantic perception
CN113421551A (en) Voice recognition method and device, computer readable medium and electronic equipment
CN117592564A (en) Question-answer interaction method, device, equipment and medium
CN111046134B (en) Dialog generation method based on replier personal characteristic enhancement
CN112364659A (en) Unsupervised semantic representation automatic identification method and unsupervised semantic representation automatic identification device
CN115525743B (en) Man-machine interaction method based on self-learning whitening network and electronic equipment
CN116958738A (en) Training method and device of picture recognition model, storage medium and electronic equipment
CN116680369A (en) Co-emotion dialogue generation method and system
Thorat et al. Improving conversation modelling using attention based variational hierarchical RNN
CN112434143B (en) Dialog method, storage medium and system based on hidden state constraint of GRU (generalized regression Unit)

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant