CN116629272A - Text generation method and system controlled by natural language - Google Patents
Text generation method and system controlled by natural language Download PDFInfo
- Publication number
- CN116629272A CN116629272A CN202310910978.0A CN202310910978A CN116629272A CN 116629272 A CN116629272 A CN 116629272A CN 202310910978 A CN202310910978 A CN 202310910978A CN 116629272 A CN116629272 A CN 116629272A
- Authority
- CN
- China
- Prior art keywords
- encoder
- text
- semantic
- descriptive
- control variable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000009826 distribution Methods 0.000 claims description 48
- 239000013598 vector Substances 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 8
- 230000014509 gene expression Effects 0.000 abstract description 7
- 238000003058 natural language processing Methods 0.000 abstract description 2
- 230000008451 emotion Effects 0.000 description 20
- 238000005070 sampling Methods 0.000 description 5
- 230000010354 integration Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000013604 expression vector Substances 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Data Mining & Analysis (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Machine Translation (AREA)
Abstract
The invention discloses a text generation method and a text generation system controlled by natural language, which belong to the technical field of natural language processing. Meanwhile, the invention enhances the representation capability of the hash code to the core semantics of the descriptive control variable through two processes of encoding the descriptive control variable into the hash code and reconstructing the descriptive variable through the hash code, and simultaneously enables the descriptive control variable to contain the diversified expression of the descriptive control variable.
Description
Technical Field
The invention discloses a text generation method and a system for natural language control, and belongs to the technical field of natural language processing.
Background
Controllable text generation refers to the generation of natural language text meeting semantic requirements under given constraint conditions, and has important practical value. In recent years, research into controllable text generation has gained prominence in emotion, style, and the like. However, most existing models cannot meet the flexible and diverse control requirements for text contents in real application scenes aiming at category variables such as 'positive', 'negative', and the like and related to external attributes of the text, and people tend to express constraints for generating the text rather than category variables by using natural language. We define a text generation task for natural language control to generate text meeting semantic constraints with the natural language describing the semantic requirements, called descriptive control variables in this invention, as control variables with brief descriptions of characters, scenes, and to generate novel segments meeting given characters and scenes, as in a machine auto-authoring task.
Text generation under descriptive control variable constraints faces two major challenges due to the inherent features of natural language expression diversity, semantic ambiguity, etc.: firstly, descriptive control variables are free and flexible in content and form and are difficult to map to a fixed category, so that a search is needed for how to acquire the representation of the descriptive control variables, thereby accommodating the diversity of the descriptive control variables and distinguishing semantic core differences; and secondly, how to establish association between descriptive control variables and generated text so as to realize control of the content.
In view of the above technical problems, the following patent documents are disclosed in the prior art:
the Chinese patent document CN115600582A proposes a controllable text generation method based on a discriminant, which takes a pre-training model as a basis to generate a framework, and respectively introduces a theme discriminant, an emotion discriminant and a writing style discriminant. In the text generation process, the pre-training language model is used for generating the prediction probability of the candidate word, and the three classes of discriminators are used for predicting the attribute probability of the candidate word conforming to the given attribute according to the generated text, and then the final candidate word sampling probability is obtained by accumulating the attribute probability and the prediction probability.
The Chinese patent document CN114510924A proposes a text generation method based on a pre-training language model, and solves the problem that the existing controllable generation method based on weighted decoding can not achieve the control of ideal style strength on the premise of ensuring the quality of generated text by introducing a weighted decoding framework capable of dynamically adjusting the weight of a controller. In the invention, the controller is used for changing the vocabulary distribution output by the pre-training model so that the whole generated text accords with a specific style. If the current decoding step is not suitable for generating words containing the target style, the method disables the controller by adjusting the weights, thereby selecting words output by the pre-training model.
The method takes a limited number of category variables as control conditions, has a limited application scene, and is difficult to be applied to the text generation requirement taking natural language as the control variable in a real scene.
Disclosure of Invention
The invention discloses a text generation method controlled by natural language.
The invention also discloses a system for realizing the generation method.
Summary of the invention:
the invention introduces a text semantic hash method aiming at a controllable text generation scene under descriptive control variable constraint and aiming at the problem of how to effectively represent the free and flexible descriptive control variable, and uses a hash code to represent the descriptive control variable. Meanwhile, the invention enhances the representation capability of the hash code to the core semantics of the descriptive control variable through two processes of encoding the descriptive control variable into the hash code and reconstructing the descriptive variable through the hash code, and simultaneously enables the descriptive control variable to contain the diversified expression of the descriptive control variable.
The invention is based on a generating framework based on a variable self-encoder, and a conditional encoder is added to establish the association between descriptive control variables and inherent generating factors of texts, namely hidden variables. The overall generation framework comprises:
first, a semantic hash self-encoder encodes descriptive control variables first, resulting in hash codes, and then reconstructs the input descriptive control variables based on the hash codes. The hash code is used as a representation of descriptive control variables for controlling text generation in a subsequent process.
And secondly, a controllable text variation self-encoder is additionally introduced into a conditional encoder based on a variation self-encoder structure to realize the control of descriptive control variables on a text generation process. In the training stage, a controllable text variation self-encoder takes a descriptive control variable and a text conforming to the variable as input, and a controllable text generation process under the constraint of a text modeling control variable conforming to the descriptive variable is reconstructed; and (3) a using stage, namely, given descriptive control variables, encoding to obtain corresponding hash codes, and generating texts conforming to the semantics of the control variables by a hash code constraint text generation process.
Technical term interpretation:
1. descriptive control variable: a piece of natural language text describing the semantic control requirements.
2. Text semantic hashing: according to text semantics, the text is mapped into binary hash codes, and meanwhile, the retrieval method of the similarity of the text is reserved.
3. Variable self-encoder, english Variational Auto Encoder: is a generating network structure based on the variational Bayesian inference, comprising two main parts of an encoder and a decoder, and is commonly used for modeling the generation process from a text intrinsic generation factor to a text, wherein the text intrinsic generation factor is called an hidden variable in a variational self-encoder. Variable self-encoders train neural network encoders and decoders by maximizing the probability of generation of all samples in a dataset, a common framework in the field of text generation.
The technical scheme of the invention is as follows:
a method for generating text in natural language control, comprising:
the generation method is executed based on a semantic hash self-encoder and a controllable text variation self-encoder;
the semantic hash self-encoder includes: the semantic encoder and the semantic decoder are respectively used for encoding and reconstructing the descriptive control variable to obtain a hash code corresponding to the descriptive control variable;
the controllable text variation self-encoder comprises: the text encoder, the conditional encoder and the text decoder, wherein the controllable text variation self-encoder encodes and reconstructs a text conforming to descriptive variation semantics under descriptive control variable constraint so as to model a text generation process under descriptive variation control, and the method specifically comprises the following steps of:
s1: the descriptive control variable is encoded and reconstructed from the encoder using semantic hashing,
the semantic hash self-encoder encodes descriptive control variables through a semantic encoderAcquisition->Hash code of bit->：
(1.)
In equation (1), the hash codeFor expressing descriptive control variables +.>Core semantics of (2); />Is a semantic encoder;
semantic decoder is based on hash codesReconstructing descriptive control variables:
(2.)
in the formula (2) of the present invention,a reconstructed descriptive control variable; />Is a semantic decoder;
the mode of coding and reconstructing descriptive control variables described in the formula (1) and the formula (2) is an unsupervised learning method, so that the semantic hash self-encoder can be pre-trained on a large-scale corpus, and the coding and decoding capacity of the semantic encoder and the semantic decoder on the descriptive control variables is further improved;
when the descriptive control variable is encoded, the descriptive control variable comprises a text form, a TF-IDF vector, an English Term Frequency-Inverse Document Frequency or a word bag vector and other various expression modes;
s2: modeling text generation under descriptive control variable constraints using controllable text variation from an encoder,
the common variable self-encoder comprises a text encoder and a text decoder, which are used for encoding and reconstructing an input text respectively, but because of lack of integration of control variables, the text generation process under the constraint of the control variables is difficult to model, and therefore, the invention adds a condition encoder on the basis of the variable self-encoder, and models the control of descriptive control variables on the text generation process;
the condition encoder uses descriptive control variablesCorresponding hash code->For input, coding to obtain a hidden variable a priori distribution +.>Wherein->To meet distribution +.>Is used for representing implicit generation factors of the text;to->Mean value (S),>is a normal distribution of variance, ++>Is a unit matrix; />And->Obtained by the following formula:
(3.)
in the formula (3) of the present invention,representing a condition encoder;
the text encoder uses descriptive control variablesAnd meet->Text of semantics->For input, the coding gets the descriptive control variable +.>And text->Hidden variable posterior distribution under conditions +.>Wherein->To meet distribution +.>Hidden variables of (a); />To->Mean value (S),>is a normal distribution of variance, ++>Is a unit matrix; />And->Obtained by the following formula:
(4.)
in the formula (4) of the present invention,representing a text encoder;
the text decoder self-distributes in samplesHidden variable +.>For input, a reconstructed text is generated>：
(5.)
In the formula (5) of the present invention,representing a text decoder;
s3: model training, namely pre-training a semantic hash self-encoder by using a large-scale corpus, and then training the whole model by using a target corpus until the model converges so as to generate a text conforming to the semantic constraint of the descriptive control variable under the condition that a section of descriptive control variable is given:
s31: pre-training a semantic hash self-encoder by using a large-scale corpus to obtain a hash code containing semantic information;
the semantic hash self-encoder takes descriptive control variables as input, the descriptive control variables can be in a text form, or in a form that the text corresponds to TF-IDF vectors, the descriptive control variables of the input are reconstructed as targets, and the difference between the control variables of the input and the reconstructed control variables is measured by using cross entropy;
s32: using a target corpus fine-tuning semantic hash self-encoder and training a controllable text variation self-encoder; after S31, using the descriptive control variable in the target corpus as the input of the semantic hash self-encoder, using the reconstructed input control variable as the target, and using the cross entropy loss function fine tuning semantic hash self-encoder to obtain the domain knowledge of the target corpus, thereby better improving the coding performance of the descriptive control variable in the target domain;
the input of the controllable text variation self-encoder is descriptive control variable in the target corpusAnd its corresponding hash code ++>And text->Wherein the input of the condition encoder is a hash code +.>The text encoder input isDescriptive control variable->And text fitting the control variable semantics +.>The method comprises the steps of carrying out a first treatment on the surface of the Controllable text variation self-encoder to maximize reconstructed text +.>The distance between the simultaneous minimization of the prior distribution and the posterior distribution of the hidden variables is targeted, in particular expressed as maximizing the objective function +.>：
(6.)
In maximizing the objective function shown in equation (6),representing text decoder basis +.>Reconstruction->Probability of->Sample from text encoder pair->Coding the resulting hidden variable posterior distribution +.>The purpose of maximizing the first term of the formula is to enable the hidden variable to reconstruct the input text efficiently +.>；/>For conditional encoder pair->Encoding the obtained prior distribution of hidden variables, ++>Representing the inter-distributionKLDistance, english Kullback-Leibler Divergence, the purpose of minimizing the second term of the formula is to constrain the distance of the hidden variable a priori distribution and posterior distribution.
At the usage stage, the controllable text generation process involves a semantic encoder in the semantic hash self-encoder, a conditional encoder in the controllable text variation self-encoder, and a text decoder:
given a descriptive control variableThe hash codes are obtained through the semantic encoder in the semantic hash self-encoder, and the condition encoder in the controllable text variation self-encoder encodes the hash codes to obtain priori distribution ++>Wherein->Is to->Mean value, & gt>Is a normal distribution of variance; />Is a unit matrix; then in a priori distribution->Sample one +.>，/>Is input to a text decoder to generate text conforming to the semantics of the descriptive control variable, in the course of which, for the same descriptive control variable, the +.>Middle sampling multiple +.>To generate a plurality of texts conforming to descriptive control variables, further improving the diversity of the generated texts.
A system for implementing a text generation method for natural language control, characterized by:
the system comprises a semantic hash self-encoder and a controllable text variation self-encoder;
the semantic hash self-encoder is configured to: in the training stage, a hash code capable of representing the core semantics of the control variable is obtained through encoding and reconstructing the descriptive control variable; in the use stage after training is completed, the contained semantic encoder encodes descriptive control variables to obtain hash codes;
the controllable text variation self-encoder is used for: in a training stage, a text conforming to descriptive control variable constraint is encoded and reconstructed under hash code constraint, and a text generation process under the hash code constraint is modeled; in the use stage after training is completed, a hash code from a semantic hash self-encoder is received, the hash code is encoded by a conditional encoder to obtain hidden variables, and then a text decoder depends on the hidden variables to generate texts conforming to descriptive control variables.
The invention has the advantages that:
1. the invention provides a text generation method for realizing natural language control, which is oriented to wider controllable application scene requirements. Aiming at the defect that the prior method takes category variables as control conditions and can only control external attributes of texts such as emotion, text length and tense generally and cannot meet the diversified control requirements of real scenes, the invention takes descriptive control variables in natural language form as control conditions, allows users to freely express the control requirements, and improves the flexibility and practicability of the text generation method.
2. The invention introduces a semantic hash method to realize the efficient coding of descriptive control variables. The method is based on a semantic hash method, and the hash codes of the control variables are obtained through encoding and reconstruction of the control variables, so that the hash codes can represent the core semantics of the text, meanwhile, the diversity of languages is contained, and the encoding quality and encoding efficiency are improved.
3. According to the invention, the variable self-encoder is used as a basic generation frame, and the constraint of the control variable on the text is fused by the conditional encoder, so that the association relation between the control variable and the generated text can be learned, the text controllability is realized, meanwhile, the diversity text conforming to the constraint can be generated through diversity sampling, and the higher-level diversity requirement of the controllable text generation field is further met.
Drawings
FIG. 1 is a block diagram of a natural language controlled text generation method of the present invention;
FIG. 2 is a schematic representation of the process of the present invention during a use phase to generate text conforming to descriptive control variables given the control variables.
Detailed Description
The invention will be described in more detail below with reference to examples and figures of the description, it being apparent that the invention can be embodied in many forms and is not limited to the examples set forth.
Example 1,
As shown in fig. 1, a text generation method controlled by natural language includes:
the generation method is executed based on a semantic hash self-encoder and a controllable text variation self-encoder;
the semantic hash self-encoder includes: the semantic encoder and the semantic decoder are respectively used for encoding and reconstructing the descriptive control variable to obtain a hash code corresponding to the descriptive control variable;
the controllable text variation self-encoder comprises: text encoder, condition encoder and text decoder. The controllable text variation self-encoder encodes and reconstructs a text conforming to descriptive variable semantics under descriptive control variable constraints, so that a text generation process under descriptive variable control can be modeled, and the method specifically comprises the following steps:
s1: the descriptive control variable is encoded and reconstructed from the encoder using semantic hashing,
the semantic hash self-encoder encodes descriptive control variables through a semantic encoderAcquisition->Hash code of bit->：
（1）
In equation (1), the hash codeFor expressing descriptive control variables +.>Core semantics of (2); />Is a semantic encoder;
semantic decoder is based on hash codesReconstructing descriptive control variables:
（2）
in the formula (2) of the present invention,a reconstructed descriptive control variable; />Is a semantic decoder;
the method for encoding and reconstructing the descriptive control variable per se described in the formula (1) and the formula (2) enables the semantic hash self-encoder to be pre-trained on a large-scale corpus, and can further improve the encoding and decoding capacities of the encoder and the decoder on the control variable;
when the descriptive control variable is coded, the descriptive control variable is converted into a TF-IDF vector, and English Term Frequency-Inverse Document Frequency is used as input of a semantic coder;
in S1, specifically, the semantic encoder in the semantic hash self-encoder uses the MLP network, english Multi-Layer permission as the base network, for a given descriptive control variableObtaining corresponding TF-IDF and English Term Frequency-Inverse Document Frequency vectors based on target corpus, and then obtaining m-dimensional feature vectors (namely, the M-dimensional feature vectors) by the MLP network through dimension reduction of the vectors>Finally, pair->Performing a binarization operation on each dimension of the block to obtain a corresponding hash code +.>：
(7)
(8)
In the formulas (7) and (8),to get->Operation of corresponding TF-IDF vectors; />Representing a two-step process; />Is a sigmoid function; />And->Respectively represent feature vector +>And Hash->Is>Dimension;
the semantic decoder takes the MLP network as a basic structure and hash codesReconstructing a TF-IDF vector corresponding to the descriptive control variable as an input;
in theory, semantically similar descriptive control variables have similar TF-IDF expression vectors, and in the steps, the encoding and reconstruction operation of the descriptive control variables ensures that semantically similar control variables have similar hash codes; the binary operation enables the hash code to retain text core semantics and ignores irrelevant details; the TF-IDF vector reconstructed by the hash code represents a class of descriptive control variables with similar semantics and different expressions;
s2: modeling text generation under descriptive control variable constraints using controllable text variation from an encoder,
the common variational self-encoder comprises a text encoder and a text decoder which are respectively used for encoding and reconstructing an input text, but because of lack of integration of control variables, the text generation process under the constraint of the control variables is difficult to model, and therefore, the invention adds a conditional encoder on the basis of the variational self-encoder, and models the control of descriptive control variables on the text generation process;
controllable text variation the text encoder in the self-encoder takes bert+mlp as a base network: first, a special character is formedAnd descriptive control variable->And text corresponding to the control variable +.>Splicing to be used as input of the BERT network; the BERT network codes the input text to obtain the feature vector corresponding to each vocabulary in the text and special characters ++>Corresponding feature vector>As a characteristic representation of the entire input text, the MLP network then follows +.>Obtaining hidden variable->Posterior distribution of (2)Wherein->To->Mean value (S),>is a normal distribution of variance, ++>Is a unit matrix; />And->Is obtained by the following formula:
(9)
(10)
the condition encoder uses the MLP network as a basic network and uses the semantic encoder to control the descriptive variableEncoding the resulting hash code +.>For input, coding to obtain a hidden variable a priori distribution +.>Wherein->Is mean->Variance->Normal distribution of (c):
(11)
the text decoder uses GPT (generated Pre-Training Transformer) as a basic network, and in the training stage, the text decoder uses the posterior distribution of the hidden variablesHidden variable +.>For input, reconstruct the input text +.>. In the use phase, in order to distribute a priori the +.>Sampling hidden variable +.>Generating text conforming to descriptive control variable semantics for input; the network such as BERT, GPT, MLP used by different parts in the model frame is an independent network, and parameters are not shared;
s3: model training, namely training the whole model by using a large-scale corpus pre-training semantic hash self-encoder and using a target corpus until convergence so as to generate a text conforming to the semantic constraint of a descriptive control variable under the condition of giving a section of descriptive control variable:
s31: pre-training a semantic hash self-encoder by using a large-scale corpus to obtain a hash code containing semantic information;
the aim of the pre-training semantic hash self-encoder is to encode more effectively, and the corpus used in the training process can be irrelevant to the target corpus; in a specific training process, the aim is to minimize the cross entropy between the reconstructed control variable and the input control variable; in order to enhance the field suitability of the semantic hash self-encoder, the self-encoder is further refined based on the target corpus;
the semantic hash self-encoder takes descriptive control variables as input, the descriptive control variables can be in a text form, or in a form that the text corresponds to TF-IDF vectors, the descriptive control variables of the input are reconstructed as targets, and the difference between the control variables of the input and the reconstructed control variables is measured by using cross entropy;
s32: using a target corpus fine-tuning semantic hash self-encoder and training a controllable text variation self-encoder; after S31, using the descriptive control variable in the target corpus as the input of the semantic hash self-encoder, using the reconstructed input control variable as the target, and using the cross entropy loss function fine tuning semantic hash self-encoder to obtain the domain knowledge of the target corpus, thereby better improving the coding performance of the descriptive control variable in the target domain;
the input of the controllable text variation self-encoder is descriptive control variable in the target corpusText corresponding to the textWherein the input of the condition encoder is the descriptive control variable +.>Corresponding hash code->The input of the text encoder is the descriptive control variable +.>And text fitting the control variable semantics +.>The method comprises the steps of carrying out a first treatment on the surface of the Controllable text variation self-encoder to maximize reconstructed text +.>Is aimed at, in particular, minimizing the distance of the hidden variable a priori distribution and posterior distribution at the same timeExpressed as maximizing the objective function +.>：
（6）
In maximizing the objective function shown in equation (6), the purpose of maximizing the first term is to enable hidden variables to effectively reconstruct the input textThe purpose of minimizing the second term is to pull the distance of the prior distribution and posterior distribution of the hidden variable,/and->Representing text decoder basis +.>Reconstruction->Probability of (2); />Sample from text encoder pair->Coding the resulting hidden variable posterior distribution +.>，/>For conditional encoder pair->Encoding the obtained prior distribution of hidden variables, ++>Representing the space between two distributionsKLDistance, english Kullback-Leibler Divergence;
as shown in fig. 2, at the usage stage, the controllable text generation process involves a semantic encoder in the semantic hash self-encoder, a conditional encoder in the controllable text variation self-encoder, and a text decoder:
given a descriptive control variableThe hash codes are obtained through the semantic encoder in the semantic hash self-encoder, and the condition encoder in the controllable text variation self-encoder encodes the hash codes to obtain priori distribution ++>Wherein->Is to->Mean value, & gt>Is a normal distribution of variance; />Is a unit matrix; then distributed a prioriSample one +.>，/>Is input to a text decoder to generate text conforming to the semantics of the descriptive control variable, in the course of which, for the same descriptive control variable, the +.>Middle sampling multiple +.>To generate a plurality ofThe text conforming to the descriptive control variable further improves the diversity of the generated text.
In various application scenarios of a text generation method controlled by natural language described in this embodiment 1, for example, in the generation of comment text with controllable emotion, a user gives a description of emotion, such as "the commodity is perfect" or "the commodity is bad", and some content hint texts "the appearance, the texture and the purpose of the commodity", and the expected model can generate comment text meeting emotion and content requirements, where the emotion requirements are not only reflected in controllable emotion polarity, but also in controllable emotion implicit strength, such as "the commodity is perfect", "the commodity can also" and the like.
The emotion is divided into two categories, namely positive and negative, 0 or 1 is used for representing the two categories and is used as a control variable of a model, and the emotion is greatly different from the emotion expression of human polynomials, so that the emotion is difficult to apply to the comment generation task with controllable emotion.
Aiming at the application requirements, the emotion description and content prompt text provided by the user are combined and then used as descriptive control variables:
firstly, a large-scale multi-field comment corpus is used for pre-training a semantic hash self-encoder, the input of the semantic hash self-encoder is a comment text, a reconstructed comment text is output, and through the training process, the semantic hash self-encoder can acquire a hash code containing semantic information under the condition that a text is given;
then, training the whole model by using comment corpus in the target field, extracting some emotion-related keywords for each comment through emotion analysis, extracting some emotion-related keywords at the same time, and then splicing the emotion keywords and the aspect keywords to form descriptive control variables, wherein comment texts are used as target texts conforming to the semantics of the control variables. The descriptive control variable is used to train the semantic hash self-encoder again, while the descriptive control variable and the target text are used to train the controllable text variation self-encoder, the objective function used by the training process is given above;
after model training is converged, a trained model is used for generating comment texts with controllable emotion, firstly, a semantic encoder in a semantic hash self-encoder is used for encoding descriptive control variables provided by users to obtain hash codes, then, a conditional encoder in the controllable text variable self-encoder is used for encoding the hash codes to obtain hidden variable priori distribution, hidden variables are sampled in the distribution, and the hidden variable priori distribution is input into a text decoder to generate comment texts conforming to user description.
As another example, in automatically authoring such a scenario: the user gives a descriptive control variable such as "generate a text about self-driving travel to the Tibet, the mood is optimistically active", and the model generates text conforming to the control variable based on understanding the control variable.
The control variables have different expression methods, such as 'I need an article on self-driving to Tibet travel', 'write an article, the theme is self-driving to Tibet travel', and the like, and the flexible diversity of languages makes the existing intelligent model difficult to effectively model the relationship between the control variables and the generated text.
Aiming at the challenges, firstly, the invention uses a pre-trained semantic hash self-encoder of large-scale data such as hundred degrees encyclopedia and the like which are irrelevant to target corpus, thereby obtaining hash codes containing semantic information;
then, using the target corpus fine-tuning semantic hash self-encoder, and simultaneously, using the target corpus to train the controllable text variation self-encoder, wherein the training process and the target are already given above;
giving a descriptive control variable, encoding the descriptive control variable by using a fine-tuned semantic hash self-encoder, and acquiring a hash code of the descriptive control variable, wherein the hash code represents the semantic core of a text, and texts which synchronously comprise the semantics of 'Tibet', 'self-driving', 'active' and the like have the same hash code; the hash code is input to a conditional coder of a controllable text variable self-coder to obtain a hidden variable, and the hidden variable is input to a text decoder to generate a text conforming to the descriptive control variable.
EXAMPLE 2,
A system for implementing a natural language controlled text generation method as described in example 1, comprising a semantic hash self-encoder, a controllable text variation self-encoder;
the semantic hash self-encoder is configured to: in the training stage, a hash code capable of expressing the core semantics of the control variable is obtained through encoding and reconstructing the descriptive control variable; in the use stage after training is completed, the contained semantic encoder encodes descriptive control variables to obtain hash codes;
the controllable text variation self-encoder is used for: in a training stage, a text conforming to descriptive control variable constraint is encoded and reconstructed under hash code constraint, and a text generation process under the hash code constraint is modeled; in the use stage after training is completed, a hash code from a semantic hash self-encoder is received, the hash code is encoded by a conditional encoder to obtain hidden variables, and then a text decoder depends on the hidden variables to generate texts conforming to descriptive control variables.
Claims (2)
1. A method for generating text in natural language control, comprising:
the generation method is executed based on a semantic hash self-encoder and a controllable text variation self-encoder;
the semantic hash self-encoder includes: the semantic encoder and the semantic decoder are respectively used for encoding and reconstructing the descriptive control variable to obtain a hash code corresponding to the descriptive control variable;
the controllable text variation self-encoder comprises: the text encoder, the condition encoder and the text decoder encode and reconstruct the text conforming to the semantic meaning of the descriptive variable under the constraint of the descriptive control variable, and specifically comprise the following steps:
s1: the descriptive control variable is encoded and reconstructed from the encoder using semantic hashing,
the semantic hash self-encoder encodes descriptive control variables through a semantic encoderAcquisition->Hash code of bit->：
（1）
In equation (1), the hash codeFor expressing descriptive control variables +.>Core semantics of (2); />Is a semantic encoder;
semantic decoder is based on hash codesReconstructing descriptive control variables:
（2）
in the formula (2) of the present invention,a reconstructed descriptive control variable; />Is a semantic decoder;
when the descriptive control variable is encoded, the descriptive control variable is used as the input of a semantic encoder in the form of text, or the descriptive control variable is converted into a TF-IDF vector, english Term Frequency-Inverse Document Frequency, or a bag-of-word vector corresponding to the descriptive control variable is used as the input of the semantic encoder;
s2: modeling text generation under descriptive control variable constraints using controllable text variation from an encoder,
the condition encoder uses descriptive control variablesCorresponding hash code->For input, the encoding is performed with a hash code +.>Cryptovaria a priori distribution for conditions +.>Wherein->To meet distribution +.>Is used for representing implicit generation factors of the text; />To->Mean value (S),>is a normal distribution of variance, ++>Is a unit matrix; />And->Obtained by the following formula:
（3）
in the formula (3) of the present invention,representing a condition encoder;
the text encoder uses descriptive control variablesAnd the compliance with descriptive control variable->Text of semantics->For input, the coding gets the descriptive control variable +.>And text->Hidden variable posterior distribution under conditions +.>WhereinTo meet distribution +.>Hidden variables of (a); />To->Mean value (S),>is a normal distribution of variance, ++>Is a unit matrix; />And->Obtained by the following formula:
（4）
in the formula (4) of the present invention,representing a text encoder;
the text decoder uses hidden variablesFor input, a reconstructed text is generated>：
（5）
In the formula (5) of the present invention,representing a text decoder;
s3: model training to satisfy the condition of given a section of descriptive control variable, generating text conforming to semantic constraint of the descriptive control variable:
s31: pre-training a semantic hash self-encoder by using a large-scale corpus to obtain a hash code containing semantic information;
the semantic hash self-encoder takes descriptive control variables as input, takes reconstructed input descriptive control variables as targets, and measures differences between the input control variables and the reconstructed control variables by using cross entropy;
s32: using a target corpus fine-tuning semantic hash self-encoder and training a controllable text variation self-encoder; the data form of the target corpus is descriptive control variable and text conforming to the semantics of the control variable;
the input of the controllable text variation self-encoder is descriptive control variable in the target corpusAnd the corresponding text->Wherein the input of the condition encoder is the descriptive control variable +.>Corresponding hash code->The input of the text encoder is the descriptive control variable +.>And text fitting the control variable semantics +.>The method comprises the steps of carrying out a first treatment on the surface of the Controllable text variation self-encoder to maximize reconstructed text +.>Is aimed at, in particular maximizing the objective function, by minimizing the distance of the hidden variable a priori distribution and the posterior distribution at the same timeCount->：
(6)
In maximizing the objective function shown in equation (6), the purpose of maximizing the first term is to enable hidden variables to effectively reconstruct the input textThe purpose of minimizing the second term is to pull the distance of the prior distribution and posterior distribution of the hidden variable,/and->Representing text decoder basis +.>Reconstruction->Probability of (2); />Sample from text encoder pair->Coding the resulting hidden variable posterior distribution，/>For conditional encoder pair->Encoding the obtained prior distribution of hidden variables, ++>Representing the space between two distributionsKLA distance;
at the usage stage, the controllable text generation process involves a semantic encoder in the semantic hash self-encoder, a conditional encoder in the controllable text variation self-encoder, and a text decoder:
given a descriptive control variableThe hash codes are obtained through the semantic encoder in the semantic hash self-encoder, and the condition encoder in the controllable text variation self-encoder encodes the hash codes to obtain priori distribution ++>Wherein->Is to->Mean value, & gt>Is a normal distribution of variance; />Is a unit matrix; then in a priori distribution->Sample one +.>，/>Is input to a text decoder to generate text that conforms to the descriptive control variable semantics.
2. A system for implementing a natural language controlled text generation method as recited in claim 1, wherein:
the system comprises a semantic hash self-encoder and a controllable text variation self-encoder;
the semantic hash self-encoder is configured to: in the training stage, a hash code capable of capturing the core semantics of the control variable is obtained through encoding and reconstructing the descriptive control variable; in the use stage after training is completed, the contained semantic encoder encodes descriptive control variables to obtain hash codes;
the controllable text variation self-encoder is used for: in a training stage, a text conforming to descriptive control variable constraint is encoded and reconstructed under hash code constraint, and a text generation process under the hash code constraint is modeled; in the use stage after training is completed, a hash code from a semantic hash self-encoder is received, the hash code is encoded by a conditional encoder to obtain hidden variables, and then a text decoder depends on the hidden variables to generate texts conforming to descriptive control variables.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310910978.0A CN116629272B (en) | 2023-07-24 | 2023-07-24 | Text generation method and system controlled by natural language |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310910978.0A CN116629272B (en) | 2023-07-24 | 2023-07-24 | Text generation method and system controlled by natural language |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116629272A true CN116629272A (en) | 2023-08-22 |
| CN116629272B CN116629272B (en) | 2023-10-10 |
Family
ID=87603035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310910978.0A Active CN116629272B (en) | 2023-07-24 | 2023-07-24 | Text generation method and system controlled by natural language |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116629272B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108510559A (en) * | 2017-07-19 | 2018-09-07 | 哈尔滨工业大学深圳研究生院 | It is a kind of based on have supervision various visual angles discretization multimedia binary-coding method |
| US20190236139A1 (en) * | 2018-01-31 | 2019-08-01 | Jungle Disk, L.L.C. | Natural language generation using pinned text and multiple discriminators |
| CN111460176A (en) * | 2020-05-11 | 2020-07-28 | 南京大学 | Multi-document machine reading understanding method based on Hash learning |
| CN112199520A (en) * | 2020-09-19 | 2021-01-08 | 复旦大学 | Cross-modal Hash retrieval algorithm based on fine-grained similarity matrix |
| CN113190699A (en) * | 2021-05-14 | 2021-07-30 | 华中科技大学 | Remote sensing image retrieval method and device based on category-level semantic hash |
| CN113255295A (en) * | 2021-04-27 | 2021-08-13 | 西安电子科技大学 | Method and system for automatically generating formalized protocol from natural language to PPTL (Power Point language) |
| CN113392180A (en) * | 2021-01-07 | 2021-09-14 | 腾讯科技（深圳）有限公司 | Text processing method, device, equipment and storage medium |
| CN113821527A (en) * | 2021-06-30 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Hash code generation method and device, computer equipment and storage medium |
| CN114416948A (en) * | 2022-01-18 | 2022-04-29 | 重庆邮电大学 | One-to-many dialog generation method and device based on semantic perception |
| CN114610940A (en) * | 2022-03-15 | 2022-06-10 | 华南理工大学 | Hash image retrieval method based on local random sensitivity self-encoder |
| CN115525759A (en) * | 2022-09-01 | 2022-12-27 | 全知科技(杭州)有限责任公司 | HTTP flow abnormity detection method based on depth self-encoder |
-
2023
- 2023-07-24 CN CN202310910978.0A patent/CN116629272B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108510559A (en) * | 2017-07-19 | 2018-09-07 | 哈尔滨工业大学深圳研究生院 | It is a kind of based on have supervision various visual angles discretization multimedia binary-coding method |
| US20190236139A1 (en) * | 2018-01-31 | 2019-08-01 | Jungle Disk, L.L.C. | Natural language generation using pinned text and multiple discriminators |
| CN111460176A (en) * | 2020-05-11 | 2020-07-28 | 南京大学 | Multi-document machine reading understanding method based on Hash learning |
| CN112199520A (en) * | 2020-09-19 | 2021-01-08 | 复旦大学 | Cross-modal Hash retrieval algorithm based on fine-grained similarity matrix |
| CN113392180A (en) * | 2021-01-07 | 2021-09-14 | 腾讯科技（深圳）有限公司 | Text processing method, device, equipment and storage medium |
| CN113255295A (en) * | 2021-04-27 | 2021-08-13 | 西安电子科技大学 | Method and system for automatically generating formalized protocol from natural language to PPTL (Power Point language) |
| CN113190699A (en) * | 2021-05-14 | 2021-07-30 | 华中科技大学 | Remote sensing image retrieval method and device based on category-level semantic hash |
| CN113821527A (en) * | 2021-06-30 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Hash code generation method and device, computer equipment and storage medium |
| CN114416948A (en) * | 2022-01-18 | 2022-04-29 | 重庆邮电大学 | One-to-many dialog generation method and device based on semantic perception |
| CN114610940A (en) * | 2022-03-15 | 2022-06-10 | 华南理工大学 | Hash image retrieval method based on local random sensitivity self-encoder |
| CN115525759A (en) * | 2022-09-01 | 2022-12-27 | 全知科技(杭州)有限责任公司 | HTTP flow abnormity detection method based on depth self-encoder |
Non-Patent Citations (3)
| Title |
|---|
| KRISTIAN KOLTHOFF 等: "Automatic Generation of Graphical User Interface Prototypes from Unrestricted Natural Language Requirements", 《2019 34TH IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING (ASE)》, pages 1234 - 1237 * |
| 周长红 等: "复杂产品协同设计流程的多视图自然语言文本生成", 《计算机集成制造系统》, vol. 24, no. 7, pages 1838 - 1849 * |
| 胡宇 等: "一种基于参考规范的专业文本生成方法", 《中文信息学报》, vol. 37, no. 3, pages 152 - 163 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116629272B (en) | 2023-10-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108153913B (en) | Training method of reply information generation model, reply information generation method and device | |
| CN109190134B (en) | Text translation method and device | |
| Chen et al. | Deep Learning for Video Captioning: A Review. | |
| CN109582952B (en) | Poetry generation method, poetry generation device, computer equipment and medium | |
| CN110059324B (en) | Neural network machine translation method and device based on dependency information supervision | |
| CN111816156A (en) | Many-to-many voice conversion method and system based on speaker style feature modeling | |
| Gontier et al. | Automated audio captioning by fine-tuning bart with audioset tags | |
| CN111581962A (en) | Text representation method based on subject word vector and hybrid neural network | |
| CN111429894A (en) | Many-to-many speaker conversion method based on SE-ResNet STARGAN | |
| CN113407663B (en) | Image-text content quality identification method and device based on artificial intelligence | |
| Zhao et al. | Videowhisper: Toward discriminative unsupervised video feature learning with attention-based recurrent neural networks | |
| CN111429893A (en) | Many-to-many speaker conversion method based on Transitive STARGAN | |
| Elshaer et al. | Transfer learning from sound representations for anger detection in speech | |
| An et al. | Speech Emotion Recognition algorithm based on deep learning algorithm fusion of temporal and spatial features | |
| CN108363685B (en) | Self-media data text representation method based on recursive variation self-coding model | |
| CN116028846A (en) | Multi-mode emotion analysis method integrating multi-feature and attention mechanisms | |
| CN114911932A (en) | Heterogeneous graph structure multi-conversation person emotion analysis method based on theme semantic enhancement | |
| Mathur et al. | A scaled‐down neural conversational model for chatbots | |
| Bie et al. | RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model | |
| CN116629272B (en) | Text generation method and system controlled by natural language | |
| Li et al. | Curriculum prompt learning with self-training for abstractive dialogue summarization | |
| WO2023226239A1 (en) | Object emotion analysis method and apparatus and electronic device | |
| CN114239575B (en) | Statement analysis model construction method, statement analysis method, device, medium and computing equipment | |
| Zhang et al. | AI-powered text generation for harmonious human-machine interaction: current state and future directions | |
| CN115422329A (en) | Knowledge-driven multi-channel screening fusion dialogue generation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |