CN117313709A - Method for detecting generated text based on statistical information and pre-training language model - Google Patents

Abstract

The invention relates to the technical field of generated text detection, and discloses a generated text detection method based on statistical information and pre-trained language models. Through a detection model composed of a statistical learning model, a deep learning model and a dynamic fusion framework, the generated text is detected. Category label; the construction method of the detection model includes: building a statistical learning model; building a deep learning model; building a dynamic fusion framework; based on the training data set, by calculating the cross-entropy loss about the category label probability distribution obtained by dynamic fusion and the real category label function to train the detection model. The statistical learning model effectively alleviates the problem of poor model transferability in the case of limited multi-domain annotation data. The deep learning model gets rid of the problem of manually designing features and can extract more implicit features. The dynamic fusion framework loses less detection effect. Under the premise, the migration ability of the model is improved.

Description

A generative text detection method based on statistical information and pre-trained language models

Technical field

The present invention relates to the technical field of generated text detection, and in particular to a generated text detection method based on statistical information and pre-trained language models.

Background technique

With the development of large-scale language models, the text they generate is getting closer and closer to human writing. But it also raises serious security concerns, namely that machine-generated text could be used to maliciously mislead people. Generated text detection systems aim to distinguish whether text is generated by machines or humans, and has become a research hotspot in the field of natural language processing in recent years. Although statistical learning models do not require a large amount of annotated data for training and are easy to migrate to new fields, their detection accuracy is often low. The deep learning model can automatically extract features, avoiding the inconvenience and effect dependence caused by manual design rules and features, and can extract more implicit features and achieve better detection results. However, training these models requires a large amount of annotated data in the domain, and when migrating to new domains, the detection effect will drop significantly. However, in many real-life scenarios, obtaining high-quality annotation data in multiple fields is usually time-consuming and labor-intensive. Therefore, how to build a high-performance generated text detection system with limited resources and data has become a major challenge.

Considering that the statistical learning model has strong transferability but poor effect and the deep learning model has poor transferability, the present invention hopes to combine the statistical learning model and the deep learning model to solve the problem of poor detection effect of generated text in multiple fields.

Contents of the invention

In order to solve the above technical problems, the present invention provides a generated text detection method based on statistical information and pre-trained language model. Statistical features such as perplexity and word frequency are obtained through the language model, and the deep features of the text are extracted through the deep learning model, and then the text is detected. The prediction results of statistical features and depth features are probabilistically calibrated respectively, and finally dynamic fusion prediction of generated text is achieved.

In order to solve the above technical problems, the present invention adopts the following technical solutions:

A generated text detection method based on statistical information and pre-trained language models. Through a detection model composed of a statistical learning model, a deep learning model and a dynamic fusion framework, the category label of the generated text is detected; the training data used in the training of the detection model The collection is recorded as ,/> Corresponding tag set/> , and/> , is a collection of tags,/> is the length of the training data set,/> Yes/> Corresponding category label; text/> is a sequence of words ,/> Represents the No./> text/> No./> in words,/> for text/> length;

Methods for building detection models include:

Step 1: Build a statistical learning model:

The statistical learning model uses an autoregressive language model; the autoregressive language model is used to obtain the generation probability of each word in the text to be detected. , counting the number of words in the text that appear in the top ten words in the vocabulary , the number of the first one hundred/> , the number of the first thousand/> ;Based on the generation probability of each word/> Compute text/> Probability/> , according to/> Compute text/> The degree of confusion/> ;Will ,/> ,/> The perplexity of the text is used as a statistical feature, and through the logistic regression classifier, the probability distribution of the category label predicted based on the statistical features of the text to be detected/> ;

Step 2: Build a deep learning model:

The deep learning model uses an auto-encoding language model. After the text to be detected is encoded by the auto-encoding language model, the starting symbol [CLS] of the text is represented by a vector. As the semantic representation of the entire text, the class label probability distribution of the text to be detected based on deep coding feature prediction is obtained through the fully connected network and the classifier network/> ;

Step 3: Build a dynamic fusion framework:

Use label smoothing to convert the original one-hot labels from The value range of ,/> Is a constant representing the degree of smoothness, adding the true probability distribution of the predicted category after label smoothing/> for:

;

in Represents the category labels predicted by the statistical learning model and the deep learning model. The loss functions of the statistical learning model and the deep learning model use the cross-entropy loss function, so here/> Used to uniformly refer to the category labels predicted by the two models, or they can also be used separately/> ,/> express. /> is the real category label, and K represents the total number of category labels; then the cross-entropy loss function of the detection model is:

;

is the original cross-entropy loss of the logistic regression classifier and classifier network, and finally through dynamic fusion, the class label probability distribution predicted and dynamically fused based on the two features/> ：/> ;wherein,/> ,/> All are weight parameters;

Step 4: Based on the training data set, calculate the and/> The cross entropy loss function/> to train the detection model.

Furthermore, in step 1, based on the generation probability of each word Compute text/> The probability hour:

;

in, represents conditional probability.

Further, in step one, according to Compute text/> The degree of confusion/> hour:

.

Further, in step one, the ,/> ,/> With the perplexity of the text as a statistical feature, through the logistic regression classifier, the probability distribution of the class label predicted by the text based on the statistical features is obtained. hour:

;

in, is a logistic regression classifier,/> Represents the splicing operation.

Further, in step 2, the vector of the starting character [CLS] of the text is represented As the semantic representation of the entire text, and through the fully connected network and the classifier network, the class label probability distribution of the text to be detected based on deep coding feature prediction is obtained/> hour:

;

in, is the activation function of the classifier network,/> It is a fully connected network,/> is the bias parameter.

Compared with the prior art, the beneficial technical effects of the present invention are:

The detection model in the present invention includes three parts: a statistical learning model, a deep learning model and a dynamic fusion framework; the statistical learning model provides statistical features, which effectively alleviates the problem of poor model transferability when multi-domain annotation data is limited. Deep learning models get rid of the problem of manually designing features and can extract more implicit features. The pre-trained language model relies on its powerful encoding capabilities to provide potential inter-word association features, improving the detection effect of the model. On the one hand, the dynamic fusion framework uses label smoothing to perform probability calibration on the model and converts the probability predicted by the model into the real probability. On the other hand, it also combines the respective advantages of the statistical learning model and the deep learning model to achieve less loss of detection effect. This greatly improves the migration ability of the model, achieves good detection results in new fields, and has a very broad application prospect.

Description of drawings

Figure 1 is a schematic diagram of a detection model in an embodiment of the present invention.

Detailed ways

A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

In the present invention, the training data set , the corresponding label set/> , and/> , tag collection/> ,/> Indicates human beings,/> Indicates the machine,/> is the length of the training data set. text/> is a sequence of words/> ,/> Represents No./> text/> No./> in words,/> for/> length. The goal of the task is to learn a pass/> to predict the correct category label/> function/> .

The detection model proposed by this invention is shown in Figure 1 and includes the following three parts: (1) statistical learning model; (2) deep learning model; (3) dynamic fusion framework.

(1) Statistical learning model

The main body of the statistical learning model uses autoregressive language models such as GPT-2, because the generation process of the autoregressive language model can better simulate the process of human text generation. These models use an autoregressive approach to progressively generate semantically coherent text by predicting the next word or token based on previously generated words or tokens. Since language models tend to sample words with higher probability of generation, the words selected by humans are more random. Therefore, choose a language model such as GPT-2 to obtain the generation probability of each word , generation probability/> Indicates given before/> under the condition of words/> The predicted probability distribution of words, and the number of words in the statistical text that appear in the top ten, top one hundred, and top one thousand in the vocabulary, respectively, are expressed as/> ,/> ,/> .

First calculate each text based on the generation probability of each word Probability/> :

;

Calculate the perplexity of each text from this :

;

Using the obtained statistical results of word ranking and the perplexity of the text as statistical features, through a logistic regression classifier, the probability distribution of the category label predicted based on the statistical features of the input text is obtained :

;

in is a logistic regression classifier,/> Represents the splicing operation.

(2) Deep learning model

The main body of the deep learning model uses auto-encoding language models such as BERT instead of auto-regressive language models. This is because auto-encoding language models usually perform better on language understanding tasks. After encoding by language models such as BERT, the vector representation of the text's starting symbol [CLS] is As a semantic representation of the entire text, the class label probability distribution of the input text predicted based on deep encoding features is then obtained through the fully connected network and the classifier network/> :

;

in is the activation function of the classifier network,/> It is a fully connected network,/> is the bias parameter.

(3) Dynamic fusion framework

When performing a binary classification task, we usually only care about whether the output of the model is greater than a certain threshold, and do not care about the confidence level. However, in the field of generative text detection, the measure of confidence is equally important. The purpose of model calibration is to keep the model predicted probability consistent with the real empirical probability, that is, the model predicted probability is as close as possible to the real probability. The present invention uses label smoothing to convert the original one-hot labels from The value range of is extended to a larger range, namely/> , of which/> is a small number indicating the degree of smoothness. Add the true probability distribution of the predicted category after label smoothing/> becomes:

.

in Represents the category labels predicted by statistical learning models and deep learning models, /> is the real category label, K represents the total number of categories, and K=2 in this embodiment. Cross entropy loss function/> Change to:

.

is the original cross-entropy loss of the logistic regression classifier and classifier network, and finally through dynamic fusion, the final class label probability distribution predicted and dynamically fused based on the two features/> :

;

in, ,/> , and/> ,/> and/> Controls the weight of each input probability distribution by adjusting/> and/> size for best results.

Based on the training data set, by calculating about and/> The optimized cross-entropy loss function/> to train the detection model.

It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as illustrative and non-restrictive from any point of view. The scope of the present invention is defined by the appended claims rather than the above description, and it is therefore intended that all claims falling within the claims All changes within the meaning and scope of equivalent elements are encompassed by the present invention, and any reference signs in a claim should not be construed as limiting the claim in question.

In addition, it should be understood that although this specification is described in terms of implementations, not each implementation only contains an independent technical solution. This description of the description is only for the sake of clarity, and those skilled in the art should take the description as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementations that can be understood by those skilled in the art.

Claims (5)

1. A generated text detection method based on statistical information and pre-trained language models. Through a detection model composed of a statistical learning model, a deep learning model and a dynamic fusion framework, the category label of the generated text is detected; the detection model is trained using The training data set is recorded as ,/> Corresponding tag set/> , and/> ,/> is a collection of tags,/> is the length of the training data set,/> Yes/> Corresponding category label; text/> is a sequence of words ,/> Represents the No./> text/> No./> in words,/> for text/> length; Methods for building detection models include: Step 1: Build a statistical learning model: The statistical learning model uses an autoregressive language model; the autoregressive language model is used to obtain the generation probability of each word in the text to be detected. , counting the number of words in the text that appear in the top ten in the vocabulary/> , the number of the first one hundred/> , the number of the first thousand/> ;Based on the generation probability of each word Compute text/> Probability/> , according to/> Compute text/> The degree of confusion/> ;Will ,/> ,/> The perplexity of the text is used as a statistical feature, and through the logistic regression classifier, the probability distribution of the category label predicted based on the statistical features of the text to be detected/> ; Step 2: Build a deep learning model: The deep learning model uses an auto-encoding language model. After the text to be detected is encoded by the auto-encoding language model, the starting symbol [CLS] of the text is represented by a vector. As the semantic representation of the entire text, the class label probability distribution of the text to be detected based on deep coding feature prediction is obtained through the fully connected network and the classifier network/> ; Step 3: Build a dynamic fusion framework: Use label smoothing to convert the original one-hot labels from The value range of ,/> Is a constant representing the degree of smoothness, adding the true probability distribution of the predicted category after label smoothing/> for: ; in Represents the category labels predicted by statistical learning models and deep learning models, /> is the real category label, and K represents the total number of category labels; then the cross-entropy loss function of the detection model is: ; is the original cross-entropy loss of the logistic regression classifier and classifier network, and finally through dynamic fusion, the class label probability distribution predicted and dynamically fused based on the two features/> ：/> ;wherein,/> ,/> All are weight parameters; Step 4: Based on the training data set, calculate the and/> The cross entropy loss function/> to train the detection model. 2. The generated text detection method based on statistical information and pre-trained language model according to claim 1, characterized in that, in step one, based on the generation probability of each word Compute text/> Probability/> hour: ; in, represents conditional probability. 3. The generated text detection method based on statistical information and pre-trained language model according to claim 1, characterized in that, in step one, according to Compute text/> The degree of confusion/> hour: . 4. The generated text detection method based on statistical information and pre-trained language model according to claim 1, characterized in that, in step one, ,/> ,/> With the perplexity of the text as a statistical feature, through the logistic regression classifier, the probability distribution of the class label predicted based on the statistical features of the text/> hour: ; in, is a logistic regression classifier,/> Represents the splicing operation. 5. The generated text detection method based on statistical information and pre-trained language model according to claim 1, characterized in that, in step two, the vector of the starting symbol [CLS] of the text is represented As a semantic representation of the entire text, the class label probability distribution of the text to be detected based on deep coding feature prediction is obtained through the fully connected network and the classifier network. hour: ; in, is the activation function of the classifier network,/> It is a fully connected network,/> is the bias parameter.