CN116776868B - Evaluation method of model generation text and computer equipment - Google Patents
Evaluation method of model generation text and computer equipment Download PDFInfo
- Publication number
- CN116776868B CN116776868B CN202311075044.6A CN202311075044A CN116776868B CN 116776868 B CN116776868 B CN 116776868B CN 202311075044 A CN202311075044 A CN 202311075044A CN 116776868 B CN116776868 B CN 116776868B
- Authority
- CN
- China
- Prior art keywords
- text
- fingerprint
- index
- model
- gene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011156 evaluation Methods 0.000 title claims abstract description 51
- 108090000623 proteins and genes Proteins 0.000 claims abstract description 49
- 238000012549 training Methods 0.000 claims abstract description 24
- 238000009826 distribution Methods 0.000 claims abstract description 12
- 238000000605 extraction Methods 0.000 claims abstract description 12
- 230000015556 catabolic process Effects 0.000 claims abstract description 6
- 238000006731 degradation reaction Methods 0.000 claims abstract description 6
- 238000005259 measurement Methods 0.000 claims abstract description 3
- 230000006870 function Effects 0.000 claims description 29
- 238000000034 method Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000012512 characterization method Methods 0.000 claims description 2
- 238000004519 manufacturing process Methods 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 6
- JEIPFZHSYJVQDO-UHFFFAOYSA-N iron(III) oxide Inorganic materials O=[Fe]O[Fe]=O JEIPFZHSYJVQDO-UHFFFAOYSA-N 0.000 description 4
- 238000013519 translation Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Abstract
The application discloses an evaluation method and computer equipment of a model generation text, which are independent of labels and are suitable for a production environment. The evaluation method is characterized in that after the three indexes of the gene, the readability and the fingerprint are respectively evaluated, comprehensive evaluation is carried out; wherein, the gene index is used for measuring the semantic relativity and homology of the model generation text and the input text, the readability index is used for measuring the degree of the model generated text which can be read by a person according to the sentence length average value and the text repeated degradation condition which are cut from the punctuation, and the fingerprint index is used for measuring the degree of consistency of semantic feature distribution of the model generated text and the training set label; specifically, a network model is extracted by training fingerprints based on a ternary twin network in advance, a fingerprint library is generated at the same time, then a model generation text is input into the fingerprint extraction network model to obtain fingerprints, and then distance measurement is carried out on the fingerprints and the fingerprint library to determine the value of a fingerprint index.
Description
Technical Field
The application belongs to the technical field of document data deep processing, and particularly relates to an evaluation method and computer equipment for model generation text.
Background
The patent deep processing is a patent rewriting technology for obtaining high added value by utilizing a text generation technology aiming at the characteristics of patent literature. At present, patent deep processing mainly comprises name deep processing, abstract deep processing, keyword indexing, IPC classification and the like, and patent abstract deep processing is that a long text is input into a trained model to generate abstract text (model generation text). Thus, it is necessary to perform a proper evaluation to provide an objective, uniform, quantitative measure for model-generated text.
Early model-generated text evaluations employed information retrieval metrics such as recall, precision, and F-value, compare model-generated text to manually written text, and measure the commonality between them. The main problem with this traditional evaluation method is that comparing model-generated text with single manually written text is too subjective.
Currently, standard metrics on which the digest system is developed are typically ROUGE and BLEU. The main task of BLEU (English full name: bilingual evaluation understudy, chinese term: bilingual evaluation candidate) is to compare the n-gram unit of the model translation with the n-gram unit of the reference translation (i.e. the manually written translation) and count the number of matches, which are location independent, the more matches the better the model translation. ROUGE (English full name: recall-Oriented Understudy for Gisting Evaluation, chinese term: evaluation replacement for Recall-oriented salient points) compares a model-generated text with a tag, i.e., a manually composed text, and calculates the number of overlapping units, such as n-gram, word sequence, and word pairs, between the model-generated text to be evaluated and the ideal text of the human composition.
AutoSummenG, presented by Giannakopoulos et al, is based on an n-gram graph and considers co-occurrence of word n-grams or character n-grams within a window. In the autosumeng method, all the different forms of a word are always converted into its lements.
ROUGE and BLEU, while effective in overall system ranking, are still lexical in nature and, depending on the tag, are not suitable for use in production environments lacking tags. AutoSummeNG is based on n-grams and considers co-occurrence of word n-grams or character n-grams within a window, which has a higher relevance to human judgment than ROUGE.
Disclosure of Invention
Based on the above, aiming at the technical problems, a new evaluation method and computer equipment for model generation text are provided, which are not dependent on labels and are suitable for production environments.
The application provides an evaluation method of a model generation text, which comprises the following steps:
generating a text for the model to be evaluated, and respectively calculating a gene index, a readability index and a fingerprint index of the text;
the gene index is used for measuring semantic relativity and homology of a model generated text and an input text, and comprises three factors including text length, relativity, mutual information and the like; the relevance is designed and realized based on word frequency and pearson correlation coefficient, and the mutual information characterization model generates the dependency degree between the text and the input text;
the readability index is used for measuring the degree of the model generated text which can be read by a person according to the sentence length average value and the text repeated degradation condition which are cut from the punctuation;
the fingerprint index is used for measuring the consistency degree of semantic feature distribution of the model generation text and the training set label; specifically, a network model is extracted by training fingerprints based on a ternary twin network in advance, a fingerprint library is generated at the same time, then a model generation text is input into the fingerprint extraction network model to obtain fingerprints, and then distance measurement is carried out on the fingerprints and the fingerprint library to determine the value of fingerprint indexes;
and carrying out comprehensive evaluation based on the calculated gene index, the calculated readability index and the calculated fingerprint index value to obtain a final evaluation result.
Alternatively, the gene index is formulated as follows:
wherein: v (V) gene Representing a gene index value; len (len) candidate Representing the number of characters of the text generated by the model; corr represents the relevance of the model generated text to the input text; mi denotes the mutual information of the model generation text and the input text.
Optionally, the formula of the readability index is as follows:
wherein: v (V) read Is the value of the readability index; total sent Representing the number of sentences of the text generated by the model, wherein the sentences refer to character strings obtained by punctuation segmentation; total char Representing the number of characters of the text generated by the model, except for punctuation; total dup Indicating the number of repetitions that occur; len (len) generate Representing the length of the text.
Optionally, the network model is extracted based on the ternary twin network training fingerprint, and a fingerprint library is generated at the same time, which specifically comprises the following steps:
firstly, preprocessing a data set to construct three data sets of an original label, a copy label and an irrelevant label;
secondly, constructing three networks sharing weight parameters, and respectively inputting three data sets of the original label, the copy label and the irrelevant label;
and updating network parameters according to the triplet loss and the classification loss, and obtaining a training set tag fingerprint library.
Further, the triplet loss function L tr The method comprises the following steps:
wherein: n represents the total number of triplet groups; max (d) represents a maximum value; d (,) represents the calculated Euclidean distance; x is x f 、And->Respectively indicate->、/>And->The characteristics obtained through network mapping; margin represents the interval of the triplet loss function for controlling the distance of positive and negative samples.
Further, the classification loss adopts a cross entropy loss function L ce The representation is specifically as follows:
wherein: n represents the total number of triplet groups; m represents the total number of label categories, and takes a value of 2; p represents the true classification probability; q represents the prediction classification probability, x ij Representing the probability that the original text x in the ith triplet belongs to (the expression applied to the calculation of the true classification probability p) or predicts (the expression applied to the prediction classification q) as the label j class.
Further, the triple loss and the cross entropy loss are combined, and the optimized fingerprint extraction network model loss function L is as follows:
wherein:representing variable weight parameters; l (L) tr Representing a triplet loss function; l (L) ce Representing a cross entropy loss function.
Optionally, the distance measure uses a hamming distance H, defined as follows:
wherein: l represents the fingerprint length; b q A fingerprint representing text to be evaluated; b represents a training set label fingerprint library; n represents the number of bits of the fingerprint;representing an exclusive-or operation;
determining fingerprint index function V based on minimum Hamming distance fingerprint The definition is as follows:
。
optionally, the comprehensive evaluation is performed to obtain a final evaluation result, and one of the following comprehensive evaluation methods is selected to be executed:
the first comprehensive evaluation method defines the following formula:
wherein: v (V) eval Representing a text evaluation value; v (V) gene Is the value of the gene index, V read Is the value of the readability index, V fingerprint Is the value of the fingerprint index; n is equal to the number of the factors in the root number;
the second comprehensive evaluation method defines the formula as follows:
wherein: w (W) gene Weight of gene index, V gene A value representing a gene index; w (W) read Weights representing readability indicators, V read A value representing a readability indicator; w (W) fingerprint Weights representing fingerprint indicators, V fingerprint A value representing a fingerprint index, and satisfies:；
if the gene index, the readability index and the fingerprint index do not need to be respectively set with weights or the weights can not be calculated, a first comprehensive evaluation method is selected; if the weights need to be set or given respectively, a second comprehensive evaluation method is selected.
The application provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that the processor realizes the steps of the model generation text evaluation method when executing the computer program.
The application has at least the following beneficial effects:
the evaluation method of the model generated text is independent of labels, is suitable for production environment, and is used for flexible and selectable comprehensive evaluation after three aspects of homology correlation degree with the original patent literature, human reading degree, semantic feature distribution consistency of training set labels and the like are respectively evaluated through three indexes such as genes, readability, fingerprints and the like.
The relevance in the gene index is designed and realized based on word frequency and pearson correlation coefficient, the weight is designed and realized simply, the sorting is quick, and the returned P value can be utilized to combine with the set significance level for verification, so that the statistical theory support is obtained. The mutual information in the gene index reflects the degree of dependence between the model generation text and the input text, and if the mutual information is far more than 0, the model generation text is highly related to the input text; if the mutual information is equal to 0, it is indicated that the model generation text and the input text are independent of each other.
The readability index is used for measuring the degree of the text generated by the model which can be read by a person, and is defined in two aspects of sentence length average value cut from punctuation and text repeated degradation condition, and the design is simple to realize.
The fingerprint index is based on a ternary twin network training fingerprint extraction network model, and a fingerprint library is generated. The model generates text input fingerprint to extract network model to obtain fingerprint, then calculates the nearest hamming distance between fingerprint and finger print in fingerprint library, then calculates fingerprint index value, and measures the consistency of semantic feature distribution of model generated text and training set label.
Drawings
FIG. 1 is a schematic diagram of an evaluation method of model generation text according to an embodiment of the present application;
FIG. 2 is a text fingerprint extraction block diagram of a ternary twin network in one embodiment of the application;
FIG. 3 is a schematic diagram of a triplet network architecture in accordance with one embodiment of the present application;
FIG. 4 illustrates three views of the integrated evaluation in one embodiment of the application.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
In one embodiment, as shown in fig. 1, there is provided an evaluation method of model generation text, including: generating a text for the model to be evaluated, and respectively calculating a gene index, a readability index and a fingerprint index of the text; and carrying out comprehensive evaluation based on the calculated gene index, the calculated readability index and the calculated fingerprint index value to obtain a final evaluation result.
1) Gene index
The genetic index is used for measuring semantic relativity and homology of the generated text and the input text of the model and comprises three factors such as text length, relativity and mutual information. The text length refers to the number of generated text characters; the relevance refers to the relevance of the generated text and the input text, and a calculation formula of the relevance corr is designed based on word frequency and pearson correlation coefficient:
（1）
wherein: n represents the size of the dictionary constructed after inputting text and all generated text word segmentation, namely the vocabulary number; x represents a word frequency list of the input text in dictionary order; y represents a word frequency list of the generated text of which the correlation degree is to be calculated, in dictionary order. Considering the performance improvement, the following equivalent and better-performance correlation calculation formula is given:
（2）
wherein: x represents a word frequency list of the input text in dictionary order; m is m x Representing word frequency average values of input texts; y represents a word frequency list of a model generation text of the correlation degree to be calculated, and the word frequency list is in dictionary order; m is m y The model representing the degree of correlation to be calculated generates a word frequency mean of the text. The scipy.stats.pearsonr packet is implemented based on equation (2) and returns the P value for verification, so the application directly calls scipy.stats.pearsonr to calculate the correlation.
The mutual information refers to average mutual information of the model generation text and the input text, namely, statistical average value in joint probability space of the model generation text and the input text. The average mutual information has the properties of non-negativity, reciprocity (symmetry), extremum and the like. Non-negative interpretation given the generated text, a portion of the uncertainty about the input text is generally always eliminated. Reciprocity (symmetry) means that the amount of information obtained from the model-generated text about the input text is equal to the amount of information obtained from the input text about the model-generated text. Extremum indicates that the amount of information obtained from one event about another event can be at most only the average self-information of the other event, and does not exceed the amount of information contained in the other event itself. The application calculates the mutual information to reflect the degree of dependence between the model generation text and the input text, and if the mutual information is far more than 0, the application indicates that the model generation text is highly relevant to the input text; if the mutual information is equal to 0, it is indicated that the model generation text and the input text are independent of each other. The calculation formula of the mutual information is defined as follows:
（3）
wherein: mi represents mutual information of the model generation text and the input text;a maximum value among a plurality of values indicating comma intervals in brackets, ensuring that the calculation result is non-negative; x represents a list of input text words; y represents the model to generate a text word list; />: representing a joint probability distribution of the input text and the model-generated text; p (x): representing a probability distribution of the input text; p (y): the representation model generates a probability distribution of the text.
A smoothed value calculation formula used in calculating the probability distribution of the input text:
（4）
wherein:: representing the sum of word frequency accumulation of the input text; />: the representation model generates a vocabulary number that appears in the text but not in the input text.
Calculating a smoothed value calculation formula used when generating a probability distribution of text:
（5）
wherein:: generating a text word frequency accumulation sum by the representation model; />: representing the number of words that occur in the input text but not in the model-generated text.
Designing gene indexes based on the factors such as the length, the relativity, the mutual information and the like, wherein a definition formula is as follows:
（6）
wherein: v (V) gene The larger the index value of the expressed gene is, the better; len (len) candidate Representing the number of characters of the text generated by the model; corr represents the relevance of model-generated text to the input text, see equation (1) (2); mi represents the mutual information of the model generation text and the input text, see formula (3).
2) Readability index
The readability index is used for measuring the degree of the model generated text which can be read by a person, and the readability index V is defined by the sentence length average value which is cut from punctuation and the repeated degradation condition of the text read The following are provided:
（7）
wherein: total sent Representing the number of sentences of the text generated by the model, wherein the sentences refer to character strings obtained by punctuation segmentation; total char Representing the number of characters of the text generated by the model, except for punctuation; total dup Indicating the number of times of repetition, and suggesting a value not less than 3; len (len) generate Representing the length of the text. In the middle ofModeling the influence of the length of sentences segmented by text punctuation on the readability, wherein the longer the readability is, the worse the readability is; />Modeling the influence of the repetition number on the readability, wherein the readability is worse as the repetition number is larger; />Modeling the influence of the average length of the repeated character strings on the readability, wherein the shorter the average length of the repeated character strings is, the worse the readability is; />Modeling is the effect of text repetition degradation on readability.
3) Fingerprint index
The fingerprint index is used for measuring the consistency degree of semantic feature distribution of the model generation text and the training set label. In the triple twinning network training stage, firstly, preprocessing a data set to construct 3 data sets of an original tag, a copy tag and an irrelevant tag; secondly, constructing 3 networks sharing weight parameters, and respectively inputting 3 data sets of an original label, a copy label and an irrelevant label; and updating network parameters according to the triplet loss and the classification loss, and obtaining a training set tag fingerprint library. In the model application stage, obtaining the fingerprint of the text to be evaluated by applying a trained fingerprint extraction network, calculating the hamming distance between the text to be evaluated and the fingerprint in the training set label fingerprint library, and calculating the fingerprint index value according to the minimum hamming distance. The triplet twinned network text fingerprint extraction framework is shown in fig. 2.
(1) Convolution kernel
Since the text character sequence is one-dimensional, the text is converted into a two-dimensional numerical matrix after the character is vectorized. The convolution kernel size dim×win, dim representing the character vector dimension size, win representing the window size.
(2) Triplet network structure
The triplet network takes a group of triplet samples as input, can learn the distance relation between the original samples and the positive and negative samples at the same time, and constrains the distance between the original samples and the positive and negative samples by using a triplet loss function, so that the characteristics of the original text and the copy text of the original text are closer, and the characteristic distance between the original text and other irrelevant texts is enlarged. Therefore, the whole framework is constructed by adopting a triplet network with shared weight parameters, as shown in fig. 3, the network weight parameters of three branches are shared, and the input is respectively an original text x and a copy text x + Extraneous text x − Extracting text features through a network, wherein the copy text x + Is obtained by carrying out synonym replacement on the original text x, adopts a triplet loss function to measure the characteristic distance, carries out network parametersAnd (5) optimizing. Triplet loss function L tr Expressed as:
（8）
wherein: n represents the total number of triplet groups; max (d) represents a maximum value; d (,) represents the calculated Euclidean distance; x is x f 、And->Respectively indicate->、/>And->The characteristics obtained through network mapping; margin represents the interval of the triplet loss function for controlling the distance of positive and negative samples.
The feature distance between the original text and the copy text is reduced by minimizing the triple loss, and the feature distance between the original text and the irrelevant text is increased. In order to enhance the sensitivity of fingerprints to the overall distribution information of sample characteristics and enhance the uniqueness of the fingerprints, the application adopts a cross entropy loss function as classification loss supplement, and the cross entropy loss function L ce The definition is as follows:
（9）
wherein: n represents the total number of triplet groups; m represents the total number of label categories, and M takes a value of 2 because the classification is used for judging whether the labels are the same type of the original text or the duplicate text; p represents the true classification probability; q represents the prediction classification probability, x ij Representing that the original text x in the ith triplet belongs to (applies to the expression calculated by the true classification probability p) or predicts(the expression applicable to the prediction class q) is the probability of the tag j class.
Combining the triple loss and the cross entropy loss, optimizing a network model loss function L, and defining as follows:
（10）
wherein:representing variable weight parameters; l (L) tr Representing a triplet loss function; l (L) ce Representing a cross entropy loss function.
(3) Training phase
The backbone network contains 5 2D convolutional layers, 1 pooling layer and 2 fully connected layers using triad twin network training that shares weight parameters. The activation function of the last full-connection layer is set as a hyperbolic tangent function tanh, the other full-connection layer activation functions are set as correction linear units ReLU, and the activation function tanh enables the corresponding output of each input after passing through the whole network structure to be inContinuous real values. And the hash function is quantized to 0 or 1, so that the storage space occupied by the text fingerprint is reduced, and the calculation efficiency of the Hamming distance is provided. Model training and fingerprint extraction can be formalized as:
（11）
wherein: given a text x i Order-makingTo learn the proper parameter w after passing through the 2D network, the feature is quantized into a binary text fingerprint b through a function sign i The fingerprint length is set to be 16-80 bits, and the application takes 64bits.
(4) Application phase
Taking the trained 2D model of one branch as a fingerprint extraction network, firstly, extracting fingerprints from all training set labels to obtain a fingerprint library; then, extracting fingerprints from each text to be evaluated through a trained fingerprint extraction network; next, a distance metric and fingerprint index value calculation is performed with the fingerprint library. The fingerprint metric uses a hamming distance H, defined as follows:
（12）
wherein: l represents the fingerprint length; b q A fingerprint representing text to be evaluated; b represents a training set label fingerprint library; n represents the number of bits of the fingerprint;representing an exclusive-or operation;
fingerprint index V designed based on minimum Hamming distance fingerprint The definition is as follows:
（13）
comprehensive evaluation is performed based on the gene index, the readability index and the fingerprint index, and three evaluation perspectives shown in fig. 4 are embodied. The present embodiment provides two comprehensive evaluation methods:
the first comprehensive evaluation method defines the following formula:
（14）
wherein: v (V) eval Representing a text evaluation value; v (V) gene Is the value of the gene index, V read Is the value of the readability index, V fingerprint Is the value of the fingerprint index; n represents the power of n, which is equal to the number of factors in the root.
The second comprehensive evaluation method defines the formula as follows:
（15）
wherein: w (W) gene Weight of gene index, V gene A value representing a gene index; w (W) read Weights representing readability indicators, V read A value representing a readability indicator; w (W) fingerprint Weights representing fingerprint indicators, V fingerprint A value representing a fingerprint index, and satisfies:。
if the gene index, the readability index and the fingerprint index do not need to be respectively set with weights or cannot be calculated, suggesting to select a first evaluation method; if weights need to be set separately or weights can be given, a second evaluation method is recommended.
In one embodiment, a computer device is also provided, where the computer device may be a server or a client device, and all or part of the flow in the method of the foregoing embodiment is implemented by using an operating program.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (8)
1. A method of evaluating model-generated text, comprising:
generating a text for the model to be evaluated, and respectively calculating a gene index, a readability index and a fingerprint index of the text;
the gene index is used for measuring semantic relativity and homology of a model generated text and an input text, and comprises three factors including text length, relativity and mutual information; the relevance is designed and realized based on word frequency and pearson correlation coefficient, and the mutual information characterization model generates the dependency degree between the text and the input text;
the readability index is used for measuring the degree of the model generated text which can be read by a person according to the sentence length average value and the text repeated degradation condition which are cut from the punctuation;
the fingerprint index is used for measuring the consistency degree of semantic feature distribution of the model generation text and the training set label; specifically, a network model is extracted by training fingerprints based on a ternary twin network in advance, a fingerprint library is generated at the same time, then a model generation text is input into the fingerprint extraction network model to obtain fingerprints, and then distance measurement is carried out on the fingerprints and the fingerprint library to determine the value of fingerprint indexes;
based on the calculated gene index, readability index and fingerprint index values, carrying out comprehensive evaluation to obtain a final evaluation result;
the gene index is expressed as follows:
V gene ＝log 2 (len candidate )×corr×log 2 (mi)
wherein: v (V) gene Representing a gene index value; len (len) candidate Representing the number of characters of the text generated by the model; corr represents the relevance of the model generated text to the input text; mi represents mutual information of the model generation text and the input text;
the formula of the readability index is as follows:
wherein: v (V) read Is the value of the readability index; total sent Representing the number of sentences of the text generated by the model, wherein the sentences refer to character strings obtained by punctuation segmentation; total char Representing the number of characters of the text generated by the model, except for punctuation; total dup Indicating the number of repetitions that occur; len (len) generate Representing the length of the text.
2. The method for evaluating model generation text according to claim 1, wherein the network model is extracted based on the ternary twin network training fingerprint, and a fingerprint library is generated at the same time, specifically comprising:
firstly, preprocessing a data set to construct three data sets of an original label, a copy label and an irrelevant label;
secondly, constructing three networks sharing weight parameters, and respectively inputting three data sets of the original label, the copy label and the irrelevant label;
and updating network parameters according to the triplet loss and the classification loss, and obtaining a training set tag fingerprint library.
3. The method for evaluating model-generated text according to claim 2, wherein the triplet loss function L tr The method comprises the following steps:
wherein: n represents the total number of triplet groups; max (d) represents a maximum value; d (,) represents the calculated Euclidean distance; x is x f 、And->Respectively represent x, x + And x - The characteristics obtained through network mapping; margin represents the interval of the triplet loss function for controlling the distance of positive and negative samples.
4. A method of evaluating model generated text according to claim 3, wherein the classification penalty employs a cross entropy penalty function L ce The representation is specifically as follows:
wherein: n represents the total number of triplet groups; m represents the total number of label categories, and takes a value of 2; p represents the true classification probability; q represents the prediction classification probability, x ij Representing the probability that the original text x in the ith triplet belongs to or is predicted to be the tag j category.
5. The method for evaluating model generated text according to claim 4, wherein the triplet loss and the cross entropy loss are combined, and the optimized fingerprint extraction network model loss function L is:
L＝γL tr +(1-γ)L ce
wherein: gamma represents a variable weight parameter; l (L) tr Representing a triplet loss function; l (L) ce Representing a cross entropy loss function.
6. The method for evaluating model-generated text according to claim 2, wherein the distance measure uses a hamming distance H defined as follows:
wherein: l represents the fingerprint length; b q A fingerprint representing text to be evaluated; b represents a training set label fingerprint library; n represents the number of bits of the fingerprint;representing an exclusive-or operation;
determining fingerprint index function V based on minimum Hamming distance fingerprint The definition is as follows:
7. the method for evaluating model-generated text according to claim 1, wherein the performing the comprehensive evaluation to obtain a final evaluation result is performed by selecting one of the following comprehensive evaluation methods:
the first comprehensive evaluation method defines the following formula:
wherein: v (V) eval Representing a text evaluation value; v (V) gene Is the value of the gene index, V read Is the value of the readability index, V fingerprint Is the value of the fingerprint index; n is equal to the number of the factors in the root number;
the second comprehensive evaluation method defines the formula as follows:
V eval ＝W gene ×V gene +W read ×V read +W fingerprint ×V fingerprint
wherein: w (W) gene Weight of gene index, V gene A value representing a gene index; w (W) read Weights representing readability indicators, V read A value representing a readability indicator; w (W) fingerprint Weights representing fingerprint indicators, V fingerprint A value representing a fingerprint index, and satisfies: w (W) fingerint +W gene +W read ＝1；
If the gene index, the readability index and the fingerprint index do not need to be respectively set with weights or the weights can not be calculated, a first comprehensive evaluation method is selected; if the weights are required to be set or given respectively, a second comprehensive evaluation method is selected.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311075044.6A CN116776868B (en) | 2023-08-25 | 2023-08-25 | Evaluation method of model generation text and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311075044.6A CN116776868B (en) | 2023-08-25 | 2023-08-25 | Evaluation method of model generation text and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116776868A CN116776868A (en) | 2023-09-19 |
| CN116776868B true CN116776868B (en) | 2023-11-03 |
Family
ID=87989926
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311075044.6A Active CN116776868B (en) | 2023-08-25 | 2023-08-25 | Evaluation method of model generation text and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116776868B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377451A (en) * | 2012-04-20 | 2013-10-30 | 马越 | Patent assessment system and method |
| CN109271629A (en) * | 2018-09-07 | 2019-01-25 | 中山大学 | Production text snippet method based on intensified learning |
| KR102019207B1 (en) * | 2018-11-12 | 2019-09-06 | 주식회사 와이즈넛 | Apparatus and method for assessing data quality for text analysis |
| CN111914084A (en) * | 2020-01-09 | 2020-11-10 | 北京航空航天大学 | Deep learning-based emotion label text generation and evaluation system |
| CN114579720A (en) * | 2022-02-17 | 2022-06-03 | 中国长江三峡集团有限公司 | Hydropower project progress intelligent assessment method based on text mining |
| CN115983877A (en) * | 2023-01-09 | 2023-04-18 | 大连理工大学 | Patent value evaluation method based on depth map and semantic learning |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8145639B2 (en) * | 2004-08-11 | 2012-03-27 | Allan Williams | System and methods for patent evaluation |
| TWI608367B (en) * | 2012-01-11 | 2017-12-11 | 國立臺灣師範大學 | Text readability measuring system and method thereof |
| US9720901B2 (en) * | 2015-11-19 | 2017-08-01 | King Abdulaziz City For Science And Technology | Automated text-evaluation of user generated text |
-
2023
- 2023-08-25 CN CN202311075044.6A patent/CN116776868B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377451A (en) * | 2012-04-20 | 2013-10-30 | 马越 | Patent assessment system and method |
| CN109271629A (en) * | 2018-09-07 | 2019-01-25 | 中山大学 | Production text snippet method based on intensified learning |
| KR102019207B1 (en) * | 2018-11-12 | 2019-09-06 | 주식회사 와이즈넛 | Apparatus and method for assessing data quality for text analysis |
| CN111914084A (en) * | 2020-01-09 | 2020-11-10 | 北京航空航天大学 | Deep learning-based emotion label text generation and evaluation system |
| CN114579720A (en) * | 2022-02-17 | 2022-06-03 | 中国长江三峡集团有限公司 | Hydropower project progress intelligent assessment method based on text mining |
| CN115983877A (en) * | 2023-01-09 | 2023-04-18 | 大连理工大学 | Patent value evaluation method based on depth map and semantic learning |
Non-Patent Citations (2)
| Title |
|---|
| BERTScore: Evaluating text generation with Bert;Tianyi Zhang 等;《arViv:1904.09675v3[cs.CL]》;第1-43页 * |
| 面向招标文件的半结构化文本自动生成;刘金硕 等;《计算机工程》;第49卷(第3期);第67-72页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116776868A (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7169369B2 (en) | Method, system for generating data for machine learning algorithms | |
| CN109960724B (en) | Text summarization method based on TF-IDF | |
| JP5338238B2 (en) | Automatic ontology generation using word similarity | |
| JP4774073B2 (en) | Methods for document clustering or categorization | |
| JP5216063B2 (en) | Method and apparatus for determining categories of unregistered words | |
| JP6838092B2 (en) | Text paraphrase method, device, server, and storage medium | |
| CN108519971B (en) | Cross-language news topic similarity comparison method based on parallel corpus | |
| JP2008084064A (en) | Text classification processing method, text classification processing device and text classification processing program | |
| CN110874536B (en) | Corpus quality evaluation model generation method and double-sentence pair inter-translation quality evaluation method | |
| CN111985228A (en) | Text keyword extraction method and device, computer equipment and storage medium | |
| CN110222192A (en) | Corpus method for building up and device | |
| CN110674301A (en) | Emotional tendency prediction method, device and system and storage medium | |
| CN111753082A (en) | Text classification method and device based on comment data, equipment and medium | |
| CN111325018A (en) | Domain dictionary construction method based on web retrieval and new word discovery | |
| CN114611491A (en) | Intelligent government affair public opinion analysis research method based on text mining technology | |
| CN113239697B (en) | Entity recognition model training method and device, computer equipment and storage medium | |
| Zaware et al. | Text summarization using tf-idf and textrank algorithm | |
| CN112016294B (en) | Text-based news importance evaluation method and device and electronic equipment | |
| CN114222000A (en) | Information pushing method and device, computer equipment and storage medium | |
| CN111581377B (en) | Text classification method and device, storage medium and computer equipment | |
| CN116776868B (en) | Evaluation method of model generation text and computer equipment | |
| JP6112536B2 (en) | Bilingual expression extraction apparatus, bilingual expression extraction method, and computer program for bilingual expression extraction | |
| JP2005078240A (en) | Method for extracting knowledge by data mining | |
| Bozorgi et al. | Cancer survivability with logistic regression | |
| Long et al. | Multi-document summarization by information distance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |