CN116629238A

CN116629238A - Text enhancement quality evaluation method, electronic device and storage medium

Info

Publication number: CN116629238A
Application number: CN202310596001.6A
Authority: CN
Inventors: 李志韬; 王健宗; 程宁
Original assignee: Ping An Technology Shenzhen Co Ltd
Current assignee: Ping An Technology Shenzhen Co Ltd
Priority date: 2023-05-24
Filing date: 2023-05-24
Publication date: 2023-08-22

Abstract

The application relates to the technical fields of data quality inspection, intelligent medical treatment and financial transaction, in particular to a text enhancement quality assessment method, electronic equipment and a storage medium. According to the text enhancement quality assessment method, an original text and a target text are required to be acquired firstly, the target text is subjected to prediction processing based on a pre-trained test language model, prediction result data are obtained, the original text and the prediction result data are subjected to coincidence comparison to obtain a first assessment index corresponding to the target text, semantic feature extraction is performed on the original text to obtain original semantic features, semantic feature extraction is performed on the target text to obtain target semantic features, the original semantic features and the target semantic features are subjected to similarity comparison to obtain a second assessment index corresponding to the target text, and finally the target text is subjected to quality assessment based on the first assessment index and the second assessment index to obtain assessment result data, so that the quality of the text generated by text enhancement processing can be objectively and accurately assessed.

Description

Text enhancement quality evaluation method, electronic device and storage medium

Technical Field

The application relates to the technical fields of data quality inspection, intelligent medical treatment and financial transaction, in particular to a text enhancement quality assessment method, electronic equipment and a storage medium.

Background

Natural language processing (Natural Language Processing, NLP) is an important direction in the fields of computer science and artificial intelligence. It is studying various theories and methods that enable effective communication between a person and a computer in natural language. With the development of artificial intelligence technology, the field of natural language processing generates multiple groups of models with different structures. For example, in many neural network models for natural language processing, differences in the structures of the neural network models are brought about based on the application differences of the activation functions, the network layers, the loss functions and the regularization methods. Various types of natural language processing models are widely used in various subdivision directions such as text classification, text matching, text summarization and the like.

In business scenarios such as smart medical treatment, financial transactions, etc., various natural language models are often used. It should be noted that various specialized terms in intelligent medical treatment and financial transaction are not easy to find in the field of learning, so that a general data set is difficult to directly use for training a natural language model of intelligent medical treatment and financial transaction, and a data set in the field of intelligent medical treatment and financial transaction is less, so that the fields often use a natural language processing model to carry out data enhancement on an original text, and an evaluation index for evaluating the text data enhancement often has defects in an evaluation dimension, and is difficult to objectively and accurately reflect the quality of text data generated by the data enhancement. Therefore, how to objectively and accurately evaluate the quality of the text generated by the text enhancement process has become a problem to be solved in the industry.

Disclosure of Invention

The present application aims to solve at least one of the technical problems existing in the prior art. Therefore, the application provides a text enhancement quality evaluation method, electronic equipment and a storage medium, which can objectively and accurately evaluate the quality of a text generated by text enhancement processing.

An embodiment of the present application provides a text enhancement quality assessment method, including:

acquiring an original text and a target text, wherein the target text is obtained by text enhancement processing of the original text;

performing prediction processing on the target text based on a pre-trained test language model to obtain prediction result data;

performing coincidence ratio comparison on the original text and the predicted result data to obtain a first evaluation index corresponding to the target text;

extracting semantic features from the original text to obtain original semantic features, and extracting semantic features from the target text to obtain target semantic features;

performing similarity comparison on the original semantic features and the target semantic features to obtain a second evaluation index corresponding to the target text;

and carrying out quality evaluation on the target text based on the first evaluation index and the second evaluation index to obtain evaluation result data.

According to some embodiments of the application, the quality evaluation of the target text based on the first evaluation index and the second evaluation index, to obtain evaluation result data, includes:

performing word frequency characteristic evaluation on the target text based on the first evaluation index to obtain first evaluation data;

performing semantic feature evaluation on the target text based on the second evaluation index to obtain second evaluation data;

and obtaining the evaluation result data according to the first evaluation data and the second evaluation data.

According to some embodiments of the present application, the predicting the target text based on the pre-trained test language model to obtain predicted result data includes:

extracting a plurality of groups of target character strings from the target text based on preset conditions;

performing prediction processing on each target character string through the test language model to obtain a predicted character string corresponding to each target character string;

and determining a plurality of predicted character strings as the predicted result data.

According to some embodiments of the present application, the comparing the degree of coincidence between the original text and the predicted result data to obtain a first evaluation index corresponding to the target text includes:

Extracting an original character string corresponding to the target character string from the original text;

and comparing the coincidence degree of the original character string with that of the predicted character string to obtain the first evaluation index.

According to some embodiments of the present application, the extracting semantic features from the original text to obtain original semantic features, and extracting semantic features from the target text to obtain target semantic features includes:

performing first word segmentation processing on the original text to obtain a plurality of first word groups;

performing second word segmentation processing on the target text to obtain a plurality of second word groups;

and extracting semantic features of each first phrase and each second phrase based on a pre-trained semantic recognition model to obtain the original semantic features of each first phrase and the target semantic features of each second phrase.

According to some embodiments of the application, the comparing the similarity between the original semantic features and the target semantic features to obtain a second evaluation index corresponding to the target text includes:

performing similarity calculation on the original semantic features of each first phrase and the target semantic features of each second phrase one by one to obtain a similarity matrix;

Obtaining a target accuracy rate and a target recall rate corresponding to the target text based on the original semantic features, the target semantic features and the similarity matrix;

and obtaining the second evaluation index according to the target accuracy rate and the target recall rate.

According to some embodiments of the application, the obtaining, based on the original semantic features, the target semantic features and the similarity matrix, a target accuracy and a target recall corresponding to the target text includes:

extracting similarity matching scores of each original semantic feature and each target semantic feature from the similarity matrix;

determining the maximum similarity score of each target semantic feature according to a plurality of similarity matching scores;

accumulating the maximum similarity score of each target semantic feature to obtain a similar accumulated value;

and obtaining the target accuracy rate and the target recall rate according to the original semantic features, the target semantic features and the similar accumulated values.

According to some embodiments of the application, the obtaining the target accuracy and the target recall corresponding to the target text based on the original semantic features, the target semantic features and the similarity matrix further includes:

Based on the reverse file frequency of each first phrase in the original text, configuring evaluation weights for each original semantic feature;

and obtaining the target accuracy rate and the target recall rate according to the original semantic features, the evaluation weights, the similarity matrix and the target semantic features.

In a second aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor, the memory storing a computer program, the processor implementing the text enhancement quality assessment method according to any one of the embodiments of the first aspect of the present application when executing the computer program.

In a third aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program, where the computer program is executed by a processor to implement a text enhancement quality assessment method according to any one of the embodiments of the first aspect of the present application.

The text enhancement quality assessment method, the electronic equipment and the storage medium have at least the following beneficial effects:

according to the text enhancement quality assessment method, an original text and a target text are required to be acquired firstly, the target text is obtained through text enhancement processing of the original text, then the target text is subjected to prediction processing based on a pre-trained test language model, prediction result data are obtained, further, the original text and the prediction result data are subjected to coincidence degree comparison to obtain a first assessment index corresponding to the target text, further, the original text is subjected to semantic feature extraction to obtain original semantic features, the target text is subjected to semantic feature extraction to obtain target semantic features, further, the original semantic features and the target semantic features are subjected to similarity comparison to obtain a second assessment index corresponding to the target text, and finally quality assessment is performed on the target text based on the first assessment index and the second assessment index to obtain assessment result data. The quality of the text generated by the text enhancement processing can be objectively and accurately evaluated through the evaluation result data. Text enhancement is carried out on technical nouns with more professions in business scenes such as intelligent medical treatment, financial transaction and the like, a plurality of texts generated after text enhancement can be obtained, and if the text enhancement quality assessment method is used for carrying out quality assessment on the part of texts, the objectivity and accuracy of an assessment link can be further improved.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Drawings

The foregoing and/or additional aspects and advantages of the application will become apparent and may be better understood from the following description of embodiments taken in conjunction with the accompanying drawings in which:

fig. 1 is a flow chart of a text enhancement quality evaluation method according to an embodiment of the present application;

FIG. 2 is a flowchart illustrating the step S102 in FIG. 1 according to an embodiment of the present application;

FIG. 3 is a flowchart illustrating step S103 in FIG. 1 according to an embodiment of the present application;

FIG. 4 is a flowchart of step S104 in FIG. 1 according to an embodiment of the present application;

FIG. 5 is a flowchart illustrating step S105 in FIG. 1 according to an embodiment of the present application;

FIG. 6 is a flowchart illustrating the step S502 in FIG. 5 according to an embodiment of the present application;

FIG. 7 is a schematic diagram illustrating another flow chart of step S502 in FIG. 5 according to an embodiment of the present application;

FIG. 8 is a flowchart illustrating step S106 in FIG. 1 according to an embodiment of the present application;

fig. 9 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the application.

In the description of the present application, a number means one or more, a number means two or more, and greater than, less than, exceeding, etc. are understood to not include the present number, and above, below, within, etc. are understood to include the present number. The description of the first and second is for the purpose of distinguishing between technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.

In the description of the present application, it should be understood that the direction or positional relationship indicated with respect to the description of the orientation, such as up, down, left, right, front, rear, etc., is based on the direction or positional relationship shown in the drawings, is merely for convenience of describing the present application and simplifying the description, and does not indicate or imply that the apparatus or element to be referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present application.

In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

In the description of the present application, unless explicitly defined otherwise, terms such as arrangement, installation, connection, etc. should be construed broadly and the specific meaning of the terms in the present application can be determined reasonably by a person skilled in the art in combination with the specific content of the technical solution. In addition, the following description of specific steps does not represent limitations on the order of steps or logic performed, and the order of steps and logic performed between steps should be understood and appreciated with reference to what is described in the embodiments.

In the related art, an evaluation index for evaluating a text enhancement model often has a defect in an evaluation dimension. Therefore, how to evaluate the quality of text generated by the text enhancement process is a problem that is in need of solving in the industry.

The following is a further description based on the accompanying drawings.

Fig. 1 is a schematic flow chart of an alternative text enhancement quality assessment method provided by the present application, which may include, but is not limited to, the following steps S101 to S106:

step S101, an original text and a target text are obtained, and the target text is obtained by text enhancement processing of the original text;

step S102, predicting a target text based on a pre-trained test language model to obtain predicted result data;

step S103, performing coincidence comparison on the original text and the predicted result data to obtain a first evaluation index corresponding to the target text;

Step S104, extracting semantic features from the original text to obtain original semantic features, and extracting semantic features from the target text to obtain target semantic features;

step S105, comparing the similarity of the original semantic features with the target semantic features to obtain a second evaluation index corresponding to the target text;

and S106, performing quality evaluation on the target text based on the first evaluation index and the second evaluation index to obtain evaluation result data.

According to the text enhancement quality assessment method, an original text and a target text are required to be acquired firstly, the target text is obtained through text enhancement processing of the original text, then the target text is subjected to prediction processing based on a pre-trained test language model, prediction result data are obtained, further, the original text and the prediction result data are subjected to coincidence degree comparison to obtain a first assessment index corresponding to the target text, further, the original text is subjected to semantic feature extraction to obtain original semantic features, the target text is subjected to semantic feature extraction to obtain target semantic features, further, the original semantic features and the target semantic features are subjected to similarity comparison to obtain a second assessment index corresponding to the target text, and finally quality assessment is performed on the target text based on the first assessment index and the second assessment index to obtain assessment result data. The quality of the text generated by the text enhancement processing can be objectively and accurately evaluated through the evaluation result data.

In step S101 of some embodiments of the present application, an original text and a target text are obtained, where the target text is obtained by performing text enhancement processing on the original text. It should be noted that the original text refers to the basic text of the text enhancement processing, and the target text is obtained by subjecting the original text to the text enhancement processing. It should be understood that text enhancement refers to a method of generating more classes of training samples by technical means in text processing tasks, such as text classification tasks. The purpose of text enhancement is to promote training effects on the target model by increasing the number of different types of training samples. In some embodiments, the text enhancement process requires a certain amount of annotation samples, and more similar samples are generated by technical means. For example, new sample material is obtained by replacing part of the word in the original sample within the synonym table or randomly deleting part of the word, etc.

It is clear that the purpose of text enhancement is to increase the types and the number of texts on the basis of original texts to obtain target texts, so that the artificial intelligent model can be trained conveniently and fully according to the target texts, and the training effect on the artificial intelligent model is improved. In some embodiments of the present application, text enhancement processing may be performed by a variety of methods in order to obtain a rich sample. For example, text enhancement can be achieved through a back-translation mode, specifically, an original text can be converted into other languages through a translation program, training samples in the other languages are further translated back to Chinese, so that the types and the numbers of the training samples are added, and a target text is obtained. For another example, text enhancement can be implemented by means of simple data expansion (Easy Data Augmentation, EDA), specifically, the change data of the original text is obtained by means of synonym substitution, random insertion, random exchange, random deletion and the like, so as to obtain the target text, and the type and the number of training samples are added, however, the text data enhancement by means of simple data expansion is difficult to control the number of samples, if too many expansion or too frequent operation is performed, the semantics of the training samples are changed, so that the quality of the target text is low. It should be noted that, in the existing text enhancement processing methods, it is difficult to directly obtain an ideal target text according to an original text, so that the embodiment of the application provides a text enhancement quality evaluation method, which can objectively and accurately evaluate the quality of a text generated by text enhancement processing, thereby facilitating the preference and improvement of various text enhancement processing methods.

In order to evaluate the quality of the text generated by the text enhancement process, it is necessary to obtain the original text before the text enhancement process and the target text after the text enhancement process.

In step S102 of some embodiments of the present application, a prediction process is performed on a target text based on a pre-trained test language model, so as to obtain prediction result data. It should be noted that the test language model refers to a language model for testing the target text. A Language Model (LM) plays an important role in natural Language processing, and is capable of predicting sentence probability of a word sequence. The language model originates from speech recognition (Speech Recognition), i.e. inputting a piece of audio data, and the speech recognition system typically generates a plurality of sentences as candidates, and in order to determine which sentences are more reasonable, the candidate sentences need to be ordered by the language model, and a word sequence with higher sentence forming probability is predicted from the candidate sentences, so as to determine the most reasonable sentences. Specifically, the language model is such a model that: for any word sequence, it can calculate the probability that the sequence is a sentence, e.g., word sequence A "today's |weather|true|good|", a good language model predicts a very high sentence probability for word sequence A, and word sequence B "today's |sleep|apple|fast", a good language model predicts a very low sentence probability for word sequence B. It should be noted that, on the basis that the language model can predict the sentence forming probability of a word sequence, the language model can be further used for predicting the probability of the next word in the case that the previous words are specified.

The test language model may be selected from various types of language models, such as an N-gram language model, a feedforward neural network language model (FeedForward Neural Network Language Models), and a recurrent neural network language model (RNN Language Models). It should be appreciated that alternatives to the language model may include, but are not limited to, the specific embodiments set forth above.

It is necessary to define the pre-training process that the test language model undergoes to train the test language model to its ability to predict the probability of the next word if the previous number of words are defined. It should be noted that, the prediction processing is performed on the target text based on the pre-trained test language model, so as to obtain the predicted result data, and the prediction processing may specifically be that a word sequence is selected from the target text by using the prediction capability of the test language model, and then the probability of the next word is predicted based on the word sequence, so as to obtain the predicted result data. It will be appreciated that the purpose of the prediction process is to derive the predicted outcome data from the target text such that the predicted outcome data is aligned with the original text in terms of overlap, thereby assessing the quality of the target text.

In some exemplary embodiments of the present application, the target text is predicted by the test language model, which may be that a part of word sequence is extracted from the target text, and then the test language model predicts according to the word sequence to obtain a confusion degree (confusion) corresponding to the part of word sequence, where the calculated confusion degree value is equal to a virtual dictionary size, and the next word to be selected needs to be selected from the virtual dictionary, and it should be pointed out that if the word to be selected in the virtual dictionary can form a sentence with the part of word sequence extracted from the target text, the word to be selected can be selected, where the sentence formed by the word to be selected and the word sequence is the prediction result data. It is clear that when the confusion degree is larger, the virtual dictionary is larger, the number of selected words is larger, the probability of the overlap ratio of the predicted result data and the original text is higher is smaller, and the quality of the target text is lower; similarly, the smaller the confusion value is, the smaller the virtual dictionary is, the fewer the candidate words can be selected, the higher the probability of the overlap ratio of the predicted result data and the original text is, and the higher the quality of the target text is.

Referring to fig. 2, step S102 according to some embodiments of the present application may include, but is not limited to, steps S201 to S203 described below.

Step S201, extracting a plurality of groups of target character strings from the target text based on preset conditions;

step S202, predicting each target character string through a test language model to obtain a predicted character string corresponding to each target character string;

step S203, a plurality of prediction strings are determined as prediction result data.

In steps S201 to S203 of some embodiments of the present application, a plurality of sets of target strings are extracted from a target text based on a preset condition, and then each target string is predicted by a test language model to obtain a predicted string corresponding to each target string, and the plurality of predicted strings are further determined as predicted result data. It should be noted that, because the pre-trained test language model can predict the probability of the next word under the condition that the previous words are clear, in order to test whether the target text generated by the text enhancement processing is associated with the original text in the statistical dimension, multiple groups of target strings need to be extracted from the target text, further, each target string is predicted by the test language model to obtain a predicted string corresponding to each target string, and further, the multiple predicted strings are determined as predicted result data so as to compare the predicted result data with the original text to obtain the first evaluation index.

It should be noted that, based on the preset conditions, the preset conditions refer to preset extraction conditions, and the preset conditions may be flexibly set according to actual requirements, for example, multiple groups of target strings are randomly extracted from the target text, and multiple groups of setting manners of each type, such as multiple groups of target strings, are extracted from the target text according to preset proportions. It should be emphasized that the original text refers to the basic text of the text enhancement processing, and the target text is obtained by subjecting the original text to the text enhancement processing. Thus, in some exemplary embodiments, each sentence in the original text is subjected to text enhancement processing, so that the type and number of the obtained text are increased, then a set of target strings is formed, and after a plurality of target strings are integrated, a target text is formed. Therefore, the target text can be integrated by a plurality of groups of target character strings, so that in order to facilitate comparison of the predicted result data with the original text, in the embodiment of the application, a plurality of groups of target character strings are firstly extracted from the target text based on the preset condition, then the predicted processing is performed on the basis of the target character strings, and the predicted character strings thus obtained are incorporated into the predicted result data, thereby facilitating comparison with corresponding sentences in the original text. Therefore, through the steps S201 to S203, a more convenient method can be provided for determining the first evaluation index, and the quality evaluation efficiency of the target text can be further improved.

In step S103 of some embodiments of the present application, the original text and the predicted result data are subjected to overlap ratio comparison, so as to obtain a first evaluation index corresponding to the target text. It is emphasized that after the target text is predicted by the test language model, a part of text with a larger sentence forming probability in the target text, that is, the predicted result data, can be obtained, so that the predicted result data includes a part of text with a larger sentence forming probability in the target text. In some exemplary embodiments of the present application, the coincidence ratio between the original text and the predicted result data is compared, so that the coincidence ratio between the original text and the partial text with a larger sentence forming probability in the target text, that is, the first evaluation index, can be determined, and the quality of the text generated by the text enhancement processing is evaluated by the first evaluation index. It should be appreciated that the first evaluation index determines whether the target text generated by the text enhancement process is associated with the original text in the statistical dimension by comparing the overlap ratio of the predicted result data and the original text, and since the purpose of text enhancement is to increase the type and the number of the text based on the original text, the higher the association degree of the target text with the original text in the statistical dimension based on the increase of the type and the number of the text, the higher the quality of the target text is.

In some specific embodiments, when the N-gram language model is used as the language model, the N-gram language model may perform a sliding window operation with the size of N on the content of the target text according to the byte stream to form a byte segment sequence with the length of N, where each byte segment is called a gram, statistics is performed on the occurrence frequencies of all the grams, filtering is performed according to a preset threshold, and a key gram list, that is, a target feature vector space of the target text, where each gram is a feature vector dimension, and prediction processing is performed according to the target feature vector space of the target text by using the N-gram model, so as to obtain predicted result data. After the prediction result data is obtained, the overlap ratio of the prediction result data and the original text can be further compared, so that the overlap ratio of the prediction result data and the original text on word frequency, namely, a first evaluation index, can be obtained, and the first evaluation index is used for evaluating the association degree of the target text and the original text on the statistical dimension.

Referring to fig. 3, step S103 according to some embodiments of the present application may include, but is not limited to, the following steps S301 to S302.

Step S301, extracting an original character string corresponding to a target character string from the original text;

Step S302, based on the coincidence ratio comparison of the original character string and the predicted character string, a first evaluation index is obtained.

In steps S301 to S302 of some embodiments of the present application, an original character string corresponding to a target character string is extracted from an original text, and then, a first evaluation index is obtained based on a coincidence ratio comparison between the original character string and a predicted character string. It should be noted that, the predicted strings are predicted results of the test language model based on the target strings, and each group of predicted strings corresponds to each sentence in the original text, so in order to compare the predicted result data with the original text in a coincidence degree, on the basis of obtaining a plurality of groups of predicted strings according to the target string prediction, it is also necessary to extract the original strings corresponding to the target strings from a plurality of sentences in the original text, and when the target strings corresponding to one group of predicted strings are consistent with the target strings corresponding to one group of original strings, the coincidence degree comparison can be performed based on the original strings and the predicted strings, so that the first evaluation index can be obtained. It is emphasized that the first evaluation index determines whether the target text generated by the text enhancement process is related to the original text in the statistical dimension by comparing the coincidence ratio of the predicted result data and the original text, and since the purpose of text enhancement is to increase the type and the number of the text based on the original text, the higher the association degree of the target text with the original text in the statistical dimension is, the higher the quality of the target text is.

Through the steps S301 to S302, the reference for the comparison of the overlap ratio can be determined in the prediction result data and the target text, so that a more convenient method can be provided for determining the first evaluation index, and the quality evaluation efficiency of the target text is further improved.

In some more specific embodiments of the present application, when the original character string is "today/weather/true/ok/go" the first type text enhancement processing and the second type text enhancement processing are respectively performed based on the original character string. The method comprises the steps that a first target character string obtained through first-class text enhancement processing is 'today/weather/good/capable of going out/strolling', prediction processing is carried out on the group of target character strings through a test language model, so that a first predicted character string is obtained, and as the 'today', 'weather', 'good', 'going out of the door', 'strolling' of the first predicted character string can find coincident characters in an original character string, corresponding first evaluation indexes are higher, and therefore the quality of the first target character string is higher; the second target character string obtained by the second type of text enhancement processing is 'present/climate/true fragrance/possibility/series gate/scatter', the group of target character strings are predicted by the test language model, so that a second predicted character string 'present/climate/happy/perhaps/series gate/chat' is obtained, and as the second predicted character string only has 'present', 'gas', the overlapped characters can be found in the original character string, the corresponding first evaluation index is lower, and therefore the quality of the second target character string is lower. It should be appreciated that the manner of obtaining the first evaluation index corresponding to the target text by performing the overlap ratio comparison between the original text and the predicted result data is various, and may include, but not limited to, the specific embodiments described above.

In steps S104 to S105 in some embodiments of the present application, semantic feature extraction is performed on an original text to obtain an original semantic feature, semantic feature extraction is performed on a target text to obtain a target semantic feature, and similarity comparison is performed on the original semantic feature and the target semantic feature to obtain a second evaluation index corresponding to the target text. It should be noted that, the original semantic feature is a feature vector characterizing the meaning of the original text sentence, and the target semantic feature is a feature vector characterizing the meaning of the target text sentence.

It should be noted that the main purpose of extracting semantic features is to reduce the number of words to be processed without damaging the core semantic information of the text, so as to reduce the vector space dimension, thereby simplifying the calculation and improving the speed and efficiency of text processing. In some embodiments, high quality semantic features may include the following features: firstly, the semantic features can be used for truly marking the text content; secondly, the semantic features have the capability of distinguishing target text from other text; thirdly, the number of semantic features cannot be too large; fourth, semantic feature separation is relatively easy to implement. The method for extracting the semantic features may be to extract the semantic features of the original text and the target text by using a natural language model, and optional natural language models include, but are not limited to: BERT pre-training models, multitasking deep neural network (Multi-Task Deep Neural Networks, MT-DNN) models, XLNet models, etc.

It should be understood that after the original semantic features and the target semantic features are obtained, the original semantic features and the target semantic features are further compared in similarity, so as to compare whether the sentence meanings of the target text and the original text are similar, thereby obtaining a second evaluation index corresponding to the target text, so as to evaluate whether the target text obtained after the text enhancement processing is similar to the sentence meaning of the original text on the basis that the types and the number of the texts are increased. It should be noted that, since the purpose of text enhancement is to increase the type and number of texts based on the original text, the greater the association of the target text with the original text in the semantic dimension, the higher the quality of the target text will be explained based on the increased type and number of texts.

In some more specific embodiments, the BERT core corresponding to the BERT pre-training model may be used as a second evaluation index, and the target text and the original text are encoded at a word level (token level), so that the accuracy (Precision), recall (Recall) and semantic evaluation index (F) of the token level are calculated according to the cosine similarity of the token encoding. The cosine similarity is adopted, so that the problem of dead plates caused by accurate matching is solved, and in addition, the context is considered in BERT coding, so that the coding of each token is fused with the context information. In addition, to take into account the importance of different token, BERTSCore may also assign weights to different words based on the Term Frequency inverse text Frequency index (Term Frequency-Inverse Document Frequency, TF-IDF), thereby more accurately calculating the BERTSCore value (Precision, recall, F).

Referring to fig. 4, step S104 according to some embodiments of the present application may include, but is not limited to, steps S401 to S403 described below.

Step S401, performing first word segmentation processing on an original text to obtain a plurality of first phrases;

step S402, performing second word segmentation processing on the target text to obtain a plurality of second word groups;

step S403, extracting semantic features of each first phrase and each second phrase based on the pre-trained semantic recognition model to obtain original semantic features of each first phrase and target semantic features of each second phrase.

In steps S401 to S402 of some embodiments of the present application, a first word segmentation process is performed on an original text to obtain a plurality of first word groups, and a second word segmentation process is performed on a target text to obtain a plurality of second word groups. It is clear that the purpose of performing the first word segmentation on the original text is to provide convenience for extracting the original semantic features of the original text, and similarly, the purpose of performing the first word segmentation on the target text is to provide convenience for extracting the target semantic features of the target text. It should be noted that word segmentation processing is the basis of natural language processing, and word segmentation accuracy directly determines the quality of semantic features. In some embodiments, because English sentences use spaces to separate words, word segmentation issues need not be considered in most cases except for certain specific words (e.g., how many, new York, etc.). However, different Chinese characters are naturally lack of separators, and readers are required to divide words and break sentences by themselves, so that when Chinese natural language processing is performed, the words are required to be divided first. Aiming at Chinese word segmentation, the current word segmentation method is mainly divided into two types, namely a dictionary-based rule matching method and a statistical-based machine learning method. First, dictionary-based word segmentation algorithms are essentially string matches. And matching the character strings to be matched with a dictionary large enough based on a certain algorithm strategy, and if the matching hits, word segmentation can be performed. According to different matching strategies, the method is divided into a forward maximum matching method, a reverse maximum matching method, two-way matching word segmentation, full segmentation path selection and the like; secondly, a word segmentation algorithm based on statistics is essentially a sequence labeling problem. We mark the words in the sentence according to their position in the word. The labels are mainly as follows: b (one word at the beginning of the word), E (the last word of the word), M (the word in the middle of the word, possibly multiple), S (the word represented by one word). For example, "today's weather is true and good", the result after labeling is "besbeesbebme", and the corresponding word segmentation result is "today/weather/true/good". It should be understood that the word segmentation process may be performed in a wide variety of ways, including, but not limited to, the specific embodiments described above.

In step S403 of some embodiments of the present application, semantic feature extraction is performed on each first phrase and each second phrase based on the pre-trained semantic recognition model, so as to obtain original semantic features of each first phrase and target semantic features of each second phrase. It is emphasized that the main purpose of extracting semantic features is to reduce the number of words to be processed without damaging the core semantic information of the text, so as to reduce the vector space dimension, thereby simplifying the calculation and improving the speed and efficiency of text processing. It should be noted that the semantic recognition model refers to a natural language model for recognizing semantic information in text, and the pre-training undergone by the semantic recognition model is just used for training the semantic feature extraction capability of the semantic recognition model. It is explicitly required that semantic features of the original text and the target text are extracted using natural language models, where alternative natural language models include, but are not limited to: BERT pre-training models, multitasking deep neural network (Multi-Task Deep Neural Networks, MT-DNN) models, XLNet models, etc.

Through the steps S401 to S403, first word segmentation is performed on the original text to obtain a plurality of first word groups, second word segmentation is performed on the target text to obtain a plurality of second word groups, and semantic feature extraction is performed on each first word group and each second word group based on a pre-trained semantic recognition model to obtain original semantic features of each first word group and target semantic features of each second word group, so that convenience can be provided for extracting the semantic features of the original text and the target text, and quality evaluation efficiency of the target text is further improved.

Referring to fig. 5, step S105 according to some embodiments of the present application may include, but is not limited to, the following steps S501 to S503.

Step S501, carrying out similarity calculation on the original semantic features of each first phrase and the target semantic features of each second phrase one by one to obtain a similarity matrix;

step S502, obtaining target accuracy and target recall corresponding to a target text based on original semantic features, target semantic features and a similarity matrix;

step S503, obtaining a second evaluation index according to the target accuracy and the target recall.

In step S501 of some embodiments of the present application, similarity calculation is performed on the original semantic features of each first phrase and the target semantic features of each second phrase to obtain a similarity matrix. It should be noted that, the similarity calculation is performed on the original semantic features of each first phrase and the target semantic features of each second phrase, so as to compare the semantic similarity of the original text and the target text, and thus, a second evaluation index for evaluating the quality of the target text in the semantic dimension is established. It should be noted that, the ways of calculating the similarity between the original semantic features of each first phrase and the target semantic features of each second phrase to obtain the similarity matrix are various, which may be to calculate the cosine similarity between the original semantic features of each first phrase and the target semantic features of each second phrase, so as to form a similarity matrix based on cosine similarity, or calculate the euclidean distance between the original semantic features of each first phrase and the target semantic features of each second phrase, so as to form a similarity matrix based on euclidean distance. It should be noted that cosine similarity can be used to measure the magnitude of the vector included angle between the original semantic features and the target semantic features, and the smaller the included angle is, the larger the cosine similarity is proved, and the more similar the two types of semantic vectors are explained. Euclidean distance is abbreviated as Euclidean distance, similarity calculation based on Euclidean distance refers to a similarity calculation method for mapping each original semantic feature and each target semantic feature in a two-dimensional coordinate system respectively and further carrying out Euclidean distance calculation based on two types of semantic features, and the smaller the Euclidean distance is, the more similar the two types of semantic vectors are explained. It should be understood that the method for performing the similarity calculation on the original semantic features of each first phrase and the target semantic features of each second phrase is various, and may include, but is not limited to, the specific embodiments mentioned above.

In step S502 to step S503 of some embodiments of the present application, a target accuracy and a target recall corresponding to the target text are obtained based on the original semantic features, the target semantic features, and the similarity matrix, and then a second evaluation index is obtained according to the target accuracy and the target recall. It should be noted that after the similarity matrix is obtained, a maximum similarity score accumulation and then normalization may be performed on the target semantic vector and the original semantic vector based on the similarity matrix, so as to obtain a target accuracy rate and a target Recall rate corresponding to the target text, and further, according to the target accuracy rate (Precision) and the target Recall rate (Recall), a semantic evaluation index (F) of the semantic dimension, that is, a second evaluation index, may be obtained.

Through the steps S502 to S503, similarity calculation is performed on the original semantic features of each first phrase and the target semantic features of each second phrase one by one to obtain a similarity matrix, and then a second evaluation index is obtained based on the original semantic features, the target semantic features and the similarity matrix. The second evaluation index reflecting that the semantic similarity of the target text and the original text is more accurate can be obtained.

Referring to fig. 6, step S502 according to some embodiments of the present application may include, but is not limited to, steps S601 to S604 described below.

Step S601, extracting similarity matching scores of each original semantic feature and each target semantic feature from a similarity matrix;

step S602, determining the maximum similarity score of each target semantic feature according to a plurality of similarity matching scores;

step S603, accumulating the maximum similarity score of each target semantic feature to obtain a similar accumulated value;

step S604, obtaining the target accuracy and the target recall according to the original semantic features, the target semantic features and the similar accumulated values.

The target accuracy and the target recall are obtained in the manner shown in the steps S601 to S604, the most similar original semantic features can be matched for each target semantic feature based on the maximum similarity score, the maximum similarity score of each target semantic feature is further accumulated to obtain a similar accumulated value, and further, the target accuracy and the target recall are obtained according to the original semantic features, the target semantic features and the similar accumulated value, so that a second evaluation index reflecting that the semantic similarity of the target text and the original text is more accurate can be obtained.

In some more specific embodiments, the calculation of the second evaluation index may be accomplished by:

first, a plurality of original semantic features of a first phrase are expressed as x= { X ₁ ,x ₂ ,x ₃ ,…,x _i -and representing the plurality of target semantic features of the second phrase as y= { Y ₁ ,y ₂ ,y ₃ ,…,y _j Where i represents the number of original semantic features and j represents the number of target semantic features.

Further, according to a plurality of original semantic features x= { X ₁ ,x ₂ ,x ₃ ,…,x _i According to a plurality of target semantic features y= { Y } ₁ ,y ₂ ,y ₃ ,…,y _j Simultaneous cosine similarity calculation formulaSimilarity calculation is carried out on each original semantic feature and each target semantic feature one by one, and a similarity matrix is obtained:

still further, based on a similarity matrixAnd calculating the target accuracy (Precision) and the target Recall (Recall) according to the following formulas to obtain a semantic evaluation index (F) of the semantic dimension:

/>

it should be noted that, the semantic evaluation index (F) in the above embodiment is a second evaluation index corresponding to the target text, so as to evaluate whether the target text obtained after the text enhancement processing is similar to the sentence meaning of the original text on the basis that the type and the number of the text are increased, and evaluate the quality of the target text from the semantic dimension. It should be understood that the manner of comparing the similarity between the original semantic features and the target semantic features to obtain the second evaluation index corresponding to the target text is various and is not limited to the specific embodiment.

Referring to fig. 7, step S502 according to some embodiments of the present application may further include, but is not limited to, the following steps S701 to S702.

Step S701, based on the reverse file frequency of each first phrase in the original text, evaluating weight is configured for each original semantic feature;

step S702, obtaining target accuracy and target recall according to the original semantic features, the evaluation weights, the similarity matrix and the target semantic features.

In steps S701 to S702 of some embodiments of the present application, firstly, based on the reverse file frequency of each first phrase in the original text, an evaluation weight is configured for each original semantic feature, and then, according to the original semantic feature, the evaluation weight, the similarity matrix and the target semantic feature, the target accuracy and the target recall are obtained. Note that Term Frequency reverse document Frequency (Term Frequency-Inverse Document Frequency, TF-IDF) is a weighting technique for information retrieval and data mining, where TF is Term Frequency (Term Frequency) and IDF is reverse document Frequency (Inverse Document Frequency). It should be noted that if a word or phrase appears frequently TF in one article is high and rarely in other articles, the word or phrase is considered to have a good category discrimination capability and is suitable for classification. Therefore, the reverse file frequency of each first phrase in the original text configures the evaluation weight for each original semantic feature, which is helpful to more accurately obtain the second evaluation index.

In step S106 of some embodiments of the present application, quality evaluation is performed on the target text based on the first evaluation index and the second evaluation index, so as to obtain evaluation result data. It should be noted that, by comparing the overlap ratio of the predicted result data and the original text, the first evaluation index determines whether the target text generated by the text enhancement processing is related to the original text in the statistical dimension, and since the purpose of text enhancement is to increase the type and the number of the text based on the original text, the higher the association degree of the target text and the original text in the statistical dimension is, the higher the quality of the target text is. In addition, the second evaluation index is used for evaluating whether the target text obtained after text enhancement processing is similar to the sentence meaning of the original text on the basis that the type and the number of the text are increased, and if the association degree of the target text with the original text in the semantic dimension is larger, the quality of the target text is higher. Based on the first evaluation index and the second evaluation index, the quality evaluation of the target text in two aspects from the statistical dimension and the semantic dimension can be performed, so that evaluation result data is obtained, and the quality of the text generated by the text enhancement processing can be objectively and accurately evaluated through the evaluation result data.

Referring to fig. 8, step S106 according to some embodiments of the present application may include, but is not limited to, steps S801 to S803 described below.

Step S801, performing word frequency characteristic evaluation on a target text based on a first evaluation index to obtain first evaluation data;

step S802, semantic feature evaluation is carried out on the target text based on a second evaluation index, so that second evaluation data are obtained;

step 803, obtaining the evaluation result data according to the first evaluation data and the second evaluation data.

In steps S801 to S803 of some embodiments of the present application, word frequency feature evaluation is performed on a target text based on a first evaluation index to obtain first evaluation data, semantic feature evaluation is performed on the target text based on a second evaluation index to obtain second evaluation data, and further, evaluation result data is obtained according to the first evaluation data and the second evaluation data. It should be noted that, by comparing the overlap ratio of the predicted result data and the original text, the first evaluation index determines whether the target text generated by the text enhancement processing is related to the original text in the statistical dimension, and since the purpose of text enhancement is to increase the type and the number of the text based on the original text, the higher the association degree between the target text and the original text in the statistical dimension is, the higher the quality of the target text is, so that word frequency characteristic evaluation can be performed on the target text based on the first evaluation index, and the first evaluation data for evaluating the quality of the target text in the statistical dimension can be obtained. In addition, the second evaluation index is used for evaluating whether the target text obtained after the text enhancement processing is similar to the sentence meaning of the original text on the basis that the type and the number of the text are increased, if the association degree of the target text and the original text in the semantic dimension is larger, the quality of the target text is higher, so that the second evaluation data for evaluating the quality of the target text from the semantic dimension can be obtained by evaluating the semantic features of the target text based on the second evaluation index. In some embodiments, the first evaluation index and the second evaluation index may be fitted to obtain a comprehensive evaluation index, and then the target text is further evaluated based on the comprehensive evaluation index to obtain evaluation result data. It should be noted that if the first evaluation index is confusion degree Perplexity, the second evaluation index is semantic evaluation index (F), wherein the smaller the Perplexity, the higher the quality of the target text evaluated in the statistical dimension, and the larger the F, the higher the quality of the target text evaluated in the semantic dimension, so for the evaluation mode of the unified two kinds of indexes, the comprehensive evaluation index may be set to be F×1/Perplexity.

Through the steps S801 to S803, the quality of the target text obtained after the text enhancement processing can be evaluated by using the first evaluation index and the second evaluation index from the two evaluation dimensions having reference values, namely the statistical dimension and the semantic dimension, respectively, so as to obtain evaluation result data, and the quality of the text generated by the text enhancement processing can be evaluated objectively and accurately by using the evaluation result data.

Fig. 9 shows an electronic device 900 provided by an embodiment of the application. The electronic device 900 includes: a processor 901, a memory 902, and a computer program stored on the memory 902 and executable on the processor 901, the computer program when executed for performing the text enhancement quality assessment method described above.

The processor 901 and the memory 902 may be connected by a bus or other means.

The memory 902, as a non-transitory computer readable storage medium, may be used to store a non-transitory software program as well as a non-transitory computer executable program, such as the text enhancement quality assessment method described in embodiments of the present application. The processor 901 implements the text enhancement quality assessment method described above by running non-transitory software programs and instructions stored in the memory 902.

The memory 902 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area. The storage data area may store text enhancement quality assessment methods described above. In addition, the memory 902 may include high-speed random access memory 902 and may also include non-transitory memory 902, such as at least one storage device memory device, flash memory device, or other non-transitory solid state memory device. In some implementations, the memory 902 optionally includes memory 902 located remotely from the processor 901, the remote memory 902 being connectable to the electronic device 900 through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The non-transitory software programs and instructions required to implement the above-described text enhancement quality assessment method are stored in the memory 902 and when executed by the one or more processors 901 perform the above-described text enhancement quality assessment method, for example, performing method steps S101 through S106 in fig. 1, method steps S201 through S203 in fig. 2, method steps S301 through S302 in fig. 3, method steps S401 through S403 in fig. 4, method steps S501 through S503 in fig. 5, method steps S601 through S604 in fig. 6, method steps S701 through S702 in fig. 7, and method steps S801 through S803 in fig. 8.

The embodiment of the application also provides a computer readable storage medium which stores computer executable instructions for executing the text enhancement quality assessment method.

In an embodiment, the computer-readable storage medium stores computer-executable instructions that are executed by one or more control processors, for example, to perform method steps S101 through S106 in fig. 1, method steps S201 through S203 in fig. 2, method steps S301 through S302 in fig. 3, method steps S401 through S403 in fig. 4, method steps S501 through S503 in fig. 5, method steps S601 through S604 in fig. 6, method steps S701 through S702 in fig. 7, and method steps S801 through S803 in fig. 8.

The above described apparatus embodiments are merely illustrative, wherein the units illustrated as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, storage device storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any information delivery media. It should also be appreciated that the various embodiments provided by the embodiments of the present application may be arbitrarily combined to achieve different technical effects.

While the preferred embodiment of the present application has been described in detail, the present application is not limited to the above embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit and scope of the present application, and these equivalent modifications or substitutions are included in the scope of the present application as defined in the appended claims.

Claims

1. A text enhancement quality assessment method, comprising:

2. The method according to claim 1, wherein the quality evaluation of the target text based on the first evaluation index and the second evaluation index to obtain evaluation result data includes:

3. The method of claim 1, wherein the predicting the target text based on the pre-trained test language model to obtain predicted result data comprises:

4. The method of claim 3, wherein the comparing the original text with the predicted result data to obtain the first evaluation index corresponding to the target text includes:

5. The method according to any one of claims 1 to 4, wherein the extracting semantic features from the original text to obtain original semantic features, and extracting semantic features from the target text to obtain target semantic features, includes:

6. The method of claim 5, wherein the comparing the similarity between the original semantic features and the target semantic features to obtain a second evaluation index corresponding to the target text comprises:

7. The method of claim 6, wherein the obtaining a target accuracy rate and a target recall rate corresponding to the target text based on the original semantic features, the target semantic features, and the similarity matrix comprises:

8. The method of claim 6, wherein the obtaining a target accuracy rate and a target recall rate corresponding to the target text based on the original semantic features, the target semantic features, and the similarity matrix, further comprises:

9. An electronic device, comprising: a memory, a processor storing a computer program, the processor implementing the text enhancement quality assessment method according to any one of claims 1 to 8 when the computer program is executed.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that is executed by a processor to implement the text enhancement quality evaluation method according to any one of claims 1 to 8.