CN117634431A - Method and system for evaluating text style conversion quality - Google Patents
Method and system for evaluating text style conversion quality Download PDFInfo
- Publication number
- CN117634431A CN117634431A CN202410108976.4A CN202410108976A CN117634431A CN 117634431 A CN117634431 A CN 117634431A CN 202410108976 A CN202410108976 A CN 202410108976A CN 117634431 A CN117634431 A CN 117634431A
- Authority
- CN
- China
- Prior art keywords
- text
- model
- similarity
- confusion
- style
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 title claims abstract description 30
- 102100033814 Alanine aminotransferase 2 Human genes 0.000 claims abstract description 30
- 101710096000 Alanine aminotransferase 2 Proteins 0.000 claims abstract description 30
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 5
- 238000012549 training Methods 0.000 claims description 28
- 238000011156 evaluation Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000011176 pooling Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 2
- 238000005259 measurement Methods 0.000 claims description 2
- 238000012544 monitoring process Methods 0.000 claims 1
- 238000013441 quality evaluation Methods 0.000 abstract description 3
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000010801 machine learning Methods 0.000 abstract description 2
- 238000003058 natural language processing Methods 0.000 description 3
- 238000001303 quality assessment method Methods 0.000 description 3
- 238000010200 validation analysis Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- 238000011524 similarity measure Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of artificial intelligence, in particular to a method and a system for evaluating text style conversion quality, which comprises the following steps: the text style classifier is used for classifying and evaluating, and textCNN is introduced as a text classifier, so that the textCNN effectively captures local characteristics of the text in a convolutional neural network structure; semantic similarity assessment, introducing BERTSCore similarity indexes; statement fluency assessment, namely calculating statement confusion degree by using a GPT-2 language model; the beneficial effects are as follows: the method and the system for evaluating the text style conversion quality provided by the invention utilize the machine learning text classifier, the BERTSCore similarity index and the statement confusion index to provide comprehensive and objective quality evaluation for the text style conversion task.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a method and a system for evaluating text style conversion quality.
Background
In the information age today, natural Language Processing (NLP) technology is evolving rapidly, offering more text processing and application possibilities to people.
Text style conversion is one of the important tasks in the NLP field in the prior art, aiming at converting text from one style to another, such as formal documents to informal spoken language. The technology has wide application prospect in the fields of advertisement, social media content generation, text creation and the like.
However, quality assessment of text style conversion has been a challenging problem. Traditionally, assessing text quality has focused mainly on grammar correctness and smoothness, however, as text generation techniques evolve, quality assessment requires more attention to semantic preservation and accuracy of style conversion. The existing assessment method usually only covers part of aspects, and lacks comprehensiveness and objectivity.
Disclosure of Invention
The invention aims to provide a method and a system for evaluating text style conversion quality, which are used for solving the problems in the background technology.
In order to achieve the above purpose, the present invention provides the following technical solutions: a method of assessing text style conversion quality, the method comprising the steps of:
the text style classifier is used for classifying and evaluating, and textCNN is introduced as a text classifier, so that the textCNN effectively captures local characteristics of the text in a convolutional neural network structure;
semantic similarity assessment, introducing BERTSCore similarity indexes;
statement fluency assessment, statement confusion is calculated using a GPT-2 language model.
Preferably, the specific operations of the text style classifier classification evaluation include:
training by using a preprocessed text data set, manually marking the style category of the converted text, using the converted text and the original text as training data, and learning the characteristic representation of different layers by the textCNN through the combination of a plurality of convolution layers and pooling layers so as to realize the training of the style classifier;
in the classification stage, the trained TextCNN model is used for distributing the converted text to different style categories, and through forward propagation of the converted text, the model distributes a confidence score for each category, selects the category with the highest confidence as the style category of the converted text, and realizes the evaluation of the accuracy of the text category.
Preferably, the specific operation of semantic similarity evaluation includes:
pre-training model selection, namely using a pre-trained BERT model as a basis for calculating similarity, and respectively encoding an original text and a converted text to obtain hidden representations of the original text and the converted text;
and calculating similarity, namely calculating BERTSCore between the original text and the converted text by using hidden representation, quantitatively evaluating whether the text conversion process keeps semantic consistency by adopting cosine similarity as similarity measurement, and indicating that the converted text is more similar to the original text in terms of semantics by higher BERTSCore score.
Preferably, the specific operation of statement fluency assessment includes:
the GPT-2 model is applied, the generated conversion text is further processed by adopting a GPT-2 language model, and the GPT-2 model generates a text with fluent language expression by inputting the converted text;
confusion computation, using the confusion of the generated text to evaluate its fluency and understandability, sentence confusion is measured by computing the uncertainty of the model's prediction of the next word in a given context, with lower confusion values indicating that the generated text is more fluent in language expression and easy to understand.
A system for evaluating text style conversion quality comprises a textCNN text classifier module, a BERTSCore similarity index module and a GPT-2 statement confusion module;
the textCNN text classifier module is used for collecting enough original texts and converted texts thereof, manually labeling style types of the original texts and dividing the data into a training set and a verification set; constructing a textCNN model, wherein the textCNN model comprises a convolution layer and a pooling layer, the convolution layer captures text features of different scales by using a plurality of convolution kernels of different sizes, and the pooling layer is used for reducing and retaining key information;
the BERTSCore similarity index module comprises BERT model selection, text coding and similarity calculation;
GPT-2 statement confusion module, including GPT-2 model application and statement confusion computation.
Preferably, the TextCNN text classifier module minimizes the loss function by performing multiple rounds of training on the training set, optimizes model parameters, monitors the performance of the model using the validation set, and stops training when performance stagnates to avoid overfitting; in the classification stage, the trained model is applied to the converted text, the confidence coefficient of each category is calculated through forward propagation, and the category with the highest confidence coefficient is selected as the style category of the converted text, so that the accuracy evaluation of the text category is realized.
Preferably, the BERT model selection: selecting a pre-trained BERT model, such as a BERT-Chinese model, as a basis for calculating the similarity;
text encoding: performing BERT coding on the original text and the converted text respectively to obtain hidden representation of the original text and the converted text, wherein the BERT coding captures context information and semantic features of the text;
similarity calculation: based on BERT coding, cosine similarity between the original text and the converted text is calculated as BERTSCore similarity score, and whether semantic consistency is maintained in the conversion process is quantitatively evaluated.
Preferably, statement confusion calculation: by inputting the generated text, calculating statement confusion of the generated text by using a GPT-2 model, wherein the lower the confusion value is, the smoother and easy to understand the generated text in language expression is represented;
and comprehensively applying a textCNN classifier, a BERTSCore similarity index and a GPT-2 confusion index.
Compared with the prior art, the invention has the beneficial effects that:
the method and the system for evaluating the text style conversion quality provided by the invention utilize the machine learning text classifier, the BERTSCore similarity index and the statement confusion index to provide comprehensive and objective quality evaluation for the text style conversion task.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
In order to make the objects, technical solutions, and advantages of the present invention more apparent, the embodiments of the present invention will be further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are some, but not all, embodiments of the present invention, are intended to be illustrative only and not limiting of the embodiments of the present invention, and that all other embodiments obtained by persons of ordinary skill in the art without making any inventive effort are within the scope of the present invention.
Embodiment one:
referring to fig. 1, the present invention provides a technical solution: a method of assessing text style conversion quality, the method comprising the steps of:
1. text style classifier classification evaluation
In the text style conversion quality evaluation method of the present invention, first, we introduce TextCNN (Convolutional Neural Network) as a text classifier. TextCNN can effectively capture local features of text in convolutional neural network structures. The training and use of the module comprises the following steps:
training phase: we use a pre-processed text data set for training. For the converted text, we manually annotate its style category and use it as training data along with the original text. The textCNN can learn different levels of characteristic representation through the combination of a plurality of convolution layers and pooling layers, so that training of the style classifier is realized.
Classification stage: the trained TextCNN model is used to assign the converted text into different style categories. By forward propagating the converted text, the model will assign a confidence score to each category. Finally, we select the category with the highest confidence as the style category of the converted text, thereby enabling an assessment of text category accuracy.
2. Semantic similarity assessment
To measure the semantic similarity between the converted text and the original text, we introduced the BERTScore similarity index. The following is a detailed description of the module:
pre-training model selection: we use a pre-trained BERT model (e.g., BERT-Chinese) as the basis for computing similarity. By encoding the original text and the converted text separately, we obtain a hidden representation of both.
Similarity calculation: using the hidden representation, we calculate BERTSCore between the original text and the converted text. By using cosine similarity as a similarity measure, we can quantitatively evaluate whether the text conversion process maintains semantic consistency. A higher BERTScore score indicates that the converted text is semantically more similar to the original text.
3. Statement fluency assessment
To evaluate the fluency and understandability of the generated converted text, we use the GPT-2 language model to calculate statement confusion. The following is a detailed description of the module:
GPT-2 model application: we employ a GPT-2 language model to further process the generated converted text. By entering the converted text, the GPT-2 model generates a text with a fluent linguistic representation.
And (5) calculating confusion degree: we use the confusion of the generated text to evaluate its fluency and understandability. Statement confusion is measured by calculating the uncertainty of the model's prediction of the next word in a given context. A lower confusion value indicates that the generated text is more fluent in language expression and easy to understand.
The invention can provide a comprehensive and objective text style conversion quality assessment method and provides an innovative solution for the field of text style conversion by comprehensively analyzing the textCNN classifier, the BERTSCore similarity index and the GPT-2 confusion index.
Embodiment two:
on the basis of the first embodiment, a system for evaluating text style conversion quality is provided, wherein the system consists of a text classifier module, a BERTSCore similarity index module and a GPT-2 statement confusion degree module;
1. textCNN text classifier module
Training data set preparation: enough original text and converted text thereof are collected, and style categories thereof are manually marked. These data are divided into training and validation sets.
TextCNN architecture: the TextCNN model is built, including a convolution layer and a pooling layer. The convolution layer uses a plurality of different sized convolution kernels to capture text features of different dimensions. The pooling layer is used for reducing and retaining key information.
Training process: the model parameters are optimized by performing multiple rounds of training on the training set using a gradient descent algorithm to minimize the loss function. The performance of the model is monitored using a validation set and training is stopped when performance stagnates to avoid overfitting.
Classification stage: in the classification stage, a trained model is applied to the converted text, and confidence of each category is calculated through forward propagation. And selecting the category with the highest confidence as the style category of the converted text, and realizing the evaluation of the accuracy of the text category.
2. BERTSCore similarity index module
BERT model selection: a pre-trained BERT model, such as BERT-Chinese, is selected as the basis for computing the similarity.
Text encoding: and performing BERT coding on the original text and the converted text respectively to obtain hidden representations of the original text and the converted text. BERT encoding is capable of capturing contextual information and semantic features of text.
Similarity calculation: based on BERT coding, cosine similarity between the original text and the converted text is calculated as BERTSCore similarity score, and whether semantic consistency is maintained in the conversion process is quantitatively evaluated.
3. GPT-2 statement confusion degree module
GPT-2 model application: the converted text is generated using a pre-trained GPT-2 language model, resulting in text with fluent language expression.
Statement confusion calculation: by inputting the generated text, the statement confusion of the generated text is calculated using the GPT-2 model. The lower the confusion value, the more fluent and easy to understand in language the generated text is represented.
The method and the system can comprehensively and objectively evaluate the quality of text style conversion by comprehensively applying the textCNN classifier, the BERTSCore similarity index and the GPT-2 confusion index.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (8)
1. A method of evaluating text style conversion quality, characterized by: the method comprises the following steps:
the text style classifier is used for classifying and evaluating, and textCNN is introduced as a text classifier, so that the textCNN effectively captures local characteristics of the text in a convolutional neural network structure;
semantic similarity assessment, introducing BERTSCore similarity indexes;
statement fluency assessment, statement confusion is calculated using a GPT-2 language model.
2. A method of assessing a quality of text style conversion in accordance with claim 1 wherein: specific operations of the text style classifier classification evaluation include:
training by using a preprocessed text data set, manually marking the style category of the converted text, using the converted text and the original text as training data, and learning the characteristic representation of different layers by the textCNN through the combination of a plurality of convolution layers and pooling layers so as to realize the training of the style classifier;
in the classification stage, the trained TextCNN model is used for distributing the converted text to different style categories, and through forward propagation of the converted text, the model distributes a confidence score for each category, selects the category with the highest confidence as the style category of the converted text, and realizes the evaluation of the accuracy of the text category.
3. A method of assessing a quality of text style conversion in accordance with claim 1 wherein: specific operations of semantic similarity evaluation include:
pre-training model selection, namely using a pre-trained BERT model as a basis for calculating similarity, and respectively encoding an original text and a converted text to obtain hidden representations of the original text and the converted text;
and calculating similarity, namely calculating BERTSCore between the original text and the converted text by using hidden representation, quantitatively evaluating whether the text conversion process keeps semantic consistency by adopting cosine similarity as similarity measurement, and indicating that the converted text is more similar to the original text in terms of semantics by higher BERTSCore score.
4. A method of assessing a quality of text style conversion in accordance with claim 1 wherein: the concrete operation of statement fluency assessment comprises the following steps:
the GPT-2 model is applied, the generated conversion text is further processed by adopting a GPT-2 language model, and the GPT-2 model generates a text with fluent language expression by inputting the converted text;
confusion computation, using the confusion of the generated text to evaluate its fluency and understandability, sentence confusion is measured by computing the uncertainty of the model's prediction of the next word in a given context, with lower confusion values indicating that the generated text is more fluent in language expression and easy to understand.
5. A system for assessing the quality of text style conversion, applied to a method for assessing the quality of text style conversion as claimed in any one of claims 1 to 4, characterized in that: the system comprises a text classifier module, a BERTSCore similarity index module and a GPT-2 statement confusion module;
the textCNN text classifier module is used for collecting enough original texts and converted texts thereof, manually labeling style types of the original texts and dividing the data into a training set and a verification set; constructing a textCNN model, wherein the textCNN model comprises a convolution layer and a pooling layer, the convolution layer captures text features of different scales by using a plurality of convolution kernels of different sizes, and the pooling layer is used for reducing and retaining key information;
the BERTSCore similarity index module comprises BERT model selection, text coding and similarity calculation;
GPT-2 statement confusion module, including GPT-2 model application and statement confusion computation.
6. The system for assessing the quality of text style conversion of claim 5 wherein: the textCNN text classifier module is used for minimizing a loss function by performing multi-round training on a training set, optimizing model parameters, monitoring the performance of a model by using a verification set, and stopping training when the performance is stagnant so as to avoid overfitting; in the classification stage, the trained model is applied to the converted text, the confidence coefficient of each category is calculated through forward propagation, and the category with the highest confidence coefficient is selected as the style category of the converted text, so that the accuracy evaluation of the text category is realized.
7. The system for assessing the quality of text style conversion of claim 5 wherein: BERT model selection: selecting a pre-trained BERT model, such as a BERT-Chinese model, as a basis for calculating the similarity;
text encoding: performing BERT coding on the original text and the converted text respectively to obtain hidden representation of the original text and the converted text, wherein the BERT coding captures context information and semantic features of the text;
similarity calculation: based on BERT coding, cosine similarity between the original text and the converted text is calculated as BERTSCore similarity score, and whether semantic consistency is maintained in the conversion process is quantitatively evaluated.
8. The system for assessing the quality of text style conversion of claim 5 wherein: statement confusion calculation: by inputting the generated text, calculating statement confusion of the generated text by using a GPT-2 model, wherein the lower the confusion value is, the smoother and easy to understand the generated text in language expression is represented;
and comprehensively applying a textCNN classifier, a BERTSCore similarity index and a GPT-2 confusion index.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410108976.4A CN117634431A (en) | 2024-01-26 | 2024-01-26 | Method and system for evaluating text style conversion quality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410108976.4A CN117634431A (en) | 2024-01-26 | 2024-01-26 | Method and system for evaluating text style conversion quality |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117634431A true CN117634431A (en) | 2024-03-01 |
Family
ID=90029033
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410108976.4A Pending CN117634431A (en) | 2024-01-26 | 2024-01-26 | Method and system for evaluating text style conversion quality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117634431A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109189985A (en) * | 2018-08-17 | 2019-01-11 | 北京达佳互联信息技术有限公司 | Text style processing method, device, electronic equipment and storage medium |
| US20210174033A1 (en) * | 2019-12-05 | 2021-06-10 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for evaluating translation quality |
| CN114757205A (en) * | 2022-05-11 | 2022-07-15 | 平安科技（深圳）有限公司 | Text evaluation method, device, equipment and storage medium |
| CN115719058A (en) * | 2022-11-28 | 2023-02-28 | 阿里巴巴（中国）有限公司 | Content analysis method, electronic equipment and storage medium |
| CN116962787A (en) * | 2023-07-31 | 2023-10-27 | 山东浪潮科学研究院有限公司 | Interaction method, device, equipment and storage medium based on video information |
| CN116975214A (en) * | 2023-02-13 | 2023-10-31 | 腾讯科技（深圳）有限公司 | Text generation method, device, storage medium and computer equipment |
-
2024
- 2024-01-26 CN CN202410108976.4A patent/CN117634431A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109189985A (en) * | 2018-08-17 | 2019-01-11 | 北京达佳互联信息技术有限公司 | Text style processing method, device, electronic equipment and storage medium |
| US20210174033A1 (en) * | 2019-12-05 | 2021-06-10 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for evaluating translation quality |
| CN114757205A (en) * | 2022-05-11 | 2022-07-15 | 平安科技（深圳）有限公司 | Text evaluation method, device, equipment and storage medium |
| CN115719058A (en) * | 2022-11-28 | 2023-02-28 | 阿里巴巴（中国）有限公司 | Content analysis method, electronic equipment and storage medium |
| CN116975214A (en) * | 2023-02-13 | 2023-10-31 | 腾讯科技（深圳）有限公司 | Text generation method, device, storage medium and computer equipment |
| CN116962787A (en) * | 2023-07-31 | 2023-10-27 | 山东浪潮科学研究院有限公司 | Interaction method, device, equipment and storage medium based on video information |
Non-Patent Citations (1)
| Title |
|---|
| 付聪: "《业务驱动的推荐系统方法与实践》", 31 January 2023, 北京：机械工业出版社, pages: 118 - 119 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021184311A1 (en) | Method and apparatus for automatically generating inference questions and answers | |
| CN108363753B (en) | Comment text emotion classification model training and emotion classification method, device and equipment | |
| CN110413741B (en) | Subjective question-oriented intelligent paper marking method | |
| CN110364251B (en) | Intelligent interactive diagnosis guide consultation system based on machine reading understanding | |
| CN111382565B (en) | Emotion-reason pair extraction method and system based on multiple labels | |
| US11409964B2 (en) | Method, apparatus, device and storage medium for evaluating quality of answer | |
| CN111339255A (en) | Target emotion analysis method, model training method, medium, and device | |
| CN113314100B (en) | Method, device, equipment and storage medium for evaluating and displaying results of spoken language test | |
| van Niekerk et al. | Analyzing speaker information in self-supervised models to improve zero-resource speech processing | |
| CN112687374B (en) | Psychological crisis early warning method based on text and image information joint calculation | |
| CN113128206B (en) | Question generation method based on word importance weighting | |
| CN113111152A (en) | Depression detection method based on knowledge distillation and emotion integration model | |
| CN113158671B (en) | Open domain information extraction method combined with named entity identification | |
| CN112307760A (en) | Deep learning-based financial report emotion analysis method and device and terminal | |
| CN114818717A (en) | Chinese named entity recognition method and system fusing vocabulary and syntax information | |
| Kawintiranon et al. | PoliBERTweet: a pre-trained language model for analyzing political content on Twitter | |
| Terada et al. | Automatic generation of fill-in-the-blank programming problems | |
| CN114742069A (en) | Code similarity detection method and device | |
| CN110852071A (en) | Knowledge point detection method, device, equipment and readable storage medium | |
| Liu et al. | Audio and video bimodal emotion recognition in social networks based on improved alexnet network and attention mechanism | |
| CN117591648A (en) | Power grid customer service co-emotion dialogue reply generation method based on emotion fine perception | |
| CN115376547B (en) | Pronunciation evaluation method, pronunciation evaluation device, computer equipment and storage medium | |
| CN115827879A (en) | Low-resource text intelligent review method and device based on sample enhancement and self-training | |
| CN115758218A (en) | Three-modal emotion analysis method based on long-time and short-time feature and decision fusion | |
| CN117634431A (en) | Method and system for evaluating text style conversion quality |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |