CN117973372A - Chinese grammar error correction method based on pinyin constraint - Google Patents

Abstract

The present invention relates to a Chinese grammar error correction method based on pinyin constraints, and belongs to the technical field of natural language processing. The present invention first constructs an end-to-end grammar error correction basic model based on the original BART model, and the model can make full use of the strong representation ability of the pre-trained language model itself to improve the error correction performance; then, after the BART encoding is completed, a detection layer is added to alleviate the problem of over-correction through effective error detection; then, the sound-like confusion set of the characters is used to construct a sound-like confusion matrix, and the sound-like confusion matrix is fused with the output of the detection layer to obtain the sound-like information of the characters containing errors in the input sentence; finally, the sound-like information is used to constrain the output probability of the decoding end, so as to obtain a more accurate error correction result.

Description

A Chinese grammar correction method based on phonetic constraints

Technical Language

The invention relates to a Chinese grammar error correction method based on pinyin constraints, belonging to the technical field of natural language processing.

Background technique

Chinese grammar correction is a key task in the field of natural language processing, which aims to identify and correct grammatical errors in Chinese text. These errors include but are not limited to incorrect word order, mismatched parts of speech, and structural errors, which may weaken the clarity and comprehensibility of the text. In view of this, the demand for Chinese grammar correction technology is gradually growing.

In order to enhance the accuracy and processing speed of Chinese grammar correction, it is crucial to develop high-performance correction models. These models can automatically detect and correct grammatical problems in texts. Furthermore, with the widespread application of Chinese texts in various fields and contexts, Chinese grammar correction models also need to adapt to different professional fields and changing usage environments to meet a wide range of users and application scenarios.

The goal of Chinese grammar error detection is to automatically detect grammatical errors in natural Chinese sentences, such as missing or redundant components, improper word order, etc. The detection tasks of Chinese grammar generally include: whether there is an error, the type of error, the location of the error, etc. Reasonable use of grammar detection can effectively improve error correction performance.

In short, Chinese grammar correction technology plays a vital role in improving text quality, enhancing user experience, and meeting diverse application needs. Grammar detection is one of the key technical means to achieve this goal. It helps ensure the correctness and professionalism of the text and reduce misunderstandings and communication barriers. In addition, it can improve the writing quality of non-native speakers, promote language learning, and in the field of natural language processing, it can enhance the accuracy of technologies such as machine translation and speech recognition. With the continuous advancement of technology, grammar detection will continue to promote the development of the field of Chinese grammar correction.

Summary of the invention

The present invention provides a Chinese grammar correction method based on pinyin constraints to solve the problem of low accuracy of Chinese grammar correction. The present invention has achieved good experimental results on the MagicData task.

The technical solution of the present invention is: a Chinese grammar error correction method based on pinyin constraints, the specific steps of the method are as follows:

Step 1: Construct a sequence-to-sequence grammatical error correction basic model based on the pre-trained model BART. The grammatical error correction basic model uses a multi-layer multi-head attention mechanism as the encoder and decoder to effectively capture context information. At the same time, it can also make full use of the strong representation ability of the BART pre-trained language model to enhance the error correction effect;

Step 2: Add a detection layer to the encoding end of the BART-based grammatical error correction basic model, and try to filter out correct sentences without correction through the detection module, thereby alleviating the over-correction problem;

Step 3: Use the phonetic confusion set of characters to construct a phonetic confusion matrix, and fuse the phonetic confusion matrix with the output of the detection layer to obtain the phonetic information of the wrong characters in the input sentence;

Step 4: Use the sound-like information to constrain the output probability of the decoding end, so as to obtain more accurate error correction results.

As a further solution of the present invention, the specific steps of Step 1 are as follows:

Step 1.1. Obtain the open-source and pre-trained Chinese BART model as our pre-trained language model;

Step 1.2. Since the BART pre-trained language model uses a multi-layer multi-head attention mechanism to build the encoder and decoder, we modify the BART model downloaded in the previous step to adapt to the task of grammatical error correction, thereby effectively utilizing the strong representation ability of the pre-trained language model to enhance the grammatical error correction model;

Step 1.3. Obtain the open access MagicData speech recognition dataset and download the speech recognition model from the Internet;

Step 1.4: Use the speech recognition model to automatically generate sentences containing phonetic errors. These sentences containing phonetic errors are combined with the correct sentences in the original dataset to form correct-incorrect sentence pairs, thereby constructing the basic dataset MagicData for grammatical correction tasks.

As a further solution of the present invention, the Step 2 includes the following:

Process the MagicData text dataset, construct the classification label of the current dataset, and add a detection layer to the encoding end of the basic model to alleviate the over-correction problem.

Specific steps are as follows:

Preprocess the labeled sentences in the training set and validation set, and classify them into two categories according to substitution errors and non-substitution errors. Substitution errors are one of the four error types in the grammatical error correction task, namely substitution errors, word order errors, redundancy errors, and missing errors. The labels of the two categories are 0 and 1 respectively. Only when a substitution error occurs is the label 1, and otherwise it is 0;

The specific steps of Step 2 are as follows:

Step 2.1. Add a detection layer after the encoder of the BART-based grammatical error correction basic model. Here, grammatical error detection is regarded as a simple binary classification task. If there is a substitution error at the current position, the output is 1, otherwise it is 0;

Step 2.2: Preprocess the labeled sentences in the training set and the validation set. If the current word has a replacement error, it is marked as 1, otherwise it is marked as 0.

Step 2.3: Take the weighted sum of the losses of the two tasks of grammatical error detection and grammatical error correction as the final loss, and update all model parameters by minimizing the final loss.

As a further solution of the present invention, the specific steps of Step 3 are as follows:

Step 3.1. Download the confusion sets of similar-sounding characters and similar-looking characters from some open source websites. De-duplicate and merge all the downloaded confusion sets to obtain the final confusion sets that include similar-sounding and similar-looking characters.

Step 3.2, pre-process the confusion set, use the tokenizer that comes with the BART-large model to id the characters in the dictionary, use the first character id as the key, and the id that sounds similar to the character as the value, and save it as a dictionary format file;

Step 3.3, read the dictionary during model training, and when the model performs word segmentation id on the input sentence, map the dictionary to the current word segmentation id, obtain the phonetic character information of the current word segmentation id, and construct a confusion phonetic matrix for the current phonetic information;

Step 3.4, multiply the confusion matrix obtained in the previous step by the result of model detection to obtain the confusion matrix of the sound-like information. Since the detection result is a binary classification, it is 1 only when a replacement error occurs. This ensures that the sound-like information will only be retained when a replacement error occurs in the current character.

As a further solution of the present invention, the specific steps of Step 4 are as follows:

Step 4.1, map the sound-like information confusion matrix obtained in Step 3.4 to a matrix of output probability dimension, and according to the sound-like information existing in the current confusion matrix, set the corresponding position in the dictionary to 1, and set the remaining positions to 0;

Step 4.2. Add the matrix with the same dimension as the output probability obtained in Step 4.1 to the final output probability to increase the probability of characters that sound similar to the current character.

The beneficial effects of the present invention are:

1. The present invention integrates common sound-similar confusion sets and shape-similar confusion sets, providing a basic confusion set for future research;

2. The present invention utilizes a detection model to alleviate the problem of over-correction and under-correction of the model.

3. The present invention utilizes sound-like information to effectively improve the accuracy of the model in correcting substitution errors.

4. The present invention uses incorrect sentences and correct sentences to construct edit sequences, and uses the edit sequences to construct labels for the detection model.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a flow chart of the present invention;

Detailed ways

Embodiment 1: As shown in FIG1 , a Chinese grammar error correction method based on pinyin constraints, the specific steps of the method are as follows:

Step 1: Construct a sequence-to-sequence grammatical error correction basic model based on the pre-trained model BART. The grammatical error correction basic model uses a multi-layer multi-head attention mechanism as the encoder and decoder to effectively capture context information. At the same time, it can also make full use of the strong representation ability of the BART pre-trained language model to enhance the error correction effect;

Download the speech recognition model and training set for preprocessing, and the Chinese BART pre-trained model.

The specific steps of Step 1 are as follows:

Step 1.1. Obtain the open-source and pre-trained Chinese BART model ¹ as our basic pre-trained language model. ;

Step 1.2. Since the BART pre-trained language model uses a multi-layer multi-head attention mechanism to build the encoder and decoder, we modify the BART model downloaded in the previous step to adapt to the task of grammatical correction, thereby effectively utilizing the strong representation ability of the pre-trained language model to enhance the performance of the grammatical correction model.

Step 1.3. Download the speech recognition model and MagicData speech recognition dataset.

Step 1.4: Use the speech recognition model to automatically generate sentences containing phonetic errors. These sentences containing phonetic errors are combined with the correct sentences in the original dataset to form correct-incorrect sentence pairs, thereby constructing the basic dataset MagicData for grammatical correction tasks.

Specifically, the downloaded MagicData speech recognition dataset and speech recognition model are used to generate the text MagicData grammatical error correction dataset; the generated MagicData data is preprocessed, mainly including deduplication, deletion of sentence pairs that are too long or too short, etc.; finally, 363.658 million relatively standardized grammatical error correction data are formed, and the number of datasets is shown in Table 1.

Table 1 Number of MagicData datasets

Dataset name Number of datasets/sentences MagicData-train 342,758 MagicData-dev 6,896 MagicData-test 14,004

Step 2: Add a detection layer to the encoding end of the BART-based grammatical error correction basic model, and try to filter out correct sentences without correction through the detection module, thereby alleviating the over-correction problem;

First, the MagicData dataset is processed to construct the classification label of the current dataset. Finally, a detection layer is added to the base model encoder to alleviate the over-correction problem.

The specific steps of Step 2 are as follows:

Step 2.1. Add a detection layer after the encoder of the BART-based grammatical error correction basic model. Here, grammatical error detection is regarded as a simple binary classification task. If there is a substitution error at the current position, the output is 1, otherwise it is 0;

Step 2.2: Preprocess the labeled sentences in the training set and the validation set. If the current word has a replacement error, it is marked as 1, otherwise it is marked as 0.

Step 2.3: Take the weighted sum of the losses of the two tasks of grammatical error detection and grammatical error correction as the final loss, and update all model parameters by minimizing the final loss.

The MagicData dataset is processed, and the wrong sentences and correct sentences are used to generate text with edit sequences. The edit sequence style is shown as "$STARTSEPL|||SEPR$KEEPplaySEPL|||SEPR$KEEPreleaseSEPL|||SEPR$KEEPworkSEPL|||SEPR$REPLACE_PalaceSEPL|||SEPR$REPLACE_ZeLiSEPL|||SEPR$REPLACE_ReasonSEPL|||SEPR$REPLACE_BenefitsSEPL|||SEPR$KEEPsongSEPL|||SEPR$KEEP". In this edit sequence, "KEEP" means that the character at the current position remains unchanged, and "REPLACE_Palace" means that the current character is replaced with "Palace". Using the edit sequence, the dataset is classified into two categories.

This basic model consists of two components, an encoder and a decoder, which work together to achieve the goal of grammatical error correction. The encoder uses a multi-head self-attention mechanism to generate rich contextual representations by modeling the context of each word in the source sentence. The decoder has a similar structure and introduces a masked multi-head sub-attention module to better capture the information of the generated words and ensure that the output sentence is grammatically and semantically accurate. At the same time, a pre-trained language model is used to enhance the model's context representation capabilities.

During the training process, the objective function is to minimize the cross entropy loss function, the formula is as follows:

θ is the trainable model parameter, x is the source sentence, y = {y ₁ ,y ₂ ,…,y _n } is the correct sentence with n words, and y _{< t} = {y ₁ ,y ₂ ,…,y _t-1 } is the word seen at the tth time step;

The modification to the base model includes the introduction of a detection layer at the end of the model. This detection layer has an independent loss function, which is added to the loss function of the original task according to a certain weight to form a total loss function. After calculating the total loss function, the gradient is calculated through back propagation, and the model parameters are updated using optimization algorithms such as gradient descent to minimize the total loss function. The total loss function of the model is shown below

in, is the detection layer loss function,/> is the error correction loss function.

In the prediction stage, beam search decoding is used to find an optimal sequence y ^* by maximizing the conditional probability P(y ^* |x; θ);

Step 3: Use the phonetic confusion set of characters to construct a phonetic confusion matrix, and fuse the phonetic confusion matrix with the output of the detection layer to obtain the phonetic information of the wrong characters in the input sentence;

The specific steps of Step 3 are as follows:

Step 3.1. Download similar-sounding character obfuscation sets from some open source websites. De-duplicate and merge all the downloaded obfuscation sets, arrange all the characters in the obfuscation sets, and put similar-sounding characters in the same row;

Step 3.2, pre-process the confusion set, use the tokenizer that comes with the BART model to id the characters in the dictionary, use the first character id as the key, and the ids of other characters that sound similar to the character as the value, and save it as a dictionary format file;

Step 3.3, read the dictionary during model training, and when the model performs word segmentation id on the input sentence, map the dictionary to the current word segmentation id, obtain the phonetic character information of the current word segmentation id, and construct a confusion phonetic matrix for the current phonetic information;

Step 3.4, multiply the confusion matrix obtained in the previous step by the result of model detection to obtain the confusion matrix of the sound-like information. Since the detection result is a binary classification, it is 1 only when a replacement error occurs. This ensures that the sound-like information will only be retained when a replacement error occurs in the current character.

Step 4: Use the sound-like information to constrain the output probability of the decoding end, so as to obtain more accurate error correction results.

The final results are shown in Table 2. Based on the MagicData data training of the basic model, we introduced the detection module and the pinyin constraint model, and the score difference between the two is significant. Compared with the basic model, the F value increased by 1 unit after the introduction of the detection module; and the F value increased by about 3 units after the introduction of the pinyin constraint module. This not only shows that there is room for improvement in the accuracy of replacing error types in the current grammatical error correction model, but also verifies the effectiveness of the pinyin constraint method we adopted.

The Chinese grammar evaluation index based on characters is used. The evaluation indexes used for Chinese grammar error correction are precision P, recall R and F _0.5 . The calculation results are as follows:

in:

TP (True Positive): The word that the model corrected from the wrong word.

FP (False Positive): The model changes the correct word to an incorrect word.

FN (False Negative): The model did not correct the wrong word.

Use the constructed MagicData text dataset as the training set to train the detection model, adjust the hyperparameters, and obtain the detection model with the best performance.

Table 2 Test results of basic model and detection model

The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims (5)

1. A Chinese grammar error correction method based on pinyin constraint is characterized in that: the method comprises the following specific steps:

step1, constructing a sequence-to-sequence grammar error correction basic model based on a pre-training model BART; the grammar error correction basic model adopts a multi-layer multi-head attention mechanism as an encoder and a decoder to effectively capture the context information, and simultaneously, the strong characterization capability of the BART pre-training language model is fully utilized to enhance the error correction effect;

Step2, adding a detection layer at the coding end of the syntax error correction basic model based on the BART, and attempting to filter out correct sentences through a detection module without correction so as to relieve the problem of excessive correction;

Step3, constructing a sound-like confusion matrix by utilizing a sound-like confusion set of the characters, and fusing the sound-like confusion matrix with the output of the detection layer to obtain sound-like information of the characters containing errors in the input sentence;

step4, constraint is carried out on the output probability of the decoding end by utilizing the sound-like information, so that a more accurate error correction result is obtained.

2. The pinyin constraint-based chinese grammar error correction method of claim 1, wherein: the specific steps of Step1 are as follows:

step1.1, acquiring a Chinese BART model which is open in source and is already pre-trained as a basic pre-training language model;

Step1.2, because the BART pre-training language model uses a multi-layer multi-head attention mechanism to construct an encoder and a decoder, the BART model downloaded in the last step is modified to adapt to the task of grammar error correction, so that the strong characterization capability of the pre-training language model is effectively utilized to enhance the performance of the grammar error correction model;

Step1.3, acquiring a MAGICDATA voice recognition data set of open access, and downloading a voice recognition model from the internet;

step1.4, automatically generating sentences containing the sound-like errors by using a voice recognition model, and combining the sentences containing the sound-like errors with correct sentences in the original dataset to form correct-error sentence pairs, thereby constructing a basic dataset MAGICDATA of a grammar error correction task.

3. The pinyin constraint-based chinese grammar error correction method of claim 1, wherein: the specific steps of Step2 are as follows:

Step2.1, adding a detection layer after an encoder of a syntax error correction basic model based on BART; syntax error detection is considered as a simple two-classification task, outputting 1 if there is a replacement error in the current position, otherwise 0;

Step2.2, preprocessing sentences with labels in the training set and the verification set, marking the sentences as 1 if the current word has a replacement error, and marking the sentences as 0 if the current word has the replacement error;

Step2.3, performing weighted summation on the loss of two tasks of grammar error detection and grammar error correction as a final loss, and updating all model parameters by minimizing the final loss.

4. The pinyin constraint-based chinese grammar error correction method of claim 1, wherein: the specific steps of Step3 are as follows:

Step3.1, download phonetic-like character confusion sets from some open-source websites. Performing de-duplication and merging on all downloaded confusion sets, arranging all characters in the confusion sets, and placing the similar characters in the same row;

Step3.2, preprocessing the confusion set, using tokenizer of the BART model to id the characters in the dictionary, taking the id of the first character as a Key, taking the ids of other characters similar to the sound of the character as Value, and storing the ids as dictionary format files;

Step3.3, reading the dictionary during model training, mapping the dictionary for the current word segmentation id when the model carries out word segmentation id on an input sentence, obtaining current word segmentation id sound-like character information, and constructing a confusion sound-like matrix for the current sound-like information;

Step3.4, multiplying the confusion sound-like matrix obtained in the last step by a result obtained by model detection to obtain a confusion matrix of sound-like information; since the detection result is classified into two types, the detection result is 1 only when a replacement error occurs; this ensures that the sound-like information is preserved only if a replacement error occurs in the current character.

5. The pinyin constraint-based chinese grammar error correction method of claim 4, wherein: the specific steps of Step4 are as follows:

Step4.1, mapping the sound-like information confusion matrix obtained in step3.4 into a matrix with output probability dimension, and setting the corresponding position in the dictionary as 1 and the rest positions as 0 according to the sound-like information existing in the current confusion matrix;

Step4.2, adding the matrix with the same dimension as the output probability obtained in step4.1 to the final output probability, and explicitly increasing the character probability similar to the current character.