CN112686045B - Method and device for evaluating text error detection model - Google Patents
Method and device for evaluating text error detection model Download PDFInfo
- Publication number
- CN112686045B CN112686045B CN202110283954.8A CN202110283954A CN112686045B CN 112686045 B CN112686045 B CN 112686045B CN 202110283954 A CN202110283954 A CN 202110283954A CN 112686045 B CN112686045 B CN 112686045B
- Authority
- CN
- China
- Prior art keywords
- text
- error
- error detection
- detection model
- subtype
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention relates to an evaluating method and a device of a text error detection model, wherein the method coarsely positions a first error type which cannot be identified by the text error detection model by performing error detection on a first error text; and then, carrying out error detection on more second error texts containing the sub-types of the first error types, thereby finely and finely positioning the sub-error types which cannot be identified by the composition error detection model. According to the scheme, through two times of error injection, the error types which cannot be identified by the text error detection model can be accurately and quickly positioned, and the accuracy of the error types acquired through the scheme of the embodiment is high. In addition, the scheme can be automatically executed without manual participation, so that the evaluation efficiency of the text error detection model can be effectively improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for evaluating a text error detection model.
Background
With the continuous development of Artificial Intelligence (AI) technology, AI has been widely used in various industries, such as finance, medical treatment, security, education, and the like. Among them, in the "AI + education" scenario, intelligent text error detection using AI is one of important application scenarios. In practical applications, a pre-trained text error detection model is usually deployed on a line to perform error detection on a text uploaded or input by a user. However, due to influencing factors such as data limitation of the training set, samples of the training set may not cover all error types, and the text error detection model may not effectively identify errors of some error types contained in the text. Therefore, the error types which cannot be identified by the text error detection model need to be searched, which is significant for optimizing the performance of the text error detection model.
The traditional method is to compare the label data with the error detection result output by the text error detection model manually, so as to determine the error type which cannot be identified by the text error detection model. Because the manual processing has a certain subjectivity, the accuracy of the error types which cannot be identified by the determined text error detection model is low.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, the present disclosure provides an evaluation method and an apparatus for a text error detection model.
In a first aspect, the present disclosure provides a method for evaluating a text error detection model,
carrying out error detection on a first error text according to a text error detection model to obtain a first error type which cannot be identified by the text error detection model;
carrying out error detection on a second error text according to the text error detection model to obtain a first target subtype which cannot be identified by the text error detection model;
wherein the second error text contains more categories of first candidate subtypes than the first error text, the first candidate subtypes being subtypes of the first error type; the first target subtype is one or more of the first candidate subtypes.
In some possible designs, the method further comprises:
the second erroneous text contains more erroneous data of the first candidate subtype than the first candidate subtype contained in the first erroneous text.
In some possible designs, the number of first erroneous texts is smaller than the number of second erroneous texts.
In some possible designs, before performing error detection on a first error text according to a text error detection model and obtaining a first error type that cannot be identified by the text error detection model, the method further includes:
matching in a standard text set according to historical data which cannot be identified by the text error detection model to obtain a first text which is successfully matched;
randomly extracting texts which are included in the standard text set and are not matched with the historical data to obtain a second text;
and converting the first text and the second text according to a preset conversion rule to obtain the first error text.
In some possible designs, the performing error detection on the second error text according to the text error detection model to obtain a first target subtype that cannot be identified by the text error detection model includes:
carrying out error detection on the second error text according to the text error detection model to obtain an error detection result corresponding to the second error text;
comparing the error detection result corresponding to the second error text with the standard error data information corresponding to the second error text to obtain a comparison result corresponding to the second error text;
and carrying out wrong word classification on the comparison result corresponding to the second wrong text to obtain the first target subtype.
In some possible designs, the performing error word classification on the comparison result corresponding to the second error text to obtain the first target subtype includes:
inputting a comparison result corresponding to the second error text into a pre-trained error word classification model, and acquiring a first target subtype output by the error word classification model; the wrong word classification model is used for performing part-of-speech analysis on wrong data in the comparison result to obtain a first target subtype contained in the comparison result.
In some possible designs, the method further comprises:
carrying out error detection on the ith error text according to the text error detection model to obtain an ith-1 target subtype which cannot be identified by the text error detection model, wherein the initial value of i is 3;
the type of an i-1 candidate subtype contained in the i-1 error text is more than that of an i-1 candidate subtype contained in the i-1 error text, and the i-1 candidate subtype is a subtype of an i-2 target subtype; the i-1 th target subtype is one or more of the i-1 th candidate subtypes;
updating i = i +1 until the updated i equals N; wherein N is an integer greater than or equal to 4.
In some possible designs, the preset transformation rules include one or more of:
random character replacement, random insertion, random deletion, Optical Character Recognition (OCR) based false replacement, Automatic Speech Recognition (ASR) based false replacement, keyboard false replacement, random substitution.
In some possible designs, the method further comprises:
and optimizing the text error detection model according to the target subtype determined by the last error detection.
In a second aspect, an embodiment of the present disclosure further provides an evaluation apparatus for a text error detection model, including:
the processing module is used for carrying out error detection on the first error text according to the text error detection model and acquiring a first error type which cannot be identified by the text error detection model;
the processing module is further configured to perform error detection on a second error text according to the text error detection model, and obtain a first target subtype which cannot be identified by the text error detection model;
wherein the second error text contains more categories of first candidate subtypes than the first error text, the first candidate subtypes being subtypes of the first error type; the first target subtype is one or more of the first candidate subtypes.
In a third aspect, an embodiment of the present disclosure further provides an electronic device, including: memory, processor, and computer program instructions;
the memory configured to store the computer program instructions;
the processor is configured to execute the computer program instructions, and the processor executes the computer program instructions to implement the method for evaluating the text error detection model according to any one of the first aspect.
In a fourth aspect, an embodiment of the present disclosure further provides a readable storage medium, including: computer program instructions;
the computer program instructions, when executed by a processor, implement a method of evaluating a text error detection model according to any of the first aspects.
The embodiment of the disclosure provides an evaluating method and device for a text error detection model, wherein the method comprises the following steps: carrying out error detection on a first error text according to a text error detection model to obtain a first error type which cannot be identified by the text error detection model; carrying out error detection on the second error text according to a text error detection model to obtain a first target subtype which cannot be identified by the text error detection model; wherein the second error text contains more categories of first candidate subtypes than the first error text, the first candidate subtypes being subtypes of the first error type; the first target subtype is one or more of the first candidate subtypes.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has at least the following advantages: 1. according to the scheme, error detection is carried out on a first error text, and a first error type which cannot be identified by a text error detection model is coarsely positioned; and then, carrying out error detection on more second error texts containing the sub-types of the first error types, thereby finely and finely positioning the sub-error types which cannot be identified by the composition error detection model. According to the scheme, through two times of error injection, the error type which cannot be identified by the text error detection model can be accurately and quickly positioned, and the accuracy of the error type is higher. 2. The method and the device can be automatically executed, manual participation is not needed, and therefore the evaluation efficiency of the text error detection model can be effectively improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a flowchart of an evaluation method of a text error detection model according to an embodiment of the present disclosure;
FIG. 2 is a flowchart of a method for evaluating a text error detection model according to another embodiment of the present disclosure;
FIG. 3 is a flowchart of a method for evaluating a text error detection model according to another embodiment of the present disclosure;
fig. 4 is a schematic diagram of a first text, a first error text corresponding to the first text, and a corresponding error detection result according to an embodiment of the disclosure;
fig. 5 is a schematic diagram of a second standard text, a second error text corresponding to the second standard text, and a corresponding error detection result provided by the present disclosure;
fig. 6 is a schematic structural diagram of an evaluation apparatus of a text error detection model according to another embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
Example one
Fig. 1 is a flowchart of an evaluation method of a text error detection model according to an embodiment of the present disclosure. The execution main body of the evaluation method of the text error detection model provided by the embodiment of the disclosure can be an evaluation device of the text error detection model provided by the embodiment of the disclosure, and the device can be realized by any software and/or hardware. Illustratively, the apparatus may be: computers, laptops, smart phones, smart wearable devices, personal digital assistants, and the like.
As shown in fig. 1, the method of the present embodiment includes:
s101, carrying out error detection on the first error text according to the text error detection model, and obtaining a first error type which cannot be identified by the text error detection model.
The text error detection model is used for carrying out error detection on the text and outputting an error detection result. Alternatively, the text error detection model may be a machine learning based neural network model. The embodiment of the present disclosure does not limit the specific type of the text error detection model, and belongs to the protection category of the present solution as long as the text error detection model has an error detection function.
The first error text in the present scheme refers to a text including a syntax error, and the number of the first error texts may be one or multiple. If the number of the first error texts is multiple, each of the first error texts may include error data of different error types, and the number of the error data of the same error type included in different first error texts may also be different.
For example, the first error text 1 includes error data corresponding to error type 1 at 10, the first error text 2 includes data corresponding to error type 2 at 10, and the first error text 3 includes error data corresponding to error type 1 at 20.
The first error text may be obtained by converting a first standard text according to a preset conversion rule, where the first standard text refers to a text that does not include a syntax error. The first standard text may be plural or one. In practical application, different types of error data can be injected into the first standard text according to different conversion rules, so that more first error texts can be obtained.
Optionally, the preset conversion rule may include one or more of the following: random character replacement, random insertion, random deletion, Optical Character Recognition (OCR) based false replacement, Automatic Speech Recognition (ASR) based false replacement, keyboard false replacement, random substitution. In addition, each of the above conversion rules may further include one or more sub-conversion rules.
Wherein, the random character replacement means that one or more Chinese characters (or characters) at a certain position are replaced irregularly, and the random character replacement is performed irregularly at the original position, for example, "today sunny day" before replacement and "today woodday" after replacement; and "applet" before the replacement and "applet" after the replacement.
The random insertion means that a random insertion is performed for a position before and/or after a certain kanji (or character). The random deletion means that one or more kanji (or characters) are randomly deleted, and in the case of a plurality of kanji (or characters), the plurality of kanji (or characters) may be continuous or discontinuous.
An erroneous replacement of OCR means to replace the OCR error-prone result, e.g. "big" before replacement and "too" after replacement. False replacement of ASR: and replacing the error-prone result after the voice recognition, wherein the replacement can be performed based on similar error-prone words, for example, "unity" before the replacement, and "agreement" after the replacement.
Keyboard error replacement: the keyboard layout is replaced with characters arranged adjacently, for example, an "applet" before the replacement and a "spple" after the replacement, because a is adjacent to s in the keyboard layout.
The random replacement means that the Chinese characters (or characters) at different positions are replaced with each other, and the different positions may be random or preset specific positions, for example, "today sunny day" before replacement and "sunny day" after replacement; for another example, the "applet" is used before the substitution and the "applet" is used after the substitution.
It is understood that each conversion rule described above corresponds to an error type, and each conversion rule includes a self-conversion rule corresponding to a seed error type. The purpose of converting the first standard text according to the above conversion rule is to inject error data of different error types into the standard text.
Illustratively, the preset conversion rule, the sub-conversion rules included in each conversion rule, and the representation of each sub-conversion rule are as shown in table 1 below:
TABLE 1
It should be understood that the above table 1 is only exemplary, and in practical applications, the kinds of the preset conversion rules and the sub-conversion rules included therein may be more or less.
The text error detection model performs error detection on the first error text to acquire a first error type which cannot be identified by the text error detection model, and the method can be realized by the following steps:
according to a possible implementation manner, the first error text is input to the text error detection model, and the text error detection model analyzes the first error text and outputs an error detection result corresponding to the first error text. And then, comparing the error detection result corresponding to the first error text with the error data information corresponding to the first error text to obtain a comparison result, wherein the comparison result comprises error data in the first error text which is not identified by the text error detection model. And then, performing part-of-speech analysis on the comparison result to acquire a first error type which is not identified by the text error detection model. Of course, in practical application, the part-of-speech analysis is performed on the comparison result, and a subtype included in the first error type that is not identified by the text error detection model can also be obtained.
The first error type may be one error type or a plurality of error types. This is related to the specific kind of the first error type contained in the first error text.
The error data information corresponding to the first error text includes: the first error text contains the error number and the position information of the error data in the first error text. The error data information may also include an error type corresponding to the error data.
Optionally, part-of-speech analysis may be performed on the comparison result corresponding to the first error text through a pre-trained wrong word classification model, where the wrong word classification model is used to analyze the part-of-speech of the input error data, so as to output the part-of-speech corresponding to the error data.
Optionally, if the error data information corresponding to the first error text includes an error type corresponding to the error data, the error type corresponding to the error data that is not identified by the text error detection model, that is, the first error type, may be determined according to the comparison result and the error data information corresponding to the first error text.
S102, carrying out error detection on the second error text according to the text error detection model, and acquiring a first target subtype which cannot be identified by the text error detection model; the category of the first candidate subtype contained in the second error text is more than that of the first candidate subtype contained in the first error text, and the first candidate subtype is a subtype of the first error type; the first target subtype is one or more of the first candidate subtypes.
Although, in some cases, in S101, the first error type and the information related to the sub-type included in the first error type may be obtained according to the comparison result between the error detection result corresponding to the first error text and the error data corresponding to the first error text, since the first error text may not cover more completely the sub-types of the first error type, it is necessary to obtain the second error text according to the first error type obtained in S101, and then obtain the first target sub-type according to the second error text.
Alternatively, the second error text may be acquired by:
and converting the second standard text according to the first error type and the preset conversion rule obtained in the step S101 to obtain a second error text. When the second standard text is converted, the category of the subtype of the first error type can be increased, that is, the category of the first candidate subtype is increased, so that the category of the first candidate subtype contained in the second error text is more than that contained in the first error text.
The preset conversion rule for converting the second standard text in this step is similar to that in table 1, and reference may be made to the foregoing description, which is not repeated herein.
According to the scheme, when the second error text is obtained, the weight of the first error type which is not identified by the text error detection model is increased in a directional mode, so that the second error text can cover the sub-type contained in the first error type comprehensively, and a foundation is provided for fine-grained positioning of the first target sub-type which cannot be identified by the text error detection model.
And carrying out error detection on the second error text according to the text error detection model to obtain a first target subtype which cannot be identified by the text error detection model, wherein the error detection can be realized by the following steps:
and a possible implementation manner is that the second error text is input into the text error detection model, the text error detection model is used for carrying out error detection on the second error text, and an error detection result corresponding to the second error text is output. And then, comparing the error detection result corresponding to the second error text with the error data information corresponding to the second error text to obtain a comparison result, wherein the comparison result comprises the error data in the second error text which is not identified by the text error detection model. And then, performing part-of-speech analysis on the comparison result to acquire a first target subtype which is not identified by the text error detection model.
Of course, in practical applications, if the first target sub-type further includes an error type of a next level, then the part-of-speech analysis is performed on the comparison result, and a sub-type included in the first target sub-type that is not identified by the text error detection model can also be obtained.
The first target subtype may be one or more. This is related to the category of the first target sub-type contained in the second error text.
The error data information corresponding to the second error text includes: the second error text contains the error number and the position information of the error data in the second error text. The error data information may also include a first candidate subtype for the error data.
Optionally, part-of-speech analysis may be performed on the comparison result corresponding to the second error text through a pre-trained wrong word classification model, where the wrong word classification model is used to analyze the part-of-speech of the input error data, so as to output the part-of-speech corresponding to the error data.
Optionally, if the error data information corresponding to the second error text includes the first candidate subtype corresponding to the error data, the first candidate subtype corresponding to the error data not identified by the text error detection model, that is, the first target subtype, may be determined according to the comparison result and the error data information corresponding to the second error text.
In the method for evaluating the text error detection model provided by the embodiment, the first error type which cannot be identified by the text error detection model is coarsely positioned by performing error detection on the first error text; and then, carrying out error detection on more second error texts containing the sub-types of the first error types, thereby finely and finely positioning the sub-error types which cannot be identified by the composition error detection model. According to the scheme, through two times of error injection, the error types which cannot be identified by the text error detection model can be accurately and quickly positioned, and the accuracy of the error types acquired through the scheme of the embodiment is high. In addition, the technical scheme provided by the embodiment can be automatically executed without manual participation, so that the evaluation efficiency of the text error detection model can be effectively improved.
On the basis of the embodiment shown in fig. 1, optionally, the number of first erroneous texts is smaller than the number of second erroneous texts. If the first error text is obtained by converting the first standard text and the second error text is obtained by converting the second standard file, the number of the first standard texts may be smaller than the number of the second standard texts.
If the number of the first error texts is smaller than that of the second error texts, the scheme firstly quickly positions a first error type which cannot be identified by a text error detection model in a coarse-grained manner through a small batch of first error texts; and then, finely positioning the first target subtype which cannot be identified by the text error detection model through a large batch of second error texts. Since the second error text contains more types of subtypes of the first error type, error data contained in the error text can point to the first target subtype more accurately, and therefore when the first target subtype is obtained through the second error text, not only is the result of the identified error type more accurate, but also the processing efficiency is higher.
Example two
Fig. 2 is a flowchart of an evaluation method of a text error detection model according to another embodiment of the present disclosure. On the basis of the embodiment shown in fig. 1, this embodiment describes in detail a specific implementation manner of obtaining the first error text. As shown in fig. 2, the method of the present embodiment includes:
s201, matching is carried out in a standard text set according to historical data which cannot be identified by the text error detection model, and a first text which is successfully matched is obtained.
The historical data can be obtained from a corresponding historical database, and the historical data which cannot be identified by the text error detection model is stored in the historical database. Since the model version may be updated iteratively, the history database may also store history data that cannot be identified by the text error detection model of different versions.
Acquiring the first text may be achieved by:
firstly, acquiring historical data from a historical database, and matching in a standard text of a standard text set according to the historical data; and converting the successfully matched standard text according to the historical data to obtain the converted standard text. The converted standard text contains error data corresponding to the historical data.
And then, respectively inputting the converted standard texts into a text error detection model, and determining whether the text error detection model of the current version can effectively identify error data corresponding to the historical data according to an error detection result output by the text error detection model.
If it is determined that the text error detection model cannot identify the error data corresponding to the historical data, all or part of the text in the converted standard text may be used as the first text.
If the number of the converted standard texts is small, all the converted standard texts can be used as the first text. If the converted standard texts are more, a part of texts can be selected as the first text. In practical application, the number of the first texts can be determined according to practical requirements.
S202, randomly extracting the texts which are included in the standard text set and do not match with the historical data, and acquiring a second text.
In the scheme, the first text is determined by the historical data, and in order to expand the limited set of the history, the scheme also obtains the second text by randomly extracting the texts which are contained in the standard text set and do not match with the historical data.
In practical application, the number of the second texts can be determined according to practical requirements.
How to obtain the first text and the second text is described below by a specific example:
illustratively, the historical data that cannot be identified by the text error detection model is obtained from the historical database as "today". The standard text set includes 10 standard texts, of which 5 standard texts include the word "today". Obtaining the first text and the second text may comprise the steps of:
step one, matching is carried out in each standard text of a standard text set according to the word of 'today', the 'today' in 5 successfully matched standard texts is replaced by 'make day', and 5 standard texts with error data injected are obtained;
step two, respectively inputting the 5 standard texts into which the error data 'order days' is injected into a text error detection model, and acquiring error detection results corresponding to the 5 standard texts output by the text error detection model;
and step three, if the text error detection model can effectively identify the error data, namely 'order day', in the text error detection model according to the error detection results respectively corresponding to the 5 standard texts injected with the error data, determining that the 5 standard texts injected with the error data 'order day' are the first texts. Alternatively, a part of the 5 standard texts into which the error data "day" is injected may be determined as the first text, for example, 3 texts of the 5 standard texts into which the error data "day" is injected may be determined as the first text.
And step four, randomly extracting the 5 standard texts which are not successfully matched with the vocabulary of 'today' to obtain a second text. For example, 2 texts are randomly extracted as the second texts from the remaining 5 standard texts which are not successfully matched with the word of "today".
Through the first step to the fourth step, a first text and a second text for acquiring a first error text are determined.
S203, converting the first text and the second text according to a preset conversion rule to obtain the first error text.
In the scheme, in order to improve the dimensionality of the error types contained in the first error text, the first text and the second text are converted according to a preset conversion rule, and more error data of the error types are injected into the first text and the second text, so that the first error text is obtained. The preset conversion rule is similar to table 1, and a specific implementation manner of obtaining the first error text according to the preset conversion rule and the first text and the second text may refer to the detailed description in the embodiment shown in fig. 1, which is not described herein again.
S204, carrying out error detection on the first error text according to the text error detection model, and acquiring a first error type which cannot be identified by the text error detection model.
S205, carrying out error detection on the second error text according to the text error detection model, and acquiring a first target subtype which cannot be identified by the text error detection model; the category of the first candidate subtype contained in the second error text is more than that of the first candidate subtype contained in the first error text, and the first candidate subtype is a subtype of the first error type; the first target subtype is one or more of the first candidate subtypes.
In this embodiment, S204 and S205 are similar to S101 and S102 in the embodiment shown in fig. 1, and reference may be made to the detailed description of the embodiment shown in fig. 1, which is not repeated herein.
According to the method provided by the embodiment, the small-batch first text and the small-batch second text are obtained through two modes of matching search and random extraction, a history set is considered, the dimension requirement of the error type is met, and the accuracy of the first error type which cannot be identified according to the text error detection model identified by the first error text is ensured. In addition, the number of the first texts and the second texts is small, and the efficiency of acquiring the first error type is high.
EXAMPLE III
Fig. 3 is a flowchart of an evaluation method of a text error detection model according to another embodiment of the present disclosure. On the basis of the embodiment shown in fig. 1, if there are more levels of error type classifications, after S102, the following steps may be further included:
s103, carrying out error detection on the ith error text according to the text error detection model to obtain an ith-1 target subtype which cannot be identified by the text error detection model, wherein the initial value of i is 3; the type of the (i-1) th candidate subtype contained in the (i-1) th error text is more than that of the (i-1) th candidate subtype contained in the (i-1) th error text, and the (i-1) th candidate subtype is a subtype of the (i-2) th target subtype; the (i-1) th target subtype is one or more of the (i-1) th candidate subtypes.
S104, updating i = i +1 until the updated i is equal to N; wherein N is an integer greater than or equal to 4.
In some cases, the error types may include more hierarchical classifications, e.g., a first error type includes a plurality of first candidate subtypes, each of which may include a plurality of second candidate subtypes, and so on. Therefore, after step S102, the error type that cannot be identified by the text error detection model can be more accurately located by the scheme of the present embodiment.
In other cases, after step S102, the accuracy of the first target sub-type that cannot be identified by the text error detection model determined in step S102 may be re-verified by the scheme of this embodiment.
In other cases, the two above cases can also be combined.
The implementation manner of S103 is similar to that of S102, and reference may be made to the detailed description in the embodiment shown in fig. 1.
According to the method provided by the embodiment, the error type distribution which cannot be identified by the text error detection model is positioned step by step, so that the error type which cannot be identified by the fine-grained text error detection model is accurately and quickly obtained. In addition, the method provided by the embodiment can also be used for verifying the error types which cannot be identified by the determined text error detection model, so that the accuracy of the evaluation result of the text error detection model is improved.
Optionally, after S104, the text error detection model may be further optimized according to the target subtype determined by the last error detection.
Illustratively, if the error type determined by the last error detection is the first target subtype, the training sample is obtained again according to the first target subtype, and the text error detection model is optimized according to the obtained training sample, so that the optimized text error detection model can effectively recognize error data of the first target subtype included in the text.
If the error type determined by the last error detection is the Nth target subtype, the training sample is obtained again according to the Nth target subtype, and the text error detection model is optimized according to the obtained training sample, so that the optimized text error detection model can effectively identify error data of the Nth target subtype contained in the text.
Example four
In a specific embodiment, determining the type of error that cannot be identified by the text error by the method of the embodiment of the present disclosure may include the following steps:
step 1: and querying historical data 'hot summer' which cannot be identified by the text error detection model from a historical database, and injecting error data 'fire summer' into the standard text according to the historical data 'hot summer'.
Step 2: and inputting the standard text injected with the error data 'fire summer day' into the text error detection model, and determining that the text error detection model cannot recognize the word 'fire summer day' according to an error detection result output by the text error detection model.
And step 3: matching in a standard text set according to the vocabulary of 'summer days with inflammation', and taking a successfully matched text as a first text; and determining the second text by means of random extraction.
And 4, step 4: and respectively converting the first text and the second text according to a preset conversion rule to obtain a first error text.
Exemplarily, the first text and the converted first text (i.e., the first error text) are as shown in fig. 4. In this embodiment, the first text is converted according to several conversion rules, such as random deletion, random insertion, similar words, and antisense words.
And 5: and respectively inputting the converted first text (namely the first error text) and the converted second text into a text error detection model, and acquiring a detection result output by the text error detection model.
Illustratively, the text error detection model performs error detection on the converted first text shown in fig. 4, and outputs an error detection result as shown in fig. 4.
Step 6: comparing a detection result output by the text error detection model with error data information corresponding to the first error text to obtain a comparison result; and analyzing the comparison result to determine a first error type which cannot be identified by the text error detection model.
Specifically, comparing the error detection result shown in fig. 4 with the error data information corresponding to the converted first text shown in fig. 4, it can be determined that the text error detection model has a poor recognition effect on the type of the random insertion error.
And 7: and converting the second standard text according to the first error type (namely, the error type of random insertion) obtained in the step 6 and a preset conversion rule to obtain a second error text, and increasing the weight of the error type of random insertion during conversion.
Exemplarily, the second standard text and the converted second standard text (i.e., the second error text) are as shown in fig. 5.
And 8: and respectively inputting the converted second standard texts (namely the second error texts) into the text error detection model, and acquiring the detection result output by the text error detection model.
And step 9: comparing the detection result output by the text error detection model with the error data information corresponding to the second error text to obtain a comparison result; and performing part-of-speech analysis on the comparison result through the wrong word classification model, and determining a first target subtype which cannot be identified by the text error detection model.
Specifically, the comparison result corresponding to the second error text and the error type of the error data included in the comparison result are shown in table 2 below:
TABLE 2
| Original text of second standard text | Converted characters | First error type | First target subtype |
| Shi | Soil for soil | Random character replacement | Similar to a character |
| Put through with | Wear like | Random character replacement | Similar to a character |
| / | Is/are as follows | Random insertion | Word aid |
| / | Is/are as follows | Random insertion | Word aid |
| Spull head | Screw child's hanging hair | Random character replacement | Similar to a character |
As can be seen from table 2, the first category that cannot be identified by the text error detection model is: the random character replaced is similar to a character and a mood auxiliary word inserted randomly.
In practical application, the 1000 standard texts are evaluated by adopting three modes of full injection, single injection and double injection, and the consumed time and the misclassification coverage rate are shown in the following table 3:
TABLE 3
| (Mode) | When in use | Coverage of misclassifications |
| Full dose injection | 2 hours | 95.3% |
| Single injection | 40 minutes | 79.9% |
| Double injection | 50 minutes | 95.1% |
Full injection: all candidate subtypes are error injected against all standard texts.
Single injection: random error injection is performed for all standard texts.
Double injection: firstly, random first error injection is carried out on a part of standard texts, and then, second error injection is carried out according to a first recognition result (a first error type) of a text error detection model, wherein the weight of the first error type during second error injection is higher than that during first error injection.
As can be seen from table 3, the evaluation method for the text error detection model provided by the present disclosure can accurately and quickly locate the error type that cannot be identified by the text error detection model by means of multiple injections, and is less in time consumption and higher in accuracy.
EXAMPLE five
Fig. 6 is a schematic structural diagram of an evaluation apparatus of a text error detection model according to an embodiment of the present disclosure. As shown in fig. 6, the evaluation apparatus 600 for a text error detection model according to this embodiment includes:
the processing module 601 is configured to perform error detection on a first error text according to a text error detection model, and obtain a first error type that cannot be identified by the text error detection model;
the processing module 601 is further configured to perform error detection on the second error text according to the text error detection model, and obtain a first target subtype that cannot be identified by the text error detection model;
wherein the second error text contains more categories of first candidate subtypes than the first error text, the first candidate subtypes being subtypes of the first error type; the first target subtype is one or more of the first candidate subtypes.
In some possible designs, the second error text contains more error data for the first candidate subtype than the first candidate subtype contained in the first error text.
In some possible designs, the number of first erroneous texts is smaller than the number of second erroneous texts.
In some possible designs, further comprising: an acquisition module 602;
the obtaining module 602 is configured to perform matching in a standard text set according to historical data that cannot be identified by the text error detection model, and obtain a first text that is successfully matched; randomly extracting texts which are included in the standard text set and are not matched with the historical data to obtain a second text; and converting the first text and the second text according to a preset conversion rule to obtain the first error text.
In some possible designs, the processing module 601 is specifically configured to perform error detection on the second error text according to the text error detection model, and obtain an error detection result corresponding to the second error text; comparing the error detection result corresponding to the second error text with the standard error data information corresponding to the second error text to obtain a comparison result corresponding to the second error text; and carrying out wrong word classification on the comparison result corresponding to the second wrong text to obtain the first target subtype.
In some possible designs, the processing module 601 is specifically configured to input a comparison result corresponding to the second wrong text into a pre-trained wrong word classification model, and obtain a first target subtype output by the wrong word classification model; the wrong word classification model is used for performing part-of-speech analysis on wrong data in the comparison result to obtain a first target subtype contained in the comparison result.
In some possible designs, the processing module 601 is further configured to perform error detection on an ith error text according to the text error detection model, to obtain an ith-1 target sub-type that cannot be identified by the text error detection model, where an initial value of i is 3;
updating i = i +1 until the updated i equals N; wherein N is an integer greater than or equal to 4.
The type of an i-1 candidate subtype contained in the i-1 error text is more than that of an i-1 candidate subtype contained in the i-1 error text, and the i-1 candidate subtype is a subtype of an i-2 target subtype; the (i-1) th target subtype is one or more of the (i-1) th candidate subtypes.
In some possible designs, the preset transformation rules include one or more of:
random character replacement, random insertion, random deletion, Optical Character Recognition (OCR) based false replacement, Automatic Speech Recognition (ASR) based false replacement, keyboard false replacement, random substitution.
In some possible designs, the processing module 601 is further configured to optimize the text error detection model according to the target subtype determined by the last error detection.
The apparatus provided in this embodiment may be used to implement the technical solution in any of the above method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.
EXAMPLE six
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 7, the electronic device 700 provided in the present embodiment includes: a memory 701 and a processor 702.
The memory 701 may be a separate physical unit, and may be connected to the processor 702 through a bus 703. The memory 701 and the processor 702 may also be integrated together, implemented by hardware, and the like.
The memory 701 is used to store program instructions that are called by the processor 702 to implement the operations of any of the method embodiments described above.
Alternatively, when part or all of the methods of the above embodiments are implemented by software, the electronic device 700 may only include the processor 702. A memory 701 for storing programs is located outside the electronic device 700 and a processor 702 is connected to the memory via circuits/wires for reading and executing the programs stored in the memory.
The Processor 702 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP.
The processor 702 may further include a hardware chip. The hardware chip may be an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), General Array Logic (GAL), or any combination thereof.
The Memory 701 may include a Volatile Memory (Volatile Memory), such as a Random-Access Memory (RAM); the Memory may also include a Non-volatile Memory (Non-volatile Memory), such as a Flash Memory (Flash Memory), a Hard Disk Drive (HDD) or a Solid-state Drive (SSD); the memory may also comprise a combination of memories of the kind described above.
The invention also provides a computer readable storage medium comprising computer program instructions which, when executed by a processor, implement a method in any of the embodiments described above.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (12)
1. An evaluation method of a text error detection model is characterized by comprising the following steps:
carrying out error detection on a first error text according to a text error detection model to obtain a first error type which cannot be identified by the text error detection model;
carrying out error detection on a second error text according to the text error detection model to obtain a first target subtype which cannot be identified by the text error detection model;
wherein the second error text contains more categories of first candidate subtypes than the first error text, the first candidate subtypes being subtypes of the first error type; the first target subtype is one or more of the first candidate subtypes.
2. The method of claim 1, further comprising:
the second erroneous text contains more erroneous data of the first candidate subtype than the first candidate subtype contained in the first erroneous text.
3. The method of claim 1, wherein the number of first erroneous texts is smaller than the number of second erroneous texts.
4. The method according to any one of claims 1 to 3, wherein before the error detecting the first erroneous text according to the text error detection model and obtaining the first error type that cannot be identified by the text error detection model, the method further comprises:
matching in a standard text set according to historical data which cannot be identified by the text error detection model to obtain a first text which is successfully matched;
randomly extracting texts which are included in the standard text set and are not matched with the historical data to obtain a second text;
and converting the first text and the second text according to a preset conversion rule to obtain the first error text.
5. The method according to claim 1, wherein the performing error detection on the second erroneous text according to the text error detection model to obtain the first target sub-type that cannot be identified by the text error detection model comprises:
carrying out error detection on the second error text according to the text error detection model to obtain an error detection result corresponding to the second error text;
comparing the error detection result corresponding to the second error text with the standard error data information corresponding to the second error text to obtain a comparison result corresponding to the second error text;
and carrying out wrong word classification on the comparison result corresponding to the second wrong text to obtain the first target subtype.
6. The method according to claim 5, wherein the performing error word classification on the comparison result corresponding to the second error text to obtain the first target subtype comprises:
inputting a comparison result corresponding to the second error text into a pre-trained error word classification model, and acquiring a first target subtype output by the error word classification model; the wrong word classification model is used for performing part-of-speech analysis on wrong data in the comparison result to obtain a first target subtype contained in the comparison result.
7. The method of claim 1, further comprising:
carrying out error detection on the ith error text according to the text error detection model to obtain an ith-1 target subtype which cannot be identified by the text error detection model, wherein the initial value of i is 3;
the type of an i-1 candidate subtype contained in the i-1 error text is more than that of an i-1 candidate subtype contained in the i-1 error text, and the i-1 candidate subtype is a subtype of an i-2 target subtype; the i-1 th target subtype is one or more of the i-1 th candidate subtypes;
updating i = i +1 until the updated i equals N; wherein N is an integer greater than or equal to 4.
8. The method of claim 4, wherein the preset transformation rules comprise one or more of the following:
random character replacement, random insertion, random deletion, Optical Character Recognition (OCR) based false replacement, Automatic Speech Recognition (ASR) based false replacement, keyboard false replacement, random substitution.
9. The method of claim 1, further comprising:
and optimizing the text error detection model according to the target subtype determined by the last error detection.
10. An evaluation apparatus for a text error detection model, comprising:
the processing module is used for carrying out error detection on the first error text according to the text error detection model and acquiring a first error type which cannot be identified by the text error detection model;
the processing module is further configured to perform error detection on a second error text according to the text error detection model, and obtain a first target subtype which cannot be identified by the text error detection model;
wherein the second error text contains more categories of first candidate subtypes than the first error text, the first candidate subtypes being subtypes of the first error type; the first target subtype is one or more of the first candidate subtypes.
11. An electronic device, comprising: memory, processor, and computer program instructions;
the memory configured to store the computer program instructions;
the processor configured to execute the computer program instructions, the processor executing the computer program instructions to implement the method of evaluating a text error detection model according to any of claims 1 to 9.
12. A readable storage medium, comprising: computer program instructions;
the computer program instructions, when executed by a processor, implement a method of evaluating a text error detection model according to any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110283954.8A CN112686045B (en) | 2021-03-17 | 2021-03-17 | Method and device for evaluating text error detection model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110283954.8A CN112686045B (en) | 2021-03-17 | 2021-03-17 | Method and device for evaluating text error detection model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112686045A CN112686045A (en) | 2021-04-20 |
| CN112686045B true CN112686045B (en) | 2021-06-29 |
Family
ID=75455630
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110283954.8A Active CN112686045B (en) | 2021-03-17 | 2021-03-17 | Method and device for evaluating text error detection model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112686045B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109543022B (en) * | 2018-12-17 | 2020-10-13 | 北京百度网讯科技有限公司 | Text error correction method and device |
| CN110135409B (en) * | 2019-04-04 | 2023-11-03 | 平安科技（深圳）有限公司 | Optimization method and device for recognition model |
| CN111950262A (en) * | 2020-07-17 | 2020-11-17 | 武汉联影医疗科技有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN112183321A (en) * | 2020-09-27 | 2021-01-05 | 深圳奇迹智慧网络有限公司 | Method and device for optimizing machine learning model, computer equipment and storage medium |
| CN112446441B (en) * | 2021-02-01 | 2021-08-20 | 北京世纪好未来教育科技有限公司 | Model training data screening method, device, equipment and storage medium |
-
2021
- 2021-03-17 CN CN202110283954.8A patent/CN112686045B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112686045A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109697162B (en) | Software defect automatic detection method based on open source code library | |
| CA2940760C (en) | Intelligent data munging | |
| CN110019843B (en) | Knowledge graph processing method and device | |
| WO2020192401A1 (en) | System and method for generating answer based on clustering and sentence similarity | |
| CN109902307B (en) | Named entity recognition method, named entity recognition model training method and device | |
| JP5356197B2 (en) | Word semantic relation extraction device | |
| CN113011533A (en) | Text classification method and device, computer equipment and storage medium | |
| CN112651238A (en) | Training corpus expansion method and device and intention recognition model training method and device | |
| CN112966068A (en) | Resume identification method and device based on webpage information | |
| CN113672931B (en) | Software vulnerability automatic detection method and device based on pre-training | |
| CN113961768B (en) | Sensitive word detection method and device, computer equipment and storage medium | |
| CN110705281B (en) | Resume information extraction method based on machine learning | |
| CN112395881B (en) | Material label construction method and device, readable storage medium and electronic equipment | |
| CN113806489A (en) | Method, electronic device and computer program product for dataset creation | |
| CN110837730B (en) | Method and device for determining unknown entity vocabulary | |
| JPWO2007132564A1 (en) | Data processing apparatus and method | |
| CN112686045B (en) | Method and device for evaluating text error detection model | |
| Barari et al. | CloniZER spell checker adaptive language independent spell checker | |
| CN110705258A (en) | Text entity identification method and device | |
| CN115953123A (en) | Method, device and equipment for generating robot automation flow and storage medium | |
| US20230075290A1 (en) | Method for linking a cve with at least one synthetic cpe | |
| CN113591480B (en) | Named entity identification method and device for electric power metering and computer equipment | |
| US11620319B2 (en) | Search platform for unstructured interaction summaries | |
| Dewantara et al. | Combination of lstm and cnn for article-level propaganda detection in news articles | |
| QasemiZadeh et al. | Adaptive language independent spell checking using intelligent traverse on a tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |