WO2020253742A1

WO2020253742A1 - Sample labeling checking method and device

Info

Publication number: WO2020253742A1
Application number: PCT/CN2020/096647
Authority: WO
Inventors: 徐青松; 李青
Original assignee: 杭州睿琪软件有限公司
Priority date: 2019-06-20
Filing date: 2020-06-17
Publication date: 2020-12-24
Also published as: CN110245716B; CN110245716A

Abstract

A sample labeling checking method and device. The method comprises: acquiring a sample to be labeled (S101); identifying, by means of an area identification model, at least one area of the sample to be labeled, and cutting the at least one area to form at least one sample area picture (S102); identifying each sample area picture by means of a pre-set identification model and performing pre-labeling processing (S103); sending the pre-labeled sample area picture to a checking unit, such that the checking unit checks a pre-labeling result of the sample area picture, and if it is checked and found that the pre-labeling result is wrong, modifying the pre-labeling result (S104); and sending the sample area picture checked by the checking unit to a verification client, such that the verification client performs verification processing on labeling information of the sample area picture checked by the checking unit (S105). By means of the method and device, the accuracy of sample labeling can be improved.

Description

Sample labeling review method and device

Technical field

The present invention relates to the field of artificial intelligence technology, in particular to a sample labeling and reviewing method, device, electronic equipment and computer readable storage medium.

Background technique

In the field of artificial intelligence, the training samples need to be annotated before model training. Usually, a manual client or a recognition model can be used to label the training samples, but this cannot guarantee the labeling accuracy of the samples.

Summary of the invention

The purpose of the present invention is to provide a sample labeling review method, device, electronic equipment and computer readable storage medium to improve the accuracy of sample labeling. The specific technical solutions are as follows:

In the first aspect, the present invention provides a sample labeling review method, which includes:

Step 1: Obtain samples to be labeled;

Step 2: Identify at least one region of the sample to be labeled through a region recognition model, and cut the at least one region to form at least one sample region picture; wherein the region recognition model is a neural network-based model;

Step 3: Recognize each sample area picture through a preset recognition model and perform pre-labeling processing; wherein the preset recognition model is a neural network-based model;

Step 4: Send the pre-labeled sample area pictures to the review unit, so that the review unit can review the pre-labeled results of the sample area pictures, and if the pre-labeled result is verified as an error, the pre-labeled The results are modified;

Step 5: Send the sample area pictures reviewed by the review unit to the verification client, so that the verification client performs verification processing on the label information of the sample area pictures reviewed by the review unit.

Optionally, step 3 recognizes each sample area picture through a preset recognition model and performs pre-labeling processing, including:

Recognize each sample area picture through a preset recognition model and perform pre-labeling processing;

Step 4 sends the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

The pre-labeled sample area picture is sent to a manual client, so that the manual client can review the pre-labeled result of the sample area picture.

Optionally, the verification client in step 5 performs verification processing on the annotation information of the sample area picture after the verification unit has reviewed, including:

For each sample area picture, the verification client checks whether the labeling information of the sample area picture after the manual client's review is accurate, and if it is not accurate, re-identifies the sample area picture.

Recognize each sample area picture through at least two preset recognition models and perform pre-labeling processing;

The sample area pictures pre-labeled by different preset recognition models are sent to each of a plurality of different manual clients, so that each manual client can review the pre-labeled results of each sample area picture.

For each sample area picture, the verification client checks whether the label information of the sample area picture after the review by different manual clients is consistent, and if they are inconsistent, the sample area picture is re-identified.

Step 4 Send the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send the pre-labeled sample area picture to a manual client, so that the manual client can review the pre-labeled result of the sample area picture; and

Recognize and label the pre-labeled sample area pictures through another preset recognition model;

Wherein, the one preset recognition model and the another preset recognition model are recognition models established by training based on different training sample sets.

For each sample area picture, the verification client verifies whether the label information of the sample area picture after the manual client's review is consistent with that of another preset recognition model. Recognition.

Optionally, after the pre-labeling processing in step 3, it also includes:

Selecting a preset number of pictures from the sample area pictures recognized by the preset recognition model, and modifying the pre-labeled result of the selected preset number of pictures to a recognition result different from the original pre-labeled result;

In step 4, the pre-labeled sample area pictures are sent to the review unit, so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send both the sample area picture of the unmodified pre-labeled result and the sample area picture of the modified pre-labeled result to the review unit, so that the review unit can check the sample area picture of the unmodified pre-labeled result and the sample of modified pre-labeled result Review the pre-annotated results of regional pictures;

After the verification client in step 5 performs verification processing on the label information of the sample area picture reviewed by the review unit, it also includes:

The verification client determines whether the verification unit is in an abnormal state according to the review result of the review unit on the preset number of pictures of which the pre-marked result has been modified.

Optionally, the review unit reviews the pre-labeled results of the sample area pictures of the unmodified pre-labeled results and the sample area pictures of the modified pre-labeled results, including:

For each sample area picture, the review unit judges whether the marked pre-marking result is correct; if not, the pre-marking result marked on the sample area picture is modified.

Optionally, the verification client terminal checks whether the verification unit is in an abnormal state according to the verification result of the verification unit on the preset number of pictures of the modified pre-annotation result, including:

For each picture in the preset number of pictures for which the pre-annotation result has been modified, determine whether the review unit has modified the pre-annotation result of the picture;

Acquiring, as the first ratio, a ratio of pictures whose pre-annotation results have been modified by the review unit among the preset number of pictures for which the pre-annotation results have been modified;

If the first ratio is less than a preset threshold, it is determined that the audit unit is in an abnormal state.

For each picture in the preset number of pictures for which the pre-labeled result has been modified, determine whether the review unit modifies the pre-labeled result of the picture to a correct recognition result;

Acquiring, as a second ratio, a ratio of the pictures that have been modified as a correct recognition result by the review unit among the preset number of pictures of the modified pre-annotation result;

If the second ratio is less than the preset threshold, it is determined that the audit unit is in an abnormal state.

In the second aspect, the present invention also provides a sample labeling and reviewing device, which includes:

Obtaining module for obtaining samples to be labeled;

A recognition module, configured to recognize at least one region of the sample to be labeled through a region recognition model, and cut the at least one region to form at least one sample region picture; wherein the region recognition model is a neural network-based model;

The pre-labeling module is used to identify each sample area picture through a preset recognition model and perform pre-labeling processing; wherein, the preset recognition model is a neural network-based model;

The review module is configured to send the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the pre-labeled sample area pictures, if the pre-labeled result is verified as If it is wrong, modify the pre-marked result;

The verification module is configured to send the sample area pictures reviewed by the review unit to the verification client, so that the verification client performs verification processing on the label information of the sample area pictures reviewed by the review unit.

Optionally, the pre-labeling module recognizes each sample area picture through a preset recognition model and performs pre-labeling processing, including:

The review module sends the pre-labeled sample area pictures to the review unit, so that the review unit can review the pre-labeled results of the pre-labeled sample area pictures, including:

Sending the pre-labeled sample area picture to a manual client, so that the manual client can review the pre-labeled result of the sample area picture;

The verification process performed by the verification client on the label information of the sample area picture after being reviewed by the verification unit includes:

Send the sample area pictures pre-labeled through different preset recognition models to each of multiple different manual clients, so that each manual client can review the pre-labeled results of each sample area picture;

Wherein, the one preset recognition model and the another preset recognition model are recognition models established by training based on different training sample sets;

Optionally, after the pre-labeling module performs pre-labeling processing, it is also used to:

Sending the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send both the sample area picture of the unmodified pre-labeled result and the sample area picture of the modified pre-labeled result to the review unit, so that the review unit can check the sample area picture of the unmodified pre-labeled result and the sample of modified pre-labeled result The pre-annotated results of regional pictures are reviewed, including:

For each sample area picture, the review unit judges whether the marked pre-marking result is correct; if not, the pre-marking result marked on the sample area picture is modified;

After the verification client performs verification processing on the annotation information of the sample area picture that has been reviewed by the review unit, it is also used to:

According to the review result of the review unit on the preset number of pictures for which the pre-marked result has been modified, it is checked whether the review unit is in an abnormal state.

Optionally, the verification client in the verification module checks whether the verification unit is in an abnormal state according to the verification result of the verification unit on the preset number of pictures of the modified pre-annotation result, including:

In a third aspect, the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus. Communication

The memory is used to store computer programs;

The processor is configured to implement the sample labeling review method described in the first aspect when executing the computer program stored on the memory.

In a fourth aspect, the present invention also provides a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the sample labeling described in the first aspect is implemented. Audit method.

Compared with the prior art, the sample labeling and reviewing method, device, electronic equipment and computer-readable storage medium provided by the present invention have the following beneficial effects:

The present invention first recognizes at least one region of the sample to be labeled through the region recognition model, and cuts to form at least a sample region picture, and then pre-labels each sample region picture through the preset recognition model, and then The pre-annotation results of the sample area pictures are reviewed by the review unit. If the pre-annotation results are found to be incorrect, the pre-annotation results will be modified. Finally, the verification client will verify the results of the pre-marking The annotation information of the pictures in the sample area is verified. It can be seen that the present invention labels samples according to the process of pre-labeling, review, and verification, which can ensure the accuracy of sample labeling, thereby improving the accuracy of model training.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic flowchart of a sample labeling and reviewing method provided by an embodiment of the present invention;

2 is a schematic structural diagram of a sample labeling and reviewing device provided by an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

Hereinafter, the method, device, electronic equipment, and computer-readable storage medium for sample labeling and review proposed by the present invention will be further described in detail with reference to the drawings and specific embodiments. According to the claims and the following description, the advantages and features of the present invention will be clearer. It should be noted that the drawings all adopt a very simplified form and all use imprecise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, it should be noted that each block in the block diagrams and/or flowcharts herein, as well as the combination of blocks in the block diagrams and/or flowcharts, can be used as dedicated hardware-based The system can be implemented, or can be implemented by a combination of dedicated hardware and computer program instructions. It is well known to those skilled in the art that implementation through hardware, implementation through software, and implementation through a combination of software and hardware are all equivalent.

In order to solve the problems of the prior art, embodiments of the present invention provide a sample labeling and reviewing method, device, electronic equipment, and computer-readable storage medium.

It should be noted that the sample labeling and reviewing method of the embodiment of the present invention can be applied to the sample labeling and reviewing device of the embodiment of the present invention, and the sample labeling and reviewing device can be configured on an electronic device. Wherein, the electronic device may be a personal computer, a mobile terminal, etc., and the mobile terminal may be a hardware device with various operating systems such as a mobile phone or a tablet computer.

Fig. 1 is a schematic flowchart of a sample labeling and reviewing method provided by an embodiment of the present invention. Please refer to Figure 1. A sample labeling review method can include the following steps:

Step S101: Obtain samples to be labeled.

Step S102: Recognizing at least one region of the sample to be labeled through a region recognition model, and cutting the at least one region to form at least one sample region picture; wherein the region recognition model is a neural network-based model.

The sample to be labeled may include various different types of picture samples, such as test papers, photos of animals and plants, scenic spots, vehicles, human faces, human body or part of human body constituents, objects, bills, etc., taking the test paper as an example, the area recognition The model will identify the area of each topic on the test paper and segment each area to form a sample image of the area. Then in step S103, a character recognition model is used to identify the character content of the sample pictures in each region and perform pre-labeling processing.

Step S103: Recognize each sample area picture through a preset recognition model and perform pre-labeling processing; wherein, the preset recognition model is a neural network-based model.

In this embodiment, the preset recognition model can be selected according to the type of the sample area picture and its annotation type. For example, if the sample area picture is a plant image, and the type of plant in the picture needs to be marked, the preset The recognition model may be a recognition model for recognizing plant species. After the recognition model recognizes the plant image, the recognition result is pre-labeled the plant image. For example, the plant image is a peach blossom image, and if the recognition model recognizes a peach blossom, the recognition result of the plant image is pre-marked as a peach blossom.

Step S104: Send the pre-labeled sample area picture to the review unit, so that the review unit can review the pre-labeled result of the sample area picture, and if the pre-labeled result is verified as an error, the pre-labeled The results are modified.

After the review unit receives the pre-labeled sample area picture, it can recognize the sample area picture, and judge whether the pre-labeled result of the sample area picture is correct according to its own recognition result, and if it is wrong, the pre-labeled result Modified to its own recognition result. For example, if the pre-annotation result of a plant picture is peach blossom, if the recognition result of the plant picture by the review unit is pear blossom, it means that the pre-annotation result is wrong, and the pre-label result is modified to be its own recognition result pear blossom .

Step S105: Send the sample area pictures reviewed by the review unit to the verification client, so that the verification client performs verification processing on the label information of the sample area pictures reviewed by the review unit.

In this embodiment, after the verification unit, the verification client is used to verify the sample area image reviewed by the verification unit to verify whether the label information reviewed by the verification unit is correct, so as to further ensure the sample area The labeling accuracy of the picture. After pre-labeling, reviewing, and testing the sample area pictures, accurate labeling information can be obtained.

In practical applications, in step S103, one or more of the preset recognition models may be used to identify and pre-label the sample area pictures, and the review unit in step S104 may be one or more manual clients , Or a combination of manual client and preset recognition model. Different preset recognition models are different recognition models established based on different training samples, so the recognition results and accuracy rates of each preset recognition model may be different.

In an implementation manner, pre-labeling processing is performed through one of the preset recognition models, and then a manual client is used as the review unit to perform pre-labeling and review processing on the sample area pictures.

Specifically, step S103 recognizes each sample area picture through a preset recognition model and performs pre-labeling processing, including:

A preset recognition model is used to recognize each sample area image and perform pre-labeling processing.

Step S104 sends the pre-labeled sample area pictures to the review unit, so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send the pre-labeled sample area pictures to a manual client, so that the artificial client can review the pre-labeled results of the sample area pictures.

In step S105, the verification client performs verification processing on the annotation information of the sample area picture after the verification unit has reviewed, including:

In this implementation manner, the sample area picture is pre-labeled by one of the preset recognition models, and then the pre-labeled result of the preset recognition model is reviewed by a manual client. If the manual client judges that the pre-labeling result is wrong, the pre-labeling result is modified.

Further, the verification client performs verification processing on the verification result of the manual client. If the verification client determines that the annotation information after verification by the manual client is accurate, the sample area The picture completes the process of labeling and reviewing. If it is inaccurate, the process of identifying, labeling and reviewing the pictures of the sample area will be performed.

For example, for a plant image, first recognize it through a recognition model for recognizing plant species. If the recognition result is A, perform pre-labeling processing to obtain pre-labeled result A; then, pre-labeled result A is obtained through an artificial client The results are reviewed. If the recognition result of the manual client is B, the pre-labeled result A is modified to B; finally, the verification client verifies whether the labeled information after the manual client’s review is accurate, The verification client recognizes the plant image. If the recognition result is B, it means that the current labeling information of the plant image is accurate. If the recognition result is not B, it means that the current labeling information of the plant image is inaccurate, and the plant image needs to be re-identified. . Through the above-mentioned labeling review process, the labeling accuracy of plant images can be improved.

In another implementation manner, pre-labeling processing is performed through one of the preset recognition models, and two manual clients are used as review units to perform pre-labeling and review processing on the sample area pictures. Finally, the verification client verifies whether the review results of the two manual clients are consistent, and if they are inconsistent, re-identify.

Specifically, in step S103, recognizing each sample area picture through a preset recognition model and performing pre-labeling processing includes:

In step S104, the pre-labeled sample area pictures are sent to the review unit, so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send the pre-labeled sample area pictures to two manual clients at the same time, so that both manual clients can review the pre-labeled results of the sample area pictures.

For each sample area picture, the verification client checks whether the label information of the sample area picture after the two manual clients review is consistent, and if they are inconsistent, the sample area picture is re-identified.

In this implementation, firstly, the sample area picture is pre-labeled through one of the preset recognition models, and then the pre-labeled results of this preset recognition model are reviewed by two manual clients. If the client judges that the pre-labeling result is wrong, it modifies the pre-labeling result.

Further, the verification client performs verification processing on the review results of the two manual clients. If the verification client determines that the marked information after the verification of the two manual clients is consistent, the sample area The picture completes the labeling review process, if inconsistent, the sample area pictures will be re-identified, labeled and reviewed.

For example, for a plant image, first perform recognition and pre-annotation processing through a recognition model for identifying plant species to obtain the pre-annotation result of the plant image; then, send the pre-annotated plant image to two A manual client, each manual client reviews the pre-labeled results of the plant image to determine whether the pre-labeled results are correct, if not correct, modify it to its own recognition results; finally, the two manual clients The verified plant image is sent to the verification client, and the verification client judges whether the labeling information after the two manual clients are the same. If they are consistent, it means that the current labeling information of the plant image is accurate; if they are inconsistent, it means that the plant image is current If the labeling information of is not accurate, the plant image needs to be re-identified. Through the above-mentioned labeling review process, the labeling accuracy of plant images can be improved.

In still another implementation manner, pre-labeling processing is performed through a plurality of the preset recognition models, and a plurality of manual clients are used as review units to perform pre-labeling and review processing on the sample area pictures.

The sample area pictures pre-labeled through different preset recognition models are sent to different manual clients at the same time, so that the manual client can review the pre-labeled results of the sample area pictures.

In this implementation manner, the sample area pictures are pre-labeled through different preset recognition models (for example, two), so that one sample area picture can generate multiple pre-labeled samples accordingly, and then These multiple pre-labeled samples are sent to different artificial clients (for example, two) at the same time, and each of the artificial clients reviews the multiple pre-labeled samples. If the artificial client judges that the pre-labeled results are incorrect , Modify the pre-labeled result.

Further, the verification client performs verification processing on the review results of each of the manual clients. For each sample area picture, if the verification client determines the labeling information reviewed by different manual clients If they are consistent, the image of the sample area completes the process of labeling and reviewing. If they are inconsistent, the process of identifying, labeling, and reviewing the image of the sample area is performed again.

For example, for a plant image, firstly use two recognition models 1 and 2 for identifying plant species to perform recognition and pre-annotation processing respectively to obtain two pre-annotated plant images. The pre-annotation result of one plant image is The recognition result of recognition model 1 and the pre-annotation result of another plant image are the recognition results of recognition model 2. Then, the two pre-annotated plant images are sent to two artificial clients at the same time, and each artificial client is Two pre-annotated plant images are reviewed to determine whether the recognition result of recognition model 1 is correct and whether the recognition result of recognition model 2 is correct. If it is incorrect, modify it to its own recognition result; finally, each artificial customer The plant image after verification by the client is sent to the verification client, and the verification client verifies whether the annotation information after the verification by the two manual clients is consistent. If they are consistent, the current annotation information of the plant image is accurate, and if they are inconsistent, it represents the plant image. If the current labeling information is not accurate, the plant image needs to be re-identified. Through the above-mentioned labeling review process, the labeling accuracy of plant images can be improved.

In yet another implementation manner, pre-labeling processing is performed through one of the preset recognition models, and one of the manual clients and the other preset recognition model are used as review units to pre-label the sample area pictures And audit processing.

Recognize and label the pre-labeled sample area pictures through another preset recognition model; wherein, the one preset recognition model and the another preset recognition model are trained and established based on different training sample sets Identify the model.

In this implementation manner, the sample area picture is pre-labeled through a preset recognition model, and then this preset recognition model is reviewed through a manual client and another preset recognition model If the manual client determines that the pre-labeling result is incorrect, the pre-labeling result is modified, and the other preset recognition model is to identify and label the sample area picture.

Further, the verification client performs verification processing on the review results of the manual client and another preset recognition model. For each sample area picture, if the verification client determines If the labeling information after the manual client review is consistent with the recognition result of another preset recognition model, the sample area picture completes the labeling review process, and if it is inaccurate, the sample area picture is re-identified, marked and reviewed Process.

For example, for a plant image, first perform recognition and pre-annotation processing through a recognition model for identifying plant species to obtain the pre-annotated result A of the plant image; then, send the pre-annotated plant image to an artificial client On the other hand, the artificial client terminal reviews the pre-annotation result. If the artificial client's recognition result of the plant image is B, then the pre-annotation result of the plant image is modified to B, and another recognition model Recognize and label the plant image to get the labeling result C; finally, the verification client verifies whether the labeling information B after the manual client’s review is consistent with the labeling result C of another recognition model. If they are consistent, the plant image is currently If the labeling information is not accurate, it means that the current labeling information of the plant image is not accurate, and the plant image needs to be re-identified. Through the above-mentioned labeling review process, the labeling accuracy of plant images can be improved.

This embodiment introduces the pre-labeling, reviewing, and verification procedures of samples through the above three implementation methods, but the technical solution of the present invention is not limited to this.

Further, during the above-mentioned pre-labeling, reviewing and verification process, it is also possible to check whether the reviewing unit is in an abnormal state, and if an abnormality occurs, the reviewing unit is corrected to further ensure the accuracy of the labeling of the sample.

Specifically, after the pre-labeling processing is performed in step S103, the method further includes:

A preset number of pictures are selected from the sample area pictures recognized by the preset recognition model, and the pre-labeled results of the selected preset number of pictures are modified into recognition results different from the original pre-labeled results.

The verification client checks whether the verification unit is in an abnormal state according to the review result of the review unit on the preset number of pictures of which the pre-marked result has been modified.

Modify the pre-labeled result of the picture to be inconsistent or different from the original pre-labeled result. For example, if the pre-labeled result of a portrait picture is female, then the pre-labeled result of the picture is modified to the original Recognition results with different pre-labeled results, such as modification to male. For another example, if the pre-annotation result of a certain plant picture is peach blossom, then the pre-annotation result of the plant picture is modified to an inconsistent or different recognition result, such as pear blossom.

In this embodiment, a preset number of pictures can be randomly selected from all sample area pictures, and the pre-labeled results of the extracted pictures can be modified to different recognition results. Since this embodiment calculates the review status of the false pre-annotation results of the extracted preset number of pictures by the review unit, infers the review status of the review unit on all sample area pictures, and then determines the review unit Whether it is in an abnormal state, therefore, in order to ensure the accuracy of subsequent statistics, the following requirements may be imposed on the number of pictures to be extracted:

The preset number is greater than or equal to the minimum sampling number N for sampling statistics;

Among them, N=Z ² ×(P×(1-P))/E ² ; Z represents the statistics related to the confidence level, which is equal to the recognition accuracy rate of the current preset recognition model; E represents the preset sampling error Value; P represents the labeling accuracy of the sample area image after being labelled by the current preset recognition model.

The corresponding relationship between Z and confidence is as follows: when the confidence is 90%, Z=1.64; when the confidence is 95%, Z=1.96; when the confidence is 95.45%, Z=2; when the confidence is 99%, Z=2.68; when the confidence level is 99.73, Z=3; the above data can be obtained by querying the statistical table. The confidence interval of this embodiment is 90%-99.99%, that is to say, it is believed that the recognition accuracy of the current preset recognition model should fall within the range of P from 90%-99.99%, and 95% can be used in this embodiment. Confidence level. In this embodiment, the sampling error value E can be set between ±5%, and P is the probability value, which can be set to 90%, that is, the labeling accuracy of the sample area image after labeling by the current preset recognition model needs to reach 90%. If the minimum sample extraction number N is equal to 100 calculated by the above calculation formula, the preset number can be set to any value greater than or equal to 100.

In this embodiment, the review unit reviews the pre-labeled results of the sample area pictures that have not modified the pre-labeled results and the sample area pictures that have modified the pre-labeled results. The review process includes: for each sample area picture, the The review unit judges whether the marked pre-marked result is correct; if it is incorrect, the pre-marked result marked on the sample area picture can be modified. For example, if the pre-labeled result of a certain picture is female, and the review unit determines that the pre-labeled result of the sample is wrong after the review, and after its own recognition, it is determined that the recognition result of the picture should be male, then The pre-labeling result of the picture can be modified to the recognition result determined by itself.

In fact, for a sample area picture whose pre-labeling result has been modified, the verification unit may not recognize that the pre-labeling result of the picture is wrong, which causes the verification unit to determine that the recognition result of the picture is correct. The review by the review unit of the sample area pictures for deliberately modifying the pre-annotated results reflects the recognition (annotation) status of all the sample area pictures by the review unit, and then checks the review status of such picture samples by the review unit It is possible to infer the labeling accuracy or the audit accuracy of the audit unit, and determine whether the audit unit is abnormal.

In an implementation manner, the verification client checks whether the verification unit is in an abnormal state according to the verification result of the verification unit on the preset number of pictures of the modified pre-annotation result, including:

It can be understood that, generally speaking, if the review unit can modify the pre-labeled result of a picture whose pre-labeled result has been modified, it can be considered that the review unit can correctly label the picture. If the proportion of pictures whose pre-annotation results have been modified by the review unit is greater than or equal to the preset threshold, it can be considered that there is no abnormality in the review unit, otherwise, it means that the review unit has abnormal.

In another implementation manner, the verification client checks whether the verification unit is in an abnormal state according to the verification result of the verification unit on the preset number of pictures of the modified pre-annotation result, including:

In this implementation manner, if the proportion of the preset number of pictures for which the pre-annotated result has been modified is modified to the correct recognition result by the review unit, the proportion of the pictures that are modified as the correct recognition result is greater than or equal to the preset threshold, it can be considered that the review unit has no abnormality, Otherwise, it means that the audit unit is abnormal. According to the proportion of the preset number of pictures of the modified pre-annotated results that are modified to the correct recognition result by the review unit, it is used to determine whether the review unit is abnormal, and to characterize the review unit The labeling accuracy rate is more accurate than the previous implementation.

When it is determined that the first ratio is less than the preset threshold, or the second ratio is less than the preset threshold, it can be determined that the audit unit is in an abnormal state, and it also means that the tagging accuracy of the audit unit does not reach the expected value, so The review unit makes corrections to make the marking accuracy rate meet the requirements.

Wherein, the minimum value X of the preset threshold value can be determined according to the following formula: 1-(1-X) ² =Q; Q represents the preset identification model labeling and the sample labeling after the auditing unit has been reviewed The target accuracy rate. The preset threshold can be set to any value equal to or greater than X, which is not limited in this embodiment.

When Q=99%, X=90% is calculated by the above formula, that is, the labeling accuracy of the audit unit needs to reach 90% or more. In this embodiment, if it is desired that the accuracy of the sample labeling after the preset recognition model labeling and the auditing unit review is more than 99%, the auditing unit will list the pre-annotated results in the preset number of pictures. The labeling accuracy rate needs to be above 90%.

To sum up, this embodiment first recognizes at least one area of the sample to be labeled through the region recognition model, and cuts to form at least a picture of the sample area, and then uses the preset recognition model to identify each sample area picture Perform pre-labeling processing, and then review the pre-labeled results of the sample area pictures through the review unit. If the pre-labeled results are wrong, modify the pre-labeled results, and finally use the verification client to verify the results of the pre-labeling. The label information of the sample area picture after the review by the review unit is verified. It can be seen that, in this embodiment, the samples are labeled according to the pre-labeling, reviewing, and verification process, which can ensure the accuracy of sample labeling, thereby improving the accuracy of model training. Further, the pre-marked results of a part of the sample area pictures are deliberately modified to the wrong recognition results, and the marking accuracy rate of the review unit can be inferred by checking the review results of the part of the sample area pictures that are deliberately incorrectly marked by the review unit. Furthermore, it is determined whether the audit unit is in an abnormal state, which realizes the rapid determination of whether the audit unit is in an abnormal state, shortens the statistical time and reduces the cost.

Corresponding to the foregoing embodiment of the sample labeling and reviewing method, an embodiment of the present invention also provides a sample labeling and reviewing device. FIG. 2 is a schematic structural diagram of a sample labeling and reviewing device provided by an embodiment of the present invention. Please refer to Figure 2. A sample labeling review device may include:

The obtaining module 201 is used to obtain samples to be labeled;

The recognition module 202 is configured to recognize at least one region of the sample to be labeled through a region recognition model, and cut the at least one region to form at least one sample region picture; wherein the region recognition model is a neural network-based model ；

The pre-labeling module 203 is configured to recognize each sample area picture through a preset recognition model and perform pre-labeling processing; wherein, the preset recognition model is a neural network-based model;

The review module 204 is configured to send the pre-labeled sample area pictures to the review unit, so that the review unit can review the pre-labeled results of the sample area pictures. Revise the pre-marked results;

The verification module 205 is configured to send the sample area pictures reviewed by the review unit to the verification client, so that the verification client performs verification processing on the label information of the sample area pictures reviewed by the review unit.

Optionally, the pre-labeling module 203 recognizes each sample area picture through a preset recognition model and performs pre-labeling processing, including:

The review module 204 sends the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

Optionally, the verification process performed by the verification client in the verification module 205 on the annotation information of the sample area picture after being reviewed by the verification unit includes:

Optionally, the verification processing performed by the verification client in the verification module 205 on the annotation information of the sample area picture after the verification unit has reviewed includes:

Optionally, after the pre-labeling module 203 performs pre-labeling processing, it is further used to:

After the verification client in the verification module 205 performs verification processing on the annotation information of the sample area picture reviewed by the verification unit, it is further used to:

Optionally, the review unit in the review module 204 reviews the pre-labeled results of the sample area pictures of the unmodified pre-labeled results and the sample area pictures of the modified pre-labeled results, including:

Optionally, the verification in the verification module 205 that the verification client checks whether the verification unit is in an abnormal state according to the verification result of the verification unit on the preset number of pictures of the modified pre-annotation result, including:

An embodiment of the present invention also provides an electronic device. FIG. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. 3, an electronic device includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304. The processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

The memory 303 is used to store computer programs;

The processor 301 is configured to implement the following steps when executing the program stored in the memory 303:

Step 1: Obtain samples to be labeled;

For the specific implementation of each step of the method and related explanation content, please refer to the method embodiment shown in FIG. 1 above, which will not be repeated here.

In addition, other implementation manners of the sample labeling review method implemented by the processor 301 executing the program stored in the memory 303 are the same as the implementation manners mentioned in the foregoing method embodiment section, and will not be repeated here.

To sum up, this embodiment first recognizes at least one area of the sample to be labeled through the region recognition model, and cuts to form at least a picture of the sample area, and then uses the preset recognition model to identify each sample area picture Perform pre-labeling processing, and then review the pre-labeled results of the sample area pictures through the review unit. If the pre-labeled results are wrong, modify the pre-labeled results, and finally use the verification client to verify the results of the pre-labeling. The label information of the sample area picture after the review by the review unit is verified. It can be seen that, in this embodiment, the samples are labeled according to the process of pre-labeling, review, and verification, which can ensure the accuracy of sample labeling, thereby improving the accuracy of model training. Further, the pre-marked results of a part of the sample area pictures are deliberately modified to the wrong recognition results, and the marking accuracy rate of the review unit can be inferred by checking the review results of the part of the sample area pictures that are deliberately incorrectly marked by the review unit. Furthermore, it is determined whether the audit unit is in an abnormal state, which realizes the rapid determination of whether the audit unit is in an abnormal state, shortens the statistical time and reduces the cost.

The electronic device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the aforementioned electronic device and other devices. The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located far away from the foregoing processor.

The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (DSP), a dedicated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the electronic device, and various interfaces and lines are used to connect various parts of the entire electronic device.

An embodiment of the present invention also provides a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the following steps can be implemented:

Step 1: Obtain samples to be labeled;

Step 2: Identify at least one region of the sample to be labeled through a region recognition model, and cut the at least one region to form at least one sample region picture; wherein, the region recognition model is a neural network-based model;

It should be noted that other embodiments of the sample labeling and reviewing method implemented when the computer program is executed by the processor are the same as the sample labeling and reviewing method mentioned in the foregoing method section, and will not be repeated here.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device, such as, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or Any suitable combination of the above. More specific examples of computer-readable storage media (non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon The protruding structure in the hole card or the groove, and any suitable combination of the above. The computer program described here can be downloaded from a computer-readable storage medium to each computing/processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer program from the network, and forwards the computer program for storage in a computer-readable storage medium in each computing/processing device. The computer program used to perform the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or any of one or more programming languages. Source code or object code written in combination, the programming language includes object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages. The computer program may be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server . In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to access connection). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer program. Read the program instructions to implement various aspects of the present invention.

The various aspects of the present invention are described herein with reference to flowcharts and/or block diagrams of methods, systems, and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart and/or block diagram and the combination of each block in the flowchart and/or block diagram can be implemented by a computer program. These computer programs can be provided to the processors of general-purpose computers, special-purpose computers, or other programmable data processing devices to produce a machine that, when executed by the processors of the computer or other programmable data processing devices, produces A device that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram. It is also possible to store these computer programs in a readable storage medium. These computer programs make computers, programmable data processing apparatuses and/or other devices work in a specific manner, so that the readable storage medium storing the computer programs includes a An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams. The computer program can also be loaded on a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process, so that A computer program executed on a computer, other programmable data processing device, or other device implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram.

It should be noted that each embodiment in this specification is described in a related manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. . In particular, for the device, electronic equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiments.

In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such existence between these entities or operations. The actual relationship or order. Moreover, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also includes Other elements of, or also include elements inherent to this process, method, article or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article, or equipment including the element. The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an", and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. As used herein, the term "and/or" includes any and all combinations of one or more related listed items. When expressions such as "at least one of" follow a list of elements, they modify the entire list of elements, but do not modify individual elements in the list. As used herein, the terms "substantially", "about" and similar terms are used as approximate terms, not degree terms, and are intended to indicate inherent deviations in measured or calculated values, which will be determined by those Recognized by those of ordinary skill in the art. In addition, when describing the embodiments of the present invention, the use of "may" refers to "one or more embodiments of the present invention." As used herein, the terms "used", "used" and "used" can be considered as synonymous with the terms "used", "used" and "used" respectively. Likewise, the term "exemplary" is intended to indicate an instance or example.

The foregoing description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes or modifications made by persons of ordinary skill in the field of the present invention based on the foregoing disclosure shall fall within the protection scope of the claims.

Claims

A sample labeling review method, characterized in that the method includes:

Step 1: Obtain samples to be labeled;

Step 2: Identify at least one region of the sample to be labeled through a region recognition model, and cut the at least one region to form at least one sample region picture; wherein the region recognition model is a neural network-based model;

Step 3: Recognize each sample area picture through a preset recognition model and perform pre-labeling processing; wherein the preset recognition model is a neural network-based model;

Step 4: Send the pre-labeled sample area pictures to the review unit, so that the review unit can review the pre-labeled results of the sample area pictures, and if the pre-labeled result is verified as an error, the pre-labeled The results are modified;

Step 5: Send the sample area pictures reviewed by the review unit to the verification client, so that the verification client performs verification processing on the label information of the sample area pictures reviewed by the review unit.
The sample labeling review method according to claim 1, wherein step 3 identifies each sample area picture through a preset recognition model and performs pre-labeling processing, comprising:

Recognize each sample area picture through a preset recognition model and perform pre-labeling processing;

Step 4 sends the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

The pre-labeled sample area picture is sent to a manual client, so that the manual client can review the pre-labeled result of the sample area picture.
The sample labeling and reviewing method according to claim 2, wherein the verification client in step 5 performs verification processing on the labeling information of the sample area picture after being reviewed by the review unit, comprising:

For each sample area picture, the verification client checks whether the labeling information of the sample area picture after the manual client's review is accurate, and if it is not accurate, re-identifies the sample area picture.
The sample labeling review method according to claim 1, wherein step 3 identifies each sample area picture through a preset recognition model and performs pre-labeling processing, comprising:

Recognize each sample area picture through at least two preset recognition models and perform pre-labeling processing;

Step 4 sends the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

The sample area pictures pre-labeled by different preset recognition models are sent to each of a plurality of different manual clients, so that each manual client can review the pre-labeled results of each sample area picture.
The sample labeling and reviewing method according to claim 4, wherein the verification client in step 5 performs verification processing on the labeling information of the sample area pictures after being reviewed by the review unit, comprising:

For each sample area picture, the verification client checks whether the label information of the sample area picture after the review by different manual clients is consistent, and if they are inconsistent, the sample area picture is re-identified.
The sample labeling review method according to claim 1, wherein step 3 identifies each sample area picture through a preset recognition model and performs pre-labeling processing, comprising:

Recognize each sample area picture through a preset recognition model and perform pre-labeling processing;

Step 4 sends the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send the pre-labeled sample area picture to a manual client, so that the manual client can review the pre-labeled result of the sample area picture; and

Recognize and label the pre-labeled sample area pictures through another preset recognition model;

Wherein, the one preset recognition model and the another preset recognition model are recognition models established by training based on different training sample sets.
The sample labeling and reviewing method according to claim 6, wherein the verification client in step 5 performs verification processing on the labeling information of the sample area picture after the review unit has reviewed, comprising:

For each sample area picture, the verification client checks whether the label information of the sample area picture after the manual client's review is consistent with that of another preset recognition model. To identify.
The sample labeling review method according to claim 1, wherein after the pre-labeling processing in step 3, the method further comprises:

Selecting a preset number of pictures from the sample area pictures recognized by the preset recognition model, and modifying the pre-labeled result of the selected preset number of pictures to a recognition result different from the original pre-labeled result;

In step 4, the pre-labeled sample area pictures are sent to the review unit, so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send both the sample area picture of the unmodified pre-labeled result and the sample area picture of the modified pre-labeled result to the review unit, so that the review unit can check the sample area picture of the unmodified pre-labeled result and the sample of modified pre-labeled result Review the pre-annotated results of regional pictures;

After the verification client in step 5 performs verification processing on the label information of the sample area picture reviewed by the review unit, it also includes:

The verification client determines whether the verification unit is in an abnormal state according to the review result of the review unit on the preset number of pictures of which the pre-marked result has been modified.
The sample labeling review method according to claim 8, wherein the reviewing unit reviews the pre-labeled results of the sample area pictures of the unmodified pre-labeled results and the sample area pictures of the modified pre-labeled results, comprising:

For each sample area picture, the review unit judges whether the marked pre-marking result is correct; if not, the pre-marking result marked on the sample area picture is modified.
The sample labeling review method according to claim 9, wherein the verification client checks the review unit according to the review result of the review unit on the preset number of pictures of the modified pre-marked result Whether it is in an abnormal state, including:

For each picture in the preset number of pictures for which the pre-annotation result has been modified, determine whether the review unit has modified the pre-annotation result of the picture;

Acquiring, as the first ratio, a ratio of pictures whose pre-annotation results have been modified by the review unit among the preset number of pictures for which the pre-annotation results have been modified;

If the first ratio is less than a preset threshold, it is determined that the audit unit is in an abnormal state.
The sample labeling review method according to claim 9, wherein the verification client checks the review unit according to the review result of the review unit on the preset number of pictures of the modified pre-marked result Whether it is in an abnormal state, including:

For each picture in the preset number of pictures for which the pre-labeled result has been modified, determine whether the review unit modifies the pre-labeled result of the picture to a correct recognition result;

Acquiring, as a second ratio, a ratio of the pictures that have been modified as a correct recognition result by the review unit among the preset number of pictures of the modified pre-annotation result;

If the second ratio is less than the preset threshold, it is determined that the audit unit is in an abnormal state.
A sample labeling review device, characterized in that the device includes:

Obtaining module for obtaining samples to be labeled;

A recognition module, configured to recognize at least one region of the sample to be labeled through a region recognition model, and cut the at least one region to form at least one sample region picture; wherein the region recognition model is a neural network-based model;

The pre-labeling module is used to identify each sample area picture through a preset recognition model and perform pre-labeling processing; wherein, the preset recognition model is a neural network-based model;

The review module is configured to send the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the pre-labeled sample area pictures, if the pre-labeled result is verified as If it is wrong, modify the pre-marked result;

The verification module is configured to send the sample area pictures reviewed by the review unit to the verification client, so that the verification client performs verification processing on the label information of the sample area pictures reviewed by the review unit.
The sample labeling and reviewing device according to claim 12, wherein the pre-labeling module recognizes each sample area picture through a preset recognition model and performs pre-labeling processing, comprising:

Recognize each sample area picture through a preset recognition model and perform pre-labeling processing;

The review module sends the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the pre-labeled sample area pictures, including:

Sending the pre-labeled sample area picture to a manual client, so that the manual client can review the pre-labeled result of the sample area picture;

The verification process performed by the verification client on the label information of the sample area picture after being reviewed by the verification unit includes:

For each sample area picture, the verification client checks whether the labeling information of the sample area picture after the manual client's review is accurate, and if it is not accurate, re-identifies the sample area picture.
The sample labeling and reviewing device according to claim 12, wherein the pre-labeling module recognizes each sample area picture through a preset recognition model and performs pre-labeling processing, comprising:

Recognize each sample area picture through at least two preset recognition models and perform pre-labeling processing;

The review module sends the pre-labeled sample area pictures to the review unit, so that the review unit can review the pre-labeled results of the pre-labeled sample area pictures, including:

Send the sample area pictures pre-labeled through different preset recognition models to each of multiple different manual clients, so that each manual client can review the pre-labeled results of each sample area picture;

The verification process performed by the verification client on the label information of the sample area picture after being reviewed by the verification unit includes:

For each sample area picture, the verification client checks whether the label information of the sample area picture after the review by different manual clients is consistent, and if they are inconsistent, the sample area picture is re-identified.
The sample labeling and reviewing device according to claim 12, wherein the pre-labeling module recognizes each sample area picture through a preset recognition model and performs pre-labeling processing, comprising:

Recognize each sample area picture through a preset recognition model and perform pre-labeling processing;

The review module sends the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the pre-labeled sample area pictures, including:

Send the pre-labeled sample area picture to a manual client, so that the manual client can review the pre-labeled result of the sample area picture; and

Recognize and label the pre-labeled sample area pictures through another preset recognition model;

Wherein, the one preset recognition model and the another preset recognition model are recognition models established by training based on different training sample sets;

The verification process performed by the verification client on the label information of the sample area picture after being reviewed by the verification unit includes:

For each sample area picture, the verification client verifies whether the label information of the sample area picture after the manual client's review is consistent with that of another preset recognition model. Recognition.
The sample labeling review device according to claim 12, wherein the pre-labeling module is further used for:

Selecting a preset number of pictures from the sample area pictures recognized by the preset recognition model, and modifying the pre-labeled result of the selected preset number of pictures to a recognition result different from the original pre-labeled result;

Sending the pre-labeled sample area pictures to the review unit so that the review unit can review the pre-labeled results of the sample area pictures, including:

Send both the sample area picture of the unmodified pre-labeled result and the sample area picture of the modified pre-labeled result to the review unit, so that the review unit can check the sample area picture of the unmodified pre-labeled result and the sample of modified pre-labeled result The pre-annotated results of regional pictures are reviewed, including:

For each sample area picture, the review unit judges whether the marked pre-marking result is correct; if not, the pre-marking result marked on the sample area picture is modified;

After the verification client performs verification processing on the annotation information of the sample area picture that has been reviewed by the review unit, it is also used to:

According to the review result of the review unit on the preset number of pictures for which the pre-annotation result has been modified, it is checked whether the review unit is in an abnormal state.
The sample labeling and reviewing device according to claim 16, wherein the verification client checks the reviewing unit according to the reviewing result of the reviewing unit on the preset number of pictures of the modified pre-marking result Whether it is in an abnormal state, including:

For each picture in the preset number of pictures for which the pre-annotation result has been modified, determine whether the review unit has modified the pre-annotation result of the picture;

Acquiring, as the first ratio, a ratio of pictures whose pre-annotation results have been modified by the review unit among the preset number of pictures for which the pre-annotation results have been modified;

If the first ratio is less than a preset threshold, it is determined that the audit unit is in an abnormal state.
The sample labeling and reviewing device according to claim 16, wherein the verification client in the verification module checks all the pictures according to the review results of the review unit on the preset number of pictures of the modified pre-labeled results. Describe whether the audit unit is in an abnormal state, including:

For each picture in the preset number of pictures for which the pre-labeled result has been modified, determine whether the review unit modifies the pre-labeled result of the picture to a correct recognition result;

Acquiring, as a second ratio, a ratio of the pictures that have been modified as a correct recognition result by the review unit among the preset number of pictures of the modified pre-annotation result;

If the second ratio is less than the preset threshold, it is determined that the audit unit is in an abnormal state.
An electronic device, characterized by comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus;

The memory is used to store computer programs;

The processor is configured to implement the method according to any one of claims 1-11 when executing the computer program stored on the memory.
A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1-11 is implemented.