WO2020253741A1 - 利用错误样本检查人工客户端状态的方法及装置 - Google Patents
利用错误样本检查人工客户端状态的方法及装置 Download PDFInfo
- Publication number
- WO2020253741A1 WO2020253741A1 PCT/CN2020/096646 CN2020096646W WO2020253741A1 WO 2020253741 A1 WO2020253741 A1 WO 2020253741A1 CN 2020096646 W CN2020096646 W CN 2020096646W WO 2020253741 A1 WO2020253741 A1 WO 2020253741A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- error
- sample
- samples
- manual client
- manual
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Definitions
- the present invention relates to the field of artificial intelligence technology, and in particular to a method, a device, an electronic device, and a computer-readable storage medium for checking the state of an artificial client using error samples.
- a manual client In the field of artificial intelligence model training, a manual client is required to label samples or to review the labeling results of samples. If the manual client is in an abnormal state during sample labeling or review, the labeling of the sample cannot be guaranteed to meet the standard , Resulting in the recognition accuracy of the trained model not up to the standard. Therefore, it is necessary to check the status of the manual client to ensure that the manual client is in a normal state during sample labeling or review, so that the accuracy of the trained model meets the standard.
- the purpose of the present invention is to provide a method, device, electronic equipment and computer-readable storage medium for checking the status of a manual client using error samples, so as to quickly determine the labeling accuracy of the manual client.
- the specific technical solutions are as follows:
- the present invention provides a method for checking the status of a manual client using error samples, the method including:
- the manual client's review of the annotation results of each sample in the sample set to be processed includes:
- the human client judges whether the marked annotation result is correct; if not, the annotation result marked by the sample is modified.
- determining whether the manual client is in an abnormal state according to the review result of the manual client on the preset number of error samples includes:
- the first ratio is less than a preset threshold, it is determined that the manual client is in an abnormal state.
- the method further includes:
- the marking accuracy rate of the manual client is determined according to the first ratio.
- checking whether the manual client is in an abnormal state according to the review result of the manual client on the preset number of error samples includes:
- the method further includes:
- the preset number is greater than or equal to the minimum sample extraction number N;
- N Z2 ⁇ (P ⁇ (1-P))/E2;
- Z represents the statistics related to the confidence level, which is the probability that the audit accuracy rate of the manual client falls within the preset confidence interval;
- E represents the preset sampling error value;
- P represents the accuracy of sample labeling in the test sample set.
- the preset threshold is a preset target audit accuracy rate of the manual client terminal.
- the obtaining a preset number of error samples includes:
- a preset number of error samples are obtained from an error sample library, where the error sample library contains error samples marked with an error annotation result.
- the method further includes:
- the feature of the error sample in the error sample library is extracted, a new error sample is generated according to the extracted feature, and the generated new error sample is added to the error sample library.
- the present invention also provides a device for checking the status of a manual client using error samples, the device comprising:
- the acquisition module is used to acquire a preset number of error samples and add them to a test sample set to form a sample set to be processed; wherein the error samples are pre-marked with the error labeling results, and the samples in the test sample set are pre-marked with labels result;
- An audit module configured to send the sample set to be processed to a manual client, so that the manual client can review the annotation results of each sample in the sample set to be processed;
- the checking module is configured to determine whether the manual client is in an abnormal state according to the review result of the preset number of error samples by the manual client.
- the manual client's review of the annotation results of each sample in the sample set to be processed includes:
- the human client judges whether the marked annotation result is correct; if not, the annotation result marked by the sample is modified.
- the inspection module is used to:
- the device further includes:
- the first determining module is configured to determine the labeling accuracy rate of the manual client according to the first ratio.
- the inspection module is used to:
- the device further includes:
- the second determining module is configured to determine the labeling accuracy rate of the manual client according to the second ratio.
- the preset number is greater than or equal to the minimum sample extraction number N;
- N Z 2 ⁇ (P ⁇ (1-P))/E 2 ;
- Z represents the statistics related to the confidence level, which is the probability that the audit accuracy rate of the manual client falls within the preset confidence interval ;
- E represents the preset sampling error value;
- P represents the accuracy of sample labeling in the test sample set.
- the preset threshold is a preset target audit accuracy rate of the manual client terminal.
- the acquiring module acquires a preset number of error samples, including:
- a preset number of error samples are obtained from an error sample library, where the error sample library contains error samples marked with an error annotation result.
- the device further includes:
- the generating module is used to extract the features of the error samples in the error sample library, generate new error samples according to the extracted features, and add the generated new error samples to the error sample library.
- the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus.
- the memory is used to store computer programs
- the processor is configured to implement the method for checking the state of an artificial client terminal using error samples as described in the first aspect when executing the computer program stored on the memory.
- the present invention also provides a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the utilization error described in the first aspect is realized.
- the present invention adds a preset number of error samples to a test sample set to form a sample set to be processed, and sends the sample set to be processed to a manual client for review.
- the result of the review can be used to infer the labeling accuracy rate of the manual client for the entire sample set to be processed, without the need to count the manual client’s labeling of the entire sample set to be processed to determine the labeling accuracy, thus realizing the rapid determination of the manual client Whether it is in an abnormal state, and shorten the statistical time, reduce the cost.
- FIG. 1 is a schematic flowchart of a method for checking the state of a manual client by using error samples according to an embodiment of the present invention
- FIG. 2 is a schematic structural diagram of an apparatus for checking the state of a manual client using error samples according to an embodiment of the present invention
- FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.
- the embodiments of the present invention provide a method, an apparatus, an electronic device, and a computer-readable storage medium for checking the status of an artificial client using error samples.
- the method for checking the state of a manual client terminal using error samples in the embodiment of the present invention can be applied to the device for checking the state of a manual client terminal using error samples in the embodiment of the present invention.
- the electronic device may be a personal computer, a mobile terminal, etc.
- the mobile terminal may be a hardware device with various operating systems such as a mobile phone or a tablet computer.
- FIG. 1 is a schematic flowchart of a method for checking the status of a manual client by using error samples according to an embodiment of the present invention. Please refer to Figure 1.
- a method of using error samples to check the status of a manual client can include the following steps:
- Step S101 Obtain a preset number of error samples and add them to a test sample set to form a sample set to be processed; wherein the error samples are pre-marked with error labeling results, and the samples in the test sample set are pre-marked with labeling results.
- the error samples may be bill pictures, vehicle pictures, face pictures, plant pictures, test paper pictures, and so on.
- the type of the sample in the test sample set needs to be the same as the type of the error sample. For example, if the sample in the test sample set is a face picture, then the preset number of error samples obtained are also face pictures.
- the wrong sample is labeled with the wrong labeling result, that is, the labeling result of the wrong sample is inconsistent or different from the real recognition result. For example, a wrong sample is a face image, and the real recognition result of the image is "Gender: Female", but it is incorrectly labeled as "Gender” :male".
- Each sample in the test sample set is also pre-marked with an annotation result.
- each sample in the test sample set is identified and annotated by a recognition model, or manually marked.
- This embodiment does not limit whether the annotation result of each sample is correct.
- This embodiment also does not limit the number of samples in the test sample set, but in order to make the judgment result of judging whether the manual client is abnormal, the number of samples should usually be set to a larger value, such as 100, 1000, etc.
- the labeling result of the sample in the error sample and the test sample set can be directly labelled on the sample picture, or it can be labelled as the attribute information of the sample picture.
- the error samples may be samples that are intentionally marked with errors manually, or samples with errors marked in them can be reviewed and filtered by obtaining a set of marked samples. Furthermore, it is possible to maintain an error sample library, add samples with errors to it, and classify and store them according to sample types. Using the error sample library can improve the efficiency of obtaining error samples, thereby improving the efficiency of checking whether the manual client is abnormal.
- the acquiring a preset number of error samples may be: acquiring a preset number of error samples from an error sample library, where the error sample library includes samples marked with an incorrectly labeled result.
- the core idea of the present invention is to infer the review status of the manual client on the entire sample set of samples to be processed by counting the manual client's review of the error labeling results of the preset number of wrong samples, and then determine whether the manual client is in Abnormal status. Therefore, in order to ensure the accuracy of subsequent statistics, the following requirements can be imposed on the number of error samples:
- the preset number is greater than or equal to the minimum sample extraction number N;
- N Z 2 ⁇ (P ⁇ (1-P))/E 2 ;
- Z represents the statistics related to the confidence level, which is the probability that the audit accuracy rate of the manual client falls within the preset confidence interval ;
- E represents the preset sampling error value;
- P represents the accuracy of sample labeling in the test sample set.
- the confidence interval of this embodiment is 90%-99.99%, that is to say, it is believed that the audit accuracy rate of the manual client should fall within the range of P at 90%-99.99%.
- This embodiment can use a 95% Confidence.
- the sampling error value E can be set within ⁇ 5%, and P is the probability value, which can be set to 90%. If the minimum sample extraction number N is equal to 100 calculated by the above calculation formula, the preset number can be set to any value greater than or equal to 100.
- the characteristics of the error samples in the error sample library can also be extracted, new error samples are generated according to the extracted features, and the generated new error samples are added to the error sample library. For example, if a wrong sample is a plant picture, the plant in it is a rose but it is incorrectly labeled as a rose, you can extract the characteristics of the rose in the wrong sample, and then obtain other samples that match the plant characteristics with the rose characteristics in the picture. Plants in other samples are also labeled as roses, thereby generating new error samples.
- Step S102 Send the sample set to be processed to a manual client, so that the manual client can review the annotation results of each sample in the sample set to be processed.
- the manual client will review the labeling results of each sample in the sample set to be processed, and the review processing includes: for each sample in the sample set to be processed, the manual client judges whether the labeling result is correct ; If not, modify the annotation results marked on the sample. If the judgment is incorrect, the recognition result marked on the sample can be modified. It should be noted that since the sample set to be processed contains two types of samples, namely the samples after labeling according to the normal labeling process, and the added samples with incorrect labeling results, the manual client will not treat these samples during the review. The two types of samples are distinguished, but the wrong samples are also regarded as samples that are labeled according to the normal labeling process.
- the labeling result of a sample is female, and the human client judges that the labeling result of the sample is wrong after review, and after its own recognition, it is determined that the sample’s recognition result should be male, then it can The labeling result of the sample is modified to the recognition result judged by itself.
- the manual client may not recognize that the sample is incorrectly labeled, which causes the manual client to judge the labeling result of the sample as correct.
- the review of the wrong samples by the manual client reflects the recognition (labeling) of the samples in the entire set of samples to be processed by the manual client, and then the manual client’s labeling accuracy can be inferred by checking the manual client’s review of such samples Or review the accuracy rate and determine whether there is an abnormality in the manual client.
- Step S103 Check whether the manual client is in an abnormal state according to the audit result of the preset number of error samples by the manual client.
- the checking whether the manual client is in an abnormal state according to the audit result of the preset number of error samples by the manual client includes:
- the first ratio is less than a preset threshold, it is determined that the manual client is in an abnormal state.
- the manual client can modify the incorrect labeling result of the wrong sample, it can be considered that the manual client can correctly label the wrong sample. If the proportion of incorrect samples whose annotation results are modified by the manual client is greater than or equal to the preset threshold, it can be considered that the artificial client has no abnormality. On the contrary, if the proportion of the incorrect samples whose annotation results are modified by the manual client is less than The preset threshold means that there is an abnormality in the manual client. Furthermore, it is possible to infer the labeling accuracy rate of the manual client for the entire sample set to be processed by the proportion of the samples whose labeling results are modified by the manual client in the wrong samples, for example, the first proportion is used as the labeling accuracy rate of the manual client .
- Method 1 For the sample set to be processed after being reviewed by the manual client, first obtain each error sample, and then determine the number of samples for which the manual client has modified the annotation result, and then obtain the error sample The manual client modifies the proportion of samples with annotated results;
- Method 2 For the sample set to be processed after being reviewed by the manual client, it is possible to first determine which samples the manual client has modified the annotation results of, and then count which of these samples belong to the wrong sample, and then obtain the error The proportion of samples in which the annotation result is modified by the manual client.
- the checking whether the manual client is in an abnormal state according to the review result of the preset number of error samples by the manual client includes:
- the proportion of the wrong samples that are modified by the manual client to the correct labeled result is greater than or equal to the preset threshold, it can be considered that the manual client has no abnormality.
- the wrong sample is modified by the manual client
- the proportion of samples for correct labeling of results is less than the preset threshold, which indicates that the manual client has an abnormality.
- the second proportion is regarded as the correct labeling accuracy of the manual client rate.
- the proportion of the wrong samples modified by the artificial client to the correct labeling result it is used to judge whether the artificial client has an abnormality and the labeling accuracy rate used to characterize the artificial client is more accurate than the previous implementation.
- Method 1 For the sample set to be processed after being reviewed by the manual client, first obtain each error sample, and then determine the number of samples in which the manual client has modified the labeling result to the correct labeling result, and then obtain the wrong sample The manual client is modified to the proportion of samples with correct annotation results;
- Method 2 For the to-be-processed sample set reviewed by the manual client, it is first possible to determine which sample labeling results the manual client has modified to the correct labeling results, and then count which of these samples belong to the wrong samples, and then obtain The proportion of the wrong samples that are modified by the manual client to correctly label the results.
- the first ratio is less than the preset threshold, or the second ratio is less than the preset threshold, it can be determined that the manual client is in an abnormal state, and it also means that the labeling accuracy of the manual client does not reach the expected value, so it can Make corrections to make the labeling accuracy meet the requirements.
- the preset threshold is a preset target audit accuracy rate of the manual client terminal, for example, the preset threshold is set to 90%. That is, if a label set with an accuracy rate of at least 90% is required, the labeling accuracy rate of the mixed error samples after being reviewed by the manual client must also reach at least 90%.
- a preset number of error samples are added to a test sample set to form a sample set to be processed, and the sample set to be processed is sent to the manual client for review.
- the audit results can infer the labeling accuracy rate of the manual client for the entire sample set to be processed, without the need to count the manual client’s labeling of the entire sample set to be processed to determine the labeling accuracy, so as to quickly determine whether the manual client is It is in an abnormal state, and the statistical time is shortened, and the cost is reduced.
- an embodiment of the present invention also provides a device for checking the status of a manual client using error samples.
- FIG. 2 is a method provided by an embodiment of the present invention. The schematic diagram of the structure of the device for checking the status of the manual client with the error sample. Please refer to Figure 2.
- An apparatus for checking the status of a manual client using error samples may include:
- the obtaining module 201 is configured to obtain a preset number of error samples, and add them to a test sample set to form a sample set to be processed; wherein the error samples are pre-marked with the error-marking result, and the samples in the test sample set are pre-marked with Mark the result;
- the review module 202 is configured to send the sample set to be processed to a manual client, so that the manual client can review the annotation results of each sample in the sample set to be processed;
- the checking module 203 is configured to check whether the manual client is in an abnormal state according to the audit result of the preset number of error samples by the manual client.
- a preset number of error samples are added to a test sample set to form a sample set to be processed, and the sample set to be processed is sent to a manual client for review.
- the manual client's labeling accuracy of the entire sample set to be processed does not need to count the manual client's labeling of the entire sample set to be processed to determine the labeling accuracy, so as to quickly determine whether the manual client is in an abnormal state, and It shortens the statistical time and reduces the cost.
- the manual client in the review module 202 reviews the annotation results of each sample in the sample set to be processed, including:
- the human client judges whether the marked annotation result is correct; if not, the annotation result marked by the sample is modified.
- the checking module 203 is specifically configured to:
- the device further includes:
- the first determining module is configured to determine the labeling accuracy rate of the manual client according to the first ratio.
- the checking module 203 is specifically configured to:
- the device further includes:
- the second determining module is configured to determine the labeling accuracy rate of the manual client according to the second ratio.
- the preset number is greater than or equal to the minimum sample extraction number N;
- N Z2 ⁇ (P ⁇ (1-P))/E2;
- Z represents the statistics related to the confidence level, which is the probability that the audit accuracy rate of the manual client falls within the preset confidence interval;
- E represents the preset sampling error value;
- P represents the accuracy of sample labeling in the test sample set.
- the preset threshold is a preset target audit accuracy rate of the manual client terminal.
- the acquiring module acquires a preset number of error samples, specifically:
- a preset number of error samples are obtained from an error sample library, where the error sample library includes samples marked with an incorrectly labeled result.
- the device further includes:
- the generating module is used to extract the features of the error samples in the error sample library, generate new error samples according to the extracted features, and add the generated new error samples to the error sample library.
- FIG. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
- an electronic device includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304.
- the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.
- the memory 303 is used to store computer programs
- the processor 301 is configured to implement the following steps when executing the program stored in the memory 303:
- the processor 301 executes the program stored in the memory 303 and implements other implementations of the method of checking the status of the manual client using error samples, which are the same as the implementations mentioned in the foregoing method embodiments, and will not be repeated here. .
- a preset number of error samples are added to a test sample set to form a sample set to be processed, and the sample set to be processed is sent to a manual client for review.
- the manual client's labeling accuracy of the entire sample set to be processed does not need to count the manual client's labeling of the entire sample set to be processed to determine the labeling accuracy, so as to quickly determine whether the manual client is in an abnormal state, and It shortens the statistical time and reduces the cost.
- the communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus.
- PCI Peripheral Component Interconnect
- EISA Extended Industry Standard Architecture
- the communication bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
- the communication interface is used for communication between the aforementioned electronic device and other devices.
- the memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk storage.
- NVM non-Volatile Memory
- the memory may also be at least one storage device located far away from the foregoing processor.
- the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (DSP), a dedicated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
- CPU central processing unit
- NP Network Processor
- DSP digital signal processor
- ASIC Application Specific Integrated Circuit
- FPGA Field-Programmable Gate Array
- An embodiment of the present invention also provides a computer-readable storage medium in which a computer program is stored.
- the computer program is executed by a processor, the method for checking the status of a manual client by using error samples is implemented. step.
- a preset number of error samples are added to a test sample set to form a sample set to be processed, and the sample set to be processed is sent to a manual client for review.
- the manual client's labeling accuracy of the entire sample set to be processed does not need to count the manual client's labeling of the entire sample set to be processed to determine the labeling accuracy, so as to quickly determine whether the manual client is in an abnormal state, and It shortens the statistical time and reduces the cost.
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Automatic Analysis And Handling Materials Therefor (AREA)
Abstract
一种利用错误样本检查人工客户端状态的方法及装置，方法包括：获取预设数量个错误样本，并加入一测试样本集中，组成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果（S101）；将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中的各个样本的标注结果进行审核（S102）；根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态（S103）。该方法可以快速判断人工客户端是否处于异常状态。
Description
本发明涉及人工智能技术领域，尤其涉及一种利用错误样本检查人工客户端状态的方法、装置、电子设备和计算机可读存储介质。
在人工智能领域的模型训练中需要人工客户端来对样本进行标注，或者对样本的标注结果进行审核，如果在进行样本标注或审核时人工客户端处于异常状态，则无法保证样本的标注情况达标，从而导致训练得到的模型的识别准确率不达标。因此，需要对人工客户端的状态进行检查，确保人工客户端在进行样本标注或审核时处于正常状态，从而使训练得到的模型的准确率达标。
目前，可以通过检查人工客户端所审核的全部样本的标注结果，根据审核结果判断人工客户端是否存在异常，然而通常样本集中样本的数量非常大，这样就需要花费较多时间才能判断人工客户端是否异常。
发明内容
本发明的目的在于提供一种利用错误样本检查人工客户端状态的方法、装置、电子设备和计算机可读存储介质，以快速确定人工客户端的标注准确率。具体技术方案如下：
第一方面，本发明提供了一种利用错误样本检查人工客户端状态的方法，所述方法包括：
获取预设数量个错误样本，并加入一测试样本集中，组成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果；
将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核；
根据所述人工客户端对所述预设数量个错误样本的审核结果，判定所述 人工客户端是否处于异常状态。
可选的，所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核，包括：
针对所述待处理样本集中每一样本，所述人工客户端判断所标注的标注结果是否正确；如果否，则对该样本所标注的标注结果进行修改。
可选的，根据所述人工客户端对所述预设数量个错误样本的审核结果，判定所述人工客户端是否处于异常状态，包括：
针对所述预设数量个错误样本中的每一样本，判断所述人工客户端是否对该样本的标注结果进行了修改；
获取所述预设数量个错误样本中被修改了标注结果的错误样本的比例，作为第一比例；
若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述方法还包括：
根据所述第一比例确定所述人工客户端的标注准确率。
可选的，根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态，包括：
针对所述预设数量个错误样本中的每一错误样本，判断所述人工客户端是否将该错误样本的错误标注结果修改为正确标注结果；
获取所述预设数量个错误样本中被所述人工客户端修改为正确标注结果的错误样本的比例，作为第二比例；
若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述方法还包括：
根据所述第二比例确定所述人工客户端的标注准确率。
可选的，所述预设数量大于等于最小样本抽取数量N；
其中，N＝Z2×(P×(1-P))/E2；Z表示与置信度相关的统计量，置信度为所述人工客户端的审核准确率落入预设的置信区间的概率；E表示预设的抽样误差值；P表示所述测试样本集中样本标注的准确率。
可选的，所述预设阈值为预先设置的所述人工客户端的目标审核准确率。
可选的，所述获取预设数量个错误样本，包括：
从错误样本库中获取预设数量个错误样本，其中，所述错误样本库包含标注有错误标注结果的错误样本。
可选的，所述方法还包括：
提取所述错误样本库中错误样本的特征，根据所提取的特征生成新的错误样本，并将生成的新的错误样本加入所述错误样本库中。
第二方面，本发明还提供了一种利用错误样本检查人工客户端状态的装置，所述装置包括：
获取模块，用于获取预设数量个错误样本，并加入一测试样本集中，组成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果；
审核模块，用于将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核；
检查模块，用于根据所述人工客户端对所述预设数量个错误样本的审核结果，判定所述人工客户端是否处于异常状态。
可选的，所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核，包括：
针对所述待处理样本集中每一样本，所述人工客户端判断所标注的标注结果是否正确；如果否，则对该样本所标注的标注结果进行修改。
可选的，所述检查模块，用于：
针对所述预设数量个错误样本中的每一样本，判断所述人工客户端是否对该样本的标注结果进行了修改；获取所述预设数量个错误样本中被修改了标注结果的错误样本的比例，作为第一比例；若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第一确定模块，用于根据所述第一比例确定所述人工客户端的标注准确率。
可选的，所述检查模块，用于：
针对所述预设数量个错误样本中的每一错误样本，判断所述人工客户端是否将该错误样本的错误标注结果修改为正确标注结果；获取所述预设数量个错误样本中被所述人工客户端修改为正确标注结果的错误样本的比例，作为第二比例；若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第二确定模块，用于根据所述第二比例确定所述人工客户端的标注准确率。
可选的，所述预设数量大于等于最小样本抽取数量N；
其中，N＝Z
2×(P×(1-P))/E
2；Z表示与置信度相关的统计量，置信度为所述人工客户端的审核准确率落入预设的置信区间的概率；E表示预设的抽样误差值；P表示所述测试样本集中样本标注的准确率。
可选的，所述预设阈值为预先设置的所述人工客户端的目标审核准确率。
可选的，所述获取模块获取预设数量个错误样本，包括：
从错误样本库中获取预设数量个错误样本，其中，所述错误样本库包含标注有错误标注结果的错误样本。
可选的，所述装置还包括：
生成模块，用于提取所述错误样本库中错误样本的特征，根据所提取的特征生成新的错误样本，并将生成的新的错误样本加入所述错误样本库中。
第三方面，本发明还提供了一种电子设备，包括处理器、通信接口、存储器和通信总线，其中，所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；
所述存储器，用于存放计算机程序；
所述处理器，用于执行所述存储器上所存放的所述计算机程序时，实现上述第一方面所述的利用错误样本检查人工客户端状态的方法。
第四方面，本发明还提供了一种计算机可读存储介质，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现上述第一方面述所述的利用错误样本检查人工客户端状态的方法。
与现有技术相比，本发明将预设数量个错误样本加入一测试样本集组成待处理样本集，将待处理样本集发给人工客户端进行审核，这样只需要检查人工客户端对错误样本的审核结果即可推断人工客户端对整个待处理样本集的标注准确率，而不需要统计人工客户端对整个待处理样本集的标注情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本发明一实施例提供的利用错误样本检查人工客户端状态的方法的流程示意图；
图2是本发明一实施例提供的利用错误样本检查人工客户端状态的装置的结构示意图；
图3是本发明一实施例提供的电子设备的结构示意图。
以下结合附图和具体实施例对本发明提出的一种利用错误样本检查人工客户端状态的方法、装置、电子设备及计算机可读存储介质作进一步详细说明。根据权利要求书和下面说明，本发明的优点和特征将更清楚。需说明的是，附图均采用非常简化的形式且均使用非精准的比例，仅用以方便、明晰地辅助说明本发明实施例的目的。
为解决现有技术的问题，本发明实施例提供了一种利用错误样本检查人工客户端状态的方法、装置、电子设备及计算机可读存储介质。
需要说明的是，本发明实施例的利用错误样本检查人工客户端状态的方法可应用于本发明实施例的利用错误样本检查人工客户端状态的装置，该利 用错误样本检查人工客户端状态的装置可被配置于电子设备上。其中，该电子设备可以是个人计算机、移动终端等，该移动终端可以是手机、平板电脑等具有各种操作系统的硬件设备。
图1是本发明一实施例提供的一种利用错误样本检查人工客户端状态的方法的流程示意图。请参考图1，一种利用错误样本检查人工客户端状态的方法可以包括如下步骤：
步骤S101，获取预设数量个错误样本，并加入一测试样本集中，组成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果。
本实施例对错误样本的类型不做限定，错误样本可以是票据图片、车辆图片、人脸图片、植物图片、试卷图片等。所述测试样本集中样本的类型需与所述错误样本的类型相同，例如，若所述测试样本集中样本是人脸图片，那么所获取的预设数量个错误样本也为人脸图片。错误样本标注有错误标注结果，即错误样本的标注结果与其真实的识别结果不一致或不同，例如一错误样本为人脸图像，图像的真实识别结果为“性别：女”，但是被错误标注为“性别：男”。
所述测试样本集中各个样本也预先标注有标注结果，例如通过一识别模型对测试样本集中各个样本进行识别和标注，或者通过人工标注，本实施例对各个样本的标注结果是否正确不做限定。本实施例对测试样本集中的样本数量亦不做限定，但是为了使判断人工客户端是否异常的判断结果更加准确，样本数量通常应设置为较大的数值，例如样本数量为100、1000等。错误样本和测试样本集中样本的标注结果可以直接标注在样本图片上，也可以标注作为样本图片的属性信息。
实际应用中，所述错误样本可以为人工故意标错的样本，也可以通过获取标注完成的样本集，审核并过滤出其中标注错误的样本。进一步的，还可以维护一个错误样本库，将标注错误的样本加入其中，并根据样本类型进行分类存储。使用错误样本库，可以提高获取错误样本的效率，进而提高检查人工客户端是否异常的效率。
因此，所述获取预设数量个错误样本，可以为：从错误样本库中获取预设数量个错误样本，其中，所述错误样本库包含标注有错误标注结果的样本。
由于本发明的核心思想是通过统计人工客户端对预设数量个错误样本的错误标注结果的审核情况，来推断人工客户端对整个待处理样本集中样本的审核情况，进而判断人工客户端是否处于异常状态，因此，为了保证后续统计的准确性，对错误样本的数量可以有如下要求：
所述预设数量大于等于最小样本抽取数量N；
其中，N＝Z
2×(P×(1-P))/E
2；Z表示与置信度相关的统计量，置信度为所述人工客户端的审核准确率落入预设的置信区间的概率；E表示预设的抽样误差值；P表示所述测试样本集中样本标注的准确率。
Z与置信度的对应关系如下：当置信度为90％时，Z＝1.64；置信度为95％时，Z＝1.96；置信度为95.45％时，Z＝2；置信度为99％时，Z＝2.68；置信度为99.73时，Z＝3；以上数据可以通过查询统计表得出。本实施例的置信区间为90％～99.99％，也就是说认为所述人工客户端的审核准确率应该在90％～99.99％的几率落入到P的范围内，本实施例可以采用95％的置信度。
本实施例中，抽样误差值E可以设定在±5％之间，P为概率值，可以设定其为90％。如果通过上述计算公式计算得到最小样本抽取数量N等于100，则所述预设数量可以设定大于等于100的任意数值。
进一步的，针对所述错误样本库，还可以提取所述错误样本库中错误样本的特征，根据所提取的特征生成新的错误样本，并将生成的新错误样本加入所述错误样本库中。例如，若某一错误样本为植物图片，其中的植物为月季，但是被错误标注为玫瑰，则可以提取错误样本中月季的特征，然后获取图片中植物特征与月季特征相匹配的其他样本，将其它样本中的植物也标注为玫瑰，从而生成新的错误样本。又如，某一错误样本为包含字符内容的图片，其中一个字符数字“0”被错误标注为字母“o”，则提取字符数字“0”的特征，将其它的数字“0”也标注为字母“o”，从而生成新的错误样本。自动归纳错误样本的特征，并自动生成更多的错误样本，可以进一步扩大错误样本库。
步骤S102，将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核。
所述人工客户端会对所述待处理样本集中各个样本的标注结果进行审核处理，审核处理包括：针对所述待处理样本集中每一样本，所述人工客户端判断所标注的标注结果是否正确；如果否，则对该样本所标注的标注结果进行修改。如果判定不正确，还可以对该样本所标注的识别结果进行修改。需要说明的是，由于待处理样本集中包含两类样本，即按照正常标注流程进行标注处理后的样本，和加入的标注有错误标注结果的错误样本，人工客户端在审核时，不会对这两类样本进行区分，而是将错误样本同样视为按照正常标注流程进行标注处理的样本。
举例而言，若某一样本的标注结果为女性，而人工客户端在审核后判定该样本的标注结果有误，且经过自身的识别后判定该样本的识别结果应该为男性，则可以对该样本所标注的标注结果修改为自身所判定的识别结果。
实际上，对于错误样本中的某一样本，人工客户端可能没有识别出该样本被标注错误，从而导致人工客户端将该样本的标注结果判定为正确。人工客户端对错误样本的审核情况反应了人工客户端对整个待处理样本集中样本的识别(标注)情况，进而通过检查人工客户端对这类样本的审核情况可以来推断人工客户端的标注准确率或审核准确率，并且判断人工客户端是否存在异常。
步骤S103，根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态。
在一种实现方式中，所述根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态，包括：
针对所述预设数量个错误样本中的每一样本，判断所述人工客户端是否对该样本的标注结果进行了修改；
获取所述预设数量个错误样本中被修改了标注结果的错误样本的比例，作为第一比例；
若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可以理解的是，通常来说，如果人工客户端能够对错误样本的错误标注结果进行修改，则可以认为人工客户端能够将该错误样本进行正确的标注。若错误样本中被人工客户端修改了标注结果的样本的比例大于等于预设阈值，可以认为人工客户端没有出现异常，反之，若错误样本中被人工客户端修改了标注结果的样本的比例小于预设阈值，则表示人工客户端出现了异常。进一步的，还可以通过错误样本中被人工客户端修改了标注结果的样本的比例，来推断人工客户端对整个待处理样本集的标注准确率，例如将第一比例作为人工客户端的标注准确率。
具体而言，确定所述预设数量个错误样本中被所述人工客户端修改了标注结果的样本的比例，可以有如下两种方式：
方式一：对于经人工客户端审核后的所述待处理样本集，首先获取各个错误样本，然后判断人工客户端对标注结果进行了修改的样本的数量，进而获取所述错误样本中被所述人工客户端修改了标注结果的样本的比例；
方式二：对于经人工客户端审核后的所述待处理样本集，首先可以判断人工客户端对哪些样本的标注结果进行了修改，然后统计出这些样本中哪些是属于错误样本的，进而获取错误样本中被所述人工客户端修改了标注结果的样本的比例。
在另一种实现方式中，所述根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态，包括：
针对所述预设数量个错误样本中的每一错误样本，判断所述人工客户端是否将该样本的错误标注结果修改为正确标注结果；
获取所述预设数量个错误样本中被所述人工客户端修改为正确标注结果的错误样本的比例，作为第二比例；
若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
在这一实现方式中，若错误样本中被人工客户端修改为正确标注结果的样本的比例大于等于预设阈值，可以认为人工客户端没有出现异常，反之，若错误样本中被人工客户端修改为正确标注结果的样本的比例小于预设阈值，则表示人工客户端出现了异常。进一步的，还可以通过错误样本中被人 工客户端修改为正确标注结果的样本的比例，来推断人工客户端对整个待处理样本集的标注准确率，例如将第二比例作为人工客户端的标注准确率。
根据错误样本中被人工客户端修改为正确标注结果的样本的比例，用来判断人工客户端是否存在异常，以及用来表征人工客户端的标注准确率，相比于上一种实现方式更加准确。
类似的，确定所述错误样本中被所述人工客户端修改为正确标注结果的样本的比例，可以有如下两种方式：
方式一：对于经人工客户端审核后的所述待处理样本集，首先获取各个错误样本，然后判断人工客户端将标注结果修改为正确标注结果的样本的数量，进而获取所述错误样本中被所述人工客户端修改为正确标注结果的样本的比例；
方式二：对于经人工客户端审核后的所述待处理样本集，首先可以判断人工客户端将哪些样本标注结果修改为正确标注结果，然后统计出这些样本中哪些是属于错误样本的，进而获取错误样本中被所述人工客户端修改为正确标注结果的样本的比例。
当判断出第一比例小于预设阈值，或第二比例小于预设阈值时，可以判定人工客户端处于异常状态，同时也表示人工客户端的标注准确率达不到期望值，因此可以对人工客户端进行修正，以使其标注准确率符合要求。
其中，所述预设阈值为预先设置的所述人工客户端的目标审核准确率，例如，设置预设阈值为90％。即如果需要一个达到至少90％准确率的标注集，那么混入的错误样本经所述人工客户端审核后的标注准确率也至少要达到90％。
综上所述，本实施例将预设数量个错误样本加入一测试样本集组成待处理样本集，将待处理样本集发给人工客户端进行审核，这样只需要检查人工客户端对错误样本的审核结果即可推断人工客户端对整个待处理样本集的标注准确率，而不需要统计人工客户端对整个待处理样本集的标注情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
相应于上述利用错误样本检查人工客户端状态的方法实施例，本发明一实施例还提供了一种利用错误样本检查人工客户端状态的装置，图2是本发明一实施例提供的一种利用错误样本检查人工客户端状态的装置的结构示意图。请参考图2，一种利用错误样本检查人工客户端状态的装置可以包括：
获取模块201，用于获取预设数量个错误样本，并加入一测试样本集中，组成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果；
审核模块202，用于将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核；
检查模块203，用于根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态。
本实施例将预设数量个错误样本加入一测试样本集组成待处理样本集，将待处理样本集发给人工客户端进行审核，这样只需要检查人工客户端对错误样本的审核结果即可推断人工客户端对整个待处理样本集的标注准确率，而不需要统计人工客户端对整个待处理样本集的标注情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
可选的，所述审核模块202中人工客户端对所述待处理样本集中各个样本的标注结果进行审核，包括：
针对所述待处理样本集中每一样本，所述人工客户端判断所标注的标注结果是否正确；如果否，则对该样本所标注的标注结果进行修改。
可选的，所述检查模块203，具体用于：
针对所述预设数量个错误样本中的每一样本，判断所述人工客户端是否对该样本的标注结果进行了修改；获取所述预设数量个错误样本中被修改了标注结果的错误样本的比例，作为第一比例；若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第一确定模块，用于根据所述第一比例确定所述人工客户端的标注准确率。
可选的，所述检查模块203，具体用于：
针对所述预设数量个错误样本中的每一错误样本，判断所述人工客户端是否将该样本的错误标注结果修改为正确标注结果；获取所述预设数量个错误样本中被所述人工客户端修改为正确标注结果的错误样本的比例，作为第二比例；若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第二确定模块，用于根据所述第二比例确定所述人工客户端的标注准确率。
可选的，所述预设数量大于等于最小样本抽取数量N；
其中，N＝Z2×(P×(1-P))/E2；Z表示与置信度相关的统计量，置信度为所述人工客户端的审核准确率落入预设的置信区间的概率；E表示预设的抽样误差值；P表示所述测试样本集中样本标注的准确率。
可选的，所述预设阈值为预先设置的所述人工客户端的目标审核准确率。
可选的，所述获取模块获取预设数量个错误样本，具体为：
从错误样本库中获取预设数量个错误样本，其中，所述错误样本库包含标注有错误标注结果的样本。
可选的，所述装置还包括：
生成模块，用于提取所述错误样本库中错误样本的特征，根据所提取的特征生成新的错误样本，并将生成的新错误样本加入所述错误样本库中。
本发明一实施例还提供了一种电子设备，图3是本发明一实施例提供的一种电子设备的结构示意图。请参考图3，一种电子设备包括处理器301、通信接口302、存储器303和通信总线304，其中，处理器301，通信接口302，存储器303通过通信总线304完成相互间的通信，
存储器303，用于存放计算机程序；
处理器301，用于执行存储器303上所存放的程序时，实现如下步骤：
获取预设数量个错误样本，并加入一测试样本集中，组成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果；
将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核；
根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态。
关于该方法各个步骤的具体实现以及相关解释内容可以参见上述图1所示的方法实施例，在此不做赘述。
另外，处理器301执行存储器303上所存放的程序而实现的利用错误样本检查人工客户端状态的方法的其他实现方式，与前述方法实施例部分所提及的实现方式相同，这里也不再赘述。
本实施例将预设数量个错误样本加入一测试样本集组成待处理样本集，将待处理样本集发给人工客户端进行审核，这样只需要检查人工客户端对错误样本的审核结果即可推断人工客户端对整个待处理样本集的标注准确率，而不需要统计人工客户端对整个待处理样本集的标注情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
上述电子设备提到的通信总线可以是外设部件互连标准(Peripheral Component Interconnect，PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，EISA)总线等。该通信总线可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
通信接口用于上述电子设备与其他设备之间的通信。
存储器可以包括随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器(Non-Volatile Memory，NVM)，例如至少一个磁盘存储器。可选的，存储器还可以是至少一个位于远离前述处理器的存储装置。
上述的处理器可以是通用处理器，包括中央处理器(Central Processing Unit，CPU)、网络处理器(Network Processor，NP)等；还可以是数字信号处理器(Digital Signal Processing，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。
本发明一实施例还提供了一种计算机可读存储介质，该计算机可读存储介质内存储有计算机程序，该计算机程序被处理器执行时实现上述的利用错误样本检查人工客户端状态的方法的步骤。
本实施例将预设数量个错误样本加入一测试样本集组成待处理样本集，将待处理样本集发给人工客户端进行审核，这样只需要检查人工客户端对错误样本的审核结果即可推断人工客户端对整个待处理样本集的标注准确率，而不需要统计人工客户端对整个待处理样本集的标注情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
需要说明的是，本说明书中的各个实施例均采用相关的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于装置、电子设备、计算机可读存储介质实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括 所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
上述描述仅是对本发明较佳实施例的描述，并非对本发明范围的任何限定，本发明领域的普通技术人员根据上述揭示内容做的任何变更、修饰，均属于权利要求书的保护范围。
Claims (20)
- 一种利用错误样本检查人工客户端状态的方法，其特征在于，所述方法包括：获取预设数量个错误样本，并加入一测试样本集中，组成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果；将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核；根据所述人工客户端对所述预设数量个错误样本的审核结果，判定所述人工客户端是否处于异常状态。
- 如权利要求1所述的利用错误样本检查人工客户端状态的方法，其特征在于，所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核，包括：针对所述待处理样本集中每一样本，所述人工客户端判断所标注的标注结果是否正确；如果否，则对该样本所标注的标注结果进行修改。
- 如权利要求2所述的利用错误样本检查人工客户端状态的方法，其特征在于，根据所述人工客户端对所述预设数量个错误样本的审核结果，判定所述人工客户端是否处于异常状态，包括：针对所述预设数量个错误样本中的每一样本，判断所述人工客户端是否对该样本的标注结果进行了修改；获取所述预设数量个错误样本中被修改了标注结果的错误样本的比例，作为第一比例；若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
- 如权利要求3所述的利用错误样本检查人工客户端状态的方法，其特征在于，所述方法还包括：根据所述第一比例确定所述人工客户端的标注准确率。
- 如权利要求2所述的利用错误样本检查人工客户端状态的方法，其特 征在于，根据所述人工客户端对所述预设数量个错误样本的审核结果，检查所述人工客户端是否处于异常状态，包括：针对所述预设数量个错误样本中的每一错误样本，判断所述人工客户端是否将该错误样本的错误标注结果修改为正确标注结果；获取所述预设数量个错误样本中被所述人工客户端修改为正确标注结果的错误样本的比例，作为第二比例；若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
- 如权利要求5所述的利用错误样本检查人工客户端状态的方法，其特征在于，所述方法还包括：根据所述第二比例确定所述人工客户端的标注准确率。
- 如权利要求1所述的利用错误样本检查人工客户端状态的方法，其特征在于，所述预设数量大于等于最小样本抽取数量N；其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度为所述人工客户端的审核准确率落入预设的置信区间的概率；E表示预设的抽样误差值；P表示所述测试样本集中样本标注的准确率。
- 如权利要求3或5所述的利用错误样本检查人工客户端状态的方法，其特征在于，所述预设阈值为预先设置的所述人工客户端的目标审核准确率。
- 如权利要求1-7中任一项所述的利用错误样本检查人工客户端状态的方法，其特征在于，所述获取预设数量个错误样本，包括：从错误样本库中获取预设数量个错误样本，其中，所述错误样本库包含标注有错误标注结果的错误样本。
- 如权利要求9所述的利用错误样本检查人工客户端状态的方法，其特征在于，所述方法还包括：提取所述错误样本库中错误样本的特征，根据所提取的特征生成新的错误样本，并将生成的新的错误样本加入所述错误样本库中。
- 一种利用错误样本检查人工客户端状态的装置，其特征在于，所述装置包括：获取模块，用于获取预设数量个错误样本，并加入一测试样本集中，组 成待处理样本集；其中，所述错误样本预先标注有错误标注结果，所述测试样本集中的样本预先标注有标注结果；审核模块，用于将所述待处理样本集发送给人工客户端，以便所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核；检查模块，用于根据所述人工客户端对所述预设数量个错误样本的审核结果，判定所述人工客户端是否处于异常状态。
- 如权利要求11所述的利用错误样本检查人工客户端状态的装置，其特征在于，所述人工客户端对所述待处理样本集中各个样本的标注结果进行审核，包括：针对所述待处理样本集中每一样本，所述人工客户端判断所标注的标注结果是否正确；如果否，则对该样本所标注的标注结果进行修改。
- 如权利要求12所述的利用错误样本检查人工客户端状态的装置，其特征在于，所述检查模块，用于：针对所述预设数量个错误样本中的每一样本，判断所述人工客户端是否对该样本的标注结果进行了修改；获取所述预设数量个错误样本中被修改了标注结果的错误样本的比例，作为第一比例；若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
- 如权利要求12所述的利用错误样本检查人工客户端状态的装置，其特征在于，所述检查模块，用于：针对所述预设数量个错误样本中的每一错误样本，判断所述人工客户端是否将该错误样本的错误标注结果修改为正确标注结果；获取所述预设数量个错误样本中被所述人工客户端修改为正确标注结果的错误样本的比例，作为第二比例；若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
- 如权利要求11所述的利用错误样本检查人工客户端状态的装置，其特征在于，所述预设数量大于等于最小样本抽取数量N；其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度为所述人工客户端的审核准确率落入预设的置信区间的概率；E表示预设的抽样误 差值；P表示所述测试样本集中样本标注的准确率。
- 如权利要求13或14所述的利用错误样本检查人工客户端状态的装置，其特征在于，所述预设阈值为预先设置的所述人工客户端的目标审核准确率。
- 如权利要求11-15中任一项所述的利用错误样本检查人工客户端状态的装置，其特征在于，所述获取模块获取预设数量个错误样本，包括：从错误样本库中获取预设数量个错误样本，其中，所述错误样本库包含标注有错误标注结果的错误样本。
- 如权利要求17所述的利用错误样本检查人工客户端状态的装置，其特征在于，所述装置还包括：生成模块，用于提取所述错误样本库中错误样本的特征，根据所提取的特征生成新的错误样本，并将生成的新的错误样本加入所述错误样本库中。
- 一种电子设备，其特征在于，包括处理器、通信接口、存储器和通信总线，其中，所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；所述存储器，用于存放计算机程序；所述处理器，用于执行所述存储器上所存放的所述计算机程序时，实现权利要求1-10中任一所述的方法。
- 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1-10中任一项所述的方法。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910538145.XA CN110263853B (zh) | 2019-06-20 | 2019-06-20 | 利用错误样本检查人工客户端状态的方法及装置 |
| CN201910538145.X | 2019-06-20 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2020253741A1 true WO2020253741A1 (zh) | 2020-12-24 |
Family
ID=67919988
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2020/096646 WO2020253741A1 (zh) | 2019-06-20 | 2020-06-17 | 利用错误样本检查人工客户端状态的方法及装置 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110263853B (zh) |
| WO (1) | WO2020253741A1 (zh) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112992299A (zh) * | 2021-02-10 | 2021-06-18 | 北京声智科技有限公司 | 信息处理方法、信息处理装置、电子设备和存储介质 |
| CN113691804A (zh) * | 2021-09-15 | 2021-11-23 | 昆山丘钛光电科技有限公司 | 一种检测功能验证方法、装置、设备和介质 |
| CN113793134A (zh) * | 2021-09-26 | 2021-12-14 | 上汽通用五菱汽车股份有限公司 | 一种车辆报警方法、装置和计算机可读存储介质 |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110263853B (zh) * | 2019-06-20 | 2023-05-23 | 杭州睿琪软件有限公司 | 利用错误样本检查人工客户端状态的方法及装置 |
| CN116307948A (zh) * | 2023-05-23 | 2023-06-23 | 飞狐信息技术（天津）有限公司 | 审核质量检测方法、装置、设备及存储介质 |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107563404A (zh) * | 2017-07-18 | 2018-01-09 | 北京工业大学 | 一种交通标志分类方法及装置 |
| WO2018119684A1 (zh) * | 2016-12-27 | 2018-07-05 | 深圳前海达闼云端智能科技有限公司 | 一种图像识别系统及图像识别方法 |
| CN108537269A (zh) * | 2018-04-04 | 2018-09-14 | 中山大学 | 一种弱交互式的物体检测深度学习方法及其系统 |
| CN108875769A (zh) * | 2018-01-23 | 2018-11-23 | 北京迈格威科技有限公司 | 数据标注方法、装置和系统及存储介质 |
| CN109784381A (zh) * | 2018-12-27 | 2019-05-21 | 广州华多网络科技有限公司 | 标注信息处理方法、装置及电子设备 |
| CN110263853A (zh) * | 2019-06-20 | 2019-09-20 | 杭州睿琪软件有限公司 | 利用错误样本检查人工客户端状态的方法及装置 |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7061510B2 (en) * | 2001-03-05 | 2006-06-13 | Digimarc Corporation | Geo-referencing of aerial imagery using embedded image identifiers and cross-referenced data sets |
| CN106294094B (zh) * | 2015-05-13 | 2020-01-31 | 腾讯科技(成都)有限公司 | 游戏服务器的测试方法、客户端、服务器及系统 |
| CN105975980B (zh) * | 2016-04-27 | 2019-04-05 | 百度在线网络技术（北京）有限公司 | 监控图像标注质量的方法和装置 |
| CN107346336B (zh) * | 2017-06-29 | 2021-06-08 | 北京百度网讯科技有限公司 | 基于人工智能的信息处理方法和装置 |
| CN108875775A (zh) * | 2018-04-27 | 2018-11-23 | 淘然视界(杭州)科技有限公司 | 一种应用于数据标注领域的评估系统及其方法 |
| CN108932724B (zh) * | 2018-05-31 | 2020-06-19 | 杭州晓图科技有限公司 | 一种基于多人协同图像标注的系统自动审核方法 |
| CN109522146A (zh) * | 2018-11-15 | 2019-03-26 | 腾讯音乐娱乐科技（深圳）有限公司 | 对客户端进行异常测试的方法、装置及存储介质 |
| CN109684947B (zh) * | 2018-12-11 | 2021-03-30 | 广州景骐科技有限公司 | 标注质量监控方法、装置、计算机设备和存储介质 |
| CN109784382A (zh) * | 2018-12-27 | 2019-05-21 | 广州华多网络科技有限公司 | 标注信息处理方法、装置以及服务器 |
-
2019
- 2019-06-20 CN CN201910538145.XA patent/CN110263853B/zh active Active
-
2020
- 2020-06-17 WO PCT/CN2020/096646 patent/WO2020253741A1/zh active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018119684A1 (zh) * | 2016-12-27 | 2018-07-05 | 深圳前海达闼云端智能科技有限公司 | 一种图像识别系统及图像识别方法 |
| CN107563404A (zh) * | 2017-07-18 | 2018-01-09 | 北京工业大学 | 一种交通标志分类方法及装置 |
| CN108875769A (zh) * | 2018-01-23 | 2018-11-23 | 北京迈格威科技有限公司 | 数据标注方法、装置和系统及存储介质 |
| CN108537269A (zh) * | 2018-04-04 | 2018-09-14 | 中山大学 | 一种弱交互式的物体检测深度学习方法及其系统 |
| CN109784381A (zh) * | 2018-12-27 | 2019-05-21 | 广州华多网络科技有限公司 | 标注信息处理方法、装置及电子设备 |
| CN110263853A (zh) * | 2019-06-20 | 2019-09-20 | 杭州睿琪软件有限公司 | 利用错误样本检查人工客户端状态的方法及装置 |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112992299A (zh) * | 2021-02-10 | 2021-06-18 | 北京声智科技有限公司 | 信息处理方法、信息处理装置、电子设备和存储介质 |
| CN112992299B (zh) * | 2021-02-10 | 2022-03-01 | 北京声智科技有限公司 | 信息处理方法、信息处理装置、电子设备和存储介质 |
| CN113691804A (zh) * | 2021-09-15 | 2021-11-23 | 昆山丘钛光电科技有限公司 | 一种检测功能验证方法、装置、设备和介质 |
| CN113793134A (zh) * | 2021-09-26 | 2021-12-14 | 上汽通用五菱汽车股份有限公司 | 一种车辆报警方法、装置和计算机可读存储介质 |
| CN113793134B (zh) * | 2021-09-26 | 2024-02-13 | 上汽通用五菱汽车股份有限公司 | 一种车辆报警方法、装置和计算机可读存储介质 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110263853B (zh) | 2023-05-23 |
| CN110263853A (zh) | 2019-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020253741A1 (zh) | 利用错误样本检查人工客户端状态的方法及装置 | |
| WO2020253742A1 (zh) | 样本标注审核方法及装置 | |
| WO2020253740A1 (zh) | 用于样本审核的人工客户端的状态检查方法及装置 | |
| WO2020253636A1 (zh) | 样本标注信息的审核方法及装置 | |
| CN110795482B (zh) | 数据对标方法、装置、及存储装置 | |
| WO2018157840A1 (zh) | 语音识别测试方法及测试终端、计算设备及存储介质 | |
| CN109473093B (zh) | 语音识别方法、装置、计算机设备及存储介质 | |
| WO2020087713A1 (zh) | 视频质检方法、装置、计算机设备及存储介质 | |
| US9836565B2 (en) | Electronic design automation method and apparatus thereof | |
| WO2020215543A1 (zh) | 数据辅助定位方法、装置、计算机设备及存储介质 | |
| US20200133962A1 (en) | Knowledge graph generating apparatus, method, and non-transitory computer readable storage medium thereof | |
| CN110990276A (zh) | 接口字段自动化测试方法、装置和存储介质 | |
| US20210192965A1 (en) | Question correction method, device, electronic equipment and storage medium for oral calculation questions | |
| CN110909361A (zh) | 一种漏洞检测方法，装置和计算机设备 | |
| CN110046086B (zh) | 用于测试的期望数据生成方法及装置和电子设备 | |
| WO2015067171A1 (zh) | Debug调试方法和装置 | |
| WO2021072872A1 (zh) | 基于字符转换的姓名存储方法、装置、计算机设备 | |
| CN117171030A (zh) | 软件运行环境检测方法、装置、设备及存储介质 | |
| CN111277465A (zh) | 一种异常数据报文检测方法、装置及电子设备 | |
| CN111399832B (zh) | 一种页面编辑方法及装置 | |
| US11482027B2 (en) | Automated extraction of performance segments and metadata values associated with the performance segments from contract documents | |
| CN110532186B (zh) | 利用验证码进行测试的方法、装置、电子设备和存储介质 | |
| CN113961672A (zh) | 信息标注方法、装置、电子设备和存储介质 | |
| CN112580334A (zh) | 一种文案处理方法、装置、服务器及存储介质 | |
| CN109165127B (zh) | 问题接口的定位方法、装置及电子设备 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20826560 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 20826560 Country of ref document: EP Kind code of ref document: A1 |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 20826560 Country of ref document: EP Kind code of ref document: A1 |