WO2020253740A1 - 用于样本审核的人工客户端的状态检查方法及装置 - Google Patents

用于样本审核的人工客户端的状态检查方法及装置 Download PDF

Info

Publication number
WO2020253740A1
WO2020253740A1 PCT/CN2020/096645 CN2020096645W WO2020253740A1 WO 2020253740 A1 WO2020253740 A1 WO 2020253740A1 CN 2020096645 W CN2020096645 W CN 2020096645W WO 2020253740 A1 WO2020253740 A1 WO 2020253740A1
Authority
WO
WIPO (PCT)
Prior art keywords
sample
sample set
manual client
client
recognition result
Prior art date
Application number
PCT/CN2020/096645
Other languages
English (en)
French (fr)
Inventor
徐青松
李青
Original Assignee
杭州睿琪软件有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州睿琪软件有限公司 filed Critical 杭州睿琪软件有限公司
Publication of WO2020253740A1 publication Critical patent/WO2020253740A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3692—Test management for test results analysis
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Definitions

  • the present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic equipment and computer-readable storage medium for checking the status of a manual client for sample review.
  • a manual client is usually used to review the labeling results of the sample. If the manual client is in an abnormal state during the sample review, the labeling of the sample cannot be guaranteed to meet the standard, resulting in training The recognition accuracy of the obtained model is not up to standard. Therefore, in order to make the accuracy of the trained model reach the standard, the manual client is required to be in a normal state during the sample review, which requires the state of the manual client to be checked.
  • the purpose of the present invention is to provide a method, device, electronic equipment and computer-readable storage medium for checking the status of a manual client for sample review, so as to quickly determine whether the manual client is in an abnormal state.
  • the specific technical solutions are as follows:
  • the present invention provides a method for checking the status of a manual client for sample review, the method including:
  • the manual client terminal reviews the processed identification results of each sample in the test sample set, including:
  • the human client judges whether the marked recognition result is correct; if not, then modifies the marked recognition result of the sample.
  • determining whether the manual client is in an abnormal state according to the review result of the manual client on the target sample set in the processed test sample set includes:
  • the first ratio is less than a preset threshold, it is determined that the manual client is in an abnormal state.
  • the method further includes:
  • the marking accuracy rate of the manual client is determined according to the first ratio.
  • checking whether the manual client is in an abnormal state includes:
  • the method further includes:
  • Q represents the preset target accuracy rate of the sample labeling in the test sample set after the identification model labeling and the manual client review.
  • the preset number is greater than or equal to the minimum sample extraction number N;
  • N Z 2 ⁇ (P ⁇ (1-P))/E 2 ;
  • Z represents the statistics related to the confidence level, and the confidence level is equal to the recognition accuracy rate of the recognition model;
  • E represents the preset sampling error value ;
  • P represents the recognition accuracy of the recognition model.
  • modifying the recognition result marked on each sample in the target sample set to the wrong recognition result includes:
  • the present invention also provides a state checking device of a manual client for sample review, the device including:
  • the labeling module is used to obtain a test sample set, use a pre-trained recognition model to identify each sample in the test sample set, and label the recognition result of each sample;
  • the modification module is used to select a preset number of samples in the test sample set to form a target sample set, and modify the recognition result marked by each sample in the target sample set to an error recognition result to form the processed test sample set;
  • An audit module configured to send the processed test sample set to a manual client, so that the manual client can review the identification results of each sample in the processed test sample set;
  • the checking module is configured to determine whether the manual client is in an abnormal state according to the review result of the target sample set in the processed test sample set by the manual client.
  • the manual client terminal reviews the processed identification results of each sample in the test sample set, including:
  • the human client judges whether the marked recognition result is correct; if not, then modifies the marked recognition result of the sample.
  • the inspection module is used to:
  • the device further includes:
  • the first determining module is configured to determine the labeling accuracy rate of the manual client according to the first ratio.
  • the inspection module is used to:
  • the device further includes:
  • the second determining module is configured to determine the labeling accuracy rate of the manual client according to the second ratio.
  • Q represents the preset target accuracy rate of the sample labeling in the test sample set after the identification model labeling and the manual client review.
  • the preset number is greater than or equal to the minimum sample extraction number N;
  • N Z 2 ⁇ (P ⁇ (1-P))/E 2 ;
  • Z represents the statistics related to the confidence level, and the confidence level is equal to the recognition accuracy rate of the recognition model;
  • E represents the preset sampling error value ;
  • P represents the recognition accuracy of the recognition model.
  • the modification module modifies the recognition result marked on each sample in the target sample set to the wrong recognition result, including:
  • the present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete mutual communication through the communication bus.
  • the memory is used to store computer programs
  • the processor is configured to implement the method for checking the status of a manual client for sample review in the first aspect when executing the computer program stored in the memory.
  • the present invention also provides a computer-readable storage medium having a computer program stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program described in the first aspect The status check method of the manual client for sample review.
  • the present invention uses the recognition model to identify the samples in the test sample set, annotates the recognition results, and deliberately modifies the recognition results of a part of the samples in the test sample set to the wrong recognition results, and the processed test sample set Send it to the manual client for review, so that you only need to check the manual client’s review results of the part of the sample that was deliberately mislabeled to infer the accuracy of the manual client’s labeling of the entire test sample set, and then determine whether the manual client is abnormal Status, without the need to count the review of the entire test sample set by the manual client to determine the accuracy of the labeling, thereby quickly determining whether the manual client is in an abnormal state, shortening the statistical time and reducing the cost.
  • FIG. 1 is a schematic flowchart of a method for checking the status of a manual client for sample review according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of the structure of a state checking device of a manual client for sample review according to an embodiment of the present invention
  • FIG. 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.
  • embodiments of the present invention provide a method, device, electronic equipment, and computer-readable storage medium for checking the status of a manual client for sample review.
  • the method for checking the status of the manual client for sample review in the embodiment of the present invention can be applied to the device for checking the status of the manual client for sample review in the embodiment of the present invention.
  • the status checking device can be configured on the electronic device.
  • the electronic device may be a personal computer, a mobile terminal, etc.
  • the mobile terminal may be a hardware device with various operating systems such as a mobile phone or a tablet computer.
  • FIG. 1 is a schematic flowchart of a method for checking the status of a manual client for sample review according to an embodiment of the present invention. Please refer to Figure 1.
  • a method for checking the status of a manual client for sample review may include the following steps:
  • Step S101 Obtain a test sample set, use a pre-trained recognition model to identify each sample in the test sample set, and mark the recognition result of each sample.
  • the recognition model may be a neural network model established through sample training, and it may be a recognition model obtained through any type of sample training, and this embodiment does not limit the type of training samples.
  • the sample can be a bill picture, and a bill recognition model can be established after training, or it can be a vehicle picture, a face picture, a plant picture, a test paper picture, etc., different sample types can be used to establish different recognition models through sample training.
  • the recognition accuracy of the recognition model is determined.
  • the process of establishing the recognition model through sample training and the manner of determining the recognition accuracy of the recognition model can be referred to the prior art, and will not be repeated here.
  • the sample type in the test sample set needs to be the same as the sample type during the training of the recognition model. For example, if the recognition model is obtained through the training of the bill picture sample, then the obtained sample in the test sample set is the bill picture.
  • This embodiment does not limit the number of samples in the test sample set, but in order to make the judgment result of judging whether the manual client is abnormal, the number of samples should usually be set to a larger value, such as 100, 1000, etc.
  • Each sample in the test sample set is recognized by the recognition model, and the recognition result of the recognition model is labeled.
  • the recognition result can be marked on the sample picture, or it can be marked as the attribute information of the sample picture.
  • step S102 a preset number of samples in the test sample set are selected as the target sample set, and the recognition result marked by each sample in the target sample set is modified into an incorrect recognition result.
  • the recognition result marked by the sample is modified to the wrong recognition result, that is, the recognition result is inconsistent or different from the original recognition result marked.
  • the recognition model is trained based on a face image sample and is used to identify whether the face in the face image is male or female, the recognition model is based on a certain face image sample in the test sample set. If the recognition result is female, then the recognition result marked by the face image sample is female, and then the recognition result marked by the face image sample is modified as an incorrect recognition result, for example, modified as a male.
  • the recognition model is trained based on plant image samples and is used to recognize the types of plants in the plant images, and the recognition result of a plant image sample in the test sample set by the recognition model is peach blossom, then The recognition result marked by the plant image sample is peach blossom, then the recognition result marked by the plant image sample is modified to an inconsistent or different recognition result, such as pear blossom.
  • a preset number of samples may be randomly selected from the test sample set as the target sample set, and the recognition result marked by the target sample set may be modified to an incorrect recognition result.
  • the preset number is greater than or equal to the minimum sampling number N for sampling statistics
  • N Z 2 ⁇ (P ⁇ (1-P))/E 2 ;
  • Z represents the statistics related to the confidence level, and the confidence level is equal to the recognition accuracy rate of the recognition model;
  • E represents the preset sampling error value ;
  • P represents the recognition accuracy of the recognition model.
  • the confidence interval of this embodiment is 90%-99.99%. That is to say, it is believed that the recognition accuracy of the recognition model should fall within the range of P from 90%-99.99%.
  • This embodiment can use a 95% Confidence.
  • the sampling error value E can be set between ⁇ 5%
  • P is the probability value, which can be set to 90%, that is, the accuracy of the sample labeling in the test sample set after being labelled by the identification model Need to reach 90%.
  • the preset number can be set to any value greater than or equal to 100. It is also possible to extract a certain proportion of samples from the test sample set as the target sample set, as long as the number of the extracted target sample sets is greater than or equal to the minimum sample extraction number N.
  • Step S103 Send the processed test sample set to the manual client, so that the manual client can review the identification results of each sample.
  • the manual client will review the recognition results of the recognition model.
  • the review processing includes: for each sample in the processed test sample set, judging whether the marked recognition result is correct; if the judgment is incorrect, it can also The recognition results marked on the sample are modified. It should be noted that since the processed test sample set contains two types of samples, namely, the samples marked with the recognition results of the recognition model, and the samples selected and deliberately marked with the wrong recognition results, the manual client will not The two types of samples are distinguished, but the target sample set that is extracted and deliberately labeled with the wrong recognition result is also regarded as the sample labeled with the recognition result of the recognition model.
  • the recognition result marked by a sample is female, and the human client judges that the recognition result marked by the sample is wrong after the review, and it has been identified by itself After it is determined that the recognition result of the sample should be male, the recognition result marked on the sample can be modified to the recognition result determined by itself.
  • the human client may not recognize that the sample is incorrectly labeled, which causes the human client to judge the recognition result of the sample as correct.
  • the manual client’s review of the deliberately mislabeled target sample set reflects the manual client’s recognition (labeling) of the samples in the entire test sample set, and the manual client can be inferred by checking the manual client’s review of such samples. The labeling accuracy or audit accuracy of the terminal, and judge whether the manual client is abnormal.
  • Step S104 Check whether the manual client is in an abnormal state according to the review result of the target sample set by the manual client.
  • the checking whether the manual client is in an abnormal state according to the review result of the target sample set by the manual client includes:
  • the first ratio is less than a preset threshold, it is determined that the manual client is in an abnormal state.
  • a human client can modify the wrong recognition result of a sample that is deliberately incorrectly labeled, it can be considered that the human client can correctly label the sample that is deliberately incorrectly labeled. If the proportion of samples in the target sample set whose recognition result has been modified by the artificial client is greater than or equal to the preset threshold, it can be considered that the artificial client has no abnormality. Conversely, if the proportion of samples whose recognition result has been modified by the artificial client in the target sample set is less than The preset threshold means that there is an abnormality in the manual client.
  • the labeling accuracy rate of the entire test sample set by the manual client based on the proportion of the samples in the target sample set whose recognition results have been modified by the manual client.
  • the first proportion is used as the labeling accuracy rate of the manual client.
  • Method 1 For the test sample set reviewed by the manual client, first obtain the target sample set extracted and deliberately labeled with incorrect recognition results, and then determine the number of samples for which the manual client has modified the labeled recognition results , And then obtain the proportion of samples in the target sample set whose recognition result has been modified by the manual client;
  • Method 2 For the test sample set reviewed by the manual client, it is possible to first determine which samples the manual client has modified the recognition results marked, and then count which of these samples are extracted and marked with incorrect recognition The target sample set of the result, and then the proportion of the samples in the target sample set whose recognition result has been modified by the artificial client is obtained.
  • the checking whether the manual client is in an abnormal state according to the review result of the target sample set by the manual client includes:
  • the target sample set is modified by the manual client to the proportion of samples with correct recognition results greater than or equal to the preset threshold, it can be considered that there is no abnormality in the manual client.
  • the target sample set is modified by the manual client
  • the proportion of samples for correct identification results is less than the preset threshold, which means that the manual client has an abnormality.
  • the second proportion is used as the labeling accuracy rate of the manual client .
  • the proportion of samples in the target sample set that are modified by the manual client to the correct recognition result it is used to determine whether the manual client has an abnormality and the labeling accuracy rate used to characterize the manual client is more accurate than the previous implementation.
  • Method 1 For the test sample set reviewed by the manual client, first obtain the target sample set extracted and deliberately labeled with the wrong recognition result, and then judge the manual client to modify the labeled recognition result to the sample with the correct recognition result , And then obtain the proportion of samples in the target sample set that have been modified to correct recognition results by the manual client;
  • Method 2 For the test sample set that has been reviewed by the manual client, it can first determine which samples the manual client has modified the recognition results marked to the correct recognition results, and then count which of these samples are extracted and marked The target sample set of the wrong recognition result is obtained, and then the proportion of the samples in the target sample set modified to the correct recognition result by the manual client is obtained.
  • the first ratio is less than the preset threshold, or the second ratio is less than the preset threshold, it can be determined that the manual client is in an abnormal state, and it also means that the labeling accuracy of the manual client does not reach the expected value, so it can Make corrections to make the labeling accuracy meet the requirements.
  • the preset threshold can be set to any value equal to or greater than X, which is not limited in this embodiment.
  • the recognition model is used to identify the samples in the test sample set, the recognition results are marked, and the recognition results of some samples in the test sample set are deliberately modified to incorrect recognition results.
  • an embodiment of the present invention also provides a device for checking the status of a manual client for sample review.
  • FIG. 2 is an example provided by an embodiment of the present invention.
  • a state checking device of a manual client for sample review may include:
  • the labeling module 201 is configured to obtain a test sample set, use a pre-trained recognition model to identify each sample in the test sample set, and label the recognition result of each sample;
  • the modification module 202 is configured to select a preset number of samples in the test sample set as a target sample set, and modify the recognition result marked by each sample in the target sample set to an incorrect recognition result;
  • the review module 203 is configured to send the processed test sample set to the manual client, so that the manual client can review the identification results of each sample;
  • the checking module 204 is configured to check whether the manual client is in an abnormal state according to the review result of the target sample set by the manual client.
  • the manual client in the review module 203 reviews the identification results of each sample, including:
  • the human client judges whether the marked recognition result is correct; if not, then modifies the marked recognition result of the sample.
  • the checking module 204 is specifically configured to:
  • the device further includes:
  • the first determining module is configured to determine the labeling accuracy rate of the manual client according to the first ratio.
  • the checking module 204 is specifically configured to:
  • the device further includes:
  • the second determining module is configured to determine the labeling accuracy rate of the manual client according to the second ratio.
  • Q represents the preset target accuracy rate of the sample labeling in the test sample set after the identification model labeling and the manual client review.
  • the preset number is greater than or equal to the minimum sample extraction number N;
  • N Z 2 ⁇ (P ⁇ (1-P))/E 2 ;
  • Z represents the statistics related to the confidence level, and the confidence level is equal to the recognition accuracy rate of the recognition model;
  • E represents the preset sampling error value ;
  • P represents the recognition accuracy of the recognition model.
  • the modification module 202 modifies the recognition result marked by each sample in the target sample set to the wrong recognition result, including:
  • this embodiment uses an identification model to identify the samples in the test sample set, annotates the recognition results, and deliberately modifies the recognition results of a part of the samples in the test sample set to wrong recognition results, and the processed test samples The set is sent to the manual client for review.
  • FIG. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
  • an electronic device includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304.
  • the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.
  • the memory 303 is used to store computer programs
  • the processor 301 is configured to implement the following steps when executing the program stored in the memory 303:
  • the communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the communication bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
  • the communication interface is used for communication between the aforementioned electronic device and other devices.
  • the memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk storage.
  • NVM non-Volatile Memory
  • the memory may also be at least one storage device located far away from the foregoing processor.
  • the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (DSP), a dedicated Circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • CPU central processing unit
  • NP Network Processor
  • DSP digital signal processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • An embodiment of the present invention also provides a computer-readable storage medium in which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned manual client state checking method for sample review is implemented Method steps.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
  • Automatic Analysis And Handling Materials Therefor (AREA)

Abstract

一种用于样本审核的人工客户端的状态检查方法及装置，方法包括：获取一测试样本集，利用预先训练好的一识别模型对测试样本集中各个样本进行识别，标注出各个样本的识别结果(S101)；选取测试样本集中预设数量个样本作为目标样本集，并将目标样本集中每一样本所标注的识别结果修改为错误识别结果(S102)；将处理后的测试样本集发送给人工客户端，以便人工客户端对各个样本的识别结果进行审核(S103)；根据人工客户端对目标样本集的审核结果，检查人工客户端是否处于异常状态(S104)。

Description

用于样本审核的人工客户端的状态检查方法及装置 技术领域
本发明涉及人工智能技术领域，尤其涉及一种用于样本审核的人工客户端的状态检查方法、装置、电子设备和计算机可读存储介质。
背景技术
在人工智能领域，在进行模型训练之前，通常使用人工客户端来对样本的标注结果进行审核，如果在进行样本审核时人工客户端处于异常状态，则无法保证样本的标注情况达标，从而导致训练得到的模型的识别准确率不达标。因此，为了使训练得到的模型的准确率达标，要求人工客户端在进行样本审核时处于正常状态，这就需要对人工客户端的状态进行检查。
目前，可以通过检查人工客户端所审核的全部样本的标注结果，根据审核结果判断人工客户端是否存在异常，然而通常样本集中样本的数量非常大，这样就需要花费较多时间才能判断人工客户端是否异常。
发明内容
本发明的目的在于提供一种用于样本审核的人工客户端的状态检查方法、装置、电子设备和计算机可读存储介质，以快速判断人工客户端是否处于异常状态。具体技术方案如下：
第一方面，本发明提供一种用于样本审核的人工客户端的状态检查方法，所述方法包括：
获取一测试样本集，利用预先训练好的一识别模型对所述测试样本集中各个样本进行识别，标注出各个样本的识别结果；
选取所述测试样本集中预设数量个样本形成目标样本集，并将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，形成处理后的所述测试样本集；
将处理后的所述测试样本集发送给人工客户端，以便所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核；
根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，判定所述人工客户端是否处于异常状态。
可选的，所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核，包括：
针对处理后的所述测试样本集中每一样本，所述人工客户端判断所标注的识别结果是否正确；如果否，则对该样本所标注的识别结果进行修改。
可选的，根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，判定所述人工客户端是否处于异常状态，包括：
针对所述目标样本集中的每一样本，判断所述人工客户端是否对该样本的错误识别结果进行了修改；
获取在所述目标样本集中被所述人工客户端修改了识别结果的样本的比例，作为第一比例；
若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述方法还包括：
根据所述第一比例确定所述人工客户端的标注准确率。
可选的，根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，检查所述人工客户端是否处于异常状态，包括：
针对所述目标样本集中的每一样本，判断所述人工客户端是否将该样本的错误识别结果修改为正确识别结果；
获取在所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例，作为第二比例；
若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述方法还包括：
根据所述第二比例确定所述人工客户端的标注准确率。
可选的，所述预设阈值的最小值X按照以下公式确定：1-(1-X) 2＝Q；
其中，Q表示预先设置的经所述识别模型标注和所述人工客户端审核后所述测试样本集中样本标注的目标准确率。
可选的，所述预设数量大于等于最小样本抽取数量N；
其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度等于所述识别模型的识别准确率；E表示预设的抽样误差值；P表示所述识别模型的识别准确率。
可选的，将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，包括：
将所述目标样本集中每一样本所标注的识别结果修改为与原始识别结果不同的识别结果。
第二方面，本发明还提供一种用于样本审核的人工客户端的状态检查装置，所述装置包括：
标注模块，用于获取一测试样本集，利用预先训练好的一识别模型对所述测试样本集中各个样本进行识别，标注出各个样本的识别结果；
修改模块，用于选取所述测试样本集中预设数量个样本形成目标样本集，并将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，形成处理后的所述测试样本集；
审核模块，用于将处理后的所述测试样本集发送给人工客户端，以便所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核；
检查模块，用于根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，判定所述人工客户端是否处于异常状态。
可选的，所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核，包括：
针对处理后的所述测试样本集中每一样本，所述人工客户端判断所标注的识别结果是否正确；如果否，则对该样本所标注的识别结果进行修改。
可选的，所述检查模块，用于：
针对所述目标样本集中的每一样本，判断所述人工客户端是否对该样本的错误识别结果进行了修改；获取在所述目标样本集中被所述人工客户端修改了识别结果的样本的比例，作为第一比例；若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第一确定模块，用于根据所述第一比例确定所述人工客户端的标注准确率。
可选的，所述检查模块，用于：
针对所述目标样本集中的每一样本，判断所述人工客户端是否将该样本的错误识别结果修改为正确识别结果；获取在所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例，作为第二比例；若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第二确定模块，用于根据所述第二比例确定所述人工客户端的标注准确率。
可选的，所述预设阈值的最小值X按照以下公式确定：1-(1-X) 2＝Q；
其中，Q表示预先设置的经所述识别模型标注和所述人工客户端审核后所述测试样本集中样本标注的目标准确率。
可选的，所述预设数量大于等于最小样本抽取数量N；
其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度等于所述识别模型的识别准确率；E表示预设的抽样误差值；P表示所述识别模型的识别准确率。
可选的，所述修改模块将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，包括：
将所述目标样本集中每一样本所标注的识别结果修改为与原始识别结果不同的识别结果。
第三方面，本发明还提供了一种电子设备，包括处理器、通信接口、存储器和通信总线，其中，所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；
所述存储器，用于存放计算机程序；
所述处理器，用于执行所述存储器上所存放的所述计算机程序时，实现上述第一方面所述的用于样本审核的人工客户端的状态检查方法。
第四方面，本发明还提供了一种计算机可读存储介质，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现上述第一方面述所述的用于样本审核的人工客户端的状态检查方法。
与现有技术相比，本发明采用识别模型对测试样本集中的样本进行识别，标注出识别结果，并将测试样本集中一部分样本的识别结果故意修改为错误识别结果，将处理后的测试样本集发给人工客户端进行审核，这样只需要检查人工客户端对故意标错的那一部分样本的审核结果即可推断人工客户端对整个测试样本集的标注准确率，进而判断人工客户端是否处于异常状态，而不需要统计人工客户端对整个测试样本集的审核情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本发明一实施例提供的用于样本审核的人工客户端的状态检查方法的流程示意图；
图2是本发明一实施例提供的用于样本审核的人工客户端的状态检查装置的结构示意图；
图3是本发明一实施例提供的电子设备的结构示意图。
具体实施方式
以下结合附图和具体实施例对本发明提出的一种用于样本审核的人工客户端的状态检查方法、装置、电子设备及计算机可读存储介质作进一步详细说明。根据权利要求书和下面说明，本发明的优点和特征将更清楚。需说明的是，附图均采用非常简化的形式且均使用非精准的比例，仅用以方便、明 晰地辅助说明本发明实施例的目的。
为解决现有技术的问题，本发明实施例提供了一种用于样本审核的人工客户端的状态检查方法、装置、电子设备及计算机可读存储介质。
需要说明的是，本发明实施例的用于样本审核的人工客户端的状态检查方法可应用于本发明实施例的用于样本审核的人工客户端的状态检查装置，该用于样本审核的人工客户端的状态检查装置可被配置于电子设备上。其中，该电子设备可以是个人计算机、移动终端等，该移动终端可以是手机、平板电脑等具有各种操作系统的硬件设备。
图1是本发明一实施例提供的一种用于样本审核的人工客户端的状态检查方法的流程示意图。请参考图1，一种用于样本审核的人工客户端的状态检查方法可以包括如下步骤：
步骤S101，获取一测试样本集，利用预先训练好的一识别模型对所述测试样本集中各个样本进行识别，标注出各个样本的识别结果。
在本实施例中，所述识别模型可以是通过样本训练建立的神经网络模型，其可以是通过任一类型的样本训练得到的识别模型，本实施例对训练用的样本类型不做限定。例如样本可以是票据图片，训练后可以建立票据识别模型，也可以是车辆图片、人脸图片、植物图片、试卷图片等，不同的样本类型通过样本训练分别可以建立不同的识别模型。当所述识别模型训练好后，所述识别模型的识别准确率也就确定了。通过样本训练建立所述识别模型的过程以及确定所述识别模型的识别准确率的方式可以参见现有技术，在此不做赘述。
所述测试样本集中的样本类型需与所述识别模型训练时的样本类型相同，例如，所述识别模型是通过票据图片样本训练得到的，那么所获取的测试样本集中样本即为票据图片。
本实施例对测试样本集中的样本数量不做限定，但是为了使判断人工客户端是否异常的判断结果更加准确，样本数量通常应设置为较大的数值，例如样本数量为100、1000等。将测试样本集中的各个样本均通过所述识别模型进行识别，并将所述识别模型的识别结果进行标注。识别结果可以标注在 样本图片上，也可以标注作为样本图片的属性信息。
步骤S102，选取所述测试样本集中预设数量个样本作为目标样本集，并将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果。
将样本所标注的识别结果修改为错误识别结果，即修改为与所标注的原始识别结果不一致或不同的识别结果。举例而言，若所述识别模型是基于人脸图像样本训练得到的，用于识别人脸图像中的人脸是男性还是女性，所述识别模型对所述测试样本集中某一人脸图像样本的识别结果为女性，则该人脸图像样本所标注的识别结果即为女性，那么将该人脸图像样本所标注的识别结果修改为错误识别结果，例如修改为男性。又如，若所述识别模型是基于植物图像样本训练得到的，用于识别植物像中的植物的类别，所述识别模型对所述测试样本集中某一植物图像样本的识别结果为桃花，则该植物图像样本所标注的识别结果即为桃花，那么将该植物图像样本所标注的识别结果修改为不一致或不同的识别结果，例如修改为梨花。
本实施例中，可以从所述测试样本集中随机地抽取预设数量个样本作为目标样本集，将目标样本集所标注的识别结果修改为错误的识别结果。由于本发明的核心思想是通过统计人工客户端对所抽取的预设数量个目标样本集的错误识别结果的审核情况，来推断人工客户端对整个测试样本集中样本的审核情况，进而判断人工客户端是否处于异常状态，因此，为了保证后续统计的准确性，对所抽取的目标样本集的数量可以有如下要求：
所述预设数量大于等于抽样统计的最小样本抽取数量N；
其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度等于所述识别模型的识别准确率；E表示预设的抽样误差值；P表示所述识别模型的识别准确率。
Z与置信度的对应关系如下：当置信度为90％时，Z＝1.64；置信度为95％时，Z＝1.96；置信度为95.45％时，Z＝2；置信度为99％时，Z＝2.68；置信度为99.73时，Z＝3；以上数据可以通过查询统计表得出。本实施例的置信区间为90％～99.99％，也就是说认为所述识别模型的识别准确率应该在90％～99.99％的几率落入到P的范围内，本实施例可以采用95％的置信度。
本实施例中，抽样误差值E可以设定在±5％之间，P为概率值，可以设定其为90％，即经所述识别模型标注后所述测试样本集中样本标注的准确率需要达到90％。
如果通过上述计算公式计算得到最小样本抽取数量N等于100，则所述预设数量可以设定大于等于100的任意数值。也可以从所述测试样本集中抽取一定比例的样本作为目标样本集，只要保证所抽取的目标样本集的数量大于等于最小样本抽取数量N即可。
步骤S103，将处理后的所述测试样本集发送给人工客户端，以便所述人工客户端对各个样本的识别结果进行审核。
所述人工客户端会对所述识别模型的识别结果进行审核处理，审核处理包括：针对处理后的测试样本集中每一样本，判断所标注的识别结果是否正确；如果判定不正确，还可以对该样本所标注的识别结果进行修改。需要说明的是，由于处理后的测试样本集中包含两类样本，即标注有所述识别模型的识别结果的样本，和被抽取并故意标注错误识别结果的样本，人工客户端在审核时，不会对这两类样本进行区分，而是将被抽取并故意标注错误识别结果的目标样本集同样视为标注有所述识别模型的识别结果的样本。
举例而言，仍以上述步骤S102中的举例为例，若某一样本所标注的识别结果为女性，而人工客户端在审核后判定该样本所标注的识别结果有误，且经过自身的识别后判定该样本的识别结果应该为男性，则可以对该样本所标注的识别结果修改为自身所判定的识别结果。
实际上，对于被抽取并故意标注错误的某一目标样本集，人工客户端可能没有识别出该样本被标注错误，从而导致人工客户端将该样本的识别结果判定为正确。人工客户端对故意标错的目标样本集的审核情况反应了人工客户端对整个测试样本集中样本的识别(标注)情况，进而通过检查人工客户端对这类样本的审核情况可以来推断人工客户端的标注准确率或审核准确率，并且判断人工客户端是否存在异常。
步骤S104，根据所述人工客户端对所述目标样本集的审核结果，检查所述人工客户端是否处于异常状态。
在一种实现方式中，所述根据所述人工客户端对所述目标样本集的审核结果，检查所述人工客户端是否处于异常状态，包括：
针对所述目标样本集中的每一样本，判断所述人工客户端是否对该样本的错误识别结果进行了修改；
获取在所述目标样本集中被所述人工客户端修改了识别结果的样本的比例，作为第一比例；
若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可以理解的是，通常来说，如果人工客户端能够对故意标注错误的样本的错误识别结果进行修改，则可以认为人工客户端能够将该故意标注错误的样本进行正确的标注。若目标样本集中被人工客户端修改了识别结果的样本的比例大于等于预设阈值，可以认为人工客户端没有出现异常，反之，若目标样本集中被人工客户端修改了识别结果的样本的比例小于预设阈值，则表示人工客户端出现了异常。进一步的，还可以通过目标样本集中被人工客户端修改了识别结果的样本的比例，来推断人工客户端对整个测试样本集的标注准确率，例如将第一比例作为人工客户端的标注准确率。
具体而言，确定所述目标样本集中被所述人工客户端修改了识别结果的样本的比例，可以有如下两种方式：
方式一：对于经人工客户端审核后的所述测试样本集，首先获取被抽取并故意标注错误识别结果的目标样本集，然后判断人工客户端对所标注的识别结果进行了修改的样本的数量，进而获取所述目标样本集中被所述人工客户端修改了识别结果的样本的比例；
方式二：对于经人工客户端审核后的所述测试样本集，首先可以判断人工客户端对哪些样本所标注的识别结果进行了修改，然后统计出这些样本中哪些是属于被抽取并标注错误识别结果的目标样本集，进而获取目标样本集中被所述人工客户端修改了识别结果的样本的比例。
在另一种实现方式中，所述根据所述人工客户端对所述目标样本集的审核结果，检查所述人工客户端是否处于异常状态，包括：
针对所述目标样本集中的每一样本，判断所述人工客户端是否将该样本 的错误识别结果修改为正确识别结果；
获取在所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例，作为第二比例；
若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
在这一实现方式中，若目标样本集中被人工客户端修改为正确识别结果的样本的比例大于等于预设阈值，可以认为人工客户端没有出现异常，反之，若目标样本集中被人工客户端修改为正确识别结果的样本的比例小于预设阈值，则表示人工客户端出现了异常。进一步的，还可以通过目标样本集中被人工客户端修改为正确识别结果的样本的比例，来推断人工客户端对整个测试样本集的标注准确率，例如将第二比例作为人工客户端的标注准确率。
根据目标样本集中被人工客户端修改为正确识别结果的样本的比例，用来判断人工客户端是否存在异常，以及用来表征人工客户端的标注准确率，相比于上一种实现方式更加准确。
类似的，确定所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例，可以有如下两种方式：
方式一：对于经人工客户端审核后的所述测试样本集，首先获取被抽取并故意标注错误识别结果的目标样本集，然后判断人工客户端将所标注的识别结果修改为正确识别结果的样本的数量，进而获取所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例；
方式二：对于经人工客户端审核后的所述测试样本集，首先可以判断人工客户端将哪些样本所标注的识别结果修改为正确识别结果，然后统计出这些样本中哪些是属于被抽取并标注错误识别结果的目标样本集，进而获取目标样本集中被所述人工客户端修改为正确识别结果的样本的比例。
当判断出第一比例小于预设阈值，或第二比例小于预设阈值时，可以判定人工客户端处于异常状态，同时也表示人工客户端的标注准确率达不到期望值，因此可以对人工客户端进行修正，以使其标注准确率符合要求。
其中，所述预设阈值的最小值X可以按照以下公式确定：1-(1-X) 2＝Q；Q表示预先设置的经所述识别模型标注和所述人工客户端审核后所述测试样本 集中样本标注的目标准确率。所述预设阈值可以设置为任意的等于大于X的数值，本实施例对此不做限定。
当Q＝99％时，通过上述公式计算出X＝90％，即人工客户端的标注准确率需要达到90％以上。本实施例中，如果希望经所述识别模型标注和所述人工客户端审核后所述测试样本集中样本标注的准确率达到99％以上，则人工客户端在故意标错的错误样本中的标注准确率需要达到90％以上。
综上所述，与现有技术相比，本实施例中采用识别模型对测试样本集中的样本进行识别，标注出识别结果，并将测试样本集中一部分样本的识别结果故意修改为错误识别结果，将处理后的测试样本集发给人工客户端进行审核，这样只需要检查人工客户端对故意标错的那一部分样本的审核结果即可推断人工客户端对整个测试样本集的标注准确率，进而判断人工客户端是否处于异常状态，而不需要统计人工客户端对整个测试样本集的审核情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
相应于上述用于样本审核的人工客户端的状态检查方法实施例，本发明一实施例还提供了一种用于样本审核的人工客户端的状态检查装置，图2是本发明一实施例提供的一种用于样本审核的人工客户端的状态检查装置的结构示意图。请参考图2，一种用于样本审核的人工客户端的状态检查装置可以包括：
标注模块201，用于获取一测试样本集，利用预先训练好的一识别模型对所述测试样本集中各个样本进行识别，标注出各个样本的识别结果；
修改模块202，用于选取所述测试样本集中预设数量个样本作为目标样本集，并将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果；
审核模块203，用于将处理后的所述测试样本集发送给人工客户端，以便所述人工客户端对各个样本的识别结果进行审核；
检查模块204，用于根据所述人工客户端对所述目标样本集的审核结果，检查所述人工客户端是否处于异常状态。
可选的，所述审核模块203中人工客户端对各个样本的识别结果进行审核，包括：
针对处理后的所述测试样本集中每一样本，所述人工客户端判断所标注的识别结果是否正确；如果否，则对该样本所标注的识别结果进行修改。
可选的，所述检查模块204，具体用于：
针对所述目标样本集中的每一样本，判断所述人工客户端是否对该样本的错误识别结果进行了修改；获取在所述目标样本集中被所述人工客户端修改了识别结果的样本的比例，作为第一比例；若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第一确定模块，用于根据所述第一比例确定所述人工客户端的标注准确率。
可选的，所述检查模块204，具体用于：
针对所述目标样本集中的每一样本，判断所述人工客户端是否将该样本的错误识别结果修改为正确识别结果；获取在所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例，作为第二比例；若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
可选的，所述装置还包括：
第二确定模块，用于根据所述第二比例确定所述人工客户端的标注准确率。
可选的，所述预设阈值的最小值X按照以下公式确定：1-(1-X) 2＝Q；
其中，Q表示预先设置的经所述识别模型标注和所述人工客户端审核后所述测试样本集中样本标注的目标准确率。
可选的，所述预设数量大于等于最小样本抽取数量N；
其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度等于所述识别模型的识别准确率；E表示预设的抽样误差值；P表示所述识别模型的识别准确率。
可选的，所述修改模块202将所述目标样本集中每一样本所标注的识别 结果修改为错误识别结果，包括：
将所述目标样本集中每一样本所标注的识别结果修改为与原始识别结果不同的识别结果。
与现有技术相比，本实施例采用识别模型对测试样本集中的样本进行识别，标注出识别结果，并将测试样本集中一部分样本的识别结果故意修改为错误识别结果，将处理后的测试样本集发给人工客户端进行审核，这样只需要检查人工客户端对故意标错的那一部分样本的审核结果即可推断人工客户端对整个测试样本集的标注准确率，进而判断人工客户端是否处于异常状态，而不需要统计人工客户端对整个测试样本集的审核情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
本发明一实施例还提供了一种电子设备，图3是本发明一实施例提供的一种电子设备的结构示意图。请参考图3，一种电子设备包括处理器301、通信接口302、存储器303和通信总线304，其中，处理器301，通信接口302，存储器303通过通信总线304完成相互间的通信，
存储器303，用于存放计算机程序；
处理器301，用于执行存储器303上所存放的程序时，实现如下步骤：
获取一测试样本集，利用预先训练好的一识别模型对所述测试样本集中各个样本进行识别，标注出各个样本的识别结果；
选取所述测试样本集中预设数量个样本作为目标样本集，并将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果；
将处理后的所述测试样本集发送给人工客户端，以便所述人工客户端对各个样本的识别结果进行审核；
根据所述人工客户端对所述目标样本集的审核结果，检查所述人工客户端是否处于异常状态。
应用本实施例提供的方案，只需要检查人工客户端对故意标错的那一部分样本的审核结果即可推断人工客户端对整个测试样本集的标注准确率，进 而判断人工客户端是否处于异常状态，而不需要统计人工客户端对整个测试样本集的审核情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
关于该方法各个步骤的具体实现以及相关解释内容可以参见上述图1所示的方法实施例，在此不做赘述。
另外，处理器301执行存储器303上所存放的程序而实现的人工客户端的标注准确率的确定方法的其他实现方式，与前述方法实施例部分所提及的实现方式相同，这里也不再赘述。
上述电子设备提到的通信总线可以是外设部件互连标准(Peripheral Component Interconnect，PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，EISA)总线等。该通信总线可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
通信接口用于上述电子设备与其他设备之间的通信。
存储器可以包括随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器(Non-Volatile Memory，NVM)，例如至少一个磁盘存储器。可选的，存储器还可以是至少一个位于远离前述处理器的存储装置。
上述的处理器可以是通用处理器，包括中央处理器(Central Processing Unit，CPU)、网络处理器(Network Processor，NP)等；还可以是数字信号处理器(Digital Signal Processing，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。
本发明一实施例还提供了一种计算机可读存储介质，该计算机可读存储介质内存储有计算机程序，该计算机程序被处理器执行时实现上述的用于样本审核的人工客户端的状态检查方法的方法步骤。
应用本实施例提供的方案，只需要检查人工客户端对故意标错的那一部 分样本的审核结果即可推断人工客户端对整个测试样本集的标注准确率，进而判断人工客户端是否处于异常状态，而不需要统计人工客户端对整个测试样本集的审核情况来确定标注准确率，从而实现了快速确定人工客户端是否处于异常状态，并且缩短了统计时间，降低了费用成本。
需要说明的是，本说明书中的各个实施例均采用相关的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于装置、电子设备、计算机可读存储介质实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
上述描述仅是对本发明较佳实施例的描述，并非对本发明范围的任何限定，本发明领域的普通技术人员根据上述揭示内容做的任何变更、修饰，均属于权利要求书的保护范围。

Claims (20)

  1. 一种用于样本审核的人工客户端的状态检查方法，其特征在于，所述方法包括：
    获取一测试样本集，利用预先训练好的一识别模型对所述测试样本集中各个样本进行识别，标注出各个样本的识别结果；
    选取所述测试样本集中预设数量个样本形成目标样本集，并将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，形成处理后的所述测试样本集；
    将处理后的所述测试样本集发送给人工客户端，以便所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核；
    根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，判定所述人工客户端是否处于异常状态。
  2. 如权利要求1所述的用于样本审核的人工客户端的状态检查方法，其特征在于，所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核，包括：
    针对处理后的所述测试样本集中每一样本，所述人工客户端判断所标注的识别结果是否正确；如果否，则对该样本所标注的识别结果进行修改。
  3. 如权利要求2所述的用于样本审核的人工客户端的状态检查方法，其特征在于，根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，判定所述人工客户端是否处于异常状态，包括：
    针对所述目标样本集中的每一样本，判断所述人工客户端是否对该样本的错误识别结果进行了修改；
    获取在所述目标样本集中被所述人工客户端修改了识别结果的样本的比例，作为第一比例；
    若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
  4. 如权利要求3所述的用于样本审核的人工客户端的状态检查方法，其特征在于，所述方法还包括：
    根据所述第一比例确定所述人工客户端的标注准确率。
  5. 如权利要求2所述的用于样本审核的人工客户端的状态检查方法，其特征在于，根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，检查所述人工客户端是否处于异常状态，包括：
    针对所述目标样本集中的每一样本，判断所述人工客户端是否将该样本的错误识别结果修改为正确识别结果；
    获取在所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例，作为第二比例；
    若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
  6. 如权利要求5所述的用于样本审核的人工客户端的状态检查方法，其特征在于，所述方法还包括：
    根据所述第二比例确定所述人工客户端的标注准确率。
  7. 如权利要求3或5所述的用于样本审核的人工客户端的状态检查方法，其特征在于，所述预设阈值的最小值X按照以下公式确定：1-(1-X) 2＝Q；
    其中，Q表示预先设置的经所述识别模型标注和所述人工客户端审核后所述测试样本集中样本标注的目标准确率。
  8. 如权利要求1所述的用于样本审核的人工客户端的状态检查方法，其特征在于，所述预设数量大于等于最小样本抽取数量N；
    其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度等于所述识别模型的识别准确率；E表示预设的抽样误差值；P表示所述识别模型的识别准确率。
  9. 如权利要求1所述的用于样本审核的人工客户端的状态检查方法，其特征在于，将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，包括：
    将所述目标样本集中每一样本所标注的识别结果修改为与原始识别结果不同的识别结果。
  10. 一种用于样本审核的人工客户端的状态检查装置，其特征在于，所述装置包括：
    标注模块，用于获取一测试样本集，利用预先训练好的一识别模型对所述测试样本集中各个样本进行识别，标注出各个样本的识别结果；
    修改模块，用于选取所述测试样本集中预设数量个样本形成目标样本集，并将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，形成处理后的所述测试样本集；
    审核模块，用于将处理后的所述测试样本集发送给人工客户端，以便所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核；
    检查模块，用于根据所述人工客户端对处理后的所述测试样本集中的所述目标样本集的审核结果，判定所述人工客户端是否处于异常状态。
  11. 如权利要求10所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述人工客户端对处理后的所述测试样本集中的各个样本的识别结果进行审核，包括：
    针对处理后的所述测试样本集中每一样本，所述人工客户端判断所标注的识别结果是否正确；如果否，则对该样本所标注的识别结果进行修改。
  12. 如权利要求11所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述检查模块，用于：
    针对所述目标样本集中的每一样本，判断所述人工客户端是否对该样本的错误识别结果进行了修改；获取在所述目标样本集中被所述人工客户端修改了识别结果的样本的比例，作为第一比例；若所述第一比例小于预设阈值，则判定所述人工客户端处于异常状态。
  13. 如权利要求12所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述装置还包括：
    第一确定模块，用于根据所述第一比例确定所述人工客户端的标注准确率。
  14. 如权利要求11所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述检查模块，用于：
    针对所述目标样本集中的每一样本，判断所述人工客户端是否将该样本 的错误识别结果修改为正确识别结果；获取在所述目标样本集中被所述人工客户端修改为正确识别结果的样本的比例，作为第二比例；若所述第二比例小于预设阈值，则判定所述人工客户端处于异常状态。
  15. 如权利要求14所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述装置还包括：
    第二确定模块，用于根据所述第二比例确定所述人工客户端的标注准确率。
  16. 如权利要求12或15所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述预设阈值的最小值X按照以下公式确定：1-(1-X) 2＝Q；
    其中，Q表示预先设置的经所述识别模型标注和所述人工客户端审核后所述测试样本集中样本标注的目标准确率。
  17. 如权利要求10所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述预设数量大于等于最小样本抽取数量N；
    其中，N＝Z 2×(P×(1-P))/E 2；Z表示与置信度相关的统计量，置信度等于所述识别模型的识别准确率；E表示预设的抽样误差值；P表示预先设置的经所述识别模型标注和所述人工客户端审核后所述测试样本集中样本标注的目标准确率。
  18. 如权利要求10所述的用于样本审核的人工客户端的状态检查装置，其特征在于，所述修改模块将所述目标样本集中每一样本所标注的识别结果修改为错误识别结果，包括：
    将所述目标样本集中每一样本所标注的识别结果修改为与原始识别结果不同的识别结果。
  19. 一种电子设备，其特征在于，包括处理器、通信接口、存储器和通信总线，其中，所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；
    所述存储器，用于存放计算机程序；
    所述处理器，用于执行所述存储器上所存放的所述计算机程序时，实现权利要求1-9中任一所述的方法。
  20. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1-9中任一项所述的方法。
PCT/CN2020/096645 2019-06-20 2020-06-17 用于样本审核的人工客户端的状态检查方法及装置 WO2020253740A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910539076.4 2019-06-20
CN201910539076.4A CN110245087B (zh) 2019-06-20 2019-06-20 用于样本审核的人工客户端的状态检查方法及装置

Publications (1)

Publication Number Publication Date
WO2020253740A1 true WO2020253740A1 (zh) 2020-12-24

Family

ID=67888373

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/096645 WO2020253740A1 (zh) 2019-06-20 2020-06-17 用于样本审核的人工客户端的状态检查方法及装置

Country Status (2)

Country Link
CN (1) CN110245087B (zh)
WO (1) WO2020253740A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113793134A (zh) * 2021-09-26 2021-12-14 上汽通用五菱汽车股份有限公司 一种车辆报警方法、装置和计算机可读存储介质

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110245087B (zh) * 2019-06-20 2023-04-18 杭州睿琪软件有限公司 用于样本审核的人工客户端的状态检查方法及装置
CN111833296B (zh) * 2020-05-25 2023-03-10 中国人民解放军陆军军医大学第二附属医院 一种骨髓细胞形态学自动检测审核系统及审核方法
CN113138916B (zh) * 2021-04-06 2024-04-30 青岛以萨数据技术有限公司 基于标注样本的图片结构化算法的自动测试方法及系统
CN116307948A (zh) * 2023-05-23 2023-06-23 飞狐信息技术（天津）有限公司 审核质量检测方法、装置、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105975980A (zh) * 2016-04-27 2016-09-28 百度在线网络技术（北京）有限公司 监控图像标注质量的方法和装置
US20180075348A1 (en) * 2016-09-09 2018-03-15 Cylance Inc. Machine learning model for analysis of instruction sequences
CN109697537A (zh) * 2017-10-20 2019-04-30 北京京东尚科信息技术有限公司 数据审核的方法和装置
CN110245087A (zh) * 2019-06-20 2019-09-17 杭州睿琪软件有限公司 用于样本审核的人工客户端的状态检查方法及装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107766809B (zh) * 2017-10-09 2020-05-19 平安科技（深圳）有限公司 电子装置、票据信息识别方法和计算机可读存储介质
CN109492549A (zh) * 2018-10-24 2019-03-19 杭州睿琪软件有限公司 一种训练样本集处理、模型训练方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105975980A (zh) * 2016-04-27 2016-09-28 百度在线网络技术（北京）有限公司 监控图像标注质量的方法和装置
US20180075348A1 (en) * 2016-09-09 2018-03-15 Cylance Inc. Machine learning model for analysis of instruction sequences
CN109697537A (zh) * 2017-10-20 2019-04-30 北京京东尚科信息技术有限公司 数据审核的方法和装置
CN110245087A (zh) * 2019-06-20 2019-09-17 杭州睿琪软件有限公司 用于样本审核的人工客户端的状态检查方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113793134A (zh) * 2021-09-26 2021-12-14 上汽通用五菱汽车股份有限公司 一种车辆报警方法、装置和计算机可读存储介质
CN113793134B (zh) * 2021-09-26 2024-02-13 上汽通用五菱汽车股份有限公司 一种车辆报警方法、装置和计算机可读存储介质

Also Published As

Publication number Publication date
CN110245087A (zh) 2019-09-17
CN110245087B (zh) 2023-04-18

Similar Documents

Publication Publication Date Title
WO2020253740A1 (zh) 用于样本审核的人工客户端的状态检查方法及装置
WO2020253742A1 (zh) 样本标注审核方法及装置
WO2020253636A1 (zh) 样本标注信息的审核方法及装置
WO2020253741A1 (zh) 利用错误样本检查人工客户端状态的方法及装置
WO2020087713A1 (zh) 视频质检方法、装置、计算机设备及存储介质
WO2021003819A1 (zh) 基于知识图谱的人机对话方法及人机对话装置
WO2018157840A1 (zh) 语音识别测试方法及测试终端、计算设备及存储介质
TW201837788A (zh) 理賠單據的字元識別方法及伺服器
CN109284355B (zh) 一种批改试卷中口算题的方法及装置
WO2019019628A1 (zh) 移动应用的测试方法、装置、测试设备及介质
US11721229B2 (en) Question correction method, device, electronic equipment and storage medium for oral calculation questions
CN110555096A (zh) 用户意图识别方法、系统、终端及介质
WO2020215550A1 (zh) 错句检测方法、装置及计算机可读存储介质
CN109189895B (zh) 一种针对口算题的题目批改方法及装置
US9836565B2 (en) Electronic design automation method and apparatus thereof
WO2020237508A1 (zh) 一种断言验证代码绑定方法及装置
CN110909361A (zh) 一种漏洞检测方法，装置和计算机设备
WO2019148706A1 (zh) web入侵检测方法、装置、计算机设备和存储介质
WO2021036681A1 (zh) 数据验证方法、装置、计算机设备和存储介质
CN111277465A (zh) 一种异常数据报文检测方法、装置及电子设备
CN115658482A (zh) 数据库测试方法及装置、电子设备及计算机可读存储介质
CN112560511A (zh) 台词翻译方法、装置及翻译模型训练方法、装置
CN112926608A (zh) 一种图像分类方法、装置、电子设备及存储介质
CN113626587A (zh) 一种文本类别识别方法、装置、电子设备及介质
CN113127635A (zh) 数据处理方法、装置及系统，存储介质和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20826651

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20826651

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20826651

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 08/09/2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20826651

Country of ref document: EP

Kind code of ref document: A1