WO2022213565A1 - 一种人工智能模型预测结果的复审方法及装置 - Google Patents

一种人工智能模型预测结果的复审方法及装置 Download PDF

Info

Publication number
WO2022213565A1
WO2022213565A1 PCT/CN2021/123302 CN2021123302W WO2022213565A1 WO 2022213565 A1 WO2022213565 A1 WO 2022213565A1 CN 2021123302 W CN2021123302 W CN 2021123302W WO 2022213565 A1 WO2022213565 A1 WO 2022213565A1
Authority
WO
WIPO (PCT)
Prior art keywords
review
prediction result
prediction
artificial intelligence
intelligence model
Prior art date
Application number
PCT/CN2021/123302
Other languages
English (en)
French (fr)
Inventor
王博
郑文杰
陈江琦
刘思言
夏卫尚
Original Assignee
全球能源互联网研究院有限公司
国网山东省电力公司电力科学研究院
国家电网有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 全球能源互联网研究院有限公司, 国网山东省电力公司电力科学研究院, 国家电网有限公司 filed Critical 全球能源互联网研究院有限公司
Publication of WO2022213565A1 publication Critical patent/WO2022213565A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G06Q10/103—Workflow collaboration or project management

Definitions

  • the present application relates to the technical field of equipment detection, and in particular to a method and device for reviewing prediction results of an artificial intelligence model.
  • artificial intelligence models are used in many industries. For example, in the field of power transmission inspection, drones take pictures of equipment such as transmission poles and towers, and screen out possible problems in the photos based on the target detection model; in the security field, based on face detection and face comparison models, find possible problems criminal suspects on the run; in the Internet field, based on text classification models, find user speech that may not comply with regulations, etc.; in the industrial field, based on image target detection models, find defective products, etc.
  • AI models are not completely accurate, manual review or manual review is often required after applying AI models to find potential problems, so it is necessary to make decisions on which samples to review.
  • the current general method is to estimate the false detection and missed detection rates of the model under different confidence conditions based on the validation set, and set the confidence threshold for manual review/manual review based on actual needs.
  • the false detection rate and missed detection rate of the model prediction results at different confidence levels are different from those based on the There is a bias in the estimated value of the validation set, so the manual review plan based on the confidence threshold set in advance will cause waste of manpower: on the one hand, the number of samples for manual review may be higher/lower than expected, resulting in higher than expected review costs or problems The number of detections is low; on the other hand, for multi-category prediction results, the workload of manual review may be concentrated in a few categories with frequent false positives, and the problem samples or filters found in review due to misallocation of review resources The number of false positive samples dropped is low.
  • the embodiment of the present application provides a method and device for reviewing the prediction result of an artificial intelligence model, so as to at least solve the problems of low number of re-examinations and low accuracy of re-examination when re-examining the prediction results of an artificial intelligence model in a way of manual re-examination. .
  • An embodiment of the present application provides a method for reviewing a prediction result of an artificial intelligence model.
  • the review method includes: obtaining a prediction result of an artificial intelligence model and a prediction category and confidence level corresponding to the prediction result; Determine the review sequence and review scope; review the prediction result based on the preset optimization conditions, the review sequence and the review scope, and obtain a review result.
  • the reexamination method further includes: adjusting the reexamination order and/or the reexamination scope of the predicted result according to the reexamination result.
  • the adjusting the review sequence and/or the review scope of the prediction result according to the review result includes: determining, according to the review result, a sample of the prediction result that has been reviewed. ; Calculate the number of correctly and incorrectly predicted samples and their corresponding proportions within a preset range in the prediction result samples that have completed the review; the preset range includes: preset reliability intervals, preset prediction category results , at least one of the categories marked by the prediction result samples.
  • the preset optimization conditions include at least one of the following three items: when the overall review amount is determined, the number of detected problem samples reaches the maximum; When the number is determined, the overall number of reviews is the smallest; when the total number of reviews is determined, the number of false positives of the artificial intelligence model determined through the review is the largest.
  • the re-examination of the prediction result based on the preset optimization conditions, the re-examination order and the re-examination scope includes: based on the prediction confidence of the artificial intelligence model, Review the prediction results from high to low, or, starting from the preset threshold, review the prediction results from low to high; when the number of reviews reaches the preset number, or the number of detected problems reaches the preset number of problems , to terminate the review.
  • the re-examination of the prediction result based on the preset optimization conditions, the re-examination order and the re-examination scope includes: based on a Bayesian method, using a validation set data design Determine the prior distribution of the accuracy of the artificial intelligence model in each category with different confidence levels; determine the review sequence based on the multi-armed bandit method, and calculate the posterior distribution of the model accuracy during the review process; based on the optimization conditions , determine the termination conditions of the review, the termination conditions include: when the number of reviews reaches the preset number, or the number of detected problems reaches the preset number, the review is terminated; based on the prior distribution, posterior distribution and The termination condition reviews the predictions above.
  • the re-examination of the prediction result based on the preset optimization conditions, the re-examination order and the re-examination scope includes: selecting an unreviewed confidence level in the prediction category The highest predicted result sample is reviewed.
  • the review method further includes: for the prediction result samples outside the review scope, determining the prediction result samples whose confidence is lower than that of the manual review sample as no problem; A sample of prediction results with a higher confidence than the human-reviewed sample is considered problematic.
  • the embodiment of the present application also provides an apparatus for reviewing a prediction result of an artificial intelligence model
  • the review apparatus includes: a prediction result obtaining module configured to obtain the prediction result of the artificial intelligence model and the prediction category and confidence level corresponding to the prediction result
  • the review sequence and scope determination module is configured to determine the review sequence and review scope according to the prediction category and confidence; the review module is configured to perform a review on the prediction results based on preset optimization conditions, the review sequence and the review scope. Review and get the review result.
  • An embodiment of the present application further provides a computer device, including: a memory and a processor, the memory and the processor are connected in communication with each other, the memory stores computer instructions, and the processor executes the Computer instructions, thereby executing the method for reviewing the prediction result of the artificial intelligence model described in the first aspect or any one of the embodiments of the first aspect.
  • Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, where the computer instructions are used to cause the computer to execute the first aspect or any implementation manner of the first aspect Methods for reviewing predictions from artificial intelligence models as described in .
  • the prediction result of the artificial intelligence model is reviewed, the process of manual review is avoided, the shortcomings of manual review of samples based on the preset scheme can be effectively avoided, the review results are more accurate, and the real-time basis of Data characteristics adjust the review strategy, more accurately estimate the review workload, and more efficiently find problem samples or false positive samples.
  • FIG. 1 shows a schematic flowchart of a method for reviewing a prediction result of an artificial intelligence model according to an embodiment of the present application
  • FIG. 2 shows a schematic structural diagram of an apparatus for reviewing a prediction result of an artificial intelligence model according to an embodiment of the present application
  • FIG. 3 shows a schematic diagram of a hardware structure of a computer device according to an embodiment of the present application.
  • the embodiment of the present application provides a method for reviewing a prediction result of an artificial intelligence model, as shown in FIG. 1 , the review method includes:
  • Step S101 Obtain the prediction result of the artificial intelligence model and the prediction category and confidence level corresponding to the prediction result.
  • the review method is mainly a scheme for reviewing the prediction result of the artificial intelligence model. Therefore, first, the prediction result of the artificial intelligence model and the prediction category and confidence level corresponding to the prediction result need to be obtained.
  • the object of the prediction result of the artificial intelligence model is not limited, for example, it may be the defect detection result applied in the field of power transmission inspection, or it may be applied in the field of The identification result of identifying the data, information, etc. appearing in the network, etc.
  • Step S102 Determine the review sequence and review range according to the prediction category and the confidence level.
  • Step S103 Review the prediction result based on the preset optimization condition, the review sequence and the review range, and obtain a review result.
  • the preset optimization condition may mainly include at least one of the following three items: when the overall review amount is determined, the number of detected problem samples reaches the maximum; when the number of detected problem samples is determined, The total number of reviews is the smallest; when the total review volume is determined, the number of false positives of the artificial intelligence model determined through the review is the largest.
  • the prediction result of the artificial intelligence model is reviewed, which can effectively avoid the shortcomings of manual review of samples based on the preset scheme, the review results are more accurate, and the review strategy can be adjusted in real time according to the characteristics of the data. Accurately estimate review workload and find problem samples or false positive samples more efficiently.
  • the review method further includes: adjusting the review order and/or review scope of the predicted result according to the review result.
  • this adjustment process mainly includes: according to the review result, determining a sample of prediction results that has been reviewed; calculating the number of correct and wrong samples predicted within a preset range in the sample of prediction results that have been reviewed.
  • the corresponding proportion; the preset range includes at least one of a preset reliability interval, a result of a preset prediction category, and a category marked by the prediction result sample.
  • the shortcomings of sample review based on the preset scheme can be effectively avoided, the review strategy can be adjusted in real time according to the data characteristics, the review workload can be more accurately estimated, and problem samples or false positive samples can be found more efficiently.
  • the prediction result is reviewed based on preset optimization conditions, the review sequence and the review scope, including: based on the artificial intelligence model Prediction confidence, review the prediction results from high to low, or, starting from a preset threshold, review the prediction results from low to high; when the number of reviews reaches the preset number, or the number of detected problems reaches the predetermined threshold.
  • the review will be terminated.
  • optimization conditions can be achieved: when the number of detected problem samples is determined, the overall number of reviews is minimized; when the overall number of reviews is determined, the number of false positives of the artificial intelligence model determined through the review is maximized.
  • the prediction result is reviewed based on preset optimization conditions, the review sequence and the review scope, including: based on a Bayesian method, using The validation set data sets the prior distribution of the accuracy of the artificial intelligence model in each category with different confidence levels; based on the multi-armed bandit method, the review sequence is determined, and the posterior distribution of the model accuracy is calculated during the review process; based on The optimization condition determines the termination condition of the review, and the termination condition includes: when the number of reviews reaches a preset number, or when the number of detected problems reaches a preset number of problems, the review is terminated; based on the prior distribution, The posterior distribution and termination conditions review the above predictions.
  • the validation set data is used to set the prior distribution of the model's accuracy in each category with different confidence levels (assuming that for different categories, at different confidence levels, the ratio of the model's accuracy to the validation set's accuracy remains unchanged) , and assume that the priori of the model accuracy when the reliability is P0 obeys Beta(tA,(1-t)A), where t is the accuracy of the validation set, and A is a predetermined integer, usually 10 to 50); based on Multi-armed bandit method, determine the review order (select the category with the largest estimate of the 90% quantile on the precision rate, and use the unreviewed and most confident samples in this category as the next sample to be reviewed), and review In the process, the posterior distribution of the model accuracy rate is calculated (for each category, select the picture with the highest confidence in the unreviewed sample, record its confidence as P, and calculate the model when the confidence of the category is P based on the reviewed pictures upper 90% quantile of precision).
  • the optimization conditions can be achieved: when the overall review quantity is determined, the number of detected problem samples is the largest; when the number of detected problem samples is determined, the overall number of reviews is minimized.
  • reviewing the prediction result based on a preset optimization condition, the review order and the review scope includes: selecting a category that is not in the prediction category. The sample of the prediction results with the highest confidence in the review is reviewed.
  • First select a category and select the unreviewed samples with the highest confidence in the category for review. If at least one condition is met, switch to the next category; otherwise, continue the review of that category. If the total number of reviews or the number of detected problem/defect samples reaches a given value, the review will be stopped.
  • the optimization conditions can be achieved: when the overall review quantity is determined, the number of detected problem samples is the largest; when the number of detected problem samples is determined, the overall number of reviews is minimized.
  • the review method further includes: for the prediction result samples that are outside the review scope, determining the prediction result samples whose confidence is lower than that of the manual review sample as no problem. ; Consider the predicted result samples with higher confidence than the manually reviewed samples as problematic.
  • Application Example 1 Applied in the field of power transmission inspection, an intelligent defect detection system for intelligent defect detection of power transmission equipment.
  • the system mainly includes the following modules:
  • the images to be detected can be entered into the system through manual uploading by users remotely, uploading images collected by drones through Wi-Fi or mobile communication network, and copying them using mobile storage devices.
  • the pictures to be inspected can be pictures taken for the inspection of all or part of the towers of a transmission line.
  • (2) Defect intelligent detection module Using the target detection model (ie, the artificial intelligence model described in the embodiment of the present application), predict the target frame position, category and confidence of the hidden dangers of the power transmission equipment that may be included in the picture, and obtain the prediction result.
  • the target detection model ie, the artificial intelligence model described in the embodiment of the present application
  • the review When reviewing images, if the ratio of the number of reviewed samples (or the number of false positives in reviewed samples) to the total number of images reaches a given value, the review will end; otherwise, first, for each category, select Unreviewed For the picture with the highest confidence in the sample, record its confidence as P, and calculate the upper 90% quantile of the model accuracy when the category confidence is P based on the reviewed pictures; select the upper 90% quantile of the accuracy For the category with the largest estimated value, the unreviewed and most confident samples in this category are returned as the next sample to be reviewed, and displayed to the user; when the user completes the review, collect and record their opinions on the sample. Review results.
  • Application example 2 applied to the Internet field, a data audit system for network transmission data.
  • the system includes the following modules:
  • the initial review module calls the AI classification model (ie artificial intelligence model) to predict the incoming data analysis request, and according to the classification result and confidence, the request of certain violation categories and the confidence exceeds the given threshold 2 of the corresponding category is identified as review If it does not pass, the request with a category violation and the confidence level is not lower than the corresponding threshold 1 and not higher than the corresponding threshold 2 is deemed to require manual review, and the classification results of other categories are deemed to have passed the review. Return the results of the preliminary review and send the request for review to the review module for review.
  • AI classification model ie artificial intelligence model
  • (2) Review module Provide the text contained in the received review request to the review module and place it on the pending review list.
  • the review speed is calculated;
  • the review request generation speed is estimated for each confidence interval for each category;
  • the model false alarm rate is estimated for each confidence interval for each category.
  • the length of the confidence interval is a preset value, which can be 1%, 5%, and so on.
  • the threshold 2 of the preliminary review module is reset according to the calculation result, and the samples whose confidence exceeds the threshold 2 in the pending review request are removed from the pending review list, and their status is set as approved.
  • the samples in the pending review list are displayed to the reviewers in the order of time they entered the list, and the status of the corresponding samples is set to approved or rejected according to the reviewers' judgment results on the samples.
  • the embodiment of the present application also provides a review device for the prediction result of an artificial intelligence model.
  • the review device includes:
  • the prediction result obtaining module 1 is configured to obtain the prediction result of the artificial intelligence model and the prediction category and confidence level corresponding to the prediction result;
  • a review order and scope determination module 2 configured to determine a review sequence and a review scope according to the predicted category and confidence
  • the review module 3 is configured to review the prediction result based on the preset optimization condition, the review sequence and the review range, and obtain a review result.
  • the review sequence and scope determination module 2 is further configured to adjust the review sequence and/or review scope of the predicted result according to the review result.
  • the review sequence and scope determination module 2 is configured to determine, according to the review results, samples of prediction results that have been reviewed; The number of correct and incorrect samples and their corresponding proportions are predicted within a preset range; the preset range includes: a preset reliability interval, the result of a preset prediction category, and at least one of the categories marked by the prediction result sample. one.
  • the preset optimization conditions include at least one of the following three items:
  • the number of false positives of the artificial intelligence model determined through the review reaches the maximum.
  • the review module 3 is configured to review the prediction results from high to low based on the prediction confidence of the artificial intelligence model, or, starting from a preset threshold, from low When it reaches the high level, the prediction result is reviewed; when the number of reviews reaches the preset number, or the number of detected problems reaches the preset number, the review is terminated.
  • the review module 3 is configured to use the validation set data to set the prior distribution of the different confidence accuracy rates of the artificial intelligence model based on the Bayesian method;
  • the multi-armed bandit method determines the review sequence, and calculates the posterior distribution of the model accuracy rate during the review process; based on the optimization condition, determines the termination condition of the review, and the termination condition includes: when the number of reviews reaches a preset number , or when the number of detected problems reaches the preset number of problems, the review is terminated; the above prediction results are reviewed based on the prior distribution, posterior distribution and termination conditions.
  • the review module 3 is configured to select unreviewed prediction result samples with the highest confidence in the prediction category for review.
  • the prediction result obtaining module 1 is configured to, for the prediction result samples outside the review range, determine the prediction result samples whose confidence is lower than that of the manual review samples as No problem; a sample of prediction results with a higher confidence level than the human-reviewed sample is considered problematic.
  • the apparatus for reviewing the prediction result of the artificial intelligence model provided by the above-mentioned embodiment performs the review, only the division of the above-mentioned program modules is used as an example for illustration. In practical applications, the above-mentioned processing can be allocated by different The program module is completed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing.
  • the apparatus for reviewing the prediction result of the artificial intelligence model provided by the above embodiments and the embodiment of the method for reviewing the prediction result of the artificial intelligence model belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
  • the prediction result of the artificial intelligence model is reviewed, which can effectively avoid the shortcomings of manual review of samples based on the preset scheme, the review results are more accurate, and the review strategy can be adjusted in real time according to the characteristics of the data. Accurately estimate review workload and find problem samples or false positive samples more efficiently.
  • the embodiment of the present application also provides a computer device, for example, the computer device may be an electronic device such as a mobile terminal and a server.
  • the computer device may include a processor 31 and a memory 32 , wherein the processor 31 and the memory 32 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 3 .
  • the processor 31 may be a central processing unit (Central Processing Unit, CPU).
  • the processor 31 may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or Other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above types of chips.
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • Other programmable logic devices discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above types of chips.
  • the memory 32 can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as the method for reviewing the prediction results of the artificial intelligence model in the embodiments of the present application.
  • Program instructions/modules eg, prediction result acquisition module 1, review sequence and scope determination module 2, and review module 3 shown in FIG. 2).
  • the processor 31 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 32, that is, the method for reviewing the prediction results of the artificial intelligence model in the above method embodiments is implemented. .
  • the memory 32 may include a storage program area and a storage data area, wherein the storage program area may store an operating system and an application program required by at least one function; the storage data area may store data created by the processor 31 and the like. Additionally, memory 32 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 32 may optionally include memory located remotely from processor 31 , which may be connected to processor 31 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • the one or more modules are stored in the memory 32, and when executed by the processor 31, execute the method for reviewing the prediction result of the artificial intelligence model in the embodiment shown in FIG. 1 .
  • Embodiments of the present application further provide a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and the computer instructions are used to make the computer execute the prediction results of the artificial intelligence model described in the embodiments of the present application method of review.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a flash memory (Flash Memory), a hard disk (Hard) Disk Drive, HDD) or solid-state hard disk (Solid-State Drive, SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memories.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Human Resources & Organizations (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Physics & Mathematics (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • General Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Development Economics (AREA)
  • Game Theory and Decision Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种人工智能模型预测结果的复审方法及装置，复审方法包括：获取人工智能模型的预测结果及预测结果对应的预测类别及置信度（S101）；根据预测类别及置信度确定复审顺序及复审范围（S102）；基于预设的优化条件、复审顺序及复审范围对预测结果进行复审，得到复审结果（S103）。通过实施本发明，针对人工智能模型的预测结果进行复审，避免了人工复审的过程，能有效规避基于预设方案进行样本人工复审的缺点，复审结果更加准确。

Description

一种人工智能模型预测结果的复审方法及装置
相关申请的交叉引用
本申请基于申请号为202110374361.2、申请日为2021年04月07日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本申请。
技术领域
本申请涉及设备检测技术领域，具体涉及一种人工智能模型预测结果的复审方法及装置。
背景技术
目前，人工智能模型在多个行业内得到应用。例如，输电巡检领域，无人机对输电杆塔等设备进行拍摄，基于目标检测模型筛查出拍摄照片中可能存在的问题；安防领域，基于人脸检测和人脸比对模型，找到可能的在逃犯罪嫌疑人；互联网领域，基于文本分类模型，找到可能不符合法规的用户言论等；工业领域，基于图片目标检测模型，找到有缺陷的产品等等。然而，由于人工智能模型并非完全准确，在应用人工智能模型找到潜在问题后，往往需要进行人工复审或人工复查，因此需要对哪些样本进行复查做出决策。目前的通用方法为：基于验证集对模型在不同置信度条件下的误检和漏检率进行估计，并结合实际需求设置人工复审/人工复查的置信度阈值。
但是，在模型预测、人工复审的系统实际应用时，由于样本采集方式、样本来源、样本特征等与训练集存在差异，模型预测结果在不同置信度水平上的误检率与漏检率与基于验证集估计的值存在偏差，因此基于事先设 置的置信度阈值进行的人工复审方案会造成人力的浪费：一方面，人工复审的样本数量可能会超出/低于预计，导致复审成本超出预计或问题检出数量偏低；另一方面，对于多类别的预测结果，人工复审的工作量可能会集中于少数几个频繁误报的类别，并因复审资源错配导致复审找出的问题样本或过滤掉的误报样本数量偏低。
发明内容
本申请实施例提供了一种人工智能模型预测结果的复审方法及装置，以至少解决人工复检的方式对人工智能模型的预测结果进行复审存在的复检数量低、复检准确率低等问题。
本申请实施例提供了一种人工智能模型预测结果的复审方法，所述复审方法包括：获取人工智能模型的预测结果及所述预测结果对应的预测类别及置信度；根据所述预测类别及置信度确定复审顺序及复审范围；基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，得到复审结果。
在本申请的一些可选实施例中，所述复审方法还包括：根据所述复审结果对所述预测结果的复审顺序和/或复审范围进行调整。
在本申请的一些可选实施例中，所述根据所述复审结果对所述预测结果的复审顺序和/或复审范围进行调整，包括：根据所述复审结果，确定已完成复审的预测结果样本；计算所述已完成复审的预测结果样本中，在预设范围内预测正确、错误的样本数量及对应的占比；所述预设范围包括：预设置信度区间、预设预测类别的结果、所述预测结果样本所标注的类别中的至少之一。
在本申请的一些可选实施例中，所述预设的优化条件包括以下三项中的至少之一：总体复审量确定的情况下，问题样本的检出数量达到最大；问题样本的检出数量确定时，总体复审数量达到最小；总体复审量确定的 情况下，通过复审确定的人工智能模型的误报数量达到最大。
在本申请的一些可选实施例中，所述基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：基于所述人工智能模型的预测置信度，从高到低对预测结果进行复审，或，从预设阈值开始，从低到高对预测结果进行复审；当复审数量达到预设数量时，或检出的问题数量达到给预设问题数量时，终止复审。
在本申请的一些可选实施例中，所述基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：基于贝叶斯方法，使用验证集数据设定所述人工智能模型在各类别不同置信度精确率的先验分布；基于多臂老虎机方法，确定所述复审顺序，并在复审过程中计算模型精确率的后验分布；基于所述优化条件，确定复审的终止条件，所述终止条件包括：当复审数量达到预设数量时，或检出的问题数量达到给预设问题数量时，终止复审；基于所述先验分布、后验分布及终止条件对上述预测结果进行复审。
在本申请的一些可选实施例中，所述基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：选取所述预测类别中未复审的置信度最高的预测结果样本进行复审。
在本申请的一些可选实施例中，所述复审方法还包括：对于处于所述复审范围之外的预测结果样本，将置信度比人工复审样本置信度低的预测结果样本判定为无问题；将置信度比人工复审样本置信度高的预测结果样本视为有问题。
本申请实施例还提供了一种人工智能模型预测结果的复审装置，所述复审装置包括：预测结果获取模块，配置为获取人工智能模型的预测结果及所述预测结果对应的预测类别及置信度；复审顺序及范围确定模块，配置为根据所述预测类别及置信度确定复审顺序及复审范围；复审模块，配 置为基于预设的优化条件、所述复审顺序及复审范围对所述预测结果进行复审，得到复审结果。
本申请实施例还提供了一种计算机设备，包括：存储器和处理器，所述存储器和所述处理器之间互相通信连接，所述存储器中存储有计算机指令，所述处理器通过执行所述计算机指令，从而执行第一方面或者第一方面的任意一种实施方式中所述的人工智能模型预测结果的复审方法。
本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储计算机指令，所述计算机指令用于使所述计算机执行第一方面或者第一方面的任意一种实施方式中所述的人工智能模型预测结果的复审方法。
通过本申请实施例的复审方法，针对人工智能模型的预测结果进行复审，避免了人工复审的过程，能有效规避基于预设方案进行样本人工复审的缺点，复审结果更加准确，并且，能够实时根据数据特点调整复审策略，更准确地预估复审工作量、更有效率地寻找问题样本或误报样本。
附图说明
通过参考附图会更加清楚的理解本申请的特征和优点，附图是示意性的而不应理解为对本申请进行任何限制，在附图中：
图1示出了本申请实施例的人工智能模型预测结果的复审方法的流程示意图；
图2示出了本申请实施例的人工智能模型预测结果的复审装置的结构示意图；
图3示出了本申请实施例的计算机设备的硬件结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本 申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请实施例提供了一种人工智能模型预测结果的复审方法，如图1所示，该复审方法包括：
步骤S101：获取人工智能模型的预测结果及所述预测结果对应的预测类别及置信度。
在本实施例中，该复审方法主要是针对人工智能模型的预测结果进行复审的方案，因此，首先需获取到人工智能模型的预测结果及该预测结果对应的预测类别及置信度。可选地，在本实施例中，该人工智能模型的预测结果所针对的对象，并不做限制，例如可以是应用在输电巡检领域的缺陷检测结果，也可以是应用在互联网领域，对网络中出现的数据、信息等进行识别的识别结果等。
步骤S102：根据所述预测类别及置信度确定复审顺序及复审范围。
步骤S103：基于预设的优化条件、所述复审顺序及复审范围对所述预测结果进行复审，得到复审结果。
在本实施例中，该预设的优化条件可以主要包括以下三项中的至少之一：总体复审量确定的情况下，问题样本的检出数量达到最大；问题样本的检出数量确定时，总体复审数量达到最小；总体复审量确定的情况下，通过复审确定的人工智能模型的误报数量达到最大。
通过本申请实施例的复审方法，针对人工智能模型的预测结果进行复审，能有效规避基于预设方案进行样本人工复审的缺点，复审结果更加准确，并且，能够实时根据数据特点调整复审策略，更准确地预估复审工作量、更有效率地寻找问题样本或误报样本。
可选地，在本申请的一些实施例中，该复审方法还包括：根据所述复审结果对所述预测结果的复审顺序和/或复审范围进行调整。
示例性的，这个调整过程主要包括：根据所述复审结果，确定已完成复审的预测结果样本；计算所述已完成复审的预测结果样本中，在预设范围内预测正确、错误的样本数量及对应的占比；所述预设范围包括：预设置信度区间、预设预测类别的结果、所述预测结果样本所标注的类别中的至少之一。
通过这个过程，能有效规避基于预设方案进行样本复审的缺点，实时根据数据特点调整复审策略，更准确地预估复审工作量、更有效率地寻找问题样本或误报样本。
可选地，在本申请的一些实施例中，上述步骤S103，基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：基于所述人工智能模型的预测置信度，从高到低对预测结果进行复审，或，从预设阈值开始，从低到高对预测结果进行复审；当复审数量达到预设数量时，或检出的问题数量达到给预设问题数量时，终止复审。基于这个过程，能够实现优化条件：问题样本的检出数量确定时，总体复审数量达到最小；总体复审量确定的情况下，通过复审确定的人工智能模型的误报数量达到最大。
可选地，在本申请的一些实施例中，上述步骤S103，基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：基于贝叶斯方法，使用验证集数据设定所述人工智能模型在各类别不同置信度精确率的先验分布；基于多臂老虎机方法，确定所述复审顺序，并在复审过程中计算模型精确率的后验分布；基于所述优化条件，确定复审的终止条件，所述终止条件包括：当复审数量达到预设数量时，或检出的问题数量达到给预设问题数量时，终止复审；基于所述先验分布、后验分布及 终止条件对上述预测结果进行复审。
在此实施例中，使用验证集数据设定模型在各类别不同置信度精确率的先验分布(假设对于不同类别，在不同置信度水平上，模型精确率与验证集精确率的比值不变，并假设置信度为P0时模型精确率的先验服从Beta(tA,(1-t)A)，其中t为验证集精确率，A为事先确定的整数，通常为10至50)；基于多臂老虎机方法，确定复审顺序(选择精确率上90％分位点估计值最大的类别，将该类别中未复审的、置信度最大的样本作为下一张待复审的样本)，并在复审过程中计算模型精确率的后验分布(对每个类别，选择未复审样本中置信度最大的图片，记其置信度为P，并根据已复审图片，计算该类别置信度为P时的模型精确率的上90％分位点)。
通过该过程，能够实现优化条件：总体复审量确定的情况下，问题样本的检出数量达到最大；问题样本的检出数量确定时，总体复审数量达到最小。
可选地，在本申请的一些实施例中，上述步骤S103，基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：选取所述预测类别中未复审的置信度最高的预测结果样本进行复审。
示例性的，对于多类别问题，预先设定n(n>＝1)个条件，其中条件i为连续ki(ki>＝1)个该类别的复审样本，模型预测结果正确的样本数量不超过ri。首先选择一个类别，选取该类别中未复审的置信度最高的样本进行复审。若满足至少一个条件，则切换至下一个类别；否则继续该类别的复审。若复审总量或检出问题/缺陷样本数量达到给定值，则停止复审。其中，对每个预设条件，ki和ri均为预先设定的正整数，例如：可预先设定两个条件，k1＝3,r1＝0,k2＝10,r2＝4，即如果对某个类别，复审到的前3个样本中不存在问题样本(r1＝0)，或连续10个样本中问题样本数量不超过4，则切换至下个类别。
通过该过程，能够实现优化条件：总体复审量确定的情况下，问题样本的检出数量达到最大；问题样本的检出数量确定时，总体复审数量达到最小。
此外，对于优化条件：总体复审量确定的情况下，通过复审确定的人工智能模型的误报数量达到最大。可以按照置信度从低往高复检，从而尽量找出误报情况，此时，预先设定n(n>＝1)个条件，其中条件i为连续ki(ki>＝1)个该类别的复审样本，模型预测结果正确的样本数量不低于ri。
可选地，在本申请的一些实施例中，该复审方法还包括：对于处于所述复审范围之外的预测结果样本，将置信度比人工复审样本置信度低的预测结果样本判定为无问题；将置信度比人工复审样本置信度高的预测结果样本视为有问题。
以下结合具体应用实例，对本申请实施例的人工智能模型预测结果的复审方法做进一步说明。
应用实例1：应用于输电巡检领域，对于输电设备进行缺陷智能检测的缺陷智能检测系统。该系统主要包括如下模块：
(1)图片采集模块。可以通过用户远程手动上传、无人机采集图像后通过Wi-Fi或移动通信网络上传、使用移动存储设备拷贝等方式，将待检测图片录入系统。待检测图片可以为一条输电线路全部或部分杆塔的巡检拍摄图片。
(2)缺陷智能检测模块。使用目标检测模型(即本申请实施例所述的人工智能模型)，预测图片中可能包含的输电设备隐患的目标框位置、类别和置信度，得到预测结果。
(3)复审模块。基于验证集得到缺陷智能检测模型各类别在不同置信度水平上的精确率。假设对于不同类别，在不同置信度水平上，模型精确率与验证集精确率的比值不变，并假设置信度为P0时模型精确率的先验服 从Beta(tA,(1-t)A)，其中t为验证集精确率，A为事先确定的整数，通常为10至50。在对图片进行复审时，若已复审样本数量(或已复审样本中的误报数量)与总图片数量之比达到给定值，则结束复审；否则，首先，对每个类别，选择未复审样本中置信度最大的图片，记其置信度为P，并根据已复审图片，计算该类别置信度为P时的模型精确率的上90％分位点；选择精确率上90％分位点估计值最大的类别，将该类别中未复审的、置信度最大的样本作为下一张待复审的样本返回，将其展示给用户；当用户完成复审后，收集并记录其对所述样本的复审结果。
(4)报告生成模块。基于复审结果得到缺陷列表，并生成缺陷报告。
应用实例2：应用于互联网领域，进行网络传输数据的数据审核系统。该系统包括如下模块：
(1)初审模块。初审模块调用AI分类模型(即人工智能模型)，对传入的数据分析请求进行预测，根据分类结果和置信度，将某些违规类别且置信度超过对应类别给定阈值2的请求认定为审核不通过，将类别违规且置信度不低于对应阈值1且不高于对应阈值2之间的请求认定为需人工复查，将其它类别的分类结果认定为审核通过。将初审结果返回，同时将需进行复查的请求发送至复审模块，等待复审。
(2)复审模块。将收到的复审请求包含的文本提供给复审模块，置于待复审列表。每隔固定时间，计算(a)复审速度；(b)估计每个类别每个置信度区间的复审请求产生速度；(c)估计每个类别每个置信度区间的模型误报率。置信度区间的长度为预先设定的值，可以为1％、5％等。根据(a)、(b)和预定条件计算最大复审工作量。然后根据最大复审工作量计算每个类别的阈值2，使得位于阈值1和2之间的各类别请求的模型误报数量达到最大。根据计算结果重新设置初审模块的阈值2，并将待复审请求中，置信度超过阈值2的样本移出待复审列表，将其状态设置为审核通过。将待复审 列表中的样本按照进入列表的时间顺序依次向审核人员展示，并根据审核人员对样本的判定结果，将对应样本的状态设置为审核通过或审核不通过。
基于上述实施例，本申请实施例还提供了一种人工智能模型预测结果的复审装置，如图2所示，该复审装置包括：
预测结果获取模块1，配置为获取人工智能模型的预测结果及所述预测结果对应的预测类别及置信度；
复审顺序及范围确定模块2，配置为根据所述预测类别及置信度确定复审顺序及复审范围；
复审模块3，配置为基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，得到复审结果。
在本申请的一些可选实施例中，所述复审顺序及范围确定模块2，还配置为根据所述复审结果对所述预测结果的复审顺序和/或复审范围进行调整。
在本申请的一些可选实施例中，所述复审顺序及范围确定模块2，配置为根据所述复审结果，确定已完成复审的预测结果样本；计算所述已完成复审的预测结果样本中，在预设范围内预测正确、错误的样本数量及对应的占比；所述预设范围包括：预设置信度区间、预设预测类别的结果、所述预测结果样本所标注的类别中的至少之一。
在本申请的一些可选实施例中，所述预设的优化条件包括以下三项中的至少之一：
总体复审量确定的情况下，问题样本的检出数量达到最大；
问题样本的检出数量确定时，总体复审数量达到最小；
总体复审量确定的情况下，通过复审确定的人工智能模型的误报数量达到最大。
在本申请的一些可选实施例中，所述复审模块3，配置为基于所述人工 智能模型的预测置信度，从高到低对预测结果进行复审，或，从预设阈值开始，从低到高对预测结果进行复审；当复审数量达到预设数量时，或检出的问题数量达到预设问题数量时，终止复审。
在本申请的一些可选实施例中，所述复审模块3，配置为基于贝叶斯方法，使用验证集数据设定所述人工智能模型在各类别不同置信度精确率的先验分布；基于多臂老虎机方法，确定所述复审顺序，并在复审过程中计算模型精确率的后验分布；基于所述优化条件，确定复审的终止条件，所述终止条件包括：当复审数量达到预设数量时，或检出的问题数量达到给预设问题数量时，终止复审；基于所述先验分布、后验分布及终止条件对上述预测结果进行复审。
在本申请的一些可选实施例中，所述复审模块3，配置为选取所述预测类别中未复审的置信度最高的预测结果样本进行复审。
在本申请的一些可选实施例中，所述预测结果获取模块1，配置为对于处于所述复审范围之外的预测结果样本，将置信度比人工复审样本置信度低的预测结果样本判定为无问题；将置信度比人工复审样本置信度高的预测结果样本视为有问题。
需要说明的是：上述实施例提供的人工智能模型预测结果的复审装置在进行复审时，仅以上述各程序模块的划分进行举例说明，实际应用中，可以根据需要而将上述处理分配由不同的程序模块完成，即将装置的内部结构划分成不同的程序模块，以完成以上描述的全部或者部分处理。另外，上述实施例提供的人工智能模型预测结果的复审装置与人工智能模型预测结果的复审方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
通过本申请实施例的复审装置，针对人工智能模型的预测结果进行复审，能有效规避基于预设方案进行样本人工复审的缺点，复审结果更加准 确，并且，能够实时根据数据特点调整复审策略，更准确地预估复审工作量、更有效率地寻找问题样本或误报样本。
本申请实施例还提供了一种计算机设备，示例性的，计算机设备可以是移动终端、服务器等电子设备。如图3所示，该计算机设备可以包括处理器31和存储器32，其中处理器31和存储器32可以通过总线或者其他方式连接，图3中以通过总线连接为例。
处理器31可以为中央处理器(Central Processing Unit，CPU)。处理器31还可以为其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等芯片，或者上述各类芯片的组合。
存储器32作为一种非暂态计算机可读存储介质，可用于存储非暂态软件程序、非暂态计算机可执行程序以及模块，如本申请实施例中的人工智能模型预测结果的复审方法对应的程序指令/模块(例如，图2所示的预测结果获取模块1、复审顺序及范围确定模块2及复审模块3)。处理器31通过运行存储在存储器32中的非暂态软件程序、指令以及模块，从而执行处理器的各种功能应用以及数据处理，即实现上述方法实施例中的人工智能模型预测结果的复审方法。
存储器32可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储处理器31所创建的数据等。此外，存储器32可以包括高速随机存取存储器，还可以包括非暂态存储器，例如至少一个磁盘存储器件、闪存器件、或其他非暂态固态存储器件。在一些实施例中，存储器32可选包括相对于处理器31远程设置的存储器，这些远程存储器可以通过网络连接至处理器31。上述 网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
所述一个或者多个模块存储在所述存储器32中，当被所述处理器31执行时，执行如图1所示实施例中的人工智能模型预测结果的复审方法。
上述计算机设备具体细节可以对应参阅图1至图2所示的实施例中对应的相关描述和效果进行理解，此处不再赘述。
本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使所述计算机执行本申请实施例所述的人工智能模型预测结果的复审方法。
本领域技术人员可以理解，实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)、随机存储记忆体(Random Access Memory，RAM)、快闪存储器(Flash Memory)、硬盘(Hard Disk Drive，HDD)或固态硬盘(Solid-State Drive，SSD)等；所述存储介质还可以包括上述种类的存储器的组合。
本申请所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。
虽然结合附图描述了本申请的实施例，但是本领域技术人员可以在不脱离本申请的精神和范围的情况下作出各种修改和变型，这样的修改和变型均落入由所附权利要求所限定的范围之内。

Claims (11)

  1. 一种人工智能模型预测结果的复审方法，所述复审方法包括：
    获取人工智能模型的预测结果及所述预测结果对应的预测类别及置信度；
    根据所述预测类别及置信度确定复审顺序及复审范围；
    基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，得到复审结果。
  2. 根据权利要求1所述的人工智能模型预测结果的复审方法，其中，所述复审方法还包括：
    根据所述复审结果对所述预测结果的复审顺序和/或复审范围进行调整。
  3. 根据权利要求2所述的人工智能模型预测结果的复审方法，其中，所述根据所述复审结果对所述预测结果的复审顺序和/或复审范围进行调整，包括：
    根据所述复审结果，确定已完成复审的预测结果样本；
    计算所述已完成复审的预测结果样本中，在预设范围内预测正确、错误的样本数量及对应的占比；所述预设范围包括：预设置信度区间、预设预测类别的结果、所述预测结果样本所标注的类别中的至少之一。
  4. 根据权利要求1所述的人工智能模型预测结果的复审方法，其中，所述预设的优化条件包括以下三项中的至少之一：
    总体复审量确定的情况下，问题样本的检出数量达到最大；
    问题样本的检出数量确定时，总体复审数量达到最小；
    总体复审量确定的情况下，通过复审确定的人工智能模型的误报数量达到最大。
  5. 根据权利要求1或4所述的人工智能模型预测结果的复审方法，其 中，所述基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：
    基于所述人工智能模型的预测置信度，从高到低对预测结果进行复审，或，从预设阈值开始，从低到高对预测结果进行复审；
    当复审数量达到预设数量时，或检出的问题数量达到预设问题数量时，终止复审。
  6. 根据权利要求1或4所述的人工智能模型预测结果的复审方法，其中，所述基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：
    基于贝叶斯方法，使用验证集数据设定所述人工智能模型在各类别不同置信度精确率的先验分布；
    基于多臂老虎机方法，确定所述复审顺序，并在复审过程中计算模型精确率的后验分布；
    基于所述优化条件，确定复审的终止条件，所述终止条件包括：当复审数量达到预设数量时，或检出的问题数量达到给预设问题数量时，终止复审；
    基于所述先验分布、后验分布及终止条件对上述预测结果进行复审。
  7. 根据权利要求1或4所述的人工智能模型预测结果的复审方法，其中，所述基于预设的优化条件、所述复审顺序及所述复审范围对所述预测结果进行复审，包括：
    选取所述预测类别中未复审的置信度最高的预测结果样本进行复审。
  8. 根据权利要求1所述的人工智能模型预测结果的复审方法，其中，所述复审方法还包括：
    对于处于所述复审范围之外的预测结果样本，将置信度比人工复审样本置信度低的预测结果样本判定为无问题；将置信度比人工复审样本置信 度高的预测结果样本视为有问题。
  9. 一种人工智能模型预测结果的复审装置，所述复审装置包括：
    预测结果获取模块，配置为获取人工智能模型的预测结果及所述预测结果对应的预测类别及置信度；
    复审顺序及范围确定模块，配置为根据所述预测类别及置信度确定复审顺序及复审范围；
    复审模块，配置为基于预设的优化条件、所述复审顺序及复审范围对所述预测结果进行复审，得到复审结果。
  10. 一种计算机设备，包括：
    存储器和处理器，所述存储器和所述处理器之间互相通信连接，所述存储器中存储有计算机指令，所述处理器通过执行所述计算机指令，从而执行如权利要求1-8中任一项所述的人工智能模型预测结果的复审方法。
  11. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使所述计算机执行如权利要求1-8中任一项所述的人工智能模型预测结果的复审方法。
PCT/CN2021/123302 2021-04-07 2021-10-12 一种人工智能模型预测结果的复审方法及装置 WO2022213565A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110374361.2 2021-04-07
CN202110374361.2A CN113095563B (zh) 2021-04-07 2021-04-07 一种人工智能模型预测结果的复审方法及装置

Publications (1)

Publication Number Publication Date
WO2022213565A1 true WO2022213565A1 (zh) 2022-10-13

Family

ID=76674770

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/123302 WO2022213565A1 (zh) 2021-04-07 2021-10-12 一种人工智能模型预测结果的复审方法及装置

Country Status (2)

Country Link
CN (1) CN113095563B (zh)
WO (1) WO2022213565A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117579625A (zh) * 2024-01-17 2024-02-20 中国矿业大学 一种用于双重预防机制的巡检任务预分发方法

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113095563B (zh) * 2021-04-07 2024-08-09 全球能源互联网研究院有限公司 一种人工智能模型预测结果的复审方法及装置
CN114612752B (zh) * 2022-03-02 2024-06-28 捻果科技(深圳)有限公司 一种对视频分析技术识别结果智能复核的方法及系统

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108805173A (zh) * 2018-05-16 2018-11-13 苏州迈为科技股份有限公司 太阳能电池片色差分选方法
CN108875963A (zh) * 2018-06-28 2018-11-23 北京字节跳动网络技术有限公司 机器学习模型的优化方法、装置、终端设备和存储介质
EP3462387A1 (en) * 2017-10-02 2019-04-03 Servicenow, Inc. Improved machine learning classification with confidence thresholds
CN111124863A (zh) * 2019-12-24 2020-05-08 北京安兔兔科技有限公司 智能设备性能测试方法、装置及智能设备
CN111753914A (zh) * 2020-06-29 2020-10-09 北京百度网讯科技有限公司 模型优化方法和装置、电子设备及存储介质
CN112508044A (zh) * 2019-09-16 2021-03-16 华为技术有限公司 人工智能ai模型的评估方法、系统及设备
CN113095563A (zh) * 2021-04-07 2021-07-09 全球能源互联网研究院有限公司 一种人工智能模型预测结果的复审方法及装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107196953B (zh) * 2017-06-14 2020-05-08 上海境领信息科技有限公司 一种基于用户行为分析的异常行为检测方法
CN107729924B (zh) * 2017-09-25 2019-02-19 平安科技（深圳）有限公司 图片复审概率区间生成方法及图片复审判定方法
CN109658042B (zh) * 2018-10-08 2023-08-25 平安科技（深圳）有限公司 基于人工智能的复审方法、装置、设备及存储介质
CN110689531A (zh) * 2019-09-23 2020-01-14 云南电网有限责任公司电力科学研究院 一种基于yolo的输电线路机巡图像缺陷自动识别方法
CN110992329B (zh) * 2019-11-28 2023-06-30 上海微创卜算子医疗科技有限公司 一种产品表面缺陷检测方法、电子设备及可读存储介质
CN111597958B (zh) * 2020-05-12 2023-04-07 西安网算数据科技有限公司 一种高度自动化的票据分类方法及系统

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3462387A1 (en) * 2017-10-02 2019-04-03 Servicenow, Inc. Improved machine learning classification with confidence thresholds
CN108805173A (zh) * 2018-05-16 2018-11-13 苏州迈为科技股份有限公司 太阳能电池片色差分选方法
CN108875963A (zh) * 2018-06-28 2018-11-23 北京字节跳动网络技术有限公司 机器学习模型的优化方法、装置、终端设备和存储介质
CN112508044A (zh) * 2019-09-16 2021-03-16 华为技术有限公司 人工智能ai模型的评估方法、系统及设备
CN111124863A (zh) * 2019-12-24 2020-05-08 北京安兔兔科技有限公司 智能设备性能测试方法、装置及智能设备
CN111753914A (zh) * 2020-06-29 2020-10-09 北京百度网讯科技有限公司 模型优化方法和装置、电子设备及存储介质
CN113095563A (zh) * 2021-04-07 2021-07-09 全球能源互联网研究院有限公司 一种人工智能模型预测结果的复审方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
GAO KUNLUN, YANG SHUAI;LIU SIYAN;LI XIANGWEI: "Transient Stability Assessment for Power System Based on One-dimensional Convolutional Neural Network ", AUTOMATION OF ELECTRIC POWER SYSTEMS, NANJING ZIDONGHUA YANJIUSUO, NANJING, CN, vol. 43, no. 12, 25 June 2019 (2019-06-25), CN , pages 18 - 26, XP055974908, ISSN: 1000-1026, DOI: 10.7500/AEPS20180911006 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117579625A (zh) * 2024-01-17 2024-02-20 中国矿业大学 一种用于双重预防机制的巡检任务预分发方法
CN117579625B (zh) * 2024-01-17 2024-04-09 中国矿业大学 一种用于双重预防机制的巡检任务预分发方法

Also Published As

Publication number Publication date
CN113095563B (zh) 2024-08-09
CN113095563A (zh) 2021-07-09

Similar Documents

Publication Publication Date Title
WO2022213565A1 (zh) 一种人工智能模型预测结果的复审方法及装置
CN111160380B (zh) 生成视频分析模型的方法及视频分析系统
CN109711744B (zh) 保洁任务自动生成、执行评价方法、保洁方法及装置
US20220405145A1 (en) Method, Apparatus, System and Electronic Device for Selecting Intelligent Analysis Algorithm
CN112883921A (zh) 一种垃圾桶满溢检测模型训练方法及垃圾桶满溢检测方法
CN113642474A (zh) 一种基于yolov5的危险区域人员监控方法
CN112613569B (zh) 图像识别方法、图像分类模型的训练方法及装置
CN110647447A (zh) 用于分布式系统的异常实例检测方法、装置、设备和介质
CN112434717B (zh) 一种模型训练方法及装置
CN111369792B (zh) 一种交通事件的分析方法、装置及电子设备
WO2020233021A1 (zh) 基于智能决策的测试结果分析方法及相关装置
CN115273191A (zh) 一种人脸聚档方法、人脸识别方法、装置、设备及介质
CN110866831A (zh) 资产活跃度等级的确定方法、装置及服务器
CN113869427A (zh) 一种场景分析方法、装置、电子设备及存储介质
CN117573477A (zh) 异常数据监控方法、装置、设备、介质和程序产品
CN112686298A (zh) 一种目标检测方法、装置及电子设备
CN111611973A (zh) 目标用户识别的方法、装置及存储介质
CN114154018B (zh) 一种面向无人系统的云边协同视频流处理方法及系统
CN114257523B (zh) 用户感知预测方法、系统、设备和计算机存储介质
CN115904883A (zh) 一种rpa流程执行可视化异常监控方法、装置及介质
WO2022218001A1 (zh) 视频分析方法及相关系统
CN113807209B (zh) 车位检测方法、装置、电子设备及存储介质
CN114067145A (zh) 无源分光器检测方法、装置、设备及介质
CN113961827A (zh) 一种区域防控风险的检测方法及装置
CN108763291B (zh) 一种数据管理方法、装置及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21935783

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21935783

Country of ref document: EP

Kind code of ref document: A1