CN117892777A - Decision risk assessment method and system for target detection model - Google Patents
Decision risk assessment method and system for target detection model Download PDFInfo
- Publication number
- CN117892777A CN117892777A CN202311867880.8A CN202311867880A CN117892777A CN 117892777 A CN117892777 A CN 117892777A CN 202311867880 A CN202311867880 A CN 202311867880A CN 117892777 A CN117892777 A CN 117892777A
- Authority
- CN
- China
- Prior art keywords
- target
- target detection
- detection model
- file
- test sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 170
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012502 risk assessment Methods 0.000 title claims abstract description 53
- 238000012360 testing method Methods 0.000 claims abstract description 83
- 238000011156 evaluation Methods 0.000 claims abstract description 69
- 238000002372 labelling Methods 0.000 claims abstract description 28
- 238000012545 processing Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 6
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 230000007246 mechanism Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 13
- 238000012549 training Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000009466 transformation Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0475—Generative networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/094—Adversarial learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a decision risk assessment method and a decision risk assessment system for a target detection model, which belong to the technical field of artificial intelligence, and comprise the following steps: inputting each test sample into a target detection model to be evaluated, and outputting a corresponding target detection result; obtaining a corresponding prediction annotation file according to the target detection result of each test sample; comparing and analyzing the key effective information in the prediction labeling file and the correct labeling file of each test sample, and determining the evaluation result of the target detection model under each test sample; and comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample. According to the invention, a comparison is carried out on the prediction result of the input data of the target detection model and the label tag data corresponding to the original sample data, the target detection model is evaluated from different dimensions, the performances such as accuracy, robustness or safety of the target detection model are determined, and a reliability guarantee mechanism is perfected.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a decision risk assessment method and a decision risk assessment system for a target detection model.
Background
In the traditional evaluation of the target detection model, after model training is completed, the performance of the model can only be calculated from the height of mAP indexes based on a data set. In recent years, the algorithm in the deep learning target detection field is faster to update iteratively, and a plurality of evaluation methods are also developed. The evaluation of the intelligent recognition model is mainly to evaluate two aspects of the model, namely recognition accuracy and recognition robustness.
The accuracy of the target detection model requires that the model can also infer and obtain a detection result with correct identification for unknown sample data of unfamiliar environments. The main reason for the low model accuracy is: the real environment factors of the model are varied, the quantity of training data sets is limited, and all modes of the objective environment cannot be completely represented, so that the generalization capability of the target detection model obtained through training is insufficient, and the target detection model is unstable in the real use process.
The robustness of the target detection model requires that the model can keep outputting accurate prediction results for input samples with small disturbance. The main reason for the lack of model robustness is: the target detection model has insufficient interpretation, the deep learning model widely used in the current AI technology is formed by connecting and combining multiple layers of neural network modules, has huge model parameters and complex system structure, and is a non-convex function which has complex structure and is difficult to express by using clear analysis.
In recent years, considering AI safety problems, the objective detection model is likely to have unexpected decision risk hidden trouble in the decision process. The accuracy of the target detection model can even exceed that of humans without any intervention, however, during the model testing phase, the target detection model is vulnerable to spoofing by the test sample to output unpredictable results, even manipulated by an attacker, and in essence, such threats can be used to test the decision correctness of the AI model. The target detection model has defects in self identification accuracy and external attack disturbance spoofing resistance to form a misleading decision risk source, so that a decision risk assessment method for the model needs to be established, and consistency and reliability of decisions are ensured.
In the prior art, when a target detection model based on a convolutional neural network algorithm is evaluated, the model is evaluated from a single angle of recognition accuracy, safety or robustness of the model, and multi-dimensional evaluation cannot be simultaneously and comprehensively performed from the comprehensive reliability of the model, namely, from the angles of recognition accuracy, robustness and safety of the model. However, decision risk assessment is carried out on the model, particularly in the military field, the comprehensive assessment is precisely carried out on the model, and the performance of the model in unfamiliar environments, severe environments and artificial intelligence anti-attack environments is required to be comprehensively tested, so that the problem to be studied and solved by the invention is precisely solved.
Disclosure of Invention
The invention provides a decision risk assessment method and a decision risk assessment system for a target detection model, which are used for solving the defects in the prior art, realizing the assessment of the target detection model from different dimensions, determining the performances of the target detection model, such as accuracy, robustness or safety, and the like, and perfecting a reliability guarantee mechanism. .
In a first aspect, the present invention provides a decision risk assessment method for a target detection model, including: inputting each test sample into a target detection model to be evaluated, and outputting a corresponding target detection result; obtaining a corresponding prediction annotation file according to the target detection result of each test sample; comparing and analyzing the key effective information in the prediction labeling file and the correct labeling file of each test sample, and determining the evaluation result of the target detection model under each test sample; the prediction annotation file and the correct annotation file belong to target annotation files, and key effective information in the target annotation files comprises: image name information of an image where each target is located, category name information of each target and label frame position information of each target; and comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample.
According to the decision risk assessment method of the target detection model provided by the invention, the types of the test samples comprise: strange samples, transform samples, and challenge samples.
According to the decision risk assessment method of the target detection model provided by the invention, key effective information in a prediction annotation file and a correct annotation file of a current test sample is compared and analyzed, and an assessment result of the target detection model under the current test sample is determined, which comprises the following steps: extracting key effective information in the prediction labeling file and the correct labeling file respectively to generate a corresponding first comparison file and a corresponding second comparison file; traversing comparison to obtain a detection result of each target in the current test sample based on the first comparison file and the second comparison file; determining the identification accuracy, false detection rate and omission factor of the target detection model according to the detection result of each target; and determining an evaluation result of the target detection model under each test sample according to the identification accuracy and the omission factor.
According to the decision risk assessment method of the target detection model provided by the invention, the detection result of any target in the current test sample is obtained, and the decision risk assessment method comprises the following steps: determining the position information of a marking frame of any target in the first comparison file and the second comparison file respectively; calculating the marking frame and cross ratio of any target according to the marking frame position information of the any target in the first comparison file and the second comparison file respectively; judging whether category name information of any target in the first comparison file and the second comparison file is consistent or not under the condition that the marking frame intersection ratio is larger than or equal to a preset intersection ratio threshold value; and determining that any target is identified correctly under the condition that the category name information is consistent.
The decision risk assessment method of the target detection model provided by the invention further comprises the following steps: under the condition that the marking frame intersection ratio is smaller than a preset intersection ratio threshold value, determining that any target is missed; and determining that any one of the targets is identified as being wrong in the case that the category name information is determined to be inconsistent.
According to the decision risk assessment method of the target detection model provided by the invention, the file types of the prediction annotation file and the correct annotation file are xml files; and the file types of the first comparison file and the second comparison file are Excel files.
According to the decision risk assessment method for the target detection model provided by the invention, the comprehensive assessment is carried out on the target detection model according to the assessment result of the target detection model under each test sample, and the decision risk assessment method comprises the following steps: acquiring a history detection record of the target detection model; analyzing the application scene of the target detection model based on the history detection record, and determining the evaluation weight corresponding to each evaluation result; and determining a final comprehensive evaluation result according to each evaluation result and the evaluation weight corresponding to each evaluation result.
In a second aspect, the present invention further provides a decision risk assessment device for a target detection model, including:
the first processing module is used for inputting each test sample into the target detection model to be evaluated and outputting a corresponding target detection result;
The second processing module is used for obtaining a corresponding prediction annotation file according to the target detection result of each test sample;
The third processing module is used for comparing and analyzing key effective information in the prediction labeling file and the correct labeling file of each test sample and determining an evaluation result of the target detection model under each test sample; the prediction annotation file and the correct annotation file belong to target annotation files, and the target annotation files comprise: image name information of an image where each target is located, category name information of each target and label frame position information of each target;
And the fourth processing module is used for comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample.
In a third aspect, the present invention provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the decision risk assessment method of any one of the object detection models described above when the program is executed.
In a fourth aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a decision risk assessment method of an object detection model as described in any of the above.
The decision risk assessment method and system for the target detection model provided by the invention have the following beneficial effects:
(1) According to the decision risk assessment method and system for the target detection model, the target detection models based on different types of test samples, different unmanned platforms for collecting data and different training convergence conditions are subjected to statistics and summarization to form assessment results, so that decision risk assessment capability of the target detection model to be assessed under different working environments and data samples can be reflected comprehensively and intuitively.
(2) The decision risk assessment method and the decision risk assessment system for the target detection model provided by the invention can start from the input end data of the model, carry out reliability assessment on the unmanned platform identification model by using 3 dimensions of strange data, transformation data and countermeasure data, and comprehensively give decision risk assessment results of the model in the whole, including failed, good and excellent class 4 grading grades according to the grading assessment results of the 3 dimensions.
(3) According to the decision risk assessment method and system for the target detection model, provided by the invention, on the basis of an intelligent recognition system, the assessment test of the model can be automatically realized by simple programming, and the threshold values such as IoU cross ratio, accuracy, omission ratio and the like can be flexibly modified according to actual requirements.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a decision risk assessment method of a target detection model provided by the invention;
FIG. 2 is a screenshot of the extraction result EXCEL of the target annotation file provided by the invention;
FIG. 3 is a schematic diagram of a comparison process according to the present invention;
FIG. 4 is a schematic diagram of evaluation grading of object detection provided by the present invention;
FIG. 5 is a schematic diagram of test results of strange samples provided by the present invention;
FIG. 6 is a schematic diagram of test results of a transformed sample provided by the present invention;
FIG. 7 is a schematic illustration of test results for challenge samples provided by the present invention;
FIG. 8 is a schematic diagram of a decision risk assessment system of a target detection model according to the present invention;
fig. 9 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that in the description of embodiments of the present invention, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
The terms "first," "second," and the like in this specification are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type, and are not limited to the number of objects, such as the first object may be one or more.
The following describes a decision risk assessment method and apparatus for a target detection model according to an embodiment of the present invention with reference to fig. 1 to 9.
Fig. 1 is a flow chart of a decision risk assessment method of a target detection model provided by the present invention, as shown in fig. 1, including but not limited to the following steps:
Step 101: and inputting each test sample into the target detection model to be evaluated, and outputting a corresponding target detection result.
When the model is subjected to decision risk assessment, the actual conditions in the real work environment are considered, and the input data of the model mainly comprise 3 types: ① Strange sample data refers to image data collected by an unmanned platform outside a training data set; ② Transforming sample data, which refers to new data obtained by some data amplification means, such as flipping, rotation, brightness adjustment, etc.; ③ The challenge sample data refers to data to which a challenge sample patch is added. The challenge sample data is briefly described below.
In 2014, the Szegedy group first proposed the concept of countering the sample, i.e., adding a fine target disturbance to the input of the neural network, could result in the model giving an erroneous output with high confidence, but it is difficult for the human eye to distinguish the sample from the real sample. In recent years, the challenge sample technology is gradually presented in the field of view of people as a brand new attack means. The study found that deep learning models are almost entirely extremely vulnerable to attack against samples. The target detector challenge sample means that, in the target detection task, by adding the challenge sample to the picture, the target detector outputs an erroneous target detection result, which contains not only the kind information but also the position (pixel coordinate) information of the target, compared with the result of the classifier.
Step 102: and obtaining a corresponding prediction annotation file according to the target detection result of each test sample.
Step 103: and comparing and analyzing the key effective information in the prediction labeling file and the correct labeling file of each test sample, and determining the evaluation result of the target detection model under each test sample.
The prediction annotation file and the correct annotation file belong to a target annotation file, and key effective information of the target annotation file comprises: image name information of an image where each target is located, category name information of each target and label frame position information of each target.
It will be appreciated that the correct annotation file is the correct and authentic target annotation file of the pre-prepared test sample data.
Optionally, the file types of the prediction annotation file and the correct annotation file are xml files.
Step 104: and comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample.
According to the decision risk assessment method of the target detection model, the prediction result of the input data and the label tag data corresponding to the original sample data are compared one by the target detection model, the target detection model is assessed from different dimensions, performances such as accuracy, robustness or safety of the target detection model are determined, and a reliability guarantee mechanism is perfected.
Based on the foregoing embodiment, as an optional embodiment, the decision risk assessment method for the target detection model provided by the present invention compares and analyzes the key valid information in the prediction annotation file and the correct annotation file of the current test sample, and determines the assessment result of the target detection model under the current test sample, and includes the following steps:
(1) And respectively extracting key effective information in the prediction labeling file and the correct labeling file to generate a corresponding first comparison file and a corresponding second comparison file.
Fig. 2 is an EXCEL screenshot of the extraction result of the target markup file provided by the present invention, as shown in fig. 2, optionally, the file type of the predicted markup file is an xml file, and the python program is used to extract key effective information of the predicted markup file to generate a first comparison file, where the first comparison file may be an EXCEL table;
Correspondingly, the file type of the correct annotation file is an xml file, and the python program is utilized to extract key effective information of the correct annotation file so as to generate a second comparison file, wherein the second comparison file can be an Excel table.
(2) And traversing comparison to obtain a detection result of each target in the current test sample based on the first comparison file and the second comparison file.
Fig. 3 is a schematic diagram of a comparison flow provided in the present invention, as shown in fig. 3, in a comparison determination link, specifically, an Excel file (a second comparison file, that is, a reference answer) needs to be correctly marked by traversing a program line by line, and each time the comparison is performed with a prediction result Excel file (a first comparison file), and whether the target is detected correctly is automatically determined by comparing a graph name, a category name, and a mark box IoU with a comparison index (a comparison threshold may be set to 0.80).
Based on the foregoing embodiments, as an alternative embodiment, a method for obtaining a detection result of any target in the current test sample is described below:
Step 201: and determining the position information of the marking frame of any target in the first comparison file and the second comparison file respectively.
Step 202: and calculating the marking frame and the cross ratio of any target according to the marking frame position information of the any target in the first comparison file and the second comparison file respectively.
The calculation mode of the label frame and the cross ratio belongs to the prior art, and is not repeated here.
Step 203: and judging whether the category name information of any target in the first comparison file and the second comparison file is consistent or not under the condition that the marking frame intersection ratio is larger than or equal to a preset intersection ratio threshold value.
In addition, under the condition that the cross ratio of the marking frame is smaller than a preset cross ratio threshold, determining that any target is missed.
Alternatively, the preset overlap ratio threshold may be 0.8.
Step 204: and determining that any target is identified correctly under the condition that the category name information is consistent.
Further, in the case that the category name information is determined to be inconsistent, it is determined that any one of the targets is identified as erroneous.
(3) And determining the identification accuracy, false detection rate and omission factor of the target detection model according to the detection result of each target.
It can be understood that the prediction result of each target has 3 conditions of correct recognition, incorrect recognition and missing detection. By counting the accumulated total number of the 3 types of targets, the model identification accuracy, false detection rate and omission rate can be calculated.
(4) And determining an evaluation result of the target detection model under each test sample according to the identification accuracy and the omission factor.
Fig. 4 is an evaluation grading schematic diagram of the target detection provided by the present invention, as shown in fig. 4, the accuracy thresholds 0.9, 0.8, 0.7 and the omission ratio threshold 0.2 are set to be respectively decision thresholds, and according to the decision flow shown in fig. 4, the model is evaluated as excellent, good, pass and fail class 4 decision risk evaluation results when evaluating based on each class of test samples.
Based on the foregoing embodiments, as an optional embodiment, the decision risk assessment method for a target detection model provided by the present invention, according to an assessment result of the target detection model under each test sample, performs comprehensive assessment on the target detection model, including: acquiring a history detection record of the target detection model; analyzing the application scene of the target detection model based on the history detection record, and determining the evaluation weight corresponding to each evaluation result; and determining a final comprehensive evaluation result according to each evaluation result and the evaluation weight corresponding to each evaluation result.
It can be understood that the invention can respectively obtain the respective evaluation results of the target detection model on strange samples, transformation samples and countermeasure samples, and further, according to the tendency degree of users, for example, according to different working environments facing the model, different weight consideration can be respectively given to the three types of evaluation results, and the decision risk evaluation results of the target detection model are comprehensively given.
In order to more fully describe the technical scheme of the invention, the following discussion of the technical scheme of the invention is carried out in combination with specific experiments.
① Model-1 strange sample test
And inputting 228 strange sample photos except the training data set, outputting a detection result, extracting a prediction labeling file to generate a first comparison file (total 959 targets), comparing the first comparison file with a second comparison file (total 937 targets) corresponding to a correct labeling file, and performing program automation evaluation to obtain a model evaluation result, wherein the model evaluation result is shown in fig. 5, the model accuracy is 0.9477, the omission ratio is 0.0488, and the accuracy is more than 0.9, so that the decision risk evaluation result of the model based on strange sample data is excellent.
The experimental results are shown in fig. 5, and fig. 5 is a schematic diagram of the experimental results of strange samples provided by the invention.
② Model-1 transform sample testing
After 69 converted photo samples subjected to brightness darkening, horizontal overturning and rotation treatment are input, a detection result is output, a prediction labeling file is extracted to generate a first comparison file (157 targets in total), comparison is carried out on the first comparison file (151 targets in total) corresponding to a correct labeling file, and a model evaluation result is obtained through program automation evaluation, and is shown in fig. 6, the model accuracy is 0.8333, the omission ratio is 0.12, and the overall evaluation result is good.
The experimental results are shown in fig. 6, and fig. 6 is a schematic diagram of the test results of the transformation sample provided by the invention.
③ Model-1 challenge sample test
The method comprises the steps of inputting 165 countermeasure sample photos added with countermeasure patches outside a training data set, outputting detection results, extracting prediction labeling files to generate first comparison files (1086 targets in total), comparing the first comparison files (1068 targets in total) corresponding to correct labeling files, and obtaining model evaluation results through program automation evaluation, wherein the model evaluation results are shown in fig. 7, the model accuracy is 0.8998, and the omission ratio is 0.0627, so that the model is good in decision risk evaluation result based on strange sample data.
The experimental results are shown in fig. 7, and fig. 7 is a schematic diagram of the test results of the challenge sample provided by the present invention.
To sum up, 3 kinds of test sample data can be obtained, and the decision risk assessment conclusion of the Model-1 target detection Model is as follows: the evaluation conclusion of the unfamiliar samples is excellent, the evaluation conclusion of the transformation samples is good, the evaluation conclusion of the countermeasure samples is good, and the decision risk evaluation conclusion is good by comprehensively evaluating the minimum as a standard.
In summary, compared with the prior art, the invention has the following beneficial effects:
(1) The invention can start from the input end data of the model, evaluate the reliability of the unmanned platform recognition model by using 3 dimensions of strange data, transformation data and countermeasure data, and comprehensively give the decision risk evaluation result of the model whole, including the class 4 grading grades of inequality, passing grade, good grade and excellent grade according to the item evaluation result of the 3 dimensions.
(2) The invention can automatically realize the evaluation test of the model by only two python codes based on the intelligent recognition system, and flexibly modify IoU cross ratio, accuracy, omission ratio and other thresholds according to actual requirements.
(3) According to the invention, based on different types of test samples, different unmanned platforms for collecting data and different training convergence conditions, formed evaluation result statistics are summarized in the table, for example, the table below shows that decision risk evaluation capability of the target detection model to be evaluated under different working environments and data samples can be reflected comprehensively and intuitively.
Fig. 8 is a schematic structural diagram of a decision risk assessment system of a target detection model provided by the present invention, as shown in fig. 8, the system includes:
The first processing module 801 is configured to input each test sample to a target detection model to be evaluated, and output a corresponding target detection result.
And a second processing module 802, configured to obtain a corresponding prediction annotation file according to the target detection result of each test sample.
And a third processing module 803, configured to compare and analyze the key effective information in the prediction annotation file and the correct annotation file of each test sample, and determine an evaluation result of the target detection model under each test sample.
The prediction annotation file and the correct annotation file belong to target annotation files, and key effective information in the target annotation files comprises: image name information of an image where each target is located, category name information of each target and label frame position information of each target;
And a fourth processing module 804, configured to comprehensively evaluate the target detection model according to the evaluation result of the target detection model under each test sample.
It should be noted that, in the decision risk assessment device for a target detection model provided in the embodiment of the present invention, the decision risk assessment method for a target detection model described in any one of the embodiments may be executed during specific operation, which is not described in detail in this embodiment.
Fig. 9 is a schematic structural diagram of an electronic device provided by the present invention, and as shown in fig. 9, the electronic device may include: processor 910, communication interface (communications interface) 920, memory 930, and communication bus 940, wherein processor 910, communication interface 920, and memory 930 communicate with each other via communication bus 940. Processor 910 may call logic instructions in memory 930 to perform a decision risk assessment method for a target detection model, the method comprising: inputting each test sample into a target detection model to be evaluated, and outputting a corresponding target detection result; obtaining a corresponding prediction annotation file according to the target detection result of each test sample; comparing and analyzing the key effective information in the prediction labeling file and the correct labeling file of each test sample, and determining the evaluation result of the target detection model under each test sample; the prediction annotation file and the correct annotation file belong to target annotation files, and key effective information in the target annotation files comprises: image name information of an image where each target is located, category name information of each target and label frame position information of each target; and comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample.
Further, the logic instructions in the memory 930 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a read-only memory (ROM), a random access memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the decision risk assessment method of the object detection model provided by the above embodiments, the method comprising: inputting each test sample into a target detection model to be evaluated, and outputting a corresponding target detection result; obtaining a corresponding prediction annotation file according to the target detection result of each test sample; comparing and analyzing the key effective information in the prediction labeling file and the correct labeling file of each test sample, and determining the evaluation result of the target detection model under each test sample; the prediction annotation file and the correct annotation file belong to target annotation files, and key effective information in the target annotation files comprises: image name information of an image where each target is located, category name information of each target and label frame position information of each target; and comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A decision risk assessment method for a target detection model, comprising:
inputting each test sample into a target detection model to be evaluated, and outputting a corresponding target detection result;
obtaining a corresponding prediction annotation file according to the target detection result of each test sample;
Comparing and analyzing the key effective information in the prediction labeling file and the correct labeling file of each test sample, and determining the evaluation result of the target detection model under each test sample; the prediction annotation file and the correct annotation file belong to target annotation files, and key effective information in the target annotation files comprises: image name information of an image where each target is located, category name information of each target and label frame position information of each target;
And comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample.
2. The decision risk assessment method of a target detection model according to claim 1, wherein the types of test samples include: strange samples, transform samples, and challenge samples.
3. The decision risk assessment method of a target detection model according to claim 1, wherein the comparing and analyzing key effective information in the prediction annotation file and the correct annotation file of the current test sample, and determining the assessment result of the target detection model under the current test sample, includes:
Extracting key effective information in the prediction labeling file and the correct labeling file respectively to generate a corresponding first comparison file and a corresponding second comparison file;
Traversing comparison to obtain a detection result of each target in the current test sample based on the first comparison file and the second comparison file;
Determining the identification accuracy, false detection rate and omission factor of the target detection model according to the detection result of each target;
And determining an evaluation result of the target detection model under each test sample according to the identification accuracy and the omission factor.
4. The method for decision risk assessment of a target detection model according to claim 3, wherein obtaining a detection result of any target in a current test sample comprises:
Determining the position information of a marking frame of any target in the first comparison file and the second comparison file respectively;
Calculating the marking frame and cross ratio of any target according to the marking frame position information of the any target in the first comparison file and the second comparison file respectively;
Judging whether category name information of any target in the first comparison file and the second comparison file is consistent or not under the condition that the marking frame intersection ratio is larger than or equal to a preset intersection ratio threshold value;
and determining that any target is identified correctly under the condition that the category name information is consistent.
5. The decision risk assessment method of an object detection model according to claim 4, further comprising:
under the condition that the marking frame intersection ratio is smaller than a preset intersection ratio threshold value, determining that any target is missed; and
And determining that any target is identified as wrong under the condition that the category name information is inconsistent.
6. The decision risk assessment method of a target detection model according to claim 3, wherein the file types of the prediction annotation file and the correct annotation file are xml files; and the file types of the first comparison file and the second comparison file are Excel files.
7. The decision risk assessment method for a target detection model according to claim 1, wherein the comprehensively assessing the target detection model according to the assessment result of the target detection model under each test sample comprises:
acquiring a history detection record of the target detection model;
Analyzing the application scene of the target detection model based on the history detection record, and determining the evaluation weight corresponding to each evaluation result;
and determining a final comprehensive evaluation result according to each evaluation result and the evaluation weight corresponding to each evaluation result.
8. A decision risk assessment system for a target detection model, comprising:
the first processing module is used for inputting each test sample into the target detection model to be evaluated and outputting a corresponding target detection result;
The second processing module is used for obtaining a corresponding prediction annotation file according to the target detection result of each test sample;
The third processing module is used for comparing and analyzing key effective information in the prediction labeling file and the correct labeling file of each test sample and determining an evaluation result of the target detection model under each test sample; the prediction annotation file and the correct annotation file belong to target annotation files, and key effective information in the target annotation files comprises: image name information of an image where each target is located, category name information of each target and label frame position information of each target;
And the fourth processing module is used for comprehensively evaluating the target detection model according to the evaluation result of the target detection model under each test sample.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the decision risk assessment method of the object detection model according to any one of claims 1 to 7 when the computer program is executed by the processor.
10. A non-transitory computer readable storage medium, having stored thereon a computer program, which when executed by a processor, implements the steps of the decision risk assessment method of the object detection model according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311867880.8A CN117892777A (en) | 2023-12-30 | 2023-12-30 | Decision risk assessment method and system for target detection model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311867880.8A CN117892777A (en) | 2023-12-30 | 2023-12-30 | Decision risk assessment method and system for target detection model |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117892777A true CN117892777A (en) | 2024-04-16 |
Family
ID=90648232
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311867880.8A Pending CN117892777A (en) | 2023-12-30 | 2023-12-30 | Decision risk assessment method and system for target detection model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117892777A (en) |
-
2023
- 2023-12-30 CN CN202311867880.8A patent/CN117892777A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108737406B (en) | Method and system for detecting abnormal flow data | |
| CN111767707B (en) | Method, device, equipment and storage medium for detecting Leideogue cases | |
| CN107104978B (en) | Network risk early warning method based on deep learning | |
| CN110381079B (en) | Method for detecting network log abnormity by combining GRU and SVDD | |
| CN109271957B (en) | Face gender identification method and device | |
| CN105072214A (en) | C&C domain name identification method based on domain name feature | |
| CN112333128B (en) | Web attack behavior detection system based on self-encoder | |
| CN111126820A (en) | Electricity stealing prevention method and system | |
| CN112632535A (en) | Attack detection method and device, electronic equipment and storage medium | |
| CN116488915A (en) | Deep learning-based Web attack detection and classification recognition method and device | |
| CN114329455B (en) | User abnormal behavior detection method and device based on heterogeneous graph embedding | |
| CN114419346B (en) | Model robustness detection method, device, equipment and medium | |
| CN116578980A (en) | Code analysis method and device based on neural network and electronic equipment | |
| CN117692242A (en) | Network attack path analysis method based on graph analysis | |
| CN114285587A (en) | Domain name identification method and device and domain name classification model acquisition method and device | |
| CN111209567B (en) | Method and device for judging perceptibility of improving robustness of detection model | |
| CN117892777A (en) | Decision risk assessment method and system for target detection model | |
| CN114638356A (en) | Static weight guided deep neural network back door detection method and system | |
| CN113989632A (en) | Bridge detection method and device for remote sensing image, electronic equipment and storage medium | |
| CN113239075A (en) | Construction data self-checking method and system | |
| CN112632541B (en) | Method, device, computer equipment and storage medium for determining malicious degree of behavior | |
| CN116863481B (en) | Service session risk processing method based on deep learning | |
| CN109508541A (en) | A kind of credible behavior library generating method based on semantic analysis | |
| CN114374561B (en) | Network security state evaluation method, device and storable medium | |
| CN115941265B (en) | Big data attack processing method and system applied to cloud service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |