CN117496306A - Multi-level robustness evaluation method and system of machine learning target detection system - Google Patents
Multi-level robustness evaluation method and system of machine learning target detection system Download PDFInfo
- Publication number
- CN117496306A CN117496306A CN202310830537.XA CN202310830537A CN117496306A CN 117496306 A CN117496306 A CN 117496306A CN 202310830537 A CN202310830537 A CN 202310830537A CN 117496306 A CN117496306 A CN 117496306A
- Authority
- CN
- China
- Prior art keywords
- robustness
- conversion
- level
- critical
- detection system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 148
- 238000011156 evaluation Methods 0.000 title claims abstract description 21
- 238000010801 machine learning Methods 0.000 title claims abstract description 14
- 238000006243 chemical reaction Methods 0.000 claims abstract description 160
- 238000000034 method Methods 0.000 claims abstract description 48
- 230000009466 transformation Effects 0.000 claims abstract description 36
- 238000004364 calculation method Methods 0.000 claims abstract description 13
- 230000007704 transition Effects 0.000 claims description 103
- 238000012549 training Methods 0.000 claims description 22
- 230000008859 change Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 4
- 238000012360 testing method Methods 0.000 abstract description 22
- 238000005516 engineering process Methods 0.000 abstract description 8
- 230000007613 environmental effect Effects 0.000 description 7
- 238000005259 measurement Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000010485 coping Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012854 evaluation process Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000013522 software testing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/776—Validation; Performance evaluation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a multi-level robustness evaluating method and system of a machine learning target detection system, wherein the method comprises the following steps: 1) Defining multi-level critical switching robustness and corresponding critical switching robustness scores for a target detection system; 2) A specific test flow of critical switching robustness; 3) The calculation method of the critical transformation robustness score and the statistics of the test result. Different levels of critical switching robustness may test and measure target detection system robustness under different stringent level requirements. The invention uses image conversion technology to simulate the input image accepted by the target detection system in different real environments, and combines the multi-level critical conversion robustness index evaluation system to cope with the robustness of different real environment condition changes under the condition of carrying out data enhancement on the original test set sample; finally, based on the guidance of the critical switching robustness, the existing target detection system can be repaired and enhanced.
Description
Technical Field
The invention relates to a multi-level robustness evaluating method and system of a machine learning target detection system, and relates to the technical fields of software engineering and artificial intelligence.
Background
In recent years, a target detection technology based on machine learning has been rapidly developed, and has been widely applied to various social scenes, such as an automatic driving system, etc., which brings great convenience to life and work of people. A target detection model based on machine learning firstly needs to be trained on a training data set marked manually so as to obtain decision logic of target positions and target categories in a predicted image, then the model is deployed into a real environment, an actual input image under a corresponding environment is received, and target detection is carried out. However, compared to conventional software systems, the decision logic of the machine-learning-based target detection system is non-interpretable, and thus cannot pass conventional software testing techniques, such as logic coverage, static analysis, etc., to test the robustness of the target detection system to maintain correct predictions against various deployment environment changes.
When the target detection system is deployed to a real environment, the real-time input image received by the system may be affected by different real-environment factors and have a large difference in data distribution from the original training set. For example, the difference of light intensities at different times of the day can cause huge difference of brightness of images input into the system, the input images can be blurred when the weather is rainy, the shake and rotation of the camera can cause the angle of the input images to change, and the like, so that the system robustness of the target detection system when the target detection system is in response to different environmental changes is still to be tested.
Disclosure of Invention
The invention aims to provide a multi-level robustness evaluating method and a multi-level robustness evaluating system for a target detection system when the target detection system is in response to different environmental changes, so that the existing target detection system is repaired and enhanced according to evaluating results, and can be deployed into various different practical application environments. The invention uses image conversion technology to simulate the input image accepted by the target detection system in different real environments, and combines the multi-level critical conversion robustness index evaluation system to cope with the robustness of different real environment condition changes under the condition of carrying out data enhancement on the original test set sample; finally, based on the guidance of the critical switching robustness, the existing target detection system can be repaired and enhanced.
In order to achieve the purpose of the invention, the technical scheme adopted by the invention is as follows:
a multi-level robustness evaluating method of a machine learning target detection system comprises the following steps:
1) For a target detection system M to be evaluated, inputting a sample x in a training data set D into the target detection system to obtain a prediction result; the target detection system M is a multi-target detection system;
2) For the prediction result M (x), performing iterative conversion on the sample x by using a conversion T, wherein the increment change of each iterative conversion is delta, and the conversion parameter of the conversion T is updated to theta when the (i+1) th conversion is performed i+1 ＝θ i +δ; sample x after the ith iteration conversion is T (x; theta) i ) Is input into the target detection system to obtain a prediction result of M (T (x; theta) i ) A) is provided; judging whether the predicted result meets the condition of the predicted result consistency under the corresponding level by using the multi-level critical conversion robustness measure index, and judging M (T (x; theta) i ) When M (x) is not equal to the i-1 st conversion parameter θ of the conversion T i-1 Critical transition robustness CTR (x; T, δ) as sample x for the corresponding level of the target detection system M at transition T and increment δ;
3) Calculating a critical conversion robustness mean value of the corresponding level of the target detection system M under the conversion T and the increment delta
4) Robustness score based on critical transitionsDetermining a critical conversion robustness score of a corresponding level of the target detection system M under the conversion T and the increment delta; wherein θ max Maximum value of critical transition robustness CTR (x; T, delta) for the corresponding level of all samples in training data set D, +.>The average value of the critical conversion robustness CTR (x; T, delta) of the corresponding level of all samples in the training data set D;
5) And determining the critical switching robustness of the target detection system M according to the critical switching robustness scores of the levels.
Further, the sample x is an image sample, and the multi-level critical transformation robustness measurement index comprises an image level measurement index, a class level measurement index and a target level measurement index; the critical switching robustness CTR (x; T, δ) of the corresponding level is the critical switching robustness of the image level, the critical switching robustness of the class level, and the critical switching robustness of the target level.
Further, for the critical transition robustness at the image level, if the number of target frames of the i-th prediction output is changed, it is determined that M (T (x; θ) i ) M (x); for critical transition robustness at class level, if the number of target frames of any class of the ith prediction output is changed, it is determined that M (T (x; θ) i ) M (x); for the critical transition robustness of the target level, if any target frame of the ith prediction output has a prediction type error or a positional shift greater than a set value, it is determined that M (T (x; θ) i ))≠M(x)。
Further, the step of judging the converted boolean variable satisfaction solving problem aiming at the output consistency of the critical transition robustness under the target level comprises the following steps:
1) The prediction result M (x) of the original image before input conversion in the target detection system M is recorded as a triplet set triplets= { (x) 1 ,y 1 ,d 1 ),(x 2 ,y 2 ,d 2 ),…,(x n ,y n ,d n ) N is }, where n>0，x i ,y i Is the center point coordinate of the ith target frame, d i The maximum offset distance of the center point allowed by the ith target frame;
2) The prediction result M (T (x; θ) of the converted image in the target detection system M is input i ) Denoted as point pair set Pairs = { (x) 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m ) -wherein m>0，x i ,y i The center point coordinates of the ith target frame;
3) Initializing a set of assigned_pairs of Boolean variables to be solved as null;
4) For each element triplet in the triplet triples of pre-conversion prediction results, matching an element pair in a pair of post-conversion prediction results, wherein the matching condition is Distance (triplet [ x, y ], pair [ x, y ]) < = triplet [ d ], and Distance is a Distance calculation function; the Boolean variable assignment_triple_i_pair_j representing the result match is used to determine the result of the matching
Adding the element into the set of assigned_pairs, wherein i represents an ith element in the triples, j represents a jth element in the triples, and assignment_triplet_ { i } -pair_ { j } represents matching the jth element in the Pairs to the ith element in the triples;
5) The satisfaction of the boolean variable set assigned _ pairs is determined using a constraint solver. If the value of the Boolean variable set can meet (SAT), the output consistency at the target level is established; otherwise, if the value of the boolean variable has conflict Unsatisfiable (UNSAT), the output consistency at the target level is not established.
The specific details of the above step 4) are as follows:
4-1) looping through each element in the Triplets;
4-2) traversing each element in the pair for the ith element triple in the Triplets. If the jth element pair in the pair can be matched to the triples, the Boolean variable assignment comprising the value of the Boolean variable that the true holds that all triples preceding the ith element in the triples do not match the jth point pair in the pair, i.e., the list, is added to the set of assigned_pairs
The values of all Boolean variables in [ (assignment_triple } -index } -pair _ { j }' foreach index < i ] are false, and are added to the set assigned_pairs. If the jth element pair in the pair cannot be matched with the triplet, the Boolean variable assignment_triplet_ { i } -pair_ { j } is false, and the value is added into the set assignment_pair;
4-3) combining the Boolean variables constructed for each element in the Triplets using logical or operators, and adding them to the solution constraints of the constraint solver.
The multi-level robustness evaluation system of the machine learning target detection system is characterized by comprising an image-level critical transformation robustness evaluation module, a class-level critical transformation robustness evaluation module, a target-level critical transformation robustness evaluation module and a comprehensive evaluation module;
Image level critical transition robustness assessment module for using image level transition T 1 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 1 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 1 ＝θ i 1 +δ 1 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 1 (x；θ i 1 ) 1 Input it to the target detection system to obtain the prediction result M (T 1 (x；θ i 1 )) 1 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at the image level, i.e. M (T 1 (x；θ i 1 )) 1 ≠M(x) 1 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 1 As sample x at transition T for the target detection system M 1 And delta 1 Critical transition robustness CTR (x; T) at lower image level 1 ,δ 1 ) Then calculate the target detection system M at transition T 1 And delta 1 Critical transition robustness mean at lower image level According to the critical switching robustness score +.>Determining that the target detection system M is in transition T 1 And delta 1 A lower image level critical transition robustness score; wherein θ max For the maximum of the critical switching robustness of all samples in the training dataset D +.>The average value of the critical conversion robustness of all sample image levels in the training data set D is obtained; m (x) 1 Inputting a prediction result obtained by the target detection system for a sample x; the target detection system M is a multi-target detection system;
A class-level critical transition robustness assessment module for using class-level transitions T 2 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 2 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 2 ＝θ i 2 +δ 2 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 2 (x；θ i 2 ) 2 Input it to the target detection system to obtain the prediction result M (T 2 (x；θ i 2 )) 2 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at class level, i.e. M (T 2 (x；θ i 2 )) 2 ≠M(x) 2 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 2 As sample x at transition T for the target detection system M 2 And delta 2 Critical transition robustness CTR (x; T) at the class-down level 2 ,δ 2 ) Then calculate the target detection system M at transition T 2 And delta 2 Critical transition robustness mean value of class-down levelAccording to the critical switching robustness score +.>Determining that the target detection system M is in transition T 2 And delta 2 Critical transition robustness scores at the class-down level; wherein (1)>The average value of the critical conversion robustness of all sample class levels in the training data set D is obtained; m (x) 2 Inputting a prediction result obtained by the target detection system for a sample x;
a target level critical transition robustness assessment module for using target level transitions T 3 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 3 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 3 ＝θ i 3 +δ 3 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 3 (x；θ i 3 ) 3 Input it to the target detection system to obtain the prediction result M (T 3 (x；θ i 3 )) 3 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at the target level, i.e. M (T 3 (x；θ i 3 )) 3 ≠M(x) 3 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 3 As sample x at transition T for the target detection system M 3 And delta 3 Critical transition robustness CTR (x; T) at lower target level 3 ,δ 3 ) The method comprises the steps of carrying out a first treatment on the surface of the Then calculate the target detection system M at transition T 3 And delta 3 Critical transition robustness mean of lower target level Robustness score based on critical transitions/>Determining that the target detection system M is in transition T 3 And delta 3 A critical transition robustness score for the lower target level; wherein (1)>The average value of the target level critical conversion robustness of all samples in the training data set D is obtained; m (x) 3 Inputting a prediction result obtained by the target detection system for a sample x;
further, for the critical transition robustness at the image level, if the number of target frames of the i-th prediction output is changed, it is determined that M (T 1 (x；θ i 1 )) 1 ≠M(x) 1 The method comprises the steps of carrying out a first treatment on the surface of the For critical transition robustness at class level, if the number of target frames of any class of the ith prediction output changes, then it is determined that M (T 2 (x；θ i 2 )) 2 ≠M(x) 2 The method comprises the steps of carrying out a first treatment on the surface of the For the critical transition robustness of the target level, if any target frame of the ith prediction output has a prediction type error or a position shift greater than a set value, it is determined that M (T 3 (x；θ i 3 )) 3 ≠M(x) 3 。
Further, the step of judging the converted boolean variable satisfaction solving problem aiming at the output consistency of the critical transition robustness under the target level comprises the following steps:
1) Inputting the predicted result M (x) of the original image before conversion in the target detection system M 3 Recorded as triplet sets three= { (x) 1 ,y 1 ,d 1 ),(x 2 ,y 2 ,d 2 ),…,(x n ,y n ,d n ) N is }, where n>0，x i ,y i Is the center point coordinate of the ith target frame, d i The maximum offset distance of the center point allowed by the ith target frame;
2) Inputting the prediction result M (T 3 (x；θ i 3 )) 3 Denoted as point pair set Pairs = { (x) 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m ) -wherein m>0，x i ,y i The center point coordinates of the ith target frame;
3) Initializing a set of assigned_pairs of Boolean variables to be solved as null;
4) For each element triplet in the triplet triples of pre-conversion prediction results, matching an element pair in a pair of post-conversion prediction results, wherein the matching condition is Distance (triplet [ x, y ], pair [ x, y ]) < = triplet [ d ], and Distance is a Distance calculation function; the Boolean variable assignment_triple_i_pair_j representing the result match is used to determine the result of the matching
Adding the element into the set of assigned_pairs, wherein i represents an ith element in the triples, j represents a jth element in the triples, and assignment_triplet_ { i } -pair_ { j } represents matching the jth element in the Pairs to the ith element in the triples;
5) The satisfaction of the boolean variable set assigned _ pairs is determined using a constraint solver. If the value of the Boolean variable set can be Satisfied (SAT), then the output consistency at the target level is true, i.e., M (T 3 (x；θ i 3 )) 3 ＝
M(x) 3 The method comprises the steps of carrying out a first treatment on the surface of the Otherwise, if there is conflict Unsatisfiable (UNSAT) in the values of the Boolean variables, the output consistency at the target level is not established, that is, M (T 3 (x；θ i 3 )) 3 ≠M(x) 3 。
The specific details of the above step 4) are as follows:
4-1) looping through each element in the Triplets;
4-2) traversing each element in the pair for the ith element triple in the Triplets. If the jth element pair in the pair can be matched to the triples, the Boolean variable assignment comprising the value of the Boolean variable that the true holds that all triples preceding the ith element in the triples do not match the jth point pair in the pair, i.e., the list, is added to the set of assigned_pairs
The values of all Boolean variables in [ (assignment_triple } -index } -pair _ { j }' foreach index < i ] are false, and are added to the set assigned_pairs. If the jth element pair in the pair cannot be matched with the triplet, the Boolean variable assignment_triplet_ { i } -pair_ { j } is false, and the value is added into the set assignment_pair;
4-3) combining the Boolean variables constructed for each element in the Triplets using logical or operators, and adding them to the solution constraints of the constraint solver.
In the object detection task, the system should make the same prediction result for images with the same semantic information. Based on the criterion, the invention provides a multi-level robustness evaluating method aiming at a target detection system, and the robustness of the system when the complex environment changes is evaluated through a plurality of input conversion technologies. The target detection system has a plurality of independent prediction target frames and belonging categories for one input image, and belongs to a multi-target prediction system. First, for a single target prediction system, a critical transition robustness (CTR, critical Transformation Robustness) index is defined as follows.
Critical transition robustness definition for single-target prediction systems: for a single-target predicted neural network M, sample x uses a transform T, iteratively increasing transform parameters θ, θ i+1 ＝θ i +δ，θ 0 =0 is the initial value, δ>0, when θ is incremented by the change δ, it is assumed that M (T (x; θ) i ) The minimum parameter value when () noteq.m (x) is θ i Record theta i-1 The critical transition robustness CTR (x; T, δ) for the model M at transition T and delta for sample x.
For one data set D, the critical switching robustness at the switching T and the delta is defined as the critical switching robustness mean for all samples in the data set, i.e.:
the mean value represents the overall case of critical transition robustness of the samples in the dataset D to measure the overall case of susceptibility of the dataset to input transitions. Based on the critical switching robustness definition of the single-target prediction system, the critical switching robustness index of the multi-level target detection system is defined as follows.
Multi-level critical transition robustness definition for target detection system: the object detection system belongs to a multi-object prediction system that simultaneously predicts each different object in one image independently. According to the characteristics, the critical switching robustness of the single-target prediction system is combined, and the critical switching robustness of the multi-level target detection system is respectively defined as the critical switching robustness of the image level, the critical switching robustness of the class level and the critical switching robustness of the target level.
(1) Critical transition robustness at image level
At this level, it is judged that the prediction result of the target detection system is changed (i.e., M (T (x; θ) i ) M (x)) is: in the prediction result of the target detection system on a certain image, the number of target frames is changed. The critical conversion robustness of the level meets the requirement that the system has the same number of prediction frames for the converted image and the image before conversion, the implementation difficulty is simple, the required calculation resources and execution time are less, and the time complexity of processing the result returned by the single-sample target detection system is O (1).
(2) Class-level critical transition robustness
At this level, it is judged that the prediction result of the target detection system is changed (i.e., M (T (x; θ) i ) M (x)) is: the number of target frames in at least one category is changed in the prediction result of the image by the target detection system. The critical conversion robustness of this level meets the requirement that the number of prediction frames of the prediction results of the system for the targets of a plurality of categories existing in the prediction results of the converted image and the image before conversion is the same, so that the prediction results of the system need to be traversed, and the number of prediction frames of the targets of different categories is counted. The implementation difficulty is moderate, the required computing resource and execution time are moderate, and the time complexity of processing the result returned by the single-sample target detection system is O (n).
(3) Target level critical transition robustness
At this level, it is judged that the prediction result of the target detection system is changed (i.e., M (T (x; θ) i ) M (x)) is: the target detection system predicts that there is a change in the prediction of at least one target frame in the prediction result of the image, including a change in the prediction category or a significant shift in the position of the prediction frame. The critical transformation robustness of this level meets the requirement that in the prediction results of the system for the transformed image and the image before transformation, for each target frame, the prediction before and after transformation of each target frame should be kept consistent except that the size and the position of the target frame generate necessary corresponding transformation along with the transformation. Under the requirement of the critical conversion robustness, the prediction result of the system needs to be traversed when the conversion parameters are increased each time, and the prediction consistency before and after conversion is judged for each detection frame. The implementation difficulty is high, the required calculation resources and the execution time are high, and the time complexity of processing the result returned by the single sample target detection system is O (n) 2 ). Aiming at the problems of high matching difficulty and high calculation complexity of the target frame in the implementation process of the critical transformation robustness of the level, the invention provides a method which models the matching problem of the target frame as a satisfiability solving problem of Boolean variables and greatly improves the calculation speed. The algorithm of the Boolean variable satisfiability solving problem modeled by the target frame matching problem is as follows:
boolean variable satisfiability solution for modeling of algorithm 1 target frame matching problem
The three levels of critical transformation robustness are sequentially stricter on the standard of judging whether the prediction result of the converted image is consistent with the prediction result of the image before conversion, namely, the critical transformation robustness of the image level is looser, the critical transformation robustness of the class level is inferior, and the critical transformation robustness of the target level is stricter. When the prediction of the target detection system does not meet the prediction result consistency requirement under the critical conversion robustness of the image level after the sample is converted, the prediction result consistency requirement under the critical conversion robustness of the class level is not necessarily met at the moment. And when the prediction result consistency requirement under the critical conversion robustness of the class level is not satisfied, the prediction result consistency requirement under the critical conversion robustness of the image level is not necessarily not satisfied. The relationship between the critical switching robustness at the target level and the critical switching robustness at the class level is similar. The three levels of critical switching robustness can be used for describing the robustness requirements of different intensities of the target detection system, and the corresponding critical switching robustness evaluation method can be used for measuring and distinguishing the robustness difference of different target detection systems.
After measuring the critical transition robustness of the target detection system under a certain input transition, in order to further accurately describe the ratio of the critical transition robustness of the target detection system to the variable range of the parameters of the input transition on the whole data set under a data set D, an input transition T and a parameter increment delta, the invention provides the index of the critical transition robustness score (CTRS, critical Transformation Robustness Score), which is defined as follows:
critical transition robustness score definition: the critical transformation robustness score is the ratio of the critical transformation robustness of the input transformation T to the variable range of the parameters of this transformation on the data set D, namely:
the critical transition robustness score may further intuitively describe the maximum change CTR (D; T, delta) of the sample that the model can withstand under an input transition and the maximum change that the transition can bring to the sampleThe ratio between them. Critical ofThe conversion robustness score index further enables the evaluation system to be more complete, the critical conversion robustness can be used for evaluating the robustness of different target detection systems under the same environment change condition, and the critical conversion robustness score can be used for evaluating the robustness of the same system under various different environment change conditions.
In the specific implementation process, the detection frames of the target and the category of the target are predicted at the same time in the prediction result of the target detection system, and certain input conversion technologies (such as rotation, translation and the like) can cause the relative positions of the target detection frames in the image to move, even cause some target detection frames to move the image boundary, which also has different degrees of influence on the prediction result of the target detection system. Therefore, under the input conversion that makes the target detection frame move, when judging whether the prediction result of the target detection system changes under a certain level of critical conversion robustness, the invention provides a conservative prediction principle, and the definition is as follows.
Conservative prediction principle definition: if the target detection system moves out of the image boundary for some predicted target frames in the original image which are not subjected to input conversion, the target frame is allowed to be successfully predicted by the target detection system and lost by the target detection system for the converted image. But other unaffected target boxes should be correctly detected and successfully classified.
In order to ensure that the robustness of the system against different environmental changes is accurately tested, the test dataset D provided to the target detection system needs to meet the consistency and diversity conditions as follows.
Data consistency: for each sample, the number of detection frames and the target category predicted by the target detection system are completely consistent with the label data.
Data diversity: the images in the dataset should be as diverse as possible, i.e. single-object, multi-category, samples of different scenes.
Critical transition robustness and critical transition robustness scoreThe body evaluation flow is shown in fig. 1. Firstly, for a target detection model M to be detected, testing a data set D, and determining an input conversion T and a parameter domain [ theta ] thereof min ,θ max ]And the parameter delta. The initialization critical switching robustness List ctr_list is empty. For all untested samples x e D, the prediction result of the model M to x is M (x), and the conversion parameter θ=θ is initialized min . The sample generated by the sample x after being input with the conversion T and the parameter theta is T (x; theta), and the prediction result of the model on the converted sample is M (T (x; theta)). The conversion parameter theta is iteratively incremented by delta until M (x) is inconsistent with M (T (x; theta)) under the requirement of the critical conversion robustness of the target detection system of a certain level, and then the critical conversion robustness of the sample x under the conversion T is marked as theta-delta. If the parameter theta is increased to theta max When the critical conversion robustness of the target detection system at a certain level is required, the critical conversion robustness of the sample x under the conversion T is marked as theta when the condition that M (x) is inconsistent with M (T (x; theta)) does not occur max . The critical switching robustness of sample x is added to the critical switching robustness List ctr_list.
And finally, according to the critical conversion robustness score of the target detection system for certain input conversion, the evaluation result of the robustness of the target detection system when coping with certain environmental changes can be obtained.
Compared with the prior art, the invention has the following positive effects:
first, a multi-level critical switching robustness index and a critical switching robustness score are defined that can measure the robustness of the target detection system to different environmental changes. Aiming at different robustness demands, the invention designs multi-level critical conversion robustness indexes and critical conversion robustness scores with different calculation complexity, which are accurate and effective in measuring the robustness of the target detection system to environmental changes, and solves the problem of robustness evaluation of the target detection system in complex and changeable environments. Meanwhile, a brand new method is provided for solving the problem of high calculation complexity of the critical switching robustness of the target level, and the calculation speed is greatly improved. Finally, the robustness of the target detection system under the condition of coping with various different real environment changes can be accurately judged by combining the evaluation standard, and the upgrading and the repairing of the target detection system are guided. Meanwhile, the method can be well expanded to other machine learning model test frameworks, and has good expandability.
Drawings
FIG. 1 is a flow chart of a test of critical transition robustness scores for an input transition target detection system.
Detailed Description
The present invention will be described in further detail with reference to the following specific examples, which are to be construed as merely illustrative, and not a limitation of the scope of the present invention. Modifications of the invention, which are various equivalents to the invention, will occur to those skilled in the art upon reading the invention, and are intended to be within the scope of the claims appended hereto.
In the embodiment, a mask target detection system with YoLoV5 as a framework is taken as an example, and a multi-level critical conversion robustness evaluation process is provided. The target detection system predicts a face target frame and a mask target frame existing in one image.
1. Selection of input transitions and parameter settings
The present example further details the testing process using two input conversion techniques, luminance change and rotation, with the test parameter settings for the input conversion shown in table 1. The present invention employs input conversion techniques for simulating various changes in a real-world environment including, but not limited to:
(1) The brightness change simulates a situation in which the brightness of the image data input into the object detection system is changed due to a change in the intensity of the light source in the environment (for example, the intensity of sunlight, the magnitude of illumination lamp power, etc. at different times of day).
(2) Image blurring and image quality compression can simulate the condition that the quality of image data input into a target detection system is reduced and a target object is blurred due to weather changes (such as rain, fog and the like) in the environment or input end equipment reasons.
(3) Changing the contrast may simulate the use of different input devices, resulting in different image contrasts being input into the object detection system due to different configurations of the devices.
(4) The rotation may simulate a situation where the angle of the image input into the object detection system changes significantly for some reason (e.g., tilting of the input device, etc.).
(5) Translation may simulate the appearance of a target object in an image at different relative positions in the image.
(6) Horizontal misclassification may simulate situations where the input device is placed differently resulting in different viewing angles of images input into the object detection system.
(7) The image scaling may simulate the situation that the distance from the target to the input device is different, resulting in different sizes of the ratio of the target in the whole image.
Table 1 input transformations and parameter settings
The setting of the maximum value and the minimum value of the input conversion parameters is required to be specifically formulated according to specific input conversion technology, parameter variable range and actual requirements of a testing party. The invention adopts the distance between the coordinate of the central point of the predicted frame of the converted image and the mapping coordinate of the central point of the corresponding predicted frame of the original image after input conversion to judge that the position of the target frame is greatly deviated under the critical conversion robustness standard of the target level, which is more than half of the diagonal distance of the original image corresponding to the target frame. This decision criterion can be adapted to the different requirements of the actual system.
2. Calculation of critical transition robustness score and test result statistics
The target detection model M is on the test data set D, and the critical transformation robustness about the input transformation T is the mean value of the critical transformation robustness of all samples in DTest dataset D for input conversionT has an initial average parameter of θ avg The model M marks the critical transition robustness score on the test dataset D with respect to the input transition T as
On the mask target detection system of the YoLoV5 architecture, the critical conversion robustness test results of the target detection system with brightness change and rotation two kinds of input conversion at three different levels are shown in tables 2, 3 and 4.
Table 2 critical transition robustness at image level
Table 3 class level critical transition robustness
Table 4 target level critical transition robustness
The experimental result shows that the three levels of critical transformation robustness are sequentially stricter on the standard of judging whether the prediction result of the converted image is consistent with the prediction result of the image before conversion, namely, the critical transformation robustness of the image level is looser, the critical transformation robustness of the class level is inferior, and the critical transformation robustness of the image level is stricter. When the prediction of the target detection system does not meet the prediction result consistency requirement under the critical conversion robustness of the image level after the sample is subjected to input conversion, the prediction result consistency requirement under the critical conversion robustness of the class level is not necessarily met at the moment. And when the prediction result consistency requirement under the critical conversion robustness of the class level is not satisfied, the prediction result consistency requirement under the critical conversion robustness of the image level is not necessarily not satisfied. As shown in the experimental results, the critical switching robustness of the class level is equal to or smaller than the critical switching robustness of the image level, wherein the critical switching robustness of the class level rotated to the right is significantly smaller than the critical switching robustness of the image level, which means that the image level only pays attention to whether the number of target frames in the prediction result is changed or not, regardless of whether the prediction categories of the target frames are consistent or not, and pays attention to whether the prediction categories of the target detection system to the detection frames are changed or not while paying attention to whether the number of the prediction frames is changed or not. Therefore, when the number of the predicted target frames in the image is unchanged and the predicted class is changed due to the input conversion technology, the critical conversion robustness of the class level can well measure the robustness of the system at the moment. At the same time, however, the critical switching robustness at the class level is also higher than the critical switching robustness at the image level for the test time. The relationship between the critical switching robustness at the target level and the critical switching robustness at the class level is similar.
In summary, the robustness testing method of the machine learning target detection system based on the input conversion technology provided by the invention can accurately and effectively test the robustness of the target detection system when the target detection system is corresponding to different environmental changes, and is suitable for various machine learning models.
Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.) comprising a memory storing a computer program configured to be executed by the processor, and a processor, the computer program comprising instructions for performing the steps in the inventive method.
Based on the same inventive concept, another embodiment of the present invention provides a computer readable storage medium (e.g., ROM/RAM, magnetic disk, optical disk) storing a computer program which, when executed by a computer, implements the steps of the inventive method.
Other embodiments of the invention:
the invention is not limited to the input conversion technique employed and its corresponding parameter ranges and incremental amounts of parameters, and the data sets required for testing.
The invention is not limited to the specific implementation manner of the target detection critical switching robustness judgment of three different levels and the specific critical switching robustness grading.
The above-disclosed embodiments of the present invention are intended to aid in understanding the contents of the present invention and to enable the same to be carried into practice, and it will be understood by those of ordinary skill in the art that various alternatives, variations and modifications are possible without departing from the spirit and scope of the invention. The invention should not be limited to what has been disclosed in the examples of the specification, but rather by the scope of the invention as defined in the claims.
Claims (10)
1. A multi-level robustness evaluating method of a machine learning target detection system comprises the following steps:
1) For a target detection system M to be evaluated, inputting a sample x in a training data set D into the target detection system to obtain a prediction result; the target detection system M is a multi-target detection system;
2) For the prediction result M (x), performing iterative conversion on the sample x by using a conversion T, wherein the increment change of each iterative conversion is delta, and the conversion parameter of the conversion T is updated to theta when the (i+1) th conversion is performed i+1 ＝θ i +δ; sample x after the ith iteration conversion is T (x; theta) i ) Is input into the target detection system to obtain a prediction result of M (T (x; theta) i ) A) is provided; judging whether the predicted result meets the condition of the predicted result consistency under the corresponding level by using the multi-level critical conversion robustness measure index, and judging M (T (x; theta) i ) When M (x) is not equal to the i-1 st conversion parameter θ of the conversion T i-1 Critical transition robustness CTR (x; T, δ) as sample x for the corresponding level of the target detection system M at transition T and increment δ;
3) Calculating a critical conversion robustness mean value of the corresponding level of the target detection system M under the conversion T and the increment delta
4) Robustness score based on critical transitionsDetermining a critical conversion robustness score of a corresponding level of the target detection system M under the conversion T and the increment delta; wherein θ max Maximum value of critical transition robustness CTR (x; T, delta) for the corresponding level of all samples in training data set D, +.>The average value of the critical conversion robustness CTR (x; T, delta) of the corresponding level of all samples in the training data set D;
5) And determining the critical switching robustness of the target detection system M according to the critical switching robustness scores of the levels.
2. The method of claim 1, wherein the sample x is an image sample, and the multi-level critical transition robustness metrics include an image level metric, a class level metric, and a target level metric; the critical switching robustness CTR (x; T, δ) of the corresponding level is the critical switching robustness of the image level, the critical switching robustness of the class level, and the critical switching robustness of the target level.
3. The method according to claim 2, wherein for the critical transition robustness at the image level, if the number of target frames of the i-th prediction output is changed, it is determined that M (T (x; θ i ) M (x); for critical transition robustness at class level, if the number of target frames of any class of the ith prediction output is changed, it is determined that M (T (x; θ) i ) M (x); for the critical transition robustness of the target level, if any target frame of the ith prediction output has a prediction type error or a positional shift greater than a set value, it is determined that M (T (x; θ) i ))≠M(x)。
4. A method according to claim 1, 2 or 3, wherein the transformation T includes, but is not limited to, an image transformation such as a luminance transformation, a rotation angle transformation, etc. on the sample x.
5. A method according to claim 3, characterized in that for a target level of critical transition robustness, M (T (x; θ i ) The method for judging whether M (x) is consistent is as follows:
1) The prediction result M (x) of the original image before input conversion in the target detection system M is recorded as a triplet set triplets= { (x) 1 ,y 1 ,d 1 ),(x 2 ,y 2 ,d 2 ),…,(x n ,y n ,d n ) N is }, where n>0，x i ,y i Is the center point coordinate of the ith target frame, d i The maximum offset distance of the center point allowed by the ith target frame;
2) The prediction result M (T (x; θ) of the converted image in the target detection system M is input i ) Denoted as point pair set Pairs = { (x) 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m ) -wherein m>0，x i ,y i The center point coordinates of the ith target frame;
3) Initializing a set of assigned_pairs of Boolean variables to be solved as null;
4) For each element triplet in the triplet triples of pre-conversion prediction results, matching an element pair in a pair of post-conversion prediction results, wherein the matching condition is Distance (triplet [ x, y ], pair [ x, y ]) < = triplet [ d ], and Distance is a Distance calculation function; adding a Boolean variable assignment_triple_i_pair_j representing result matching to the set assignment_pair, wherein i represents an ith element in the triple, j represents a jth element in the pair, and assignment_triple_i_pair_j represents matching the jth element in the pair to the ith element in the triple;
5) Determining the satisfiability of the Boolean variable set assigned_pairs by using a constraint solver; if the value of the Boolean variable set can be satisfied, the consistency of the output at the target level is established, i.e., M (T (x; θ) i ) M (x); otherwise, if the value of the Boolean variable has conflict, the output consistency under the target level is not satisfied, namely
M(T(x；θ i ))≠M(x)。
6. The multi-level robustness evaluation system of the machine learning target detection system is characterized by comprising an image-level critical transformation robustness evaluation module, a class-level critical transformation robustness evaluation module, a target-level critical transformation robustness evaluation module and a comprehensive evaluation module;
image level critical transition robustness assessment module for using image level transition T 1 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 1 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 1 ＝θ i 1 +δ 1 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 1 (x；θ i 1 ) 1 Input it to the target detection system to obtain the prediction result M (T 1 (x；θ i 1 )) 1 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at the image level, i.e. M (T 1 (x；θ i 1 )) 1 ≠M(x) 1 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 1 As sample x at transition T for the target detection system M 1 And delta 1 Critical transition robustness CTR (x; T) at lower image level 1 ,δ 1 ) Then calculate the target detection system M at transition T 1 And delta 1 Critical transition robustness mean at lower image levelAccording to the critical switching robustness score +.> Determining that the target detection system M is in transition T 1 And delta 1 A lower image level critical transition robustness score; wherein θ max For the maximum of the critical switching robustness of all samples in the training dataset D +.>The average value of the critical conversion robustness of all sample image levels in the training data set D is obtained; m (x) 1 Inputting a prediction result obtained by the target detection system for a sample x; the target detection system M is a multi-target detection system;
a class-level critical transition robustness assessment module for using class-level transitions T 2 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 2 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 2 ＝θ i 2 +δ 2 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 2 (x；θ i 2 ) 2 Input it to the target detection system to obtain the prediction result M (T 2 (x；θ i 2 )) 2 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at class level, i.e. M (T 2 (x；θ i 2 )) 2 ≠M(x) 2 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 2 As sample x at transition T for the target detection system M 2 And delta 2 Critical transition robustness CTR (x; T) at the class-down level 2 ,δ 2 ) Then calculate the target detection system M at transition T 2 And delta 2 Critical transition robustness at the class-down level Value ofAccording to the critical switching robustness score +.>Determining that the target detection system M is in transition T 2 And delta 2 Critical transition robustness scores at the class-down level; wherein,
the average value of the critical conversion robustness of all sample class levels in the training data set D is obtained; m (x) 2 Inputting a prediction result obtained by the target detection system for a sample x;
a target level critical transition robustness assessment module for using target level transitions T 3 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 3 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 3 ＝θ i 3 +δ 3 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 3 (x；θ i 3 ) 3 Input it to the target detection system to obtain the prediction result M (T 3 (x；θ i 3 )) 3 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at the target level, i.e. M (T 3 (x；θ i 3 )) 3 ≠M(x) 3 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 3 As sample x at transition T for the target detection system M 3 And delta 3 Critical transition robustness CTR (x; T) at lower target level 3 ,δ 3 ) The method comprises the steps of carrying out a first treatment on the surface of the Then calculate the target detection system M at transition T 3 And delta 3 Critical transition robustness mean of lower target levelAccording to the critical switching robustness score +. > Determining that the target detection system M is in transition T 3 And delta 3 A critical transition robustness score for the lower target level; wherein (1)>The average value of the target level critical conversion robustness of all samples in the training data set D is obtained; m (x) 3 Inputting a prediction result obtained by the target detection system for a sample x;
the comprehensive evaluation module is used for converting the robustness score CTRS (D; T) according to the critical of each level 1 ,δ 1 )、CTRS(D；T 2 ,δ 2 )、CTRS(D；T 3 ,δ 3 ) The critical switching robustness of the object detection system M is determined.
7. The system of claim 6, wherein for critical transition robustness at the image level, if the number of target frames of the ith prediction output changes, determining M (T 1 (x；θ i 1 )) 1 ≠M(x) 1 The method comprises the steps of carrying out a first treatment on the surface of the For critical transition robustness at class level, if the number of target frames of any class of the ith prediction output changes, then it is determined that M (T 2 (x；θ i 2 )) 2 ≠M(x) 2 The method comprises the steps of carrying out a first treatment on the surface of the For the critical transition robustness of the target level, if any target frame of the ith prediction output has a prediction type error or a position shift greater than a set value, it is determined that M (T 3 (x；θ i 3 )) 3 ≠M(x) 3 。
8. The system of claim 7, wherein for a target level of critical transition robustness, M (T 3 (x；θ i 3 )) 3 、M(x) 3 The method for judging whether the two methods are consistent is as follows:
1) Inputting the predicted result M (x) of the original image before conversion in the target detection system M 3 Recorded as triplet sets three= { (x) 1 ,y 1 ,d 1 ),(x 2 ,y 2 ,d 2 ),…,(x n ,y n ,d n ) N is }, where n>0，x i ,y i Is the center point coordinate of the ith target frame, d i The maximum offset distance of the center point allowed by the ith target frame;
2) Inputting the prediction result M (T 3 (x；θ i 3 )) 3 Denoted as point pair set Pairs = { (x) 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m ) -wherein m>0，x i ,y i The center point coordinates of the ith target frame;
3) Initializing a set of assigned_pairs of Boolean variables to be solved as null;
4) For each element triplet in the triplet triples of pre-conversion prediction results, matching an element pair in a pair of post-conversion prediction results, wherein the matching condition is Distance (triplet [ x, y ], pair [ x, y ]) < = triplet [ d ], and Distance is a Distance calculation function; adding a Boolean variable assignment_triple_i_pair_j representing result matching to the set assignment_pair, wherein i represents an ith element in the triple, j represents a jth element in the pair, and assignment_triple_i_pair_j represents matching the jth element in the pair to the ith element in the triple;
5) Determining the satisfiability of the Boolean variable set assigned_pairs by using a constraint solver; if the value of the Boolean variable set can be satisfied, the consistency of the output at the target level is established, namely M (T 3 (x；θ i 3 )) 3 ＝M(x) 3 The method comprises the steps of carrying out a first treatment on the surface of the Otherwise, if the value of the Boolean variable has conflict, the output consistency at the target level is not satisfied, namely M (T 3 (x；θ i 3 )) 3 ≠M(x) 3 ；
The specific details of the above step 4) are as follows:
4-1) looping through each element in the Triplets;
4-2) traversing each element in the pair for the ith element triplet in the Triplets; if the jth element pair in the pair can be matched with the triplet, the Boolean variable assignment is true, the Boolean variable assignment contained in the Boolean variable assignment is false, and the Boolean variable assignment comprises that all Triplets before the ith element in the triplet are not matched with the jth point pair in the pair, namely, all Boolean variables in a list [ ' assignment_triplet { index } ' pair } ' for index < i ] are false, and the Boolean variable assignment is added into the set assigned_pair; if the jth element pair in the pair cannot be matched with the triplet, the Boolean variable assignment_triplet_ { i } -pair_ { j } is false, and the value is added into the set assignment_pair;
4-3) combining the Boolean variables constructed for each element in the Triplets using logical or operators, and adding them to the solution constraints of the constraint solver.
9. A server comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program comprising instructions for performing the steps of the method of any of claims 1 to 5.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310830537.XA CN117496306A (en) | 2023-07-06 | 2023-07-06 | Multi-level robustness evaluation method and system of machine learning target detection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310830537.XA CN117496306A (en) | 2023-07-06 | 2023-07-06 | Multi-level robustness evaluation method and system of machine learning target detection system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117496306A true CN117496306A (en) | 2024-02-02 |
Family
ID=89681618
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310830537.XA Pending CN117496306A (en) | 2023-07-06 | 2023-07-06 | Multi-level robustness evaluation method and system of machine learning target detection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117496306A (en) |
-
2023
- 2023-07-06 CN CN202310830537.XA patent/CN117496306A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| TWI742382B (en) | Neural network system for vehicle parts recognition executed by computer, method for vehicle part recognition through neural network system, device and computing equipment for vehicle part recognition | |
| CN110222831B (en) | Robustness evaluation method and device of deep learning model and storage medium | |
| CN108875600A (en) | A kind of information of vehicles detection and tracking method, apparatus and computer storage medium based on YOLO | |
| CN111126134B (en) | Radar radiation source deep learning identification method based on non-fingerprint signal eliminator | |
| CN109711283A (en) | A kind of joint doubledictionary and error matrix block Expression Recognition algorithm | |
| CN111191629A (en) | Multi-target-based image visibility detection method | |
| CN111598182A (en) | Method, apparatus, device and medium for training neural network and image recognition | |
| CN110826581B (en) | Animal number identification method, device, medium and electronic equipment | |
| CN113298152A (en) | Model training method and device, terminal equipment and computer readable storage medium | |
| CN111429481B (en) | Target tracking method, device and terminal based on adaptive expression | |
| WO2020109774A1 (en) | Verification of perception systems | |
| Ren et al. | Environment influences on uncertainty of object detection for automated driving systems | |
| Cruz et al. | Closing the simulation-to-reality gap using generative neural networks: Training object detectors for soccer robotics in simulation as a case study | |
| CN113505886A (en) | Countermeasure sample generation method, system, terminal and medium based on fuzzy test | |
| Yan et al. | ARTDL: Adaptive random testing for deep learning systems | |
| Zhang et al. | Generalizing universal adversarial perturbations for deep neural networks | |
| Jaganeshwari et al. | an Automated Testing Tool Based on Graphical User Interface With Exploratory Behavioural Analysis | |
| CN117496306A (en) | Multi-level robustness evaluation method and system of machine learning target detection system | |
| Moskalenko et al. | TETRIS: Towards Exploring the Robustness of Interactive Segmentation | |
| CN114359667A (en) | Strength coherent identification method and equipment based on generating type countermeasure network | |
| Tavares et al. | Co-design System for Tracking Targets Using Template Matching | |
| Nazarkevych et al. | Method of Dataset Filling and Recognition of Moving Objects in Video Sequences based on YOLO. | |
| CN118447234B (en) | Imaging optimization method and system of astronomical telescope based on edge calculation | |
| CN118037738B (en) | Asphalt pavement crack pouring adhesive bonding performance detection method and equipment | |
| CN117407319B (en) | Target generation method and device for radar system software test and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |