CN117496306A - Multi-level robustness evaluation method and system of machine learning target detection system - Google Patents

Multi-level robustness evaluation method and system of machine learning target detection system Download PDF

Info

Publication number
CN117496306A
CN117496306A CN202310830537.XA CN202310830537A CN117496306A CN 117496306 A CN117496306 A CN 117496306A CN 202310830537 A CN202310830537 A CN 202310830537A CN 117496306 A CN117496306 A CN 117496306A
Authority
CN
China
Prior art keywords
robustness
level
conversion
critical
detection system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310830537.XA
Other languages
Chinese (zh)
Inventor
梁哲恒
张金波
沈伍强
沈佳泉
崔磊
钱正浩
曾纪钧
周纯
裴求根
张小陆
龙震岳
周昉昉
姚潮生
李凯
张震
吴鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Power Grid Co Ltd
Information Center of Guangdong Power Grid Co Ltd
Original Assignee
Guangdong Power Grid Co Ltd
Information Center of Guangdong Power Grid Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Power Grid Co Ltd, Information Center of Guangdong Power Grid Co Ltd filed Critical Guangdong Power Grid Co Ltd
Priority to CN202310830537.XA priority Critical patent/CN117496306A/en
Publication of CN117496306A publication Critical patent/CN117496306A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776Validation; Performance evaluation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07Target detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a multi-level robustness evaluating method and system of a machine learning target detection system, wherein the method comprises the following steps: 1) Defining multi-level critical switching robustness and corresponding critical switching robustness scores for a target detection system; 2) A specific test flow of critical switching robustness; 3) The calculation method of the critical transformation robustness score and the statistics of the test result. Different levels of critical switching robustness may test and measure target detection system robustness under different stringent level requirements. The invention uses image conversion technology to simulate the input image accepted by the target detection system in different real environments, and combines the multi-level critical conversion robustness index evaluation system to cope with the robustness of different real environment condition changes under the condition of carrying out data enhancement on the original test set sample; finally, based on the guidance of the critical switching robustness, the existing target detection system can be repaired and enhanced.

Description

一种机器学习目标检测系统的多级鲁棒性评测方法及系统A multi-level robustness evaluation method and system for machine learning target detection systems

技术领域Technical field

本发明涉及一种机器学习目标检测系统的多级鲁棒性评测方法及系统,涉及软件工程与人工智能技术领域。The invention relates to a multi-level robustness evaluation method and system for a machine learning target detection system, and relates to the technical fields of software engineering and artificial intelligence.

背景技术Background technique

近年来,基于机器学习的目标检测技术飞速发展,已经被广泛应用于社会的各种不同场景之中,例如自动驾驶系统等,其给人们的生活和工作带来了极大的便利。一个基于机器学习的目标检测模型首先需要在人工标注好的训练数据集上训练从而获得预测图像中目标位置和目标类别的决策逻辑,然后再将模型部署到现实环境中,接收相应环境下的实际输入图像并进行目标检测。然而,相比于传统的软件系统,基于机器学习的目标检测系统的决策逻辑具有不可解释性,因此无法通过传统的软件测试技术,例如逻辑覆盖、静态分析等,测试目标检测系统应对各种部署环境变化时保持正确预测的鲁棒性。In recent years, target detection technology based on machine learning has developed rapidly and has been widely used in various scenarios in society, such as autonomous driving systems. It has brought great convenience to people's lives and work. A target detection model based on machine learning first needs to be trained on a manually labeled training data set to obtain the decision logic for predicting target positions and target categories in images, and then deploy the model to a real environment to receive actual data in the corresponding environment. Input images and perform object detection. However, compared with traditional software systems, the decision-making logic of target detection systems based on machine learning is uninterpretable, so it cannot pass traditional software testing technologies, such as logic coverage, static analysis, etc. Test target detection systems should cope with various deployments Robustness to maintain correct predictions when the environment changes.

在将目标检测系统部署到现实环境时,系统接收的实时输入图像可能受到不同现实环境因素的影响而与原始训练集在数据分布上存在较大差异。例如,一天中不同时刻光线强度的不同会使得输入到系统中的图像亮度差异巨大,天气为雨天时会使得输入图像模糊,摄像头的晃动与旋转会使得输入的图像的角度产生变化等,因此目标检测系统应对不同环境变化时的系统鲁棒性还有待测试。When deploying the target detection system to a real environment, the real-time input images received by the system may be affected by different real-world environmental factors and have a large difference in data distribution from the original training set. For example, the difference in light intensity at different times of the day will cause huge differences in the brightness of the image input to the system. When the weather is rainy, the input image will be blurred. The shaking and rotation of the camera will change the angle of the input image. Therefore, the target The system robustness of the detection system in response to different environmental changes has yet to be tested.

发明内容Contents of the invention

本发明的目的在于提出一种目标检测系统应对不同环境变化时的多级鲁棒性评测方法及系统,从而根据评测结果对现有目标检测系统进行修复与增强,使其能够被部署到各种不同的实际应用环境之中。本发明使用图像转换技术模拟目标检测系统在不同现实环境下所接受的输入图像,在对原始测试集样本进行数据增强的情况下,结合多级临界转换鲁棒性指标评测系统应对不同现实环境条件变化的鲁棒性;最后基于临界转换鲁棒性的指导,可以对现有目标检测系统进行修复与增强。The purpose of the present invention is to propose a multi-level robustness evaluation method and system for a target detection system to respond to changes in different environments, so as to repair and enhance the existing target detection system based on the evaluation results so that it can be deployed in various in different practical application environments. The present invention uses image conversion technology to simulate the input images accepted by the target detection system in different real-world environments. In the case of data enhancement of the original test set samples, it combines with the multi-level critical conversion robustness index evaluation system to cope with different real-world environmental conditions. Robustness to changes; finally, based on the guidance of critical transition robustness, the existing target detection system can be repaired and enhanced.

为了实现本发明的目的,本发明采用的技术方案如下:In order to achieve the purpose of the present invention, the technical solutions adopted by the present invention are as follows:

一种机器学习目标检测系统的多级鲁棒性评测方法,其步骤包括:A multi-level robustness evaluation method for machine learning target detection systems, the steps of which include:

1)对于一个待评测的目标检测系统M,将训练数据集D中的样本x输入该目标检测系统得到预测结果;所述目标检测系统M为多目标检测系统;1) For a target detection system M to be evaluated, input the sample x in the training data set D into the target detection system to obtain the prediction result; the target detection system M is a multi-target detection system;

2)针对预测结果M(x),使用转换T对该样本x进行迭代转换,每次迭代转换的递增变化量为δ,第i+1次转换时转换T的转换参数更新为θi+1=θi+δ;第i次迭代转换后的样本x为T(x;θi),将其输入该目标检测系统得到预测结果为M(T(x;θi));使用多级别的临界转换鲁棒性度量指标判断预测结果是否满足相应级别下预测结果一致性的条件,当对应级别下的临界转换鲁棒性判断M(T(x;θi))≠M(x)时,将第i-1次转换时转换T的转换参数θi-1作为样本x对于所述目标检测系统M在转换T和增量δ下对应级别的临界转换鲁棒性CTR(x;T,δ);2) For the prediction result M(x), use transformation T to iteratively transform the sample x. The incremental change of each iterative transformation is δ. At the i+1th transformation, the transformation parameter of transformation T is updated to θ i+1. = θ i + δ; the sample x converted in the i-th iteration is T(x; θ i ), which is input into the target detection system and the prediction result is M(T(x; θ i )); using multi-level The critical transition robustness measurement index determines whether the prediction result meets the consistency conditions of the prediction results at the corresponding level. When the critical transition robustness judgment at the corresponding level is M(T(x; θ i ))≠M(x), The conversion parameter θ i-1 of the conversion T at the i-1th conversion is taken as the critical conversion robustness CTR (x; T, δ of the corresponding level of the target detection system M under the conversion T and the increment δ of the sample x. );

3)计算所述目标检测系统M在转换T和增量δ下对应级别的临界转换鲁棒性均值3) Calculate the critical transformation robustness mean value of the corresponding level of the target detection system M under transformation T and increment δ

4)根据临界转换鲁棒性得分确定所述目标检测系统M在转换T和增量δ下对应级别的临界转换鲁棒性得分;其中,θmax为训练数据集D中所有样本对应级别的临界转换鲁棒性CTR(x;T,δ)的最大值,/>为训练数据集D中所有样本对应级别的临界转换鲁棒性CTR(x;T,δ)的平均值;4) Score based on critical transition robustness Determine the critical conversion robustness score of the corresponding level of the target detection system M under the conversion T and increment δ; where θ max is the critical conversion robustness CTR (x; T , the maximum value of δ),/> is the average value of the critical transformation robustness CTR (x; T, δ) of the corresponding levels of all samples in the training data set D;

5)根据各级别的临界转换鲁棒性得分确定所述目标检测系统M的临界转换鲁棒性。5) Determine the critical conversion robustness of the target detection system M according to the critical conversion robustness scores of each level.

进一步的,所述样本x为图像样本,所述多级别的临界转换鲁棒性度量指标包括图像级别的度量指标、类级别的度量指标和目标级别的度量指标;对应级别的临界转换鲁棒性CTR(x;T,δ)为图像级别的临界转换鲁棒性、类级别的临界转换鲁棒性和目标级别的临界转换鲁棒性。Further, the sample x is an image sample, and the multi-level critical transformation robustness metrics include image-level metrics, class-level metrics and target-level metrics; the corresponding level of critical transformation robustness is CTR(x; T, δ) is the critical transformation robustness at the image level, the critical transformation robustness at the class level, and the critical transformation robustness at the target level.

进一步的,对于图像级别的临界转换鲁棒性,如果第i次预测输出的目标框数量发生改变,则判定M(T(x;θi))≠M(x);对于类级别的临界转换鲁棒性,如果第i次预测输出的任一类别的目标框数量发生改变,则判定M(T(x;θi))≠M(x);对于目标级别的临界转换鲁棒性,如果第i次预测输出的任一目标框发生预测类别错误或位置偏移大于设定值,则判定M(T(x;θi))≠M(x)。Furthermore, for the critical transformation robustness at the image level, if the number of target frames in the i-th prediction output changes, it is determined that M(T(x; θ i ))≠M(x); for the critical transformation at the class level Robustness, if the number of target frames of any category in the i-th prediction output changes, it is determined that M(T(x; θ i ))≠M(x); for critical conversion robustness at the target level, if If any target frame in the i-th prediction output has a prediction category error or the position deviation is greater than the set value, then it is determined that M(T(x; θ i ))≠M(x).

进一步的,针对目标级别下临界转换鲁棒性的输出一致性判断所转化的布尔变量可满足性求解问题,其步骤包括:Further, to solve the problem of satisfiability of Boolean variables transformed from the output consistency judgment of critical transformation robustness at the target level, the steps include:

1)输入转换前原图像在目标检测系统M中的预测结果M(x)记为三元组集合Triplets={(x1,y1,d1),(x2,y2,d2),…,(xn,yn,dn)},其中,n>0,xi,yi为第i个目标框的中心点坐标,di为第i个目标框所允许的中心点最大偏移距离;1) The prediction result M(x) of the original image in the target detection system M before input conversion is recorded as a triplet set = {(x 1 , y 1 , d 1 ), (x 2 , y 2 , d 2 ), …, (x n ,y n ,d n )}, where n>0, x i , y i are the center point coordinates of the i-th target frame, and di is the maximum allowable center point of the i-th target frame offset distance;

2)输入转换后图像在目标检测系统M中的预测结果M(T(x;θi))记为点对集合Pairs={(x1,y1),(x2,y2),…,(xm,ym)},其中,m>0,xi,yi为第i个目标框的中心点坐标;2) The prediction result M (T (x; θ i )) of the input converted image in the target detection system M is recorded as the point pair set Pairs={(x 1 , y 1 ), (x 2 , y 2 ),… ,(x m ,y m )}, where m>0, x i , y i are the center point coordinates of the i-th target frame;

3)待求解的布尔变量的集合assigned_pairs初始化为空;3) The set assigned_pairs of Boolean variables to be solved is initialized to empty;

4)为转换前预测结果三元组Triplets中的每一个元素triplet,匹配一个转换后预测结果Pairs中的一个元素pair,匹配条件为Distance(triplet[x,y],pair[x,y])<=triplet[d],其中Distance为距离计算函数;将表示结果匹配的布尔变量assign_triplet_{i}_pair_{j}4) For each element triplet in the pre-conversion prediction result triplet, match an element pair in the post-conversion prediction result Pairs, and the matching condition is Distance(triplet[x,y],pair[x,y]) <=triplet[d], where Distance is the distance calculation function; assign_triplet_{i}_pair_{j}, a Boolean variable representing the result matching

加入到集合assigned_pairs中,其中i表示Triplets中第i个元素,j表示Pairs中第j个元素,assign_triplet_{i}_pair_{j}表示将Pairs中第j个元素匹配给Triplets中第i个元素;Add to the set assigned_pairs, where i represents the i-th element in Triplets, j represents the j-th element in Pairs, assign_triplet_{i}_pair_{j} represents matching the j-th element in Pairs to the i-th element in Triplets;

5)使用约束求解器对布尔变量集合assigned_pairs的可满足性进行判定。若布尔变量集合的取值可以满足(SAT),则目标级别下的输出一致性成立;否则,布尔变量的取值存在冲突不可满足(UNSAT),则目标级别下的输出一致性不成立。5) Use the constraint solver to determine the satisfiability of the Boolean variable set assigned_pairs. If the values of the Boolean variable set can satisfy (SAT), the output consistency at the target level is established; otherwise, if the values of the Boolean variables conflict and cannot be satisfied (UNSAT), the output consistency at the target level is not established.

以上步骤4)的具体细节如下:The specific details of step 4) above are as follows:

4-1)循环遍历Triplets中的每一个元素;4-1) Loop through each element in Triplets;

4-2)对于Triplets中的第i个元素triplet,遍历Pairs中的每一个元素。如果Pairs中的第j个元素pair可以匹配给triplet,则布尔变量assign_triplet_{i}_pair_{j}为真,加入到集合assigned_pairs中,此布尔变量为真所蕴含的布尔变量赋值包括Triplets中第i个元素之前的所有三元组均未匹配Pairs中的第j个点对,即列表4-2) For the i-th element triplet in Triplets, traverse every element in Pairs. If the j-th element pair in Pairs can be matched to triplet, then the Boolean variable assign_triplet_{i}_pair_{j} is true and is added to the set assigned_pairs. The Boolean variable assignment implied by this Boolean variable being true includes the i-th element in Triplets. All triples before elements do not match the j-th point pair in Pairs, that is, the list

['assign_triplet_{index}_pair_{j}'foreach index<i]中所有布尔变量的取值均为假,将其加入到集合assigned_pairs中。如果Pairs中的第j个元素pair无法匹配给triplet,则布尔变量assign_triplet_{i}_pair_{j}为假,加入到集合assigned_pairs中;The values of all Boolean variables in ['assign_triplet_{index}_pair_{j}'foreach index<i] are false, and they are added to the set assigned_pairs. If the j-th element pair in Pairs cannot be matched to triplet, the Boolean variable assign_triplet_{i}_pair_{j} is false and added to the set assigned_pairs;

4-3)对于Triplets中的每一个元素所构造的布尔变量之间,使用逻辑或算子进行组合,将其加入到约束求解器的求解约束中。4-3) Use logical OR operators to combine the Boolean variables constructed by each element in Triplets, and add them to the solving constraints of the constraint solver.

一种机器学习目标检测系统的多级鲁棒性评测系统,其特征在于,包括图像级别的临界转换鲁棒性评估模块、类级别的临界转换鲁棒性评估模块、目标级别的临界转换鲁棒性评估模块和综合评估模块;A multi-level robustness evaluation system for a machine learning target detection system, which is characterized by including an image-level critical conversion robustness evaluation module, a class-level critical conversion robustness evaluation module, and a target-level critical conversion robustness evaluation module. Sexual Assessment Module and Comprehensive Assessment Module;

图像级别的临界转换鲁棒性评估模块,用于使用图像级别的转换T1对训练数据集D中的样本x进行迭代转换,每次迭代转换的递增变化量为δ1,第i+1次转换时转换T的转换参数更新为θi+1 1=θi 11;第i次迭代转换后的样本x为T1(x;θi 1)1,将其输入到目标检测系统得到预测结果M(T1(x;θi 1))1;当转换后的输出不满足图像级别下输出一致性的要求时,即M(T1(x;θi 1))1≠M(x)1时,将第i-1次转换时转换T的转换参数θi-1 1作为样本x对于所述目标检测系统M在转换T1和增量δ1下图像级别的临界转换鲁棒性CTR(x;T11),然后计算所述目标检测系统M在转换T1和增量δ1下图像级别的临界转换鲁棒性均值 根据临界转换鲁棒性得分/>确定所述目标检测系统M在转换T1和增量δ1下图像级别的临界转换鲁棒性得分;其中,θmax为训练数据集D中所有样本的临界转换鲁棒性的最大值,/>为训练数据集D中所有样本图像级别临界转换鲁棒性的平均值;M(x)1为样本x输入该目标检测系统得到的预测结果;所述目标检测系统M为多目标检测系统;The image-level critical transformation robustness evaluation module is used to iteratively transform the sample x in the training data set D using the image-level transformation T 1 . The incremental change of each iterative transformation is δ 1 , the i+1th time During conversion, the conversion parameter of conversion T is updated to θ i+1 1 = θ i 1 + δ 1 ; the sample x converted in the i-th iteration is T 1 (x; θ i 1 ) 1 , which is input to the target detection system The prediction result M(T 1 (x; θ i 1 )) 1 is obtained; when the converted output does not meet the requirements of output consistency at the image level, that is, M(T 1 (x; θ i 1 )) 1 ≠M (x) When 1 , the conversion parameter θ i-1 1 of conversion T at the i-1th conversion is used as the critical conversion risk of the image level of the target detection system M under conversion T 1 and increment δ 1 of sample x. Rodability CTR (x; T 1 , δ 1 ), and then calculate the image-level critical transformation robustness mean of the target detection system M under transformation T 1 and increment δ 1 Score based on critical transition robustness/> Determine the critical transformation robustness score of the target detection system M at the image level under transformation T 1 and increment δ 1 ; where θ max is the maximum value of the critical transformation robustness of all samples in the training data set D, / > is the average value of the critical transformation robustness of all sample image levels in the training data set D; M(x) 1 is the prediction result obtained by inputting sample x into the target detection system; the target detection system M is a multi-target detection system;

类级别的临界转换鲁棒性评估模块,用于使用类级别的转换T2对训练数据集D中的样本x进行迭代转换,每次迭代转换的递增变化量为δ2,第i+1次转换时转换T的转换参数更新为θi+1 2=θi 22;第i次迭代转换后的样本x为T2(x;θi 2)2,将其输入到目标检测系统得到预测结果M(T2(x;θi 2))2;当转换后的输出不满足类级别下输出一致性的要求时,即M(T2(x;θi 2))2≠M(x)2时,将第i-1次转换时转换T的转换参数θi-1 2作为样本x对于所述目标检测系统M在转换T2和增量δ2下类级别的临界转换鲁棒性CTR(x;T22),然后计算所述目标检测系统M在转换T2和增量δ2下类级别的临界转换鲁棒性均值根据临界转换鲁棒性得分/>确定所述目标检测系统M在转换T2和增量δ2下类级别的临界转换鲁棒性得分;其中,/>为训练数据集D中所有样本类级别临界转换鲁棒性的平均值;M(x)2为样本x输入该目标检测系统得到的预测结果;The class-level critical transformation robustness evaluation module is used to iteratively transform the sample x in the training data set D using the class-level transformation T 2 . The incremental change of each iterative transformation is δ 2 , the i+1th time During conversion, the conversion parameter of conversion T is updated to θ i+1 2i 22 ; the sample x converted in the i-th iteration is T 2 (x; θ i 2 ) 2 , which is input to the target detection system The prediction result M(T 2 (x; θ i 2 )) 2 is obtained; when the converted output does not meet the requirements of output consistency at the class level, that is, M(T 2 (x; θ i 2 )) 2 ≠M (x) 2 , the conversion parameter θ i-1 2 of conversion T at the i-1th conversion is used as the critical conversion risk of the class level of the target detection system M under conversion T 2 and increment δ 2 of sample x. Rodability CTR (x; T 2 , δ 2 ), and then calculate the class-level critical transformation robustness mean of the target detection system M under transformation T 2 and increment δ 2 Score based on critical transition robustness/> Determine the critical transformation robustness score of the target detection system M at the class level under transformation T 2 and increment δ 2 ; where, /> is the average value of critical transformation robustness at the class level of all samples in the training data set D; M(x) 2 is the prediction result obtained by inputting sample x to the target detection system;

目标级别的临界转换鲁棒性评估模块,用于使用目标级别的转换T3对训练数据集D中的样本x进行迭代转换,每次迭代转换的递增变化量为δ3,第i+1次转换时转换T的转换参数更新为θi+1 3=θi 33;第i次迭代转换后的样本x为T3(x;θi 3)3,将其输入到目标检测系统得到预测结果M(T3(x;θi 3))3;当转换后的输出不满足目标级别下输出一致性的要求时,即M(T3(x;θi 3))3≠M(x)3时,将第i-1次转换时转换T的转换参数θi-1 3作为样本x对于所述目标检测系统M在转换T3和增量δ3下目标级别的临界转换鲁棒性CTR(x;T33);然后计算所述目标检测系统M在转换T3和增量δ3下目标级别的临界转换鲁棒性均值 根据临界转换鲁棒性得分/>确定所述目标检测系统M在转换T3和增量δ3下目标级别的临界转换鲁棒性得分;其中,/>为训练数据集D中所有样本目标级别临界转换鲁棒性的平均值;M(x)3为样本x输入该目标检测系统得到的预测结果;The target-level critical transformation robustness evaluation module is used to iteratively transform the sample x in the training data set D using the target-level transformation T 3. The incremental change of each iterative transformation is δ 3 , the i+1th time During conversion, the conversion parameter of conversion T is updated to θ i+1 3i 33 ; the sample x converted in the i-th iteration is T 3 (x; θ i 3 ) 3 , which is input to the target detection system The prediction result M(T 3 (x; θ i 3 )) 3 is obtained; when the converted output does not meet the requirements for output consistency at the target level, that is, M(T 3 (x; θ i 3 )) 3 ≠M (x) When 3 , the conversion parameter θ i-1 3 of conversion T at the i-1th conversion is used as the critical conversion risk of the target level of the target detection system M under conversion T 3 and increment δ 3 of sample x. Robustness CTR (x; T 3 , δ 3 ); then calculate the target level critical conversion robustness mean of the target detection system M under the conversion T 3 and increment δ 3 Score based on critical transition robustness/> Determine the critical conversion robustness score of the target level of the target detection system M under the conversion T 3 and the increment δ 3 ; where, /> is the average value of critical conversion robustness of all sample target levels in the training data set D; M(x) 3 is the prediction result obtained by inputting sample x to the target detection system;

进一步的,对于图像级别的临界转换鲁棒性,如果第i次预测输出的目标框数量发生改变,则判定M(T1(x;θi 1))1≠M(x)1;对于类级别的临界转换鲁棒性,如果第i次预测输出的任一类别的目标框数量发生改变,则判定M(T2(x;θi 2))2≠M(x)2;对于目标级别的临界转换鲁棒性,如果第i次预测输出的任一目标框发生预测类别错误或位置偏移大于设定值,则判定M(T3(x;θi 3))3≠M(x)3Furthermore, for the critical transformation robustness at the image level, if the number of target frames in the i-th prediction output changes, it is determined that M(T 1 (x; θ i 1 )) 1 ≠M(x) 1 ; for the class The critical conversion robustness of the level, if the number of target frames of any category in the i-th prediction output changes, it is determined that M(T 2 (x; θ i 2 )) 2 ≠M(x) 2 ; for the target level The critical transformation robustness of ) 3 .

进一步的,针对目标级别下临界转换鲁棒性的输出一致性判断所转化的布尔变量可满足性求解问题,其步骤包括:Further, to solve the problem of satisfiability of Boolean variables transformed from the output consistency judgment of critical transformation robustness at the target level, the steps include:

1)输入转换前原图像在目标检测系统M中的预测结果M(x)3记为三元组集合Triplets={(x1,y1,d1),(x2,y2,d2),…,(xn,yn,dn)},其中,n>0,xi,yi为第i个目标框的中心点坐标,di为第i个目标框所允许的中心点最大偏移距离;1) The predicted result M(x) 3 of the original image in the target detection system M before input conversion is recorded as a triplet set = {(x 1 , y 1 , d 1 ), (x 2 , y 2 , d 2 ) ,...,(x n ,y n ,d n )}, where n>0, x i , y i are the center point coordinates of the i-th target frame, and di is the allowed center point of the i-th target frame Maximum offset distance;

2)输入转换后图像在目标检测系统M中的预测结果M(T3(x;θi 3))3记为点对集合Pairs={(x1,y1),(x2,y2),…,(xm,ym)},其中,m>0,xi,yi为第i个目标框的中心点坐标;2) Input the predicted result of the converted image in the target detection system M (T 3 (x; θ i 3 )) 3, which is recorded as the point pair set Pairs={(x 1 , y 1 ), (x 2 , y 2 ),…,(x m ,y m )}, where m>0, x i , y i are the center point coordinates of the i-th target frame;

3)待求解的布尔变量的集合assigned_pairs初始化为空;3) The set assigned_pairs of Boolean variables to be solved is initialized to empty;

4)为转换前预测结果三元组Triplets中的每一个元素triplet,匹配一个转换后预测结果Pairs中的一个元素pair,匹配条件为Distance(triplet[x,y],pair[x,y])<=triplet[d],其中Distance为距离计算函数;将表示结果匹配的布尔变量assign_triplet_{i}_pair_{j}4) For each element triplet in the pre-conversion prediction result triplet, match an element pair in the post-conversion prediction result Pairs, and the matching condition is Distance(triplet[x,y],pair[x,y]) <=triplet[d], where Distance is the distance calculation function; assign_triplet_{i}_pair_{j}, a Boolean variable representing the result matching

加入到集合assigned_pairs中,其中i表示Triplets中第i个元素,j表示Pairs中第j个元素,assign_triplet_{i}_pair_{j}表示将Pairs中第j个元素匹配给Triplets中第i个元素;Add to the set assigned_pairs, where i represents the i-th element in Triplets, j represents the j-th element in Pairs, assign_triplet_{i}_pair_{j} represents matching the j-th element in Pairs to the i-th element in Triplets;

5)使用约束求解器对布尔变量集合assigned_pairs的可满足性进行判定。若布尔变量集合的取值可以满足(SAT),则目标级别下的输出一致性成立,即M(T3(x;θi 3))35) Use the constraint solver to determine the satisfiability of the Boolean variable set assigned_pairs. If the value of the Boolean variable set can satisfy (SAT), the output consistency at the target level is established, that is, M(T 3 (x; θ i 3 )) 3 =

M(x)3;否则,布尔变量的取值存在冲突不可满足(UNSAT),则目标级别下的输出一致性不成立,即M(T3(x;θi 3))3≠M(x)3M(x) 3 ; otherwise, the values of the Boolean variables are conflicting and unsatisfiable (UNSAT), and the output consistency at the target level is not established, that is, M(T 3 (x; θ i 3 )) 3 ≠M(x) 3 .

以上步骤4)的具体细节如下:The specific details of step 4) above are as follows:

4-1)循环遍历Triplets中的每一个元素;4-1) Loop through each element in Triplets;

4-2)对于Triplets中的第i个元素triplet,遍历Pairs中的每一个元素。如果Pairs中的第j个元素pair可以匹配给triplet,则布尔变量assign_triplet_{i}_pair_{j}为真,加入到集合assigned_pairs中,此布尔变量为真所蕴含的布尔变量赋值包括Triplets中第i个元素之前的所有三元组均未匹配Pairs中的第j个点对,即列表4-2) For the i-th element triplet in Triplets, traverse every element in Pairs. If the j-th element pair in Pairs can be matched to triplet, then the Boolean variable assign_triplet_{i}_pair_{j} is true and is added to the set assigned_pairs. The Boolean variable assignment implied by this Boolean variable being true includes the i-th element in Triplets. All triples before elements do not match the j-th point pair in Pairs, that is, the list

['assign_triplet_{index}_pair_{j}'foreach index<i]中所有布尔变量的取值均为假,将其加入到集合assigned_pairs中。如果Pairs中的第j个元素pair无法匹配给triplet,则布尔变量assign_triplet_{i}_pair_{j}为假,加入到集合assigned_pairs中;The values of all Boolean variables in ['assign_triplet_{index}_pair_{j}'foreach index<i] are false, and they are added to the set assigned_pairs. If the j-th element pair in Pairs cannot be matched to triplet, the Boolean variable assign_triplet_{i}_pair_{j} is false and added to the set assigned_pairs;

4-3)对于Triplets中的每一个元素所构造的布尔变量之间,使用逻辑或算子进行组合,将其加入到约束求解器的求解约束中。4-3) Use logical OR operators to combine the Boolean variables constructed by each element in Triplets, and add them to the solving constraints of the constraint solver.

在目标检测任务中,系统对于具有相同语义信息的图像应该做出相同的预测结果。基于此准则,本发明提出了一种针对目标检测系统的多级鲁棒性评测方法,通过多种输入转换技术,评估系统应对复杂环境变化时的鲁棒性。目标检测系统对于一个输入图像具有多个独立的预测目标框和所属类别,属于多目标预测系统。首先针对单目标预测系统,定义临界转换鲁棒性(CTR,Critical Transformation Robustness)指标如下。In object detection tasks, the system should make the same prediction results for images with the same semantic information. Based on this criterion, the present invention proposes a multi-level robustness evaluation method for target detection systems, which uses multiple input conversion technologies to evaluate the system's robustness in response to complex environmental changes. The target detection system has multiple independent predicted target frames and categories for an input image, and is a multi-target prediction system. First, for the single-objective prediction system, the critical transformation robustness (CTR, Critical Transformation Robustness) index is defined as follows.

单目标预测系统的临界转换鲁棒性定义:对于单目标预测的神经网络M,样本x使用转换T,迭代的递增转换参数θ,θi+1=θi+δ,θ0=0为初始值,δ>0,则在θ以变化量δ进行递增时,假设满足M(T(x;θi))≠M(x)时的最小参数值为θi,记θi-1为样本x对于模型M在转换T和增量δ下的临界转换鲁棒性CTR(x;T,δ)。Definition of critical transformation robustness of single-target prediction system: For the neural network M of single-target prediction, the sample x uses transformation T, the iterative incremental transformation parameter θ, θ i+1 = θ i + δ, θ 0 = 0 is the initial value, δ>0, then when θ increases by the change amount δ, assuming that M(T(x; θ i ))≠M(x) is satisfied, the minimum parameter value is θ i , and θ i-1 is the sample The critical transformation robustness CTR(x; T, δ) of x for model M under transformation T and increment δ.

对于一个数据集D,在转换T和增量δ下的临界转换鲁棒性定义为数据集中所有样本的临界转换鲁棒性均值,即:For a data set D, the critical transformation robustness under transformation T and increment δ is defined as the mean critical transformation robustness of all samples in the data set, that is:

均值代表了数据集D中样本的临界转换鲁棒性的整体情况,用以度量数据集对于输入转换敏感性的整体情况。基于单目标预测系统的临界转换鲁棒性定义,多级别目标检测系统的临界转换鲁棒性指标定义如下。The mean represents the overall critical transformation robustness of the samples in the data set D, and is used to measure the overall sensitivity of the data set to input transformations. Based on the definition of critical transition robustness of single target prediction system, the critical transition robustness index of multi-level target detection system is defined as follows.

目标检测系统的多级别临界转换鲁棒性定义:目标检测系统属于多目标预测系统,其同时独立地预测一张图像中的各个不同目标。针对这种特点,结合单目标预测系统的临界转换鲁棒性,分别定义多级别的目标检测系统的临界转换鲁棒性为图像级别的临界转换鲁棒性、类级别的临界转换鲁棒性以及目标级别的临界转换鲁棒性。Definition of multi-level critical transition robustness of target detection system: The target detection system is a multi-target prediction system, which independently predicts different targets in an image at the same time. In view of this characteristic, combined with the critical conversion robustness of the single target prediction system, the critical conversion robustness of the multi-level target detection system is defined as image-level critical conversion robustness, class-level critical conversion robustness and Target level critical transition robustness.

(1)图像级别的临界转换鲁棒性(1) Critical transformation robustness at image level

在此级别下,判断目标检测系统的预测结果发生改变(即,M(T(x;θi))≠M(x))的条件为:目标检测系统对某图像的预测结果中,目标框的数量发生了改变。此级别的临界转换鲁棒性满足要求为系统对于转换后的图像与转换前的图像的预测框的数量相同,实现难度简单,所需计算资源和执行时间较少,对单样本目标检测系统返回的结果进行处理的时间复杂度为O(1)。At this level, the condition for judging that the prediction result of the target detection system has changed (i.e., M(T(x; θ i ))≠M(x)) is: in the prediction result of the target detection system for an image, the target frame The number has changed. This level of critical conversion robustness satisfies the requirement that the system has the same number of prediction frames for the converted image as for the image before conversion. It is easy to implement and requires less computing resources and execution time. It returns to the single-sample target detection system. The time complexity of processing the results is O(1).

(2)类级别的临界转换鲁棒性(2) Class-level critical transition robustness

在此级别下,判断目标检测系统的预测结果发生改变(即,M(T(x;θi))≠M(x))的条件为:目标检测系统对图像的预测结果中,至少存在一个类别的目标框的数量发生了改变。此级别的临界转换鲁棒性满足要求为系统对于转换后的图像与转换前的图像的预测结果中,对于存在的多个类别的目标的预测结果,其预测框的数量相同,因此需要遍历系统的预测结果,统计不同类别目标的预测框数量。实现难度适中,所需计算资源和执行时间适中,对单样本目标检测系统返回的结果进行处理的时间复杂度为O(n)。At this level, the condition for judging that the prediction result of the target detection system has changed (i.e., M(T(x; θ i ))≠M(x)) is: among the prediction results of the target detection system for the image, there is at least one The number of target boxes for categories has changed. The critical transformation robustness of this level meets the requirement that the system has the same number of prediction boxes for the prediction results of the converted image and the pre-conversion image for the prediction results of multiple categories of targets, so it is necessary to traverse the system The prediction results, count the number of prediction boxes for different categories of targets. The implementation difficulty is moderate, the required computing resources and execution time are moderate, and the time complexity of processing the results returned by the single-sample target detection system is O(n).

(3)目标级别的临界转换鲁棒性(3) Target-level critical transition robustness

在此级别下,判断目标检测系统的预测结果发生改变(即,M(T(x;θi))≠M(x))的条件为:目标检测系统对图像的预测结果中,至少存在一个目标框的预测发生了改变,这种改变包括预测类别发生了改变或预测框的位置发生了很大的偏移。此级别的临界转换鲁棒性满足要求为系统对于转换后的图像与转换前的图像的预测结果中,对于每一个目标框,除目标框的大小和位置随转换产生必要的相应变换外,每一个目标框转换前后的预测应保持一致。在此临界转换鲁棒性的要求下,需要当每一次转换参数的递增时遍历系统的预测结果,对每一个检测框进行转换前后预测一致性的判断。实现难度大,所需计算资源和执行时间大,对单样本目标检测系统返回的结果进行处理的时间复杂度为O(n2)。本发明针对此级别的临界转换鲁棒性具体实施过程中,目标框匹配难度大,计算复杂性大的问题,提出了一种的方法,即将目标框的匹配问题建模为布尔变量的可满足性求解问题,大幅提高计算速度。目标框匹配问题所建模的布尔变量可满足性求解问题的算法如下:At this level, the condition for judging that the prediction result of the target detection system has changed (i.e., M(T(x; θ i ))≠M(x)) is: among the prediction results of the target detection system for the image, there is at least one The prediction of the target box has changed. This change includes a change in the prediction category or a large shift in the position of the prediction box. The critical transformation robustness of this level meets the requirements for the system's prediction results of the converted image and the pre-converted image. For each target frame, in addition to the necessary corresponding transformations in the size and position of the target frame with the conversion, each The predictions before and after a target box transformation should be consistent. Under the requirement of critical conversion robustness, it is necessary to traverse the prediction results of the system when each conversion parameter is incremented, and judge the prediction consistency before and after conversion for each detection frame. It is difficult to implement, requires large computing resources and execution time, and the time complexity of processing the results returned by the single-sample target detection system is O(n 2 ). In the specific implementation process of this level of critical conversion robustness, the present invention proposes a method to solve the problem of high difficulty in matching target frames and high computational complexity, which is to model the matching problem of target frames as a satisfiable Boolean variable. Solve problems efficiently and greatly improve calculation speed. The algorithm for solving the satisfiability problem of Boolean variables modeled by the target box matching problem is as follows:

算法1目标框匹配问题建模的布尔变量可满足性求解Algorithm 1 Boolean variable satisfiability solution for target frame matching problem modeling

三个级别的临界转换鲁棒性在判断转换后图像的预测结果与转换前图像的预测结果是否一致的标准上依次更加严格,即图像级别的临界转换鲁棒性最宽松,类级别的临界转换鲁棒性次之,目标级别的临界转换鲁棒性最严格。当样本在经过转换后,目标检测系统的预测不满足图像级别的临界转换鲁棒性下的预测结果一致性要求时,此时也一定不满足类级别的临界转换鲁棒性下的预测结果一致性要求。而当其不满足类级别的临界转换鲁棒性下的预测结果一致性要求时,却不一定不满足图像级别的临界转换鲁棒性结果一致性要求。目标级别的临界转换鲁棒性与类级别的临界转换鲁棒性之间的关系与此类似。三个级别的临界转换鲁棒性可以刻画目标检测系统的不同强度的鲁棒性需求,相应的临界转换鲁棒性评测方法可以衡量和区分不同目标检测系统的鲁棒性差异。The three levels of critical transformation robustness are sequentially more stringent in judging whether the prediction result of the transformed image is consistent with the prediction result of the pre-transformation image, that is, the critical transformation robustness of the image level is the loosest, and the critical transformation robustness of the class level is more stringent. Robustness is second, and critical transition robustness at the target level is the most stringent. When the sample is converted and the prediction result of the target detection system does not meet the consistency requirements of the prediction results under the critical transformation robustness of the image level, it must not also meet the consistency of the prediction results under the critical transformation robustness of the class level. sexual requirements. And when it does not meet the consistency requirements of prediction results under critical transformation robustness at the class level, it does not necessarily fail to meet the consistency requirements for critical transformation robustness results at the image level. The relationship between critical transition robustness at the target level and critical transition robustness at the class level is similar. The three levels of critical transition robustness can characterize the different intensity robustness requirements of the target detection system, and the corresponding critical transition robustness evaluation method can measure and distinguish the robustness differences of different target detection systems.

在度量了某种输入转换下目标检测系统的临界转换鲁棒性后,为了进一步准确描述在一个数据集D,输入转换T和参数增量δ下,目标检测系统的临界转鲁棒性占整个数据集上输入转换的参数可变范围的比例,本发明提出了临界转换鲁棒性得分(CTRS,CriticalTransformation Robustness Score)指标,定义如下:After measuring the critical transition robustness of the target detection system under a certain input transformation, in order to further accurately describe the critical transition robustness of the target detection system under a data set D, input transformation T and parameter increment δ, it accounts for the entire As the proportion of the variable range of input transformation parameters on the data set, the present invention proposes the Critical Transformation Robustness Score (CTRS) index, which is defined as follows:

临界转换鲁棒性得分定义:临界转换鲁棒性得分为输入转换T的临界转换鲁棒性与此转换的参数在数据集D上的可变范围的比值,即:Definition of critical transformation robustness score: The critical transformation robustness score is the ratio of the critical transformation robustness of the input transformation T to the variable range of the parameters of this transformation on the data set D, that is:

临界转换鲁棒性得分可以进一步直观地描述在一个输入转换下,模型所能承受的样本的最大改变量CTR(D;T,δ)与此转换所能带给样本的最大改变量之间的比例。临界转换鲁棒性得分指标进一步让评测系统更加完整,临界转换鲁棒性可以用于评测不同目标检测系统在同一环境改变条件下的鲁棒性强弱,而临界转换鲁棒性得分则可以用于评测同一系统对于多种不同环境改变条件下的鲁棒性强弱。The critical transformation robustness score can further intuitively describe the maximum change CTR (D; T, δ) of the sample that the model can withstand under an input transformation and the maximum change that the transformation can bring to the sample. the ratio between. The critical transition robustness score index further makes the evaluation system more complete. The critical transition robustness can be used to evaluate the robustness of different target detection systems under the same environmental change conditions, while the critical transition robustness score can be used It is used to evaluate the robustness of the same system to a variety of different environmental changes.

多级别临界转换鲁棒性在具体实施过程中,由于目标检测系统的预测结果中会同时预测目标的检测框和目标所属类别,而某些输入转换技术(例如旋转,平移等)会引起目标检测框在图像中的相对位置发生移动,甚至导致一些目标检测框移动出图像边界,这对于目标检测系统的预测结果也产生了不同程度的影响。因此,在使得目标检测框发生移动的输入转换下,判断在某级别的临界转换鲁棒性下目标检测系统的预测结果是否发生改变时,本发明提出了保守预测原则,定义如下。Multi-level critical transformation robustness During the specific implementation process, since the prediction results of the target detection system will simultaneously predict the detection frame and the category of the target, some input transformation techniques (such as rotation, translation, etc.) will cause target detection The relative position of the frame in the image moves, even causing some target detection frames to move out of the image boundary, which also has varying degrees of impact on the prediction results of the target detection system. Therefore, when judging whether the prediction result of the target detection system changes under a certain level of critical transition robustness under an input transformation that causes the target detection frame to move, the present invention proposes a conservative prediction principle, which is defined as follows.

保守预测原则定义:如果目标检测系统对于未经过输入转换的原图像中的某些预测目标框经过输入转换后,目标框的部分被移出图像边界,则对于转换后的图像,既允许该目标框被目标检测系统成功预测出,也允许该目标框被目标检测系统丢失。但其他未受影响的目标框应该被正确的检测出并且成功分类。Conservative prediction principle definition: If the target detection system undergoes input conversion for some predicted target frames in the original image that has not undergone input conversion, and part of the target frame is moved out of the image boundary, then for the converted image, the target frame is allowed Successfully predicted by the target detection system, the target frame is also allowed to be lost by the target detection system. But other unaffected object boxes should be correctly detected and successfully classified.

为了保证系统应对不同环境变化的鲁棒性能力被准确的测试,提供给目标检测系统的测试数据集D需要满足一致性与多样性条件如下。In order to ensure that the system's robustness to different environmental changes is accurately tested, the test data set D provided to the target detection system needs to meet the consistency and diversity conditions as follows.

数据一致性:对于每一张样本,目标检测系统预测的检测框个数和目标类别,与标签数据完全一致。Data consistency: For each sample, the number of detection frames and target categories predicted by the target detection system are completely consistent with the label data.

数据多样性:数据集中的图像应尽量具有多样性,即单目标,多目标,多类别,不同场景的样本。Data diversity: The images in the data set should be as diverse as possible, that is, samples of single targets, multiple targets, multiple categories, and different scenes.

临界转换鲁棒性及临界转换鲁棒性得分的具体评测流程如图1所示。首先对于待测的目标检测模型M,测试数据集D,确定输入转换T及其参数域[θminmax]与参数增量δ。初始化临界转换鲁棒性列表CTR_List为空。对于所有的未测试的样本x∈D,模型M对x的预测结果为M(x),初始化转换参数θ=θmin。样本x经过输入转换T及参数θ后生成的样本为T(x;θ),且模型对于转换后的样本的预测结果为M(T(x;θ))。以增量δ迭代地递增转换参数θ,直到在某级别的目标检测系统临界转换鲁棒性要求下,M(x)与M(T(x;θ))不一致,则对于样本x在转换T下的临界转换鲁棒性记为θ-δ。若当参数θ一直递增到θmax时,在某级别的目标检测系统临界转换鲁棒性要求下,仍未出现M(x)与M(T(x;θ))不一致的情况,则对于样本x在转换T下的临界转换鲁棒性记为θmax。将样本x的临界转换鲁棒性加入临界转换鲁棒性列表CTR_List。The specific evaluation process of critical transition robustness and critical transition robustness score is shown in Figure 1. First, for the target detection model M to be tested and the test data set D, the input transformation T and its parameter domain [θ min , θ max ] and parameter increment δ are determined. The initial critical transition robustness list CTR_List is empty. For all untested samples x∈D, the prediction result of model M for x is M(x), and the conversion parameter θ=θ min is initialized. The sample x generated after the input transformation T and parameter θ is T(x; θ), and the model’s prediction result for the converted sample is M(T(x; θ)). Iteratively increase the transformation parameter θ with an increment δ until M(x) is inconsistent with M(T(x; θ)) under the critical transformation robustness requirements of a certain level of target detection system, then for sample x after transformation T The critical transition robustness under is denoted as θ-δ. If when the parameter θ has been increased to θ max , under the critical conversion robustness requirements of a certain level of target detection system, there is still no inconsistency between M(x) and M(T(x; θ)), then for the sample The critical transformation robustness of x under transformation T is denoted by θ max . Add the critical transition robustness of sample x to the critical transition robustness list CTR_List.

最后根据目标检测系统对于某种输入转换的临界转换鲁棒性得分,可以得到目标检测系统在应对某种环境变化时的鲁棒性能力的评测结果。Finally, based on the critical conversion robustness score of the target detection system for a certain input conversion, the evaluation results of the target detection system's robustness ability in responding to certain environmental changes can be obtained.

与现有技术相比,本发明的积极效果为:Compared with the existing technology, the positive effects of the present invention are:

首先,定义了能够度量目标检测系统应对不同环境变化时的鲁棒性能力的多级别临界转换鲁棒性指标与临界转换鲁棒性得分。针对不同的鲁棒性需求,本发明设计了不同计算复杂度的多级别临界转换鲁棒性指标及临界转换鲁棒性得分,其在度量目标检测系统应对环境变化的鲁棒性能力方面准确有效,解决了目标检测系统在复杂多变环境下的鲁棒性评测问题。同时,针对目标级别的临界转换鲁棒性计算复杂度大的问题,提出了一种全新的方法,大幅提高了计算速度。最后,结合评估标准可以准确判断目标检测系统在应对各种不同的现实环境变化下的鲁棒性能力,指导目标检测系统的升级与修复。同时,此方法可以很好地扩展到其它机器学习模型测试框架下,具有良好的可扩展性。First, a multi-level critical transition robustness index and a critical transition robustness score that can measure the robustness of the target detection system in response to different environmental changes are defined. In response to different robustness requirements, the present invention designs multi-level critical transition robustness indicators and critical transition robustness scores with different computational complexity, which are accurate and effective in measuring the robustness of the target detection system in response to environmental changes. , which solves the problem of robustness evaluation of target detection systems in complex and changeable environments. At the same time, a new method is proposed to solve the problem of high computational complexity of critical transition robustness at the target level, which greatly improves the calculation speed. Finally, combined with the evaluation criteria, the robustness of the target detection system in responding to various real-world environmental changes can be accurately judged, and the upgrade and repair of the target detection system can be guided. At the same time, this method can be well extended to other machine learning model testing frameworks and has good scalability.

附图说明Description of drawings

图1是一种输入转换下目标检测系统的临界转换鲁棒性得分的测试流程图。Figure 1 is a test flow chart of the critical transition robustness score of a target detection system under input transition.

具体实施方式Detailed ways

下面结合具体实施例对本发明做进一步地详细说明,应理解这些实施例仅用于说明本发明,而不用于限制本发明的范围。在阅读了本发明之后,本领域技术人员对本发明的各种等价形式的修改均落于本申请所附权利要求所限定的范围。The present invention will be further described in detail below with reference to specific examples. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, those skilled in the art will make modifications to various equivalent forms of the present invention and all fall within the scope defined by the appended claims of this application.

本实例以YoLoV5为架构的口罩目标检测系统为例,给出多级别临界转换鲁棒性的评测过程。该目标检测系统预测一张图像中存在的人脸目标框和口罩目标框。This example takes the mask target detection system based on YoLoV5 as an example to give the evaluation process of multi-level critical transition robustness. This target detection system predicts face target frames and mask target frames that exist in an image.

1.输入转换的选择与参数设置1. Input conversion selection and parameter settings

本实例以亮度变化和旋转两种输入转换技术为例进一步详细说明测试过程,输入转换的测试参数设置如表1所示。本发明为模拟现实环境的各种不同的变化情况所采用的输入转换技术包括但不限于:This example uses the two input conversion technologies of brightness change and rotation to further explain the test process in detail. The test parameter settings of the input conversion are shown in Table 1. The input conversion technology used by the present invention to simulate various changes in the real environment includes but is not limited to:

(1)亮度变化模拟由于环境中光源强度的改变(例如每天不同时刻太阳光的强度,照明灯功率的大小的改变等),导致输入到目标检测系统中的图像数据的亮度发生改变的情况。(1) Brightness change simulates the situation where the brightness of the image data input to the target detection system changes due to changes in the intensity of the light source in the environment (such as the intensity of sunlight at different times of the day, changes in the power of lighting lamps, etc.).

(2)图像模糊化与图像质量压缩可以模拟由于环境中天气的改变(例如下雨,起雾等)或输入端设备原因,导致输入到目标检测系统中的图像数据质量下降,目标物体模糊不清的情况。(2) Image blurring and image quality compression can simulate the degradation of the image data input to the target detection system due to changes in weather in the environment (such as rain, fog, etc.) or input device problems, resulting in blurring of the target object. clear situation.

(3)改变对比度可以模拟使用不同输入设备时,由于设备的配置不同,导致输入到目标检测系统中的图像对比度不同的情况。(3) Changing the contrast can simulate the situation where the contrast of the images input to the target detection system is different due to different configurations of the devices when using different input devices.

(4)旋转可以模拟由于某些原因(例如输入设备发生倾斜等)导致输入到目标检测系统中的图像角度发生较大改变的情况。(4) Rotation can simulate the situation where the angle of the image input to the target detection system changes significantly due to certain reasons (such as tilt of the input device, etc.).

(5)平移可以模拟图像中的目标物体出现在图像中不同的相对位置的情况。(5) Translation can simulate the situation where the target object in the image appears in different relative positions in the image.

(6)水平错切可以模拟输入设备所放置的位置不同而导致输入到目标检测系统中图像的视角不同的情况。(6) Horizontal miscutting can simulate the situation where the input device is placed in a different position, resulting in a different viewing angle of the image input into the target detection system.

(7)图片缩放可以模拟目标距离输入设备的距离大小不同,从而导致目标在整个图像中所占比的大小不同的情况。(7) Image scaling can simulate the situation where the distance between the target and the input device is different, resulting in a different proportion of the target in the entire image.

表1输入转化及参数设置Table 1 Input conversion and parameter settings

输入转换参数的最大值与最小值的设置需要根据具体的输入转换技术,参数可变范围以及测试方的实际需求具体制定。本发明对于目标级别的临界转换鲁棒性标准下,目标框的位置发生较大偏移的判断的标准采用转换变换后图像的预测框中心点坐标与原图像的对应预测框的中心点经过输入转换后的映射坐标之间的距离,大于原图像对应目标框的对角线距离的一半。此判断标准可随实际系统的不同要求做出适配。The settings of the maximum and minimum values of the input conversion parameters need to be specifically formulated based on the specific input conversion technology, parameter variable range, and the actual needs of the test party. Under the critical conversion robustness standard of the target level, the present invention uses the input of the coordinates of the center point of the prediction frame of the transformed image and the center point of the corresponding prediction frame of the original image to determine whether the position of the target frame has shifted significantly. The distance between the converted mapping coordinates is greater than half of the diagonal distance of the corresponding target frame in the original image. This judgment standard can be adapted according to different requirements of the actual system.

2.临界转换鲁棒性得分的计算与测试结果统计2. Calculation of critical transition robustness score and statistics of test results

目标检测模型M在测试数据集D上,关于输入转换T的临界转换鲁棒性为所有D中样本的临界转换鲁棒性均值测试数据集D对于输入转换T的初始平均参数为θavg,则模型M在测试数据集D上关于输入转换T的临界转换鲁棒性得分记为 The target detection model M is on the test data set D. The critical transformation robustness with respect to the input transformation T is the mean of the critical transformation robustness of all samples in D. The initial average parameter of the test data set D for the input transformation T is θ avg , then the critical transformation robustness score of the model M on the test data set D with respect to the input transformation T is recorded as

在YoLoV5架构的口罩目标检测系统上,亮度变化和旋转两种输入转换在三种不同级别下的目标检测系统临界转换鲁棒性测试结果如表2,表3和表4所示。On the YoLoV5 architecture mask target detection system, the critical conversion robustness test results of the target detection system at three different levels for the two input conversions of brightness change and rotation are shown in Table 2, Table 3 and Table 4.

表2图像级别的临界转换鲁棒性Table 2 Critical transformation robustness at image level

表3类级别的临界转换鲁棒性Table 3 Critical transition robustness at class level

表4目标级别的临界转换鲁棒性Table 4 Critical transition robustness at target level

实验结果表明,三个级别的临界转换鲁棒性在判断转换后图像的预测结果与转换前图像的预测结果是否一致的标准上依次更加严格,即图像级别的临界转换鲁棒性最宽松,类级别的临界转换鲁棒性次之,图像级别的临界转换鲁棒性最严格。当样本在经过输入转换后,目标检测系统的预测不满足图像级别的临界转换鲁棒性下的预测结果一致性要求时,此时也一定不满足类级别的临界转换鲁棒性下的预测结果一致性要求。而当其不满足类级别的临界转换鲁棒性下的预测结果一致性要求时,却不一定不满足图像级别的临界转换鲁棒性结果一致性要求。如实验结果所示,类级别的临界转换鲁棒性均等于或小于图像级别的临界转换鲁棒性,其中右旋转的类级别的临界转换鲁棒性明显小于图像级别的临界转换鲁棒性,这说明图像级别仅仅关注预测结果中目标框的数量是否发生了改变,而不考虑目标框的预测类别是否一致,而类级别的临界转换鲁棒性在关注预测框的数量是否发生改变的同时,也关注目标检测系统对于检测框的预测类别是否也发生了改变。因此当图像中预测目标框的数量未发生变化而预测类别由于输入转换技术而已经发生改变时,类级别的临界转换鲁棒性可以很好的度量系统此时鲁棒性能力。但同时,类级别的临界转换鲁棒性对于测试时间的要求也要高于图像级别的临界转换鲁棒性。目标级别的临界转换鲁棒性与类级别的临界转换鲁棒性之间的关系与此类似。Experimental results show that the three levels of critical conversion robustness are sequentially more stringent in judging whether the prediction result of the converted image is consistent with the prediction result of the pre-conversion image, that is, the critical conversion robustness of the image level is the loosest, and the class The robustness of critical transformations at the image level is second, and the critical transformation robustness at the image level is the most stringent. When a sample undergoes input conversion and the prediction result of the target detection system does not meet the prediction result consistency requirements under the critical conversion robustness at the image level, the prediction results under the critical conversion robustness at the class level must not meet the prediction results at this time. Consistency requirements. And when it does not meet the consistency requirements of prediction results under critical transformation robustness at the class level, it does not necessarily fail to meet the consistency requirements for critical transformation robustness results at the image level. As shown in the experimental results, the critical transformation robustness of the class level is equal to or smaller than the critical transformation robustness of the image level. Among them, the critical transformation robustness of the right-rotated class level is significantly smaller than the critical transformation robustness of the image level. This shows that the image level only focuses on whether the number of target frames in the prediction results has changed, without considering whether the prediction categories of the target frames are consistent, while the critical transformation robustness of the class level focuses on whether the number of prediction frames has changed. Also pay attention to whether the target detection system's prediction category for the detection frame has also changed. Therefore, when the number of predicted target boxes in the image has not changed but the predicted categories have changed due to the input transformation technology, the class-level critical transformation robustness can be a good measure of the system's robustness capability at this time. But at the same time, the critical transformation robustness at the class level has higher testing time requirements than the critical transformation robustness at the image level. The relationship between critical transition robustness at the target level and critical transition robustness at the class level is similar.

综上所述,本发明提出的一种基于入转换技术的机器学习目标检测系统的鲁棒性测试方法,可以准确有效的测试目标检测系统应对不同的环境变化时的鲁棒性能力,且适用于各种机器学习模型。In summary, the invention proposes a robustness testing method for a machine learning target detection system based on input conversion technology, which can accurately and effectively test the robustness of the target detection system in response to different environmental changes, and is applicable for various machine learning models.

基于同一发明构思,本发明的另一实施例提供一种电子装置(计算机、服务器、智能手机等),其包括存储器和处理器,所述存储器存储计算机程序,所述计算机程序被配置为由所述处理器执行,所述计算机程序包括用于执行本发明方法中各步骤的指令。Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, and the computer program is configured to be The computer program includes instructions for executing each step of the method of the present invention.

基于同一发明构思,本发明的另一实施例提供一种计算机可读存储介质(如ROM/RAM、磁盘、光盘),所述计算机可读存储介质存储计算机程序,所述计算机程序被计算机执行时,实现本发明方法的各个步骤。Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM/RAM, magnetic disk, optical disk). The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, , implement each step of the method of the present invention.

本发明的其它实施方式:Other embodiments of the invention:

本发明不限定采用的输入转换技术及其相应的参数范围与参数的递增量,以及测试所需用到的数据集。The present invention does not limit the input conversion technology adopted and its corresponding parameter range and parameter increment, as well as the data set required for testing.

本发明不限定三种不同级别的目标检测临界转换鲁棒性判定的具体实现方式及具体的临界转换鲁棒性等级划分。The present invention does not limit the specific implementation of three different levels of critical transition robustness determination for target detection and the specific classification of critical transition robustness levels.

以上公开的本发明的具体实施例,其目的在于帮助理解本发明的内容并据以实施,本领域的普通技术人员可以理解,在不脱离本发明的精神和范围内,各种替换、变化和修改都是可能的。本发明不应局限于本说明书的实施例所公开的内容,本发明的保护范围以权利要求书界定的范围为准。The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. Those of ordinary skill in the art can understand that various substitutions, changes and modifications can be made without departing from the spirit and scope of the present invention. Modifications are possible. The present invention should not be limited to the contents disclosed in the embodiments of this specification. The protection scope of the present invention shall be defined by the claims.

Claims (10)

1. A multi-level robustness evaluating method of a machine learning target detection system comprises the following steps:
1) For a target detection system M to be evaluated, inputting a sample x in a training data set D into the target detection system to obtain a prediction result; the target detection system M is a multi-target detection system;
2) For the prediction result M (x), performing iterative conversion on the sample x by using a conversion T, wherein the increment change of each iterative conversion is delta, and the conversion parameter of the conversion T is updated to theta when the (i+1) th conversion is performed i+1 =θ i +δ; sample x after the ith iteration conversion is T (x; theta) i ) Is input into the target detection system to obtain a prediction result of M (T (x; theta) i ) A) is provided; judging whether the predicted result meets the condition of the predicted result consistency under the corresponding level by using the multi-level critical conversion robustness measure index, and judging M (T (x; theta) i ) When M (x) is not equal to the i-1 st conversion parameter θ of the conversion T i-1 Critical transition robustness CTR (x; T, δ) as sample x for the corresponding level of the target detection system M at transition T and increment δ;
3) Calculating a critical conversion robustness mean value of the corresponding level of the target detection system M under the conversion T and the increment delta
4) Robustness score based on critical transitionsDetermining a critical conversion robustness score of a corresponding level of the target detection system M under the conversion T and the increment delta; wherein θ max Maximum value of critical transition robustness CTR (x; T, delta) for the corresponding level of all samples in training data set D, +.>The average value of the critical conversion robustness CTR (x; T, delta) of the corresponding level of all samples in the training data set D;
5) And determining the critical switching robustness of the target detection system M according to the critical switching robustness scores of the levels.
2. The method of claim 1, wherein the sample x is an image sample, and the multi-level critical transition robustness metrics include an image level metric, a class level metric, and a target level metric; the critical switching robustness CTR (x; T, δ) of the corresponding level is the critical switching robustness of the image level, the critical switching robustness of the class level, and the critical switching robustness of the target level.
3. The method according to claim 2, wherein for the critical transition robustness at the image level, if the number of target frames of the i-th prediction output is changed, it is determined that M (T (x; θ i ) M (x); for critical transition robustness at class level, if the number of target frames of any class of the ith prediction output is changed, it is determined that M (T (x; θ) i ) M (x); for the critical transition robustness of the target level, if any target frame of the ith prediction output has a prediction type error or a positional shift greater than a set value, it is determined that M (T (x; θ) i ))≠M(x)。
4. A method according to claim 1, 2 or 3, wherein the transformation T includes, but is not limited to, an image transformation such as a luminance transformation, a rotation angle transformation, etc. on the sample x.
5. A method according to claim 3, characterized in that for a target level of critical transition robustness, M (T (x; θ i ) The method for judging whether M (x) is consistent is as follows:
1) The prediction result M (x) of the original image before input conversion in the target detection system M is recorded as a triplet set triplets= { (x) 1 ,y 1 ,d 1 ),(x 2 ,y 2 ,d 2 ),…,(x n ,y n ,d n ) N is }, where n>0,x i ,y i Is the center point coordinate of the ith target frame, d i The maximum offset distance of the center point allowed by the ith target frame;
2) The prediction result M (T (x; θ) of the converted image in the target detection system M is input i ) Denoted as point pair set Pairs = { (x) 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m ) -wherein m>0,x i ,y i The center point coordinates of the ith target frame;
3) Initializing a set of assigned_pairs of Boolean variables to be solved as null;
4) For each element triplet in the triplet triples of pre-conversion prediction results, matching an element pair in a pair of post-conversion prediction results, wherein the matching condition is Distance (triplet [ x, y ], pair [ x, y ]) < = triplet [ d ], and Distance is a Distance calculation function; adding a Boolean variable assignment_triple_i_pair_j representing result matching to the set assignment_pair, wherein i represents an ith element in the triple, j represents a jth element in the pair, and assignment_triple_i_pair_j represents matching the jth element in the pair to the ith element in the triple;
5) Determining the satisfiability of the Boolean variable set assigned_pairs by using a constraint solver; if the value of the Boolean variable set can be satisfied, the consistency of the output at the target level is established, i.e., M (T (x; θ) i ) M (x); otherwise, if the value of the Boolean variable has conflict, the output consistency under the target level is not satisfied, namely
M(T(x;θ i ))≠M(x)。
6. The multi-level robustness evaluation system of the machine learning target detection system is characterized by comprising an image-level critical transformation robustness evaluation module, a class-level critical transformation robustness evaluation module, a target-level critical transformation robustness evaluation module and a comprehensive evaluation module;
image level critical transition robustness assessment module for using image level transition T 1 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 1 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 1 =θ i 11 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 1 (x;θ i 1 ) 1 Input it to the target detection system to obtain the prediction result M (T 1 (x;θ i 1 )) 1 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at the image level, i.e. M (T 1 (x;θ i 1 )) 1 ≠M(x) 1 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 1 As sample x at transition T for the target detection system M 1 And delta 1 Critical transition robustness CTR (x; T) at lower image level 11 ) Then calculate the target detection system M at transition T 1 And delta 1 Critical transition robustness mean at lower image levelAccording to the critical switching robustness score +.> Determining that the target detection system M is in transition T 1 And delta 1 A lower image level critical transition robustness score; wherein θ max For the maximum of the critical switching robustness of all samples in the training dataset D +.>The average value of the critical conversion robustness of all sample image levels in the training data set D is obtained; m (x) 1 Inputting a prediction result obtained by the target detection system for a sample x; the target detection system M is a multi-target detection system;
a class-level critical transition robustness assessment module for using class-level transitions T 2 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 2 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 2 =θ i 22 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 2 (x;θ i 2 ) 2 Input it to the target detection system to obtain the prediction result M (T 2 (x;θ i 2 )) 2 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at class level, i.e. M (T 2 (x;θ i 2 )) 2 ≠M(x) 2 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 2 As sample x at transition T for the target detection system M 2 And delta 2 Critical transition robustness CTR (x; T) at the class-down level 22 ) Then calculate the target detection system M at transition T 2 And delta 2 Critical transition robustness at the class-down level Value ofAccording to the critical switching robustness score +.>Determining that the target detection system M is in transition T 2 And delta 2 Critical transition robustness scores at the class-down level; wherein,
the average value of the critical conversion robustness of all sample class levels in the training data set D is obtained; m (x) 2 Inputting a prediction result obtained by the target detection system for a sample x;
a target level critical transition robustness assessment module for using target level transitions T 3 Performing iterative conversion on samples x in the training data set D, wherein the incremental change of each iterative conversion is delta 3 The conversion parameter of the conversion T at the i+1th conversion is updated to θ i+1 3 =θ i 33 The method comprises the steps of carrying out a first treatment on the surface of the Sample x after the ith iterative conversion is T 3 (x;θ i 3 ) 3 Input it to the target detection system to obtain the prediction result M (T 3 (x;θ i 3 )) 3 The method comprises the steps of carrying out a first treatment on the surface of the When the converted output does not meet the requirement of output consistency at the target level, i.e. M (T 3 (x;θ i 3 )) 3 ≠M(x) 3 At that time, the conversion parameter theta of the i-1 st conversion time conversion T is calculated i-1 3 As sample x at transition T for the target detection system M 3 And delta 3 Critical transition robustness CTR (x; T) at lower target level 33 ) The method comprises the steps of carrying out a first treatment on the surface of the Then calculate the target detection system M at transition T 3 And delta 3 Critical transition robustness mean of lower target levelAccording to the critical switching robustness score +. > Determining that the target detection system M is in transition T 3 And delta 3 A critical transition robustness score for the lower target level; wherein (1)>The average value of the target level critical conversion robustness of all samples in the training data set D is obtained; m (x) 3 Inputting a prediction result obtained by the target detection system for a sample x;
the comprehensive evaluation module is used for converting the robustness score CTRS (D; T) according to the critical of each level 11 )、CTRS(D;T 22 )、CTRS(D;T 33 ) The critical switching robustness of the object detection system M is determined.
7. The system of claim 6, wherein for critical transition robustness at the image level, if the number of target frames of the ith prediction output changes, determining M (T 1 (x;θ i 1 )) 1 ≠M(x) 1 The method comprises the steps of carrying out a first treatment on the surface of the For critical transition robustness at class level, if the number of target frames of any class of the ith prediction output changes, then it is determined that M (T 2 (x;θ i 2 )) 2 ≠M(x) 2 The method comprises the steps of carrying out a first treatment on the surface of the For the critical transition robustness of the target level, if any target frame of the ith prediction output has a prediction type error or a position shift greater than a set value, it is determined that M (T 3 (x;θ i 3 )) 3 ≠M(x) 3
8. The system of claim 7, wherein for a target level of critical transition robustness, M (T 3 (x;θ i 3 )) 3 、M(x) 3 The method for judging whether the two methods are consistent is as follows:
1) Inputting the predicted result M (x) of the original image before conversion in the target detection system M 3 Recorded as triplet sets three= { (x) 1 ,y 1 ,d 1 ),(x 2 ,y 2 ,d 2 ),…,(x n ,y n ,d n ) N is }, where n>0,x i ,y i Is the center point coordinate of the ith target frame, d i The maximum offset distance of the center point allowed by the ith target frame;
2) Inputting the prediction result M (T 3 (x;θ i 3 )) 3 Denoted as point pair set Pairs = { (x) 1 ,y 1 ),(x 2 ,y 2 ),…,(x m ,y m ) -wherein m>0,x i ,y i The center point coordinates of the ith target frame;
3) Initializing a set of assigned_pairs of Boolean variables to be solved as null;
4) For each element triplet in the triplet triples of pre-conversion prediction results, matching an element pair in a pair of post-conversion prediction results, wherein the matching condition is Distance (triplet [ x, y ], pair [ x, y ]) < = triplet [ d ], and Distance is a Distance calculation function; adding a Boolean variable assignment_triple_i_pair_j representing result matching to the set assignment_pair, wherein i represents an ith element in the triple, j represents a jth element in the pair, and assignment_triple_i_pair_j represents matching the jth element in the pair to the ith element in the triple;
5) Determining the satisfiability of the Boolean variable set assigned_pairs by using a constraint solver; if the value of the Boolean variable set can be satisfied, the consistency of the output at the target level is established, namely M (T 3 (x;θ i 3 )) 3 =M(x) 3 The method comprises the steps of carrying out a first treatment on the surface of the Otherwise, if the value of the Boolean variable has conflict, the output consistency at the target level is not satisfied, namely M (T 3 (x;θ i 3 )) 3 ≠M(x) 3
The specific details of the above step 4) are as follows:
4-1) looping through each element in the Triplets;
4-2) traversing each element in the pair for the ith element triplet in the Triplets; if the jth element pair in the pair can be matched with the triplet, the Boolean variable assignment is true, the Boolean variable assignment contained in the Boolean variable assignment is false, and the Boolean variable assignment comprises that all Triplets before the ith element in the triplet are not matched with the jth point pair in the pair, namely, all Boolean variables in a list [ ' assignment_triplet { index } ' pair } ' for index < i ] are false, and the Boolean variable assignment is added into the set assigned_pair; if the jth element pair in the pair cannot be matched with the triplet, the Boolean variable assignment_triplet_ { i } -pair_ { j } is false, and the value is added into the set assignment_pair;
4-3) combining the Boolean variables constructed for each element in the Triplets using logical or operators, and adding them to the solution constraints of the constraint solver.
9. A server comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program comprising instructions for performing the steps of the method of any of claims 1 to 5.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.
CN202310830537.XA 2023-07-06 2023-07-06 Multi-level robustness evaluation method and system of machine learning target detection system Pending CN117496306A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310830537.XA CN117496306A (en) 2023-07-06 2023-07-06 Multi-level robustness evaluation method and system of machine learning target detection system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310830537.XA CN117496306A (en) 2023-07-06 2023-07-06 Multi-level robustness evaluation method and system of machine learning target detection system

Publications (1)

Publication Number Publication Date
CN117496306A true CN117496306A (en) 2024-02-02

Family

ID=89681618

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310830537.XA Pending CN117496306A (en) 2023-07-06 2023-07-06 Multi-level robustness evaluation method and system of machine learning target detection system

Country Status (1)

Country Link
CN (1) CN117496306A (en)

Similar Documents

Publication Publication Date Title
TWI742382B (en) Neural network system for vehicle parts recognition executed by computer, method for vehicle part recognition through neural network system, device and computing equipment for vehicle part recognition
CN113657202B (en) Component identification method, training set construction method, device, equipment and storage medium
WO2020030052A1 (en) Animal count identification method, device, medium, and electronic apparatus
CN114972268B (en) A method, device, electronic device and storage medium for generating defective images
CN117522174B (en) Territorial space planning space data mutation checking method, application system and cloud system
WO2024179409A9 (en) Three-dimensional industrial anomaly detection method and apparatus, storage medium, and electronic device
Ren et al. Environment influences on uncertainty of object detection for automated driving systems
Mosalam et al. Artificial intelligence in vision-based structural health monitoring
Duan et al. Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model
CN116630751B (en) Trusted target detection method integrating information bottleneck and uncertainty perception
CN118521858A (en) Aviation deep learning model robustness assessment method for challenge attack
CN117496306A (en) Multi-level robustness evaluation method and system of machine learning target detection system
CN117893490A (en) Industrial product defect detection method, device, equipment and storage medium
CN116030076A (en) Tunnel point cloud multi-target segmentation integration method and system based on causal reasoning
CN118629337B (en) Control method of display module, electronic equipment and chip
CN119169489B (en) A method for identifying insulator defects
CN119942334B (en) Road remote sensing image target detection method based on composite feature enhancement fusion module
CN118887244B (en) Multi-mode image-based multi-task defect segmentation method and system
CN118447234B (en) Imaging optimization method and system of astronomical telescope based on edge calculation
CN117853817B (en) A smart community garbage classification alarm management method based on image recognition
CN118762249A (en) Adversarial sample generation method and device for power inspection system
CN109145379A (en) A kind of building storey height figure intelligence drawing system and management method
CN119131623A (en) Small-scale target detection method, device, medium and terminal for power line inspection
CN118012768A (en) Intelligent software integrated test method
CN119600268A (en) Target detection method, system and equipment for automatic disassembly system of transformer

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB03 Change of inventor or designer information

Inventor after: Liang Zheheng

Inventor after: Zhang Xiaolu

Inventor after: Long Zhenyue

Inventor after: Zhou Fangfang

Inventor after: Yao Chaosheng

Inventor after: Li Kai

Inventor after: Zhang Zhen

Inventor after: Wu Peng

Inventor after: Zhang Jinbo

Inventor after: Shen Wuqiang

Inventor after: Shen Guiquan

Inventor after: Cui Lei

Inventor after: Qian Zhenghao

Inventor after: Zeng Jijun

Inventor after: Zhou Chun

Inventor after: Pei Qiugen

Inventor before: Liang Zheheng

Inventor before: Zhang Xiaolu

Inventor before: Long Zhenyue

Inventor before: Zhou Fangfang

Inventor before: Yao Chaosheng

Inventor before: Li Kai

Inventor before: Zhang Zhen

Inventor before: Wu Peng

Inventor before: Zhang Jinbo

Inventor before: Shen Wuqiang

Inventor before: Shen Jiaquan

Inventor before: Cui Lei

Inventor before: Qian Zhenghao

Inventor before: Zeng Jijun

Inventor before: Zhou Chun

Inventor before: Pei Qiugen

CB03 Change of inventor or designer information