WO2023207064A1 - 基于弱光补偿的MaskRCNN渗水检测方法及系统 - Google Patents

基于弱光补偿的MaskRCNN渗水检测方法及系统 Download PDF

Info

Publication number
WO2023207064A1
WO2023207064A1 PCT/CN2022/134451 CN2022134451W WO2023207064A1 WO 2023207064 A1 WO2023207064 A1 WO 2023207064A1 CN 2022134451 W CN2022134451 W CN 2022134451W WO 2023207064 A1 WO2023207064 A1 WO 2023207064A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
maskrcnn
water seepage
layer
module
Prior art date
Application number
PCT/CN2022/134451
Other languages
English (en)
French (fr)
Inventor
周玉权
周万竣
欧阳济凡
黄祖良
刘欣
王劲
蔡喜昌
翁正
林杰胜
许晓萌
冯文嵛
赵少华
Original Assignee
清远蓄能发电有限公司
南方电网调峰调频(广东)储能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清远蓄能发电有限公司, 南方电网调峰调频(广东)储能科技有限公司 filed Critical 清远蓄能发电有限公司
Publication of WO2023207064A1 publication Critical patent/WO2023207064A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G06T5/75—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
    • Y02E10/00—Energy generation through renewable energy sources
    • Y02E10/20—Hydro energy

Definitions

  • the invention relates to the technical field of optimization of identification algorithms for inspection robots in complex indoor environments, and in particular to the MaskRCNN water seepage detection method and system based on weak light compensation.
  • the present invention provides a MaskRCNN water seepage detection method and system based on low light compensation, by importing the inspection image to be detected into the MBLLEN model for low light enhancement, and then importing the low light enhanced image.
  • the MaskRCNN model detects water accumulation areas, which can not only perform effective target detection, but also achieve accurate segmentation of the boundaries of the target area.
  • the method of the present invention adopts the following technical solution to implement: MaskRCNN water penetration detection method based on low light compensation, including the following steps:
  • the system of the present invention adopts the following technical solutions to realize: MaskRCNN water penetration detection system based on low light compensation, including:
  • Fusion sample data enhancement module used to combine the captured water images with the water seepage images collected online, and use fused sample data to enhance and expand the sample data set;
  • Data annotation module Used to annotate the amplified data set using Lableme and generate label files for water accumulation areas;
  • Enhancement operation module used to enhance the marked data set, flip, zoom, and change the color gamut of the image. After the operation is completed, the image will be restored to the original image pixel size;
  • MaskRCNN model training module Use the enhanced data set to train the MaskRCNN model
  • Low-light enhanced water seepage area image acquisition module Import the image to be detected into the MBLLEN model, obtain the feature maps of each level through the feature extraction module FEM layer, and obtain the feature map of each layer after low-light enhancement through the EM layer of the enhancement module. Input the enhanced feature map into the FM layer of the fusion module to obtain the final low-light enhanced water seepage area image;
  • Water seepage area position acquisition module Import the final output low-light enhanced water seepage image into the MaskRCNN model.
  • the water seepage image is calculated through convolution to obtain the feature map, the candidate area is obtained through RPN, and the final water seepage area position is obtained through the ROI Align layer.
  • the present invention has the following advantages and beneficial effects:
  • the present invention can not only perform effective target detection, but also achieve accurate segmentation of the target area. border.
  • Figure 1 is a flow chart of the method of the present invention
  • Figure 2(a) is a schematic diagram of capturing water accumulation images according to the present invention.
  • Figure 2(b) is a schematic diagram of images of accumulated water and seepage collected online
  • Figure 2(c) is a schematic diagram of fusion sample data enhancement according to the present invention.
  • Figure 3 is a schematic diagram of the image of the water accumulation area marked by Labelme software
  • Figure 4 is a schematic structural diagram of the MaskRCNN model
  • Figure 5 is a schematic diagram of the MBLLEN model structure
  • Figure 6(a) is a low-light image
  • Figure 6(b) is the enhanced image.
  • this embodiment uses the MaskRCNN water penetration detection method based on low light compensation, including the following steps:
  • step S1 the specific process of fusion sample data enhancement in step S1 is as follows:
  • the formula for fusion is as follows:
  • Image(R,G,B) n ⁇ Image1(R,G,B)+(1-n)Image2(R,G,B)
  • Image(R,G,B) is the fusion image
  • Image1(R,G,B) and Image2(R,G,B) are the two original images
  • R, G, and B are the three channels of the image.
  • step S2 the specific process of data annotation in step S2 is as follows:
  • Use multi-line segments and multi-point methods to mark the water seepage area use Labelme marking tool to mark the polygonal outline of the water seepage area, set the label name of the water seepage outline, and generate corresponding json files for the marked samples.
  • the json format files are stored in the samples. Contour and image information of the target area.
  • MaskRCNN is an instance segmentation framework, used for effective target detection and accurate segmentation of the boundaries of the target area.
  • the MaskRCNN model is mainly composed of the feature extraction framework ResNet and RPN modules; ResNet uses a multi-layer convolution structure to extract features of the image to be detected, and RPN is used to generate multiple ROI areas; MaskRCNN uses the RoI Align layer instead of RoI Pooling, and uses double lines sexual interpolation maps the multiple ROI feature areas generated by RPN to a unified 7*7 size; finally, the multiple ROI areas generated by the RPN layer are classified and the regression operation of the positioning frame is performed, and the fully convolutional neural network FCN is used to generate the water seepage area The corresponding Mask.
  • the loss function Loss of MaskRCNN is defined as:
  • L cls is the classification error
  • L box is the error caused by the positioning box
  • L mask is the error caused by the mask Mask
  • L cls Introducing log-likelihood loss (Log-likelihood Loss) to construct the classification error L cls , the calculation formula of L cls is as follows:
  • this embodiment introduces the pre-trained weights (mask_rcnn_coco.h5) on the COCO data set for fine tuning.
  • mask_rcnn_coco.h5 the pre-trained weights
  • the MBLLEN model is a multi-layer feature low-light enhancement deep learning network model. Image features of different layers are extracted through convolution calculations, and the feature maps of different layers are input into multiple sub-networks for enhancement. .
  • the MBLLEN model mainly includes feature extraction module (Feature Extraction Module, FEM), enhancement module (Enhancement Module, EM), and fusion module (Fusion Module, FM).
  • FEM feature extraction module
  • the feature extraction module FEM consists of a unidirectional 10-layer network structure, 32 3 ⁇ 3 convolution kernels, a convolution step of 1, and a ReLU activation function.
  • the feature extraction module does not use pooling layers; the output of each layer is simultaneously It is the input of the convolutional layer of the next feature extraction module FEM and the input of the corresponding convolutional layer of the enhancement module EM. Since the feature extraction module FEM contains 10 feature extraction layers, the enhancement module EM contains 10 sub-network structures with the same structure. The EM layer sub-network structure is all 1 convolution layer, 3 convolution layers and 3 deconvolution layers. The fusion module FM fuses all images output from the EM subnet and uses 3-channel 1 ⁇ 1 convolution kernel convolution to obtain the final enhanced result.
  • structure loss (Str)
  • VGG pre-trained VGG content loss
  • region loss (Region loss)
  • L SSIM is the structural similarity between the enhanced image and the real image
  • L MS-SSIM is the multi-level structural similarity
  • Pre-trained VGG content loss minimizes the absolute difference between the enhanced image and the real image output by the pre-trained VGG-19 network.
  • the loss function formula is as follows:
  • E and G are the enhanced image and the real image respectively
  • W i,j , H i,j , C i,j respectively represent the dimensions of the pre-trained VGG feature map
  • ⁇ i,j represents the jth node of the VGG-19 network Convolutional layer, i-th feature map
  • x, y, z represent the width, height, and number of channels of the feature map respectively;
  • E L and G L are the low-light areas of the enhanced image and the real image respectively
  • E H and G H are the non-low light areas of the enhanced image and the real image respectively
  • w L and w H are 4 and 1 respectively
  • m L is the width of the G L image
  • n L is the height of the G L image
  • m H is the width of the G H image
  • n H is the height of the G H image.
  • the low-light data set is synthesized on the basis of the PASCAL VOC data set, and Gamma correction and Poisson noise with a Peak value of 200 are added as the low-light input image, and the original image is used as the real image.
  • the enhanced image experimental results of low-light water seepage images are shown in Figure 6(a) and Figure 6(b).
  • the present invention proposes a MaskRCNN water penetration detection system based on low light compensation, including:
  • Fusion sample data enhancement module used to combine the captured water images with the water seepage images collected online, and use fused sample data to enhance and expand the sample data set;
  • Data annotation module Used to annotate the amplified data set using Lableme and generate label files for water accumulation areas;
  • Enhancement operation module used to enhance the marked data set, flip, zoom, and change the color gamut of the image. After the operation is completed, the image will be restored to the original image pixel size;
  • MaskRCNN model training module Use the enhanced data set to train the MaskRCNN model
  • Low-light enhanced water seepage area image acquisition module Import the image to be detected into the MBLLEN model, obtain the feature maps of each level through the feature extraction module FEM layer, and obtain the feature map of each layer after low-light enhancement through the EM layer of the enhancement module. Input the enhanced feature map into the FM layer of the fusion module to obtain the final low-light enhanced water seepage area image;
  • Water seepage area position acquisition module Import the final output low-light enhanced water seepage image into the MaskRCNN model.
  • the water seepage image is calculated through convolution to obtain the feature map, the candidate area is obtained through RPN, and the final water seepage area position is obtained through the ROI Align layer.

Abstract

本发明涉及基于弱光补偿的MaskRCNN渗水检测方法及系统，其方法包括：S1、采用融合样本数据增强扩充样本数据集；S2、对扩增后的数据集用Lableme进行数据标注，生成积水区域的标记文件；S3、对标记的数据集进行增强操作；S4、利用增强的数据集训练MaskRCNN模型；S5、将待检测图像导入MBLLEN模型，获取最终弱光增强的渗水区域图像；S6、将最终输出的弱光增强渗水图像导入MaskRCNN模型，渗水图像经过卷积计算获得特征图，经过RPN获得候选区域，经过ROI Align层获得最终的渗水区域位置。本发明通过将待检测的巡检图像导入MBLLEN模型进行弱光增强，再将弱光增强的图像导入MaskRCNN模型进行积水区域检测，不仅可以进行有效的目标检测，而且还可以实现精准分割目标区域的边界。

Description

基于弱光补偿的MaskRCNN渗水检测方法及系统 技术领域
本发明涉及室内复杂环境下巡检机器人识别算法优化技术领域，尤其涉及基于弱光补偿的MaskRCNN渗水检测方法及系统。
背景技术
水轮机组在运行过程中，经常出现主轴密封频繁漏水的现象，这严重的影响了机组的稳定运行。对电缆遍布的水轮机层来说，更容易造成电路短路等事故。漏水量大时，存在水轮机层严重积水隐患。同时，水轮机层设备滴水漏水造成的设备故障应该及时检修，以维护生产的稳定运行。因此，对水轮机层设备的巡检区域进行定期、全面地滴水漏水，分析整体渗水状况，按其渗水形式及渗水程度进行及时的修补维护工作，将有效地提高安全系数，降低因渗漏水造成的经济损失和安全隐患。然而，水轮机层由于光线条件差，即使在补光的情况下，巡检机器人拍摄的图像也难以区分渗水漏水区域的边界。
发明内容
为解决现有技术所存在的技术问题，本发明提供基于弱光补偿的MaskRCNN渗水检测方法及系统，通过将待检测的巡检图像导入MBLLEN模型进行弱光增强，再将弱光增强的图像导入MaskRCNN模型进行积水区域检测，不仅可以进行有效的目标检测，而且还可以实现精准分割目标区域的边界。
本发明方法采用以下技术方案来实现：基于弱光补偿的MaskRCNN渗水检测方法，包括以下步骤：
S1、进行积水图像拍摄，在网上收集积水渗水图像，采用融合样本数据增强扩充样本数据集；
S2、对扩增后的数据集用Lableme进行数据标注，生成积水区域的标记文件；
S3、对标记的数据集进行增强操作，对图片进行翻转、缩放、色域变化操作，操作完成后将图片恢复到原图像素大小；
S4、利用增强的数据集训练MaskRCNN模型；
S5、将待检测图像导入MBLLEN模型，经过特征提取模块FEM层获得各层级的特征图，特征图经过增强模块EM层获得每层特征图经过弱光增强后的图片，将增强后的特征图输入融合模块FM层，获取最终弱光增强的渗水区域图像；
S6、将最终输出的弱光增强渗水图像导入MaskRCNN模型，渗水图像经过卷积计算获得特征图，经过RPN获得候选区域，经过ROI Align层获得最终的渗水区域位置。
本发明系统采用以下技术方案来实现：基于弱光补偿的MaskRCNN渗水检测系统，包括：
融合样本数据增强模块：用于将拍摄的积水图像与网上收集的积水渗水图像，采用融合样本数据增强扩充样本数据集；
数据标注模块：用于对扩增后的数据集采用Lableme进行数据标注，生成积水区域的标记文件；
增强操作模块：用于对标记的数据集进行增强操作，对图片进行翻转、缩放、色域变化操作，操作完成后将图片恢复到原图像素大小；
MaskRCNN模型训练模块：利用增强的数据集训练MaskRCNN模型；
弱光增强渗水区域图像获取模块：将待检测图像导入MBLLEN模型，经过特征提取模块FEM层获得各层级的特征图，特征图经过增强模块EM层获得每层特征图经过弱光增强后的图片，将增强后的特征图输入融合模块FM层，获取最终弱光增强的渗水区域图像；
渗水区域位置获取模块：将最终输出的弱光增强渗水图像导入MaskRCNN模型，渗水图像经过卷积计算获得特征图，经过RPN获得候选区域，经过ROI Align层获得最终的渗水区域位置。
本发明与现有技术相比，具有如下优点和有益效果：
本发明通过将待检测的巡检图像导入MBLLEN模型进行弱光增强，再将弱光增强的图像导入MaskRCNN模型进行积水区域检测，不仅可以进行有效的目标检测，而且还可以实现精准分割目标区域的边界。
附图说明
图1是本发明的方法流程图；
图2(a)是本发明积水图像拍摄示意图；
图2(b)是网上收集积水渗水图像示意图；
图2(c)是本发明融合样本数据增强示意图；
图3是Labelme软件标注积水区域图像示意图；
图4是MaskRCNN模型的结构示意图；
图5是MBLLEN模型结构示意图；
图6(a)是弱光图像；
图6(b)是增强图像。
具体实施方式
下面结合实施例及附图对本发明作进一步详细的描述，但本发明的实施方式不限于此。
实施例
如图1所示，本实施例基于弱光补偿的MaskRCNN渗水检测方法，包括以下步骤：
S1、进行积水图像拍摄，在网上收集类似的积水渗水图像，采用融合样本数据增强扩充样本数据集；
S2、对扩增后的数据集用Lableme进行数据标注，生成积水区域的标记文件；
S3、对标记的数据集进行增强操作，对图片进行翻转、缩放、色域变化等操作，操作完成后将图片恢复到原图像素大小；
S4、利用增强的数据集训练MaskRCNN模型；
S5、将待检测图像导入MBLLEN模型，经过特征提取模块FEM层获得各层级的特征图，特征图经过增强模块EM层获得每层特征图经过弱光增强后的图片，将增强后的特征图输入融合模块FM层，获取最终弱光增强的渗水区域图像；
S6、将最终输出的弱光增强渗水图像导入MaskRCNN模型，渗水图像经过卷积计算获得特征图，经过RPN获得候选区域，经过ROI Align层获得最终的渗水区域位置。
如图2(a)、图2(b)、图2(c)所示，本实施例中，步骤S1中融合样本数据增强的具体过程如下：
从训练集中随机选择两张图片并采用数据增强方法，包括翻转、增加噪声、剪切等进行增强，按照随机权重将两张积水图像融合到一起，以增加样本的多样性；具体地，两张图像融合的公式如下所示：
Image(R,G,B)＝η×Image1(R,G,B)+(1-η)Image2(R,G,B)
η＝rand(0.3-0.7)          (1)
其中，Image(R,G,B)为融合图像，Image1(R,G,B)和Image2(R,G,B)为原始的两幅图像，η＝rand(0.3-0.7)表示融合权重为0.3-0.7的随机数，R,G,B为图像的三个通道。
如图3所示，本实施例中，步骤S2中数据标注的具体过程如下：
采用多线段、多点的方式对渗水区域进行标注；利用Lableme标记工具标注渗水区域的多边形轮廓，设置渗水轮廓的标签名称，标记的样本生成与之对应的json文件，json格式的文件存储样本中目标区域的轮廓和图像信息。
如图4所示，本实施例中，MaskRCNN为实例分割框架，用于进行有效的目标检测和精准分割目标区域的边界。MaskRCNN模型主要由特征提取框架ResNet和RPN模块构成；ResNet利用多层卷积结构提取待检测图像的特征，RPN用于生成多个ROI区域；MaskRCNN采用RoI Align层代替了RoI Pooling，并采用双线性插值将RPN生成的多个ROI特征区域映射到统一的7*7尺寸；最后，对RPN层生成的多个ROI区域进行分类和定位框的回归操作，采用全卷积神经网络FCN生成渗水区域对应的Mask。
本实施例中，MaskRCNN的损失函数Loss定义为：
Loss＝L cls+L box+L mask       (2)
其中，L cls为分类误差，L box为定位框产生的误差，L mask为掩膜Mask造成的误差；
引入对数似然损失(Log-likelihood Loss)构造分类误差L cls，L cls的计算公式如下：
Figure PCTCN2022134451-appb-000001
其中，X、Y为分别为测试分类和真实分类，N为输入样本量，M为可能的类别数，p ij表示样本x i的模型预测输出为类别j的概率分布；y ij表示样本x i的真实类别是否为类别j；为了增加损失函数的鲁棒性，定位框产生的误差L box采用L1损失；ROI区域内的像素采用sigmoid 函数求相对熵，得到平均相对熵误差L mask。
为了使少量标记的数据集在MaskRCNN获得更好的泛化性能，本实施例引入了COCO数据集上预训练的权重(mask_rcnn_coco.h5)进行fine tuning。使用MBLLEN和MaskRCNN积水区域检测模型进行分类；将待检测的巡检图像导入MBLLEN模型进行弱光增强，再将弱光增强的图像导入MaskRCNN模型进行积水区域检测，输出标记好的渗水区域。
如图5所示，本实施例中，MBLLEN模型是一个多层特征弱光增强深度学习网络模型，通过卷积计算提取不同层的图像特征，并将不同层的特征图输入多个子网络进行增强。MBLLEN模型主要包含特征提取模块(Feature Extraction Module,FEM)、增强模块(Enhancement Module,EM)、融合模块(Fusion Module,FM)。其中，特征提取模块FEM由单向10层网络结构构成，32个3×3卷积核，卷积步长为1，ReLU激活函数，特征提取模块不采用池化层；每一层的输出同时为下一个特征提取模块FEM卷积层的输入和增强模块EM对应卷积层的输入。由于特征提取模块FEM包含10层特征提取层，因此，增强模块EM包含10个结构相同的子网络结构。EM层子网络结构都是1层卷积层、3层卷积层和3层反卷积层。融合模块FM融合所有从EM子网输出的图像，使用3通道1×1卷积核卷积得到最终增强结果。
为了训练MBLLEN模型使其可以补偿图像弱光，分别定义了结构损失(Structure loss,Str)、预训练VGG内容损失(VGG)和区域损失(Region loss)。
具体地，损失函数的公式如下所示：
Loss＝L Str+L VGG/i,j+L Region      (4)
其中，结构损失主要用于减小增强图像和真实图像的结构扭曲和畸变，具体公式表示如下：
L Str＝L SSIM+L MS-SSIM       (5)
其中，L SSIM为增强图像和真实图像的结构相似度，L MS-SSIM为多层级结构相似度；
预训练VGG内容损失，最小化增强图像和真实图像在预训练VGG-19网络输出的绝对差值，损失函数公式如下：
Figure PCTCN2022134451-appb-000002
其中，E和G分别是增强图像和真实图像，W i,j、H i,j、C i,j分别表示预训练VGG的特征图的维度；Φ i,j表示VGG-19网络第j个卷积层，第i个特征图；x、y、z分别表示特征图的宽度，高度，通道数；
区域损失，通过分割图像40％最暗像素值来近似估计整图暗光区域，得到如下损失函数：
Figure PCTCN2022134451-appb-000003
其中，E L和G L分别是增强图像和真实图像的弱光区域，E H和G H分别是增强图像和真实图像的非弱光区域，w L和w H分别为4和1；m L是G L图像的宽；n L是G L图像的高；m H是G H 图像的宽；n H是G H图像的高。
通过在PASCAL VOC数据集基础上合成得到弱光照数据集，分别加入Gamma矫正和Peak值为200的Poisson噪声作为弱光输入图像，原图像作为真实图像。弱光渗水图像的增强图像实验结果如图6(a)、图6(b)所示。
基于相同的发明构思，本发明提出基于弱光补偿的MaskRCNN渗水检测系统，包括：
融合样本数据增强模块：用于将拍摄的积水图像与网上收集的积水渗水图像，采用融合样本数据增强扩充样本数据集；
数据标注模块：用于对扩增后的数据集采用Lableme进行数据标注，生成积水区域的标记文件；
增强操作模块：用于对标记的数据集进行增强操作，对图片进行翻转、缩放、色域变化操作，操作完成后将图片恢复到原图像素大小；
MaskRCNN模型训练模块：利用增强的数据集训练MaskRCNN模型；
弱光增强渗水区域图像获取模块：将待检测图像导入MBLLEN模型，经过特征提取模块FEM层获得各层级的特征图，特征图经过增强模块EM层获得每层特征图经过弱光增强后的图片，将增强后的特征图输入融合模块FM层，获取最终弱光增强的渗水区域图像；
渗水区域位置获取模块：将最终输出的弱光增强渗水图像导入MaskRCNN模型，渗水图像经过卷积计算获得特征图，经过RPN获得候选区域，经过ROI Align层获得最终的渗水区域位置。
上述实施例为本发明较佳的实施方式，但本发明的实施方式并不受上述实施例的限制，其他的任何未背离本发明的精神实质与原理下所作的改变、修饰、替代、组合、简化，均应为等效的置换方式，都包含在本发明的保护范围之内。

Claims (9)

  1. 基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，包括以下步骤：
    S1、进行积水图像拍摄，在网上收集积水渗水图像，采用融合样本数据增强扩充样本数据集；
    S2、对扩增后的数据集用Lableme进行数据标注，生成积水区域的标记文件；
    S3、对标记的数据集进行增强操作，对图片进行翻转、缩放、色域变化操作，操作完成后将图片恢复到原图像素大小；
    S4、利用增强的数据集训练MaskRCNN模型；
    S5、将待检测图像导入MBLLEN模型，经过特征提取模块FEM层获得各层级的特征图，特征图经过增强模块EM层获得每层特征图经过弱光增强后的图片，将增强后的特征图输入融合模块FM层，获取最终弱光增强的渗水区域图像；
    S6、将最终输出的弱光增强渗水图像导入MaskRCNN模型，渗水图像经过卷积计算获得特征图，经过RPN获得候选区域，经过ROI Align层获得最终的渗水区域位置。
  2. 根据权利要求1所述的基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，步骤S1中融合样本数据增强的具体过程如下：
    从训练集中随机选择两张图片并采用数据增强方法，包括翻转、增加噪声、剪切进行增强，按照随机权重将两张积水图像融合到一起，以增加样本的多样性；两张图像融合的公式如下所示：
    Image(R,G,B)＝η×Image1(R,G,B)+(1-η)Image2(R,G,B)
    η＝rand(0.3-0.7)       (1)
    其中，Image(R,G,B)为融合图像，Image1(R,G,B)和Image2(R,G,B)为原始的两幅图像，η＝rand(0.3-0.7)表示融合权重为0.3-0.7的随机数，R,G,B为图像的三个通道。
  3. 根据权利要求1所述的基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，步骤S2中数据标注的具体过程如下：
    采用多线段、多点的方式对渗水区域进行标注；利用Lableme标记工具标注渗水区域的多边形轮廓，设置渗水轮廓的标签名称，标记的样本生成与之对应的json文件，json格式的文件存储样本中目标区域的轮廓和图像信息。
  4. 根据权利要求1所述的基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，步骤S4中MaskRCNN模型由特征提取框架ResNet和RPN模块构成；ResNet利用多层卷积结构提取待检测图像的特征，RPN用于生成多个ROI区域；MaskRCNN采用RoI Align层代替了RoI Pooling，并采用双线性插值将RPN生成的多个ROI特征区域映射到统一的7*7尺寸；最后，对RPN层生成的多个ROI区域进行分类和定位框的回归操作，采用全卷积神经网络FCN生成渗水区域对应的Mask。
  5. 根据权利要求4所述的基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，MaskRCNN的损失函数Loss定义为：
    Loss＝L cls+L box+L mask   (2)
    其中，L cls为分类误差，L box为定位框产生的误差，L mask为掩膜Mask造成的误差；
    引入对数似然损失构造分类误差L cls，L cls的计算公式如下：
    Figure PCTCN2022134451-appb-100001
    其中，X、Y为分别为测试分类和真实分类，N为输入样本量，M为可能的类别数，p ij表示样本x i的模型预测输出为类别j的概率分布；y ij表示样本x i的真实类别是否为类别j；
    定位框产生的误差L box采用L1损失；ROI区域内的像素采用sigmoid函数求相对熵，得到平均相对熵误差L mask。
  6. 根据权利要求1所述的基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，步骤S4中训练MaskRCNN模型包括引入COCO数据集上预训练的权重进行fine tuning。
  7. 根据权利要求1所述的基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，步骤S5中MBLLEN模型具体实现过程如下：
    S51、将MBLLEN模型分为特征提取模块FEM、增强模块EM和融合模块FM；
    S52、特征提取模块FEM由单向10层网络结构构成，32个3×3卷积核，卷积步长为1，ReLU激活函数；每一层的输出同时为下一个特征提取模块FEM卷积层的输入和增强模块EM对应卷积层的输入；
    S53、增强模块EM包含10个结构相同的子网络结构，均是1层卷积层、3层卷积层和3层反卷积层；
    S54、融合模块FM融合所有从增强模块EM子网输出的图像，使用3通道1×1卷积核卷积得到最终增强结果。
  8. 根据权利要求7所述的基于弱光补偿的MaskRCNN渗水检测方法，其特征在于，MBLLEN模型的训练过程如下：
    定义结构损失、预训练VGG内容损失和区域损失；
    损失函数的公式如下所示：
    Loss＝L Str+L VGG/i,j+L Region   (4)
    其中，结构损失具体公式表示如下：
    L Str＝L SSIM+L MS-SSIM   (5)
    其中，L SSIM为增强图像和真实图像的结构相似度，L MS-SSIM为多层级结构相似度；
    预训练VGG内容损失，公式如下：
    Figure PCTCN2022134451-appb-100002
    其中，E和G分别是增强图像和真实图像，W i,j、H i,j、C i,j分别表示预训练VGG的特征图的维度；Φ i,j表示VGG-19网络第j个卷积层，第i个特征图；x、y、z分别表示特征图的宽度，高度，通道数；
    区域损失，通过分割图像40％最暗像素值获取整图暗光区域，得到如下损失函数：
    Figure PCTCN2022134451-appb-100003
    其中，E L和G L分别是增强图像和真实图像的弱光区域，E H和G H分别是增强图像和真实图像的非弱光区域，w L和w H分别为4和1；m L是G L图像的宽；n L是G L图像的高；m H是G H图像的宽；n H是G H图像的高。
  9. 基于弱光补偿的MaskRCNN渗水检测系统，其特征在于，包括：
    融合样本数据增强模块：用于将拍摄的积水图像与网上收集的积水渗水图像，采用融合样本数据增强扩充样本数据集；
    数据标注模块：用于对扩增后的数据集采用Lableme进行数据标注，生成积水区域的标记文件；
    增强操作模块：用于对标记的数据集进行增强操作，对图片进行翻转、缩放、色域变化操作，操作完成后将图片恢复到原图像素大小；
    MaskRCNN模型训练模块：利用增强的数据集训练MaskRCNN模型；
    弱光增强渗水区域图像获取模块：将待检测图像导入MBLLEN模型，经过特征提取模块FEM层获得各层级的特征图，特征图经过增强模块EM层获得每层特征图经过弱光增强后的图片，将增强后的特征图输入融合模块FM层，获取最终弱光增强的渗水区域图像；
    渗水区域位置获取模块：将最终输出的弱光增强渗水图像导入MaskRCNN模型，渗水图像经过卷积计算获得特征图，经过RPN获得候选区域，经过ROI Align层获得最终的渗水区域位置。
PCT/CN2022/134451 2022-04-29 2022-11-25 基于弱光补偿的MaskRCNN渗水检测方法及系统 WO2023207064A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210464625.8A CN115240020A (zh) 2022-04-29 2022-04-29 基于弱光补偿的MaskRCNN渗水检测方法及系统
CN202210464625.8 2022-04-29

Publications (1)

Publication Number Publication Date
WO2023207064A1 true WO2023207064A1 (zh) 2023-11-02

Family

ID=83667997

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/134451 WO2023207064A1 (zh) 2022-04-29 2022-11-25 基于弱光补偿的MaskRCNN渗水检测方法及系统

Country Status (3)

Country Link
CN (1) CN115240020A (zh)
LU (1) LU505937B1 (zh)
WO (1) WO2023207064A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117315446A (zh) * 2023-11-29 2023-12-29 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) 一种面向复杂环境下水库溢洪道异常智能识别方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115240020A (zh) * 2022-04-29 2022-10-25 清远蓄能发电有限公司 基于弱光补偿的MaskRCNN渗水检测方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020047738A1 (zh) * 2018-09-04 2020-03-12 安徽中科智能感知大数据产业技术研究院有限责任公司 多尺度特征融合网络结合定位模型的害虫自动计数方法
CN113469177A (zh) * 2021-06-30 2021-10-01 河海大学 基于深度学习的排水管道缺陷检测方法及系统
CN115240020A (zh) * 2022-04-29 2022-10-25 清远蓄能发电有限公司 基于弱光补偿的MaskRCNN渗水检测方法及系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110675415B (zh) * 2019-12-05 2020-05-15 北京同方软件有限公司 一种基于深度学习增强实例分割的道路积水区域检测方法
CN114298145A (zh) * 2021-11-22 2022-04-08 三峡大学 基于深度学习的透水混凝土孔隙智能识别与分割方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020047738A1 (zh) * 2018-09-04 2020-03-12 安徽中科智能感知大数据产业技术研究院有限责任公司 多尺度特征融合网络结合定位模型的害虫自动计数方法
CN113469177A (zh) * 2021-06-30 2021-10-01 河海大学 基于深度学习的排水管道缺陷检测方法及系统
CN115240020A (zh) * 2022-04-29 2022-10-25 清远蓄能发电有限公司 基于弱光补偿的MaskRCNN渗水检测方法及系统

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS: "Interpretation of low light enhancement paper--MBLLEN ", CSDN-BLOG, 8 January 2020 (2020-01-08), XP093105014, Retrieved from the Internet <URL:https://blog.csdn.net/m0_37833142/article/details/103888633> [retrieved on 20231123] *
ANONYMOUS: "Summary of loss function of Fast(er) rcnn ", CSDN-BLOG, 10 September 2019 (2019-09-10), XP093105016, Retrieved from the Internet <URL:https://blog.csdn.net/qq_17846375/article/details/100687504> [retrieved on 20231123] *
HE KAIMING; GKIOXARI GEORGIA; DOLLAR PIOTR; GIRSHICK ROSS: "Mask R-CNN", 2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), IEEE, 22 October 2017 (2017-10-22), pages 2980 - 2988, XP033283165, DOI: 10.1109/ICCV.2017.322 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117315446A (zh) * 2023-11-29 2023-12-29 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) 一种面向复杂环境下水库溢洪道异常智能识别方法
CN117315446B (zh) * 2023-11-29 2024-02-09 江西省水利科学院(江西省大坝安全管理中心、江西省水资源管理中心) 一种面向复杂环境下水库溢洪道异常智能识别方法

Also Published As

Publication number Publication date
LU505937B1 (de) 2024-04-29
CN115240020A (zh) 2022-10-25

Similar Documents

Publication Publication Date Title
WO2023207064A1 (zh) 基于弱光补偿的MaskRCNN渗水检测方法及系统
CN111223088B (zh) 一种基于深层卷积神经网络的铸件表面缺陷识别方法
CN110175982B (zh) 一种基于目标检测的缺陷检测方法
CN106127684B (zh) 基于双向递归卷积神经网络的图像超分辨率增强方法
CN113609896A (zh) 基于对偶相关注意力的对象级遥感变化检测方法及系统
CN110598698B (zh) 基于自适应区域建议网络的自然场景文本检测方法和系统
US20220366682A1 (en) Computer-implemented arrangements for processing image having article of interest
CN110097110B (zh) 一种基于目标优化的语义图像修复方法
CN111696079A (zh) 一种基于多任务学习的表面缺陷检测方法
CN114943689A (zh) 基于半监督学习的钢铁冷轧退火炉元器件检测方法
CN111914720A (zh) 一种输电线路绝缘子爆裂识别方法及装置
CN116205876A (zh) 基于多尺度标准化流的无监督笔记本外观缺陷检测方法
CN111915565B (zh) 一种基于yolact算法对输变电线路瓷瓶裂缝实时分析方法
CN111368637B (zh) 一种基于多掩模卷积神经网络的搬运机器人识别目标方法
CN111091533B (zh) 一种基于改进ssd算法的电池片el缺陷检测方法
Wang et al. Efficient multi-branch dynamic fusion network for super-resolution of industrial component image
Huang et al. Crack detection of masonry structure based on thermal and visible image fusion and semantic segmentation
CN116385465A (zh) 图像分割模型的构建及图像分割方法、系统、设备和介质
Gupta et al. Post disaster mapping with semantic change detection in satellite imagery
CN115410184A (zh) 一种基于深度神经网络的目标检测车牌识别方法
CN115291141A (zh) 一种电缆及其附件的故障诊断方法及系统
CN114898096A (zh) 一种人物图像的分割和标注方法及系统
CN111144513B (zh) 一种结合生成对抗网络与传统图像处理技术的工业产品喷涂缺陷数据增强方法
CN111089865B (zh) 一种基于f-rcnn的缺陷电缆检测方法
CN112818965B (zh) 多尺度影像目标检测方法、系统、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22939879

Country of ref document: EP

Kind code of ref document: A1