CN115063434A - A low-low-light image instance segmentation method and system based on feature denoising - Google Patents

Abstract

The invention relates to a low-light image instance segmentation method and system based on feature denoising, and belongs to the technical field of computer vision. Firstly, calibrating according to the characteristics of the image sensor, and constructing a full-element physical quantity noise model. And then training the example segmentation deep convolutional network by using a noise disturbance suppression loss function and an example segmentation task loss function. The low-light image is subjected to example segmentation by a convolutional neural network, and the low-light image does not need to be preprocessed. And finally, performing target detection segmentation on the denoised features by utilizing the trained detection segmentation sub-network to obtain a final instance segmentation result. The method is simple to realize, high in performance, strong in robustness, extremely low in extra calculation amount overhead of dealing with low dim light, and beneficial to realizing low-delay and high-speed low dim light example segmentation. The method can synthesize the high-quality simulation low-light image data set only by utilizing the existing natural image data set, thereby reducing the generation cost of the low-light image data.

Description

A method and system for instance segmentation of low and low light images based on feature denoising

technical field

The invention relates to a method and system for instance segmentation of images under low and low light conditions, in particular to a low and low light image instance segmentation method and system based on feature denoising, belonging to the technical field of computer vision.

Background technique

Low low light environment, that is, the environment with low light intensity, such as the environment under night city lights, moonlight, starlight conditions, etc. An image captured under low-low-light conditions is a low-low-light image. At this time, the light is insufficient, the image sensor collects less photons, and the image signal is weak, resulting in low-low-light images with strong noise and low signal-to-noise ratio, and the scene information is seriously lost and difficult to recover. A RAW image is the raw data that the image sensor converts the captured light source signal into a digital signal, which is an unprocessed, uncompressed format. RAW images retain more information and have better dynamic range than sRGB images in the commonly used JPEG format.

Instance segmentation is a technique that can extract semantic information from images. The technology can intelligently identify the class, location, size, and shape of objects of interest from images under natural lighting conditions, and describe them with class labels, object bounding boxes, and pixel masks or polygons. This technology is widely used in image editing, intelligent security, autonomous driving and satellite image interpretation, and has high practical value and application potential. Currently, mainstream instance segmentation methods are based on deep learning and deep convolutional neural networks, and rely on normal light image datasets for learning and training. Due to the lack of low-low-light image datasets and corresponding end-to-end methods and systems, when it is applied to low-low-light images, it is necessary to perform image preprocessing such as denoising and enhancement. Image denoising refers to converting a noisy image into a clean image, and image enhancement refers to converting a blurred image with lower brightness from a brighter, clearer image. Image denoising and enhancement require complex calculation process, which increases the computational complexity of the entire system, and often has poor effect on low-low-light images, resulting in the speed and accuracy of the final instance segmentation system being difficult to meet the actual needs.

Denoising refers to the process of reducing the noise in the signal and improving the signal-to-noise ratio of the signal. Since low-low-light images have strong noise, these noises can cause deep convolutional neural networks for instance segmentation to be severely perturbed in the process of feature extraction from images. Therefore, the image features extracted by the shallow layer of the network have obvious high-frequency noise, and the image features extracted by the deep network have a low semantic response to the target of interest, which ultimately makes it impossible to accurately extract the instance segmentation results of the target of interest. Adaptive feature denoising, that is, adaptively reducing the disturbance of image features caused by image noise according to the noise of the input features, while reducing the high-frequency noise of the shallow extracted features of the network, while retaining more effective feature signals, and improving the deep network features. Semantic responses to objects of interest.

SUMMARY OF THE INVENTION

The purpose of the present invention is to propose a feature-based denoising method based on the existing requirements for intelligent image recognition of low and low light scenes at night, aiming at the shortcomings of instance segmentation technology in low and low light conditions, such as high computational complexity and low performance. Low-low-light image instance segmentation method and system. The invention realizes end-to-end fast and high-performance low-low-light image instance segmentation, does not need to perform complex denoising and enhancement preprocessing on the low-low-light image, and reduces the computational complexity of the entire instance segmentation.

The present invention is realized by the following technical solutions.

A low-low-light image instance segmentation method based on feature denoising, comprising the following steps:

Step 101: Synthesize a simulated low-low-light image dataset.

Step 102: Use the noise disturbance suppression loss function and the instance segmentation task loss function to train the instance segmentation deep convolutional network.

Step 103: Use the trained deep convolutional neural network to perform feature extraction and feature denoising on the low-low-light image. No preprocessing is required for low-low-light images.

Step 104: Use the trained detection and segmentation sub-network to perform target detection and segmentation on the denoised features to obtain a final instance segmentation result.

In order to achieve the purpose of the present invention, the present invention further proposes a low-low-light image instance segmentation system based on feature denoising, including a simulation low-low-light image synthesis module, a noise disturbance suppression learning module, image feature extraction and feature denoising. Module, target detection segmentation and extraction module.

beneficial effect

Compared with the prior art, the method and system of the present invention have the following advantages:

1. This method does not rely on additional low-low-light image denoising, enhancement and other processing modules. The training process is carried out end-to-end, with simple implementation, high performance and strong robustness.

2. This method has extremely low extra computational overhead for dealing with low and low light, which is beneficial to realize low-latency, high-speed low-low light instance segmentation.

3. The method can reduce the cost of generating low-low-light image data. The data-driven deep convolutional neural network image instance segmentation method requires a large amount of low-low-light image data, while the collection and production of traditional low-low-light image datasets requires a lot of manpower and material resources. This method does not require manual data collection, and only needs to use existing natural The image dataset can be synthesized into a high-quality simulated low-low-light image dataset.

Description of drawings

Figure 1 is a flow chart of the method of the present invention.

FIG. 2 is a schematic diagram of the method of the present invention for simulating low-low-light image synthesis and the noise disturbance suppression learning method in the training process.

FIG. 3 is a schematic diagram of the internal details of the adaptive feature denoising downsampling layer according to the method of the present invention.

FIG. 4 is a schematic diagram of a learnable low-pass filtering convolution block according to the method of the present invention.

Figure 5 is a flow chart of the system of the present invention.

Detailed ways

In order to better illustrate the purpose and advantages of the present invention, the content of the invention will be further described below with reference to the accompanying drawings.

A low-low-light image instance segmentation method based on feature denoising, comprising the following steps:

Step 101: Synthesize a simulated low-low-light image dataset.

At present, the learning-based instance segmentation methods mainly rely on high-quality datasets to achieve good performance, and there is currently a lack of public low-low-light image datasets, and the collection and production of real low-low-light image datasets suitable for instance segmentation production cycle The long-term cost is high. Therefore, the present invention considers synthesizing the low-low-light image data set to train the instance segmentation model, so as to reduce the production cost of the low-low-light image instance segmentation data set.

The production of simulated low and low light image data is shown in Figure 2. First, the Unprocess operation can be used to convert ordinary natural light RGB images into RAW image data. Then, simulated noise injection was used to simulate the noise in the image in low low light conditions. Wherein, the injected simulation noise can be generated using a Gaussian noise model, a Gaussian Poisson mixed noise model, a full-element physical quantity noise model, etc., preferably the full-element physical quantity noise model is used. The full-element physical quantity noise model needs to be calibrated according to the sensor characteristics to obtain physical noise components such as shot noise, readout noise, color cast noise, random line noise, etc., to achieve more realistic noise simulation, and to achieve better low-low light images. Instance segmentation effect. Through these two steps, large-scale natural light image data can be transformed into simulated low-low light image data. The proposed method is suitable for many types of low-low-light images (RAW images, sRGB images, etc.), preferably RAW images.

Step 102: Use the noise disturbance suppression loss function and the instance segmentation task loss function to train the instance segmentation deep convolutional network. As shown in Figure 2, instance segmentation models (Mask R-CNN, HTC, Cascade Mask R-CNN, YOLACT, PointRend, etc.) are applied.

Specifically, the total loss function L(θ) is expressed as:

L(θ)=L _IS (x; θ)+αL _IS (x'; θ)+βL _DS (x,x'; θ) (1)

Among them, L _IS and L _DS represent the instance segmentation task loss function and noise disturbance suppression loss function respectively; x and x′ represent the simulated clean image and simulated noisy image respectively; θ is the model parameter; α and β are the weights of the loss function. Among them, L _IS is adjusted according to the different models used for instance segmentation; L _DS is expressed as:

Among them, f ⁽ⁱ⁾ is the feature extracted by the (i)th layer of the instance segmentation network f. _LDS can use the clean image features extracted on clean images as a guide to make the features extracted by the model on noisy images as close to clean image features as possible, thereby reducing the interference of higher levels of noise in low-low-light images on network feature extraction. , which is beneficial to achieve robust low-low-light image instance segmentation.

During the whole training process, there is no need to process low-light images, and the feature extraction network and detection and segmentation sub-network of the instance segmentation network are completed by one-time training in an end-to-end manner.

Step 103: Use the trained instance segmentation deep convolutional neural network to perform instance segmentation on the low-low-light image, and the low-low-light image does not need to be preprocessed.

Ordinary instance segmentation models can only perform feature extraction on images, and high-intensity noise in low-low-light images will introduce high-frequency noise into the extracted image features, resulting in lower responsivity to the target of interest.

This method first extracts features from the image. During feature extraction, features are denoised using adaptive feature denoising downsampling layers and learnable low-pass filtered convolution blocks. The denoised features greatly reduce the feature perturbation caused by higher levels of noise in low-low-light images, and the adaptive feature denoising downsampling layer brings less additional computation, and the learnable low-pass filter convolution block does not. The additional computational cost is beneficial to realize low-latency and robust low-light image instance segmentation.

The adaptive feature denoising downsampling layer, through the downsampling process of the convolutional neural network, makes full use of the local features for weighted averaging, reducing the noise level in the features.

In order to retain more target features while reducing feature noise. As shown in Figure 3, the adaptive feature denoising downsampling layer adaptively predicts different low-pass filters for each channel and each position of the feature map. The features after adaptive low-pass filtering and 2 times downsampling are the features after denoising:

表示预测W_c,i,j所依赖的局部特征位置；GP()表示全局池化(Global Pooling)操作。p、q分别表示卷积核上的横向、纵向坐标。Among them, X and Y are the input feature map and the denoised feature map respectively; W is the weighted weight predicted by the adaptive downsampling layer according to the input feature map; (c, i, j) represents the channel dimension and wide dimension of the feature map. , coordinates in high dimensions; S represents the position around the spatial position (i, j); φ() represents the weight prediction function of the adaptive downsampling layer. In order to ensure that the output weight is low-pass, the weight prediction function will predict The weights are normalized by softmax;

Represents the local feature position on which the prediction W _c,i,j depends; GP() represents the global pooling (Global Pooling) operation. p and q represent the horizontal and vertical coordinates on the convolution kernel, respectively.

The learnable low-pass filtering convolution block improves the robustness of ordinary convolutional layers to feature noise by explicitly utilizing a branch with a learnable low-pass filtering. Its structure is shown in Figure 4 and contains two branches. , one is the same branch as ordinary convolution, and the other is a branch consisting of learnable low-pass filtering and 1×1 convolution. The learnable low-pass filter weight can be obtained from training and normalized by the softmax function to ensure that it is a low-pass filter, which can locally smooth the feature to reduce the noise level in the feature, and pass the 1×1 convolution. The denoised features are fused into the normal convolution branch. At the same time, the learnable low-pass filtered convolutional blocks can be transformed into ordinary convolutional layers during inference using reparameterization techniques, thus not adding any computational effort. The re-parameterization is to convert multiple parallel branches in the convolutional neural network into an equivalent single-branch structure through parameter fusion. In this embodiment, the learnable low-pass filter convolution block is re-parameterized into the parameters of a single 3×3 convolution layer, which is obtained by the following formula:

W′ _3×3 [h,t,p,q]＝W _3×3 [h,t,h,t]+(W _1×1 [h,t,1,1]*W _LLPF [1,t ,p,q]) (5)

C₁、C₂分别表示卷积层输入通道数、输出通道数，R表示实数；

为可学习的低通滤波卷积块中的普通3×3卷积层参数；

为1×1卷积层参数；

为可学习的低通滤波器参数，可学习的低通滤波器对每个输入通道共享一套3×3的权重；h、t分别表示在输出通道、输入通道的坐标，并且h∈{1,2,…,C₂}，t∈{1,2,…,C₁}；p、q分别表示卷积核的横向、纵向坐标，p、q∈{1,2,3}。Among them, W′ _3×3 is the convolution layer parameter after reparameterization of the learnable low-pass filter convolution block,

C ₁ and C ₂ respectively represent the number of input channels and output channels of the convolution layer, and R represents a real number;

are the common 3×3 convolutional layer parameters in the learnable low-pass filtered convolutional block;

is a 1×1 convolutional layer parameter;

is a learnable low-pass filter parameter, and the learnable low-pass filter shares a set of 3 × 3 weights for each input channel; h and t represent the coordinates of the output channel and input channel, respectively, and h∈{1 ,2,…,C ₂ }, t∈{1,2,…,C ₁ }; p and q represent the horizontal and vertical coordinates of the convolution kernel, respectively, p, q∈{1,2,3}.

Step 104: Use the trained detection and segmentation sub-network to perform target detection and segmentation on the denoised features to obtain a final instance segmentation result.

In order to achieve the purpose of the present invention, the present invention further proposes an end-to-end low-low-light image instance segmentation system based on adaptive feature denoising, as shown in FIG. Disturbance suppression learning module 20 , image feature extraction and feature denoising module 30 , target detection segmentation extraction module 40 .

Among them, the simulated low and low light image synthesis module 10 is used to establish a simulated low and low light image data set for training the instance segmentation model. This module can assemble the input natural light image data into a simulated low and low light image data set.

The noise disturbance suppression learning module 20 is used to guide the instance segmentation model to learn robust image features, so that it can reduce the feature disturbance caused by higher level noise in low and low light images. This module uses the simulated low and low light image data set to learn the training model using noise disturbance suppression, and outputs the trained instance segmentation model.

The image feature extraction and feature denoising module 30 uses adaptive low-pass filtering to reduce the noise level of image features in the instance segmentation network, and extracts stable and clean image features on the noisy low-light image to achieve robust Instance segmentation of low-low-light images.

The target detection, segmentation and extraction module 40 can identify and extract the category, position, size, and shape of the target of interest from the denoised image features, and obtain the final low-low-light image instance segmentation result.

The connection relationship between the above modules is as follows:

The output end of the simulated low and low light image synthesis module 10 is connected to the input end of the noise disturbance suppression learning module 20 .

The output end of the noise disturbance suppression learning module 20 is connected to the input end of the image feature extraction and feature denoising module 30 .

The output end of the image feature extraction and feature denoising module 30 is connected to the input end of the target detection, segmentation and extraction module 40 .

Claims (8)

1. a low-low-light image instance segmentation method based on feature denoising, is characterized in that, comprises the following steps: Step 101: Synthesize a simulated low and low light image dataset; Step 102: Use the noise disturbance suppression loss function and the instance segmentation task loss function to train the instance segmentation deep convolutional network; Step 103: Use the trained deep convolutional neural network to perform feature extraction and feature denoising on the low-low-light image; Step 104: Use the trained detection and segmentation sub-network to perform target detection and segmentation on the denoised features to obtain a final instance segmentation result. 2. a kind of low-low-light image instance segmentation method based on feature denoising as shown in claim 1, it is characterized in that, in step 101, first, the ordinary natural illumination RGB image is converted into RAW image data; Then, use simulation Noise injection to simulate noise in images in low light conditions. 3 . The method for segmenting low-low-light image instances based on feature denoising according to claim 2 , wherein the injected simulation noise uses a full-element physical quantity noise model. 4 . 4. A low-low-light image instance segmentation method based on feature denoising as shown in claim 1, wherein in step 102, the total loss function L(θ) is expressed as: L(θ)=L _IS (x; θ)+αL _IS (x'; θ)+βL _DS (x,x'; θ) (1) Among them, L _IS and L _DS represent the instance segmentation task loss function and noise disturbance suppression loss function respectively; x and x′ represent the simulated clean image and simulated noisy image respectively; θ is the model parameter; α and β are the weights of the loss function; Among them, L _IS is adjusted according to the different models used for instance segmentation, and L _DS is expressed as:

Among them, f ⁽ⁱ⁾ is the feature extracted by the instance segmentation network f (i) layer; During the whole training process, the feature extraction network and detection and segmentation sub-network of the instance segmentation network are completed by one-time training in an end-to-end manner. 5. A low-low-light image instance segmentation method based on feature denoising as shown in claim 4, wherein the _LDS uses the clean image features extracted from the clean image as a guide, so that the model can be extracted on the noisy image. The features are as close as possible to the clean image features. 6. A low-low-light image instance segmentation method based on feature denoising as shown in claim 1, characterized in that, in step 103, first feature extraction is performed on the image, and in the feature extraction process, adaptive features are used to remove the image. Noise downsampling layers and learnable low-pass filtered convolution blocks to denoise features; The adaptive feature denoising downsampling layer, through the downsampling process of the convolutional neural network, makes full use of the local features for weighted averaging to reduce the noise level in the features; The adaptive feature denoising downsampling layer adaptively predicts different low-pass filters for each channel and each position of the feature map; the features after adaptive low-pass filtering and 2 times downsampling are the denoised features. feature:

Among them, X and Y are the input feature map and the denoised feature map respectively; W is the weighted weight predicted by the adaptive downsampling layer according to the input feature map; (c, i, j) represents the channel dimension and wide dimension of the feature map. , coordinates in high dimensions; S represents the position around the spatial position (i, j); φ() represents the weight prediction function of the adaptive downsampling layer. In order to ensure that the output weight is low-pass, the weight prediction function will predict The weights are normalized by softmax;

Represents the local feature position on which the prediction W _c,i,j depends; GP() represents the global pooling operation; p and q represent the horizontal and vertical coordinates on the convolution kernel, respectively. 7. A low-low-light image instance segmentation method based on feature denoising as shown in claim 6, wherein the learnable low-pass filtering convolution block is obtained by explicitly using a Filtering branch to improve the robustness of ordinary convolutional layers to feature noise, including two branches, one is the same branch as ordinary convolution, and the other is a branch composed of learnable low-pass filtering and 1×1 convolution ; The learnable low-pass filter weights are obtained from training and normalized by the softmax function to ensure that they are low-pass filters, which can locally smooth the features to reduce the noise level in the features, and denoise them through 1×1 convolution. The latter features are fused into the ordinary convolution branch; at the same time, the learnable low-pass filtering convolution block uses the reparameterization technique to become an ordinary convolution layer during inference; The reparameterization is to transform multiple parallel branches in the convolutional neural network into an equivalent single branch structure through parameter fusion; the learnable low-pass filter convolution block is reparameterized into a single 3×3 convolutional layer. The parameters of , are obtained by the following formula: W′ _3×3 [h,t,p,q]＝W _3×3 [h,t,h,t]+(W _1×1 [h,t,1,1]*W _LLPF [1,t ,p,q]) (5) Among them, W′ _3×3 is the convolution layer parameter after reparameterization of the learnable low-pass filter convolution block,

C ₁ and C ₂ respectively represent the number of input channels and output channels of the convolution layer, and R represents a real number;

are the common 3×3 convolutional layer parameters in the learnable low-pass filtered convolutional block;

is a 1×1 convolutional layer parameter;

is a learnable low-pass filter parameter, and the learnable low-pass filter shares a set of 3 × 3 weights for each input channel; h and t represent the coordinates of the output channel and input channel, respectively, and h∈{1 ,2,…,C ₂ }, t∈{1,2,…,C ₁ }; p and q represent the horizontal and vertical coordinates of the convolution kernel, respectively, p, q∈{1,2,3}. 8. A low-low-light image instance segmentation system based on feature denoising, characterized in that it includes a simulation low-low-light image synthesis module, a noise disturbance suppression learning module, an image feature extraction and feature denoising module, and a target detection, segmentation and extraction module. ; Among them, the simulated low and low light image synthesis module is used to establish a simulated low and low light image data set for training the instance segmentation model; this module assembles the input natural lighting image data into a simulated low and low light image data set; The noise perturbation suppression learning module is used to guide the instance segmentation model to learn robust image features, enabling it to reduce feature perturbations caused by higher levels of noise in low-low-light images; this module uses a simulated low-low-light image dataset to use noise perturbation suppression Learn and train the model, and output the trained instance segmentation model; The image feature extraction and feature denoising module uses adaptive low-pass filtering to reduce the noise level of image features in the instance segmentation network, and extracts stable and clean image features from the noisy low-light image; The target detection, segmentation and extraction module identifies and extracts the category, position, size and shape of the target of interest from the denoised image features, and obtains the final low-low-light image instance segmentation result; The connection relationship between the above modules is as follows: The output end of the simulated low and low light image synthesis module is connected with the input end of the noise disturbance suppression learning module; The output end of the noise disturbance suppression learning module is connected with the input end of the image feature extraction and feature denoising module; The output end of the image feature extraction and feature denoising module is connected with the input end of the target detection, segmentation and extraction module.

Images