WO2023082453A1 - 一种图像处理方法及装置 - Google Patents

一种图像处理方法及装置 Download PDF

Info

Publication number
WO2023082453A1
WO2023082453A1 PCT/CN2021/143563 CN2021143563W WO2023082453A1 WO 2023082453 A1 WO2023082453 A1 WO 2023082453A1 CN 2021143563 W CN2021143563 W CN 2021143563W WO 2023082453 A1 WO2023082453 A1 WO 2023082453A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
feature
noise
processed
module
Prior art date
Application number
PCT/CN2021/143563
Other languages
English (en)
French (fr)
Inventor
程剑杰
Original Assignee
深圳须弥云图空间科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳须弥云图空间科技有限公司 filed Critical 深圳须弥云图空间科技有限公司
Priority to KR1020247015759A priority Critical patent/KR20240127952A/ko
Priority to EP21963889.7A priority patent/EP4432215A1/en
Publication of WO2023082453A1 publication Critical patent/WO2023082453A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure relates to the field of computer technology, in particular to an image processing method and device.
  • embodiments of the present disclosure provide an image processing method, device, computer equipment, and computer-readable storage medium, so as to solve the problem of poor image restoration quality in the prior art.
  • the first aspect of the embodiments of the present disclosure provides an image processing method, the method comprising:
  • the weather type acquiring environmental noise features corresponding to the weather type in the image to be processed
  • an image processing device includes:
  • An image acquisition module configured to acquire images to be processed
  • a type determination module configured to determine the weather type corresponding to the image to be processed
  • a feature acquisition module configured to acquire environmental noise features corresponding to the weather type in the image to be processed according to the weather type
  • An image generation module configured to obtain the non-noise image features of the image to be processed according to the environmental noise features, and generate the The denoised image corresponding to the image to be processed.
  • a third aspect of the embodiments of the present disclosure provides a computer device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor implements the steps of the above method when executing the computer program.
  • a fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above method are implemented.
  • the embodiments of the present disclosure have the following beneficial effects: the embodiments of the present disclosure can acquire the image to be processed first; then, the weather type corresponding to the image to be processed can be determined; then, according to the weather type, Acquire the environmental noise features corresponding to the weather type in the image to be processed; then, according to the environmental noise features, obtain the non-noise image features of the image to be processed, and, according to the environmental noise features, The non-noise image features and the image to be processed generate a denoised image corresponding to the image to be processed.
  • this embodiment can obtain the environmental noise characteristics corresponding to the weather type for different weather types, and can perform denoising processing on the image to be processed according to the environmental noise characteristics corresponding to the weather type, and can use non-noise image features to denoise Therefore, the method provided by this embodiment can remove the noise caused by various weather types from the image to be processed, and can realize It can also restore image detail information, so that the same technical framework can be used to solve the problem of image quality degradation under more weather conditions and improve the effect of image denoising and enhancement.
  • FIG. 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure
  • FIG. 2 is a flowchart of an image processing method provided by an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of the network architecture of Resnet18 provided by an embodiment of the present disclosure.
  • FIG. 4 is a schematic diagram of a network architecture of an image enhancement model provided by an embodiment of the present disclosure
  • FIG. 5 is a schematic diagram of a network architecture of a noise feature extraction module provided by an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of a network architecture of a feature aggregation dense convolution module provided by an embodiment of the present disclosure
  • Fig. 7 is a block diagram of an image processing device provided by an embodiment of the present disclosure.
  • FIG. 8 is a schematic diagram of a computer device provided by an embodiment of the present disclosure.
  • the existing image enhancement algorithm due to the existing image enhancement algorithm based on deep learning and physical modeling, it can only solve the image quality degradation under certain extreme weather conditions, such as only realizing image deraining, or only realizing image Defog, but cannot use the same technical framework to solve image quality degradation under more weather conditions; and, in the process of denoising and restoring images taken under extreme weather, scenes often appear unclear and image details are lost
  • the problem is that it is difficult to completely remove noise such as rain lines, fog, and snowflakes in the image, and the image restoration quality is poor.
  • the present invention provides an image processing method.
  • this embodiment can obtain the environmental noise characteristics corresponding to the weather type for different weather types, and can process the image to be processed according to the environmental noise characteristics corresponding to the weather type Denoising processing, and image restoration and restoration of the denoised area can be performed using non-noise image features; therefore, the method provided in this embodiment can remove noise caused by various weather types from the image to be processed, and , it can realize the restoration of image detail information while removing the noise caused by various weather types, so that the same technical framework can be used to solve the problem of image quality degradation under more weather conditions and improve image denoising and enhancement Effect.
  • this embodiment of the present invention can be applied to the application scenario shown in FIG. 1 .
  • a terminal device 1 and a server 2 may be included.
  • the terminal device 1 can be hardware or software.
  • the terminal device 1 When the terminal device 1 is hardware, it can be various electronic devices that have the functions of collecting images, storing images and supporting communication with the server 2, including but not limited to smart phones, tablet computers, laptop computers, digital cameras , a monitor, a video recorder and a desktop computer, etc.; when the terminal device 1 is software, it can be installed in the electronic device as above.
  • the terminal device 1 may be implemented as multiple software or software modules, or may be implemented as a single software or software module, which is not limited in this embodiment of the present disclosure.
  • various applications may be installed on the terminal device 1, such as image collection applications, image storage applications, instant chat applications, and the like.
  • Server 2 may be a server that provides various services, for example, a background server that receives requests sent by terminal devices that establish a communication connection with it, and the background server can receive and analyze requests sent by terminal devices, and generate processing result.
  • the server 2 may be one server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in this embodiment of the present disclosure.
  • the server 2 may be hardware or software. When the server 2 is hardware, it may be various electronic devices that provide various services for the terminal device 1 . When the server 2 is software, it can be a plurality of software or software modules that provide various services for the terminal device 1, or a single software or software module that provides various services for the terminal device 1. No limit.
  • the terminal device 1 and the server 2 can be connected through a network for communication.
  • the network can be a wired network connected by coaxial cable, twisted pair and optical fiber, or a wireless network that can realize the interconnection of various communication devices without wiring, for example, Bluetooth (Bluetooth), Near Field Communication (Near Field Communication, NFC), infrared (Infrared), etc., which are not limited in this embodiment of the present disclosure.
  • the user can determine the image to be processed through the terminal device 1 , and choose to perform denoising processing on noise caused by weather in the image to be processed.
  • the server 2 may first determine the weather type corresponding to the image to be processed. Then, the server 2 may acquire, according to the weather type, the environmental noise features corresponding to the weather type in the image to be processed. Next, the server 2 can obtain the non-noise image features of the image to be processed according to the environmental noise features, and generate the The denoised image corresponding to the image to be processed. Finally, the server 2 can send the denoised image to the terminal device 1, so that the terminal device 1 can display the denoised image to the user.
  • this embodiment can obtain the environmental noise features corresponding to the weather type for different weather types, and can perform denoising processing on the image to be processed according to the environmental noise features corresponding to the weather type, and can use non-noise image features to Image restoration is performed on the denoised area; therefore, the method provided in this embodiment can remove the noise caused by various weather types from the image to be processed, and can realize the removal of noise caused by various weather types At the same time, it can also restore image detail information, so that the same technical framework can be used to solve the problem of image quality degradation under more weather conditions and improve the effect of image denoising and enhancement.
  • terminal device 1, the server 2, and the network may be adjusted according to actual requirements of the application scenario, which is not limited in this embodiment of the present disclosure.
  • Fig. 2 is a flowchart of an image processing method provided by an embodiment of the present disclosure.
  • An image processing method in FIG. 2 may be executed by the terminal device or the server in FIG. 1 .
  • the image processing method includes:
  • the image to be processed may be understood as an image or a video frame that requires denoising processing of noise caused by weather.
  • an image or a video frame of a video taken under extreme weather such as heavy fog, heavy rain, heavy snow, etc.
  • the terminal device can provide a page through which the user can upload an image and click a preset button to trigger denoising processing on the noise caused by the weather in the image. At this time, the This image is the image to be processed.
  • the characteristics of noise caused by different weather are different, for example, the shape and distribution of noise caused by different weather types are different.
  • the weather type may include rain, snow, and fog; if the weather type is rain, the noise caused by the weather type is a rain line, and the shape of the rain line is a line, and the rain line
  • the distribution mode of the rain line can include the direction of the rain line and the density of the rain line.
  • the density of the rain line can be divided into three types according to the density of the rain line: high density, medium density, and small density.
  • the direction of the rain line can be understood as the rain line The rain falling direction corresponding to the line; if the weather type is snow, the noise caused by the weather type is snowflakes, and the shape of the snowflakes is flake, and the distribution of the snowflakes can include the direction of the snowflakes and the density of the snowflakes.
  • the density can be divided into three categories according to the density of snowflakes: large density, medium density, and small density.
  • the direction of snowflakes can be understood as the falling direction of snowflakes corresponding to snowflakes; if the weather type is fog, the noise caused by this weather type is fog , and the distribution method of the mist may include the concentration of the mist, and the distribution position of the mist is a random position. Therefore, after the image to be processed is acquired, the weather type corresponding to the image to be processed can be determined first, so that the environmental noise feature can be extracted according to the characteristics of noise corresponding to different weather types.
  • an image processing algorithm may be used to classify the image to be processed by weather type, so as to determine the weather type corresponding to the image to be processed.
  • the neural network can also be used to classify the weather type of the image to be processed, so as to determine the weather type corresponding to the image to be processed. It should be noted that in this embodiment, other classification methods may also be used to classify the image to be processed by weather type, which will not be repeated here.
  • S203 According to the weather type, acquire environmental noise features corresponding to the weather type in the image to be processed.
  • the environmental noise characteristics corresponding to the weather type in the image to be processed can be obtained first according to the noise characteristics corresponding to the weather type of the image to be processed, so that the follow-up can be based on
  • the environmental noise feature corresponding to the weather type removes the noise corresponding to the weather type.
  • the environmental noise feature corresponding to a weather type can be understood as a feature that can reflect the distribution of the environmental noise corresponding to the weather type in the image to be processed, for example, the environmental noise feature corresponding to a weather type may include In the image to be processed, the shape, distribution density, distribution concentration, and distribution area of the noise corresponding to the weather type; it should be noted that the feature form of the environmental noise feature can be a feature map, of course, it can also be in the form of a feature matrix. This is not limited in this embodiment.
  • the noise distribution information related to the noise characteristics corresponding to the weather type can be extracted from the image to be processed according to the noise characteristics corresponding to the weather type . Then, the environmental noise feature corresponding to the weather type can be generated according to the noise distribution information.
  • the distribution information of the rain line such as the shape of the rain line, the direction of the rain line, the density of the rain line, the position of the rain line, etc. in the image to be processed can be extracted, and can be
  • the rain line noise feature is generated according to the distribution information of the rain line in the image to be processed. It can be understood that if the weather type is rain, the environmental noise feature is the rain line noise feature.
  • the distribution information of snowflakes such as the shape of the snowflakes in the image to be processed, the direction of the snowflakes, the density of the snowflakes, the position of the snowflakes, etc. can be extracted, and according to the image to be processed
  • the snowflake distribution information in generates snowflake noise features. It can be understood that if the weather type is snow, the environmental noise features are snowflake noise features.
  • the distribution information of the fog such as the concentration of the fog in the image to be processed, the location of the fog, etc. can be extracted, and can be generated according to the distribution information of the fog in the image to be processed
  • Fog noise features it can be understood that if the weather type is fog, the environmental noise features are fog noise features.
  • S204 Obtain non-noise image features of the image to be processed according to the environmental noise features, and generate the image to be processed according to the environmental noise features, the non-noise image features, and the image to be processed The corresponding denoised image.
  • the non-noise image feature is a feature that can reflect the details of the non-noise image area in the image to be processed, for example, the non-noise image feature can include texture features, edge features, color features, shape features, Feature information such as spatial features.
  • the environmental noise feature can be used to remove the noise corresponding to the weather type in the image to be processed; for example, the environmental noise feature can be used to determine the position of the noise corresponding to the weather type in the image to be processed, and the pixel corresponding to the position of the noise The pixel value of is adjusted to the preset initial value. It should be noted that since the environmental noise features are extracted based on the noise characteristics corresponding to the weather type, the acquired environmental noise features can more clearly reflect the noise corresponding to the weather type in the image to be processed (such as rain lines, fog, etc. , snowflakes), so using the environmental noise feature to remove the noise corresponding to the weather type in the image to be processed can improve the removal effect of the noise corresponding to the weather type in the image to be processed.
  • the method provided in this embodiment can be compatible to remove the noise caused by multiple weather types, that is, for each The noise caused by one type of weather can be removed by using the method provided in this embodiment, so that the same technical framework can be used to solve the problem of image quality degradation under more weather conditions.
  • the non-noise image feature of the image to be processed can be used to restore the image detail information to the area after the noise is removed.
  • the non-noise image features include the features of the details of the non-noise image area in the image to be processed, that is, the non-noise image feature can more clearly reflect the physical characteristics of the non-noise image area in the image to be processed; therefore, it can be Use the non-noise image features to calculate the feature relationship between the area after noise removal and the non-noise image area, so that the image details of the area after noise removal can be restored according to the non-noise image features, and the denoising image corresponding to the image to be processed can be obtained.
  • the degree of image detail recovery of the image to be processed after noise removal can be improved, thereby reducing the degree of distortion of the image to be processed after noise removal, thereby improving the clarity of the denoised image corresponding to the image to be processed.
  • the embodiment of the present disclosure can acquire the image to be processed first; then, the weather type corresponding to the image to be processed can be determined; type, to obtain the environmental noise feature corresponding to the weather type in the image to be processed; then, according to the environmental noise feature, the non-noise image feature of the image to be processed can be obtained, and, according to the environmental noise features, the non-noise image features, and the image to be processed, to generate a denoised image corresponding to the image to be processed.
  • this embodiment can obtain the environmental noise characteristics corresponding to the weather type for different weather types, and can perform denoising processing on the image to be processed according to the environmental noise characteristics corresponding to the weather type, and can use non-noise image features to denoise Therefore, the method provided by this embodiment can remove the noise caused by various weather types from the image to be processed, and can realize It can also restore image detail information, so that the same technical framework can be used to solve the problem of image quality degradation under more weather conditions and improve the effect of image denoising and enhancement.
  • the weather type classification model may be a residual network obtained by training based on sample images and weather type labels corresponding to the sample images.
  • the residual network may be a Resent18 network as shown in FIG. 3 .
  • several sets of training samples may be preset, and each set of training samples includes a sample image and a weather type label corresponding to the sample image.
  • the method of using the preset training samples to train the residual network can be as follows: input the sample image in each group of training samples into the residual network to obtain the forecast weather type corresponding to the sample image, and then, according to the corresponding The weather type label and the predicted weather type determine the loss value, and then use the loss value to adjust the network parameters of the residual network until the residual network meets the training conditions, such as the network parameter fitting of the residual network, Or if the number of training times meets the preset training threshold, a trained weather type classification model can be obtained. For example, assuming that the preset weather types include rain, fog, and snow, after inputting the image to be processed into the trained weather type classification model, the weather type of the image to be processed can be obtained as one of rain, fog, and snow .
  • S203 “According to the weather type, acquire the environmental noise features corresponding to the weather type in the image to be processed” may include the following steps:
  • S203a According to the weather type, determine an image enhancement model corresponding to the weather type; wherein, the image enhancement model includes a noise feature extraction module.
  • different image enhancement models may be set for different weather types. It should be noted that the model network architectures of the image enhancement models corresponding to different weather types are the same, and the only difference is that the training samples of the image enhancement models corresponding to different weather types are different.
  • the sample image in the training sample of the image enhancement model corresponding to the weather type "rain” is an image with rain line noise
  • the sample denoising image corresponding to the sample image is an image that removes rain line noise and restores image details
  • the sample image in the training sample of the image enhancement model corresponding to the weather type "snow” is an image with snowflake noise
  • the sample denoising image corresponding to the sample image is an image that removes snowflake noise and restores image details
  • the weather type "fog” The sample images in the training samples of the corresponding image enhancement model are images with fog noise
  • the sample denoising images corresponding to the sample images are images with fog noise removed and image details restored.
  • model network architectures of the image enhancement models corresponding to different weather types are the same, but the model network parameters of the image enhancement models corresponding to different weather types may be different; it should be emphasized that, Since the implementation methods described later are all introduced for the implementation methods at the network architecture level, the implementation methods described later are applicable to image enhancement models corresponding to all weather types.
  • the same image enhancement model can be set for all weather types, that is, all weather types correspond to the same image enhancement model.
  • the image The enhanced model is trained based on the sample images corresponding to all weather types and the denoised images corresponding to the sample images. It should be noted that, in this embodiment, a case where different image enhancement models are respectively set for different weather types will be used as an example for illustration.
  • the image enhancement model corresponding to the weather type can be determined among all trained image enhancement models according to the weather type .
  • the image enhancement model corresponding to each weather type is provided with a type identification corresponding to the weather type. Therefore, after obtaining the weather type of the image to be processed, according to the type identification corresponding to the weather type, all Among the trained image enhancement models, it is determined that the type identifier is the same as the type identifier corresponding to the weather type.
  • S203b Input the image to be processed into the noise feature extraction module to obtain the environmental noise feature corresponding to the weather type in the image to be processed.
  • the image enhancement model may include a noise feature extraction module. After the image to be processed is obtained, the image enhancement model can be used to obtain the environmental noise feature corresponding to the weather type in the image to be processed, so that the environmental noise feature can be used to perform noise removal.
  • the noise feature extraction module may be a multi-scale feature extraction network module (Feature Block).
  • the noise feature extraction module It may include N dilated convolution layers and aggregated convolution layers, wherein the convolution kernel dilation sizes of each dilated convolution layer are different, and N is a positive integer greater than 1.
  • the expansion size of the convolution kernel of each expansion convolution layer is different, the number of interval pixels between adjacent pixels collected by each expansion convolution layer during the convolution calculation process is different , so that the size of the receptive field of each expansion convolution layer will be different, and the first noise features extracted by each expansion convolution layer will also be different (for example, the size is different, the image feature information contained are not the same); in this way, the first noise features of different scales can be extracted, such as rain lines of different sizes and fog of different concentrations, and more context space information can be aggregated during the training process, and the training parameters are fewer, making Models are easier to train to fit.
  • the image to be processed may be firstly input into the N dilated convolution layers respectively to obtain N first noise features, where each first noise feature has a different size.
  • the aggregation convolutional layer can be used to aggregate the N first noise features to obtain the environmental noise features corresponding to the weather type in the image to be processed, due to the different scales of the first noise features
  • the included image feature information is different, so that in the process of using the N first noise features to aggregate to obtain environmental noise features, more context space information can be aggregated, and the information in the aggregated environmental noise features can be enriched , so that the extraction of noise features in the image to be processed can be significantly improved, that is, the acquired environmental noise features can more clearly reflect the noise corresponding to the weather type in the image to be processed.
  • the image to be processed is respectively input into the first dilated convolution layer, the second dilated convolution layer, the third dilated convolution layer and the fourth dilated convolution layer to obtain four first noise features.
  • the two first noise features output by the layer are input to the second aggregation convolutional layer to obtain the second aggregation sub-feature; the first sub-aggregation feature and the second sub-aggregation feature are input to the third aggregation convolution layer to obtain the image to be processed
  • the environmental noise characteristics corresponding to the weather type in is respectively input into the first dilated convolution layer, the second dilated convolution layer, the third dilated convolution layer and the fourth dilated convolution layer to obtain four first noise features.
  • a convolution layer with a convolution kernel of 1*1 is used to aggregate features of different scales. After three feature aggregations, the feature information (that is, environmental noise features) is more abundant. For the environment of weather images Noise feature extraction has been significantly improved.
  • S204 "obtain the non-noise image features of the image to be processed according to the environmental noise features, and generate The denoising image corresponding to the image to be processed" may include the following steps:
  • S204a Input the environmental noise feature into the convolution layer to obtain a first feature map.
  • the image enhancement model further includes a convolutional layer. After the environmental noise features are acquired, the environmental noise features are input into the convolution layer for convolution processing to obtain a first feature map.
  • S204b Input the first feature map into the encoding network layer to obtain P downsampled feature maps and P non-noise image features, where P is a positive integer greater than 1.
  • the image enhancement model further includes an encoding network layer.
  • the encoding network layer may include P cascaded encoding network modules, and each encoding network module includes a feature aggregation dense convolution module and a maximum pooling layer; in one implementation, as shown in Figure 4,
  • An encoding network module can include a feature aggregation dense convolution module and a maximum pooling layer.
  • the feature aggregation dense convolution module can be FJDB (Feature Joint Dense Block) in Figure 4
  • the maximum pooling layer can be Figure 4. Maxpooling in .
  • the first feature map in the encoding stage, can be input into the encoding network layer, and the first feature is extracted by using P cascaded encoding network modules in the encoding network layer to obtain P down-sampled feature map and P non-noise image features;
  • the P cascaded coding network modules of the coding network layer can be understood as a downsampling convolutional layer, and the feature maps output by the P cascading coding network modules The sizes are different, and the size of the feature map output by the encoding network module that is ranked later is smaller.
  • the downsampling feature map can be understood as the sum of the features of the same scale, and the downsampling feature map can be used to remove the noise caused by weather;
  • the non-noise image feature is the feature in the pooling stage (that is, the encoding network module) Extraction stage)
  • the image details lost in this stage are recorded by Pooling Indices, and the non-noise image features can be used to guide the recovery encoding network module in the upsampling stage (that is, the subsequent decoding stage).
  • the detailed features lost in the extraction stage can be understood as the sum of the features of the same scale, and the downsampling feature map can be used to remove the noise caused by weather;
  • the non-noise image feature is the feature in the pooling stage (that is, the encoding network module) Extraction stage)
  • the image details lost in this stage are recorded by Pooling Indices, and the non-noise image features can be used to guide the recovery encoding network module in the upsampling stage (that is, the
  • the first feature map can be input into the feature aggregation dense convolution module in the first encoding network module to obtain the downsampled feature map and a non-noise feature map output by the maximum pooling layer in the first encoding network module image features.
  • the downsampled feature map output by the i-1th encoding network module can be input into the feature aggregation dense convolution module in the i-th encoding network module to obtain the downsampling feature output by the maximum pooling layer in the i-th encoding network module graph and a non-noise image feature; i is a positive integer greater than 1 and less than or equal to P.
  • the encoding network layer may include three cascaded encoding network modules, which are respectively the first encoding network module, the second encoding network module and the third encoding network module; and each Each encoding network module includes a feature aggregation dense convolution module (FIDB) and a maximum pooling layer (Maxpooling).
  • FIDB feature aggregation dense convolution module
  • Maxpooling maximum pooling layer
  • the first feature map F1 is input to the first encoding network module, and the first encoding network module outputs a downsampling feature map F2 and a non-noise image feature; the downsampling feature map F2 output by the first encoding network module is input to the second
  • the feature aggregation dense convolution module in the first encoding network module obtains the down-sampling feature map F3 and a non-noise image feature output by the maximum pooling layer output in the second encoding network module; the output of the second encoding network module
  • the downsampling feature map F 3 is input to the feature aggregation dense convolution module in the third encoding network module, and the downsampling feature map F 4 and a non-noise image feature output by the maximum pooling layer in the third encoding network module are obtained.
  • S205c Input the P downsampled feature maps and the P non-noise image features into a decoding network layer to obtain a denoising feature map;
  • the image enhancement model further includes a decoding network layer.
  • the decoding network layer includes P cascaded decoding network modules, and each decoding network module includes a feature aggregation dense convolution module and an upsampling maximum pooling layer; in one implementation, as shown in Figure 4 , a decoding network module can include a feature aggregation dense convolution module and an upsampling maximum pooling layer, for example, the feature aggregation dense convolution module can be FIDB (Feature Joint Dense Block) in Figure 4, upsampling maximum pooling The layer can be UpMaxpooling in Figure 4. Wherein, each decoding network module corresponds to an encoding network module, and the input of the decoding network module includes a downsampled feature map output by its corresponding encoding network module and a non-noise image feature.
  • FIDB Feature Joint Dense Block
  • the image to be processed, P downsampled feature maps and P non-noise image features can be input into P cascaded decoding network modules of the decoding network layer, so that the P stages
  • the connected decoding network module uses the P downsampled feature maps to perform noise removal corresponding to the weather type, and uses the P non-noise image features and the restoration of image detail information to obtain a denoising feature map .
  • the P cascaded decoding network modules of the decoding network layer can be understood as an upsampling convolutional layer, and the sizes of the feature maps output by the P cascaded decoding network modules are different, and the order is more in The size of the feature map output by the subsequent decoding network module is larger.
  • the long-term spatial feature dependency can be calculated by integrating the encoding network layer and the decoding network layer.
  • the pooling index (Pooling Indices) is used to record the loss and supplement of the image detail information in the downsampling stage (that is, the encoding stage) and the upsampling stage (that is, the decoding stage), and each maximum pooling layer (Maxpooling) in the encoding stage Both correspond to an upsampling maximum pooling layer (UpMaxpooling), and the maximum pooling layer (Maxpooling) guides the maximum pooling layer (UpMaxpooling) to upsample through the pooling index, so that more can be recovered in the upsampling phase Image details.
  • the downsampling feature map output by the Pth encoding network module can be input into the feature aggregation dense convolution module in the first decoding network module, and a non-noise image output by the Pth encoding network module
  • the feature is input to the upsampling maximum pooling layer in the first decoding network module to obtain an upsampling feature map output by the upsampling maximum pooling layer.
  • the downsampling feature map output by the P-jth encoding network module and the upsampling feature map output by the upsampling maximum pooling layer in the jth decoding network module can be input into the feature aggregation dense in the 1+jth decoding network module Convolution module; and, input a non-noisy image feature output by the P-jth encoding network module into the upsampling maximum pooling layer in the 1+jth decoding network module to obtain the 1+jth
  • the upsampling feature map output by the upsampling maximum pooling layer in the Pth decoding network module can be used as the denoising feature map.
  • the encoding network layer may include three cascaded encoding network modules, which are respectively the first encoding network module, the second encoding network module and the third encoding network module; the decoding The network layer may include three cascaded decoding network modules, namely a first decoding network module, a second decoding network module and a third decoding network module.
  • the subsampled feature map F4 output by the third encoding network module can be input into the feature aggregation dense convolution module in the first decoding network module, and a non-noise image feature output by the third encoding network module input to the upsampling maximum pooling layer in the first decoding network module to obtain an upsampling feature map F 5 output by the upsampling maximum pooling layer.
  • the downsampling feature map F 3 output by the second encoding network module and the upsampling feature map F 5 output by the upsampling maximum pooling layer in the first decoding network module can be input into the features in the second decoding network module Aggregating dense convolution modules; and, inputting a non-noise image feature output by the second encoding network module into an upsampling maximum pooling layer in the second decoding network module to obtain the second decoding network module An upsampled feature map F 6 output by the middle upsampling max pooling layer.
  • the downsampling feature map F 2 output by the first encoding network module and the upsampling feature map F 6 output by the upsampling maximum pooling layer in the second decoding network module can be input into the features in the third decoding network module Aggregating dense convolution modules; and, inputting a non-noise image feature output by the first encoding network module into the upsampling maximum pooling layer in the third decoding network module to obtain the third decoding network module
  • the upsampling feature map F 7 output by the upsampling maximum pooling layer in the middle upsampling network module can be used as the denoising feature map.
  • S206d Input the denoising feature map, the first feature map, the environmental noise features, and the image to be processed into the output layer to obtain a denoising image corresponding to the image to be processed.
  • the image enhancement model further includes an output layer.
  • the output layer can use the denoising feature map, the first The feature map, the environmental noise feature and the image to be processed are fused to obtain a denoised image corresponding to the image to be processed.
  • the output layer includes a first output layer and a second output layer, wherein the first output layer includes a convolutional layer with a convolution kernel size of 1*1 (1*1 1Conv) and a convolutional layer (Conv), the second output layer includes a convolutional layer (Conv) and an activation function layer (such as a Tanh function layer, namely Tanh).
  • the first output layer includes a convolutional layer with a convolution kernel size of 1*1 (1*1 1Conv) and a convolutional layer (Conv)
  • the second output layer includes a convolutional layer (Conv) and an activation function layer (such as a Tanh function layer, namely Tanh).
  • the input of the first output layer is the denoising feature map and the first feature map
  • the output of the first output layer is the first output feature map
  • the input of the second output layer is the first output feature map and the environmental noise feature F 0
  • the fusion feature of the second output layer is the second output feature map; then, the second output feature map can be fused with the image to be processed to obtain a denoised image corresponding to the image to be processed.
  • the feature aggregation dense convolution modules mentioned in the above embodiments all include M dilated convolution layers, densely connected network modules and fully connected layers.
  • the convolution kernel expansion size of each expansion convolution layer is different, and M is a positive integer greater than 1.
  • the M dilated convolutional layers are used to generate M second noise features according to feature maps (such as first feature maps, downsampled feature maps, and upsampled feature maps), wherein the size of each second noise feature is different.
  • feature maps such as first feature maps, downsampled feature maps, and upsampled feature maps
  • the feature maps can be respectively input into M dilated convolutional layers to obtain M second noise features.
  • the convolution kernel with a size of 3*3 can well extract noise features such as rain lines, fog, and snowflakes.
  • the number of channels of the feature map is kept the same in the encoding stage and the decoding stage, that is, the number of channels of the feature map output in the encoding network layer and the decoding network layer is the same. It can be seen that this implementation uses dilated convolution with different dilation sizes of the convolution kernel, and aggregates features of different scales together, which can improve the image information contained in the extracted features.
  • the use of multi-scale aggregation feature technology will make the extraction of noise features better, for example, it will be better than the use of a single size convolution kernel.
  • the features extracted by convolution are more abundant, so that the acquired environmental noise features can more clearly reflect the physical characteristics of the noise (such as rain lines, fog, snowflakes) corresponding to the weather type in the image to be processed, which in turn can improve the quality of the image to be processed.
  • the sharpness of the denoised image corresponding to the image can be feature blocks of different scales, and the output is feature blocks of unchanged size.
  • the densely connected network module is used to perform convolution calculation processing and data screening processing on the M second noise features to obtain multiple convolution feature maps.
  • the densely connected network module includes multiple densely connected modules, and each densely connected module includes a convolutional layer (Conv) and an activation function layer (such as a ReLu function layer, ie ReLu).
  • the fully connected layer is used to aggregate the M second noise features and the plurality of convolutional feature maps to obtain aggregated features.
  • the fully connected layer includes a concat function layer (that is, Concat) and a convolution layer (1*1Conv) with a convolution kernel of 1*1.
  • the densely connected network module includes 3 densely connected modules, which are respectively the first densely connected module, the second densely connected module and the third densely connected module, wherein, the M second densely connected modules
  • the fusion feature obtained after the noise feature is fused can be used as the input of the first dense connection module, and the fusion feature obtained after the fusion of M second noise features and the convolution feature map output by the first dense connection module can be used as the first dense connection module.
  • the input of the second dense connection module, the fusion feature obtained by fusing the M second noise features, the convolution feature map output by the first dense connection module and the convolution feature map output by the second dense connection module can be used as the second dense connection module.
  • the convolutional feature map output by the dense connection module can be used as the input of the fully connected layer, and the feature map output by the fully connected layer can be input to the feature maps of M expansion convolution layers (such as the first feature map, the downsampling feature map , upsampled feature maps) are fused (that is, aggregated) to obtain aggregated features.
  • the fused features obtained after the M second noise features are fused are input into a densely connected network module, and then the Concat layer in the fully connected layer is used to combine
  • the features of all stages are aggregated, so that the feature information in the downsampled feature map and non-noise image features obtained in the encoding stage can be described in more detail, and noise information such as rain lines, fog, and snowflakes can be removed in the decoding stage ( That is, the noise feature) restores more image details, thereby reducing the degree of distortion of the denoised image corresponding to the image to be processed.
  • the loss function of the image enhancement model may be a mean absolute error function.
  • the noise information i.e., noise features
  • this embodiment describes the prediction after passing through the noise information such as rain lines, fog, and snowflakes
  • the MAE Mean Absolute Error
  • the model parameters of the image enhancement model can be adjusted using the loss value until the training completion conditions are met, such as the number of training times reaching the preset number or image enhancement Model parameter fit for the model.
  • the MAE function is as shown in the following formula:
  • H, W, and C represent the height, width, and number of channels of the input image of the image enhancement model
  • Y i, j, k represent the real noise-free image
  • i, j, k Respectively represent the height, width, and number of channels of the image
  • L represents the loss value
  • Fig. 7 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure. As shown in Figure 7, the image processing device includes:
  • a type determining module 702 configured to determine the weather type corresponding to the image to be processed
  • a feature acquisition module 703, configured to acquire environmental noise features corresponding to the weather type in the image to be processed according to the weather type;
  • An image generation module 704 configured to obtain the non-noise image features of the image to be processed according to the environmental noise features, and generate the Denoising image corresponding to the image to be processed.
  • the type determining module 702 is configured to:
  • the image to be processed is input into the trained weather type classification model to obtain the corresponding weather type of the image to be processed;
  • the weather type classification model is a residual network obtained by training based on sample images and weather type labels corresponding to the sample images.
  • the feature acquisition module 703 is configured to:
  • the image enhancement model includes a noise feature extraction module
  • the noise feature extraction module Inputting the image to be processed into the noise feature extraction module to obtain the environmental noise feature corresponding to the weather type in the image to be processed; wherein, the environmental noise feature reflects that the environmental noise corresponding to the weather type is in The characteristics of the distribution in the image to be processed.
  • the noise feature extraction module includes N dilated convolution layers and aggregated convolution layers, wherein the convolution kernel dilation sizes of each dilated convolution layer are different, and N is a positive value greater than 1. integer;
  • the feature acquisition module 703 is specifically used for:
  • the aggregated convolutional layer is used to aggregate the N first noise features to obtain environmental noise features corresponding to the weather type in the image to be processed.
  • the image enhancement model further includes a convolutional layer, an encoding network layer, a decoding network layer, and an output layer;
  • the image generation module 704 is configured to:
  • the first feature map is input into the encoding network layer to obtain P downsampling feature maps and P non-noise image features; wherein, P is a positive integer greater than 1;
  • the encoding network layer includes P cascaded encoding network modules, and each encoding network module includes a feature aggregation dense convolution module and a maximum pooling layer;
  • the image generating module 704 is specifically used for:
  • the decoding network layer includes P cascaded decoding network modules, and each decoding network module includes a feature aggregation dense convolution module and an upsampling maximum pooling layer;
  • the image generating module 704 is specifically used for:
  • the upsampled feature map output by the upsampled maximum pooling layer in the Pth decoding network module is used as the denoising feature map.
  • the feature aggregation dense convolution module includes M dilated convolutional layers, a densely connected network module, and a fully connected layer; wherein the convolution kernel dilation sizes of each dilated convolutional layer are different, and M is a positive integer greater than 1;
  • the M dilated convolution layers are used to generate M second noise features according to the feature map, wherein the size of each second noise feature is different;
  • the densely connected network module is used to perform convolution calculation processing and data screening processing on the M second noise features to obtain multiple convolution feature maps;
  • the fully connected layer is used to aggregate the M second noise features and the plurality of convolutional feature maps to obtain aggregated features.
  • the loss function of the image enhancement model is a mean absolute error function.
  • the weather type includes at least one of the following: rain, snow, fog;
  • the environmental noise feature is the rain line noise feature; if the weather type is snow, the environmental noise feature is the snowflake noise feature; if the weather type is fog, the environmental noise feature is the characteristic of fog noise.
  • the image processing device includes: an image acquisition module, used to acquire an image to be processed; a type determination module, used to determine the weather type corresponding to the image to be processed; a feature acquisition module, used to obtain the image according to The weather type, acquiring the environmental noise feature corresponding to the weather type in the image to be processed; an image generation module, configured to obtain the non-noise image feature of the image to be processed according to the environmental noise feature, and, Generate a denoised image corresponding to the image to be processed according to the environmental noise feature, the non-noise image feature, and the image to be processed.
  • this embodiment can obtain the environmental noise characteristics corresponding to the weather type for different weather types, and can perform denoising processing on the image to be processed according to the environmental noise characteristics corresponding to the weather type, and can use non-noise image features to denoise Therefore, the method provided by this embodiment can remove the noise caused by various weather types from the image to be processed, and can realize It can also restore image detail information, so that the same technical framework can be used to solve the problem of image quality degradation under more weather conditions and improve the effect of image denoising and enhancement.
  • FIG. 8 is a schematic diagram of a computer device 8 provided by an embodiment of the present disclosure.
  • the computer device 8 of this embodiment includes: a processor 801 , a memory 802 , and a computer program 803 stored in the memory 802 and capable of running on the processor 801 .
  • the processor 801 executes the computer program 803
  • the steps in the foregoing method embodiments are implemented.
  • the processor 801 executes the computer program 803 the functions of the modules/modules in the above-mentioned device embodiments are realized.
  • the computer program 803 can be divided into one or more modules/modules, and one or more modules/modules are stored in the memory 802 and executed by the processor 801 to complete the present disclosure.
  • One or more modules/modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 803 in the computer device 8 .
  • the computer equipment 8 may be computer equipment such as desktop computers, notebooks, palmtop computers, and cloud servers.
  • the computer device 8 may include, but is not limited to, a processor 801 and a memory 802 .
  • FIG. 8 is only an example of computer equipment 8, and does not constitute a limitation to computer equipment 8. It may include more or less components than those shown in the illustration, or combine certain components, or different components. , for example, computer equipment may also include input and output equipment, network access equipment, bus, and so on.
  • the processor 801 can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), on-site Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the storage 802 may be an internal storage module of the computer device 8 , for example, a hard disk or a memory of the computer device 8 .
  • Memory 802 also can be the external storage device of computer equipment 8, for example, the plug-in type hard disk equipped on computer equipment 8, smart memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash memory card ( Flash Card), etc.
  • the memory 802 may also include both an internal storage module of the computer device 8 and an external storage device.
  • the memory 802 is used to store computer programs and other programs and data required by the computer equipment.
  • the memory 802 can also be used to temporarily store data that has been output or will be output.
  • Module completion means that the internal structure of the device is divided into different functional modules or modules to complete all or part of the functions described above.
  • the functional modules and modules in the embodiments can be integrated into one processing module, or each module can exist separately physically, or two or more modules can be integrated into one module, and the above-mentioned integrated modules can either use hardware It can also be implemented in the form of software function modules.
  • the functional modules and the specific names of the modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present disclosure. For the specific working process of modules and modules in the above-mentioned system, reference may be made to the corresponding process in the aforementioned method embodiments, which will not be repeated here.
  • modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementation should not be considered beyond the scope of the present disclosure.
  • the disclosed apparatus/computer equipment and methods may be implemented in other ways.
  • the device/computer device embodiments described above are only illustrative, for example, the division of modules or modules is only a logical function division, and there may be other division methods in actual implementation, and multiple modules or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms.
  • a module described as a separate component may or may not be physically separated, and a component shown as a module may or may not be a physical module, that is, it may be located in one place, or may also be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional module in each embodiment of the present disclosure may be integrated into one processing module, each module may exist separately physically, or two or more modules may be integrated into one module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules.
  • the integrated modules/modules are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure realizes all or part of the processes in the methods of the above embodiments, and can also be completed by instructing related hardware through computer programs.
  • the computer programs can be stored in computer-readable storage media, and the computer programs can be processed. When executed by the controller, the steps in the above-mentioned method embodiments can be realized.
  • a computer program may include computer program code, which may be in source code form, object code form, executable file, or some intermediate form or the like.
  • the computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access Memory (Random Access Memory, RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in computer readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, computer readable media may not Including electrical carrier signals and telecommunication signals.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Processing (AREA)

Abstract

一种图像处理方法及装置。该方法包括：获取待处理图像；确定待处理图像对应的天气类型；根据天气类型，获取待处理图像中与天气类型对应的环境噪声特征；根据环境噪声特征，得到待处理图像的非噪声图像特征，以及，根据环境噪声特征、非噪声图像特征和待处理图像，生成待处理图像对应的去噪图像。由此能够使用同一种技术框架解决更多天气条件下的图像质量退化问题以及提高了图像去噪、增强的效果。

Description

一种图像处理方法及装置 技术领域
本公开涉及计算机技术领域，尤其涉及一种图像处理方法及装置。
背景技术
当室外环境出现大雾、大雨、大雪等极端天气时，室外摄像监控设备所拍摄到的图像会存在场景不清晰、图像细节丢失的问题，限制了极端天气下图像识别和视频监控在交通监控、目标跟踪、自主导航等领域的应用。
现有的基于深度学习和物理建模的图像增强算法，只能解决某一种极端天气条件下的图像质量退化，比如只能实现图像去雨，或者只能实现图像去雾，而不能够使用同一种技术框架解决更多天气条件下的图像质量退化；并且，在极端天气下所拍摄的图像进行去噪恢复的过程中，常常会出现场景不清晰、图像细节丢失的问题，即导致出现难以彻底地去除图像中的雨线、雾气、雪花等噪声，图像恢复质量较差的问题。
发明内容
有鉴于此，本公开实施例提供了一种图像处理方法、装置、计算机设备及计算机可读存储介质，以解决现有技术中图像恢复质量较差的问题。
本公开实施例的第一方面，提供了一种图像处理方法，所述方法包括：
获取待处理图像；
确定所述待处理图像对应的天气类型；
根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；
根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。
本公开实施例的第二方面，提供了一种图像处理装置，所述装置包括：
图像获取模块，用于获取待处理图像；
类型确定模块，用于确定所述待处理图像对应的天气类型；
特征获取模块，用于根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；
图像生成模块，用于根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。
本公开实施例的第三方面，提供了一种计算机设备，包括存储器、处理器以及存储在存储器中并且可以在处理器上运行的计算机程序，该处理器执行计算机程序时实现上述方法的步骤。
本公开实施例的第四方面，提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机程序，该计算机程序被处理器执行时实现上述方法的步骤。
本公开实施例与现有技术相比存在的有益效果是：本公开实施例可以先获取待处理图像；然后，可以确定所述待处理图像对应的天气类型；接着，可以根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；紧接着，可以根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。由于本实施例可以针对不同的天气类型获取该天气类型对应的环境噪声特征，并可以根据该天气类型对应的环境噪声特征对待处理图像进行去噪处理，以及，可以利用非噪声图像特征对去噪后的区域进行图像 还原恢复；因此，本实施例所提供的方法可以将各种天气类型所导致的噪声从待处理图像中去除，以及，能够实现在去除各种天气类型所导致的噪声的同时还可以恢复图像细节信息，从而可以实现能够使用同一种技术框架解决更多天气条件下的图像质量退化问题以及提高了图像去噪、增强的效果。
附图说明
为了更清楚地说明本公开实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图。
图1是本公开实施例的应用场景的场景示意图；
图2是本公开实施例提供的图像处理方法的流程图；
图3是本公开实施例提供的Resnet18的网络架构示意图；
图4是本公开实施例提供的图像增强模型的网络架构示意图；
图5是本公开实施例提供的噪声特征提取模块的网络架构示意图；
图6是本公开实施例提供的特征聚合密集卷积模块的网络架构示意图；
图7是本公开实施例提供的图像处理装置的框图；
图8是本公开实施例提供的计算机设备的示意图。
具体实施方式
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、技术之类的具体细节，以便透彻理解本公开实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本公开。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本公开的描述。
下面将结合附图详细说明根据本公开实施例的一种图像处理方法和装置。
在现有技术中，由于现有的基于深度学习和物理建模的图像增强算法，只能解决某一种极端天气条件下的图像质量退化，比如只能实现图像去雨，或者只能实现图像去雾，而不能够使用同一种技术框架解决更多天气条件下的图像质量退化；并且，在极端天气下所拍摄的图像进行去噪恢复的过程中，常常会出现场景不清晰、图像细节丢失的问题，即导致出现难以彻底地去除图像中的雨线、雾气、雪花等噪声，图像恢复质量较差的问题。
为了解决上述问题。本发明提供了一种图像处理方法，在本方法中，由于本实施例可以针对不同的天气类型获取该天气类型对应的环境噪声特征，并可以根据该天气类型对应的环境噪声特征对待处理图像进行去噪处理，以及，可以利用非噪声图像特征对去噪后的区域进行图像还原恢复；因此，本实施例所提供的方法可以将各种天气类型所导致的噪声从待处理图像中去除，以及，能够实现在去除各种天气类型所导致的噪声的同时还可以恢复图像细节信息，从而可以实现能够使用同一种技术框架解决更多天气条件下的图像质量退化问题以及提高了图像去噪、增强的效果。
举例说明，本发明实施例可以应用到如图1所示的应用场景。在该场景中，可以包括终端设备1和服务器2。
终端设备1可以是硬件，也可以是软件。当终端设备1为硬件时，其可以是具有采集图像、存储图像功能且支持与服务器2通信的各种电子设备，包括但不限于智能手机、平板电脑、膝上型便携计算机、数码照像机、监控器、录像机和台式计算机等；当终端设备1为软件时，其可以安装在如上该的电子设备中。终端设备1可以实现为多个软件或软件模块，也可以实现为单个软件或软件模块，本公开实施例对此不作限制。进一步地，终端设备1上可以安装有各种应用，例如图像采集应用、图像存储应用、即时聊天应用等。
服务器2可以是提供各种服务的服务器，例如，对与其建立通信连接的终端设备发送的 请求进行接收的后台服务器，该后台服务器可以对终端设备发送的请求进行接收和分析等处理，并生成处理结果。服务器2可以是一台服务器，也可以是由若干台服务器组成的服务器集群，或者还可以是一个云计算服务中心，本公开实施例对此不作限制。
需要说明的是，服务器2可以是硬件，也可以是软件。当服务器2为硬件时，其可以是为终端设备1提供各种服务的各种电子设备。当服务器2为软件时，其可以是为终端设备1提供各种服务的多个软件或软件模块，也可以是为终端设备1提供各种服务的单个软件或软件模块，本公开实施例对此不作限制。
终端设备1与服务器2可以通过网络进行通信连接。网络可以是采用同轴电缆、双绞线和光纤连接的有线网络，也可以是无需布线就能实现各种通信设备互联的无线网络，例如，蓝牙(Bluetooth)、近场通信(Near Field Communication，NFC)、红外(Infrared)等，本公开实施例对此不作限制。
具体地，用户可以通过终端设备1确定待处理图像，以及选择需要对该待处理图像中由于天气原因所导致的噪声进行去噪处理。服务器2接收到该待处理图像后，服务器2可以先确定所述待处理图像对应的天气类型。然后，服务器2可以根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征。接着，服务器2可以根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、和所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。最后，服务器2可以将该去噪图像向终端设备1发送，以便终端设备1向用户展示该去噪图像。这样，由于本实施例可以针对不同的天气类型获取该天气类型对应的环境噪声特征，并可以根据该天气类型对应的环境噪声特征对待处理图像进行去噪处理，以及，可以利用非噪声图像特征对去噪后的区域进行图像还原恢复；因此，本实施例所提供的方法可以将各种天气类型所导致的噪声从待处理图像中去除，以及，能够实现在去除各种天气类型所导致的噪声的同时还可以恢复图像细节信息，从而可以实现能够使用同一种技术框架解决更多天气条件下的图像质量退化问题以及提高了图像去噪、增强的效果。
需要说明的是，终端设备1和服务器2以及网络的具体类型、数量和组合可以根据应用场景的实际需求进行调整，本公开实施例对此不作限制。
需要注意的是，上述应用场景仅是为了便于理解本公开而示出，本公开的实施方式在此方面不受任何限制。相反，本公开的实施方式可以应用于适用的任何场景。
图2是本公开实施例提供的一种图像处理方法的流程图。图2的一种图像处理方法可以由图1的终端设备或服务器执行。如图2所示，该图像处理方法包括：
S201：获取待处理图像。
在本实施例中，待处理图像可以理解为需要对由于天气原因所导致的噪声进行去噪处理的图像或视频帧。例如，可以将在极端天气(比如大雾、大雨、大雪等)下所拍摄的图像或视频的视频帧作为待处理图像。作为一种示例，终端设备可以提供一个页面，用户可以通过该页面上传图像，并点击预设按键，以触发对该图像中由于天气原因所导致的噪声进行去噪处理，此时，便可以将该图像作为待处理图像。
S202：确定所述待处理图像对应的天气类型。
由于不同天气所导致的噪声的特点是不相同的，例如不同天气类型所导致的噪声的形状、分布方式是不相同的。举例来说，在一种实现方式中，天气类型可以包括雨、雪、雾；若天气类型为雨，则该天气类型所导致的噪声为雨线，而雨线的形状为线条状，雨线的分布方式可以包括雨线的方向和雨线的密度，其中，雨线的密度可以根据雨线的密集程度分为大密度、中密度、小密度等三类，雨线的方向可以理解为雨线对应的雨水落下方向；若天气类型为雪，则该天气类型所导致的噪声为雪花，而雪花的形状为片状，雪花的分布方式可以包括雪花的方向和雪花的密度，其中，雪花的密度可以根据雪花的密集程度分为大密度、中密度、小密度等三类，雪花的方向可以理解为雪花对应的雪花落下方向；若天气类型为雾，则该天气类型所导致的噪声为雾气，而雾气的分布方式可以包括雾气的浓度，而雾气分布位置 为随机位置。因此，获取到待处理图像后，可以先确定待处理图像对应的天气类型，以便后续可以根据不同天气类型对应的噪声的特点进行环境噪声特征的提取。
在本实施例中，可以利用图像处理算法对待处理图像进行天气类型分类处理，以确定待处理图像对应的天气类型。当然，还可以利用神经网络对待处理图像进行天气类型分类处理，以确定待处理图像对应的天气类型。需要说明的是，本实施例还可以采用其它分类方式对待处理图像进行天气类型分类，在此不再一一赘述。
S203：根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征。
为了将待处理图像中由于天气原因所导致的噪声去除，可以先根据待处理图像的天气类型对应的噪声特点，获取所述待处理图像中与该天气类型对应的环境噪声特征，以便后续可以根据与该天气类型对应的环境噪声特征去除该天气类型对应的噪声。可以理解的是，一天气类型对应的环境噪声特征可以理解为能够反映该天气类型对应的环境噪声在所述待处理图像中的分布情况的特征，例如，一天气类型对应的环境噪声特征可以包括在待处理图像中该天气类型对应的噪声的形状、分布密度、分布浓度、分布区域等情况；需要说明的是，环境噪声特征的特征形式可以为特征图，当然也可以为特征矩阵的形式，在本实施例中对此不进行限定。
也就是说，在本实施例中，确定待处理图像的天气类型后，可以先根据该天气类型对应的噪声特点，在待处理图像中提取与该天气类型对应的噪声特点相关的噪声分布情况信息。然后，可以根据该噪声分布情况信息生成该天气类型对应的环境噪声特征。
举例说明，假设待处理图像的天气类型为雨，则可以提取待处理图像中的雨线的形状、雨线的方向、雨线的密度、雨线的位置等雨线的分布情况信息，并可以根据待处理图像中的雨线的分布情况信息生成雨线噪声特征，可以理解的是，若所述天气类型为雨，则所述环境噪声特征为雨线噪声特征。
继续举例说明，假设待处理图像的天气类型为雪，则可以提取待处理图像中的雪花的形状、雪花的方向、雪花的密度、雪花的位置等雪花的分布情况信息，并可以根据待处理图像中的雪花的分布情况信息生成雪花噪声特征，可以理解的是，若所述天气类型为雪，所述环境噪声特征为雪花噪声特征。
继续举例说明，假设待处理图像的天气类型为雾，则可以提取待处理图像中的雾气的浓度、雾气的位置等雾气的分布情况信息，并可以根据待处理图像中的雾气的分布情况信息生成雾气噪声特征，可以理解的是，若所述天气类型为雾，所述环境噪声特征为雾气噪声特征。
S204：根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、和所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。
在本实施例中，在获取到环境噪声特征后，可以将待处理图像中除环境噪声特征对应的区域以外的区域作为非噪声图像区域，并可以对非噪声图像区域进行特征提取，得到非噪声图像特征。可以理解的是，非噪声图像特征为能够反映待处理图像中非噪声图像区域的细节的特征，例如，非噪声图像特征可以包括非噪声图像区域的纹理特征、边缘特征、颜色特征、形状特征、空间特征等特征信息。
然后，可以利用环境噪声特征对待处理图像中天气类型对应的噪声去除；例如，可以利用环境噪声特征，确定待处理图像中天气类型对应的噪声的位置，以及，将噪声的位置所对应的像素点的像素值调整为预设初始值。需要说明的是，由于环境噪声特征是基于天气类型对应的噪声特点所提取的，因此，所获取到的环境噪声特征能够更加清晰地反映待处理图像中天气类型对应的噪声(比如雨线、雾气、雪花)的物理特征，故利用利用环境噪声特征对待处理图像中天气类型对应的噪声去除，可以提高待处理图像中天气类型对应的噪声的去除效果。还需要强调的是，由于实施例中是针对不同天气类型对应的噪声特点所提取的环境 噪声特征，因此，本实施例所提供的方法可以兼容去除多种天气类型所导致的噪声，即针对每一种天气类型所导致的噪声均可以采用本实施例所提供的方法进行去除，从而可以实现能够使用同一种技术框架解决更多天气条件下的图像质量退化问题。
接着，可以利用待处理图像的非噪声图像特征，对去除噪声后的区域进行图像细节信息的恢复还原。需要说明的是，由于非噪声图像特征包括了待处理图像中非噪声图像区域的细节的特征，即非噪声图像特征能够更加清晰地反映待处理图像中非噪声图像区域的物理特征；因此，可以利用非噪声图像特征计算去除噪声后的区域与非噪声图像区域之间的特征联系，从而可以根据非噪声图像特征对去除噪声后的区域进行图像细节的还原，得到待处理图像对应的去噪图像，例如，根据非噪声图像特征计算去除噪声后的区域中像素点的目标像素值，并将去除噪声后的区域中像素点的像素值调整为目标像素值。这样，便可以提高去除噪声后的待处理图像的图像细节恢复程度，从而可以减少去除噪声后的待处理图像的失真程度，进而提高了待处理图像对应的去噪图像的清晰程度。
可见，本公开实施例与现有技术相比存在的有益效果是：本公开实施例可以先获取待处理图像；然后，可以确定所述待处理图像对应的天气类型；接着，可以根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；紧接着，可以根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。由于本实施例可以针对不同的天气类型获取该天气类型对应的环境噪声特征，并可以根据该天气类型对应的环境噪声特征对待处理图像进行去噪处理，以及，可以利用非噪声图像特征对去噪后的区域进行图像还原恢复；因此，本实施例所提供的方法可以将各种天气类型所导致的噪声从待处理图像中去除，以及，能够实现在去除各种天气类型所导致的噪声的同时还可以恢复图像细节信息，从而可以实现能够使用同一种技术框架解决更多天气条件下的图像质量退化问题以及提高了图像去噪、增强的效果。
接下来，将介绍S202的一种实现方式，即如何利用神经网络对待处理图像进行天气类型的分类。在本实施例中，S202“确定所述待处理图像对应的天气类型”可以包括以下步骤：
将所述待处理图像输入已训练的天气类型分类模型，得到所述待处理图像对应的天气类型。
在本实施例中，所述天气类型分类模型可以为基于样本图像以及样本图像对应的天气类型标签进行训练所得到的残差网络。在一种实现方式中，所述残差网络可以为如图3所示的Resent18网络。本实施例中，可以预先设置有若干组训练样本，每组训练样本包括样本图像和该样本图像对应的天气类型标签。利用预设的训练样本对残差网络进行训练的方式可以为：将每组训练样本中的样本图像输入残差网络中，得到该样本图像对应的预测天气类型，接着，根据该样本图像对应的天气类型标签和该预测天气类型确定损失值，紧接着，利用该损失值对该残差网络的网络参数进行调整，直至该残差网络满足训练条件，例如该残差网络的网络参数拟合，或者训练次数满足预设训练阈值，便可以得到已训练的天气类型分类模型。举例来说，假设预设的天气类型包括雨、雾、雪，则将待处理图像输入已训练的天气类型分类模型后，可以得到待处理图像的天气类型为雨、雾、雪中的一种。
接下来，将介绍S203的一种实现方式，即如何获取所述待处理图像中与所述天气类型对应的环境噪声特征。在本实施例中，S203“根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征”可以包括以下步骤：
S203a：根据所述天气类型，确定所述天气类型对应的图像增强模型；其中，所述图像增强模型包括噪声特征提取模块。
在本实施例的一种实现方式中，可以针对不同天气类型分别设置不同的图像增强模型。需要说明的是，不同的天气类型所对应的图像增强模型的模型网络架构均是相同的，区别仅在于不同的天气类型所对应的图像增强模型的训练样本是不相同。举例来说，假设天气类型“雨”对应的图像增强模型的训练样本中的样本图像为存在雨线噪声的图像，样本图像对应 的样本去噪图像为去除雨线噪声且恢复图像细节的图像；假设天气类型“雪”对应的图像增强模型的训练样本中的样本图像为存在雪花噪声的图像，样本图像对应的样本去噪图像为去除雪花噪声且恢复图像细节的图像；假设天气类型“雾”对应的图像增强模型的训练样本中的样本图像为存在雾气噪声的图像，样本图像对应的样本去噪图像为去除雾气噪声且恢复图像细节的图像。可以理解的是，不同的天气类型所对应的图像增强模型的模型网络架构均是相同的，但是不同的天气类型所对应的图像增强模型的模型网络参数可能是不相同的；需要强调的是，由于后续介绍的实现方法均是针对网络架构层面的实现方式进行介绍的，因此，后续介绍的实现方式适用于所有天气类型对应的图像增强模型。当然，在本实施例的另一种实现方式中，所有天气类型可以设置同一个图像增强模型，即所有天气类型均对应同一个图像增强模型，可以理解的是，在本实现方式中，该图像增强模型是基于所有天气类型对应的样本图像以及样本图像对应的去噪图像训练得到的。需要说明的是，在本实施例中，将主要以针对不同天气类型分别设置不同的图像增强模型的情况进行举例说明。
在针对不同天气类型分别设置不同的图像增强模型的情况下，获取到待处理图像的天气类型后，可以根据该天气类型，在所有已训练的图像增强模型中确定该天气类型对应的图像增强模型。作为一种示例，每种天气类型对应的图像增强模型均设置有与天气类型对应的类型标识，因此，在获取到待处理图像的天气类型后，可以根据该天气类型对应的类型标识，在所有已训练的图像增强模型中确定类型标识与该天气类型对应的类型标识相同的图像增强模型。
S203b：将所述待处理图像输入所述噪声特征提取模块，得到所述待处理图像中与所述天气类型对应的环境噪声特征。
在本实施例中，如图4所示，图像增强模型可以包括噪声特征提取模块。在获取到待处理图像后，可以先利用图像增强模型，获取待处理图像中与天气类型对应的环境噪声特征，以便后续利用该环境噪声特征进行噪声去除。
在一种实现方式中，如图4所示，噪声特征提取模块可以为多尺度特征提取网络模块(Feature Block)。该噪声特征提取模块提取环境噪声特征的过程可以定义为F 0＝H 0(I 0)，I 0表示待处理图像，H 0表示Feature Block的特征计算函数，F 0为得到的环境噪声特征。
为了能够提取不同形状、不同分布方式的噪声的特征(比如不同方向、不同大小的雨线噪声特征，不同浓度的雾气噪声特征，不同方向、不同大小的雪花噪声特征)，所述噪声特征提取模块可以包括N个膨胀卷积层和聚合卷积层，其中，每个膨胀卷积层的卷积核膨胀尺寸均不相同，且N为大于1的正整数。需要说明的是，由于每个膨胀卷积层的卷积核膨胀尺寸均不相同，因此，每个膨胀卷积层在卷积计算的过程中所采集的相邻像素之间的间隔像素数量不同，从而每个膨胀卷积层的感受野的大小也会不一样，进而每个膨胀卷积层所提取的第一噪声特征也会不相同(比如，尺寸大小不相同，所包含的图像特征信息也不相同)；这样可以提取到不同尺度的第一噪声特征，比如不同大小的雨线、不同浓度的雾气，在训练过程中能够聚合更多的上下文空间信息，并且训练的参数更少，使得模型更容易训练达到拟合。
具体地，在本实现方式中，可以先将所述待处理图像分别输入所述N个膨胀卷积层，得到N个第一噪声特征，其中，每个第一噪声特征的尺寸大小不相同。然后，可以利用所述聚合卷积层对所述N个第一噪声特征进行聚合处理，得到所述待处理图像中与所述天气类型对应的环境噪声特征，由于不同尺度的第一噪声特征所包含的图像特征信息不相同，使得利用所述N个第一噪声特征进行聚合得到环境噪声特征的过程中，可以聚合更多的上下文空间信息，以及使得聚合得到的环境噪声特征中的信息更加丰满，从而可以使得待处理图像中的噪声的特征的提取有显著的提高，即使得所获取到的环境噪声特征能够更加清晰地反映待处理图像中天气类型对应的噪声。
接下来将结合图5对S203b的具体实现方式进行举例说明。如图5所示，噪声特征提取模块可以包括4个膨胀卷积层和3个聚合卷积层；具体地，4个膨胀卷积层包括第一膨胀 卷积层(Conv 3*3 DF＝1)、第二膨胀卷积层(Conv 3*3 DF＝2)、第三膨胀卷积层(Conv 3*3 DF＝3)和第四膨胀卷积层(Conv 3*3 DF＝4)，所述聚合卷积层包括第一聚合卷积层(Conv 1*1 DF＝1)、第二聚合卷积层(Conv 1*1 DF＝1)和第三聚合卷积层(Conv 1*1 DF＝1)；其中，第一膨胀卷积层为卷积核为3*3、DF＝1(DF＝1代表卷积计算的过程中所采集的相邻像素之间没有间隔)的卷积层，第二膨胀卷积层为卷积核为3*3、DF＝2(DF＝2代表卷积计算的过程中所采集的相邻像素之间的间隔数量为一个像素)的卷积层，第三膨胀卷积层为卷积核为3*3、DF＝3(DF＝3代表卷积计算的过程中所采集的相邻像素之间的间隔数量为两个像素)的卷积层，第四膨胀卷积层为卷积核为3*3、DF＝4(DF＝4代表卷积计算的过程中所采集的相邻像素之间的间隔数量为三个像素)的卷积层，第一聚合卷积层、第二聚合卷积层和第三聚合卷积层均为卷积核为1*1、DF＝1的卷积层。具体地，将待处理图像分别输入第一膨胀卷积层、第二膨胀卷积层、第三膨胀卷积层和第四膨胀卷积层，得到四个第一噪声特征。将第一膨胀卷积层、第二膨胀卷积层输出的2个第一噪声特征输入第一聚合卷积层，得到第一子聚合特征；将第三膨胀卷积层、第四膨胀卷积层输出的2个第一噪声特征输入第二聚合卷积层，得到第二聚合子特征；将第一子聚合特征和第二子聚合特征输入第三聚合卷积层，得到所述待处理图像中与所述天气类型对应的环境噪声特征。需要说明的是，本示例中采用卷积核为1*1的卷积层来聚合不同尺度的特征，经过3次特征聚合得到的特征信息(即环境噪声特征)更加丰满，对于天气图像的环境噪声特征提取有显著的提高。
接下来，将介绍S204的一种实现方式，即如何生成待处理图像对应的去噪图像。在本实施例中，S204“根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像”可以包括以下步骤：
S204a：将所述环境噪声特征输入所述卷积层，得到第一特征图。
在本实施例中，如图4所示，所述图像增强模型还包括卷积层。在获取到环境噪声特征后，将所述环境噪声特征输入所述卷积层进行卷积处理，得到第一特征图。
S204b：将所述第一特征图输入所述编码网络层，得到P个下采样特征图和P个非噪声图像特征，其中，P为大于1的正整数。
在本实施例中，如图4所示，所述图像增强模型还包括编码网络层。所述编码网络层可以包括P个级联的编码网络模块，且每个编码网络模块均包括一特征聚合密集卷积模块和最大池化层；在一种实现方式中，如图4所示，一个编码网络模块可以包括一个特征聚合密集卷积模块和一个最大池化层，比如，特征聚合密集卷积模块可以为图4中的FJDB(Feature Joint Dense Block)，最大池化层可以为图4中的Maxpooling。
在本实施例中，在编码阶段中，可以将第一特征图输入所述编码网络层，利用编码网络层的P个级联的编码网络模块对第一特征进行特征提取，得到P个下采样特征图和P个非噪声图像特征；需要说明的是，编码网络层的P个级联的编码网络模块可以理解为下采样卷积层，P个级联的编码网络模块所输出的特征图的尺寸是不相同的，且排序越在后面的编码网络模块所输出的特征图的尺寸越小。其中，下采样特征图可以理解为相同尺度的特征的相加之和，下采样特征图可以用于去除天气原因所导致的噪声；非噪声图像特征为在池化阶段(即编码网络模块进行特征提取阶段)通过池化索引(Pooling Indices)所记录到的该阶段所丢失的图像细节，非噪声图像特征可以用于通过在上采样阶段(即后续的解码阶段)中指导恢复编码网络模块进行特征提取阶段中所丢失的细节特征。
具体地，可以将所述第一特征图输入第1个编码网络模块中的特征聚合密集卷积模块，得到第1个编码网络模块中的最大池化层输出的一下采样特征图和一非噪声图像特征。可以将第i-1个编码网络模块输出的下采样特征图输入第i个编码网络模块中的特征聚合密集卷积模块，得到第i个编码网络模块中的最大池化层输出的一下采样特征图和一非噪声图像特征；i为大于1且小于或等于P的正整数。
作为一种示例，如图4所示，所述编码网络层可以包括3个级联的编码网络模块，分 别为第一编码网络模块、第二编码网络模块和第三编码网络模块；且每个编码网络模块均包括一特征聚合密集卷积模块(即FIDB)和一个最大池化层(Maxpooling)。第一特征图F 1输入第一编码网络模块，第一编码网络模块输出一下采样特征图F 2和一非噪声图像特征；将第1个编码网络模块输出的下采样特征图F 2输入第2个编码网络模块中的特征聚合密集卷积模块，得到第2个编码网络模块中的最大池化层输出的一下采样特征图F 3和一非噪声图像特征；将第2个编码网络模块输出的下采样特征图F 3输入第3个编码网络模块中的特征聚合密集卷积模块，得到第3个编码网络模块中的最大池化层输出的一下采样特征图F 4和一非噪声图像特征。
S205c：将所述P个下采样特征图和所述P个非噪声图像特征输入解码网络层，得到去噪特征图；
在本实施例中，如图4所示，所述图像增强模型还包括解码网络层。所述解码网络层包括P个级联的解码网络模块，且每个解码网络模块均包括一特征聚合密集卷积模块和上采样最大池化层；在一种实现方式中，如图4所示，一个解码网络模块可以包括一个特征聚合密集卷积模块和一个上采样最大池化层，比如，特征聚合密集卷积模可以为图4中的FIDB(Feature Joint Dense Block),上采样最大池化层可以为图4中的UpMaxpooling。其中，每个解码网络模块均对应一个编码网络模块，解码网络模块的输入包括与其对应的编码网络模块所输出的下采样特征图和一非噪声图像特征。
在本实施例中，在解码阶段总，可以将待处理图像、P个下采样特征图和P个非噪声图像特征输入解码网络层的P个级联的解码网络模块，以便所述P个级联的解码网络模块利用所述P个下采样特征图对待处理图像进行天气类型对应的噪声去除，以及，利用所述P个非噪声图像特征，以及图像细节信息的恢复还原，得到去噪特征图。需要说明的是，解码网络层的P个级联的解码网络模块可以理解为上采样卷积层，P个级联的解码网络模块所输出的特征图的尺寸是不相同的，且排序越在后面的解码网络模块所输出的特征图的尺寸越大。可见，本实施例通过将编码网络层和解码网络层集成在一起，从而能够计算长期的空间特征依赖关系。其中采用了池化索引(Pooling Indices)来记录下采样阶段(即编码阶段)和上采样阶段(即解码阶段)的图像细节信息的丢失和补充，编码阶段的每一个最大池化层(Maxpooling)都对应一个上采样最大池化层(UpMaxpooling)，并由最大池化层(Maxpooling)通过池化索引来指导最大池化层(UpMaxpooling)进行上采样，这样就能够在上采样阶段恢复更多的图像细节。
具体地，可以将第P个编码网络模块输出的一下采样特征图输入第1个解码网络模块中的特征聚合密集卷积模块，以及，将所述第P个编码网络模块输出的一非噪声图像特征输入所述第1个解码网络模块中的上采样最大池化层，得到所述上采样最大池化层输出的一上采样特征图。可以将第P-j个编码网络模块输出的一下采样特征图以及第j个解码网络模块中上采样最大池化层输出的一上采样特征图，输入第1+j个解码网络模块中的特征聚合密集卷积模块；以及，将所述第P-j个编码网络模块输出的一非噪声图像特征输入所述第1+j个解码网络模块中的上采样最大池化层，得到所述第1+j个解码网络模块中上采样最大池化层输出的一上采样特征图；j为等于或大于1且小于P的正整数。可以将第P个解码网络模块中上采样最大池化层输出的上采样特征图作为去噪特征图。
作为一种示例，如图4所示，所述编码网络层可以包括3个级联的编码网络模块，分别为第一编码网络模块、第二编码网络模块和第三编码网络模块；所述解码网络层可以包括3个级联的解码网络模块，分别为第一解码网络模块、第二解码网络模块和第三解码网络模块。可以将第3个编码网络模块输出的一下采样特征图F 4输入第1个解码网络模块中的特征聚合密集卷积模块，以及，将所述第3个编码网络模块输出的一非噪声图像特征输入所述第1个解码网络模块中的上采样最大池化层，得到所述上采样最大池化层输出的一上采样特征图F 5。可以将第2个编码网络模块输出的一下采样特征图F 3以及第1个解码网络模块中上采样最大池化层输出的一上采样特征图F 5，输入第2个解码网络模块中的特征聚合密集卷 积模块；以及，将所述第2个编码网络模块输出的一非噪声图像特征输入所述第2个解码网络模块中的上采样最大池化层，得到所述第2解码网络模块中上采样最大池化层输出的一上采样特征图F 6。可以将第1个编码网络模块输出的一下采样特征图F 2以及第2个解码网络模块中上采样最大池化层输出的一上采样特征图F 6，输入第3个解码网络模块中的特征聚合密集卷积模块；以及，将所述第1个编码网络模块输出的一非噪声图像特征输入所述第3个解码网络模块中的上采样最大池化层，得到所述第3解码网络模块中上采样最大池化层输出的一上采样特征图F 7，可以将第3个解码网络模块中上采样最大池化层输出的上采样特征图F 7作为去噪特征图。
S206d：将所述去噪特征图、所述第一特征图、所述环境噪声特征和所述待处理图像输入所述输出层，得到所述待处理图像对应的去噪图像。
在本实施例中，如图4所示，所述图像增强模型还包括输出层。将去噪特征图、第一特征图、环境噪声特征和所述待处理图像输入所述输出层，得到所述待处理图像对应的去噪图像，例如输出层可以将去噪特征图、第一特征图、环境噪声特征和待处理图像相融合，得到待处理图像对应的去噪图像。
在一种实现方式中，如图4所示，所述输出层包括第一输出层和第二输出层，其中，第一输出层包括卷积核大小为1*1的卷积层(1*1Conv)和一卷积层(Conv)，第二输出层包括一卷积层(Conv)和一激活函数层(比如Tanh函数层，即Tanh)。其中，第一输出层的输入为去噪特征图和第一特征图，第一输出层的输出为第一输出特征图；第二输出层的输入为第一输出特征图和环境噪声特征F 0的融合特征，第二输出层的输出为第二输出特征图；接着，可以将第二输出特征图与待处理图像进行融合，得到待处理图像对应的去噪图像。
需要说明的是，在上述实施例所提及的特征聚合密集卷积模块均包括M个膨胀卷积层、密集连接网络模块和全连接层。其中，每个膨胀卷积层的卷积核膨胀尺寸均不相同，且M为大于1的正整数。
所述M个膨胀卷积层用于根据特征图(比如第一特征图、下采样特征图、上采样特征图)生成M个第二噪声特征，其中，每个第二噪声特征的尺寸大小不相同；具体地，可以将特征图分别输入M个膨胀卷积层，得到M个第二噪声特征。例如，如图6所示，一个特征聚合密集卷积模块可以包括3个膨胀卷积层，三个膨胀卷积层分别为第一子膨胀卷积层(Conv 3*3 DF＝1)、第二子膨胀卷积层(Conv 3*3 DF＝2)和第三子膨胀卷积层(Conv 3*3 DF＝3)；其中，第一子膨胀卷积层为卷积核为3*3、DF＝1(DF＝1代表卷积计算的过程中所采集的相邻像素之间没有间隔)的膨胀卷积层，第二子膨胀卷积层为卷积核为3*3、DF＝2(DF＝2代表卷积计算的过程中所采集的相邻像素之间的间隔数量为一个像素)的膨胀卷积层，第三子膨胀卷积层为卷积核为3*3、DF＝3(DF＝3代表卷积计算的过程中所采集的相邻像素之间的间隔数量为两个像素)的膨胀卷积层；本实施例采用三种卷积核膨胀尺寸的膨胀卷积来聚合不同尺度的特征信息(即特征图)，其中，尺寸为3*3的卷积核能够很好的提取雨线、雾气、雪花等噪声特征。另外，在编码阶段和解码阶段保持特征图的通道数一致，即编码网络层和解码网络层中所输出的特征图的通道数是相同的。可见，本实施了利用卷积核膨胀尺寸不同的膨胀卷积，并将不同尺度的特征聚合在一起，可以提高所提取的特征所包含的图像信息更加丰富；需要说明的是，对于雨线、雪花、雾气这种形状大小不规则且会随着风向随时变化的噪声特征而言，采用多尺度聚合特征的技术手段会使得提取噪声特征的效果更好，比如会比采用单一大小卷积核的卷积提取到的特征更丰富，从而可以使得获取到的环境噪声特征能够更加清晰地反映待处理图像中天气类型对应的噪声(比如雨线、雾气、雪花)的物理特征，进而可以提高待处理图像对应的去噪图像的清晰程度。其中，在一种实现方式中，特征聚合密集卷积模块的输入可以是不同尺度的特征块，输出是不改变尺寸的特征块。
所述密集连接网络模块用于对所述M个第二噪声特征进行卷积计算处理以及数据筛选处理，得到多个卷积特征图。在一种实现方式中，如图6所示，密集连接网络模块包括多个密集连接模块，且每个密集连接模块包括一个卷积层(Conv)和一个激活函数层(比如ReLu 函数层，即ReLu)。
所述全连接层用于对所述M个第二噪声特征和所述多个卷积特征图进行聚合处理，得到聚合特征。在一种实现方式中，如图6所示，所述全连接层包括一concat函数层(即Concat)和一卷积核为1*1的卷积层(1*1Conv)。
结合图6举例说明，如图6所示，假设密集连接网络模块包括3个密集连接模块，分别为第一密集连接模块、第二密集连接模块和第三密集连接模块，其中，M个第二噪声特征进行融合后所得到的融合特征可以作为第一密集连接模块的输入，M个第二噪声特征进行融合后所得到的融合特征和第一密集连接模块所输出的卷积特征图可以作为第二密集连接模块的输入，M个第二噪声特征进行融合后所得到的融合特征、第一密集连接模块所输出的卷积特征图和第二密集连接模块所输出的卷积特征图可以作为第三密集连接模块的输入；M个第二噪声特征进行融合后所得到的融合特征、第一密集连接模块所输出的卷积特征图、第二密集连接模块所输出的卷积特征图、第三密集连接模块所输出的卷积特征图可以作为全连接层的输入，可以将全连接层所输出的特征图与输入M个膨胀卷积层的特征图(比如第一特征图、下采样特征图、上采样特征图)相融合(即进行聚合处理)，得到聚合特征。
可见，本实施例M个尺度不同的第二噪声特征进行融合后，将M个第二噪声特征进行融合后所得到的融合特征输入一个密集连接网络模块，然后通过全连接层中的Concat层将所有阶段的特征聚合，这样能够使编码阶段中所得到的下采样特征图和非噪声图像特征中的特征信息更加细节的刻画，并能够在解码阶段中去除雨线、雾气、雪花等噪声信息(即噪声特征)后恢复更多的图像细节，从而实现减少待处理图像对应的去噪图像的失真程度。
还需要说明的是，所述图像增强模型的损失函数可以为平均绝对误差函数。需要说明的是，在本实施例中，由于雨线、雾气、雪花等噪声信息(即噪声特征)比较稀疏，并且本实施例刻画的是经过去处雨线、雾气、雪花等噪声信息之后的预测去噪图像f(x)与真实无噪声图像Y的相差程度，因此本实施例可以选择对稀疏特征比较敏感的MAE(Mean Absolute Error，平均绝对误差)函数作为训练的损失函数。这样，在利用损失函数确定用于模型训练的输入图像对应的损失值后，可以利用损失值对图像增强模型的模型参数进行调整，直至满足训练完成条件，例如训练次数达到预设次数或者图像增强模型的模型参数拟合。
在一种实现方式中，MAE函数如下公式所示：
Figure PCTCN2021143563-appb-000001
上式中，
Figure PCTCN2021143563-appb-000002
代表图像增强模型所输出的预测去噪图像；H、W、C分别代表图像增强模型的输入图像的高度、宽度、通道数；Y i，j，k代表真实无噪声图像；i、j、k分别代表图像的高度、宽度、通道数；L代表损失值。
上述所有可选技术方案，可以采用任意结合形成本公开的可选实施例，在此不再一一赘述。
下述为本公开装置实施例，可以用于执行本公开方法实施例。对于本公开装置实施例中未披露的细节，请参照本公开方法实施例。
图7是本公开实施例提供的图像处理装置的示意图。如图7所示，该图像处理装置包括：
图像获取模块701，用于获取待处理图像；
类型确定模块702，用于确定所述待处理图像对应的天气类型；
特征获取模块703，用于根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；
图像生成模块704，用于根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。
在一些实施例中，所述类型确定模块702，用于：
将所述待处理图像输入已训练的天气类型分类模型，得到所述待处理图像对应的天气 类型；
其中，所述天气类型分类模型为基于样本图像以及样本图像对应的天气类型标签进行训练所得到的残差网络。
在一些实施例中，所述特征获取模块703，用于：
根据所述天气类型，确定所述天气类型对应的图像增强模型；其中，所述图像增强模型包括噪声特征提取模块；
将所述待处理图像输入所述噪声特征提取模块，得到所述待处理图像中与所述天气类型对应的环境噪声特征；其中，所述环境噪声特征为反映所述天气类型对应的环境噪声在所述待处理图像中的分布情况的特征。
在一些实施例中，所述噪声特征提取模块包括N个膨胀卷积层和聚合卷积层，其中，每个膨胀卷积层的卷积核膨胀尺寸均不相同，且N为大于1的正整数；
所述所述特征获取模块703，具体用于：
将所述待处理图像分别输入所述N个膨胀卷积层，得到N个第一噪声特征，其中，每个第一噪声特征的尺寸大小不相同；
利用所述聚合卷积层对所述N个第一噪声特征进行聚合处理，得到所述待处理图像中与所述天气类型对应的环境噪声特征。
在一些实施例中，所述图像增强模型还包括卷积层、编码网络层、解码网络层和输出层；
所述图像生成模块704，用于：
将所述环境噪声特征输入所述卷积层，得到第一特征图；
将所述第一特征图输入所述编码网络层，得到P个下采样特征图和P个非噪声图像特征；其中，P为大于1的正整数；
将所述P个下采样特征图和所述P个非噪声图像特征输入解码网络层，得到去噪特征图；
将所述去噪特征图、所述第一特征图、所述环境噪声特征和所述待处理图像输入所述输出层，得到所述待处理图像对应的去噪图像。
在一些实施例中，所述编码网络层包括P个级联的编码网络模块，且每个编码网络模块均包括一特征聚合密集卷积模块和最大池化层；
所述图像生成模块704，具体用于：
将所述第一特征图输入第1个编码网络模块中的特征聚合密集卷积模块，得到第1个编码网络模块中的最大池化层输出的一下采样特征图和一非噪声图像特征；
将第i-1个编码网络模块输出的下采样特征图输入第i个编码网络模块中的特征聚合密集卷积模块，得到第i个编码网络模块中的最大池化层输出的一下采样特征图和一非噪声图像特征；i为大于1且小于或等于P的正整数。
在一些实施例中，所述解码网络层包括P个级联的解码网络模块，且每个解码网络模块均包括一特征聚合密集卷积模块和上采样最大池化层；
所述图像生成模块704，具体用于：
将第P个编码网络模块输出的一下采样特征图输入第1个解码网络模块中的特征聚合密集卷积模块，以及，将所述第P个编码网络模块输出的一非噪声图像特征输入所述第1个解码网络模块中的上采样最大池化层，得到所述上采样最大池化层输出的一上采样特征图；
将第P-j个编码网络模块输出的一下采样特征图以及第j个解码网络模块中上采样最大池化层输出的一上采样特征图，输入第1+j个解码网络模块中的特征聚合密集卷积模块；以及，将所述第P-j个编码网络模块输出的一非噪声图像特征输入所述第1+j个解码网络模块中的上采样最大池化层，得到所述第1+j个解码网络模块中上采样最大池化层输出的一上采样特征图；j为等于或大于1且小于P的正整数；
将第P个解码网络模块中上采样最大池化层输出的上采样特征图作为去噪特征图。
在一些实施例中，所述特征聚合密集卷积模块包括M个膨胀卷积层、密集连接网络模块和全连接层；其中，每个膨胀卷积层的卷积核膨胀尺寸均不相同，且M为大于1的正整数；
所述M个膨胀卷积层用于根据特征图生成M个第二噪声特征，其中，每个第二噪声特征的尺寸大小不相同；
所述密集连接网络模块用于对所述M个第二噪声特征进行卷积计算处理以及数据筛选处理，得到多个卷积特征图；
所述全连接层用于对所述M个第二噪声特征和所述多个卷积特征图进行聚合处理，得到聚合特征。
在一些实施例中，所述图像增强模型的损失函数为平均绝对误差函数。
在一些实施例中，所述天气类型包括以下至少一种：雨、雪、雾；
若所述天气类型为雨，所述环境噪声特征为雨线噪声特征；若所述天气类型为雪，所述环境噪声特征为雪花噪声特征；若所述天气类型为雾，所述环境噪声特征为雾气噪声特征。
根据本公开实施例提供的技术方案，图像处理装置包括：图像获取模块，用于获取待处理图像；类型确定模块，用于确定所述待处理图像对应的天气类型；特征获取模块，用于根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；图像生成模块，用于根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。由于本实施例可以针对不同的天气类型获取该天气类型对应的环境噪声特征，并可以根据该天气类型对应的环境噪声特征对待处理图像进行去噪处理，以及，可以利用非噪声图像特征对去噪后的区域进行图像还原恢复；因此，本实施例所提供的方法可以将各种天气类型所导致的噪声从待处理图像中去除，以及，能够实现在去除各种天气类型所导致的噪声的同时还可以恢复图像细节信息，从而可以实现能够使用同一种技术框架解决更多天气条件下的图像质量退化问题以及提高了图像去噪、增强的效果。
应理解，上述实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本公开实施例的实施过程构成任何限定。
图8是本公开实施例提供的计算机设备8的示意图。如图8所示，该实施例的计算机设备8包括：处理器801、存储器802以及存储在该存储器802中并且可以在处理器801上运行的计算机程序803。处理器801执行计算机程序803时实现上述各个方法实施例中的步骤。或者，处理器801执行计算机程序803时实现上述各装置实施例中各模块/模块的功能。
示例性地，计算机程序803可以被分割成一个或多个模块/模块，一个或多个模块/模块被存储在存储器802中，并由处理器801执行，以完成本公开。一个或多个模块/模块可以是能够完成特定功能的一系列计算机程序指令段，该指令段用于描述计算机程序803在计算机设备8中的执行过程。
计算机设备8可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算机设备。计算机设备8可以包括但不仅限于处理器801和存储器802。本领域技术人员可以理解，图8仅仅是计算机设备8的示例，并不构成对计算机设备8的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件，例如，计算机设备还可以包括输入输出设备、网络接入设备、总线等。
处理器801可以是中央处理模块(Central Processing Unit，CPU)，也可以是其它通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其它可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
存储器802可以是计算机设备8的内部存储模块，例如，计算机设备8的硬盘或内存。存储器802也可以是计算机设备8的外部存储设备，例如，计算机设备8上配备的插接式硬 盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。进一步地，存储器802还可以既包括计算机设备8的内部存储模块也包括外部存储设备。存储器802用于存储计算机程序以及计算机设备所需的其它程序和数据。存储器802还可以用于暂时地存储已经输出或者将要输出的数据。
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，仅以上述各功能模块、模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块、模块完成，即将装置的内部结构划分成不同的功能模块或模块，以完成以上描述的全部或者部分功能。实施例中的各功能模块、模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中，上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。另外，各功能模块、模块的具体名称也只是为了便于相互区分，并不用于限制本公开的保护范围。上述系统中模块、模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述或记载的部分，可以参见其它实施例的相关描述。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的模块及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本公开的范围。
在本公开所提供的实施例中，应该理解到，所揭露的装置/计算机设备和方法，可以通过其它的方式实现。例如，以上所描述的装置/计算机设备实施例仅仅是示意性的，例如，模块或模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，多个模块或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通讯连接可以是通过一些接口，装置或模块的间接耦合或通讯连接，可以是电性，机械或其它的形式。
作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络模块上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
另外，在本公开各个实施例中的各功能模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
集成的模块/模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读存储介质中。基于这样的理解，本公开实现上述实施例方法中的全部或部分流程，也可以通过计算机程序来指令相关的硬件来完成，计算机程序可以存储在计算机可读存储介质中，该计算机程序在被处理器执行时，可以实现上述各个方法实施例的步骤。计算机程序可以包括计算机程序代码，计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。计算机可读介质可以包括：能够携带计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、电载波信号、电信信号以及软件分发介质等。需要说明的是，计算机可读介质包含的内容可以根据司法管辖区内立法和专利实践的要求进行适当的增减，例如，在某些司法管辖区，根据立法和专利实践，计算机可读介质不包括电载波信号和电信信号。
以上实施例仅用以说明本公开的技术方案，而非对其限制；尽管参照前述实施例对本公开进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本公开各实施例技术方案的精神和范围，均应包含在本公开的保 护范围之内。

Claims (13)

  1. 一种图像处理方法，其特征在于，所述方法包括：
    获取待处理图像；
    确定所述待处理图像对应的天气类型；
    根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；
    根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。
  2. 根据权利要求1所述的图像处理方法，其特征在于，所述确定所述待处理图像对应的天气类型，包括：
    将所述待处理图像输入已训练的天气类型分类模型，得到所述待处理图像对应的天气类型；
    其中，所述天气类型分类模型为基于样本图像以及样本图像对应的天气类型标签进行训练所得到的残差网络。
  3. 根据权利要求1所述的图像处理方法，其特征在于，所述根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征，包括：
    根据所述天气类型，确定所述天气类型对应的图像增强模型；其中，所述图像增强模型包括噪声特征提取模块；
    将所述待处理图像输入所述噪声特征提取模块，得到所述待处理图像中与所述天气类型对应的环境噪声特征；其中，所述环境噪声特征为反映所述天气类型对应的环境噪声在所述待处理图像中的分布情况的特征。
  4. 根据权利要求3所述的图像处理方法，其特征在于，所述噪声特征提取模块包括N个膨胀卷积层和聚合卷积层，其中，每个膨胀卷积层的卷积核膨胀尺寸均不相同，且N为大于1的正整数；
    所述将所述待处理图像输入所述噪声特征提取模块，得到所述待处理图像中与所述天气类型对应的环境噪声特征，包括：
    将所述待处理图像分别输入所述N个膨胀卷积层，得到N个第一噪声特征，其中，每个第一噪声特征的尺寸大小不相同；
    利用所述聚合卷积层对所述N个第一噪声特征进行聚合处理，得到所述待处理图像中与所述天气类型对应的环境噪声特征。
  5. 根据权利要求3所述的图像处理方法，其特征在于，所述图像增强模型还包括卷积层、编码网络层、解码网络层和输出层；
    所述根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像，包括：
    将所述环境噪声特征输入所述卷积层，得到第一特征图；
    将所述第一特征图输入所述编码网络层，得到P个下采样特征图和P个非噪声图像特征；其中，P为大于1的正整数；
    将所述P个下采样特征图和所述P个非噪声图像特征输入解码网络层，得到去噪特征图；
    将所述去噪特征图、所述第一特征图、所述环境噪声特征和所述待处理图像输入所述输出层，得到所述待处理图像对应的去噪图像。
  6. 根据权利要求5所述的图像处理方法，其特征在于，所述编码网络层包括P个级联的编码网络模块，且每个编码网络模块均包括一特征聚合密集卷积模块和最大池化层；
    所述将所述第一特征图输入所述编码网络层，得到P个下采样特征图和P个非噪声图像特征，包括：
    将所述第一特征图输入第1个编码网络模块中的特征聚合密集卷积模块，得到第1个编码网络模块中的最大池化层输出的一下采样特征图和一非噪声图像特征；
    将第i-1个编码网络模块输出的下采样特征图输入第i个编码网络模块中的特征聚合密集卷积模块，得到第i个编码网络模块中的最大池化层输出的一下采样特征图和一非噪声图像特征；i为大于1且小于或等于P的正整数。
  7. 根据权利要求6所述的图像处理方法，其特征在于，所述解码网络层包括P个级联的解码网络模块，且每个解码网络模块均包括一特征聚合密集卷积模块和上采样最大池化层；
    所述将所述P个下采样特征图和所述P个非噪声图像特征输入解码网络层，得到去噪特征图，包括：
    将第P个编码网络模块输出的一下采样特征图输入第1个解码网络模块中的特征聚合密集卷积模块，以及，将所述第P个编码网络模块输出的一非噪声图像特征输入所述第1个解码网络模块中的上采样最大池化层，得到所述上采样最大池化层输出的一上采样特征图；
    将第P-j个编码网络模块输出的一下采样特征图以及第j个解码网络模块中上采样最大池化层输出的一上采样特征图，输入第1+j个解码网络模块中的特征聚合密集卷积模块；以及，将所述第P-j个编码网络模块输出的一非噪声图像特征输入所述第1+j个解码网络模块中的上采样最大池化层，得到所述第1+j个解码网络模块中上采样最大池化层输出的一上采样特征图；j为等于或大于1且小于P的正整数；
    将第P个解码网络模块中上采样最大池化层输出的上采样特征图作为去噪特征图。
  8. 根据权利要求6所述的图像处理方法，其特征在于，所述特征聚合密集卷积模块包括M个膨胀卷积层、密集连接网络模块和全连接层；其中，每个膨胀卷积层的卷积核膨胀尺寸均不相同，且M为大于1的正整数；
    所述M个膨胀卷积层用于根据特征图生成M个第二噪声特征，其中，每个第二噪声特征的尺寸大小不相同；
    所述密集连接网络模块用于对所述M个第二噪声特征进行卷积计算处理以及数据筛选处理，得到多个卷积特征图；
    所述全连接层用于对所述M个第二噪声特征和所述多个卷积特征图进行聚合处理，得到聚合特征。
  9. 根据权利要求3所述的图像处理方法，其特征在于，所述图像增强模型的损失函数为平均绝对误差函数。
  10. 根据权利要求1所述的图像处理方法，其特征在于，所述天气类型包括以下至少一种：雨、雪、雾；
    若所述天气类型为雨，所述环境噪声特征为雨线噪声特征；若所述天气类型为雪，所述环境噪声特征为雪花噪声特征；若所述天气类型为雾，所述环境噪声特征为雾气噪声特征。
  11. 一种图像处理装置，其特征在于，所述装置包括：
    图像获取模块，用于获取待处理图像；
    类型确定模块，用于确定所述待处理图像对应的天气类型；
    特征获取模块，用于根据所述天气类型，获取所述待处理图像中与所述天气类型对应的环境噪声特征；
    图像生成模块，用于根据所述环境噪声特征，得到所述待处理图像的非噪声图像特征，以及，根据所述环境噪声特征、所述非噪声图像特征和所述待处理图像，生成所述待处理图像对应的去噪图像。
  12. 一种计算机设备，包括存储器、处理器以及存储在所述存储器中并且可以在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求1所述方法的步骤。
  13. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在 于，所述计算机程序被处理器执行时实现如权利要求1所述方法的步骤。
PCT/CN2021/143563 2021-11-15 2021-12-31 一种图像处理方法及装置 WO2023082453A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020247015759A KR20240127952A (ko) 2021-11-15 2021-12-31 이미지 처리 방법 및 장치
EP21963889.7A EP4432215A1 (en) 2021-11-15 2021-12-31 Image processing method and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111348360.7 2021-11-15
CN202111348360.7A CN114140346A (zh) 2021-11-15 2021-11-15 一种图像处理方法及装置

Publications (1)

Publication Number Publication Date
WO2023082453A1 true WO2023082453A1 (zh) 2023-05-19

Family

ID=80393231

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/143563 WO2023082453A1 (zh) 2021-11-15 2021-12-31 一种图像处理方法及装置

Country Status (4)

Country Link
EP (1) EP4432215A1 (zh)
KR (1) KR20240127952A (zh)
CN (1) CN114140346A (zh)
WO (1) WO2023082453A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197487A (zh) * 2023-09-05 2023-12-08 东莞常安医院有限公司 一种免疫胶体金诊断试纸条自动识别系统

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114565540B (zh) * 2022-04-30 2022-07-22 深圳市巨力方视觉技术有限公司 基于多路对照图像去噪用机器视觉集成系统
CN116258647B (zh) * 2023-02-20 2023-11-28 阿里云计算有限公司 图像去噪方法，天气图像修复方法及计算设备
CN116310598B (zh) * 2023-05-16 2023-08-22 常州海图信息科技股份有限公司 一种用于恶劣天气下的障碍物检测方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190206091A1 (en) * 2017-12-29 2019-07-04 Baidu Online Network Technology (Beijing) Co., Ltd Method And Apparatus For Compressing Image
CN110717863A (zh) * 2019-08-16 2020-01-21 天津大学 一种基于生成对抗网络的单图像去雪方法
CN111970510A (zh) * 2020-07-14 2020-11-20 浙江大华技术股份有限公司 视频处理方法、存储介质及计算装置
CN112801888A (zh) * 2021-01-06 2021-05-14 杭州海康威视数字技术股份有限公司 图像处理方法、装置、计算机设备及存储介质
CN113436101A (zh) * 2021-06-28 2021-09-24 杭州电子科技大学 基于高效通道注意力机制的龙格库塔模块去雨的方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190206091A1 (en) * 2017-12-29 2019-07-04 Baidu Online Network Technology (Beijing) Co., Ltd Method And Apparatus For Compressing Image
CN110717863A (zh) * 2019-08-16 2020-01-21 天津大学 一种基于生成对抗网络的单图像去雪方法
CN111970510A (zh) * 2020-07-14 2020-11-20 浙江大华技术股份有限公司 视频处理方法、存储介质及计算装置
CN112801888A (zh) * 2021-01-06 2021-05-14 杭州海康威视数字技术股份有限公司 图像处理方法、装置、计算机设备及存储介质
CN113436101A (zh) * 2021-06-28 2021-09-24 杭州电子科技大学 基于高效通道注意力机制的龙格库塔模块去雨的方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197487A (zh) * 2023-09-05 2023-12-08 东莞常安医院有限公司 一种免疫胶体金诊断试纸条自动识别系统
CN117197487B (zh) * 2023-09-05 2024-04-12 东莞常安医院有限公司 一种免疫胶体金诊断试纸条自动识别系统

Also Published As

Publication number Publication date
EP4432215A1 (en) 2024-09-18
CN114140346A (zh) 2022-03-04
KR20240127952A (ko) 2024-08-23

Similar Documents

Publication Publication Date Title
WO2023082453A1 (zh) 一种图像处理方法及装置
WO2020125495A1 (zh) 一种全景分割方法、装置及设备
CN105657402B (zh) 一种深度图恢复方法
CN111079764B (zh) 一种基于深度学习的低照度车牌图像识别方法及装置
CN112861575A (zh) 一种行人结构化方法、装置、设备和存储介质
CN110825900A (zh) 特征重构层的训练方法、图像特征的重构方法及相关装置
CN115293992B (zh) 基于无监督权重深度模型的偏振图像去雾方法及装置
CN115082306A (zh) 一种基于蓝图可分离残差网络的图像超分辨率方法
CN113393385B (zh) 基于多尺度融合的无监督去雨方法、系统、装置及介质
Babu et al. An efficient image dahazing using Googlenet based convolution neural networks
CN113177483B (zh) 视频目标分割方法、装置、设备以及存储介质
Shit et al. An encoder‐decoder based CNN architecture using end to end dehaze and detection network for proper image visualization and detection
CN112686828B (zh) 视频去噪方法、装置、设备及存储介质
WO2024159888A1 (zh) 复原图像的方法、装置、计算机设备、程序产品及存储介质
CN113920023A (zh) 图像处理方法及装置、计算机可读介质和电子设备
CN113688839A (zh) 视频处理方法及装置、电子设备、计算机可读存储介质
CN112734675A (zh) 一种基于金字塔模型和非局部增强密集块的图像去雨方法
CN110310269B (zh) 基于极平面多尺度Gabor特征相似度的光场图像质量评价方法
CN113496138A (zh) 稠密点云数据的生成方法、装置、计算机设备及存储介质
CN112052863B (zh) 一种图像检测方法及装置、计算机存储介质、电子设备
CN114429602A (zh) 语义分割方法、装置、电子设备及存储介质
CN113610856A (zh) 训练图像分割模型和图像分割的方法和装置
CN112215853A (zh) 图像分割方法、装置、电子设备和计算机可读介质
CN116681625B (zh) 基于多尺度对比学习的无监督图像去雨方法、装置及终端
CN118410724B (zh) 一种输电线路异物识别方法、系统、计算机设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21963889

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2021963889

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2021963889

Country of ref document: EP

Effective date: 20240612