CN117495717A - Image defogging method capable of restoring target polarization characteristics - Google Patents

Image defogging method capable of restoring target polarization characteristics Download PDF

Info

Publication number
CN117495717A
CN117495717A CN202311457664.6A CN202311457664A CN117495717A CN 117495717 A CN117495717 A CN 117495717A CN 202311457664 A CN202311457664 A CN 202311457664A CN 117495717 A CN117495717 A CN 117495717A
Authority
CN
China
Prior art keywords
image
defogging
polarization
polarized
haze
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311457664.6A
Other languages
Chinese (zh)
Inventor
贾启龙
侯清铠
刘志晨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dalian Maritime University
Original Assignee
Dalian Maritime University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dalian Maritime University filed Critical Dalian Maritime University
Priority to CN202311457664.6A priority Critical patent/CN117495717A/en
Publication of CN117495717A publication Critical patent/CN117495717A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/21—Polarisation-affecting properties
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Biochemistry (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides an image defogging method capable of restoring target polarization characteristics, which comprises the following steps: constructing a haze medium polarization transmission model based on active light source polarization imaging; constructing a haze imaging model based on the constructed haze medium polarization transmission model; based on the constructed haze imaging model and an unsupervised multitask learning theory, an unsupervised polarized image defogging neural network model is constructed, and defogging of a single polarized image is realized in an end-to-end mode; and designing a loss function of the guided image defogging model based on the constructed unsupervised polarized image defogging neural network model, and guiding the unsupervised polarized image defogging neural network model to defog. In order to solve the problems of transmissivity estimation, atmospheric light estimation and image defogging, the invention designs a neural network based on unsupervised multi-task learning to realize defogging of a single image, designs a loss function based on multi-task learning, guides the learning process of the neural network, realizes the enhancement effect of a target, and improves the performance of target detection and identification.

Description

Image defogging method capable of restoring target polarization characteristics
Technical Field
The invention relates to the technical fields of maritime search and rescue, intelligent navigation of ships, military false target identification, automatic driving, target detection and target identification, in particular to an image defogging method capable of restoring target polarization characteristics.
Background
Haze is a typical atmospheric phenomenon that occurs when dust, smoke and other particulate matter accumulate in the air. The transmission of light in a haze medium can cause phenomena such as optical signal attenuation, scattering, absorption and the like, so that the detection, transmission and imaging performances of the photoelectric imaging equipment are greatly reduced. The light is scattered by the medium, so that the propagation direction of the light is changed, and the light deviates from the field of view. At the same time, ambient light outside the field of view is scattered by the particles into the field of view, obscuring the detector imaging. The low visibility of the hazy image results in reduced performance of many visual tasks, such as object detection. Therefore, image defogging has been widely studied as a pretreatment step and a visual enhancement technique, and a remarkable effect has been achieved.
The polarization characteristics of light are easily perceived by animals such as certain insects (e.g., ants), but are not perceived by humans. Compared with the traditional photoelectric detection technology, the polarization detection has the following advantages: 1) The advantage of "highlighting the target"; 2) The advantages of 'cloud penetration and fog penetration'; 3) The advantage of "discriminating true from false". It has been shown that the rate of polarization decay in polarization imaging is slower than the rate of light intensity decay in intensity imaging, i.e. the light travels the same distance along its path, the magnitude of light intensity decay being greater than that of polarization. Theoretical and experimental researches show that the polarization characteristics of the artificial target and the natural environment are obviously different, and the artificial target and the natural environment particularly react on the dielectric material. For example, plastics, glass and metals have higher polarization than atmospheric and seawater, which have higher polarization than soil, rock and vegetation. The polarization characteristics of the substances enable polarization imaging detection to distinguish targets from backgrounds, improve image contrast, identify stealth and camouflage targets, and improve detection and identification performances of the targets. The polarized image defogging and target enhancement are core problems of vision-based automatic driving, target detection and target identification systems, and an intelligent vehicle or ship needs to identify and detect targets in a field of view by means of a vision system in the driving process, so that the automatic driving system is guided based on detection results, and the accuracy of target detection and identification is guaranteed under the condition of severe weather by means of a reliable image defogging method.
The existing image defogging and target enhancement methods mainly comprise three methods, namely image defogging based on prior information, image restoration based on supervised learning and image restoration based on unsupervised learning. The image defogging method based on the unsupervised learning is one of the common methods in the image defogging field at present because of the end-to-end defogging process and no prior knowledge of images. Meanwhile, due to the complexity of a real haze environment, general image priori knowledge is difficult to find, and a haze-free image corresponding to a haze image cannot be obtained, so that an image defogging method based on the priori knowledge and supervised learning is not feasible.
Compared with the image defogging method based on priori knowledge and supervised learning, the image defogging method based on unsupervised learning shows remarkable universality. The defogging method does not need to resort to priori knowledge of the image and does not need to train defogging models on the defogging image and the foggy image set in advance, so that the defogging method is widely applied to solving the problem of defogging the image. The deep neural network is widely applied in the field of computer vision due to the strong feature extraction and the end-to-end learning mechanism. Image defogging based on supervised deep learning has achieved a number of representative achievements, the main idea of which is to decompose the image defogging problem into several sub-problems to be solved separately. Furthermore, aiming at the problem that the defogging image corresponding to the foggy image can not be obtained, an unsupervised depth image defogging method is further provided, and defogging effects equivalent to other methods are obtained. The method is independent of priori knowledge of the image and the image training set, so that stability and universality of the image under a complex imaging environment can be ensured.
Disclosure of Invention
According to the technical problem that the image priori knowledge is unknown and the fog-free and foggy image pair cannot be obtained, the image defogging method capable of restoring the target polarization characteristic is provided. The invention designs an effective neural network based on unsupervised multitask learning to realize defogging of a single image in order to solve the problems of transmissivity estimation, atmospheric light estimation and image defogging. On the basis, a loss function based on multi-task learning is further designed to guide the learning process of the neural network, so that a single image defogging method with universality is realized, the enhancement effect of a target is realized, and the performance of target detection and recognition is further improved.
The invention adopts the following technical means:
an image defogging method capable of restoring a target polarization characteristic, comprising:
s1, constructing a haze medium polarization transmission model based on active light source polarization imaging;
s2, constructing a haze imaging model based on the constructed haze medium polarization transmission model;
s3, based on the constructed haze imaging model and an unsupervised multitask learning theory, an unsupervised polarized image defogging neural network model is constructed, and defogging of a single polarized image is realized in an end-to-end mode;
s4, designing a loss function of the guide image defogging model based on the constructed unsupervised polarized image defogging neural network model, and guiding the unsupervised polarized image defogging neural network model to defog.
Further, the step S1 specifically includes:
s11, supposing that the target reflects lightThe Stokes vectors of the emergent light transmitted by the haze medium are S= [ S ] 0 ，S 1 ，S 2 ，S 3 ] T And S '= [ S ]' 0 ，S′ 1 ，S′ 2 ，S′ 3 ] T The polarization transmission model of the haze medium is expressed as:
S′＝MS
wherein M represents a Mueller matrix;
s12, according to a polarization transmission model of the haze medium, solving a Mueller matrix by changing Stokes vectors of incident light and measuring Stokes vectors of emergent light at the same time, wherein when the incident light is natural light, horizontal linear polarized light, +45 DEG linear polarized light and right-handed circular polarized light respectively, the corresponding Stokes vectors are [1,0] T ，[1，1，0，0] T ，[1，0，1，0] T ，[1，0，0，1] T At this time, stokes vectors S' are calculated as follows:
calculating elements in the Mueller matrix according to the formula in step S12;
s13, measuring corresponding polarized images when the laser transmitting end polarizing plate and the receiving end polarization detecting plate are placed at different polarization angles;
s14, inverting the Stokes vector of the incident light based on the polarization transmission model of the haze medium in the step S11, the measured polarized image of the emergent light and the Stokes vector thereof, wherein the Stokes vector of the incident light is as follows:
wherein,represents the Moore-Penrose inverse matrix of M.
Further, the step S2 specifically includes:
s21, splicing a plurality of single-channel polarized images into a multi-channel image in order to fully utilize the polarization information of the target;
s22, set I 0 ，I 45 And I 90 Three polarized images taken when the polarizer was 0 °,45 ° and 90 °, respectively, were selected I 0 、I 45 And I 90 Splicing into a three-channel image, wherein I 0 、I 45 And I 90 R, G and B channels as stitched images; i= [ I ] for spliced image 0 ,I 45 ,I 90 ]A representation; due to I 135 The polarization degree and the polarization angle are respectively defined by I 0 、I 45 And I 90 Deriving, so that all polarization information of the scene is contained in the stitched three-channel polarized image;
s23, supposing that each spliced image I follows the following haze imaging model:
I＝Jt+I[argmin(t)](1-t)+n e
wherein n is e Representing random noise, I [ argmin (t)]The pixel having the lowest transmittance in I is represented by J, the defogging image is represented by J, and the transmittance of the haze medium is represented by t.
Further, in the step S3, the constructed unsupervised polarized image defogging neural network model includes: the image defogging device comprises a common feature extraction module, an image defogging module and a transmissivity estimation module, wherein:
the common feature extraction module consists of a convolution layer and a group of residual blocks, wherein each residual block consists of the convolution layer, a batch normalization layer and a ReLU activation layer;
the image defogging module is characterized in that an input layer and an output layer are convolution layers respectively, an output layer channel is 3, and the rest parts are residual blocks;
the transmissivity estimation module is characterized in that the input layer and the output layer are convolution layers respectively, the output layer channel is 1, and the rest parts are residual blocks.
Further, the common feature extraction module is used for extracting common features from the input foggy polarized image, inputting the extracted common features into the image defogging module and the transmissivity estimation module, and respectively estimating defogging images and the transmissivity; the input of the image defogging module is the output of the common characteristic extraction module and is used for learning defogging images according to the input foggy polarized images; the input of the transmissivity estimation module is the output of the common characteristic extraction module and is responsible for estimating the transmissivity, so that the atmospheric light is estimated according to the transmissivity.
Further, in the step S4, the designed loss function of the guided image defogging model includes three parts, the first part is used for limiting the output of the neural network to approach to the foggy polarized image, the second part is used for limiting the output of the defogging module to be a foggy image, and the third part is used for limiting the output of the transmittance estimation module to be accurate transmittance.
Further, the step S4 specifically includes:
s41, based on the Bayes theorem, the defogging of the image is expressed as the following formula:
P(J，t|I)∝P(I|J，t)P(J)P(t)
wherein P (J, t|I) represents posterior distribution, P (I|J, t) represents likelihood, and P (J) and P (t) represent a haze-free image and a transmittance prior, respectively;
s42, according to the haze imaging model in the step S2, obtaining:
n e ＝I-Jt-I[argmin(t)](1-t)
s43, assume n e Obeying gaussian distributionThen there are:
where N is the number of pixels, trace (&) is the trace of the square matrix, and Σ is the covariance matrix;
s44, calculating maximum likelihood estimation values of J and t, wherein the calculation formula is as follows:
in the formula, I F The frobenio us norm of the matrix;
applying the formula in step S44 to limit the output of the neural network to approach the foggy polarized image;
s45, assuming that one-step distribution of J follows a position of zero and a scale of r 1 Laplace distribution of (a):
wherein,gradient operators in the horizontal direction and the vertical direction of the image are represented;
s46, applying a second-order gradient prior to the defogging image J besides the first-order prior to capture smaller-scale edges in the gradient domain and inhibit step artifacts, wherein the second-order prior adopts a position of 0 and a scale of r 2 The Laplace distribution of (2) is described as follows:
P 2 (J)＝L(ΔJ|0，r 2 1)
where Δ represents the second-order Laplace operator, and the a priori P (J) is expressed as:
P(J)＝P 1 (J)P 2 (J)
s47, calculating a defogging image J, wherein the calculation formula is as follows:
s48, introducing a loss function for learning the transmittance t, and adopting a transmittance estimation method in defogging a dark channel to improve the estimation accuracy of the transmittance t, wherein the transmittance t is required to have the following form:
wherein Ω (x) represents a local region of the image centered on pixel x, I (y) represents pixel y of image I, a represents atmospheric light, and superscript c represents one color channel of { R, G, B }, n t Representing the residual error; n is n t The use of (c) makes t robust to noise, assuming n t Obeying gaussian distributionThen there are:
in Sigma -1 The superscript c, which is the inverse of the covariance matrix Σ, represents one color channel in { R, G, B };
s49, calculating a maximum likelihood estimation value of t, wherein the calculation formula is as follows:
s410, summarizing the above, defining a loss function for image defogging and transmittance estimation is:
wherein I F Frobenious norm, f representing matrix t (I) And f t (I) Respectively representing the output of the defogging module and the transmissivity estimation module, wherein S represents the Stokes vector corresponding to the defogging image J, and S' represents the Stokes vector corresponding to the foggy image; by introducing into the loss functionThe Stokes vector of the defogging image can be limited to approach the Stokes vector of the incident light, and the polarization characteristic of the incident light can be restored.
Compared with the prior art, the invention has the following advantages:
1. aiming at the problems of low visibility of a target image, fuzzy target image and the like in a haze environment, the invention provides a polarized image defogging method capable of restoring the polarization characteristic of the target based on an unsupervised neural network model, a haze imaging model and a haze polarization transmission model.
2. The image defogging method capable of restoring the target polarization characteristics can improve the visibility of the target image in the haze environment, and is beneficial to intelligent navigation, maritime search and rescue, automatic driving, target detection and target identification of ships.
3. According to the image defogging method capable of restoring the target polarization characteristic, which is provided by the invention, the polarization image defogging adopts an end-to-end unsupervised learning theory, model training is not required on a large-scale image set, and defogging of a single image can be realized.
4. The image defogging method capable of restoring the target polarization characteristic provided by the invention can realize false target identification based on the restored target polarization characteristic and the polarization characteristic difference between the artificial target and the natural environment.
Based on the reasons, the method can be widely popularized in the fields of maritime search and rescue, intelligent navigation of ships, military false target identification, automatic driving, target detection, target identification and the like.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to the drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of the method of the present invention.
Fig. 2 is a schematic diagram of an active light source polarization imaging experimental apparatus according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of stitching polarized images according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of an unsupervised polarized image defogging neural network model according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
As shown in fig. 1, the present invention provides an image defogging method capable of restoring a target polarization characteristic, comprising:
s1, constructing a haze medium polarization transmission model based on active light source polarization imaging;
s2, constructing a haze imaging model based on the constructed haze medium polarization transmission model;
s3, based on the constructed haze imaging model and an unsupervised multitask learning theory, an unsupervised polarized image defogging neural network model is constructed, and defogging of a single polarized image is realized in an end-to-end mode;
s4, designing a loss function of the guide image defogging model based on the constructed unsupervised polarized image defogging neural network model, and guiding the unsupervised polarized image defogging neural network model to defog.
In specific implementation, as a preferred embodiment of the present invention, the step S1 specifically includes:
s11, supposing that Stokes vectors of target reflected light and emergent light transmitted through haze medium are S= [ S ] 0 ，S 1 ，S 2 ，S 3 ] T And S '= [ S ]' 0 ，S′ 1 ，S′ 2 ，S′ 3 ] T The polarization transmission model of the haze medium is expressed as:
S′＝MS
wherein M represents a Mueller matrix;
s12, according to a polarization transmission model of the haze medium, solving a Mueller matrix by changing Stokes vectors of incident light and measuring Stokes vectors of emergent light at the same time, wherein when the incident light is natural light, horizontal linear polarized light, +45 DEG linear polarized light and right-handed circular polarized light respectively, the corresponding Stokes vectors are [1,0] T ，[1，1，0，0] T ，[1，0，1，0] T ，[1，0，1，0] T At this timeThe Stokes vectors s' are calculated as:
according to the formula in step S12, the elements in the Mueller matrix are calculated, as shown in table 1, the measurement method corresponding to the polarized image in table 1 is shown in table 2, and the polarized image measurement experiment is shown in fig. 2.
TABLE 1 calculation results for each element of Mueller matrix
Table 2 polarization image measurement scheme
S13, for measuring the polarized images shown in the table 2, designing an experimental system shown in the table 2, and measuring the polarized images corresponding to the polarized sheets at the laser transmitting end and the polarized sheets at the receiving end when the polarized sheets are placed at different polarized angles; the experiment uses laser light with active 450nm,532nm and 670nm as light sources.
S14, inverting the Stokes vector of the incident light based on the polarization transmission model of the haze medium in the step S11, the measured polarized image of the emergent light and the Stokes vector thereof, wherein the Stokes vector of the incident light is as follows:
wherein,represents the Moore-Penrose inverse matrix of M.
In specific implementation, as a preferred embodiment of the present invention, the step S2 specifically includes:
s21, the single-channel polarized image contains limited information, and any two different polarized images contain complementary information about the same scene. In order to fully utilize the polarization information of the target, a plurality of single-channel polarization images are spliced into a multi-channel image; fig. 3 shows a schematic diagram of stitching single-channel polarized images into multi-channel polarized images.
S22, set I 0 ，I 45 And I 90 Three polarized images taken when the polarizer was 0 °,45 ° and 90 °, respectively, were selected I 0 、I 45 And I 90 Splicing into a three-channel image, wherein I 0 、I 45 And I 90 R, G and B channels as stitched images; i= [ I ] for spliced image 0 ,I 45 ,I 90 ]A representation; due to I 135 The polarization degree and the polarization angle are respectively defined by I 0 、I 45 And I 90 Deriving, so that all polarization information of the scene is contained in the stitched three-channel polarized image;
s23, supposing that each spliced image I follows the following haze imaging model:
I＝Jt+I[argmin(t)](1-t)+n e
wherein n is e Representing random noise, I [ arg min (t)]The pixel having the lowest transmittance in I is represented by J, the defogging image is represented by J, and the transmittance of the haze medium is represented by t. In the formula in step S23, the original haze is calculatedAtmospheric light A in the imaging model is replaced by I [ arg min (t)]. The first reason is that such a replacement can reduce an unknown parameter a, and the second reason is that a polarized image defogging neural network of simple structure can be constructed because it only needs to estimate J and t.
It must be emphasized that in the haze imaging model proposed in this embodiment, the atmospheric light is not visible because it is estimated by I [ arg min (t) ]. This is possible because atmospheric light is always at an infinite distance, i.e., given a pixel x, then it means t (x) to 0 when the depth of field of that pixel is distance d (x) to → infinity. On the other hand, as the distance increases, haze becomes denser, and the transmittance of light becomes lower. The above analysis shows that atmospheric light can be estimated from I [ arg min (t) ].
In specific implementation, as a preferred embodiment of the present invention, in step S3, as shown in fig. 4, the constructed unsupervised polarized image defogging neural network model includes: the image defogging device comprises a common feature extraction module, an image defogging module and a transmissivity estimation module, wherein:
the common feature extraction module consists of a convolution layer and a group of residual blocks, wherein each residual block consists of the convolution layer, a batch normalization layer and a ReLU activation layer;
the image defogging module is characterized in that an input layer and an output layer are convolution layers respectively, an output layer channel is 3, and the rest parts are residual blocks;
the transmissivity estimation module is characterized in that the input layer and the output layer are convolution layers respectively, the output layer channel is 1, and the rest parts are residual blocks.
Given a foggy polarized image I, a neural network for learning the defocused image J from I is introduced. The neural network provided by the embodiment of the invention is shown in fig. 4, and is constructed based on a haze imaging model and an unsupervised multitask learning theory. The neural network learns defogging images from the foggy images in an unsupervised manner. In other words, the neural network can defog a single fogged polarized image without prior training on large-scale foggy and foggy image pairs. Specifically, the goal of neural networks is to learn defogging images and transmittance so that they can be merged together according to a haze imaging model and approach the input hazy image.
In a specific implementation, as a preferred embodiment of the present invention, the common feature extraction module is configured to extract a common feature from an input foggy polarized image, and input the extracted common feature into the image defogging module and the transmittance estimation module, where the extracted common feature is used to estimate a defogging image and a transmittance respectively; the sense of developing neural networks comes from the multitasking learning based on the parameter sharing mechanism. The benefit of the common feature extraction module is that the complexity of the model can be simplified. Furthermore, it can reduce the risk of overfitting of the neural network by learning common representative features of the images. Since the common feature extraction module consists of residual blocks, it can also avoid the gradient vanishing problem. The input of the image defogging module is the output of the common characteristic extraction module and is used for learning defogging images according to the input foggy polarized images; the input of the transmissivity estimation module is the output of the common characteristic extraction module and is responsible for estimating the transmissivity and estimating the atmospheric light according to the transmissivity. With the defogging image, transmittance, and atmospheric light estimated, they can be combined together according to the haze imaging model. Moreover, the combined image needs to be as close as possible to the foggy polarized image.
In particular, as a preferred embodiment of the present invention, in order to ensure that the output of the image defogging module and the transmittance estimation module is an accurate defogging image and transmittance, a loss function is designed to guide defogging of the neural network. In the step S4, the designed loss function of the guided image defogging model includes three parts, the first part is used for limiting the output of the neural network to approach to the foggy polarized image, the second part is used for limiting the output of the defogging module to be a foggy image, and the third part is used for limiting the output of the transmissivity estimation module to be accurate transmissivity.
In specific implementation, as a preferred embodiment of the present invention, the step S4 specifically includes:
s41, based on the Bayes theorem, the defogging of the image is expressed as the following formula:
P(J，t|I)∝P(I|J，t)P(J)P(t)
wherein P (J, t|I) represents posterior distribution, P (I|J, t) represents likelihood, and P (J) and P (t) represent a haze-free image and a transmittance prior, respectively;
s42, according to the haze imaging model in the step S2, obtaining:
n e ＝I-Jt-I[argmin(t)](1-t)
s43, assume n e Obeying gaussian distributionThen there are:
where N is the number of pixels, trace (&) is the trace of the square matrix, and Σ is the covariance matrix;
s44, calculating maximum likelihood estimation values of J and t, wherein the calculation formula is as follows:
in the formula, I F The frobenio us norm of the matrix;
applying the formula in step S44 to limit the output of the neural network to approach the foggy polarized image;
s45, in order to capture details from the foggy image I, a multi-order gradient prior is applied to the defogging image J, because the defogging image is full of edge and detail information, which can be described by the first and second order gradients of J. Based on the above observations of defogging images, it is assumed that defogging images are piecewise continuous, which can be measured by total variation priors. Thus, assume that a one-step distribution of J follows a position of zero and a scale of r 1 Laplace distribution of (a):
wherein the method comprises the steps of，Gradient operators in the horizontal direction and the vertical direction of the image are represented;
s46, applying a second-order gradient prior to the defogging image J besides the first-order prior to capture smaller-scale edges in the gradient domain and inhibit step artifacts, wherein the second-order prior adopts a position of 0 and a scale of r 2 The Laplace distribution of (2) is described as follows:
P 2 (J)＝L(ΔJ|0，r 2 1)
where Δ represents the second-order Laplace operator, and the a priori P (J) is expressed as:
P(J)＝P 1 (J)P 2 (J)
s47, calculating a defogging image J, wherein the calculation formula is as follows:
s48, a loss function for learning the transmittance t is introduced, and as described above, the transmittance t can be estimated by the formula in step S44. However, the formula in step S44 does not guarantee a high quality transmittance estimation result. In order to improve the accuracy of the estimation of the transmittance t, the transmittance estimation method in defogging of the dark channel is adopted, and the transmittance t is required to have the following form:
wherein Ω (x) represents a local region of the image centered on pixel x, I (y) represents pixel y of image I, a represents atmospheric light, and superscript c represents one color channel of { R, G, B }, n t Representing the residual error; n is n t The use of (c) makes t robust to noise, assuming n t Obeying gaussian distributionThen there are:
in Sigma -1 The superscript c, which is the inverse of the covariance matrix Σ, represents one color channel in { R, G, B };
s49, calculating a maximum likelihood estimation value of t, wherein the calculation formula is as follows:
s410, in summary, defining a loss function for image defogging and transmittance estimation as follows:
wherein I F Frobenious norm, f representing matrix t (I) And f t (I) Respectively representing the output of the defogging module and the transmissivity estimation module, wherein S represents the Stokes vector corresponding to the defogging image J, and S' represents the Stokes vector corresponding to the foggy image; by introducing into the loss functionThe Stokes vector of the defogging image can be limited to approach the Stokes vector of the incident light, and the polarization characteristic of the incident light can be restored.
The loss function defined in step S411 measures the foggy polarized images I and f J (I)f t (I)+I[arg min f t (I)](1-f t (I) Error between the two). Given a foggy polarized image I, the image defogging problem is reduced to solve for f J (I) Approximately J and f t (I) And t, so that the loss function becomes minimal. The defogging image J is learned and obtained in an unsupervised mode, namely J is estimated only according to the defogging image, and the neural network does not need to be trained in advance. Furthermore, this is also a multi-task learning problem, since the target of image defogging includes learning of J and t. It must be emphasized thatIt is true that J and t are estimated simultaneously, not sequentially. Unlike prior-based image defogging methods (which learn t, a, and J sequentially, rather than simultaneously), image defogging is an end-to-end learning process.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.

Claims (7)

1. An image defogging method capable of restoring a target polarization characteristic, comprising:
s1, constructing a haze medium polarization transmission model based on active light source polarization imaging;
s2, constructing a haze imaging model based on the constructed haze medium polarization transmission model;
s3, based on the constructed haze imaging model and an unsupervised multitask learning theory, an unsupervised polarized image defogging neural network model is constructed, and defogging of a single polarized image is realized in an end-to-end mode;
s4, designing a loss function of the guide image defogging model based on the constructed unsupervised polarized image defogging neural network model, and guiding the unsupervised polarized image defogging neural network model to defog.
2. The image defogging method capable of restoring a target polarization characteristic according to claim 1, wherein the step S1 specifically comprises:
s11, supposing that Stokes vectors of target reflected light and emergent light transmitted through haze medium are S= [ S ] 0 ，S 1 ，S 1 ，S 3 ] T And S '= [ S ]' 0 ，S′ 1 ，S′ 2 ，S′ 3 ] T The polarization transmission model of the haze medium is expressed as:
S′＝MS
wherein M represents a Mueller matrix;
s12, according to a polarization transmission model of the haze medium, solving a Mueller matrix by changing Stokes vectors of incident light and measuring Stokes vectors of emergent light at the same time, wherein when the incident light is natural light, horizontal linear polarized light, +45 DEG linear polarized light and right-handed circular polarized light respectively, the corresponding Stokes vectors are [1,0] T ，[1，1，0，0] T ，[1，0，1，0] T ，[1，0，0，1] T At this time, stokes vectors S' are calculated as follows:
calculating elements in the Mueller matrix according to the formula in step S12;
s13, measuring corresponding polarized images when the laser transmitting end polarizing plate and the receiving end polarization detecting plate are placed at different polarization angles;
s14, inverting the Stokes vector of the incident light based on the polarization transmission model of the haze medium in the step S11, the measured polarized image of the emergent light and the Stokes vector thereof, wherein the Stokes vector of the incident light is as follows:
wherein,represents the Moore-Penrose inverse matrix of M.
3. The image defogging method capable of restoring a target polarization characteristic according to claim 1, wherein the step S2 specifically comprises:
s21, splicing a plurality of single-channel polarized images into a multi-channel image in order to fully utilize the polarization information of the target;
s22, set I 0 ，I 45 And I 90 Three polarized images taken when the polarizer was 0 °,45 ° and 90 °, respectively, were selected I 0 、I 45 And I 90 Splicing into a three-channel image, wherein I 0 、I 45 And I 90 R, G and B channels as stitched images; i= [ I ] for spliced image 0 ,I 45 ,I 90 ]A representation; due to I 135 The polarization degree and the polarization angle are respectively defined by I 0 、I 45 And I 90 Deriving, so that all polarization information of the scene is contained in the stitched three-channel polarized image;
s23, supposing that each spliced image I follows the following haze imaging model:
I＝Jt+[argmin(t)](1-t)+n e
wherein n is e Representing random noise, I [ arg min (t)]The pixel having the lowest transmittance in I is represented by J, the defogging image is represented by J, and the transmittance of the haze medium is represented by t.
4. The image defogging method capable of restoring target polarization characteristics according to claim 1, wherein in the step S3, the constructed unsupervised polarization image defogging neural network model comprises: the image defogging device comprises a common feature extraction module, an image defogging module and a transmissivity estimation module, wherein:
the common feature extraction module consists of a convolution layer and a group of residual blocks, wherein each residual block consists of the convolution layer, a batch normalization layer and a ReLU activation layer;
the image defogging module is characterized in that an input layer and an output layer are convolution layers respectively, an output layer channel is 3, and the rest parts are residual blocks;
the transmissivity estimation module is characterized in that the input layer and the output layer are convolution layers respectively, the output layer channel is 1, and the rest parts are residual blocks.
5. The image defogging method capable of restoring a target polarization characteristic according to claim 4, wherein the common feature extraction module is used for extracting a common feature from an inputted foggy polarization image, and inputting the extracted common feature into the image defogging module and the transmittance estimation module for estimating a defogging image and a transmittance, respectively; the input of the image defogging module is the output of the common characteristic extraction module and is used for learning defogging images according to the input foggy polarized images; the input of the transmissivity estimation module is the output of the common characteristic extraction module and is responsible for estimating the transmissivity, so that the atmospheric light is estimated according to the transmissivity.
6. The image defogging method capable of restoring a target polarization characteristic according to claim 1, wherein in the step S4, the designed loss function of the guiding image defogging model comprises three parts, the first part is used for limiting the output of the neural network to be close to the fogged polarization image, the second part is used for limiting the output of the defogging module to be a fogless image, and the third part is used for limiting the output of the transmittance estimation module to be an accurate transmittance.
7. The image defogging method capable of restoring a target polarization characteristic according to claim 1, wherein the step S4 specifically comprises:
s41, based on the Bayes theorem, the defogging of the image is expressed as the following formula:
P(J,t|I)∝P(I|J,t)P(J)P(t)
wherein P (J, t|I) represents posterior distribution, P (I|J, t) represents likelihood, and P (J) and P (t) represent a haze-free image and a transmittance prior, respectively;
s42, according to the haze imaging model in the step S2, obtaining:
n e ＝I-Jt-I[argmin(t)](1-t)
s43, assume n e Obeying gaussian distributionThen there are:
where N is the number of pixels, trace (&) is the trace of the square matrix, and Σ is the covariance matrix;
s44, calculating maximum likelihood estimation values of J and t, wherein the calculation formula is as follows:
in the formula, I F The frobenio us norm of the matrix;
applying the formula in step S44 to limit the output of the neural network to approach the foggy polarized image;
s45, assuming that one-step distribution of J follows a position of zero and a scale of r 1 Laplace distribution of (a):
wherein,gradient operators in the horizontal direction and the vertical direction of the image are represented;
s46, in addition to the first order prior, applying a second order gradient prior to the defogging image J,capturing smaller-scale edges in gradient domain and inhibiting ladder artifacts, and adopting a position of 0 and a scale of r for second order prior 2 The Laplace distribution of (2) is described as follows:
P 2 (J)＝L(ΔJ|0,r 2 1)
where Δ represents the second-order Laplace operator, and the a priori P (J) is expressed as:
P(J)＝P 1 (J)P 2 (J)
s47, calculating a defogging image J, wherein the calculation formula is as follows:
s48, introducing a loss function for learning the transmittance t, and adopting a transmittance estimation method in defogging a dark channel to improve the estimation accuracy of the transmittance t, wherein the transmittance t is required to have the following form:
wherein Ω (x) represents a local region of the image centered on pixel x, I (y) represents pixel y of image I, a represents atmospheric light, and superscript c represents one color channel of { R, G, B }, n t Representing the residual error; n is n t The use of (c) makes t robust to noise, assuming n t Obeying gaussian distributionThen there are:
in Sigma -1 An inverse matrix of the covariance matrix Σ;
s49, calculating a maximum likelihood estimation value of t, wherein the calculation formula is as follows:
s410, in summary, defining a loss function for image defogging and transmittance estimation as follows:
wherein I F Frobenious norm, f representing matrix t (I) And f t (I) Respectively representing the output of the defogging module and the transmissivity estimation module, wherein S represents the Stokes vector corresponding to the defogging image J, and S' represents the Stokes vector corresponding to the foggy image; by introducing into the loss functionThe Stokes vector of the defogging image can be limited to approach the Stokes vector of the incident light, and the polarization characteristic of the incident light can be restored.
CN202311457664.6A 2023-11-03 2023-11-03 Image defogging method capable of restoring target polarization characteristics Pending CN117495717A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311457664.6A CN117495717A (en) 2023-11-03 2023-11-03 Image defogging method capable of restoring target polarization characteristics

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311457664.6A CN117495717A (en) 2023-11-03 2023-11-03 Image defogging method capable of restoring target polarization characteristics

Publications (1)

Publication Number Publication Date
CN117495717A true CN117495717A (en) 2024-02-02

Family

ID=89682249

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311457664.6A Pending CN117495717A (en) 2023-11-03 2023-11-03 Image defogging method capable of restoring target polarization characteristics

Country Status (1)

Country Link
CN (1) CN117495717A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117911282A (en) * 2024-03-19 2024-04-19 华中科技大学 Construction method and application of image defogging model

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117911282A (en) * 2024-03-19 2024-04-19 华中科技大学 Construction method and application of image defogging model
CN117911282B (en) * 2024-03-19 2024-05-28 华中科技大学 Construction method and application of image defogging model

Similar Documents

Publication Publication Date Title
Yu et al. Underwater-GAN: Underwater image restoration via conditional generative adversarial network
Santra et al. Learning a patch quality comparator for single image dehazing
Namer et al. Skyless polarimetric calibration and visibility enhancement
CN117495717A (en) Image defogging method capable of restoring target polarization characteristics
Fayaz et al. Underwater image restoration: A state‐of‐the‐art review
US20240161479A1 (en) Polarized Image Enhancement using Deep Neural Networks
Alterman et al. Detecting motion through dynamic refraction
Lu et al. Computer vision for ocean observing
WO2019089049A1 (en) Systems and methods for improved feature extraction using polarization information
Li et al. Real‐Time Visualization System for Deep‐Sea Surveying
Moghimi et al. Real-time underwater image resolution enhancement using super-resolution with deep convolutional neural networks
Cai et al. Semi-supervised visual tracking of marine animals using autonomous underwater vehicles
Hardy et al. Improving space domain awareness through unequal-cost multiple hypothesis testing in the space surveillance telescope
Edge et al. A generative approach for detection-driven underwater image enhancement
Song et al. Optical imaging and image restoration techniques for deep ocean mapping: a comprehensive survey
Zhou et al. Underwater occlusion object recognition with fusion of significant environmental features
Spandana et al. Underwater Image Enhancement and Restoration Using Cycle GAN
Uemura et al. Marine organisms tracking and recognizing using yolo
CN112507965A (en) Target identification method and system of electronic lookout system
Kumar et al. Saliency based shape extraction of objects in unconstrained underwater environment
Verma et al. FCNN: fusion-based underwater image enhancement using multilayer convolution neural network
Kumar et al. Saliency subtraction inspired automated event detection in underwater environments
Li et al. UStark: underwater image domain-adaptive tracker based on Stark
CN111881982A (en) Unmanned aerial vehicle target identification method
Quintana et al. Towards automatic recognition of mining targets using an autonomous robot

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination