CN113096033B

CN113096033B - Low-light image enhancement method based on Retinex model self-adaptive structure

Info

Publication number: CN113096033B
Application number: CN202110304780.9A
Authority: CN
Inventors: 王润雨; 齐娜; 齐景仲; 朱青
Original assignee: Beijing University of Technology
Current assignee: Beijing University of Technology
Priority date: 2021-03-22
Filing date: 2021-03-22
Publication date: 2024-05-28
Anticipated expiration: 2041-03-22
Also published as: CN113096033A

Abstract

The invention discloses a low-light image enhancement method based on a Retinex model self-adaptive structure, which is improved according to low-light conditions and comprises the following steps of: TSRI algorithms for input image illumination structure and reflective texture extraction are presented. According to the characteristics of the low-illumination image, the sparse reflection image L0 is introduced, and the problem that the reflection image is not smooth enough in the Retinex algorithm is solved; for the Retinex low-light image enhancement algorithm. The TSRI algorithm is used to process the luminance component in HSV space.

Description

Low-light image enhancement method based on Retinex model self-adaptive structure

Technical Field

The invention relates to a low-illumination image enhancement method based on a Retinex model self-adaptive structure, and belongs to the technical field of image processing.

Background

With the vigorous development of social media such as Facebook and Youtube, some people take their daily lives or record them and share them with other people through the internet, which almost forms a family meal. However, many photographs are taken in low light environments due to backlight, exposure, or dark environments. Despite improvements in both technology and equipment, the poor quality of taking photographs is unavoidable, not to mention that a large number of photographs remain a number of problems in the past.

In a low light environment. First, these photographs, i.e., low light images, are subject to low definition, low contrast and significant noise. Post-processing techniques are required to improve the visual quality of these images. Second, the quality of the images and videos captured by the optical imaging devices can be reduced, which can also reduce the performance of certain systems, such as those used for intelligent traffic analysis, visual surveillance, and consumer electronics. Low light conditions in a night-time environment can produce images and videos with low contrast, reducing visibility. Although professional equipment and advanced photography can alleviate these drawbacks to some extent, the inherent factors of noise are unavoidable and cannot be addressed at the hardware level. Without a sufficient amount of light, the output of the camera sensor is often masked by the inherent noise of the system. While longer exposure times can effectively improve signal-to-noise ratio and produce noise-free images, it suffers from new problems such as motion blur. In image processing and analysis, image enhancement plays an important role. The main purpose of image enhancement is to process an image to make it more suitable for certain specific fields than the original image. Today, image enhancement techniques are used in a variety of scientific and engineering fields. Such as atmospheric science, astrophotography, biomedical, computer vision, and the like.

Therefore, software-level low-light image enhancement techniques are very popular in photography. In addition, this technique may benefit from a number of computer vision algorithms (object detection, tracking, etc.). As their performance is highly dependent on the sharpness of the target scene.

The low-illumination image can be used after being enhanced, and the existing enhancement algorithm has limited improvement on the image quality. The enhancement algorithm based on Retinex theory can simultaneously give consideration to brightness enhancement, detail enhancement and color fidelity, and is comprehensive in image quality enhancement and is an important point of low-illumination image enhancement research.

In the Retinex model, an image is decomposed into two parts, namely an incident component and a reflected component, the incident component is a description of information such as a light source and brightness of a shooting environment, and the reflected component is feedback of information on the surface of an object, so that the intrinsic properties of the object are described. Solving the incident and reflected components from an image is a problem and cannot be accurately solved. Therefore, some scholars obtain the approximate solution of the incident component through the Gaussian function and the image convolution, and further calculate the reflection component reflecting the essential attribute of the image, so that the purpose of enhancement is achieved. In order to simplify the calculation amount and the processing convenience, and simultaneously more accord with the human vision, the logarithm processing is generally carried out on the formula (1), and the multiplication operation is converted into the addition and subtraction operation, namely

R (x, y) is the output result of the Retinex algorithm. With the continuous development of the Retinex algorithm, many enhancement algorithms based on the Retinex theory appear, and typical algorithms are an SSR algorithm, an MSR algorithm and MSRCR algorithm.

The low-illuminance image generally refers to an image acquired in a shooting environment where the light brightness is dark. Typical low-light images are of two types, one type is that the overall brightness of the image is relatively low; another type of image contains brighter regions, but the overall brightness of the target region of interest is very low.

Disclosure of Invention

The main research content of the method is to provide a low-illumination image enhancement method based on a Retinex model self-adaptive structure. Several common Retinex algorithms are applied to the field of low-illumination image enhancement to achieve good effects, and the contrast and details are improved. However, the conventional Retinex algorithm also has certain limitations. Early path-based methods, i.e., the reflection component, could be calculated by the product of ratios on some random paths. These methods require careful parameter adjustment and can result in high computational costs. The total variation method introduces smooth and reflective assumptions into the variation model to estimate illumination and reflection. But this approach is overly smooth in reflectivity due to the side effects of logarithmic transformation. The method firstly provides a new weighting matrix Retinex image enhancement algorithm with a self-adaptive structure, namely a TSRI algorithm for extracting an illumination structure and a reflection texture of an input image. According to the characteristics of the low-illumination image, the sparse reflection map L0 is introduced, the problem that the reflection map is not smooth enough in the Retinex algorithm is solved, and meanwhile, the effects of adding different weighting matrixes and controlling the iteration times of the weighting matrixes on an output result map are explored.

A structure and texture weighting matrix is proposed:

The Retinex model decomposes the observed scene into its illumination and reflection components. This problem is highly ill-conditioned and requires regularization of the solution space taking into account the appropriate lighting and a priori of reflectivity. Qualitatively, the illumination should be smooth, capturing the structure of objects in the scene, while the reflectivity should present the physical characteristics of the observed scene, capturing its texture information. Previous structure-texture decomposition methods typically use a TV regularization model to preserve edges. These TV regularization models simply force the gradient similarity of the scene and extract the structure of the object. There are two methods of texture decomposition. One is to derive the structure directly using structure-preserving techniques, another approach is to extract the structure from the estimated texture matrix, however, these techniques are prone to ringing effects. To better understand the role of these techniques in structure/texture extraction, new filters, i.e. mean local variances, are proposed.

Where Ω is a local block around each pixel of O, the size of which is set to 3x3 in all experiments. However, the filter appearing in the equation cannot be directly applied to a specific problem, and therefore, by exponential growth or decay, it is proposed that local derivatives will be able to extract structural and texture information. Adding an index to the filter allows for more flexibility in separating structure and texture extraction. As also described by Retinex theory, the larger derivatives are due to the change in reflectivity, while the smaller derivatives occur in smooth illumination.

As also described by Retinex theory, the larger derivatives are due to the change in reflectivity, while the smaller derivatives occur in smooth illumination. Based on this observation, by setting I ₀＝R₀＝O^0.5, it is proposed that

Where γs >1 and γt <1 are two exponential parameters for adjusting the structure and texture perception of illumination and reflectance decomposition.

Proposed are structure and texture TSRI objective functions:

A structure and texture aware Retinex model is proposed to estimate both the illumination I and the reflectivity R of the observed image O. In order to make the model as simple as possible, the illumination and reflectance components are regularized using the L2 norm.

Where S0 and T0 are defined two matrices, the structure should be small enough to preserve the edges of objects in the scene and large enough to suppress detail, and the texture map should be small enough to display detail. Therefore, L0 sparse representation of the reflectograms is added to get more accurate results.

The solution of the algorithm is optimized by using the alternating direction least square method:

Since the objective function is two separable variables I and R, the problem can be solved by an alternating direction multiplier algorithm, the two separate sub-problems being convex, ill-posed, and solved alternately. Matrix variables I ₀＝R₀＝O^0.5 are initialized. I _k and R _k are denoted as illuminance and reflectance variables at the kth iteration (k=0, 1,2, …), respectively, L being the number of iterations. By optimizing one variable at a time while fixing the other, the two variables can be updated alternately. The method encounters an L0 range in the solving process, and can be solved by using a method of introducing auxiliary variables h _p and v _p in a reflection diagram to enable R to be converted into a least square method with alternating directions. In addition, verification is made as to whether the weighting matrix should be updated simultaneously when the illumination map and the reflection map are iteratively updated.

The TSRI algorithm of the adaptive structure achieves good effect on the low-illumination image, but in experiments, the effect of enhancing the low-illumination image is found to be improved. When the traditional enhancement algorithm estimates the reflection component of the low-illumination image, the effect of human eyes on perceiving the image is easily reduced, and a great deal of loss of image details is caused. In order to solve the above problems in the conventional operations, the model improves the effect of the image by changing parameters in the model when the image is processed, and satisfactory results are obtained.

Drawings

Fig. 1 is a brief description of Retinex model image enhancement.

Fig. 2 is a flow chart of a method.

Fig. 3 is a structure texture extraction diagram.

FIG. 4 is a graph of reflectance versus number of iterations of a weighting matrix.

Fig. 5 is a graph comparing the latest low light enhancement algorithm.

FIG. 6 is the effect of low-intensity image enhancement algorithm brightness and saturation on image visual effect based on RIST model in the present method.

Detailed Description

Firstly, a Retinex enhancement algorithm of a self-adaptive weighting matrix for a low-illumination image is provided, and a new exponential local derivative is introduced into the algorithm so as to better utilize global properties of the derivative, and related derivatives are promoted into a structure and texture map; the algorithm input image extracts illumination and reflection information respectively, and carries out L0 sparse expression on the reflection map; the algorithm solves the problem of inappropriateness in a better way using an alternating direction least squares method. The improvement scheme is as follows:

first, a new objective function will be applied to the solution of L0 smoothing, so the method of L0 solution will be described in detail herein.

On the input map, R is the input image, and W is the output result. Gradient ofThe color difference between adjacent pixels of each pixel p in the x and y directions is calculated. The gradient measure is:

Calculate P as A number other than 0. W can be defined as:

W _p is the sum of the reflectogram gradients. The above formula also ensures the structural similarity of the images. (7) The equation is discrete and difficult to solve, so the alternate direction least squares method is also employed. The auxiliary variables h _p and v _p are introduced and correspond to

The retrieve objective function is:

C (h, v) = #p||h _p|+|v_p |noteq0) and Is a parameter to control the similarity between (h, v) and its similar gradient. When/>When large enough, (8) is similar to (7). The alternating direction least squares method is used to solve for W and (h, v).

A) Solving W: the sub-problem minimization for W (8) is given by:

By omitting the phase not involving W, equation (9) is obtained, and deriving it, the minimum solution can be obtained.

B) The objective function for (h, v) is:

c (h, v) is the number of non-0 elements in (h, v), then (10) can be written as:

Since H (|h _p|+|v_p |) is a binary function, if H (|h _p|+|v_p |) =0, then return to 1, otherwise

Returning to 0. Each pixel p in (11) is:

The minimum E _P reached under this condition is:

By the derivation of the above formula, the minimum value E _P can be obtained, and all pixels are added to obtain the optimal solution. Automatically adjusting in a fixed maximum and minimum value. The algorithm iterates 20 times.

In summary, a solution of L0 of the input map R can be obtained.

The framework of the L0 solving method mainly comprises the steps of structural texture extraction and illumination component iterative optimization synthesis. By applying the Retinex model to the input image, weighting matrices are added, structure and reflection components are extracted, and reflection structure variation constraints are applied to these components. For the solving step, since it is an ill-posed problem, a least square method is employed to solve the reflection component by fixing the illumination component and the illumination component by fixing the reflection component, respectively. And finally obtaining the output low light enhancement image.

According to the proposed weighting matrix, an optimized objective function is proposed again:

o is the observed image, I represents the scene illumination map representing the brightness of the object, and R represents the surface reflection representing the physical characteristics of the scene. S ₀ and T ₀ are weighting matrices that are set forth in terms of requirements that the structure should be small enough to preserve edges of objects in the scene, yet large enough to suppress detail, and that the texture map should be small enough to display detail. L0 sparseness is carried out on the reflection diagram, and smoothness is ensured.

Since the objective function is two variables I and R that are separable, the problem can be solved by an alternate direction multiplier (ADMM) algorithm. Matrix variables I ₀＝R₀＝O^0.5 are initialized. I _k and R _k are denoted as illuminance and reflectance variables at the kth iteration (k=0, 1,2, …), respectively, L being the number of iterations. By optimizing one variable at a time while repairing the other, both variables can be updated alternately. According to illumination and reflectivity of each iteration, S ₀ and T ₀ can be updated in an iterative mode at the same time. And finally, setting the derivative to be zero to obtain the solution of illumination and reflectivity.

Algorithm optimization:

a) Update R at fixed I in the kth iteration: : the optimization problem for R becomes:

To solve this problem, it is converted into a vector. For this purpose, the vectorization operator vec (·) is used, representing the vectors o=vec (O), i=vec (I), r _k＝vec(R_k),s₀＝vec(S₀), which have a length of nm. Representing Toeplitz matrix of discrete gradient operator by G to obtain . Use/> Representing a matrix with r _k,s₀ located on the main diagonal. The problem is then converted into a standard least squares regression problem:

the above formula can be converted into:

Wherein,

Setting the r' _k+1 derivative to 0 yields the solution:

Regarding the r ₀, during the experiment, it is noted that during the process of extracting the illumination structure, the edge of the illumination map is not very significant, and in order to remove small non-zero gradients, smooth unimportant details are smoothed, a smoother reflection map is obtained, and an L0 normal form can be introduced to sparsely express the reflection map. The main edges are effectively sharpened by increasing the sharpness of the transition while eliminating manageable low-amplitude structures.

L0 smoothing on the reflection map R can be obtained:

regarding the smooth solving of the reflection map L0, the method proposed by the foregoing may be employed.

B) Similarly, update I when R is fixed in the kth iteration: : the method comprises the following steps:

The algorithm is as follows:

For the Retinex-based TSRI algorithm, a typical low-light image was selected for testing. The main steps for enhancing the image are as follows:

Step 1: inputting a low-illumination image, extracting R, G, B color channels, and converting the color channels into a v channel of an hsv space for experiments;

Step 2: adding a weight matrix S0 to the initial illumination map I0, and adding a weight matrix R0 to the initial reflection map R0;

Step 3: l0 smoothing is carried out on the reflection diagram R0;

Step 4: solving Ik by a conjugate gradient method, solving Rk by the conjugate gradient method, and carrying out iterative updating;

step 5: iterative K times to update Sk and Tk on the basis of Ik and Rk, wherein K is more than or equal to 20;

Step 6: gamma correction is carried out on the illumination map;

step 7: obtaining a result illumination map and a result reflection map;

Step 8: and outputting a final enhancement result.

Through verification, the enhancement algorithm is found to improve the image quality and enrich the detail information of the image, and the method is proved to have stronger capability of enhancing the detail information of the low-illumination image.

In addition, in order to make the enhancement effect more natural, brightness correction and saturation correction are introduced.

Brightness correction: the brightness of an image and the brightness perceived by the human eye are not linear and are affected by a number of factors, including physical and psychological factors. The human visual system is limited in the perception of brightness and the stimulus to vision must reach a certain proportion to be perceived by the human eye, this minimum proportion value being called the human eye recognition threshold. According to weber-fishena law, there is the following relationship:

Where Δi is the change in luminance, I is the actual physical luminance, and the relationship between the luminance perceived by the human eye and the physical luminance of the image can be obtained by integrating both sides of equation (22) simultaneously:

K＝c₁·log I+c₂ (23)

From the expression of formula (23), it can be seen that the process of perceiving brightness by human eyes is nonlinear, satisfying the logarithmic characteristic. Therefore, when adjusting the brightness of an image, nonlinear adjustment is required. When the image is actually observed, the perception of brightness by the human eye is influenced by the illumination intensity of the object on the one hand and also depends on the structure of the image on the other hand. The brightness of the image background also affects the subjective perception of the observer. Since the visual characteristics of human eyes have directionality, are more sensitive to the vertical direction and the horizontal direction and are weaker in the brightness variation in other directions, the weights in the vertical direction and the horizontal direction can be selectively increased when the background brightness is estimated, and the weights in the inclined direction are correspondingly weakened. The average background intensity of the pixel points under the RIST model is:

wherein alpha and beta are weight coefficients, and Q and D are respectively an upper, lower, left and right 4 neighborhood of the image pixel point and a 4 neighborhood on a diagonal line.

For places with higher background brightness, brightness enhancement is required to be limited, so that the phenomenon of unnatural brightness of the enhanced image is avoided. The result after brightness adjustment of the V component is as follows:

through the adjustment of the formula, the problem of brightness enhancement in the image can be effectively improved, and the enhanced image brightness is more in line with the visual characteristics of human eyes.

Saturation correction: in the RGB color space, changing the color channel values causes a large change in the image, for example, increasing the R component values causes the image to redder and increasing the R and G component values causes the image to yellow. In the HSV color space, however, increasing the brightness of the image does not shift the image color toward other colors because the brightness and color components are separate. However, the brightness increase in the high brightness region can lead to the saturation becoming lighter, and the image color becoming lighter, as shown in fig. 6. The image (b) shows the color fading phenomenon of the high brightness area of the image after the brightness is increased. The result after the saturation correction is shown in the graph (c). The image effect after correction is better than the simple brightness improvement effect.

After the luminance component is enhanced by using the Retinex algorithm, the internal relationship between the luminance and the saturation of the image changes, resulting in color fading of the enhanced image, so that the saturation component needs to be adjusted to make the saturation follow the luminance component and change synchronously. The image brightness and saturation are not completely independent and can be cooperatively changed according to the correlation coefficient of the two. The adjustment is thus made using the following formula, such as the saturation of the image:

S′(x,y)＝S(x,y)+t(V′(x,y)-ρ(x,y)×V(x,y)) (26)

where ρ (x, y) represents the correlation coefficient of the image, defined as follows:

omega is the n-n neighborhood window of the image, inside which the luminance mean is defined And saturation mean valueThe formula is as follows:

Wherein,

The S component is changed along with the change of the V component through the correlation coefficient of the V component and the S component, the saturation is correspondingly changed while the brightness is adjusted, the color fading of the image can be avoided, and the final enhancement result is more natural.

In the description of the present invention, unless otherwise indicated, the meaning of "a plurality" is two or more.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof; the present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein; any reference sign in a claim should not be construed as limiting the claim concerned.

Furthermore, it should be understood that although the present disclosure describes embodiments, not every embodiment is provided with a separate embodiment, and that this description is provided for clarity only, and that the disclosure is not limited to the embodiments described in detail below, and that the embodiments described in the examples may be combined as appropriate to form other embodiments that will be apparent to those skilled in the art.

Claims

1. The low-light image enhancement method based on the Retinex model self-adaptive structure is characterized by comprising the following steps of: the method comprises the following steps:

Step 3: l0 smoothing is carried out on the reflection diagram R0;

Step 6: gamma correction is carried out on the illumination map;

step 7: obtaining a result illumination map and a result reflection map;

step 8: outputting a final enhancement result;

Respectively extracting illumination and reflection information from an input image, and carrying out L0 sparse expression on a reflection image; adopting an alternate direction least square method;

aiming at a Retinex low-illumination image enhancement algorithm based on a TSRI model; according to the relation between the visual characteristics of human eyes and the background brightness of the image, carrying out nonlinear adjustment on the enhanced brightness component, so that the visual effect of the enhanced image accords with the visual characteristics of human eyes; and correcting the saturation by utilizing the related characteristics of the brightness and the saturation to ensure that the color sense of the image is more full.