CN109447930B - Wavelet Domain Light Field All-Focus Image Generation Algorithm - Google Patents

Abstract

The wavelet domain light field all-focus image generation algorithm of the present invention belongs to the field of all-focus image fusion, the invention effectively avoids the block effect of the traditional spatial domain light-field image fusion algorithm, and obtains a high-quality light field all-focus image. The 4D light field data obtained by the light field camera is spatially transformed and projected to obtain a multi-focus image for all-focus image fusion. Wavelet decomposition is performed on each frame of the multi-focus image to extract high and low frequency sub-image sets, and a regional balanced Laplacian is proposed. The high-frequency and low-frequency wavelet coefficients of the fused image are respectively constructed by the Si operator and the pixel visibility function to realize image fusion, and its performance is better than the traditional area definition evaluation function. The raw data of the light field camera is used to calculate the fused all-in-focus image, and compared with the traditional image fusion algorithm, the visual effect of the human eye is better, and the objective image index has also been improved.

Description

Wavelet Domain Light Field All-Focus Image Generation Algorithm

technical field

The invention belongs to the field of all-focus image fusion, in particular to an all-focus image generation algorithm in a wavelet domain light field.

Background technique

With the rise of the new subject field of computational photography and the development of light field imaging theory, light field cameras have become the focus of attention in many fields at home and abroad in the past decade. Compared with the traditional camera, the microlens light field camera adds a microlens array after the main lens and records the position and direction information of the spatial light. The recording of multi-dimensional light field information provides convenience for post-image processing and application of light field cameras, such as digital refocusing, all-focus image generation and depth information calculation using light field cameras. The light field camera can calculate the refocusing image of any depth in space after a single shot, which is the most prominent technical highlight of the light field camera. Based on this, the acquisition of high-quality texture images of various light fields and the calculation of high-precision depth information have been deeply studied. Since it is not limited by the traditional camera to obtain multi-focus images through multiple focus shooting, all-focus image fusion based on light field digital refocusing technology has become an important branch application of light field cameras. Reconstruction, generation of light field video files is also significant.

At present, all-focus fusion for traditional images is mainly divided into spatial domain and transform domain. The spatial domain performs sharpness evaluation based on pixels or blocks, and extracts high-quality pixels from different images to form an all-focus image. The calculation time is fast but there are blocks. effect problem. The transform domain decomposes the image into sub-images of different resolution layers or different frequency bands. By evaluating the reconstructed resolution layers or sub-images to construct a refocusing image, the blocking effect can be effectively avoided. As a general method in the transform domain, wavelet transform decomposes the image to be fused into a series of frequency channels, and uses the decomposed tower structure to construct high and low frequency sub-images. The inverse transformation results in an all-in-focus image. The quality of wavelet transform fusion image depends on the selection of fusion rules for high and low frequency sub-images: for low-frequency sub-images, the mean value calculation method is generally used to achieve fusion; for high-frequency sub-images, Sobel operator, Prewitt operator and Laplacian are often used. Evaluate and establish fusion rules by using Si operator and so on.

The second derivative of the traditional Laplacian operator in the x and y directions may have opposite signs, and the existing Laplacian algorithm has low anti-noise ability, and the microlens calibration error will cause refocusing The image produces local noise. In addition, the existing low-frequency signal fusion weighted average method will reduce the contrast of the fused image and lose some useful information in the original image.

SUMMARY OF THE INVENTION

Aiming at the problems of low contrast of images captured by existing civilian-grade light field cameras, limited resolution of multi-focus image sets obtained by digital refocusing technology, and local noise caused by calibration errors, the present invention aims to provide a wavelet-based Domain light field all-focus image generation algorithm, this algorithm realizes light field all-focus image generation by establishing high-frequency coefficient area equalization Laplacian operator and low-frequency coefficient pixel visibility function. The conversion to an all-focus image, and the image fusion quality is improved compared with the traditional algorithm.

In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: a wavelet domain light field all-focus image generation algorithm is implemented according to the following steps:

Step 1): Decode the original image of the light field to obtain a 4D light field, select different α _n (n=1, 2, 3...), and use digital refocusing technology to obtain refocusing images of different spatial depths

Step 2) Calculate the wavelet high and low frequency sub-images of the refocusing image of each frame

Step 3) respectively adopting the regional balanced Laplacian BL operator and the pixel visibility PV function as the image fusion sharpness evaluation index for the high and low frequency sub-images to realize the fusion of the high and low frequency coefficients;

Step 4) The high and low frequency coefficients are subjected to wavelet inverse transformation to obtain a fused all-focus image.

Further, the BL operator adopted in step 3) is the regional equilibrium Laplacian operator, and the expression of this operator is as follows:

表示权重因子，距离中心点越近的点，权重因子越大，对拉普拉斯算子值贡献越大，反之，距离中心点越远，对拉普拉斯算子值贡献越小。Among them, S×T represents the size of the equalization area, and S and T can only take odd numbers; s and t represent the second-order guide step size in the horizontal and vertical directions;

Indicates the weight factor. The closer the point is to the center point, the larger the weight factor, and the greater the contribution to the Laplacian value. On the contrary, the farther away from the center point, the smaller the contribution to the Laplacian value.

Further, in step 3), the PV function is a pixel visibility function, and the specific expression is as follows:

表示S×T区域像素的平均灰度值。Among them, S×T represents the rectangular neighborhood with the current pixel as the center, and S and T can only take odd numbers; s and t represent the scanning step size in the horizontal and vertical directions in the rectangular neighborhood;

Represents the average gray value of pixels in the S×T area.

Further, the fusion rules of low-frequency coefficients and high-frequency coefficients are the same. Taking the fusion of high-frequency coefficients as an example, the rules are as follows:

代表不同空间深度各重聚焦图像经小波分解后各自的高频子图像，n＝1,2,3…N，N表示参与全聚焦图像融合的重聚焦图像帧数；

代表任意2幅高频子图像对应点的均衡拉普拉斯算子的差值；max[·]、min[·]为取最大值、最小值操作；H_H为自定义门限阈值(H_H取0.1，因为当两者差值小于0.1时表示两者区域均衡拉普拉斯值差异很小可以忽略不计)，当差值的最小值大于门限阈值时，取N帧图像中均衡拉普拉斯能量最大者所对应的高频系数作为融合系数，当两者差值小于门限阈值时，由多帧图像高频系数乘以权重因子来决定最后的融合系数，其中权重因子

in,

Represents the respective high-frequency sub-images of each refocusing image at different spatial depths after wavelet decomposition, n=1, 2, 3...N, N represents the number of refocusing image frames participating in the fusion of all-focus images;

Represents the difference of the equalization Laplacian of the corresponding points of any two high-frequency sub-images; max[ ], min[ ] are the maximum and minimum operations; H _H is the custom threshold (H _H Take 0.1, because when the difference between the two is less than 0.1, it means that the difference between the two regional equalized Laplaces is very small and can be ignored), when the minimum value of the difference is greater than the threshold, the equalized Laplacian in N frames of images is taken. The high-frequency coefficient corresponding to the one with the largest energy is used as the fusion coefficient. When the difference between the two is less than the threshold, the final fusion coefficient is determined by multiplying the high-frequency coefficient of the multi-frame image by the weighting factor. The weighting factor

The invention adopts the method of wavelet transform to perform image fusion. Firstly, the 4D light field is decoded and the digital refocusing algorithm is used to obtain multi-focus images of different depths. The high and low frequency sub-image sets are constructed by wavelet decomposition and tower reconstruction of each multi-focus image set. Finally, the regional balanced Laplacian is proposed. The high and low frequency wavelet coefficients of the fused image are respectively constructed by the operator and the pixel visibility function to realize image fusion. This algorithm effectively realizes the conversion of the original light field data to the all-focus image, effectively avoids the block effect of the traditional spatial image fusion algorithm, and obtains a high-quality light-field all-focus image, and the image fusion quality is improved compared with the traditional algorithm.

Description of drawings

The present invention will be described in further detail below with reference to the accompanying drawings.

Figure 1 is a flow chart of the algorithm of the present invention.

Figure 2 is a biplane parameterized model of the light field.

Figure 3 is a schematic diagram of the digital refocusing of the light field camera.

FIG. 4 is a schematic diagram of the BL operator.

Figure 5 is a demonstration diagram of the fusion process of Leaves sample images.

Figure 6 is a comparison diagram of different fusion algorithms of Flower sample images.

Figure 7 is a comparison diagram of different fusion algorithms of Forest sample images.

Figure 8 is a comparison diagram of different fusion algorithms of Zither sample images.

Detailed ways

In order to make the objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

As shown in Figure 1, the specific process of this algorithm is:

Step 1) The original image of the light field is decoded to obtain a 4D light field, select different α _n (n=1, 2, 3...), and use digital refocusing technology to obtain refocusing images of different spatial depths

Step 2) Calculate the wavelet high and low frequency sub-images of the refocusing image of each frame

Step 3) High and low frequency sub-images respectively adopt BL operator and PV function as image fusion sharpness evaluation indicators, to realize the fusion of high and low frequency coefficients;

Step 4) Finally, the fused all-focus image is obtained by inverse wavelet transform.

The specific process of step 1) is as follows: According to the biplane parameterized model of the light field, as shown in Figure 2, any ray in space can be determined by its intersection with the two planes, and the main lens plane of the light field camera is set as (u, v) plane, the sensor plane is the (x, y) plane, the 4D light field recorded by the light field camera is _LF (x, y, u, v), the integral image of the focal plane of the plenoptic camera can be obtained by the classical light radiation formula :

where _F represents the distance between the main lens plane and the focal plane, and X×Y×U×V represents the size of the 4D light field matrix LF (x, y, u, v). If the image plane is moved from F to F', the new 4D light field matrix is represented by L _F' (x', y', u', v'), and the refocused image of the camera focal plane is represented as:

Let F′=α _n ·F, for the convenience of graphical representation, take a tangent plane in the 4D space to obtain the geometric relationship between the coordinates, as shown in Figure 3. According to the similar triangle principle, it can be obtained that the coordinates of the new light field and the original light field satisfy:

x'=u+(xu)·α _n =α _n ·x+(1-α _n )·u (3)

u′=u (4)

Similarly, we can get:

y′=v+(yv)·α _n =α _n ·y+(1-α _n )·v (5)

v′=v (6)

Equations (3)-(6) can be expressed in matrix form:

表示坐标变换矩阵，具体形式如下：Among them, [x', y', u', v'] ^T represents the transpose of the row vector [x', y', u', v'],

Represents the coordinate transformation matrix, the specific form is as follows:

Equation (7) is also equivalent to the following:

According to formula (9), formula (2) can be rewritten as:

By changing the value of α _n , the purpose of changing the position of the image plane can be achieved, and then refocusing pictures of different spatial depths can be obtained.

The specific process of step 2) is as follows: According to the image fusion theory of wavelet transform, the image to be fused is decomposed into a series of frequency channels through wavelet transform, and the high and low frequency sub-images are constructed by using the decomposed tower structure. The process can be described as: :

where (x, y) represents the image coordinate system, (i, j) represents the wavelet domain coordinate system, W[ ] represents the wavelet tower decomposition operator, and W _H [ ] represents the high frequency coefficients extracted after the wavelet tower decomposition ( high-frequency sub-image), W _L [·] represents the extraction of low-frequency coefficients (low-frequency sub-image) after wavelet tower decomposition.

The BL operator adopted in step 3) is the regional equilibrium Laplacian operator, and the expression of this operator is as follows:

表示权重因子，距离中心点越近的点，权重因子越大，对拉普拉斯算子值贡献越大，反之，距离中心点越远，对拉普拉斯算子值贡献越小。图4为S＝5、T＝5时的均衡拉普拉斯算子。Among them, S×T represents the size of the equalization area, and S and T can only take odd numbers; s and t represent the second-order guide step size in the horizontal and vertical directions;

Indicates the weight factor. The closer the point is to the center point, the larger the weight factor, and the greater the contribution to the Laplacian value. On the contrary, the farther away from the center point, the smaller the contribution to the Laplacian value. Figure 4 shows the equalized Laplacian operator when S=5 and T=5.

The high-frequency sub-image of the wavelet transform reflects the brightness mutation characteristics of the image, that is, the boundary characteristics. The Laplacian operator can sharpen the boundaries and lines of any direction, and maintain the isotropic characteristics. It is widely used in sharpness evaluation. In view of the situation that the second derivative of the Laplacian operator in the x direction and the y direction may have opposite signs, and at the same time, the influence of the surrounding points on the current position definition evaluation function is fully considered. The invention proposes a regional balanced Laplacian The operator achieves energy balance by increasing the number and direction of the second derivative.

Considering that the microlens calibration error will cause local noise in the refocusing image, and the Laplacian operator is sensitive to noise, bilateral filtering preprocessing is performed before the high-frequency sub-image is fused. The present invention is based on regional equalization Laplacian The high-frequency coefficient fusion rules of the Lass operator are as follows:

代表不同空间深度各重聚焦图像经小波分解后各自的高频子图像，n＝1,2,3···N，N表示参与全聚焦图像融合的重聚焦图像帧数；D(BL_αn(i,j))代表任意2幅高频子图像对应点的均衡拉普拉斯算子的差值；max[·]、min[·]为取最大值、最小值操作；HH为自定义门限阈值(HH取0.1，因为当两者差值小于0.1时表示两者区域均衡拉普拉斯值差异很小可以忽略不计)，当差值的最小值大于门限阈值时，取N帧图像中均衡拉普拉斯能量最大者所对应的高频系数作为融合系数，当两者差值小于门限阈值时，由多帧图像高频系数乘以权重因子来决定最后的融合系数，其中权重因子

in,

Represents the respective high-frequency sub-images of each refocusing image at different spatial depths after wavelet decomposition, n=1, 2, 3...N, N represents the number of refocusing image frames participating in all-focus image fusion; D(BL _αn ( i,j)) represents the difference between the equalized Laplacian operators of the corresponding points of any two high-frequency sub-images; max[ ], min[ ] are the maximum and minimum operations; HH is the user-defined threshold Threshold (HH takes 0.1, because when the difference between the two is less than 0.1, it means that the difference between the two regional equilibrium Laplacian values is very small and can be ignored), when the minimum value of the difference is greater than the threshold, the equalization in N frame images is taken. The high-frequency coefficient corresponding to the one with the largest Laplacian energy is used as the fusion coefficient. When the difference between the two is less than the threshold, the final fusion coefficient is determined by multiplying the high-frequency coefficient of the multi-frame image by the weighting factor. The weighting factor

The low-frequency coefficient obtained by the wavelet tower decomposition in step 2) mainly reflects the average grayscale feature of the original image. The easiest way to calculate the low-frequency fusion coefficient is the weighted average method, but the weighted average method will reduce the contrast of the fused image and lose some useful information in the original image. In addition, some gradient calculation methods such as spatial frequency method and point sharpness operator are also applied to the calculation of low-frequency fusion coefficients. In the light field image low-frequency coefficient fusion of the present invention, the concept of image visibility (Image visibility, VI for short) based on human visual characteristics is used for reference, and its definition is as follows:

表示图像I(i,j)的平均值；γ表示视觉常量，其取值范围为0.6～0.7，VI的值越大，代表图像可见度越高。where P×Q represents the size of the image I(i,j);

Represents the average value of the image I(i,j); γ represents the visual constant, and its value ranges from 0.6 to 0.7. The larger the value of VI, the higher the visibility of the image.

In the fusion process of low-frequency sub-images, if the formula (15) is directly used for calculation, only the VI value of the whole image can be obtained, and it cannot be used for regional or pixel-level fusion of multiple images. In order to reasonably establish an effective low-frequency coefficient evaluation index, we improve Equation (15) and establish a pixel-based image visibility function (Pixel visibility, PV for short), the specific expression is as follows:

表示S×T区域像素的平均灰度值。在低频系数融合过程中，采用与高频系数相同的融合规则。Among them, S×T represents the rectangular neighborhood centered on the current pixel, and S and T can only take odd numbers; s and t represent the scanning step size in the horizontal and vertical directions in the rectangular neighborhood;

Represents the average gray value of pixels in the S×T area. In the fusion process of low frequency coefficients, the same fusion rules as high frequency coefficients are used.

Finally, the fused high and low frequency coefficients are subjected to wavelet inverse transform to obtain the fused all-focus image.

The above is a detailed description of the wavelet domain light field all-focus image generation algorithm of the present invention, and the validity of the algorithm is verified by a specific example below.

The present invention uses the original image captured by the Lytro light field camera to carry out the experiment. Figure 5(a) is the original image of the light field, and Figures 5(b), (c), and (d) are three different spaces calculated according to formula (10) when α=0.52, α=0.78, and α=0.98, respectively A multi-focus image of depth, where the depth of focus gradually changes from the foreground to the background. Figure 5(e) is an all-focus image calculated by the method of the present invention, the definition of the area framed by the red dotted line is obviously higher than that of the corresponding area of (b), and the clarity of the area framed by the yellow dotted line is obviously higher than that of the corresponding area of (c) , the clarity of the area framed by the white dashed line is obviously higher than that of the corresponding area in (d), it can be seen that the algorithm of the present invention can effectively use the original image of the light field to obtain a fully focused image.

In order to visually evaluate the advantages of the algorithm proposed by the present invention, three classical image fusion methods based on wavelet transform (Sobel algorithm, Prewitt algorithm, Laplace algorithm) are selected for comparison with the algorithm proposed by the present invention. Field original image (Flower, Forest, Zither).

Figure 6, Figure 7, and Figure 8 are the corresponding experimental results: (a) and (b) of each figure are the refocusing images obtained when the three original light field images correspond to α=1 and α=2, respectively. The depth of focus is from The foreground is transformed into the background; (c), (d), (e), and (f) of Figures 6-8 are all-focus images obtained by the Sobel algorithm, the Prewitt algorithm, the traditional Laplace algorithm and the algorithm of the present invention. From the visual effect, the sharpness of the fusion image obtained by the Sobel algorithm and the Prewitt algorithm in Fig. 6 is obviously inferior to the algorithm of the present invention in the rectangular area framed by the dotted line; the clarity obtained by the Sobel algorithm and the Prewitt algorithm in the area framed by the dotted line in Fig. 7 is also obviously inferior to the algorithm of the present invention. Fig. 8 adopts the sharpness of the area framed by the corresponding dotted line (checking the position of the dotted line frame) of the plant leaves fused by the Prewitt algorithm also obviously not as good as the algorithm of the present invention; Explain that the light field all-focus image fusion method of the present invention has a certain visual effect. Advantage.

In addition, considering the visual limitation of human eyes, the present invention further selects some objective evaluation indicators to evaluate the image quality to verify the superiority of the algorithm of the present invention. The present invention selects information entropy (E), average gradient (AG), image sharpness (FD) and edge intensity (EI) as evaluation indicators respectively, and compares the all-focus images obtained by various methods in Fig. 6 , Fig. 7 and Fig. 8 quality is evaluated.

Among them, E is a physical quantity that measures the size of the information, and the larger the value, the greater the amount of image information. AG can sensitively reflect the contrast ability of the image to small details. The higher the value, the stronger its ability. FD stands for image clarity, and the higher the value, the better the clarity. EI reflects the edge strength of the image. The higher the value, the clearer the edge of the image. The corresponding results of the specific evaluation indicators are shown in Table 1, Table 2, and Table 3.

It can be seen from the data in the comparison table that the algorithm of the present invention is superior to the other three traditional wavelet transform methods in the four objective evaluation indexes of the image, which reflects the feasibility and effectiveness of the algorithm of the present invention.

Table 1 Comparison of performance indicators of different fusion algorithms for Flower sample images

Table 1 Comparison of performance indexes of different fusion algorithms for Flower sample images

E FD AG EI Sobel's algorithm 6.8676 6.8991 6.2470 66.5340 Prewitt's algorithm 6.8634 6.3270 5.8326 62.6420 Laplace algorithm 6.8830 7.6837 6.8668 72.3073 The algorithm of the invention 6.8896 7.8498 7.0055 73.7203

Table 2 Comparison of performance indicators of different fusion algorithms for Cucurbit sample images

Table 2 Comparison of performance indexes of different fusion algorithms for Cucurbit sample images

E FD AG EI Sobel's algorithm 5.7544 2.9136 2.5328 26.5157 Prewitt's algorithm 5.7492 2.5766 2.2905 24.2735 Laplace algorithm 5.8011 3.5235 3.0018 31.0134 The algorithm of the invention 5.8099 3.6305 3.0875 31.9033

Table 3 Comparison of performance indicators of different fusion algorithms for Zither sample images

Table 3 Comparison of performance indexes of different fusion algorithms for Zither sample images

E FD AG EI Sobel's algorithm 6.2935 5.1865 4.4675 48.3854 Prewitt's algorithm 6.2695 4.5182 4.0184 43.7566 Laplace algorithm 6.2716 5.6773 4.8649 52.4474 The algorithm of the invention 6.2987 6.2987 4.9425 53.1501

The invention completes the calculation of the original image of the light field to the all-focus image, realizes the fusion of the all-focus image by the method based on the wavelet domain sharpness evaluation, and avoids the block effect caused by the traditional spatial domain image fusion algorithm. Firstly, the 4D light field data obtained by decoding is spatially transformed and projected to obtain a multi-focus image for all-focus image fusion. Then, the high and low frequency sub-image sets are constructed by wavelet decomposition and tower reconstruction of each multi-focus image set. Image fusion. In the fusion of wavelet high-frequency sub-images, the present invention proposes a sharpness evaluation function based on the area equalization Laplacian operator; in the fusion of wavelet low-frequency sub-images, the present invention proposes a sharpness evaluation function based on pixel visibility, to improve the fusion quality of all-in-focus images. The above experiments show that, compared with the traditional algorithm based on wavelet transform, the method proposed in the present invention improves the final fusion image from subjective vision to objective indicators.

The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned embodiments, and within the scope of knowledge possessed by those of ordinary skill in the art, various kind of change.

Claims (2)

1. A method for generating an all-focus image in a wavelet domain light field, characterized in that, it is realized according to the following steps: Step 1): Decode the original image of the light field to obtain a 4D light field, select different α _n , n=1, 2, 3...N, and use digital refocusing technology to obtain refocusing images of different spatial depths

Step 2) Calculate the wavelet high and low frequency sub-images of each frame of the refocusing image

Step 3) respectively adopting the regional balanced Laplacian BL operator and the pixel visibility PV function as the image fusion sharpness evaluation index for the high and low frequency sub-images to realize the fusion of high and low frequency coefficients; The BL operator is a regional equilibrium Laplace operator, and the expression of the operator is as follows:

Among them, S×T represents the size of the equalization area, and S and T can only take odd numbers; s and t represent the second-order guide step size in the horizontal and vertical directions;

Represents the weight factor, the closer the point is to the center point, the larger the weight factor, the greater the contribution to the Laplacian operator value, and vice versa, the farther away from the center point, the smaller the contribution to the Laplacian operator value; The PV function is the pixel visibility function, and the specific expression is as follows:

Among them, S×T represents the rectangular neighborhood with the current pixel as the center, and S and T can only take odd numbers; s and t represent the scanning step size in the horizontal and vertical directions in the rectangular neighborhood;

Represents the average gray value of pixels in the S×T area; Step 4) The high and low frequency coefficients are subjected to wavelet inverse transformation to obtain a fused all-focus image. 2. The wavelet domain light field all-focus image generation method according to claim 1, characterized in that: the low-frequency coefficient and high-frequency coefficient fusion rules are the same, and the high-frequency coefficient fusion is taken as an example, and the rules are as follows:

in,

Represents the respective high-frequency sub-images of each refocusing image at different spatial depths after wavelet decomposition, n=1, 2, 3...N, N represents the number of refocusing image frames participating in all-focus image fusion;

Represents the difference of the equalization Laplacian of the corresponding points of any two high-frequency sub-images; max[ ], min[ ] are the operations of taking the maximum and minimum values; H _H is the user-defined threshold threshold, when the difference When the minimum value of the value is greater than the threshold, the high-frequency coefficient corresponding to the one with the largest equalized Laplacian energy in the N frames of images is taken as the fusion coefficient. The final fusion coefficient is determined by the weight factor, where the weight factor

Images