TW201804434A - Adaptive high dynamic range image fusion algorithm - Google Patents

Abstract

Adaptive high dynamic range image fusion algorithm has steps comprising of: building a learn and image combination model, fetching a database, and performing a training and HDR combination. The combination model receives original images and combines the images as HDR images with referencing to network parameters. Training step inputs database in model and compares original images and HRD images to generate the parameters. Thus, the present invention is able to learn the images with different HDR from different users' picture taking habit, the output images may become nearly perfect.

Description

Adaptive high dynamic range image synthesis algorithm

The invention relates to a learnable image synthesis method, in particular to an adaptive high dynamic range image synthesis algorithm.

The blackest/whitest ratio (ie Contrast Ratio) that can be displayed in a typical digital camera depends on the sensitivity of the light-sensing element. The light-sensing element used in digital cameras generally has a contrast ratio of about 1: 4,096, however, the contrast of human-readable images is as high as 1:100,000, which is about 24 times that of a typical digital camera. Therefore, when taking pictures with a digital camera, the image details are easily lost due to too dark shadows or too bright parts. To this end, there is a high dynamic range image (High Dynamic Range Image) synthesis technology, which can be used to reproduce the image and increase the dynamic range of the image captured by the digital camera to reduce the loss of image details.

Early high dynamic range images (HDRI) were mainly performed by manual post-production. Photographers can use the Auto Exposure Bracketing (AEB) built in general digital cameras to shoot images, which is automatic when shooting images. The bracketing function enables the digital camera to capture multiple images with different exposures, and then combines multiple images into a high-dynamic range image with image-based software.

The existing image processing software has a high dynamic range image synthesis function, and one of the synthesis techniques utilizes the contrast (Contrast), saturation (Saturation) and exposure (Well-Exposedness) of each pixel in the input image, referred to as CSE. The product of the three is used as the composite weight, and the complex pixels corresponding to the position are synthesized according to the combined weights, thereby obtaining a high dynamic range image.

At present, high dynamic range image synthesis technology has been gradually applied to smart phones or high-end digital cameras. Users can obtain high dynamic range images through the operation of smart phones or digital cameras after shooting images, but due to the above high dynamics Range image synthesis technology involves high-level image processing expertise, so there is currently no user to adjust network parameters, composite weights or synthesis formulas, so that the existing high dynamic range built into smart devices or digital cameras The image composite image function cannot be adapted to users of different shooting habits, and the synthetic effect is still not good. Most users still have to use the image synthesis software to make high dynamic range image synthesis in the future. In addition to technology, it takes time to manually perform image post-processing.

In view of the technical defects that the existing high dynamic range image synthesis method cannot adapt to the shooting habits of different users, the present invention proposes an adaptive high dynamic range image synthesis algorithm, which includes:

Establishing a machine learning and image synthesis model, the machine learning and image synthesis model is pre-set with a plurality of network parameters, and after receiving a plurality of original images with different exposures, synthesizing a reference high dynamic range image according to the network parameters;

Obtaining an image database, the image database comprising a plurality of image groups, each image group comprising a plurality of original images of different exposures;

Performing a training step, the plurality of image groups of the image database are input into the machine learning and image synthesis model, and the machine learning and image synthesis model is compared with the original image and the corresponding reference high dynamic range image, and the network is adjusted. parameter;

For high dynamic range image synthesis, a set of images to be synthesized is input into the trained machine learning and image synthesis model, so that the machine learning and image synthesis model synthesizes an output high dynamic range image according to the adjusted network parameters.

The above adaptive high dynamic range image synthesis algorithm can be optimized by the user to provide an image database for the machine learning and image synthesis model, thereby adapting to different shooting habits, and the above adaptation is high. The dynamic range image synthesis algorithm uses the principle of machine learning to adjust the network parameters. The user does not need to have the expertise of image processing to operate, which is conducive to popularization.

Referring to FIG. 1 , the main steps of the adaptive high dynamic range image synthesis algorithm of the present invention include:

Establishing a machine learning and image synthesis model, the machine learning and image synthesis model is pre-set with a plurality of network parameters, and after receiving a plurality of original images with different exposures, synthesizing a reference high dynamic range image according to network parameters, In the embodiment, the calculation of the weight of the image synthesis is performed by using the neural network, so that the weight of the composite network can be adjusted by adjusting the complex network parameters, and the detailed steps are described later;

Obtaining an image database, the image database comprising a plurality of image groups, each image group comprising a plurality of original images of different exposures;

Performing a training step, the plurality of image groups of the image database are input into the machine learning and image synthesis model, and the machine learning and image synthesis model is compared with the original image and the corresponding reference high dynamic range image, and the network is adjusted. parameter;

For high dynamic range image synthesis, a set of images to be synthesized is input into the trained machine learning and image synthesis model, so that the machine learning and image synthesis model synthesizes an output high dynamic range image according to the adjusted network parameters. .

Please further refer to FIG. 2, the above machine learning and image synthesis model, the execution step includes a weight calculation program, a high dynamic range image synthesis program and a parameter optimization program, which can be achieved by hardware or software, and the weight calculation program is After receiving the contrast, saturation, and brightness of each pixel of the original image, calculating a composite weight according to the complex network parameter, the high dynamic range image synthesis program receives the complex original image and the weight calculation program calculates the corresponding Combining the weight of the pixel, and synthesizing the complex original image into the reference high dynamic range image according to the composite weight, the parameter optimization program receives the original image and the corresponding reference high dynamic range image, and compares the reference high dynamic range image and corresponding After the original image, adjust the network parameters of the weight calculation program, and the steps are detailed later.

The weight calculation program is implemented by a type of neural network, and its architecture diagram is as shown in FIG. 3, which includes an input layer, a hidden layer and an output layer, and the input layer receives the contrast, saturation and pixel of each pixel of the original image. The brightness layer is converted according to the complex internal function by an excitation function to output the composite weight Y. In this embodiment, the complex network parameter includes at least an input weight w ₁₁ ~ w _3h and an excitation function φ ₁ ~ φ _h and the output weight β ₁ ~β _k , the output layer outputs the combined weight corresponding to each pixel, and the contrast, saturation and brightness of each pixel of the original image are calculated by the following methods:

The contrast parameter C is obtained by pixel graying and Laplace conversion;

The saturation parameter S is obtained using the HSV model:

The brightness parameter L is the gray scale value of the pixel

The excitation function of the hidden layer of the above weight calculation program is as follows:

（w為輸入權重、x為輸入、b為偏權值），w與b為介於0至1之間隨機決定之值。

(w is the input weight, x is the input, b is the bias value), and w and b are randomly determined values between 0 and 1.

，再由後續訓練步驟進一步調整網路參數。Moreover, when the weight calculation program calculates the combined weight for the first time, the network parameter can be initially set such that the combined weight is the product of the multiplication of contrast, saturation, and brightness.

Then, the network parameters are further adjusted by subsequent training steps.

係利用高斯曲線取得：

，

為像素之正規化灰階值，

設為一固定值0.2。Exposure parameter

Use the Gaussian curve to obtain:

,

Normalized grayscale values for pixels,

Set to a fixed value of 0.2.

Please further refer to FIG. 4, in order to reduce the phenomenon of synthetic seams in high dynamic range images, the high dynamic range image synthesis program uses Laplacian pyramid and Gaussian pyramid image processing for high dynamic range image synthesis, which includes the following step:

Performing a multi-level Gaussian pyramid transformation and a Laplacian pyramid transformation on a plurality of original images;

According to the composite weight of each pixel of the original image, a composite weight map is obtained, corresponding to the combined weight of each pixel, and the combined weight map is synthesized according to the position of the corresponding pixel, and the weighted graph is converted into a Gaussian pyramid;

Multiplying the converted complex original image with the corresponding pair of converted composite weight maps according to the level of level, and then superimposing the images of each level after multiplication, as shown in FIG. 5, to obtain the reference. High dynamic range image.

，G為經過高斯金字塔轉換的合成權重圖，L為經過拉普拉斯金字塔轉換的原始影像。）(which is

, G is a composite weight map transformed by Gaussian pyramid, and L is the original image transformed by Laplacian pyramid. )

The above Gaussian pyramid is transformed as follows: (R is Reduce, the higher the Gaussian pyramid level, the smaller the image size)

Where i and j are the coordinates of the pixel points, l is the number of pyramid layers, and w is

The above Laplacian pyramid conversion is as follows:

Where E is Enhance, the higher the Laplacian pyramid level, the smaller the image size.

=

=

The calculation formula of the superposition is as follows:

=

=

Where N is the largest pyramid level;

L and g represent the Laplace and Gaussian pyramids, respectively.

Further, referring to FIG. 6 , when the adaptive high dynamic range image synthesis algorithm performs the training step, the parameter optimization program receives the original image and the corresponding high dynamic range image synthesized by the high dynamic range image synthesis program. After that, proceed as follows:

Obtaining a plurality of original images and corresponding reference high dynamic range images;

Comparing the structural similarity (SSIM) of the reference high dynamic range image with the complex original image, and determining whether each pixel in the original image is normally exposed by whether the grayscale value is between 255*0.1 and 255*0.9. In practice, a reference value can be set. If the structural similarity is calculated to be lower than the reference value and there is no overexposure, it is determined that the exposure is normal, and the basis for determining whether the exposure is normal can be set other conditions, and is not limited to the above conditions;

If the exposure is normal, the combined weights of the corresponding pixels are increased to obtain a new combined weight. In this embodiment, after each judgment, the combined weight is multiplied by (1+0.1) to increase the combined weight;

According to the previous step, the new composite weight is calculated as the learning target, and the network parameters of the weight calculation program are adjusted, and the detailed adjustment manner is described later;

Obtaining a new composite weight, and obtaining a new reference high dynamic range image, and returning to the step of comparing the similarity between the reference high dynamic range image and the original image structure, the weight calculation program recalculates the new composite weight, and The high dynamic range image synthesis program synthesizes a new reference high dynamic range image, and then calculates the structural similarity and adjusts the network parameters by the optimization program.

*HThe above parameter optimization program adjusts the excitation function by calculating a new network parameter by using Online Sequential ELM (OSELM). This embodiment takes the calculation of a new output weight β ₁ ~ β _k as an example. The above-mentioned online sequence limit machine learning system is shown in FIG. 7. In this embodiment, the output weight β in the complex network parameter of the above-mentioned excitation function H ₀ is recalculated, as follows:

*H

Where β _d+1 is the adjusted output weight, and T is the combined weight output after the excitation function is again passed.

The actual experimental data shows that the number of nodes of the neural network in the above weight calculation program exceeds 50, which can cause the error rate to converge and reduce to a stable range. Therefore, the node of the above-mentioned neural network can be set to 50, The embodiment is preferably set to 100.

The above parameter optimization program repeatedly adjusts the network parameters of the neural network based on the recursive operation to achieve the purpose of learning, and the number of recursive times can be preset by an optimization number, and the number of times of adjusting the network parameters is repeatedly performed to achieve the optimization. After the number of times, the optimization program is stopped, or the condition for terminating the training with the SSIM index value is set, or the two are used as the conditions for stopping the training.

The adaptive high dynamic range image synthesis algorithm of the invention can be built in a digital camera or a smart portable device, and the photo taken by the user is used as an image database to train the machine learning and image synthesis model to Adjusting the network parameters of the weight calculation program, the machine learning and image synthesis model can be adapted to the usage habits of different users.

The purpose of the parameter optimization program in the above machine learning and image synthesis model is to adjust the network parameters of the weight calculation program, and it is not necessary to perform parameter optimization through the parameter optimization program every time the high dynamic range image synthesis is performed, that is, when using After training the machine learning and image synthesis model with the self-photographed image database, most of the network parameters have been converged, and the subsequent input photos can directly use the previously adjusted network parameters for high dynamic range image synthesis output. Output high dynamic range images.

The adaptive high dynamic range image synthesis algorithm of the invention can not only achieve the purpose of adapting to different user habits, but also improve the quality of image synthesis, and present image synthesis quality by statistical data. The following three image indicators are used together with FIG. 7 and Figure 8 shows the data.

The experiment uses the image index value as an indicator to evaluate the high dynamic range shadow sample quality. The larger the index value, the higher the image quality, and the image index value is calculated as ASSIM*STD*IE*1000, where:

為各像素的SSIM指標值，p為總像素數量，M、N為各行、列的總像素數。

among them:

For the SSIM index value of each pixel, p is the total number of pixels, and M and N are the total number of pixels in each row and column.

為平均亮度，

為座標(i,j)像素的亮度。

among them:

For average brightness,

Is the brightness of the coordinates (i, j) pixels.

Where: L is the total order of gray-scale pixels, and p _i is the probability of the i-th pixel.

The experimental data shown in Figure 7 shows that the image1~image36 are 36 sets of test images, and two index values are calculated for each test image. The left column is labeled with a traditional high dynamic range image (not adjusted). Synthetic weight) The image index value calculated after comparison with the original image, and the right column indicates the image index value calculated by comparing the high dynamic range image synthesized by the adaptive high dynamic range image synthesis algorithm of the present invention with the original image, which can be seen in the figure. The present invention can improve the quality of the synthesized image compared with the conventional high dynamic range image synthesis method, and the statistical data of FIG. 8 shows that the adaptive high dynamic range image synthesis algorithm of the present invention has better image optimization effect. .

Figure 1 is a flow chart of the present invention. Figure 2 is a detailed flow chart of the execution of the training steps in Figure 1. Figure 3 is a schematic diagram of the implementation of the weight calculation procedure in Figure 2 with a neural network. Figure 4 is a detailed flow chart of the steps of the high dynamic range image synthesis procedure of Figure 2. Figure 5: Schematic diagram of the Gaussian and Laplacian pyramid transformation in Figure 4. Figure 6 is a detailed flow chart of the steps of the parameter optimization procedure in Figure 2. Figure 7 is a statistical graph of experimental data comparing the present invention with the prior art. Figure 8 is a diagram showing another experimental data of the present invention compared with the prior art.

Claims (10)

An adaptive high dynamic range image synthesis algorithm includes: establishing a machine learning and image synthesis model, the machine learning and image synthesis model is pre-set with a plurality of network parameters, and receiving a plurality of original images with different exposures Then, synthesizing a reference high dynamic range image according to the network parameter; obtaining an image database, the image database comprising a plurality of image groups, each image group comprising a plurality of original images of different exposures; performing a training step The plurality of image groups of the image database are input into the machine learning and image synthesis model, and the machine learning and image synthesis model is compared with the original image and the corresponding reference high dynamic range image, and the network parameters are adjusted; high dynamic range image synthesis is performed. The machine learning and image synthesis model after the training of the image to be synthesized is input, so that the machine learning and image synthesis model synthesizes an output high dynamic range image according to the adjusted network parameters. The adaptive high dynamic range image synthesis algorithm according to claim 1, the machine learning and image synthesis model performs the following steps: Performing a weight calculation program, after receiving the contrast, saturation and brightness of each pixel of the original image, The complex network parameter is calculated to output a composite weight; performing a high dynamic range image synthesis program, which receives the complex original image and the combined weight calculated by the weight calculation program for each pixel, and the complex number is determined according to the composite weight The original image is synthesized into the reference high dynamic range image; and a parameter optimization program is executed, and the original image and the corresponding reference high dynamic range image are received, and the reference high dynamic range image and the corresponding original image are compared, and the weight calculation program is adjusted. Network parameters. The adaptive high dynamic range image synthesis algorithm according to claim 2, wherein the weight calculation program is implemented by a type of neural network, the neural network comprising an input layer, a hidden layer and an output layer, the input layer The contrast, saturation, and brightness of each pixel of the original image are received, and the hidden layer is converted according to the complex internal function by an excitation function to output the combined weight. The adaptive high dynamic range image synthesis algorithm according to claim 2, wherein the high dynamic range image synthesis program comprises the following steps: performing complex Gaussian pyramid transformation and Laplacian pyramid transformation on the complex original image; The composite weight of each pixel obtains a weight map, and the weighted map is converted into a Gaussian pyramid; the converted complex original image is multiplied by the corresponding composite weight map according to the relative level. The images of each level are multiplied to obtain the reference high dynamic range image. The adaptive high dynamic range image synthesis algorithm according to claim 2, wherein the optimization program comprises the steps of: obtaining a plurality of original images and corresponding reference high dynamic range images; comparing the reference high dynamic range image with the complex original image Structural similarity to determine whether each pixel in the original image is normally exposed; if normal exposure, increase the composite weight of the corresponding pixel to obtain a new composite weight; according to the previous step, calculate a new composite weight as the learning target, adjust the above Weight calculation program network parameters; obtain new composite weights, and obtain new reference high dynamic range images, and return to the step of comparing the similarity between the reference high dynamic range image and the original image structure. The adaptive high dynamic range image synthesis algorithm according to claim 2, wherein the optimization program sets an optimization number, and stops executing the optimization program after repeatedly performing the adjustment network parameter times to reach the optimization number. The adaptive high dynamic range image synthesis algorithm according to claim 3, wherein the high dynamic range image synthesis program comprises the following steps: performing complex Gaussian pyramid transformation and Laplacian pyramid transformation on the complex original image; The composite weight of each pixel obtains a weight map, and the weighted map is converted into a Gaussian pyramid; the converted complex original image is multiplied by the corresponding composite weight map according to the relative level. The images of each level are multiplied to obtain the reference high dynamic range image. The adaptive high dynamic range image synthesis algorithm according to claim 7, the optimization program comprising the steps of: obtaining a plurality of original images and corresponding reference high dynamic range images; comparing the reference high dynamic range image with the complex original image Structural similarity to determine whether each pixel in the original image is normally exposed; if there is no normal exposure, increase the combined weight of the corresponding pixel to obtain a new composite weight; calculate a new composite weight as a learning target according to the previous step, adjust The network parameters of the above weight calculation program; obtaining a new composite weight, and obtaining a new reference high dynamic range image, and returning to the step of comparing the similarity between the reference high dynamic range image and the original image structure. The adaptive high dynamic range image synthesis algorithm according to claim 8, wherein the optimization program sets an optimization number, and stops executing the optimization program after repeatedly performing the adjustment network parameter times to reach the optimization number. The adaptive high dynamic range image synthesis algorithm according to claim 8, wherein the complex network parameter includes at least an input weight, an excitation function, and an output weight, and the optimization program adjusts an output weight in the network parameter of the weight calculation program.