KR20230077854A - Deep Learning Multiple Exposure Fusion Method, device, and program Based on Feature Boosting Weights Using Single Image - Google Patents

Abstract

The present invention relates to a multi-exposure convergence technique based on feature enhancement weights using a single image, which divides one image into multiple regions and applies each gamma correction coefficient of the image in which the amount of information is maximized in each divided region. A method of applying feature enhancement weights that generate multiple multiple exposure images by applying gamma correction to the entire area of a single image and variably applying deep learning weights for multiple exposure convergence according to the amount of information to obtain an improved image; It relates to devices and programs. Through the present invention, the learning efficiency is superior to that of the existing methods, and a more improved image quality improvement effect can be obtained.

Description

Deep learning multiple exposure fusion method, device, and program based on feature boosting weights using single image

The present invention relates to a multi-exposure image fusion method and apparatus, and more particularly, to generate a multi-exposure image input to generate a multi-exposure fusion image using one image, and to generate the multi-exposure image. It is about a deep learning multiple exposure convergence technique that generates and applies feature enhancement weights according to the amount of information in .

Recently, due to the development of digital imaging equipment, it is possible to acquire images and images of higher resolution than in the past. However, these devices still had a problem in that they could not express actual images as they were because the sensor sensitivity and pixel operating range were narrower than the human visual range.

In order to solve this problem, a multi-exposure fusion technique has been conventionally proposed. Multi-Exposure Fusion technology is a technology that extracts necessary information from multiple images with different exposure values and fuses them to create a higher quality image. Saturation and well-exposedness are calculated and a weight map is applied to each pixel value to create a fusion image. However, in order to apply the multi-exposure convergence technology, several images obtained from physically the same location must be secured, and for this purpose, there is a problem in that an expensive physical device for securing multiple-exposure images at the same time point is required.

In addition, the multi-exposure convergence technology requires arbitrarily setting weights for each of contrast, color tone, and exposure information, and such arbitrarily set weights directly affect the quality of the final convergence image, making it impossible to obtain a high-quality image or inconsistent quality. It acts as a cause of not getting improvement. In addition, since blurring due to the average filter effect occurs in the process of fusion processing of multiple images, there is a problem in that a Laplacian filter for correcting this is additionally required.

Meanwhile, in order to solve the problem of arbitrarily setting weights, a feature fusion auto-encoder model (FFAM) based on deep learning has been proposed. The FFAM technique extracts the features of an input image from an auto-encoder in the encoder region to improve image quality, and fuses the extraction result in a fusion layer (or bottleneck layer). A method of restoring this in the decoder area is used. However, this feature fusion auto-encoder model (FFAM) applies uniform weights to each image for encoders composed of as many images as input in the fusion layer, and then simply synthesizes the result and passes it to the decoder. Therefore, there is a problem in that the convergence speed is low and the characteristics of each input multi-exposure image cannot be effectively reflected in the fusion layer.

In addition, a technique of fusing two HDR images having a difference in exposure time has been proposed as a technique for obtaining a multi-exposure fusion image effect from a single image in order to solve the problem of securing multiple images. This technology sets the short exposure conversion function and the long exposure conversion function as the luminance conversion function, uses the set luminance conversion function to convert the weight of each pixel of the image used for convergence to a value between 0 and 1, and then The final multi-exposure fusion image is generated through a weight normalization process, a weight mixing process based on a Gaussian function, and a weighted average luminance fusion process. However, this study has a problem in that uniform multi-exposure images are generated by determining, setting, and applying each of the short-exposure conversion function and the long-exposure conversion function as linear functions in generating images used for multiple exposures.

T. Mertens, J. Kautz, and F. V. Reeth, “Exposure Fusion,” Proceedings of 15th Pacific Conference on Computer Graphics and Applications, IEEE Computer Graphics and Applications, pp. 382-390, 2007 J. -H. Ryu, J.-W. Kim, J.-O. Kim, "Texture-enhanced multi-exposure fusion using texture decomposition and cascaded convolutional autoencoder," Proc. of ITC-CSCC 2019, Jeju, Korea, June. 2019. in Heon Kim, “An Image Merging Method for Two High Dynamic Range Images of Different Exposure,” Journal of Korea Multimedia Society Vol. 13, no. 4, p. 526-534, 2010

The present invention is to solve the problems of the existing technology as described above, and to generate and apply a high-quality image with the maximum amount of information for each area from a single image, to create a multiple exposure image based on the amount of image information, and each image includes The purpose of this study is to provide a method for converging images more effectively by generating and applying feature enhancement weights according to the amount of information.

In addition, in order to solve the above problems, the present invention generates multiple exposure images based on the amount of image information so that a high-quality image with the maximum amount of information for each area can be generated and applied from a single image, and It is an object of the present invention to provide an apparatus for more effectively converging images by generating and applying feature enhancement weights according to the present invention.

In order to achieve the above object, the present invention divides a single image into a plurality of regions to form a plurality of divided regions, and obtains an optimal gamma correction coefficient for each of the plurality of divided regions. An acquisition step, a gamma-corrected image derivation step of deriving a plurality of gamma-corrected images by applying the optimal gamma-correction coefficient to the entire area of the provided single image, and a feature enhancement generating a feature-enhancement weight ratio corresponding to the plurality of gamma-corrected images. A multi-exposure image fusing method from a single image is provided, comprising: generating a weight ratio; and fusing the plurality of gamma-corrected images by applying the feature enhancement weights to the plurality of gamma-corrected images.

According to an embodiment of the present invention, the obtaining of the optimal gamma correction coefficient for each of the plurality of divided regions may include two gamma correction coefficients when a difference value among the gamma correction coefficients derived for each region is less than or equal to a set threshold in the range of 0.1 or more and less than 1.0. A segmentation region filtering step of removing any one of the correction coefficients may be further included.

According to one embodiment of the present invention, the plurality of divided regions may be formed by overlapping each region.

According to one embodiment of the present invention, the plurality of divided areas may be formed in a size within a range of 30% to 75% of the total size of the provided single image.

According to an embodiment of the present invention, in the obtaining of the optimal gamma correction coefficient, a gamma correction coefficient obtained when the amount of information of the corrected image derived for each of the divided regions is maximum may be derived as the optimal gamma correction coefficient.

According to an embodiment of the present invention, the obtaining of the optimal gamma correction coefficient may include setting a search range of the gamma correction coefficient, setting an arbitrary gamma correction coefficient, and setting an increment for the arbitrary gamma correction coefficient. a divisional area gamma correction step of deriving a corrected image by performing gamma correction on each of the divided areas by applying the arbitrary gamma correction coefficient, and applying the increment to the arbitrary gamma correction coefficient. and a gamma correction coefficient changing step of changing the gamma correction coefficient by performing a gamma correction coefficient change, wherein the divided region gamma correction step and the gamma correction coefficient changing step are repeated within a search range of the gamma correction coefficient to obtain corrections for each divided region. A gamma correction coefficient when the amount of information of an image is maximum can be derived as the optimal gamma correction coefficient.

According to an embodiment of the present invention, the feature enhancement weight ratio generating step may include: calculating chromaticity weights corresponding to the plurality of gamma-corrected images; calculating texture weights corresponding to the plurality of gamma-corrected images; Calculating feature enhancement weights corresponding to the plurality of gamma-corrected images by summing the chromaticity weights and the texture weights corresponding to the plurality of gamma-corrected images, and the feature enhancement corresponding to the plurality of gamma-corrected images. The method may further include generating a feature enhancement weight ratio by generating a weight ratio.

An apparatus for fusing multiple exposure images from a single image according to an embodiment of the present invention includes: a division region forming unit for forming a plurality of division regions by dividing a provided single image into a plurality of regions; and performing optimal gamma correction for each of the plurality of division regions. An optimal gamma correction coefficient acquisition unit for acquiring a coefficient, a gamma correction image derivation unit for deriving a plurality of gamma correction images by applying the optimal gamma correction coefficient to the entire area of the provided single image, and features corresponding to the plurality of gamma correction images. A feature enhancement weight ratio generating unit generating an enhancement weight ratio, and an image fusion unit fusing the plurality of gamma-corrected images by applying the feature enhancement weights to the plurality of gamma-corrected images.

According to an embodiment of the present invention, a multi-exposure image fusing device from a single image

The optimal gamma correction coefficient obtaining unit may further include a segmented region filtering unit that removes one of the two gamma correction coefficients when a difference value among the gamma correction coefficients derived for each region is less than or equal to a set threshold value within a range of 0.1 or more and less than 1.0.

According to an embodiment of the present invention, the optimal gamma correction coefficient acquisition unit of the multi-exposure image fusing device from a single image calculates the optimal gamma correction coefficient when the amount of information of the corrected image derived for each divided area is maximum. can be derived as a coefficient.

According to an embodiment of the present invention, an optimal gamma correction coefficient acquisition unit of a multiple exposure image fusion device from a single image, a search range setting unit for setting a search range of a gamma correction coefficient, and a gamma correction coefficient for setting an arbitrary gamma correction coefficient A coefficient setting unit, an increment setting unit for setting an increment for the arbitrary gamma correction coefficient, and gamma correction for each of the divided regions by applying the arbitrary gamma correction coefficient to derive a corrected image. a correction unit and a gamma correction coefficient changing unit for changing the gamma correction coefficient by applying the increment to the arbitrary gamma correction coefficient, wherein the partition gamma correction step and the gamma correction coefficient within a search range of the gamma correction coefficient By repeating the change, a gamma correction coefficient when the amount of information of the corrected image derived for each divided area is maximum can be derived as the optimal gamma correction coefficient.

According to an embodiment of the present invention, a feature enhancement weight ratio generator of a multi-exposure image fusion device from a single image includes a chromaticity weight calculator that calculates chromaticity weights corresponding to the plurality of gamma-corrected images, and the plurality of gamma-corrected images. A texture weight calculation unit that calculates texture weights corresponding to corrected images, and feature enhancement weights corresponding to the plurality of gamma-corrected images are calculated by summing the chromaticity weights and the texture weights corresponding to the plurality of gamma-corrected images. and a feature enhancement weight calculator that generates a ratio of the feature enhancement weights corresponding to the plurality of gamma-corrected images.

According to one embodiment of the present invention, a division region forming step of forming a plurality of division regions by dividing a provided single image into a plurality of regions. An optimal gamma correction coefficient obtaining step of obtaining an optimal gamma correction coefficient for each of the plurality of divided regions; a gamma correction image derivation step of deriving a plurality of gamma corrected images by applying the optimal gamma correction coefficient to the entire region of the single image provided; a feature enhancement weight ratio generating step of generating a feature enhancement weight ratio corresponding to a plurality of gamma-corrected images; and an image fusion step of fusing the plurality of gamma-corrected images by applying the feature enhancement weights to the plurality of gamma-corrected images. A multiple exposure image fusion program from a single image stored on a recording medium is provided for execution.

According to the present invention, a multi-exposure image can be generated from a single image, and thus an expensive physical device for securing multiple-exposure images at the same time point is not required. In addition, a more precise high-quality image can be obtained by synthesizing multiple exposure images by applying a feature boosting weight to each image rather than simply synthesizing them.

1 is a flowchart of a method of obtaining a multiple exposure image from a single image according to an embodiment of the present invention.
2 is an exemplary view of a case where each divided area is set to a size of 25% of the entire provided image.
3 is an exemplary view of a case in which each divided area is set to a size of 40% of the entire provided image.
4 is an exemplary view showing a correction image according to a change in information amount according to a change in a unit gamma correction coefficient and a result of applying an optimal gamma correction coefficient.
5 is an exemplary diagram of the basic structure of a conventional feature fusion auto-encoder model (FFAM) in which two images are input.
6 shows an embodiment of a deep learning model for generating a multi-exposure fusion image according to the present invention.
7 illustrates a diagram for calculating feature enhancement weights based on texture and chromaticity information according to an embodiment of the present invention.
8 shows a process for calculating texture weights.
9 is an example of a process of calculating chromaticity weights according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the examples.

In one embodiment of the present invention, a single image is divided into a plurality of regions, a gamma correction coefficient is determined for each divided region when the amount of information is maximum, and a multi-exposure image is generated using the gamma correction coefficient, and the image for each region is included. Provides a method and apparatus for generating a high-quality exposure fusion image by generating and applying feature enhancement weights through deep learning based on the amount of information being processed.

Figure 1 discloses a flow chart for generating multiple multiple exposure images from a single image in accordance with the present invention.

Referring to FIG. 1, a method for generating a multiple exposure image from a single image according to the present invention divides a single image into multiple regions, sets an initial value of a gamma correction coefficient for each divided region, and performs gamma correction, By repeating the calculation of the amount of information in each area through the calculation of the gradient, a gamma correction coefficient is obtained when the amount of information in each area is maximized. Here, the gamma correction coefficient at the time when the maximum amount of information is obtained for each area is defined as an 'optimal gamma correction coefficient'. Here, since the optimal gamma correction coefficient is determined based on the point in time when the maximum amount of information is obtained for each region, the same number of gamma correction coefficients as the number of regions obtained by dividing a single image is obtained.

Indices used to calculate the amount of information for each area are not only gradients applied, but various indices such as corner points, chromaticity, and edge points that determine the characteristics and quality of an image can be applied.

When the optimal value of the gamma correction coefficient corresponding to each segmented area is obtained, after the segmentation filtering process of removing the corrected image having the same or within the closest range among each optimal gamma correction factor, each area of a single image is A plurality of multi-exposure images gamma-corrected are generated by applying a gamma correction coefficient.

Here, a method of dividing an area of a single image into one or more arbitrary areas based on a given image may be used. If a single image is divided into a plurality of areas, the size (width x height) of each area may be set to be the same. If the size of the segmented area is small compared to the size of the entire image, the size of the segmented area should be at least within the range of 30% to 75% of the entire image size, as the segmented area may cause serious distortion in the entire surrounding area. It is desirable to set them so that they overlap between regions.

Meanwhile, FIG. 2 is an example of a case where each divided area is set to a size of 25% of the entire provided image, and FIG. 3 is an example of a case where each divided area is set to a size of 40% of the entire provided image.

In general, an object of interest exists in the center of an image, which means that there is a lot of information in the center. At this time, when the size of the partitioned area is set to 25%, the amount of information in the center of each area is distributed to each divided area, and the amount of information distributed to each divided area tends to make a sharp difference in the optimal gamma correction coefficient for each area. Such rapid differences in gamma correction coefficients between regions may lead to heterogeneous results in generating a fusion image.

In order to minimize this effect, it is necessary to set a separate area in the center of the image. In this way, in order to minimize the occurrence of rapid gamma correction coefficient differences between regions, it is preferable that the divided regions are set to have overlapping regions, and the size of the divided regions may also be set such that overlapping regions exist between the regions. Since the difference in the optimal gamma correction coefficient between divided regions increases as the overlapping region decreases, it is preferable to set the size of the divided region to 40-50%.

After region division, an arbitrary gamma correction coefficient may be set and gamma correction of the provided image may be performed using the set gamma correction coefficient according to Equation (1) below.

In Equation (1), as the value of g decreases relative to 1, the brightness of the corrected image increases, and as the value of g increases from 1, the corrected image becomes darker.

In the present invention, the optimal gamma correction coefficient is searched to determine the optimal gamma correction coefficient when the maximum amount of information is extracted for each region. Increments can also be arbitrarily set.

However, if the size of the increment of the gamma correction coefficient is set too large, the difference due to gamma correction in each divided area becomes large, and the difference in the information amount of each corrected image increases, making it difficult to find an appropriate gamma correction coefficient. It is possible to find a more optimal gamma correction coefficient because the deviation of is reduced, but there is a problem in that the time for deriving the gamma correction coefficient increases as the number of iterations of gamma conversion increases. Therefore, in order to derive the optimal gamma correction coefficient and reduce the execution time, the size of the increment of the gamma correction coefficient can be set within the range of more than 0 and less than 1.0, more preferably within the range of 0.05-0.3. .

In addition, if the search range of the gamma correction coefficient is too wide, the execution time increases. Therefore, the search range of the gamma correction coefficient can be preferably set to 0.2 to 5.0.

When a gamma change correction coefficient for search is set, a corrected image is generated through Equation (1), and a gradient can be calculated from the corrected image.

If the image I(x,y) is viewed as a function representing the brightness of a pixel at (x,y), and the size and direction of the gradient are obtained using the following equation (2) at each pixel position, how much the pixel corresponds to the image It's easy to see if it's close to the edge of the contained object, and what the direction of the edge is.

Equation (2) relates to the change in brightness between adjacent pixels based on the position of the image pixel (x, y), and since there is a negative value, it is converted to an absolute value, and when the gradients of all pixels converted to absolute values are accumulated, The amount of information in the corresponding area can be calculated. That is, in general, gradients can be used to find the edges and directions of objects, colors, etc. included in an image. The amount of information in each image is how many objects are included in a single image. And/or it can be measured by whether there is a change in color, illuminance, etc. As a result, the value obtained by adding the absolute value of the above gradient, which is the amount of change thereof, can be interpreted as the amount of information included in one image.

Equation (2) is not the only method for calculating the amount of information, and Laplacian, Jacobian, Hessian, and the like may be used. In one embodiment of the present invention, the amount of information was calculated using a gradient, and the method for calculating the amount of information for each area using Equation (2) is as shown in Equation (3) below.

The amount of information according to the gamma correction coefficient for each area can be calculated using Equation (3), and the calculated information amount value is compared to obtain the gamma correction coefficient when the amount of information is maximum as each optimal gamma correction coefficient. can do. The gamma correction coefficient when the amount of information is maximum is It is determined using Equation (4).

4 shows that the optimal gamma correction coefficient search range is 0.1 to 8.0 and the increment value of the gamma coefficient is 0.1, using Equation (4) to determine the optimal gamma correction coefficient when the maximum amount of information is reached. It is an exemplary view showing the change in the amount of information according to the change in the correction coefficient and the corrected image that is the result of applying the optimal gamma correction coefficient of 2.0 when the amount of information is maximum.

As shown in FIG. 4, while changing the gamma correction coefficient, the sum of the absolute values of the gradients is compared to determine when the amount of information is maximum, and the gamma correction coefficient at that time can be derived as the optimal gamma correction coefficient for the divided area. there is.

On the other hand, if the optimal gamma correction coefficients corresponding to each of the n divided regions are determined using Equation (3), a filtering process for the divided regions may be performed to remove unnecessary images.

Segmentation filtering removes divisions with similar gamma correction coefficients using Equation (5) because there is no difference between corrected images when the optimal gamma correction coefficients determined for each region are the same or are located within the proximity range. Only high-quality multi-exposure images containing the maximum amount of information can be selected.

는 분할영역 필터를 위한 기준 임계값으로서 0.1 ~ 1.0의 범위 내에 설정되며 기본값으로는 감마보정계수 증분값으로 설정하는 것이 바람직하다. 또한 이러한 분할영역 필터를 위한 비교는 분할영역 수가 n일 때,

의 조합 수만큼 수행되어야 한다.In Equation (5) above

is set within the range of 0.1 to 1.0 as a reference threshold for the segmentation filter, and is preferably set to a gamma correction coefficient increment value as a default value. Also, the comparison for this partition filter is when the number of partitions is n,

should be performed as many as the number of combinations of

A multi-exposure image can be obtained by removing similar gamma correction coefficients from the entire region of the initially provided single image through segmentation filtering and performing gamma correction using each remaining optimum gamma correction coefficient.

Thus, a multiple-exposure fusion image to which feature enhancement weights according to the present invention are applied can be obtained from the multiple-exposure image obtained by applying the gamma correction coefficient derived from the divided regions of the single image.

The method for generating and applying feature enhancement weights according to the present invention does not simply synthesize multi-exposure images obtained by applying gamma correction coefficients derived from divided regions of a single image as described above, but feature enhancement weights (Feature boosting weights) for each image. Weight) can be applied and synthesized to obtain a more precise, high-quality image.

In the existing feature fusion auto-encoder model (FFAM, Feature Fusion Auto-encoder Model), the characteristics of input images are extracted as information amounts, and the extracted information amounts are corresponding to each image in a fusion layer (also called a fusion layer or bottleneck layer). By applying a feature enhancement weight to the result of the encoder, a feature amplification effect can be obtained in the learning process in proportion to the amount of feature extracted information for each image. Due to the feature amplification effect according to the application of such feature enhancement weights, a superior multi-exposure fusion image can be obtained compared to the application of the existing feature fusion auto-encoder model (FFAM).

5 is an exemplary view of an image fusion structure in which two images are input as a basic structure of a conventional feature fusion auto-encoder model (FFAM).

Looking at the structure of the conventional feature fusion auto-encoder model (FFAM) shown in FIG. 5, the decoder ( decoder). As a result, it can be interpreted that no weight is applied to an encoder corresponding to each image or that a weight of 1 is equally applied. However, such a conventional feature fusion auto-encoder model (FFAM) has a disadvantage in that an optimal fusion image cannot be obtained because it does not properly reflect the characteristics of an image.

However, according to an embodiment of the present invention as shown in FIG. 6 , each image is a feature fusion auto-encoding model (FFAM, Feature Fusion Auto-encoder Model) generated through the encoder (encoder) of the result of each input image. By calculating and applying reinforcement weights, and transferring the results to a decoder, fast convergence is achieved in the learning process by applying the characteristics of each image as weights, and a precise high-quality multi-exposure fusion image can be generated.

According to one embodiment of the present invention, feature enhancement weights corresponding to each image can be generated using the quality of each image or the amount of feature-extracted information. An input image may be a color image to derive a feature-enhancing weighted multi-exposure fusion image according to the present invention, but even in the case of a black-and-white image, it may be applied in the same way as a color image, except that the characteristics related to saturation are not considered. there is.

In one embodiment of the present invention, a method for deriving feature enhancement weights when an input image is a color image will be described. In color images, elements that can define the quality and/or characteristics of an image can be set to texture, colorfulness, and other various types, and various information that can quantitatively define the characteristics or quality. And the method can also be applied to the feature enhancement weight method of the present invention.

In the present invention, the method for calculating the feature enhancement weight is limited to texture and chromaticity, but is not limited to the method according to the information type of texture and chromaticity.

The texture includes the boundary of an object in the image and contrast information of the image, and the chromaticity includes saturation information of the image.

7 illustrates a diagram for calculating feature enhancement weights based on texture and chromaticity information according to an embodiment of the present invention.

The feature enhancement weight calculation process of FIG. 7 includes a weight calculation process for texture, a weight calculation process for chromaticity, and a process for calculating feature enhancement weights by integrating these weights. The weight calculation process may increase depending on the type of quantified information that determines the characteristics and quality of an image.

In one embodiment of the present invention, for one image, the texture is a gradient, and the chromaticity is calculated by calculating the standard deviation of the R (RED) / G (GREEN) / B (BLUE) channels constituting each pixel, and each weight map (weight map) is created, and a plurality of generated weight maps are synthesized to calculate the weight value of the corresponding image.

The process of calculating the weight corresponding to the texture may include calculating a texture weight map by converting the image into a gray scale image, and calculating texture weights from the texture weight map.

In the texture weight map calculation step, a color image is converted into a black-and-white image, and a gradient for each pixel is calculated and generated from the converted black-and-white image. Various methods may be applied to the gradient applied at this time, as described above in the multi-exposure image correction coefficient search method, and in one embodiment of the present invention, a texture weight map is calculated using Laplacian.

)를 적용하여 질감 가중치 값을 스칼라 값으로 산출한다. 8 shows a process for calculating texture weights. The texture weight calculation step is L1-norm (

) is applied to calculate the texture weight value as a scalar value.

9 illustrates a process of calculating chromaticity weights according to an embodiment of the present invention. The process of calculating a weight corresponding to chromaticity according to an embodiment of the present invention may include obtaining a chromaticity weight map for an input color image and calculating chromaticity weights using the chromaticity weight map. .

In the chromaticity weight calculation step, the standard deviation of the pixel values of the R (RED) / G (GREEN) / B (BLUE) channels located in each pixel of the input color image is calculated as a chromaticity weight map, and the calculated chromaticity It can be calculated as a scalar value by applying L1_norm to the weight map.

In addition to the above method, various methods using the same and/or other factors may be applied to the method for calculating the chromaticity weight. can When using the Hasler and Susstrunk model, chromaticity can be calculated using Equation (6).

The feature enhancement weight corresponds to the Fused Norm of FIG. 7 . As shown in FIG. 7 , the sum of texture weights and chromaticity weights calculated one by one for each input multi-exposure image can be calculated using Equation (7).

는 입력되는 이미지에 대하여 0~1사이의 값이 될 수 있도록 수학식(8)을 이용하여 가중치 비율로 계산할 수 있다.The weight calculated by Equation (7) is calculated one by one corresponding to each input image, and the calculated weight is substantially the total amount of information of one image and becomes a scalar value. Feature enhancement weight for each image calculated in this way

can be calculated as a weight ratio using Equation (8) so that it can be a value between 0 and 1 for an input image.

)을 가중평균값으로 전환하여 특징강화 가중치를 산출 및 적용할 수 있다.Equation (8) is a weight ratio for each image, and means that an image having a high weight ratio includes a larger amount of feature extraction information. However, in applying the calculated weight ratio to the fusion layer of the Feature Fusion Auto-encoder Model (FFAM), the node value of the encoder has a value in the range of 0 to 1, Since the weight ratio also has a value between 0 and 1, when the weight ratio is applied, the amplification effect may be reduced due to the problem that the value of the encoder node converges to a value of 0. Therefore, using Equation (9), the weight ratio (

) into a weighted average value, the feature enhancement weight can be calculated and applied.

)가 높은 이미지에 대해서 높은 기댓값을 적용하며, 인코더(encoder)의 노드와의 곱으로 0으로 급격히 수렴하는 것을 방지함으로서 학습과정에서 특징 추출된 정보량의 가중치에 따른 증폭효과를 기대할 수 있다.Since the feature enhancement weight value calculated using Equation (9) above is an expectation value, the weight (

) is applied to a high expected value, and by preventing rapid convergence to 0 by multiplication with the node of the encoder, an amplification effect according to the weight of the amount of information extracted during the learning process can be expected.

The application of the feature enhancement weights of the present invention is not applied only to the deep learning model. Even in the traditional multi-exposure fusion technique, feature enhancement weights are applied according to the characteristics of each image, and thus have the effect of generating a high-quality exposure fusion image.

As described above, the present invention has been described by the limited embodiments and drawings, but those skilled in the art can make various modifications and variations from the above description.

Therefore, other implementations, other embodiments and equivalents of the claims are also within the scope of the claims of the present invention.

Claims (15)

a division region forming step of forming a plurality of division regions by dividing the provided single image into a plurality of regions;
an optimal gamma correction coefficient obtaining step of obtaining an optimal gamma correction coefficient for each of the plurality of divided regions;
a gamma correction image derivation step of deriving a plurality of gamma correction images by applying the optimal gamma correction coefficient to the entire area of the provided single image;
a feature enhancement weight ratio generating step of generating feature enhancement weight ratios corresponding to the plurality of gamma-corrected images; and
an image fusion step of fusing the plurality of gamma-corrected images by applying the feature enhancement weight to the plurality of gamma-corrected images;
containing
Multiple exposure image fusion method from a single image. The method of claim 1 , wherein the obtaining of an optimal gamma correction coefficient for each of the plurality of divided regions comprises:
segmentation filtering step of removing one of the two gamma correction coefficients when the difference value among the gamma correction coefficients derived for each region is less than or equal to a set threshold in the range of 0.1 or more and less than 1.0;
Multiple exposure image fusion method from a single image, characterized in that it further comprises. The method of claim 1 , wherein the plurality of divided regions are formed by overlapping each region. [Claim 4] The method of claim 1 or 3, wherein the plurality of divided regions are formed in a size within a range of 30% to 75% of the total size of the provided single image. The method according to claim 1, wherein the obtaining of the optimal gamma correction coefficient derives a gamma correction coefficient when the amount of information of the corrected image derived for each of the divided regions is maximum as the optimal gamma correction coefficient. Multiple exposure image fusion method. The method of claim 1 or 5, wherein the obtaining of the optimal gamma correction coefficient comprises:
setting a search range for gamma correction coefficients;
Setting a random gamma correction coefficient;
an increment setting step of setting an increment for the arbitrary gamma correction coefficient;
a gamma correction step of deriving a corrected image by performing gamma correction on each of the divided areas by applying the arbitrary gamma correction coefficient; and
a gamma correction coefficient changing step of changing the gamma correction coefficient by applying the increment to the arbitrary gamma correction coefficient;
including,
The optimal gamma correction coefficient is obtained by repeating the division gamma correction step and the gamma correction coefficient change step within the search range of the gamma correction coefficient when the amount of information of the corrected image derived for each division is maximum. characterized by deriving as a coefficient
Multiple exposure image fusion method from a single image. The method according to claim 1, wherein the feature enhancement weight ratio generating step
calculating chromaticity weights corresponding to the plurality of gamma-corrected images;
calculating texture weights corresponding to the plurality of gamma-corrected images;
calculating feature enhancement weights corresponding to the plurality of gamma-corrected images by summing the chromaticity weights and the texture weights corresponding to the plurality of gamma-corrected images; and
generating a feature enhancement weight ratio by generating a feature enhancement weight ratio corresponding to the plurality of gamma-corrected images;
Multiple exposure image fusion method from a single image, characterized in that it further comprises. a division region forming unit dividing a single image into a plurality of regions to form a plurality of division regions;
an optimal gamma correction coefficient obtaining unit acquiring an optimal gamma correction coefficient for each of the plurality of divided regions;
a gamma correction image derivation unit for deriving a plurality of gamma correction images by applying the optimal gamma correction coefficient to the entire area of the provided single image;
a feature enhancement weight ratio generating unit generating feature enhancement weight ratios corresponding to the plurality of gamma-corrected images; and
an image fusion unit fusing the plurality of gamma-corrected images by applying the feature enhancement weight to the plurality of gamma-corrected images;
containing
Multi-exposure image fusing device from a single image. The method of claim 8, wherein the optimum gamma correction coefficient obtaining unit
a segmented region filtering unit for removing one of the two gamma correction coefficients when a difference value among the gamma correction coefficients derived for each region is less than or equal to a set threshold value in the range of 0.1 or more and less than 1.0;
Multiple exposure image fusion device from a single image, characterized in that it further comprises. 9. The apparatus of claim 8, wherein the plurality of divided regions are formed by overlapping each region. [Claim 11] The apparatus of claim 8 or 10, wherein the plurality of divided regions are formed in a size within a range of 30% to 75% of the total size of the provided single image. 9. The method according to claim 8 , wherein the optimal gamma correction coefficient acquisition unit derives a gamma correction coefficient when the amount of information of the corrected image derived for each divided area is maximum as the optimal gamma correction coefficient. Exposure image fusion device. The method according to claim 8 or 12, wherein the optimal gamma correction coefficient obtaining unit
a search range setting unit for setting a search range of gamma correction coefficients;
a gamma correction coefficient setting unit for setting an arbitrary gamma correction coefficient;
an increment setting unit configured to set an increment for the arbitrary gamma correction coefficient;
a segmented area gamma correction unit for deriving a corrected image by performing gamma correction on each of the divided areas by applying the arbitrary gamma correction coefficient; and
a gamma correction coefficient changing unit for changing the gamma correction coefficient by applying the increment to the arbitrary gamma correction coefficient;
including,
The gamma correction step and the change of the gamma correction coefficient are repeated within the search range of the gamma correction coefficient, and the gamma correction coefficient when the information amount of the corrected image derived for each divided region is maximum is determined as the optimal gamma correction coefficient. characterized in that derived by
Multi-exposure image fusing device from a single image. The method according to claim 8, wherein the feature enhancement weight ratio generator
a chromaticity weight calculator to calculate chromaticity weights corresponding to the plurality of gamma-corrected images;
a texture weight calculator to calculate texture weights corresponding to the plurality of gamma-corrected images; and
a feature enhancement weight calculation unit calculating a feature enhancement weight corresponding to the plurality of gamma-corrected images by summing the chromaticity weights and the texture weights corresponding to the plurality of gamma-corrected images;
The multi-exposure image fusing device from a single image, further comprising generating ratios of the feature enhancement weights corresponding to the plurality of gamma-corrected images. a division region forming step of forming a plurality of division regions by dividing the provided single image into a plurality of regions;
an optimal gamma correction coefficient obtaining step of obtaining an optimal gamma correction coefficient for each of the plurality of divided regions;
a gamma correction image derivation step of deriving a plurality of gamma correction images by applying the optimal gamma correction coefficient to the entire area of the provided single image;
a feature enhancement weight ratio generating step of generating feature enhancement weight ratios corresponding to the plurality of gamma-corrected images; and
An image fusion step of fusing the plurality of gamma-corrected images by applying the feature enhancement weight to the plurality of gamma-corrected images; a multi-exposure image fusion program from single images stored in a recording medium.

Images