KR102020464B1 - Color-mono Dual Camera Image Fusion Method, System and Computer-readable Medium - Google Patents

Abstract

The present invention relates to a color-mono dual camera imaging image fusion method, and a system and a computer-readable medium thereof, and more specifically, to a color-mono dual camera imaging image fusion method which fuses a color image and a mono image photographed through two separated cameras so as to increase an image quality of low level light imaging, and to a system and a computer-readable medium thereof.

Description

Color-mono Dual Camera Image Fusion Method, System and Computer-readable Medium}

The present invention relates to a method, a system and a computer-readable medium for fusion of color-mono dual camera photographed images, and more particularly, to image quality of low light photographing by fusion of color images and mono images photographed through two separate cameras. A method, system, and computer-readable medium for fusion of color-mono dual-camera photographed images that can improve image quality.

Until recently, the market for complementary metal-oxide-semiconductor (CMOS) image sensors (CIS) for smartphone cameras has been competing to make small sensors while supporting high resolution images. Recently, sensors that have been reduced to 1.0 μm pixel size are on the market. As the size of the pixel becomes smaller and the height of the sensor can be reduced, the size of the camera portion can be reduced in devices such as smartphones, making it possible to manufacture products with an elegant design.

However, due to the reduced size of the pixel has a low dynamic range (Low Dynamic Range), due to the trade-offs that increase the noise, congestion occurs in the pixel size competition.

On the other hand, in the smart phone market, users are demanding to obtain high-quality photos of DSLR (Digital Single-Lens Reflex) cameras from small size smartphone cameras. In particular, there is a great demand from users for improving image quality in low-light photography, acquiring high-resolution images such as optical zoom, and high-quality photographs with reduced motion blur.

Among these requirements, it is necessary to solve the problem that the small size sensor causes excessive noise in low light shooting and deterioration in sharpness due to malfunction of the auto focus function due to the dark environment. have.

Recently, a method of using a dual lens camera (that is, a dual camera in which two image sensors and a lens are arranged side by side) has been proposed to improve the image quality of a low-light image, which is limited to one color CIS sensor. However, there is a difficulty in the process of fusing the two captured images. This is because parallax compensation is inevitable because there is pixel disparity between two captured stereo images, but in this process, there is an absence of an algorithm that prevents deterioration of the quality of the fused image due to incorrect disparity estimation. Therefore, there is a need to develop an algorithm that compensates for such drawbacks.

The present invention provides a method, system and computer-readable medium for fusion of color-mono dual-camera photographed images that can improve the image quality of low-light imaging by fusing color images and mono images captured by two separate cameras. For that purpose.

In order to solve the above problems, the present invention provides a fusion method of a color-mono dual camera photographed image, comprising: an image input step of receiving a mono image and a color image photographed through a color-mono dual camera; A preprocessing step of performing preprocessing on the input mono image and the color image; A parallax estimating step of estimating a parallax from the pre-processed color image and the pre-processed mono image; A joint denoising step of removing noise of the pre-processed color image based on the pre-processed mono image to generate a denoised color image; And a color information transfer step of transferring color information from the denoised color image to the mono image which has been preprocessed to generate an improved color image having improved image quality. It provides a, fusion method of the color-mono dual camera photographing image.

In the present invention, the color information transferring step, based on the degree of difference for each partial region of the mono image that has been pre-processed for the color image that has been pre-processed, the color information of the partial region of the mono image has been pre-processed Disparity compensation color information generation step of generating disparity compensation color information for each partial region by inputting color information of the partial region of the denoised color image corresponding to the estimated parallax; Color hole filling that fills color information derived based on the disparity compensation color information of another partial area into which one or more color information is input, in a color hole in which the color information is not input among the disparity compensation color information for each partial area. step; And an image merging step of generating an improved color image by merging the disparity compensation color information for each partial region filled with the color information through the color hole filling step into the mono image which has been preprocessed. It may include.

In the present invention, the step of generating the disparity compensation color information, wherein the difference between the partial region of the same coordinates of the denoised color image and the mono image is less than the BJND value of the BJND model representing the difference between the binocular image, the partial A masking generation step of generating masking in the area; And a disparity compensation color information transferring step of generating disparity compensation color information by inputting color information of the denoised color image in consideration of the disparity estimated for the masked area; It may include.

In the present invention, the color hole filling step may derive color information to be filled in the color hole based on an artificial neural network algorithm.

In the present invention, the color hole filling step may derive color information to be filled in the color hole based on the convolution neural network.

In the present invention, the convolutional neural network may be configured as a cross-channel auto encoder that uses the color difference information as the color difference.

In the present invention, the pre-processing step, the geometric correction step of aligning the mono image and the color image through a linear transformation of the input mono image and the color image; A normalization step of generating a pre-processed color image by adjusting the brightness of the color image in which the geometric correction step is performed according to the brightness of the mono image; And generating a grayscale image by removing color information of the preprocessed color image. It may include.

In the present invention, the parallax estimating step estimates the parallax of the grayscale image and the mono image, and the joint denoising step removes the noise of the preprocessed color image based on the mono image which has been preprocessed. Decal images can be generated.

In the present invention, the joint de-noising step removes noise based on the mono image and the pre-processed color image when the parallax estimated in the parallax estimation step is less than a preset denoise reference value, and is estimated in the parallax estimating step. When the parallax is greater than or equal to a preset denoise reference value, the noise may be removed based on the preprocessed color image.

In the present invention, the denoise reference value may be determined based on a BJND model representing a difference discrimination between binocular images.

In order to solve the above problems, the present invention is a fusion system of a color-mono dual camera photographed image, the image input unit for receiving a mono image and a color image photographed through a color-mono dual camera; A preprocessing unit performing preprocessing on the input mono image and the color image; A parallax estimator for estimating a parallax from the color image pre-processed and the mono image pre-processed; A joint denoising unit for generating a denoised color image by removing noise of the pre-processed color image based on the mono-image which has been preprocessed; And a color information transfer unit configured to transfer color information from the denoised color image to the mono image which has been preprocessed to generate an improved color image having improved image quality. It provides a, fusion system of color-mono dual camera photographing image.

In the present invention, the color information transfer unit is based on the color information of the partial region of the mono-image subjected to the pre-processing, based on the degree of difference for each partial region of the mono-image that has been pre-processed for the color image subjected to pre-processing A parallax compensation color information generation unit for generating parallax compensation color information for each subregion by inputting color information of a partial region of the denoised color image in consideration of the estimated parallax; Color hole filling that fills color information derived based on the disparity compensation color information of another partial area into which one or more color information is input, in a color hole in which the color information is not input among the disparity compensation color information for each partial area. part; And an image merger which merges the disparity compensation color information for each partial region filled with the color information received from the color hole filling unit into the mono image that has been preprocessed to generate an improved color image. It may include.

In the present invention, the disparity compensation color information generating unit is the partial region when the difference in the partial region of the same coordinates of the denoised color image and the mono image is less than the BJND value of the BJND model indicating the difference between the binocular image Masking generation unit for generating a masking on; And a parallax compensation color information transmitting unit configured to generate parallax compensation color information by inputting color information of the denoised color image in consideration of the parallax estimated for the masked area; It may include.

In the present invention, the color hole filling unit may derive color information to be filled in the color hole based on an artificial neural network algorithm.

In the present invention, the color hole filling unit may derive color information to be filled in the color hole based on the convolutional neural network.

In the present invention, the convolutional neural network may be configured as a cross-channel auto encoder that uses the color difference information as the color difference.

In the present invention, the pre-processing unit, the geometric correction unit for aligning the mono image and the color image through a linear transformation of the input mono image and the color image; A normalizer for generating a pre-processed color image by adjusting the brightness of the color image received from the geometric correction unit according to the brightness of the mono image; And a monochrome unit which generates a grayscale image by removing color information of the preprocessed color image. It may include.

In the present invention, the parallax estimator estimates the parallax of the grayscale image and the mono image, and the joint denoising unit removes noise of the preprocessed color image based on the mono image which has been preprocessed to de-noise the color. An image can be generated.

In the present invention, the joint denoising unit removes noise based on the mono image and the pre-processed color image when the parallax estimated by the parallax estimating unit is less than a preset denoise reference value, and the parallax estimated by the parallax estimating unit. If is equal to or greater than a preset denoise reference value, the noise may be removed based on the preprocessed color image.

In the present invention, the denoise reference value may be determined based on a BJND model representing a difference discrimination between binocular images.

In order to solve the above problems, the present invention is a computer-readable medium for realizing the fusion of color-mono dual camera photographed image, the computer-readable medium includes at least one processor and at least one memory Storing instructions for causing a computing device to perform the following steps, the steps comprising: an image input step of receiving a mono image and a color image captured by a color-mono dual camera; A preprocessing step of performing preprocessing on the input mono image and the color image; A parallax estimating step of estimating a parallax from the color image and the mono image subjected to preprocessing; A joint denizing step of generating a denoised color image by removing noise of the preprocessed color image based on the preprocessed color image and the mono image; And a color information transferring step of generating an improved color image having improved image quality by transferring color information from the de-noise color image to the mono image. It provides a computer-readable medium comprising a.

In order to solve the above problems, the present invention provides a fusion method of a color-mono dual camera photographed image, comprising: an image input step of receiving a mono image and a color image photographed through a color-mono dual camera; A preprocessing step of performing preprocessing on the input mono image and the color image; A parallax estimating step of estimating a parallax from the pre-processed color image and the pre-processed mono image; A color information transferring step of transferring color information from the color image subjected to preprocessing to the mono image subjected to preprocessing to generate an improved color image with improved image quality; It provides a, fusion method of the color-mono dual camera photographing image.

In the present invention, the color information transferring step, based on the degree of difference for each partial region of the mono image that has been pre-processed for the color image that has been pre-processed, the color information of the partial region of the mono image has been pre-processed Disparity compensation color information generation step of generating disparity compensation color information for each partial region by inputting color information of the partial region of the color image which has undergone a corresponding preprocessing in consideration of the estimated parallax; Color hole filling that fills color information derived based on the disparity compensation color information of another partial area into which one or more color information is input, in a color hole in which the color information is not input among the disparity compensation color information for each partial area. step; And an image merging step of generating an improved color image by merging the disparity compensation color information for each partial region filled with the color information through the color hole filling step into the mono image which has been preprocessed. It may include.

According to an embodiment of the present invention, it is possible to achieve an effect of obtaining an image having excellent image quality by fusing a mono image obtained through a mono sensor and a color image obtained through a color sensor.

According to an embodiment of the present invention, the color information is transferred from a color image to a mono image having a higher signal-to-noise ratio (SNR) and image detail information in a low-light shooting environment, thereby obtaining an image having excellent image quality. Can be exercised.

According to an embodiment of the present invention, by estimating the parallax, and joint denoising based on the estimated reliability of the parallax, it is possible to exert an excellent denoising effect.

According to an embodiment of the present invention, the masking is generated based on the difference between the color image and the mono image, and the color information is transmitted based on the masking, thereby improving the accuracy of the color information transmission.

According to one embodiment of the present invention, after receiving color information based on masking, color hole filling through deep learning can be performed, thereby minimizing color bleeding and improving color reproducibility.

1 is a diagram schematically illustrating a fusion system of color-mono dual camera captured images according to an embodiment of the present invention.
2 is a flowchart schematically illustrating steps of a fusion method of a color-mono dual camera captured image according to an exemplary embodiment of the present invention.
3 is a flowchart schematically showing the detailed steps of the preprocessing step according to an embodiment of the present invention.
4 is a flowchart schematically showing the detailed steps of the color information transfer step according to an embodiment of the present invention.
5 is a diagram schematically illustrating a structure of a convolutional neural network used in a color hole peeling step according to an embodiment of the present invention.
6 is a flowchart schematically showing the detailed steps of the parallax compensation color information generation step according to an embodiment of the present invention.
7 is a flowchart schematically illustrating a process of processing a color image and a mono image according to an embodiment of the present invention.
8 is a block diagram schematically illustrating an internal configuration of a fusion device of a color-mono dual camera captured image according to an embodiment of the present invention.
9 is a block diagram schematically illustrating an internal configuration of a preprocessor according to an embodiment of the present invention.
10 is a block diagram schematically showing an internal configuration of a color information transmitting unit according to an embodiment of the present invention.
11 is a block diagram illustrating an example of an internal configuration of a computing device according to an embodiment of the present disclosure.

In the following, various embodiments and / or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will also be appreciated by one of ordinary skill in the art that this aspect (s) may be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative aspects of the one or more aspects. However, these aspects are exemplary and some of the various methods in the principles of the various aspects may be used and the descriptions described are intended to include all such aspects and their equivalents.

Moreover, various aspects and features will be presented by a system that may include a number of devices, components, and / or modules, and the like. The various systems may include additional devices, components, and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in connection with the drawings. It must be understood and recognized.

As used herein, “an embodiment”, “an example”, “aspect”, “an example”, etc., may not be construed as having any aspect or design described being better or advantageous than other aspects or designs. . The terms '~ part', 'component', 'module', 'system', 'interface', etc. used generally mean a computer-related entity, for example, hardware, hardware And a combination of software and software.

In addition, the terms "comprises" and / or "comprising" mean that such features and / or components are present, but exclude the presence or addition of one or more other features, components, and / or groups thereof. It should be understood that it does not.

In addition, terms including ordinal numbers such as first and second may be used to describe various components, but the components are not limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein including technical or scientific terms are generally understood by those skilled in the art to which the present invention belongs. Has the same meaning as Terms such as those defined in the commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and ideally or excessively formal meanings, unless explicitly defined in the embodiments of the present invention. Not interpreted as

1 is a diagram schematically illustrating a fusion system of color-mono dual camera captured images according to an embodiment of the present invention.

Referring to FIG. 1, a fusion system of a color-mono dual camera photographing image according to an exemplary embodiment of the present invention includes a mono-CIS (CMOS image sensor) photographing a photographing target 300 through two lenses 110 and 210, respectively. And a 100) and a color CIS (CMOS image sensor, 200), and the image fusion device 1000 to improve the image quality by fusing the mono image and the color image taken by the mono CIS 100 and the color CIS 200 It can be configured as.

The mono CIS 100 without the color filter array acquires a mono image without color information, but generally has higher light efficiency than the color CIS 200 in which photons are lost while passing through the color filter array. It is possible to acquire images with higher sharpness and signal-to-noise ratio (SNR) and a lot of detail information. As described above, the mono CIS 100 and the color CIS 200 complement each other, and are photographed using the mono CIS 100 having excellent image quality in a low light shooting environment and the color CIS 200 capable of acquiring color information. By fusing a stereo image into one, it is possible to obtain an image having excellent image quality even in a low-light shooting environment.

In the present invention, the image quality is improved by introducing an image fusion algorithm that transfers information between the photographed mono-color images into one resultant image.

2 is a flowchart schematically illustrating steps of a fusion method of a color-mono dual camera captured image according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the fusion method of the color-mono dual camera photographed image according to the exemplary embodiment of the present invention is performed by the following steps.

First, the fusion method of the color-mono dual camera photographed image according to an embodiment of the present invention performs an image input step S100 of receiving a mono image and a color image photographed through the color-mono dual camera. In the image input step S100, the mono image captured by the mono CIS 100 as shown in FIG. 1 and the color image captured by the color CIS 200 are received.

Thereafter, a preprocessing step (S200) of performing preprocessing on the input mono image and the color image is performed. In the preprocessing step (S200), preprocessing necessary to perform fusion on the mono image and the color image is performed. The preprocessing may perform all preprocessing necessary for subsequent processing steps such as color space conversion, geometric correction, normalization, and monochrome. By normalizing the mono image and the color image received through such a pre-processing, the mono image and the color image received are converted into a form suitable for fusion processing.

In an embodiment of the present invention, the mono image and the color image received in the preprocessing step S200 may be converted into a lab color space. The Lab color space is divided into luminance (L) information and color (ab) information and stored, thereby facilitating matching and color information transfer of the mono image and the color image.

Thereafter, a parallax estimating step (S300) of estimating parallaxes from the pre-processed color image and the pre-processed mono image is performed. The parallax estimating step (S300) estimates parallax by comparing the color image and the mono image. In the process of deriving parallax from such stereoscopic images, there is a high possibility of error, and if such errors occur at pixels near the boundary of the image object, serious structural distortion may occur in the image and the overall cognitive quality of the image may be degraded. The derived parallax only serves as an estimate.

In an embodiment of the present invention, the disparity is obtained by dividing the mono image and the color image into partial regions and creating a disparity map by estimating and storing the disparity of the corresponding partial region for each coordinate of the partial region. It can be estimated. Preferably, the partial region may consist of one pixel.

At this time, as a method for estimating the parallax between the mono image and the color image, various known algorithms may be used.

Thereafter, a joint denoising step (S400) of generating a denoised color image by removing noise of the pre-processed color image based on the pre-processed mono image is performed. However, in another embodiment of the present invention, such a joint denizing step (S400) may be omitted.

In an embodiment of the present invention, the joint denizing step (S400) performs denoising with reference to the mono image as reference information to remove noise of the pre-processed color image. Since the mono image is an image obtained through the mono CIS 100 and has a higher SNR and a lot of detail information, noise of the color image that has been preprocessed by performing denoising using the mono image as reference information is obtained. Can be removed effectively.

However, although the mono image has a higher SNR and a lot of detail information, parallax exists because the image is photographed at a position different from that of the color image. Therefore, it is necessary to remove the noise in consideration of the parallax.

In this case, in the joint denoising step S400, when the parallax estimated in the parallax estimating step S300 is less than a preset denoise reference value, noise is removed based on the mono image and the pre-processed color image. When the parallax estimated in the estimating step is equal to or greater than a preset denoise reference value, noise may be removed based on the preprocessed color image.

As such, when the estimated disparity between the preprocessed color image and the mono image is equal to or greater than a predetermined value, it is determined that the information of the mono image is not suitable for removing noise of the preprocessed color image, and thus the information of the preprocessed color image itself. If only the noise is removed and the estimated parallax is less than a predetermined value, it is determined that the noise is suitable to remove the noise of the pre-processed color image by referring to the information of the mono-image. By removing the noise of the image, it is possible to minimize the influence of the occurrence of error in the parallax estimation while performing denoising considering the estimated parallax.

In this case, the denoise reference value may be determined based on a BJND model representing the difference between the binocular images. The BJND (Binocular Just Noticeable Difference) model represents the minimum discrepancy that can be recognized by both eyes of the human being according to luminance, and the discrepancy discrepancy that can be recognized by a person in the BJND model may be used as a denoise reference value. . As described above, when the estimated difference is less than the discernible difference, it is possible to perform the denoising using the mono image as reference information. If the difference is more than a human perceptible identification, the noise is removed only by the information of the preprocessed color image itself.

Thereafter, the color information transfer step (S500) is performed to transfer the color information from the de-noise color image to the mono image which has been preprocessed to generate an improved color image with improved image quality.

In the color information transferring step (S500), the color information of the denoised color image is transferred to the color information of the mono image to generate an improved color image with improved image quality. In this case, since the mono image and the denoised color image have parallaxes, it is necessary to compensate the parallax to transmit color information.

In this case, the estimated parallax may have a large number of errors, and distortion of an image may occur due to the errors. Therefore, in an embodiment of the present invention, the color information is not transmitted directly to reflect the parallax, the color information is transmitted only in an area of low probability of parallax, and the previously transmitted color of an area of high probability of error. The color information transmitting step S500 may be performed by deriving color information based on the information.

More specifically, in an embodiment of the present invention, the degree of difference between the corresponding partial region of the mono image and the color image is derived, the masking is generated for an area whose difference is less than a predetermined reference, and the generated masking A color fill for transmitting color information of the denoised color image based on the estimated parallax for an area, and deriving color information based on the color information transmitted for the color hole in which the color information is not transmitted; Color information generated by performing may be transferred to the mono image.

3 is a flowchart schematically showing the detailed steps of the preprocessing step (S200) according to an embodiment of the present invention.

Referring to FIG. 3, the preprocessing step (S200) according to an embodiment of the present invention includes a geometric correction step (S210), a normalization step (S220), and a monochrome step (S230).

The geometric correction step (S210) aligns the mono image and the color image through a linear transformation of the input mono image and the color image. Since the mono image and the color image are images obtained through separate lenses and CIS, angles and magnifications of the captured images may not exactly match. The geometric correction step (S210) is performed by matching a partial region of the color image corresponding to the mono image partial region so that the matched partial regions may be located at the same coordinate of each image. This is done by converting the above. In more detail, the geometric correction step S210 may be performed by deriving an epipolar line from the mono image and the color image to match the epipolar line.

In the geometric correction step (S210), both the mono image and the color image may be converted to be aligned, or the color image may be converted to be aligned with the mono image to perform alignment.

The normalization step (S220) generates a pre-processed color image by adjusting the brightness of the color image in which the geometric correction step (S210) is performed according to the brightness of the mono image. In the normalization step (S220), the brightness difference between the two images is adjusted to increase the accuracy in estimating the parallax for the matched partial region of the two images. The normalization step S220 may be performed by image histogram matching or least squares method.

The monochrome step (S230) generates a grayscale image by removing color information of the preprocessed color image. As such, by generating a grayscale image from which color information of a color image is removed, parallax may be more accurately estimated through matching with the mono image.

In performing the preprocessing step (S200), the mono image and the color image are converted into a lab color space and performed to simplify the normalization step (S220) and the monochrome step (S230).

In the exemplary embodiment of performing the preprocessing step (S200), the parallax estimating step (S300) estimates the parallax of the grayscale image and the mono image, and the joint denoising step (S400) includes the preprocessing color image and The noise of the preprocessed color image may be removed based on the mono image.

In the parallax estimating step (S300), the parallaxes of the grayscale image and the mono image may be estimated. Thus, by comparing the grayscale image from which the color information has been removed by preprocessing the color image with the mono image, the parallax is estimated, thereby making it possible to estimate the parallax more accurately.

In addition, in the joint denizing step (S400), a noise of the preprocessed color image is removed based on the disparity estimated in the mono image and the parallax estimating step (S300) to generate a denoised color image including color information. can do.

4 is a flowchart schematically showing the detailed steps of the color information transfer step (S500) according to an embodiment of the present invention.

Referring to FIG. 4, the color information transfer step S500 may include a parallax compensation color information generation step S510, a color hole peeling step S520, and an image merge step S530. .

The disparity compensation color information generation step (S510) is based on the degree of difference for each partial region of the mono image which has been preprocessed for the color image which has been preprocessed, and the color information of the partial region of the mono image which has been preprocessed. Disparity compensation color information for each subregion is generated by inputting color information of a partial region of the denoised color image corresponding to the estimated parallax.

More specifically, in an embodiment of the present invention, the degree of difference between the corresponding partial region of the denoised color image and the mono image is derived, and the masking is generated and generated for an area whose difference is less than a predetermined criterion. Color information of the denoised color image may be transmitted based on the disparity estimated for the masked area.

The color hole peeling step (S520) is derived based on the disparity compensation color information of another partial region in which one or more color information is input to a color hole in which color information is not input among the disparity compensation color information of each partial region. Fill in the color information. The disparity compensation color information generated in the disparity compensation color information generation step (S510) is input color information of the denoised color image considering the parallax, but color information is not input to all areas, There is a color hole that is not input. There are many known methods for filling color information for such color holes. For example, there are colorization techniques based on the optimization by the least-squares method. However, in this method, color bleeding distortion may occur around the boundary of an object.

In order to solve this problem, in the embodiment of the present invention, the color hole filling step S520 may derive color information to be filled in the color hole based on an artificial neural network algorithm. In this case, the artificial neural network may be a convolutional neural network. As described above, the deep learning method based on the convolutional neural network can suppress the color bleeding caused by deriving and filling the color information into the color hole, and enhance the color reproducibility by filling the accurate color information. The convolutional neural network will be described later with reference to FIG. 5.

The image merging step S530 merges the disparity compensation color information for each partial region filled with the color information through the color hole filling step S520 into the mono image which has undergone preprocessing to generate an improved color image.

In the image merging step (S530), color information is added to brightness information of the mono image by adding the disparity compensation color information generated through the color hole peeling step (S520) to the color information of the mono image. An image can be generated. In an embodiment of the present invention, the color information is an ab color value in a Lab color space, and the ab color value is merged with an L luminance value stored in the mono image to generate an improved color image having Lab color information. .

5 is a diagram schematically illustrating a structure of a convolutional neural network used in a color hole peeling step according to an embodiment of the present invention.

Referring to FIG. 5, the mono image is one channel data including only brightness information, and the parallax compensation color information is two channel data including only color information. The mono image and the parallax compensation color information may be input to a convolutional neural network, respectively.

In one embodiment of the present invention, the convolutional neural network may be configured as a cross-channel auto encoder that uses the color difference information as the color difference.

More specifically, referring to FIG. 5, the convolutional neural network includes ten convolution blocks ranging from the first convolutional block to the tenth convolutional block. Each of the convolution blocks includes two to three convolution and active function pairs, wherein the first convolution block and the tenth convolution block have a size of 64 channels in height H and width W, and a second convolution block. And the 9th convolution block has a size of 128 channels in height H / 2 and a width W / 2, and the 3rd and 8th convolution blocks have a size of 256 channels in height H / 4 and a width W / 4. The fourth to seventh convolution blocks have a size of 512 channels at height H / 8 and width W / 8. Preferably, the activity function is ReLU (Rectified Linear Unit).

When the spatial resolution decreases, such as from the first convolution block to the second convolution block, from the second convolution block to the third convolution block, and from the third convolution block to the fourth convolution block, by downsampling Reduce the spatial resolution, and increase the spatial resolution as in the seventh convolution block to the eighth convolution block, the eighth convolution block to the ninth convolution block, and the ninth convolution block to the tenth convolution block. In this case, the spatial resolution is increased by upsampling.

Batch normalization is performed after each of the convolution blocks.

The output of the first convolution block of the parallax compensation color information is connected to the first convolution block of the mono image.

The output of the first convolution block is connected to the tenth convolution block, the output of the second convolution block to the ninth convolution block, and the output of the third convolution block to the eighth convolution block.

The last layer of the tenth convolution block is a 1 x 1 kernel to generate two channels of output colors. In one embodiment of the present invention, the output colors of the two channels may be ab color values of a Lab color space.

This convolutional neural network is learned in the following way.

The mapping function of the convolutional neural network may be expressed as follows.

X is a mono image, H is parallax compensation color information, and θ is a parameter of the convolutional neural network.

At this time, the loss function indicating the degree of agreement between the output value and the actual data can be expressed as follows.

At this time, the convolutional neural network performs the learning of the convolutional neural network by deriving the parameter θ satisfying the following equation.

6 is a flowchart schematically showing the detailed steps of the parallax compensation color information generation step (S510) according to an embodiment of the present invention.

Referring to FIG. 6, the disparity compensation color information generation step S510 according to an embodiment of the present invention may include a masking generation step S511 and a disparity compensation color information delivery step S512.

In the masking generation step S511, when the difference between the partial regions of the same coordinates of the denoised color image and the mono image is less than the BJND value of the BJND model representing the difference between the binocular images, the masking is generated in the partial region. do.

As described above, the BJND model represents a minimum discrimination difference that can be recognized by both eyes according to luminance. When the difference between the denoised color image and the mono image is less than the BJND value of the BJND model, The human eye cannot recognize the difference between the denoised color image and the mono image. As such, when the degree of difference of the partial region of the same coordinate is too low to be perceived by a human, masking is generated in the corresponding region, thereby distinguishing the region of low difference from the region of high difference.

The process of performing the masking generation step (S511) is expressed as a formula.

H ( x, y ) is masking in the subregion of (x, y) coordinates, and S ( x, y ) is the degree of difference in the subregion of (x, y) coordinates. The BJND (x, y) is a BJND value in the partial region of the (x, y) coordinate.

As such, masking is generated when the BJND value at a specific coordinate is less than the degree of difference, and masking is not generated when the BJND value is more than the degree of difference.

The disparity compensation color information transferring step (S512) generates disparity compensation color information by inputting color information of the denoised color image in consideration of the disparity estimated for the masked area. Since the masked region is a region having a low difference between the denoised color image and the mono image, the masking is generated in consideration of the parallax in which color information of the denoised color image is estimated. Will be entered.

The process of performing the parallax compensation color information transferring step (S510) is represented by the following equation.

( x,y )는 (x,y) 좌표의 부분영역에서의 시차보상색상정보 이고, a( x,y )는 (x,y) 좌표의 부분영역에서의 디노이즈드칼라영상의 색상정보이다. 상기 α 및 β는 상기 (x,y) 좌표의 부분영역에서의 시차이다.remind

( x, y ) is the parallax compensation color information in the subregion of (x, y) coordinates, and a ( x, y ) is the color information of the denoised color image in the subregion of (x, y) coordinates. . Α and β are parallaxes in the partial region of the (x, y) coordinates.

In the case where the masking is generated ( H ( x, y ) = 1), the color information of the denoised color image that compensates for the parallax is input to the parallax compensation color information, and an area ( H ( x, y ) = 0) creates color holes by specifying color information as 0. The color hole generated as described above receives color information through the color hole filling step (S520).

7 is a flowchart schematically illustrating a process of processing a color image and a mono image according to an embodiment of the present invention.

Referring to FIG. 7, a fusion method of a color-mono dual camera photographed image according to an embodiment of the present invention receives a mono image 10 and a color image 20.

Thereafter, the geometrically corrected color image 21 aligned with the mono image 10 is generated based on the color image 20 received in the geometric correction step S210.

Thereafter, in the normalization step (S220), the brightness of the geometrically corrected color image 21 is adjusted to generate a preprocessed color image 22.

Thereafter, in the monochrome step (S230), color information of the preprocessed color image 22 is removed to generate a grayscale image 23.

Thereafter, in the parallax estimating step (S300), the parallax of the mono image 10 and the gray scale image 23 is estimated.

Subsequently, in the joint denoising step S400, the noise of the preprocessed color image 22 is removed based on the disparity estimated in the mono image 10 and the parallax estimation step S300. 24).

Subsequently, in the generation of the parallax compensating color information (S510), the denoise considering the parallax estimated in the color information of the mono image based on the degree of difference between the mono image 10 and the denoised color image 24 is performed. Disparity compensation color information 30 is generated by inputting color information of the decolor image.

Thereafter, in the color hole filling step (S520), color information derived based on the input parallax compensation color information 30 is filled into color holes in which color information is not input among the parallax compensation color information 30. The parallax compensation color information 31 filled with the color information is generated.

Subsequently, in the image merging step S530, the monochromatic image 10 merges the parallax compensation color information 31 filled with the color information to generate an improved color image 40.

By generating an improved color image 40 in which the mono image 10 and the color image 20 are fused by the color-mono dual camera through the above process, an image of excellent image quality is obtained in a low-light shooting environment. Can exert.

8 is a block diagram schematically illustrating an internal configuration of a fusion device of a color-mono dual camera captured image according to an embodiment of the present invention.

Referring to FIG. 8, the image fusion apparatus 1000 according to an embodiment of the present invention may include an image input unit 1100, a preprocessor 1200, a parallax estimator 1300, a joint denoising unit 1400, and color information transfer. Part 1500 is included.

The image input unit 1100 receives a mono image and a color image captured by the color-mono dual camera. The image input unit 1100 receives the mono image captured by the mono CIS 100 as shown in FIG. 1 and the color image captured by the color CIS 200. This is the same as performing the image input step (S100).

The preprocessor 1200 performs preprocessing on the input mono image and the color image. The preprocessing unit 1200 performs preprocessing necessary to perform fusion on the mono image and the color image. The preprocessing may perform all preprocessing necessary for subsequent processing steps such as color space conversion, geometric correction, normalization, and monochrome. By normalizing the mono image and the color image received through such a pre-processing, the mono image and the color image received are converted into a form suitable for fusion processing. This is the same as performing the preprocessing step (S200).

The parallax estimator 1300 estimates the parallax from the color image which has been preprocessed and the mono image which has been preprocessed. The parallax estimator 1300 estimates the parallax by comparing the color image and the mono image. In the process of deriving parallax from such stereoscopic images, there is a high possibility of error, and if such errors occur at pixels near the boundary of the image object, serious structural distortion may occur in the image and the overall cognitive quality of the image may be degraded. The derived parallax only serves as an estimate.

In an embodiment of the present invention, the disparity is obtained by dividing the mono image and the color image into partial regions and creating a disparity map by estimating and storing the disparity of the corresponding partial region for each coordinate of the partial region. It can be estimated. Preferably, the partial region may consist of one pixel.

At this time, as a method for estimating the parallax between the mono image and the color image, various known algorithms may be used.

This process is the same as performing the parallax estimation step (S300).

The joint denoising unit 1400 generates a de-noise color image by removing noise of the pre-processed color image based on the mono-image which has been preprocessed.

In an embodiment of the present invention, the joint denoising unit 1400 performs denoising using the mono image as reference information to remove noise of the pre-processed color image. Since the mono image is an image obtained through the mono CIS 100 and has a higher SNR and a lot of detail information, noise of the color image that has been preprocessed by performing denoising using the mono image as reference information is obtained. Can be removed effectively.

However, although the mono image has a higher SNR and a lot of detail information, parallax exists because the image is photographed at a position different from that of the color image. Therefore, it is necessary to remove the noise in consideration of the parallax.

In this case, when the disparity estimated by the disparity estimating unit 1300 is less than a preset denoise reference value, the joint denoising unit 1400 removes the noise based on the mono image and the pre-processed color image, When the parallax estimated in the estimating step is equal to or greater than a preset denoise reference value, noise may be removed based on the preprocessed color image.

In this case, the denoise reference value may be determined based on a BJND model representing the difference between the binocular images.

This process is the same as performing the joint denizing step (S400).

The color information transmitting unit 1500 generates the improved color image with improved image quality by transferring color information from the denoised color image to the mono image.

The color information transmitting unit 1500 may generate an improved color image having improved image quality by transferring color information of the denoised color image to color information of the mono image. In this case, since the mono image and the denoised color image have parallaxes, it is necessary to compensate the parallax to transmit color information.

In this case, the estimated parallax may have a large number of errors, and distortion of an image may occur due to the errors. Therefore, in an embodiment of the present invention, the color information is not transmitted directly to reflect the parallax, the color information is transmitted only in an area of low probability of parallax, and the previously transmitted color of an area of high probability of error. The improved color image may be generated by deriving color information based on the information. This is the same as performing the color information transfer step (S500).

9 is a block diagram schematically illustrating an internal configuration of the preprocessor 1200 according to an embodiment of the present invention.

Referring to FIG. 9, the preprocessor 1200 according to an embodiment of the present invention includes a geometric correction unit 1210, a normalization unit 1220, and a monochrome unit 1230.

The geometric correction unit 1210 aligns the mono image and the color image through a linear transformation of the input mono image and the color image. Since the mono image and the color image are images obtained through separate lenses and CIS, angles and magnifications of the captured images may not exactly match. The geometric correction unit 1210 matches a partial region of the color image corresponding to the mono image partial region so that the matched partial regions may be located at the same coordinates of each image. Convert the above. More specifically, alignment may be performed by deriving epipolar lines from the mono image and the color image and matching the epipolar lines.

The geometric correction unit 1210 may perform alignment by converting both the mono image and the color image, or perform the alignment by converting the color image according to the mono image. This is the same as performing the geometric correction step (S210).

The normalization unit 1220 generates a pre-processed color image by adjusting the brightness of the color image received from the geometric correction unit 1210 according to the brightness of the mono image. The normalizer 1220 adjusts the brightness difference between the two images to increase the accuracy in estimating the parallax for the matched partial region of the two images. The normalizer 1220 may use a method such as image histogram matching or least squares method. This is the same as performing the normalization step (S220).

The monochrome unit 1230 generates a grayscale image by removing color information of the preprocessed color image. As such, by generating a grayscale image from which color information of a color image is removed, parallax may be more accurately estimated through matching with the mono image. This is the same as performing the monochrome step (S230).

In an exemplary embodiment including the preprocessor 1200, the parallax estimator 1300 estimates the parallax between the grayscale image and the mono image, and the joint denizing unit 1400 performs the preprocessing color image. The noise of the preprocessed color image may be removed based on the mono image.

The parallax estimator 1300 may estimate the parallax of the grayscale image and the mono image. Thus, by comparing the grayscale image from which the color information has been removed by preprocessing the color image with the mono image, the parallax is estimated, thereby making it possible to estimate the parallax more accurately.

In addition, the joint denoising unit 1400 generates a denoised color image including color information by removing noise of the preprocessed color image based on the mono image and the parallax estimated by the parallax estimating unit 1300. can do.

10 is a block diagram schematically illustrating an internal configuration of a color information transmitting unit 1500 according to an embodiment of the present invention.

Referring to FIG. 10, the color information transmitting unit 1500 includes a parallax compensating color information generating unit 1510, a color hole filling unit 1520, and an image merger 1530.

The parallax compensation color information generation unit 1510 may perform color information of the partial region of the mono image, which has been preprocessed, based on the degree of difference for each partial region of the mono image, which has been preprocessed. Disparity compensation color information for each subregion is generated by inputting color information of a partial region of the denoised color image corresponding to the estimated parallax.

More specifically, in an embodiment of the present invention, the degree of difference between the corresponding partial region of the denoised color image and the mono image is derived, and the masking is generated and generated for an area whose difference is less than a predetermined criterion. Color information of the denoised color image may be transmitted based on the disparity estimated for the masked area. This is the same as performing the parallax compensation color information generation step (S510).

In one embodiment of the present invention, the parallax compensation color information generation unit 1510 may include a masking generation unit 1511 and a parallax compensation color information transmission unit 1512.

The masking generator 1511 generates masking in the partial region when the difference between the partial regions of the same coordinates of the denoised color image and the mono image is less than the BJND value of the BJND model representing the difference between the binocular images. do. This is the same as performing the masking generation step (S511).

The parallax compensation color information transmitting unit 1512 generates parallax compensation color information by inputting color information of the denoised color image in consideration of the parallax estimated for the region in which the masking is generated. Since the masked region is a region having a low difference between the denoised color image and the mono image, the masking is generated in consideration of the parallax in which color information of the denoised color image is estimated. Will be entered. This is the same as performing the parallax compensation color information transferring step (S512).

The color hole peeling unit 1520 is derived based on the disparity compensation color information of another partial area in which one or more color information is input to a color hole in which the color information is not input among the disparity compensation color information for each partial area. Fill in the color information. In an embodiment of the present invention, the color hole filling unit 1520 may derive color information to be filled in the color hole based on an artificial neural network algorithm. In this case, the artificial neural network may be a convolutional neural network. As described above, the deep learning method based on the convolutional neural network can suppress the color bleeding caused by deriving and filling the color information into the color hole, and enhance the color reproducibility by filling the accurate color information. This is the same as performing the color hole filling step (S520).

The image merger 1530 merges the disparity compensation color information filled with the color information received from the color hole filling unit 1520 into the mono image that has been preprocessed to generate an improved color image.

The image merger 1530 merges the disparity compensation color information generated by the color hole filling unit 1520 into the color information of the mono image, thereby adding color information to brightness information of the mono image and improving the color image. Can be generated. In an embodiment of the present invention, the color information is an ab color value in a Lab color space, and the ab color value is merged with an L luminance value stored in the mono image to generate an improved color image having Lab color information. . This is the same as performing the image merging step S530.

11 is a block diagram illustrating an example of an internal configuration of a computing device according to an embodiment of the present disclosure.

The entire abnormal usage detection system may be implemented by a computing device to be described later, or two or more of the computing devices to be described later may construct the abnormal usage detection system.

Preferably, the situation information analysis module and the abnormality detection module is preferably implemented by a separate computing device. Meanwhile, the profile storage unit and the policy storage unit may be configured as separate computing devices, unlike other components of the abnormality detection module.

As shown in FIG. 11, the computing device 11000 may include at least one processor 11100, a memory 11200, a peripheral interface 11300, and an input / output subsystem ( I / O subsystem 11400, power circuit 11500, and communication circuit 11600 at least. In this case, the computing device 11000 may correspond to the user terminal A connected to the service server B or the service server B described above.

The memory 11200 may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or nonvolatile memory. have. The memory 11200 may include a software module, an instruction set, or other various data necessary for the operation of the computing device 11000.

In this case, accessing the memory 11200 from another component such as the processor 11100 or the peripheral device interface 11300 may be controlled by the processor 11100.

The peripheral interface 11300 may couple the input and / or output peripherals of the computing device 11000 to the processor 11100 and the memory 11200. The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

Input / output subsystem 11400 may couple various input / output peripherals to peripheral interface 11300. For example, the input / output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor or keyboard, a mouse, a printer, or a touch screen or a sensor, as necessary, to the peripheral interface 11300. According to another aspect, the input / output peripherals may be coupled to the peripheral interface 11300 without passing through the input / output subsystem 11400.

The power circuit 11500 may supply power to all or part of the components of the terminal. For example, power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), charging systems, power failure detection circuits, power converters or inverters, power status indicators or power sources. It can include any other components for creation, management, distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, the communication circuit 11600 may include an RF circuit to transmit and receive an RF signal, also known as an electromagnetic signal, to enable communication with other computing devices.

This embodiment of FIG. 11 is only one example of the computing device 11000, and the computing device 11000 may include some components shown in FIG. 11, or may further include additional components not shown in FIG. 11, or 2. It may have a configuration or arrangement that combines two or more components. For example, the computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, in addition to the components illustrated in FIG. 11, and various communication schemes (WiFi, 3G, LTE) in the communication circuit 1160. , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software, including integrated circuits specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that may be executed by various computing devices and may be recorded in a computer readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed in a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmitter (not shown) for transmitting the file at the request of the user terminal.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments are, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs). May be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of these, and configure the processing device to operate as desired, or process it independently or in combination. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device for the purpose of interpreting or providing instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computing devices so that they are stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

According to an embodiment of the present invention, it is possible to achieve an effect of obtaining an image having excellent image quality by fusing a mono image obtained through a mono sensor and a color image obtained through a color sensor.

According to an embodiment of the present invention, the color information is transferred from a color image to a mono image having a higher signal-to-noise ratio (SNR) and image detail information in a low-light shooting environment, thereby obtaining an image having excellent image quality. Can be exercised.

According to an embodiment of the present invention, by estimating the parallax, and joint denoising based on the estimated reliability of the parallax, it is possible to exert an excellent denoising effect.

According to an embodiment of the present invention, the masking is generated based on the difference between the color image and the mono image, and the color information is transmitted based on the masking, thereby improving the accuracy of the color information transmission.

According to one embodiment of the present invention, after receiving color information based on masking, color hole filling through deep learning can be performed, thereby minimizing color bleeding and improving color reproducibility.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved. Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (23)

As a fusion method of the color mono mono camera photography image,
An image input step of receiving a mono image and a color image captured by the color-mono dual camera;
A preprocessing step of performing preprocessing on the input mono image and the color image;
A parallax estimating step of estimating a parallax from the pre-processed color image and the pre-processed mono image;
A joint denoising step of removing noise of the pre-processed color image based on the pre-processed mono image to generate a denoised color image; And
A color information transfer step of transferring color information from the de-noise color image to the mono image which has been pre-processed to generate an improved color image having improved image quality; Including,
The color information transfer step,
Color information is transmitted to the mono image for an area where the possibility of parallax error is less than or equal to a predetermined criterion, and color information is derived based on the previously transmitted color information for an area where the possibility of parallax error exceeds a predetermined criterion. , Fusion method of color-mono dual cameras.
As a fusion method of the color mono mono camera photography image,
An image input step of receiving a mono image and a color image captured by the color-mono dual camera;
A preprocessing step of performing preprocessing on the input mono image and the color image;
A parallax estimating step of estimating a parallax from the pre-processed color image and the pre-processed mono image;
A joint denoising step of removing noise of the pre-processed color image based on the pre-processed mono image to generate a denoised color image; And
A color information transfer step of transferring color information from the de-noise color image to the mono image which has been pre-processed to generate an improved color image having improved image quality; Including,
The color information transfer step,
Based on the degree of difference for each partial region of the mono image, which has been preprocessed, the color image corresponding to the color information of the partial region of the mono image having been preprocessed in consideration of the disparity estimated Disparity compensation color information generation step of generating disparity compensation color information for each partial region by inputting color information of the partial region of the denoised color image;
Color hole filling that fills color information derived based on the disparity compensation color information of another partial area into which one or more color information is input, in a color hole in which the color information is not input among the disparity compensation color information for each partial area. step; And
An image merging step of merging the disparity compensation color information for each partial region filled with the color information through the color hole filling step into the mono image which has been preprocessed to generate an improved color image; Including, the fusion method of the color-mono dual camera photographing image.
The method according to claim 2,
The disparity compensation color information generation step,
A masking generation step of generating masking in the partial region when the difference between the partial regions of the same coordinates of the de-nosed color image and the mono image is less than the BJND value of the BJND model indicating the difference between the binocular images; And
A parallax compensation color information transferring step of generating parallax compensation color information by inputting color information of the denoised color image in consideration of the parallax estimated for the masked area; Including, the fusion method of the color-mono dual camera photographing image.
The method according to claim 2,
The color hole peeling step,
A color-mono dual camera photographing method for deriving color information to be filled in the color hole based on an artificial neural network algorithm.
The method according to claim 4,
The color hole peeling step,
A method of fusing color-mono dual camera images, which derives color information to be filled in a color hole based on a convolutional neural network.
The method according to claim 5,
The convolutional neural network,
And a cross-channel auto-encoder that uses the disparity compensation color information as a color hint.
delete delete delete delete As a fusion system of color mono mono camera photography images,
An image input unit configured to receive a mono image and a color image captured by the color-mono dual camera;
A preprocessing unit performing preprocessing on the input mono image and the color image;
A parallax estimator for estimating a parallax from the color image pre-processed and the mono image pre-processed;
A joint denoising unit for generating a denoised color image by removing noise of the pre-processed color image based on the mono-image which has been preprocessed; And
A color information transfer unit configured to transfer color information from the de-noise color image to the mono image which has been pre-processed to generate an improved color image with improved image quality; Including,
The color information transfer unit,
Color information is transmitted to the mono image for an area where the possibility of parallax error is less than or equal to a predetermined criterion, and color information is derived based on the previously transmitted color information for an area where the possibility of parallax error exceeds a predetermined criterion. , Fusion system of color-mono dual cameras.
As a fusion system of color mono mono camera photography images,
An image input unit configured to receive a mono image and a color image captured by the color-mono dual camera;
A preprocessing unit performing preprocessing on the input mono image and the color image;
A parallax estimator for estimating a parallax from the color image pre-processed and the mono image pre-processed;
A joint denoising unit for generating a denoised color image by removing noise of the pre-processed color image based on the mono-image which has been preprocessed; And
A color information transfer unit configured to transfer color information from the de-noise color image to the mono image which has been pre-processed to generate an improved color image with improved image quality; Including,
The color information transfer unit,
Based on the degree of difference for each partial region of the mono image, which has been preprocessed, the color image corresponding to the color information of the partial region of the mono image having been preprocessed in consideration of the disparity estimated A disparity compensation color information generation unit for generating disparity compensation color information for each partial region by inputting color information of the partial region of the denoised color image;
Color hole filling that fills color information derived based on the disparity compensation color information of another partial area into which one or more color information is input, in a color hole in which the color information is not input among the disparity compensation color information for each partial area. part; And
An image merger which merges the disparity compensation color information for each partial region filled with the color information received from the color hole filling unit into the mono image that has been preprocessed to generate an improved color image; Containing, a fusion system of color-mono dual camera photographing image.
The method according to claim 12,
The parallax compensation color information generation unit,
A masking generator for generating masking in the partial region when the difference between the partial regions of the same coordinates of the de-noised color image and the mono image is less than the BJND value of the BJND model indicating the difference between the binocular images; And
A parallax compensation color information transmitter configured to generate parallax compensation color information by inputting color information of the denoised color image in consideration of the parallax estimated for the masked area; Containing, a fusion system of color-mono dual camera photographing image.
The method according to claim 12,
The color hole filling unit,
A color-mono dual camera photographing image fusion system for deriving color information to be filled in the color hole based on an artificial neural network algorithm.
The method according to claim 14,
The color hole filling unit,
A fusion system of color-mono dual camera photographing images which derives color information to be filled in a color hole based on a convolutional neural network.
The method according to claim 15,
The convolutional neural network,
A fusion system of color-mono dual camera photographing images, comprising a cross-channel auto encoder that uses the color difference information as color hints.
delete delete delete delete A computer-readable medium for implementing a fusion of color-mono dual camera photographed images,
The computer-readable medium stores instructions for causing a computing device including one or more processors and one or more memories to perform the following steps, the steps:
An image input step of receiving a mono image and a color image captured by the color-mono dual camera;
A preprocessing step of performing preprocessing on the input mono image and the color image;
A parallax estimating step of estimating a parallax from the color image and the mono image subjected to preprocessing;
A joint denizing step of generating a denoised color image by removing noise of the preprocessed color image based on the preprocessed color image and the mono image; And
A color information transfer step of generating an improved color image with improved image quality by transferring color information from the denoise color image to the mono image; Including,
The color information transfer step,
Color information is transmitted to the mono image for an area where the possibility of parallax error is less than or equal to a predetermined criterion, and color information is derived based on the previously transmitted color information for an area where the possibility of parallax error exceeds a predetermined criterion. , Computer-readable media.
A computer-readable medium for implementing a fusion of color-mono dual camera photographed images,
The computer-readable medium stores instructions for causing a computing device including one or more processors and one or more memories to perform the following steps, the steps:
An image input step of receiving a mono image and a color image captured by the color-mono dual camera;
A preprocessing step of performing preprocessing on the input mono image and the color image;
A parallax estimating step of estimating a parallax from the pre-processed color image and the pre-processed mono image;
A joint denoising step of removing noise of the pre-processed color image based on the pre-processed mono image to generate a denoised color image; And
A color information transfer step of transferring color information from the de-noise color image to the mono image which has been pre-processed to generate an improved color image having improved image quality; Including,
The color information transfer step,
Based on the degree of difference for each partial region of the mono image, which has been preprocessed, the color image corresponding to the color information of the partial region of the mono image having been preprocessed in consideration of the disparity estimated Disparity compensation color information generation step of generating disparity compensation color information for each partial region by inputting color information of the partial region of the denoised color image;
Color hole filling that fills color information derived based on the disparity compensation color information of another partial area into which one or more color information is input, in a color hole in which the color information is not input among the disparity compensation color information for each partial area. step; And
An image merging step of merging the disparity compensation color information for each partial region filled with the color information through the color hole filling step into the mono image which has been preprocessed to generate an improved color image; A computer-readable medium comprising a.
delete

Images