KR102418000B1 - Method for performing stereo matching by using color image and monochrome image and device using the same - Google Patents

Abstract

According to the present invention, a method for performing stereo matching by using a color image and a monochrome image includes the steps of: (a) allowing a learning device to generate a denoised monochrome image for learning and a denoised image for learning when at least one color/monochrome stereo pair for learning is acquired; (b) allowing the learning device to generate a final color reconstruction image for learning; and (c) allowing the learning device to learn at least one parameter included in a disparity network, an occlusion network, a denoising network, and a colorization network using summation losses. Accordingly, it is possible to obtain a color reconstruction image with high sharpness.

Description

Method for performing stereo matching using a color image and a monochrome image, and an apparatus using the same

The present invention relates to a method for performing stereo matching using a color image and a monochrome image and an apparatus using the same, and more particularly, to a disparity network and an occlusion network. network) from each of the disparity map according to the matching cost between the color image and the monochrome image and the occlusion map according to the difference between the color image and the monochrome image, and the disparity map and The present invention relates to a method of obtaining a clear image in which color is restored using a denoised color image and a denoised monochrome image generated by removing noise from an occlusion map and a denoising network, and an apparatus using the same.

As the computing power that can be operated by mobile devices increases, various companies developing smartphones and various shooting devices such as Huawei, LG, Samsung, and Apple are adjusting and merging multiple images taken simultaneously by multiple cameras. We are actively researching multi-camera setups that improve the quality of captured images. Interestingly, some commercial multi-camera setups include asymmetric camera configurations that use different Field of View (FoV) or different spectral properties simultaneously.

Among the asymmetric camera configurations, as in Huawei's P9, P10, and P20 series, a stereo camera with an RGB-Monochrome setup using a color camera and a monochrome camera is being put to practical use.

In a stereo camera configured with this RGB-Monochrome setup, the color camera has a Bayer color filter on the image sensor that separates the incoming light into one of three primary colors (red, green, or blue) depending on its wavelength, which is the color of the captured image. It is effective in acquiring information, but since the filter blocks a lot of incoming light, it can amplify image noise in low-light conditions. On the other hand, a monochrome camera can provide better light efficiency and a clearer image than a color camera because it receives all the light that enters each pixel.

FIG. 1 schematically shows the characteristics of a stereo camera configured with a conventional RGB-Monochrome setup, and FIG. 1 (a) shows the spectral sensitivity of a color camera and a monochrome camera, and FIG. 1 ( b) shows images taken by a color camera and a monochrome camera, respectively.

As can be seen from FIG. 1A , in the same spectral region, that is, at the same wavelength, a monochrome camera provides better spectral sensitivity than a color camera. Even when comparing the images taken by the monochrome camera and the color camera in FIG. 1B , it can be seen that the noise of the color image compared to the monochrome image is much higher due to the difference in light efficiency between the color camera and the monochrome camera. Therefore, RGB-Monochrome setup can provide better quality images when using both color image and monochrome image information.

However, when asymmetric images such as a color image and a monochrome image are used as a stereo pair, there is a problem in that it is difficult to accurately match the images due to a difference in non-linear spectral sensitivity between the asymmetric image pairs. That is, the image luminance recorded by the color camera is different from the image luminance recorded by the monochrome camera due to the difference in spectral luminance between the color camera and the monochrome camera and the spatially varying illuminance, thereby degrading the performance of stereo matching.

In addition, in the process of performing stereo matching between asymmetric images such as color images and monochrome images, if an occlusion area occurs between images or if incorrect matching is performed, a color-bleeding error, that is, color bleeding is issued. can do.

In addition, there is a problem that it may take time to generate an output if stereo matching is repeatedly performed to increase accuracy between asymmetric images.

Accordingly, there is a need for an improvement method to solve the above problems.

An object of the present invention is to solve all of the above problems.

In addition, the present invention performs stereo matching by combining color information of a color image and luminance information of a monochrome image by using a color image and a monochrome image as a stereo pair, but additionally generates a disparity map and an occlusion map to achieve high clarity. It aims to obtain a color restoration image.

In addition, the present invention uses an end-to-end convolutional neural network (CNN) structure to generate a high-accuracy disparity map and a clear color reconstructed image even with a relatively short processing time. The purpose.

Another object of the present invention is to restore a clear color image without noise by adding color information of a color image from which noise has been removed with reference to a disparity map to a monochrome image from which noise has been removed.

Another object of the present invention is to prevent color bleeding that may occur due to inaccurate stereo matching using an occlusion map.

In order to achieve the object of the present invention as described above and to realize the characteristic effects of the present invention to be described later, the characteristic configuration of the present invention is as follows.

According to an aspect of the present invention, in a method for performing stereo matching using a color image and a monochrome image, (a) a color image for learning corresponding to a color camera constituting a stereo camera and the When at least one learning color and monochrome stereo pair including a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained, the learning apparatus (i) converts the learning color and monochrome stereo pair into a disparity network (disparity network) ) to cause the disparity network to generate a training disparity map corresponding to the training monochrome image according to a matching cost between the training color image and the training monochrome image of the training color and monochrome stereo pair, and , (ii) input the color and monochrome stereo pair for training into an occlusion network to cause the occlusion network to perform occlusion for training according to the difference between the training color image and the training monochrome image generate an occlusion map, and (iii) input the training color and monochrome stereo pairs into a denoising network, causing the denoising network to generate an occlusion map in each of the training color image and the training monochrome image. generating a denoised monochrome image for learning and a denoised color image for learning from which noise has been removed; (b) the learning device warps the denoised color image for learning with reference to (i) (i-1) the disparity map for learning to generate a warped denoised color image for learning and (i-2) concatenating the warped denoised color image for training and the denoised monochrome image for training to generate an initial color restored image for learning, (ii) the initial color restored image for training , input the learning occlusion map and the learning monochrome image to a colorization network, causing the colorization network to refer to the occlusion map and according to the color information of the initial color restoration image for learning, the learning monochrome image restoring the color of to generate a final color restored image for learning; and (c) a first loss generated by the learning device by referring to the learning disparity map and a disparity GT (ground truth) map, and a second loss generated by referring to the learning occlusion map and the occlusion GT map. , a third loss generated by referring to the denoised monochrome image for learning, the denoised color image for learning, and a corresponding denoised GT image, and a restored chrominance value for learning of the final color restored image for learning The disparity network, the occlusion network, the denoising network, and the colorization network are included in the disparity network, the occlusion network, and the colorization network using the summation loss generated by giving weight to the fourth loss generated by referring to the GT chrominance value and the corresponding GT chrominance value. A method comprising a; learning the at least one parameter is disclosed.

As an example, in step (a), the learning apparatus inputs the learning monochrome image and the learning color image to a disparity network, causing the disparity network to perform (i) an encoder of the disparity network ) encodes the monochromatic image for training through the 1_1 sub-encoder included in ) to generate a 1_1 feature map for training with respect to the monochromatic image for training, and 1_2 sub-encoder included in the encoder of the disparity network to generate a first_2 feature map for training for the color image for training by encoding the color image for training through By performing a correlation operation on the 1_1 feature map and the first_2 feature map for learning and calculating the matching cost, a correlation feature map for learning corresponding to the color and monochrome stereo pair for learning is generated, and (iii) of the disparity network. A second feature map for learning is generated by encoding the correlation feature map for learning through a second sub-encoder included in the encoder, and (iv) the second feature map for learning through a decoder of the disparity network. A method characterized in that by decoding to generate the disparity map for learning is disclosed.

As an example, the learning apparatus causes the disparity network to perform, as the correlation operation, a first feature vector for learning corresponding to at least one first patch included in at least one first area on the first_1 feature map for learning. and a second feature vector for learning corresponding to at least one second patch included in at least one second region on the first_2 feature map for learning corresponding to the first region to generate the correlation feature map for learning Disclosed is a method characterized in that to do so.

As an example, the learning apparatus causes the disparity network to perform (i) a first intermediate feature map for learning included in the first_1 sub-encoder, the first_2 sub-encoder, and the second sub-encoder. ), wherein the first intermediate feature map for learning includes the first_1 feature map for learning, the first_2 feature map for learning and the second feature map for learning. Among the at least one decoding layer, a predetermined first intermediate feature map for learning generated in a layer is included in the decoder to generate a second intermediate feature map for learning corresponding to the first intermediate feature map for learning. At least one concatenation feature map for learning is generated by concatenation with a second intermediate feature map for learning generated in a predetermined decoding layer corresponding to a predetermined encoding layer, and the concatenation for learning A method characterized in that inputting each feature map into a subsequent layer of each of the predetermined decoding layers corresponding to the predetermined second intermediate feature map for learning used to generate the concatenation feature map for learning is initiated

As an example, the learning apparatus includes: (i) a pixel value of each pixel on a warped color image for training generated by warping the color image for training with reference to the disparity map for training, and each pixel on the monochrome image for training generating the difference map for learning by calculating the difference between the pixel values of Let the occlusion network display the occlusion between the warped color image for training and the monochrome image for training by referring to the features extracted from the learning difference map and the features of the second learning feature map as binary values for each pixel. Disclosed is a method characterized in that it generates the occlusion map for learning.

As an example, the learning device causes the occlusion network to (i) encode the difference map for learning through an encoder of the occlusion network to generate a third feature map for learning, (ii) the occlusion Disclosed is a method characterized in that the third feature map for learning and the second feature map for learning are concatenated through a decoder of a network and then decoded to generate the occlusion map for learning.

As an example, in step (a), the learning device causes the denoising network to perform at least one convolution operation on each of the learning monochrome image and the learning color image, and at least one batch normalization ) operation and at least one ReLU operation to remove the latent noiseless image for learning corresponding to each of the learning monochrome image and the learning color image from each of the learning monochrome image and the learning color image. Acquire a predicted residual image for training, and remove the noise from the training monochrome image and the training color image by referring to the training residual image corresponding to each of the training monochrome image and the training color image. Disclosed is a method comprising generating a denoised monochrome image and a denoised color image for training.

As an example, in step (b), the learning device causes the colorization network to (i) at least one pixel extracted from the initial color restoration image for training with reference to the occlusion region information of the occlusion map for training. Correcting a color bleeding error for a feature value to generate a corrected pixel feature value, and (ii) each region separated by at least one edge on the learning monochrome image used as a reference image. By using at least a part of the corrected pixel feature values as a color seed of each region for color restoration, the final color information for each pixel on the learning monochrome image is restored to obtain the final color restoration image for training. Disclosed is a method characterized in that it is caused to generate.

As an example, in step (b), the learning device causes the colorization network to (i) use an encoder of the colorization network to obtain the initial color restoration image for learning, the occlusion map for learning, and the learning use The monochrome image is concatenated and then encoded to generate a fourth feature map for training, and (ii) the fourth feature map for training is decoded using a decoder of the colorization network to decode the final color for training. Disclosed is a method comprising generating a reconstructed image.

As an example, in step (b), the learning device concatenates the warped denoised monochrome image for training and the denoised color image for training in a channel direction, and the warped denoised color image for training The color information of the chrominance channel constitutes the U channel and the V channel, and the luminance information of the denoised monochrome image for training constitutes the Y channel, which is the luminance channel. Disclosed is a method characterized in that

According to another aspect of the present invention, in a method for performing stereo matching using a color image and a monochrome image, (a) a color image for learning corresponding to a color camera constituting a stereo camera and the When at least one learning color and monochrome stereo pair including a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained, the learning device (1) (i) converts the learning color and monochrome stereo pair into a disparity network (disparity network) to cause the disparity network to generate a learning disparity map corresponding to the learning monochrome image according to a matching cost between the learning color image and the learning monochrome image of the learning color and monochrome stereo pair (ii) input the color and monochrome stereo pair for training into an occlusion network, so that the occlusion network causes the training color image and the training monochrome image according to the difference generate an occlusion map for training, and (iii) input the training color and monochrome stereo pair into a denoising network, causing the denoising network to cause the training color image and the training monochrome image. A process of generating a denoised monochrome image for training and a denoised color image for training from which noise has been removed from each, (2) (i) (i-1) Referring to the disparity map for training, the Warped the noised color image to generate a warped denoised color image for training, (i-2) convert the warped denoised color image for training and the denoised monochrome image for training by concatenation generating an initial color restored image for training, (ii) inputting the initial color restored image for training, the training occlusion map, and the training monochrome image to a colorization network, causing the colorization network to perform the occlusion map A process of restoring the color of the monochrome image for training according to the color information of the initial color restored image for training with reference to generate a final color restored image for training, and (3) the training disparity map and disparity GT (ground truth) ) the first loss generated by referring to the map, the second loss generated by referring to the occlusion map for learning and the occlusion GT map, the denoised monochrome image for learning, the denoised color image for learning, and the corresponding data loss Weights are respectively applied to the third loss generated by referring to the noised GT image, the restored chrominance value for training of the final color restored image for training, and the fourth loss generated by referring to the corresponding GT chrominance value. In a state in which the process of learning at least one parameter included in the disparity network, the occlusion network, the denoising network, and the colorization network is performed using the summation loss generated by giving the test apparatus, the stereo obtaining at least one test color and monochrome stereo pair including a test color image corresponding to the color camera constituting a camera and a test monochrome image corresponding to the monochrome camera constituting the stereo camera; (b) the test device (i) inputs the test color and monochrome stereo pair to the disparity network, so that the disparity network causes the test color image and the test color image of the test color and monochrome stereo pair; generate a test disparity map corresponding to the test monochrome image according to a matching cost between the test monochrome images, (ii) input the test color and monochrome stereo pair into the occlusion network, and cause the occlusion network to generate a test occlusion map according to a difference between the test color image and the test monochrome image, and (iii) input the test color and monochrome stereo pair into the denoising network. causing the denoising network to generate a denoised monochrome image for testing and a denoised color image for testing from which noises are removed from each of the color image for testing and the monochrome image for testing; and (c) the test device generates a warped denoised color image for testing by (i) (i-1) warping the denoised color image for testing with reference to the disparity map for testing, (i-2) concatenating the warped denoised color image for the test and the denoised monochrome image for the test to generate an initial color restored image for testing, (ii) the initial color restored image for the test; The test occlusion map and the test monochrome image are input to the colorization network, so that the colorization network refers to the occlusion map and according to the color information of the initial color restoration image for test, the test monochrome image A method comprising the steps of restoring the color of to generate a final color restored image for testing is disclosed.

As an example, in step (b), the test apparatus inputs the monochrome image for testing and the color image for testing to a disparity network, and causes the disparity network to (i) an encoder of the disparity network Encodes the test monochrome image through the 1_1 sub-encoder included in (encoder) to generate a test 1_1 feature map for the test monochrome image, and includes it in the encoder of the disparity network encoding the test color image through the first_2 sub-encoder to generate a test first_2 feature map for the test color image, (ii) a correlation layer included in the encoder of the disparity network ( A correlation layer for testing corresponding to the color and monochrome stereo pair for testing by performing a correlation operation on the first_1 feature map for testing and the first_2 feature map for testing through a correlation layer) and calculating the matching cost (iii) encoding the correlation feature map for testing through a second sub-encoder included in the encoder of the disparity network to generate a second feature map for testing, (iv) the disparity network A method is disclosed, comprising decoding the second feature map for testing through a decoder of a network to generate the disparity map for testing.

As an example, the test apparatus causes the disparity network to perform a test first patch corresponding to at least one first patch included in at least one first area on the test first_1 feature map as the correlation operation. The test nose by performing a dot product operation on a feature vector and a second feature vector for testing corresponding to at least one second patch included in at least one second region on the first_2 feature map for testing corresponding to the first region Disclosed is a method comprising generating a relation feature map.

As an example, the test device includes: (i) a pixel value of each pixel on a warped color image for testing generated by warping the color image for testing with reference to the disparity map for testing and the monochrome for testing generating the difference map for the test by calculating the difference between the pixel values of each pixel on the image; Inputs to the occlusion network, causing the occlusion network to refer to the features extracted from the difference map for testing and features of the second feature map for testing, and the warped color image for testing and the monochrome image for testing Disclosed is a method characterized by generating the occlusion map for the test in which the occlusion of the liver is expressed as a binary value for each pixel.

According to another aspect of the present invention, in a learning apparatus for performing stereo matching using a color image and a monochrome image, instructions for performing stereo matching using a color image and a monochrome image are provided. stored memory; and a processor that performs an operation for performing stereo matching using a color image and a monochrome image according to the instructions stored in the memory, wherein the processor corresponds to (I) a color camera constituting the stereo camera When at least one learning color and monochrome stereo pair including a learning color image to be used and a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained, (i) the learning color and monochrome stereo pair is combined with a disparity network ( disparity network), causing the disparity network to generate a training disparity map corresponding to the training monochrome image according to a matching cost between the training color image and the training monochrome image of the training color and monochrome stereo pair (ii) input the color and monochrome stereo pair for learning into an occlusion network, so that the occlusion network causes the learning according to the difference between the color image for learning and the monochrome image for learning generate an occlusion map, and (iii) input the training color and monochrome stereo pair into a denoising network to cause the denoising network to each of the training color image and the training monochrome image. A process of generating a denoised monochrome image for training and a denoised color image for training from which the noise in warping the denoised color image to generate a warped denoised color image for training, (i-2) concatenating the warped denoised color image for training and the denoised monochrome image for training (concatenation) for learning generate an initial color restored image, (ii) input the initial color restored image for training, the training occlusion map, and the training monochrome image to a colorization network, causing the colorization network to generate the occlusion map A process of restoring the color of the monochrome image for training according to the color information of the initial color restored image for training with reference to generate a final color restored image for training, and (III) the training disparity map and disparity GT (ground truth) The first loss generated by referring to the map, the second loss generated by referring to the occlusion map for learning and the occlusion GT map, the denoised monochrome image for learning, the denoised color image for learning, and the corresponding denoise Weights are given to the third loss generated by referring to the de GT image, the restoration chrominance value for training of the final color restoration image for training, and the fourth loss generated by referring to the GT chrominance value corresponding thereto, respectively. Disclosed is a learning apparatus that performs a process of learning at least one parameter included in the disparity network, the occlusion network, the denoising network, and the colorization network using the generated summation loss.

As an example, in the process (I), the processor inputs the monochrome image for training and the color image for training to a disparity network, and causes the disparity network to: (i) an encoder of the disparity network Encoding the monochromatic image for learning through the 1_1 sub-encoder included in generates a 1_1 feature map for learning for the monochromatic image for training, and using the 1_2 sub-encoder included in the encoder of the disparity network to generate a first_2 feature map for training with respect to the color image for training by encoding the color image for training through (ii) the first_1 for training through a correlation layer included in the encoder of the disparity network By performing a correlation operation on the feature map and the first_2 feature map for learning to calculate the matching cost, a correlation feature map for learning corresponding to the color and monochrome stereo pair for learning is generated, and (iii) the disparity network A second feature map for learning is generated by encoding the correlation feature map for learning through a second sub-encoder included in the encoder, and (iv) the second feature map for learning through a decoder of the disparity network. A learning apparatus is disclosed, characterized in that it generates the disparity map for learning by decoding.

As an example, the processor may cause the disparity network to perform, as the correlation operation, a first feature vector for learning corresponding to at least one first patch included in at least one first area on the first_1 feature map for learning; To generate the correlation feature map for learning by performing a dot product operation on a second feature vector for learning corresponding to at least one second patch included in at least one second region on the first_2 feature map for learning corresponding to the first region A learning device is disclosed, characterized in that

As an example, the processor may cause the disparity network to: (i) a first intermediate feature map for learning included in the first_1 sub-encoder, the first_2 sub-encoder, and the second sub-encoder; A predetermined encoding layer that is at least a part of at least one encoding layer for generating - the first intermediate feature map for learning includes the first_1 feature map for learning, the first_2 feature map for learning, and the second feature map for learning The predetermined first intermediate feature map for learning generated in at least one decoding layer is included in the decoder to generate a second intermediate feature map for learning corresponding to the first intermediate feature map for learning Concatenate with a predetermined second intermediate feature map for learning generated in a predetermined decoding layer corresponding to the encoding layer of A learning apparatus characterized in that each map is input to a subsequent layer of each predetermined decoding layer corresponding to the predetermined second intermediate feature map for learning used to generate the concatenation feature map for learning is initiated.

As an example, the processor may include: (i) a pixel value of each pixel on a warped color image for training generated by warping the color image for training with reference to the disparity map for training and a pixel value of each pixel on the monochrome image for training generating the difference map for learning by calculating the difference between pixel values, (ii) inputting the second feature map for learning and the difference map for learning used to generate the learning disparity map to the occlusion network, The occlusion network between the warped color image for training and the monochrome image for training is displayed as a binary value for each pixel by having the occlusion network refer to the features extracted from the learning difference map and the features of the second learning feature map. Disclosed is a learning apparatus characterized in that it generates the occlusion map for learning.

As an example, the processor is configured to cause the occlusion network to: (i) encode a difference map for training through an encoder of the occlusion network to generate a third feature map for training, (ii) the occlusion network Disclosed is a learning apparatus characterized in that the third feature map for learning and the second feature map for learning are concatenated and then decoded to generate the occlusion map for learning through a decoder of .

As an example, in the process (I), the processor causes the denoising network to perform at least one convolution operation on each of the monochrome image for training and the color image for training, and at least one batch normalization. Calculation and at least one ReLU operation are performed to predict the output obtained by removing the latent noiseless image for learning corresponding to each of the learning monochrome image and the learning color image from each of the learning monochrome image and the learning color image. A residual image for training is obtained, and the noise is removed from the training monochrome image and the training color image by referring to the training residual image corresponding to each of the training monochrome image and the training color image. Disclosed is a learning apparatus characterized by generating a noised monochrome image and a denoised color image for learning.

As an example, in the process (II), the processor causes the colorization network to: (i) at least one pixel feature extracted from the initial color restoration image for training with reference to the occlusion region information of the occlusion map for training. Correcting a color bleeding error on a value to produce a corrected pixel feature value, and (ii) the color of each region separated by at least one edge on the training monochrome image used as a reference image. By using at least a part of the corrected pixel feature values as a color seed of each region to restore Disclosed is a learning apparatus characterized in that to do so.

As an example, in the process (II), the processor causes the colorization network to: (i) use an encoder of the colorization network to perform the training initial color restoration image, the training occlusion map, and the training monochrome The image is concatenated and then encoded to generate a fourth feature map for training, and (ii) the fourth feature map for training is decoded using a decoder of the colorization network to restore the final color for training. Disclosed is a learning apparatus characterized in that it generates an image.

As an example, in the process (II), the processor concatenates the warped denoised monochrome image for training and the denoised color image for training in a channel direction, and the The color information constitutes the U channel and the V channel, which are the chrominance channels, and the luminance information of the denoised monochrome image for training constitutes the Y channel, which is the luminance channel To generate the initial color restoration image for training A learning apparatus characterized in that is disclosed.

According to another aspect of the present invention, in a test apparatus for performing stereo matching using a color image and a monochrome image, instructions for performing stereo matching using a color image and a monochrome image are provided. stored memory; and a processor that performs an operation for performing stereo matching using a color image and a monochrome image according to the instructions stored in the memory, wherein the processor corresponds to (I) a color camera constituting the stereo camera When at least one learning color and monochrome stereo pair including a learning color image to be used and a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained, the learning apparatus is configured to: (1) (i) the learning color and monochrome A stereo pair is input to a disparity network, so that the disparity network corresponds to the learning monochrome image according to a matching cost between the learning color image and the learning monochrome image of the learning color and monochrome stereo pair and (ii) input the training color and monochrome stereo pair to an occlusion network, causing the occlusion network to dip between the training color image and the training monochrome image. generate an occlusion map for learning according to difference, and (iii) input the learning color and monochrome stereo pair into a denoising network, causing the denoising network to cause the learning color A process of generating a denoised monochrome image for training and a denoised color image for training by removing noise from each of the image and the monochrome image for training, (2) (i) (i-1) the disparity map for training warped the denoised color image for learning with reference to generate a warped denoised color image for learning, (i-2) the warped denoised color image for learning and the denoise for learning Concatena a monochrome image tion) to generate an initial color restored image for training, (ii) input the initial color restored image for learning, the occlusion map for learning, and the monochrome image for learning into a colorization network, causing the colorization network to A process of restoring the color of the learning monochrome image according to the color information of the initial color restored image for training with reference to an occlusion map to generate a final color restored image for training, and (3) the training disparity map and disparity GT (ground truth) The first loss generated by referring to the map, the second loss generated by referring to the occlusion map for training and the occlusion GT map, the denoised monochrome image for training and the denoised color image for training, and the The third loss generated by referring to the corresponding denoised GT image, the restored chrominance value for training of the final color restored image for training, and the fourth loss generated by referring to the corresponding GT chrominance value In a state in which a process of learning at least one parameter included in the disparity network, the occlusion network, the denoising network, and the colorization network is performed using the summation loss generated by assigning weights to each, the stereo camera a process of obtaining at least one test color and monochrome stereo pair comprising a test color image corresponding to the color camera constituting ; to generate a test disparity map corresponding to the test monochrome image according to the matching cost, (ii) input the test color and monochrome stereo pair to the occlusion network to cause the occlusion network to generate a test occlusion map according to a difference between the test color image and the test monochrome image (iii) input the test color and monochrome stereo pair to the denoising network to cause the denoising network to remove noise from each of the test color image and the test monochrome image. a process to generate a de-monochrome image and a denoised color image for testing, and (III) (i) (i-1) warping the denoised color image for testing with reference to the disparity map for testing and testing generating a warped denoised color image for the test, (i-2) concatenating the warped denoised color image for the test and the denoised monochrome image for the test to generate an initial color restored image for the test, , (ii) inputting the initial color restoration image for testing, the occlusion map for testing, and the monochrome image for testing into the colorization network, causing the colorization network to refer to the occlusion map to determine the initial color for testing Disclosed is a test apparatus that performs a process of restoring the color of the monochrome image for testing according to color information of the restored image to generate a final color restored image for testing.

As an example, in the process (II), the processor inputs the monochrome image for testing and the color image for testing to a disparity network to cause the disparity network to: (i) an encoder of the disparity network ( Encodes the test monochrome image through the first_1 sub-encoder included in the encoder to generate a test first_1 feature map for the test monochrome image, and includes the encoder in the disparity network. Encoding the color image for testing through the first_2 sub-encoder to generate a first_2 feature map for testing with respect to the color image for testing, (ii) a correlation layer included in the encoder of the disparity network layer) to calculate the matching cost by performing a correlation operation on the 1_1 feature map for testing and the 1_2 feature map for testing through the test correlation feature map corresponding to the color and monochrome stereo pair for testing. (iii) encoding the correlation feature map for testing through a second sub-encoder included in the encoder of the disparity network to generate a second feature map for testing, (iv) the disparity network A test apparatus is disclosed, characterized in that the disparity map for the test is generated by decoding the second feature map for the test through a decoder of .

As an example, the processor is configured to cause the disparity network, as the correlation operation, to be a test first feature corresponding to at least one first patch included in at least one first area on the test first_1 feature map. Correlation for testing by performing a dot product operation on a vector and a second feature vector for testing corresponding to at least one second patch included in at least one second region on the first_2 feature map for testing corresponding to the first region Disclosed is a test apparatus characterized in that it is configured to generate a feature map.

As an example, the processor is configured to: (i) warp the test color image with reference to the test disparity map, and the pixel value of each pixel on the test warped color image and the test monochrome image generating the difference map for the test by calculating the difference between the pixel values of each pixel on the image; By inputting to the occlusion network, the occlusion network causes the test between the warped color image for testing and the monochrome image for testing by referring to the features extracted from the difference map for testing and the features of the second feature map for testing. Disclosed is a test apparatus configured to generate the occlusion map for testing in which occlusion is expressed as a binary value for each pixel.

In addition to this, a computer-readable recording medium for recording a computer program for executing the method of the present invention is further provided.

The present invention is an end including a disparity network, an occlusion network, and a colorization network composed of an encoder and a decoder, respectively, as well as a denoising network -Using an end-to-end convolutional neural network (CNN) structure, there is an effect of generating a high-accuracy disparity map and a clear color reconstructed image together with a relatively short processing time.

In addition, the present invention has an effect of restoring a clear color image without noise by adding color information of a color image from which noise has been removed with reference to a disparity map to a monochrome image from which noise has been removed.

In addition, the present invention has an effect of preventing color bleeding that may occur due to inaccurate stereo matching by using the occlusion map.

In addition, the present invention has an effect of improving the accuracy of the occlusion map by additionally using information on the disparity map when generating the occlusion map.

The accompanying drawings for use in the description of the embodiments of the present invention are only a part of the embodiments of the present invention, and for those of ordinary skill in the art to which the present invention pertains (hereinafter, "those of ordinary skill in the art"), the invention Other drawings may be obtained based on these drawings without any work being done.
1 schematically shows the difference in light efficiency between a color image and a monochrome image,
2 schematically shows a learning apparatus for learning a Color and Monochrome Stereo Network (CMSNet) that performs stereo matching using a color image and a monochrome image according to an embodiment of the present invention. ,
3 schematically shows the structure of a color and monochrome stereo network network that performs stereo matching using a color image and a monochrome image according to an embodiment of the present invention;
4 schematically shows a method of generating a color image for learning and a monochrome image for learning to be used for learning of a network according to an embodiment of the present invention;
5 schematically shows the structure of a disparity network that generates a disparity map from a stereo pair consisting of a color image and a monochrome image according to an embodiment of the present invention;
6 schematically shows the structure of an occlusion network for generating an occlusion map from a stereo pair according to an embodiment of the present invention;
7 schematically illustrates the color bleeding error correction effect according to the use of an occlusion map according to an embodiment of the present invention;
8 schematically shows the structure of a denoising network that removes noise from a monochrome image and a color image according to an embodiment of the present invention;
9 schematically shows a comparison of a disparity map generated from a stereo pair without noise and a stereo pair from which noise has been removed according to an embodiment of the present invention;
10 schematically illustrates the structure of a colorization network that generates a color restored image using a stereo pair, a disparity map, and an occlusion map according to an embodiment of the present invention;
11 schematically shows a test apparatus for testing a color and monochrome stereo network that performs stereo matching using a color image and a monochrome image according to an embodiment of the present invention;
12 schematically shows a disparity map generated according to an embodiment of the present invention by comparing it with a disparity map generated from various other networks;
13 schematically shows a color restored image generated according to an embodiment of the present invention by comparing it with color restored images generated from other various networks;
14 schematically shows a color image used as an input and a color restored image generated as an output according to an embodiment of the present invention by comparison;
15 schematically illustrates a disparity map generated from a stereo pair including a color image and a near-infrared (NIR) image according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following detailed description of the present invention refers to the accompanying drawings, which show by way of illustration a specific embodiment in which the present invention may be practiced, in order to clarify the objects, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention.

Also, throughout this description and claims, the word "comprises" and variations thereof are not intended to exclude other technical features, additions, components or steps. Other objects, advantages and characteristics of the present invention will become apparent to a person skilled in the art in part from this description and in part from practice of the present invention. The following illustrations and drawings are provided by way of illustration and are not intended to limit the invention.

Moreover, the invention encompasses all possible combinations of the embodiments indicated herein. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented in other embodiments with respect to one embodiment without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

Hereinafter, in order to enable those of ordinary skill in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a color and monochrome stereo network (CMSNet) that performs stereo matching using a color image and a monochrome image according to an embodiment of the present invention according to an embodiment of the present invention. A schematic diagram of a learning apparatus 1000 for

Referring to FIG. 2 , the learning apparatus 1000 includes a memory 1001 storing instructions for learning a color and monochrome stereo network that performs stereo matching using a color image and a monochrome image, and instructions stored in the memory 1001 . The processor 1002 may include a processor 1002 for learning a color and monochrome stereo network that performs stereo matching using a color image and a monochrome image in correspondence with each other.

Specifically, the learning device 1000 is typically a computing device (eg, a device that may include a computer processor, memory, storage, input and output devices, other components of a conventional computing device; electronic devices such as routers, switches, etc.) A desired system using a combination of communication devices; electronic information storage systems such as network attached storage (NAS) and storage area networks (SANs)) and computer software (ie, instructions that cause the computing device to function in a particular way). performance may be achieved.

In addition, the processor of the computing device may include a hardware configuration such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, and a data bus. In addition, the computing device may further include an operating system and a software configuration of an application for performing a specific purpose.

However, a case in which the computing device includes an integrated processor in which a medium, a processor, and a memory are integrated for implementing the present invention is not excluded.

A method for learning a network for performing stereo matching using a color image and a monochrome image according to an embodiment of the present invention using the learning apparatus 1000 configured as described above will be described with reference to FIGS. 3 to 10 . As follows.

First, FIG. 3 schematically illustrates the structure of a color and monochrome stereo network for performing stereo matching using a color image and a monochrome image according to an embodiment of the present invention. According to FIG. 3 , the color and monochrome stereo network includes a disparity network 100 , an occlusion network 200 , a denoising network 300 , and a colorization network 400 .

Referring to FIG. 3 , first, the learning apparatus 1000 includes at least one learning color image including a learning color image corresponding to the color camera constituting the stereo camera and a learning monochrome image corresponding to the monochrome camera constituting the stereo camera, and A monochrome stereo pair can be obtained.

Here, the stereo camera is composed of two cameras having a predetermined interval, one camera may be composed of a color camera, and the other camera may be composed of a monochrome camera. Accordingly, the learning apparatus 1000 may acquire a color and monochrome stereo pair for learning including a monochrome image photographed using a stereo camera including a monochrome camera and a color camera and the color image.

In addition, the learning apparatus 1000 performs data augmentation on a stereo data set consisting only of an existing color image in order to secure a color and monochrome stereo pair for learning including a sufficient amount of color images and monochrome images for learning. Thus, it is possible to obtain a color image for learning corresponding to the color image obtained from the color camera and a monochrome image for learning corresponding to the monochrome image obtained from the monochrome camera. Such data augmentation may be performed with reference to the characteristics of each of the color camera and the monochrome camera.

As an example, a method of generating a monochrome image for learning and a color image for learning will be described with reference to FIG. 4 .

First, as in FIG. 4A , decolorization, random gamma mapping, and signal-dependent Gaussian noise addition are performed on a color image located on the left side of an existing color stereo pair. By doing so, it is possible to generate a learning monochrome image corresponding to the left image on the stereo pair according to the present invention. Here, gamma mapping is to perform correction to transform the light intensity signal on the image, and by adding noise having a normal distribution, such as Gaussian noise, it is possible to reproduce general noise generated in the image.

{0.005, 0.015}의 기설정된 표준 편차를 가지는 신호 의존성 가우시안 노이즈를 무작위로 추가하여 학습용 모노크롬 이미지를 생성할 수 있다. That is, in order to generate a monochrome image for training, the color image located on the left side of the existing stereo pair is decolorized by performing normal RGB to grayscale conversion, and then random gamma mapping ( For example, a range of gamma values: [0.6, 0.9]) can be applied. In addition to this,

A monochromatic image for training may be generated by randomly adding signal-dependent Gaussian noise having a preset standard deviation of {0.005, 0.015}.

Next, referring to FIG. 4B , a raw image is generated from a color image located on the right side of an existing color stereo pair, and then signal-dependent Gaussian noise is added, demosaicing. By performing random adjustments of king and contrast, hue and saturation, it corresponds to the right image on the stereo pair according to the present invention, and for learning that reproduces lower light efficiency and resolution than the monochrome image for learning You can create color images.

{0.03, 0.05}의 학습용 모노크롬 이미지보다 상대적으로 높은 표준 편차를 갖는 무작위 신호 의존성 노이즈를 추가하여 학습용 컬러 이미지를 생성할 수 있다.Specifically, in order to generate a color image for training, pixels of each color channel are sampled from a fixed grid having a stride of 2 with respect to a color image located on the right side of the existing stereo pair, and then pixels with a BGGR Bayer pattern are Pixels can be collected from a rectangular grid of 2Х2 pixels to create a grayscale RAW format. The lower resolution of the color image than the monochrome image is then associated with demosaicing, so conventional demosaicing techniques can be used to convert the raw image back to a color image. In addition, it performs color augmentation including saturation (eg range [0, 1]), contrast (eg range [0, 2]), and hue (eg range [-1, 1]) to provide a different spectrum simulate the sensitivity

A color image for training can be generated by adding random signal-dependent noise having a relatively higher standard deviation than the monochrome image for training of {0.03, 0.05}.

Meanwhile, in the above configuration of the stereo camera according to the present invention, the monochrome camera is positioned on the left and the color camera is on the right. It may be reversed, and accordingly, positions of the respective color images for generating the training monochrome image and the training color image in the existing color stereo pair may also be opposite to each other.

Referring back to FIG. 3 , the learning apparatus 1000 inputs a training color and monochrome stereo pair to a disparity network 100 , and causes the disparity network 100 to select a color and monochrome stereo pair for training. According to the matching cost between the color image for learning and the monochrome image for learning, a disparity map for learning corresponding to the monochrome image for learning may be generated.

Next, the learning apparatus 1000 inputs the learning color and monochrome stereo pair to the occlusion network 200, and causes the occlusion network 200 to differentiate between the learning color image and the learning monochrome image ( difference) to generate an occlusion map.

In addition, the learning apparatus 1000 inputs the learning color and monochrome stereo pair to the denoising network 300, and the denoising network 300 removes noise from each of the learning color image and the learning monochrome image. It is possible to generate a denoised monochrome image for training and a denoised color image for training.

Next, the learning apparatus 1000 generates a warped denoised color image for learning by warping the denoised color image for learning with reference to the disparity map for learning, and a warped denoised color image for learning. And the denoised monochrome image for training is concatenated to generate an initial color restored image for learning, and the initial color restored image for learning, the occlusion map for learning, and the monochrome image for learning are input to a colorization network 400 Thus, the colorization network 400 may restore the color of the monochrome image for learning according to the color information of the initial color restored image for learning with reference to the occlusion map to generate the final color restored image for training. Here, the warping may be performed through differentiable bilinear interpolation, and the learning apparatus 1000 concatenates the warped denoised monochrome image for training and the denoised color image for training in the channel direction. , the color information of the warped denoised color image for training constitutes the U channel and V channel, which are the chrominance channels, and the luminance information of the denoised monochrome image for training constitutes the Y channel, which is the luminance channel. An initial color restoration image for training can be generated.

Next, the learning apparatus 1000 includes a first loss generated by referring to a disparity map for learning and a disparity ground truth (GT) map, a second loss generated by referring to an occlusion map for learning and an occlusion GT map for learning, and a second loss for learning. Reconstruction chrominance values for training and GT corresponding to the third loss and final color reconstructed image for training generated by referring to the denoised monochrome image, the denoised color image for training, and the corresponding denoised GT image The disparity network 100 , the occlusion network 200 , the denoising network 300 , and the colorization network 400 use the summation loss generated by giving weight to the fourth loss generated by referring to the chrominance value. ) can be learned at least one parameter included in the .

A method of performing stereo matching using a color image and a monochrome image using the disparity network 100, the occlusion network 200, the denoising network 300, and the colorization network 400 as described above is shown in FIGS. It will be described in detail with reference to FIG. 10 as follows.

First, FIG. 5 schematically shows the structure of a disparity network 100 for generating a disparity map from a stereo pair consisting of a color image and a monochrome image according to an embodiment of the present invention. will be.

Referring to FIG. 5 , the learning apparatus 1000 first inputs a monochrome image for learning and a color image for learning to the disparity network 100 , and causes the disparity network 100 to cause the disparity network 100 to be an encoder of the disparity network 100 . ) by encoding the monochromatic image for learning through the 1_1 sub-encoder included in ) to generate a 1_1 feature map for learning for the monochromatic image for training, and the 1_2 sub-encoder included in the encoder of the disparity network 100 Through this, it is possible to encode the training color image to generate the first_2 feature map for training with respect to the training color image. As an example, the learning apparatus 1000 performs a 7*7 convolution operation and a 5*5 convolution operation on each of the monochrome image for learning and the color image for learning through the 1_1 sub-encoder and the 1_2 sub-encoder, respectively, to obtain the monochromatic image for learning and the color for learning. The first_1 feature map and the first_2 feature map having an increased number of channels and a reduced size compared to the image may be generated. In this case, each of the 7*7 convolution operation and the 5*5 convolution operation may perform at least one convolution operation with different strides, and a feature map of a preset size through zero padding in the convolution operation. You can also make this output.

Then, the learning apparatus 1000 causes the disparity network 100 to correlate to the 1_1 feature map for learning and the 1_2 feature map for learning through a correlation layer included in the encoder of the disparity network 100 . By performing an operation and calculating a matching cost, a correlation feature map for learning corresponding to a color and monochrome stereo pair for learning may be generated.

Here, the learning apparatus 1000 causes the disparity network 100 to perform a first feature vector for learning corresponding to at least one first patch included in at least one first area on the first_1 feature map for learning as a correlation operation. and a second feature vector for learning corresponding to at least one second patch included in at least one second region on the first_2 feature map for learning corresponding to the first region to generate a correlation feature map for learning have. Here, the first feature vector for learning and the second feature vector for learning may include gradient information that may indicate edge information on an image according to a change in image intensity or color included in the first patch and the second patch, respectively. However, the present invention is not limited thereto.

Then, the learning apparatus 1000 causes the disparity network 100 to encode the correlation feature map for learning through the second sub-encoder included in the encoder of the disparity network 100 to generate a second feature map for learning. can As an example, the learning apparatus 1000 may perform at least one 3*3 convolution operation on the correlation feature map for learning through the second sub-encoder. In this case, the 3*3 convolution operation may perform at least one or more convolution operations having different strides.

Subsequently, the learning apparatus 1000 may cause the disparity network 100 to decode the second feature map for learning through a decoder of the disparity network 100 to generate a learning disparity map. Here, the generated disparity map may indicate depth information corresponding to a learning monochrome image that is a left image of a color and monochrome stereo pair for learning. For example, the learning apparatus 1000 may generate a disparity map for learning whose size is enlarged compared to that of the second feature map for learning by performing at least one 3*3 deconvolution operation on the second feature map for learning through a decoder. have. In this case, the final stage of the decoder may generate a disparity map for learning having one channel through a convolution operation using a 3*3 filter having one channel.

Meanwhile, the learning apparatus 1000 corresponds to the encoder and the decoder of the disparity network 100 in order to preserve both high-level information and precise local information about objects on the image. It is possible to connect intermediate feature maps generated in one or more layers.

According to this, the learning apparatus 1000 causes the disparity network 100 to generate a first intermediate feature map for learning included in the 1_1 sub-encoder, the 1_2 sub-encoder, and the second sub-encoder at least one encoding. At least one of a first intermediate feature map for learning generated from a predetermined encoding layer that is at least a part of the layers included in the decoder to generate a second intermediate feature map for learning corresponding to the first intermediate feature map for learning At least one learning concatenation feature map may be generated by concatenating with a predetermined second intermediate feature map for learning generated in a predetermined decoding layer corresponding to a predetermined encoding layer among the decoding layers of . Here, the first intermediate feature map for learning includes a first_1 feature map for learning, a first_2 feature map for learning, and a second feature map for learning. Also, as an example, as shown in FIG. 5 , the learning apparatus 1000 concatenates the first intermediate feature map for learning and the second intermediate feature map for learning, performs a 3*3 convolution operation, and then performs bilinear interpolation ) operation, then 3*3 deconvolution operation can be performed.

)로서 생성할 수 있다.In order to learn the disparity network 100 configured as described above, the learning apparatus 1000 may generate a first loss with reference to a learning disparity map and a disparity GT map. Specifically, in order to train the disparity network 100, a reverse Huber loss is applied to the first loss (

) can be created as

[Equation 1]

와

각각은 학습용 디스패리티 맵과 디스패리티 GT 맵을 나타낸다.

은 이미지 상의 픽셀 수를 나타내며,

는 이미지의 픽셀 인덱스를 나타낸다. 또한,

는 절대 연산이며,

는 그 값이

로 지정(

)된 변수이다. here,

Wow

Each represents a learning disparity map and a disparity GT map.

represents the number of pixels on the image,

represents the pixel index of the image. In addition,

is an absolute operation,

is its value

specified as (

) is a variable.

Next, FIG. 6 schematically shows the structure of an occlusion network 200 that generates an occlusion map from a color and monochrome stereo pair for learning according to an embodiment of the present invention.

In order to generate the occlusion map, the learning apparatus 1000 refers to the training disparity map and warps the training color image to create a pixel value of each pixel on the training warped color image and each pixel on the training monochrome image. Generates a learning difference map that calculates the difference between pixel values of ) with reference to the features extracted from the difference map for learning and the features of the first feature map for learning, the occlusion between the warped color image for training and the monochrome image for training is displayed as a binary value for each pixel. map) can be created.

Accordingly, as shown in FIG. 6 , the learning apparatus 1000 first causes the occlusion network 200 to encode the difference map for learning through the encoder of the occlusion network 200 to generate a third feature map for learning. have. For example, as shown in FIG. 6 , the learning apparatus 1000 may perform at least one 3*3 convolution operation on the learning difference map through an encoder.

Next, the learning apparatus 1000 concatenates the third feature map for learning and the second feature map for learning through the decoder of the occlusion network 200, and then decodes it to generate an occlusion map for learning. have. For example, as shown in FIG. 6 , the learning apparatus 1000 concatenates the third feature map for learning and the second feature map for learning through the concatenation layer of the decoder, and then performs at least one 3*3 deconvolution operation and at least one 3*3 convolution operation of . In this case, the final stage of the decoder may generate an occlusion map for learning having two channels through a convolution operation using a 3*3 filter having two channels.

Here, the reason the difference map for learning is used as an input of the occlusion network 200 is to generate a feature map for texture similarities as a third feature map for learning. In addition, the occlusion network 200 intends to increase the accuracy of the occlusion map for learning generated by additionally using the disparity information of the second feature map for learning to the texture similarity information of the third feature map for learning.

)를 구하기 위한 손실 함수로써는 바이너리 크로스 엔트로피(binary cross-entropy)를 사용할 수 있다.Meanwhile, in order to learn the occlusion network 200 configured as described above, the learning apparatus 1000 may generate a second loss with reference to the occlusion map for learning and the occlusion GT map. Specifically, the second loss (

), binary cross-entropy can be used as a loss function.

[Equation 2]

는 바이너리 마스크(binary mask; 오클루션된 영역은 1, 오클루션되지 않은 영역은 0)를 나타내며,

는 오클루션된 영역의 예측 확률을 나타낸다.here,

represents a binary mask (1 for an occluded area, 0 for an unoccluded area),

denotes the predicted probability of the occluded region.

When the color is reconstructed using the occlusion map generated by this method, a more accurate color reconstructed image can be generated compared to the related art.

That is, referring to FIG. 7 , in the occlusion GT map of FIG. 7 (a) and in the state in which occlusion exists, the color is restored without reflecting the occlusion information as in FIG. 7 (b). In this case, a color bleeding error, that is, color bleeding, occurs in some areas of the occlusion area, whereas in the state in which the occlusion map according to the present invention is generated as shown in FIG. 7C . , it can be seen that the color bleeding phenomenon is corrected when the color is restored by reflecting the occlusion information according to the occlusion map generated in (c) of FIG. 7 as in (d) of FIG.

Next, FIG. 8 schematically illustrates the structure of a denoising network 300 for removing noise from a monochrome image and a color image according to an embodiment of the present invention.

Referring to FIG. 8 , the learning apparatus 1000 causes the denoising network 300 to perform at least one convolution operation, at least one batch normalization operation and Obtain a residual image for training by performing at least one ReLU operation, and remove noise from the monochrome image for training and the color image for training by referring to the training residual image corresponding to each of the training monochrome image and the training color image. You can create a noised monochrome image and a denoised color image for training. Here, the residual image for training may be a prediction of the output obtained by removing a latent noiseless image for training corresponding to each of the training monochrome image and the training color image from each of the training monochrome image and the training color image.

According to FIG. 8 , the denoising network 300 is composed of 17 convolutional layers to which batch normalization and Rectifier Linear Unit (ReLU) activation are applied except for the first layer and the last layer to generate a residual image for training. Here, the advantage of using batch normalization is that the training residual image follows a Gaussian distribution utilizing the Gaussian normalization step of batch normalization. In addition, each convolutional layer can be initialized with orthogonal regularization, which can be effective in suppressing noise in the image and preserving details in the image. At this time, in the last convolutional layer of the denoising network 300, if the target to be denoised is a monochrome image, a convolution operation with one channel is performed, and if the target to be denoised is a color image, 3 A convolution operation with channels can be performed.

)를 통해 학습용 디노이즈드 모노크롬 이미지 및 학습용 디노이즈드 컬러 이미지와 이에 대응하는 디노이즈드 GT 이미지 간의 유클리디안 거리(Euclidean Distance)를 최소화함으로써 디노이징 네트워크(300)를 학습시킬 수 있다.In order to train the denoising network 300 configured as above, the learning apparatus 1000 generates a third loss by referring to the denoised monochrome image for learning, the denoised color image for learning, and the denoised GT image corresponding thereto. can Specifically, the learning device 1000 is a third loss (

), the denoising network 300 can be trained by minimizing the Euclidean distance between the denoised monochrome image for learning and the denoised color image for learning and the corresponding denoised GT image.

[Equation 3]

는 디노이즈드 GT 이미지를 나타내고,

는 노이즈가 제거되지 않은 상태의 학습용 컬러 이미지와 학습용 모노크롬 이미지를 나타내며,

은 학습용 모노크롬 이미지와 학습용 컬러 이미지 각각에 대응하는 학습용 잔차 이미지를 나타낸다. 그리고,

는 L₂ 노름(L₂ norm)을 나타내며,

는 이미지 채널의 인덱스를 나타낸다. 또한,

는 컬러 채널의 개수를 나타내며, 학습용 모노크롬 이미지가 디노이징 네트워크(300)의 입력일 경우

는 1이 된다. here,

represents the denoised GT image,

represents a color image for training and a monochrome image for training in a state where noise is not removed,

denotes a training residual image corresponding to each of the training monochrome image and the training color image. and,

represents the L ₂ norm (L ₂ norm),

represents the index of the image channel. In addition,

represents the number of color channels, and when a monochrome image for learning is an input of the denoising network 300 ,

becomes 1.

Meanwhile, FIG. 9 shows a comparison between a disparity map generated from a stereo pair from which noise has been removed and a disparity map generated from a stereo pair from which noise has been removed according to an embodiment of the present invention. That is, in a state where there is a disparity GT map as in FIG. 9(a), as in FIG. 9(b), a disparity map generated from a stereo pair from which noise is removed and as in FIG. 9(c) Comparing the disparity map generated from the stereo pair from which the noise is not removed, compared to the disparity map generated from the stereo pair from which the noise has not been removed in FIG. In the disparity map generated from the stereo pair from which the noise has been removed, it can be seen that disparity information in some areas is missing. Therefore, a stereo pair consisting of a denoised color image for training and a denoised monochrome image for training from which noise has been removed produces an inaccurate disparity map than a stereo pair without noise removal because it is difficult to accurately match. A color image for training and a monochrome image for training in which is not removed are used.

Next, FIG. 10 schematically illustrates the structure of a colorization network 400 that generates a color reconstructed image using a stereo pair, a disparity map, and an occlusion map according to an embodiment of the present invention. .

In order to generate the final color reconstructed image, the learning apparatus 1000 causes the colorization network 400 to refer to the occlusion area information of the training occlusion map to at least one pixel feature value extracted from the initial color reconstructed image for training. Each region for generating a corrected pixel feature value by correcting a color bleeding error for By using at least some of the corrected pixel feature values as a color seed of , final color information for each pixel on the learning monochrome image may be restored to generate a final color restored image for training.

Accordingly, as shown in FIG. 10 , the learning device 1000 causes the colorization network 400 to control the initial color restoration image for learning, the occlusion map for learning, and the monochrome image for learning using the encoder of the colorization network 400 . After concatenation and encoding, the fourth feature map for learning is generated, and the fourth feature map for learning is decoded using a decoder of the colorization network 400 to generate the final color reconstructed image for learning. have. Here, referring to FIG. 10 , the colorization network 400 performs a convolution operation on each input instead of directly concatenating the initial color restoration image for training, the occlusion map for training, and the monochrome image for training to make the number of channels of each input the same. After that, concatenation can be performed. 10 shows a simplified structure of the colorization network 400 for convenience of illustration, but the colorization network 400 sequentially performs convolution-convolution-batch normalization with first to fourth convolutional blocks and deconvolution-convolution -Convolution-convolution-upsampling may be composed of a fifth convolution block to an eighth convolution block. In addition, a subsequent ReLU activation layer exists in all convolutional layers except for a layer that performs prediction. According to this structure, feature maps generated by the encoder of the colorization network 400 may be gradually halved in space and a feature dimension may be doubled. Conversely, feature maps generated by the decoder of the colorization network 400 may restore spatial resolution while halving feature dimensions. In addition, similar to the disparity network 100 , the colorization network 400 enables the decoder of the colorization network 400 to more accurately recover spatial resolution through a skip connection connecting the corresponding layers of the encoder and the decoder. can make it

As an example, referring to the example of FIG. 10, the learning apparatus 1000 applies a 3*3 convolution operation to each of the initial color restoration image for learning, the occlusion map for learning, and the monochrome image for learning through the encoder of the colorization network 400. Then, The fourth feature map for learning may be performed by concatenating and performing at least one 3*3 convolution operation, at least one batch normalization operation, and at least one bilinear interpolation operation. Then, the learning apparatus 1000 performs at least one 4*4 deconvolution operation on the fourth feature map for learning through the decoder of the colorization network 400, and the intermediate feature map output from the corresponding encoder layer. A catering operation, at least one 3*3 convolution operation, and at least one batch normalization operation may be performed.

)를 통해 학습용 복원 크로미넌스 값(

)과 GT 크로미넌스 값(

) 간의 유클리디안 거리(Euclidean Distance)를 줄임으로써 컬러화 네트워크(400)를 학습시킬 수 있다.In order to train the colorization network 400 configured as described above, the learning apparatus 1000 generates a fourth loss by referring to a restored chrominance value for learning of the final color restored image for training and a GT chrominance value corresponding thereto. can do. Specifically, the learning apparatus 1000 is a fourth loss (

) through the restoration chrominance value for training (

) and the GT chrominance value (

), the colorization network 400 can be trained by reducing the Euclidean distance.

[Equation 4]

Here, the learning apparatus 1000 obtains the Y channel of the YCbCr channel from the decolorized image obtained by decolorizing the color image located on the left side of the existing stereo pair used in the process of generating the learning monochrome image, and the learning color image The CbCr channel is obtained by converting the color image located on the right side of the existing stereo pair used in the process of creation to the YCbCr channel, and the above Y channel and CbCr channel are concatenated to convert back to the RGB channel. You can create a GT chrominance value from

In conclusion, the learning apparatus 1000 performs the disparity network 100 and occlusion using the summation loss generated by giving weights to the first loss, the second loss, the third loss, and the fourth loss generated as described above. At least one parameter included in the network 200 , the denoising network 300 , and the colorization network 400 may be learned.

[Equation 5]

,

,

각각은 하이퍼-파라미터(hyper-parameter)이며, 각각을 1, 100, 0.1과 같은 값으로 설정할 수 있다.here,

,

,

Each is a hyper-parameter, and each can be set to a value such as 1, 100, or 0.1.

As described above, a method for the test device to test the color and monochrome stereo network in a state where the color and monochrome stereo network for performing stereo matching is learned using the color image and the monochrome image can be described as follows.

First, FIG. 11 schematically shows a test apparatus for testing a color and monochrome stereo network that performs stereo matching using a color image and a monochrome image according to an embodiment of the present invention.

Referring to FIG. 11 , the test device 2000 includes a memory 2001 storing instructions for testing a color and monochrome stereo network performing stereo matching using a color image and a monochrome image, and instructions stored in the memory 2001 . The processor may include a processor 2002 for testing a color and monochrome stereo network that performs stereo matching using a color image and a monochrome image in correspondence with each other.

Specifically, test device 2000 is typically a computing device (eg, a device that may include a computer processor, memory, storage, input and output devices, other components of a conventional computing device; electronic devices such as routers, switches, etc.) A desired system using a combination of communication devices; electronic information storage systems such as network attached storage (NAS) and storage area networks (SANs)) and computer software (ie, instructions that cause the computing device to function in a particular way). performance may be achieved.

In addition, the processor of the computing device may include a hardware configuration such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, and a data bus. In addition, the computing device may further include an operating system and a software configuration of an application for performing a specific purpose.

However, a case in which the computing device includes an integrated processor in which a medium, a processor, and a memory are integrated for implementing the present invention is not excluded.

A method of testing a network for performing stereo matching using a color image and a monochrome image according to an embodiment of the present invention using the test apparatus 2000 according to an embodiment of the present invention configured as described above will be described below. same as Hereinafter, detailed description of parts that can be easily understood from the description with reference to FIGS. 3 to 10 will be omitted.

First, the test device 2000 is a color camera constituting a stereo camera in a state in which the disparity network 100 , the occlusion network 200 , the denoising network 300 , and the colorization network 400 are learned as described above. At least one test color and monochrome stereo pair including a test color image corresponding to , and a test monochrome image corresponding to a monochrome camera constituting the stereo camera may be acquired.

Next, the test device 2000 inputs the color and monochrome stereo pair for testing to the disparity network 100 to cause the disparity network 100 to perform a test color image and a test color image of the test color and monochrome stereo pair. According to the matching cost between the monochrome images, a disparity map for testing corresponding to the monochrome image for testing may be generated.

Specifically, the test device 2000 inputs a monochrome image for testing and a color image for testing to the disparity network 100 , and causes the disparity network 100 to include The test monochrome image is encoded through the 1_1 sub-encoder to generate a test 1_1 feature map for the test monochrome image, and the test color image is generated through the 1_2 sub-encoder included in the encoder of the disparity network 100 . Encoding to generate a test 1_2 feature map for a color image for testing, and a correlation layer included in the encoder of the disparity network 100 to code the code into the first 1_1 feature map for testing and 1_2 feature map for testing By performing a relation operation and calculating a matching cost, a correlation feature map for testing corresponding to a color and monochrome stereo pair for testing is generated, and a test code is generated through the second sub-encoder included in the encoder of the disparity network 100 . The relation feature map may be encoded to generate a second feature map for testing, and a disparity map for testing may be generated by decoding the second feature map for testing through a decoder of the disparity network 100 .

Here, the test apparatus 2000 causes the disparity network 100 to perform a test first patch corresponding to at least one first patch included in at least one first area on the test 1_1 feature map as a correlation operation. A correlation feature map for testing by performing a dot product operation on the feature vector and a second feature vector for testing corresponding to at least one second patch included in at least one second region on the first_2 feature map for testing corresponding to the first region can be made to create

In addition, the test device 2000 causes the disparity network 100 to generate a first intermediate feature map for testing included in the first_1 sub-encoder, the first_2 sub-encoder, and the second sub-encoder at least one encoding. A first intermediate feature map for testing generated from a predetermined encoding layer that is at least a part of the layers is included in the decoder to generate a second intermediate feature map for testing corresponding to the first intermediate feature map for testing Concatenate with a predetermined second intermediate feature map for a test generated in a predetermined decoding layer corresponding to a predetermined encoding layer among at least one decoding layer to generate at least one concatenation feature map for testing, , input each of the concatenation feature maps for testing into subsequent layers of each predetermined decoding layer corresponding to the predetermined second intermediate feature map for testing used to generate the concatenation feature map for testing. have. Here, the first intermediate feature map for testing may include a first_1 feature map for testing, a first_2 feature map for testing, and a second feature map for testing.

Next, the test device 2000 inputs a color and monochrome stereo pair for testing to the occlusion network 200 , and causes the occlusion network 200 according to the difference between the color image for testing and the monochrome image for testing. You can create an occlusion map for testing.

Specifically, the test device 2000 refers to the test disparity map and warps the test color image to generate a pixel value of each pixel on a warped color image for testing and a pixel value of each pixel on a monochrome image for testing. A difference map for testing is generated by calculating the difference between 200) by referring to the features extracted from the difference map for testing and the features of the second feature map for testing, and expressing the occlusion between the warped color image for testing and the monochrome image for testing as binary values for each pixel. You can create an occlusion map for

That is, the test device 2000 causes the occlusion network 200 to encode the difference map for testing through the encoder of the occlusion network 200 to generate a third feature map for testing, and the occlusion network ( 200), the third feature map for testing and the second feature map for testing are concatenated and then decoded to generate an occlusion map for testing.

In addition, the test device 2000 inputs a test color and monochrome stereo pair to the denoising network 300 , and causes the denoising network 300 to remove noise from each of the test color image and test monochrome image. You can create a denoised monochrome image for use and a denoised color image for testing.

Specifically, the test device 2000 causes the denoising network 300 to perform at least one convolution operation, at least one batch normalization operation, and at least one ReLU operation on each of the monochrome image for testing and the color image for testing. Denoise for test by acquiring a residual image for testing, and removing noise from the monochrome image for testing and color image for testing by referring to the residual image for testing corresponding to each of the monochrome image for testing and the color image for testing It can be configured to generate de-monochrome images and denoised color images for testing.

Next, the test device 2000 warps the denoised color image for the test by referring to the disparity map for the test to generate a warped denoised color image for the test, and includes the warped denoised color image for the test and The denoised monochrome image for the test is concatenated to generate an initial color restored image for the test, and the initial color restored image for the test, the occlusion map for the test, and the monochrome image for the test are input to the colorization network 400 for colorization The network 400 may generate a final color restored image for testing by reconstructing the color of the monochrome image for testing according to color information of the initial color restored image for testing with reference to the occlusion map. Here, the test device 2000 concatenates the warped denoised monochrome image for the test and the denoised color image for the test in the channel direction, so that the color information of the warped denoised color image for the test is chrominance. It is possible to generate an initial color restoration image for testing in which the U channel and V channel, which are (chrominance) channels, are configured, and the luminance information of the denoised monochrome image for testing constitutes the Y channel, which is a luminance channel.

Specifically, the test device 2000 causes the colorization network 400 to cause a color bleeding error with respect to at least one pixel feature value extracted from the initial color restoration image for testing with reference to the occlusion area information of the occlusion map for testing. at least among the corrected pixel feature values as a color seed of each region to generate a corrected pixel feature value by correcting Using a portion, the final color information for each pixel on the test monochrome image may be restored to generate the final color restored image for the test.

Accordingly, the test device 2000 causes the colorization network 400 to concatenate the initial color restoration image for testing, the occlusion map for testing, and the monochrome image for testing using the encoder of the colorization network 400 , and then The fourth feature map for testing is generated by encoding, and the fourth feature map for testing is decoded using a decoder of the colorization network 400 to generate a final color restored image for testing.

Meanwhile, FIGS. 12 to 15 show the performance of a Color and Monochrome Stereo Network (CMSNet) that performs stereo matching using a color image and a monochrome image according to an embodiment of the present invention.

First, FIG. 12 schematically shows a disparity map generated according to an embodiment of the present invention by comparing it with a disparity map generated from other various networks. Other networks used for comparison include ANCC (Heo et al 2011), JDMCC (Heo et al 2013), DASC (Kim et al) that perform multi-spectral/cross-channel stereo matching. 2015), CCNG (Holloway et al 2015), ITER (Jeon et al 2016), and DMC (Zhi et al 2018) networks. In addition, when the color and monochrome stereo network of the present invention is different from the configuration described above and a stereo pair consisting of only RGB images is used as an input (CMSNet RGB Pair), the encoder of the disparity network 100 When using concatenation layer instead of correlation layer (CMSNet Feat. denoise) was also compared.

In conclusion, as a result of testing the different network configurations as above with the FlyingThings3D data set, Middlebury data set, Monkaa data set, and KITTI data set, the disparity map generated with the configuration of the present invention produces the most similar result to the disparity GT map. can be seen doing

And, FIG. 13 schematically shows a color restored image generated according to an embodiment of the present invention by comparing it with color restored images generated from various other networks. Other networks used for comparison include BM3D (Dabov et al 2007), non-local means (Buades et al 2005), DnCNN (Zhang et al 2017a), and WAVG (Im et al. al 2019a) and ITER (Jeon et al 2016), a network that performs denoising on multiple images, were included. In addition, when the color and monochrome stereo network of the present invention is different from the configuration described above, and only the initial color restored image is generated as the final color restored image because the colorization network 400 is not used (CMSNet Init) Map), a case in which the final color reconstructed image is generated without using the occlusion map generated in the occlusion network 200 (CMSNet w/o occ) was also compared.

As a result of testing different network configurations as above with the FlyingThings3D data set, Middlebury data set, and Monkaa data set, it can be seen that the final color restored image created with the configuration of the present invention produces the result most similar to the GT color restored image. have.

Next, FIG. 14 schematically illustrates a comparison of a color image used as an input and a color restored image generated as an output according to an embodiment of the present invention. Here, FIG. 14 shows the results of actual data photographed under low-illuminance conditions at night. 14 (a) and 14 (b) show monochrome images and corresponding color images obtained by photographing different objects in different situations, and FIGS. 14 (d) and 14 (e) lists the final color restored image and disparity map generated by inputting a monochrome image and a corresponding color image to the color and monochrome stereo network of the present invention, and (c) of FIG. 14 shows the color image and the final color restored image The restoration effect is shown by comparing some areas of

According to FIG. 14, the color and monochrome stereo network of the present invention can vividly reproduce depth discontinuous characteristics and microstructures such as statues, plants, or golf clubs included in the image in the third row, etc. can be seen In particular, as can be seen from the images in columns 1 and 2, it is not very helpful for the cameras to perform automatic exposure on areas that are underexposed due to insufficient light, but it is not very helpful to recover these areas, but using the network configuration of the present invention. Thus, a brighter image than the input color image can be obtained by fusing the color and monochrome stereo pair and the disparity map. In addition, using the monochrome image to generate the luminance channel of the color reconstructed image helps to reduce the noise of the image and improve the resolution. It can be seen that the clear

In addition, FIG. 15 schematically illustrates an example of a disparity map generated from a stereo pair consisting of an RGB image and a near-infrared (NIR) image according to an embodiment of the present invention.

Referring to FIG. 15 , the color and monochrome stereo network of the present invention learned with the learning color image and the learning color image generated through data augmentation as above is converted into RGB image and near-infrared image without additional fine-tuning process. It has been shown that a disparity map can be effectively generated even for a configured stereo pair.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of the computer-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, and a magneto-optical medium such as a floppy disk. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for carrying out the processing according to the present invention, and vice versa.

In the above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , various modifications and variations can be devised from these descriptions by those of ordinary skill in the art to which the present invention pertains.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and not only the claims described below but also all modifications equivalently or equivalently to the claims described below belong to the scope of the spirit of the present invention. will do it

1000: learning device
1001: memory
1002: Processor
2000: test device
2001: Memory
2002: Processor
100: disparity network
200: occlusion network
300: denoising network
400: colorization network

Claims (28)

A method of performing stereo matching using a color image and a monochrome image, the method comprising:
(a) When at least one learning color and monochrome stereo pair including a learning color image corresponding to the color camera constituting the stereo camera and a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained, the learning device , (i) inputting the training color and monochrome stereo pair into a disparity network, causing the disparity network to match between the training color image and the training monochrome image of the training color and monochrome stereo pair generate a learning disparity map corresponding to the learning monochrome image according to the cost; to generate an occlusion map for learning according to the difference between the image and the monochrome image for learning, (iii) input the color and monochrome stereo pair for learning into a denoising network, causing the denoising network to generate a denoised monochrome image for learning and a denoised color image for learning from which noises are removed from each of the color image for training and the monochrome image for training;
(b) the learning device warps the denoised color image for learning with reference to (i) (i-1) the disparity map for learning to generate a warped denoised color image for learning and (i-2) concatenating the warped denoised color image for training and the denoised monochrome image for training to generate an initial color restored image for learning, (ii) the initial color restored image for training , input the learning occlusion map and the learning monochrome image to a colorization network, causing the colorization network to refer to the occlusion map and according to the color information of the initial color restoration image for learning, the learning monochrome image restoring the color of to generate a final color restored image for learning; and
(c) a first loss generated by the learning device with reference to the learning disparity map and a disparity GT (ground truth) map, and a second loss generated by referring to the learning occlusion map and occlusion GT map; The third loss generated by referring to the denoised monochrome image for learning, the denoised color image for learning, and the corresponding denoised GT image, and the restored chrominance value for learning of the final color restored image for learning; The disparity network, the occlusion network, the denoising network, and the colorization network are included in the disparity network, the occlusion network, and the colorization network using the summation loss generated by giving weights to the fourth loss generated by referring to the corresponding GT chrominance value. learning at least one parameter;
How to include. According to claim 1,
In step (a),
The learning apparatus inputs the learning monochrome image and the learning color image to a disparity network, causing the disparity network to perform (i) a first_1 sub-encoder included in an encoder of the disparity network. Encoding the learning monochrome image to generate a first_1 feature map for learning for the learning monochrome image, and encoding the color image for learning through the first_2 sub-encoder included in the encoder of the disparity network. generate a first_2 feature map for training with respect to a color image for training, and (ii) a first_1 feature map for training and a first_2 feature map for training through a correlation layer included in the encoder of the disparity network By performing a correlation operation to calculate the matching cost, a correlation feature map for training corresponding to the training color and monochrome stereo pair is generated, and (iii) a second sub-encoder included in the encoder of the disparity network. to generate a second feature map for learning by encoding the correlation feature map for learning through (iv) decoding the second feature map for learning through a decoder of the disparity network to generate the learning disparity map A method characterized in that it is made. 3. The method of claim 2,
The learning apparatus causes the disparity network to perform, as the correlation operation, a first feature vector for learning corresponding to at least one first patch included in at least one first area on the first_1 feature map for learning and the first feature vector for learning as the correlation operation. To generate the correlation feature map for learning by performing a dot product operation on a second feature vector for learning corresponding to at least one second patch included in at least one second region on the first_2 feature map for learning corresponding to region 1 How to characterize. 3. The method of claim 2,
The learning apparatus causes the disparity network to: (i) a first intermediate feature map for learning included in the first_1 sub-encoder, the first_2 sub-encoder, and the second sub-encoder - the The first intermediate feature map for training includes the first_1 feature map for training, the first_2 feature map for training, and the second feature map for training. The encoding of the predetermined first intermediate feature map for learning is included in the decoder to generate a second intermediate feature map for learning corresponding to the first intermediate feature map for learning, among at least one decoding layer. Concatenate with a predetermined second intermediate feature map for learning generated in a predetermined decoding layer corresponding to the layer to generate at least one concatenation feature map for learning, each of the concatenation feature maps for learning to input into a subsequent layer of each predetermined decoding layer corresponding to the predetermined second intermediate feature map for training used to generate the concatenation feature map for training. 3. The method of claim 2,
The learning apparatus includes: (i) a pixel value of each pixel on a warped color image for training generated by warping the training color image with reference to the training disparity map, and a pixel value of each pixel on the training monochrome image generating a difference map for learning by calculating the difference between occlusion between the warped color image for training and the monochromatic image for training with reference to the features extracted from the learning difference map and the features of the second learning feature map as binary values for each pixel. A method comprising generating a solution map. 6. The method of claim 5,
The learning apparatus causes the occlusion network to (i) encode the difference map for learning through an encoder of the occlusion network to generate a third feature map for learning, (ii) a decoder of the occlusion network Method characterized in that the third feature map for learning and the second feature map for learning are concatenated and then decoded to generate the occlusion map for learning. According to claim 1,
In step (a),
The learning device causes the denoising network to perform at least one convolution operation, at least one batch normalization operation, and at least one ReLU operation on each of the monochrome image for training and the color image for training. to predict the output obtained by removing the latent noiseless image for learning corresponding to each of the learning monochrome image and the learning color image from each of the learning monochrome image and the learning color image, and a residual image for learning obtained, and removing the noise from the learning monochrome image and the learning color image with reference to the learning residual image corresponding to each of the learning monochrome image and the learning color image to obtain the learning denoised monochrome image and the learning denoise image A method for generating a de-color image. According to claim 1,
In step (b),
The learning apparatus causes the colorization network to perform (i) color bleeding for at least one pixel feature value extracted from the initial color restoration image for training with reference to occlusion area information of the occlusion map for training. A color of each region for generating a corrected pixel feature value by correcting an error, and (ii) restoring a color of each region divided by at least one edge on the learning monochrome image used as a reference image The method according to claim 1, wherein the final color information for each pixel on the training monochrome image is restored using at least some of the corrected pixel feature values as a seed to generate the final color restored image for training. According to claim 1,
In step (b),
The learning apparatus causes the colorization network to (i) use an encoder of the colorization network to concatenate the initial color restoration image for learning, the occlusion map for learning, and the monochrome image for learning. Next, to generate a fourth feature map for learning by encoding, (ii) decoding the fourth feature map for learning using a decoder of the colorization network to generate the final color restored image for learning Way. According to claim 1,
In step (b),
The learning device concatenates the warped denoised monochrome image for training and the denoised color image for training in a channel direction, so that the color information of the warped denoised color image for training is chrominance Method characterized in that generating the initial color restoration image for training constituting the U channel and the V channel, which are channels, and the luminance information of the denoised monochrome image for training constituting the Y channel, which is a luminance channel. A method of performing stereo matching using a color image and a monochrome image, the method comprising:
(a) When at least one learning color and monochrome stereo pair including a learning color image corresponding to the color camera constituting the stereo camera and a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained, the learning device , (1) (i) input the training color and monochrome stereo pair to a disparity network, causing the disparity network to generate the training color image and the training monochrome image of the training color and monochrome stereo pair generate a learning disparity map corresponding to the learning monochrome image according to a matching cost between to generate an occlusion map for learning according to the difference between the color image for learning and the monochrome image for learning, and (iii) the color and monochrome stereo pair for learning to a denoising network (2) a process for causing the denoising network to generate a denoised monochrome image for training and a denoised color image for training in which noise in each of the training color image and the training monochrome image has been removed by input, (2) ( i) (i-1) warping the denoised color image for training with reference to the disparity map for training to generate a warped denoised color image for training, (i-2) for the training The warped denoised color image and the denoised monochrome image for learning are concatenated to generate an initial color restored image for learning, (ii) the initial color restored image for learning, the occlusion map for learning, and the learning for Monochrome image colorization network ( colorization network), the colorization network refers to the occlusion map and restores the color of the learning monochrome image according to the color information of the initial color restored image for training to generate a final color restored image for training , and (3) a first loss generated by referring to the disparity map for training and a disparity GT (ground truth) map, a second loss generated by referring to the occlusion map for learning and the occlusion GT map, and the D for training The third loss generated by referring to the noised monochrome image, the denoised color image for training, and the corresponding denoised GT image, and the restored chrominance value for learning of the final color restored image for training and corresponding values At least one included in the disparity network, the occlusion network, the denoising network, and the colorization network using the summation loss generated by assigning weights to the fourth loss generated by referring to the GT chrominance value In a state in which the process of learning the parameters is performed, the test device includes a test color image corresponding to the color camera constituting the stereo camera and a test monochrome image corresponding to the monochrome camera constituting the stereo camera obtaining at least one test color and monochrome stereo pair;
(b) the test device (i) inputs the test color and monochrome stereo pair to the disparity network, so that the disparity network causes the test color image and the test color image of the test color and monochrome stereo pair; generate a test disparity map corresponding to the test monochrome image according to a matching cost between the test monochrome images, (ii) input the test color and monochrome stereo pair into the occlusion network, and cause the occlusion network to generate a test occlusion map according to a difference between the test color image and the test monochrome image, and (iii) input the test color and monochrome stereo pair into the denoising network. causing the denoising network to generate a denoised monochrome image for testing and a denoised color image for testing from which noises are removed from each of the color image for testing and the monochrome image for testing; and
(c) the test device generates a warped denoised color image for testing by (i) (i-1) warping the denoised color image for testing with reference to the disparity map for testing, ( i-2) concatenating the warped denoised color image for the test and the denoised monochrome image for the test to generate an initial color restored image for the test, (ii) the initial color restored image for the test, the Input the occlusion map for test and the monochrome image for test into the colorization network, and cause the colorization network to refer to the occlusion map and select the color information of the initial color restoration image for test according to the color information of the test monochrome image. reconstructing the color to produce a final color reconstructed image for testing;
How to include. 12. The method of claim 11,
In step (b),
The test apparatus inputs the test monochrome image and the test color image to a disparity network to cause the disparity network to perform (i) a first_1 sub-encoder included in an encoder of the disparity network. encodes the test monochrome image through to generate a test 1_1 feature map for the test monochrome image, and uses the first_2 sub-encoder included in the encoder of the disparity network for the test encode a color image to generate a test first_2 feature map for the test color image, and (ii) the test first_1 through a correlation layer included in the encoder of the disparity network. By performing a correlation operation on the feature map and the first_2 feature map for testing to calculate the matching cost, a correlation feature map for testing corresponding to the color and monochrome stereo pair for testing is generated, and (iii) the disparity Encoding the correlation feature map for testing through a second sub-encoder included in the encoder of the network to generate a second feature map for testing, (iv) the test through a decoder of the disparity network and decoding a second feature map for use to generate the disparity map for testing. 13. The method of claim 12,
The test apparatus causes the disparity network to perform a first feature vector for testing corresponding to at least one first patch included in at least one first area on the first_1 feature map for testing as the correlation operation; The correlation feature map for testing by performing a dot product operation on a second feature vector for testing corresponding to at least one second patch included in at least one second region on the first_2 feature map for testing corresponding to the first region A method characterized in that to create a. 13. The method of claim 12,
The test apparatus includes: (i) a pixel value of each pixel on a warped color image for testing generated by warping the color image for testing with reference to the disparity map for testing, and each pixel value on the monochrome image for testing. a difference map for a test obtained by calculating a difference between pixel values of a pixel is generated; By inputting into , the occlusion network causes the occlusion between the warped color image for testing and the monochrome image for testing by referring to the features extracted from the difference map for testing and the features of the second feature map for testing. and generating the occlusion map for the test expressed as a binary value for each pixel. In the learning apparatus for performing stereo matching using a color image and a monochrome image,
a memory storing instructions for performing stereo matching using a color image and a monochrome image; and
a processor that performs an operation for performing stereo matching using a color image and a monochrome image according to the instructions stored in the memory;
including,
The processor, (I) at least one learning color and monochrome stereo pair comprising a learning color image corresponding to the color camera constituting the stereo camera and a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained , (i) inputting the training color and monochrome stereo pair into a disparity network, causing the disparity network to match between the training color image and the training monochrome image of the training color and monochrome stereo pair generate a learning disparity map corresponding to the learning monochrome image according to the cost; to generate an occlusion map for learning according to the difference between the image and the monochrome image for learning, (iii) input the color and monochrome stereo pair for learning into a denoising network, a process for causing the denoising network to generate a denoised monochrome image for training and a denoised color image for training in which noise in each of the training color image and the training monochrome image has been removed, (II) (i) ( i-1) warping the denoised color image for learning with reference to the disparity map for learning to generate a warped denoised color image for learning, (i-2) the warped denoised color image for learning The noised color image and the denoised monochrome image for learning are concatenated to generate an initial color restored image for training, and (ii) the initial color restored image for learning, the occlusion map for learning, and the learning monochrome image colorization network (c olorization network), the colorization network refers to the occlusion map and restores the color of the learning monochrome image according to the color information of the initial color restored image for training to generate a final color restored image for training , and (III) a first loss generated by referring to the disparity map for training and a disparity GT (ground truth) map, a second loss generated by referring to the occlusion map for training and the occlusion GT map, and the D for training The third loss generated by referring to the noised monochrome image, the denoised color image for training, and the corresponding denoised GT image, and the restored chrominance value for learning of the final color restored image for training and corresponding values At least one included in the disparity network, the occlusion network, the denoising network, and the colorization network using the summation loss generated by assigning weights to the fourth loss generated by referring to the GT chrominance value A learning device that performs the process of learning parameters. 16. The method of claim 15,
In the process (I) above,
The processor inputs the learning monochrome image and the learning color image to a disparity network, causing the disparity network to (i) through a 1_1 sub-encoder included in an encoder of the disparity network. Encoding a monochrome image for learning to generate a first_1 feature map for learning for the monochromatic image for learning, and encoding the color image for learning through the first_2 sub-encoder included in the encoder of the disparity network for the learning to generate a first_2 feature map for learning for a color image, and (ii) to the first_1 feature map for learning and the first_2 feature map for learning through a correlation layer included in the encoder of the disparity network By performing a correlation operation to calculate the matching cost, a correlation feature map for learning corresponding to the learning color and monochrome stereo pair is generated, and (iii) through a second sub-encoder included in the encoder of the disparity network. Encoding the correlation feature map for learning to generate a second feature map for learning, (iv) decoding the second feature map for learning through a decoder of the disparity network to generate the learning disparity map Learning device, characterized in that. 17. The method of claim 16,
The processor is configured to cause the disparity network to perform, as the correlation operation, a first feature vector for learning corresponding to at least one first patch included in at least one first area on the first_1 feature map for learning and the first and generating the correlation feature map for learning by performing a dot product operation on a second feature vector for learning corresponding to at least one second patch included in at least one second region on the first_2 feature map for learning corresponding to the region. learning device with 17. The method of claim 16,
The processor is configured to cause the disparity network to: (i) a first intermediate feature map for learning included in the first_1 sub-encoder, the first_2 sub-encoder and the second sub-encoder - for the learning The first intermediate feature map includes the first_1 feature map for training, the first_2 feature map for training, and the second feature map for training. The predetermined encoding layer among at least one decoding layer that includes a predetermined first intermediate feature map for learning in the decoder to generate a second intermediate feature map for learning corresponding to the first intermediate feature map for learning Concatenate with a predetermined second intermediate feature map for learning generated in a predetermined decoding layer corresponding to , to generate at least one concatenation feature map for learning, and each of the concatenation feature maps for learning and input to a subsequent layer of each predetermined decoding layer corresponding to the predetermined second intermediate feature map for learning used to generate the concatenation feature map for learning. 17. The method of claim 16,
The processor is configured to: (i) between a pixel value of each pixel on a warped color image for training generated by warping the color image for training with reference to the disparity map for training and a pixel value of each pixel on the monochrome image for training A difference map for learning is generated by calculating the difference, and (ii) the second feature map for learning and the difference map for learning used to generate the learning disparity map are input to the occlusion network to the occlusion network. The training occlusion in which the occlusion between the warped color image for training and the monochrome image for training is expressed as a binary value for each pixel by referring to the feature extracted from the difference map for learning and the feature of the second feature map for learning. A learning device, characterized in that it generates a map. 20. The method of claim 19,
The processor causes the occlusion network to (i) encode the learning difference map through an encoder of the occlusion network to generate a third feature map for learning, and (ii) a decoder of the occlusion network. Learning apparatus, characterized in that the third feature map for learning and the second feature map for learning are concatenated and then decoded to generate the occlusion map for learning. 16. The method of claim 15,
In the process (I) above,
The processor causes the denoising network to perform at least one convolution operation, at least one batch normalization operation, and at least one ReLU operation on each of the monochrome image for training and the color image for training. From each of the learning monochrome image and the learning color image, the output obtained by removing the latent noiseless image for learning corresponding to each of the learning monochrome image and the learning color image is predicted. A residual image for learning is obtained. and removing the noise from the learning monochrome image and the learning color image with reference to the learning residual image corresponding to each of the learning monochrome image and the learning color image, and the denoised monochrome image for learning and the denoised for learning A learning device, characterized in that it generates a color image. 16. The method of claim 15,
In the process (II) above,
The processor is configured to cause the colorization network to: (i) a color bleeding error for at least one pixel feature value extracted from the initial color restoration image for training with reference to occlusion region information of the training occlusion map to generate a corrected pixel feature value, and (ii) a color seed of each region for reconstructing the color of each region divided by at least one edge on the learning monochrome image used as a reference image (seed) by using at least a part of the corrected pixel feature values to restore final color information for each pixel on the learning monochrome image to generate the final color restored image for training. 16. The method of claim 15,
In the process (II) above,
The processor causes the colorization network to (i) concatenate the initial color restoration image for learning, the occlusion map for learning, and the monochrome image for learning using an encoder of the colorization network. Encoding to generate a fourth feature map for learning, (ii) decoding the fourth feature map for learning using a decoder of the colorization network to generate the final color restored image for learning Device. 16. The method of claim 15,
In the process (II) above,
The processor concatenates the warped denoised monochrome image for training and the denoised color image for training in a channel direction, so that the color information of the warped denoised color image for training is a chrominance channel Learning apparatus, characterized in that generating the initial color restoration image for training constituting the U channel and the V channel, and the luminance information of the denoised monochrome image for training constituting the Y channel which is a luminance channel. In the test apparatus for performing stereo matching using a color image and a monochrome image,
a memory storing instructions for performing stereo matching using a color image and a monochrome image; and
a processor that performs an operation for performing stereo matching using a color image and a monochrome image according to the instructions stored in the memory;
including,
The processor, (I) at least one learning color and monochrome stereo pair comprising a learning color image corresponding to the color camera constituting the stereo camera and a learning monochrome image corresponding to the monochrome camera constituting the stereo camera is obtained , a learning device, (1) (i) input the training color and monochrome stereo pair to a disparity network, and cause the disparity network to generate the training color image and the training color image of the training color and monochrome stereo pair to generate a learning disparity map corresponding to the learning monochrome image according to a matching cost between the learning monochrome images, (ii) input the learning color and monochrome stereo pair into an occlusion network, and Let the solution network generate an occlusion map for training according to the difference between the training color image and the training monochrome image, and (iii) denoising the training color and monochrome stereo pair with a denoising network ( A process of causing the denoising network to generate a denoised monochrome image for learning and a denoised color image for learning in which noise from each of the learning color image and the learning monochrome image has been removed by input to the denoising network; (2) (i) (i-1) warping the denoised color image for training with reference to the disparity map for training to generate a warped denoised color image for training, (i- 2) generating an initial color restored image for learning by concatenating the warped denoised color image for learning and the denoised monochrome image for learning, (ii) the initial color restored image for learning, the occlusion for learning Map and the learning monochrome image Input to a colorization network, causing the colorization network to restore the color of the monochrome image for learning according to the color information of the initial color restoration image for training with reference to the occlusion map to generate a final color restoration image for training (3) a first loss generated by referring to the disparity map for learning and a disparity GT (ground truth) map, and a second loss generated by referring to the occlusion map for learning and the occlusion GT map; The third loss generated by referring to the denoised monochrome image for learning, the denoised color image for learning, and the corresponding denoised GT image, and the restored chrominance value for learning of the final color restored image for learning; The disparity network, the occlusion network, the denoising network, and the colorization network are included in the disparity network, the occlusion network, and the colorization network using the summation loss generated by assigning weights to the fourth loss generated by referring to the corresponding GT chrominance value. In a state in which the process of learning at least one parameter is performed, a test color image corresponding to the color camera constituting the stereo camera and a test monochrome image corresponding to the monochrome camera constituting the stereo camera are included a process of obtaining at least one test color and monochrome stereo pair, (II) (i) inputting the test color and monochrome stereo pair into the disparity network, causing the disparity network to: generate a test disparity map corresponding to the test monochrome image according to a matching cost between the test color image and the test monochrome image of the monochrome stereo pair; (ii) the test color and monochrome stereo pair; is input to the occlusion network, causing the occlusion network to generate a test occlusion map according to the difference between the monochrome images for the image, (iii) input the test color and monochrome stereo pair to the denoising network, so that the denoising network causes the test color a process for generating a denoised monochrome image for testing and a denoised color image for testing in which noises in the image and each of the monochrome images for testing are removed, and (III) (i) (i-1) the diss for testing warping the denoised color image for the test with reference to the parity map to generate a warped denoised color image for the test, (i-2) the warped denoised color image for the test and the denoised for the test concatenating the de-monochrome image to generate an initial color restored image for testing, (ii) inputting the initial color restored image for testing, the occlusion map for testing, and the monochrome image for testing into the colorization network, wherein the A test apparatus performing a process of causing a colorization network to generate a final color restored image for testing by reconstructing the color of the monochrome image for test according to color information of the initial color restored image for testing with reference to the occlusion map . 26. The method of claim 25,
In the process (II) above,
The processor inputs the test monochrome image and the test color image to a disparity network to cause the disparity network to perform (i) a first_1 sub-encoder included in an encoder of the disparity network. Encodes the monochrome image for testing through a first_1 feature map for testing with respect to the monochrome image for testing, and uses the first_2 sub-encoder included in the encoder of the disparity network for the test color encode an image to generate a first_2 feature map for testing for the color image for testing, and (ii) the first_1 feature for testing through a correlation layer included in the encoder of the disparity network. By performing a correlation operation on the map and the first_2 feature map for testing to calculate the matching cost, a correlation feature map for testing corresponding to the color and monochrome stereo pair for testing is generated, and (iii) the disparity network encodes the correlation feature map for testing through a second sub-encoder included in the encoder of The test apparatus according to claim 1, wherein the second feature map is decoded to generate the disparity map for the test. 27. The method of claim 26,
The processor is configured to cause the disparity network to perform, as the correlation operation, a first feature vector for testing corresponding to at least one first patch included in at least one first area on the first_1 feature map for testing and the The correlation feature map for testing is obtained by performing a dot product operation on a second feature vector for testing corresponding to at least one second patch included in at least one second region on the first_2 feature map for testing corresponding to the first region. A test device, characterized in that to generate. 27. The method of claim 26,
The processor is configured to: (i) a pixel value of each pixel on a warped color image for testing generated by warping the color image for testing with reference to the disparity map for testing and each pixel on the monochrome image for testing generate a difference map for testing by calculating the difference between pixel values of By inputting, the occlusion network performs occlusion between the warped color image for testing and the monochrome image for testing by referring to the features extracted from the difference map for testing and the features of the second feature map for testing. A test device, characterized in that it generates the occlusion map for the test expressed as a binary value for a pixel.

Images