KR20210082901A - Method for raw-to-rgb mapping using two-stage u-net with misaligned data, recording medium and device for performing the method - Google Patents

Abstract

A mapping method from raw to RGB using misaligned data based on a two-step U-net structure comprises the steps of: obtaining a misaligned data set of an input raw image and an output RGB image from different cameras; generating a first U-net neural network model consisting of a plurality of pooling layers and a plurality of upsampling layers; learning the data set through the first U-net neural network model using a loss function defined as a sum of pixel loss, feature loss, and color loss between an output image and a target image; generating a second U-net neural network model composed of a plurality of pooling layers and a plurality of upsampling layers; and by using the loss function, learning the result of the learning through the first U-net neural network model through the second U-net neural network model learning, and performing alignment in data pairs of the raw image and the RGB image. Accordingly, a convolutional neural network model can be trained to generate an RGB image with DSLR-level quality by using an image obtained from a raw sensor as an input.

Description

A method of mapping from raw to RGB using misaligned data based on a two-step U-Net structure, and a recording medium and device for performing the same. DEVICE FOR PERFORMING THE METHOD}

The present invention relates to a raw to RGB mapping method using misaligned data based on a two-step U-Net structure, a recording medium and an apparatus for performing the same, and more particularly, a raw Bayer obtained from a smartphone camera based on deep learning. It relates to a method of mapping an image to an RGB image acquired with a DSLR camera.

[Description of state-funded R&D]

This study was conducted with the support of the Information and Communication Planning and Evaluation Institute with the funding of the government (Ministry of Science and ICT) in 2019 (No. 2014-3-00077, global multi-object of interest tracking and situation by large-scale real-time video analysis) development of predictive technology).

Smartphone cameras are developing rapidly, but due to physical limitations (small sensor size, small lens size, hardware cost, etc.), images of lower quality than those shot with a DSLR camera are obtained.

Therefore, research on improving the image quality of an image captured by a smartphone camera has been recently conducted. Some smartphones support saving the raw image data acquired from the camera sensor. These raw images are converted into RGB images through an image signal processor mounted on the camera.

However, the prior art uses only RGB images in order to improve the quality of mobile camera images to the level of a DSLR camera. Accordingly, the RGB image is processed by raw sensor data through an image signal processor (ISP) already mounted in the camera, and there is a problem in that data is lost.

US 8,508,612 B2 KR 10-1996730 B1 US 2017/0270122 A1

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” in International Journal of Computer Vision (IJCV), 60(2):91-110, 2004. E. Schwartz, R. Giryes and A. M. Bronstein, "DeepISP: Toward Learning an End-to-End Image Processing Pipeline," in IEEE Transactions on Image Processing, vol. 28, no. 2, pp. 912-923, Feb. 2019.

Accordingly, the technical problem of the present invention was conceived in this regard, and an object of the present invention is to provide a mapping method from raw to RGB using misaligned data based on a two-step U-Net structure.

Another object of the present invention is to provide a recording medium in which a computer program is recorded for performing a raw-to-RGB mapping method using misaligned data based on the two-step U-Net structure.

Another object of the present invention is to provide an apparatus for performing a raw-to-RGB mapping method using misaligned data based on the two-step U-Net structure.

The raw to RGB mapping method using misaligned data based on two-step U-Net structure according to an embodiment for realizing the object of the present invention is a mismatched data set of input raw image and output RGB image from different cameras obtaining a; Generating a first U-Net neural network model consisting of a plurality of pooling (Pooling) layers and a plurality of upsampling (Upsampling) layers; learning the data set through the first U-Net neural network model using a loss function defined as a sum of pixel loss, feature loss, and color loss between an output image and a target image; generating a second U-Net neural network model composed of a plurality of pooling layers and a plurality of upsampling layers; and using the loss function, learning the result learned through the first U-Net neural network model through learning the second U-Net neural network model, and aligning the raw image and RGB image data pairs into pairs. includes ;

In an embodiment of the present invention, the step of learning through the first U-Net neural network model may include defining the loss function using a difference between an output image and a target image for the data set.

In an embodiment of the present invention, the step of defining the loss function includes: calculating a pixel loss that is a pixel difference between an output image and a target image using the data set; calculating a feature loss for extracting a feature from a perceptual loss based on features above a preset level; and calculating a color loss from the cosine distance between the RGB color vectors.

In an embodiment of the present invention, the step of calculating a feature loss for extracting a feature from the perceptual loss based on the features of the preset level or higher may extract a feature from a Rectified Linear Unit (ReLU) 4 or higher layer.

In an embodiment of the present invention, the calculating of the color loss from the cosine distance between the RGB color vectors may include reducing the image by 2 times.

In an embodiment of the present invention, the raw-to-RGB mapping method using the misaligned data based on the two-step U-Net structure is based on the learned data pair of the raw image and the RGB image, from the raw image in real time. The method may further include outputting an RGB image.

In an embodiment of the present invention, the first U-Net neural network model and the second U-Net neural network model may include a channel attention mechanism and a long skip connection.

In a computer-readable storage medium according to an embodiment for realizing another object of the present invention, a computer program for performing a raw-to-RGB mapping method using misaligned data based on a two-step U-Net structure is recorded has been

A raw to RGB mapping device using misaligned data based on a two-step U-Net structure according to an embodiment for realizing another object of the present invention is a plurality of pooling layers and a plurality of upsampling ( Upsampling) a first U-Net neural network model composed of layers; a second U-Net neural network model consisting of a plurality of pooling layers and a plurality of upsampling layers, and continuous with the first U-Net neural network model to form a W-Net structure; a loss function unit defining a loss function defined as a sum of pixel loss, feature loss, and color loss between the output image and the target image; And by using the loss function to learn the mismatched data set of the input raw image and the output RGB image from different cameras through the first U-Net neural network model and the second U-Net neural network model, the raw image and the RGB image and an image learning unit that aligns data pairs of

In an embodiment of the present invention, the loss function unit includes: a pixel loss unit for calculating a pixel loss that is a pixel difference between an output image and a target image using the data set; a feature loss unit calculating a feature loss for extracting features from a perceptual loss based on features above a preset level; and a color loss unit that calculates color loss from the cosine distance between RGB color vectors.

In an embodiment of the present invention, the feature loss unit may extract features from a rectified linear unit (ReLU) 4 or higher layer, and the color loss unit may reduce an image by 2 times.

In an embodiment of the present invention, the raw-to-RGB mapping device using the misaligned data based on the two-step U-Net structure is based on the learned raw image and the data pair of the RGB image, It may further include an image converter for outputting an RGB image.

In an embodiment of the present invention, the first U-Net neural network model and the second U-Net neural network model may include a channel attention mechanism and a long skip connection.

According to the raw to RGB mapping method using mismatched data based on the two-step U-Net structure, the U-Net neural network model is trained by shooting the same scene with different cameras, and then the U-Net neural network model is additionally added to the output. Apply and learn Accordingly, it is possible to convert an image taken with a smartphone into a quality comparable to that of a DSLR image.

1 is a block diagram of a raw-to-RGB mapping apparatus using misaligned data based on a two-step U-Net structure according to an embodiment of the present invention.
2 is a diagram showing an example of a misaligned raw image and an RGB image.
3 is a conceptual diagram showing the structure of one neural network used in the present invention.
4 is a conceptual diagram showing a two-step W-Net structure formed by the first U-Net neural network model and the second U-Net neural network model of the present invention.
FIG. 5 is a detailed block diagram of the loss function unit of FIG. 1 .
6 is a flowchart of a raw-to-RGB mapping method using misaligned data based on a two-step U-Net structure according to an embodiment of the present invention.
7 is a table showing a performance comparison according to the improvement of the network structure of the present invention.
8 is a table showing a performance comparison according to the loss function combination of the present invention.
9 is a table showing performance comparison in the data set of the present invention.
10 is a diagram showing an example of a result according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0023] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be embodied in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a raw-to-RGB mapping apparatus using misaligned data based on a two-step U-Net structure according to an embodiment of the present invention.

A raw-to-RGB mapping device (10, hereinafter device) using mismatched data based on a two-step U-Net structure according to the present invention is a data pair that is misaligned to obtain an RGB image of DSLR-level quality from a raw image taken with a mobile device. It is a learning period for learning pairs.

Referring to FIG. 1 , an apparatus 10 according to the present invention includes a first U-Net neural network model 200 , a second U-Net neural network model 400 , a loss function unit 100 and an image learning unit 500 . includes In an embodiment, the device 10 may further include an image converter (not shown).

The first U-Net neural network model 200 and the second U-Net neural network model 400 are each composed of a plurality of pooling layers and a plurality of upsampling layers, and are connected to each other to W- Net structure is formed.

The loss function unit 100 defines a loss function defined as the sum of pixel loss, feature loss, and color loss between the output image and the target image.

The image learning unit 500 uses the loss function to input raw images and output RGB images from different cameras through the first U-Net neural network model 200 and the second U-Net neural network model 400 . It learns the misaligned data set of , and sorts it into data pairs of raw and RGB images.

As an embodiment, the input raw image may be an image captured by a smartphone, and the output RGB image may be an image captured by a high-quality DSLR camera.

The image converter may output the RGB image from the raw image in real time based on the learned data pair of the raw image and the RGB image. For example, you can output a raw image shot with a smartphone at the level of an image shot with a DSLR camera.

In the device 10 of the present invention, software (application) for performing mapping from raw to RGB using misaligned data based on a two-step U-Net structure may be installed and executed, and the first U-Net neural network model ( 200), the configuration of the second U-Net neural network model 400, the loss function unit 100 and the image learning unit 500 is based on the two-step U-Net structure executed in the device 10. It can be controlled by software to perform mapping from raw to RGB using misaligned data.

The device 10 may be a separate terminal or a module of the terminal. In addition, the configuration of the first U-Net neural network model 200, the second U-Net neural network model 400, the loss function unit 100 and the image learning unit 500 is formed as an integrated module, It may consist of one or more modules. However, on the contrary, each configuration may be formed of a separate module.

The device 10 may be movable or stationary. The apparatus 10 may be in the form of a server or an engine, and may be a device, an application, a terminal, a user equipment (UE), a mobile station (MS), or a wireless device. (wireless device), may be called other terms such as a handheld device (handheld device).

The device 10 may execute or manufacture various software based on an operating system (OS), that is, the system. The operating system is a system program for software to use the hardware of the device, and is a mobile computer operating system such as Android OS, iOS, Windows Mobile OS, Bada OS, Symbian OS, Blackberry OS, and Windows series, Linux series, Unix series, It can include all computer operating systems such as MAC, AIX, and HP-UX.

Conventional technologies used only RGB images to improve the quality of mobile camera images to the level of DSLR cameras. However, RGB images are processed by raw sensor data through an image signal processor (ISP) already installed in the camera, and data loss may have occurred.

Therefore, in the present invention, a convolutional neural network model that generates an RGB image of DSLR-level quality is learned by using an image obtained from a raw sensor as an input.

Also, in general, SIFT keypoint matching is used to align pairs of images captured by different cameras, which are also not perfectly aligned due to the presence of errors.

2 is a diagram showing an example of a misaligned raw image and an RGB image. Referring to FIG. 2 , the image taken with a smartphone on the left and the same scene with a DSLR camera on the right, or a vertical shift phenomenon was observed. As such, when the same scene is photographed with different cameras, the images cannot be completely identical.

In order to solve the problem of such a data set, the present invention uses a neural network structure and loss function different from the existing ones. A detailed description thereof will be described with reference to the drawings below.

3 is a conceptual diagram showing the structure of one neural network used in the present invention. 4 is a conceptual diagram showing a two-step W-Net structure formed by the first U-Net neural network model and the second U-Net neural network model of the present invention.

First, the neural network structure used in the present invention is a U-Net structure, and two neural network models are used. That is, as shown in FIG. 4 , a neural network structure in which the first U-Net neural network model 200 and the second U-Net neural network model 400 are connected is formed.

3 shows only one neural network structure for convenience. Referring to FIG. 3 , U-Net consists of a plurality of pooling layers and upsampling layers, so it is effective when learning with misaligned data.

The present invention improves the performance by further modifying the U-Net structure. To this end, more useful features are extracted by applying a channel attention mechanism as shown in FIG. 3 (CA-Convs part). In addition, the learning of the neural network is facilitated by adding a long skip connection.

As shown in Figure 4, the present invention consists of two steps by successively attaching two identically improved U-Nets, and this is called W-Net. In the learning process, the first U-Net 200 is learned using the loss function proposed in the present invention, and after the learning is completed, the second U-Net 400 is learned using the same loss function.

FIG. 5 is a detailed block diagram of the loss function unit of FIG. 1 .

Referring to FIG. 5 , the loss function unit 100 converts the data set into a pixel loss unit 110 for calculating a pixel loss that is a pixel difference between an output image and a target image, and a perceptual ( perceptual) includes a feature loss unit 130 for calculating a feature loss that extracts features from the loss, and a color loss unit 150 for calculating a color loss from a cosine distance between RGB color vectors.

In an embodiment, the loss function unit 100 may further include a loss synthesis unit 170 that defines a final loss function by summing the pixel loss, the feature loss, and the color loss.

와 타겟 영상

의 차이를 비교할 때, 데이터의 정렬에 덜 가변적이고 색상 차이에 더 민감한 손실 함수를 사용한다.As already described, the loss function proposed in the present invention consists of a combination of three losses. In the present invention, the output image of the neural network model

and target video

When comparing the differences in , we use a loss function that is less variable in the alignment of the data and more sensitive to color differences.

손실을 사용한다. 픽셀 손실인

은 아래의 수학식 1과 같이 정의된다.First, the basic pixel loss

use loss. loss of pixels

is defined as in Equation 1 below.

[Equation 1]

However, due to data misalignment, if only pixel loss is used, a blurry result image is obtained, so the following loss function is further used.

Second, we use perceptual loss based on high-level features as feature loss. Because high-level features are generated through multiple down-sampling, they are less variable in data discrepancy.

In the present invention, features can be extracted from the previously learned rule4_1 or rule5_1 layer of VGG-19. If the feature loss is used together with the pixel loss, the resulting image can be obtained with a sharper quality than when only the pixel loss is used. Equation 2 below defines the feature loss.

[Equation 2]

는 VGG-19의 rule4_1 또는 rule5_1 계층을 나타낸다.here,

represents the rule4_1 or rule5_1 layer of VGG-19.

Third, we use color loss to effectively learn the color conversion from raw to RGB. Color loss is defined as the cosine distance between RGB color vectors. If you train the neural network by combining color loss, you can get a more accurate color result image. Equation 3 below defines color loss.

[Equation 3]

는 영상의 2배 축소를 의미하고,

는

번째 픽셀에 대한 인덱스이다.here,

means 2x reduction of the image,

is

It is the index of the th pixel.

In summary, the loss function proposed in the present invention is defined as the sum of the pixel loss, the feature loss, and the color loss, and is expressed as Equation 4 below.

[Equation 4]

As described above, in the present invention, the first U-Net neural network model 200 is learned using the newly derived loss function, and when the learning is completed, the same loss function is used in the second U-Net neural network model 400 . to learn

As a result of this learning, alignment data of a data pair of a raw image and an RGB image can be acquired, and an RGB image can be generated from the raw image in real time by using the learned data pair of the raw image and the RGB image. can be printed out.

6 is a flowchart of a raw-to-RGB mapping method using misaligned data based on a two-step U-Net structure according to an embodiment of the present invention.

The raw-to-RGB mapping method using misaligned data based on the two-step U-Net structure according to the present embodiment may be performed in substantially the same configuration as the device 10 of FIG. 1 .

Accordingly, the same components as those of the device 10 of FIG. 1 are given the same reference numerals, and repeated descriptions are omitted. In addition, the mapping method from raw to RGB using mismatched data based on the two-step U-Net structure according to this embodiment is software (application) for mapping from raw to RGB using misaligned data based on the two-step U-Net structure. can be executed by

In the present invention, a convolutional neural network model that generates an RGB image of DSLR-level quality is learned by using an image obtained from a raw sensor as an input.

Referring to FIG. 6 , the raw-to-RGB mapping method using mismatched data based on the two-step U-Net structure according to the present embodiment first acquires mismatched data sets of input raw images and output RGB images from different cameras. do (step S10).

As an embodiment, the input raw image may be an image captured by a smartphone, and the output RGB image may be an image captured by a high-quality DSLR camera.

A first U-Net neural network model consisting of a plurality of pooling layers and a plurality of upsampling layers is generated (step S20).

In the present invention, the first U-Net neural network model and the second U-Net neural network model are each composed of a plurality of pooling layers and a plurality of upsampling layers, and are connected to each other to form a W-Net structure. do.

The present invention improves the performance by further modifying the U-Net structure. To this end, more useful features are extracted by applying a channel attention mechanism as shown in FIG. 3 (CA-Convs part). In addition, the learning of the neural network is facilitated by adding a long skip connection.

First, the data set is learned through the first U-Net neural network model by using a loss function defined as the sum of pixel loss, feature loss, and color loss between an output image and a target image (step S30).

와 타겟 영상

의 차이를 비교할 때, 데이터의 정렬에 덜 가변적이고 색상 차이에 더 민감한 손실 함수를 사용한다.The loss function proposed in the present invention consists of a combination of three losses. In the present invention, the output image of the neural network model

and target video

When comparing the differences in , we use a loss function that is less variable in the alignment of the data and more sensitive to color differences.

손실을 사용한다. 픽셀 손실인

은 상기 수학식 1과 같이 정의된다. 그러나, 데이터의 어긋남 때문에, 픽셀 손실만을 사용하면 흐릿한 결과 영상을 얻게 되므로 아래의 손실 함수를 더 이용한다.First, the basic pixel loss

use loss. loss of pixels

is defined as in Equation 1 above. However, due to data misalignment, if only pixel loss is used, a blurry result image is obtained, so the following loss function is further used.

Second, we use perceptual loss based on high-level features as feature loss. Because high-level features are generated through multiple down-sampling, they are less variable in data discrepancy.

In the present invention, features can be extracted from the previously learned rule4_1 or rule5_1 layer of VGG-19. If the feature loss is used together with the pixel loss, the resulting image can be obtained with a sharper quality than when only the pixel loss is used. Equation 2 above defines the feature loss.

Third, we use color loss to effectively learn the color conversion from raw to RGB. Color loss is defined as the cosine distance between RGB color vectors. If you train the neural network by combining color loss, you can get a more accurate color result image. Equation 3 above defines color loss.

In conclusion, the loss function proposed in the present invention is defined as the sum of the pixel loss, the feature loss, and the color loss, and is expressed as Equation 4 above.

A second U-Net neural network model composed of a plurality of pooling layers and a plurality of upsampling layers is generated (step S50).

The first U-Net neural network model is trained using the newly derived loss function, and when the learning is completed, the second U-Net neural network model is trained using the same loss function.

In other words, by using the loss function, the result learned through the first U-Net neural network model is learned through the second U-Net neural network model learning, and the data pairs of the raw image and the RGB image are arranged. do (step S60).

Accordingly, based on the learned data pair of the raw image and the RGB image, it is possible to output the RGB image from the raw image in real time. For example, you can output a raw image shot with a smartphone at the level of an image shot with a DSLR camera.

Hereinafter, experimental results for evaluating the performance of the present invention are described.

An RGB image with DSLR-level quality is obtained from the input raw image taken with a smartphone. The neural network model developed in the present invention was trained and tested using the Zurich RAW2RGB (ZRR) dataset published in the advanced in image manipulation (AIM) raw-to-RGB mapping challenge in the International conference on computer vision (ICCV). . Performance evaluation indicators use peak signal to noise ratio (PSNR) and structural similarity index (SSIM).

7 shows the performance improvement according to the improvement of the network structure, and FIG. 8 shows the performance improvement according to the combination of the loss functions. Through this, it can be confirmed that the performance of the neural network structure and loss function proposed in the present invention is greatly improved.

9 is a table showing performance comparison in the data set of the present invention.

Referring to FIG. 9 , it can be seen that the present invention obtained higher performance compared to other techniques participating in the challenge. The highest performance was achieved in terms of PSNR and SSIM. Another evaluation index, the mean opinion score (MOS), is a subjective evaluation by a human, and the first place came in second place with a slight difference.

10 is a view showing an example of a result according to the present invention, and it can be confirmed that an RGB image of a DSLR-level quality is obtained from a raw image taken with a mobile device.

This method of mapping from raw to RGB using mismatched data based on the two-step U-Net structure is implemented as an application or in the form of a program command that can be executed through various computer components and recorded on a computer-readable recording medium. can be The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer-readable recording medium are specially designed and configured for the present invention, and may be known and available to those skilled in the computer software field.

Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although described above with reference to the embodiments, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention described in the claims below You will understand.

The present invention can obtain alignment data of a data pair of a raw image and an RGB image that is misaligned, and using the learned data pair of a raw image and an RGB image, from a smartphone image captured in real time It can output DSLR-quality images.

10: Raw to RGB mapping device
100: loss function part
200: first U-Net neural network model
400: second U-Net neural network model
500: video learning unit
110: pixel loss part
130: feature loss part
150: color loss part
170: lossy synthesis unit

Claims (13)

obtaining a mismatched data set of an input raw image and an output RGB image from different cameras;
Generating a first U-Net neural network model consisting of a plurality of pooling (Pooling) layers and a plurality of upsampling (Upsampling) layers;
learning the data set through the first U-Net neural network model using a loss function defined as a sum of pixel loss, feature loss, and color loss between an output image and a target image;
generating a second U-Net neural network model composed of a plurality of pooling layers and a plurality of upsampling layers; and
using the loss function, learning the result learned through the first U-Net neural network model through the second U-Net neural network model learning, and aligning the raw image and the RGB image data pair; A mapping method from raw to RGB using misaligned data based on a two-step U-Net structure, including
According to claim 1, wherein the step of learning through the first U-Net neural network model,
A mapping method from raw to RGB using misaligned data based on a two-step U-Net structure, comprising the step of defining the loss function using the difference between the output image and the target image in the data set.
3. The method of claim 2, wherein defining the loss function comprises:
calculating a pixel loss that is a pixel difference between an output image and a target image using the data set;
calculating a feature loss for extracting a feature from a perceptual loss based on features above a preset level; and
A method of mapping from raw to RGB using misaligned data based on a two-step U-Net structure, including; calculating a color loss from the cosine distance between RGB color vectors.
The method of claim 3 , wherein calculating the feature loss for extracting features from the perceptual loss based on features above the preset level comprises:
ReLU (Rectified Linear Unit) A mapping method from raw to RGB using misaligned data based on a two-step U-Net structure that extracts features from 4 or higher layers.
4. The method of claim 3, wherein calculating the color loss from the cosine distance between the RGB color vectors comprises:
A mapping method from raw to RGB using misaligned data based on a two-step U-Net structure that reduces the image by 2 times.
According to claim 1,
Mapping from raw to RGB using mismatched data based on a two-step U-Net structure, further comprising outputting an RGB image from a raw image in real time based on a data pair of the learned raw image and RGB image Way.
According to claim 1,
The first U-Net neural network model and the second U-Net neural network model are misaligned data based on a two-step U-Net structure, including a channel attention mechanism and a long skip connection. Mapping method from raw to RGB using .
8. The method according to any one of claims 1 to 7,
A computer-readable storage medium in which a computer program is recorded for performing the raw-to-RGB mapping method using the misaligned data based on the two-step U-Net structure.
A first U-Net neural network model consisting of a plurality of pooling layers and a plurality of upsampling layers;
a second U-Net neural network model consisting of a plurality of pooling layers and a plurality of upsampling layers, and continuous with the first U-Net neural network model to form a W-Net structure;
a loss function unit defining a loss function defined as a sum of pixel loss, feature loss, and color loss between the output image and the target image; and
Using the loss function, the first U-Net neural network model and the second U-Net neural network model are used to learn the mismatched data set of the input raw image and the output RGB image from different cameras. A mapping device from raw to RGB using misaligned data based on a two-step U-Net structure, including; an image learning unit that aligns data pairs.
The method of claim 9, wherein the loss function unit,
a pixel loss unit for calculating a pixel loss that is a pixel difference between an output image and a target image using the data set;
a feature loss unit calculating a feature loss for extracting features from a perceptual loss based on features above a preset level; and
A color loss unit that calculates color loss from the cosine distance between RGB color vectors; a raw to RGB mapping device using misaligned data based on a two-step U-Net structure.
11. The method of claim 10,
The feature loss unit extracts features from a ReLU (Rectified Linear Unit) 4 or higher layer, and the color loss unit reduces the image by a factor of 2 and uses mismatched data based on a two-step U-Net structure. Raw to RGB mapping device .
10. The method of claim 9,
Based on the learned raw image and the data pair of the RGB image, it further includes an image converter that outputs an RGB image from a raw image in real time. mapping device.
10. The method of claim 9,
The first U-Net neural network model and the second U-Net neural network model are misaligned data based on a two-step U-Net structure, including a channel attention mechanism and a long skip connection. Raw to RGB mapping device using .

Images