KR20230165676A - Method for correcting image by using neural network, and computing apparatus executing neural network for correcting image - Google Patents

Abstract

A method of correcting an image using a neural network model according to an embodiment of the present disclosure includes the steps of a processor of a computing device inputting an input image and a correction parameter to a first neural network model, and the processor Obtaining an inference image corresponding to the correction parameter from the first neural network model, wherein the first neural network model includes the inference image output when the input image and the correction parameter are input. and, a model learned to minimize the difference between label images corresponding to the correction parameters, wherein the label image corresponding to the correction parameters is, when the correction parameters are applied to at least one image correction algorithm, Since the input image is an image corrected by at least one image correction algorithm, and the first neural network model learns the relationship between the correction parameter and the label image, the image output from the first neural network model is changed by changing the correction parameter. Control of the inferred image is possible.

Description

Method for correcting an image using a neural network model and a computing device executing a neural network model for image correction {METHOD FOR CORRECTING IMAGE BY USING NEURAL NETWORK, AND COMPUTING APPARATUS EXECUTING NEURAL NETWORK FOR CORRECTING IMAGE}

The present disclosure relates to a method of correcting an image using a neural network model and a computing device that executes the neural network model for image correction.

Due to recent developments in neural network technology, neural networks are being used in various fields. In particular, processing speed and image characteristics can be improved by utilizing neural networks in the field of image signal processing to perform image correction or restoration.

For example, in mobile terminals such as smartphones, a program is installed to correct images taken with an equipped camera. If a program implemented to include existing image correction algorithms can be replaced with a neural network, there are advantages in various aspects. You can expect it.

A method of correcting an image using a neural network model, disclosed as a technical means for achieving a technical problem, includes the steps of a processor of a computing device inputting an input image and a correction parameter to a first neural network model. And a step of the processor acquiring an inference image corresponding to the correction parameter from the first neural network model, wherein the first neural network model outputs an inference image when the input image and the correction parameter are input. A model learned to minimize the difference between the inferred image and a label image corresponding to the correction parameter, and the label image corresponding to the correction parameter is a model in which the correction parameter has been applied to at least one image correction algorithm. When the input image is an image corrected by the at least one image correction algorithm, and the first neural network model learns the relationship between the correction parameter and the label image, the first neural network model is changed by changing the correction parameter. It is possible to control the inferred image output from .

A computing device executing a neural network model for image correction, disclosed as a technical means for achieving a technical task, includes an input/output interface for receiving input from a user, a memory storing a program for correcting an input image, and a processor; , the processor executes the program, inputs an input image and correction parameters to a first neural network model, and obtains an inferred image corresponding to the correction parameters from the first neural network model, and the first neural network model is configured to: It is a model learned to minimize the difference between the inferred image output when the input image and the correction parameter are input, and the label image corresponding to the correction parameter, and the label image corresponding to the correction parameter has the correction parameter at least When applied to one image correction algorithm, the input image is an image corrected by the at least one image correction algorithm, and the first neural network model learns the relationship between the correction parameter and the label image, so that the correction parameter It is possible to control the inferred image output from the first neural network model by changing .

A computer-readable recording medium disclosed as a technical means for achieving a technical problem may store a program for executing at least one of the embodiments of the disclosed method on a computer.

A computer program disclosed as a technical means for achieving a technical problem may be stored in a medium for performing at least one of the embodiments of the disclosed method on a computer.

FIG. 1 is a diagram for explaining the operation of a first neural network model 100 according to an embodiment of the present disclosure.
FIG. 2 is a diagram for explaining the operation of the second neural network model 200 according to an embodiment of the present disclosure.
FIG. 3 is a diagram for explaining a process of training the first neural network model 100 according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating a process of learning the second neural network model 200 according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating a process in which the label image generation module 300 generates the label image 3 from the input image 1 according to an embodiment of the present disclosure.
FIG. 6 is a diagram for explaining the strength value 11 applied to the contrast ratio improvement algorithm 310 according to an embodiment of the present disclosure.
FIG. 7 is a diagram for explaining the gamma value 12 applied to the brightness correction algorithm 320 according to an embodiment of the present disclosure.
Figure 8 is applied to the exposure fusion algorithm 330 according to an embodiment of the present disclosure. This is a drawing to explain the value (13).
FIG. 9 is a diagram illustrating a method of applying a correction parameter as an input to a first neural network model, according to an embodiment of the present disclosure.
FIG. 10 is a diagram schematically showing components included in the mobile terminal 1000 according to an embodiment of the present disclosure.
Figure 11 is a diagram schematically showing the detailed configuration of an ISP according to an embodiment of the present disclosure.
FIG. 12 is a diagram illustrating a specific embodiment in which neural network models 100 and 200 according to an embodiment of the present disclosure are included in the image correction module 1013b of FIG. 10.
13 to 15 are flowcharts illustrating a method of correcting an image using a neural network model according to embodiments of the present disclosure.
FIG. 16 is a diagram illustrating a change in an inferred image when a correction parameter input to a first neural network model is changed according to an embodiment of the present disclosure.
FIG. 17 is a diagram illustrating a situation in which a user adjusts correction parameters for brightness adjustment through a UI displayed on the screen of a mobile terminal 1000, according to an embodiment of the present disclosure.
FIG. 18 is a block diagram illustrating the configuration of a computing device for performing learning on neural network models 100 and 200, according to an embodiment of the present disclosure.
FIG. 19 is a diagram illustrating an embodiment in which the cloud server 1900 corrects an image received from the mobile terminal 1000 using the neural network models 100 and 200, according to an embodiment of the present disclosure. .
20 and 21 are flowcharts for explaining a method of training the first neural network model 100 and the second neural network model 200, respectively, according to embodiments of the present disclosure.
Figures 22 and 23 are diagrams for explaining a method of updating a neural network model according to an embodiment of the present disclosure.

In describing the present disclosure, description of technical content that is well known in the technical field to which the present disclosure belongs and that is not directly related to the present disclosure will be omitted. This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanation. In addition, the terms described below are terms defined in consideration of the functions in the present disclosure, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

For the same reason, some components are exaggerated, omitted, or schematically shown in the attached drawings. Additionally, the size of each component does not entirely reflect its actual size. In each drawing, identical or corresponding components are assigned the same reference numbers.

The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure is complete and to fully inform those skilled in the art of the disclosure of the scope of the disclosure. One embodiment of the present disclosure may be defined according to the claims. Like reference numerals refer to like elements throughout the specification. Additionally, when describing an embodiment of the present disclosure, if it is determined that a detailed description of a related function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of the functions in the present disclosure, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

In one embodiment, each block of the flow diagram diagrams and combinations of the flow diagram diagrams may be performed by computer program instructions. Computer program instructions may be mounted on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, and the instructions performed through the processor of the computer or other programmable data processing equipment may perform the functions described in the flowchart block(s). You can create the means to carry them out. The computer program instructions may also be stored in a computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular way. The instructions may also produce manufactured items containing instruction means that perform the functions described in the flow diagram block(s). Computer program instructions may also be mounted on a computer or other programmable data processing equipment.

Additionally, each block in a flowchart diagram may represent a module, segment, or portion of code containing one or more executable instructions for executing specified logical function(s). In one embodiment, it is possible for functions mentioned in blocks to occur out of order. For example, two blocks shown in succession may be performed substantially simultaneously or may be performed in reverse order depending on their functions.

The term '~unit' used in an embodiment of the present disclosure may refer to software or hardware components such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and '~unit' refers to a specific role. can be performed. Meanwhile, '~ part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. In one embodiment, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, May include subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided through specific components or specific '~parts' can be combined to reduce their number or separated into additional components. Additionally, in one embodiment, ‘˜part’ may include one or more processors.

Embodiments of the present disclosure relate to a method of correcting an image using a neural network model. Before describing specific embodiments, the meaning of terms frequently used in this specification are defined.

An 'input image' is an image that is input to a neural network model, and an 'inference image' is an image that the neural network model outputs by correcting the input image. According to one embodiment, the input image may be a linear RGB image generated from a plurality of raw images. According to one embodiment, 'raw image' is an image output from an image sensor (e.g. CCD sensor, CMOS sensor, etc.), and may mean an image in Bayer format with only one color channel per pixel. . Additionally, according to one embodiment, one linear RGB image can be obtained by using temporal information of the raw images and performing demosaicing and denoising on a plurality of raw images taken before and after a specific point in time. You can.

‘Image characteristic’ refers to the brightness, contrast, and color temperature of the image.

‘Image correction’ refers to the operation of adjusting the characteristics of an image, and ‘image correction algorithm’ refers to an algorithm for adjusting the characteristics of an image.

‘Correction parameter’ refers to a parameter applied to an image correction algorithm when correcting an image using an image correction algorithm. In other words, the degree to which image characteristics are adjusted can be determined according to the value of the correction parameter. Specific examples of correction parameters will be described below.

A ‘label image’ is an image used as training data to perform supervised learning on a neural network model according to embodiments of the present disclosure, and in particular, ground truth data (ground truth data). This refers to the image used as truth data. The 'label image generation module' is a component that generates a label image by correcting an input image using at least one image correction algorithm.

The neural network model according to embodiments of the present disclosure may include a 'first neural network model' that corrects the input image and outputs an inferred image, and a 'second neural network model' that infers the optimal correction parameter for a given input image. You can. How the first neural network model and the second neural network model can be used in an actual device will be described in detail below with reference to FIGS. 10 and 11.

One. Description of the basic operation of the first neural network model and the second neural network model

First, with reference to FIGS. 1 and 2, the roles of the first neural network model 100 and the second neural network model 200 will be briefly described, and then with reference to FIGS. 3 and 4, the first neural network to perform those roles will be described briefly. How to train the model 100 and the second neural network model 200 will be described.

FIG. 1 is a diagram for explaining the operation of a first neural network model 100 according to an embodiment of the present disclosure. Referring to FIG. 1, when the first neural network model 100 according to one embodiment receives an input image 1 and a correction parameter 10, it outputs an inferred image 2 corresponding to the correction parameter 10. You can. At this time, the inferred image (2) corresponds to an image in which the input image (1) has been corrected based on the correction parameter (10). How the first neural network model 100 is learned to output such an inferred image 2 will be described in detail below with reference to FIG. 3.

In general, the input image (1) may have a problem in which the entire or partial area of the image is dark or the contrast ratio is low, making it difficult for objects in the image to be recognized. The first neural network model 100 may serve to increase the brightness or improve the contrast ratio of the input image 1 to solve this problem.

The first neural network model 100 may output the inferred image 2 as shown in FIG. 1, but does not output the image itself and uses a filter ( filter) or map information can also be output. That is, the input image 1 may be converted into the inferred image 2 using the filter or map information output by the first neural network model 100.

The first neural network model 100 may be implemented to include various types of deep learning networks, for example, it may be implemented as ResNet (Residual Network), a type of convolution neural network (CNN). It is not limited to this.

FIG. 2 is a diagram for explaining the operation of the second neural network model 200 according to an embodiment of the present disclosure. Referring to FIG. 2, the second neural network model 200 according to one embodiment may receive the input image 1, infer the correction parameter 10, and output it. The correction parameter 10 output from the second neural network model 200 is input to the first neural network model 100, and the first neural network model 100 is input to the correction parameter 10 as previously described with reference to FIG. 1. The corresponding inferred image (2) can be output.

The second neural network model 200 can infer a correction parameter 10 for correcting the input image 1 to have image characteristics that the user may like. At this time, the direction in which the input image 1 is corrected due to the correction parameter 10 inferred by the second neural network model 200 may be determined during the learning process of the second neural network model 200. Details are provided below. This will be described with reference to FIG. 4 .

The second neural network model 200 may also be implemented to include various types of deep learning networks, for example, it may be implemented as ResNet (Residual Network), a type of convolution neural network (CNN). It is not limited to this.

2. (Learning process) Method of training the first neural network model and the second neural network model

Hereinafter, a method of training the first neural network model 100 and the second neural network model 200 will be described with reference to FIGS. 3 and 4.

According to one embodiment, learning of the neural network models 100 and 200 is performed not by a device on which the neural network models 100 and 200 are mounted (e.g. the mobile terminal 1000 of FIG. 10) but by another external device (e.g. FIG. 18). It may be performed by a computing device 1800). Of course, depending on the embodiment, a device on which the neural network models 100 and 200 are mounted may perform learning of the neural network models 100 and 200. Hereinafter, for convenience of explanation, the processor 1830 of the computing device 1800 of FIG. 18 executes a program stored in the memory 1820 to learn the neural network models 100 and 200 described in FIGS. 3 and 4. Assume that you are performing: That is, the operations performed by the label image generation module 300 or the optimizer 400 and the calculation of the loss function in FIGS. 3 and 4 are actually performed by the processor ( 1830) can be seen as carrying it out.

The processor 1830 of the computing device 1800 of FIG. 18 may include a CPU 1831, a GPU 1832, and an NPU 1833, and executes a program stored in the memory 1820 using at least one of these. By doing so, learning about the neural network model (100, 200) can be performed. The input/output interface 1810 of the computing device 1800 may receive commands or display information related to learning of the neural network models 100 and 200.

FIG. 3 is a diagram for explaining a process of training the first neural network model 100 according to an embodiment of the present disclosure. Referring to FIG. 3 , the label image generation module 300 may output a label image 3 by correcting the input image 1 using at least one image correction algorithm. When the label image creation module 300 corrects the input image 1, the correction parameter 10 is applied to the image correction algorithm. The image correction algorithm used by the label image creation module 300, and the correction parameter 10 This will be described in detail with reference to FIGS. 5 to 8.

FIG. 5 is a diagram illustrating a process in which the label image generation module 300 generates the label image 3 from the input image 1 according to an embodiment of the present disclosure. Referring to FIG. 5, the label image generation module 300 may use a contrast enhancement algorithm 310, a brightness correction algorithm 320, and an exposure fusion algorithm 330. . However, the image correction algorithms used by the label image generation module 300 disclosed in FIG. 5 are merely examples, and the label image generation module 300 may also use various types of image correction algorithms.

FIG. 5 shows which parameters are specifically included in the correction parameter 10 and which image correction algorithm each parameter corresponds to. Referring to FIG. 5, a strength value (11) may be applied to the contrast ratio improvement algorithm 310, a gamma value (12) may be applied to the brightness correction algorithm 320, and the exposure fusion algorithm ( 330), the α value (13) can be applied. The parameters 11, 12, and 13 applied to each image correction algorithm will be described with reference to FIGS. 6 to 8.

FIG. 6 is a diagram for explaining the strength value 11 applied to the contrast ratio improvement algorithm 310 according to an embodiment of the present disclosure. The contrast ratio improvement algorithm 310 according to one embodiment is a method of adjusting the contrast ratio using a curve.

Referring to the graph 600 of FIG. 6, the curvature of the S-curve changes depending on the strength value 11, and in particular, as the strength value 11 increases, the curvature of the S-curve increases. Able to know. As the slope of the S-curve increases, the tone is stretched, thus increasing the contrast ratio. Therefore, as the strength value 11 increases, the contrast ratio improvement algorithm 310 can increase the contrast ratio of the input image 1.

FIG. 7 is a diagram for explaining the gamma value 12 applied to the brightness correction algorithm 320 according to an embodiment of the present disclosure. The graph 700 of FIG. 7 shows a gamma curve according to the gamma value (12). The x-axis of the graph 700 represents the contrast expression value, and the y-axis represents brightness. Referring to the graph 700 of FIG. 7, it can be seen that as the gamma value 12 decreases, the brightness increases. Accordingly, as the gamma value 12 becomes smaller, the brightness correction algorithm 320 can increase the brightness of the input image 1.

FIG. 8 is a diagram for explaining the α value 13 applied to the exposure fusion algorithm 330 according to an embodiment of the present disclosure. The exposure fusion algorithm 330 generates a processed image (X') with different exposure from the base image (X), and multiplies the base image (X) and the processed image (X') by weights W and (1-W), respectively. After combining them, a fused image (F) can be output. According to one embodiment, the base image (X) may correspond to the input image (1) of FIG. 5, and the output fused image (F) may correspond to the label image (3) of FIG. 5.

The exposure fusion algorithm 330 can generate a processed image (X') from the basic image (X) according to Equation 1 below.

If the exposure fusion algorithm 330 is performed according to the method described above, the larger the α value (13), the brighter the fusion image (F) can be compared to the basic image (X), and the smaller the β value, the brighter the fusion image (F) can be. may become brighter compared to the basic image (X). In Figure 5, the α value (13) is shown as being used as a correction parameter for the exposure fusion algorithm 330, but the β value is used as a correction parameter instead of the α value (13), or both the α value (13) and the β value are used as the correction parameter. may also be used as a correction parameter.

The label image generation module 300 adjusts at least one of the characteristics of the input image 1 by applying the correction parameters 11, 12, and 13 to the image correction algorithms 310, 320, and 330 described above. Video (3) can be output. The label image generation module 300 may use a combination of at least one of the image correction algorithms 310, 320, and 330 shown in FIG. 5, or may use another type of image correction algorithm not shown in FIG. 5. there is.

According to one embodiment, the order of the parameter values 11, 12, and 13 included in the correction parameter 10 may remain the same during learning and inference. In other words, according to one embodiment, the order of the parameter types 11, 12, and 13 in the correction parameter 10 used during learning may remain the same even during use.

For example, if the correction parameter 10 is in the form of a column vector, the elements of each row are the same type of parameter (parameters applied to the same image correction algorithm) during learning and inference. ) may apply.

To explain with a specific example, it is as follows.

As in the embodiment shown in FIG. 5, the element of the first row of the correction parameter (column vector) 10 used during learning is the parameter 11 applied to the contrast ratio improvement algorithm 310, and the two If the element in the third row was the parameter 12 applied to the brightness correction algorithm 320, and the element in the third row was the parameter 13 applied to the exposure fusion algorithm 330, the correction parameter used during inference would also be the first The element in the first row is the parameter 11 applied to the contrast ratio improvement algorithm 310, the element in the second row is the parameter 12 applied to the brightness correction algorithm 320, and the element in the third row is the exposure fusion algorithm. It may be parameter 13 applied to 330.

In relation to this, description is given with reference to FIG. 9 as follows.

Referring to FIG. 9, the two parameter values 11 and 12 are converted to the input image 1 and the grayscale image converted from the input image 1 and scaled concatenation (see 1201 in FIG. 12) and then applied to the first neural network. It is input into model 100. That is, grayscale images in which the parameter values 11 and 12 are multiplied (or divided or added) are attached to the input image 1 in the channel direction and input to the first neural network model 100. At this time, the channel order is the same as during learning. Since they must be the same during inference, as described above, the order of the parameter values 11, 12, and 13 included in the correction parameter 10 must remain the same during learning and during inference.

Additionally, according to one embodiment, the number of bits of each of the parameter values 11, 12, and 13 included in the correction parameter 10 may be kept the same during learning and inference. A detailed description of an embodiment in which the order of the parameter values 11, 12, and 13 included in the correction parameter 10 and the number of bits of each of the parameter values 11, 12, and 13 are kept the same during learning and inference. If you do so, it is as follows.

It is assumed that the correction parameter 10 used when learning the neural network models 100 and 200 is binary data of “01010110”. In order from the beginning, 3 bits of data (“010”) are parameters 11 to be applied to the contrast ratio improvement algorithm 310, and the following 2 bits of data (“10”) are used to apply the brightness correction algorithm 320. It is assumed that this is the parameter 12 to be applied, and the last 3 bits of data (“110”) are the parameters 13 to be applied to the exposure fusion algorithm 330.

The number of bits of each parameter 11, 12, and 13 and the order between the parameters 11, 12, and 13 must remain the same even when using (inferring) the neural network models 100 and 200. For example, if binary data of "10111100" is input as the correction parameter 10 to the first neural network model 100 during inference, 3 bits of data ("101") are input to the contrast ratio improvement algorithm 310 in order from the beginning. ), the next 2 bits of data ("11") become the parameters 12 to be applied to the brightness correction algorithm 320, and the last 3 bits of data ("100") becomes a parameter 13 to be applied to the exposure fusion algorithm 330.

Returning to FIG. 3 , the label image 3 output by the label image generation module 300 may be used as training data (in particular, ground truth data) for learning the first neural network model 100. When the input image 1 and the correction parameter 10 are input to the first neural network model 100, the inferred image 2 output from the first neural network model 100 is made to be as similar as possible to the label image 3. The first neural network model 100 can be trained. That is, according to one embodiment, the first neural network model 100 includes an inference image 2 output when the input image 1 and the correction parameter 10 are input, and a label image corresponding to the correction parameter 10 ( 3) It may be a model learned to minimize the differences between At this time, the label image 3 corresponding to the correction parameter 10 means that when the correction parameter 10 is applied to at least one image correction algorithm, the input image 1 is corrected by at least one image correction algorithm. It could be a video.

In other words, the optimizer 400 can update the first neural network model 100 so that the loss value of the loss function 20 representing the difference between the inference image 2 and the label image 3 is minimized. there is. At this time, the loss function 20 may be composed of a combination of Mean Absolute Error (MAE), Mean Square Error (MSE), and SSIM (Structural Similarity Index Measure).

As described above, when learning the first neural network model 100, the input image 1, the label image 3, and the correction parameter 10 can be used as training data. According to one embodiment, a plurality of label images (3) are generated by changing the correction parameter (10) for several input images (1), and the input image (1), label image (3) and correction collected in this way are The first neural network model 100 can be learned by using a combination of the parameters 10 as training data.

In this way, when learning the first neural network model 100, the correction parameter 10 is used as an input to the first neural network model 100, so that the relationship between the correction parameter 10 and the label image 3 is established by using the correction parameter 10 as an input to the first neural network model 100. The model 100 can be trained. In other words, the first neural network model 100 can be viewed as learning how to generate the corrected label image 3 when a correction parameter 10 is applied to the input image 1. If the input image 1 input to the learned first neural network model 100 is kept the same and only the correction parameter 10 is changed, the inferred image 2 output from the first neural network model 100 is also changed. Accordingly, the user can control the inferred image 2 output from the first neural network model 100 to have desired image characteristics by adjusting the correction parameter 10 input to the first neural network model 100. A specific embodiment of applying the correction parameter 10 as an input to the first neural network model 100 will be described below with reference to FIG. 9.

FIG. 9 is a diagram illustrating a method of applying a correction parameter as an input to a first neural network model, according to an embodiment of the present disclosure. In the embodiment shown in Figure 9, it is assumed that the correction parameter 10 includes a first parameter 11 and a second parameter 12.

Referring to FIG. 9, the input image 1 is converted into a gray scale image by the decolorize module 500. The first operator 911 performs multiplication on the gray-scale image and the first parameter 11, and the second operator 912 performs multiplication on the gray-scale image and the second parameter 12. . The third operator 920 performs concatenation on the input image 1, the output of the first operator 911, and the output of the second operator 912 and outputs it to the first neural network model 100. can do. That is, according to one embodiment, the parameters 11 and 12 included in the correction parameter 10 are respectively applied to the grayscale image converted from the input image 1, and then used together with the input image 1 to form a first neural network model ( 100) can be entered.

Meanwhile, in FIG. 9, it is shown that the grayscale image is multiplied by the first parameter 11 and the second parameter 12 and then concatenated on the input image. However, according to one embodiment, the grayscale image is converted to a grayscale image using an operation other than multiplication. It is also possible to process and parameters (11, 12). For example, the grayscale image may be divided into the first and second parameters 11 and 12 and then concatenated into the input image, or the first and second parameters 11 and 12 may be added to the grayscale image. You can then concatenate the input image.

FIG. 4 is a diagram illustrating a process of learning the second neural network model 200 according to an embodiment of the present disclosure.

First, the reason for using the second neural network model 200 will be explained.

The second neural network model 200 can generally infer correction parameters for correcting the input image 1 to have image characteristics that many users would consider good (e.g. image characteristics determined to be optimal by the designer of the neural network model).

Therefore, by using the second neural network model 200, correction parameters for correcting the input image 1 to have optimal image characteristics are automatically generated (inferred) even if the user does not set or adjust the correction parameters every time, Accordingly, the input image 1 can be corrected and presented to the user.

For example, when a user captures an image through a terminal equipped with neural network models 100 and 200, the captured image is corrected according to the correction parameter inferred by the second neural network model 200 and previewed on the terminal. It can be displayed on the screen.

Therefore, the user of the terminal can first check the image with good image characteristics (brightness, contrast ratio, etc.) through a preview, and adjust the correction parameters to change the image characteristics only if the preview is not to their liking.

Referring to FIG. 4, a second neural network model 200 that receives the input image 1, infers the correction parameter 10, and outputs it has been added. As shown in FIG. 4, with both the first neural network model 100 and the second neural network model 200 connected, learning on the two neural network models 100 and 200 may be performed simultaneously, and the first neural network model 100 may be trained first, and the second neural network model 200 may be trained while the learned first neural network model 100 is fixed (the parameters included in the first neural network model 100 are fixed). Additionally, the first neural network model 100 and the second neural network model 200 may be trained separately.

When learning the second neural network model 200, the measured characteristic value 40, which quantitatively quantifies the characteristics of the label image 3 (e.g. brightness, contrast ratio, color temperature, etc.), is compared with the preset target characteristic value 50. , the second neural network model 200 can be updated so that the difference between the two is minimized. The target characteristic value 50 can be set in advance to a value desired by the user (administrator). That is, according to one embodiment, the second neural network model 200 is an image in which the correction parameter 10 inferred by the second neural network model 200 when the input image 1 is input and the label image 3 are preset. It may be a model learned to minimize the difference between correction parameters that enable the label image generation module 300 to output an image corresponding to the target feature value 50 in FIG. 4.

In other words, the optimizer 400 uses the second neural network model 200 to minimize the loss value of the second loss function 30, which represents the difference between the measured characteristic value 40 and the target characteristic value 50. can be updated. At this time, the second loss function 30 may be composed of a combination of Mean Absolute Error (MAE), Mean Square Error (MSE), and Structural Similarity Index Measure (SSIM).

According to one embodiment, the measurement characteristic value 40 and the target characteristic value 50 used when learning the second neural network model 200 may include a plurality of values corresponding to each of a plurality of image characteristics. For example, the measured characteristic value 40 and the target characteristic value 50 may include a first characteristic value that quantitatively quantifies the brightness of the image, and a second characteristic value that quantitatively quantifies the color temperature of the image.

In the embodiment shown in FIG. 4, the measurement characteristic value 40 for the label image 3 is acquired for learning the second neural network model 200, and the obtained measurement characteristic value 40 is used as a preset target characteristic. Compare with value (50). However, according to one embodiment, the measurement characteristic value 40 for the image generated in the middle of the process of converting the input image 1 to the label image 3 (hereinafter referred to as 'intermediate label image') is obtained, , the second neural network model 200 may be trained by comparing the obtained measured characteristic value 40 with a preset target characteristic value 50.

A specific example will be described with reference to FIG. 5 as follows.

According to one embodiment, when measuring a specific value for learning the second neural network model 200, some image correction algorithms among the plurality of image correction algorithms 310, 320, and 330 included in the label image generation module 300 The second neural network model 200 may be trained by measuring feature values for the image to which only the image (middle label image) is applied, and comparing the measured feature values with preset target feature values.

In Figure 5, when the input image 1 is input to the label image generation module 300, the input image 1 goes through the contrast ratio improvement algorithm 310, the brightness correction algorithm 320, and the exposure fusion algorithm 330 in that order. Assume that it is converted to a label image (3).

For example, a measurement that quantitatively quantifies the 'brightness' of the intermediate label image (the input image to the exposure fusion algorithm 330) to which only the contrast improvement algorithm 310 and the brightness correction algorithm 320 have been applied to the input image 1. Characteristic values may be acquired, and the second neural network model 200 may be trained so that the difference (loss) between the acquired measured characteristic value and the preset target characteristic value for 'brightness' is minimized. When learning the second neural network model 200 in this way, the parameter 13 applied to the exposure fusion algorithm 330 is set to minimize separate loss regardless of the target brightness (brightness corresponding to the target characteristic value). It can be learned.

According to one embodiment, the second neural network model 200 may include a plurality of neural network models. For example, when the second neural network model 200 infers the first correction parameter and the second correction parameter, a separate neural network model for inferring each correction parameter may exist. And, if necessary, the second neural network model 200 may be changed to infer the first to third correction parameters by adding a neural network model for inferring the third correction parameter to the second neural network model 200.

3. (Usage process) How to correct an image using the first neural network model and the second neural network model

A method of correcting an image using the first neural network model 100 and the second neural network model 200 learned according to the method described above will be described. First, with reference to FIGS. 10 and 11 , we will describe how the neural network models 100 and 200 according to embodiments of the present disclosure can be used in actual devices, and then use the neural network models 100 and 200 to This explains how to correct images.

FIG. 10 is a diagram schematically showing components included in the mobile terminal 1000 according to an embodiment of the present disclosure. In Figure 10, the device on which the first neural network model 100 and the second neural network model 200 are mounted is shown as a mobile terminal 1000, but it is not limited to this and can be used in all types of devices with an image capture function, such as a camera. The neural network models 100, 200 can be mounted, and furthermore, the neural network models 100, 200 can be mounted on various types of devices that do not have an image capture function but can perform image signal processing. Self-explanatory.

Referring to FIG. 10, the mobile terminal 1000 may include a camera module 1010 and a main processor 1020.

The camera module 1010 may include a lens module 1011, an image sensor 1012, and an ISP (Image Signal Processor) 1013, and these components (1011, 1012, and 1013) form one chip. It can also be implemented in the form of a SoC (System on Chip) mounted on a .

The image sensor 1012 receives light transmitted through the lens module 1011 and outputs an image, and the ISP 1013 can restore and correct the image output from the image sensor 1012 and output it as a final image. That is, when a user captures an image through the mobile terminal 1000, the final image restored and corrected through the ISP 1013 is displayed to the user.

The ISP 1013 may include an image reconstruction module 1013a and an image correction module 1013b, and the first neural network model 100 and the second neural network model 200 may be used to configure the image It may be included in the correction module 1013b. At this time, the image restoration module 1013a and the image correction module 1013b can be viewed as software units that perform specific functions or roles in the overall program that performs the image processing process.

The image restoration module 1013a performs demosaicing and denoising on the raw image output from the image sensor 1012 and outputs a linear RGB image. And the image correction module 1013b can correct the linear RGB image and output it as a final image. Hereinafter, specific embodiments of the image restoration module 1013a and the image correction module 1013b will be described with reference to FIG. 11.

Figure 11 is a diagram schematically showing the detailed configuration of an ISP according to an embodiment of the present disclosure. The sensor 1110 of FIG. 11 may correspond to the image sensor 1012 of FIG. 10, and the Neuro-ISP 1120 of FIG. 11 corresponds to a specific embodiment of the ISP 1013 of FIG. 10. Neuro-ISP (1120) is a component that processes image signals using a neural network. According to one embodiment, Neuro-ISP (1120) does not apply AI to a portion of the ISP on a module basis, but rather uses a single neural network end-to-end. -It may be an End-based ISP.

Referring to FIG. 11, the sensor 1110 may output a plurality of raw images. At this time, 'raw image' is an image output from the camera's image sensor, and may refer to an image in Bayer format with only one color channel per pixel.

BurstNet 1121 of FIG. 11 may correspond to the image restoration module 1013a of FIG. 10, and MasteringNet 1122 of FIG. 11 may correspond to the image correction module 1013b of FIG. 10. It can be.

The burstnet 1121 can receive a plurality of raw images and output one linear RGB image. A plurality of raw images input to the burstnet 1121 are a plurality of images taken before and after a specific point in time, and the burstnet 1121 uses the temporal information of the raw images and performs demosaicing and denoising. By doing so, one linear RGB image can be output. The linear RGB image output in this way may correspond to the input image (1) input to the neural network model in embodiments of the present disclosure.

The mastering net 1122 performs correction on the linear RGB image. For example, the mastering net 1122 adjusts the white balance (WB), the gamma value, or the linear RGB image. , the sRGB image can be output as the final image by adjusting the image characteristics by performing processing such as global tone mapping and local tone mapping.

Neural network models according to embodiments of the present disclosure may be included in the mastering net 1122 and serve to correct a linear RGB image (input image).

Meanwhile, the above description assumes that the input image (1) is a linear RGB image and the inference image (2) is an sRGB image, but the input image (1) and the inference image (2) may be images of various formats. For example, the input image 1 and the inference image 2 may be any one of a non-linear RGB image, an sRGB image, an AdobeRGB image, a YCbCr image, and a Bayer image.

The format of the input image (1) and the inference image (2) may be determined depending on the purpose or environment for which the neural network model is used. However, the format of the input image 1 used during learning may remain the same during inference, and by a similar principle, the format of the inference image 2 used during learning may remain the same during inference.

For example, if an input image (1) of a specific format was used when learning the first neural network model 100, the first neural network model 100 receives an input of the same format (as the input image (1) used during learning). By receiving and processing the image (1), the inferred image (2) can be output.

In the embodiment shown in FIG. 10, the first neural network model 100 and the second neural network model 200 are mounted on the camera module 1010, but differently, the neural network models are installed in the main processor 1020 of the mobile terminal 1000. (100, 200) may be implemented to be mounted. The main processor 1020 may include a CPU 1011, GPU 1012, NPU 1013, and memory 1014, and at least one of the CPU 1011, GPU 1012, and NPU 1013 is memory. Neural network models 100 and 200 may be implemented by executing the program stored in 1014.

According to another embodiment, the neural network models 100 and 200 may not be mounted on a device where images are captured, but may be implemented by an external device such as a cloud server. For example, as shown in FIG. 19, when the mobile terminal 1000 transmits raw images acquired through the image sensor 1012 to the cloud server 1900, the processor 1910 of the cloud server 1900 A linear RGB image is generated from the raw images using the restoration module 1911, and the linear RGB image is corrected using the first neural network model 100 and the second neural network model 200 included in the image correction module 1912. Afterwards, the corrected image can be stored in the cloud server 1900 or transmitted to the mobile terminal 1000.

Or, according to another embodiment, the mobile terminal 1000 restores the linear RGB image from the raw images, transmits the restored linear RGB image to the cloud server 1900, and the processor 1910 of the cloud server 1900 After correcting the image using the neural network models 100 and 200, the corrected image can be stored in the cloud server 1900 or transmitted to the mobile terminal 1000.

FIG. 12 is a diagram illustrating a specific embodiment in which neural network models 100 and 200 according to an embodiment of the present disclosure are included in the image correction module 1013b of FIG. 10.

Referring to FIG. 12 , the image correction module 1200 may include a first preprocessor 1210, a second preprocessor 1220, a first neural network model 100, and a second neural network model 200.

The first preprocessor 1210 is a component for converting the input image 1 into a grayscale image, and includes the Auto White Balance (AWB) gain and the color correction matrix included in the metadata. , CCM) and Gamma are applied to the input image (1), and then the input image (1) is converted to a grayscale image by performing channel wise MAX to extract the maximum value of RGB for each pixel. You can print it out.

The second preprocessor 1220 extracts the mean and variance of pixel values from the grayscale image output from the first preprocessor 1210, and applies the extracted mean and variance to the input image (1) and grayscale. It is output using scaled concatenation along with the video. The specific operation structure for performing scaled concatenation is shown in area 1201.

The second neural network model 200 can receive the output of the second preprocessor 1220 and infer the correction parameters (α, β). The correction parameters (α, β) inferred by the second neural network model 200 may be scaled concatenated with the input image 1 and the grayscale image and input to the first neural network model 100.

The first neural network model 100 can output a filter 1230 corresponding to the input image 1 and the correction parameters (α, β), and using the filter 1230 output in this way, the input image 1 It can be converted into an inferred image (2).

According to one embodiment, the correction parameters (α, β) inferred by the second neural network model 200 may be adjusted by the user. When the user adjusts the correction parameters (α, β), the first neural network model 100 outputs a filter 1230 corresponding to the adjusted correction parameters (α, β), and uses the filter 1230 output in this way. Thus, the input image (1) can be converted into the inferred image (2).

According to one embodiment, as described above, the user previews the image corrected according to the correction parameters (α, β) deduced by the second neural network model 200 and checks the correction parameters (α, β) to have the desired image characteristics. α, β) can be adjusted.

Hereinafter, a method of correcting an image using a neural network model will be described with reference to FIGS. 13 to 15. Hereinafter, for convenience of explanation, the ISP 1013 of the mobile terminal 1000 of FIG. 10 will be described as performing the steps of FIGS. 13 to 15. However, the present invention is not limited thereto, and all or part of the steps of FIGS. 13 to 15 may be performed by the main processor 1020 of the mobile terminal 1000 or the processor 1910 of the cloud server 1900 of FIG. 19 .

Referring to FIG. 13, in step 1301, the ISP 1013 may input an input image and correction parameters to the first neural network model 100. The first neural network model 100 may be a model learned to minimize the difference between an inferred image output when an input image and a correction parameter are input, and a label image corresponding to the correction parameter. At this time, the label image corresponding to the correction parameter may be an image in which the input image has been corrected by at least one image correction algorithm when the correction parameter is applied to at least one image correction algorithm.

The input image may be an image output by the image restoration module 1013a by restoring raw images output from the image sensor 1012. The correction parameter may be a preset value (e.g. a value used when learning the first neural network model 100) or a value adjusted according to the user's input. An embodiment in which correction parameters are adjusted according to user input will be described in detail below with reference to FIGS. 15 to 17.

In step 1302, the ISP 1013 may obtain an inferred image corresponding to the correction parameter from the first neural network model 100. The ISP 1013 may display the obtained inferred image on the screen of the mobile terminal 100 or store it in the memory 1014.

Figure 14 is a diagram for explaining an embodiment in which a second neural network model infers a correction parameter. The flow chart of FIG. 14 includes detailed steps included in step 1301 of FIG. 13.

Referring to FIG. 14, in step 1401, the ISP 1013 may input an input image to the second neural network model 200. The second neural network model 200 is a model learned to minimize the difference between the correction parameters inferred by the second neural network model 200 when an input image is input and the correction parameters that allow the label image to have preset image characteristics. You can.

In step 1402, the ISP 1013 obtains the correction parameters deduced by the second neural network model 200, and in step 1403, the ISP 1013 inputs the correction parameters together with the input image into the first neural network model 100. there is.

Figure 15 is a diagram for explaining an embodiment in which a user adjusts correction parameters. The flow chart of FIG. 15 includes detailed steps included in step 1301 of FIG. 13.

Referring to FIG. 15, in step 1501, the ISP 1013 may receive input for adjusting correction parameters through the input/output interface (e.g. touch screen, microphone, etc.) of the mobile terminal 1000. In step 1502, the ISP 1013 may change the preset first correction parameter to a second correction parameter based on the input received in step 1501. In step 1503, the ISP 1013 may input the second correction parameter into the first neural network model 100 along with the input image.

An example in which a user adjusts correction parameters for brightness control through the UI displayed on the screen of the mobile terminal 1000 is shown in FIG. 17 .

Referring to FIG. 17, the image subject to correction (an inferred image corresponding to a preset first correction parameter) is displayed in the first area 1710 of the UI screen 1700 for adjusting the characteristics of the image, and the UI Tools for adjusting the characteristics of the image may be displayed at the bottom of the screen 1700.

In the UI screen 1700 shown in FIG. 17, it is assumed that the user has selected a tool to adjust the brightness of the image. In the second area 1720 of the UI screen 1700, information that the characteristic of the image currently being adjusted is 'brightness' and a value expressing the current brightness level in numbers may be displayed. A slider for adjusting brightness is displayed in the third area 1730 of the UI screen 1700, and the user can adjust the brightness of the input image by moving the slider through touch input.

If the user moves the slider in the third area 1730 of the UI screen 1700 of FIG. 17 to increase the brightness of the input image, the value displayed in the second area 1720 increases, and the ISP 1013 increases the image brightness. The first correction parameter may be adjusted to the second correction parameter to increase the brightness.

When a user captures an image using the mobile terminal 1000 and selects a tool to adjust the brightness of the image, the second neural network model 200 infers is displayed in the first area 1710 of the UI screen 1700. Depending on the correction parameters, an image in which the input image (e.g. a linear RGB image generated from a raw image that is a captured image, etc.) has been corrected may be displayed. When the user adjusts the brightness through the tools displayed in the second area 1720 and the third area 1730 of the UI screen 1700, the correction parameter is adjusted accordingly (e.g. increased by 2 times as in FIG. 16), An image in which the input image has been corrected according to the adjusted correction parameters may be displayed in the first area 1710.

As described above, the first neural network model 100 learns the relationship between correction parameters and label images, so when the correction parameters are changed, the label image corresponding to the changed correction parameters can be inferred. FIG. 16 is a diagram illustrating a change in an inferred image when a correction parameter input to a first neural network model is changed according to an embodiment of the present disclosure.

For convenience of explanation, it is assumed in FIG. 16 that the correction parameter 10 includes only parameters related to brightness control. In addition, the first neural network model 100 was learned using a label image generated by an image correction algorithm that adjusts brightness, and the image correction algorithm used during learning corrects the image so that the brightness increases as the value of the correction parameter increases. Assume that you do.

In the example shown in Figure 16, the correction parameter 10 is doubled. The brightness of the label image corresponding to the doubled correction parameter 10 increases compared to the label image corresponding to the existing correction parameter 10. Accordingly, the brightness of the inferred image 2 output from the first neural network model 100 also increases as the correction parameter 10 increases.

Meanwhile, in the embodiment shown in FIG. 17, the user applies an input to adjust the correction parameter by touching the UI screen 1700. However, according to another embodiment, the user may adjust the correction parameter through voice input. there is. For example, the user may input a voice (e.g. “Hi Bixby, make the picture brighter.”) that instructs to adjust the characteristics of the image through a speaker provided in the mobile terminal 1000.

Hereinafter, a method of training the first neural network model 100 and the second neural network model 200 will be described with reference to FIGS. 20 and 21, respectively. Hereinafter, for convenience of explanation, the processor 1830 of the computing device 1800 of FIG. 18 will be described as performing the steps of FIGS. 20 and 21. However, the present invention is not limited to this, and all or part of the steps of FIGS. 20 and 21 may be performed by the main processor 1020 of the mobile terminal 1000 of FIG. 10 .

Referring to FIG. 20, in step 2001, the processor 1830 may generate a label image by correcting the input image by applying a correction parameter to at least one image correction algorithm.

In step 2002, the processor 1830 may input correction parameters and an input image to the first neural network model, and in step 2003, the processor 1830 may obtain an inferred image output from the first neural network model.

In step 2004, the processor 1830 may update the first neural network model so that the difference between the label image and the inferred image is minimized.

Referring to FIG. 21, the processor 1830 may input an input image into a second neural network model in step 2101 and obtain correction parameters output from the second neural network model in step 2102.

In step 2103, the processor 1830 may generate a label image by correcting the input image by applying correction parameters to at least one image correction algorithm.

The processor 1830 obtains measurement characteristic values expressing the image characteristics of the label image in numbers in step 2104, and updates the second neural network model so that the difference between the measurement characteristic value and the preset target characteristic value is minimized in step 2105. You can.

If the neural network models 100 and 200 according to the embodiments described above are mounted on the mobile terminal 1000, there is an advantage that the image correction module 1013b can be easily updated. Hereinafter, a method of updating a neural network model according to an embodiment will be described with reference to FIGS. 22 and 23.

FIG. 22 shows an example in which the image correction module 1013b included in the ISP 1013 of the mobile terminal 1000 of FIG. 10 is implemented to include image correction algorithms rather than a neural network model. Referring to FIG. 22, the image correction module 1013b is implemented to include a contrast ratio improvement algorithm A (2211) and a brightness correction algorithm (2220).

In FIG. 22, it is assumed that an update to the image correction algorithms included in the image correction module 1013b is performed. The contents of the update are as follows.

1) Replace contrast ratio improvement algorithm A with contrast ratio improvement algorithm B

2) Remove brightness correction algorithm

3) Add exposure fusion algorithm

The result of performing the update is shown on the right side of FIG. 22. Referring to FIG. 22, in the image correction module 1013b, contrast ratio improvement algorithm A (2211) has been replaced with contrast ratio improvement algorithm B (2212), the brightness correction algorithm (2220) has been removed, and the exposure fusion algorithm (2230) has been added. It has been done.

In this way, when an update to the image correction module 1013b is implemented while the image correction module 1013b is implemented to include image correction algorithms, the following problems exist.

First, whenever a change (replacement, removal, addition, etc.) is made to the image correction algorithm, optimization work is required to improve processing speed, which requires a lot of time and resources.

Second, some image correction algorithms are implemented as separate hardware, so in order to add the corresponding image correction algorithm, the hardware must be replaced or added, especially when the image correction module 1013b is implemented on the mobile terminal 1000. Replacing or adding hardware comes with many restrictions.

FIG. 23 shows image correction when the image correction module 1013b included in the ISP 1013 of the mobile terminal 1000 of FIG. 10 is implemented to include neural network models 100 and 200 according to embodiments of the present disclosure. An embodiment of updating the module 1013b is shown.

Referring to FIG. 23, in order to update the first neural network model 100 included in the image correction module 1013b, learning on the first neural network model 100 must be performed again. As described above, learning of the first neural network model 100 may be performed by a separate computing device 1800 shown in FIG. 18, so hereinafter, the computing device 1800 is used to learn the first neural network model 100. Assume that you are learning about

As described above, the label image generation module 300 may be composed of image correction algorithms corresponding to image correction characteristics to be learned in the first neural network model 100. Referring to FIG. 23, the processor 1830 initially configures the label image generation module 300 to include a contrast ratio improvement algorithm A (3211) and a brightness correction algorithm 3220, and learns the first neural network model 100. After doing so, it was mounted on the image correction module 1013b of the mobile terminal 1000.

Afterwards, if there is a need to adjust the image correction characteristics of the image correction module 1013b of the mobile terminal 1000, the processor 1830 allows the label image generation module 300 to use the contrast ratio improvement algorithm B 3212 and the exposure fusion algorithm 3230. ) and the first neural network model 100 can be trained again. When learning of the first neural network model 100 is completed, the processor 1830 transmits the newly learned first neural network model 100 to the mobile terminal 1000, and the mobile terminal 100 receives the received first neural network model. By installing 100, the image correction module 1013b can be updated.

In this way, when the image correction module 1013b is implemented to include the neural network models 100 and 200 according to embodiments of the present disclosure, there is no need to perform optimization even if the image correction module 1013b is updated. In addition, since the computing device 1800 for training the neural network models 100 and 200 generally includes sufficient hardware resources, it is necessary to change (replace) the image correction algorithms included in the label image generation module 300 for new learning. , deletion, addition, etc.) can be done freely.

A method of correcting an image using a neural network model according to an embodiment of the present disclosure includes the steps of a processor of a computing device inputting an input image and a correction parameter to a first neural network model, and the processor Obtaining an inference image corresponding to the correction parameter from the first neural network model, wherein the first neural network model includes the inference image output when the input image and the correction parameter are input. and, a model learned to minimize the difference between label images corresponding to the correction parameters, wherein the label image corresponding to the correction parameters is, when the correction parameters are applied to at least one image correction algorithm, Since the input image is an image corrected by at least one image correction algorithm, and the first neural network model learns the relationship between the correction parameter and the label image, the image output from the first neural network model is changed by changing the correction parameter. Control of the inferred image is possible.

According to one embodiment, when the input image and the first correction parameter are input to the first neural network model, the first inferred image output from the first neural network model is the input image and the second correction parameter to the first neural network model. When a correction parameter is input, it may be different from the second inferred image output from the first neural network model.

According to one embodiment, the step of inputting the correction parameter includes the processor inputting the input image to a second neural network model, the processor obtaining the correction parameter inferred by the second neural network model, and A step of the processor inputting the correction parameters together with the input image into the first neural network model, wherein the second neural network model includes correction parameters that the second neural network model infers when the input image is input. , It may be a model learned to minimize the difference between correction parameters that allow the label image to have preset image characteristics.

According to one embodiment, the step of inputting the correction parameter includes the processor receiving an input for adjusting the correction parameter through an input/output interface of the computing device, and the processor performing a preset adjustment based on the input. It may include changing a first correction parameter to a second correction parameter, and inputting the second correction parameter to the first neural network model together with the input image by the processor.

According to one embodiment, in the step of acquiring the inferred image, the processor acquires an inferred image corresponding to the second correction parameter from the first neural network model, and the first neural network model includes the input image and the Minimize the difference between a first inferred image output when a first correction parameter is input and a first label image corresponding to the first correction parameter, and a first inferred image output when the input image and the second correction parameter are input. 2 A model learned to minimize the difference between an inferred image and a second label image corresponding to the second correction parameter, and the first label image is a model in which the first correction parameter has been applied to the at least one image correction algorithm. When the input image is a corrected image by the at least one image correction algorithm, and the second label image is the at least one image when the second correction parameter is applied to the at least one image correction algorithm The input image may be a corrected image using a correction algorithm.

According to one embodiment, the first neural network model includes generating the label image by correcting the input image by applying the correction parameter to the at least one image correction algorithm, the correction parameter and the input image Learning by inputting into a first neural network model, acquiring the inferred image output from the first neural network model, and updating the first neural network model so that the difference between the label image and the inferred image is minimized. It can be.

According to one embodiment, the second neural network model includes inputting the input image into the second neural network model, obtaining the correction parameter output from the second neural network model, and the at least one image correction algorithm. generating the label image by correcting the input image by applying the correction parameter to the label image, obtaining a measurement characteristic value expressing the image characteristic of the label image in numbers, and determining the difference between the measurement specific value and a preset target characteristic value. It can be learned by updating the second neural network model so that the difference is minimized.

According to one embodiment, the step of inputting the input image and the correction parameter to the first neural network model includes converting the input image into a gray scale image by the processor, and converting the input image into a gray scale image by the processor. Generating a first gray-scale image by applying a first parameter included in the gray-scale image to the gray-scale image, generating a second gray-scale image by the processor applying a second parameter included in the correction parameter to the gray-scale image, and It may include the step of a processor inputting the input image, the first gray-scale image, and the second gray-scale image into the first neural network model.

According to one embodiment, the obtaining of the inferred image includes the processor acquiring at least one of a filter and map information from the first neural network model, and the processor obtaining the filter and the map. It may include converting the input image into the inferred image using at least one piece of information.

According to one embodiment, the step of inputting the input image and the correction parameter to the first neural network model includes the processor using a raw image acquired through an image sensor of the camera from a camera included in the computing device. images), the processor generating the input image using the raw images, and the processor inputting the input image and the correction parameter into the first neural network model.

According to one embodiment, inputting the input image and the correction parameter to the first neural network model includes the computing device receiving raw images from an external device, and the processor using the raw images to It may include generating an input image and allowing the processor to input the input image and the correction parameters into the first neural network model.

A computing device executing a neural network model for image correction according to an embodiment of the present disclosure includes an input/output interface for receiving input from a user, a memory storing a program for correcting an input image, and a processor, the processor By executing the program, inputting an input image and correction parameters to a first neural network model, and obtaining an inferred image corresponding to the correction parameters from the first neural network model, the first neural network model includes the input image and It is a model learned to minimize the difference between the inferred image output when the correction parameter is input and the label image corresponding to the correction parameter, and the label image corresponding to the correction parameter is an image in which the correction parameter is at least one image. When applied to a correction algorithm, the input image is a corrected image by the at least one image correction algorithm, and the first neural network model learns the relationship between the correction parameter and the label image, so by changing the correction parameter The inferred image output from the first neural network model can be controlled.

According to one embodiment, when inputting the correction parameter, the processor inputs the input image to a second neural network model, obtains the correction parameter inferred by the second neural network model, and then sets the correction parameter to the It is input to the first neural network model together with the input image, and the second neural network model includes a correction parameter that the second neural network model infers when the input image is input, and a correction parameter that allows the label image to have preset image characteristics. It may be a model learned to minimize differences between correction parameters.

According to one embodiment, when inputting the correction parameter, the processor receives an input for adjusting the correction parameter through the input/output interface, and based on the input, performs a second correction of the preset first correction parameter. After changing the parameter, the second correction parameter can be input to the first neural network model along with the input image.

According to one embodiment, when acquiring the inferred image, the processor acquires an inferred image corresponding to the second correction parameter from the first neural network model, and the first neural network model includes the input image and the second correction parameter. 1 Minimize the difference between a first inferred image output when a correction parameter is input and a first label image corresponding to the first correction parameter, and a second image output when the input image and the second correction parameter are input. A model learned to minimize the difference between an inferred image and a second label image corresponding to the second correction parameter, wherein the first label image is used when the first correction parameter is applied to the at least one image correction algorithm. , the input image is an image corrected by the at least one image correction algorithm, and the second label image is a corrected image when the second correction parameter is applied to the at least one image correction algorithm. The input image may be a corrected image using an algorithm.

According to one embodiment, the first neural network model generates the label image by correcting the input image by applying the correction parameter to the at least one image correction algorithm, and combines the correction parameter and the input image with the first neural network model. 1 It can be learned by inputting a neural network model, obtaining the inferred image output from the first neural network model, and then updating the first neural network model so that the difference between the label image and the inferred image is minimized.

According to one embodiment, the second neural network model inputs the input image to the second neural network model, obtains the correction parameter output from the second neural network model, and executes the at least one image correction algorithm. The label image is generated by correcting the input image by applying a correction parameter, and measurement characteristic values expressing the image characteristics of the label image in numbers are obtained, and then the difference between the measurement specific value and the preset target characteristic value is minimized. It can be learned by updating the second neural network model so that .

According to one embodiment, when inputting the input image and the correction parameter to the first neural network model, the processor converts the input image into a gray scale image and uses a first parameter included in the correction parameter. is applied to the gray-scale image to generate a first gray-scale image, and a second parameter included in the correction parameter is applied to the gray-scale image to generate a second gray-scale image. Then, the input image, the first gray-scale image, and The second grayscale image may be input to the first neural network model.

According to one embodiment, when acquiring the inferred image, the processor obtains at least one of a filter and map information from the first neural network model, and then obtains at least one of the filter and map information. The input image can be converted into the inferred image using .

According to one embodiment, when inputting the input image and the correction parameter to the first neural network model, the processor uses a raw image obtained from a camera included in the computing device through an image sensor of the camera. ) are received, the input image is generated using the raw images, and then the input image and the correction parameters can be input to the first neural network model.

According to one embodiment, when inputting the input image and the correction parameters to the first neural network model, the processor receives raw images from an external device, generates the input image using the raw images, and then , the processor may input the input image and the correction parameter into the first neural network model.

Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and stored in a computer-readable medium. In this disclosure, “application” and “program” refer to one or more computer programs, software components, instruction sets, procedures, functions, or objects suitable for implementation in computer-readable program code. ), may represent a class, instance, related data, or part thereof. “Computer-readable program code” may include various types of computer code, including source code, object code, and executable code. “Computer-readable media” means read only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disc (CD), digital video disc (DVD), or various types of memory, It may include various types of media that can be accessed by a computer.

Additionally, device-readable storage media may be provided in the form of non-transitory storage media. Here, a ‘non-transitory storage medium’ is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is semi-permanently stored in the storage medium and cases where data is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. Computer-readable media can be any available media that can be accessed by a computer and can include both volatile and non-volatile media, removable and non-removable media. Computer-readable media includes media on which data can be permanently stored and media on which data can be stored and later overwritten, such as rewritable optical disks or erasable memory devices.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. A computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or through an application store, or on two user devices (e.g., smart device). It can be distributed (e.g., downloaded or uploaded) directly between phones, or online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be stored on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be at least temporarily stored or created temporarily.

The foregoing description of the present disclosure is for illustrative purposes, and a person skilled in the art to which the present disclosure pertains will understand that the present disclosure can be easily modified into another specific form without changing its technical idea or essential features. will be. For example, the described techniques are performed in an order different from the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or in a different configuration. Appropriate results may be achieved through substitution or substitution by elements or equivalents. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present disclosure. do.

Claims (20)

Inputting, by a processor of a computing device, an input image and a correction parameter to a first neural network model; and
Comprising the step of the processor acquiring an inference image corresponding to the correction parameter from the first neural network model,
The first neural network model is,
A model learned to minimize the difference between the inferred image output when the input image and the correction parameter are input, and a label image corresponding to the correction parameter,
The label image corresponding to the correction parameter is an image in which the input image is corrected by the at least one image correction algorithm when the correction parameter is applied to the at least one image correction algorithm,
Since the first neural network model learns the relationship between the correction parameter and the label image, the inferred image output from the first neural network model can be controlled by changing the correction parameter. According to paragraph 1,
When the input image and the first correction parameter are input to the first neural network model, the first inferred image output from the first neural network model is, when the input image and the second correction parameter are input to the first neural network model. A method, characterized in that it is different from the second inferred image output from the first neural network model. According to paragraph 1,
The step of inputting the correction parameters is,
inputting the input image to a second neural network model by the processor;
obtaining, by the processor, the correction parameter inferred by the second neural network model; and
Including the step of the processor inputting the correction parameters together with the input image into the first neural network model,
The second neural network model is,
A method in which the model is learned to minimize the difference between a correction parameter inferred by the second neural network model when the input image is input and a correction parameter that causes the label image to have preset image characteristics. According to paragraph 1,
The step of inputting the correction parameters is,
receiving, by the processor, an input for adjusting the correction parameter through an input/output interface of the computing device;
changing, by the processor, a preset first correction parameter to a second correction parameter based on the input; and
Method comprising the step of the processor inputting the second correction parameter into the first neural network model along with the input image. According to paragraph 4,
The step of acquiring the inferred image is,
The processor obtains an inferred image corresponding to the second correction parameter from the first neural network model,
The first neural network model is,
Minimize the difference between a first inferred image output when the input image and the first correction parameter are input and a first label image corresponding to the first correction parameter, and the input image and the second correction parameter are input. It is a model learned to minimize the difference between the second inferred image output when
The first label image is an image in which the input image is corrected by the at least one image correction algorithm when the first correction parameter is applied to the at least one image correction algorithm,
The second label image is an image in which the input image is corrected by the at least one image correction algorithm when the second correction parameter is applied to the at least one image correction algorithm. According to paragraph 1,
The first neural network model is,
generating the label image by correcting the input image by applying the correction parameter to the at least one image correction algorithm;
Inputting the correction parameters and the input image into the first neural network model;
Obtaining the inferred image output from the first neural network model; and
A method characterized in that it is learned by updating the first neural network model so that the difference between the label image and the inferred image is minimized. According to paragraph 3,
The second neural network model is,
Inputting the input image into the second neural network model;
Obtaining the correction parameters output from the second neural network model;
generating the label image by correcting the input image by applying the correction parameter to the at least one image correction algorithm;
Obtaining measurement characteristic values expressing image characteristics of the label image in numbers; and
A method characterized in that it is learned by updating the second neural network model so that the difference between the measurement specific value and a preset target characteristic value is minimized. According to paragraph 1,
Inputting the input image and the correction parameters into the first neural network model includes:
converting the input image into a gray scale image by the processor;
generating a first gray-scale image by the processor applying a first parameter included in the correction parameter to the gray-scale image;
generating a second gray-scale image by the processor applying a second parameter included in the correction parameter to the gray-scale image; and
The method comprising the processor inputting the input image, the first gray-scale image, and the second gray-scale image into the first neural network model. According to paragraph 1,
The step of acquiring the inferred image is,
The processor acquiring at least one of a filter and map information from the first neural network model; and
Method comprising the step of converting the input image into the inferred image by the processor using at least one of the filter and map information. According to paragraph 1,
Inputting the input image and the correction parameters into the first neural network model includes:
Receiving, by the processor, raw images obtained through an image sensor of the camera from a camera included in the computing device;
generating, by the processor, the input image using the raw images; and
The method comprising the processor inputting the input image and the correction parameters into the first neural network model. According to paragraph 1,
Inputting the input image and the correction parameters into the first neural network model includes:
The computing device receiving raw images from an external device;
generating, by the processor, the input image using the raw images; and
The method comprising the processor inputting the input image and the correction parameters into the first neural network model. A computer-readable recording medium on which a program for performing the method of any one of claims 1 to 11 is recorded on a computer. an input/output interface for receiving input from a user;
a memory in which a program for correcting an input image is stored; and
Includes a processor,
The processor executes the program,
Entering an input image and correction parameters into a first neural network model, and obtaining an inferred image corresponding to the correction parameters from the first neural network model,
The first neural network model is,
A model learned to minimize the difference between the inferred image output when the input image and the correction parameter are input, and the label image corresponding to the correction parameter,
The label image corresponding to the correction parameter is an image in which the input image is corrected by the at least one image correction algorithm when the correction parameter is applied to the at least one image correction algorithm,
Since the first neural network model learns the relationship between the correction parameter and the label image, the computing device is capable of controlling the inferred image output from the first neural network model by changing the correction parameter. According to clause 13,
When the input image and the first correction parameter are input to the first neural network model, the first inferred image output from the first neural network model is, when the input image and the second correction parameter are input to the first neural network model. A computing device, characterized in that it is different from the second inferred image output from the first neural network model. According to clause 13,
When inputting the correction parameter, the processor
Inputting the input image into a second neural network model, obtaining the correction parameter inferred by the second neural network model, and then inputting the correction parameter together with the input image into the first neural network model,
The second neural network model is,
A computing device that is a model learned to minimize the difference between a correction parameter inferred by the second neural network model when the input image is input and a correction parameter that causes the label image to have preset image characteristics. According to clause 13,
When inputting the correction parameter, the processor
Receives an input for adjusting the correction parameter through the input/output interface, changes a preset first correction parameter to a second correction parameter based on the input, and then exchanges the second correction parameter with the input image. A computing device that inputs to the first neural network model. According to clause 16,
In acquiring the inferred image, the processor
Obtaining an inferred image corresponding to the second correction parameter from the first neural network model,
The first neural network model is,
Minimize the difference between a first inferred image output when the input image and the first correction parameter are input and a first label image corresponding to the first correction parameter, and the input image and the second correction parameter are input. It is a model learned to minimize the difference between the second inferred image output when
The first label image is an image in which the input image is corrected by the at least one image correction algorithm when the first correction parameter is applied to the at least one image correction algorithm,
The second label image is an image in which the input image is corrected by the at least one image correction algorithm when the second correction parameter is applied to the at least one image correction algorithm. According to clause 13,
The first neural network model is,
The label image is generated by correcting the input image by applying the correction parameter to the at least one image correction algorithm, inputting the correction parameter and the input image into the first neural network model, and generating the label image from the first neural network model. A computing device characterized in that it is learned by updating the first neural network model so that the difference between the label image and the inferred image is minimized after obtaining the output inferred image. According to clause 15,
The second neural network model is,
Input the input image into the second neural network model, obtain the correction parameter output from the second neural network model, and apply the correction parameter to the at least one image correction algorithm to correct the input image to obtain the label. After generating an image and obtaining a measurement characteristic value expressing the image characteristic of the label image in numbers, learning is performed by updating the second neural network model so that the difference between the measurement specific value and a preset target characteristic value is minimized. A computing device characterized by: According to clause 13,
The processor inputs the input image and the correction parameters into the first neural network model,
Convert the input image to a gray scale image, apply the first parameter included in the correction parameter to the gray scale image to generate a first gray scale image, and apply the second parameter included in the correction parameter to the gray scale image. A computing device that generates a second gray-scale image by applying it to an image and then inputs the input image, the first gray-scale image, and the second gray-scale image into the first neural network model.

Images