KR20230166870A - Image signal processing method using neural network, and computing appratus for performing the same - Google Patents

Abstract

An image processing method using a neural network model according to an embodiment of the present disclosure includes the steps of acquiring an image captured through an image sensor, confirming a shooting environment (shooting context) of the image, and restoring the image according to the shooting environment. It may include selecting a neural network model included in at least one of an image reconstruction module and an image correction module, and processing the image using the selected neural network model.

Description

Image signal processing method using a neural network model and computing device for performing the same {IMAGE SIGNAL PROCESSING METHOD USING NEURAL NETWORK, AND COMPUTING APPRATUS FOR PERFORMING THE SAME}

The present disclosure relates to an image signal processing method using a neural network model and a computing device for performing the same. In detail, a neural network model is selected according to the context in which the image was captured, and the image captured using the selected neural network model is captured. It is about restoration and correction methods.

Due to recent developments in neural network technology, neural networks are being used in various fields. In particular, processing speed and image characteristics can be improved by utilizing neural networks in the field of image signal processing to perform image correction or restoration. For example, in mobile terminals such as smartphones, a program is installed to correct images taken with an equipped camera. If a program implemented to include existing image correction algorithms can be replaced with a neural network, there are advantages in various aspects. You can expect it.

Meanwhile, the quality of the captured image may be affected depending on the context in which the shooting is performed. For example, images captured at night have a lot of noise, and images captured in zoom mode may not show colors or edges clearly due to magnification.

Since the neural network model is trained to correct the image characteristics in a preset direction, good image quality cannot be guaranteed in all environments if a fixed neural network model is used.

An image processing method using a neural network model, which was disclosed as a technical means for achieving a technical task, includes the steps of acquiring an image captured through an image sensor 100, confirming the shooting context of the image, and the steps of: Selecting a neural network model included in at least one of an image reconstruction module 1100 and an image correction module 1200 according to the shooting environment, and converting the image using the selected neural network model. It may include processing steps.

Disclosed as a technical means for achieving a technical problem, a computing device for processing an image signal using a neural network model includes a memory storing a program for processing an image signal and at least one processor, and the at least one By executing the program, the processor acquires an image captured through the image sensor 100, checks the shooting environment of the image, and operates an image reconstruction module 1100 according to the shooting environment. ) and the image correction module 1200, the neural network model included in at least one is selected, and the image can be processed using the selected neural network model.

A computer-readable recording medium disclosed as a technical means for achieving a technical problem may store a program for executing at least one of the embodiments of the disclosed method on a computer.

A computer program disclosed as a technical means for achieving a technical problem may be stored in a medium for performing at least one of the embodiments of the disclosed method on a computer.

FIG. 1 is a diagram illustrating the configuration of an image signal processing module (ISP) using a neural network model, according to an embodiment of the present disclosure.
FIG. 2 is a diagram illustrating a specific example of an image signal processing module (ISP) using a neural network model, according to an embodiment of the present disclosure.
FIG. 3 is a diagram illustrating the configuration of a user terminal that performs image signal processing using a neural network model, according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating an example in which a cloud server corrects an image received from a user terminal using an ISP, according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating a method of selecting a neural network model included in an image restoration module based on a shooting environment, according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating a method of training a neural network model included in an image restoration module according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating a method of selecting a neural network model of an image correction module according to an image restoration module, according to an embodiment of the present disclosure.
8 to 10 are diagrams for explaining a method of selecting an image sensor, an image restoration module, and an image correction module according to a shooting environment according to embodiments of the present disclosure.
11 to 14 are flowcharts for explaining a method of processing an image signal using a neural network model according to embodiments of the present disclosure.
FIG. 15 is a diagram illustrating a method of changing the combination of an image sensor, an image restoration module, and an image correction module according to the type of an object recognized in an image, according to an embodiment of the present disclosure.
FIG. 16 is a diagram for explaining the operation of a first neural network model according to an embodiment of the present disclosure.
FIG. 17 is a diagram for explaining the operation of a second neural network model according to an embodiment of the present disclosure.
Figure 18 is a diagram for explaining a process of learning a first neural network model according to an embodiment of the present disclosure.
Figure 19 is a diagram for explaining a process of learning a second neural network model according to an embodiment of the present disclosure.
FIG. 20 is a diagram illustrating a process in which a label image generation module generates a label image from an input image according to an embodiment of the present disclosure.
Figure 21 is a diagram for explaining the strength value applied to the contrast ratio improvement algorithm according to an embodiment of the present disclosure.
FIG. 22 is a diagram for explaining gamma values applied to a brightness correction algorithm according to an embodiment of the present disclosure.
Figure 23 is a diagram for explaining the α value applied to the exposure fusion algorithm according to an embodiment of the present disclosure.
FIG. 24 is a diagram illustrating a method of applying a correction parameter as an input to a first neural network model, according to an embodiment of the present disclosure.
FIG. 25 is a diagram illustrating a specific example in which neural network models according to an embodiment of the present disclosure are included in an image correction module.
26 to 28 are flowcharts for explaining a method of correcting an image using a neural network model according to embodiments of the present disclosure.
FIG. 29 is a diagram illustrating a change in an inferred image when a correction parameter input to a first neural network model is changed according to an embodiment of the present disclosure.
FIG. 30 is a diagram illustrating a situation in which a user adjusts correction parameters for brightness adjustment through a UI displayed on the screen of a mobile terminal, according to an embodiment of the present disclosure.
FIG. 31 is a block diagram illustrating the configuration of a computing device for performing learning on neural network models, according to an embodiment of the present disclosure.
Figures 32 and 33 are flow charts for explaining a method of training a first neural network model and a second neural network model, respectively, according to embodiments of the present disclosure.
Figures 34 and 35 are diagrams for explaining a method of updating a neural network model according to an embodiment of the present disclosure.

In describing the present disclosure, description of technical content that is well known in the technical field to which the present disclosure belongs and that is not directly related to the present disclosure will be omitted. This is to convey the gist of the present disclosure more clearly without obscuring it by omitting unnecessary explanation. In addition, the terms described below are terms defined in consideration of the functions in the present disclosure, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

For the same reason, some components are exaggerated, omitted, or schematically shown in the attached drawings. Additionally, the size of each component does not entirely reflect its actual size. In each drawing, identical or corresponding components are assigned the same reference numbers.

The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the disclosure is complete and to fully inform those skilled in the art of the disclosure of the scope of the disclosure. One embodiment of the present disclosure may be defined according to the claims. Like reference numerals refer to like elements throughout the specification. Additionally, when describing an embodiment of the present disclosure, if it is determined that a detailed description of a related function or configuration may unnecessarily obscure the gist of the present disclosure, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of the functions in the present disclosure, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

In one embodiment, each block of the flow diagram diagrams and combinations of the flow diagram diagrams may be performed by computer program instructions. Computer program instructions may be mounted on a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, and the instructions performed through the processor of the computer or other programmable data processing equipment may perform the functions described in the flowchart block(s). You can create the means to carry them out. The computer program instructions may also be stored in a computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular way. The instructions may also produce manufactured items containing instruction means that perform the functions described in the flow diagram block(s). Computer program instructions may also be mounted on a computer or other programmable data processing equipment.

Additionally, each block in a flowchart diagram may represent a module, segment, or portion of code containing one or more executable instructions for executing specified logical function(s). In one embodiment, it is possible for functions mentioned in blocks to occur out of order. For example, two blocks shown in succession may be performed substantially simultaneously or may be performed in reverse order depending on their functions.

The term '~unit' used in an embodiment of the present disclosure may refer to software or hardware components such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and '~unit' refers to a specific role. can be performed. Meanwhile, '~ part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. In one embodiment, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, May include subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided through specific components or specific '~parts' can be combined to reduce their number or separated into additional components. Additionally, in one embodiment, ‘˜part’ may include one or more processors.

Embodiments of the present disclosure relate to a method of processing image signals using a neural network model. Before describing specific embodiments, the meaning of terms frequently used in this specification are defined.

‘Shooting context’ refers to various factors that affect video shooting or all matters related to video shooting, and can especially refer to factors that affect the quality or characteristics of the video being shot. ‘Shooting environment’ may include ‘shooting conditions’ and ‘shooting mode’, and their respective meanings are as follows.

'Shooting conditions' are conditions under which an image is shot, and may include at least one of shooting time (e.g. daytime or night), shooting location (e.g. indoors or outdoors), and ISO (film speed).

'Shooting mode' refers to a mode for shooting images, and may be set in a shooting device such as a camera, for example. According to one embodiment, the shooting mode may include a normal shooting mode, a night shooting mode, and a zoom shooting mode.

'ISP (Image Signal Processing)' refers to the operation of processing digital image signals. In this disclosure, the module that performs ISP is defined as 'ISP'.

‘Image reconstruction’ refers to the process of demosaicing the image of the Bayer pattern (RGGB, GBRG, BGGR, GRBG) extracted from the sensor into an RGB pattern. The demosaicing process may include an operation to improve the quality of the image by interpolating some pixel values included in the image or removing noise.

‘Image correction’ refers to the operation of adjusting image characteristics, and ‘image characteristic’ refers to the brightness, contrast, and color temperature of the image. it means. Additionally, ‘image correction algorithm’ refers to an algorithm for controlling the characteristics of an image.

‘Correction parameter’ refers to a parameter applied to an image correction algorithm when correcting an image using an image correction algorithm. In other words, the degree to which image characteristics are adjusted can be determined according to the value of the correction parameter. Specific examples of correction parameters will be described below.

A ‘label image’ is an image used as training data to perform supervised learning on a neural network model according to embodiments of the present disclosure, and in particular, ground truth data (ground truth data). This refers to the image used as truth data. The 'label image generation module' is a component that generates a label image by correcting an input image using at least one image correction algorithm.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

FIG. 1 is a diagram illustrating the configuration of an image signal processing module (ISP) using a neural network model, according to an embodiment of the present disclosure. Referring to FIG. 1 , the ISP 1000 according to an embodiment may include an image reconstruction module 1100 and an image correction module 1200. At this time, the image restoration module 1100 and the image correction module 1200 are components that are implemented by executing a program stored in the memory by a processor of a computing device that performs image processing. A computing device that performs image processing may be a device that has a shooting function and an arithmetic processing function, such as a smartphone or digital camera. Even if it does not have a shooting function, it can receive an image file or video file and perform a video signal processing process. It can be various types of devices (e.g. laptop, cloud server). The configuration of a computing device that performs image processing according to embodiments of the present disclosure will be described in detail below with reference to FIG. 3.

The image sensor 100 is a component that performs imaging and outputs an image signal, and may be, for example, a CCD sensor or a CMOS sensor. In addition, various types of image sensors 100 may be used. According to one embodiment, the image sensor 100 may output a raw image in Bayer format with only one color channel per pixel. However, the image sensor 100 is not limited to this and may output images in various formats. Additionally, according to one embodiment, the image sensor 100 may include a plurality of image sensors of different types. In the present disclosure, 'different types of image sensors' may include cases where the types of matching lenses are different even if the image sensors themselves have the same hardware specifications. In other words, assuming that the image sensor 100 includes a first image sensor and a second image sensor, and that the first image sensor and the second image sensor have the same hardware specifications, the first image sensor has a general lens. If they are matched and a wide-angle lens is matched to the second image sensor, in this disclosure, the first image sensor and the second image sensor are expressed as different types of image sensors. Of course, the image sensor 100 may include image sensors with different hardware specifications.

The image restoration module 1100 is a module for restoring an image received from the image sensor 100. According to one embodiment, the image restoration module 1100 performs lens shading correction (LSC), bad pixel correction (BPC), and demosaicing on the input image received from the image sensor 100. At least one restoration operation of demosaic and denoise may be performed.

As shown in FIG. 1, the image restoration module 1100 may include a hardware (HW) ISP 1110, a software (SW) ISP 1120, and an AI restoration ISP 1130 as detailed components. Depending on the shooting environment, at least one of the detailed configurations included in the image restoration module 1100 may be selected to perform a restoration operation. How the detailed configuration of the image restoration module 1100 is selected according to the shooting environment will be described in detail below.

According to one embodiment, the HW ISP 1110 and SW ISP 1120 may be modules divided based on the level of complexity of the process being performed. For example, the HW ISP 1110 may be a module that processes only simple processes, and the SW ISP 1120 may be a module that processes relatively complex processes using software algorithms.

Among the detailed components included in the image restoration module 1100, the AI restoration ISP 1130 is a neural network-based module and may include a plurality of neural network models. As shown in FIG. 1, the AI restoration ISP 1130 may include a plurality of model parameters (restoration parameters), and each restoration parameter represents a neural network model. According to one embodiment, the AI restoration ISP 1130 may set a neural network model to one of a plurality of restoration parameters depending on the shooting environment and perform an image processing process.

The image correction module 1200 is a module for performing image correction. The image correction module 1200 can make the image better to view by increasing the brightness or improving the contrast ratio of the image received from the image restoration module 1100. According to one embodiment, the image correction module 1200 adjusts the white balance (WB), performs color correction (CC), or adjusts the gamma value for the received image. , image characteristics can be adjusted by performing processing such as global tone mapping, local tone mapping, HDR effect, etc.

According to one embodiment, the image correction module 1200 may be composed of a neural network-based module (AI correction ISP) and may include a plurality of neural network models. As shown in FIG. 1, the AI correction ISP 1200 may include a plurality of model parameters (correction parameters), and each correction parameter represents one neural network model. According to one embodiment, the AI correction ISP 1200 may set a neural network model to one of a plurality of correction parameters depending on the shooting environment and perform an image processing process.

When shooting in a low-light environment at night, it may be difficult to identify objects in the image due to the low brightness or contrast ratio of the image. Additionally, if you increase the ISO value by shooting in night mode to shoot brightly in a low-light environment, the brightness will improve, but the amount of noise included in the image may increase, which may reduce the quality. Additionally, when shooting in zoom mode, the shape or color of the object may not be expressed clearly due to the enlargement of the image.

As such, the quality of the image may deteriorate depending on the shooting environment (shooting conditions and shooting mode), but embodiments of the present disclosure provide a method of obtaining optimized high-quality images (e.g. still images, moving images) in any situation. . According to embodiments of the present disclosure, image quality can be improved by selecting detailed configurations (particularly, neural network models) included in the image restoration module 1100 and the image correction module 1200 according to the shooting environment. Additionally, embodiments of the present disclosure can improve image quality by selecting one of various types of image sensors 100 according to the shooting environment.

In addition, camera-related hardware (e.g., not only direct components for shooting, such as lenses and image sensors, but also processors for processing captured images) may be different for each device, and according to embodiments of the present disclosure, such Despite differences in hardware, technical effects that can provide high quality video above a certain level can be expected.

First, a specific example of a neural network-based ISP will be described with reference to FIG. 2, and a specific example of a computing device that performs image processing according to embodiments of the present disclosure will be described with reference to FIG. 3, and then Therefore, we explain in detail how to select a neural network model.

FIG. 2 is a diagram illustrating a specific example of an image signal processing module (ISP) using a neural network model, according to an embodiment of the present disclosure.

The sensor 200 of FIG. 2 may correspond to the image sensor 100 of FIG. 1, and the Neuro-ISP 2000 of FIG. 2 corresponds to a specific embodiment of the ISP 1000 of FIG. 1. Neuro-ISP (2000) is a component that processes image signals using a neural network. According to one embodiment, Neuro-ISP (2000) may be a method of applying a neural network to a portion of the ISP on a module basis, or in one embodiment According to Neuro-ISP (2000), it may be an ISP based on a single neural network end-to-end.

BurstNet 2130 of FIG. 2 corresponds to a specific embodiment of the image restoration module 1130 of FIG. 1, and MasteringNet 2200 of FIG. 2 corresponds to the image correction module 1200 of FIG. 1. It corresponds to a specific example of.

Referring to FIG. 2, the sensor 200 may output a plurality of raw images. According to one embodiment, the raw image may be a Bayer format image with only one color channel per pixel.

The burstnet 2130 can receive a plurality of raw images and output one linear RGB image. A plurality of raw images input to the burstnet 2130 are a plurality of images taken before and after a specific point in time. The burstnet 2130 uses the temporal information of the raw images and performs LSC, BPC, and align and fusion. By performing (align & fusion), demosaicing, and denoising, one linear RGB image can be output.

The mastering net 2200 can perform correction on linear RGB images. According to one embodiment, the mastering net 2200 adjusts the white balance (WB), performs color correction (CC), or adjusts the gamma value for the linear RGB image. By performing processing such as global tone mapping, local tone mapping, HDR effect, etc., image characteristics can be adjusted and the sRGB image can be output as the final image. To this end, the mastering net 2200 may receive a white balance gain (WBG) and a color correction matrix (CCM) from the sensor 200.

Meanwhile, in the above description, it was assumed that the image output from the sensor 200 is a Bayer image, the image output from the burst net 2130 is a linear RGB image, and the image output from the mastering net 2200 is an sRGB image. , but is not limited to this, and each video may be of various formats. For example, the above images may be any one of a non-linear RGB image, an sRGB image, an AdobeRGB image, a YCbCr image, or a Bayer image. However, the format of the input image used when learning the neural network model included in each of the burst net 2130 and the mastering net 2200 may remain the same during inference, and the format of the inference image used during learning follows a similar principle. The same can remain true during this inference.

FIG. 3 is a diagram illustrating the configuration of a user terminal that performs image signal processing using a neural network model, according to an embodiment of the present disclosure. In FIG. 3, the user terminal 3000 that performs video signal processing is shown as a smartphone, but it is not limited to this and the ISP 3130 can be mounted on any type of device with a video capture function, such as a camera. Furthermore, it is obvious that the ISP 3130 can be installed in various types of devices that do not have a shooting function but can perform video signal processing.

Referring to FIG. 3, the user terminal 3000 may include a camera module 3100 and a main processor 3200.

The camera module 3100 may include a lens module 3110, an image sensor 3120, and an ISP 3130, and these components (3110, 3120, and 3130) are mounted on one chip (SoC). It may also be implemented in the form of a System on Chip. The camera module 3100 may include a separate processor, through which the ISP 3130 can be executed.

The image sensor 3120 receives light transmitted through the lens module 3110 and outputs an image, and the ISP 3130 can restore and correct the image output from the image sensor 3120 and output it as a final image. That is, when a user captures an image through the user terminal 3000, the final image restored and corrected through the ISP 3130 is displayed to the user.

The ISP 3130 may include an image restoration module 3131 and an image correction module 3132, and the operation of each component is the same as previously described with reference to FIGS. 1 and 2.

In the embodiment shown in FIG. 3, the ISP 3130 is mounted on the camera module 3100. Alternatively, the ISP 3130 may be mounted on the main processor 3200 of the user terminal 3000. The main processor 3200 may include a CPU 3210, GPU 3220, NPU 3230, and memory 3240, and at least one of the CPU 3210, GPU 3220, and NPU 3230 is memory. ISP 3130 can also be implemented by executing the program stored in 3240.

According to another embodiment, at least some of the components included in the ISP 3130 may not be mounted on the device where the image is captured, but may be implemented by an external device such as a cloud server. For example, as shown in FIG. 4, an ISP 4000 including an image restoration module 4100 and an image correction module 4200 may be implemented in the cloud server 400, and the user terminal 3000 may When raw images acquired through the sensor 3120 are transmitted to the cloud server 400, the image restoration module 4100 of the cloud server 400 generates a linear RGB image from the raw images, and the image correction module 4200 After correcting the linear RGB image, the corrected image can be stored in the cloud server 400 or transmitted to the user terminal 3000.

Or, according to another embodiment, the user terminal 3000 restores the linear RGB image from the raw images and then transmits the restored linear RGB image to the cloud server 400, and the cloud server 400 provides an image correction module ( After correcting the image using 4200), the corrected image can be stored in the cloud server 400 or transmitted to the user terminal 3000.

Therefore, in the following embodiments, the operations performed by the ISP 1000 of FIG. 1 are actually performed by the user terminal 3000, the cloud server 400, or various other computing devices individually or by a combination of two or more computing devices. It can be performed by .

1. Neural network model selection in image restoration module

A method of selecting a neural network model in the AI restoration ISP 1130 of the image restoration module 1100 of FIG. 1 will be described. FIG. 5 is a diagram illustrating a method of selecting a neural network model included in the image restoration module 1100 based on a shooting environment, according to an embodiment of the present disclosure. As shown in FIG. 5, it is assumed that the AI restoration ISP 1130 includes a neural network model for daytime use (daytime model parameters 1131) and a neural network model for nighttime use (night model parameters 1132).

According to an embodiment of the present disclosure, the AI restoration ISP 1130 selects the weekly model parameter 1131 when the noise included in the input image 51 is less than a certain standard, and conversely, the AI restoration ISP 1130 selects the weekly model parameter 1131 and conversely selects the If is above a certain standard, the night model parameter 1132 can be selected.

(1) Neural network model selection based on ISO value

According to one embodiment, the amount of noise included in the input image 51 can be estimated based on the ISO value of the input image 51, and therefore, based on the ISO value among shooting conditions (time, location, ISO, etc.) You can select the neural network model of the AI restoration ISP (1130). For example, the AI restoration ISP 1130 may check the ISO value from the metadata of the input image 51 and select a neural network model (model parameter) based on the confirmed ISO value. At this time, the input image 51 may be a raw image output from the image sensor 100.

Table 1 below shows the amount of noise estimated according to ISO value. Shot noise according to ISO value can be measured differently depending on the device that performs shooting, and read noise is a value calculated from shot noise using a specific formula. Therefore, the values included in Table 1 only correspond to one example, and the amount of noise estimated may vary depending on the ISO value. Additionally, the section can be further subdivided by adding data between the numbers included in Table 1 through interpolation.

If the shot noise included in the input image 51 is set to select the night model parameter (1132) if it is more than 0.001, and the day model parameter (1131) if it is less than 0.001, the AI restoration ISP (1130) is set to select the ISO value If the ISO value is above 1600, the neural network model can be set with the night model parameter (1132), and if the ISO value is less than 1600, the neural network model can be set with the daytime model parameter (1131).

(2) Neural network model selection based on shooting time and shooting location

According to one embodiment, the AI restoration ISP 1130 may determine whether the input image 51 was captured in a low-light environment based on the capture time and location of the input image 51 and select a neural network model accordingly. For example, the AI restoration ISP 1130 obtains information about the time and place where the image was captured (e.g. location estimated using a GPS signal, cellular base station, or WiFi AP) from the metadata of the input image 51. And, a neural network model (model parameters) may be selected based on the acquired information. According to one embodiment, the AI restoration ISP 1130 may select model parameters for a combination of time (day/night) and location (indoor/outdoor) as follows. In general, since it can be expected to be in a low-light environment only when shooting outdoors at night, model parameters are selected according to the rules below, but the rules are not limited to this and can be changed depending on the need or situation.

1) Daytime & Outdoor => Daytime model parameters

2) Daytime & Indoor => Daytime model parameters

3) Night & Outdoor => Night Model Parameters

4) Night & Indoor => Day model parameters

Additionally, according to one embodiment, the AI restoration ISP 1130 may select model parameters by considering the ISO value along with information about time and place to increase selection accuracy. For example, as a result of judgment based on information about time and place, it is necessary to select daytime model parameters, but if the ISO value is above a certain standard, the AI restoration ISP 1130 may select nighttime model parameters. In other words, the AI restoration ISP 1130 may primarily select model parameters based on time and location, and secondarily determine whether the model parameters need to be changed by considering the ISO value.

Meanwhile, according to one embodiment, the AI restoration ISP 1130 may determine the amount of noise included in the input image 51 by analyzing the input image 51 and select a neural network model according to the result. For example, as a result of analyzing the pixel values of the input image 51, the AI restoration ISP 1130 selects the night model parameter 1132 if the noise included is greater than the preset standard, and if it is less than the preset standard, the AI restoration ISP 1130 selects the day model parameter 1132. (1132) can be selected.

(3) Selection of neural network model according to shooting mode or type of image sensor

According to an embodiment of the present disclosure, the AI restoration ISP 1130 may select a neural network model according to the shooting mode or type of image sensor. For example, the AI restoration ISP (1130) can set the neural network model with the daytime model parameter (1131) in normal shooting mode (daytime shooting mode), and set the neural network model with the night model parameter (1132) in night shooting mode. there is.

Or, according to one embodiment, if the image sensor 100 includes a plurality of image sensors (see FIG. 8) and one of the plurality of image sensors is selected to capture the input image 51, the selected image sensor A corresponding neural network model may be selected in the AI restoration ISP 1130. To this end, the type of image sensor corresponding to the neural network models included in the AI restoration ISP 1130 may be predetermined. For example, if the image sensor 100 includes an image sensor for night photography (e.g. an image sensor suitable for use with a wide-angle lens) and the input image 51 is captured using this, the AI restoration ISP 1130 The neural network model can be set with the night model parameters (1132).

In the embodiment shown in Figure 5, the AI restoration ISP (1130) includes two model parameters (1131, 1132), but is not limited to this, three or more model parameters are included in the AI restoration ISP (1130), and The standard ISO value and noise section can also be further subdivided.

2. Learning of the neural network model included in the image restoration module

As described above, the AI restoration ISP 1130 selects a neural network model according to the amount of noise included in the input image, and the selected neural network model can perform an appropriate restoration process according to the amount of noise. Below, we will explain how to learn the neural network models included in the AI restoration ISP (1130) for this purpose.

According to an embodiment of the present disclosure, model parameters of the AI restoration ISP (1130) are adjusted to minimize the difference between the image output when an image containing noise is input to the AI restoration ISP (1130) and the image from which the noise has been removed. Learning can be performed by updating.

FIG. 6 is a diagram illustrating a method of training a neural network model included in an image restoration module according to an embodiment of the present disclosure. Referring to FIG. 6, a noise image 62 on which noise simulation 610 has been performed on the input image 61 is input to the AI restoration ISP 1130, and the AI restoration ISP 1130 generates an inferred image 63 therefrom. can be output. At this time, the input image 61 is a label image to be compared with the inferred image 63 and may correspond to ground truth data. Noise simulation 610 refers to the operation of adding noise to the input image 61 to generate a noise image 62.

The method of generating training data by performing noise simulation 610 on the input image 61 is as follows. The input image 61 may be a clean image containing little noise, and noise may be added by performing a noise simulation 610 while varying the noise intensity on various input images 61 stored in the database. Image 62 can be acquired. As described above, since the input image 61 corresponds to a label image, a pair of the input image 61 and the noise image 62 can be used as training data.

According to one embodiment, training data can be generated in advance by performing noise simulation 610 on all input images 61 stored in the database. Alternatively, according to one embodiment, training data may be generated by performing noise simulation 610 while adjusting the intensity whenever training data is needed during the learning process. (on-the-fly method)

When generating training data for learning a daytime model, a noise image 62 can be generated by adding noise of an intensity corresponding to the daytime model to the input image 61 through noise simulation 610. For example, in the embodiment described above with reference to Table 1, the noise image 62 can be generated by adding shot noise of less than 0.001 to the input image 61 through noise simulation 610.

When generating training data for learning a night model using a similar principle, noise image 62 can be generated by adding noise of intensity corresponding to the night model to the input image 61 through noise simulation 610. there is. For example, in the embodiment described above with reference to Table 1, the noise image 62 can be generated by adding shot noise of 0.001 or more to the input image 61 through noise simulation 610.

The optimizer 630 may update the model parameters of the AI restoration ISP 1130 to minimize the results output when the inferred image 63 and the input image 61 are input to the loss function 620. Accordingly, the AI restoration ISP 1130 can be trained to infer an image as close as possible to the image from which noise has been removed when an image with noise added is input.

The noise simulation 610, calculation of the loss function 620, and parameter update of the optimizer 630 described above may be performed by the processor of the computing device responsible for learning the AI restoration ISP 1130.

3. Neural network model selection in image correction module

According to an embodiment of the present disclosure, a neural network model of the image correction module 1200 may be selected according to a process performed by the image restoration module 1100. For example, if the image restoration module 1100 includes a plurality of detailed configurations and the process performed for each detailed configuration is different, the neural network of the image correction module 1200 according to the detailed configuration selected in the image restoration module 1100 A model may be selected.

FIG. 7 is a diagram illustrating a method of selecting a neural network model of an image correction module according to an image restoration module, according to an embodiment of the present disclosure. Referring to FIG. 7, the HW ISP 1110 is selected in the image restoration module 1100. It is assumed that the HW ISP 1110 performs demosaicing during the restoration process, but does not perform denoising.

The HW ISP 1110 may transmit information about the process it performs (demosaicing is performed, denoising is not performed) to the image correction module 1200. The image correction module 1200 can know that denoising was not performed in the image restoration step through the information received from the HW ISP 1110, and therefore selects a second correction parameter supporting the denoise function to use the neural network model. can be set. At this time, the second correction parameter may correspond to a neural network model previously learned to correspond to the HW ISP (1110). In other words, the second correction parameter may be a neural network model learned to perform not only white balance adjustment, color correction, and gamma value adjustment, but also denoise in order to correct an image on which denoising has not been performed.

As shown in FIG. 7, the second correction parameter adjusts the white balance (WB) using the white balance gain (WBG) and color correction matrix (CCM) obtained from the image sensor 100. , color correction (CC) and gamma value adjustment can also be performed.

Although not shown in the drawing, if the SW ISP 1120 is selected in the image restoration module 1100 and the SW ISP 1120 performs demosaicing and denoising, the image correction module 1200 performs demosaicing and denoising. ) A learned neural network model (model parameter) may be selected to correspond to, and at this time, the selected neural network model may only perform white balance adjustment, color correction, and gamma value adjustment without performing denoising. That is, depending on what process is performed in the image restoration module 1100, a neural network model may be selected in the image correction module 1200.

As described above, according to one embodiment, the neural network models (model parameters) included in the image correction module 1200 may be learned in advance to correspond to each detailed configuration of the image restoration module 1100, and the image restoration module ( When a detailed configuration is selected in 1100), a neural network model learned to correspond to the selected detailed configuration may be selected in the image correction module 1200. And for this purpose, the image correction module 1200 may receive information about the process performed by the image restoration module 1100 from the image restoration module 1100.

So far, we have explained how to select a neural network model in the AI restoration ISP 1130 based on shooting conditions and how to select a neural network model in the image correction module 1200 according to the process performed by the image restoration module 1100. did.

Below, based on what has been explained so far, embodiments of selecting the type of image sensor, the detailed configuration of the image restoration module, and the image correction module according to the shooting environment will first be described, and then the neural network model included in the image correction module. Explains how to learn.

4. Selection of image sensor, image restoration module, and image correction module according to the shooting environment

According to one embodiment, the type of image sensor, the detailed configuration of the image restoration module, and the image correction module may be selected according to the shooting environment (shooting conditions and shooting mode). 8 to 10 are diagrams for explaining a method of selecting an image sensor, an image restoration module, and an image correction module according to a shooting environment according to embodiments of the present disclosure. As shown in FIGS. 8 to 10 , it is assumed that the image sensor 100 includes a plurality of image sensors. Additionally, it is assumed that the shooting modes include daytime shooting mode (normal shooting mode) (FIG. 8), zoom shooting mode (FIG. 9), and night shooting mode (FIG. 10).

According to one embodiment, the shooting mode can be set by the user through the UI of a computing device (e.g. the user terminal 3000 in FIG. 3) that performs image signal processing. For example, a user can run a camera application on the user terminal 3000 and select an appropriate shooting mode depending on the situation before shooting. When the shooting mode is set, the user terminal 3000 can select the type of the image sensor 100 and the detailed configuration of the image restoration module 1100 and the image correction module 1200 accordingly.

(1) Selection of image sensor

According to an embodiment of the present disclosure, the type of image sensor 100 may be selected based on the shooting mode. For example, the image sensor 100 may include a general image sensor, a wide-angle image sensor, and an ultra-wide-angle image sensor, and these may be matched to each shooting mode as needed.

In the embodiments of FIGS. 8 to 10 , it is assumed that the first image sensor is matched to the daytime shooting mode, the second image sensor is matched to the zoom shooting mode, and the third image sensor is matched to the night shooting mode. Referring to Figures 8 to 10, it can be seen that a corresponding image sensor is selected according to each shooting mode.

Additionally, according to one embodiment, the type of image sensor 100 may be selected based on shooting conditions. For example, if it is determined to be a low-light environment considering the time and place where photography is performed, an image sensor 100 suitable for night photography may be selected.

(2) Selection of image restoration module

According to an embodiment of the present disclosure, the detailed configuration of the image restoration module 1100 may be selected according to the shooting mode or the type of image sensor 100 used when shooting an image. To this end, the detailed configurations of the image restoration module 1100 may be matched to each shooting mode or to the type of the image sensor 100.

Referring to Figures 8 to 10, when in daytime shooting mode (when the first image sensor is selected), the HW ISP (1110) is selected, and when in zoom shooting mode (when the second image sensor is selected) It can be seen that the SW ISP 1120 has been selected, and in the night shooting mode (when the third image sensor is selected), the AI restoration ISP 1130 has been selected.

Referring to FIG. 10, the second restoration parameter is selected in the AI restoration ISP 1130, and as previously described with reference to FIG. 5, the shooting conditions (e.g. ISO value included in the metadata of the image captured by the third image sensor) ) A second restoration parameter may be selected from among a plurality of restoration parameters based on .

Additionally, according to one embodiment, when selecting the detailed configuration of the image restoration module 1100, not only the shooting mode or the type of the image sensor 100 but also the shooting conditions may be further considered. For example, in the embodiment shown in FIG. 8, when the first image sensor is selected due to the daytime shooting mode, the HW ISP 1110 may be selected in the image restoration module 1100, and even in this case, if the ISO value is constant If it exceeds the standard, either the SW ISP (1120) or the AI restoration ISP (1130) may be selected.

Additionally, according to one embodiment, the detailed configuration of the image restoration module 1100 may be selected considering only shooting conditions. For example, the detailed configuration corresponding to each section of the ISO value is predetermined, and the image restoration module 1100 may select the detailed configuration according to the ISO value obtained from the metadata of the image.

In summary, the detailed configuration of the image restoration module 1100 may be selected based on at least one of the type of image sensor 100, shooting mode, and shooting conditions. In particular, if the type of image sensor 100 is selected according to the shooting environment (shooting mode and shooting conditions), the detailed configuration of the image restoration module 1100 may be selected accordingly.

(3) Selection of image correction module

As previously described with reference to FIG. 7 , according to one embodiment, the neural network model of the image correction module 1200 may be selected according to the process performed by the image restoration module 1100. However, the process performed by the image restoration module 1100 may be determined depending on which detailed configuration is selected in the image restoration module 1100, and the detailed configuration of the image restoration module 1100 may be determined by the type of image sensor, shooting conditions, and shooting conditions. It may be selected based on at least one of the modes. Therefore, in summary, the neural network model of the image correction module 1200 may also be selected based on at least one of the type of image sensor, shooting mode, and shooting conditions.

According to an embodiment of the present disclosure, there may be a neural network model of the image correction module 1200 corresponding to each combination of the image sensor 100 and the image restoration module 1100. Therefore, according to one embodiment, when the shooting mode is determined, the type of the image sensor 100 and the detailed configuration of the image restoration module 1100 may be selected according to the shooting mode, and the selected image sensor 100 and the image restoration module ( 1100), any one of a plurality of neural network models included in the image correction module 1200 may be selected. Additionally, according to one embodiment, shooting conditions (e.g. ISO value) may also be considered when selecting the image sensor 100, the image restoration module 1100, and the image correction module 1200.

According to one embodiment, the type of image sensor 100 may be reflected when selecting a neural network model in the image correction module 1200, so the image correction module 1200 receives image sensor selection information (which image Information about whether a sensor has been selected) can be obtained.

Meanwhile, the image correction module 1200 can obtain the white balance gain (WBG) and color correction matrix (CCM) from the image sensor 100 and use them to perform white balance adjustment, color correction, and gamma value adjustment. there is.

Referring to FIG. 8 , in the daytime shooting mode, the first image sensor may be selected in the image sensor 100, and the HW ISP 1110 corresponding to the first image sensor may be selected in the image restoration module 1100. According to one embodiment, shooting conditions may also be considered when selecting the detailed configuration of the image restoration module 1100. For example, even if it is daytime shooting mode and the first image sensor is selected, if the ISO value is higher than a certain standard, Instead of the HW ISP 1110, either the SW ISP 1120 or the AI restoration ISP 1130 may be selected.

In the image correction module 1200, a first correction parameter corresponding to the HW ISP 1110 may be selected. At this time, the first correction parameter may be a neural network model learned to correspond to the HW ISP (1110). For example, if the HW ISP 1110 only performs demosaicing and does not denoise, the first correction parameter may be a neural network model learned to perform denoising as well as white balance adjustment, color correction, and gamma value adjustment. there is.

Meanwhile, as described above, the type of image sensor 100 can be considered when selecting a neural network model in the image correction module 1200. For example, the HW ISP 1110 was selected in the same way in the image restoration module 1100. , if the second or third image sensor is selected in the image sensor 100, a correction parameter other than the first correction parameter may be selected in the image correction module 1200. Accordingly, the neural network models included in the image correction module 1200 may be learned to correspond to the combination of the image sensor 100 and the image restoration module 1100. Additionally, neural network models included in the image correction module 1200 may be learned to correspond to the shooting environment.

Referring to FIG. 9 , in the zoom shooting mode, the second image sensor may be selected in the image sensor 100, and the SW ISP 1120 corresponding to the second image sensor may be selected in the image restoration module 1100. According to one embodiment, the second image sensor is an image sensor capable of capturing multiple frames and may be an image sensor that matches a wide-angle lens, and may be easily enlarged by combining a plurality of frames obtained through the second image sensor. You can.

According to one embodiment, in zoom shooting mode, when the second image sensor transmits an image physically enlarged by a certain number to the SW ISP (1120), the SW ISP (1120) performs a center crop on the received image. After performing , the cropped area can be enlarged by a certain number of times through an algorithm. Additionally, the SW ISP 1120 can further perform demosaicing and super resolution (SR) to output a linear RGB image.

According to one embodiment, shooting conditions may also be considered when selecting the detailed configuration of the image restoration module 1100. For example, even if it is zoom shooting mode and the second image sensor is selected, if the ISO value is higher than a certain standard, The AI restoration ISP (1130) may be selected instead of the SW ISP (1120). Additionally, within the AI restoration ISP 1130, a neural network model (restoration parameter) may be selected according to the ISO value.

In the image correction module 1200, a second correction parameter corresponding to the SW ISP 1120 may be selected. At this time, the second correction parameter may be a neural network model learned to correspond to the zoom shooting mode and the SW ISP (1120). For example, the second correction parameter may be a neural network model learned in advance to correct the image to have image characteristics suitable for the enlarged image. At this time, 'image characteristics suitable for the enlarged image' may not be uniformly determined, but may be set in advance by the user or administrator based on objective statistical data or individual or multiple preferences. Also, for example, the second correction parameter may be a neural network model learned to correct image characteristics (e.g., correct to increase contrast ratio) so that objects in the final image appear as clearly and clearly as possible.

Also, according to one embodiment, the SW ISP 1120 corresponding to the zoom shooting mode performs center cropping, demosaicing, and super resolution (but does not perform denoising), and accordingly, the second correction parameter includes white balance adjustment, It may be a neural network model trained to perform color correction and gamma value adjustment as well as denoise.

Meanwhile, as described above, the type of image sensor 100 can be considered when selecting a neural network model in the image correction module 1200. For example, the SW ISP 1120 was selected in the same way in the image restoration module 1100. , if the first image sensor or the third image sensor is selected in the image sensor 100, a correction parameter other than the second correction parameter may be selected in the image correction module 1200. Accordingly, the neural network models included in the image correction module 1200 may be learned to correspond to the combination of the image sensor 100 and the image restoration module 1100. Additionally, neural network models included in the image correction module 1200 may be learned to correspond to the shooting environment.

Referring to FIG. 10, in the night shooting mode, the third image sensor may be selected in the image sensor 100, and the AI restoration ISP 1130 corresponding to the third image sensor may be selected in the image restoration module 1100. . According to one embodiment, the third image sensor may be an image sensor matching a wide-angle lens. As previously described with reference to FIG. 5 , the AI restoration ISP 1130 may select one (second restoration parameter) from a plurality of neural network models (restoration parameters) based on the ISO value. Additionally, the second restoration parameter selected by the AI restoration ISP 1130 may perform denoising at an intensity determined based on the ISO value.

According to one embodiment, shooting conditions may also be considered when selecting the detailed configuration of the image restoration module 1100. For example, even if it is night shooting mode and a third image sensor is selected, if the ISO value is lower than a certain standard, Instead of the AI restoration ISP 1130, either the SW ISP 1120 or the HW ISP 1110 may be selected.

In the image correction module 1200, a third correction parameter corresponding to the second restoration parameter of the AI restoration ISP 1120 may be selected. At this time, the third correction parameter may be a neural network model learned to correspond to the second restoration parameter of the AI restoration ISP (1120). For example, the third correction parameter may be a neural network model learned to correct image characteristics to brighten dark areas as clearly as possible (e.g. to increase the brightness of a specific area).

In addition, according to one embodiment, if demosaicing and denoising are performed in the second restoration parameter of the AI restoration ISP 1120, the third correction parameter does not perform denoising, but white balance adjustment, color correction, gamma value adjustment, etc. It may be a neural network model trained to perform.

Meanwhile, as described above, the type of image sensor 100 can be considered when selecting a neural network model in the image correction module 1200. For example, in the image restoration module 1100, the AI restoration ISP 1130 is selected in the same way. However, if the second or third image sensor is selected in the image sensor 100, a correction parameter other than the third correction parameter may be selected in the image correction module 1200. Accordingly, the neural network models included in the image correction module 1200 may be learned to correspond to the combination of the image sensor 100 and the image restoration module 1100. Additionally, neural network models included in the image correction module 1200 may be learned to correspond to the shooting environment.

As described above, the neural network model of the image correction module 1200 may be selected based on at least one of the type of image sensor 100, shooting mode, and shooting conditions. In particular, if the type of image sensor 100 and the detailed configuration of the image restoration module 1100 are selected according to the shooting environment (shooting mode and shooting conditions), the neural network model of the image correction module 1200 may be selected accordingly. .

Additionally, according to one embodiment, the neural network model of the image correction module 1200 may be selected to correspond to each shooting mode, and the selected neural network model may be selected in advance in the direction of appropriately correcting image characteristics according to the corresponding shooting mode. It may be something learned. At this time, the 'direction of appropriately correcting image characteristics according to each shooting mode' may be set in advance by the user or administrator based on objective statistical data or individual or majority preferences.

Below, the processes of the embodiments described so far will be described with reference to the flow chart. 11 to 14 are flowcharts for explaining a method of processing an image signal using a neural network model according to embodiments of the present disclosure.

Referring to FIG. 11, in step 1101, a computing device (e.g. user terminal 3000 of FIG. 3) may acquire an image captured through an image sensor. At this time, the image sensor may be selected from among a plurality of different types of image sensors based on at least one of shooting conditions and shooting mode.

In step 1102, the computing device can check the shooting environment (shooting conditions, shooting mode) of the image. According to one embodiment, the computing device may obtain shooting conditions, such as the time and place where the image was captured, and the ISO value, from the metadata of the image. If the computing device includes an image sensor, that is, if an image is captured through the computing device, the computing device can directly check the shooting mode set at the time of shooting. Alternatively, if the computing device does not include an image sensor and only receives captured images and performs image signal processing, the computing device may check the capturing mode from metadata of the received images.

According to one embodiment, the shooting environment may include at least one of shooting conditions and shooting modes. The shooting conditions may include at least one of the ISO value of the image and the time and place at which the image was captured. The shooting mode may include at least one of a normal shooting mode, a night shooting mode, and a zoom shooting mode.

In step 1103, the computing device may select a neural network model included in at least one of the image restoration module and the image correction module according to the shooting environment. Detailed steps included in step 1103 are shown in FIG. 12.

Referring to FIG. 12 , in step 1201, the computing device may select a detailed configuration of the image restoration module based on at least one of the shooting environment and the type of the selected image sensor. Detailed steps included in step 1201 are shown in FIG. 13.

Referring to FIG. 13, in step 1301, the computing device may initially select a detailed configuration based on at least one of the shooting mode or the type of the selected image sensor.

In step 1302, the computing device may maintain or change the primarily selected detailed configuration based on the shooting conditions. For example, if it is necessary to change the primarily selected detailed configuration considering the ISO value included in the shooting condition, the computing device may select a new detailed configuration based on the ISO value. According to one embodiment, the detailed configuration of the image restoration module corresponding to the shooting mode or type of image sensor may be specified in advance.

Returning to FIG. 12, according to one embodiment, in selecting the detailed configuration of the image restoration module in step 1201, the computing device uses a plurality of neural network models included in the image restoration module based on the ISO value included in the shooting condition. You can choose any one of them. At this time, the neural network model selected in the image restoration module includes a denoise function and may be a neural network model learned using an image containing noise corresponding to the ISO value included in the shooting conditions.

In step 1202, the computing device may select one of a plurality of neural network models included in the image correction module based on at least one of the shooting environment, the type of the selected image sensor, and the detailed configuration of the selected image restoration module. According to one embodiment, the computing device may select a neural network model corresponding to a combination of the type of the selected image sensor and the detailed configuration of the selected image restoration module. At this time, the selected neural network model may be a neural network model learned to correct image characteristics according to the shooting environment in which the image was captured.

Returning to FIG. 11 , in step 1104, the computing device may process the image using the selected neural network model.

Figure 14 shows a flow chart to explain an embodiment of sequentially selecting the type of image sensor, the detailed configuration of the image restoration module, and the neural network model of the image correction module according to the shooting environment.

Referring to FIG. 14, in step 1401, the computing device may select the type of image sensor based on shooting conditions and shooting mode. According to one embodiment, the type of image sensor corresponding to each combination of shooting conditions and shooting modes may be predetermined. Or, according to one embodiment, the type of image sensor corresponding to each shooting mode may be predetermined. Or, according to one embodiment, the type of image sensor corresponding to each shooting mode is predetermined, so the computing device basically selects the type of image sensor according to the shooting mode, but if the shooting condition meets a specific standard, another image sensor is selected. You can also select .

In step 1402, the computing device may select the detailed configuration of the image restoration module according to the type of image sensor selected. To this end, the detailed configuration of the image restoration module corresponding to each type of image sensor may be predetermined. For example, if the selected image sensor is an image sensor for performing zoom photography, detailed configuration for performing center cropping and enlargement may be selected in the image restoration module.

In step 1403, the computing device may select one of a plurality of neural network models included in the image correction module according to a combination of the type of the selected image sensor and the detailed configuration of the selected image restoration module. At this time, the selected neural network model may be a neural network model learned to correct image characteristics according to the shooting environment corresponding to the combination of the type of the selected image sensor and the detailed configuration of the selected image restoration module. For example, if the shooting environment corresponding to the combination of the type of image sensor selected and the detailed configuration of the selected image restoration module is shooting in zoom shooting mode in low light, the neural network model selected in the image correction module is based on brightness and contrast ratio. It could be a neural network model trained to increase .

In step 1404, the computing device may process the image captured by the selected image sensor using the detailed configuration of the selected image restoration module and the neural network model of the selected image correction module.

4. Application Example - Selection of an image sensor, image restoration module, and image correction module reflecting object recognition results

Hereinafter, as a specific embodiment, when shooting in object recognition mode (e.g. AI camera mode that recognizes objects), the method of changing the combination of the image sensor, image restoration module, and image correction module according to the object recognition result is shown in Figure 15. It is explained with reference to .

Referring to FIG. 15, an image is captured by the first image sensor according to the object recognition mode, and the captured image may be processed through the AI restoration ISP 1130 of the image restoration module 1100. In the embodiment of FIG. 15, the AI restoration ISP 1130 processes the image signal with the second restoration parameter set, and the second restoration parameter is based on the shooting mode (object recognition mode) and shooting conditions (e.g. ISO value). So it may be selected.

In the embodiment of FIG. 15 , a third correction parameter corresponding to a combination of the first image sensor and the second restoration parameter is selected in the image correction module 1200. According to one embodiment, the third correction parameter may be a neural network model learned to correct image characteristics in the direction of increasing object recognition accuracy.

The object recognition module 1510 may recognize an object from the image output by the image correction module 1200. According to one embodiment, the object recognition module 1510 may request a change to the image sensor 100 based on the object recognition result. Additionally, according to one embodiment, the object recognition module 1510 may recommend a different shooting mode to the user or automatically change the shooting mode based on the object recognition result.

For example, if the moon is recognized as a result of object recognition, the object recognition module 1510 requests the image sensor 100 to change to an image sensor matched to a wide-angle lens, or instructs the user to change to a wide-angle shooting mode. It is recommended to change, or it can automatically change to wide-angle shooting mode.

If the type or shooting mode of the image sensor 100 is changed according to the object recognition result, the detailed configuration of the image restoration module 1100 and the neural network model of the image correction module 1200 may also be changed accordingly.

In this way, if the type of image sensor 100, the detailed configuration of the image restoration module 1100, and the neural network model of the image correction module 1200 are changed according to the object recognition result, the image signal is processed again according to the changed settings to produce the final image. can be output.

In the embodiment shown in FIG. 15, the object recognition module 1510 recognizes an object from the image output from the image correction module 1200, but the object recognition module 1510 may also perform object recognition at another location. For example, the object recognition module 1510 may recognize an object from an image output from the image sensor 100 or may recognize an object from an image output from the image restoration module 1100.

5. Learning of the neural network model included in the image correction module

As described above, the neural network models included in the image correction module 1200 can be learned by setting the correction direction of image characteristics differently for each shooting environment (e.g. increasing contrast ratio in zoom shooting mode, increasing erection in night shooting mode, etc.). Hereinafter, a method of learning a neural network model included in the image correction module 1200 will be described with reference to FIGS. 16 to 33.

In the following embodiments, it is assumed that the neural network models included in the image correction module 1200 each include two neural network models (a first neural network model and a second neural network model). In other words, each correction parameter included in the image correction module 1200 includes both parameters for setting the first neural network model and parameters for setting the second neural network model.

Each neural network model included in the image correction module 1200 according to embodiments of the present disclosure includes a 'first neural network model' that corrects the input image and outputs an inferred image, and an optimal correction parameter for a given input image. It may include a ‘second neural network model’ for inference.

(1) Description of the basic operation of the first neural network model and the second neural network model

First, with reference to FIGS. 16 and 17, the roles of the first neural network model 1210 and the second neural network model 1220 will be briefly described, and then with reference to FIGS. 18 and 19, the first neural network model to perform those roles will be described briefly. How to train the model 1210 and the second neural network model 1220 will be described.

FIG. 16 is a diagram for explaining the operation of the first neural network model 1210 according to an embodiment of the present disclosure. Referring to FIG. 16, when the first neural network model 1210 according to one embodiment receives the input image 161 and the correction parameter 1610, it outputs the inference image 162 corresponding to the correction parameter 1610. You can. At this time, the inferred image 162 corresponds to an image in which the input image 161 has been corrected based on the correction parameter 1610. How the first neural network model 1210 is learned to output the inferred image 162 will be described in detail below with reference to FIG. 18.

In general, the input image 161 may have a problem in which the entire or partial area of the image is dark or the contrast ratio is low, making it difficult for objects in the image to be recognized. The first neural network model 1210 may serve to increase the brightness or improve the contrast ratio of the input image 161 to solve this problem.

The first neural network model 1210 may output the inferred image 162 as shown in FIG. 16, but does not output the image itself and uses a filter ( FIG. filter) or map information can also be output. That is, the input image 161 may be converted into the inferred image 162 using the filter or map information output by the first neural network model 1210.

The first neural network model 1210 may be implemented to include various types of deep learning networks, for example, it may be implemented as ResNet (Residual Network), a type of convolution neural network (CNN). It is not limited to this.

FIG. 17 is a diagram for explaining the operation of the second neural network model 1220 according to an embodiment of the present disclosure. Referring to FIG. 17, the second neural network model 1220 according to one embodiment may receive the input image 161, infer the correction parameter 1610, and output it. The correction parameter 1610 output from the second neural network model 1220 is input to the first neural network model 1210, and the first neural network model 1210 is input to the correction parameter 1610 as previously described with reference to FIG. 16. The corresponding inferred image 162 can be output.

The second neural network model 1220 can infer a correction parameter 1610 for correcting the input image 161 to have image characteristics that the user may like. At this time, the direction in which the input image 161 is corrected due to the correction parameter 1610 inferred by the second neural network model 1220 may be determined during the learning process of the second neural network model 1220. Details are provided below. This will be described with reference to FIG. 19.

The second neural network model 1220 may also be implemented to include various types of deep learning networks, for example, it may be implemented as ResNet (Residual Network), a type of convolution neural network (CNN). It is not limited to this.

(2) (Learning process) Method of training the first neural network model and the second neural network model

Hereinafter, a method of learning the first neural network model 1210 and the second neural network model 1220 will be described with reference to FIGS. 18 and 19.

According to one embodiment, the learning of the neural network models 1210 and 1220 is performed using a device other than the device on which the neural network models 1210 and 1220 are mounted (e.g. the user terminal 2000 in FIG. 3) and another external device (e.g. FIG. 31). It can be performed by the computing device 500). Of course, depending on the embodiment, a device equipped with the neural network models 1210 and 1220 may perform learning of the neural network models 1210 and 1220. Hereinafter, for convenience of explanation, the processor 530 of the computing device 500 of FIG. 31 executes the program stored in the memory 520 to learn the neural network models 1210 and 1220 described in FIGS. 18 and 19. Assume that you are performing: That is, the operations performed by the label image generation module 1810 or the optimizer 1820 and the calculation of the loss function in FIGS. 18 and 19 are actually performed by the processor ( 530) can be seen as carrying out this.

The processor 530 of the computing device 500 of FIG. 31 may include a CPU 531, a GPU 532, and an NPU 533, and executes a program stored in the memory 520 using at least one of these. By doing so, learning about the neural network models (1210, 1220) can be performed. The input/output interface 510 of the computing device 500 may receive commands or display information related to learning of the neural network models 1210 and 1220.

FIG. 18 is a diagram for explaining a process of learning the first neural network model 1210 according to an embodiment of the present disclosure. Referring to FIG. 18 , the label image generation module 1810 may output a label image 163 obtained by correcting the input image 161 using at least one image correction algorithm. When the label image creation module 1810 corrects the input image 161, a correction parameter 1610 is applied to the image correction algorithm. The image correction algorithm used by the label image creation module 1810, and the correction parameter 1610 This will be described in detail with reference to FIGS. 20 to 23.

FIG. 20 is a diagram illustrating a process in which the label image generation module 1810 generates the label image 163 from the input image 161 according to an embodiment of the present disclosure. Referring to FIG. 20, the label image generation module 1810 may use a contrast enhancement algorithm 1811, a brightness correction algorithm 1812, and an exposure fusion algorithm 1813. . However, the image correction algorithms used by the label image generation module 1810 disclosed in FIG. 20 are only examples, and the label image generation module 1810 may also use various types of image correction algorithms.

FIG. 20 shows which parameters are specifically included in the correction parameter 1610 and which image correction algorithm each parameter corresponds to. Referring to FIG. 20, a strength value 1611 may be applied to the contrast ratio improvement algorithm 1811, a gamma value 1612 may be applied to the brightness correction algorithm 1812, and the exposure fusion algorithm ( 1813), the α value (1613) can be applied. The parameters 1611, 1612, and 1613 applied to each image correction algorithm will be described with reference to FIGS. 21 to 23.

FIG. 21 is a diagram for explaining the strength value 1611 applied to the contrast ratio improvement algorithm 1811 according to an embodiment of the present disclosure. The contrast ratio improvement algorithm 1811 according to one embodiment is a method of adjusting the contrast ratio using a curve.

Referring to the graph 2100 of FIG. 21, the curvature of the S-curve changes depending on the strength value 1611, and in particular, as the strength value 1611 increases, the curvature of the S-curve increases. Able to know. As the slope of the S-curve increases, the tone is stretched, thus increasing the contrast ratio. Accordingly, as the strength value 1611 increases, the contrast ratio improvement algorithm 1811 can increase the contrast ratio of the input image 161.

FIG. 22 is a diagram for explaining the gamma value 1612 applied to the brightness correction algorithm 1812 according to an embodiment of the present disclosure. The graph 2200 of FIG. 22 shows a gamma curve according to the gamma value 1612. The x-axis of the graph 2200 represents the contrast expression value, and the y-axis represents brightness. Referring to the graph 2200 of FIG. 22, it can be seen that the brightness increases as the gamma value 1612 decreases. Accordingly, as the gamma value 1612 becomes smaller, the brightness correction algorithm 1812 can increase the brightness of the input image 161.

FIG. 23 is a diagram for explaining the α value 1613 applied to the exposure fusion algorithm 1813 according to an embodiment of the present disclosure. The exposure fusion algorithm 1813 generates a processed image (X') with different exposure from the base image (X), and multiplies the base image (X) and the processed image (X') by weights W and (1-W), respectively. After combining them, a fused image (F) can be output. According to one embodiment, the base image (X) may correspond to the input image 161 of FIG. 20, and the output fused image (F) may correspond to the label image 163 of FIG. 20.

The exposure fusion algorithm 1813 can generate a processed image (X') from the basic image (X) according to Equation 1 below.

If the exposure fusion algorithm 1813 is performed according to the method described above, the larger the α value 1613, the brighter the fusion image (F) becomes compared to the basic image (X), and the smaller the β value, the brighter the fusion image (F) becomes. may become brighter compared to the basic image (X). In Figure 5, it is shown that the α value 1613 is used as a correction parameter for the exposure fusion algorithm 1813. However, the β value is used as a correction parameter instead of the α value 1613, or both the α value 1613 and the β value are used as a correction parameter. may also be used as a correction parameter.

The label image generation module 1810 adjusts at least one of the characteristics of the input image 161 by applying correction parameters 1611, 1612, and 1613 to the image correction algorithms 1811, 1812, and 1813 described above. Video 163 can be output. The label image generation module 1810 may use a combination of at least one of the image correction algorithms 1811, 1812, and 1813 shown in FIG. 20, or may use another type of image correction algorithm not shown in FIG. 20. there is.

According to one embodiment, the order of the parameter values 1611, 1612, and 1613 included in the correction parameter 1610 may remain the same during learning and inference. In other words, according to one embodiment, the order of parameter types 1611, 1612, and 1613 in the correction parameter 1610 used during learning may remain the same even during use.

For example, if the correction parameter 1610 is in the form of a column vector, the elements of each row are the same type of parameter (parameters applied to the same image correction algorithm) during learning and inference. ) may apply.

To explain with a specific example, it is as follows.

As in the embodiment shown in FIG. 20, the element of the first row of the correction parameter (column vector) 1610 used during learning is the parameter 1611 applied to the contrast ratio improvement algorithm 1811, and the two If the element in the third row is a parameter 1612 applied to the brightness correction algorithm 1812, and the element in the third row is a parameter 1613 applied to the exposure fusion algorithm 1813, the correction parameter used during inference is also the first The element in the first row is the parameter 1611 applied to the contrast ratio improvement algorithm 1811, the element in the second row is the parameter 1612 applied to the brightness correction algorithm 1812, and the element in the third row is the exposure fusion algorithm. It may be a parameter (1613) applied to (1813).

In relation to this, it is explained with reference to FIG. 24 as follows.

Referring to FIG. 24, the two parameter values 1614 and 1615 are converted to the input image 161 and the grayscale image converted from the input image 161 and scaled concatenation (see 2501 in FIG. 25) and then applied to the first neural network. It is input into model 1210. That is, grayscale images in which the parameter values 1614 and 1615 are multiplied (or divided or added) are attached to the input image 161 in the channel direction and input to the first neural network model 1210. At this time, the channel order is the same as during training. Since they must be the same during inference, as described above, the order of the parameter values 1614 and 1615 included in the correction parameter 1610 must remain the same during learning and during inference.

Also, according to one embodiment, the number of bits of each of the parameter values 1611, 1612, and 1613 included in the correction parameter 1610 in the embodiment shown in FIG. 20 will remain the same during learning and inference. You can. A detailed description of an embodiment in which the order of the parameter values 1611, 1612, and 1613 included in the correction parameter 1610 and the number of bits of each of the parameter values 1611, 1612, and 1613 are kept the same during learning and inference. If you do so, it is as follows.

It is assumed that the correction parameter 1610 used when learning the neural network models 1210 and 1220 is binary data of “01010110”. In order from the beginning, 3 bits of data (“010”) are parameters 1611 to be applied to the contrast ratio improvement algorithm 1811, and the following 2 bits of data (“10”) are used to apply the brightness correction algorithm 1812. It is assumed that this is the parameter 1612 to be applied, and that the last 3 bits of data (“110”) are the parameters 1613 to be applied to the exposure fusion algorithm 1813.

The number of bits of each parameter (1611, 1612, 1613) and the order between the parameters (1611, 1612, 1613) must remain the same even when using (inference) the neural network models (1210, 1220). For example, if binary data of "10111100" is input as the correction parameter 1610 to the first neural network model 1210 during inference, 3 bits of data ("101") are input to the contrast ratio improvement algorithm 1811 in order from the beginning. ), the next 2 bits of data ("11") become the parameters 1612 to be applied to the brightness correction algorithm 1812, and the last 3 bits of data ("100") becomes a parameter 1613 to be applied to the exposure fusion algorithm 1813.

Returning to FIG. 18 , the label image 163 output by the label image generation module 1810 can be used as training data (particularly, ground truth data) for learning the first neural network model 1210. When the input image 161 and the correction parameter 1610 are input to the first neural network model 1210, the inferred image 162 output from the first neural network model 1210 is made to be as similar as possible to the label image 163. The first neural network model 1210 can be trained. That is, according to one embodiment, the first neural network model 1210 includes an inference image 162 output when the input image 161 and the correction parameter 1610 are input, and a label image corresponding to the correction parameter 1610 ( 163) It may be a model learned to minimize the differences between At this time, the label image 163 corresponding to the correction parameter 1610 means that when the correction parameter 1610 is applied to at least one image correction algorithm, the input image 161 is corrected by at least one image correction algorithm. It could be a video.

In other words, the optimizer 1820 can update the first neural network model 1210 so that the loss value of the loss function 1620 representing the difference between the inference image 162 and the label image 163 is minimized. there is. At this time, the loss function 1620 may be composed of a combination of Mean Absolute Error (MAE), Mean Square Error (MSE), and Structural Similarity Index Measure (SSIM).

As described above, when learning the first neural network model 1210, the input image 161, the label image 163, and the correction parameter 1610 may be used as training data. According to one embodiment, a plurality of label images 163 are generated by changing the correction parameters 1610 for several input images 161, and the input images 161, label images 163, and corrections collected in this way are generated. The first neural network model 1210 can be learned using a combination of parameters 1610 as training data.

In this way, when learning the first neural network model 1210, the correction parameter 1610 is used as an input to the first neural network model 1210, so that the relationship between the correction parameter 1610 and the label image 163 is converted into the first neural network model 1210. The model 1210 can be trained. In other words, the first neural network model 1210 can be viewed as learning how to generate the corrected label image 163 when a correction parameter 1610 is applied to the input image 161. If the input image 161 input to the learned first neural network model 1210 is kept the same and only the correction parameter 1610 is changed, the inferred image 162 output from the first neural network model 1210 is also changed. Accordingly, the user can control the inferred image 162 output from the first neural network model 1210 to have desired image characteristics by adjusting the correction parameter 1610 input to the first neural network model 1210. A specific embodiment of applying the correction parameter 1610 as an input to the first neural network model 1210 will be described below with reference to FIG. 24.

FIG. 24 is a diagram illustrating a method of applying a correction parameter as an input to a first neural network model, according to an embodiment of the present disclosure. In the embodiment shown in FIG. 24, it is assumed that the correction parameter 1610 includes a first parameter 1614 and a second parameter 1615.

Referring to FIG. 24, the input image 161 is converted into a gray scale image by the decolorize module 2410. The first operator 2421 performs multiplication on the gray-scale image and the first parameter 1614, and the second operator 2422 performs multiplication on the gray-scale image and the second parameter 1615. . The third operator 2423 performs concatenation on the input image 161, the output of the first operator 2421, and the output of the second operator 2422 and outputs it to the first neural network model 1210. can do. That is, according to one embodiment, the parameters 1614 and 1615 included in the correction parameter 1610 are each applied to the grayscale image converted from the input image 161, and then together with the input image 161, the first neural network model ( 1210).

Meanwhile, in FIG. 24, it is shown that the grayscale image is multiplied by the first parameter 1614 and the second parameter 1615 and then concatenated on the input image. However, according to one embodiment, the grayscale image is converted to a grayscale image using an operation other than multiplication. It is also possible to process the parameters 1614 and 1615. For example, the grayscale image may be divided into the first and second parameters 1614 and 1615 and then concatenated into the input image, or the first and second parameters 1614 and 1615 may be added to the grayscale image. You can then concatenate the input image.

FIG. 19 is a diagram illustrating a process of learning a second neural network model 1220 according to an embodiment of the present disclosure.

First, the reason for using the second neural network model 1220 will be explained.

The second neural network model 1220 can generally infer correction parameters for correcting the input image 161 to have image characteristics that many users would consider good (e.g. image characteristics determined to be optimal by the designer of the neural network model). Therefore, by using the second neural network model 1220, correction parameters for correcting the input image 161 to have optimal image characteristics are automatically generated (inferred) even if the user does not set or adjust the correction parameters every time, Accordingly, the input image 161 can be corrected and presented to the user. For example, when a user captures an image through a terminal equipped with neural network models 1210 and 1220, the captured image is corrected according to the correction parameter inferred by the second neural network model 1220 and previewed on the terminal. It can be displayed on the screen.

According to one embodiment, as described above, the neural network model included in the image correction module 1200 may be trained in advance to appropriately correct image characteristics according to the shooting environment. Accordingly, the user or administrator can set a desired correction direction in advance and train the second neural network model 1220 to infer correction parameters for correcting image characteristics in the set direction. In this case, the user or administrator can ensure that the image is appropriately corrected according to the shooting environment by simply setting the correction direction of the image characteristics without the need to directly adjust the correction parameter 1610.

Therefore, the user of the terminal can first check the image with good image characteristics (brightness, contrast ratio, etc.) through a preview, and adjust the correction parameters to change the image characteristics only if the preview is not to their liking.

Referring to FIG. 19, a second neural network model 1220 that receives the input image 161, infers the correction parameter 1610, and outputs it has been added. As shown in FIG. 19, with both the first neural network model 1210 and the second neural network model 1220 connected, learning for the two neural network models 1210 and 1220 may be performed simultaneously, and the first neural network model 1210 may be trained first, and the second neural network model 1220 may be trained while the learned first neural network model 1210 is fixed (the parameters included in the first neural network model 1210 are fixed). Additionally, the first neural network model 1210 and the second neural network model 1220 may be trained separately.

When learning the second neural network model 1220, the measured characteristic value 1910, which quantitatively quantifies the characteristics of the label image 163 (e.g. brightness, contrast ratio, color temperature, etc.), is compared with the preset target characteristic value 1920. , the second neural network model 1220 can be updated so that the difference between the two is minimized. The target characteristic value 1920 can be set in advance to a value desired by the user (administrator). That is, according to one embodiment, the second neural network model 1220 is an image in which the correction parameter 1610 inferred by the second neural network model 1220 when the input image 161 is input and the label image 163 are preset. It may be a model learned to minimize the difference between correction parameters that enable the label image generation module 1810 to output an image corresponding to the target feature value 1920 in FIG. 19.

In other words, the optimizer 1820 operates the second neural network model 1220 so that the loss value of the second loss function 1622 representing the difference between the measured characteristic value 1910 and the target characteristic value 1920 is minimized. can be updated. At this time, the second loss function 1622 may be composed of a combination of Mean Absolute Error (MAE), Mean Square Error (MSE), and Structural Similarity Index Measure (SSIM).

According to one embodiment, the measurement characteristic value 1910 and the target characteristic value 1920 used when learning the second neural network model 1220 may include a plurality of values corresponding to each of a plurality of image characteristics. For example, the measurement characteristic value 1910 and the target characteristic value 1920 may include a first characteristic value that quantitatively quantifies the brightness of the image, and a second characteristic value that quantitatively quantifies the color temperature of the image.

In the embodiment shown in FIG. 19, the measurement characteristic value 1910 for the label image 163 is acquired for learning the second neural network model 1220, and the obtained measurement characteristic value 1910 is used as a preset target characteristic. Compare with value (1920). However, according to one embodiment, the measurement characteristic value 1910 for the image generated in the middle of the process of converting the input image 161 to the label image 163 (hereinafter referred to as 'intermediate label image') is obtained, , the second neural network model 1220 may be learned by comparing the obtained measured characteristic value 1910 with a preset target characteristic value 1920.

A specific example will be described with reference to FIG. 20 as follows.

According to one embodiment, when measuring a specific value for learning the second neural network model 1220, some image correction algorithms among a plurality of image correction algorithms 1811, 1812, and 1813 included in the label image generation module 1810 The second neural network model 1220 may be trained by measuring feature values for the image to which only the image (middle label image) is applied, and comparing the measured feature values with preset target feature values.

In Figure 20, when the input image 161 is input to the label image generation module 1810, the input image 161 goes through the contrast ratio improvement algorithm 1811, the brightness correction algorithm 1812, and the exposure fusion algorithm 1813 in that order. Assume that it is converted to a label image (163).

For example, a quantitative measurement of the 'brightness' of the intermediate label image (the input image to the exposure fusion algorithm 1813) to which only the contrast improvement algorithm 1811 and the brightness correction algorithm 1812 have been applied to the input image 161. The characteristic value may be acquired, and the second neural network model 1220 may be trained so that the difference (loss) between the obtained measured characteristic value and the preset target characteristic value for 'brightness' is minimized. When learning the second neural network model 1220 in this way, the parameters 1613 applied to the exposure fusion algorithm 1813 are set to minimize separate loss regardless of the target brightness (brightness corresponding to the target characteristic value). It can be learned.

According to one embodiment, the second neural network model 1220 may include a plurality of neural network models. For example, when the second neural network model 1220 infers the first correction parameter and the second correction parameter, a separate neural network model for inferring each correction parameter may exist. And, if necessary, the second neural network model 1220 may be changed to infer the first to third correction parameters by adding a neural network model for inferring the third correction parameter to the second neural network model 1220.

(3) (Usage process) How to correct the image using the first neural network model and the second neural network model

As seen above, according to one embodiment, the ISP 3130 may be included in the camera module 3100 of the user terminal 3000 of FIG. 3, and the first neural network model 1210 and the second neural network model 1220 may be It may be included in the image correction module 3132 of the ISP 3130.

FIG. 25 is a diagram illustrating a specific embodiment in which neural network models 1210 and 1220 according to an embodiment of the present disclosure are included in the image correction module 3132 of FIG. 3.

Referring to FIG. 25, the image correction module 2500 according to one embodiment includes a first preprocessor 2510, a second preprocessor 2520, a first neural network model 1210, and a second neural network model 1220. It can be included.

The first preprocessor 2510 is a component for converting the input image 161 into a grayscale image, and includes the Auto White Balance (AWB) gain and the Color Correction Matrix included in the metadata. , CCM) and Gamma are applied to the input image 1, and then the input image 161 is converted to a grayscale image by performing channel wise MAX to extract the maximum value of RGB for each pixel. You can print it out.

The second preprocessor 2520 extracts the mean and variance of pixel values from the grayscale image output from the first preprocessor 2510, and applies the extracted mean and variance to the input image 161 and grayscale. It is output using scaled concatenation along with the video. The specific operation structure for performing scaled concatenation is shown in area 2501.

The second neural network model 1220 can receive the output of the second preprocessor 2520 and infer the correction parameters (α, β). The correction parameters (α, β) inferred by the second neural network model 1220 may be scaled concatenated with the input image 161 and the grayscale image and input to the first neural network model 1210.

The first neural network model 1210 can output a filter 2530 corresponding to the input image 161 and the correction parameters (α, β), and using the filter 2530 output in this way, the input image 161 It can be converted into an inference image 162.

According to one embodiment, the correction parameters (α, β) inferred by the second neural network model 1220 may be adjusted by the user. When the user adjusts the correction parameters (α, β), the first neural network model 1210 outputs a filter 2530 corresponding to the adjusted correction parameters (α, β), and uses the filter 2530 output in this way. Thus, the input image 161 can be converted into the inferred image 162.

According to one embodiment, as described above, the user previews the image corrected according to the correction parameters (α, β) deduced by the second neural network model 1220 and checks the correction parameter ( α, β) can be adjusted.

Hereinafter, a method of correcting an image using a neural network model will be described with reference to FIGS. 26 to 28. Hereinafter, for convenience of explanation, it will be described that the ISP 3130 of the user terminal 3000 of FIG. 3 (corresponding to the ISP 1000 of FIG. 1) performs the steps of FIGS. 26 to 28. However, the present invention is not limited thereto, and all or part of the steps in FIGS. 26 to 28 may be performed by the main processor 3200 of the user terminal 3000 or the processor of the cloud server 400 of FIG. 4 .

Referring to FIG. 26, in step 2601, the ISP 3130 may input an input image and correction parameters to the first neural network model 1210. The first neural network model 1210 may be a model learned to minimize the difference between an inferred image output when an input image and a correction parameter are input, and a label image corresponding to the correction parameter. At this time, the label image corresponding to the correction parameter may be an image in which the input image has been corrected by at least one image correction algorithm when the correction parameter is applied to at least one image correction algorithm.

The input image may be an image that the image restoration module 1100 restores and output from the raw images output from the image sensor 100. The correction parameter may be a preset value (e.g. a value used when learning the first neural network model 1210) or a value adjusted according to the user's input. An embodiment in which correction parameters are adjusted according to user input will be described in detail below with reference to FIGS. 28 to 30.

In step 2602, the ISP 3130 may obtain an inferred image corresponding to the correction parameter from the first neural network model 1210. The ISP 3130 may display the obtained inferred image on the screen of the user terminal 3000 or store it in the memory 3240.

Figure 27 is a diagram for explaining an embodiment in which a second neural network model infers a correction parameter. The flowchart of FIG. 27 includes detailed steps included in step 2601 of FIG. 26.

Referring to FIG. 27, in step 2701, the ISP 3130 may input an input image to the second neural network model 1220. The second neural network model 1220 is a model learned to minimize the difference between the correction parameter inferred by the second neural network model 1220 when an input image is input and the correction parameter that causes the label image to have preset image characteristics. You can.

In step 2702, the ISP 3130 obtains the correction parameters deduced by the second neural network model 1220, and in step 2703, the ISP 3130 inputs the correction parameters together with the input image into the first neural network model 1210. there is.

Figure 28 is a diagram for explaining an embodiment in which a user adjusts correction parameters. The flow chart of FIG. 28 includes detailed steps included in step 2801 of FIG. 28.

Referring to FIG. 28, in step 2801, the ISP 3130 may receive input for adjusting correction parameters through the input/output interface (e.g. touch screen, microphone, etc.) of the user terminal 3000. In step 2802, the ISP 3130 may change the preset first correction parameter to the second correction parameter based on the input received in step 2801. In step 2803, the ISP 3130 may input the second correction parameter together with the input image into the first neural network model 1210.

An example in which a user adjusts correction parameters for brightness control through the UI displayed on the screen of the user terminal 3000 is shown in FIG. 30 .

Referring to FIG. 30, the image subject to correction (an inferred image corresponding to a preset first correction parameter) is displayed in the first area 310 of the UI screen 300 for adjusting the characteristics of the image, and the UI Tools for adjusting the characteristics of the image may be displayed at the bottom of the screen 300.

In the UI screen 300 shown in FIG. 30, it is assumed that the user has selected a tool to adjust the brightness of the image. In the second area 320 of the UI screen 300, information that the characteristic of the image currently being adjusted is 'brightness' and a value expressing the current brightness level in numbers may be displayed. A slider for adjusting brightness is displayed in the third area 330 of the UI screen 300, and the user can adjust the brightness of the input image by moving the slider through touch input.

If the user moves the slider in the third area 330 of the UI screen 300 of FIG. 30 to increase the brightness of the input image, the value displayed in the second area 320 increases, and the ISP 3130 increases the image brightness. The first correction parameter may be adjusted to the second correction parameter to increase the brightness.

When a user captures an image using the user terminal 3000 and selects a tool to adjust the brightness of the image, the second neural network model 1220 inferred is displayed in the first area 310 of the UI screen 300. Depending on the correction parameters, an image in which the input image (e.g. a linear RGB image generated from a raw image that is a captured image, etc.) has been corrected may be displayed. When the user adjusts the brightness through the tools displayed in the second area 320 and the third area 330 of the UI screen 300, the correction parameter is adjusted accordingly (e.g. increased by 2 times as in FIG. 16), An image in which the input image has been corrected according to the adjusted correction parameters may be displayed in the first area 310 .

As described above, the first neural network model 1210 learns the relationship between correction parameters and label images, so when the correction parameters are changed, the label image corresponding to the changed correction parameters can be inferred. FIG. 29 is a diagram illustrating a change in an inferred image when a correction parameter input to a first neural network model is changed according to an embodiment of the present disclosure.

For convenience of explanation, it is assumed in FIG. 29 that the correction parameter 1610 includes only parameters related to brightness control. In addition, the first neural network model 1210 was learned using a label image generated by an image correction algorithm that adjusts brightness, and the image correction algorithm used during learning corrects the image so that the brightness increases as the value of the correction parameter increases. Assume that you do.

In the example shown in Figure 29, the correction parameter 1610 is doubled. The brightness of the label image corresponding to the doubled correction parameter 1610 increases compared to the label image corresponding to the existing correction parameter 1610. Accordingly, the brightness of the inferred image 162 output from the first neural network model 1210 also increases as the correction parameter 1610 increases.

Meanwhile, in the embodiment shown in FIG. 30, the user applies an input to adjust the correction parameter by touching the UI screen 300. However, according to another embodiment, the user may adjust the correction parameter through voice input. there is. For example, the user may input a voice (e.g., “Hi Bixby, make the picture brighter.”) instructing to adjust the characteristics of the image through a speaker provided in the user terminal 3000.

Hereinafter, a method for training the first neural network model 1210 and the second neural network model 1220 will be described with reference to FIGS. 32 and 33, respectively. Hereinafter, for convenience of explanation, it will be described that the processor 530 of the computing device 500 of FIG. 31 performs the steps of FIGS. 32 and 33. However, the present invention is not limited to this and all or part of the steps of FIGS. 32 and 33 may be performed by the main processor 3200 of the user terminal 3000 of FIG. 3 .

Referring to FIG. 32, in step 3201, the processor 530 may generate a label image by correcting the input image by applying a correction parameter to at least one image correction algorithm.

In step 3202, the processor 530 inputs correction parameters and an input image to the first neural network model, and in step 3203, the processor 530 may obtain an inferred image output from the first neural network model.

In step 3204, the processor 530 may update the first neural network model so that the difference between the label image and the inferred image is minimized.

Referring to FIG. 33, the processor 530 may input an input image into a second neural network model in step 3301 and obtain correction parameters output from the second neural network model in step 3302.

In step 3203, the processor 530 may generate a label image by correcting the input image by applying correction parameters to at least one image correction algorithm.

The processor 530 obtains measurement characteristic values expressing the image characteristics of the label image in numbers in step 3204, and updates the second neural network model so that the difference between the measurement characteristic value and the preset target characteristic value is minimized in step 3205. You can.

If the neural network models 1210 and 1220 according to the embodiments described above are mounted on the user terminal 3000, etc., the image correction module 3132 (corresponding to the image correction module 1200 in FIG. 1) can be easily updated. There is an advantage. Hereinafter, a method of updating a neural network model according to an embodiment will be described with reference to FIGS. 34 and 35.

FIG. 34 shows an example in which the image correction module 3132 included in the ISP 3130 of the user terminal 3000 of FIG. 3 is implemented to include image correction algorithms rather than a neural network model. Referring to FIG. 34, the image correction module 3132 is implemented to include a contrast ratio improvement algorithm A (3132a) and a brightness correction algorithm (3132b).

In Figure 34, it is assumed that an update to the image correction algorithms included in the image correction module 3132 is performed. The contents of the update are as follows.

1) Replace contrast ratio improvement algorithm A with contrast ratio improvement algorithm B

2) Remove brightness correction algorithm

3) Add exposure fusion algorithm

The result of performing the update is shown on the right side of FIG. 34. Referring to FIG. 34, in the image correction module 3132, contrast ratio improvement algorithm A (3132a) has been replaced with contrast ratio improvement algorithm B (3132c), the brightness correction algorithm (3132b) has been removed, and the exposure fusion algorithm (3132d) has been added. It has been done.

In this way, when an update to the image correction module 3132 is implemented while the image correction module 3132 is implemented to include image correction algorithms, the following problems exist.

First, whenever a change (replacement, removal, addition, etc.) is made to the image correction algorithm, optimization work is required to improve processing speed, which requires a lot of time and resources.

Second, some image correction algorithms are implemented as separate hardware, so in order to add the corresponding image correction algorithm, the hardware must be replaced or added, especially when the image correction module 3132 is implemented on the user terminal 3000. Replacing or adding hardware comes with many restrictions.

35 shows image correction when the image correction module 3132 included in the ISP 3130 of the user terminal 3000 of FIG. 3 is implemented to include neural network models 1210 and 1220 according to embodiments of the present disclosure. An embodiment of updating the module 3132 is shown.

Referring to FIG. 35, in order to update the first neural network model 1210 included in the image correction module 3132, learning on the first neural network model 1210 must be performed again. As described above, learning about the first neural network model 1210 may be performed by a separate computing device 500 shown in FIG. 31, so hereinafter, the computing device 500 is used to learn the first neural network model 1210. Assume that you are learning about

As described above, the label image generation module 1810 may be composed of image correction algorithms corresponding to image correction characteristics to be learned in the first neural network model 1210. Referring to FIG. 35, the processor 530 initially configures the label image generation module 1810 to include a contrast ratio improvement algorithm A (3132a) and a brightness correction algorithm (3132b), and learns the first neural network model (1210). After doing so, it was mounted on the image correction module 3132 of the user terminal 3000.

Afterwards, if there is a need to adjust the image correction characteristics of the image correction module 3132 of the user terminal 3000, the processor 530 allows the label image generation module 1810 to use the contrast ratio improvement algorithm B (3132c) and the exposure fusion algorithm (3132d). ) and the first neural network model 1210 can be trained again. When learning of the first neural network model 1210 is completed, the processor 530 transmits the newly learned first neural network model 1210 to the user terminal 3000, and the user terminal 3000 receives the received first neural network model. The image correction module 3132 can be updated by installing 1210.

In this way, when the image correction module 3132 is implemented to include the neural network models 1210 and 1220 according to embodiments of the present disclosure, there is no need to perform optimization even if the image correction module 3132 is updated. In addition, since the computing device 500 for training the neural network models 1210 and 1220 generally includes sufficient hardware resources, it is possible to change (replace) the image correction algorithms included in the label image generation module 1810 for new learning. , deletion, addition, etc.) can be done freely.

An image processing method using a neural network model according to an embodiment of the present disclosure includes obtaining an image captured through the image sensor 100, confirming a shooting environment of the image, and determining the shooting environment of the image. Accordingly, selecting a neural network model included in at least one of the image reconstruction module 1100 and the image correction module 1200 and processing the image using the selected neural network model. may include.

According to one embodiment, the shooting environment includes at least one of a shooting condition and a shooting mode, and the shooting condition includes the ISO value of the image, the time and place at which the image was shot, and the shooting mode may include at least one of a normal shooting mode, a night shooting mode, and a zoom shooting mode.

According to one embodiment, the image sensor 100 may be selected from among a plurality of different types of image sensors based on at least one of the shooting conditions and the shooting mode.

According to one embodiment, the step of selecting the neural network model includes selecting a detailed configuration of the image restoration module 1100 based on at least one of the shooting environment and the type of the selected image sensor, and the shooting environment , selecting one of a plurality of neural network models included in the image correction module 1200 based on at least one of the type of the selected image sensor 100 and the detailed configuration of the selected image restoration module 1100. May include steps.

According to one embodiment, the step of selecting the detailed configuration of the image restoration module 1100 includes primarily selecting the detailed configuration based on at least one of the shooting mode or the type of the selected image sensor 100. A step of maintaining or changing the primarily selected detailed configuration based on the step and the shooting conditions, and detailing the image restoration module 1100 corresponding to the shooting mode or the type of the image sensor 100. The configuration may be pre-specified.

According to one embodiment, the step of selecting the detailed configuration of the image restoration module 1100 includes selecting one of a plurality of neural network models included in the image restoration module 1100 based on the ISO value included in the shooting condition. You can choose either one.

According to one embodiment, the selected neural network model includes a denoise function and may be a neural network model learned using an image containing noise corresponding to the ISO value.

According to one embodiment, the step of selecting one of a plurality of neural network models included in the image correction module 1200 includes the type of the selected image sensor 100 and the detailed configuration of the selected image restoration module 1100. You can select a neural network model that corresponds to a combination of .

According to one embodiment, the selected neural network model may be a neural network model learned to correct image characteristics according to the shooting environment.

According to one embodiment, the neural network model included in the image correction module 1200 includes a first neural network model 1210 and a second neural network model 1220, and the first neural network model 1210 is the It is a model learned to minimize the difference between the inferred image output when the input image and correction parameters are input to the first neural network model 1210 and the label image corresponding to the correction parameter, and the label image corresponding to the correction parameter is , when the correction parameter is applied to at least one image correction algorithm, the input image is an image corrected by the at least one image correction algorithm, and the correction parameter input to the first neural network model 1210 is the first neural network model 1210. 2 It is a correction parameter inferred when the input image is input to the neural network model 1220, and the second neural network model 1220 is a correction parameter inferred by the second neural network model 1220 when the input image is input. Wow, it may be a model learned to minimize the difference between correction parameters that allow the label image to have preset image characteristics.

A computing device for processing an image signal using a neural network model according to an embodiment of the present disclosure includes a memory storing a program for processing an image signal and at least one processor, wherein the at least one processor is configured to: By executing the program, an image captured through the image sensor 100 is acquired, a shooting context of the image is confirmed, and an image reconstruction module 1100 and an image are generated according to the shooting environment. After selecting a neural network model included in at least one of the image correction modules 1200, the image may be processed using the selected neural network model.

According to one embodiment, the shooting environment includes at least one of a shooting condition and a shooting mode, and the shooting condition includes the ISO value of the image, the time and place at which the image was shot, and the shooting mode may include at least one of a normal shooting mode, a night shooting mode, and a zoom shooting mode.

According to one embodiment, the image sensor 100 may be selected from among a plurality of different types of image sensors 100 based on at least one of the shooting conditions and the shooting mode.

According to one embodiment, when selecting the neural network model, the at least one processor determines the detailed configuration of the image restoration module 1100 based on at least one of the shooting environment and the type of the selected image sensor 100. After selecting, based on at least one of the shooting environment, the type of the selected image sensor 100, and the detailed configuration of the selected image restoration module 1100, a plurality of neural networks included in the image correction module 1200 You can choose any one of the models.

According to one embodiment, when selecting the detailed configuration of the image restoration module 1100, the at least one processor primarily based on at least one of the shooting mode or the type of the selected image sensor 100. After selecting the detailed configuration, the primarily selected detailed configuration is maintained or changed based on the shooting conditions, and the detailed configuration of the image restoration module 1100 corresponding to the shooting mode or type of the image sensor 100 is changed. This can be specified in advance.

According to one embodiment, in selecting the detailed configuration of the image restoration module 1100, the at least one processor selects a plurality of images included in the image restoration module 1100 based on the ISO value included in the shooting condition. You can select any one of the neural network models.

According to one embodiment, the selected neural network model includes a denoise function and may be a neural network model learned using an image containing noise corresponding to the ISO value.

According to one embodiment, when selecting one of a plurality of neural network models included in the image correction module 1200, the at least one processor determines the type of the selected image sensor 100 and the selected image restoration module ( You can select a neural network model that corresponds to a combination of the detailed configurations of 1100).

According to one embodiment, the selected neural network model may be a neural network model learned to correct image characteristics according to the shooting environment.

According to one embodiment, the neural network model included in the image correction module 1200 includes a first neural network model 1210 and a second neural network model 1220, and the first neural network model 1210 is the It is a model learned to minimize the difference between the inferred image output when the input image and correction parameters are input to the first neural network model 1210 and the label image corresponding to the correction parameter, and the label image corresponding to the correction parameter is , when the correction parameter is applied to at least one image correction algorithm, the input image is an image corrected by the at least one image correction algorithm, and the correction parameter input to the first neural network model 1210 is the first neural network model 1210. 2 It is a correction parameter inferred when the input image is input to the neural network model 1220, and the second neural network model 1220 is a correction parameter inferred by the second neural network model 1220 when the input image is input. Wow, it may be a model learned to minimize the difference between correction parameters that allow the label image to have preset image characteristics.

Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and stored in a computer-readable medium. In this disclosure, “application” and “program” refer to one or more computer programs, software components, instruction sets, procedures, functions, or objects suitable for implementation in computer-readable program code. ), may represent a class, instance, related data, or part thereof. “Computer-readable program code” may include various types of computer code, including source code, object code, and executable code. “Computer-readable media” means read only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disc (CD), digital video disc (DVD), or various types of memory, It may include various types of media that can be accessed by a computer.

Additionally, device-readable storage media may be provided in the form of non-transitory storage media. Here, a ‘non-transitory storage medium’ is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is semi-permanently stored in the storage medium and cases where data is stored temporarily. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. Computer-readable media can be any available media that can be accessed by a computer and can include both volatile and non-volatile media, removable and non-removable media. Computer-readable media includes media on which data can be permanently stored and media on which data can be stored and later overwritten, such as rewritable optical disks or erasable memory devices.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. A computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or through an application store, or on two user devices (e.g., smart device). It can be distributed (e.g., downloaded or uploaded) directly between phones, or online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be stored on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be at least temporarily stored or created temporarily.

The foregoing description of the present disclosure is for illustrative purposes, and a person skilled in the art to which the present disclosure pertains will understand that the present disclosure can be easily modified into another specific form without changing its technical idea or essential features. will be. For example, the described techniques are performed in an order different from the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or in a different configuration. Appropriate results may be achieved through substitution or substitution by elements or equivalents. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present disclosure. do.

Claims (21)

In an image processing method using a neural network model,
Obtaining an image captured through the image sensor 100;
Confirming the shooting context of the video;
selecting a neural network model included in at least one of an image reconstruction module 1100 and an image correction module 1200 according to the shooting environment; and
Method comprising processing the image using the selected neural network model. According to paragraph 1,
The shooting environment includes at least one of a shooting condition and a shooting mode,
The shooting conditions include at least one of the ISO value of the image and the time and place at which the image was captured,
The method characterized in that the shooting mode includes at least one of a normal shooting mode, a night shooting mode, and a zoom shooting mode. According to any one of paragraphs 1 and 2,
The image sensor 100,
A method of selecting a plurality of different types of image sensors based on at least one of the shooting conditions and the shooting mode. According to any one of claims 1 to 3,
The step of selecting the neural network model is,
selecting a detailed configuration of the image restoration module 1100 based on at least one of the shooting environment and the type of the selected image sensor; and
Based on at least one of the shooting environment, the type of the selected image sensor 100, and the detailed configuration of the selected image restoration module 1100, one of a plurality of neural network models included in the image correction module 1200 A method comprising the step of selecting. According to any one of claims 1 to 4,
The step of selecting the detailed configuration of the image restoration module 1100 is:
initially selecting a detailed configuration based on at least one of the shooting mode or the type of the selected image sensor 100; and
Maintaining or changing the primarily selected detailed configuration based on the shooting conditions,
A method characterized in that the detailed configuration of the image restoration module 1100 corresponding to the shooting mode or type of the image sensor 100 is specified in advance. According to any one of claims 1 to 5,
The step of selecting the detailed configuration of the image restoration module 1100 is:
A method characterized in that one of a plurality of neural network models included in the image restoration module 1100 is selected based on the ISO value included in the shooting condition. According to any one of claims 1 to 6,
The selected neural network model is,
A method comprising a denoise function and being a neural network model learned using an image containing noise corresponding to the ISO value. According to any one of claims 1 to 7,
The step of selecting one of a plurality of neural network models included in the image correction module 1200 includes:
A method characterized by selecting a neural network model corresponding to a combination of the type of the selected image sensor 100 and the detailed configuration of the selected image restoration module 1100. According to any one of claims 1 to 8,
The selected neural network model is,
A method characterized in that it is a neural network model learned to correct image characteristics according to the shooting environment. According to any one of claims 1 to 9,
The neural network model included in the image correction module 1200 is:
Includes a first neural network model 1210 and a second neural network model 1220,
The first neural network model 1210 is a model learned to minimize the difference between the inferred image output when the input image and correction parameters are input to the first neural network model 1210 and the label image corresponding to the correction parameter. and
The label image corresponding to the correction parameter is an image in which the input image is corrected by the at least one image correction algorithm when the correction parameter is applied to the at least one image correction algorithm,
The correction parameter input to the first neural network model 1210 is a correction parameter inferred when the input image is input to the second neural network model 1220,
The second neural network model 1220 is configured to minimize the difference between a correction parameter that the second neural network model 1220 infers when the input image is input and a correction parameter that causes the label image to have preset image characteristics. A method characterized by being a learned model. In a computing device for processing image signals using a neural network model,
A memory in which a program for processing video signals is stored; and
Contains at least one processor,
The at least one processor executes the program,
An image captured through the image sensor 100 is acquired, the shooting context of the image is confirmed, and an image reconstruction module 1100 and an image correction module are used according to the shooting environment. A computing device that selects a neural network model included in at least one of the modules (1200) and then processes the image using the selected neural network model. According to clause 11,
The shooting environment includes at least one of a shooting condition and a shooting mode,
The shooting conditions include at least one of the ISO value of the image and the time and place at which the image was captured,
A computing device characterized in that the shooting mode includes at least one of a normal shooting mode, a night shooting mode, and a zoom shooting mode. According to any one of claims 11 and 12,
The image sensor 100,
A computing device, wherein a plurality of image sensors 100 of different types are selected based on at least one of the shooting conditions and the shooting mode. According to any one of claims 11 to 13,
The at least one processor selects the neural network model,
After selecting the detailed configuration of the image restoration module 1100 based on at least one of the shooting environment and the type of the selected image sensor 100,
Based on at least one of the shooting environment, the type of the selected image sensor 100, and the detailed configuration of the selected image restoration module 1100, one of a plurality of neural network models included in the image correction module 1200 A computing device characterized in that selection. According to any one of claims 11 to 14,
The at least one processor selects a detailed configuration of the image restoration module 1100,
After initially selecting a detailed configuration based on at least one of the shooting mode or the type of the selected image sensor 100,
Maintaining or changing the primarily selected detailed configuration based on the shooting conditions,
A computing device, wherein a detailed configuration of the image restoration module 1100 corresponding to the shooting mode or type of the image sensor 100 is specified in advance. According to any one of claims 11 to 15,
The at least one processor selects a detailed configuration of the image restoration module 1100,
A computing device characterized in that one of a plurality of neural network models included in the image restoration module 1100 is selected based on the ISO value included in the shooting condition. According to any one of claims 11 to 16,
The selected neural network model is,
A computing device comprising a denoise function and being a neural network model learned using an image containing noise corresponding to the ISO value. According to any one of claims 11 to 17,
The at least one processor selects one of a plurality of neural network models included in the image correction module 1200,
A computing device characterized in that it selects a neural network model corresponding to a combination of the type of the selected image sensor 100 and the detailed configuration of the selected image restoration module 1100. According to any one of claims 11 to 18,
The selected neural network model is,
A computing device characterized in that it is a neural network model learned to correct image characteristics according to the shooting environment. According to any one of claims 11 to 19,
The neural network model included in the image correction module 1200 is:
Includes a first neural network model 1210 and a second neural network model 1220,
The first neural network model 1210 is a model learned to minimize the difference between the inferred image output when the input image and correction parameters are input to the first neural network model 1210 and the label image corresponding to the correction parameter. and
The label image corresponding to the correction parameter is an image in which the input image is corrected by the at least one image correction algorithm when the correction parameter is applied to the at least one image correction algorithm,
The correction parameter input to the first neural network model 1210 is a correction parameter inferred when the input image is input to the second neural network model 1220,
The second neural network model 1220 is configured to minimize the difference between a correction parameter that the second neural network model 1220 infers when the input image is input and a correction parameter that causes the label image to have preset image characteristics. A computing device characterized by a learned model. A computer-readable recording medium on which a program for performing the method of any one of claims 1 to 10 is recorded on a computer.

Images