KR102210940B1 - Electronic apparatus and image processing method thereof - Google Patents

Abstract

An electronic device is disclosed. The electronic device is electrically connected to a memory storing at least one command and a memory, and by executing the command, obtains a noise map representing the quality of the input image from the input image, and generates a plurality of layers of the input image and the noise map. And a processor for obtaining an output image with improved quality of the input image by applying it to the included training network model, the processor providing a noise map to at least one intermediate layer among a plurality of layers, and the training network model is a plurality of samples It may be an artificial intelligence model obtained by learning a relationship between an image, a noise map for each sample image, and an original image corresponding to each sample image through an artificial intelligence algorithm.

Description

Electronic device and its image processing method {ELECTRONIC APPARATUS AND IMAGE PROCESSING METHOD THEREOF}

The present disclosure relates to an electronic device and an image processing method thereof, and more particularly, to an electronic device that acquires an output image with improved quality of input data, and an image processing method thereof.

In addition, the present disclosure relates to an artificial intelligence (AI) system that simulates functions such as cognition and judgment of the human brain by using a machine learning algorithm, and its application.

In recent years, artificial intelligence systems that implement human-level intelligence have been used in various fields. Unlike existing rule-based smart systems, artificial intelligence systems are systems in which machines learn, judge, and become smarter. As the artificial intelligence system is used, the recognition rate improves and the user's taste can be understood more accurately, and the existing rule-based smart system is gradually being replaced by a deep learning-based artificial intelligence system.

Artificial intelligence technology consists of machine learning (for example, deep learning) and component technologies using machine learning.

Machine learning is an algorithm technology that classifies/learns the features of input data by itself, and element technology is a technology that simulates functions such as cognition and judgment of the human brain using machine learning algorithms such as deep learning. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

These artificial intelligence technologies can be used to remove various and complex types of image noise. In particular, in the case of streaming video, since visually unpleasant noise such as block edge and ringing noise is caused in a compression process at a low bit rate, compression noise reduction (CAR) is important.

For noise removal, generally an image degradation model y = x + e is used. Here, y is an observed image with noise consisting of a clean image x and noise e. A number of image noise reduction studies estimate x using image estimation models, including non-local self-similarity (NSS) models and sparse models. BM3D or WNN is a representative method of using the NSS model, but there is a limit to achieving high noise reduction performance.

Recently, discriminative learning methods have been developed to overcome these limitations. A trainable nonlinear response diffusion (TNRD) has been proposed that provides better performance compared to an NSS model such as BM3D, but the fact that it is learned only for a specific noise model may be a disadvantage.

In addition, a DnCNN employing a residual learning strategy and batch normalization has been proposed, but has a disadvantage in that there is no separate CNN for image quality evaluation.

In addition, FFDNet, which removes spatially varying noise by using a non-uniform noise level map as an input, was proposed, but the point and noise map assumed to be given a noise map are used only once for the first layer, so other layers of FFDNet are There is a limitation that the noise map cannot be fully utilized.

And, in the case of a residual dense network (RDN) for Single Image Super-Resolution (SISR), there is an advantage that all hierarchical features with residual high density blocks can be fully used, but a specific model suitable for each noise level There is a limit to learning only.

Accordingly, there is a need to develop a method for adaptively removing spatially changing noise in an input image.

The present disclosure is in accordance with the above-described necessity, and an object of the present disclosure is to provide an electronic device and an image processing method for adaptively improving quality according to an input image.

An electronic device according to an embodiment of the present disclosure for achieving the above object is a memory storing at least one command and a quality of the input image from the input image by being electrically connected to the memory and executing the command. And a processor for obtaining an output image with improved quality of the input image by applying the input image and the noise map to a learning network model including a plurality of layers, and the processor The noise map is provided to at least one intermediate layer among a plurality of layers, and the learning network model uses an artificial intelligence algorithm to determine the relationship between a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image. It may be an artificial intelligence model obtained by learning through.

In addition, the learning network model further includes at least one sub-layer, and the processor processes the noise map using the at least one sub-layer, and provides the processed noise map to the at least one intermediate layer. can do.

Here, the processor provides a plurality of channels and additional channels corresponding to output data output from a previous layer of each of the at least one intermediate layer to each of the at least one intermediate layer, and the additional channel is the at least one intermediate layer. It may be the processed noise map output from a sub-layer corresponding to each layer.

Meanwhile, the processor may obtain the output image by mixing output data of an output layer among the plurality of layers and the input image.

In addition, the processor obtains the noise map by applying the input image to a noise map generation model including a plurality of layers, and the noise map generation model is a relationship between the plurality of sample images and noise maps for each sample image. It may be an artificial intelligence model obtained by learning through an artificial intelligence algorithm.

Meanwhile, the processor may provide the noise map to each of the plurality of layers, or may provide the noise map to each of the plurality of layers except for the input layer.

In the learning network model, each of the plurality of sample images provided as an input layer among the plurality of layers and a noise map of each of the plurality of sample images provided as the at least one intermediate layer are sequentially obtained by the plurality of layers. It may be an artificial intelligence model obtained by learning a relationship between an output image obtained by processing and an original image corresponding to each of the plurality of sample images through the artificial intelligence algorithm.

Meanwhile, each of the plurality of sample images may be a compressed image obtained by compressing an original image, and a noise map for each sample image may be a noise map obtained from each sample image and an original image corresponding to each sample image.

In addition, the processor may obtain an output video with improved quality of the video by applying each of the plurality of frames included in the video to the learning network model as the input image.

Meanwhile, the electronic device further includes a display, and the processor converts the resolution of the output image based on the resolution of the display, controls the display to display the image having the converted resolution, and the resolution is converted. The image may be a 4K UHD (Ultra High Definition) image or an 8K UHD image.

And, the processor divides the input image into an object area and a background area, and applies the input image, the noise map, and information on the background area to the learning network model to improve the quality of the background area, and the input The quality of the object region is improved by applying an image, the noise map, and information on the object region to another learning network model, and the output image is based on the object region with improved quality and the background region with improved quality Can be obtained.

Here, the learning network model may be a learning network model for removing noise, and the other learning network model may be a learning network model for upscaling the resolution of an image.

Meanwhile, the processor divides the input image into an object area and a background area, and applies at least one of information on the object area or information on the background area to the learning network model to improve the quality of the input image. Images can be acquired.

Meanwhile, in the image processing method of an electronic device according to an embodiment of the present disclosure, the step of obtaining a noise map representing the quality of the input image from the input image, the input image as an input layer among a plurality of layers included in the learning network model And providing the noise map to at least one intermediate layer among the plurality of layers, and applying the input image and the noise map to the learning network model to obtain an output image with improved quality of the input image. Including the step, wherein the learning network model may be an artificial intelligence model obtained by learning a relationship between a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image through an artificial intelligence algorithm. .

In addition, the learning network model further includes at least one sub-layer, and the providing step is to process the noise map using the at least one sub-layer, and the processed noise map with the at least one intermediate layer. Can provide.

Here, in the providing step, a plurality of channels and additional channels corresponding to output data output from a previous layer of each of the at least one intermediate layer are provided to each of the at least one intermediate layer, and the additional channel is the at least one It may be the processed noise map output from sub-layers corresponding to each of the intermediate layers of.

Meanwhile, in the obtaining of the output image, the output image may be obtained by mixing output data of an output layer among the plurality of layers and the input image.

And, in the obtaining of the noise map, the noise map is obtained by applying the input image to a noise map generation model including a plurality of layers, and the noise map generation model is applied to the plurality of sample images and each sample image. It may be an artificial intelligence model obtained by learning the relationship of the noise map to Korea through an artificial intelligence algorithm.

Meanwhile, in the providing step, the noise map may be provided to each of the plurality of layers, or the noise map may be provided to each of the plurality of layers except for an input layer.

In the learning network model, each of the plurality of sample images provided as an input layer among the plurality of layers and a noise map of each of the plurality of sample images provided as the at least one intermediate layer are sequentially obtained by the plurality of layers. It may be an artificial intelligence model obtained by learning a relationship between an output image obtained by processing and an original image corresponding to each of the plurality of sample images through the artificial intelligence algorithm.

Meanwhile, each of the plurality of sample images may be a compressed image obtained by compressing an original image, and a noise map for each sample image may be a noise map obtained from each sample image and an original image corresponding to each sample image.

In the acquiring of the output image, each of a plurality of frames included in the video may be applied as the input image to the learning network model to obtain an output video with improved quality of the video.

On the other hand, the step of converting the resolution of the output image based on the resolution of the display of the electronic device, and further comprising the step of displaying the image of which the resolution has been converted, the converted image is 4K Ultra High Definition (UHD) ) Images or 8K UHD images.

According to various embodiments of the present disclosure as described above, the electronic device obtains a noise map from the input image to more accurately identify the quality of the input image, and uses a learning network model that adaptively operates based on the noise map. The quality of the input image can be improved.

1 is a diagram for describing an example implementation of an electronic device according to an embodiment of the present disclosure.
2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
3A and 3B are diagrams for describing a learning network model and a noise map generation model according to various embodiments of the present disclosure.
4 is a diagram illustrating a method of learning a noise map generation model according to an embodiment of the present disclosure.
5 is a diagram for describing a learning method of a learning network model that improves image quality according to an embodiment of the present disclosure.
6 is a block diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure.
7 is a block diagram illustrating a configuration for learning and using a learning network model according to an embodiment of the present disclosure.
8A and 8B are diagrams for describing performance of a noise map generation model according to various embodiments of the present disclosure.
9A and 9B are diagrams for describing performance of a learning network model that improves the quality of an input image according to various embodiments of the present disclosure.
10A to 10D are diagrams for describing various extended embodiments of the present disclosure.
11 is a flowchart illustrating an image processing method of an electronic device according to an embodiment of the present disclosure.

Terms used in the embodiments of the present disclosure have selected general terms that are currently widely used as possible while taking functions of the present disclosure into consideration, but this may vary according to the intention or precedent of a technician working in the field, the emergence of new technologies, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, not the name of a simple term.

In the present specification, expressions such as "have," "may have," "include," or "may include" are the presence of corresponding features (eg, elements such as numbers, functions, actions, or parts). And does not exclude the presence of additional features.

The expression A or/and at least one of B is to be understood as representing either “A” or “B” or “A and B”.

Expressions such as "first," "second," "first," or "second," as used herein may modify various elements regardless of their order and/or importance, and one element It is used to distinguish it from other components and does not limit the components.

Some component (eg, a first component) is "(functionally or communicatively) coupled with/to)" to another component (eg, a second component) or " When referred to as "connected to", it should be understood that a component may be directly connected to another component or may be connected through another component (eg, a third component).

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "comprise" are intended to designate the existence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other It is to be understood that the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance the possibility of being excluded.

In the present disclosure, a "module" or "unit" performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module except for the "module" or "unit" that needs to be implemented with specific hardware and implemented as at least one processor (not shown). Can be.

In the present specification, the term user may refer to a person using an electronic device or a device (eg, an artificial intelligence electronic device) using an electronic device.

Hereinafter, an exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings.

1 is a diagram for describing an example implementation of an electronic device 100 according to an embodiment of the present disclosure.

As shown in FIG. 1, the electronic device 100 includes a TV, a monitor, a smart phone, a tablet PC, a notebook PC, a head mounted display (HMD), a near eye display (NED), a large format display (LFD), and a digital signage. It may be implemented as a device having a display such as a (digital signage), a digital information display (DID), a video wall, and a projector display.

Alternatively, the electronic device 100 may be a device that provides an image to a device equipped with an external display, such as a server, a BD player, a disk player, or a streaming box.

However, the present invention is not limited thereto, and the electronic device 100 may be any device as long as it can process an image.

The electronic device 100 may receive various types of images. Specifically, the electronic device 100 may receive any one of SD (Standard Definition), HD (High Definition), Full HD, and Ultra HD images. Alternatively, the electronic device 100 may receive an image compressed by MPEG (eg, MP2, MP4, MP7, etc.), AVC, H.264, HEVC, or the like.

For example, as illustrated in FIG. 1, the electronic device 100 may receive an uncompressed image 10-1 having the same resolution as the display of the electronic device 100. In this case, the electronic device 100 may display the image 10-1 without image processing on the received image 10-1, but always receive an image with guaranteed quality, such as the image 10-1. It does not become.

For example, the electronic device 100 has the same resolution as the display of the electronic device 100, but may receive the image 10-2 deteriorated by compression. In this case, the electronic device 100 needs to improve the quality of the deteriorated image 10-2.

Alternatively, the electronic device 100 may receive an image 10-3 having a lower resolution than the display of the electronic device 100. In this case, upscaling may be performed on the image 10-3, but the quality may be lowered. Accordingly, the electronic device 100 needs to improve the quality of the image 10-3 before being upscaled, and then upscale the image with improved quality or improve the quality of the upscaled image.

In addition, the electronic device 100 may receive various types of images, and it is necessary to improve quality in consideration of characteristics of each image. Hereinafter, a method of improving image quality of the electronic device 100 and various embodiments will be described.

2 is a block diagram illustrating a configuration of an electronic device 100 according to an embodiment of the present disclosure. As shown in FIG. 2, the electronic device 100 includes a memory 110 and a processor 120.

The memory 110 is electrically connected to the processor 120 and may store data necessary for various embodiments of the present disclosure. For example, the memory 110 may be implemented as an internal memory such as ROM (eg, electrically erasable programmable read-only memory (EEPROM)) or RAM included in the processor 120, or It may be implemented as a separate memory from 120. In this case, the memory 110 may be implemented in the form of a memory embedded in the electronic device 100 according to the purpose of data storage, or may be implemented in a form of a memory that is detachable to the electronic device 100. For example, in the case of data for driving the electronic device 100, it is stored in a memory embedded in the electronic device 100, and in the case of data for an extended function of the electronic device 100, the electronic device 100 is detachable. It can be stored in memory whenever possible. Meanwhile, in the case of a memory embedded in the electronic device 100, a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.)), a non-volatile memory ( Examples: OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g. NAND flash or NOR flash, etc.) ), a hard drive, or a solid state drive (SSD), and in the case of a memory that is detachable to the electronic device 100, a memory card (for example, compact flash (CF), SD ( secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory that can be connected to a USB port (e.g., USB memory) or the like.

The memory 110 may store a learning network model used to improve the quality of the input image. Here, the learning network model may be a machine learning model based on a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image. For example, the training network model may be a model trained on a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image (Convolution Neural Network). Here, the CNN is a multilayer neural network with a special connection structure designed for voice processing and image processing.

However, this is only an embodiment, and the learning network model may be a learning network model based on various neural networks such as a recurrent neural network (RNN) and a deep neural network (DNN).

Meanwhile, the noise map may represent the quality of the input image. For example, the noise map includes information indicating the quality of each pixel included in the input image, and in this case, the size of the noise map may be the same as the size of the input image. For example, if the input image has a size of 4 × 4, the noise map may also have a size of 4 × 4. However, the present invention is not limited thereto, and if the noise map indicates the quality of the input image, various methods may be used for the shape and information display method. For example, the unit information of the noise map may not correspond to each pixel value of the input image, but may correspond to an average value for each area of a preset size of the input image.

The memory 110 may further store a noise map generation model for obtaining a noise map of the input image. Here, the noise map generation model may be a machine learning model based on a plurality of sample images and a noise map for each sample image.

A detailed description of the learning network model and the noise map generation model used to improve the quality of the input image will be described later with reference to the drawings.

The processor 120 is electrically connected to the memory 110 and controls the overall operation of the electronic device 100.

According to an embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a timing controller (T-CON) that processes digital image signals, but is limited thereto. It is not a central processing unit (CPU), a microcontroller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), or a communication processor. processor (CP)), or one or more of an ARM processor, or may be defined in a corresponding term In addition, the processor 120 includes a system on chip (SoC) with a built-in processing algorithm, and a large scale integration (LSI). It may be implemented in the form of a field programmable gate array (FPGA).

The processor 120 may image-process the input image to obtain an output image with improved quality of the input image.

In particular, the processor 120 obtains a noise map representing the quality of the input image from the input image, and applies the input image and the noise map to a learning network model including a plurality of layers to produce an output image with improved quality of the input image. Can be obtained. As the noise map obtained from the input image is used in the process of improving the quality of the input image, quality improvement is adaptively performed according to the type of the input image, and the quality improvement effect may be improved.

Here, the learning network model may be an artificial intelligence model obtained by learning a relationship between a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image through an artificial intelligence algorithm. In addition, the plurality of layers of the learning network model may include an input layer, an intermediate layer, and an output layer. The input layer is a layer in which an operation is performed first among a plurality of layers, an output layer is a layer in which an operation is performed last among a plurality of layers, and an intermediate layer may be a layer disposed between the input layer and the output layer.

In addition, the processor 120 may provide a noise map to at least one intermediate layer among the plurality of layers. In particular, the processor 120 may provide a noise map to each of a plurality of layers, or may provide a noise map to each of the plurality of layers except for an input layer. Through this operation, the quality of the image is continuously reflected in the process of improving the quality of the input image, so that the quality improvement performance may be improved.

For example, if a noise map is provided only as an input layer among a plurality of layers, a characteristic of the noise map may be diluted while passing through the plurality of layers, resulting in a decrease in quality improvement performance.

Alternatively, if the noise map is provided only to the output layer among the plurality of layers, the noise map is not reflected while passing through the plurality of layers, and as a result, quality improvement is achieved with the noise map reflected only in the output layer. In general, considering that the more layers of the learning network model, the better the performance, and if the noise map is reflected in only one output layer, the performance may be degraded.

Therefore, when the processor 120 provides the noise map to at least one intermediate layer, quality improvement performance may be improved compared to the case of providing the noise map to only the input layer or the output layer.

Meanwhile, the learning network model further includes at least one sub-layer, and the processor 120 may process a noise map using at least one sub-layer and provide a noise map processed with at least one intermediate layer. .

The processor 120 may provide a plurality of channels and additional channels corresponding to output data output from a previous layer of each of the at least one intermediate layer to each of the at least one intermediate layer. Here, the additional channel may be a processed noise map output from a sub-layer corresponding to each of the at least one intermediate layer. That is, the processor 120 does not mix the output data output from the previous layer of each of the at least one intermediate layer and the processed noise map output from the sub-layer corresponding to each of the at least one intermediate layer, but connects it in parallel. It may be provided on each of at least one intermediate layer.

Meanwhile, the processor 120 may provide an input image as an input layer among a plurality of layers included in the learning network model. In this case, the training network model is an output obtained by sequentially processing a noise map of each of the plurality of sample images provided as an input layer among the plurality of layers and each of the plurality of sample images provided as at least one intermediate layer. It may be an artificial intelligence model obtained by learning a relationship between an image and an original image corresponding to each of the plurality of sample images through an artificial intelligence algorithm.

Alternatively, the processor 120 may provide an input image to an input layer and obtain an output image by mixing the output data and the input image of the output layer. That is, the processor 120 may provide an input layer not only to the input layer but also to the rear end of the output layer. In this case, the training network model is an output obtained by sequentially processing a noise map of each of the plurality of sample images provided as an input layer among the plurality of layers and each of the plurality of sample images provided as at least one intermediate layer. It may be an artificial intelligence model obtained by learning a relationship between an output image obtained by mixing data and each of the plurality of sample images, and an original image corresponding to each of the plurality of sample images through an artificial intelligence algorithm.

Here, each of the plurality of sample images is a compressed image obtained by compressing an original image, and a noise map for each sample image may be a noise map obtained from each sample image and an original image corresponding to each sample image.

Meanwhile, the processor 120 may obtain a noise map by applying the input image to a noise map generation model including a plurality of layers. Here, the noise map generation model may be an artificial intelligence model obtained by learning a relationship between a plurality of sample images and a noise map for each sample image through an artificial intelligence algorithm.

Meanwhile, when a training network model for improving image quality is trained and when a noise map generation model is trained, the same plurality of sample images and noise maps for each sample image may be used. However, the present invention is not limited thereto, and training data when training a training network model for improving image quality and training data when training a noise map generation model may be different from each other.

Meanwhile, the electronic device 100 further includes a display (not shown), and the processor 120 converts the resolution of the output image based on the resolution of the display, and controls the display to display the converted image. have. Here, the image whose resolution is converted may be a 4K UHD (Ultra High Definition) image or an 8K UHD image.

Alternatively, the processor 120 may obtain an output video with improved quality of the video by applying each of a plurality of frames included in the video to the learning network model as an input image. For example, the processor 120 decodes a video, applies each frame of the decoded video as an input image to a learning network model to improve quality, and combines a plurality of frames with improved quality to output an improved quality. You can also get a video. Here, the processor 120 may obtain a noise map of each frame and use the obtained noise map to improve the quality of each frame.

As described above, the processor 120 may adaptively improve the quality of the input image as the noise map is obtained from the input image. Also, as the processor 120 provides a noise map to at least one intermediate layer among a plurality of layers included in the learning network model, the processor 120 may perform image processing while continuously reflecting the quality of the input image. Accordingly, the quality improvement performance of the input image can be improved.

Hereinafter, the operation of the processor 120 will be described in more detail through the drawings.

3A and 3B are diagrams for describing a learning network model and a noise map generation model according to various embodiments of the present disclosure.

First, the processor 120 may obtain a noise map by applying an input image to a noise map generation model (Quality Estimation Convolution Neural Network, QECNN, 310), as shown in FIG. 3A.

The noise map generation model 310 may include a plurality of convolutional layers. For example, each of the plurality of convolution layers may perform convolution on input data using a 5×5 kernel. However, the present invention is not limited thereto, and any number of different types of kernels may be used. In addition, one convolution layer may perform convolution on input data using each of a plurality of kernels.

Some of the plurality of convolutional layers may process input data using a Relu (Rectified Linear Unit) function after performing convolution. The Relu function is a function that converts to 0 if the input value is less than 0, and outputs the input value as it is if the input value is greater than 0. However, the present invention is not limited thereto, and some of the plurality of convolutional layers may process input data using a sigmoid function.

The processor 120 may apply the input image to a training network model (Compression Artifact Reduction Convolution Neural Network, CARCNN, 320-1) to obtain an output image with improved quality of the input image. In this case, the processor 120 may provide a noise map to at least one intermediate layer among the plurality of layers. For example, as illustrated in FIG. 3A, the processor 120 may provide a noise map to each of the remaining layers except for the input layer among the plurality of layers.

The learning network model 320-1 may include a plurality of convolutional layers and a plurality of sub-convolutional layers 330. For example, each of the plurality of convolution layers performs convolution on the input data using a 3 × 3 kernel, and the plurality of sub-convolution layers 330 uses a 1 × 1 kernel to map the noise map. Convolution can be performed. can do. However, the present invention is not limited thereto, and any number of different types of kernels may be used. In addition, one convolution layer may perform convolution on input data using each of a plurality of kernels.

Some of the plurality of convolutional layers may process input data using a Relu function after performing convolution. Other parts of the plurality of convolutional layers may process input data using a batch normalization and a Relu function after performing convolution. Batch normalization is the task of making the distribution of each layer the same to secure a fast learning speed.

Output data output from the input layer among the plurality of convolutional layers may be divided into channels corresponding to the number of kernels included in the input layer. In addition, the output data output from the sub-convolution layer 330 corresponding to the input layer among the plurality of sub-convolution layers 330 is concatenated with the output data output from the input layer, so that the next layer of the input layer is performed. Can be entered. For example, output data consisting of 36 channels is output from the input layer, output data consisting of one channel is output from the sub-convolution layer 330 corresponding to the input layer, and output data consisting of a total of 37 channels. May be input as a next layer of the input layer. The number of channels may vary according to the characteristics of each of the plurality of convolutional layers, and similar operations are performed in the remaining convolutional layers.

The processor 120 may obtain an output image with improved quality of the input image by mixing the output data and the input image of the output layer among the plurality of convolution layers.

As described above, the processor 120 acquires a noise map corresponding to the input image, and continuously reflects the noise map in the process of improving the quality of the input image, thereby improving quality improvement performance.

Meanwhile, the learning network model 320-1 of FIG. 3A shows a form in which a plurality of sub-convolution layers 330 are added to continuously reflect a noise map in the form of a DnCNN (Denoise Convolution Neural Network), but FIG. 3B As described above, the learning network model 320-2 may be implemented in a form in which a plurality of sub-convolution layers 330 for continuously reflecting a noise map are added to a RDN (Residual Dense Network) form. The RDB (Residual Dense Block) layer of FIG. 3B includes a plurality of convolution layers in a form in which a residual block and a Dense block are combined, and outputs of each of the plurality of convolution layers are sequentially input to the next convolution layer, and additionally It can also be input as a convolutional layer placed at the location. In addition, output data obtained by mixing first input data of the RDB layer and data passing through the last layer may be output from the RDB layer. In the case of FIG. 3B as well, the same noise map generation model 310 as in FIG. 3A may be used.

However, the present invention is not limited thereto, and any basic model may be used as long as the learning network model 320-1 can continuously reflect the noise map.

Meanwhile, the models of FIGS. 3A and 3B are implemented as software and stored in the memory 110, and the processor 120 reads data for performing the operation of each layer from the memory 110 and processes the input image. You can do it.

4 is a diagram illustrating a method of learning a noise map generation model according to an embodiment of the present disclosure.

The noise map generation model may be an artificial intelligence model obtained by learning a relationship between a plurality of sample images and a noise map for each sample image through an artificial intelligence algorithm. For example, as shown in FIG. 4, the noise map generation model includes output data according to the input of the first sample image 410-1 and the first noise map 420-of the first sample image 410-1. The relationship of 1) can be learned through artificial intelligence algorithms. And, the same learning process is repeated for the relationship of the remaining data pairs ((410-2, 420-2), (410-3, 420-3), (410-4, 420-4), ...) Thus, a noise map generation model can be obtained. Here, the noise map for each sample image may be a noise map obtained through a rule-based preset algorithm. For example, the first sample image 410-1 to the fourth sample image 410-4 are images of JPEG quality 10, 30, 50, and 90, respectively, and the first noise maps 420-1 to 4 The noise maps 420-4 may be noise maps for the first to fourth sample images 410-1 to 410-4, respectively.

Meanwhile, the noise map generation model may be a model learned by a device other than the electronic device 100. However, the present invention is not limited thereto, and the processor 120 of the electronic device 100 may learn the noise map generation model.

5 is a diagram for describing a learning method of a learning network model that improves image quality according to an embodiment of the present disclosure.

The learning method of the learning network model may be an artificial intelligence model obtained by learning a relationship between a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image through an artificial intelligence algorithm. For example, as shown in FIG. 5, the training network model inputs a first sample image 520-1 and an input of a first noise map 530-1 for the first sample image 520-1. The relationship between the output data according to and the original image 510 corresponding to the first sample image 520-1 may be learned through an artificial intelligence algorithm. In addition, a learning network model may be obtained by repeating the same learning process for the relationship between the remaining data groups (520-2, 530-2, 510), (520-3, 530-3, 510). , The noise map for each sample image may be a noise map obtained through a rule-based preset algorithm.

Meanwhile, in FIG. 5, although the original image 510 is illustrated as being of one type, a plurality of original images may be used in the actual learning process. That is, in addition to the original image 510 of FIG. 5, a plurality of sample images obtained by compressing the additional original image and the additional original image at various compression rates, and a noise map for each sample image may be used in the learning process.

Meanwhile, the learning network model may be a model learned by a device other than the electronic device 100. However, the present invention is not limited thereto, and the processor 120 of the electronic device 100 may learn the learning network model.

6 is a block diagram illustrating a detailed configuration of an electronic device 100 according to an embodiment of the present disclosure.

Referring to FIG. 6, the electronic device 100 may include a memory 110, a processor 120, an input unit 130, a display 140, and a user interface 150.

At least one command may be stored in the memory 110. The processor 120 may execute a command stored in the memory 110 to perform an operation of obtaining a noise map of an input image, an operation of improving the quality of an input image, and a learning operation of each artificial intelligence model as described above. A detailed description of the configuration shown in FIG. 6 that overlaps the configuration shown in FIG. 2 will be omitted.

The input unit 130 receives various types of content, for example, an input image. For example, the input unit 130 is AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Bluetooth, Zigbee, wired/wireless LAN (Local Area Network), WAN, Ethernet, IEEE 1394, HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), external device (e.g., source device), external storage medium (e.g., USB), external server (e.g., web hard), etc. through communication methods such as DVI (Digital Visual Interface), etc. A video signal may be input from a streaming or download method. Here, the input image may be a digital signal, but is not limited thereto. Also, a video may be received through the input unit 130.

The display 140 includes a liquid crystal display (LCD), an organic light-emitting diode (OLED), a light-emitting diode (ED), a micro LED, a liquid crystal on silicon (LCoS), a digital light processing (DLP), and a quantum dot) It can be implemented in various forms such as a display panel.

The processor 120 may control the display 140 to display an output image with improved quality of the input image.

The user interface 150 may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen or a remote control receiver capable of performing the above-described display function and manipulation input function. Here, the button may be various types of buttons such as a mechanical button, a touch pad, a wheel, etc. formed in an arbitrary area such as a front portion, a side portion, or a rear portion of the external body of the electronic device 100.

The processor 120 may perform an operation to improve the quality of an input image according to a user command input through the user interface 150.

7 is a block diagram illustrating a configuration for learning and using a learning network model according to an embodiment of the present disclosure. Although learning and image processing may be performed in separate devices, in FIG. 7, for convenience of description, it will be described that the electronic device 100 learns a learning network model.

Referring to FIG. 7, the processor 120 may include at least one of a learning unit 710 and an image processing unit 720.

The learning unit 710 may generate or train a model that obtains a noise map from an input image and a model that improves the quality of the input image. The learning unit 710 may generate a recognition model having a determination criterion by using the collected training data.

For example, the learning unit 710 may generate, learn, or update a model for obtaining a noise map from the input image by using the input image and the noise map of the input image as training data. In addition, the learning unit 710 generates, trains, or updates a model for acquiring the original image from the input image and the noise map by using the input image, the noise map for the input image, and the original image corresponding to the input image as training data. I can make it.

The image processing unit 720 may use predetermined data (eg, an input image) as input data of the learning network model to obtain output data with improved quality of the predetermined data.

As an example, the image processing unit 720 may obtain a noise map of the input image and obtain an output image with improved quality of the input image based on the noise map.

At least a portion of the learning unit 710 and at least a portion of the image processing unit 720 may be implemented as a software module or manufactured in the form of at least one hardware chip and mounted on the electronic device 100. For example, at least one of the learning unit 710 and the image processing unit 720 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU or application). processor) or a graphics dedicated processor (for example, a GPU), and mounted on various electronic devices or object recognition devices described above. At this time, the dedicated hardware chip for artificial intelligence is a dedicated processor specialized in probability calculation, and has higher parallel processing performance than conventional general-purpose processors, so it can quickly process computation tasks in artificial intelligence fields such as machine learning. When the learning unit 710 and the image processing unit 720 are implemented as a software module (or a program module including an instruction), the software module is a computer-readable non-transitory readable recording medium (non-transitory). transitory computer readable media). In this case, the software module may be provided by an OS (Operating System) or a predetermined application. Alternatively, some of the software modules may be provided by an operating system (OS), and some of the software modules may be provided by a predetermined application.

In this case, the learning unit 710 and the image processing unit 720 may be mounted on one electronic device, or may be mounted on separate electronic devices, respectively. For example, one of the learning unit 710 and the image processing unit 720 may be included in the electronic device 100, and the other may be included in an external server. In addition, the learning unit 710 and the image processing unit 720 may provide model information built by the learning unit 710 to the image processing unit 720 or input to the learning unit 1120 through wired or wireless. Data may be provided to the learning unit 710 as additional learning data.

8A and 8B are diagrams for describing performance of a noise map generation model according to various embodiments of the present disclosure.

First, FIG. 8A shows a mean square error of a first noise map according to image quality and a second noise map output from a noise map generation model, and a LIVE1 image data set was used. For example, when compressing the original image, a first noise map may be obtained from the compressed image through a rule-based preset algorithm, and a second noise map may be obtained by applying the compressed image to a noise map generation model. In addition, the mean square error of the first noise map and the second noise map may be obtained.

Q (compression factor) in FIG. 8A represents the quality according to compression, and the closer to the original image from 10 to 90. In addition, when there are 8 layers, when there are 12 layers, and when there are 16 layers, the average squared error may decrease as the number of layers increases. This is more clearly illustrated in FIG. 8B.

In addition, when the number of layers is more than a certain number, it can be seen that a second noise map very similar to the first noise map is obtained regardless of Q.

9A and 9B are diagrams for describing performance of a learning network model that improves the quality of an input image according to various embodiments of the present disclosure.

9A shows the average PSNR/SSIM result calculated after reducing compression lockdown by various methods for compressed images having a Q of 10 to 90 in a classic5 or LIVE1 video data set. Peak Signal-to-noise Ratio (PSNR) is the maximum signal-to-noise ratio, representing the power of noise to the maximum power that a signal can have, and SSIM (Structural Similarity Index) is a structural similarity index, which is a distortion caused by compression and transformation. Represents the similarity to the original image.

As shown in FIG. 9A, it can be seen that the higher Q is, the better the quality improvement performance, the better the QEDnCNN performance than the conventional DnCNN, and the QERDN performance is improved than the conventional RDN.

9B is a view comparing QEDnCNN and QERDN, and it can be seen that in general, QEDnCNN has better performance than QERDN.

10A to 10D are diagrams for describing various extended embodiments of the present disclosure.

The processor 120 may improve the quality of the input image by classifying the object area and the background area in the input image. For example, as shown in FIG. 10A, the processor 120 divides the original image into an object image including only an object and a background image including only the remaining areas excluding the object, and determines the quality of each of the object image and the background image. It is also possible to obtain an output image with improved quality by synthesizing an object image with improved quality and a background image with improved quality.

The processor 120 may identify the object area and the background area of the input image in various ways. For example, the processor 120 may identify an object having a predetermined shape in the input image based on the pixel value, and identify the remaining area except the area where the object is identified as the background area.

Alternatively, the processor 120 may identify an object having a predetermined shape in the input image by using an artificial intelligence model for object recognition, and identify the remaining area except the area where the object is identified as a background area.

The above example is only an embodiment, and the processor 120 may identify the object area and the background area in the input image in any number of ways.

The processor 120 may process an image by classifying an object area and a background area using a plurality of image quality improvement models. For example, the processor 120 improves the quality of the object area in the input image by applying the input image and the object area information in the input image to the first image quality improvement model 1010, as shown in FIG. 10B, The quality of the background area in the input image may be improved by applying the input image and the background area information in the input image to the second image quality improvement model 1020. In addition, the processor 120 may obtain an output image with improved quality of the input image by synthesizing the object region with improved quality and the background region with improved quality.

Here, the object region information may be the same as the lower left of FIG. 10A, and the background region information may be the same as the lower right of FIG. 10A. Alternatively, the object region information and the background region information may not include pixel values, but may include only region information for distinguishing the object region from the background region. For example, the object region information may be an image indicating an object region as 1 and a background region as 0, and the background region information may be an image indicating a background region as 1 and an object region as 0.

The first image quality improvement model 1010 may include a first noise map generation model for obtaining a noise map from an input image and a first learning network model for improving quality of an object region of the input image. In addition, the second image quality improvement model 1020 may include a second noise map generation model for obtaining a noise map from the input image and a second learning network model for improving the quality of a background region of the input image.

The first noise map generation model may be a model for generating a noise map of an object area, and the second noise map generation model may be a model for generating a noise map of a background area.

The first learning network model may be a model for improving the image quality of the object region, and the second learning network model may be a model for improving the image quality of the background region. To this end, different sample images may be used for the first training network model and the second training network model during a training process. For example, the first training network model is created by lowering the resolution of the original image and the original image and then training the up-scaled image, and the second training network model trains the original image and an image with noise added to the original image. Can be created. In this case, the processor 120 obtains a clear result such as that the resolution of the object region is enlarged using the first learning network model, and obtains a result from which noise of the background region is removed using the second learning network model. I can. The processor 120 may perform different image processing on the object area and the background area through the above method.

10B illustrates that only the input image is applied to the first noise map generation model and the second noise generation model, but is not limited thereto. For example, object region information as well as the input image may be additionally applied to the first noise map generation model, and background region information as well as the input image may be additionally applied to the second noise map generation model.

Alternatively, the processor 120 may process an image by classifying an object region and a background region using one image quality improvement model. For example, as shown in FIG. 10C, the processor 120 additionally applies at least one of object region information or background region information as well as input image and noise map to the learning network model (CARCNN) to determine the quality of the input image. It is also possible to obtain this improved output image.

Here, the object region information and the background region information may be the same as in FIG. 10B. Alternatively, as object region information and background region information, one image representing the object region as 1 and the background region as 0 may be used. However, the present invention is not limited thereto, and any method may be used as long as the object area and the background area can be distinguished.

The learning network model may be a model trained to correspond to types of object region information and background region information. For example, when one image in which the object area is expressed as 1 and the background area is expressed as 0 is used, the same type of image may be used in the learning process.

In addition, the plurality of sample images used in the learning process of the learning network model may also be sample images having different quality improvement methods of the object region and the background region. For example, the object area of the plurality of sample images may be an area whose quality is improved to a higher level than the background area. Here, the plurality of sample images may be obtained through deterioration of the original image. That is, the plurality of sample images may be obtained through a method in which the object area and the background area of the original image are deteriorated to different levels. Alternatively, each of the plurality of sample images may be a compressed image in which the object area and the background area of the corresponding original image are compressed in different ways.

That is, the learning network model of FIG. 10C may perform quality improvement by identifying the object region and the background region of the input image, and classifying the object region and the background region.

In FIG. 10C, the image quality improvement model includes a noise map generation model (QECNN) and a training network model, and it is illustrated that only the input image is applied to the noise map generation model, but is not limited thereto. For example, the processor 120 may obtain a noise map for the input image by additionally applying at least one of object region information or background region information as well as the input image to the noise map generation model.

Alternatively, the processor 120 may divide the input image into a plurality of blocks, divide each block into an object area and a background area, and process the object area and the background area through a separate artificial intelligence model. For example, as shown in FIG. 10D, the processor 120 may sequentially divide the input image into blocks of a preset size and identify whether each block is an object area or a background area. Further, the processor 120 applies the block identified as the object area to the first image quality improvement model 1030 to obtain the first output block 1050-1, and converts the block identified as the background area to the second image quality. The second output block 1050-2 is obtained by applying to the improved model 1040, and an output image may be obtained by merging the first output block 1050-1 and the second output block 1050-2. .

Here, in the learning process of the first image quality improvement model 1030, a plurality of sample blocks representing object regions may be used, and in the learning process of the second image quality improvement model 1040, a plurality of sample blocks representing background regions Can be used.

As described above, the processor 120 may improve the quality of the input image by dividing the object area and the background area in the input image.

11 is a flowchart illustrating an image processing method of an electronic device according to an embodiment of the present disclosure.

First, a noise map indicating the quality of the input image is obtained from the input image (S1110). In addition, an input image is provided as an input layer among a plurality of layers included in the learning network model, and a noise map is provided to at least one intermediate layer among the plurality of layers (S1120). Then, an output image with improved quality of the input image is obtained by applying the input image and the noise map to the learning network model (S1130). Here, the learning network model may be an artificial intelligence model obtained by learning a relationship between a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image through an artificial intelligence algorithm.

Here, the learning network model further includes at least one sub-layer, and in the providing step (S1120), a noise map is processed using at least one sub-layer, and a noise map processed with at least one intermediate layer is provided. I can.

In the providing step (S1120), a plurality of channels and an additional channel corresponding to output data output from a previous layer of each of the at least one intermediate layer are provided to each of the at least one intermediate layer, and the additional channel is at least one intermediate layer. It may be a processed noise map output from a sub-layer corresponding to each layer.

Meanwhile, in the step of obtaining the output image (S1130), the output image may be obtained by mixing the output data and the input image of the output layer among a plurality of layers.

And, in the step of obtaining the noise map (S1110), the noise map is obtained by applying the input image to a noise map generation model including a plurality of layers, and the noise map generation model includes a plurality of sample images and noise for each sample image. It may be an artificial intelligence model obtained by learning the relationship of the map through an artificial intelligence algorithm.

Meanwhile, in the providing operation S1120, a noise map may be provided to each of the plurality of layers, or a noise map may be provided to each of the plurality of layers except for the input layer.

In addition, the training network model is an output image obtained by sequentially processing a noise map of each of a plurality of sample images provided as an input layer among a plurality of layers and each of a plurality of sample images provided as at least one intermediate layer. , And an artificial intelligence model obtained by learning a relationship between an original image corresponding to each of the plurality of sample images through an artificial intelligence algorithm.

Meanwhile, each of the plurality of sample images is a compressed image in which an original image is compressed, and a noise map for each sample image may be a noise map obtained from each sample image and an original image corresponding to each sample image.

In the step of acquiring the output image (S1130 ), each of a plurality of frames included in the video may be applied as an input image to a learning network model to obtain an output video with improved quality of the video.

On the other hand, the step of converting the resolution of the output image based on the resolution of the display of the electronic device and the step of displaying the converted image, the resolution converted image is a 4K UHD (Ultra High Definition) image or 8K It can be a UHD image.

According to various embodiments of the present disclosure as described above, the electronic device obtains a noise map from the input image to more accurately identify the quality of the input image, and uses a learning network model that adaptively operates based on the noise map. The quality of the input image can be improved. That is, since the electronic device obtains a noise map from an input image and uses it to remove noise, a noise removal effect of a spatially changing image is excellent, and compression artifacts can be reduced.

Meanwhile, according to an example of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media. I can. The device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, electronic device A) according to the disclosed embodiments. When an instruction is executed by a processor, the processor may perform a function corresponding to the instruction directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transient' means that the storage medium does not contain a signal and is tangible, but does not distinguish between semi-permanent or temporary storage of data in the storage medium.

In addition, according to an embodiment of the present disclosure, the method according to various embodiments described above may be included in a computer program product and provided. Computer program products can be traded between sellers and buyers as commodities. The computer program product may be distributed online in the form of a device-readable storage medium (eg, compact disc read only memory (CD-ROM)) or through an application store (eg, Play StoreTM). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

In addition, according to an embodiment of the present disclosure, various embodiments described above are in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. Can be implemented in In some cases, the embodiments described herein may be implemented by the processor itself. According to software implementation, embodiments such as procedures and functions described in the present specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing a processing operation of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When a computer instruction stored in such a non-transitory computer-readable medium is executed by a processor of a specific device, a specific device causes a specific device to perform a processing operation in the device according to the various embodiments described above. The non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as registers, caches, and memory. Specific examples of non-transitory computer-readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, each of the constituent elements (eg, modules or programs) according to the various embodiments described above may be composed of a singular or plural entity, and some sub-elements of the above-described sub-elements are omitted, or other sub-elements are omitted. Components may be further included in various embodiments. Alternatively or additionally, some constituent elements (eg, a module or a program) may be integrated into one entity, and functions performed by each corresponding constituent element prior to the consolidation may be performed identically or similarly. Operations performed by modules, programs, or other components according to various embodiments are sequentially, parallel, repetitively or heuristically executed, or at least some operations are executed in a different order, omitted, or other operations are added. Can be.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is generally in the technical field belonging to the disclosure without departing from the gist of the disclosure claimed in the claims. Of course, various modifications may be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present disclosure.

100: electronic device 110: memory
120: processor 130: input unit
140: display 150: user interface

Claims (20)

A memory storing at least one instruction; And
Electrically connected to the memory,
By executing the command, a noise map representing the quality of the input image is obtained from an input image, and an output image with improved quality is obtained by applying the input image and the noise map to a learning network model including a plurality of layers. Including;
The processor,
Providing the noise map to at least one intermediate layer among the plurality of layers,
The learning network model,
An artificial intelligence model obtained by learning a relationship between a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image through an artificial intelligence algorithm,
Each of the sample images is a compressed image in which the corresponding original image is compressed,
The noise map for each of the sample images is obtained from each of the sample images and an original image corresponding to each of the sample images. The method of claim 1,
The learning network model,
It further includes at least one sub-layer,
The processor,
The electronic device comprising: processing the noise map using the at least one sub-layer and providing the processed noise map with the at least one intermediate layer. The method of claim 2,
The processor,
A plurality of channels and additional channels corresponding to output data output from a previous layer of each of the at least one intermediate layer are provided to each of the at least one intermediate layer,
The additional channel is the processed noise map output from a sub-layer corresponding to each of the at least one intermediate layer. The method of claim 1,
The processor,
The electronic device, wherein the output image is obtained by mixing output data of an output layer among the plurality of layers and the input image. The method of claim 1,
The processor,
Applying the input image to a noise map generation model including a plurality of layers to obtain the noise map,
The noise map generation model,
The electronic device, which is an artificial intelligence model obtained by learning a relationship between the plurality of sample images and a noise map for each sample image through an artificial intelligence algorithm. The method of claim 1,
The processor,
The electronic device comprising: providing the noise map to each of the plurality of layers, or providing the noise map to each of the plurality of layers except for an input layer. The method of claim 1,
The learning network model,
An output image obtained by sequentially processing each of the plurality of sample images provided as an input layer among the plurality of layers and a noise map of each of the plurality of sample images provided as the at least one intermediate layer by the plurality of layers, And an artificial intelligence model obtained by learning a relationship between an original image corresponding to each of the plurality of sample images through the artificial intelligence algorithm. The method of claim 1,
Each of the plurality of sample images,
The original image is a compressed compressed image,
The noise map for each of the sample images is,
The electronic device, which is a noise map obtained from each of the sample images and an original image corresponding to each of the sample images. The method of claim 1,
The processor,
An electronic device for obtaining an output video with improved quality of the video by applying each of a plurality of frames included in the video to the learning network model as the input image. The method of claim 1,
It further includes a display;
The processor,
Converting the resolution of the output image based on the resolution of the display, and controlling the display to display the converted image,
The image in which the resolution is converted,
An electronic device that is a 4K Ultra High Definition (UHD) image or an 8K UHD image. The method of claim 1,
The processor,
Divide the input image into an object area and a background area,
Applying the input image, the noise map, and information on the background region to the learning network model to improve the quality of the background region,
Improving the quality of the object region by applying the input image, the noise map, and information on the object region to another learning network model,
The electronic device of claim 1, wherein the output image is obtained based on the object region with improved quality and the background region with improved quality. The method of claim 11,
The learning network model is a learning network model for removing noise,
The electronic device, wherein the other learning network model is a learning network model for upscaling the resolution of an image. The method of claim 1,
The processor,
Divide the input image into an object area and a background area,
The electronic device, wherein at least one of information on the object area or information on the background area is applied to the learning network model to obtain an output image with improved quality. In the image processing method of an electronic device,
Obtaining a noise map representing the quality of the input image from the input image;
Providing an input image to an input layer among a plurality of layers included in the learning network model, and providing the noise map to at least one intermediate layer of the plurality of layers; And
Comprising: applying the input image and the noise map to the learning network model to obtain an output image with improved quality; Including,
The learning network model,
An artificial intelligence model obtained by learning a relationship between a plurality of sample images, a noise map for each sample image, and an original image corresponding to each sample image through an artificial intelligence algorithm,
Each of the sample images is a compressed image in which the corresponding original image is compressed,
The noise map for each sample image is obtained from each sample image and an original image corresponding to each sample image. The method of claim 14,
The learning network model,
It further includes at least one sub-layer,
The providing step,
Processing the noise map using the at least one sub-layer and providing the processed noise map with the at least one intermediate layer. The method of claim 15,
The providing step,
A plurality of channels and additional channels corresponding to output data output from a previous layer of each of the at least one intermediate layer are provided to each of the at least one intermediate layer,
The additional channel is the processed noise map output from a sub-layer corresponding to each of the at least one intermediate layer. The method of claim 14,
The step of obtaining the output image,
The image processing method of obtaining the output image by mixing output data of an output layer among the plurality of layers and the input image. The method of claim 14,
Obtaining the noise map,
Applying the input image to a noise map generation model including a plurality of layers to obtain the noise map,
The noise map generation model,
An image processing method that is an artificial intelligence model obtained by learning a relationship between the plurality of sample images and a noise map for each sample image through an artificial intelligence algorithm. The method of claim 14,
The providing step,
Providing the noise map to each of the plurality of layers, or providing the noise map to each of the plurality of layers except for an input layer. The method of claim 14,
The learning network model,
An output image obtained by sequentially processing each of the plurality of sample images provided as an input layer among the plurality of layers and a noise map of each of the plurality of sample images provided as the at least one intermediate layer by the plurality of layers, And an artificial intelligence model obtained by learning a relationship between an original image corresponding to each of the plurality of sample images through the artificial intelligence algorithm.

Images