TWI768323B - Image processing apparatus and image processing method thereof - Google Patents

Abstract

An image processing apparatus applies an image to a first learning network model to optimize the edges of the image, applies the image to a second learning network model to optimize the texture of the image, and applies a first weight to the first image and a second weight to the second image based on information on the edge areas and the texture areas of the image to acquire an output image.

Description

Image processing device and image processing method therefor

The present disclosure relates to an image processing apparatus and an image processing method thereof, and more particularly, to an image processing apparatus and an image processing method for enhancing the characteristics of an image by using a learning network model.

Stimulated by the development of electronic technology, various types of electronic devices have been developed and released. In particular, in recent years, image processing apparatuses have been used in various places (eg, homes, offices, and public spaces) and are continuing to develop.

Recently, high-resolution display panels (eg, 4K Ultra High Definition (UHD) television (TV)) have been introduced and widely released. However, the availability of high-resolution content for reproduction on such high-resolution display panels is limited. Therefore, various techniques for generating high-resolution content from low-resolution content are being developed. In particular, there is an increasing need for efficient processing of the large number of operations necessary to produce high-resolution content within limited processing resources.

In addition, recently, artificial intelligence systems replicating human-level intelligence have been used in various fields. Different from the traditional rule-based intelligent system, artificial intelligence system refers to the machine in which the machine learns, judges and implements autonomously. processing system. Since the artificial intelligence system operates iteratively, the system shows a more improved recognition rate and, for example, becomes able to understand user preferences more correctly. Therefore, traditional rule-based intelligent systems are gradually being replaced by deep learning-based artificial intelligence systems.

Artificial intelligence technology consists of machine learning (eg, deep learning) and element technology that utilizes machine learning.

Machine learning refers to algorithmic techniques for autonomous classification/learning of the characteristics of input data. Meanwhile, elemental technology refers to a technology that simulates the functions of the human brain (eg, cognition and judgment) by using machine learning algorithms (eg, deep learning) and includes, for example, language understanding, visual understanding, inference/prediction, knowledge representation ) and operation control and other technical fields.

Attempts have been made to enhance the characteristics of images by using artificial intelligence techniques in conventional image processing devices. However, there is a problem that for the performance of conventional image processing apparatuses, the processing amount of the operations required to generate high-resolution images is limited and takes a lot of time. Therefore, there is a need for a technology that enables an image processing device to generate high-resolution images and provide images by performing only a few operations.

The present disclosure aims to solve the above-mentioned needs and provide an image processing device and an image processing method thereof to obtain a high-resolution image with improved image characteristics by using a plurality of learning network models.

An image processing method for achieving the above object according to an embodiment of the present disclosure includes: a memory, which stores computer-readable instructions; and a processor, which is configured to to execute the computer-readable instructions to: apply an input image as a first input to a first learning network model and obtain a first image from the first learning network model, the first image including and applying the input image as a second input to a second learning network model and obtaining a second image from the second learning network model, the second The image includes an enhanced texture optimized based on the texture of the input image. The processor identifies edge regions and texture regions included in the image and applies first weights to the first image and second weights to the first image based on information about the edge regions and the texture regions the second image, and obtain an optimized output image from the input image based on the first weight applied to the first image and the second weight applied to the second image.

Additionally, the first type of the first learning network model is different from the second type of the second learning network model.

Additionally, the first learning network model can be a deep learning model that optimizes the edges of the input image by using multiple layers or is trained to use multiple pre-learning filters to one of the machine learning models that optimize the edges of the input image.

Additionally, the second learning network model may be a deep learning model that optimizes the texture of the input image by using multiple layers or by using multiple pre-learned filters for the One of the machine learning models that are optimized for the texture of the input image.

At the same time, the processor may be based on the edge region and the texture region to obtain the first weight corresponding to the edge region and the second weight corresponding to the texture region.

Additionally, the processor may scale down the input image to obtain a scaled-down image having a resolution less than that of the input image. Additionally, the first learning network model may obtain the first image with the enhanced edge from the first learning network model that scales the scaled-down image, and the second A learning network model may obtain the second image with the enhanced texture from the second learning network model that scales up the scaled-down image.

In addition, the processor may obtain area detection information for which the edge area and the texture area have been identified based on the scaled-down image, and provide the area detection information and the image to the A first learning network model and the second learning network model.

Additionally, the first learning network model may obtain the first image by scaling up the edge region, and the second learning network model may obtain the first image by scaling up the texture region Describe the second image.

In addition, the first image and the second image may be a first afterimage and a second afterimage, respectively. Additionally, the processor may apply the first weight to the first afterimage based on the edge region and the second weight to the second afterimage based on the texture region, and then apply the The first afterimage, the second afterimage, and the input image are mixed to obtain the output image.

Meanwhile, the second learning network model may be the following model: the model storing a plurality of filters corresponding to each of a plurality of image patterns, and classifying each of the image blocks included in the image into one of the plurality of image patterns, and assigning At least one filter of the plurality of filters corresponding to the classified image pattern is applied to the image block and provides the second image.

Here, the processor may accumulate index information of image patterns corresponding to each of the image blocks being classified and identify the image as one of a natural image or a graphic image based on the index information and the first weight and the second weight are adjusted based on a result of identifying the input image as one of the natural image or the graphic image.

Here, the processor may increase at least one of the first weight or the second weight based on the input image being identified as the natural image, and based on the input image being identified as the natural image reducing at least one of the first weight or the second weight by using the graphic image.

Meanwhile, an image processing method of an image processing apparatus according to an embodiment of the present disclosure includes the following steps: applying an input image as a first input to a first learning network model; acquiring a first image from the first learning network model, the first image includes an enhanced edge optimized based on the edge of the input image; applying the input image as a second input to a second learning network model; obtaining the first learning network model from the second learning network model; Two images, the second image including an enhanced texture optimized based on the texture of the input image; identifying the edge region of the edge included in the input image; identifying the texture included in the input image region; applying a first weight to the first image based on the edge region; based on the texture region applying a second weight to the second image; and obtaining an optimum from the input image based on the first weight applied to the first image and the second weight applied to the second image converted output image.

Here, the first learning network model and the second learning network model may be different types of learning network models from each other.

Additionally, the first learning network model can be a deep learning model that optimizes the edges of the input image by using multiple layers or is trained to use multiple pre-learning filters to one of the machine learning models that optimize the edges of the input image.

Additionally, the second learning network model can be a deep learning model that optimizes the texture of the input image by using multiple layers or is trained to use multiple pre-learning filters one of the machine learning models that optimize the texture of the input image.

In addition, the image processing method may include the step of: obtaining the first weight and the second weight based on ratio information of the edge region in the input image and the texture region in the input image .

In addition, the image processing method may include the step of scaling down the input image to obtain a scaled-down image having a resolution smaller than that of the input image. Meanwhile, the first learning network model can obtain the first image by scaling up the scaled down image, and the second learning network model can obtain the first image by scaling up the scaled down image image to acquire the second image.

Here, the image processing method may include the following steps: acquiring first area detection information for identifying the edge area of the input image and second area identifying the texture area of the input image detecting information, and providing the region detection information and the image to the first learning network model and the second learning network model, respectively.

Meanwhile, the first learning network model can acquire the first image by scaling up the edge region, and the second learning network model can acquire the texture region by scaling up the texture region. Describe the second image.

In addition, the first image and the second image may be a first afterimage and a second afterimage, respectively.

According to the various embodiments of the present disclosure as described above, high-resolution images are generated by applying mutually different learning network models to images, and the amount of operations required to generate high-resolution images is reduced, and thus it is possible to A high-resolution image is generated within the limited resources of the image processing device and can be provided to the user.

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32: Index information

10: Image/Output Image

10’: Input image/image

20, 30: output image

30': Final output image

100: Image processing device/sound output device

110: Memory

120, 1200: Processor

130: Input

140: Display

150: Exporter

160: User Interface

810: Laplace Filter

820: Gradient Vector

830: Filter/Search

840: Application

850: index matrix

860: Filter Database (DB)

1210: Learning Parts

1220: Identify parts

S310, S320, S330, S340, S350, S410, S420, S430, S440, S450, S610, S620, S630, S640, S650, S710, S720, S730, S740, S1310, S1320, S1330: Operation

FIG. 1 is a diagram illustrating an implementation example of an image processing apparatus according to an embodiment of the present disclosure.

FIG. 2 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present disclosure.

FIG. 3 is a diagram illustrating a first learning network model and a second learning network model according to an embodiment of the present disclosure.

FIG. 4 is a scaled-down diagram illustrating an embodiment of the present disclosure.

FIG. 5 is a diagram illustrating a deep learning model and a machine learning model according to an embodiment of the present disclosure.

FIG. 6 is a diagram illustrating a first learning network model and a second learning network model according to another embodiment of the present disclosure.

FIG. 7 is a diagram illustrating a first learning network model and a second learning network model according to another embodiment of the present disclosure.

FIG. 8 is a diagram schematically illustrating an operation of a second learning network model according to an embodiment of the present disclosure.

FIG. 9 is a diagram illustrating index information according to an embodiment of the present disclosure.

FIG. 10 is a diagram illustrating a method of obtaining a final output image according to an embodiment of the present disclosure.

FIG. 11 is a block diagram showing a detailed configuration of the image processing apparatus shown in FIG. 2 .

12 is a block diagram illustrating a configuration of an image processing apparatus for learning and using a learning network model according to an embodiment of the present disclosure.

FIG. 13 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be explained in detail with reference to the accompanying drawings.

Considering the functions set forth in the present disclosure, general terms that are traditionally widely used are selected as terms used in the embodiments of the present disclosure as much as possible. However, the terminology may be changed according to the intention of those skilled in the art in the relevant field or the appearance of new technologies. changes. In addition, in certain cases, there may be designated terms, and in such cases, the meanings of the terms will be elaborated in the relevant descriptions in the present disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the overall content of the present disclosure, rather than only based on the names of the terms.

In this specification, expressions such as "have", "may have", "include" and "may include" should be considered to indicate the presence of such a property (eg , such as numerical values, functions, operations, and components), and the terms are not intended to exclude the presence of additional characteristics.

Additionally, the expression "at least one of A and/or B" should be construed to mean either "A" or "B" or "A and B".

Additionally, the expressions "first", "second", etc. used in this specification may be used to describe various elements without regard to any order and/or importance. Additionally, such expressions are used only to distinguish one element from another, and are not intended to limit the element.

Meanwhile, the descriptions in this disclosure refer to an element (eg, a first element) with another element (eg, a second element) "(operably or communicatively) coupled" or "(operably or communicatively) ) coupled to" another element (eg, a second element) or "connected to" another element (eg, a second element) should be construed to mean that the one element is directly coupled to the other element, or all The one element is coupled to the other element via a further element (eg, a third element).

Singular expressions include plural expressions unless the context clearly defines otherwise. In addition, in this disclosure, for example, "include" and "have" Terms such as "have" should be construed as indicating the presence of such features, numbers, steps, operations, elements, components or combinations thereof stated in the specification, rather than pre-exclusion of other features, numbers, steps, operations, elements, The presence or possibility of addition of one or more of the components or combinations thereof.

In addition, in the present disclosure, a "module" or "unit" may perform at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "units" may be integrated into at least one module and may be implemented as at least one processor, but need to be implemented as "modules" or "units" of specific hardware except.

In addition, in this specification, the term "user" may refer to a person or a device (eg, an artificial intelligence electronic device) operating an electronic device.

Hereinafter, embodiments of the present disclosure will be explained in more detail with reference to the accompanying drawings.

FIG. 1 is a diagram illustrating an implementation example of an image processing apparatus according to an embodiment of the present disclosure.

The image processing apparatus 100 may be implemented as a television (TV) as shown in FIG. 1 . However, the image processing apparatus 100 is not limited thereto, and the image processing apparatus 100 may be implemented as any of the following apparatuses equipped with image processing functions and/or display functions: for example, smart phones, tablet personal computers (personal computers, etc.) PC), laptop personal computer, head mounted display (HMD), near eye display (NED), large format display (LFD), digital signage (digital signage), digital information Display (digital information display, DID), video wall (video wall), projector display, camera, camcorder (camcorder), printer, etc. without limitation.

The image processing device 100 can receive images of various resolutions or compressed images. For example, the image processing device 100 may receive the image 10 formatted according to any of the following images: standard definition (SD) images, high definition (HD) images, full HD images, and Ultra HD video. In addition, the image processing device 100 may receive the image 10 in the following encoding formats or compression formats: Moving Picture Experts Group (MPEG) (eg, MP2, MP4, MP7, etc.), advanced video coding (advanced video) coding, AVC), H.264, high efficiency video coding (high efficiency video coding, HEVC) and so on.

Although the image processing apparatus 100 according to the disclosed embodiment is implemented as a UHD television, due to the limited availability of UHD content, there are many situations where only the following images are available: eg standard definition (SD) images, high definition (HD) images and full HD images (hereinafter referred to as low-resolution images) and other images. In this case, a method of upscaling a low-resolution input image to a UHD image (hereinafter referred to as a high-resolution image) and providing the resulting image can be provided. As an example, a low-resolution image can be applied as input to the learning network model, so that the low-resolution image can be upscaled, and thus the high-resolution image can be acquired as output for display on the image processing device 100 .

However, in order to enlarge a low-resolution image to a high-resolution image, a large number of complex processing operations are generally required to transform the image data. Therefore, it is necessary to have There is a high-performance and high-complexity image processing apparatus 100 to perform this transformation. As an example, in order to scale up a 60P image in an SD class with a resolution of 820×480 to a high-resolution image, the image processing apparatus 100 should perform an operation of 820×480×60 pixels per second. Therefore, there is a need for a processing unit with high performance, such as a central processing unit (CPU) or a graphics processing unit (GPU) or a combination thereof. As another example, the image processing apparatus 100 should perform an operation of 3840×2160×60 pixels per second to scale up a 60P image in a UHD level with a resolution of 4K to an image with a resolution of 8K. Therefore, there is a need for a processing unit capable of handling a large number of operations, such as at least 24 times the amount that would be the case in the SD class for upscaling images.

Accordingly, in the following, various embodiments will be described that provide an image processing apparatus 100 that reduces the amount of operations required to scale up a lower resolution image to a higher resolution image, and thus enables image processing The limited resources of the device 100 are maximized.

Additionally, various embodiments will be described in which the image processing apparatus 100 acquires an output image while enhancing or enhancing at least one image characteristic of various characteristics of the input image.

FIG. 2 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present disclosure.

According to FIG. 2 , the image processing apparatus 100 includes a memory 110 and a processor 120 .

The memory 110 is electrically connected to the processor 120 and can store and execute the present disclosure necessary information for various implementations. For example, the memory 110 may be implemented as an internal memory, such as a read-only memory (ROM) included in the processor 120 (eg, an electrically erasable programmable read-only memory) (electrically erasable programmable read-only memory, EEPROM)), random access memory (random access memory, RAM), etc., or a memory separate from the processor 120.

The memory 110 may be implemented in the form of memory embedded in the image processing device 100 , or implemented as being attachable to or from the image processing device 100 , depending on the use of the stored data. form of detached memory. For example, in the case where the data is used to operate the image processing device 100 , the data may be stored in a memory embedded in the image processing device 100 , and in the case where the data is used for extended functions of the image processing device 100 , the data may be stored in a memory that may be attached to or detached from the image processing device 100 . In the case where the memory 110 is implemented as a memory embedded in the image processing device 100 , the memory 110 may be at least one of the following: volatile memory (eg, dynamic RAM (DRAM) ), static RAM (static RAM, SRAM), synchronous dynamic RAM (synchronous dynamic RAM, SDRAM), etc.) or non-volatile memory (for example, one time programmable ROM (OTPROM), programmable ROM (programmable ROM, PROM), erasable and programmable ROM (erasable and programmable ROM, EPROM), electrically erasable and programmable ROM (electrically erasable and programmable ROM, EEPROM), mask ROM (mask ROM), flash ROM (flash ROM), flash memory (eg, NAND flash or NOR flash, etc.), hard drive or solid state drive (solid state drive, SSD)).

Meanwhile, in the case where the memory 110 is implemented as a memory that can be attached to or detached from the image processing device 100 , the memory 110 may be a memory card (eg, a compact flash). , CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multimedia card (multi-media card, MMC, etc.), external memory (eg, USB memory) that can be connected to a universal serial bus (USB) port, etc.

According to embodiments of the present disclosure, the memory 110 may store at least one program for causing instructions to be executed by the processor 120 . Here, the instructions may be instructions for the processor 120 to obtain an output image by applying the image 10 to the learning network.

According to another embodiment of the present disclosure, the memory 110 may store the learning network model according to various embodiments of the present disclosure.

The learning network model according to the embodiment of the present disclosure is a judgment model based on an artificial intelligence algorithm and trained based on a plurality of images, and the learning network may be a neural network-based model. The trained judgment model can be designed to simulate human intelligence and decision-making on a computer, and can include a number of weighted network nodes that simulate the neurons of a human neural network. Each of the plurality of network nodes may form connections to simulate synaptic activity of neurons that transmit and receive signals through synapses. Additionally, the trained judgment model may include, for example, a machine learning model, a neural network model, or a deep learning model developed from a neural network model. In a deep learning model, multiple network nodes may be located in different depths (or layers) from each other and transmit and receive data according to convolutional connections.

As an example, the learning network model may be a convolution neural network (CNN) model trained on images. CNN is a multi-layer neural network with a specific connection structure designed for speech processing, image processing, etc. At the same time, learning network models is not limited to CNNs. For example, the learning network model may be implemented as at least one deep neural network (DNN) model among the following models: recurrent neural network (RNN) model, long short-term memory network (long short term memory network, LSTM) model, gated recurrent unit (GRU) model or generative adversarial network (generative adversarial network, GAN) model.

For example, the learning network model can restore or convert low-resolution images to high-resolution images based on Super-resolution GAN (SRGAN). Meanwhile, the memory 110 according to the embodiment of the present disclosure can store a plurality of learning network models of the same type or different types. The number and type of learning network models is not limited. However, according to another embodiment of the present disclosure, at least one learning network model according to various embodiments of the present disclosure may be stored in at least one of an external device or an external server.

The processor 120 is electrically connected to the memory 110 and controls the overall operation of the image processing apparatus 100 .

According to an embodiment of the present disclosure, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, an artificial intelligence (AI) processor, and a timing controller for processing digital signals , T-CON). However, the processor 120 is not limited thereto, and the processor 120 may include a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), One or more of a controller, an application processor (AP), a communication processor (CP), and an Advanced RISC Machine (ARM) processor, or may be defined by terms. In addition, the processor 120 may be implemented as a system on chip (SoC) or a large scale integration (LSI) in which processing algorithms are stored, or as a field programmable gate array (field programmable gate array). , FPGA) form to implement.

The processor 120 may apply the image 10 as input to the learning network model and obtain images with improved, enhanced, optimized or enhanced image characteristics. Here, the characteristics of the image 10 may mean at least one of edge directions, edge strengths, textures, gray values, brightness, contrast, or gamma values according to a plurality of pixels included in the image 10 . For example, the processor 120 may apply the imagery to the learning network model and obtain images in which edges and textures have been enhanced. Here, the edge of the image may mean an area in which the values of spatially adjacent pixels change sharply. For example, an edge can be an area where the brightness of the image changes sharply from a low value to a high value or from a high value to a low value. The texture of an image can be viewed as the same feature in the image. Distinctive patterns or shapes of sexual areas. At the same time, the texture of the image can also be composed of fine edges, and thus the processor 120 can obtain the edge components equal to or greater than the first threshold intensity (or threshold thickness) and the components less than the second threshold intensity (or threshold thickness) therein. Image with improved edge components. Here, the first threshold strength may be a value for dividing edge components according to an embodiment of the present disclosure, and the second threshold strength may be a value for dividing texture components according to an embodiment of the present disclosure, And the first threshold intensity and the second threshold intensity may be predetermined values or values set based on characteristics of the image. However, in the following, for convenience of explanation, the above-mentioned characteristics will be referred to as edge and texture.

Meanwhile, the image processing apparatus 100 according to the embodiment of the present disclosure may include a plurality of learning network models. Each of the plurality of learned network models may enhance different properties of the image 10 . This will be explained in detail with reference to FIG. 3 .

FIG. 3 is a diagram illustrating a first learning network model and a second learning network model according to an embodiment of the present disclosure.

3 , at operation S310 , the processor 120 according to an embodiment of the present disclosure may apply the image 10 as an input to a first learning network model and obtain a first image in which the edge of the image 10 has been improved as an output. At operation S320, the image 10 may also be supplied as an input to the second learning network model and a second image in which the texture of the image 10 has been improved is obtained as an output.

Meanwhile, the image processing apparatus 100 according to the embodiment of the present disclosure may use the first learning network model and the second learning network model based on different artificial intelligence algorithms in parallel. Alternatively, the image 10 may be processed sequentially by the first learning network model and the second learning network model. Here, the first learning network model can be obtained by A model trained using larger resources than the resources of the second learning network model. Here, the resources may be various data necessary to train and/or process the learning network model and may include, for example, whether immediate learning is performed, the amount of learning data, the number of convolutional layers included in the learning network model, the number of parameters , the capacity of the memory used in the learning network model, the extent to which the learning network uses the GPU, etc.

For example, the GPU provided in the image processing device 100 may include a texture unit, a special function unit (SFU), an arithmetic logic device, and the like. Here, a texture unit is a resource for adding materials or textures to the image 10, and a specific functional unit is a resource for processing complex operations such as square root, reciprocal, and algebraic functions. Meanwhile, an integer arithmetic logic unit (ALU) is a resource for processing floating point, integer arithmetic, comparison and data movement. A geometric unit is a resource for computing the position or viewpoint of an object, the direction of a light source, and so on. A raster unit is a resource that projects 3D data onto a 2D screen. In this case, the deep learning model may use various resources included in the GPU to learn and operate compared to the machine learning model. Meanwhile, the resources of the image processing apparatus 100 are not limited to the resources of the GPU, and the resources may be resources of various components included in the image processing apparatus 100 , such as the storage area of the memory 110 and power.

The first learning network model and the second learning network model according to the embodiments of the present disclosure may be different types of learning network models.

As an example, the first learning network model may be one of a deep learning-based model or a machine learning model, and the deep learning-based model learns based on multiple Like to improve the edges of the image 10, a machine learning model is trained to improve the edges of the image by using a number of pre-learned filters. The second learning network model can be a deep learning model or a machine learning based model. The deep learning model learns to improve the texture of the image by using multiple layers. The machine learning based model is trained to improve the texture of the image by using multiple layers. Pre-learning database (DB) and multiple pre-learning filters to improve image texture. Here, the pre-learning DB can be a plurality of filters corresponding to each of the plurality of image patterns, and the second learning network model can identify the image patterns corresponding to the image blocks included in the image 10 and use the The texture of the image 10 is optimized by using the filter of the plurality of filters corresponding to the recognized pattern. According to an embodiment of the present disclosure, the first learning network model may be a deep learning model, and the second learning network model may be a machine learning model.

Machine learning models include multiple pre-learners that learn in advance based on various information and data input methods, such as supervised learning, unsupervised learning, and semi-supervised learning filter, and the filter to be applied to image 10 is identified among the plurality of filters.

A deep learning model is a model that performs learning based on a large amount of data and includes multiple hidden layers between an input layer and an output layer. Therefore, the deep learning model may require additional resources of the image processing apparatus 100 to perform learning and operation than the resources of the machine learning model.

As another example, the first learning network model and the second learning network model may be models based on the same artificial intelligence algorithm, but with different sizes or configurations set. For example, the second learning network model may be a low-complexity model having a size smaller than that of the first learning network model. Here, the size and complexity of the learning network model may be proportional to the number of convolutional layers and the number of parameters constituting the model. In addition, according to an embodiment of the present disclosure, the second learning network model may be a deep learning model, and the first learning network model may be a deep learning model using fewer convolutional layers than the second learning network model.

As yet another example, each of the first learning network model and the second learning network model may be a machine learning model. For example, the second learning network model may be a low-complexity machine learning model having a size smaller than that of the first learning network model.

Also, various embodiments of the present disclosure have been explained based on the assumption that the first learning network model is a model trained by using more resources than the second learning network model, but this is only an example and the present disclosure is not Just that. For example, the first learning network model and the second learning network model can be models of the same or similar complexity, and the second learning network model can be achieved by using more resources than the first learning network model. resources to train the model.

At operation S330 , the processor 120 according to an embodiment of the present disclosure may identify edge regions and texture regions included in the image 10 . Then, at operation S340, the processor 120 may apply the first weight to the first image and the second weight to the second image based on the information about the edge region and the texture region. As an example, the processor 120 may obtain the first weight corresponding to the edge region and the first weight corresponding to the texture region based on information about the ratio of the edge region to the texture region included in the image 10 second weight. For example, if there are more edge areas than texture areas according to the ratio, the processor 120 may apply a greater weight to the first image in which the edge area has been improved than the second image in which the texture has been improved. As another example, if there are more texture areas than edge areas according to the ratio, the processor 120 may apply a greater weight to the second image in which the texture has been improved than the first image in which the edge has been improved. Next, the processor 120 may obtain the output image 10 based on the first image to which the first weight has been applied and the second image to which the second weight has been applied.

As yet another example, the first and second images obtained from the first learned network model and the second learned network model may be residual images. Here, the residual image may be an image including only residual information in addition to the original image. As an example, a first learning network model may identify edge regions in the image 10 and optimize the identified edge regions and obtain the first image. The second learning network model can identify texture regions in the image 10 and optimize the identified texture regions and acquire the second image.

Next, the processor 120 may mix the image 10 with the first image and the second image and obtain the output image 20 . Here, blending may be the process of adding the corresponding pixel value of each of the first image and the second image to the value of each pixel included in image 10 . In this case, the output image 20 may be an image with enhanced edges and textures due to the first image and the second image.

The processor 120 according to an embodiment of the present disclosure may apply the first weight and the second weight to the first image and the second image, respectively, and then mix the image with the image 10 , and thus obtain the output image 20 .

As another example, processor 120 may divide image 10 into regions area. Next, processor 120 may identify a ratio of edge area to texture area for each of the plurality of areas. For the first region in which the ratio of edge regions is high among the plurality of regions, the processor 120 may set the first weight to a value greater than the second weight. In addition, the processor 120 may set the second weight to a larger value than the first weight for the second area in which the ratio of the texture area is high among the plurality of areas.

Next, at operation S340, the processor 120 may mix the first image and the second image to which the weights have been applied with the image 10 and obtain an output image. The image 10 and the output image 20 corresponding to the image 10 can be expressed as Equation 1 below.

[Equation 1] Y _res = Y _img + a * Network_Model 1( Y _img )+b* Network_Model 2( Y _img )

Here, Y _img means image 10, Network_Model1( Y _img ) means the first image, Network_Model2( Y _img ) means the second image, "a" means the first weight corresponding to the first image, and "b" ” means the second weight corresponding to the second image.

Meanwhile, as yet another example, the processor 120 may apply the image 10 as an input to a third learning network model and obtain a first weight for applying to the first image and a second weight for applying to the second image. For example, the third learning network model may be trained to identify edge regions and texture regions included in the image 10 and enhance the edge regions based on the ratio of the identified edge regions and texture regions, characteristics of the image 10, etc. The first weight of and the second weight to enhance the texture area.

FIG. 4 is a scaled-down diagram illustrating an embodiment of the present disclosure.

Referring to FIG. 4 , at operation S410, the processor 120 may identify edge regions and texture regions in the input image 10' and obtain a first weight corresponding to the edge regions and a second weight corresponding to the texture region. As an example, the processor 120 may apply a guided filter to the input image 10' and identify edge regions and texture regions. The guided filter may be a filter used to divide the image 10 into a base layer and a detail layer. The processor 120 may identify edge regions based on the base layer and texture regions based on the detail layer.

Next, at operation S420, the processor 120 may scale down the input image 10' and acquire the image 10 with a resolution smaller than that of the input image 10'. As an example, processor 120 may apply subsampling to input image 10' and scale down the resolution of input image 10' to a target resolution. Here, the target resolution may be a low resolution lower than the resolution of the input image 10'. For example, the target resolution may be the resolution of the original image corresponding to the input image 10'. Here, the resolution of the original image may be estimated by a resolution estimation program, or identified based on additional information received with the input image 10', but the resolution and identification of the original image are not limited thereto. Meanwhile, the processor 120 may apply various known downscaling methods other than subsampling, and thus acquire the image 10 corresponding to the input image 10'.

As an example, if the input image 10 ′ is a UHD image with a resolution of 4K, in order to apply the input image 10 ′ as an input to the first learning network model and the second learning network model and obtain the output image 20 , it is necessary to compare the When the SD image with a resolution of 820×480 is applied to the first learning network model and the second learning network model, the line buffer memory is at least 5.33 times (3840/820) larger. In addition, there is the following problem: as the amount of operations required for the first learning network model to acquire the first image increases, storing the The memory 110 space and required CPU/GPU performance for each intermediate operation result increases exponentially.

Therefore, the processor 120 according to an embodiment of the present disclosure can apply the scaled-down input image 10 to the first learning network model and the second learning network model to reduce the size of the first learning network model and the second learning network model. The amount of operations required in the road model, the storage space of the memory 110, and the like.

At operation S430 , when the scaled-down image 10 is input, the first learning network model according to an embodiment of the present disclosure may implement a scale-up that enhances high-frequency components corresponding to edges included in the input image 10 And get a high-resolution first image. Meanwhile, at operation S440, the second learning network model may perform upscaling to enhance high frequency components corresponding to textures included in the image 10 and acquire a high-resolution second image. Here, the resolution of the first image and the second image may be the same as the resolution of the input image 10'. For example, if the input image 10 is an image of 4K resolution and the scaled-down image 10 is an image of 2K resolution, the first learning network model and the second learning network model can perform the scaling up of the image 10 And obtain a 4K resolution image as the output of image 10 .

At operation S450, the processor 120 according to an embodiment of the present disclosure may blend the scaled-up first image and the second image with the input image 10' and obtain an image in which edges and textures in the input image 10' have been enhanced High-resolution output image 20 . According to the embodiment shown in FIG. 4, the process of acquiring the input image 10' and the output image 20 corresponding to the input image 10' can be expressed as Equation 2 below.

[Equation 2] Y _res = Y _org + a * Network_Model 1( DownScaling ( Y _org ))+b* Network_Model 2( DownS caling ( Y _org ))

Here, Y _org means the input image 10 ′, DownScaling( Y _org ) means the image 10 , Network_Model1(DownScaling( Y _org )) means the first image, Network_Model2(DownScaling( Y _org )) means the second image, "a" means the first weight corresponding to the first image, and "b" means the second weight corresponding to the second image.

FIG. 5 is a diagram illustrating a deep learning model and a machine learning model according to an embodiment of the present disclosure.

5, as described above, the first learning network model may be a deep learning model that learns to enhance the edges of the image 10 by using multiple layers, and the second learning network model may be trained to enhance the edges of the image 10 by using multiple layers. A machine learning model of a pre-learned filter to enhance the texture of the image 10.

According to an embodiment of the present disclosure, a deep learning model may be modeled as a deep structure including ten or more layers in total in a configuration in which two convolutional layers and one pooling layer are repeated. In addition, deep learning models can be implemented by using various types of activation functions (such as Identity Function, Logistic Sigmoid Function, Hyperbolic Tangent (tanh) function, rectified linear unit, ReLU) function, leaky ReLU function (Leaky ReLU Function), etc.) to implement the operation. In addition, deep learning models can be sized differently by performing padding, striding, etc., during the process of performing convolutions. Here, padding means to fill in a specific value (eg, pixel value) that is generally larger than a predetermined size around the received input value. Stride means that the weighting matrix when performing convolution shift interval. For example, if stride=3, then the learning network model can perform convolution on the input values while shifting the weight matrix by as many as three spaces at a time.

According to an embodiment of the present disclosure, the deep learning model can learn to optimize a characteristic of the image 10 that has high user sensitivity, and the machine learning model can use a plurality of pre-learning filters to optimize the image 10 at least one of the remaining properties of . For example, a situation may be assumed in which there is a close relationship between the transparency of the edge region (eg, edge direction, edge intensity) and the user's perceived transparency of the image 10 . The image processing device 100 may enhance the edges of the image 10 by using a deep learning model, and as an example of the remaining characteristics, the image processing device 100 may enhance the texture by using a machine learning model. Since the deep learning model learns based on a larger amount of data than the machine learning model and performs iterative operations, it is assumed that the processing result of the deep learning model is better than the processing result of the machine learning model. However, the present disclosure is not necessarily limited thereto, and both the first learning network model and the second learning network model may be implemented as deep learning based models or as machine learning based models. As another example, the first learning network model may be implemented as a machine learning based model and the second learning network model may be implemented as a deep learning based model.

Additionally, although various embodiments of the present disclosure are explained based on the assumption that the first learned network model enhances edges and the second learned network model enhances texture, the specific operations of the learned network model are not limited thereto. For example, a situation may be assumed in which there is the closest relationship between the processing level of the noise of the image 10 and the transparency of the image 10 perceived by the user. In this case, the image processing apparatus 100 may By performing image processing on the noise of the image 10 using a deep learning model, and as an example of the remaining image properties, the image processing device 100 can enhance texture by using a machine learning model. As another example, if there is the closest relationship between the processing degree of the brightness of the image 10 and the transparency of the image 10 perceived by the user, the image processing apparatus 100 may perform imaging on the brightness of the image 10 by using a deep learning model processing, and as an example of the remaining image characteristics, the image processing device 100 may filter noise by using a machine learning model.

FIG. 6 is a diagram illustrating a first learning network model and a second learning network model according to an embodiment of the present disclosure.

The processor 120 according to an embodiment of the present disclosure may scale down the input image 10' and obtain the image 10 with a relatively lower resolution at operation S610, and obtain area detection information at operation S620, which has been based on A scaled-down image 10' identifies edge regions and textured regions. According to the embodiment shown in FIG. 5, the processor 120 may identify edge regions and texture regions included in the input image 10' of the resolution of the original image. 6, the processor 120 can identify edge regions and texture regions included in the image 10 in which the resolution of the input image 10' has been scaled down to the target resolution.

Then, the processor 120 according to an embodiment of the present disclosure may provide the region detection information and the image 10 to the first learning network model and the second learning network model, respectively.

At operation S630, the first learning network model according to an embodiment of the present disclosure may perform a scaling-up that only enhances the edge region of the image 10 based on the region detection information. At operation S640, the second learning network model may detect information based on the region The information performs upscaling that enhances only the textured regions of the image 10 .

As another example, the processor 120 may provide an image including only some of the pixel information included in the image 10 to the learning network model based on the region detection information. Since the processor 120 only provides some information included in the image 10 instead of the image 10 to the learning network model, the amount of operations performed by the learning network model can be reduced. For example, the processor 120 may provide an image including only pixel information corresponding to the edge region to the first learning network model and provide an image including only pixel information corresponding to the texture region based on the region detection information to the second learning network model.

Then, the first learning network model can scale up the edge region and acquire the first image, and the second learning network model can scale up the texture region and acquire the second image.

Next, at operation S650, the processor 120 may add the first image and the second image to the input image 10' and obtain the output image 20.

FIG. 7 is a diagram illustrating a first learning network model and a second learning network model according to another embodiment of the present disclosure.

Referring to FIG. 7 , at operation S710, the processor 120 according to an embodiment of the present disclosure may apply the input image 10' as an input to the first learning network model and acquire the first image. As an example, since the first learning network model performs upscaling to enhance the high frequency components corresponding to the edges included in the input image 10 ′ at operation S710 , the processor 120 may acquire the high-resolution first image. Here, the first image may be an afterimage. Afterimages can include only residual information in addition to the original image. image of the news. Residual information may indicate the difference between each pixel or group of pixels of the original image and the high-resolution image.

In addition, at operation S720, the processor 120 according to an embodiment of the present disclosure may apply the input image 10' as an input to the second learning network model and acquire the second image. As an example, the processor 120 may acquire a high-resolution second image because the second learning network model performs upscaling that enhances high frequency components corresponding to textures included in the input image 10'. Here, the second image may be an afterimage. Residual information may indicate the difference between each pixel or group of pixels of the original image and the high-resolution image.

According to embodiments of the present disclosure, the first learning network model and the second learning network model, respectively, implement a scale-up that enhances at least one of the characteristics of the input image 10', and thus is compared to the input image 10'. , the first image and the second image have high resolution. For example, if the resolution of the input image 10' is 2K, the resolution of the first image and the second image may be 4K, and if the resolution of the input image 10' is 4K, the resolution of the first image and the second image The resolution can be 8K.

At operation S730, the processor 120 according to an embodiment of the present disclosure may scale up the input image 10' and acquire a third image. According to an embodiment of the present disclosure, the image processing apparatus 100 may include a separate processor for scaling up the input image 10', and the processor 120 may scale up the input image 10' and obtain a high-resolution third image. For example, the processor 120 may use bilinear interpolation, bicubic interpolation, cubic spline interpolation, Lanczos interpolation interpolation), edge directed interpolation (EDI), etc. perform scaling up of the input image 10'. Also, this is only an example and the processor 120 may upscale the input image 10' based on various upscaling (or super-resolution) methods.

As another example, the processor 120 may apply the input image 10' as an input to a third learning network model and obtain a high-resolution third image corresponding to the input image 10'. Here, the third learning network model may be a deep learning based model or a machine learning based model. According to the embodiment of the present disclosure, if the resolution of the input image 10' is 4K, the resolution of the third image may be 8K. In addition, according to the embodiment of the present disclosure, the resolutions of the first image to the third image may be the same.

Next, at operation S740 , the processor 120 may mix the first image to the third image and obtain the output image 20 .

The processor 120 according to an embodiment of the present disclosure may mix the first afterimage, the second afterimage, and the third afterimage to obtain an output image, and the first afterimage is scaled up by enhancing the edges in the input image 10' The input image 10', the second residual image scales the input image 10' by enhancing the texture in the input image 10', and the third residual image scales the input image 10'. Here, the processor 120 may identify edge regions in the input image 10' and apply the identified edge regions to the first learning network model, and enhance the edge regions, and thus obtain a scaled-up first residual image. Additionally, the processor 120 may identify textured regions in the input image 10' and apply the identified textured regions to the second learning network model, and enhance the textured regions, and thereby obtain a scaled-up second residual image. At the same time, this is only instance and configuration and operation are not limited to this. For example, the processor 120 may apply the input image 10' to the first learning network model and the second learning network model. The first learning network model may then identify edge regions and enhance the identified edge regions based on edge characteristics among various image characteristics of the input image 10', and thus obtain a scaled-up high-resolution first residual image. The second learning network model may identify textured regions and enhance the identified textured regions based on texture properties among various image properties of the input image 10', and thereby obtain a scaled-up high-resolution second residual image.

In addition, the processor 120 according to an embodiment of the present disclosure can scale up the input image 10' and obtain a high-resolution third image. Here, the third image may be an image obtained by scaling up the original image instead of the residual image.

According to an embodiment of the present disclosure, the processor 120 may mix the first image to the third image and obtain an output image 20 with a higher resolution than the input image 10'. Here, the output image 20 may be a scaled-up image in which edge and texture regions have been enhanced, rather than an image in which only the resolution has been scaled up. Meanwhile, this is only an example and the processor 120 may obtain a plurality of afterimages in which various image characteristics of the input image 10' have been enhanced and blend a third image that scales up the input image 10' with the plurality of afterimages, And an output image 20 is obtained based on the result of mixing the images.

FIG. 8 is a diagram schematically illustrating an operation of a second learning network model according to an embodiment of the present disclosure.

The processor 120 according to an embodiment of the present disclosure can take the image 10 as an input Applying to a second learning network model and obtaining a second image in which the texture has been enhanced.

A second learning network model according to an embodiment of the present disclosure may store a plurality of filters corresponding to each of a plurality of image patterns. Here, the plurality of image patterns may be classified according to the characteristics of the image blocks. For example, the first image pattern may be an image pattern with a large number of lines in the horizontal direction, and the second image pattern may be an image pattern with a large number of lines in the rotational direction. The plurality of filters may be pre-learned filters by artificial intelligence algorithms.

In addition, the second learning network model according to the embodiment of the present disclosure can read image blocks of a predetermined size from the image 10 . Here, the image block may be a group of a plurality of pixels including a target pixel included in the image 10 and a plurality of surrounding pixels. As an example, the second learning network model may read a first image block of size 3×3 pixels on the upper left end of the image 10 and perform image processing on the first image block. Then, the second learning network model can scan from the upper left end of the image 10 to the right by an amount as much as a unit pixel, and read a second image block with a size of 3×3 pixels and execute the execution on the second image block image processing. The second learning network model can perform image processing on the image 10 by scanning the pixel blocks. Meanwhile, the second learning network model can autonomously read the first image block to the n th image block from the image 10, and the processor 120 can sequentially apply the first image block to the n th image block as input to the The second learns the network model and performs image processing on the image 10 .

To detect high frequency components in the image block, the second learning network model may apply a filter of predetermined size to the image block. As an example, the second learning network model may apply a Laplacian filter 810 of size 3x3 corresponding to the size of the image block to the image block, and thus eliminate the Low frequency components and detect high frequency components. As another example, the second learning network model may be applied to the image by applying various types of filters (eg, Sobel, Prewitt, Robert, Canny, etc.) block to obtain the high frequency components of the image 10 .

Next, the second learning network model may calculate gradient vectors 820 based on the high frequency components obtained from the image blocks. Specifically, the second learning network model may calculate horizontal and vertical gradients and calculate gradient vectors based on the horizontal and vertical gradients. Here, the gradient vector may express the amount of change based on each pixel with respect to pixels located in a predetermined direction. Additionally, the second learning network model may classify the image block into one of a plurality of image patterns based on the directionality of the gradient vector.

Next, the second learned network model may search for the filter (performing a filter search) 830 to be applied to the high frequency components detected from the image 10 by using the index matrix 850 . Specifically, the second learning network model may identify index information indicative of the pattern of the image block based on the index matrix, and search 830 the filter corresponding to the index information. For example, if the index information corresponding to the image block is identified as index information 32 among the index information 1 to 32 indicating the pattern of the image block, the second learning network model may be derived from the plurality of filters The filters mapped to the index information 32 are obtained in . Meanwhile, the above specific index values are only examples and the index information may be decreased or increased according to the number of filters. In addition, index information can be expressed in various ways other than integers.

After that, the second learning network model may obtain at least one filter of the plurality of filters included in the filter database (DB) 860 based on the search result, And applying 840 the at least one filter to the image block to obtain a second image. As an example, the second learning network model may identify the filter corresponding to the pattern of the image block among the plurality of filters based on the search result and apply the identified filter to the image block, and then obtain the texture area in which the texture area has been The second image is scaled up.

Here, the filters included in the filter database 860 can be acquired according to the result of learning the relationship between the low-resolution image blocks and the high-resolution image blocks by the artificial intelligence algorithm. For example, the second learning network model can learn a low-resolution first image block and a high-resolution second image in which the texture area of the first image block has been scaled up by an artificial intelligence algorithm The relationship between the blocks is identified, and the filter to be applied to the first image block is identified and the identified filter is stored in the filter database 860 . However, this is only an example and the present disclosure is not limited thereto. For example, the second learning network model may identify a filter that enhances at least one of the various characteristics of the image block by using the result of the learning using an artificial intelligence algorithm and store the identified filter in a Filter database 860.

FIG. 9 is a diagram illustrating index information according to an embodiment of the present disclosure.

The processor 120 according to an embodiment of the present disclosure may accumulate the index information of the image patterns corresponding to each classified image block and obtain an accumulation result. Referring to FIG. 9 , the processor 120 may acquire index information corresponding to the image pattern of the image block among the index information indicating the image pattern. The processor 120 may then accumulate the index information for each of the plurality of image blocks included in the image 10, and thus obtain an accumulation result, as shown in FIG. 9 .

The processor 120 may analyze the accumulated results and identify the image 10 as one of a natural image or a graphic image. For example, if the number of image blocks that do not include a pattern (or show no directionality) among the image blocks included in the image 10 is equal to or greater than a threshold value, then based on the accumulated results, the processor 120 may convert the image 10 is recognized as a graphic image. As another example, if the number of image blocks that do not include patterns among the image blocks included in the image 10 is less than a threshold value, the processor 120 may recognize the image 10 as a natural image based on the accumulated results. As yet another example, if the number of image blocks with patterns in the vertical direction or patterns in the horizontal direction is equal to or greater than the threshold value, the processor 120 may recognize the image 10 as a natural image based on the accumulated results. Meanwhile, the identification and classification of the images are only exemplary and the threshold value may be specified according to the purpose of the manufacturer, the setting of the user, and the like.

As another example, the processor 120 may calculate the number and ratio of specific index information based on the accumulated result, and identify the type of the image 10 as a natural image or a graphic image based on the accumulated result. For example, the processor 120 may calculate at least three features based on the accumulated results.

If specific index information in the index information is information indicating an image block whose pattern is not recognized (or does not show directionality), the processor 120 may calculate the ratio of the index information from the accumulated result. In the following, image blocks of unrecognized patterns are generally referred to as image blocks including flat areas. The proportion of the image block including the flat area in the entire image block can be calculated based on Equation 3 below.

[Equation 3]

Here, Histogram[i] means the number of image blocks with index information "i" identified based on the accumulation result. In addition, based on the assumption that the index information indicating the image block including the flat area is 32, Histogram[32] means the number of image blocks with index information 32, and P1 means that the image block including the flat area is in the whole image The proportion in the block.

If the image block includes a pattern, the processor 120 may identify whether the pattern is located in a central area inside the image block based on the index information. As an example, compared to the image blocks whose index information is 1-12 and 17-31, the pattern of the image blocks whose index information is 13-16 may be located in the central area inside the block. Hereinafter, the image blocks in which the image patterns are located in the central area inside the image blocks are generally referred to as the image blocks distributed in the center. Next, the processor 120 may calculate the proportion of the image blocks distributed in the center based on the following Equation 4 based on the accumulated results.

，以辨識除包括平坦區域的影像區塊之外的包括圖案的影像區塊的數目。另外，處理器120可計算居中分佈的影像區塊的數目

。同時，具有索引資訊13至15的影像區塊僅為其中圖案位於影像區塊內部的中心區域 的情形的實例且本揭露未必僅限於此。作為另一實例，可基於索引資訊11至17的數目來計算P2。 Here, the processor 120 may count the number of image blocks with index information 1 to 31

, to identify the number of image blocks including patterns other than image blocks including flat areas. In addition, the processor 120 may calculate the number of image blocks distributed in the center

. Meanwhile, the image blocks with the index information 13 to 15 are only examples of the case where the pattern is located in the central area inside the image block and the present disclosure is not necessarily limited thereto. As another example, P2 may be calculated based on the number of index information 11-17.

Then, the processor 120 may obtain average index information of the image 10 based on the index information of each of the plurality of image blocks included in the image 10 . According to an embodiment of the present disclosure, the processor 120 may calculate the average index information based on Equation 5 below.

Here, "i" means index information, Histogram[i] means the number of image blocks corresponding to index information i, and P3 means average index information.

The processor 120 according to an embodiment of the present disclosure may calculate the "Y" value based on Equation 6 below.

[Equation 6] Y = W 1* P 1+ W 2* P 2+ W 3* P 3+ Bias

Here, P1 means the proportion of image blocks including flat areas, P2 means the proportion of image blocks distributed in the middle, and P3 means the average index information. In addition, W1, W2, W3 and Bias refer to parameters pre-learned by using an artificial intelligence algorithm model.

If the Y value exceeds 0, the processor 120 according to an embodiment of the present disclosure may recognize the image 10 as a graphic image, and if the "Y" value is equal to or less than 0, the processor 120 may recognize the image 10 as a natural image.

Then, the processor 120 may adjust the first weight and the second weight respectively corresponding to the first image and the second image based on the identification result. As an example, if the image 10 is identified as a natural image, the processor 120 may increase at least one of the first weight corresponding to the first image or the second weight corresponding to the second image. Additionally, processor 120 may increase at least one of the parameters "a" or "b" in Equation 1 and Equation 2. Meanwhile, if the image 10 is a natural image, the processor 120 can acquire a high-resolution image, since the first image in which the edges have been improved or the second image in which the texture has been improved is added to the image 10 or the input image 10 ', so the transparency of the high-resolution image has been improved, and thus the processor 120 can increase at least one of the first weight or the second weight.

As another example, if the image 10 is identified as a graphic image, the processor 120 may reduce at least one of the first weight corresponding to the first image or the second weight corresponding to the second image. Additionally, processor 120 may reduce at least one of the parameters "a" or "b" in Equation 1 and Equation 2. Meanwhile, if the image 10 is a graphic image, since the first image in which the edge has been enhanced or the second image in which the texture has been enhanced is added to the image 10 or the input image 10 ′, the processor 120 can obtain the distortion in which the distortion occurs. , and thus the processor 120 can reduce at least one of the first weight or the second weight, and thus minimize the occurrence of distortion.

Here, the graphic image may be an image manipulated with an image of the real world, or an image newly created by using a computer, an imaging device, or the like. For example, graphic images may include example images, computer graphic (CG) images, animation images, etc. generated by using known software. Natural images can be removed image other than the shape image. For example, natural images may include real-world images, landscape images, portrait images, etc. captured by a photographing device.

FIG. 10 is a diagram illustrating a method of obtaining a final output image according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, in the case where the final output image 30 ′ (ie, the display image) is an image with a higher resolution than that of the output image 30 , at operation S350 , the processor 120 may scale up the output image 30 and obtain the final output image 30'. For example, if the output image 30 is a 4K UHD image and the final output image is an 8K image, the processor 120 may scale the output image 30 to an 8K UHD image and obtain the final output image 30'. Meanwhile, according to another embodiment of the present disclosure, a separate processor for performing the upscaling of the output image 30 may be provided in the image processing device 100 . For example, the image processing apparatus 100 may include a first processor and a second processor, and obtain the output image 30 in which edges and textures have been enhanced by using the first processor and obtain high-resolution images by using the second processor The final output image 30 ′ of the resolution zooms in on the resolution of the output image 30 .

Meanwhile, each of the first learning network model and the second learning network model according to various embodiments of the present disclosure may be an on-device machine in which the image processing apparatus 100 performs learning by itself without relying on an external device Learning model (on-device machine learning model). Also, this is just an example and some learning network models may be implemented based on on-device operations, and other learning network models may be implemented based on external servers.

FIG. 11 is a block diagram showing a detailed configuration of the image processing apparatus shown in FIG. 2 .

According to FIG. 11 , the image processing apparatus 100 ′ includes a memory 110 , a processor 120 , an input device 130 , a display 140 , an output device 150 and a user interface 160 . Meanwhile, in explaining the components shown in FIG. 11 , redundant explanation of components similar to those shown in FIG. 2 will be omitted.

According to embodiments of the present disclosure, memory 110 may be implemented as a single memory that stores data generated from various operations in accordance with the present disclosure.

The memory 110 may be implemented to include a first memory to a third memory.

The first memory can store at least a part of the image input through the input device 130 . Specifically, the first memory can store at least some regions of the input image frame. In such a configuration, the at least some regions may be regions necessary to perform image processing in accordance with embodiments of the present disclosure. Meanwhile, according to an embodiment of the present disclosure, the first memory may be implemented as an N-row memory. For example, the N-row memory may be a memory having as much capacity as 17 rows in the horizontal direction, but the memory is not limited thereto. For example, in the case of inputting a full HD image of 1080 pixels (with a resolution of 1920×1080), only the image area of 17 lines in the full HD image can be stored in the first memory. As mentioned above, the reason why the first memory is implemented as N-line memory and only stores some areas of the input image frame for image processing is that the memory capacity of the first memory is limited due to hardware limitations. Meanwhile, the second memory may be a memory area allocated to the learning network model in the entire area of the memory 110 .

The third memory stores therein the first image and the second image and the output The memory for outputting the image, and according to various embodiments of the present disclosure, the third memory may be implemented as memory of various sizes. According to an embodiment of the present disclosure, the processor 120 applies the scaled down image 10 of the input image 10' to the first learning network model and the second learning network model, and is thus stored from the first learning network model and the second learning network model. The size of the third memory of the first image and the second image acquired by the two learning network models may be implemented to be the same or similar to the size of the first memory.

The input device 130 may be a communication interface (eg, a wired Ethernet interface or a wireless communication interface) for receiving various types of content (eg, image signals from an image source). For example, the input device 130 can be accessed from an external device (eg, source device), an external storage medium (eg, USB), an external device (eg, a source device), an external storage medium (eg, USB), an A server (eg, a webhard), etc., receives video signals in a streaming method or a download method: AP-based Wi-Fi (Wireless Local Area Network (LAN) network), Bluetooth , Zigbee, Wired/Wireless Local Area Network (LAN), Wide Area Network (WAN), Ethernet, Long Trem Evolution (LTE), 5th Generation Mobile Communication Technology ( 5th-generation, 5G), Institute of Electrical and Electronic Engineers (IEEE) 1394, High Definition Multimedia Interface (HDMI), Mobile High-Definition Link, MHL), Universal Serial Bus (USB), Display Port (DP), Thunderbolt (Thunderbolt), Video Graphic Array (VGA) port, Red, Green and Blue (RGB) port, D-Ultra Small (D-subminiature, D-SUB), digital visual interface (Digital Visual Interface, DVI) and so on. Specifically, the 5G communication system is communication using ultra-high frequency (millimeter wave (mmWave)) frequency bands (eg, millimeter wave frequency bands such as 26, 28, 38, and 60 gigahertz frequency bands), and the image processing device 100 UHD video in 4K and 8K can be sent or received in a streaming environment.

Here, the video signal may be a digital signal, but the video signal is not limited to this.

The display 140 may be implemented in various forms such as liquid crystal display (LCD), organic light-emitting diode (OLED), light-emitting diode (LED), micro LED, Quantum dot light-emitting diode (QLED), liquid crystal on silicon (LCoS), digital light processing (DLP) and quantum dot (QD) display panels . Specifically, the processor 120 according to the embodiment of the present disclosure can control the display 140 to display the output image 30 or the final output image 30'. Here, the final output image 30' may include 4K or 8K real-time UHD images, streaming images, and the like.

The output device 150 outputs an acoustic signal.

For example, the outputter 150 may convert the digital audio signal processed at the processor 120 into an analog audio signal, amplify the signal, and output the signal. For example, the outputter 150 may include at least one speaker unit, a digital-to-analog (D/A) converter, an audio amplifier, etc., and the outputter 150 may output at least one channel. According to an embodiment of the present disclosure, the outputter 150 may be implemented to output each A multi-channel sound signal. In this case, the processor 120 can control the output unit 150 to perform enhancement processing on the input of the audio signal, so as to correspond to the enhancement processing of the input image, and output the signal. For example, the processor 120 may convert the input two-channel acoustic signal into a virtual multi-channel (eg, 5.1-channel) acoustic signal, or identify the location within the environment where the image processing device 100 is placed in a room or building and convert the The signal processing is a stereo signal optimized for the space, or provides an audio signal optimized according to the type of input image (eg, the genre of the content). Meanwhile, the user interface 160 can be implemented as a device such as a button, a touchpad, a mouse, and a keyboard, or as a touchscreen, which can receive user input to perform both the above-mentioned display function and the remote control reception of the manipulation input function. device. The remote control transceiver can receive the remote control signal from the external remote control device or transmit the remote control signal to the external remote control device by at least one communication method among infrared communication, bluetooth communication or Wi-Fi communication. Meanwhile, although not shown in FIG. 9, according to embodiments of the present disclosure, free filtering to remove noise of the input image may be applied before image processing. For example, significant noise can be removed by applying a smoothing filter (eg, Gaussian filter, steering filter, etc.) that filters the input image by comparing images of predetermined waveguides.

12 is a block diagram illustrating a configuration of an image processing apparatus for learning and using a learning network model according to an embodiment of the present disclosure.

Referring to FIG. 12 , the processor 1200 may include at least one of a learning part 1210 or a discriminating part 1220 . The processor 1200 in FIG. 12 may correspond to the processor 120 of the image processing apparatus 100 in FIG. 2 or the processor of the data learning server.

For learning and using the first learning network model and the second learning network model The processor 1200 of the image processing apparatus 100 of the type 1200 may include at least one of the learning component 1210 or the identifying component 1220 .

The learning component 1210 according to an embodiment of the present disclosure may acquire an image in which the image characteristic of the image 10 has been enhanced, and acquire an output image based on the image 10 and the image in which the image characteristic of the image 10 has been enhanced. Next, the learning component 1210 may generate or train a discrimination model with criteria for minimizing distortion of the image 10 and obtaining a high-resolution scaled-up image corresponding to the image 10 . In addition, the learning component 1210 can generate a discrimination model with judgment criteria by using the collected learning data.

As an example, learning component 1210 may generate, train, or update a learning network model such that at least one of edge or texture regions of output image 30 is enhanced to more than edge or texture regions of input image 10'.

The discrimination component 1220 can use predetermined data (eg, input images) as input data for a trained discrimination model, and thus estimate objects for discrimination or conditions included in the predetermined data.

At least a portion of the learning component 1210 and at least a portion of the identification component 1220 may be implemented as a software module or fabricated in the form of at least one hardware chip and mounted on an image processing device. For example, at least one of the learning component 1210 or the discrimination component 1220 may be fabricated in the form of a hardware chip dedicated to artificial intelligence (AI), or as a conventional general-purpose processor (eg, a CPU or application processor) Or part of a graphics-specific processor (eg, GPU), and installed on the above-mentioned various types of image processing devices or object recognition devices. Here, the hardware dedicated to artificial intelligence The bulk chip is a special-purpose processor for probabilistic operations and has higher parallel processing performance than traditional general-purpose processors, and can quickly process operations in the field of artificial intelligence (such as machine learning). In the case where the learning component 1210 and the identifying component 1220 are implemented as one or more software modules (or program modules including instructions), the software modules may be stored on a non-transitory computer readable medium (non-transitory computer readable medium). medium). In this case, the software module may be provided by an operating system (OS) or by a specific application. Alternatively, a portion of the software module may be provided by an operating system (OS) and other portions may be provided by a specific application.

In this case, the learning part 1210 and the discriminating part 1220 may be installed on one image processing device, or may be installed on separate image processing devices respectively. For example, one of the learning component 1210 and the identifying component 1220 may be included in the image processing device 100, and the other may be included in an external server. In addition, the learning component 1210 and the identifying component 1220 may be connected in a wired or wireless manner, or may be a larger software module or a separate software module for an application. The model information constructed by the learning component 1210 may be provided to the identifying component 1220, and the data input to the identifying component 1220 may be provided to the learning component 1210 as additional learning materials.

FIG. 13 is a flowchart illustrating an image processing method according to an embodiment of the present disclosure.

According to the image processing method shown in FIG. 13 , first, at operation S1310 , the image is applied to the first learning network model and the first image in which the edge of the image has been enhanced is obtained.

Next, at operation S1320, the image is applied to the second learning network model and a second image in which the texture of the image has been enhanced is obtained.

Next, at operation S1330, an edge region and a texture region included in the image are identified, and a first weight is applied to the first image and a second weight is applied to the second image based on the information about the edge region and the texture region , and get the output image.

Here, the first learning network model and the second learning network model may be different types of learning network models from each other.

The first learning network model according to an embodiment of the present disclosure can be a deep learning model that learns to enhance the edge of an image by using multiple layers or a machine that is trained to enhance the edge of an image by using multiple pre-learning filters One of the learning models.

In addition, the second learning network model according to an embodiment of the present disclosure may be a deep learning model that learns to optimize the texture of an image by using multiple layers or is trained to use multiple pre-learning filters to One of the machine learning models that optimizes the texture of an image.

In addition, the operation S1330 of obtaining the output image may include the following steps: obtaining a first weight corresponding to the edge area and a second weight corresponding to the texture area based on the ratio information of the edge area and the texture area.

In addition, the image processing method according to the embodiment of the present disclosure may include the step of scaling down the input image and obtaining an image with a resolution smaller than that of the input image. At the same time, the first learning network model can enhance the edge of the image by performing a button The first image is acquired by scaling up, and the second learning network model may acquire the second image by performing the scaling up that enhances the texture of the image.

In addition, the image processing method according to the embodiment of the present disclosure may include the following steps: acquiring area detection information, the area detection information identifies the edge area and the texture area based on the scaled-down image, and the area detection information and the image are respectively provided to The first learning network model and the second learning network model.

Here, the step of providing the region detection information and the image to the first learning network model and the second learning network model respectively may include the following steps: based on the region detection information, only the pixel information corresponding to the edge region will be included. The image is provided to the first learning network model and the image including only the pixel information corresponding to the texture region is provided to the second learning network model. Meanwhile, the first learning network model can acquire the first image by scaling up the edge area, and the second learning network model can acquire the second image by scaling up the texture area.

In addition, the first image and the second image according to the embodiment of the present disclosure may be the first afterimage and the second afterimage, respectively. In addition, in the operation S1330 of obtaining the output image, the first weight may be applied to the first afterimage and the second weight may be applied to the second afterimage, and then the afterimage and the image may be mixed to obtain the output image.

Furthermore, the second learning network model may be a model that stores a plurality of filters corresponding to each of the plurality of image patterns and classifies each of the image blocks included in the image into the plurality of one of the image patterns, and applying at least one of the plurality of filters corresponding to the classified image pattern to the image area block and provide a second image.

In addition, the operation S1330 of acquiring an output image according to an embodiment of the present disclosure may include the following steps: accumulating index information of image patterns corresponding to each of the classified image blocks and identifying the type of the image based on the accumulation result ( such as one of natural images or graphic images), and adjust the weights based on the recognition results.

Here, the step of adjusting the weight may include the steps of increasing at least one of a first weight corresponding to the first image or a second weight corresponding to the second image based on the image being identified as a natural image, and based on the image Recognized as a graphic image, at least one of the first weight or the second weight is reduced.

The output image may be a 4K Ultra High Definition (UHD) image, and the image processing method according to an embodiment of the present disclosure may include the step of upscaling the output image to an 8K UHD image.

Meanwhile, various embodiments of the present disclosure can be applied to all electronic devices capable of performing image processing (eg, image receiving devices (such as set top boxes and image processing devices)) and image processing devices.

In addition, the various embodiments described above can be implemented by a recording medium readable by a computer or a computer-like device using software, hardware, or a combination thereof. In some cases, the embodiments described in this specification may be implemented as the processor 120 itself. Depending on the implementation of the software, the embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more of the functions and operations described in this specification.

Meanwhile, for implementing the sound input according to the above-described various embodiments of the present disclosure Computer instructions for processing operations of the device 100 may be stored in a non-transitory computer readable medium. The processing operations at the audio output device 100 according to the various embodiments described above are performed by the specific device when the computer-readable instructions stored in such a non-transitory computer-readable medium are executed by the processor of the specific device.

A non-transitory computer-readable medium refers to a medium that stores data semi-permanently rather than for a short period of time (eg, scratchpad, cache, and memory) and is readable by a machine. Specific examples of non-transitory computer-readable media include compact disc (CD), digital versatile disk (DVD), hard disk, Blu-ray disc, USB, memory card, ROM, and the like.

While embodiments of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described above, and it will be apparent to those familiar with the present disclosure without departing from the gist of the present disclosure as claimed in the scope of the appended claims. Various modifications may be made by those skilled in the art. In addition, such modifications are not intended to be interpreted independently of the technical idea or prospect of the present disclosure.

100: Image processing device/sound output device

110: Memory

120: Processor

Claims (15)

An image processing device, comprising: a memory storing computer-readable instructions; and a processor configured to execute the computer-readable instructions to: provide an input image as a first input of a first neural model, from which obtaining a first image from the first neural model, the first image including enhanced edges optimized based on edges of the input image, providing the input image as a second input for a second neural model, from the first image Two neural models obtain a second image, the second image including an enhanced texture optimized based on the texture of the input image, and identify the edge region of the edge included in the input image to obtain a region corresponding to the edge of the input image. a first weight of the edge area, identifying a texture area of the texture included in the input image to obtain a second weight corresponding to the texture area, providing the first weight to the area including the enhanced edge the first image, the second image provided with the second weight to the second image including the enhanced texture, and based on the first image provided with the first weight and the second image provided with the second weight The second image is described to obtain the output image. The image processing apparatus of claim 1, wherein the first type of the first neural model is different from the second type of the second neural model. The image processing device according to claim 1, wherein the first god The model is a deep learning model that optimizes the edges of the input image by using multiple layers or is trained to optimize the edges of the input image by using multiple pre-learned filters One of the optimized machine learning models. The image processing apparatus of claim 1, wherein the second neural model is a deep learning model that optimizes the texture of the input image by using a plurality of layers or is trained by using one of a plurality of machine learning models that pre-learn filters to optimize the texture of the input image. The image processing device of claim 1, wherein the processor executing the computer-readable instructions is further configured to: based on the edge region in the input image and the edge region in the input image The first weight and the second weight are obtained from the scale information of the texture area. The image processing device of claim 1, wherein the processor executing the computer-readable instructions is further configured to: scale down the input image to obtain a resolution having a smaller resolution than the input image A scaled-down image of a resolution of the first image having the enhanced edge, providing the scaled-down image as the second input to the second neural model, and providing the scaled-down image from the first image that scales the scaled-down image two neural models type to obtain the second image with the enhanced texture. The image processing device of claim 1, wherein the processor executing the computer-readable instructions is further configured to: obtain first region detection information for identifying the edge region of the input image and second region detection information identifying the textured region of the input image, providing the input image and the first region detection information as the first input to the first neural model, and The input image and the second region detection information are provided as the second input to the second neural model. The image processing device of claim 7, wherein the first neural model obtains the first image by scaling up the edge region, and the second neural model obtains the first image by scaling up the texture area to obtain the second image. The image processing device according to claim 1, wherein the second neural model is a model: the model stores a plurality of filters corresponding to a plurality of image patterns, and the image blocks included in the input image Each image block in is classified into an image pattern of the plurality of image patterns, and at least one filter of the plurality of filters corresponding to the image pattern is provided to the image block. The image processing device of claim 1, wherein the first image is a first afterimage and the second image is a second afterimage, and wherein the processor executing the computer-readable instructions further is configured with: The first weight is provided to the first afterimage based on the edge region, the second weight is provided to the second afterimage based on the texture region, and the first weight and the After the second weight, the first residual image, the second residual image and the input image are mixed to obtain the output image. The image processing device of claim 10, wherein the processor executing the computer-readable instructions is further configured to: perform processing of image patterns corresponding to each of the image blocks of the input image Index information is accumulated, identifying the input image as one of a natural image or a graphic image based on the index information, and based on identifying the input image as one of the natural image or the graphic image The first weight and the second weight are adjusted as a result. The image processing device of claim 11, wherein the processor executing the computer-readable instructions is further configured to: increase the first weight based on the input image being recognized as the natural image or at least one of the second weights, and reducing at least one of the first weights or the second weights based on the input image being identified as the graphic image. An image processing method of an image processing device, comprising: providing an input image as a first input to a first neural model; obtaining a first image from the first neural model, the first image including enhanced edges optimized based on edges of the input image; providing the input an image as a second input to a second neural model; obtaining a second image from the second neural model, the second image including an enhanced texture optimized based on the texture of the input image; identifying including an edge region of the edge to obtain a first weight corresponding to the edge region; identifying a texture region included in the input image to obtain a second weight corresponding to the texture region; providing the providing the first weight to the first image including the enhanced edge; providing the second weight to the second image including the enhanced texture; and providing the first weight based on the first image an image and the second image provided with the second weight to obtain an output image. The image processing method of claim 13, wherein the first neural model is a deep learning model that optimizes the edges of the input image by using a plurality of layers or is trained by using one of a plurality of machine learning models that pre-learn filters to optimize the edges of the input image. The image processing method of claim 13, wherein the second neural model is a deep learning model that optimizes the texture of the input image by using multiple layers or is trained by using multiple pre-learned filters to one of the machine learning models that optimize the texture of the input image.

Images