TW202044196A - Image processing apparatus and image processing method thereof - Google Patents

Abstract

An image processing apparatus applies an image to a first learning network model to optimize the edges of the image, applies the image to a second learning network model to optimize the texture of the image, and applies a first weight to the first image and a second weight to the second image based on information on the edge areas and the texture areas of the image to acquire an output image.

Description

Image processing device and its image processing method

The present disclosure relates to an image processing device and an image processing method thereof, and more specifically, to an image processing device and an image processing method thereof that enhance the characteristics of an image by using a learning network model.

Stimulated by the development of electronic technology, various types of electronic devices have been developed and released. Specifically, in recent years, image processing devices have been used in various places (for example, homes, offices, and public spaces) and are continuously developing.

Recently, high-resolution display panels (for example, 4K Ultra High Definition (UHD) televisions (TV)) have been introduced and widely released. However, the usability of high-resolution content for reproduction on such high-resolution display panels is limited. Therefore, various technologies for generating high-resolution content from low-resolution content are being developed. Specifically, the demand for efficient processing of a large number of operations necessary to generate high-resolution content within limited processing resources is increasing.

In addition, recently, artificial intelligence (artificial intelligence) systems that replicate human-level intelligence have been used in various fields. Unlike traditional rule-based intelligent systems, artificial intelligence systems refer to systems in which machines perform autonomous learning, judgment, and processing. As the artificial intelligence system performs iterative operations, the system shows a more improved recognition rate, and for example, becomes able to understand user preferences more correctly. Therefore, traditional rule-based intelligent systems are gradually being replaced by artificial intelligence systems based on deep learning.

Artificial intelligence technology consists of machine learning (for example, deep learning) and element technology using machine learning.

Machine learning refers to an algorithm technique that autonomously classifies/learns the characteristics of input data. At the same time, elemental technology refers to the technology that simulates the functions of the human brain (for example, cognition and judgment) by using machine learning algorithms (for example, deep learning) and includes, for example, language understanding, visual understanding, inference/prediction, knowledge representation ) And technical fields such as operation control.

Attempts have been made to enhance the image characteristics by using artificial intelligence technology in traditional image processing devices. However, there is the following problem: For the performance of the conventional image processing device, the amount of processing required to generate high-resolution images is limited and takes a lot of time. Therefore, there is a need for a technology that enables image processing devices to generate high-resolution images and provide images by performing only a few operations.

The present disclosure aims to solve the aforementioned needs and provide a high-resolution image processing device with improved image characteristics and an image processing method thereof by using multiple learning network models.

According to an embodiment of the disclosure, an image processing method for achieving the above-mentioned purpose includes: a memory storing computer-readable instructions; and a processor configured to execute the computer-readable instructions to: take the input image as A first input is applied to a first learning network model to obtain a first image from the first learning network model, the first image including an enhanced edge optimized based on the edge of the input image; and The input image is applied as a second input to a second learning network model and a second image is obtained from the second learning network model. The second image includes an enhanced texture optimized based on the texture of the input image. The processor recognizes the edge area and the texture area included in the image, and based on the information about the edge area and the texture area, applies a first weight to the first image and a second weight to The second image obtains an optimized output image from the input image based on the first weight applied to the first image and the second weight applied to the second image.

In addition, the first type of the first learning network model is different from the second type of the second learning network model.

In addition, the first learning network model may be a deep learning model that optimizes the edge of the input image by using multiple layers or is trained to use multiple pre-learning filters. One of the machine learning models that optimizes the edge of the input image.

In addition, the second learning network model may be a deep learning model that optimizes the texture of the input image by using multiple layers, or a deep learning model that optimizes the texture of the input image by using multiple pre-learning filters. One of the machine learning models optimized for the texture of the input image.

At the same time, the processor may obtain the first weight corresponding to the edge region and the second weight corresponding to the texture region based on the ratio information of the edge region and the texture region.

In addition, the processor may scale down the input image to obtain a scaled down image with a resolution lower than that of the input image. In addition, the first learning network model may obtain the first image with the enhanced edge from the first learning network model that scales the scaled down image, and the second The learning network model may obtain the second image with the enhanced texture from the second learning network model that scales up the scaled-down image.

In addition, the processor may obtain area detection information in which the edge area and the texture area have been identified based on the scaled-down image, and provide the area detection information and the image to the The first learning network model and the second learning network model.

In addition, the first learning network model can obtain the first image by scaling up the edge area, and the second learning network model can obtain the first image by scaling up the texture area. The second image.

In addition, the first image and the second image may be a first afterimage and a second afterimage, respectively. In addition, the processor may apply the first weight to the first afterimage based on the edge region and apply the second weight to the second afterimage based on the texture region, and then apply the The first afterimage, the second afterimage, and the input image are mixed to obtain the output image.

At the same time, the second learning network model may be the following model: the model stores a plurality of filters corresponding to each of a plurality of image patterns, and the image blocks included in the image Each is classified into one of the plurality of image patterns, and at least one filter of the plurality of filters corresponding to the classified image pattern is applied to the image block and provides the first Two images.

Here, the processor may accumulate index information of the image pattern corresponding to each of the classified image blocks, and recognize the image as a natural image or a graphic image based on the index information And adjust the first weight and the second weight based on the result of recognizing the input image as one of the natural image or the graphic image.

Here, the processor may increase at least one of the first weight or the second weight based on the input image being recognized as the natural image, and based on the input image being recognized as the natural image. The graphic image reduces at least one of the first weight or the second weight.

At the same time, an image processing method of an image processing device according to an embodiment of the disclosure includes the following steps: applying an input image as a first input to a first learning network model; acquiring the first image from the first learning network model, The first image includes an enhanced edge optimized based on the edge of the input image; the input image is applied as a second input to a second learning network model; the first image is obtained from the second learning network model Two images, the second image includes an enhanced texture optimized based on the texture of the input image; identifying the edge region of the edge included in the input image; identifying the texture included in the input image Region; applying a first weight to the first image based on the edge region; applying a second weight to the second image based on the texture region; and based on the first image applied to the first image The weight and the second weight applied to the second image obtain an optimized output image from the input image.

Here, the first learning network model and the second learning network model may be different types of learning network models.

In addition, the first learning network model may be a deep learning model that optimizes the edge of the input image by using multiple layers or is trained to use multiple pre-learning filters. One of the machine learning models that optimizes the edge of the input image.

In addition, the second learning network model can be a deep learning model that optimizes the texture of the input image by using multiple layers, or can be trained to use multiple pre-learning filters. One of the machine learning models that optimize the texture of the input image.

In addition, the following step: obtaining the first weight and the second weight based on the ratio information between the edge area in the input image and the texture area in the input image.

In addition, the image processing method may include the following steps: scaling down the input image to obtain a scaled down image with a resolution lower than that of the input image. At the same time, the first learning network model can obtain the first image by scaling up the scaled down image, and the second learning network model can be scaled up by scaling up the scaled down image Image to obtain the second image.

Here, the image processing method may include the following steps: acquiring first area detection information for recognizing the edge area of the input image and a second area for recognizing the texture area of the input image Detecting information, and providing the area detection information and the image to the first learning network model and the second learning network model, respectively.

At the same time, the first learning network model can obtain the first image by scaling up the edge area, and the second learning network model can obtain the first image by scaling up the texture area. The second image.

In addition, the first image and the second image may be a first afterimage and a second afterimage, respectively.

According to various embodiments of the present disclosure as described above, a high-resolution image is generated by applying different learning network models to the image, and the amount of operations required to generate the high-resolution image is reduced, and therefore, High-resolution images are generated within the limited resources of the image processing device and can be provided to users.

Hereinafter, the disclosure will be described in detail with reference to the accompanying drawings.

Taking into account the functions described in this disclosure, try to choose general terms that are traditionally and widely used as terms used in the embodiments of this disclosure. However, the terms can be changed according to the intention of those familiar with the technology in the related field or the emergence of new technology. In addition, in certain situations, there may be designated terms, and in this case, the meaning of the terms will be elaborated in the relevant description in this disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of the disclosure, and not only based on the names of the terms.

In this manual, expressions such as "have", "may have", "include" and "may include" shall be regarded as indicating the existence of such characteristics (for example , Such as numerical values, functions, operations and components), and the terms are not intended to exclude the existence of additional features.

In addition, the expression "at least one of A and/or B" should be interpreted as meaning either "A" or "B" or "A and B".

In addition, the expressions "first", "second", etc. used in this specification can be used to describe various elements without regard to any order and/or importance. In addition, this expression is only used to distinguish one element from another element, and is not intended to limit the element.

At the same time, the description in this disclosure couples one element (for example, the first element) with another element (for example, the second element) "(operably or communicably)" or "(operably or communicably) ) Coupled to" another element (for example, a second element) or "connected to" another element (for example, a second element) shall be interpreted as meaning that the one element is directly coupled to the other element, or The one element is coupled to the other element via another element (for example, a third element).

Unless clearly defined differently in the context, the singular expression includes the plural expression. In addition, in this disclosure, terms such as "include" and "have" should be regarded as indicating the existence of such characteristics, numbers, steps, operations, elements, components, or combinations thereof described in the specification , Rather than precluding the possibility of the existence or addition of one or more of other characteristics, numbers, steps, operations, elements, components or combinations thereof.

In addition, in this disclosure, the "module" or "unit" can perform at least one function or operation, and can be implemented as hardware or software or as a combination of hardware and software. In addition, multiple "modules" or multiple "units" can be integrated into at least one module and can be implemented as at least one processor, but need to be implemented as "modules" or "units" of specific hardware except.

In addition, in this specification, the term "user" may refer to a person or device that operates an electronic device (for example, an artificial intelligence electronic device).

Hereinafter, the embodiments of the present disclosure will be explained in more detail with reference to the accompanying drawings.

FIG. 1 is a diagram showing an implementation example of an image processing device according to an embodiment of the disclosure.

The image processing device 100 may be implemented as a television (TV) as shown in FIG. 1. However, the image processing device 100 is not limited to this, and the image processing device 100 can be implemented as any of the following devices equipped with an image processing function and/or a display function: for example, a smart phone, a tablet personal computer (personal computer, PC), laptop personal computer, head mounted display (HMD), near eye display (NED), large format display (LFD), digital signage, digital information Display (digital information display, DID), video wall (video wall), projector display, camera, camcorder, printer, etc., without limitation.

The image processing device 100 can receive images of various resolutions or various compressed images. For example, the image processing device 100 may receive an image 10 formatted according to any of the following images: standard definition (SD) images, high definition (HD) images, full HD images, and Ultra HD video. In addition, the image processing device 100 can receive the image 10 in the following encoding format or compression format, for example: Moving Picture Experts Group (MPEG) (for example, MP2, MP4, MP7, etc.), advanced video coding (advanced video) coding, AVC), H.264, high efficiency video coding (HEVC), etc.

Although the image processing device 100 according to the embodiment of the present disclosure is implemented as a UHD TV, due to the limited availability of UHD content, there are many situations where only the following images are available: for example, standard definition (SD) images, high definition (HD) images And full HD images (hereinafter referred to as low-resolution images). In this case, it is possible to provide a method of amplifying a low-resolution input image to a UHD image (hereinafter referred to as a high-resolution image) and providing the resulting image. As an example, a low-resolution image can be applied as an input to the learning network model, so that the low-resolution image can be enlarged, and therefore, the high-resolution image can be acquired as an output for display on the image processing device 100.

However, in order to enlarge a low-resolution image to a high-resolution image, a lot of complex processing operations are generally required to transform the image data. Therefore, an image processing device 100 with high performance and high complexity is required to perform this transformation. As an example, in order to scale up a 60P image in the SD class with a resolution of 820×480 to a high-resolution image, the image processing device 100 should perform an operation of 820×480×60 pixels per second. Therefore, a processing unit with high performance is required, such as a central processing unit (CPU) or a graphics processing unit (GPU) or a combination thereof. As another example, the image processing device 100 should perform an operation of 3840×2160×60 pixels per second to scale up a 60P image in a UHD level with a resolution of 4K to an image with a resolution of 8K. Therefore, there is a need for a processing unit capable of processing a large number of operations, such as an amount of at least 24 times that in the case of scaling up images in the SD level.

Therefore, in the following, various embodiments of providing the image processing device 100 will be described. The image processing device 100 reduces the amount of operations required to scale up a lower resolution image to a higher resolution image, and thus enables the image processing The limited resources of the device 100 are maximized.

In addition, various embodiments in which the image processing device 100 acquires an output image while enhancing or enhancing at least one of the various characteristics of the input image will be described.

FIG. 2 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the disclosure.

According to FIG. 2, the image processing device 100 includes a memory 110 and a processor 120.

The memory 110 is electrically connected to the processor 120 and can store data necessary to execute various embodiments of the present disclosure. For example, the memory 110 may be implemented as an internal memory, such as a read-only memory (ROM) included in the processor 120 (for example, an electrically erasable programmable read-only memory) (Electrically erasable programmable read-only memory, EEPROM)), random access memory (random access memory, RAM), etc., or a memory separate from the processor 120.

The memory 110 can be implemented in the form of a memory embedded in the image processing device 100 according to the use of stored data, or can be implemented to be attachable to or from the image processing device 100 The form of detached memory. For example, when the data is used to operate the image processing device 100, the data can be stored in a memory embedded in the image processing device 100, and when the data is used for the extended functions of the image processing device 100 Here, the data can be stored in a memory that can be attached to or detached from the image processing device 100. In the case where the memory 110 is implemented as a memory embedded in the image processing device 100, the memory 110 may be at least one of the following memories: volatile memory (for example, dynamic RAM (DRAM) ), static RAM (static RAM, SRAM), synchronous dynamic RAM (synchronous dynamic RAM, SDRAM), etc.) or non-volatile memory (for example, one time programmable ROM (OTPROM), programmable ROM) (Programmable ROM, PROM), erasable and programmable ROM (erasable and programmable ROM, EPROM), electrically erasable and programmable ROM (electrically erasable and programmable ROM, EEPROM), mask ROM (mask ROM) ), flash ROM (flash ROM), flash memory (for example, NAND flash or NOR flash, etc.), hard drive or solid-state drive state drive, SSD)).

Meanwhile, in the case that the memory 110 is implemented as a memory that can be attached to or detached from the image processing device 100, the memory 110 may be a memory card (for example, a compact flash , CF), secure digital (SD), micro secure digital (micro secure digital, Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multimedia card (Multi-media card, MMC), etc.), external memory (for example, USB memory) that can be connected to the universal serial bus (USB) port.

According to the embodiment of the present disclosure, the memory 110 can store at least one program for executing instructions by the processor 120. Here, the instruction may be an instruction for the processor 120 to obtain an output image by applying the image 10 to the learning network.

According to another embodiment of the present disclosure, the memory 110 can store the learning network model according to various embodiments of the present disclosure.

The learning network model according to the embodiment of the present disclosure is a judgment model based on artificial intelligence algorithms and training based on multiple images, and the learning network may be a neural network-based model. The trained judgment model can be designed to simulate human intelligence and decision-making on a computer, and can include multiple network nodes with weights that simulate neurons of a human neural network. Each of the plurality of network nodes can form a connection relationship to simulate the synaptic activity of neurons that transmit and receive signals through the synapse. In addition, the trained judgment model may include, for example, a machine learning model, a neural network model, or a deep learning model developed from a neural network model. In the deep learning model, multiple network nodes can be located in different depths (or layers) from each other and transmit and receive data according to convolutional connections.

As an example, the learning network model may be a convolution neural network (CNN) model trained on images. CNN is a multilayer neural network with a specific connection structure designed for voice processing, image processing, etc. At the same time, learning network models is not limited to CNN. For example, the learning network model can be implemented as at least one deep neural network (DNN) model among the following models: recurrent neural network (RNN) model, long and short-term memory network (Long short term memory network, LSTM) model, gated recurrent unit (GRU) model, or generative adversarial network (GAN) model.

For example, the learning network model can be based on super-resolution GAN (Super-resolution GAN, SRGAN) to restore or convert low-resolution images to high-resolution images. At the same time, the memory 110 according to the embodiment of the present disclosure can store multiple learning network models of the same type or different types. The number and types of learning network models are not limited. However, according to another embodiment of the present disclosure, at least one learning network model according to various embodiments of the present disclosure may be stored in at least one of an external device or an external server.

The processor 120 is electrically connected to the memory 110 and controls the overall operation of the image processing device 100.

According to the embodiment of the present disclosure, the processor 120 can be implemented as a digital signal processor (DSP), a microprocessor, an artificial intelligence (AI) processor, and a timing controller for processing digital signals. , T-CON). However, the processor 120 is not limited to this, and the processor 120 may include a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), One or more of a controller, an application processor (AP), a communication processor (CP), and an advanced RISC machine (Advanced RISC Machine, ARM) processor may be defined by terms. In addition, the processor 120 can be implemented as a system on chip (SoC) or large scale integration (LSI) in which processing algorithms are stored, or as a field programmable gate array (field programmable gate array). , FPGA).

The processor 120 can apply the image 10 as an input to the learning network model and obtain an image with improved, enhanced, optimized, or enhanced image characteristics. Here, the characteristics of the image 10 may refer to at least one of the edge direction, edge intensity, texture, gray value, brightness, contrast, or gamma value of a plurality of pixels included in the image 10. For example, the processor 120 may apply the image to the learning network model and obtain an image in which the edges and textures have been enhanced. Here, the edge of the image may mean an area in which the values of spatially adjacent pixels sharply change. For example, the edge may be an area where the brightness of the image changes sharply from a low value to a high value or sharply changes from a high value to a low value. The texture of an image can be a unique pattern or shape of areas in the image that are regarded as the same. At the same time, the texture of the image can also be composed of fine edges, and therefore the processor 120 can obtain edge components that are equal to or greater than the first threshold intensity (or threshold thickness) and those that are less than the second threshold intensity (or threshold thickness). An image with improved edge components. Here, the first threshold intensity may be a value used to divide the edge component according to the embodiment of the present disclosure, and the second threshold strength may be a value used to divide the texture component according to the embodiment of the present disclosure, And the first threshold intensity and the second threshold intensity may be predetermined values or values set based on the characteristics of the image. However, in the following, for ease of explanation, the above-mentioned characteristics will be referred to as edges and textures.

At the same time, the image processing device 100 according to the embodiment of the disclosure may include multiple learning network models. Each of the multiple learning network models can enhance different characteristics of the image 10. This will be explained in detail with reference to FIG. 3.

FIG. 3 is a diagram showing a first learning network model and a second learning network model according to an embodiment of the disclosure.

3, at operation S310, the processor 120 according to an embodiment of the present disclosure may apply the image 10 as an input to the first learning network model and obtain a first image in which the edge of the image 10 has been improved as an output. At operation S320, the image 10 may also be supplied as an input to the second learning network model and a second image in which the texture of the image 10 has been improved is obtained as an output.

At the same time, the image processing device 100 according to the embodiment of the disclosure can use the first learning network model and the second learning network model based on different artificial intelligence algorithms in parallel. Alternatively, the image 10 can be continuously processed by the first learning network model and the second learning network model. Here, the first learning network model may be a model that is trained by using larger resources than those of the second learning network model. Here, the resource can be various data necessary for training and/or processing the learning network model and can include, for example, whether to implement real-time learning, the amount of learning data, the number of convolutional layers included in the learning network model, and the number of parameters , The capacity of the memory used in the learning network model, the extent to which the GPU is used by the learning network, etc.

For example, the GPU provided in the image processing device 100 may include a texture unit, a special function unit (SFU), an arithmetic logic device, and so on. Here, the texture unit is a resource used to add material or texture to the image 10, and the specific functional unit is a resource used to process complex operations such as square root, reciprocal, and algebraic functions. At the same time, the integer arithmetic logic unit (ALU) is a resource for processing floating point, integer arithmetic, comparison, and data movement. The geometric unit is a resource for calculating the position or viewpoint of an object, the direction of the light source, etc. A raster unit is a resource that projects three-dimensional data on a two-dimensional screen. In this case, the deep learning model can use various resources included in the GPU to perform learning and operation compared to the machine learning model. At the same time, the resources of the image processing device 100 are not limited to the resources of the GPU, and the resources may be resources of various components included in the image processing device 100, such as the storage area of the memory 110, power, etc.

The first learning network model and the second learning network model according to the embodiment of the present disclosure may be different types of learning network models.

As an example, the first learning network model may be one of a deep learning-based model or a machine learning model. The deep learning-based model learns to improve the edges of the image 10 based on multiple images, and the machine learning model is trained by Use multiple pre-learning filters to improve the edges of the image. The second learning network model can be a deep learning model or a machine learning-based model. The deep learning model learns to improve the texture of the image by using multiple layers, and the machine learning-based model is trained by using multiple image-based Pre-learning database (database, DB) and multiple pre-learning filters to improve the texture of the image. Here, the pre-learning DB may be a plurality of filters corresponding to each of the plurality of image patterns, and the second learning network model may recognize the image pattern corresponding to the image block included in the image 10. The texture of the image 10 is optimized by using the filter corresponding to the recognized pattern among the plurality of filters. According to the embodiment of the present disclosure, the first learning network model may be a deep learning model, and the second learning network model may be a machine learning model.

Machine learning models include multiple pre-learning based on various information and data input methods (such as supervised learning, unsupervised learning, and semi-supervised learning). Filter, and the filter to be applied to the image 10 is identified among the plurality of filters.

The deep learning model is a model that performs learning based on a large amount of data and includes multiple hidden layers located between the input layer and the output layer. Therefore, the deep learning model may require additional resources of the image processing device 100 that have more resources than the machine learning model to perform learning and operation.

As another example, the first learning network model and the second learning network model may be models based on the same artificial intelligence algorithm, but have different sizes or configurations. For example, the second learning network model may be a low-complexity model having a size smaller than that of the first learning network model. Here, the size and complexity of the learning network model can be proportional to the number of convolutional layers and the number of parameters constituting the model. In addition, according to the embodiment of the disclosure, the second learning network model may be a deep learning model, and the first learning network model may be a deep learning model that uses fewer convolutional layers than the second learning network model.

As yet another example, each of the first learning network model and the second learning network model may be a machine learning model. For example, the second learning network model may be a low-complexity machine learning model having a size smaller than that of the first learning network model.

At the same time, various embodiments of the present disclosure have been explained based on the assumption that the first learning network model is a model trained by using more resources than the second learning network model, but this is only an example and this disclosure is not Limited to this. For example, the first learning network model and the second learning network model can be models with the same or similar complexity, and the second learning network model can be used by using more resources than the first learning network model. Resources for training models.

At operation S330, the processor 120 according to an embodiment of the present disclosure can recognize the edge area and the texture area included in the image 10. Then, at operation S340, the processor 120 may apply the first weight to the first image and the second weight to the second image based on the information about the edge region and the texture region. As an example, the processor 120 may obtain the first weight corresponding to the edge region and the second weight corresponding to the texture region based on information about the ratio of the edge region and the texture region included in the image 10. For example, if there are more edge areas than texture areas according to the ratio, the processor 120 may apply a larger weight to the first image in which the edge area has been improved than the second image in which the texture has been improved. As another example, if there are more texture regions than the edge regions according to the ratio, the processor 120 may apply a greater weight to the second image in which the texture has been improved than the first image in which the edges have been improved. Then, the processor 120 may obtain the output image 10 based on the first image to which the first weight has been applied and the second image to which the second weight has been applied.

As another example, the first image and the second image obtained from the first learning network model and the second learning network model may be afterimages. Here, the residual image may be an image that only includes residual information in addition to the original image. As an example, the first learning network model can identify the edge region in the image 10 and optimize the identified edge region to obtain the first image. The second learning network model can identify the texture area in the image 10 and optimize the identified texture area to obtain the second image.

Then, the processor 120 can mix the image 10 with the first image and the second image and obtain the output image 20. Here, the blending may be a process of adding the corresponding pixel value of each of the first image and the second image to the value of each pixel included in the image 10. In this case, due to the first image and the second image, the output image 20 may be an image with enhanced edges and textures.

The processor 120 according to the embodiment of the present disclosure can apply the first weight and the second weight to the first image and the second image, respectively, and then mix the image with the image 10, and thus obtain the output image 20.

As another example, the processor 120 may divide the image 10 into multiple regions. Then, the processor 120 can recognize the ratio of the edge area to the texture area of each of the plurality of areas. For the first area with a high proportion of the edge area among the plurality of areas, the processor 120 may set the first weight to a value larger than the second weight. In addition, for the second area with a high proportion of the texture area in the plurality of areas, the processor 120 may set the second weight to a value larger than the first weight.

Then, in operation S340, the processor 120 may mix the first image and the second image to which the weight has been applied with the image 10 and obtain an output image. The image 10 and the output image 20 corresponding to the image 10 can be expressed as Equation 1 below. [Equation 1] Y _res = Y _img + a * Network _ Model 1( Y _img )+b* Network _ Model 2( Y _img )

Here, Y_img means image 10, Network_Model1 (Y_img) means the first image, Network_Model2 (Y_img) means the second image, "a" means the first weight corresponding to the first image, and "b" means The second weight corresponding to the second image.

At the same time, as another example, the processor 120 may apply the image 10 as an input to the third learning network model and obtain the first weight applied to the first image and the second weight applied to the second image. For example, the third learning network model can be trained to recognize the edge regions and texture regions included in the image 10 and enhance the edge regions based on the output of the recognized edge region and the texture region, the characteristics of the image 10, etc. The first weight and the second weight to strengthen the texture area.

FIG. 4 is a diagram showing a scaled-down view according to an embodiment of the present disclosure.

Referring to FIG. 4, at operation S410, the processor 120 may recognize the edge area and the texture area in the input image 10' and obtain a first weight corresponding to the edge area and a second weight corresponding to the texture area. As an example, the processor 120 may apply a guided filter to the input image 10' and identify edge regions and texture regions. The steering filter may be a filter for dividing the image 10 into a base layer and a detail layer. The processor 120 may recognize the edge area based on the base layer and recognize the texture area based on the detail layer.

Then, in operation S420, the processor 120 may scale down the input image 10' and obtain an image 10 with a resolution lower than that of the input image 10'. As an example, the processor 120 may apply sub-sampling to the input image 10' and scale down the resolution of the input image 10' to the target resolution. Here, the target resolution may be a lower resolution than the resolution of the input image 10'. For example, the target resolution may be the resolution of the original image corresponding to the input image 10'. Here, the resolution of the original image can be estimated by a resolution estimation program, or can be identified based on the additional information received with the input image 10', but the resolution and identification of the original image are not limited to this. At the same time, the processor 120 may apply various known scaling methods other than sub-sampling, and thus obtain the image 10 corresponding to the input image 10'.

As an example, if the input image 10' is a UHD image with a resolution of 4K, in order to apply the input image 10' as input to the first learning network model and the second learning network model and obtain the output image 20, it is necessary to The SD image with a resolution of 820×480 is applied to the first learning network model and the second learning network model at least 5.33 times (3840/820) larger line buffer memory. In addition, there is the following problem: as the amount of operations required for the first learning network model to obtain the first image increases, intermediate operations of each of the multiple hidden layers included in the first learning network model are stored As a result, the space of the memory 110 and the required CPU/GPU performance increase exponentially.

Therefore, the processor 120 according to the embodiment of the present disclosure can apply the scaled-down input image 10 to the first learning network model and the second learning network model, so as to reduce the first learning network model and the second learning network model. The amount of operations required in the road model, the storage space of the memory 110, and so on.

At operation S430, when the scaled-down image 10 is input, the first learning network model according to the embodiment of the present disclosure may implement the scale-up that enhances the high-frequency components corresponding to the edges included in the input image 10 And get the first high-resolution image. At the same time, at operation S440, the second learning network model may implement a scale-up that enhances the high-frequency components corresponding to the texture included in the image 10 and obtain a high-resolution second image. Here, the resolution of the first image and the second image may be the same as the resolution of the input image 10'. For example, if the input image 10 is a 4K resolution image and the scaled-down image 10 is a 2K resolution image, the first learning network model and the second learning network model can scale the image 10 And obtain a 4K resolution image as the output of image 10.

In operation S450, the processor 120 according to an embodiment of the present disclosure may mix the scaled-up first image and the second image with the input image 10' and obtain an enhanced edge and texture of the input image 10' High-resolution output image 20. According to the embodiment shown in FIG. 4, the process of acquiring the input image 10' and the output image 20 corresponding to the input image 10' can be expressed as the following equation 2. [Equation 2] Y _res = Y _org + a * Network _ Model 1( DownScaling ( Y _org ))+b* Network _ Model 2( DownScaling ( Y _org ))

Here, Y_org means the input image 10', DownScaling(Y_org) means image 10, Network_Model1(DownScaling(Y_org)) means the first image, Network_Model2(DownScaling(Y_org)) means the second image, and “a” means Refers to the first weight corresponding to the first image, and "b" means the second weight corresponding to the second image.

Fig. 5 is a diagram showing a deep learning model and a machine learning model according to an embodiment of the present disclosure.

5, as described above, the first learning network model can be a deep learning model that learns to enhance the edge of the image 10 by using multiple layers, and the second learning network model can be trained to use multiple layers. A pre-learning filter to enhance the machine learning model of the texture of the image 10.

According to the embodiment of the present disclosure, a configuration in which two convolutional layers and one pooling layer are repeated can be used to model the deep learning model into a deep structure including ten or more layers in total. In addition, the deep learning model can use various types of activation functions (such as identity function, logistic sigmoid function, hyperbolic tangent, tanh), and linear rectification. unit, ReLU) function, Leaky ReLU Function (Leaky ReLU Function, etc.) to perform operations. In addition, in the process of performing convolution, the deep learning model can be sized differently by performing padding, striding, etc. Here, filling means filling in a specific value (for example, pixel value) that is as large as a predetermined size around the received input value. Stride means the shift interval of the weighting matrix when performing convolution. For example, if stride=3, the learning network model can perform convolution on the input value when the weight matrix is shifted by three spaces at once.

According to the embodiment of the present disclosure, the deep learning model can learn to optimize one of the various characteristics of the image 10 that is highly sensitive to the user, and the machine learning model can use multiple pre-learning filters to optimize the image 10 At least one of the remaining characteristics of the system is optimized. For example, it can be assumed that there is a close relationship between the transparency of the edge region (for example, edge direction, edge strength) and the transparency of the image 10 perceived by the user. The image processing device 100 can enhance the edge of the image 10 by using a deep learning model, and as an example of the remaining characteristics, the image processing device 100 can enhance the texture by using a machine learning model. Since the deep learning model learns based on a large amount of data more than the machine learning model and performs iterative operations, it is assumed that the processing result of the deep learning model is better than the processing result of the machine learning model. However, the present disclosure is not necessarily limited to this, and both the first learning network model and the second learning network model can be implemented as a model based on deep learning or as a model based on machine learning. As another example, the first learning network model may be implemented as a machine learning-based model and the second learning network model may be implemented as a deep learning-based model.

In addition, although the various embodiments of the disclosure are explained based on the assumption that the first learning network model strengthens the edge and the second learning network model strengthens the texture, the specific operation of the learning network model is not limited to this. For example, it can be assumed that there is the closest relationship between the processing degree of the noise of the image 10 and the transparency of the image 10 perceived by the user. In this case, the image processing device 100 can perform image processing on the noise of the image 10 by using a deep learning model, and as an example of other image characteristics, the image processing device 100 can enhance the texture by using a machine learning model. As another example, if there is the closest relationship between the processing degree of the brightness of the image 10 and the transparency of the image 10 perceived by the user, the image processing device 100 can perform image processing on the brightness of the image 10 by using a deep learning model. Processing, and as an example of other image characteristics, the image processing device 100 can filter noise by using a machine learning model.

Fig. 6 is a diagram showing a first learning network model and a second learning network model according to an embodiment of the disclosure.

The processor 120 according to the embodiment of the present disclosure may scale down the input image 10' at operation S610 and obtain a relatively low-resolution image 10, and obtain area detection information at operation S620, which is based on The image 10' is scaled down to identify the edge area and texture area. According to the embodiment shown in FIG. 5, the processor 120 can recognize the edge area and the texture area included in the input image 10' of the original image resolution. 6, the processor 120 can identify the edge area and texture area included in the image 10 in which the resolution of the input image 10' has been scaled down to the target resolution.

Then, the processor 120 according to the embodiment of the disclosure can provide the area detection information and the image 10 to the first learning network model and the second learning network model, respectively.

In operation S630, the first learning network model according to the embodiment of the present disclosure may implement a scaled enlargement that only enhances the edge area of the image 10 based on the area detection information. In operation S640, the second learning network model may implement a scaled enlargement that only enhances the texture area of the image 10 based on the area detection information.

As another example, the processor 120 may provide an image including only some of the pixel information included in the image 10 to the learning network model based on the area detection information. Since the processor 120 only provides some information included in the image 10 instead of the image 10 to the learning network model, the amount of operations performed by the learning network model can be reduced. For example, the processor 120 may provide an image including only pixel information corresponding to the edge region to the first learning network model based on the region detection information and provide an image including only pixel information corresponding to the texture region Go to the second learning network model.

Then, the first learning network model can scale up the edge area and obtain the first image, and the second learning network model can scale up the texture area and obtain the second image.

Next, at operation S650, the processor 120 may add the first image and the second image to the input image 10' and obtain the output image 20.

FIG. 7 is a diagram showing a first learning network model and a second learning network model according to another embodiment of the present disclosure.

Referring to FIG. 7, at operation S710, the processor 120 according to an embodiment of the present disclosure may apply the input image 10' as an input to the first learning network model and obtain the first image. As an example, since the first learning network model implements the enhanced scaling up of the high-frequency components corresponding to the edges included in the input image 10' at operation S710, the processor 120 can obtain a high-resolution first image. Here, the first image may be an afterimage. The residual image may be an image that only includes residual information in addition to the original image. The residual information can indicate the difference between each pixel or pixel group of the original image and the high-resolution image.

In addition, in operation S720, the processor 120 according to an embodiment of the present disclosure may apply the input image 10' as an input to the second learning network model and obtain the second image. As an example, since the second learning network model implements a scaled up that strengthens the high-frequency components corresponding to the texture included in the input image 10', the processor 120 can obtain a high-resolution second image. Here, the second image may be an afterimage. The residual information can indicate the difference between each pixel or pixel group of the original image and the high-resolution image.

According to the embodiment of the present disclosure, the first learning network model and the second learning network model respectively implement an enhanced scaling up of at least one of the characteristics of the input image 10', and therefore are compared with the input image 10' , The first image and the second image have high resolution. For example, if the resolution of the input image 10' is 2K, the resolution of the first image and the second image may be 4K, and if the resolution of the input image 10' is 4K, the resolution of the first image and the second image The resolution can be 8K.

At operation S730, the processor 120 according to an embodiment of the present disclosure may scale up the input image 10' and obtain a third image. According to an embodiment of the present disclosure, the image processing device 100 may include a separate processor that scales up the input image 10', and the processor 120 may scale up the input image 10' and obtain a high-resolution third image. For example, the processor 120 may use bilinear interpolation, bicubic interpolation, cubic spline interpolation, or Lanczos interpolation. , Edge directed interpolation (EDI), etc., implement proportional enlargement of the input image 10'. Meanwhile, this is only an example and the processor 120 can scale up the input image 10' based on various scale up (or super resolution) methods.

As another example, the processor 120 may apply the input image 10' as an input to the third learning network model and obtain a high-resolution third image corresponding to the input image 10'. Here, the third learning network model may be a model based on deep learning or a model based on machine learning. According to the embodiment of the present disclosure, if the resolution of the input image 10' is 4K, the resolution of the third image may be 8K. In addition, according to the embodiment of the disclosure, the resolution of the first image to the third image may be the same.

Then, in operation S740, the processor 120 may mix the first image to the third image and obtain the output image 20.

The processor 120 according to the embodiment of the present disclosure can mix the first afterimage, the second afterimage, and the third afterimage to obtain an output image. The first afterimage is scaled up by enhancing the edges in the input image 10' The input image 10', the second afterimage enlarges the input image 10' proportionally by enhancing the texture in the input image 10', and the third afterimage enlarges the input image 10' proportionally. Here, the processor 120 can recognize the edge area in the input image 10' and apply the recognized edge area to the first learning network model, and strengthen the edge area, and thus obtain the first afterimage that is scaled up. In addition, the processor 120 can recognize the texture area in the input image 10' and apply the recognized texture area to the second learning network model, and strengthen the texture area, and thus obtain a second afterimage that is scaled up. At the same time, this is only an example and the configuration and operation are not limited to this. For example, the processor 120 can apply the input image 10' to the first learning network model and the second learning network model. Then, the first learning network model can identify the edge area based on the edge characteristics of the various image characteristics of the input image 10' and enhance the identified edge area, and thus obtain the first afterimage with a high resolution that is scaled up. The second learning network model can identify the texture area based on the texture characteristics of the various image characteristics of the input image 10' and enhance the recognized texture area, and thus obtain the second afterimage with a high resolution that is scaled up.

In addition, the processor 120 according to the embodiment of the present disclosure can scale up the input image 10' and obtain a high-resolution third image. Here, the third image may be an image obtained by scaling up the original image instead of the afterimage.

According to an embodiment of the present disclosure, the processor 120 can mix the first image to the third image and obtain an output image 20 with a higher resolution than the input image 10'. Here, the output image 20 may be a scaled-up image in which the edge area and texture area have been enhanced, rather than an image in which only the resolution has been scaled-up. At the same time, this is only an example and the processor 120 can obtain multiple afterimages in which various image characteristics of the input image 10' have been enhanced, and mix the third image of the input image 10' that is scaled up with multiple afterimages. And the output image 20 is obtained based on the result of mixing the images.

Fig. 8 is a diagram schematically showing the operation of the second learning network model according to an embodiment of the present disclosure.

The processor 120 according to the embodiment of the present disclosure can apply the image 10 as an input to the second learning network model and obtain the second image in which the texture has been enhanced.

The second learning network model according to an embodiment of the present disclosure can store multiple filters corresponding to each of multiple image patterns. Here, the multiple image patterns can be classified according to the characteristics of the image blocks. For example, the first image pattern may be an image pattern with a large number of lines in the horizontal direction, and the second image pattern may be an image pattern with a large number of lines in the rotation direction. The plurality of filters may be filters that are pre-learned by artificial intelligence algorithms.

In addition, the second learning network model according to the embodiment of the disclosure can read image blocks of a predetermined size from the image 10. Here, the image block may be a group of multiple pixels including the target pixel and multiple surrounding pixels included in the image 10. As an example, the second learning network model can read the first image block with a size of 3×3 pixels on the upper left end of the image 10 and perform image processing on the first image block. Then, the second learning network model can scan from the upper left end of the image 10 to the right by as much as the unit pixel, and read the second image block with the size of 3×3 pixels and execute the second image block Image processing. By scanning the pixel blocks, the second learning network model can perform image processing on the image 10. At the same time, the second learning network model can autonomously read the first image block to the nth image block from the image 10, and the processor 120 can sequentially apply the first image block to the nth image block as input to The second learns the network model and performs image processing on the image 10.

To detect high frequency components in the image block, the second learning network model can apply a filter of a predetermined size to the image block. As an example, the second learning network model can apply a 3×3 Laplacian filter 810 corresponding to the size of the image block to the image block, thereby eliminating low-frequency components in the image 10 And detect high frequency components. As another example, the second learning network model can be applied to images by applying various types of filters (such as Sobel, Prewitt, Robert, Canny, etc.) Block to obtain the high frequency components of the image 10.

Then, the second learning network model can calculate the gradient vector 820 based on the high frequency components obtained from the image block. Specifically, the second learning network model can calculate the horizontal gradient and the vertical gradient, and calculate the gradient vector based on the horizontal gradient and the vertical gradient. Here, the gradient vector may express the amount of change based on each pixel relative to a pixel located in a predetermined direction. In addition, the second learning network model can classify the image block into one of a plurality of image patterns based on the directionality of the gradient vector.

Next, the second learning network model can search for a filter to be applied to the high frequency components detected from the image 10 (perform a filter search) 830 by using the index matrix 850. Specifically, the second learning network model can identify index information indicating the pattern of the image block based on the index matrix, and search 830 for the filter corresponding to the index information. For example, if the index information corresponding to the image block is recognized as the index information 32 among the index information from 1 to 32 indicating the pattern of the image block, the second learning network model can be derived from the plurality of filters Get the filter mapped to index information 32 in. At the same time, the above specific index values are only examples and the index information can be reduced or increased according to the number of filters. In addition, the index information can be expressed in various ways other than integers.

After that, the second learning network model may obtain at least one filter of the plurality of filters included in the filter database (DB) 860 based on the search result, and apply 840 the at least one filter to the image Block to obtain the second image. As an example, the second learning network model can identify the filter corresponding to the pattern of the image block among the plurality of filters based on the search result and apply the identified filter to the image block, thereby obtaining the texture area The second image that is scaled up.

Here, the filters included in the filter database 860 can be obtained according to the result of learning the relationship between the low-resolution image block and the high-resolution image block by an artificial intelligence algorithm. For example, the second learning network model can learn a low-resolution first image block and a high-resolution second image in which the texture area of the first image block has been scaled up by an artificial intelligence algorithm The relationship between the blocks is recognized, and the filter to be applied to the first image block is identified and the recognized filter is stored in the filter database 860. However, this is only an example and the present disclosure is not limited to this. For example, the second learning network model can identify a filter that enhances at least one of the various characteristics of the image block by using the result of the learning of an artificial intelligence algorithm, and store the identified filter in Filter database 860.

FIG. 9 is a diagram showing index information according to an embodiment of the present disclosure.

The processor 120 according to the embodiment of the present disclosure can accumulate the index information of the image pattern corresponding to each image block classified and obtain the accumulation result. 9, the processor 120 may obtain index information corresponding to the image pattern of the image block among the index information indicating the image pattern. Then, the processor 120 may accumulate the index information of each of the plurality of image blocks included in the image 10, and thus obtain the accumulation result, as shown in FIG. 9.

The processor 120 can analyze the accumulation result and recognize the image 10 as one of a natural image or a graphic image. For example, if the number of image blocks that do not include patterns (or do not display directionality) in the image blocks included in the image 10 is equal to or greater than the threshold, based on the cumulative result, the processor 120 may 10Identified as a graphic image. As another example, if the number of image blocks that do not include a pattern among the image blocks included in the image 10 is less than the threshold, the processor 120 may recognize the image 10 as a natural image based on the accumulation result. As yet another example, if the number of image blocks having a pattern in the vertical direction or a pattern in the horizontal direction is equal to or greater than the threshold, the processor 120 may recognize the image 10 as a natural image based on the accumulation result. At the same time, the identification and classification of images are only exemplary, and thresholds can be specified according to the manufacturer's purpose, the user's setting, and so on.

As another example, the processor 120 may calculate the number and proportion of specific index information based on the accumulation result, and identify the type of the image 10 as a natural image or a graphic image based on the accumulation result. For example, the processor 120 may calculate at least three characteristics based on the accumulated result.

If the specific index information in the index information is information indicating an image block whose pattern is not recognized (or does not show directionality), the processor 120 may calculate the ratio of the index information from the accumulation result. In the following, image blocks of unrecognized patterns are generally referred to as image blocks including flat areas. The proportion of the image block including the flat area in the entire image block can be calculated based on Equation 3 below. [Equation 3]

Here, Histogram[i] means the number of image blocks with index information "i" identified based on the accumulation result. In addition, based on the assumption that the index information indicating the image block including the flat area is 32, Histogram [32] means the number of image blocks with the index information 32, and P1 means that the image block including the flat area is in the entire image The proportion in the block.

If the image block includes a pattern, the processor 120 can identify whether the pattern is located in the central area inside the image block based on the index information. As an example, compared to image blocks with index information of 1 to 12 and 17 to 31, the pattern of image blocks with index information of 13 to 16 may be located in the central area inside the block. Hereinafter, the image block in which the image pattern is located in the central area inside the image block is generally referred to as a centered image block. Then, the processor 120 may calculate the proportion of the image blocks in the center distribution based on the accumulation result and the following Equation 4. [Equation 4]

），以辨識除包括平坦區域的影像區塊之外的包括圖案的影像區塊的數目。另外，處理器120可計算居中分佈的影像區塊的數目（

）。同時，具有索引資訊13至15的影像區塊僅為其中圖案位於影像區塊內部的中心區域的情形的實例且本揭露未必僅限於此。作為另一實例，可基於索引資訊11至17的數目來計算P2。Here, the processor 120 may calculate the number of image blocks with index information 1 to 31 (

) To identify the number of image blocks including patterns other than image blocks including flat areas. In addition, the processor 120 may calculate the number of image blocks distributed in the center (

). Meanwhile, the image block with index information 13 to 15 is only an example of the case where the pattern is located in the central area inside the image block and the present disclosure is not necessarily limited to this. As another example, P2 can be calculated based on the number of index information 11-17.

Then, the processor 120 may obtain the average index information of the image 10 based on the index information of each of the plurality of image blocks included in the image 10. According to an embodiment of the present disclosure, the processor 120 may calculate the average index information based on Equation 5 below. [Equation 5]

Here, "i" means index information, Histogram[i] means the number of image blocks corresponding to index information i, and P3 means average index information.

The processor 120 according to an embodiment of the present disclosure can calculate the "Y" value based on the following Equation 6. [Equation 6] Y = W 1* P 1+ W 2* P 2 + W 3* P 3+ Bias

Here, P1 means the proportion of image blocks including flat areas, P2 means the proportion of image blocks distributed in the center, and P3 means average index information. In addition, W1, W2, W3, and Bias mean parameters learned in advance by using an artificial intelligence algorithm model.

If the Y value exceeds 0, the processor 120 according to the embodiment of the present disclosure can recognize the image 10 as a graphic image, and if the "Y" value is equal to or less than 0, the processor 120 can recognize the image 10 as a natural image.

Then, the processor 120 may adjust the first weight and the second weight respectively corresponding to the first image and the second image based on the identification result. As an example, if the image 10 is recognized as a natural image, the processor 120 may increase at least one of the first weight corresponding to the first image or the second weight corresponding to the second image. In addition, the processor 120 can increase at least one of the parameters “a” or “b” in Equation 1 and Equation 2. At the same time, if the image 10 is a natural image, the processor 120 can obtain a high-resolution image, because the first image with improved edges or the second image with improved texture is added to the image 10 or the input image 10 ', therefore, the transparency of the high-resolution image has been improved, and therefore the processor 120 may increase at least one of the first weight or the second weight.

As another example, if the image 10 is recognized as a graphic image, the processor 120 may reduce at least one of the first weight corresponding to the first image or the second weight corresponding to the second image. In addition, the processor 120 can reduce at least one of the parameters “a” or “b” in Equation 1 and Equation 2. At the same time, if the image 10 is a graphic image, since the first image in which the edge has been enhanced or the second image in which the texture has been enhanced is added to the image 10 or the input image 10', the processor 120 can obtain the distortion Therefore, the processor 120 can reduce at least one of the first weight or the second weight, and thus minimize the occurrence of distortion.

Here, the graphic image may be an image manipulated to an image of the actual world, or an image newly created by using a computer, an imaging device, or the like. For example, graphic images may include exemplified images, computer graphic (CG) images, animated images, etc., generated by using known software. Natural images can be other images except graphic images. For example, natural images may include real-world images, landscape images, portrait images, etc. taken by a photographing device.

FIG. 10 is a diagram illustrating a method of obtaining a final output image according to an embodiment of the disclosure.

According to the embodiment of the present disclosure, in the case that the final output image 30' (ie, the display image) is an image with a resolution larger than that of the output image 30, in operation S350, the processor 120 may scale up the output Image 30 and obtain the final output image 30'. For example, if the output image 30 is a 4K UHD image and the final output image is an 8K image, the processor 120 may scale the output image 30 to an 8K UHD image and obtain the final output image 30'. At the same time, according to another embodiment of the present disclosure, a separate processor for performing the scaling of the output image 30 may be provided in the image processing device 100. For example, the image processing device 100 may include a first processor and a second processor, and by using the first processor to obtain the output image 30 in which the edges and textures have been enhanced, and by using the second processor to obtain the high The resolution of the final output image 30', and the final output image 30' enlarges the resolution of the output image 30.

At the same time, each of the first learning network model and the second learning network model according to various embodiments of the present disclosure can be an on-device machine in which the image processing device 100 performs learning by itself without relying on external devices. Learning model (on-device machine learning model). At the same time, this is only an example and some learning network models can be implemented based on operations on the device, and other learning network models can be implemented based on operations on an external server.

FIG. 11 is a block diagram showing the detailed configuration of the image processing apparatus shown in FIG. 2.

According to FIG. 11, the image processing device 100 includes a memory 110, a processor 120, an input 130, a display 140, an output 150 and a user interface 160. Meanwhile, when explaining the components shown in FIG. 11, redundant explanation of components similar to the components shown in FIG. 2 will be omitted.

According to the embodiment of the present disclosure, the memory 110 may be implemented as a single memory that stores data generated from various operations according to the present disclosure.

The memory 110 may be implemented to include a first memory to a third memory.

The first memory can store at least a part of the image input through the input 130. Specifically, the first memory can store at least some areas of the input image frame. In this configuration, the at least some areas may be areas necessary for performing image processing according to the embodiment of the disclosure. At the same time, according to the embodiment of the present disclosure, the first memory can be implemented as N rows of memory. For example, the N-row memory may be a memory having as much capacity as 17 rows in the horizontal direction, but the memory is not limited to this. For example, in the case of inputting a full HD video with 1080 pixels (resolution of 1,920chines), only the 17-line image area of the full HD video can be stored in the first memory. As described above, the reason why the first memory is implemented as N rows of memory and only stores some areas of the input image frame for image processing is that the memory capacity of the first memory is limited due to hardware limitations. Meanwhile, the second memory may be a memory area allocated to the learning network model in the entire area of the memory 110.

The third memory is a memory in which the first image, the second image and the output image are stored, and according to various embodiments of the present disclosure, the third memory can be implemented into memories of various sizes. According to the embodiment of the present disclosure, the processor 120 applies the image 10 scaled down to the input image 10' to the first learning network model and the second learning network model, and therefore stores from the first learning network model and the second learning network model. The size of the third memory of the first image and the second image acquired by the second learning network model can be implemented to be the same or similar to the size of the first memory.

The input 130 may be a communication interface (for example, a wired Ethernet interface or a wireless communication interface) that receives various types of content (for example, an image signal from an image source). For example, the input 130 can be from an external device (eg, a source device), an external storage medium (eg, USB), or an external device via one or more networks such as the Internet by using the following communication methods. Servers (for example, webhard), etc. receive video signals by streaming or downloading methods: AP-based Wi-Fi (Wireless Local Area Network (LAN) network), Bluetooth , Zigbee, wired/wireless local area network (LAN), wide area network (WAN), Ethernet, long-term evolution (Long Trem Evolution, LTE), 5th generation mobile communication technology ( 5th-generation, 5G), Institute of Electrical and Electronic Engineers (IEEE) 1394, High Definition Multimedia Interface (HDMI), Mobile High-Definition Link (Mobile High-Definition Link, MHL), Universal Serial Bus (USB), Display Port (DP), Thunderbolt (Thunderbolt), Video Graphic Array (VGA) Port, Red Green Blue (RGB) Port, D-Ultra Small size (D-subminiature, D-SUB), digital visual interface (Digital Visual Interface, DVI), etc. Specifically, the 5G communication system uses ultra-high frequency (millimeter wave (mmWave)) frequency bands (for example, millimeter wave frequency bands such as 26, 28, 38, and 60 gigahertz bands) for communication, and the image processing device 100 It can transmit or receive 4K and 8K UHD images in a streaming environment.

Here, the image signal may be a digital signal, but the image signal is not limited to this.

The display 140 may be implemented in various forms such as: liquid crystal display (LCD), organic light-emitting diode (OLED), light-emitting diode (ED), micro LED, Quantum dot light-emitting diode (QLED), liquid crystal on silicon (LCoS), digital light processing (DLP) and quantum dot (QD) display panels . Specifically, the processor 120 according to the embodiment of the present disclosure can control the display 140 to display the output image 30 or the final output image 30'. Here, the final output image 30' may include 4K or 8K real-time UHD images, streaming images, etc.

The output unit 150 outputs an acoustic signal.

For example, the output unit 150 can convert the digital audio signal processed at the processor 120 into an analog audio signal, amplify the signal, and output the signal. For example, the output device 150 may include at least one speaker unit, a digital-to-analog (D/A) converter, an audio amplifier, etc., and the output device 150 may output at least one channel. According to the embodiment of the present disclosure, the output unit 150 can be implemented to output various multi-channel acoustic signals. In this case, the processor 120 can control the output unit 150 to perform enhancement processing on the audio signal input to correspond to the enhancement processing of the input image, and output the signal. For example, the processor 120 may convert the input two-channel sound signal into a virtual multi-channel (for example, 5.1 channel) sound signal, or recognize the location where the image processing device 100 is placed in the environment of a room or building and convert The signal processing is a stereo sound signal optimized for the space, or an sound signal optimized according to the type of input image (for example, the genre of the content) is provided. At the same time, the user interface 160 can be implemented as a button, a touchpad, a mouse, and a keyboard, or as a touch screen, which can receive user input to perform remote control of both the above-mentioned display function and manipulation input function. Device. The remote control transceiver can receive a remote control signal from an external remote control device or transmit a remote control signal to the external remote control device by at least one communication method of infrared communication, Bluetooth communication or Wi-Fi communication. At the same time, although not shown in FIG. 9, according to the embodiments of the present disclosure, a free filter that removes noise from the input image can be applied before the image processing. For example, it is possible to remove obvious noise by applying a smoothing filter (such as a Gaussian filter, a steering filter, etc.) that filters the input image by comparing images of predetermined waveguides.

FIG. 12 is a block diagram showing the configuration of an image processing device for learning and using a learning network model according to an embodiment of the disclosure.

Referring to FIG. 12, the processor 1200 may include at least one of a learning part 1210 or a discrimination part 1220. The processor 1200 in FIG. 12 may correspond to the processor 120 of the image processing device 100 or the processor of the data learning server in FIG. 2.

The processor 1200 of the image processing device 100 for learning and using the first learning network model and the second learning network model may include at least one of a learning component 1210 or a distinguishing component 1220.

The learning component 1210 according to the embodiment of the present disclosure can obtain an image in which the image characteristics of the image 10 have been enhanced, and obtain an output image based on the image 10 and the images in which the image characteristics of the image 10 have been enhanced. Next, the learning component 1210 can generate or train a standard discrimination model for minimizing the distortion of the image 10 and obtaining a high-resolution scaled-up image corresponding to the image 10. In addition, the learning component 1210 can generate a discrimination model with judgment criteria by using the collected learning data.

As an example, the learning component 1210 may generate, train or update a learning network model so that at least one of the edge area or texture area of the output image 30 is enhanced to be more than the edge area or texture area of the input image 10'.

The discrimination component 1220 may use predetermined data (for example, an input image) as input data of the trained discrimination model, and thus estimate the object used for discrimination or the conditions included in the predetermined data.

At least a part of the learning component 1210 and at least a part of the identification component 1220 can be implemented as a software module or manufactured in the form of at least one hardware chip, and installed on the image processing device. For example, at least one of the learning component 1210 or the discrimination component 1220 may be manufactured in the form of a hardware chip dedicated to artificial intelligence (AI), or manufactured into a traditional general-purpose processor (for example, a CPU or an application processor) Or a part of a dedicated graphics processor (for example, GPU), and is installed on the above-mentioned various types of image processing devices or object recognition devices. Here, the hardware chip dedicated to artificial intelligence is a dedicated processor specifically used in probabilistic calculations and has higher parallel processing performance than traditional general-purpose processors, and can quickly process operations in the field of artificial intelligence (such as machine learning) . In the case where the learning component 1210 and the identification component 1220 are implemented as one or more software modules (or program modules including instructions), the software modules can be stored in a non-transitory computer readable medium (non-transitory computer readable medium). medium). In this case, the operating system (OS) or a specific application can provide software modules. Alternatively, part of the software module can be provided by the operating system (OS), and other parts can be provided by a specific application.

In this case, the learning component 1210 and the discrimination component 1220 can be installed on one image processing device, or can be installed on separate image processing devices. For example, one of the learning component 1210 and the identification component 1220 may be included in the image processing device 100, and the other may be included in an external server. In addition, the learning component 1210 and the identification component 1220 may be connected in a wired or wireless manner, or may be a larger software module or a separate software module for applications. The model information constructed by the learning component 1210 can be provided to the identifying component 1220, and the data input to the identifying component 1220 can be provided to the learning component 1210 as additional learning data.

FIG. 13 is a flowchart showing an image processing method according to an embodiment of the disclosure.

According to the image processing method shown in FIG. 13, first, at operation S1310, the image is applied to the first learning network model and the first image in which the edge of the image has been enhanced is obtained.

Then, in operation S1320, the image is applied to the second learning network model and a second image in which the texture of the image has been enhanced is obtained.

Next, in operation S1330, the edge area and the texture area included in the image are identified, and based on the information about the edge area and the texture area, the first weight is applied to the first image and the second weight is applied to the second image , And get the output image.

Here, the first learning network model and the second learning network model may be different types of learning network models.

The first learning network model according to an embodiment of the present disclosure may be a deep learning model that learns to enhance the edge of an image by using multiple layers or a machine trained to enhance the edge of an image by using multiple pre-learning filters One of the learning models.

In addition, the second learning network model according to the embodiment of the present disclosure can be a deep learning model that learns to optimize the texture of an image by using multiple layers, or can be trained to use multiple pre-learning filters. One of the machine learning models that optimize the texture of the image.

In addition, the operation S1330 of obtaining the output image may include the following steps: obtaining a first weight corresponding to the edge area and a second weight corresponding to the texture area based on the ratio information of the edge area and the texture area.

In addition, the image processing method according to the embodiment of the disclosure may include the steps of scaling down the input image and obtaining an image with a resolution lower than that of the input image. At the same time, the first learning network model can obtain the first image by implementing a scaled enlargement that enhances the edges of the image, and the second learning network model can obtain the first image by implementing a scaled enlargement that enhances the texture of the image. Acquire the second image.

In addition, the image processing method according to the embodiment of the present disclosure may include the following steps: acquiring area detection information, identifying edge areas and texture areas based on the scaled-down image, and providing the area detection information and images to The first learning network model and the second learning network model.

Here, the step of providing the area detection information and images to the first learning network model and the second learning network model respectively may include the following steps: based on the area detection information, only the pixel information corresponding to the edge area will be included. The image is provided to the first learning network model and the image including only the pixel information corresponding to the texture area is provided to the second learning network model. At the same time, the first learning network model can obtain the first image by scaling up the edge area, and the second learning network model can obtain the second image by scaling the texture area.

In addition, the first image and the second image according to the embodiment of the present disclosure may be a first afterimage and a second afterimage, respectively. In addition, in the operation S1330 of obtaining the output image, the first weight may be applied to the first afterimage and the second weight may be applied to the second afterimage, and then the afterimage and the image may be mixed to obtain the output image.

In addition, the second learning network model may be a model that stores multiple filters corresponding to each of the multiple image patterns and classifies each of the image blocks included in the image into the multiple One of the image patterns, and at least one filter corresponding to the classified image pattern among the plurality of filters is applied to the image block and provides a second image.

In addition, the operation S1330 of obtaining an output image according to an embodiment of the present disclosure may include the following steps: accumulating index information of the image pattern corresponding to each of the classified image blocks and identifying the type of the image based on the accumulation result ( For example, one of a natural image or a graphic image), and the weight is adjusted based on the recognition result.

Here, the step of adjusting the weight may include the following steps: based on the image being identified as a natural image, increasing at least one of a first weight corresponding to the first image or a second weight corresponding to the second image, and based on the image It is identified as a graphic image, and at least one of the first weight or the second weight is reduced.

The output image may be a 4K ultra-high definition (UHD) image, and the image processing method according to an embodiment of the disclosure may include a step of scaling the output image to an 8K UHD image.

At the same time, the various embodiments of the present disclosure can be applied to all electronic devices capable of performing image processing (such as image receiving devices (such as set top boxes and image processing devices)) and image processing devices.

In addition, the various embodiments described above can be implemented by using a recording medium read by a computer or a computer-like device using software, hardware, or a combination thereof. In some cases, the embodiments described in this specification may be implemented as the processor 120 itself. According to the software implementation, the embodiments such as procedures and functions described in this specification can be implemented as separate software modules. Each of the software modules can perform one or more of the functions and operations described in this specification.

At the same time, computer instructions used to implement the processing operations of the sound output device 100 according to the various embodiments of the present disclosure can be stored in a non-transitory computer readable medium. When the computer-readable instructions stored in such a non-transitory computer-readable medium are executed by the processor of the specific device, the processing operations at the sound output device 100 according to the various embodiments described above are executed by the specific device.

Non-transitory computer-readable media refers to media that store data semi-permanently instead of storing data in a short period of time (for example, temporary memory, cache memory, and memory) and can be machine-readable. As a specific example of a non-transitory computer-readable medium, it can be compact disc (CD), digital versatile disk (DVD), hard disk, Blu-ray disc, USB, memory card, ROM, etc.

Although the embodiments of the present disclosure have been shown and described, the present disclosure is not limited to the above-mentioned specific embodiments, and it is obvious that you should be familiar with the present disclosure without departing from the gist of the present disclosure required by the scope of the appended application. Those skilled in the art can make various modifications. In addition, this modification is not intended to be explained separately from the technical ideas or prospects of this disclosure.

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32: index information 10: Image/output image 10’: Input image/video 20, 30: output image 30’: Final output image 100: image processing device/sound output device 110: memory 120, 1200: processor 130: Input 140: display 150: output device 160: User Interface 810: Laplacian filter 820: gradient vector 830: Filter/Search 840: application 850: index matrix 860: Filter Database (DB) 1210: learning parts 1220: Identify parts S310, S320, S330, S340, S350, S410, S420, S430, S440, S450, S610, S620, S630, S640, S650, S710, S720, S730, S740, S1310, S1320, S1330: Operation

FIG. 1 is a diagram showing an implementation example of an image processing device according to an embodiment of the disclosure.

FIG. 2 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the disclosure.

FIG. 3 is a diagram showing a first learning network model and a second learning network model according to an embodiment of the disclosure.

FIG. 4 is a diagram showing a scaled-down view according to an embodiment of the present disclosure.

Fig. 5 is a diagram showing a deep learning model and a machine learning model according to an embodiment of the present disclosure.

FIG. 6 is a diagram showing a first learning network model and a second learning network model according to another embodiment of the present disclosure.

FIG. 7 is a diagram showing a first learning network model and a second learning network model according to another embodiment of the present disclosure.

Fig. 8 is a diagram schematically showing the operation of the second learning network model according to an embodiment of the present disclosure.

FIG. 9 is a diagram showing index information according to an embodiment of the present disclosure.

FIG. 10 is a diagram illustrating a method of obtaining a final output image according to an embodiment of the disclosure.

FIG. 11 is a block diagram showing the detailed configuration of the image processing apparatus shown in FIG. 2.

FIG. 12 is a block diagram showing the configuration of an image processing device for learning and using a learning network model according to an embodiment of the disclosure.

FIG. 13 is a flowchart showing an image processing method according to an embodiment of the disclosure.

100: image processing device/sound output device

110: memory

120: processor

Claims (15)

An image processing device, including: Memory, storing computer readable instructions; and The processor is configured to execute the computer-readable instructions to: Apply the input image as the first input to the first learning network model, Acquiring a first image from the first learning network model, the first image including an enhanced edge optimized based on the edge of the input image, Applying the input image as the second input to the second learning network model, Acquiring a second image from the second learning network model, the second image including an enhanced texture optimized based on the texture of the input image, Identifying the edge area of the edge included in the input image, Identifying the texture area of the texture included in the input image, Applying a first weight to the first image based on the edge area, Applying a second weight to the second image based on the texture area, and Based on the first weight applied to the first image and the second weight applied to the second image, an optimized output image is obtained from the input image. The image processing device according to claim 1, wherein the first type of the first learning network model is different from the second type of the second learning network model. The image processing device according to claim 1, wherein the first learning network model is a deep learning model that optimizes the edge of the input image by using multiple layers or is trained to borrow One of the machine learning models used to optimize the edge of the input image by using a plurality of pre-learning filters. The image processing device according to claim 1, wherein the second learning network model is a deep learning model that optimizes the texture of the input image by using multiple layers, or by using multiple layers One of the machine learning models in which a pre-learning filter is used to optimize the texture of the input image. The image processing device according to claim 1, wherein the processor that executes the computer-readable instructions is further configured to: The first weight and the second weight are obtained based on the ratio information of the edge area in the input image and the texture area in the input image. The image processing device according to claim 1, wherein the processor that executes the computer-readable instructions is further configured to: Scaling down the input image to obtain a scaled down image with a resolution smaller than that of the input image, Applying the scaled down image as the first input to the first learning network model, Acquiring the first image with the enhanced edge from the first learning network model that scales up the scaled down image, Applying the scaled down image as the second input to the second learning network model, and Acquiring the second image with the enhanced texture from the second learning network model that scales up the scaled-down image. The image processing device according to claim 1, wherein the processor that executes the computer-readable instructions is further configured to: Acquiring first area detection information for identifying the edge area of the input image and second area detection information for identifying the texture area of the input image, Applying the input image and the first region detection information as the first input to the first learning network model, and Applying the second area detection information of the input image as the second input to the second learning network model. The image processing device according to claim 7, wherein the first learning network model obtains the first image by scaling up the edge area, and The second learning network model obtains the second image by scaling up the texture area. The image processing device according to claim 1, wherein the second learning network model is a model: the model stores a plurality of filters corresponding to a plurality of image patterns, and converts the images included in the input image Each image block in the block is classified into an image pattern in the plurality of image patterns, and at least one filter of the plurality of filters corresponding to the image pattern is applied to the image block . The image processing device according to claim 1, wherein the first image is a first afterimage and the second image is a second afterimage, and The processor that executes the computer-readable instructions is further configured to: Applying the first weight to the first afterimage based on the edge region, Applying the second weight to the second afterimage based on the texture area, and The first afterimage, the second afterimage, and the input image are mixed to obtain the output image. The image processing device according to claim 10, wherein the processor that executes the computer-readable instructions is further configured to: Accumulate index information of the image pattern corresponding to each of the image blocks of the input image, Identifying the input image as one of a natural image or a graphic image based on the index information, and The first weight and the second weight are adjusted based on a result of identifying the input image as one of the natural image or the graphic image. The image processing device according to claim 11, wherein the processor that executes the computer-readable instructions is further configured to: Based on the input image being identified as the natural image, increasing at least one of the first weight or the second weight, and Based on the input image being recognized as the graphic image, reducing at least one of the first weight or the second weight. An image processing method of an image processing device includes: Applying the input image as the first input to the first learning network model; Acquiring a first image from the first learning network model, the first image including an enhanced edge optimized based on the edge of the input image; Applying the input image as a second input to a second learning network model; Acquiring a second image from the second learning network model, the second image including an enhanced texture optimized based on the texture of the input image; Identifying the edge area of the edge included in the input image; Identifying the texture area included in the input image; Applying a first weight to the first image based on the edge region; Applying a second weight to the second image based on the texture area; and Based on the first weight applied to the first image and the second weight applied to the second image, an optimized output image is obtained from the input image. The image processing method according to claim 13, wherein the first learning network model is a deep learning model that optimizes the edge of the input image by using multiple layers or is trained to borrow One of the machine learning models used to optimize the edge of the input image by using a plurality of pre-learning filters. The image processing method according to claim 13, wherein the second learning network model is a deep learning model that optimizes the texture of the input image by using multiple layers or is trained to borrow One of the machine learning models that uses a plurality of pre-learning filters to optimize the texture of the input image.

Images