KR102246110B1 - Display apparatus and image processing method thereof - Google Patents

Abstract

An image processing apparatus is disclosed. It includes a memory storing at least one instruction and a processor electrically connected to the memory, and the processor applies the input image to the learning network model by executing the instruction, thereby generating a pixel texture patch corresponding to the pixel block included in the input image. An output image is obtained by applying it to a block, and the learning network model stores a texture patch corresponding to each of a plurality of classes classified based on the characteristics of the image, and a texture patch corresponding to each of the plurality of classes based on the input image To learn.

Description

Image processing apparatus and image processing method thereof [Display apparatus and image processing method thereof]

The present invention relates to an image processing apparatus and a control method thereof, and more particularly, to an image processing apparatus for reconstructing a texture component of an input image, and an image processing method thereof.

In addition, the present invention relates to an artificial intelligence (AI) system that simulates functions such as cognition and judgment of the human brain by using a learning network model and an application thereof.

With the development of electronic technology, various types of electronic devices are being developed and distributed. In particular, image processing apparatuses used in various places such as homes, offices, and public places have been continuously developed in recent years.

Recently, high-resolution display panels such as 4K UHD TVs have been released and are widely spread. However, high-quality high-resolution content is still lacking. Accordingly, various technologies for generating high-resolution content from low-resolution content are required. In addition, texture loss of content may occur due to image compression such as MPEG/H.264/HEVC, and accordingly, a technique for restoring the lost texture component is required.

In addition, in recent years, artificial intelligence systems that implement human-level intelligence have been used in various fields. Unlike existing rule-based smart systems, artificial intelligence systems are systems where machines learn, judge, and become smarter. As the artificial intelligence system is used, the recognition rate improves and the user's taste can be understood more accurately, and the existing rule-based smart system is gradually being replaced by a deep learning-based artificial intelligence system.

Artificial intelligence technology consists of machine learning (for example, deep learning) and component technologies using machine learning.

Machine learning is an algorithm technology that classifies/learns the features of input data by itself, and element technology is a technology that simulates functions such as cognition and judgment of the human brain using machine learning algorithms such as deep learning. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology that recognizes and applies/processes human language/text, and includes natural language processing, machine translation, dialogue system, question and answer, and speech recognition/synthesis. Visual understanding is a technology that recognizes and processes objects like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, and image improvement. Inference prediction is a technique that logically infers and predicts information by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation/classification), knowledge management (data utilization), and the like. Motion control is a technology that controls autonomous driving of a vehicle and movement of a robot, and includes movement control (navigation, collision, travel), operation control (behavior control), and the like.

Meanwhile, in the conventional image processing apparatus, there is a problem of applying a fixed texture patch to restore a lost texture component or applying a texture patch that is not compatible with an image. Accordingly, there has been a demand for a technique for generating a texture suitable for an image.

The present invention is in accordance with the above-described necessity, and an object of the present invention is to provide an image processing apparatus and an image processing method for improving the detail of an input image by using a texture patch learned based on the characteristics of the input image. It is in the offering.

An image processing apparatus according to an embodiment of the present disclosure for achieving the above object includes a memory storing at least one instruction and a processor electrically connected to the memory, and the processor, by executing the instruction , Applying an input image to a learning network model to obtain a texture patch corresponding to a pixel block included in the input image, and applying the obtained texture patch to the pixel block to obtain an output image, the learning network model , A texture patch corresponding to each of a plurality of classes classified based on an image characteristic is stored, and a texture patch corresponding to each of the plurality of classes is learned based on the input image.

Here, the learning network model identifies one of the plurality of classes based on the characteristics of the pixel block, outputs a texture patch corresponding to the identified class, and a first between the pixel block and the identified class. Whether to update the texture patch may be identified by comparing a similarity and a second similarity between the texture patch and the identified class.

In addition, the learning network model may replace a texture patch corresponding to the identified class with the pixel block based on the first and second similarities, or replace the pixel block with a texture patch corresponding to the identified class. Can be added.

In addition, if the first similarity is less than the second similarity based on the comparison result, the learning network model maintains the texture patch corresponding to the identified class, and based on the comparison result, the first similarity If the first similarity is greater than the second similarity, the texture patch may be updated based on the pixel block.

In addition, the learning network model, if there are a plurality of texture patches corresponding to the identified class, based on the correlation (correlation) of each of the pixel block and the plurality of texture patches, any one of the plurality of texture patches Can be identified.

In addition, the learning network model may learn the texture patch based on at least one of a storage timing of a texture patch corresponding to each of the plurality of classes or an application frequency of the texture patch.

In addition, the learning network model, if it is identified that the pixel block does not correspond to any one of the plurality of classes based on the characteristic of the pixel block, generates a new class based on the characteristic of the pixel block, and the new The pixel block may be mapped to a class and stored.

In addition, the plurality of classes may be classified based on at least one of an average pixel value, pixel coordinates, variance, edge intensity, edge direction, or color.

In addition, the processor obtains a weight for the texture patch based on a correlation between the obtained texture patch and the pixel block, and applies the weighted texture patch to the pixel block to obtain the output image. Can be obtained.

In addition, the output image may be a 4K Ultra High Definition (UHD) image or an 8K UHD image.

Meanwhile, an image processing method of an image processing apparatus according to an embodiment of the present disclosure includes the steps of obtaining a texture patch corresponding to a pixel block included in the input image by applying an input image to a learning network model, and obtaining a texture patch corresponding to a pixel block included in the input image. And obtaining an output image by applying the obtained texture patch, wherein the learning network model stores a texture patch corresponding to each of a plurality of classes classified based on a characteristic of the image, and based on the input image Thus, a texture patch corresponding to each of the plurality of classes is learned.

Here, the learning network model identifies one of the plurality of classes based on the characteristics of the pixel block, outputs a texture patch corresponding to the identified class, and a first between the pixel block and the identified class. Whether to update the texture patch may be identified by comparing a similarity and a second similarity between the texture patch and the identified class.

Here, the learning network model replaces the texture patch corresponding to the identified class with the pixel block based on the first and second similarities, or adds the pixel block as a texture patch corresponding to the identified class. can do.

In addition, if the first similarity is less than the second similarity based on the comparison result, the learning network model maintains the texture patch corresponding to the identified class, and based on the comparison result, the first similarity If the first similarity is greater than the second similarity, the texture patch may be updated based on the pixel block.

In addition, the learning network model, if there are a plurality of texture patches corresponding to the identified class, based on the correlation (correlation) of each of the pixel block and the plurality of texture patches, any one of the plurality of texture patches Can be identified.

In addition, the learning network model may learn the texture patch based on at least one of a storage timing of a texture patch corresponding to each of the plurality of classes or an application frequency of the texture patch.

In addition, the learning network model, if it is identified that the pixel block does not correspond to any one of the plurality of classes based on the characteristic of the pixel block, generates a new class based on the characteristic of the pixel block, and the new The pixel block may be mapped to a class and stored.

In addition, the learning network model identifies a class corresponding to each of a plurality of pixel blocks included in the input image, and the memory corresponding to at least one of the plurality of classes based on the identification frequency of each of the plurality of classes. You can change the size of your storage space.

Here, the learning network model deletes a texture patch corresponding to a class identified less than a preset number of times based on the identification frequency from the memory, and allocates a storage space reserved according to the deletion of the texture patch to the remaining classes. can do.

In addition, the plurality of classes may be classified based on at least one of an average pixel value, pixel coordinates, variance, edge intensity, edge direction, or color.

In addition, the obtaining of the output image may include obtaining a weight for the texture patch based on a correlation between the obtained texture patch and the pixel block, and applying the weighted texture patch to the pixel block. Applying to may include the step of obtaining the output image.

In addition, the output image may be a 4K Ultra High Definition (UHD) image or an 8K UHD image.

In addition, in a non-transitory computer-readable medium storing a computer command that causes the image processing device to perform an operation when executed by a processor of the image processing device according to an embodiment of the present disclosure, the operation is an input image Applying to a learning network model to obtain a texture patch corresponding to a pixel block included in the input image, and applying the obtained texture patch to the pixel block to obtain an output image, wherein the learning network The model stores texture patches corresponding to each of a plurality of classes classified based on characteristics of an image, and learns a texture patch corresponding to each of the plurality of classes based on the input image.

As described above, according to various embodiments of the present disclosure, it is possible to improve the detail of the entire image by generating a texture using a texture patch learned based on the characteristics of the image.

1 is a diagram for describing an example implementation of an image processing apparatus according to an exemplary embodiment of the present disclosure.
2 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present disclosure.
3 is a diagram for describing a pixel block according to an exemplary embodiment of the present disclosure.
4 is a diagram for describing a texture patch according to an exemplary embodiment of the present disclosure.
5 is a diagram illustrating a learning network model according to an embodiment of the present disclosure.
6 is a diagram for describing a class and a texture patch according to an embodiment of the present disclosure.
7 is a diagram illustrating a model for learning an input image according to an embodiment of the present disclosure.
8 is a diagram for describing a class according to another exemplary embodiment of the present disclosure.
9 is a diagram for describing a learning result according to an embodiment of the present disclosure.
10 is a diagram for describing a class according to another exemplary embodiment of the present disclosure.
11 is a block diagram illustrating a detailed configuration of the electronic device shown in FIG. 2.
12 is a block diagram illustrating a configuration of an image processing apparatus for learning and using a learning network model according to an embodiment of the present disclosure.
13 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

Terms used in the embodiments of the present disclosure have selected general terms that are currently widely used as possible while considering functions in the present disclosure, but this may vary according to the intention or precedent of a technician working in the field, the emergence of new technologies, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the overall contents of the present disclosure, not a simple name of the term.

In this specification, expressions such as "have," "may have," "include," or "may include" are the presence of corresponding features (eg, elements such as numbers, functions, actions, or parts). And does not exclude the presence of additional features.

The expression of at least one of A or/and B is to be understood as representing either “A” or “B” or “A and B”.

Expressions such as "first," "second," "first," or "second," as used herein may modify various elements regardless of order and/or importance, and It is used to distinguish it from other components and does not limit the components.

Some component (eg, the first component) is “(functionally or communicatively) coupled with/to)” to another component (eg, the second component) or “ When referred to as "connected to", it should be understood that a component can be directly connected to another component, or can be connected through another component (eg, a third component).

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "comprise" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other It is to be understood that the presence or addition of features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude the possibility of preliminary exclusion.

In the present disclosure, a "module" or "unit" performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module except for the "module" or "unit" that needs to be implemented with specific hardware and implemented as at least one processor (not shown). Can be.

In the present specification, the term user may refer to a person using an electronic device or a device (eg, an artificial intelligence electronic device) using an electronic device.

Hereinafter, an embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings.

1 is a diagram for describing an example implementation of an image processing apparatus according to an exemplary embodiment of the present disclosure.

The image processing apparatus 100 may be implemented as a TV as shown in FIG. 1, but is not limited thereto, and is not limited to, and is not limited to, a smart phone, a tablet PC, a notebook PC, a head mounted display (HMD), a near eye display (NED), and a LFD. (Large format display), Digital Signage (digital signage), DID (Digital Information Display), video wall (video wall), projector display, such as a device equipped with a display function can be applied without limitation.

The image processing apparatus 100 may receive images of various resolutions or various compressed images. For example, the image processing apparatus 100 may receive any one of standard definition (SD), high definition (HD), full HD, and ultra HD images. In addition, the image processing apparatus 100 may receive an image in a compressed form using MPEG (eg, MP2, MP4, MP7, etc.), AVC, H.264, HEVC, or the like.

According to an embodiment, even if the image processing apparatus 100 is implemented as a UHD TV, images such as SD (Standard Definition), HD (High Definition), and Full HD (hereinafter referred to as low-resolution images) are not sufficient because UHD content itself is insufficient. It is often entered. In this case, a method of expanding the input low-resolution image into a UHD image (hereinafter referred to as a high-resolution image) may be used. However, in the process of expanding the image, there is a problem that the texture of the image is blurred, resulting in deterioration of detail. Here, the texture of the image means a unique pattern or shape of an area that is regarded as the same feature in the image.

According to another embodiment, even when a high-resolution image is input, there is a problem in that the texture is lost due to image compression or the like, resulting in deterioration of detail. This is because digital images require more data as the number of pixels increases, and when large amounts of data are compressed, loss of texture due to compression is inevitable.

Accordingly, in the following, various embodiments of improving the detail of an image by restoring a texture component lost in various cases as described above will be described.

2 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present disclosure.

Referring to FIG. 2, the image processing apparatus 100 includes a memory 110 and a processor 120.

The memory 110 is electrically connected to the processor 120 and may store data necessary for various embodiments of the present disclosure. For example, the memory 110 may be implemented as an internal memory such as ROM (eg, electrically erasable programmable read-only memory (EEPROM)) or RAM included in the processor 120, or It may be implemented as a separate memory from 120. In this case, the memory 110 may be implemented in the form of a memory embedded in the image processing apparatus 100 depending on the purpose of data storage, or may be implemented in the form of a memory that is detachable to the image processing apparatus 100. For example, data for driving the image processing device 100 is stored in a memory embedded in the image processing device 100, and in the case of data for an extended function of the image processing device 100, the image processing device 100 ) Can be stored in a removable memory. Meanwhile, in the case of a memory embedded in the image processing apparatus 100, a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.), non-volatile memory) (E.g. one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g. NAND flash or NOR flash) Etc.), a hard drive, or a solid state drive (SSD), and in the case of a memory that is detachable to the image processing apparatus 100, a memory card (for example, a compact flash (CF)), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory that can be connected to the USB port (e.g. For example, it may be implemented in a form such as a USB memory).

According to an embodiment, the memory 110 may store a learning network model used to acquire a texture patch corresponding to a pixel block included in the input image 10. Here, the learning network model may be a machine-learned model based on a plurality of images. For example, the training network model may be a model trained on a convolution neural network (CNN) based on a plurality of sample images and input images 10. Here, CNN is a multilayer neural network with a special connection structure designed for voice processing and image processing. In particular, the CNN can variously filter the image through preprocessing on the pixel and recognize the characteristics of the image. For example, a characteristic of a pixel block having a preset size included in the input image 10 may be recognized. On the other hand, it goes without saying that the learning network model is not limited to CNN. For example, it goes without saying that the image processing apparatus 100 may use a learning network model based on various neural networks, such as a recurrent neural network (RNN) and a deep neural network (DNN).

Meanwhile, the “texture patch” may mean a patch applied to a corresponding pixel block in order to improve the texture of the pixel block. Here, since "patch" is a term applied for convenience in consideration of functions, various terms other than the term "patch" may be applied to the present embodiment. For example, since each patch has a structure in which a plurality of patch values are arranged in a pixel-based matrix form, it may be referred to as a mask in consideration of this form. As will be described later, as the texture patch is applied to the corresponding pixel block, the texture of the pixel block may be improved and the fineness of the pixel block may be increased. Meanwhile, the image processing apparatus 100 may not apply a previously determined texture patch to the pixel block regardless of the characteristics of the pixel block, but may apply the updated texture patch using the learning network model.

The processor 120 is electrically connected to the memory 110 and controls the overall operation of the image processing apparatus 100.

According to an embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a timing controller (T-CON) that processes digital image signals, but is limited thereto. It is not a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), or a communication processor. processor (CP)), or one or more of an ARM processor, or may be defined in a corresponding term In addition, the processor 140 includes a system on chip (SoC) and a large scale integration (LSI) with a built-in processing algorithm. It may be implemented in the form of a field programmable gate array (FPGA).

The processor 120 image-processes the input image to obtain an output image. Specifically, the processor 120 may obtain an output image by performing texture enhancement processing on the input image. Here, the output image may be an Ultra High Definition (UHD) image, particularly, a 4K UHD image or an 8K UHD image, but is not limited thereto.

In particular, the processor 120 according to an embodiment of the present disclosure may obtain a texture patch to be used for texture enhancement processing. Specifically, the processor 120 may obtain a texture patch corresponding to a pixel block included in the input image 10 by applying the input image 10 to the learning network model. Here, the pixel block means a set of adjacent pixels including at least one pixel.

3 is a diagram for describing a pixel block according to an exemplary embodiment of the present disclosure.

Referring to FIG. 3, in the image frame constituting the input image 10, the processor 120 may divide a plurality of pixels included in the image frame in units of pixel blocks 20 and input them to the learning network model. . According to an embodiment, the processor 120 may sequentially input a plurality of pixel blocks 20 constituting an image frame into the learning network model. Subsequently, the learning network model may output texture patches 30-1, ... 30-n corresponding to each of the input plurality of pixel blocks 20-1, ... 20-n.

The processor 120 according to the exemplary embodiment of the present disclosure may identify the input image 10 as a 5*5 pixel block 20, but the size of the pixel block is not limited thereto, and 3*3, It can be implemented in various sizes in the form of N*N such as 4*4. For example, the processor 120 may convert the input image 10 into pixels of various sizes according to the purpose of image processing, the resolution of the input image 10 (for example, FHD), and the resolution of the output image (UHD, 8K). Of course, it can be divided into blocks 20. Hereinafter, for convenience of explanation, a group of pixels of a predetermined size having a structure arranged in a matrix form in an image frame constituting the input image 10 is assumed to be a pixel block 20 obtained from the input image 10. Do it.

Returning to FIG. 2, the processor 120 according to an embodiment of the present disclosure may obtain a texture patch corresponding to the pixel block 20 by applying the learning network model to the input image 20. A detailed description of this will be made with reference to FIG. 4.

4 is a diagram for describing a texture patch according to an exemplary embodiment of the present disclosure.

4 is a diagram in which pixels constituting the input image 10 are expressed as pixel values. The processor 120 according to an embodiment of the present disclosure may obtain a texture patch 30 corresponding to the pixel block 20 by applying a learning network model to the input image 10. Here, application means inputting the input image 10 to the learning network model, and the output of the learning network may be the texture patch 30.

Here, the learning network model may output a texture patch 30 corresponding to the pixel block 20 included in the input image 10 and may perform learning based on the pixel block 20.

The learning network model according to an embodiment may include a plurality of classes classified based on any one of various characteristics of an image, and may include a texture patch 30 corresponding to each of the plurality of classes. For example, the learning network model may store a plurality of classes classified based on an edge direction among characteristics of an image, and may include a texture patch 30 matched to each of the plurality of classes. As another example, the learning network model may store a plurality of classes classified based on a grayscale average value of each pixel block 20 among the characteristics of an image, and may include a texture patch 30 matched to each of the plurality of classes. have.

Meanwhile, it goes without saying that the image processing apparatus 100 according to an embodiment of the present disclosure may include a plurality of learning network models. As an example, the image processing apparatus 100 classifies classes based on an edge direction, classifies classes based on a first learning network model that performs learning on a texture patch 30, a gray scale average value, and performs learning. It may include a plurality of learning network models, such as a second learning network model to perform, a third learning network model that classifies classes based on color coordinates, and performs learning. The image processing apparatus 100 according to an embodiment identifies any one of a plurality of learning network models based on the characteristics of the input image 10 and applies the identified learning network model to the input image 10 to patch a texture. (30) can be obtained. For example, the image processing apparatus 100 is a pre-process of identifying any one learning network model for obtaining an appropriate texture patch 30 based on the characteristics of the input image 10 among a plurality of learning network models. It may contain a learning network model. The preprocessing learning network model classifies a class based on the edge direction among the plurality of learning network models, for example, if the colors of a plurality of pixels constituting the input image 10 are distributed within a similar color based on the characteristics of the image. And, it is possible to identify a first learning network model that outputs the texture patch 30.

The learning network model according to an embodiment of the present disclosure may perform learning based on the input image 10. For example, the learning network model identifies a first similarity of the pixel block 20 with respect to the class corresponding to the pixel block 20 included in the input image 10, and the texture matched to the corresponding class for the class The second degree of similarity of the patch 30 can be identified. Subsequently, it may be identified whether the texture patch 30 is updated based on the first and second similarities. For example, if the first similarity is greater than the second similarity, the learning network model determines that the obtained texture patch 30 is not suitable for texture enhancement of the input image 10, and the pixel of the input image 10 An update may be performed based on block 20. The learning network model outputs a texture patch 30 corresponding to another pixel block 20' belonging to the same class as the corresponding pixel block 20 among various pixel blocks constituting the input image 10. The updated texture patch 30 ′ may be output based on the pixel block 20 instead of the texture patch 30. Accordingly, the texture patch 30 output from the learning network model may be suitable for texture enhancement of the input image 10. As another example, when the second similarity degree is greater than the first similarity degree, the learning network model may determine that the obtained texture patch 30 is suitable for texture enhancement of the corresponding input image 10 and maintain the texture patch 30.

Meanwhile, an operation of a learning network model for classifying (or identifying) a class corresponding to the pixel block 20 among a plurality of classes according to an embodiment of the present disclosure may be referred to as a classifier, a class identifier, or the like. . Here, when the pixel block 20 included in the input image 10 is input, the classifier may identify a class suitable for the pixel block 20 from among a plurality of classes. For example, the classifier may identify an edge direction of the pixel block 20 and a degree of similarity between the identified edge direction and an edge direction defining each of the plurality of classes. Subsequently, the classifier may identify one class having the greatest similarity among the plurality of classes as a class corresponding to the corresponding pixel block 20.

The learning network model according to an embodiment of the present disclosure includes a model (eg, a classifier model) for identifying a class corresponding to the pixel block 20 and a texture corresponding to the pixel block 20 and the pixel block 20. It may mean a combination of a model that performs self-learning on the texture patch 30 by comparing the similarity of the patch 30. The learning network model according to an embodiment may be an on-device machine learning model in which the image processing apparatus 100 performs training on its own without depending on an external device. On the other hand, this is an embodiment, and the learning network model may be implemented in a form that the classifier model operates on an on-device basis, and a model that performs learning on a texture patch operates on an external server basis. Of course.

Accordingly, the learning network model may store a texture patch 30 corresponding to each of a plurality of classes classified and learned based on the characteristics of the image. The learning network model may output a texture patch corresponding to the input image 10 and at the same time learn a texture patch 30 corresponding to each of a plurality of classes based on pixel values included in the input image 10. .

Referring to FIG. 4, the learning network model may identify one class corresponding to the pixel block 20 from among a plurality of classes based on the characteristics of the pixel block 20. For example, the learning network model may store a plurality of classes classified based on an edge direction (or edge pattern) among various characteristics of an image. Here, the edge may mean a point at which a pixel value (or brightness of a pixel) changes from a low value to a high value or from a high value to a low value. The edge may mean a boundary line between objects generated according to various objects included in the image. The learning network model according to an example may identify one class corresponding to the edge direction (or border direction) of the pixel block 20 from among a plurality of classes. In addition, the learning network model may identify one class that is most similar (or most suitable) to the edge direction of the pixel block 20 among the plurality of classes. Subsequently, the learning network model may output a texture patch 30 corresponding to the identified class. Returning to FIG. 2, the processor 120 according to an embodiment of the present disclosure may perform texture enhancement processing by applying a texture patch output from the learning network model to the input image 10.

5 is a diagram illustrating a learning network model according to an embodiment of the present disclosure.

The learning network model according to an embodiment of the present disclosure may store a plurality of classes classified based on characteristics of an image and at least one texture patch corresponding to each of the plurality of classes. Referring to FIG. 5, the learning network model may include first to nth classes classified based on an edge direction among image characteristics. In addition, the learning network model may include texture patches corresponding to each of the first to nth classes. Here, the characteristic of the image may mean at least one of an average, dispersion, pixel coordinate, edge intensity, edge direction, or color of pixel values included in the pixel block 20. Accordingly, the learning network model according to an embodiment may include a plurality of classes divided based on at least one of an average of pixel values, a variance, a pixel coordinate, an edge intensity, an edge direction, or a color. On the other hand, the learning network model can generate a plurality of classes based on various features that can be identified from the pixel block 20, in addition to the above-described examples, and whether the corresponding pixel block 20 corresponds to which of the plurality of classes. Of course, can be identified. For example, the learning network model may classify classes based on color coordinates, and may identify a class corresponding to the pixel block 20 based on an average of color coordinates of pixels included in the pixel block 20. have.

Referring to FIG. 5, in an image frame constituting an input image 10, the processor 120 according to an embodiment divides a plurality of pixels included in the image frame into pixel blocks 20 to provide a learning network model. You can enter in According to an embodiment, the processor 120 may sequentially input a plurality of pixel blocks 20 constituting an image frame into the learning network model. Subsequently, the learning network model may output texture patches 30-1, ... 30-n corresponding to each of the input plurality of pixel blocks 20-1, ... 20-n.

For example, the learning network model may identify a class corresponding to the first pixel block 20-1 from among a plurality of classes based on the characteristics of the first pixel block 20-1. For example, the learning network model identifies the edge direction of the first pixel block 20-1 based on the pixels constituting the first pixel block 20-1, and the identified edge direction is among a plurality of classes. It is possible to identify which class corresponds to. Specifically, the learning network model may identify a similarity between a plurality of classes and the first pixel block 20-1. For example, in the learning network model, if the edge direction of the first pixel block 20-1 is 0°, in the first class (Class #1) compared to the second to eighth classes (Class #2-Class #8) A high degree of similarity (or goodness of fit) can be obtained. Here, the first class Class#1 may mean a class defined based on the edge direction 0°. Subsequently, the learning network model may identify a first class (Class #1) as a class corresponding to the first pixel block 20-1. Subsequently, the processor 120 may acquire the first texture patch 30-1 corresponding to the first class (Class #1) through the learning network model.

As another example, if the second pixel block 20-2 is identified as corresponding to a second class (Class #2) among a plurality of classes, the learning network model is a second texture corresponding to the second class (Class #2). A patch 30-2 may be provided.

Meanwhile, in FIG. 5, for convenience of explanation, the learning network model includes first to eighth classes divided based on the edge direction, and each of the classes is one texture patch, that is, the first to eighth texture patches 30. -1, ... 30-8), but is not limited thereto.

In the learning network model according to an embodiment, if a corresponding pixel block is identified as not corresponding to any one of a plurality of classes based on the characteristics of the pixel block 20, a new class is generated based on the characteristics of the corresponding pixel block 20. Can be created and saved by mapping a pixel block to a new class. For example, if the similarity between the pixel block 20 and the plurality of classes are all less than the threshold value, the learning network model may generate a new class other than the plurality of classes based on the characteristics of the pixel block 20.

Referring to FIG. 5, according to an embodiment, if the similarity between the first to eighth classes and the fourth pixel block 20-4 is less than a threshold value, that is, the fourth pixel block 20-4 is If the class corresponding to) is not identified, the ninth class may be generated based on the characteristics of the fourth pixel block 20-4. For example, if a plurality of classes are classified based on the edge direction, the learning network model identifies the edge directions of pixels constituting the fourth pixel block 20-4, and the ninth class based on the identified edge direction. Can be created. Subsequently, the learning network model may map and store the fourth pixel block 20-4 to the ninth class. For example, the learning network model may store the fourth pixel block 20-4 as a texture patch corresponding to the newly generated ninth class.

Returning to FIG. 2, in the learning network model according to an embodiment of the present disclosure, when the texture patch 30 matching the class corresponding to the pixel block 20 is identified, the similarity between the pixel block 20 and the class, texture Whether to update the texture patch 30 may be identified based on the similarity between the patch 30 and the class. Here, the learning network model compares the similarity (or goodness of fit) between the criteria defining the class and the pixel block 20 and the similarity between the criteria defining the class and the texture patch 30 matched to the corresponding class to identify whether or not to be updated. can do. Referring to FIG. 5, the learning network model may include a plurality of classes classified based on an edge direction. Among the plurality of classes, a first class (Class#1) may mean a class in which an edge direction is defined as 0°, and a fifth class (Class#5) may mean a class in which an edge direction is defined as 90°. When the first pixel block 20-1 is input, the learning network model identifies a first class (Class#1) with the largest similarity among a plurality of classes based on the edge direction of the first pixel block 20-1. can do. Subsequently, the first texture patch by comparing the similarity between the first class (Class#1) and the first pixel block 20-1 and the similarity between the first class (Class#1) and the first texture patch 30-1. Whether to update (30-1) can be identified.

A detailed description of this will be made with reference to FIG. 6.

6 is a diagram for describing a class and a texture patch according to an embodiment of the present disclosure.

Referring to FIG. 6, the learning network model may identify a class corresponding to the pixel block 20 from among a plurality of classes based on the characteristics of the pixel block 20. For example, if the pixel block 20 includes an edge direction of 65°, the learning network model is a fourth class defined as an edge direction of 67.5° among the first to eighth classes (Class#1-Class#8). (Class #4) can be identified. Subsequently, the learning network model may acquire the texture patch 30 corresponding to the identified fourth class (Class #4).

Subsequently, the learning network model determines whether the texture patch 30 is updated based on the similarity between the pixel block 20 and the fourth class (Class#4) and the similarity between the texture patch 30 and the fourth class (Class#4). Can be identified. Here, it goes without saying that the similarity may be measured using various types of similarity measurement algorithms, fitness measurement algorithms, and machine learning algorithms. For example, it is possible to compare histograms based on grayscale values, calculate the Euclidean distance, etc. to identify similarity, and as another example, to identify similarity based on an algorithm model trained by a convolution neural network (CNN). Of course.

For example, if the edge direction of the texture patch 30 matched to the fourth class (Class#4) is 50° according to the result of learning based on the conventional input image 10' and the sample image of the learning network model. Can be assumed. Since the edge direction defining the fourth class (Class#4) is 67.5°, the learning network model is a texture patch that includes the edge direction of 50° with a first similarity of the pixel block 20 including the edge direction of 65°. It has a larger value than the second similarity of (30), and it can be identified that the pixel block 20 is suitable for the fourth class (Class#4). The learning network model may replace the texture patch 30 based on the pixel block 20. Subsequently, in the learning network model, when another pixel block 20' included in the input image 10 is input and the other pixel block 20' corresponds to the fourth class (Class#4), the edge direction of 65° is An updated texture patch 30 ′ may be output based on the included pixel block 20. Subsequently, the processor 120 may generate a texture of the other pixel block 20' based on the texture patch 30'.

As another example, it may be assumed that the second similarity between the class and the texture patch 30 matched to the corresponding class is greater than the first similarity between the class corresponding to the pixel block and the pixel block 20. In this case, the learning network model can identify that the texture patch 30 is suitable for texture generation of the input image 10 and the pixel block 20, and can maintain the texture patch 30 as it is.

Meanwhile, according to an embodiment of the present disclosure, since the learning network model updates the texture patch 30 in the process of obtaining the texture patch 30 corresponding to the pixel block 20 included in the input image 10, An image processing model including a texture patch 30 suitable for texture enhancement of the input image 10 may be generated.

For example, if the learning network model is applied to the input image 10 including objects such as forest and grass, the learning network model is the similarity between the pixel block 20 constituting the input image 10 and the class and a previously stored texture. The previously stored texture patch 30 may be maintained by comparing the similarity between the patch 30 and the classes, or the previously stored texture patch 30 may be replaced with the corresponding pixel block 20. According to an embodiment, when a learning network model is applied to another pixel block 20 ′ included in the input image 10, the learning network model is updated based on the pixel block 20 in a preceding process. ') can be identified. In this case, since the updated texture patch 30 ′ is a patch obtained from the input image 10, it may have a high correlation with other pixel blocks 20 ′ included in the same input image 10, and a high degree of suitability. . Accordingly, the processor 120 may apply the updated texture patch 30 ′ to the other pixel block 20 ′ to generate a texture and obtain an output image with improved detail.

Returning to FIG. 2, the learning network model according to an embodiment of the present disclosure according to an embodiment of the present disclosure is selected from the storage timing of the texture patch 30 corresponding to each of the plurality of classes or the frequency of application of the texture patch 30. The texture patch 30 may be learned based on at least one.

As an example, the learning network model may learn the texture patch 30 based on the input image 10, and may additionally consider the storage timing of the previously stored texture patch 30. For example, when the learning network model identifies that the storage timing of the texture patch 30 corresponding to the pixel block 20 included in the input image 10 has passed a certain period, the texture patch 30 is converted to the pixel block. Can be replaced with (20). If the storage time of the texture patch 30 is old, it may mean that the degree of fitness with the input image 10 and the similarity with the class of the matching relationship are low, so that the learning network model is a pixel block included in the input image 10 Learning may be performed based on (20) and the texture patch 30 may be updated. The learning network model maps the pixel block 20 included in the input image 10 to a texture patch 30 of a class corresponding to the pixel block 20, and maps the newly mapped texture patch 30 to the input image ( 10) can be used to create the texture.

As another example, in the learning network model according to an embodiment of the present disclosure, when the first similarity between the pixel block 20 and the class and the second similarity between the texture patch 30 and the class are the same, when the texture patch 30 is stored The texture patch 30 may be updated based on the frequency of use, and the like. For example, if the first and second similarities are the same, the pixel block 20 may be more suitable for texture generation of the input image 10 than the previously stored texture patch 30. The texture patch 30 can be updated. As another example, if the first and second similarities are the same, it goes without saying that the learning network model may add the pixel block 20 in addition to the texture patch 30.

However, this does not mean that the texture patch 30 must be updated when the storage timing of the texture patch 30 elapses for a certain period of time, for example.

As another example, the learning network model may learn the texture patch 30 based on the frequency of application of the texture patch 30. For example, if it is identified that a specific texture patch 30 is frequently used for texture generation of another input image 10' in addition to the image 10 currently being input, the specific texture patch 30 is associated with the corresponding class. It may mean that the suitability of is high, and it is usefully applicable to texture generation. On the contrary, if the specific texture patch 30 is identified as having a low frequency used to generate the texture of the input image 10, the learning network model can identify the texture patch 30 as having a low suitability with the class of the mapping relationship. have. In this case, the learning network model may replace the texture patch 30 with the pixel block 20 included in the input image 10. For example, based on the characteristics of the pixel block 20, it is identified that a specific class among a plurality of classes is a class corresponding to the pixel block 20, and the storage time of the texture patch 30 corresponding to the identified class is If a certain period has elapsed or the frequency of application to the image is less than the threshold number, the learning network model may replace the texture patch 30 with the corresponding pixel block 20.

7 is a diagram illustrating a model for learning an input image according to an embodiment of the present disclosure.

Referring to FIG. 7, the learning network model may not store texture patches 30 corresponding to some of the plurality of classes. For example, as shown in FIG. 5, rather than storing the first to eighth texture patches 30-1, ..., 30-8 corresponding to each of the first to eighth classes, a plurality of Some of the classes may store the texture patch 30 in a mapping relationship, and the other classes may not store the texture patch 30. In this case, the learning network model may acquire and store the texture patch 30 based on the input image 10. For example, it may be assumed that the learning network model identifies a class corresponding to the pixel block 20 included in the input image 10 and does not include the texture patch 30 corresponding to the identified class. The learning network model may map and store the corresponding pixel block 20 to the identified class.

Meanwhile, although it has been described that the class includes only one texture patch 30, it is not necessarily limited thereto. For example, the first class may include at least two texture patches 30 corresponding to the first class. According to an embodiment, the learning network model may identify a class of the pixel block 20 included in the input image 10 and add the pixel block 20 as a texture patch 30 to the identified class. Here, the learning network model does not delete or replace the previously stored texture patch 30, but uses the previously stored texture patch 30 as a first texture patch and the pixel block 20 as a second texture patch. It can be saved by mapping to the corresponding class.

In the learning network model according to an embodiment, if a plurality of texture patches 30 corresponding to a class identified as corresponding to the pixel block 20 exist, each of the pixel block 20 and the plurality of texture patches 30 Any one of a plurality of texture patches may be identified based on the correlation. For example, it may be assumed that a class corresponding to the pixel block 20 is a fourth class, and a texture patch having a mapping relationship with the fourth class is a first to third texture patch. The learning network model may identify a correlation between a pixel block and each of the first to third texture patches, and may identify a texture patch having a highest correlation value among the identified correlations. The texture patch having the highest correlation value may mean that it is a patch having the highest degree of suitability for texture generation of the pixel block 20. Subsequently, the learning network model may generate a texture by applying the identified texture patch to the pixel block 20.

8 is a diagram for describing a class according to another exemplary embodiment of the present disclosure.

Referring to FIG. 8, the learning network model according to an embodiment may classify the pixel block 20 into any one of the first to sixteenth classes based on the characteristics of an image. Subsequently, the learning network model may identify the classified class and the texture patch 30 of the mapping relationship. Subsequently, the identified texture patch 30 may be applied to the pixel block 20.

Meanwhile, the learning network model can classify classes according to various criteria. In addition, the number of classes is not fixed or limited, and it goes without saying that the learning network model may delete a specific class from among a plurality of classes or create an additional class other than a plurality of classes.

For convenience of description, classes are classified based on the edge direction, and description has been made on the assumption that each of the plurality of classes includes a single texture patch, but is not limited thereto. For example, the learning network model according to an embodiment may be classified into first to nth classes based on the distribution of color coordinates, and the color coordinate distribution of the pixel block 20 included in the input image 10 Based on the first to nth classes, a corresponding class may be identified. As another example, it goes without saying that the learning network model may be classified into first to nth classes based on an average grayscale value and a variance of grayscale values.

9 is a diagram for describing a learning result according to an embodiment of the present disclosure.

Referring to FIG. 9, the training network model provides a texture patch 30 corresponding to each of a plurality of pixel blocks 20 constituting an input image 10, and the processor 120 uses the texture patch 30 as a pixel. By applying it to the block 20, an output image with improved detail may be obtained.

As the learning network model performs learning based on the pixel block 20 included in the input image 10, the plurality of classes and texture patches 30 included in the learning network model before and after the input of the image 10 are different. can do. For example, before inputting an image, the learning network model may include a texture patch 30 learned based on another image or sample image previously input. The learning network model identifies and identifies the similarity between the pixel block 20 included in the input image 10 and the class corresponding to the pixel block 20, and the similarity between the texture patch 30 mapped to the corresponding class and the class. The texture patch 30 may be updated based on the result. For example, the learning network model may replace the texture patch 30 with the pixel block 20 or maintain the texture patch 30.

9 is a diagram illustrating a training result of a learning network model based on an input image 10. Referring to FIG. 9, some of the plurality of classes included in the learning network model have a mapping relationship texture patch 30 replaced with a pixel block 20 included in the input image 10. As another example, the texture patch 30 of the mapping relationship is maintained for the rest of the plurality of classes.

In Figs. 5, 6, and 7, the arrows shown in the pixel block 20 show the classes corresponding to the corresponding pixel blocks 20, and the arrows shown in the pixel block 20 in Fig. 9 are the learning of the learning network model. It means that the texture patch 30 has been replaced with the corresponding pixel block 20 according to the result. For example, referring to FIG. 9, a texture patch 30 corresponding to each of classes 2, 4, and 6 has been replaced with a pixel block 20 included in the input image 10.

Returning to FIG. 2, the processor 120 according to an embodiment of the present disclosure may obtain a weight for the texture patch 30 based on a correlation between the texture patch 30 and the pixel block 20. Subsequently, the processor 120 may obtain an output image by applying the texture patch 30 ′ to which the weight is applied to the pixel block 20.

The correlation (or correlation or correlation) between the pixel block 20 included in the input image 10 and the texture patch 30 obtained from the training network model is calculated as a constant value, and two variables x, y This refers to a relationship that is assumed to be related to each other, and the degree of the relationship can be expressed by a number called a correlation coefficient. For example, the correlation coefficient can be expressed as a number between -1.0 and +1.0. Regardless of the sign, the greater the absolute value of the number, the greater the relationship. For example, a negative (-) value may indicate a negative correlation, and a positive (+) value may indicate a positive correlation.

For example, the pixel value I = [i ₀ , i ₁ , ..., i _n-1 ] included in the corresponding pixel block 20, the value R[n] = [ Assuming r ₀ , r ₁ , ..., r _n-1 ], the correlation value C[n] can be obtained as E[I*R[n]] = i _i * r _i.

Alternatively, if the average of the pixel values included in the target pixel block is m(I) and the average of the values included in the texture patch R[n] is m(R[n]), the correlation value is Equation 1 as follows: Can be obtained based on.

Meanwhile, according to another embodiment of the present disclosure, the average of the texture patches 30 may be 0. This is to keep the brightness of the entire input image 10 unchanged even when the texture patch 30 is applied. According to an embodiment, if the average of the texture patches 30 is 0, Equation 1 may be derived as Equation 2 below.

In the learning network model according to an embodiment of the present disclosure, if the correlation between the pixel block 20 and the texture patch 30 corresponding to the pixel block 20 is greater than or equal to a threshold value, the class of the pixel block 20 is The corresponding texture patch 30 can be maintained. As another example, in the learning network model, if the correlation between the pixel block 20 and the texture patch 30 corresponding to the pixel block 20 is less than a threshold value, the texture patch 30 is based on the pixel block 20. As another example, the processor 120 may obtain a value obtained by multiplying the obtained correlation value by a preset proportional constant as a weight corresponding to the texture patch 30. For example, the processor 120 may obtain a weight in the range of 0 to 1 based on the correlation value. According to an example, when a weight of 0 is applied to the texture patch 30 according to the correlation, the texture patch 30 is not added to the target pixel block 20. For example, in flat areas or areas with strong edges, there is a high probability that the correlation for all classes and all texture patches is very low, so no texture is produced. In this case, it is possible to prevent a ringing phenomenon that may occur in the edge region, and it is possible to prevent unnecessary texture from being added to the flat region.

However, according to another embodiment of the present disclosure, similarity information between the pixel block 20 and the texture patch 30 may be obtained by various cost functions in addition to the above-described correlation. For example, as a cost function for determining similarity, MSE (Mean Square Error), SAD (Sum of Absolute Difference), MAD (Median Absolute Deviation), and correlation may be used. For example, when MSE is applied, the MSE of the target pixel block may be calculated, and similarity between the target pixel block 20 and the texture patch 30 may be obtained from the viewpoint of the MSE. For example, a similarity weight may be determined based on the MSE difference.

The processor 120 may apply the obtained weight to each of the texture patches 30 and apply the weighted texture patch 30 to the target pixel block 20 to obtain an output image. Here, the application may be a method of adding a value included in a corresponding region of the texture patch 30 to which a weight is applied to each pixel value included in the target pixel block 20. However, it is not limited thereto, and of course, additional processing other than simple addition may be performed.

According to another embodiment of the present disclosure, when the texture patch 30 is obtained, the processor 120 applies frequency filtering to the texture patch 30 and applies the texture patch 30 to which frequency filtering is applied to a target pixel block. You may. That is, the processor 120 may modify the frequency range of the texture patch 30 by applying frequency filtering before adding the texture patch 30 to the input image. For example, the processor 120 may generate a high-frequency texture using a high-pass filter, or may generate a low-frequency texture using a low-pass filter. Equation 3 below shows a process of obtaining an output image O by adding the filtered texture Filter(T) to the input image I.

For example, the processor 120 may apply a low-pass filter such as Gaussian blurring (or Gaussian filtering) to the texture patch 30. Gaussian blurring is a method of blurring using a Gaussian filter based on a Gaussian probability distribution. When a Gaussian filter is applied to the texture patch 30, high-frequency components are cut off to perform a blur process. The processor 12 according to an embodiment may obtain a blurred texture patch 30' by performing Gaussian filtering on all pixel values included in the texture patch 30. Subsequently, the processor 120 may obtain an output image by applying the blurred texture patch 30 ′ to the corresponding pixel block 20.

Meanwhile, the above-described image processing process, that is, texture enhancement processing may be performed before or after scaling of an image according to an exemplary embodiment. For example, the above-described image processing may be performed after scaling to enlarge a low-resolution image to a high-resolution image, or scaling may be performed after performing the above-described image processing in a process of decoding a compressed image.

The learning network model according to another embodiment of the present disclosure may acquire a plurality of texture patches to which different weights are applied.

For example, the learning network model may identify a class corresponding to the pixel block 20 and obtain first to nth texture patches corresponding to the class. Then, the learning network model may identify a correlation between the pixel block 20 and the first to nth texture patches. For example, the learning network model acquires a first weight based on the correlation between the pixel block 20 and the first texture patch, and obtains a second weight based on the correlation between the pixel block 20 and the second texture patch. Can be obtained. Subsequently, the learning network model multiplies the first weight by the first texture patch, multiplies the second weight by the second texture patch, and the first texture patch multiplied by the first weight and the second texture multiplied by the second weight. An output image may be obtained by applying the patch to the target pixel block 20.

According to an embodiment of the present disclosure, the weight may be determined in a preset range, for example, between 0 and 1 according to the correlation. For example, in the learning network model, when the correlation between the pixel block 20 and the acquired texture patch 30 is minimum, the weight is determined as 0, and when the correlation is maximum, the weight is determined as 1, and the correlation is The weight can be determined to increase linearly between the minimum and maximum.

10 is a diagram for describing a class according to another exemplary embodiment of the present disclosure.

Referring to FIG. 10, the learning network model may add or delete texture patches 30 for each class in the process of performing training.

According to an embodiment, the learning network model may delete a texture included in a specific class or store a plurality of texture patches in a specific class as learning is performed based on a plurality of pixel blocks included in the input image 10. . Accordingly, the learning network model may allocate the same storage space for storing texture patches to each of a plurality of classes, or allocate more storage space to a specific class than the rest of the classes.

According to an embodiment, the learning network model identifies a class of each of a plurality of pixel blocks included in the input image 10, and a memory 110 corresponding to at least one of the plurality of classes based on the identification frequency of each of the plurality of classes. ) You can change the size of the storage space. For example, the learning network model may increase the size of the storage space of the memory 110 by additionally allocating a storage space for storing a texture patch to a class that is identified by a predetermined frequency or more according to the identification frequency. Here, the preset frequency may mean a case in which 20% or more of a specific class is identified relative to the total number of pixel blocks. On the other hand, this is only an exemplary embodiment, and it goes without saying that the preset number may be variously set, such as 10%. As another example, the learning network model may increase the size of a storage space corresponding to the most identified class based on the identification frequency.

For example, it may be assumed that a plurality of pixel blocks among a plurality of pixel blocks included in the input image 10 correspond to the fourth class. The learning network model may increase the size of the storage space on the memory 110 corresponding to the fourth class.

According to an embodiment, when a pixel block is identified as corresponding to a fourth class, the learning network model identifies a first similarity between the corresponding pixel block and the fourth class, and calculates a second similarity between texture patches previously stored in the fourth class. Can be identified. Subsequently, if the first similarity is smaller than the second similarity, the learning network model may maintain a previously stored texture patch and additionally store a corresponding pixel block in the fourth class. Here, the previously stored texture patch may have a line priority than the corresponding pixel block.

As another example, if the first similarity is greater than the second similarity, the learning network model may additionally store the corresponding pixel block in the fourth class. Here, the priority of the previously stored texture patch is changed to a later priority, and the corresponding pixel block may have a prior order than the previously stored texture patch.

In another embodiment, the learning network model may change the size of the storage space of the memory 110 so that a preset number of texture patches can be stored in the most identified class based on the identification frequency for each of a plurality of classes. The size of the storage space can be changed so that a smaller number of texture patches than a preset number can be stored in the many identified classes. For example, the training network model can change the size of the storage space so that up to 10 texture patches can be stored in the fourth class identified most, and up to 6 texture patches are stored in the second class identified the second most. You can change the size of the storage space to allow it. On the other hand, the specific number is only an example, and it goes without saying that the number of storeable texture patches may be variously set.

On the other hand, the learning network model does not always add the corresponding pixel block as a texture patch corresponding to the identified class, and if the similarity between the pixel block and the identified class is less than a preset value, it may not add the corresponding pixel block. Of course. For example, if the similarity between the corresponding pixel block and the identified class is less than 50%, the learning network model may not add the corresponding pixel block as a texture patch of the identified class.

As another example, in identifying the classes of each of the plurality of pixel blocks included in the input image 10, the learning network model may delete from the memory 110 a texture patch corresponding to a class identified less than a preset number of times. Subsequently, the learning network model may allocate the storage space of the memory 110 reserved according to the deletion of the texture patch to the remaining classes.

For example, if the number of pixel blocks corresponding to the third class is less than a preset number according to the result of identifying each class of a plurality of pixel blocks, the learning network model deletes a texture patch previously stored in the third class, A storage space for storing a texture patch corresponding to the third class may be allocated to the remaining classes. Accordingly, the learning network model may increase the size of a storage space in a corresponding class so that a plurality of texture patches can be stored in a class identified with a high frequency.

As another example, the learning network model may reallocate a storage space pre-allocated to the class to the remaining classes by deleting a class that is identified with a small number of assumptions based on the frequency of identification.

11 is a block diagram illustrating a detailed configuration of the electronic device shown in FIG. 2.

Referring to FIG. 11, the image processing apparatus 100 includes a memory 110, a processor 120, an input unit 130, a display 140, an output unit 150, and a user interface 160. In the description of FIG. 11, detailed descriptions of the configurations overlapping with those of FIG. 2 will be omitted.

According to an embodiment of the present disclosure, the memory 110 may be implemented as a single memory that stores data generated in various operations according to the present disclosure.

However, according to another embodiment of the present disclosure, the memory 110 may be implemented to include first to third memories.

The first memory may store at least some of the images input through the input unit 130. In particular, the first memory may store at least a partial region of the input image frame. In this case, at least some of the areas may be areas required to perform image processing according to an exemplary embodiment of the present disclosure. According to an embodiment, the first memory may be implemented as an N-line memory. For example, the N-line memory may be a memory having a capacity equivalent to 17 lines in the vertical direction, but is not limited thereto. For example, when a 1080p (1,920×1,080 resolution) Full HD image is input, only 17 lines of the image area of the Full HD image is stored in the first memory. As described above, the reason why the first memory is implemented as an N-line memory, and only some regions of the input image frames are stored for image processing is that the memory capacity of the first memory is limited according to hardware limitations. The second memory is a memory for storing at least one obtained texture patch 30, and the like, and may be implemented as a memory of various sizes according to various embodiments of the present disclosure. For example, according to an embodiment of the present disclosure, when all texture components corresponding to each pixel value of the input image are acquired and stored, and then applied to the input image, the second memory is equal to or equal to the size of the input image. It can be implemented in a large size. According to another embodiment, when a texture component is applied in an image unit corresponding to the size of the first memory, or a texture component obtained in a pixel line unit is applied in a pixel line unit, it is implemented with an appropriate size for the image processing. It could be. On the other hand, it goes without saying that the second memory may mean a memory area allocated to the learning network model among the entire area of the memory 110.

The third memory is a memory for storing an image-processed output image by applying the obtained texture component, and may be implemented as a memory having various sizes according to various embodiments of the present disclosure. For example, according to an embodiment of the present disclosure, when all texture components corresponding to each pixel value of the input image 10 are applied to obtain and display an output image, the third memory It can be implemented in the same or larger size. According to another embodiment, when an image is output in an image unit corresponding to the size of the first memory or a line unit corresponding to a patch size is output, the image may be implemented in an appropriate size for storing the image.

However, when the output image is overwritten in the first memory or the second memory, or the output image is not stored and is directly displayed, the third memory may not be required.

The input unit 130 receives various types of content, for example, an image signal. For example, the input unit 130 is AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Bluetooth, Zigbee, wired/wireless LAN (Local Area Network), WAN, Ethernet, IEEE 1394, HDMI. (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB ( D-subminiature), through a communication method such as DVI (Digital Visual Interface), from an external device (e.g., a source device), an external storage medium (e.g., USB), an external server (e.g., web hard), etc. Video signals can be input by streaming or downloading. Here, the image signal may be a digital signal, but is not limited thereto.

The display 140 includes a liquid crystal display (LCD), an organic light-emitting diode (OLED), a light-emitting diode (ED), a micro LED, liquid crystal on silicon (LCoS), digital light processing (DLP), and quantum dot) It can be implemented in various forms such as a display panel.

The output unit 150 outputs an acoustic signal.

For example, the output unit 150 may convert a digital sound signal processed by the processor 120 into an analog sound signal, amplify it, and output it. For example, the output unit 150 may include at least one speaker unit, a D/A converter, an audio amplifier, and the like capable of outputting at least one channel. For example, the output unit 150 may include an L-channel speaker and an R-channel speaker for reproducing L and R channels, respectively. However, the present invention is not limited thereto, and the output unit 150 may be implemented in various forms. As another example, the output unit 150 may be implemented in the form of a sound bar that reproduces an L channel, an R channel, and a center channel.

The user interface 160 may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen or a remote control receiver capable of performing the above-described display function and manipulation input function. Here, the button may be various types of buttons such as a mechanical button, a touch pad, a wheel, etc. formed in an arbitrary area such as a front portion, a side portion, or a rear portion of the main body of the image processing apparatus 100.

Meanwhile, although not shown in FIG. 11, it is possible to additionally apply pre-filtering to remove noise from an input image before image processing according to an exemplary embodiment of the present disclosure. For example, noticeable noise may be removed by applying a smoothing filter such as a Gaussian filter, a guided filter that filters an input image against a preset guidance, and the like.

12 is a block diagram illustrating a configuration of an image processing apparatus for learning and using a learning network model according to an embodiment of the present disclosure.

Referring to FIG. 12, the processor 120 may include at least one of a learning unit 1210 and a recognition unit 1220. The processor 120 of FIG. 12 may correspond to the processor 140 of the image processing apparatus 100 of FIG. 2 or a processor of a data learning server (not shown).

The learning unit 1210 generates a recognition model having a criterion for identifying a class of the pixel block 20 and a recognition model having a criterion for acquiring a texture patch 30 corresponding to the pixel block 20 according to the class. Or you can learn. The learning unit 1210 may generate a recognition model having a determination criterion by using the collected training data.

For example, the learning unit 1210 may generate, learn, or update a recognition model for determining a class corresponding to the pixel block 20 by using the pixel block 20 included in the image as training data.

As another example, the learning unit 1210 generates, learns, or generates a recognition model that determines whether the texture patch 30 is updated by comparing the similarity between the pixel block 20 and the class and the similarity between the texture patch 30 and the class. Can be updated.

The recognition unit 1220 may use predetermined data (eg, an input image) as input data of the learned recognition model to estimate a recognition object or situation included in the predetermined data.

For example, the recognition unit 1220 may use the pixel block 20 of the input image 10 as input data of the learned recognition model to identify the class and texture patch 30 of the corresponding pixel block 20. .

At least a portion of the learning unit 1210 and at least a portion of the recognition unit 1220 may be implemented as a software module or manufactured in the form of at least one hardware chip and mounted on the image processing apparatus. For example, at least one of the learning unit 1210 and the recognition unit 1220 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU or application). processor) or a graphics dedicated processor (eg, a GPU) and mounted on the aforementioned various image processing devices or object recognition devices. At this time, the dedicated hardware chip for artificial intelligence is a dedicated processor specialized in probability calculation, and has higher parallel processing performance than conventional general-purpose processors, so it can quickly process computation tasks in the field of artificial intelligence such as machine learning. When the learning unit 1210 and the recognition unit 1220 are implemented as a software module (or a program module including an instruction), the software module is a computer-readable non-transitory readable recording medium (non-transitory). transitory computer readable media). In this case, the software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, some of the software modules may be provided by an operating system (OS), and some of the software modules may be provided by a predetermined application.

In this case, the learning unit 1210 and the recognition unit 1220 may be mounted on one image processing device, or may be mounted on separate image processing devices, respectively. For example, one of the learning unit 1210 and the recognition unit 1220 may be included in the image processing apparatus 100, and the other may be included in an external server. In addition, the learning unit 1210 and the recognition unit 1220 may provide model information built by the learning unit 1210 to the recognition unit 1220 through wired or wireless, or input to the learning unit 1220. Data may be provided to the learning unit 1210 as additional learning data.

13 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.

According to the image processing method illustrated in FIG. 13, a texture patch corresponding to a pixel block included in the input image is obtained by applying an input image to a learning network model (S1310).

Subsequently, an output image is obtained by applying the obtained texture patch to the pixel block (S1320).

Here, the learning network model stores texture patches corresponding to each of a plurality of classes classified based on characteristics of an image, and may learn a texture patch corresponding to each of the plurality of classes based on an input image.

In addition, the learning network model according to an embodiment identifies one of a plurality of classes based on a characteristic of a pixel block, outputs a texture patch corresponding to the identified class, and a first degree of similarity between the pixel block and the identified class. And comparing a second degree of similarity between the texture patch and the identified class to identify whether to update the texture patch.

The learning network model according to an embodiment may replace a texture patch corresponding to a class identified based on the first and second similarities with a pixel block, or may add a pixel block as a texture patch corresponding to the identified class. .

In addition, the learning network model maintains a texture patch corresponding to the identified class if the first similarity is less than the second similarity based on the comparison result, and the first similarity is greater than the second similarity based on the comparison result. If it is a value, the texture patch can be updated based on the pixel block.

In the learning network model according to an embodiment of the present disclosure, when there are a plurality of texture patches corresponding to an identified class, any one of a plurality of texture patches is based on a correlation between a pixel block and a plurality of texture patches. Can be identified.

In addition, the learning network model may learn a texture patch based on at least one of a storage timing of a texture patch corresponding to each of a plurality of classes or an application frequency of a texture patch.

In the learning network model according to an embodiment of the present disclosure, when a pixel block is identified as not corresponding to any one of a plurality of classes based on a characteristic of a pixel block, a new class is generated based on the characteristic of the pixel block and a new class is generated. You can map and store a block of pixels in a class.

Here, the plurality of classes may be classified based on at least one of an average pixel value, pixel coordinates, variance, edge intensity, edge direction, or color.

In step S1320 of obtaining an output image according to an embodiment of the present disclosure, a step of acquiring a weight for a texture patch based on a correlation between the acquired texture patch and a pixel block, and a texture patch to which the weight is applied are pixelated. It may include applying to the block to obtain an output image.

Here, the output image may be a 4K Ultra High Definition (UHD) image or an 8K UHD image.

However, it goes without saying that the various embodiments of the present disclosure may be applied not only to an image processing device, but also to all electronic devices capable of image processing, such as an image receiving device such as a set-top box and an image processing device.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor 120 itself. According to software implementation, embodiments such as procedures and functions described in the present specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing the processing operation of the sound output device 100 according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. I can. When the computer instructions stored in the non-transitory computer-readable medium are executed by the processor of the specific device, the specific device causes the specific device to perform the processing operation in the sound output device 100 according to the various embodiments described above.

The non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as a register, cache, and memory. Specific examples of the non-transitory computer-readable medium may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is not departing from the gist of the disclosure claimed in the claims. Various modifications are possible by those skilled in the art of course, and these modifications should not be individually understood from the technical idea or perspective of the present disclosure.

110: memory 120: processor

Claims (20)

A memory storing at least one instruction; And
Including; a processor electrically connected to the memory,
The processor,
By executing the above command,
Apply the input image to the training network model,
Obtaining a texture patch corresponding to the pixel block included in the input image from the learning network model,
Applying the texture patch to the pixel block to obtain an output image,
The learning network model,
Storing a plurality of texture patches corresponding to each of a plurality of classes classified based on the characteristics of the pixel block
Identifying any one of the plurality of classes based on the characteristic of the pixel block included in the input image,
Outputs the texture patch corresponding to the identified class,
Updating at least one texture patch among the plurality of texture patches based on the input image. The method of claim 1,
The learning network model,
Comparing a first similarity between the pixel block included in the input image and the identified class and a second similarity between the texture patch and the identified class to identify whether to update the texture patch. The method of claim 2,
The learning network model,
The image processing apparatus comprising: replacing a texture patch corresponding to the identified class with the pixel block based on the first and second similarities, or adding the pixel block as a texture patch corresponding to the identified class. The method of claim 2,
The learning network model,
If the first similarity is a value smaller than the second similarity based on the comparison result, the texture patch corresponding to the identified class is maintained, and
If the first similarity is greater than the second similarity based on the comparison result, the texture patch is updated based on the pixel block. The method of claim 2,
The learning network model,
When there are a plurality of texture patches corresponding to the identified class, any one of the plurality of texture patches is identified based on a correlation between the pixel block and the plurality of texture patches. The method of claim 1,
The learning network model,
An image processing apparatus for learning the texture patch based on at least one of a storage timing of a texture patch corresponding to each of the plurality of classes or an application frequency of the texture patch. The method of claim 1,
The learning network model,
If the pixel block is identified as not corresponding to any one of the plurality of classes based on the characteristic of the pixel block included in the input image, a new class is selected based on the characteristic of the pixel block included in the input image. An image processing apparatus for generating and storing the pixel block by mapping the pixel block to the new class. The method of claim 1,
The learning network model,
Identifying a class corresponding to each of a plurality of pixel blocks included in the input image,
The image processing apparatus, wherein the size of the storage space of the memory corresponding to at least one of the plurality of classes is changed based on the identification frequency of each of the plurality of classes. The method of claim 8,
The learning network model,
The image processing apparatus, wherein a texture patch corresponding to a class identified less than a preset number of times based on the identification frequency is deleted from the memory, and a storage space reserved according to the deletion of the texture patch is allocated to the remaining classes. The method of claim 1,
The plurality of classes,
An image processing device that is classified based on at least one of an average pixel value, pixel coordinates, variance, edge intensity, edge direction, or color. The method of claim 1,
The processor,
Obtaining a weight for the texture patch based on a correlation between the obtained texture patch and the pixel block,
An image processing apparatus for obtaining the output image by applying the texture patch to which the weight is applied to the pixel block. The method of claim 1,
The output image,
An image processing device that is a 4K UHD (Ultra High Definition) video or an 8K UHD video. In the image processing method of the image processing apparatus,
Applying an input image to a learning network model to obtain a texture patch corresponding to a pixel block included in the input image from the learning network model;
Including; applying the texture patch to the pixel block to obtain an output image; Including,
The learning network model,
A plurality of texture patches corresponding to each of the plurality of classes classified based on the characteristics of the pixel block are stored,
Identifying any one of the plurality of classes based on the characteristic of the pixel block included in the input image,
Outputs the texture patch corresponding to the identified class,
Updating at least one texture patch among the plurality of texture patches based on the input image. The method of claim 13,
The learning network model,
Comparing a first similarity between the pixel block included in the input image and the identified class and a second similarity between the texture patch and the identified class to identify whether to update the texture patch. The method of claim 14,
The learning network model,
Replacing a texture patch corresponding to the identified class with the pixel block to the first and second similarity degrees, or adding the pixel block as a texture patch corresponding to the identified class. The method of claim 14,
The learning network model,
If the first similarity is a value smaller than the second similarity based on the comparison result, the texture patch corresponding to the identified class is maintained, and
If the first similarity is greater than the second similarity based on the comparison result, the texture patch is updated based on the pixel block. The method of claim 14,
The learning network model,
When there are a plurality of texture patches corresponding to the identified class, any one of the plurality of texture patches is identified based on a correlation between the pixel block and each of the plurality of texture patches. The method of claim 13,
The learning network model,
An image processing method for learning the texture patch based on at least one of a storage timing of a texture patch corresponding to each of the plurality of classes or an application frequency of the texture patch. The method of claim 13,
The learning network model,
If the pixel block is identified as not corresponding to any one of the plurality of classes based on the characteristic of the pixel block included in the input image, a new class is selected based on the characteristic of the pixel block included in the input image. The image processing method of generating and mapping the pixel block to the new class and storing it. The method of claim 13,
The plurality of classes,
An image processing method that is classified based on at least one of an average pixel value, pixel coordinates, variance, edge intensity, edge direction, or color.

Images