KR20220067463A - Electronic apparatus and controlling method thereof - Google Patents

Abstract

Disclosed is an electronic device dividing an input image into a plurality of areas and converting the divided area based on different style information. The electronic device includes: a display; a memory storing a learned artificial intelligence model; and a processor obtaining a high-resolution output image having critical resolution or greater from a low-resolution input image under the critical resolution by using the learned artificial intelligence model. The processor also controls the display to display the obtained high-resolution image. The processor identifies a first area including first property information in the low-resolution input image and a second area including second property information. The processor obtains a first image and a second image. Test processing of a first intensity is performed on the first area in the first image. The test processing of a second intensity different from the first intensity is performed on the second area in the second image. The processor can individually scale up the first image and the second image. Also, the processor obtains a high-resolution image by combining the first image and the second image scaled up.

Description

ELECTRONIC APPARATUS AND CONTROLLING METHOD THEREOF

The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for changing resolution by dividing an input image into a plurality of regions, and a method for controlling the same.

An artificial intelligence system is a computer system that implements human-level intelligence. It is a system in which a machine learns and judges on its own, and the recognition rate improves the more it is used.

Artificial intelligence technology consists of machine learning (deep learning) technology that uses an algorithm that classifies/learns the characteristics of input data by itself, and element technology that uses machine learning algorithms to simulate functions such as cognition and judgment of the human brain.

Element technologies are, for example, linguistic understanding technology for recognizing human language/text, visual understanding technology for recognizing objects as from human vision, reasoning/prediction technology for logically reasoning and predicting by judging information, and human experience information It may include at least one of a knowledge expression technology that processes the data as knowledge data, an autonomous driving of a vehicle, and a motion control technology that controls the movement of a robot.

Artificial intelligence models can be used to transform low-resolution images into high-resolution images.

An artificial intelligence model trained based on perceptual loss and adversarial loss may be used to generate a super-resolution image. Artificial intelligence models using cognitive loss and adversarial loss can improve the overall visual quality of reconstructions. In particular, when an image conversion operation is performed using a generative adversarial network (GAN), it may be used to generate a high-resolution image.

However, there may be a problem that an unnatural region cannot be transformed in a state where re-learning is not performed. Also, there is a limitation in switching the continuous image effect. In addition, the same image conversion method is applied to the entire area, and an image conversion method is not different for each area, thereby generating an unnatural image.

In addition, the general image transformation model has a problem that the result is monotonous and unnatural because it calculates the cognitive loss for the entire area of the image in the same functional space.

The present disclosure has been devised to improve the above problems, and an object of the present disclosure is to provide an electronic device that divides an input image into a plurality of regions and converts the divided regions based on different style information, and a control method thereof .

An electronic device according to an embodiment of the present invention for achieving the above object is a high-resolution output image above the threshold resolution from a low-resolution input image less than a threshold resolution using a display, a memory in which a learned artificial intelligence model is stored, and the learned artificial intelligence model and a processor capable of obtaining 2 regions are identified, and a first image in which texturing of a first intensity is performed on the first region and a second image in which texturing of a second intensity different from the first intensity is performed on the second region is obtained and upscaling the first image and the second image, respectively, and combining the upscaled first and second images to obtain the high-resolution image.

Meanwhile, the first characteristic information may include object region information and the second characteristic information may include background region information, and the processor identifies the region including the object region information as the first region. and an area including the background area information may be identified as the second area.

Meanwhile, if the type of the object included in the first region is the first type, the processor may perform texture processing with an intensity corresponding to the first type on the first region, and If the object type is the second type, texture processing with an intensity corresponding to the second type may be performed on the first area.

Meanwhile, the processor may input the low-resolution input image into a first artificial intelligence model to obtain identification information for the first region, the first intensity, identification information for the second region, and the second intensity, and , the first artificial intelligence model may be trained to output identification information for regions having different characteristics in the input image and texture intensity for each identified region when an image is input.

Meanwhile, the identification information on the first region may include coordinate information on the first region included in the low-resolution input image, and the identification information on the second region includes the second region included in the low-resolution input image. 2 may include coordinate information for the region.

On the other hand, the processor inputs an image including identification information for the first region and the second region, the first intensity, and the second intensity to a second artificial intelligence model, and receives the texture-processed and upscaling second image. First and second images may be acquired.

Meanwhile, the second artificial intelligence model may be implemented to include a residual-in-residual density block (RRDB) that performs partial feature transformation (SFT).

Meanwhile, the second artificial intelligence model may include a layer having a plurality of parameters, an image identified as a plurality of regions having different characteristics, and a plurality of images obtained based on texture intensity corresponding to the plurality of regions may be compared with a reference image to calculate a loss value, and the plurality of parameters may be learned based on the calculated loss value.

Meanwhile, the second artificial intelligence model may calculate the loss value based on at least one of a reconstruction loss, an adversarial loss, and a perceptual loss.

Meanwhile, the second artificial intelligence model may be implemented as a Generative Adversarial Network (GAN).

According to an embodiment of the present disclosure, a control method of an electronic device for storing an artificial intelligence model trained to acquire a high-resolution output image equal to or greater than the threshold resolution from a low-resolution input image less than a threshold resolution includes first characteristic information from the low-resolution input image identifying a first area including a first area and a second area including second characteristic information; obtaining a second image subjected to texturing of a second intensity different from the intensity, upscaling the first image and the second image, respectively, and combining the upscaled first and second images to obtain the and acquiring a high-resolution image.

Meanwhile, the first characteristic information may include object region information and the second characteristic information may include background region information, and the step of identifying the first region and the second region includes the object region information. An area including the area may be identified as the first area, and an area including the background area information may be identified as the second area.

Meanwhile, in the acquiring of the first image and the second image, if the type of the object included in the first region is the first type, texture processing with an intensity corresponding to the first type is performed on the first region. If the type of the object included in the first region is the second type, texture processing with an intensity corresponding to the second type may be performed on the first region.

On the other hand, the step of inputting the low-resolution input image into a first artificial intelligence model to obtain identification information for the first region, the first intensity, identification information for the second region, and the second intensity and the first artificial intelligence model is trained to output identification information for regions having different characteristics in the input image and texture intensity for each identified region when an image is input.

Meanwhile, the identification information on the first region may include coordinate information on the first region included in the low-resolution input image, and the identification information on the second region includes the second region included in the low-resolution input image. 2 may include coordinate information for the region.

Meanwhile, the steps of obtaining the first image and the second image and upscaling the first image and the second image, respectively, include an image including identification information for the first area and the second area , by inputting the first intensity and the second intensity into a second artificial intelligence model, the first and second images subjected to the texture processing and upscaling may be obtained.

Meanwhile, the second artificial intelligence model may be implemented to include a residual-in-residual density block (RRDB) that performs partial feature transformation (SFT).

Meanwhile, the second artificial intelligence model may include a layer having a plurality of parameters, an image identified as a plurality of regions having different characteristics, and a plurality of images obtained based on texture intensity corresponding to the plurality of regions may be compared with a reference image to calculate a loss value, and the plurality of parameters may be learned based on the calculated loss value.

Meanwhile, the second artificial intelligence model may calculate the loss value based on at least one of a reconstruction loss, an adversarial loss, and a perceptual loss.

Meanwhile, the second artificial intelligence model may be implemented as a Generative Adversarial Network (GAN).

1 is a diagram for describing an image conversion operation according to an exemplary embodiment.
2 is a diagram for describing an image conversion operation according to another exemplary embodiment.
3 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.
FIG. 4 is a block diagram illustrating a detailed configuration of the electronic device of FIG. 3 .
5 is a flowchart illustrating an image conversion operation according to an exemplary embodiment.
6 is a flowchart for describing the image conversion operation of FIG. 5 in detail.
7 is a diagram for explaining an operation of dividing an input image into a plurality of regions.
8 is a flowchart illustrating an operation in which an image transformation model is learned.
9 is a view for explaining an image conversion model according to an embodiment of the present disclosure.
10 is a diagram for explaining a calculation used in an image conversion operation.
11 is a diagram for explaining a calculation used for a learning operation according to an embodiment of the present disclosure.
12 is a view for specifically explaining an image conversion model according to another embodiment of the present disclosure.
13 is a diagram for describing a basic block used in an image transformation model according to an embodiment of the present disclosure.
14 is a view for explaining a style identification model according to an embodiment of the present disclosure.
15 is a diagram for explaining a calculation used for a learning operation according to another embodiment of the present disclosure.
16 is a diagram for explaining an effect of an image conversion operation of an electronic device according to various embodiments of the present disclosure;
17 is a diagram for comparing image conversion operations of an electronic device according to an exemplary embodiment.
18 is a diagram for comparing image conversion operations of an electronic device according to another exemplary embodiment.
19 is a diagram for comparing image conversion operations of an electronic device according to another exemplary embodiment.
20 is a diagram for explaining a method of controlling an electronic device according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

Terms used in the embodiments of the present disclosure are selected as currently widely used general terms as possible while considering the functions in the present disclosure, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

The expression "at least one of A and/or B" is to be understood as indicating either "A" or "B" or "A and B".

As used herein, expressions such as "first," "second," "first," or "second," can modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it should be understood that an element may be directly connected to another element or may be connected through another element (eg, a third element).

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "consisting of" are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification exist, and are intended to indicate that one or more other It should be understood that this does not preclude the possibility of addition or presence of features or numbers, steps, operations, components, parts, or combinations thereof.

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “units” are integrated into at least one module and implemented with at least one processor (not shown) except for “modules” or “units” that need to be implemented with specific hardware. can be

In this specification, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

Hereinafter, an embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings.

Functions related to artificial intelligence according to the present disclosure are operated through a processor and a memory. The processor may consist of one or a plurality of processors. At this time, one or more processors are general-purpose processors such as CPUs, APs, Digital Signal Processors (DSPs), etc., graphics-only processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-only processors such as NPUs (Neural Network Processing Units). can be One or a plurality of processors control to process input data according to a predefined operation rule or artificial intelligence model stored in the memory. Alternatively, when one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

A predefined action rule or artificial intelligence model is characterized in that it is created through learning. Here, being made through learning means that a basic artificial intelligence model is learned using a plurality of learning data by a learning algorithm, so that a predefined action rule or artificial intelligence model set to perform a desired characteristic (or purpose) is created means burden. Such learning may be performed in the device itself on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between the operation result of a previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), There may be a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Networks, but is not limited thereto.

Meanwhile, knowledge expression is a technology for automatically processing human experience information into knowledge data, and may include knowledge construction (data generation/classification), knowledge management (data utilization), and the like.

1 is a diagram for describing an image conversion operation according to an exemplary embodiment.

Referring to FIG. 1 , the image conversion model 101 may receive an input image 10 and generate an output image 20 . Here, the input image 10 may mean a low-resolution image, and the output image 20 may mean a high-resolution image. Here, the image conversion operation performed by the image conversion model 101 may be image super resolution. Here, the image conversion model 101 may correspond to a super-resolution network.

Here, the image conversion model 101 may perform a super-resolution operation using the style identification model 102 . Here, the style identification model 102 may determine a style applied to an image conversion operation. Specifically, the style identification model 102 may include a plurality of style information. In addition, the style identification model 102 may determine a style corresponding to the input image 10 .

Here, the style information may mean a style map. Here, the style map may be described as a region map, a control map, or the like.

The image conversion model 101 may transmit information on the input image 10 to the style identification model 102 , and the style identification model 102 may transmit information about the input image 10 to the input image ( 10) can identify a suitable style. In addition, the style identification model 102 may transmit the identified style to the image conversion model 101 .

Here, the image conversion model 101 may super-resolution the input image 10 into the output image 20 based on the style identified in the style identification model 102 .

2 is a diagram for describing an image conversion operation according to another exemplary embodiment.

Referring to FIG. 2 , the image conversion model 101 may receive an input image 10 and generate an output image 20 . Here, the image transformation model 101 may include the style identification model 102 . Here, the style identification model 102 may determine a style corresponding to the input image 10 . Specifically, the style identification model 102 may identify a style suitable for the input image 10 , and the image conversion model 101 converts the input image 10 into the output image 20 based on the identified style. can be resolved.

3 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 3 , the electronic device 100 may include a memory 110 and a processor 120 .

The electronic device 100 according to various embodiments of the present specification may include, for example, at least one of a smartphone, a tablet PC, a mobile phone, a desktop PC, a laptop PC, a PDA, and a portable multimedia player (PMP). . In some embodiments, the electronic device 100 may include, for example, at least one of a television, a digital video disk (DVD) player, and a media box (eg, Samsung HomeSync™, Apple TV™, or Google TV™).

The memory 110 is implemented as an internal memory such as a ROM (eg, electrically erasable programmable read-only memory (EEPROM)) included in the processor 120, a RAM, or the like, or with the processor 120 It may be implemented as a separate memory. In this case, the memory 110 may be implemented in the form of a memory embedded in the electronic device 100 or may be implemented in the form of a memory detachable from the electronic device 100 depending on the purpose of data storage. For example, data for driving the electronic device 100 is stored in a memory embedded in the electronic device 100 , and data for an extended function of the electronic device 100 is detachable from the electronic device 100 . It can be stored in any available memory.

Meanwhile, in the case of a memory embedded in the electronic device 100 , a volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory ( Examples: one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (such as NAND flash or NOR flash, etc.) ), a hard drive, or a solid state drive (SSD), and in the case of a removable memory in the electronic device 100, a memory card (eg, compact flash (CF), SD ( secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory that can be connected to the USB port (e.g., USB memory) and the like.

The processor 120 may perform an overall control operation of the electronic device 100 . Specifically, the processor 120 functions to control the overall operation of the electronic device 100 .

The processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals, but is not limited thereto, and the central processing unit ( central processing unit (CPU), micro controller unit (MCU), micro processing unit (MPU), controller, application processor (AP), graphics-processing unit (GPU) or communication processor (CP)), may include one or more of an ARM processor, or may be defined by a corresponding term In addition, the processor 120 is a SoC (System on Chip) or LSI (large scale integration) in which a processing algorithm is embedded. It may be implemented in the form of a field programmable gate array (FPGA), and the processor 120 may perform various functions by executing computer executable instructions stored in the memory 110. have.

The processor 120 may control the display 140 to obtain a high-resolution output image having a resolution higher than or equal to a threshold resolution from a low-resolution input image less than the threshold resolution by using the learned artificial intelligence model, and display the obtained high-resolution image. Then, the processor 120 identifies a first region including the first characteristic information and a second region including the second characteristic information in the low-resolution input image, and performs texture processing of a first intensity on the first region. acquiring a second image to which texturing of a second intensity different from the first intensity is performed on the first image and the second region, upscaling the first image and the second image, respectively, and performing the upscaled first and second images The two images can be combined to obtain a high-resolution image.

Here, the artificial intelligence model may be a model that converts a low-resolution image less than the threshold resolution into a high-resolution image greater than or equal to the threshold resolution. Here, the threshold resolution can be changed by a user's setting. For example, the processor 120 may change the input image having the first resolution to the second resolution having a size larger than the first resolution. Here, the processor 120 may control the display 140 to display the generated high-resolution image.

According to an embodiment, the artificial intelligence model may be stored in the memory 110 of the electronic device 100 . Accordingly, the processor 120 may convert the low-resolution image into a high-resolution image by using the artificial intelligence model stored in the memory 110 .

According to another embodiment, the artificial intelligence model may be stored in an external server. Accordingly, the processor 120 may transmit the low-resolution image to the external server through the communication interface 130 . Here, the external server may convert the received low-resolution image into a high-resolution image, and transmit the converted high-resolution image back to the electronic device 100 . The processor 120 may receive a high-resolution image from an external server through the communication interface 130 . In addition, the processor 120 may control the display 140 to display the received high-resolution image.

Here, the processor 120 may identify (or classify) the first region and the second region in the low-resolution input image. Here, the first region may mean a region in which the first characteristic information is identified in the entire region of the low-resolution input image, and the second region may refer to an region in which the second characteristic information is identified in the entire region of the low-resolution input image. have.

Meanwhile, the first characteristic information may include object region information and the second characteristic information may include background region information, identifying an area including the object region information as the first region and generating a region including the background region information. Two areas can be identified.

For example, the processor 120 may identify the object region and the background region from the entire region of the low-resolution input image. Here, the object area may mean an area including people, things, and the like. Also, the object area may mean an area requiring a lot of texture processing. Here, the background area may mean an area including the sky, clouds, sea, and the like. Also, the background area may mean an area that does not require much texture processing.

Here, the characteristic information may include object area information or background area information. Here, the object region information may mean a characteristic indicating that an object is included, and the background region information may indicate a characteristic indicating that the background is included.

Here, the processor 120 may acquire characteristic information based on the low-resolution input image. Specifically, the processor 120 may analyze the low-resolution input image to determine characteristic information for each location. In addition, the processor 120 may divide the entire region of the low-resolution input image into a plurality of regions based on the determined characteristic information.

For example, the processor 120 may analyze the low-resolution image to obtain a position having an object characteristic and a position having a background characteristic according to a position. In addition, the processor 120 may identify the plurality of regions based on the obtained position having the object characteristic and the position having the background characteristic. Specifically, the processor 120 may group positions having object characteristics to determine one region, and group positions having background characteristics to determine one region.

Here, although an embodiment in which two regions are divided for convenience of description has been described, the region may be divided into three or more regions instead of two regions according to user settings.

Meanwhile, if the type of the object included in the first area is the first type, the processor 120 performs texture processing with an intensity corresponding to the first type on the first area, and the type of the object included in the first area is In the case of the second type, texture processing with an intensity corresponding to the second type may be performed on the first area.

Here, the processor 120 may divide the object region into a plurality of regions based on the object type in the region having object characteristics. In addition, when regions having different object types are identified, the processor 120 may perform texture processing of different intensity according to the object type.

For example, the processor 120 may distinguish an area including a person from an area including a building in the object area. Then, texture processing of the intensity corresponding to the person (eg, level 3) is performed on the area including the person, and the texture of the intensity (eg, level 5) corresponding to the building is performed on the area including the building. processing can be performed.

Here, strength information such as the first strength and the second strength may be described as style information. A different intensity for texture processing may mean a different style. The style information may include information indicating which processing to be performed on a specific region. For example, the first style information may include information indicating that texture processing of a first intensity should be performed, and the second style information may include information indicating that texture processing of a second intensity should be performed. The processor 120 may acquire the first style information and the second style information based on each of the first characteristic information corresponding to the first region and the second characteristic information corresponding to the second region included in the input image 10 . and inputting the first style information and the second style information into the artificial intelligence model, respectively, to obtain a high-resolution image corresponding to the first region and a high-resolution image corresponding to the second region, and a high-resolution image corresponding to the first region The output image 20 may be acquired based on the image and the high-resolution image corresponding to the second region.

Here, the processor 120 may acquire the input image 10 . Here, the input image 10 may mean a low-resolution image.

Also, the processor 120 may divide (or identify) the entire region corresponding to the acquired input image 10 into a plurality of regions. Specifically, the processor 120 may analyze the input image 10 to determine whether the characteristics are different for each region. This is because, if characteristics of a plurality of regions included in the input image 10 are different, it is necessary to perform a high-resolution conversion operation in a different manner. Accordingly, the processor 120 may distinguish a plurality of regions having different characteristics included in the input image 10 .

Specifically, the processor 120 may identify an object in the input image 10 , and may classify a plurality of regions based on the identified object.

Meanwhile, the processor 120 may identify the first object and the second object included in the input image 10 , and based on the position of the first object and the position of the second object, the first object including the first object The region and the second region including the second object may be divided. Here, the characteristic information may include a characteristic corresponding to the divided (or identified) area.

Meanwhile, the first characteristic information may include characteristic information of the first object, and the second characteristic information may include characteristic information of the second object.

For example, the characteristic information may include the type and name of the object. When the first object is a background object, the first characteristic information may be a background characteristic, and may additionally include a specific background such as a sky background, a forest background, a building background, and a ground background.

Also, if the second object is a thing object, the second characteristic information may be a thing characteristic. If the second object is an object object, the second characteristic information may be an object characteristic, and may additionally mean a human object, an animal object, a building object, a metal surface object, or the like.

Meanwhile, the first style information may include information for transforming an image corresponding to the first region based on a first level of a predetermined image style, and the second style information includes a first level and a predetermined image style. Information for transforming an image corresponding to the second region based on a different second level may be included.

Here, the predetermined image style may mean a style to be additionally considered while the artificial intelligence model generates a high-resolution image. Here, the artificial intelligence model may generate a high-resolution image based on at least one style among a plurality of image styles. For example, the predetermined image style may mean a style reflecting a texture effect.

In addition, the predetermined image style may be reflected in the image conversion operation in a plurality of levels. Here, the level may mean information indicating the degree of reflection of a predetermined image style. For example, a style reflecting a texture effect may be reflected in an image conversion operation in three levels. Here, the first level is a level that minimally reflects the texture effect or does not reflect the texture effect at all, the second level is a level that reflects the texture effect to an intermediate degree, and the third level is a level that reflects the texture effect to the maximum. can Accordingly, the processor 120 may identify a level corresponding to the acquired characteristic information, and perform an image conversion operation based on the identified level.

The style information may include a predetermined image style or level information of a predetermined image style. Here, when a level corresponding to the predetermined image style does not exist, the style information may not include level information. In addition, level information may already be reflected in the predetermined image style itself. For example, the style information includes at least one of a first style reflecting a texture effect of a first level, a second style reflecting a texture effect of a second level, and a third style reflecting a texture effect of a third level can do.

Meanwhile, the artificial intelligence model may include a style identification model 102 including a plurality of image styles and an image conversion model 101 for converting an input image 10 into an output image 20 .

Meanwhile, the processor 120 inputs the low-resolution input image to the first artificial intelligence model to provide identification information for the first region, a first intensity that is a texture intensity to process the first region, identification information for the second region, and the second A second intensity, which is the texture intensity to process the region, is acquired, and when an image is input, the first artificial intelligence model learns to output identification information for regions with different characteristics in the input image and texture intensity for each identified region. can

Here, the first artificial intelligence model may be the style identification model 102 of FIG. 12 . Here, the first artificial intelligence model may receive an image and output a region and a texture intensity corresponding to the region. Specifically, in the first AI model, the image may be divided into a plurality of regions, a characteristic corresponding to the divided region may be acquired, and a texture intensity corresponding to the acquired characteristic may be acquired. For example, the first artificial intelligence model divides the image into a first region and a second region, and determines that the first region needs texturing of a first intensity and that the second region needs texturing of a second intensity. can

Here, the identification information on the first region may mean location information or coordinate information of the first region. Here, the identification information on the second region may mean location information or coordinate information of the second region.

Meanwhile, the identification information for the first region includes coordinate information for the first region included in the low-resolution input image, and the identification information for the second region includes coordinate information for the second region included in the low-resolution input image. can do.

Here, the coordinate information may mean location information on an image.

On the other hand, the processor 120 inputs the image including the identification information for the first region and the second region, the first intensity, and the second intensity to the second artificial intelligence model, and the first and second regions subjected to texture processing and upscaling. 2 images can be acquired.

Here, the processor 120 may input identification information for each of the plurality of regions and texture intensity information for each of the plurality of regions to the second artificial intelligence model. Here, the second artificial intelligence model may be the image transformation model 101 of FIG. 12 .

Here, the second artificial intelligence model may acquire the first image by performing texture processing of the first intensity on the first region. In addition, the second artificial intelligence model may obtain a second image by performing texture processing of a second intensity on the second region.

Here, the second artificial intelligence model may acquire the upscaled first image by upscaling the acquired first image. In addition, the second artificial intelligence model may acquire an upscaled second image by upscaling the acquired second image.

Meanwhile, the second artificial intelligence model may be implemented to include a residual-in-residual density block (RRDB) that performs partial feature transformation (SFT). Specifically, the image transformation model 101 included in the artificial intelligence model may use the RRDB including the SFT layer as a basic block. A detailed description related thereto will be described later with reference to FIG. 13 .

Here, the image transformation model 101 and the style identification model 102 may be implemented as separate artificial intelligence models. By performing each model separately and performing separate operations, data processing speed can be improved or operation algorithms can be simplified.

Here, the image conversion model 101 may perform an operation of converting a low-resolution image into a high-resolution image, and may additionally consider the style information identified in the style identification model 102 in the conversion operation.

For example, in the style identification model 102 , the style information that reflects the texture effect to the minimum in the first area and the texture effect to the maximum in the second area is generated, and the generated style information is used in the image conversion model 101 . can be sent to Here, the image conversion model 101 converts the image corresponding to the first region into a high-resolution image while minimizing the texture effect based on the style information received from the style identification model 102 and corresponding to the second region. Images can be converted to high-resolution images while maximally reflecting the texture effect.

In addition, the processor 120 may obtain the first characteristic information and the second characteristic information based on the image transformation model 101 , and based on the style identification model 102 , the first style corresponding to the first characteristic information Second style information corresponding to the information and the second characteristic information may be acquired.

Meanwhile, the processor 120 may convert the first region into a low-resolution first image based on the first style information, and convert the second region into a low-resolution second image based on the second style information, A high-resolution image corresponding to the first area may be obtained by upscaling the first image, and a high-resolution image corresponding to the second area may be obtained by upscaling the second image.

Here, the processor 120 may perform an image conversion operation based on the image conversion model 101 , and may perform a low-resolution processing operation, an upscaling operation, and a high-resolution processing operation. Here, the low-resolution processing operation may refer to an operation of converting the input image 10 before the upscaling operation. Here, the low-resolution processing operation may be performed by the low-resolution processing unit 1201 of FIG. 12 . Here, the upscaling operation may be performed by the upscaling unit 1202 of FIG. 12 . Here, the high-resolution processing operation may be performed by the high-resolution processing unit 1203 of FIG. 12 .

Here, various embodiments using the image conversion model 101 and the style identification model 102 have been described in FIGS. 1 and 2 . In addition, a specific operation will be described later with reference to FIG. 12 .

Meanwhile, the above description was related to the operation of converting the input image 10 into the output image 20 . In the transformation operation, the image transformation model 101 and the style identification model 102 may be used, and the parameters used in the image transformation operation in the image transformation model 101 may be already learned final parameters. That is, the image conversion model 101 may have already completed learning based on the plurality of input images and the reference image corresponding to the plurality of input images. Based on the image conversion model 101 that has been trained, the electronic device 100 may perform an image conversion operation.

On the other hand, the second artificial intelligence model includes a layer having a plurality of parameters, an image identified as a plurality of regions having different characteristics, and a plurality of images obtained based on texture intensity corresponding to the plurality of regions with a reference image A loss value may be calculated by comparison, and a plurality of parameters may be learned based on the calculated loss value.

A process in which the image transformation model 101 is trained will be described. The artificial intelligence model may additionally use the loss value calculation model 103 for learning.

Specifically, the artificial intelligence model corresponds to a style identification model 102 including a plurality of image styles, an image transformation model 101 for converting an input image 10 into an output image 20 , and an image transformation model 101 . and a loss value calculation model 103 used in a learning operation to determine a plurality of parameters to be , the loss value calculation model 103 may acquire a loss corresponding to the training output image based on the style information, and the image conversion model 101 may determine a plurality of parameters based on the loss.

Meanwhile, the second artificial intelligence model may calculate a loss value based on at least one of a reconstruction loss, an adversarial loss, and a perceptual loss.

Meanwhile, the second artificial intelligence model may be implemented as a Generative Adversarial Network (GAN).

Here, the cognitive loss is obtained based on the style information, and the image transformation model 101 may determine a plurality of parameters based on at least one loss.

Here, the image used in the loss value acquisition operation may be the output image 20 generated by the image transformation model 101 and the reference image 30 corresponding to the input image 10 . Here, the reference image 30 may mean a high-resolution image of the input image 10 and may mean ground truth.

Here, the reconstruction loss may be information including a difference value between the output image 20 and the reference image 30 . A detailed calculation related to the reconstruction loss will be described later in Equation 1111 of FIG. 11 .

Here, the cognitive loss may be a perceptual loss, and is acquired after passing the training images (at least one of the input image 10, the output image 20, or the reference image 30) to another deep learning-based image classification network. It may mean a loss between the feature maps that become Here, the image-based classification network may mean an image transformation network corresponding to style information. For example, the image transformation model 101 may use the first image transformation network to reflect the first style information. Here, the loss value calculation model 103 may obtain a difference value between the output image 20 and the reference image 30 based on the first image transformation network. A detailed calculation process will be described later in Equations 1121 and 1122 of FIG. 11 .

Here, the adversarial loss may mean a loss obtained based on a Generative Adversarial Network (GAN). Here, the hostility loss may mean a loss obtained based on a difference between a probability value of determining a real image (reference image) as real and a probability value of determining a fake image (output image) as fake. A detailed calculation process will be described later in Equations 1131 , 1132 , and 1133 of FIG. 11 .

On the other hand, the loss value calculation model 103 may obtain a discriminator loss based on a discriminator included in the loss value calculation model 103 , and the image conversion model 101 is a discriminator A plurality of parameters may be determined based on the loss.

Here, the discriminator may refer to a classification network used in a Generative Adversarial Network (GAN). Here, the discriminator may perform an operation of determining a real image (reference image) as real and a fake image (output image) as fake.

Here, the discriminator loss may mean a loss obtained in the process of learning the discriminator's discriminator operation itself.

The adversarial loss and the discriminator loss can be common in that they are obtained in the probability of judging the real image (reference image) as real and the operation of judging the fake image (output image) as fake. However, there is a difference in that the adversarial loss is a loss obtained in the image conversion operation and the discriminator loss is a loss obtained in the discrimination operation. A detailed calculation process will be described later in Equations 1501, 1502, and 1503 of FIG. 15 .

Meanwhile, the electronic device 100 may optimize the image transformation model 101 by using a weighted sum of a plurality of perceptual losses to perform image transformation based on various style information.

Also, the electronic device 100 may generate an adaptive high-resolution image for each of a plurality of regions by using the image transformation model 101 using a conditional target function capable of generating a reconstruction style.

Meanwhile, there may be various embodiments of converting input data into an output image using an artificial intelligence model.

According to an embodiment, the electronic device 100 stores the artificial intelligence model, receives input data, obtains an output image corresponding to the received input data based on the artificial intelligence model, and provides the output image to the user. can provide Here, providing the output image to the user may mean displaying the image on a display or outputting audio through a speaker. For example, the TV (electronic device) may receive an input image, obtain an output image by changing the resolution of the received input image, and may directly display the obtained output image on the display of the TV (electronic device).

According to another embodiment, the electronic device 100 stores the artificial intelligence model, receives input data, obtains output data corresponding to the received input data based on the artificial intelligence model, and uses the obtained output data. It can be transmitted to an external device. Here, the external device may be a user terminal device such as a TV or a smart phone. The electronic device 100 is only a device for acquiring an output image, and an operation of substantially providing the output image to a user may be performed by an external device. For example, a TV (external device) may receive an input image and transmit the received input image to an artificial intelligence server (electronic device). The artificial intelligence server (electronic device) may convert the received input image into an output image, and transmit the converted output image back to the TV (external device).

According to another embodiment, the electronic device 100 and the external device may acquire output data together. Assume that a plurality of operations are required to obtain the output data. Here, some of the plurality of operations may be performed by the electronic device 100 , and others of the plurality of operations may be performed by an external device. Accordingly, in order to obtain the output data, not only the electronic device 100 storing the artificial intelligence model but also an external device may be required. For example, a TV (external device) may receive an input image and divide the received input image into a plurality of regions. In addition, the TV (external device) may acquire style information corresponding to each of a plurality of divided regions. In addition, the TV (external device) may transmit the input image and style information corresponding to each of the plurality of regions to the artificial intelligence server (electronic device). Here, the artificial intelligence server (electronic device) may acquire an output image based on an input image received from a TV (external device) and style information corresponding to each of the plurality of regions. And, the artificial intelligence server (electronic device) may transmit the output image to the TV (external device). In addition, the TV (external device) may display the output image received from the artificial intelligence server (electronic device). Specifically, the TV (external device) may store the style identification model 102 , and the artificial intelligence server (electronic device) may store the image conversion model 101 .

Meanwhile, in the above description, it has been described that the electronic device 100 performs texture processing to obtain a high-resolution image. However, according to an implementation example, the electronic device 100 may perform various processing operations other than texture processing.

Meanwhile, in the above description, it has been described that the upscaling operation is performed after the texture processing is performed. However, according to an implementation example, the electronic device 100 may first perform an upscaling operation and then perform texture processing on the upscaled image. Here, the upscaling operation may be applied to each of the plurality of regions, and texture processing may also be applied to each of the plurality of upscaled regions.

Meanwhile, although only a simple configuration constituting the electronic device 100 has been illustrated and described above, various configurations may be additionally provided when implemented. This will be described below with reference to FIG. 4 .

FIG. 4 is a block diagram illustrating a detailed configuration of the electronic device of FIG. 3 .

Referring to FIG. 4 , the electronic device 100 may include a memory 110 , a processor 120 , a communication interface 130 , a display 140 , an operation interface 150 , and an input/output interface 160 .

Meanwhile, redundant descriptions of the same operations as those described above among the operations of the memory 110 and the processor 120 will be omitted.

The communication interface 130 is configured to communicate with various types of external devices according to various types of communication methods. The communication interface 130 includes a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

The communication interface 130 may communicate with a Wi-Fi module and a Bluetooth module using a Wi-Fi method and a Bluetooth method, respectively. In the case of using a Wi-Fi module or a Bluetooth module, various types of connection information such as an SSID and a session key are first transmitted and received, and various types of information can be transmitted/received after communication connection using this.

The infrared communication module communicates according to the infrared data association (IrDA) technology, which wirelessly transmits data in a short distance using infrared that is between visible light and millimeter wave.

In addition to the above-described communication methods, the wireless communication module includes Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE Advanced (LTE-A), 4th Generation (4G), 5G It may include at least one communication chip that performs communication according to various wireless communication standards such as (5th Generation).

In addition, the communication interface 130 is at least one of a wired communication module for performing communication using a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, an optical fiber cable, or an Ultra Wide-Band (UWB) module. may include

According to an example, the communication interface 130 may use the same communication module (eg, a Wi-Fi module) to communicate with an external device such as a remote control device and an external server.

According to another example, the communication interface 130 may use different communication modules to communicate with an external device such as a remote control device and an external server. For example, the communication interface 130 may use at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may use a Bluetooth module to communicate with an external device such as a remote control device. However, this is only an embodiment, and when communicating with a plurality of external devices or external servers, the communication interface 130 may use at least one communication module among various communication modules.

The display 140 may be implemented as various types of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), and the like. The display 140 may also include a driving circuit, a backlight unit, and the like, which may be implemented in the form of an a-si TFT, a low temperature poly silicon (LTPS) TFT, or an organic TFT (OTFT). Meanwhile, the display 140 may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display, or the like.

Also, according to an embodiment of the present disclosure, the display 140 may include a bezel housing the display panel as well as a display panel for outputting an image. In particular, according to an embodiment of the present disclosure, the bezel may include a touch sensor (not shown) for detecting user interaction.

According to an embodiment, the electronic device 100 may include a display 140 . Specifically, the electronic device 100 may directly display the acquired image or content on the display 140 .

Meanwhile, according to another embodiment, the electronic device 100 may not include the display 140 . The electronic device 100 may be connected to an external display device, and may transmit an image or content stored in the electronic device 100 to the external display device. Specifically, the electronic device 100 may transmit the image or content to the external display device together with a control signal for controlling the external display device to display the image or content. Here, the external display device may be connected to the electronic device 100 through the communication interface 130 or the input/output interface 160 . For example, the electronic device 100 may not include a display such as a set top box (STB). Also, the electronic device 100 may include only a small display capable of displaying only simple information such as text information. Here, the electronic device 100 may transmit the image or content to the external display device through the communication interface 130 by wire or wirelessly or to the external display device through the input/output interface 160 .

The manipulation interface 150 may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen capable of performing the above-described display function and manipulation input function together. Here, the button may be various types of buttons such as a mechanical button, a touch pad, a wheel, etc. formed in an arbitrary area such as the front, side, or rear of the exterior of the main body of the electronic device 100 .

Input/output interface 160 is HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt (Thunderbolt), VGA (Video Graphics Array) port, The interface may be any one of an RGB port, a D-subminiature (D-SUB), and a digital visual interface (DVI).

The input/output interface 160 may input/output at least one of audio and video signals. Depending on the implementation, the input/output interface 160 may include a port for inputting and outputting only an audio signal and a port for inputting and outputting only a video signal as separate ports, or may be implemented as a single port for inputting and outputting both an audio signal and a video signal.

Meanwhile, the electronic device 100 may transmit at least one of audio and video signals to an external device (eg, an external display device or an external speaker) through the input/output interface 160 . Specifically, an output port included in the input/output interface 160 may be connected to an external device, and the electronic device 100 may transmit at least one of an audio and video signal to the external device through the output port.

The electronic device 100 may further include a microphone 170 . A microphone is a component for receiving a user's voice or other sound and converting it into audio data.

The microphone 170 may receive a user's voice in an activated state. For example, the microphone 170 may be integrally formed in the upper side, the front direction, the side direction, or the like of the electronic device 100 . The microphone 170 includes a microphone for collecting analog user voice, an amplifier circuit for amplifying the collected user voice, an A/D conversion circuit for sampling the amplified user voice and converting it into a digital signal, and a noise component from the converted digital signal. It may include various configurations such as a filter circuit that removes the

There may be various embodiments in which the electronic device 100 performs an operation corresponding to a user voice signal received through the microphone 170 .

For example, the electronic device 100 may control the display 140 based on a user voice signal received through the microphone 170 . For example, when a user voice signal for displaying content A is received, the electronic device 100 may control the display 140 to display content A.

As another example, the electronic device 100 may control an external display device connected to the electronic device 100 based on a user voice signal received through the microphone 170 . Specifically, the electronic device 100 may generate a control signal for controlling the external display device so that an operation corresponding to the user's voice signal is performed on the external display device, and transmit the generated control signal to the external display device. Here, the electronic device 100 may store a remote control application for controlling the external display device. In addition, the electronic device 100 may transmit the generated control signal to the external display device using at least one communication method among Bluetooth, Wi-Fi, and infrared. For example, when a user voice signal for displaying content A is received, the electronic device 100 may transmit a control signal for controlling content A to be displayed on the external display device to the external display device. Here, the electronic device 100 may refer to various terminal devices in which a remote control application, such as a smart phone or an AI speaker, can be installed.

As another example, the electronic device 100 may use the remote control device to control an external display device connected to the electronic device 100 based on a user voice signal received through the microphone 170 . Specifically, the electronic device 100 may transmit a control signal for controlling the external display device to the remote control device so that an operation corresponding to the user voice signal is performed on the external display device. In addition, the remote control device may transmit a control signal received from the electronic device 100 to the external display device. For example, when a user voice signal for displaying content A is received, the electronic device 100 transmits a control signal for controlling content A to be displayed on the external display device to the remote control device, and the remote control device receives control signal may be transmitted to an external display device.

5 is a flowchart illustrating an image conversion operation according to an exemplary embodiment.

Referring to FIG. 5 , the electronic device 100 may acquire an input image 10 ( S505 ). Then, the electronic device 100 may divide the input image 10 into a plurality of regions (S510). Here, the electronic device 100 may divide the input image 10 into a plurality of regions based on the object identified in the input image 10 . For example, the electronic device 100 may classify a background object and an object object in the input image 10 , and may identify a region including the background object and a region including the object object.

Also, the electronic device 100 may obtain characteristic information corresponding to each of the plurality of divided regions (S515). Here, the characteristic information may include characteristics corresponding to objects included in the plurality of regions. For example, the characteristic information may be information related to a background object or information related to a thing object.

Also, the electronic device 100 may identify style information corresponding to the characteristic information for each of a plurality of regions ( S520 ). Here, the style information may mean a style used for an image conversion operation. For example, the style information may mean at least one of texture, sharpness, and contrast. For example, the style information may include at least one of a texture (texture) intensity, a sharpness intensity, and a contrast intensity. For example, the first style information may include information indicating that texturing should be performed with a first intensity, and the second style information may include information indicating that texturing should be performed with a second intensity. have.

Also, the electronic device 100 may acquire the output image 20 based on the identified style information (S525). Here, the electronic device 100 may convert the input image 10 into the output image 20 based on the identified style information.

6 is a flowchart for describing the image conversion operation of FIG. 5 in detail.

Referring to FIG. 6 , the electronic device 100 may acquire an input image 10 ( S605 ). Then, the electronic device 100 may divide the input image 10 into a first area and a second area (S610). Then, the electronic device 100 may obtain first characteristic information corresponding to the first area and second characteristic information corresponding to the second area ( S615 ).

Also, the electronic device 100 may obtain first style information corresponding to the first characteristic information and obtain (or identify) second style information corresponding to the second characteristic information (S620).

Also, the electronic device 100 converts the image corresponding to the first region into a high-resolution image based on the first style information and converts the image corresponding to the second region into a high-resolution image based on the second style information. It can be done (S625).

Also, the electronic device 100 may acquire an output image based on high-resolution images corresponding to each region ( S630 ). Specifically, the electronic device 100 may obtain the output image 20 that is the entire image by combining the converted high-resolution image corresponding to the first region and the high-resolution image corresponding to the second region.

7 is a diagram for explaining an operation of dividing an input image into a plurality of regions.

Referring to FIG. 7 , the electronic device 100 may analyze the input image 10 to divide the input image 10 into a plurality of regions.

According to an embodiment, the electronic device 100 may identify an object included in the input image 10 and may divide the input image 10 into a plurality of regions based on the identified position of the object.

According to another embodiment, the electronic device 100 identifies an outline (or boundary line) included in the input image 10 and divides the input image 10 into a plurality of regions based on the identified outline (or boundary line). can

The reason for dividing the input image 10 into a plurality of regions is to perform image conversion of a different style for each region. For example, a background area may hardly need a texture effect and an object area may need a texture effect. Here, an image having a higher degree of completeness can be obtained without adding a texture effect even to the background area. Accordingly, the electronic device 100 may perform image conversion operations of different styles in different regions.

Here, the electronic device 100 may divide the entire area of the input image 10 into a first area 710 and a second area 720 .

Meanwhile, according to an embodiment, after identifying the first area 710 , the electronic device 100 may identify an area other than the first area 710 as the second area 720 .

8 is a flowchart illustrating an operation in which an image transformation model is learned.

Referring to FIG. 8 , the electronic device 100 may acquire a low-resolution image and a high-resolution image (S805). Here, the low-resolution image may be used as a learning input image in the image conversion operation. In addition, the high-resolution image may be used as a reference image for obtaining a loss value in an image conversion operation.

Also, the electronic device 100 may divide the low-resolution image into a plurality of regions (S810). Then, the electronic device 100 may obtain characteristic information corresponding to each of the plurality of regions (S815). Then, the electronic device 100 may identify style information corresponding to the characteristic information for each of a plurality of regions (S820).

Also, the electronic device 100 may convert the low-resolution image into an output image based on the style information identified in step S820 . Here, the operation of converting the output image may be performed for each area divided in step S810. Here, the output image may be an estimated image that meets the purpose of the image transformation model 101 . If the image conversion model 101 is a model for super-resolution of an image, the output image may be a high-resolution image generated by the image conversion model 101 .

Here, the electronic device 100 may acquire a loss value between the output image acquired in step S825 and the high-resolution image acquired in step S805 based on the pre-stored loss function. Since the high-resolution image already corresponds to a high-resolution correct answer image, it may be a criterion for how accurately image conversion is performed through the image conversion model 101 . Accordingly, the electronic device 100 may acquire a loss value in order to determine how much the output image 20 obtained by the image conversion model 101 matches the high-resolution image.

Here, the electronic device 100 may determine an image conversion parameter of the style identification model 102 for minimizing the loss ( S835 ).

Then, the electronic device 100 may identify whether to end the learning (S840). If it is determined that the learning is finished (S840-Y), the electronic device 100 may confirm the parameter determined in step S835. Here, the criterion for identifying the end of learning may be a loss value. For example, when the loss value is less than the threshold value, the electronic device 100 may end learning.

If it is identified that learning is not finished (S840-N), the electronic device 100 may repeat steps S805 to S840 by acquiring a new low-resolution image and a new high-resolution image.

Meanwhile, according to an embodiment, the electronic device 100 may perform an image conversion operation on the same low-resolution image and the same high-resolution image according to changed parameters. Specifically, the electronic device 100 may change the parameter used in the image conversion model 101 . Then, the electronic device 100 may perform steps S805 to S840 again based on the changed parameter.

9 is a view for explaining an image conversion model according to an embodiment of the present disclosure.

Referring to FIG. 9 , the electronic device 100 may include an image transformation model 101 , a style identification model 102 , and a loss value calculation model 103 . Here, the image conversion model 101 may obtain the input image 10 and convert it into the output image 20 . Here, the style identification model 102 may determine style information corresponding to the input image 10 , and transmit the determined style information to the style identification model 102 . The image conversion model 101 may convert the input image 10 into the output image 20 based on the style information transmitted from the style identification model 102 .

Here, the loss value calculation model 103 may obtain a loss value based on the output image 20 and the reference image 30 . Here, the reference image 30 may be a high-resolution image of the input image 10 . The loss value calculation model 103 may include a loss function 901 . Here, the loss value calculation model 103 may calculate a loss value between the output image 20 and the reference image 30 based on the loss function 901 .

Here, the loss value may include at least one of a mean squared error (MSE), a cognitive loss, or an adversarial loss.

Here, the electronic device 100 may acquire a cognitive loss by using the layer result of the pre-learned classification network as a feature space. In addition, the electronic device 100 may optimize the image transformation model 101 by reducing an error (distance) in the feature space. Each layer included in the image conversion model 101 may be differently disposed according to a reconstruction style of an image or a texture generation shape.

Also, the loss value calculation model 103 may differently weight and sum various types of loss values based on the style information received from the style identification model 102 . In addition, the electronic device 100 may learn to use various restoration styles for a single image transformation model 101 to perform a super-resolution operation.

10 is a diagram for explaining a calculation used in an image conversion operation.

Referring to FIG. 10 , the image conversion model 101 may convert an image based on Equation 1005 . Specifically, I^-HR may mean the output image 20 . And, G_θ may mean an image conversion function using θ as a parameter. I-LR may refer to the input image 10 . T may mean style information. As a result, the output image I^-HR may be an image generated by the image conversion function G_θ, and the image conversion function G_θ converts the input image I-LR based on the style information T can do.

Here, Equation 1010 may be an equation for determining the θ parameter of the image transformation model 101 . θ^ may mean a finally determined parameter. Ii^-HR may mean the i-th output image 20 , Ii-HR may mean the i-th reference image 30 , and Ti may mean the i-th style information.

Here, arg min(f(θ)) may mean a function that determines θ that minimizes f(θ). L(a,b) may mean a function for calculating a loss value between a and b. Therefore, L(Ii^-HR, Ii-HR |Ti) means a function that calculates the loss value between the output image (Ii^-HR) and the reference image (Ii-HR) considering the style information (Ti) can do.

11 is a diagram for explaining a calculation used for a learning operation according to an embodiment of the present disclosure.

Referring to FIG. 11 , Equation 1100 may be used for a loss value calculation operation in a learning operation for determining a parameter of the image transformation model 101 . The electronic device 100 may obtain a loss value between the output image 20 and the reference image 30 using Equation 1100 . Here, the electronic device 100 may obtain a loss value by using the loss value calculation model 103 .

Referring to Equation 1100, O may mean a loss value function. O may be λ_rec*L_rec+λ_per*L_per+λ_adv*L_adv.

Here, L_rec may mean a reconstruction loss, L_per may mean a perceptual loss, and L_adv may mean an adversarial loss. And, λ_rec may mean a weight of reconstruction loss, λ_per may mean a weight of cognitive loss, and λ_adv may mean a weight of hostility loss. λ_rec, λ_per, and λ_adv may have different values. λ_rec, λ_per, and λ_adv may have a value between 0 and 1, and the sum of λ_rec, λ_per, and λ_adv may be 1.

Here, the reconstruction loss (L_rec) may be calculated based on Equation (1111). Here, E may mean a function for calculating an average value. Here, Ii^-HR may mean an i-th output image. Here, Ii-HR may be an i-th reference image. Here, ||Ii^-HR-Ii-HR||_1 may be the sum of absolute values of the Euclidean distance difference values between the i-th output image and the i-th reference image. And, E[||Ii^-HR-Ii-HR||_1] may mean an average value of difference values between the i-th output image and the i-th reference image.

Here, the cognitive loss L_per may be calculated based on Equation 1121. Here, l may mean a style. Here, λ_l may be a weight related to a function to which style information is applied. Here, w_l(T) may be a weight related to the style information itself. And, w_l(T) may be changed based on style information (or style map), which is a random variable during learning.

Accordingly, λ_l may mean a weight corresponding to a specific function among a plurality of functions that perform an image conversion operation based on style information. And, w_l(T) may mean a weight corresponding to a specific style among a plurality of styles.

Here, the perceptual loss may be a weighted sum of multiple cognitive losses at different levels of functional space.

Here, L_l may mean a loss value of style l, and may be calculated based on Equation 1122. Here, φ_l may mean a function that transforms an image based on style l. Here, φ_l(Ii^-HR) may mean a function for converting the i-th output image Ii^-HR into a feature space corresponding to style l. Here, φ_l(Ii-HR) may mean a function that transforms the i-th reference image Ii-HR into a feature space corresponding to style l. ||Ii^-HR-Ii-HR||_2 may be a sum of squares of difference values between the i-th output image Ii^-HR and the i-th reference image Ii-HR. And, E[||Ii^-HR-Ii-HR||_2] is the square root of the sum of the square values of the difference values between the i-th output image (Ii^-HR) and the i-th reference image (Ii-HR). It can mean a function.

Here, Equation 1111 may be a calculation using the L1 norm, and Equation 1122 may be a calculation using the L2 norm.

Here, the hostility loss L_adv may be calculated based on Equation 1131. Here, E_I^-HR(f(x)) may mean an average value of f(x) for the output images. Here, f(x) may be log(D~(I^-HR), where E_I-HR(f(x)) may mean an average value of f(x) for reference images. Here, f(x) may be log(D~(I-HR).

In addition, D~(I-HR) may be calculated based on Equation 1132, and D~(I-HR) may mean the probability of identifying a real image (reference image) as real (reference image). can Here, D~(I^-HR) may be calculated based on Equation (1133), and D~(I^-HR) represents the probability of identifying a fake image (output image) as a fake (output image). can mean Here, C(I^-HR) may mean a value related to an output image obtained through the discriminator. And, C(I-HR) may mean a value related to the reference image obtained through the discriminator.

12 is a view for specifically explaining an image conversion model according to another embodiment of the present disclosure.

Referring to FIG. 12 , the electronic device 100 may include an image conversion model 101 , a style identification model 102 , and a loss value calculation model 103 .

In a general image transformation operation, the image transformation model 101 and the style identification model 102 may be used, and in the image learning operation, the image transformation model 101 , the style identification model 102 and the loss calculation model 103 are All can be used.

The image conversion model 101 may include a low resolution processing unit 1201 , an upscaling unit 1202 , and a high resolution processing unit 1203 .

Here, the low resolution processing unit 1201 may correspond to a super resolution (SR) branch and may include at least one convolutional layer and a plurality of basic blocks. Here, the low resolution processing unit 1201 may convert the image in a low resolution state before the input image 10 is upscaled. Here, the upscaling unit 1202 may upscale the converted low-resolution image. Here, the high resolution processing unit 1203 may convert the upscaled image based on at least one convolutional layer and a ReLU layer. Here, the low resolution processing unit 1201 may be a residual in residual density block (RRDB) including a spatial feature transform (SFT) layer. Here, the SFT layer may be used to learn a mapping function for outputting a modulation parameter based on the style information (T). Therefore, the SFT layer may be an essential layer for the optimization of the target.

Here, the style identification model 102 may include a user control unit 1204 , a style identification network 1205 , a style map generation unit 1206 , and a condition network 1207 . Here, the user control unit 1204 may receive a selection from the user of a style to be used for the conversion of the input image 10 directly. Here, the style identification network 1205 may identify a style suitable for the input image 10 as a network for generating a style map. Here, the style map generator 1206 may generate a style map corresponding to the input image 10 based on a style determined through at least one of the user control unit 1204 and the style identification network 1205 . Here, the conditional network 1207 may transmit the generated style map to the image transformation model 101 . Specifically, the conditional network 1207 may transmit the generated style map to a basic block included in the image transformation model 101 . Specifically, the conditional network 1207 may transmit the style map to a feature modulation layer (eg, Spatial Feature Transform (SFT)) layer of the image transformation model 101 . Meanwhile, all convolutional layers included in the conditional network 1207 may use a 1*1 kernel to avoid interference from other regions.

Here, the style map may include information for classifying an object, a background, and a boundary, and may include style information to be applied to an object, a background, and a boundary. Meanwhile, according to another embodiment, the electronic device 100 may concatenate additional information to the input image to reflect the style information to the image conversion model 101 .

Here, the style map may be used to control edges of a plurality of regions and a texture restoration style through a Spatial Feature Transform (SFT) layer.

Accordingly, the image conversion model 101 may generate the output image 20 by using the style map generated by the style identification model 102 . Meanwhile, the above-described operation may be an operation performed in a general image conversion operation.

Meanwhile, a learning operation may be required to determine various parameters used in the image transformation model 101 . Here, the loss value calculation model 103 may be used for the learning operation.

Here, the loss value calculation model 103 may include a loss function 1208 , a discriminator 1209 , and a VGG network 1210 . Here, the loss function 1208 may obtain a loss value between the output image 20 and the reference image 30 . Here, the discriminator 1209 may obtain a discriminator loss corresponding to the output image 20 and the reference image 30 . Here, the VGG network may mean a style application function corresponding to a plurality of styles. Here, the style identification model 102 may transmit the generated style map to the loss value calculation model 103 . In addition, the loss value calculation model 103 may calculate a loss value based on the style map obtained from the style identification model 102 .

Meanwhile, the loss calculation model 103 may change the calculation of the loss function 901 based on the style information (or style map) transmitted from the style identification model 102 . In addition, the loss value calculation model 103 is based on the style information (or style map) transmitted from the style identification model 102 Mean Squared Error (MSE), cognitive loss or perception in various levels of the characteristic space. A loss value may be obtained by combining at least one of the losses.

13 is a diagram for describing a basic block used in an image transformation model according to an embodiment of the present disclosure.

Referring to FIG. 13 , the basic block included in the low resolution processing unit 1201 may be a residual in residual density block (RRDB) including a plurality of unit blocks 1301 , 1302 , 1303 , 1304 and a convolutional layer 1305 . . Here, the unit blocks 1301 , 1302 , 1303 , and 1304 may include a Spatial Feature Transform (SFT) layer, a convolutional layer, and a Leaky ReLU (LRelu) layer.

Here, each SFT layer may receive style information (or style map) generated by the style identification model 102 .

14 is a diagram for describing a style identification model according to an embodiment of the present disclosure.

Referring to FIG. 14 , the style map generator 1206 may generate a style map 1405 based on an input image 1401 . Specifically, the style map generator 1206 may include convolutional layers 1402 and 1404 and a plurality of basic blocks 1403-1, 1403-2, and 1403-n. The style map 1405 may be expressed in a color between white and black depending on whether a style is applied. For example, a pixel corresponding to white in the style map 1405 may be a pixel to which a texture style is not applied at all. Also, a pixel corresponding to black in the style map 1405 may be a pixel to which a texture style is maximally applied. Also, a pixel corresponding to gray in the style map 1405 may be a pixel to which a texture style is applied to an intermediate degree.

Meanwhile, although the style map 1405 is expressed as a color, the style map 1405 may be expressed as matrix data having a value between 0 and 1.

15 is a diagram for explaining a calculation used for a learning operation according to another embodiment of the present disclosure.

Referring to FIG. 15 , the discriminator loss L_dis may be calculated based on Equation 1501. Here, E_I-HR(f(x)) may mean an average value of f(x) for reference images. Here, f(x) may be log(D~(I-HR). Here, E_I^-HR(f(x)) may mean an average value of f(x) for output images. Here, f(x) may be log(D~(I^-HR).

Equations 1502 and 1503 may correspond to Equations 1132 and 1133 of FIG. 11 . Here, D~(I-HR) may be calculated based on Equation 1502, and D~(I-HR) means the probability of identifying a real image (reference image) as a real (reference image). can Here, D~(I^-HR) may be calculated based on Equation 1503, and D~(I^-HR) represents the probability of identifying a fake image (output image) as a fake (output image). can mean Here, C(I^-HR) may mean a value related to an output image obtained through the discriminator. And, C(I-HR) may mean a value related to the reference image obtained through the discriminator.

16 is a diagram for explaining an effect of an image conversion operation of an electronic device according to various embodiments of the present disclosure;

Referring to FIG. 16 , in a graph 1605 , the x-axis may be Peak Signal-to-Noise Ratio (PSNR) and the y-axis may be Learned Perceptual Image Patch Similarity (LPIPS). The PSNR may be a value indicating a difference in values in units of pixels, and a higher PSNR may mean a lower cognitive loss. In addition, LPIPS may mean cognitive similarity, and the lower the value, the better the performance.

SRResNET, NatSR, SRGAN, SFTGAN, ESRGAN, and SPSR may refer to various super-resolution models. Here, FxSR-D may be an embodiment in which the image conversion model is super-resolution based on first style information (eg, a style that minimally reflects a texture effect). Here, the FxSR-P may be an embodiment in which the image conversion model is super-resolution based on the second style information (eg, a style that maximally reflects a texture effect).

In the graph 1605, in that FxSR-D and FxSR-P are located on the left and bottom, it can be evaluated that the performance is superior to that of other super-resolution models.

17 is a diagram for comparing image conversion operations of an electronic device according to an exemplary embodiment.

Referring to FIG. 17 , various image conversion methods may be applied to input images 1710 , 1720 , and 1730 . The images 1711 , 1721 , and 1731 corresponding to HR may refer to reference images. The images 1712 , 1722 , and 1732 corresponding to FxSR-D may be images in which super-resolution is performed in a style (T=0) that minimally reflects a texture effect. In addition, the images 1713 , 1723 , and 1733 corresponding to FxSR-E may be images in which super-resolution is performed in a style (T=0.5) that reflects the texture effect in the middle. Also, the images 1714 , 1724 , and 1734 corresponding to FxSR-P may be images in which super-resolution is performed in a style (T=1) that maximizes the texture effect.

Here, the electronic device 100 identifies a specific area 1710-1 from the input image 1710, and first generates an image (eg, an image including a building object) corresponding to the specific area 1710-1. can be resolved. The image 1711 may be a reference image corresponding to the specific region 1710-1. The images 1712 , 1713 , and 1714 may be images in which super-resolution is performed in a style that reflects the texture effect differently. Image 1714 may be most similar to base image 1711 .

Here, the electronic device 100 identifies a specific region 1720 - 1 in the input image 1720 , and an image corresponding to the specific region 1720 - 1 (eg, including a scratched brick object). image) can be super-resolution. The image 1721 may be a reference image corresponding to the specific region 1720-1. The images 1722 , 1723 , and 1724 may be images in which super-resolution is performed in a style that reflects the texture effect differently. Image 1724 may be most similar to base image 1721 .

Here, the electronic device 100 identifies a specific region 1730-1 in the input image 1730 and selects an image (eg, an image including a plant object) corresponding to the specific region 1730-1. can be resolved. The image 1731 may be a reference image corresponding to the specific area 1730-1. The images 1732 , 1733 , and 1734 may be images in which super-resolution is performed in a style that reflects the texture effect differently. Image 1734 may be most similar to base image 1731 .

18 is a diagram for comparing image conversion operations of an electronic device according to another exemplary embodiment.

Referring to FIG. 18 , an image 1800 may be a reference image, an image 1800-2 may be an image corresponding to a first region 1801 , and an image 1800-3 may be a second region 1802 . ) may be an image corresponding to

The style maps 1810-1, 1820-1, 1830-1, 1840-1, and 1850-1 may refer to data indicating to what extent a style effect (eg, a texture effect) is to be reflected. Here, black may mean data that reflects the texture effect to the minimum, and white may mean data that reflects the texture effect to the maximum. Gray may mean data reflecting a texture effect as a medium. Accordingly, the degree to which the texture effect is reflected may be different depending on the degree of gray.

Also, the image 1810 may be an image in which super-resolution is performed in a style (T=0) that minimizes the texture effect for the entire area. Also, the image 1810-1 may mean a style map. The image 1810 - 2 may be an image corresponding to the first region 1801 of the image 1810 . The image 1810 - 3 may be an image corresponding to the second region 1802 of the image 1810 .

Also, the image 1820 may be an image in which super-resolution is performed in a style (T=1) that maximizes the texture effect for the entire area. Also, the image 1820-1 may mean a style map. The image 1820 - 2 may be an image corresponding to the first region 1801 of the image 1820 . The image 1820 - 3 may be an image corresponding to the second region 1802 of the image 1820 .

Also, the image 1830 may be an image in which super-resolution is performed in a style that reflects the texture effect differently for each region. Also, the image 1830-1 may mean a style map. The image 1830 - 2 may be an image corresponding to the first area 1801 of the image 1830 . The image 1830 - 3 may be an image corresponding to the second region 1802 of the image 1830 .

Also, according to an embodiment, the image 1840 may be an image in which super-resolution is performed in a style in which a user artificially designates an area to reflect a texture effect differently. Also, the image 1840 - 1 may mean a style map. The image 1840 - 2 may be an image corresponding to the first region 1801 of the image 1840 . The image 1840 - 3 may be an image corresponding to the second region 1802 of the image 1840 .

Also, according to another embodiment, the image 1850 may be an image in which super-resolution is performed in a style in which a user artificially designates a region to reflect a texture effect differently. Also, the image 1850 - 1 may mean a style map. The image 1850 - 2 may be an image corresponding to the first area 1801 of the image 1850 . The image 1850 - 3 may be an image corresponding to the second area 1802 of the image 1850 .

The image 1820 that maximally reflects the texture effect for the entire area and the image 1830 generated by automatically determining the style map through the style identification network (1205 in Fig. 12) may be most similar to the basic image 1800. have. However, in consideration of the naturalness of the image, the image 1830 may be more similar to the basic image 1800 than the image 1820 .

19 is a diagram for comparing image conversion operations of an electronic device according to another exemplary embodiment.

Referring to FIG. 19 , images 1910 and 1920 may be input images and may include specific regions 1910 - 1 and 1920 - 1 . The images 1910 - 2 and 1920 - 2 may refer to style maps generated according to an embodiment in which a style map is automatically determined through a style identification network ( 1205 of FIG. 12 ). The images 1910 - 3 and 1920 - 3 may refer to style maps generated according to an embodiment in which the user directly determines the style map through the user control unit ( 1204 of FIG. 12 ).

Here, the images 1911 , 1912 , 1913 , 1914 , 1915 , and 1916 may be images corresponding to a specific region 1910 - 1 among the entire region of the input image 1910 . The images 1921 , 1922 , 1923 , 1924 , 1925 , and 1926 may be images corresponding to a specific region 1920 - 1 among the entire region of the input image 1920 .

The images 1911 and 1921 may be reference images corresponding to the input images 1910 and 1920 . Also, the images 1912 and 1922 may be images in which super-resolution is performed in a style (T=0) that maximally reflects the texture effect for the entire area. Also, the images 1913 and 1923 may be images in which super-resolution is performed in a style that reflects a texture effect differently for each region. Also, the images 1914 and 1924 may be images in which super-resolution is performed in a style in which a user artificially designates an area to reflect a texture effect differently. Also, the images 1915 and 1925 may be images corresponding to specific regions 1910 - 1 and 1920 - 1 among the entire regions of the images 1910 - 2 and 1920 - 2 . Also, the images 1916 and 1926 may be images corresponding to specific regions 1910 - 1 and 1920 - 1 among the entire regions of the images 1910 - 3 and 1920 - 3 .

Meanwhile, here, the images 1910-2, 1910-3, 1915, 1916, 1920-2, 1920-3, 1925, and 1926 may mean a style map image.

Images (1912, 1922) that maximally reflected the texture effect for the entire area and images (1913, 1923) reflecting the texture effect by automatically determining the style map through the style identification network (1205 in Fig. 12) are the default images (1911) , 1922). However, considering the naturalness of the image, the images 1913 and 1923 may be closer to the basic images 1911 and 1922 than the images 1912 and 1922 .

In addition, the style map images 1915 and 1925 generated by automatically determining the style map through the style identification network ( 1205 in FIG. 12 ) may be more sophisticated than the style map images 1916 and 1926 generated directly by the user. .

20 is a diagram for explaining a method of controlling an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 20 , a control method of an electronic device for storing an artificial intelligence model trained to obtain a high-resolution output image equal to or greater than a threshold resolution from a low-resolution input image less than the threshold resolution is a first method including first characteristic information in the low-resolution input image. A step of identifying a region and a second region including the second characteristic information (S2005), a first image in which texture processing of a first intensity is performed on the first region, and a second region different from the first intensity on the second region A high-resolution image is obtained by obtaining a second image that has been subjected to intensity texture processing (S2010), upscaling the first image and the second image, respectively (S2015), and combining the upscaled first and second images It may include a step of obtaining (S2020).

Meanwhile, the first characteristic information may include object region information and the second characteristic information may include background region information, and the step of identifying the first region and the second region ( S2005 ) includes object region information. The area may be identified as the first area, and the area including the background area information may be identified as the second area.

Meanwhile, in the step of acquiring the first image and the second image ( S2010 ), if the type of the object included in the first region is the first type, texture processing with an intensity corresponding to the first type may be performed on the first region. If the type of the object included in the first area is the second type, texture processing with an intensity corresponding to the second type may be performed on the first area.

Meanwhile, the method may further include inputting a low-resolution input image into the first artificial intelligence model to obtain identification information for the first region, first intensity, identification information for the second region, and second intensity, the first The artificial intelligence model is trained to output identification information for regions with different characteristics in the input image and texture intensity for each identified region when an image is input, a control method.

Meanwhile, the identification information on the first region may include coordinate information on the first region included in the low-resolution input image, and the identification information on the second region includes coordinate information on the second region included in the low-resolution input image. may include

On the other hand, the step of obtaining the first image and the second image (S2010) and the step of upscaling the first image and the second image (S2015), respectively, is an image including identification information for the first area and the second area , first intensity and second intensity may be input to the second artificial intelligence model to obtain first and second images subjected to texture processing and upscaling.

Meanwhile, the second artificial intelligence model may be implemented to include a residual-in-residual density block (RRDB) that performs partial feature transformation (SFT).

Meanwhile, the second artificial intelligence model may include a layer having a plurality of parameters, based on an image identified as a plurality of regions having different characteristics and a plurality of images obtained based on texture intensity corresponding to the plurality of regions. A loss value may be calculated by comparing with the image, and a plurality of parameters may be learned based on the calculated loss value.

Meanwhile, the second artificial intelligence model may calculate a loss value based on at least one of a reconstruction loss, an adversarial loss, and a perceptual loss.

Meanwhile, the second artificial intelligence model may be implemented as a Generative Adversarial Network (GAN).

Meanwhile, the control method of the electronic device 100 as shown in FIG. 20 may be executed on the electronic device 100 having the configuration of FIG. 3 or 4 , and may also be executed on the electronic device 100 having other configurations.

Meanwhile, the above-described methods according to various embodiments of the present disclosure may be implemented in the form of an application that can be installed in an existing electronic device.

In addition, the above-described methods according to various embodiments of the present disclosure may be implemented only by software upgrade or hardware upgrade of an existing electronic device.

In addition, various embodiments of the present disclosure described above may be performed through an embedded server provided in an electronic device or an external server of at least one of an electronic device and a display device.

Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). can A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include the electronic device according to the disclosed embodiments. When the instruction is executed by the processor, the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

In addition, according to an embodiment of the present disclosure, the method according to the various embodiments described above may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

In addition, each of the components (eg, a module or a program) according to the various embodiments described above may be composed of a single or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be omitted. Components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity, so that functions performed by each corresponding component prior to integration may be performed identically or similarly. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallelly, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. can

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and is generally used in the technical field belonging to the present disclosure without departing from the gist of the present disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

100: electronic device
110: memory
120: processor
130: communication interface
140: display

Claims (20)

In an electronic device,
display;
a memory in which the trained artificial intelligence model is stored; and
A processor for controlling the display to obtain a high-resolution output image above the threshold resolution from a low-resolution input image less than the threshold resolution using the learned artificial intelligence model and display the obtained high-resolution image,
The processor is
identifying a first region including first characteristic information and a second region including second characteristic information in the low-resolution input image;
acquiring a first image in which texturing of a first intensity is performed on the first area and a second image in which texturing of a second intensity different from the first intensity is performed on the second area;
upscaling the first image and the second image, respectively;
and combining the upscaled first and second images to obtain the high-resolution image. According to claim 1,
The first characteristic information includes object region information and the second characteristic information includes background region information,
The processor is
and identifying an area including the object area information as the first area and identifying an area including the background area information as the second area. 3. The method of claim 2,
The processor is
If the type of the object included in the first region is the first type, texture processing of an intensity corresponding to the first type is performed on the first region;
If the type of the object included in the first area is the second type, the electronic device performs texture processing with an intensity corresponding to the second type on the first area. According to claim 1,
The processor is
inputting the low-resolution input image into a first artificial intelligence model to obtain identification information for the first region, the first intensity, identification information for the second region, and the second intensity;
The first artificial intelligence model,
When an image is input, the electronic device is trained to output identification information for regions having different characteristics in the input image and texture intensity for each identified region. 5. The method of claim 4,
Identification information for the first area,
including coordinate information for the first region included in the low-resolution input image,
Identification information for the second area,
The electronic device comprising coordinate information on the second region included in the low-resolution input image. 5. The method of claim 4,
The processor is
An image including identification information for the first region and the second region, and first and second images subjected to texture processing and upscaling by inputting the first intensity and the second intensity into a second artificial intelligence model to obtain, an electronic device. 7. The method of claim 6,
The second artificial intelligence model,
An electronic device implemented to include a Residual-in-Residual Dense Block (RRDB) that performs Partial Feature Transformation (SFT). 7. The method of claim 6,
The second artificial intelligence model,
A layer having a plurality of parameters,
Comparing the image identified as a plurality of regions having different characteristics and a plurality of images obtained based on the texture intensity corresponding to the plurality of regions with a reference image to calculate a loss value,
learning the plurality of parameters based on the calculated loss value. 9. The method of claim 8,
The second artificial intelligence model,
An electronic device for calculating the loss value based on at least one of a reconstruction loss, an adversarial loss, and a perceptual loss. 9. The method of claim 8,
The second artificial intelligence model,
An electronic device implemented as a Generative Adversarial Network (GAN). A method for controlling an electronic device for storing an artificial intelligence model trained to obtain a high-resolution output image equal to or greater than the threshold resolution from a low-resolution input image less than a threshold resolution,
identifying a first region including first characteristic information and a second region including second characteristic information in the low-resolution input image;
obtaining a first image on which the texturing process of a first intensity is performed on the first area and a second image on which the texturing process of a second intensity different from the first intensity is performed on the second area;
upscaling the first image and the second image, respectively; and
Combining the upscaled first and second images to obtain the high-resolution image; including, a control method. 12. The method of claim 11,
The first characteristic information includes object region information and the second characteristic information includes background region information,
The step of identifying the first region and the second region comprises:
and identifying an area including the object area information as the first area and identifying an area including the background area information as the second area. 13. The method of claim 12,
Acquiring the first image and the second image comprises:
If the type of the object included in the first region is the first type, texture processing of an intensity corresponding to the first type is performed on the first region;
If the type of the object included in the first region is the second type, texture processing of the intensity corresponding to the second type is performed on the first region. 12. The method of claim 11,
The method further comprising: inputting the low-resolution input image into a first artificial intelligence model to obtain identification information for the first region, the first intensity, identification information for the second region, and the second intensity;
The first artificial intelligence model,
A control method that is learned to output identification information for regions having different characteristics in the input image and texture intensity for each identified region when an image is input. 15. The method of claim 14,
Identification information for the first area,
including coordinate information for the first region included in the low-resolution input image,
Identification information for the second area,
A control method comprising coordinate information on the second region included in the low-resolution input image. 15. The method of claim 14,
Acquiring the first image and the second image and upscaling the first image and the second image, respectively,
An image including identification information for the first region and the second region, and first and second images subjected to texture processing and upscaling by inputting the first intensity and the second intensity into a second artificial intelligence model to obtain, a control method. 17. The method of claim 16,
The second artificial intelligence model,
A control method implemented to include a Residual-in-Residual Dense Block (RRDB) that performs SFT (Spartial Feature Transformation). 17. The method of claim 16,
The second artificial intelligence model,
A layer having a plurality of parameters,
Comparing the image identified as a plurality of regions having different characteristics and a plurality of images obtained based on the texture intensity corresponding to the plurality of regions with a reference image to calculate a loss value,
Learning the plurality of parameters based on the calculated loss value, a control method. 19. The method of claim 18,
The second artificial intelligence model,
A control method for calculating the loss value based on at least one of a reconstruction loss, an adversarial loss, or a perceptual loss. 19. The method of claim 18,
The second artificial intelligence model,
A control method, implemented as a Generative Adversarial Network (GAN).

Images