KR102269034B1 - Apparatus and method of using AI metadata related to image quality - Google Patents

Abstract

a memory storing one or more instructions; one or more processors to execute the one or more instructions stored in the memory; and an output unit, wherein the one or more processors execute the one or more instructions and use a first artificial intelligence (AI) network to correspond to the type of object included in the first image among a plurality of predefined objects Creates AI metadata including class information including at least one class, and at least one class map indicating a region corresponding to each class in the first image, and encodes the first image to create an encoded image and outputting the encoded image and the AI metadata through the output unit.

Description

Apparatus and method of using AI metadata related to image quality

Embodiments of the present disclosure relate to an image providing apparatus, a method for controlling an image providing apparatus, an image reproducing apparatus, a method for controlling an image providing apparatus, and a computer program.

When a digital image is transmitted between devices through a network, various technologies such as encoding and image downscaling are used to reduce the amount of transmitted data. However, in the process of reducing the amount of data of the image and reconstructing and playing back the image in the device that received the image, the image quality of the original image cannot be reproduced, resulting in deterioration of image quality.

In order to improve the image quality restored by the image reproducing apparatus, an attempt has been made to improve the image quality by extracting features of the input image in the process of reproducing the encoded input image. However, the process of extracting image features from the input image as described above involves considerable resources and processing time, so there is a problem in that the processing load in the image reproducing apparatus is increased and the processing speed is reduced.

Embodiments of the present disclosure are for lightening the AI quality network used for generation of AI metadata.

In addition, embodiments of the present disclosure are for reducing the weight of an AI network used for image quality processing in an image reproducing apparatus.

In addition, embodiments of the present disclosure are for improving the performance of an AI image quality network by generating AI metadata from an original image.

According to an aspect of an embodiment of the present disclosure, a memory for storing one or more instructions; one or more processors to execute the one or more instructions stored in the memory; and an output unit, wherein the one or more processors execute the one or more instructions and use a first artificial intelligence (AI) network to correspond to the type of object included in the first image among a plurality of predefined objects Creates AI metadata including class information including at least one class, and at least one class map indicating a region corresponding to each class in the first image, and encodes the first image to create an encoded image and outputting the encoded image and the AI metadata through the output unit.

In addition, according to an embodiment of the present disclosure, the one or more processors execute the one or more instructions, input the first image to the first AI network, and apply to each type of the plurality of predefined objects. A class including generating a plurality of segmentation probability maps for , defining at least one class corresponding to a type of an object included in the first image based on the plurality of segmentation probability maps, and including the at least one class information may be generated, and the at least one class map for each of the at least one class may be generated from the plurality of segmentation probability maps.

In addition, according to an embodiment of the present disclosure, the one or more processors execute the one or more instructions to calculate an average value of pixels except for a pixel having a value of 0 for each of the plurality of segmentation probability maps, and , the at least one class may be defined by selecting some of the plurality of predefined objects based on the size of the average value of each of the plurality of segmentation probability maps.

In addition, according to an embodiment of the present disclosure, the one or more processors execute the one or more instructions to display at least some of the objects that are not selected as the at least one class among the plurality of predefined objects. The class map may be generated by mapping to one class and synthesizing the segmentation probability map of the object mapped to the at least one class and the segmentation probability map of the class to which the corresponding object is mapped.

In addition, according to an embodiment of the present disclosure, the one or more processors execute the one or more instructions to generate, from the first AI network, a segmentation probability map for a frequency for each of a plurality of predefined frequency domains. and frequency information including information on a frequency domain for the first image based on the segmentation probability map for the frequency, and at least one frequency map corresponding to each of the frequency domains included in the frequency information. AI metadata can be generated.

In addition, according to an embodiment of the present disclosure, the first image includes a plurality of frames, the class information and the at least one class map are generated for each of the plurality of frames, and the one or more processors include: By executing the one or more instructions, at least one sequence including at least one frame is defined from the plurality of frames, and for each sequence, a class included in the at least one frame included in the sequence is defined. Generates sequence class information indicating information and frame class information indicating information on a class included in each of the at least one frame included in the corresponding sequence, wherein the frame class information includes a class included in the sequence class information represents a combination of classes included in the corresponding frame, and the number of bits may be smaller than the sequence class information.

Also, according to an embodiment of the present disclosure, the one or more processors execute the one or more instructions, and based on the at least one class map, each pixel has a representative value corresponding to one of the at least one class. may generate a lightweight class map having , generate lightweight AI metadata including the class information and the lightweight class map, and output the encoded image and the lightweight class map through the output unit.

Also, according to an embodiment of the present disclosure, the first AI network includes at least one convolution layer and at least one max pooling layer, and features from the first image. A 1-1 AI network that generates a map (feature map); and a first layer group including at least one convolutional layer and at least one activation layer, the first layer group receiving and processing the feature map from the 1-1 AI network, and upscaling the output of the first layer group and an upscaler comprising at least one convolutional layer and at least one min pooling layer, and receiving the output of the upscaler as input to generate a segmentation probability map for each of the plurality of predefined objects A 1-2 AI network including a 2-layer group may be included.

In addition, according to an embodiment of the present disclosure, the first AI network is trained in connection with a second AI network, and the second AI network receives the encoded image and the AI metadata to decode the encoded image. It may be included in the device and perform image quality processing of image data corresponding to the encoded image from the AI metadata.

Also, according to an embodiment of the present disclosure, the one or more processors may execute the one or more instructions to perform downscale processing and encoding processing on the first image to generate the encoded image.

In addition, according to another aspect of an embodiment of the present disclosure, using a first artificial intelligence (AI) network, at least one class corresponding to the type of object included in the first image among a plurality of predefined objects is included. generating AI metadata including class information and at least one class map indicating a region corresponding to each object in the first image; generating an encoded image by encoding the first image; and outputting the encoded image and the AI metadata.

In addition, according to another aspect of an embodiment of the present disclosure, a memory for storing one or more instructions; one or more processors to execute the one or more instructions stored in the memory; an input unit receiving an encoded image corresponding to the first image and AI metadata corresponding to the encoded image; and an output unit, wherein the AI metadata includes class information including at least one class corresponding to a type of an object included in the first image, and at least an area corresponding to each class in the first image. one class map, wherein the one or more processors execute the one or more instructions to decode the encoded image to generate a second image corresponding to the first image, and use a second AI network, An image reproducing apparatus is provided, which generates a third image on which image quality improvement has been performed, from the second image and the AI metadata, and outputs the third image through the output unit.

Also, according to an embodiment of the present disclosure, the second AI network includes at least one convolution layer, at least one modulation layer, and at least one activation layer. and a 2-1 AI network, wherein the at least one modulation layer may process a feature map input to the modulation layer based on the modulation parameter set generated from the AI metadata.

In addition, according to an embodiment of the present disclosure, the 2-1 AI network includes a plurality of modulation layers, and the second AI network includes a modulation parameter corresponding to each of the plurality of modulation layers from the AI metadata. a generation network, wherein each modulation parameter generation network may generate the set of modulation parameters for a corresponding modulation layer.

In addition, according to an embodiment of the present disclosure, the modulation parameter set includes a first arithmetic modulation parameter synthesized by a multiplication operation on a data value of an input feature map, and the first arithmetic modulation parameter and the input feature map a second arithmetic modulation parameter synthesized by an addition operation to a result of multiplying data values of , wherein the at least one modulation layer comprises the input feature based on the first arithmetic modulation parameter and the second arithmetic modulation parameter. Processing can be performed on the map.

Further, according to an embodiment of the present disclosure, the 2-1 AI network includes a plurality of residual blocks, and each of the plurality of residual blocks includes at least one convolutional layer, at least one modulation layer, and at least A main stream including one activation layer and generating a residual version processing result value, and at least one second skip processing for generating a prediction version processing result by bypassing at least one layer included in the corresponding block a path, and the 2-1 AI network may include at least one first skip processing path that skips at least one residual block among the plurality of residual blocks to generate a processing result value of a prediction version. .

In addition, according to an embodiment of the present disclosure, the second AI network includes an upscaler that receives the output of the 2-1 AI network and performs upscaling, and a third It may include a second image quality processing unit that generates and outputs an image and includes at least one machine-learned layer.

In addition, according to an embodiment of the present disclosure, the AI metadata input through the input unit includes a lightweight class map in which at least one class map for each of the at least one class is lightened, and the one or more processors, By executing the one or more instructions, a restored class map for each of the at least one class is generated from the lightweight class map, and the restored class map may have a value indicating whether each pixel corresponds to the corresponding class. .

According to another aspect of an embodiment of the present disclosure, the method may further include: receiving an encoded image corresponding to a first image and AI metadata corresponding to the encoded image; generating a second image corresponding to the first image by decoding the encoded image; generating, from the second image and the AI metadata, a third image on which image quality improvement has been performed, by using a second AI network; and outputting the third image, wherein the AI metadata includes class information including at least one class corresponding to a type of an object included in the first image, and each class in the first image A method for controlling an image reproducing apparatus is provided, including at least one class map indicating a region corresponding to .

In addition, according to another aspect of an embodiment of the present disclosure, in a computer-readable recording medium storing computer program instructions for performing a method of controlling an image reproducing apparatus when executed by a processor, the image reproducing apparatus controlling the image reproducing apparatus The method may include: receiving an encoded image corresponding to the first image and AI metadata corresponding to the encoded image; generating a second image corresponding to the first image by decoding the encoded image; generating, from the second image and the AI metadata, a third image on which image quality improvement has been performed, by using a second AI network; and outputting the third image, wherein the AI metadata includes class information including at least one class corresponding to a type of an object included in the first image, and each class in the first image A recording medium is provided, comprising at least one class map indicating an area corresponding to .

According to embodiments of the present disclosure, there is an effect of reducing the weight of the AI image quality network used for generating AI metadata.

In addition, according to embodiments of the present disclosure, there is an effect of reducing the weight of the AI network used for image quality processing in the image reproducing apparatus.

Also, according to embodiments of the present disclosure, there is an effect of improving the performance of an AI image quality network by generating AI metadata from an original image.

1 is a diagram illustrating structures of an image providing apparatus and an image reproducing apparatus according to an embodiment of the present disclosure.
2 is a diagram illustrating a structure of a processor of an image providing apparatus according to an exemplary embodiment.
3 is a diagram illustrating a processing process of a meta information extracting unit according to an embodiment of the present disclosure.
4 is a diagram illustrating a process of generating object information and class information from a first image in an image characteristic analysis process, according to an embodiment of the present disclosure.
5 is a diagram illustrating an operation of an information generating network of a meta information extracting unit according to an embodiment of the present disclosure.
6 is a diagram illustrating an operation of a meta information lightweight unit of an image providing apparatus according to an embodiment of the present disclosure.
7 is a diagram illustrating an operation of a meta information compression unit according to an embodiment of the present disclosure.
8 is a diagram illustrating a method of compressing class information in a meta information compression unit according to an embodiment of the present disclosure.
9 is a diagram illustrating an operation of a meta information compression unit of an image providing apparatus according to an embodiment of the present disclosure.
10 is a flowchart illustrating a method of controlling an image providing apparatus according to an embodiment of the present disclosure.
11 is a diagram illustrating structures of an input unit, a processor, and an output unit of an image reproducing apparatus according to an embodiment of the present disclosure.
12 is a diagram illustrating an operation of a meta information synthesizing unit according to an embodiment of the present disclosure.
13 is a diagram illustrating a structure of a residual block 1212 according to an embodiment of the present disclosure.
14 is a diagram illustrating operations of a 2-1 AI network and a 2-2 AI network according to an embodiment of the present disclosure.
15A is a diagram illustrating structures of a 2-1 AI network and a 2-2 AI network according to an embodiment of the present disclosure.
15B is a diagram illustrating structures of a 2-1 AI network and a 2-2 AI network according to an embodiment of the present disclosure.
16 is a diagram illustrating operations of a metadata synthesizing unit and a metadata processing unit according to an embodiment of the present disclosure.
17 is a diagram for describing a method of learning a first AI network of an image providing apparatus and a second AI network of an image reproducing apparatus according to an embodiment of the present disclosure.
18 is a flowchart illustrating a method of controlling an image reproducing apparatus according to an embodiment of the present disclosure.
19 is a diagram illustrating the configuration of an image providing apparatus and an image reproducing apparatus according to an embodiment of the present disclosure.

This specification clarifies the scope of the claims, and explains and discloses the principles of the embodiments so that those of ordinary skill in the art to which the embodiments of the present disclosure pertain can practice the embodiments described in the claims. The disclosed embodiments may be implemented in various forms.

Like reference numerals refer to like elements throughout. This specification does not describe all elements of the embodiments, and general content in the technical field to which the embodiments of the present disclosure pertain or overlapping between the embodiments will be omitted. The term "module" or "unit" used in this specification may be implemented in one or a combination of two or more of software, hardware, or firmware, and according to embodiments, a plurality of "modules" or "units" may be one It is also possible that it is implemented as an element of , or that one “module” or “part” includes a plurality of elements.

In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present disclosure, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the specification are only identifiers for distinguishing one component from other components.

In addition, in this specification, when a component is referred to as "connected" or "connected" with another component, the component may be directly connected or directly connected to the other component, but in particular It should be understood that, unless there is a description to the contrary, it may be connected or connected through another element in the middle.

Hereinafter, the principle of operation and various embodiments of the embodiments of the present disclosure will be described with reference to the accompanying drawings.

In the present specification, 'a first artificial intelligence (AI) network' and a 'second AI network' refer to an artificial intelligence model that is machine-learned using a plurality of training data. The first AI network and the second AI network may include various machine learning models, for example, include a deep neural network (DNN) structure.

Also, in this specification, an 'image' or 'picture' may correspond to a still image, a moving picture composed of a plurality of continuous still images (or frames), or a video.

In addition, in the present specification, a 'first image' refers to an image to be encoded, and an 'encoded image' refers to an image generated by encoding the first image. In addition, the 'second image' refers to an image generated by decoding the encoded image, and the 'third image' refers to an image on which image quality improvement processing is performed on the second image.

1 is a diagram illustrating structures of an image providing apparatus and an image reproducing apparatus according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the image providing apparatus 110 generates an encoded image and AI metadata from the first image and transmits them to the image reproducing apparatus 150, and the image reproducing apparatus 150 generates the encoded image and AI The received metadata is received, the encoded image is decoded, and the image quality improvement process is performed to output the received encoded image. The image providing apparatus 110 and the image reproducing apparatus 150 may be connected through a communication network or various input/output interfaces.

The image providing apparatus 110 may be implemented as various types of electronic devices. According to an embodiment, the image providing apparatus 110 may correspond to a server of a service provider that provides image content. Also, the image providing device 110 may be implemented in the form of a cloud server as well as a physically independent electronic device.

The image providing apparatus 110 includes a processor 112 , a memory 116 , and an output unit 118 .

The processor 112 controls the overall operation of the image providing apparatus 110 . The processor 112 may be implemented with one or more processors. The processor 112 may execute an instruction or a command stored in the memory 116 to perform a predetermined operation.

The processor 112 may receive the first image, extract AI metadata related to image quality processing, and encode the first image to generate an encoded image. The processor 112 may extract AI metadata from the first image by using the first AI network 114 .

AI metadata refers to metadata related to image quality improvement processing extracted from the first image. The AI metadata includes, from the first image, information related to the type of object in the image. AI metadata is extracted from the first image using the first AI network 114 .

The processor 112 may process the input image data using the first artificial intelligence (AI) network 114 . According to an embodiment, the first AI network 114 may be provided in the image providing device 110 . According to another embodiment, the first AI network 114 may be provided in another device, and the image providing device 110 may use the first AI network 114 provided in an external device through a communication interface or the like. In the present specification, an embodiment in which the first AI network 114 is provided in the image providing device 110 is mainly described, but the embodiment of the present disclosure is not limited thereto.

The first AI network 114 refers to an artificial intelligence model that is machine-learned using a plurality of training data. The first AI network 114 may include a machine learning model having various structures, for example, a deep neural network (DNN) structure. In addition, the first AI network 114 may have a structure such as, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), or the like. In addition, the first AI network 114 may employ a machine learning algorithm such as a Spatial Feature Transform Generative Adversarial Network (SFTGAN) or an Enhanced Super-resolution Generative Adversarial Network (ESRGAN).

The first AI network 114 may include a storage space and a processor for processing data stored in each storage space. The first AI network 114 may define at least one node and a layer, and define data processing between the nodes and layers, by an algorithm and parameter values that define processing between data in each storage space. The first AI network 114 may include a plurality of nodes and a plurality of layers, and may generate output data by processing input data by transferring data between the plurality of nodes and the plurality of layers.

The memory 116 stores computer program instructions, information, and contents necessary for the operation of the image providing apparatus 110 . The memory 116 may include a volatile storage medium, a non-volatile storage medium, or a combination thereof. The memory 116 may be implemented with various types of storage media. The memory 116 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk.

The output unit 118 outputs data or control signals generated or processed by the image providing apparatus 110 . The output unit 118 outputs the encoded image and AI metadata generated by the image providing device 110 . The output unit 118 may include various types of output interfaces such as a communication interface, a display, a speaker, and a touch screen.

According to an embodiment, the output unit 118 may include a communication unit or a communication interface. The output unit 118 may output data or signals to an external device such as the image reproducing apparatus 150 through a communication interface. The communication interface may output the encoded image and AI metadata to the image reproducing apparatus 150 directly or via another device.

The image reproducing apparatus 150 may be implemented as various types of electronic devices. According to an embodiment, the image reproducing apparatus 150 may correspond to a user terminal. Also, the image reproducing apparatus 150 may be implemented as various types of electronic devices, such as a portable communication terminal, a smart phone, a wearable device, a laptop computer, a desktop computer, a tablet PC, and a kiosk.

The image reproducing apparatus 150 may include an input unit 152 , a processor 154 , a memory 158 , and an output unit 160 .

The input unit 152 receives data or a control signal and transmits it to the processor 154 . The input unit 152 receives the encoded image and AI metadata output from the image providing device 110 .

The input unit 152 may include a communication unit or a communication interface. Also, it may include an input/output interface such as a touch screen, a keyboard, a mouse, and a touch pad.

The processor 154 controls the overall operation of the image reproducing apparatus 150 . Processor 154 may be implemented with one or more processors. The processor 154 may execute an instruction or a command stored in the memory 158 to perform a predetermined operation.

The processor 154 may process the input image data using the second artificial intelligence (AI) network 156 . The processor 154 receives the encoded image and AI metadata input from the image providing device 110 , obtains a second image through decoding processing for the encoded image, and uses the AI metadata for the second image Thus, the third image may be obtained by performing image quality improvement processing. The processor 154 may obtain a third image output from the second image and AI metadata input by using the second AI network 156 .

According to an embodiment, the image reproducing apparatus 150 may receive the low-resolution encoded image and perform upscaling on the second image obtained from the encoded image together. The processor 156 may perform image quality improvement processing and upscaling processing on the second image. The processor 156 may perform image quality improvement processing and upscaling processing using the second AI network 156 .

According to an embodiment, the second AI network 156 may be provided in the image reproducing apparatus 150 . According to another embodiment, the second AI network 156 may be provided in another device, and the image reproducing device 150 may use the second AI network 156 provided in the external device through a communication interface or the like. In the present specification, an embodiment in which the second AI network 156 is provided in the image reproducing apparatus 150 is mainly described, but the embodiment of the present disclosure is not limited thereto.

The second AI network 156 refers to a machine-learning artificial intelligence model using a plurality of training data. The second AI network 156 may include a machine learning model having various structures, for example, a deep neural network (DNN) structure. In addition, the second AI network 156 may have a structure such as, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), or the like. In addition, the second AI network 156 may employ a machine learning algorithm such as a Spatial Feature Transform Generative Adversarial Network (SFTGAN) or an Enhanced Super-resolution Generative Adversarial Network (ESRGAN).

The second AI network 156 may include a storage space and a processor that performs processing of data stored in each storage space. The second AI network 156 may define at least one node and layer, and define data processing between the nodes and layers, by an algorithm and parameter values that define processing between data in each storage space. The second AI network 156 may include a plurality of nodes and a plurality of layers, and may generate output data by processing input data by transferring data between the plurality of nodes and the plurality of layers.

The memory 158 stores computer program instructions, information, and contents necessary for the operation of the image reproducing apparatus 150 . Memory 158 may include volatile storage media, non-volatile storage media, or a combination thereof. As described above with respect to the memory 116 of the image providing apparatus 110 , the memory 158 may be implemented as various types of storage media.

The output unit 160 outputs data or control signals generated or processed by the image reproducing apparatus 150 . The output unit 160 outputs the third image from the image reproducing apparatus 150 . The output unit 118 may include various types of output interfaces such as a communication interface, a display, a speaker, and a touch screen.

According to an embodiment, the output unit 160 may output the third image by displaying it corresponding to the display.

According to another embodiment, the output unit 160 may include a communication unit or a communication interface. The output unit 160 may transmit the third image to an external device, a display device, or a multimedia device through a communication interface. For example, the output unit 160 may transmit display data of the third image to an external display device.

2 is a diagram illustrating a structure of a processor of an image providing apparatus according to an exemplary embodiment.

In the present disclosure, blocks within the processor 112 or 154 may correspond to at least one software processing block or at least one dedicated hardware processor, and a combination thereof. Blocks defined in the processor 112 or 154 in the present disclosure are only an example of a software processing unit for performing embodiments of the present disclosure, and a processing unit that performs embodiments of the present disclosure in various ways other than the processing unit disclosed in the present disclosure can be defined.

According to an embodiment, the processor 112 of the image providing apparatus 110 may include a metadata generator 210 and an encoder 220 . The metadata generator 210 receives the first image and generates AI metadata.

According to an embodiment, the first image may be downscaled and encoded by the processor 112a. For example, the first image may be downscaled by the metadata generator 210 , the encoder 220 , or a separate downscaler block. According to an embodiment, the metadata generator 210 generates AI metadata from a high-resolution high-definition image before downscaling. The image providing apparatus 110 may generate AI metadata without loss of information related to image quality by generating AI metadata from a high-resolution high-definition image.

The metadata generator 210 may use the first AI network 114 in some processing. The first AI network 114 may be provided in the processor 112a, in the image providing device 110 as a separate dedicated processor, or in an external device.

The meta data generating unit 210 may include a meta information extracting unit 212 and a meta information lightening unit 214 . Also, according to an embodiment, the meta data generating unit 210 may further include a meta information compressing unit 216 . The meta information compression unit 216 may be omitted in some embodiments.

The meta information extraction unit 212 extracts AI metadata from the first image. The AI metadata may include class information indicating the type of object included in the first image and a class map indicating a class corresponding to each region in the first image. The class map is information defining a class corresponding to each pixel with respect to a pixel array corresponding to the first image. The class map may be generated for each of at least one class included in the class information of the first image. For example, when four classes of default, sky, grass, and plant are defined for the first image and the class information defines four classes, the class map is generated for each of default, sky, grass, and plant, so a total of 4 class maps can be generated.

The meta information extractor 212 may detect an object from the first image, define a predetermined number of classes corresponding to the types of the detected objects, and map the objects that are not defined as classes to the predetermined class. The object represents an object included in the image. The type of object may be defined in various ways according to embodiments, for example, sky, water, grass, mountain, building, plant, or animal. ) and the like. If an object corresponding to water is detected in the first image, but the water object is not included in class information because the number of pixels corresponding to the water object is small, the area corresponding to the water object is one of the classes defined for the first image. can be mapped to The class map may be implemented in the form of a probability map indicating a probability corresponding to a corresponding class for each pixel. For example, a class map corresponding to sky may indicate a probability that each pixel corresponds to sky. The resolution of the class map may be the same as the resolution of the first image or may be smaller than the resolution of the first image.

The meta information extraction unit 212 may extract AI metadata from the first image by using the first AI network 114 . According to an embodiment, the meta information extraction unit 212 may use the first AI network 114 to extract a class map from the first image and class information.

The meta information extraction unit 212 may generate AI metadata in a form before the AI metadata undergoes weight reduction processing, and the AI metadata may include class information and a plurality of class maps corresponding to each class.

The meta information weight reduction unit 214 performs a weight reduction process for reducing the data capacity of the AI metadata extracted by the meta information extraction unit 212 . The meta information weight reduction unit 214 reduces the number of bits of the AI metadata by reducing the weight of the plurality of class maps, thereby reducing the weight of the AI metadata. The meta information lightweight unit 214 generates a lightweight class map by defining a representative value for each location (pixel) in one map. A representative value of the lightweight class map indicates a class corresponding to each pixel. Accordingly, the lightweight AI metadata includes a lightweight class map and class information.

The meta-information compression unit 216 reduces the size of the lightweight class map by reducing the horizontal and vertical sizes of the class map lightweighted by the meta-information lightweight unit 214 . The meta information compression unit 216 compresses the lightweight class map by applying a lossless compression method used for image compression. The meta information compression unit 216 compresses the lightweight class map and class information to generate and output compressed AI metadata. The compressed AI metadata includes a lightweight class map and class information.

The encoder 220 encodes the first image to generate an encoded image. The encoder 220 may encode the first image using various types of image encoding algorithms. The encoder 220 generates prediction data by predicting the first image, generates residual data corresponding to a difference between the first image and the prediction data, and transforms the spatial domain residual data into frequency domain components. A process of transforming, a process of quantizing the residual data transformed into a frequency domain component, and a process of entropy encoding the quantized residual data may be performed. This encoding process uses frequency conversion such as MPEG-2, H.264 AVC (Advanced Video Coding), MPEG-4, HEVC (High Efficiency Video Coding), VC-1, VP8, VP9 and AV1 (AOMedia Video 1). It may be implemented through one of the image compression methods.

The compressed AI metadata may be included in the header of the encoded image and output, or may be output separately from the encoded image. According to an embodiment, the meta-information extractor 212 generates image characteristic information and an image characteristic map corresponding to other types of image characteristics in addition to the class information and the class map, and the AI metadata includes additional image characteristic information and the image. It may include a feature map. The additional image characteristics may include at least one of frequency, texture, semantics, or imaging parameters or a combination thereof.

According to an embodiment, the meta information extractor 212 may generate frequency information and a frequency map by analyzing frequency information from the first image. AI metadata may include frequency information and a frequency map. The frequency information may include information on frequency regions included in the first image. The frequency map may indicate, for each frequency domain, a probability corresponding to the corresponding frequency domain in the first image. For example, five frequency ranges may be defined, and five frequency maps corresponding to each frequency region may be generated. According to an embodiment, the low-frequency region is a flat region within the image, and since the improvement in detail by image quality processing is small, it is excluded from the image characteristic information, and the image characteristic information can be defined only for a high-frequency region of a predetermined frequency or higher. .

According to another embodiment, the meta information extractor 212 may generate texture information and a texture map by analyzing texture information from the first image. AI metadata may include texture information and texture maps. The texture information is information corresponding to the density in each region of the first image. The meta information extractor 212 may calculate a density from each region of the first image and generate texture information according to the density value. A high density may indicate that a high complexity and various patterns exist within a corresponding region. The density (Spatial Information, SI) may be calculated based on Equation 1 below.

[Equation 1]

Spatial Information(SI) = stdev[Sobel(I) _HR )]

To calculate the density, the first image may be filtered with a _{Sobel filter (Sobel(I HR )).} Here, I _HR represents an image input to the filter, and corresponds to the first image in the present embodiment. Next, a standard deviation is calculated for pixels in the first image processed by the Sobel filter (stdev[Sobel(I _HR )]). According to the distribution of the density values, predetermined regions according to the density values may be defined, and regions of the density values detected in the first region may be included in the texture information. The texture map is map information indicating a region of the first image corresponding to each density region included in the texture information. The texture map may be expressed in the form of a probability map.

According to another embodiment, the meta information extractor 212 may extract shooting parameter information and shooting environment information applied to the first image and record the extracted information in AI metadata. Shooting parameters include, for example, zoom parameter, aperture information, focal length information, flash usage information, white balance information, ISO information, exposure time information, shooting mode information, out-focusing information, AF (Auto-focusing) information, AE ( Auto-exposure) information, person detection information, scene information, and the like may be included. The photographing environment information may include illuminance information. The image characteristic map may be generated together when a value corresponding to each pixel in the first image can be defined in the corresponding shooting parameter information or the shooting environment information. When a value corresponding to each pixel in the first image cannot be defined, an image characteristic value may be defined for the first image. For example, zoom parameters, aperture information, focal length information, flash usage information, white balance information, ISO information, exposure time information, shooting mode information, out-focusing information, AF (Auto-focusing) information, AE (Auto-exposure) information ) information, scene information, and illuminance information, an image characteristic value corresponding to the first image may be defined. The person detection information may include an image characteristic map indicating an area in which a person is detected in the first image or an image characteristic value indicating a coordinate of an area in which the person is detected.

In the present disclosure, an embodiment in which the meta information extraction unit 212 generates AI metadata including class information and a class map will be mainly described. However, embodiments of the present disclosure are not limited to class information and a class map.

3 is a diagram illustrating a processing process of a meta information extracting unit according to an embodiment of the present disclosure.

The meta information extraction unit 212a receives the first image 302 as input, generates and outputs AI meta data. The first image 302 may correspond to the high-definition image before the image encoding process. Also, the first image 302 may be a high-resolution image before the downscaling process.

The meta information extraction unit 212a receives the first image 302 and performs image characteristic analysis processing 312 . The image characteristic analysis processing 312 generates image characteristic information from the first image 302 and the plurality of segmentation probability maps. The image characteristic information includes class information indicating a type of a class included in the first image 302 . In the image characteristic analysis processing 312 , image characteristic information is generated from the first image 302 using various types of image characteristic analysis algorithms. According to an embodiment, the image characteristic analysis processing 312 may generate class information by extracting a main class included in the first image 302 from a plurality of segmentation probability maps generated by the information generating network 316 . have. The image characteristic analysis processing 312 may use, as an image characteristic analysis algorithm, for example, a Fourier transform, a wavelet transform, or the like, capable of analyzing frequency information. In addition, the image characteristic analysis processing 312 may use a Sobel filter, a Laplacian filter, a Canny filter, etc. capable of analyzing edge information in an image as an image characteristic analysis algorithm. Also, the meta information extraction unit 212a may use morphology erosion and dilation operations capable of analyzing texture information in an image using an image characteristic analysis algorithm. The meta information extractor 212a may combine a plurality of image characteristic analysis algorithms in the image characteristic analysis process 312 to subdivide image characteristics into various image characteristics. For example, the meta information extraction unit 212a may calculate at least one or a combination of frequency information, edge information, and texture information by using a plurality of image characteristic analysis algorithms in the image characteristic analysis process 312 . have.

The image characteristic analysis processing 312 selects some of the objects detected in the first image 302 using a plurality of segmentation probability maps generated by the information generation network 316 and defines a predetermined number of classes. have. As described above, when the number of types of detected objects is greater than a predetermined number, the meta information extracting unit 212a may define a predetermined number of classes by extracting a predetermined number of objects in the order of their corresponding regions being wide. . For example, in the image characteristic analysis processing 312 , when a total of five objects of sky, grass, plant, water, and default are detected from the first image 302 and the number of predetermined classes is four, the corresponding region By extracting a predetermined number of objects in this wide order, sky, grass, plant, and default may be defined as classes, and water may not be defined as classes. According to an embodiment, the meta-information extractor 212a may always include a default class indicating an area that is not classified as a specific object in the class, regardless of the size of the area.

The information generating network 316 receives the first image 302 and generates a segmentation probability map for a plurality of predefined objects. For example, 32 objects are predefined for the information generating network 316 , and the information generating network 316 may receive the first image 302 and generate a segmentation probability map for each of the 32 objects. . The segmentation probability map indicates a probability that each pixel or region of the first image 302 corresponds to an object corresponding to the corresponding segmentation probability map. The information generating network 316 includes the first AI network 114 or 1 AI network 114 is available. According to an embodiment, the first AI network 114 may generate a plurality of segmentation probability maps from the first image 302 by using the SFTGAN algorithm.

In addition, the meta information extraction unit 212a performs a meta data generation process 318 for generating AI meta data including class information and a class map. In the meta data generation processing 318, a segmentation probability map of an object included in the class information generated in the image characteristic analysis processing 312 is extracted from among the plurality of segmentation probability maps generated by the information generation network 316, and a class map to define In addition, the meta data generation process 318 outputs AI meta data including a plurality of class maps and class information. The meta information extraction unit 212a may generate AI metadata including class information and a class map based on a predetermined standard.

4 is a diagram illustrating a process of generating object information and class information from a first image in an image characteristic analysis process, according to an embodiment of the present disclosure.

According to an embodiment, the metadata extractor 212a extracts information indicating the type of object included in the first image 302 from the first image 302 in the image characteristic analysis process 312 . The information generating network 316 includes a plurality of segmentation probability maps 412a, 412b, ..., 412h corresponding to predefined objects such as sky, water, grass, mountains, buildings, plants, and animals in the first image 302 . create In the image characteristic analysis process 312 , class information is generated from the plurality of segmentation probability maps 412a, 412b, ..., 412h generated by the information generating network 316 .

The plurality of segmentation probability maps 412a, 412b, ..., 412h are generated for each object and are probability maps indicating a probability that each region of the first image corresponds to the corresponding object. In the present disclosure, a probability map indicating a probability corresponding to a predetermined class is referred to as a class map, and a probability map indicating a probability corresponding to a predetermined object is referred to as a segmentation probability map. The class map may be defined for classes included in the class information after class information is defined in the image characteristic analysis process 312 .

According to an embodiment, the plurality of segmentation probability maps 412a , 412b , ..., 412h may indicate a probability corresponding to each object type in each pixel or region of the first image 302 . The plurality of segmentation probability maps 412a , 412b , ..., 412h may be defined with the same resolution as the first image 302 , or may be defined with a lower resolution than the first image 302 .

The plurality of segmentation probability maps 412a, 412b, ..., 412h may include segmentation probability maps 412a, 412b, ..., 412h corresponding to at least one predefined object type, respectively. Each segmentation probability map 412a, 412b, ..., 412h includes information indicating whether each pixel or region of the first image 302 corresponds to a corresponding object type or a probability corresponding to the corresponding object type. For example, the segmentation probability map 412b corresponding to the sky has a value of 1 in the region of the first image 302 corresponding to the sky, and has a value of 0 in the region of the first image 302 that does not correspond to the sky. can have a value. As another example, each pixel of the segmentation probability maps 412a, 412b, ..., 412h may have a positive real value representing a probability. Also, a segmentation probability map 412a corresponding to a default value may be calculated, which may indicate a region that does not correspond to any of the predefined object types. According to an embodiment, each pixel value of the plurality of segmentation probability maps 412a, 412b, ..., 412h is, for the corresponding pixel, the plurality of segmentation probability maps 412a, 412b, ..., 412h, respectively. may be set such that the sum of pixel values of .

In the image characteristic analysis process 312 , the meta data extraction unit 212a performs an average of the segmentation probability maps 412a, 412b, ..., 412h from the plurality of segmentation probability maps 412a, 412b, ..., 412h. The class to be included in class information is determined by deriving the top n classes. The image characteristic analysis process 312 may be performed through the steps of the segmentation probability map process 430 , the upper rank extraction 440 , and the class information generation 450 .

The segmentation probability map processing 430 calculates an average value for non-zero pixel values for each of the plurality of segmentation probability maps 412a, 412b, ..., 412h, and determines a pixel value lower than the average value in the corresponding class. Remove. For example, if the average value is 0.4 for a non-zero pixel value in the segmentation probability map 412b for the Sky object, pixel values smaller than 0.4 are removed from the segmentation probability map 412b. This processing is for preventing misclassification and removing unnecessary values.

Next, the upper rank extraction process 440 determines the top k segmentation probability maps 412a, 412b, 412f based on the average value of each segmentation probability map 412a, 412b, ..., 412h. k is a predefined natural number as the number of classes. A high average value of the segmentation probability maps 412a, 412b, ..., 412h means that the area occupied by the corresponding object in the first image 320 is large. The image characteristic analysis processing 312 extracts n objects of higher rank and defines them as classes, so that the type of object occupying a large area in the first image 320 may be accurately extracted. The segmentation probability maps 412a , 412b , and 412f selected as the upper rank are one embodiment, and in each embodiment, the segmentation probability map of the upper rank may be determined in various ways.

Next, the class information generation process 450 defines objects corresponding to the segmentation probability maps 412a, 412b, and 412f of the higher rank as classes, and generates the class information 420 . The class information 420 indicates a plurality of classes selected as a higher rank. The class information 420 is, for example, information on classes whose average values of the segmentation probability maps 412a, 412b, and 412f belong to the top k, and includes Default, Sky, and Bulding in the example of FIG. 4 .

According to an embodiment, the metadata extraction unit 212a selects an object that is not selected as a class from among the objects whose detection values of the segmentation probability maps 412a, 412b, ..., 412h are not 0, the selected object, that is, Map to one of the classes. For example, an object corresponding to a plant (corresponding to the segmentation probability map 412g) is not selected in the top three objects, but may be mapped and transmitted to a class corresponding to the sky among the defined classes. The types of objects mapped to each other may be predefined. Among the predefined object types, object types that similarly restore detail when applied to an actual AI network may be predefined as objects mapped to each other. For example, human hair and animal fur may be defined as objects mapped to each other. The metadata extractor 212a may perform mapping between objects by using information about objects that are mapped to each other previously stored in the memory 116 or the like. According to an embodiment, when an object mapped to a certain object is not defined, the corresponding object may be mapped to an object corresponding to a default.

The image characteristic analysis processing 312 may generate a class map in which a plurality of segmentation probability maps 412a, 412b, ..., 412h are synthesized by performing the object mapping processing. For example, the image characteristic analysis processing 312 performs the segmentation probability map 412a, 412b, 412f of the object selected as a class, and at least one of the segmentation probability maps 412c, 412d, 412e, 412g, 412h can be synthesized to create a class map. The synthesis of the segmentation probability maps 412a, 412b, ..., 412h may be performed using averaging of pixel values, weighted averaging, or the like. When the mapping processing for the segmentation probability maps 412a, 412b, ..., 412h is performed, the meta data generation processing 318 receives the class map generated by the mapping processing from the image characteristic analysis processing 312 . The metadata generation processing 318 may receive the class information and the class map from the image characteristic segmentation processing 312 to generate AI metadata. As another example, the image characteristic analysis processing 312 may perform object mapping to each other. , and transmits information about the mapping between objects to the meta data generation process 318 . The meta data generation process 318 generates a class map from the segmentation probability map generated in the information generation network 316 based on the information on the inter-object mapping. In the meta data generation process 138, the synthesis of the segmentation probability maps 412a, 412b, ..., 412h may be performed using averaging of pixel values, weighted averaging, or the like.

5 is a diagram illustrating an operation of an information generating network of a meta information extracting unit according to an embodiment of the present disclosure.

The information generating network 316 receives the first image 502 and generates a segmentation probability map 504 for a plurality of predefined objects. The plurality of objects may be predefined in the design process of the information generating network 316 . For example, 32 objects are predefined for the information generation network 316 , and the information generation network 316 receives the first image 302 and generates a segmentation probability map 504 for each of the 32 objects. can do. The information generation network 316 is trained to generate a segmentation probability map 504 for preset objects. The information generation network 316 includes a plurality of computation layers, and performs processing by the plurality of computation layers based on parameter values generated by learning by a plurality of training data.

According to an embodiment, the information generating network 316 may be configured to generate the segmentation probability map 504 based on not only the object but also other image characteristic information such as a frequency characteristic and a texture characteristic. For example, the information generating network 316 may generate and output the segmentation probability map 504 for each of a plurality of preset frequency domains.

The information generating network 316 may correspond to the first AI network 114a including a plurality of layers or may use the first AI network 114a. The first AI network 114a may include a 1-1 AI network 510 and a 1-2 AI network 520 . The first AI network 114a may have, for example, a CNN structure. In the present disclosure, the 1-1 AI network 510 and the 1-2 AI network 520 are terms for referring to a group of a plurality of layers, and the first AI network 114a is the 1-1 AI network ( 510) and the 1-2 AI network 520 are not meant to be implemented separately. The first AI network 114a may be configured in various ways including a plurality of layers, and may include one or a plurality of layer groups.

The 1-1 AI network 510 generates a feature map from the first image. The feature map is data obtained by extracting predetermined implicit information from the first image.

The 1-1 AI network 510 may include a plurality of layers and have a single forward pass structure. According to an embodiment, the 1-1 AI network 510 may include a combination of at least one convolutional layer and at least one maximum pooling layer. For example, the 1-1 AI network 510 may have a structure in which an arrangement of a predetermined number of convolutional layers and a maximum pooling layer is repeated. The first image 502 and class information input to the 1-1 AI network 510 are sequentially processed and transmitted through a plurality of layers.

The 1-1 AI network 510 generates a feature map by processing the first image as a plurality of convolutional layers. The 1-1 AI network 510 processes the intermediate data, which is a result of processing the input data through a predetermined number of convolutional layers, using the maximum pooling layer, and repeats the process of extracting only the main feature values from the intermediate data. can do. The 1-1 AI network 510 may generate a downscaled feature map from the first image. The feature map output from the 1-1 AI network 510 is input to the 1-2 AI network 520 .

The convolution layer performs convolution processing on input data using a predetermined filter kernel. A predetermined feature is extracted from the input data by the filter kernel. For example, the convolutional layer may perform a convolution operation on input data using a filter kernel having a size of 3*3. The convolutional layer determines respective pixel values of the feature map through multiplication and addition operations between parameter values of the filter kernel and pixel values in input data corresponding thereto. The parameter values of the filter kernel and parameters of the convolution operation may be predetermined by learning. The convolutional layer may additionally perform a padding process to generate output data having the same size as the input data.

A maximum pooling layer applies a predetermined filter to input data, extracts a maximum value from among input data in the filter, and outputs it. For example, the maximum value pooling layer extracts and outputs the maximum value within the 2*2 filter while moving the 2*2 filter with respect to the input data. The maximum pooling layer may reduce the size of the input data by a ratio of the filter size by applying a filter to the input data in units of blocks corresponding to the filter size. For example, a maximum pooling layer using a 2*2 filter reduces the horizontal and vertical lengths of input data by half. The maximum value pooling layer may be used to transfer only the main value in the input data to the next layer.

The 1-2 AI network 520 receives the feature map and generates a plurality of segmentation probability maps 504 . The 1-2 AI network 520 may include a plurality of layers and have a single forward pass structure. According to an embodiment, the 1-2 AI network 520 may include various combinations of at least one convolutional layer, at least one activation layer, and at least one minimum value pooling layer.

The activation layer performs processing for imparting non-linear characteristics to the input data. The activation layer may perform an operation based on, for example, a sigmoid function, a Tanh function, a Rectified Linear Unit (ReLU) function, and the like, but is not limited thereto.

The min pooling layer applies a predetermined filter to input data, extracts a minimum value from among the input data in the filter, and outputs it. For example, the minimum value pooling layer extracts and outputs the minimum value within the 2*2 filter while moving the 2*2 filter with respect to the input data. The minimum pooling layer may reduce the size of the input data by a ratio of the filter size by applying a filter to the input data in units of blocks corresponding to the filter size. For example, a minimum pooling layer using a 2*2 filter reduces the horizontal and vertical lengths of input data by half.

According to an embodiment, the 1-2 AI network 520 may include a first layer group 522 , an upscaler 524 , and a second layer group 526 . The first layer group 522 , the upscaler 524 , and the second layer group 526 may be disposed in a single forward pass structure.

The first layer group 522 receives the feature map, performs non-linear processing on the feature map, and outputs it. The first layer group 522 may include a combination of at least one convolutional layer and at least one activation layer. For example, the first layer group 522 may have a structure in which a convolutional layer and an activation layer are alternately disposed.

The upscaler 524 receives the output of the first layer group 522 and performs an upscaling process. The upscaler 524 may be implemented using various types of upscaling algorithms. According to an embodiment, the upscaler may correspond to an AI upscaler having a DNN structure.

The second layer group 526 receives the output of the upscaler 524 and performs additional processing. The second layer group 526 may include a combination of at least one convolutional layer and at least one minimum value pooling layer. For example, the second layer group 526 may have a structure in which a plurality of convolutional layers are disposed and a minimum value pooling layer is disposed last.

When other image characteristics other than the object type are analyzed together, the first AI network 114a corresponding to each image characteristic may be separately provided. For example, when class, frequency, and texture are analyzed as image characteristics, a first AI network corresponding to a class, a first AI network corresponding to a frequency, and a first AI network corresponding to a texture are separately provided. can be

6 is a diagram illustrating an operation of a meta information lightweight unit of an image providing apparatus according to an embodiment of the present disclosure.

The meta data generating unit 210 of the processor 112a includes a meta information lightening unit 214a. The meta information weight reduction unit 214a receives the AI metadata generated by the meta information extraction unit 212 and performs a weight reduction process for reducing the data capacity of the AI metadata.

The meta-information lightweight unit 214a receives the class information and the class map output from the meta-information extraction unit 212, and provides a lightweight class map 612 having representative value information for each location and a class corresponding to each representative value. By converting the indicated class information 614, lightweight AI metadata 610 may be generated.

The meta-information lightweight unit 214a, from the class information and class maps 620a, 620b, 620c, and 620d, each pixel has a representative value of the class of the segmentation area corresponding to the pixel. Lightweight class map 612. create The lightweight class map 612 is a map having representative value information corresponding to each pixel. According to an embodiment, a color corresponding to each representative value is defined, and each pixel of the lightweight class map 612 may have a color corresponding to the value of the corresponding pixel ( 616 ).

The representative value of each pixel may be determined based on probability values displayed on the plurality of class maps 420 . For example, the representative value of the lightweight class map 612 may be defined as a value corresponding to a class having the highest probability value in the corresponding pixel. The number of representative values of the lightweight class map 612 may be determined based on the number of classes (k) included in the class information. Accordingly, the number of bits allocated to each pixel of the lightweight class map 612 may be determined as _{(log 2 k) based on the number of classes (k) included in the class information.} In addition, the lightweight class map 612 _{may be expressed as (log 2} k)*w*h bits. Here, w denotes the width of the lightweight class map, and h denotes the height of the lightweight class map. The width and height of the lightweight class map may be the same as the first image 602 , or may be smaller than the width and height of the first image 602 .

The meta-information lightweight unit 214a includes information on a class type included in the class information generated by the meta-information extraction unit 212 and information on a class corresponding to each representative value of the lightweight class map 612 . Class information 614 is generated. For example, if the selected class is water, grass, default, the class information 614 may indicate that the selected class is water, grass, default, a first value is assigned to water, a second value is assigned to pool, and , information that a third value is assigned for the default. The first value, the second value, and the third value may be defined as values as small as possible so as to be expressed by the minimum number of bits.

The meta information lightweight unit 214a outputs the lightweight AI metadata 610 including the lightweight class map 612 and the lightweight class information 614 .

When the AI metadata is not lightweight, a plurality of class maps 620a, 620b, 620c, and 620d include class maps 620a, 620b, 620c, and 620d for each class, k for one frame image *w*h bits may be required. In addition, if the class maps 620a, 620b, 620c, and 620d indicate the probability of corresponding to the corresponding class for each pixel (618), it may be necessary to generate a class map as a floating value to represent the probability value, When the level of the floating value is 32, 32*k*w*h bits may be required for a plurality of class maps of one frame image. On the other hand, when the AI metadata is lightened as in the present embodiment, the _{AI metadata can be expressed as (log 2} k)*w*h bits for one frame image. Therefore, by lightening the AI metadata, the number of bits of the AI metadata _{can be reduced to ((log 2} k) / (ck)) level compared to before the weight reduction. Here, c denotes the number of bits allocated to one pixel of the existing class maps 620a, 620b, 620c, and 620d (32 bits in the previous example). Therefore, according to the present embodiment, there is an effect that the number of bits required for AI metadata can be significantly reduced.

7 is a diagram illustrating an operation of a meta information compression unit according to an embodiment of the present disclosure.

The meta-information compression unit 216 compresses the class information lightened by the meta-information lightweight unit 214 . The meta information compression unit 216 may perform compression on class information and compression on a lightweight class map. Compression of class information will be described with reference to FIGS. 7 and 8 , and compression of a lightweight class map will be described with reference to FIG. 9 .

According to an embodiment, the class information 730 may have a bit value allocated to each of the predefined objects 710 . For example, the class information 730 assigns a value of 1 to the object 710 selected as a class, for each of the predefined objects 710 , and assigns a value of 0 to the remaining objects 710 . can For example, if there are 8 predefined objects 710 for AI metadata, 8 bits of class information may be defined, and each object 710 may be allocated to each bit. Also, the bit value corresponding to each object 710 may have a value of 0 or 1 depending on whether the corresponding object 710 is selected as a class. Accordingly, the class information of FIG. 7 indicates class information 730 in which a default class, an empty class, and a full class are selected as classes.

The meta information compression unit 216 may perform compression processing on the class information 730 . The meta information compression unit 216 may compress the class information 730 and the class map using a compression method used for symbol compression, and transmit the compressed information as a bitstream. The meta information compression unit 216 may compress the class information 730 and the class map using, for example, run-length encoding, Huffman coding, or arithmetic coding.

8 is a diagram illustrating a method of compressing class information in a meta information compression unit according to an embodiment of the present disclosure. In FIG. 8, as in the embodiment described above with reference to FIG. 7, an embodiment in which one bit is allocated to each object and class information is indicated by values of 1 and 0 will be mainly described.

According to an embodiment, the first image is video data including a plurality of frames, and class information may be defined for each sequence including the plurality of frames (Frame_1, Frame_2, ..., Frame_n).

According to an embodiment, the number of frames included in the sequence may be preset to a predetermined number.

According to another embodiment, the number of frames included in the sequence may be dynamically defined based on the image content of the first image. For example, when an image characteristic change of a predetermined reference value or more is detected in the first image, when a scene change is detected, etc., different sequences may be distinguished and defined. In addition, the maximum value of the number of frames included in the sequence is set, and the meta information extraction unit 212a sets the maximum number of frames included in the sequence even if it is not detected that the criterion for sequence change is not satisfied in the image content of the first image When a value is reached, the sequence can be further defined so that each sequence contains only frames below the maximum value.

According to another embodiment, a frame included in a sequence may be determined based on a candidate class detected in a frame of the corresponding sequence. The metadata extractor 212a may define a sequence such that the number of candidate classes having a detection value greater than or equal to a predetermined value in a frame of the corresponding sequence does not exceed the predetermined number. For example, when there are eight predefined objects, the meta information compression unit 216 may define a sequence such that the number of objects having a detection value greater than or equal to a predetermined value in a corresponding sequence frame does not exceed four.

According to an embodiment, the class information 810 may include sequence class information 812 and frame class information 814 . The class information 810 may include as much frame class information 814 as the number corresponding to the number of frames included in the sequence. In addition, the class information 810 may include additional information such as a frame range and the number of frames included in a sequence corresponding to the class information 810 . In addition, according to an embodiment, the class information 810 includes additional information such as the length of the corresponding class information 810 and the range of the sequence class information 812 and the frame class information 814 in the corresponding class information 810 . may include

The sequence class information 812 indicates candidate classes detected from frames included in the corresponding sequence. The sequence class information 812 may include the number of bits corresponding to the number of predefined candidate classes. When the number of candidate classes selected from the class information of each frame is k, the sequence class information 812 may have a value of 1 for a number of candidate classes greater than or equal to k. For example, when the frame class information 814 is information indicated by selecting and representing the upper three candidate classes of the detection values of object detection among the candidate classes, the sequence class information 812 includes three or more values of 1 (eg, For example, it can have 4 values of 1). In this case, the sequence class information means that frames included in the corresponding sequence include objects of candidate classes corresponding to a predetermined number (eg, three) of the four candidate classes.

The frame class information 814 is information corresponding to each frame (Frame_1, Frame_2, ..., Frame_n) included in the sequence, and includes information about a candidate class corresponding to an object detected in each frame. The frame class information 814 may be sequentially arranged according to a frame order and recorded in a bit stream.

The frame class information 614 may have a data size (number of bits) corresponding to the number of bits having a value of 1 in the sequence class information 812 . For example, when the number of bits having a value of 1 in the sequence class information 812 is 4, a plurality of frame class information 814 corresponding to the sequence may be defined as 4 bits. Accordingly, according to an embodiment, the number of bits of the frame class information 814 may vary according to the sequence class information 812 . According to another embodiment, the number of bits of the frame class information 814 is preset, and the sequence class information 812 may have a value of 1 as many as the number of bits corresponding to the number of bits of the frame class information 814 . . To this end, the sequence may be defined to have a value of 1 or less corresponding to the number of bits of the frame class information 814 .

9 is a diagram illustrating an operation of a meta information compression unit of an image providing apparatus according to an embodiment of the present disclosure.

The meta information compression unit 216 downscales the lightweight class map 902 generated by the meta information lightweight unit 214a, compresses the lightweight class map 902, and generates a compressed class map 904 . to output Compression processing of the lightweight class map 902 may use a lossless compression method used for image compression. For example, the meta information compression unit 216 may compress the lightweight class map 902 by grouping pixel values in the lightweight class map 902 and extracting maximum probability class information. The meta information compression unit 216 may compress the lightweight class map 902 using, for example, run-length encoding, Huffman coding, or arithmetic coding. According to an embodiment, the meta information compression unit 216 may compress the lightweight class information using a predetermined compression method. The meta information compression unit 216 generates compressed AI metadata including the compressed class information and the compressed class map 904 .

According to the experimental results based on the present disclosure by the present applicant, it was confirmed that similar detail is generated without deterioration of image quality in the image reproducing apparatus even when the lightweight class map 902 of the AI metadata is compressed. Therefore, according to the present embodiment, there is an effect of reducing the capacity of AI metadata without deterioration of image quality, thereby increasing the efficiency of data storage and transmission.

10 is a flowchart illustrating a method of controlling an image providing apparatus according to an embodiment of the present disclosure.

Each step of the method for controlling an image providing apparatus of the present disclosure may be performed by various types of electronic devices including a processor, a memory, and an output unit, and using a machine learning model. The present specification will be mainly described with respect to an embodiment in which the image providing apparatus 110 according to embodiments of the present disclosure performs a method for controlling the image providing apparatus. Accordingly, the embodiments described for the image providing apparatus 110 are applicable to the embodiments of the method for controlling the image providing apparatus, and on the contrary, the embodiments described for the method for controlling the image providing apparatus 110 are for the image providing apparatus 110 . Applicable to the embodiments. The method for controlling an image providing apparatus according to the disclosed embodiments is not limited to being performed by the image providing apparatus 110 disclosed herein, and may be performed by various types of electronic devices.

The image providing apparatus receives the first image (S1002).

The image providing apparatus generates AI metadata by performing AI metadata extraction processing on the first image (S1004). The image providing apparatus generates image characteristic information and an image characteristic map by performing a predetermined image characteristic analysis on the first image. The image providing apparatus generates a plurality of segmentation probability maps for predefined objects by inputting the first image to the first AI network. The image providing apparatus generates class information, which is information about an object included in the first image, from a plurality of segmentation probability maps, and generates a class map for each class defined in the class information. The image providing apparatus generates AI metadata including class information and a class map.

Next, the image providing apparatus performs a process of reducing the weight of AI metadata including class information and class map. The image providing apparatus lightens the plurality of class maps into one lightweight class map.

Next, the image providing apparatus compresses the lightweight class map to generate a compressed class map. Lightweight class information may be further compressed. The image providing apparatus generates and outputs compressed AI metadata including compressed class information and a compressed class map. Since the contents of the AI metadata generation, weight reduction, and compression processing described above for the metadata generator 210 are similarly applicable to the AI metadata generation operation of step S1004, a redundant description will be omitted.

Also, the image providing apparatus generates an encoded image by performing an encoding process on the first image (S1006). The image providing apparatus may encode the first image using various types of image encoding algorithms. The image providing apparatus predicts the first image to generate prediction data, generates residual data corresponding to a difference between the first image and the prediction data, and transforms the residual data, which is a spatial domain component, into a frequency domain component. ), a process of quantizing the residual data transformed into a frequency domain component, and a process of entropy encoding the quantized residual data may be performed. This encoding process uses frequency conversion such as MPEG-2, H.264 AVC (Advanced Video Coding), MPEG-4, HEVC (High Efficiency Video Coding), VC-1, VP8, VP9 and AV1 (AOMedia Video 1). It may be implemented through one of the image compression methods.

`Next, the image providing apparatus outputs the compressed AI metadata and the encoded image (S1008). The image providing apparatus may output compressed AI metadata and encoded images through various types of output interfaces such as a communication interface, a display, a speaker, and a touch screen.

11 is a diagram illustrating structures of an input unit, a processor, and an output unit of an image reproducing apparatus according to an embodiment of the present disclosure.

The image reproducing apparatus 150 receives the encoded image and compressed AI metadata through the input unit 152 . The encoded image and the compressed AI metadata are data corresponding to the encoded image and the compressed AI metadata generated by the image providing apparatus 110 . The input unit 152 outputs the encoded image to the decoder 1110 , and outputs compressed AI metadata to the image quality processing unit 1120 .

The input unit 152 may correspond to a communication unit that receives an encoded image and compressed AI metadata from an external device, an input/output interface, and the like.

The processor 154a includes a decoding unit 1110 and an image quality processing unit 1120 .

The decoder 1110 generates a second image by performing a decoding process on the encoded image. The decoder 1110 may decode the encoded image by using a decoding method corresponding to an image encoding algorithm applied to the encoded image. Information on the image encoding algorithm applied to the encoded image may be obtained from the header of the encoded image or additional information input together with the encoded image. The encoded image may be decoded using a method corresponding to the encoding method used by the encoder 220 of the image providing apparatus 110 . The decoder 1110 entropy-decodes image data to generate quantized residual data, inverse quantizes the quantized residual data, generates prediction data, and generates a second image using the prediction data and the residual data. It may include a process of restoration, and the like. Such a decoding process corresponds to one of the video compression methods using frequency conversion, such as MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 used in the process of encoding the encoded video. It can be implemented through an image restoration method.

The image quality processing unit 1120 receives the decoded second image and the compressed AI metadata, performs image quality improvement processing, and generates and outputs a third image. The image quality improvement processing may include various types of existing image quality processing for the decoded second image, and image quality processing using AI metadata according to embodiments of the present disclosure. The image quality processing unit 1120 may include an input information processing unit 1122 , a meta information synthesis unit 1124 , a meta information restoration unit 1126 , and a meta information processing unit 1128 .

AI metadata input through the input unit 152 is input to the meta information restoration unit 1126 . The meta information restoration unit 1126 receives the compressed AI metadata input from the image providing device 110 and performs a decompression process 1130 and a light weight restoration process 1132 to recover a plurality of compressed AI metadata. It restores the class map and class information of The plurality of class maps and class information correspond to the class map and class information output from the meta information extraction unit 212 of the image providing apparatus 110 .

The meta information restoration unit 1126 performs a decompression process 1130 on the AI metadata that is lightened by the meta information lightweight unit 214 of the image providing device 110 and compressed by the meta information compression unit 216 . Lightweight class map and lightweight class information are generated. The decompression process 1130 may be performed by a decoding process corresponding to the encoding algorithm used by the meta information compression unit 216 of the image providing apparatus 110 . The lightweight class map and the lightweight class information generated by the decompression process 1130 are the lightweight class map 812 and the lightweight class previously generated by the meta information lightweight unit 214a of the image providing device 110 . It corresponds to information 814 .

In addition, the meta information restoration unit 1126 performs the lightweight restoration process 1132 to generate a plurality of restoration class maps and class information from the lightweight class map and the lightweight class information. The lightweight restoration process 1132 generates a plurality of restoration class maps corresponding to each class by using the representative value of the lightweight class map. The lightweight restoration process 1132 generates a restoration class map corresponding to each class based on the information about the class corresponding to each representative value recorded in the lightweight class information. For example, the light weight restoration process 1132 obtains information that the first representative value corresponds to the water class from the light weight class information, extracts the area corresponding to the first representative value from the light weight class map, and classifies the water class Create a restoration class map corresponding to . In this way, the lightweight restoration process 1132 generates a restoration class map corresponding to each representative value, and generates a plurality of restoration class maps for each class recorded in the lightweight class information. In addition, class information in which information about classes corresponding to a plurality of class maps is recorded is generated by the lightweight restoration process 1132 . In the plurality of restored class maps generated by the lightweight restoration process 1132 , information on probability is removed from the class map generated by the meta information extraction unit 212 of the image providing apparatus, and whether each pixel corresponds to a corresponding class Shows only information about Accordingly, the restored class map may have a smaller capacity than the class map generated by the meta information extractor 212 of the image providing apparatus. The class information generated by the lightweight restoration process 1132 corresponds to the class information generated by the meta information extraction unit 212 .

The meta information processing unit 1128 generates modulation parameters for image quality processing from the AI metadata restored by the meta information restoration unit 1126 , that is, a plurality of restored class maps and class information. The meta-information processing unit 1128 includes a 2-2 AI network 1150 that generates modulation parameters from a plurality of reconstructed class maps and class information. The modulation parameter is a parameter applied to the modulation layer of the 2-1 AI network 1140 of the meta information synthesis unit 1124 . The 2-1 AI network 1140 includes a plurality of modulation layers. The 2-2 AI network 1150 of the meta information processing unit 1128 generates modulation parameters for each of the plurality of modulation layers included in the 2-1 AI network 1140 . In addition, the AI metadata includes a plurality of restored class maps and class information corresponding to each frame, and the 2-2 AI network 1150 of the meta information processing unit 1128 generates a modulation parameter corresponding to each frame. can do.

The input information processing unit 1122 processes the second image decoded by the decoding unit 1110 into a form required by the meta information synthesis unit 1124 and inputs it to the meta information synthesis unit 1124 . According to an embodiment, the input information processing unit 1122 may generate a feature map and output it to the meta information synthesis unit 1124 . According to an embodiment, the input information processing unit 1122 may include a combination of at least one convolutional layer and at least one activation layer for generating the feature map.

The meta information synthesizing unit 1124 receives the feature map output from the input information processing unit 1122 and the modulation parameter generated by the meta information processing unit 1128, and performs image quality improvement processing on the second image to generate it output the third image. The meta information synthesizer 1124 may perform image quality improvement processing on the second image using the 2-1 AI network 1140 .

The meta information synthesis unit 1124 receives the feature map from the input information processing unit 1122 and performs image quality processing based on AI metadata. The meta-information synthesizing unit 1124 includes a 2-1 AI network 1140 that performs meta-information synthesizing processing. The meta information synthesizing unit 1124 applies the modulation parameters input from the meta information processing unit 1128 to a plurality of modulation layers included in the 2-1 AI network 1140 to perform image quality processing on the second image. carry out According to an embodiment, the modulation processing by the 2-1 AI network 1140 is performed using an affine transformation based on a modulation parameter.

The output unit 160 receives and outputs the third image generated by the meta information synthesis unit 1124 . According to an embodiment, the output unit 160 may correspond to a display displaying the third image. According to another embodiment, the output unit 160 may correspond to a communication unit that transmits the third image to an external device.

12 is a diagram illustrating an operation of a meta information synthesizing unit according to an embodiment of the present disclosure.

The meta information synthesis unit 1124 includes a 2-1 AI network 1140 including at least one layer. The 2-1 AI network 1140 may include a combination of at least one convolutional layer and at least one modulation layer. Also, the 2-1 AI network 1140 may further include at least one activation layer. Also, the 2-1 AI network 1140 may include an upscaler 1220 that receives the feature map generated from the low-resolution second image and upscales the resolution of the input data.

The 2-1 AI network 1140 may include a first image quality processing unit 1210 including at least one residual block 1212 , an upscaler 1220 , and a second image quality processing unit 1230 . have.

In the present specification, the first image corresponds to the high-resolution original image, the second image corresponds to the downscaled low-resolution image, and the third image corresponds to the super-resolution image upscaled by the restoration process of the image reproducing apparatus 150 . ) corresponds to

The first image quality processing unit 1210 includes a plurality of residual blocks 1212 . The first image quality processing unit 1210 receives the feature vector and performs image quality improvement processing. The first image quality processing unit 1210 may include a first skip processing path 1214 that skips the plurality of residual blocks 1212 . The first skip processing path 1214 may transfer the input data as it is, or may perform a predetermined process to transfer the input data. The first image quality processing unit 1210 generates a processing result value of a residual version by using the plurality of residual blocks 1212 , and processes a prediction version through the first skip processing path 1214 . Produces a result value. Also, the first image quality processing unit 1210 sums the processing result value of the residual version and the processing result value of the prediction version, and outputs it as the processing result value of the first image quality processing unit 1210 .

According to an embodiment, a start point and an end point of the first skip processing path 1214 may be determined to skip all of the plurality of residual blocks 1212 . The starting point and the ending point of the first skip processing path 1214 may be determined differently according to embodiments. Also, the number of the first skip processing paths 1214 may be variously determined, and the first image quality processing unit 1210 may include one or more first skip processing paths 1214 .

Referring to FIG. 13 , the structure of the residual block 1212 will be described.

13 is a diagram illustrating a structure of a residual block 1212 according to an embodiment of the present disclosure.

The residual block 1212 receives and processes the feature map. The residual block 1212 may include at least one convolutional layer or a combination of at least one modulation layer. Also, according to an embodiment, the residual block 1212 may further include at least one activation layer. According to an embodiment, the residual block 1212 may have a structure in which layers are repeatedly arranged in the order of a convolutional layer, a modulation layer, and an activation layer.

The residual block 1212 may include a second skip processing path 1320 that bypasses the main stream 1310 including a plurality of layers and proceeds from the input end to the output end. The main stream 1310 generates a residual version of the processing result value for the input data, and the second skip processing path 1320 generates a prediction version of the processing result value. The residual block 1212 sums the processing result value of the residual version and the processing result value of the prediction version, and outputs it as the processing result value of the residual block 1212 .

According to an embodiment, a start point and an end point of the second skip processing path 1320 may be determined to skip all of the plurality of layers in the residual block 1212 . The starting point and the ending point of the second skip processing path 1320 may be determined differently according to embodiments. Also, the number of second skip processing paths 1320 may be variously determined, and the residual block 1212 may include one or more second skip processing paths 1320 .

According to the present embodiment, due to the structure including the main stream 1310 and the second skip processing path 1320 , the main stream including a plurality of layers corresponds to the output of the 2-1 AI network 1140 . By learning only the difference between the super-resolution image and the low-resolution image corresponding to the input of the 2-1 AI network 1140 , and transmitting the remaining information through the second skip processing path 1320 , the detail learning efficiency can be increased. . _{Here, the residual data F mian-stream} (I _LR ) used for learning the main stream 1310 is a difference image between the super-resolution image I _SR and the low-resolution image I _LR and may be defined as follows. .

[Equation 2]

F _mian-stream (I _LR ) = I _SR - I _LR

The modulation layer performs modulation processing on input data. The modulation layer may, for example, perform an affine transformation process on input data. A separate modulation parameter may be defined for each of the plurality of modulation layers included in the first image quality processing unit 1210 . The meta-information processing unit 1128 may individually generate modulation parameters for each of the plurality of modulation layers included in the first image quality processing unit 1210 and output them to the meta-information synthesis unit 1124 . To this end, the 2-2 AI network 1150 of the meta information processing unit 1128 may separately include a network corresponding to each modulation layer.

Referring again to FIG. 12 , the processing result of the first image quality processing unit 1210 is input to the upscaler 1220 . The upscaler 1220 upscales the processing result of the first image quality processing unit 1210 . If, after the upscaling of the second image is performed before the 2-1 AI network 1140 , the feature map of the second image with high resolution is input to the 2-1 AI network 1140 , the second- 1 In the AI network 1140 , the upscaler 1220 may be omitted. The upscaler 1220 may be implemented using various upscaling algorithms, and may be implemented as an AI network having a DNN structure.

The second image quality processing unit 1230 generates and outputs a third image by performing additional image quality processing on the output of the upscaler 1220 . If the upscaler 1220 is omitted from the 2-1 AI network 1140 , the second image quality processing unit 1230 performs additional image quality processing on the processing result value of the first image quality processing unit 1210 . The second image quality processing unit 1230 may include a combination of at least one convolutional layer and at least one activation layer.

14 is a diagram illustrating operations of a 2-1 AI network and a 2-2 AI network according to an embodiment of the present disclosure.

According to an embodiment, the meta-information processing unit 1128 generates a modulation parameter from the AI metadata received from the image providing device 110 and restored, and the second-first AI network 1140 of the meta-information synthesis unit 1124 . ) is output. The meta information processing unit 1128 may include the 2-2 AI network 1150 or may use the 2-2 AI network 1150 provided in the external device.

The 2-2 AI network 1150 includes a combination of at least one convolutional layer and at least one activation layer. The modulation parameter may include a plurality of modulation parameter sets, and the 2-2 AI network 1150 may include separate AI networks 1410 and 1420 for each of the plurality of modulation parameter sets. For one modulation layer, a modulation parameter set including a plurality of parameters is output, and the 2-2 AI network 1150 may generate and output a plurality of modulation parameter sets for a plurality of modulation layers. The 2-2 AI network 1150 may generate and output a first modulation parameter set for the first modulation layer and may generate and output a second modulation parameter set for the second modulation layer. The 2-2 AI network 1150 includes AI networks 1410a and 1420a corresponding to the first modulation layer, AI networks 1410b and 1420b corresponding to the second modulation layer, and AI networks corresponding to the third modulation layer ( 1410c, 1420c). As the number of modulation layers increases, the number of AI networks 1410 and 1420 in the 2-2 AI network 1140 also increases. According to an embodiment, the modulation parameter set includes a first modulation parameter and a second modulation parameter, and the 2-2 AI network 1150 includes a first modulation parameter generator 1410 that generates the first modulation parameter, and A second modulation parameter generator 1420 may be included. The first modulation parameter generator 1410 and the second modulation parameter generator 1420 may be configured in a structure that shares some layers with each other, or may be configured separately without a shared layer.

The 2-2 AI network 1150 generates a plurality of modulation parameter sets and outputs them to the 2-1 AI network 1140 . The 2-1 AI network 1140 sets a modulation parameter set for each of at least one modulation layer by using the plurality of modulation parameter sets input from the 2-2 AI network 1150 . The modulation layer of the 2-1 AI network 1140 performs modulation processing on input data based on the set modulation parameter set.

The plurality of modulation parameter sets output from the 2-2 AI network 1150 may be recorded in the form of a vector or tensor of a predetermined dimension. For example, the 2-2 AI network 1150 outputs a plurality of modulation parameter sets to the 2-1 AI network through 64 channels, and each channel corresponds to each element of a vector corresponding to the plurality of modulation parameter sets. can be matched. The number of channels of the 2-2 AI network 1150 may be determined according to the number of elements included in the plurality of modulation parameter sets.

15A is a diagram illustrating structures of a 2-1 AI network and a 2-2 AI network according to an embodiment of the present disclosure.

According to an embodiment, the 2-2 AI network 1150 is configured by the number of modulation layers 1512a, 1512b, 1512c, 1512d included in the first image quality processing unit 1210 of the 2-1 AI network 1140 . modulation parameter generation networks 1520a, 1520b, 1520c, and 1520d. The plurality of modulation parameter generation networks 1520a, 1520b, 1520c, and 1520d receive AI metadata (AI_META) as input, respectively, and modulation layers 1512a and 1512b corresponding to the modulation parameter generation networks 1520a, 1520b, 1520c, 1520d. , 1512c, 1512d) are generated and outputted to the modulation parameters (P_SET1, P_SET2, P_SET3, P_SET4). Each of the modulation parameter generation networks 1520a , 1520b , 1520c , and 1520d may include a first modulation parameter generator 1410 and a second modulation parameter generator 1420 . According to an embodiment, the plurality of modulation parameter generating networks 1520a, 1520b, 1520c, and 1520d may share some layers or may not share layers, depending on the embodiment.

15B is a diagram illustrating structures of a 2-1 AI network and a 2-2 AI network according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the image reproducing apparatus 150 includes an image providing apparatus. A plurality of AI metadata corresponding to a plurality of image characteristics may be received from 110 . The plurality of image characteristics may include, for example, at least one of an object, a frequency, a texture, a semantic, or an imaging parameter or a combination thereof. The image reproducing apparatus 150 may include modulation parameter generating networks 1540a, 1540b, 1540c, and 1540d corresponding to respective image characteristics. For example, the image reproducing apparatus 150 may receive the first AI metadata AI_META1 corresponding to the object characteristic and the second AI metadata AI_META2 corresponding to the frequency characteristic. The first AI metadata AI_META1 may include class information and a plurality of class maps, and the second AI metadata AI_MEAT2 may include frequency information and a plurality of frequency maps. The first AI metadata (AI_MEAT1) and the second AI metadata (AI_MEATA2) are transmitted to the image reproducing apparatus 150 in a lightweight and compressed form, and are decompressed and lightened and restored by the meta information restoration unit 1126 . It is transmitted to the 2-2 AI network 1150 through

The modulation parameter generation networks 1540a, 1540b, 1540c, and 1540d receive the first AI metadata (AI_META1) and the second AI metadata (AI_META2) as inputs, and receive a first modulation parameter set (P_SET1) and a second modulation parameter set, respectively. (P_SET2), a third modulation parameter set (P_SET3), and a fourth modulation parameter set (P_SET4) are generated. As the types of input image characteristics increase, the modulation parameter generation networks 1540a, 1540b, 1540c, and 1540d receive combinations of AI metadata corresponding to respective image characteristics and generate modulation parameters. (1540a, 1540b, 1540c, 1540d) can be learned.

According to an embodiment, each of the modulation parameter generating networks 1540a, 1540b, 1540c, and 1540d may each correspond to a predetermined image characteristic. For example, the modulation parameter generating networks 1540a and 1540b may correspond to object characteristics, and the modulation parameter generating networks 1540c and 1540d may correspond to frequency characteristics. The modulation parameter generation networks 1540a and 1540b corresponding to object characteristics input a first modulation parameter set (P_SET1) and a second modulation parameter set (P_SET2) to the modulation layers 1532a and 1532b of the first residual block 1530a. do. The modulation parameter generating networks 1540c and 1540d corresponding to the frequency characteristic receive the second AI metadata AI_META2 and generate a third modulation parameter set P_SET3 and a fourth modulation parameter set P_SET4, respectively. The modulation parameter generation networks 1540c and 1540d corresponding to the frequency characteristic input the third modulation parameter set P_SET3 and the fourth modulation parameter set P_SET4 to the modulation layers 1532c and 1532d of the second residual block 1530b. do.

The modulation parameter generating networks 1540a, 1540b, 1540c, and 1540d corresponding to each image characteristic may be activated as AI metadata corresponding to the corresponding image characteristic is input. The image reproducing apparatus 150 includes modulation parameter generating networks 1540a, 1540b, 1540c, and 1540d corresponding to a plurality of image characteristics, and a modulation parameter generating network 1540a according to image characteristics corresponding to input AI metadata. 1540b, 1540c, 1540d) can be selectively activated. As the AI metadata is input, the meta information processing unit 1128 recognizes image characteristics corresponding to the AI metadata, and generates modulation parameter generating networks 1540a, 1540b, and 1540c corresponding to the image characteristics of the AI metadata. 1540d) to pass AI metadata. In addition, modulation parameter sets (P_SET1, P_SET2, P_SET3, P_SET4) are generated from the modulation parameter generation networks 1540a, 1540b, 1540c, and 1540d corresponding to the image characteristics of the corresponding AI metadata, and the modulation layers 1532a, 1532b, 1532c, 1532d), the corresponding modulation layers 1532a, 1532b, 1532c, and 1532d are activated, and image quality processing by the activated modulation layers 1532a, 1532b, 1532c, and 1532d for the second image may be performed. .

16 is a diagram illustrating operations of a metadata synthesizing unit and a metadata processing unit according to an embodiment of the present disclosure.

The metadata synthesizing unit may include at least one residual block 1212 , and each residual block 1212 may include at least one modulation layer 1640 . Each modulation layer 1640 receives a modulation parameter set individually generated for each modulation layer 1640 .

According to an embodiment, the modulation parameter set input to the modulation layer 1640 of the residual block 1212 includes a first operation modulation parameter (a _i ) for performing a multiplication operation with the input data, and a first operation modulation parameter (a i ) for performing an addition operation with the input data. 2 contains the computational modulation parameter (b _{i ). The input value f i} of the modulation layer 1640 may have the form of an input feature map fi(w, h, c) 1642 having a width w, a height c, and a height h. where w, c, and h represent the number of element values and are defined as natural numbers. The first arithmetic modulation parameter (a _i ) may have the form of a weight map (ai(w, h, c)) having the same size as _{f i (w, h, c).} The second arithmetic modulation parameter b _i may have the form of a weight map bi(w, h, c) having the same size as f _{i (w, h, c).} The input feature map 1642, the first computational modulation parameter (a _i ), and the second computational modulation parameter (b _i ) may each have w*h*c element values.

The 2-2 AI network 1150 receives AI metadata and generates a first arithmetic modulation parameter (a _i ) and a second arithmetic modulation parameter (b _i ) corresponding to each modulation layer 1640 . The 2-2 AI network 1150 includes a common layer 1610 commonly used for generation processing of the first arithmetic modulation parameter (a _i ) and generation processing of the second arithmetic modulation parameter (b _{i ), the first arithmetic modulation} a first arithmetic modulation parameter generating layer 1620 to generate a parameter a _i , and a second arithmetic modulation parameter generating layer 1630 to generate a second arithmetic modulation parameter _{b i .} The arrangement and connection structures of common layers and individual layers in the generation processing of the first operation modulation parameter (a _i ) and the generation processing of the second operation modulation parameter (b _{i ) may be variously determined.}

값(1646)을 생성한다. 변조 레이어(1640)는 곱셈 결과 값(1644)과 제2 연산 변조 파라미터(b_i)를 더할 때, 동일 위치의 요소 값을 더한다.

값(1646)은 수학식 3과 같이 정의될 수 있다.The modulation layer 1640 receives the first operation modulation parameter a _i , and performs a multiplication operation with the input feature map f _{i input from the previous layer.} The modulation layer 1640 may perform a point-wise multiplication operation of multiplying element values at the same location when multiplying the _{input feature map f i} and the first operation modulation parameter a _{i .} The modulation layer 1640 is an output value of the modulation layer 1640 by adding the multiplication result value 1644 using the first arithmetic modulation parameter (a _i ) and the second arithmetic modulation parameter (b _{i ).}

Generate the value 1646. When the modulation layer 1640 adds the multiplication result value 1644 and the second operation modulation parameter b _i , the modulation layer 1640 adds element values at the same position.

The value 1646 may be defined as in Equation (3).

[Equation 3]

Each of the plurality of modulation layers 1640 generates an operation result value as in Equation 3 and outputs it to the next layer. The metadata processing unit 1128 may perform image quality improvement processing by processing the plurality of modulation layers 1640 included in the 2-1 AI network 1140, and obtain a high-quality image in which the texture of the image is restored. have.

는 수학식 4와 같이 정의될 수 있다. 본 실시예에 따르면, 변조 레이어(1640)의 출력 값은 다차수 함수에 의해 정의된다. 제2-2 AI 네트워크(1140)는 각각의 변조 레이어(1640)에 대해 수학식 4에서 정의된 a_i, b_i, ..., 및 n_i를 생성하여 출력할 수 있다. According to another embodiment of the present disclosure, an output value of the modulation layer 1640

can be defined as in Equation (4). According to this embodiment, the output value of the modulation layer 1640 is defined by a multi-order function. The 2-2 AI network 1140 may generate and output _{a i} , b _i , ..., and n _i defined in Equation 4 for each modulation layer 1640 .

[Equation 4]

는 수학식 5와 같이 정의될 수 있다. 본 실시예에 따르면, 변조 레이어(1640)의 출력 값은 로그 함수에 의해 정의된다. 제2-2 AI 네트워크(1140)는 각각의 변조 레이어(1640)에 대해 수학식 5에서 정의된 a_i, b_i, ..., 및 n_i를 생성하여 출력할 수 있다.According to another embodiment of the present disclosure, an output value of the modulation layer 1640

may be defined as in Equation 5. According to this embodiment, the output value of the modulation layer 1640 is defined by a log function. The 2-2 AI network 1140 may generate and output _{a i} , b _i , ..., and n _i defined in Equation 5 for each modulation layer 1640 .

[Equation 5]

는 수학식 6과 같이 정의될 수 있다. 본 실시예에 따르면, 변조 레이어(1640)의 출력 값은 지수 함수에 의해 정의된다. 제2-2 AI 네트워크(1140)는 각각의 변조 레이어(1640)에 대해 수학식 6에서 정의된 a_i, b_i, ..., 및 n_i를 생성하여 출력할 수 있다.According to another embodiment of the present disclosure, an output value of the modulation layer 1640

can be defined as in Equation (6). According to this embodiment, the output value of the modulation layer 1640 is defined by an exponential function. The 2-2 AI network 1140 may generate and output _{a i} , b _i , ..., and n _i defined in Equation 6 for each modulation layer 1640 .

[Equation 6]

According to an embodiment, the 2-1 AI network 1140 may include a combination of modulation layers 1640 having different modulation functions. For example, the first modulation layer may perform modulation processing according to Equation 4, and the second modulation layer may perform modulation processing according to Equation 6.

According to an embodiment, the function of the modulation layer 1640 of the 2-1 AI network 1140 may vary depending on the type of input image characteristics and the combination of image characteristics. For example, when only AI metadata for an object is input, the 2-1 AI network 1140 performs image quality processing by the modulation layer 1640 using Equation 3, and AI metadata for the object When AI metadata for and frequency are received together, image quality processing by the modulation layer 1640 may be performed using Equation (4). The processor 1120 may change the type of modulation processing performed by the modulation layer 1640 according to the type and combination of image characteristics corresponding to the input AI metadata. For example, the AI metadata includes information on the type of the corresponding image characteristic, and the meta-information processing unit 1128 modulates the modulation processing corresponding to the type of the modulation processing according to the type of the image characteristic corresponding to the AI metadata. A parameter may be generated and output to the meta information synthesizing unit 1124 . The meta information synthesizing unit 1124 may set a type of modulation processing based on the modulation processing parameter and perform the modulation processing.

The modulation layer 1640 may perform a modulation operation according to Equation 3 on the input feature map to improve the quality of the second image. The modulation layer 1640 reconstructs the object-related texture of the second image by a modulation operation using the modulation parameter set determined by the 2-2 AI network 1150 to restore the third image having a texture almost similar to the original image. create In particular, by the modulated image using the modulation parameter set determined by the 2-2 AI network 1150, the detailed part of the image related to the image characteristics such as the object and frequency of the original image is restored, so that the quality of the third image is improved. can be improved.

17 is a diagram for explaining a method of learning a first AI network of an image providing apparatus and a second AI network of an image reproducing apparatus according to an embodiment of the present disclosure.

According to an embodiment, the second AI network of the image reproducing apparatus may be learned using a plurality of training data including a low-resolution image (LR), AI metadata, and a high-resolution image (HR). According to an embodiment, the learning processing unit 1700 for learning the second AI network inputs a low-resolution image (LR) and AI metadata to the second AI network 156, and the second AI network 156 generates The super-resolution image SR may be compared with the high-resolution image HR, and the second AI network 156 may be updated based on the comparison result. The second AI network 156 compares the high-resolution image (HR) and the super-resolution image (SR) using various types of Generative Adversarial Network (GAN) algorithms, and the second AI network 156 according to the comparison result. can be updated.

According to an embodiment, the second AI network may be trained based on an Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN) algorithm. The learning processing unit 1700 predicts a probability that a _{real image (x r} ) is relatively more realistic than a face image (x _{f ) using a relativistic discriminator.} Here, the real image (x _r ) corresponds to the high-resolution image (HR), and the fake image (x _f ) corresponds to the super-resolution image. To this end, the learning processing unit 1700 compares the super-resolution image SR and the high-resolution image HR using the _{relativistic discrimination unit D Ra .} The relativistic determination unit D _Ra calculates a determination result value based on Equation 7 below.

[Equation 7]

D _Ra (x _r , x _f ) = σ(C(x) _r )) - E _xf ((C(x _f )))

Here, σ denotes a sigmoid function, C(x) denotes the output of a discriminator that determines whether the input image is an actual image, and E _xf (.) denotes a mini-batch represents the average operation for all spurious data. The learning processing unit 1700 performs learning by updating the parameter values of the second AI network 156 based on the processing result of the relativistic discrimination unit DRa.

According to an embodiment, the first AI network 114 and the second AI network 156 may be linked and learned. The learning processing unit 1700 generates a super-resolution image SR by processing the first AI network 114 and the second AI network 156 using training data including the high-resolution image HR, and the high-resolution image SR is generated. The first AI network 114 and the second AI network 156 may be updated based on the comparison result of the image HR and the super-resolution image SR.

According to another embodiment, the learning process of the first AI network 114 and the second AI network 156 includes individual learning for learning the first AI network 114 and the second AI network 156, respectively, and the second AI network 156. It may include joint learning of the first AI network 114 and the second AI network 156 . In the individual learning, the first AI network 114 and the second AI network 156 are individually learned using a plurality of training data including high-resolution image (HR), low-resolution image (LR), and AI metadata. make it Next, linkage learning is performed using the high-resolution image (HR) as training data for the individually learned first AI network 114 and the second AI network 156 . The individually learned first AI network 114 and the second AI network 156 may be further updated by the associative learning process.

18 is a flowchart illustrating a method of controlling an image reproducing apparatus according to an embodiment of the present disclosure.

Each step of the method for controlling an image reproducing apparatus of the present disclosure may be performed by various types of electronic devices having an input unit, a processor, a memory, and an output unit, and using a machine learning model. The present specification will be mainly described with respect to an embodiment in which the image reproducing apparatus 150 according to embodiments of the present disclosure performs the method of controlling the image reproducing apparatus. Accordingly, the embodiments described for the image reproducing apparatus 150 are applicable to the embodiments of the method for controlling the image reproducing apparatus, and on the contrary, the embodiments described for the method for controlling the image reproducing apparatus 150 are for the image reproducing apparatus 150 . Applicable to the embodiments. The method for controlling an image reproducing apparatus according to the disclosed embodiments is not limited to being performed by the image reproducing apparatus 150 disclosed herein, and may be performed by various types of electronic devices.

The video reproducing apparatus receives the encoded video and AI metadata (S1802). AI metadata is generated by the image providing device. AI metadata may be input to the image reproducing device in a lightweight and compressed form. The AI metadata includes class information corresponding to the encoded image and a plurality of class maps.

Next, the image reproducing apparatus generates a second image by decoding the encoded image (S1804). The image reproducing apparatus generates a second image from the encoded image encoded by the image providing apparatus by using a decoding process corresponding to the encoding process used in the image providing apparatus. Since the decoding process is the same as or similar to the operation of the decoding unit 1110 described above, a redundant description will be omitted.

Also, the image reproducing apparatus generates a third image on which image quality improvement processing is performed by using the second AI network from the AI metadata and the second image (S1806). The image reproducing apparatus performs decompression processing and light weight restoration processing on AI metadata. AI metadata is restored in the form of class information and a plurality of restored class maps by decompression processing and lightweight restoration processing. The class information and the plurality of reconstructed class maps are converted into a plurality of modulation parameter sets by the 2-2 AI network. The 2-2 AI network generates a modulation parameter set for each modulation layer in the residual block for image quality processing. The image reproducing apparatus performs image quality processing on the second image using the 2-1 AI network. The 2-1 AI network includes at least one residual block, and each residual block includes at least one modulation layer. The modulation layer is parameterized by the modulation parameter set generated by the 2-2 AI network. The feature map generated from the second image is processed by at least one residual block. A modulation layer in the residual block performs image quality processing on the feature map. The modulation layer may perform image quality processing on the feature map using, for example, an affine transform. The 2-1 AI network generates and outputs a third image by performing image quality processing on the second image. Since the image quality improvement process is the same as or similar to the operation of the image quality processing unit 1120 described above, a redundant description will be omitted.

Next, the image reproducing apparatus outputs a third image (S1808). The image reproducing apparatus may display the third image or transmit it to another device and output it. Since the output processing is the same as or similar to the operation of the output unit 160 described above, a redundant description will be omitted.

19 is a diagram illustrating the configuration of an image providing apparatus and an image reproducing apparatus according to an embodiment of the present disclosure.

The image providing apparatus and the image reproducing apparatus may be implemented as various types of electronic devices 1900 . The electronic device 1900 includes a processor 1910 , a memory 1920 , and an input/output unit 1930 . The processor 112 of the image providing device 110 corresponds to the processor 1910 of the electronic device 1900 , and the memory 116 of the image providing device 110 corresponds to the memory 1920 of the electronic device 1900 . and the output unit 118 of the image providing apparatus 110 may correspond to the input/output unit 1930 of the electronic device 1900 . The input unit 152 and the output unit 160 of the image reproducing apparatus 150 correspond to the input/output unit 1930 of the electronic device 1900, and the processor 154 of the image reproducing apparatus 150 includes the electronic device ( Corresponds to the processor 1910 of the 1900 , and the memory 158 of the image reproducing apparatus 150 may correspond to the memory 1920 of the electronic device 1900 .

Processor 1910 may include one or more processors. The processor 1910 may include a dedicated processor such as a central control unit, an image processing unit, and an AI processing unit.

The memory 1920 may include a volatile storage medium, a non-volatile storage medium, or a combination thereof. Also, the memory 1920 may include various types of memory, such as a main memory, a cache memory, a register, and a non-volatile memory. The memory 1920 may be implemented as various types of storage media. For example, the memory 1920 may include a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory). etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), It may include at least one type of storage medium among a magnetic memory, a magnetic disk, and an optical disk.

The input/output unit 1930 may include various types of input/output interfaces and communication units 1940 . The input/output unit 1930 may include a display unit, a touch screen, a touch pad, a communication unit, or a combination thereof. The communication unit 1940 may include various types of communication modules. The communication unit 1940 may include a short-range communication unit 1942 , a mobile communication unit 1944 , or a broadcast receiving unit 1946 . The short-range communication unit 1942 communicates with Bluetooth, Bluetooth Low Energy (BLE), near field communication, RFID (Radio Frequency Identification), WLAN (Wi-Fi), Zigbee, and infrared (IrDA) communication. , WFD (Wi-Fi Direct), UWB (ultra wideband), Ant+ communication, or a combination thereof may perform communication.

Meanwhile, the above-described embodiments of the present disclosure can be written as a program that can be executed on a computer, and the written program can be stored in a medium.

The medium may be to continuously store a computer executable program, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

Meanwhile, the above-described AI network model may be implemented as a software module. When implemented as a software module (eg, a program module including instructions), the AI network model may be stored in a computer-readable recording medium.

In addition, the AI network model may be integrated in the form of a hardware chip to become a part of the above-described image providing apparatus or image reproducing apparatus. For example, the AI network model may be built in the form of a dedicated hardware chip for artificial intelligence, or as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU). may be manufactured.

In addition, the AI network model may be provided in the form of downloadable software. The computer program product may include a product (eg, a downloadable application) in the form of a software program distributed electronically through a manufacturer or an electronic market. For electronic distribution, at least a portion of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer or an electronic market, or a storage medium of a relay server.

In the above, the technical idea of the present disclosure has been described in detail with reference to preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments, and those of ordinary skill in the art within the scope of the technical spirit of the present disclosure Various modifications and changes are possible by the person.

Claims (20)

a memory storing one or more instructions;
one or more processors to execute the one or more instructions stored in the memory; and
including an output;
The one or more processors by executing the one or more instructions,
Class information including at least one class corresponding to the type of object included in the first image among a plurality of predefined objects using a first artificial intelligence (AI) network, and each class in the first image generate AI metadata including at least one class map representing a region corresponding to
Encoding the first image to generate an encoded image,
outputting the encoded image and the AI metadata through the output unit;
The one or more processors by executing the one or more instructions,
inputting the first image to the first AI network to generate a plurality of segmentation probability maps for each of the plurality of predefined object types;
defining at least one class corresponding to a type of an object included in the first image based on the plurality of segmentation probability maps;
generating class information including the at least one class;
and generating the at least one class map for each of the at least one class from the plurality of segmentation probability maps. delete According to claim 1,
The one or more processors by executing the one or more instructions,
For each of the plurality of segmentation probability maps, calculating an average value of pixels except for a pixel having a value of 0;
An image providing apparatus for defining the at least one class by selecting some of the plurality of predefined objects based on the size of an average value of each of the plurality of segmentation probability maps. 4. The method of claim 3,
The one or more processors by executing the one or more instructions,
mapping at least some of the objects that are not selected as the at least one class among the plurality of predefined objects to the at least one class,
and generating the class map by synthesizing the segmentation probability map of the object mapped to the at least one class and the segmentation probability map of the class to which the corresponding object is mapped. According to claim 1,
The one or more processors by executing the one or more instructions,
generating, from the first AI network, a segmentation probability map for a frequency for each of a plurality of predefined frequency domains;
AI including frequency information including information on a frequency domain for the first image, and at least one frequency map corresponding to each of the frequency domains included in the frequency information, based on the segmentation probability map for the frequency An image providing device that generates metadata. According to claim 1,
The first image includes a plurality of frames, the class information and the at least one class map are generated for each of the plurality of frames,
The one or more processors by executing the one or more instructions,
defining at least one sequence including at least one frame from the plurality of frames,
For each sequence, sequence class information indicating information on a class included in the at least one frame included in the sequence, and information on a class included in each of the at least one frame included in the sequence create frame class information representing;
The frame class information indicates a combination of classes included in a corresponding frame among classes included in the sequence class information, and the number of bits is smaller than that of the sequence class information. According to claim 1,
The one or more processors by executing the one or more instructions,
Based on the at least one class map, each pixel generates a lightweight class map having a representative value corresponding to one of the at least one class,
Generate lightweight AI metadata including the class information and the lightweight class map,
An image providing apparatus for outputting the encoded image and the lightweight class map through the output unit. According to claim 1,
The first AI network,
a 1-1 AI network including at least one convolution layer and at least one max pooling layer and generating a feature map from the first image; and
A first layer group including at least one convolutional layer and at least one activation layer and receiving and processing the feature map from the 1-1 AI network, upscaling the output of the first layer group a second comprising an upscaler, and at least one convolutional layer and at least one min pooling layer, and receiving an output of the upscaler to generate a segmentation probability map for each of the plurality of predefined objects An image providing device comprising a 1-2 AI network including a layer group. According to claim 1,
The first AI network is learned in connection with the second AI network,
The second AI network is included in an apparatus for decoding the encoded image by receiving the encoded image and the AI metadata, and performs image quality processing on image data corresponding to the encoded image from the AI metadata. Device. According to claim 1,
The one or more processors by executing the one or more instructions,
and generating the encoded image by performing downscale processing and encoding processing on the first image. Class information including at least one class corresponding to the type of object included in the first image among a plurality of predefined objects using a first artificial intelligence (AI) network, and each object in the first image generating AI metadata including at least one class map indicating a region corresponding to ;
generating an encoded image by encoding the first image; and
outputting the encoded image and the AI metadata;
The step of generating the AI metadata includes:
generating a plurality of segmentation probability maps for each of the plurality of predefined object types by inputting the first image to the first AI network;
defining at least one class corresponding to a type of an object included in the first image based on the plurality of segmentation probability maps;
generating class information including the at least one class;
and generating the at least one class map for each of the at least one class from the plurality of segmentation probability maps. a memory storing one or more instructions;
one or more processors to execute the one or more instructions stored in the memory;
an input unit receiving an encoded image corresponding to the first image and AI metadata corresponding to the encoded image; and
including an output;
The AI metadata includes class information including at least one class corresponding to the type of object included in the first image, and at least one class map indicating a region corresponding to each class in the first image and,
The one or more processors by executing the one or more instructions,
generating a second image corresponding to the first image by decoding the encoded image;
using a second AI network to generate a third image on which image quality improvement processing is performed from the second image and the AI metadata;
output the third image through the output unit,
The second AI network includes a 2-1 AI network including at least one convolution layer, at least one modulation layer, and at least one activation layer,
The at least one modulation layer processes a feature map input to the modulation layer based on a modulation parameter set generated from the AI metadata. delete 13. The method of claim 12,
The 2-1 AI network includes a plurality of modulation layers,
wherein the second AI network includes a modulation parameter generation network corresponding to each of the plurality of modulation layers from the AI metadata, each of the modulation parameter generation networks generating the modulation parameter set for a corresponding modulation layer; video playback device. 15. The method of claim 14,
The modulation parameter set is a first arithmetic modulation parameter synthesized by a multiplication operation on the data value of the input feature map, and a multiplication result of the first arithmetic modulation parameter and the data value of the input feature map by an addition operation. a second computational modulation parameter to be synthesized;
and the at least one modulation layer performs processing on the input feature map based on the first arithmetic modulation parameter and the second arithmetic modulation parameter. 13. The method of claim 12,
The 2-1 AI network includes a plurality of residual blocks,
Each of the plurality of residual blocks comprises:
a main stream comprising at least one convolutional layer, at least one modulation layer, and at least one activation layer, for generating a residual version of a processing result value; and
at least one second skip processing path for generating a processing result of a prediction version by bypassing at least one layer included in the corresponding block;
The 2-1 AI network includes at least one first skip processing path that skips at least one residual block among the plurality of residual blocks to generate a processing result value of a prediction version. 13. The method of claim 12,
The second AI network,
an upscaler that receives the output of the 2-1 AI network and performs upscale processing; and
and a second image quality processing unit receiving the output of the upscaler, generating and outputting a third image, and including at least one machine-learned layer. 13. The method of claim 12,
The AI metadata input through the input unit includes a lightweight class map in which at least one class map for each of the at least one class is lightened,
The one or more processors by executing the one or more instructions,
From the lightweight class map, generate a restored class map for each of the at least one class,
The reconstructed class map has a value indicating whether each pixel corresponds to a corresponding class. receiving an encoded image corresponding to the first image and AI metadata corresponding to the encoded image;
generating a second image corresponding to the first image by decoding the encoded image;
generating, from the second image and the AI metadata, a third image on which image quality improvement has been performed, by using a second AI network; and
outputting the third image,
The AI metadata includes class information including at least one class corresponding to the type of object included in the first image, and at least one class map indicating a region corresponding to each class in the first image and,
The second AI network includes a 2-1 AI network including at least one convolution layer, at least one modulation layer, and at least one activation layer,
The at least one modulation layer processes a feature map input to the corresponding modulation layer based on a modulation parameter set generated from the AI metadata. A computer-readable recording medium storing computer program instructions for performing a method of controlling an image reproducing apparatus when executed by a processor, the method comprising:
receiving an encoded image corresponding to the first image and AI metadata corresponding to the encoded image;
generating a second image corresponding to the first image by decoding the encoded image;
generating, from the second image and the AI metadata, a third image on which image quality improvement has been performed, by using a second AI network; and
outputting the third image,
The AI metadata includes class information including at least one class corresponding to the type of object included in the first image, and at least one class map indicating a region corresponding to each class in the first image and,
The second AI network includes a 2-1 AI network including at least one convolution layer, at least one modulation layer, and at least one activation layer,
The at least one modulation layer processes a feature map input to the modulation layer based on a modulation parameter set generated from the AI metadata.

Images