KR102589858B1 - Encoding apparatus and operating method for the same, and AI up scaling apparatus and operating method for the same - Google Patents

Abstract

It relates to a decoding device, comprising a communication unit that receives AI encoded data generated as a result of AI downscaling and first encoding of the original video, a processor that separates the AI encoded data into video data and AI data, and an input/output unit, Based on the image data, the processor may control the input/output unit to obtain a second image by first decoding the first image obtained by AI downscaling the original image and transmit the second image and AI data to an external device. there is.

Description

Decoding apparatus and operating method thereof, and AI up scaling apparatus and operating method {Encoding apparatus and operating method for the same, and AI up scaling apparatus and operating method for the same}

Various embodiments relate to a decoding device and operating method for decoding a compressed image, and an AI upscaling device and operating method including a deep neural network for AI upscaling an image.

The video is encoded by a codec that complies with a certain data compression standard, for example, the MPEG (Moving Picture Expert Group) standard, and is then stored in a recording medium in the form of a bitstream or transmitted through a communication channel.

With the development and distribution of hardware that can play and store high-resolution/high-quality images, the need for codecs that can effectively encode and decode high-resolution/high-quality images is increasing.

Various embodiments may provide a decoding device and a method of operating the same that can restore a compressed image and transmit the restored image and data necessary for AI upscaling of the restored image to the AI upscale device.

In addition, it is possible to provide an AI upscale device and an operating method that can receive video and AI data from a decoding device and AI upscale the video using a DNN for upscaling.

A decoding device according to an embodiment includes a communication unit that receives AI encoded data generated as a result of AI downscaling and first encoding of an original image, a processor that separates the AI encoded data into image data and AI data, and input/output a unit, wherein the processor obtains a second image by first decoding the first image obtained by AI downscaling the original image based on the image data, and transmits the second image and the AI data to an external device. The input/output unit can be controlled to transmit to .

The input/output unit according to an embodiment includes a High Definition Multimedia Interface (HDMI), and the processor may transmit the second image and the AI data to the external device through the HDMI.

The processor according to one embodiment may transmit the AI data in the form of a VSIF (Vendor Specific InfoFrame) packet.

The input/output unit according to one embodiment includes a DP (Display Port), and the processor may transmit the second image and the AI data to the external device through the DP.

AI data according to one embodiment may be data related to the AI downscaling of the original image.

AI data according to one embodiment may include at least one of information indicating whether AI upscaling is applied to the second image and information about a DNN for upscaling the second image in response to the AI downscaling. You can.

A method of operating a decoding device according to an embodiment includes receiving AI encoded data generated as a result of AI downscaling and first encoding of an original image, dividing the AI encoded data into image data and AI data, Based on the image data, obtaining a second image by first decoding the first image obtained by AI downscaling the original image, and transmitting the second image and the AI data to an external device through an input/output interface. It may include the step of transmitting to .

An AI upscaling device according to an embodiment is an input/output device that uses a first DNN to receive a second image corresponding to a first image obtained by AI downscaling an original image, and AI data related to the AI downscaling. a unit, a memory storing one or more instructions, and a processor executing the one or more instructions stored in the memory, wherein the processor, based on the AI data, relates to a second DNN corresponding to the first DNN. Obtain information, AI upscale the second image using the second DNN determined according to the obtained information, and the input/output unit includes a High Definition Multimedia Interface (HDMI), and the second DNN Video and the AI data are received through the HDMI, and the AI data may be received in the form of a VSIF (Vendor Specific InfoFrame) packet.

The decoding device according to one embodiment can efficiently transmit the restored image and AI data to the AI upscale device through an input/output interface.

An AI upscale device according to an embodiment can efficiently receive restored images and AI data from a decoding device through an input/output interface.

Figure 1 is a diagram for explaining an AI encoding process and an AI decoding process according to an embodiment.
Figure 2 is a block diagram showing the configuration of an AI decoding device according to an embodiment.
Figure 3 is an exemplary diagram showing a second DNN for AI upscaling of a second image.
Figure 4 is a diagram for explaining a convolution operation by a convolution layer.
Figure 5 is an exemplary diagram showing the mapping relationship between various image-related information and various DNN setting information.
Figure 6 is a diagram showing a second image composed of a plurality of frames.
Figure 7 is a block diagram showing the configuration of an AI encoding device according to an embodiment.
Figure 8 is an exemplary diagram showing a first DNN for AI downscaling of an original image.
Figure 9 is a diagram for explaining a method of training the first DNN and the second DNN.
Figure 10 is a diagram for explaining the training process of the first DNN and the second DNN by the training device.
FIG. 11 is an exemplary diagram illustrating a device for AI downscaling of an original image and a device for AI upscaling of a second image.
Figure 12 is a diagram showing an AI decoding system according to an embodiment.
Figure 13 is a diagram showing the configuration of a decoding device according to an embodiment.
Figure 14 shows AI data in the form of metadata according to an embodiment.
Figure 15 is a diagram showing a case where AI data is received in bitstream form according to an embodiment.
Figure 16 shows an AI codec syntax table according to one embodiment.
Figure 17 is a block diagram showing the configuration of an AI upscale device according to an embodiment.
Figure 18 is a diagram illustrating an example in which a decoding device and an AI upscale device transmit and receive data through HDMI, according to an embodiment.
Figure 19 is a diagram showing HF-VSDB included in EDID information according to an embodiment.
Figure 20 is a diagram showing the header structure and content structure of a VSIF (Vendor Specific InfoFrame) packet according to an embodiment.
Figure 21 is a diagram illustrating an example in which AI data is defined in a VSIF packet according to an embodiment.
Figure 22 is a flowchart showing a method of operating a decoding device according to an embodiment.
Figure 23 is a flowchart showing a method by which a decoding device transmits a second video and AI data through HDMI according to an embodiment.
Figure 24 is a flowchart showing a method of operating an AI upscale device according to an embodiment.
Figure 25 is a block diagram showing the configuration of a decoding device according to an embodiment.
Figure 26 is a block diagram showing the configuration of an AI upscale device according to an embodiment.

Since the present disclosure can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail through detailed description. However, this is not intended to limit the embodiments of the present disclosure, and the present disclosure should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the various embodiments.

In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the present disclosure, the detailed descriptions will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of the specification are merely identifiers to distinguish one component from another component.

In addition, in this specification, when a component is referred to as "connected" or "connected" to another component, the component may be directly connected or directly connected to the other component, but specifically Unless there is a contrary description, it should be understood that it may be connected or connected through another component in the middle.

In addition, in this specification, components expressed as 'unit (unit)', 'module', etc. are two or more components combined into one component, or one component is divided into two or more components for each more detailed function. It may be differentiated into In addition, each of the components described below may additionally perform some or all of the functions of other components in addition to the main functions that each component is responsible for, and some of the main functions of each component may be different from other components. Of course, it can also be performed exclusively by a component.

Additionally, in this specification, 'image' or 'picture' may refer to a still image, a moving image consisting of a plurality of consecutive still images (or frames), or a video.

Additionally, in this specification, 'DNN (deep neural network)' is a representative example of an artificial neural network model that simulates brain nerves, and is not limited to an artificial neural network model using a specific algorithm.

Additionally, in this specification, 'parameter' is a value used in the calculation process of each layer forming a neural network and may include, for example, a weight used when applying an input value to a predetermined calculation equation. Additionally, parameters can be expressed in matrix form. Parameters are values set as a result of training, and can be updated through separate training data as needed.

Additionally, in this specification, 'first DNN' refers to a DNN used for AI downscaling of an image, and 'second DNN' refers to a DNN used for AI upscaling of an image.

Additionally, in this specification, 'DNN setting information' is information related to elements constituting a DNN and includes the parameters described above. The first DNN or the second DNN can be set using the DNN setting information.

Additionally, in this specification, 'original image' refers to an image that is the subject of AI encoding, and 'first image' refers to an image obtained as a result of AI downscaling of the original image during the AI encoding process. In addition, the 'second image' refers to an image obtained through first decoding in the AI decoding process, and the 'third image' refers to an image obtained by AI upscaling the second image in the AI decoding process.

Additionally, in this specification, 'AI downscale' refers to a process that reduces the resolution of an image based on AI, and 'first encoding' refers to an encoding process using a frequency transformation-based image compression method. Additionally, 'first decoding' refers to decoding processing using a frequency conversion-based image restoration method, and 'AI upscale' refers to processing that increases the resolution of the image based on AI.

FIG. 1 is a diagram illustrating an artificial intelligence (AI) encoding process and an AI decoding process according to an embodiment.

As mentioned above, as the resolution of an image rapidly increases, the amount of information processed for encoding/decoding increases, and accordingly, a method to improve the efficiency of encoding and decoding the image is needed.

As shown in FIG. 1, according to an embodiment of the present disclosure, the first image 115 is obtained by AI downscaling (110) the original image (105) with high resolution. In addition, since the first encoding 120 and the first decoding 130 are performed on the first image 115 of relatively small resolution, the first encoding 120 and the first decoding are performed on the original image 105. Compared to the case of performing the first decoding 130, the processing bit rate can be significantly reduced.

As described in detail with reference to FIG. 1, in one embodiment, in the AI encoding process, the original image 105 is AI downscaled (110) to obtain the first image (115), and the first image (115) is converted to the first image (115). Encode (120). In the AI decoding process, AI encoded data including AI data and image data obtained as a result of AI encoding is received, a second image 135 is obtained through the first decoding 130, and the second image 135 is The third image 145 is obtained through AI upscaling (140).

Looking at the AI encoding process in more detail, when the original image 105 is input, the original image 105 is AI downscaled (110) to obtain the first image 115 of a certain resolution or quality. At this time, AI downscale 110 is performed based on AI, and AI for AI downscale 110 must be joint trained with AI for AI upscale 140 of the second image 135. do. This is because, when AI for AI downscaling (110) and AI for AI upscaling (140) are trained separately, between the original image (105) that is subject to AI encoding and the third image (145) restored through AI decoding This is because the difference increases.

In an embodiment of the present disclosure, AI data can be used to maintain this linkage during the AI encoding process and AI decoding process. Therefore, the AI data acquired through the AI encoding process must include information representing the upscale target, and in the AI decoding process, the second image 135 is AI upscaled ( 140) must be done.

AI for AI downscale (110) and AI for AI upscale (140) can be implemented with a deep neural network (DNN). As described later with reference to FIG. 9, the first DNN and the second DNN are jointly trained by sharing loss information under a predetermined target, so the AI encoding device uses the target information used when the first DNN and the second DNN are jointly trained. is provided to the AI decoding device, and the AI decoding device can AI upscale 140 to the target resolution of the second image 135 based on the provided target information.

Describing the first encoding 120 and the first decoding 130 shown in FIG. 1 in detail, the first image 115 that has been AI downscaled (110) from the original image (105) is the first encoding (120). The amount of information can be reduced through . The first encoding 120 includes a process of predicting the first image 115 and generating prediction data, a process of generating residual data corresponding to the difference between the first image 115 and the prediction data, and a spatial domain component. It may include a process of transforming the residual data into frequency domain components, a process of quantizing the residual data converted to frequency domain components, and a process of entropy encoding the quantized residual data. This first encoding process 120 includes MPEG-2, H.264 AVC (Advanced Video Coding), MPEG-4, HEVC (High Efficiency Video Coding), VC-1, VP8, VP9, and AV1 (AOMedia Video 1). It can be implemented through one of the image compression methods using frequency transformation.

The second image 135 corresponding to the first image 115 may be restored through the first decoding 130 of the image data. The first decoding 130 includes a process of entropy decoding image data to generate quantized residual data, a process of dequantizing the quantized residual data, a process of converting residual data of a frequency domain component into a spatial domain component, and prediction data. It may include a process of generating and a process of restoring the second image 135 using prediction data and residual data. This first decoding (130) process is image compression using frequency conversion such as MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 used in the first encoding (120) process. It can be implemented through an image restoration method corresponding to one of the methods.

The AI encoded data acquired through the AI encoding process may include image data obtained as a result of the first encoding 120 of the first image 115 and AI data related to the AI downscale 110 of the original image 105. You can. Image data can be used in the first decoding (130) process, and AI data can be used in the AI upscaling (140) process.

Video data may be transmitted in bitstream form. The image data may include data obtained based on pixel values in the first image 115, for example, residual data that is the difference between the first image 115 and the predicted data of the first image 115. . Additionally, the image data includes information used in the first encoding 120 process of the first image 115. For example, the image data includes prediction mode information used to first encode the first image 115 (120), motion information, and information related to quantization parameters used in the first encoding (120). can do. The video data is the video compression method used in the first encoding 120 process among video compression methods using frequency transformation, such as MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1. It may be generated according to rules, for example, syntax.

AI data is used for AI upscaling (140) based on the second DNN. As described above, since the first DNN and the second DNN are trained in conjunction, the AI data includes information that allows accurate AI upscaling 140 of the second image 135 through the second DNN to be performed. . In the AI decoding process, the second image 135 may be AI upscaled 140 to the target resolution and/or image quality based on AI data.

AI data can be transmitted along with video data in the form of a bitstream. Alternatively, depending on the implementation, AI data may be transmitted separately from video data in the form of frames or packets. Image data and AI data obtained as a result of AI encoding may be transmitted through the same network or different networks.

Figure 2 is a block diagram showing the configuration of an AI decoding device 200 according to an embodiment.

Referring to FIG. 2, the AI decoding device 200 according to an embodiment may include a receiving unit 210 and an AI decoding unit 230. The receiving unit 210 may include a communication unit 212, a parsing unit 214, and an output unit 216. The AI decoding unit 230 may include a first decoding unit 232 and an AI upscaling unit 234.

The receiving unit 210 receives and parses the AI-encoded data obtained as a result of AI encoding, separates image data from AI data, and outputs it to the AI decoder 230.

Specifically, the communication unit 212 receives AI encoded data obtained as a result of AI encoding through the network. AI encoded data obtained as a result of AI encoding includes image data and AI data. Video data and AI data can be received through homogeneous or heterogeneous networks.

The parsing unit 214 receives the AI encoded data received through the communication unit 212, parses it, and divides it into image data and AI data. For example, by reading the header of data obtained from the communication unit 212, it is possible to distinguish whether the data is video data or AI data. In one example, the parsing unit 214 separates image data and AI data through the header of the data received through the communication unit 212 and transmits it to the output unit 216, and the output unit 216 separates the image data from the AI data. Data is transmitted to the first decoding unit 232 and the AI upscaling unit 234. At this time, the image data included in the AI encoded data is image data acquired through a certain codec (e.g., MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9 or AV1) It can also be confirmed that it is. In this case, the corresponding information can be transmitted to the first decoding unit 232 through the output unit 216 so that the video data can be processed with the confirmed codec.

In one embodiment, the AI-encoded data parsed by the parsing unit 214 includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and floptical disks. It may also be obtained from a data storage medium including the same magneto-optical medium.

The first decoder 232 restores the second image 135 corresponding to the first image 115 based on the image data. The second image 135 acquired by the first decoding unit 232 is provided to the AI upscaling unit 234. Depending on the implementation, first decoding-related information such as prediction mode information, motion information, and quantization parameter information included in the image data may be further provided to the AI upscale unit 234.

The AI upscaling unit 234 that receives the AI data AI upscales the second image 135 based on the AI data. Depending on the implementation, AI upscaling may be performed by further using first decoding-related information such as prediction mode information and quantization parameter information included in the image data.

Although the receiving unit 210 and the AI decoding unit 230 according to one embodiment are described as separate devices, they can be implemented through a single processor. In this case, the receiving unit 210 and the AI decoding unit 230 may be implemented as a dedicated processor, or may be implemented through a combination of a general-purpose processor such as an AP, CPU, or GPU and S/W. Additionally, in the case of a dedicated processor, it may be implemented including a memory for implementing an embodiment of the present disclosure, or may be implemented including a memory processing unit for using an external memory.

Additionally, the receiving unit 210 and the AI decoding unit 230 may be composed of a plurality of processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of multiple general-purpose processors such as AP, CPU, and GPU and S/W. Likewise, the AI upscaling unit 234 and the first decoding unit 232 may each be implemented with different processors.

AI data provided to the AI upscaling unit 234 includes information that allows AI upscaling of the second image 135. At this time, the upscale target must correspond to the downscale of the first DNN. Therefore, AI data must include information that can confirm the downscale target of the first DNN.

Specific examples of information included in AI data include information on the difference between the resolution of the original image 105 and the resolution of the first image 115, and information related to the first image 115.

The difference information may be expressed as information about the degree of resolution conversion of the first image 115 compared to the original image 105 (for example, resolution conversion rate information). And, because the resolution of the first image 115 can be known through the resolution of the restored second image 135 and the degree of resolution conversion can be confirmed through this, the difference information can be expressed only with the resolution information of the original image 105. It may be possible. Here, resolution information may be expressed as horizontal/vertical screen size, or as a ratio (16:9, 4:3, etc.) and the size of one axis. Additionally, if there is preset resolution information, it may be expressed in the form of an index or flag.

In addition, the information related to the first image 115 includes at least one of the bitrate of the image data obtained as a result of the first encoding of the first image 115 and the codec type used during the first encoding of the first image 115. It may contain information about

The AI upscale unit 234 may determine an upscale target of the second image 135 based on at least one of difference information and information related to the first image 115 included in the AI data. For example, the upscale target may indicate to what resolution the second image 135 should be upscaled. When the upscale target is determined, the AI upscale unit 234 AI upscales the second image 135 through the second DNN to obtain the third image 145 corresponding to the upscale target.

Before explaining how the AI upscale unit 234 AI upscales the second image 135 to match the upscale target, refer to FIGS. 3 and 4 for the AI upscale process through the second DNN. Explain.

FIG. 3 is an exemplary diagram showing the second DNN 300 for AI upscaling of the second image 135, and FIG. 4 shows the convolution operation in the first convolution layer 310 shown in FIG. 3. It is showing.

As shown in FIG. 3, the second image 135 is input to the first convolution layer 310. The 3 As a result of convolution processing, four feature maps are generated by four filter kernels. Each feature map represents unique characteristics of the second image 135. For example, each feature map may represent vertical characteristics, horizontal characteristics, or edge characteristics of the second image 135.

Referring to FIG. 4, the convolution operation in the first convolution layer 310 will be described in detail.

Through multiplication and addition operations between the parameters of the filter kernel 430 with a size of 3 A feature map 450 may be created. Since four filter kernels are used in the first convolution layer 310, four feature maps can be generated through a convolution operation process using four filter kernels.

In FIG. 4 , I1 to I49 displayed in the second image 135 represent pixels of the second image 135, and F1 to F9 displayed in the filter kernel 430 represent parameters of the filter kernel 430. Additionally, M1 to M9 displayed in the feature map 450 represent samples of the feature map 450.

Figure 4 illustrates that the second image 135 includes 49 pixels, but this is only an example. If the second image 135 has a resolution of 4K, for example, 3840 Can contain pixels.

In the convolution operation process, pixel values of I1, I2, I3, I8, I9, I10, I15, I16, and I17 of the second image 135 and F1, F2, F3, F4, and F5 of the filter kernel 430 , F6, F7, F8, and F9, each multiplication operation is performed, and a value obtained by combining the result values of the multiplication operation (for example, an addition operation) may be assigned as the value of M1 of the feature map 450. If the stride of the convolution operation is 2, each of the pixel values of I3, I4, I5, I10, I11, I12, I17, I18, and I19 of the second image 135 and F1 and F2 of the filter kernel 430 , F3, F4, F5, F6, F7, F8, and F9 are each multiplied, and a value obtained by combining the resulting values of the multiplication operation may be assigned as the value of M2 of the feature map 450.

While the filter kernel 430 moves according to the stride until it reaches the last pixel of the second image 135, a convolution operation is performed between the pixel values in the second image 135 and the parameters of the filter kernel 430. By performing this, a feature map 450 having a predetermined size can be obtained.

According to the present disclosure, through joint training of the first DNN and the second DNN, the parameters of the second DNN, for example, the parameters of the filter kernel used in the convolutional layers of the second DNN (e.g., the filter The values of F1, F2, F3, F4, F5, F6, F7, F8, and F9) of kernel 430 may be optimized. As described above, the AI upscale unit 234 determines an upscale target corresponding to the downscale target of the first DNN based on the AI data, and sets the parameters corresponding to the determined upscale target to the convolution of the second DNN. It can be determined by the parameters of the filter kernel used in the solution layers.

The convolution layers included in the first DNN and the second DNN can be processed according to the convolution operation process described in relation to FIG. 4, but the convolution operation process described in FIG. 4 is only an example and is limited thereto. That is not the case.

Referring again to FIG. 3, the feature maps output from the first convolution layer 310 are input to the first activation layer 320.

The first activation layer 320 may provide non-linear characteristics to each feature map. The first activation layer 320 may include, but is not limited to, a sigmoid function, a Tanh function, and a Rectified Linear Unit (ReLU) function.

Giving nonlinear characteristics in the first activation layer 320 means changing and outputting some sample values of the feature map, which is the output of the first convolution layer 310. At this time, the change is performed by applying non-linear characteristics.

The first activation layer 320 determines whether to transfer sample values of feature maps output from the first convolution layer 310 to the second convolution layer 330. For example, some sample values of the feature maps are activated by the first activation layer 320 and transmitted to the second convolution layer 330, and some sample values are activated by the first activation layer 320. It is deactivated and is not transmitted to the second convolution layer 330. Unique characteristics of the second image 135 indicated by the feature maps are emphasized by the first activation layer 320.

The feature maps 325 output from the first activation layer 320 are input to the second convolution layer 330. One of the feature maps 325 shown in FIG. 3 is a result of the feature map 450 described with reference to FIG. 4 processed in the first activation layer 320.

3 The output of the second convolution layer 330 is input to the second activation layer 340. The second activation layer 340 may provide non-linear characteristics to input data.

The feature maps 345 output from the second activation layer 340 are input to the third convolution layer 350. 3X3X1 shown in the third convolution layer 350 shown in FIG. 3 illustrates convolution processing to create one output image using one filter kernel of size 3 The third convolution layer 350 is a layer for outputting the final image and generates one output using one filter kernel. According to the example of the present disclosure, the third convolution layer 350 may output the third image 145 through the result of the convolution operation.

DNN setting information indicating the number of filter kernels, filter kernel parameters, etc. of the first convolution layer 310, second convolution layer 330, and third convolution layer 350 of the second DNN 300 is As will be described later, there may be a plurality of DNNs, and the plurality of DNN configuration information must be linked to the plurality of DNN configuration information of the first DNN. Linkage between the plurality of DNN configuration information of the second DNN and the plurality of DNN configuration information of the first DNN may be implemented through linked learning of the first DNN and the second DNN.

Figure 3 shows that the second DNN (300) includes three convolutional layers (310, 330, 350) and two activation layers (320, 340), but this is only an example and an implementation example. Depending on this, the number of convolutional layers and activation layers can be changed in various ways. Additionally, depending on the implementation, the second DNN 300 may be implemented through a recurrent neural network (RNN). In this case, it means changing the CNN structure of the second DNN 300 according to the example of the present disclosure to the RNN structure.

In one embodiment, the AI upscale unit 234 may include at least one Arithmetic logic unit (ALU) for the above-described convolution operation and activation layer operation. ALU can be implemented as a processor. For the convolution operation, the ALU may include a multiplier that performs a multiplication operation between sample values of the feature map output from the second image 135 or the previous layer and sample values of the filter kernel, and an adder that adds the resultant values of the multiplication. You can. In addition, for the operation of the activation layer, the ALU uses a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and compares the multiplication result with a predetermined value to determine the input sample value. It may include a comparator that determines whether to pass on to the next layer.

Below, a method by which the AI upscale unit 234 AI upscales the second image 135 to match the upscale target will be described.

In one embodiment, the AI upscale unit 234 may store a plurality of DNN setting information that can be set in the second DNN.

Here, the DNN configuration information may include information about at least one of the number of convolutional layers included in the second DNN, the number of filter kernels for each convolutional layer, and the parameters of each filter kernel. A plurality of DNN configuration information may each correspond to various upscale targets, and the second DNN may operate based on DNN configuration information corresponding to a specific upscale target. The second DNN may have a different structure depending on the DNN configuration information. For example, depending on some DNN configuration information, the second DNN may include three convolutional layers, and depending on other DNN configuration information, the second DNN may include four convolutional layers.

In one embodiment, the DNN configuration information may include only parameters of the filter kernel used in the second DNN. In this case, the structure of the second DNN is not changed, but only the parameters of the internal filter kernel may vary depending on the DNN configuration information.

The AI upscaling unit 234 may obtain DNN setting information for AI upscaling of the second image 135 from among a plurality of DNN setting information. Each of the plurality of DNN setting information used here is information for acquiring the third image 145 of a predetermined resolution and/or predetermined image quality, and is trained in conjunction with the first DNN.

For example, any one of the plurality of DNN setting information may be a third image 145 with a resolution twice as large as that of the second image 135, for example, a second image of 2K (2048*1080). It may include information for acquiring the third image 145 of 4K (4096*2160), which is twice as large as the second image 135, and the other DNN setting information is 4K resolution higher than the resolution of the second image 135. Information for acquiring the third image 145 of twice the resolution, for example, the third image 145 of 8K (8192*4320), which is 4 times larger than the second image 135 of 2K (2048*1080). may include.

Each of the plurality of DNN setting information is created in connection with the DNN setting information of the first DNN of the AI encoding device 600, and the AI upscale unit 234 has an enlargement ratio corresponding to the reduction ratio of the DNN setting information of the first DNN. Accordingly, one DNN setting information among a plurality of DNN setting information is acquired. For this purpose, the AI upscale unit 234 must check the information of the first DNN. In order for the AI upscale unit 234 to confirm the information of the first DNN, the AI decoding device 200 according to an embodiment receives AI data including information of the first DNN from the AI encoding device 600. .

In other words, the AI upscale unit 234 uses the information received from the AI encoding device 600 to confirm the information targeted by the DNN setting information of the first DNN used to obtain the first image 115. And, it is possible to obtain DNN setting information of the second DNN trained in conjunction with it.

When DNN setting information for AI upscaling of the second image 135 among the plurality of DNN setting information is obtained, input data may be processed based on the second DNN operating according to the obtained DNN setting information.

For example, when one DNN setting information is acquired, the first convolution layer 310, the second convolution layer 330, and the third convolution layer ( 350) For each, the number of filter kernels included in each layer and the parameters of the filter kernel are set to the values included in the obtained DNN configuration information.

Specifically, the parameters of the 3 It is set, and if there is a change in the DNN setting information later, it can be replaced with {2, 2, 2, 2, 2, 2, 2, 2, 2}, which are parameters included in the changed DNN setting information.

The AI upscaling unit 234 can obtain DNN setting information for upscaling the second image 135 among a plurality of DNN setting information based on the information included in the AI data, and is used to obtain the DNN setting information. AI data that is used is explained in detail.

In one embodiment, the AI upscaling unit 234 may obtain DNN setting information for upscaling the second image 135 among a plurality of DNN setting information based on difference information included in AI data. For example, based on the difference information, the resolution of the original image 105 (e.g., 4K (4096*2160)) is higher than the resolution of the first image 115 (e.g., 2K (2048*1080)). If it is confirmed to be twice as large, the AI upscale unit 234 can obtain DNN setting information that can increase the resolution of the second image 135 by two times.

In another embodiment, the AI upscale unit 234 sets a DNN to AI upscale the second image 135 among a plurality of DNN setting information based on information related to the first image 115 included in the AI data. Information can be obtained. The AI upscale unit 234 may predetermine a mapping relationship between image-related information and DNN setting information and obtain DNN setting information mapped to information related to the first image 115.

Figure 5 is an exemplary diagram showing the mapping relationship between various image-related information and various DNN setting information.

Through the embodiment according to FIG. 5, it can be seen that the AI encoding/AI decoding process of an embodiment of the present disclosure does not only consider changes in resolution. As shown in Figure 5, DNN setting information is provided by individually or all considering resolutions such as SD, HD, and Full HD, bitrates such as 10Mbps, 15Mbps, and 20Mbps, and codec information such as AV1, H.264, and HEVC. A choice can be made. To take this into account, training that considers each element in the AI training process must be conducted in conjunction with the encoding and decoding processes (see Figure 9).

Therefore, according to the training content, as shown in FIG. 5, when a plurality of DNN setting information is provided based on image-related information including codec type, image resolution, etc., the first image received in the AI decoding process ( 115) DNN setting information for AI upscaling of the second image 135 can be obtained based on the related information.

That is, the AI upscale unit 234 matches the image-related information shown on the left side of the table shown in FIG. 5 with the DNN setting information on the right side of the table, so that it can use DNN setting information according to the image-related information.

As illustrated in FIG. 5, from the information related to the first image 115, the resolution of the first image 115 is SD, and the bit rate of the image data obtained as a result of the first encoding of the first image 115 is 10 Mbps. , If it is confirmed that the first image 115 is first encoded with the AV1 codec, the AI upscale unit 234 may use A DNN setting information among a plurality of DNN setting information.

In addition, from the information related to the first image 115, the resolution of the first image 115 is HD, the bit rate of the image data obtained as a result of the first encoding is 15Mbps, and the first image 115 is encoded with the H.264 codec. If it is confirmed that it has been first encoded, the AI upscale unit 234 can use the B DNN setting information among the plurality of DNN setting information.

In addition, from the information related to the first image 115, the resolution of the first image 115 is Full HD, the bit rate of the image data obtained as a result of the first encoding of the first image 115 is 20Mbps, and the first image ( If 115) is confirmed to be first encoded with the HEVC codec, the AI upscale unit 234 uses the C DNN setting information among the plurality of DNN setting information, the resolution of the first image 115 is Full HD, and the 1 If the bit rate of the image data obtained as a result of the first encoding of the image 115 is 15 Mbps and the first image 115 is confirmed to be first encoded with the HEVC codec, the AI upscale unit 234 uses a plurality of DNNs Among the setting information, D DNN setting information can be used. One of the C DNN setting information and the D DNN setting information is selected depending on whether the bit rate of the image data obtained as a result of the first encoding of the first image 115 is 20 Mbps or 15 Mbps. When the first image 115 of the same resolution is first encoded with the same codec, the bit rates of the image data are different, which means that the image quality of the restored images is different. Therefore, the first DNN and the second DNN can be trained in conjunction based on a predetermined image quality, and accordingly, the AI upscale unit 234 trains the DNN according to the bitrate of the image data, which represents the image quality of the second image 135. Setting information can be obtained.

In another embodiment, the AI upscale unit 234 uses information provided from the first decoder 232 (prediction mode information, motion information, quantization parameter information, etc.) and the first image 115 included in the AI data. Considering all related information, DNN setting information for AI upscaling of the second image 135 from among the plurality of DNN setting information may be obtained. For example, the AI upscale unit 234 receives quantization parameter information used in the first encoding process of the first image 115 from the first decoder 232, and converts the first image 115 from the AI data. It is also possible to check the bitrate of the image data obtained as a result of encoding and obtain DNN setting information corresponding to the quantization parameters and bitrate. Even if the bit rate is the same, there may be differences in the quality of the restored image depending on the complexity of the image. The bit rate is a value representing the entire first image 115 to be encoded, and is the value of each frame within the first image 115. Image quality may vary. Therefore, considering the prediction mode information, motion information, and/or quantization parameters that can be obtained for each frame from the first decoder 232, a DNN setting more suitable for the second image 135 compared to using only AI data Information can be obtained.

Additionally, depending on the implementation, AI data may include an identifier of mutually promised DNN setting information. The identifier of the DNN setting information is an upscale target corresponding to the downscale target of the first DNN, and is DNN setting information trained in conjunction between the first DNN and the second DNN to AI upscale the second image 135. This is information to distinguish between pairs. After obtaining the identifier of the DNN setting information included in the AI data, the AI upscaling unit 234 can AI upscale the second image 135 using the DNN setting information corresponding to the identifier of the DNN setting information. . For example, an identifier indicating each of a plurality of DNN setting information configurable in the first DNN and an identifier indicating each of a plurality of DNN setting information configurable in the second DNN may be pre-designated. In this case, the same identifier may be designated for each pair of DNN setting information that can be set for each of the first DNN and the second DNN. The AI data may include an identifier of DNN setting information set in the first DNN for AI downscaling of the original image 105. The AI upscaling unit 234, which has received the AI data, may AI upscale the second image 135 using DNN setting information indicated by an identifier included in the AI data among a plurality of DNN setting information.

Additionally, depending on the implementation, AI data may include DNN setting information. The AI upscaling unit 234 may obtain DNN setting information included in AI data and then AI upscale the second image 135 using the DNN setting information.

Depending on the implementation, if the information constituting the DNN setting information (e.g., the number of convolution layers, the number of filter kernels for each convolution layer, the parameters of each filter kernel, etc.) is stored in the form of a lookup table, The AI upscale unit 234 obtains DNN setting information by combining some of the lookup table values based on the information included in the AI data, and AI upscales the second image 135 using the obtained DNN setting information. It can also be scaled.

Depending on the implementation, when the structure of the DNN corresponding to the upscale target is determined, the AI upscale unit 234 may obtain DNN setting information corresponding to the determined structure of the DNN, for example, parameters of the filter kernel. .

The AI upscale unit 234 acquires DNN setting information of the second DNN through AI data including information related to the first DNN, and outputs the second image (135) through the second DNN set with the obtained DNN setting information. ) is AI upscaled, which can reduce memory usage and computation compared to upscaling by directly analyzing the characteristics of the second image 135.

In one embodiment, when the second image 135 consists of a plurality of frames, the AI upscale unit 234 may independently obtain DNN setting information for each predetermined number of frames, or may provide common information for all frames. DNN configuration information can also be obtained.

FIG. 6 is a diagram illustrating a second image 135 composed of a plurality of frames.

As shown in FIG. 6, the second image 135 may be composed of frames corresponding to t0 to tn.

In one example, the AI upscale unit 234 may obtain DNN configuration information of the second DNN through AI data and AI upscale the frames corresponding to t0 to tn based on the obtained DNN configuration information. That is, frames corresponding to t0 to tn may be AI upscaled based on common DNN configuration information.

In another example, the AI upscale unit 234 uses some of the frames corresponding to t0 to tn, for example, frames corresponding to t0 to ta, as 'A' DNN setting information obtained from AI data. AI can be upscaled, and frames corresponding to ta+1 to tb can be AI upscaled using 'B' DNN setting information obtained from AI data. Additionally, the AI upscaling unit 234 can AI upscale the frames corresponding to tb+1 to tn using 'C' DNN setting information obtained from AI data. In other words, the AI upscale unit 234 independently acquires DNN setting information for each group including a predetermined number of frames among a plurality of frames, and uses the independently acquired DNN setting information for the frames included in each group to provide AI It can be upscaled.

In another example, the AI upscale unit 234 may independently obtain DNN setting information for each frame constituting the second image 135. That is, if the second image 135 consists of three frames, the AI upscale unit 234 AI upscales the first frame with the DNN setting information obtained in relation to the first frame, and The second frame can be AI upscaled using the DNN setting information obtained in relation to and the third frame can be AI upscaled using the DNN setting information acquired in relation to the third frame. DNN setting information is obtained based on the information provided from the above-described first decoder 232 (prediction mode information, motion information, quantization parameter information, etc.) and information related to the first image 115 included in the AI data. Depending on the method, DNN setting information may be independently obtained for each frame constituting the second image 135. This is because mode information, quantization parameter information, etc. can be independently determined for each frame constituting the second image 135.

In another example, the AI data may include information indicating up to which frames the DNN setting information obtained based on the AI data is valid. For example, if the AI data includes information that the DNN setting information is valid up to the ta frame, the AI upscale unit 234 AI upscales t0 to ta frames with the DNN setting information obtained based on the AI data. do. And, if other AI data includes information that the DNN setting information is valid up to the tn frame, the AI upscale unit 234 uses the DNN setting information obtained based on the other AI data for ta+1 to tn frames. AI can be upscaled.

Below, with reference to FIG. 7 , the AI encoding device 600 for AI encoding of the original image 105 will be described.

FIG. 7 is a block diagram showing the configuration of an AI encoding device 600 according to an embodiment.

Referring to FIG. 7, the AI encoding device 600 may include an AI encoding unit 610 and a transmitting unit 630. The AI encoder 610 may include an AI downscale unit 612 and a first encoder 614. The transmission unit 630 may include a data processing unit 632 and a communication unit 634.

Figure 7 shows the AI encoder 610 and the transmitter 630 as separate devices, but the AI encoder 610 and the transmitter 630 can be implemented through one processor. In this case, it may be implemented with a dedicated processor, or it may be implemented through a combination of a general-purpose processor such as an AP, CPU, or GPU and S/W. Additionally, in the case of a dedicated processor, it may be implemented including a memory for implementing an embodiment of the present disclosure, or may be implemented including a memory processing unit for using an external memory.

Additionally, the AI encoding unit 610 and the transmitting unit 630 may be composed of a plurality of processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of multiple general-purpose processors such as AP, CPU, and GPU and S/W. The AI downscale unit 612 and the first encoder 614 may also be implemented with different processors.

The AI encoder 610 performs AI downscaling of the original image 105 and first encoding of the first image 115, and transmits the AI data and image data to the transmission unit 630. The transmission unit 630 transmits AI data and image data to the AI decoding device 200.

The image data includes data obtained as a result of first encoding of the first image 115. The image data may include data obtained based on pixel values in the first image 115, for example, residual data that is the difference between the first image 115 and the predicted data of the first image 115. . Additionally, the image data includes information used in the first encoding process of the first image 115. For example, the image data may include prediction mode information used to first encode the first image 115, motion information, and information related to quantization parameters used to first encode the first image 115. .

The AI data includes information that allows the AI upscale unit 234 to AI upscale the second image 135 to an upscale target corresponding to the downscale target of the first DNN. In one example, the AI data may include difference information between the original image 105 and the first image 115. Additionally, AI data may include information related to the first image 115. Information related to the first image 115 includes the resolution of the first image 115, the bitrate of the image data obtained as a result of the first encoding of the first image 115, and the information used when first encoding the first image 115. It may contain information about at least one of the codec types provided.

In one embodiment, the AI data may include an identifier of mutually agreed DNN setting information so that the second image 135 can be AI upscaled to an upscale target corresponding to the downscale target of the first DNN. .

Additionally, in one embodiment, AI data may include DNN setting information that can be set to the second DNN.

The AI downscale unit 612 may obtain the AI downscaled first image 115 from the original image 105 through the first DNN. The AI downscale unit 612 may determine the downscale target of the original image 105 based on a predetermined standard.

In order to acquire the first image 115 that meets the downscale target, the AI downscale unit 612 may store a plurality of DNN setting information that can be set in the first DNN. The AI downscale unit 612 acquires DNN setting information corresponding to the downscale target among a plurality of DNN setting information, and AI downscales the original image 105 through the first DNN set with the obtained DNN setting information. .

Each of the plurality of DNN setting information may be trained to obtain the first image 115 of a predetermined resolution and/or predetermined image quality. For example, one of the plurality of DNN setting information may be a first image 115 with a resolution that is 1/2 times smaller than the resolution of the original image 105, for example, 4K (4096*2160). It may include information for acquiring the first image 115 of 2K (2048*1080), which is 1/2 times smaller than the original image 105, and the other DNN setting information is the resolution of the original image 105. A first image 115 with a resolution that is 1/4 times smaller than the original image 105 of 8K (8192*4320). ) may include information to obtain.

Depending on the implementation, if the information constituting the DNN setting information (e.g., the number of convolution layers, the number of filter kernels for each convolution layer, the parameters of each filter kernel, etc.) is stored in the form of a lookup table, The AI downscale unit 612 may obtain DNN setting information by combining some selected lookup table values according to the downscale target, and may AI downscale the original image 105 using the obtained DNN setting information.

Depending on the implementation, the AI downscale unit 612 may determine the structure of the DNN corresponding to the downscale target and obtain DNN setting information corresponding to the determined structure of the DNN, for example, parameters of the filter kernel. .

The plurality of DNN setting information for AI downscaling of the original image 105 may have optimized values by jointly training the first DNN and the second DNN. Here, each DNN configuration information includes at least one of the number of convolution layers included in the first DNN, the number of filter kernels for each convolution layer, and the parameters of each filter kernel.

The AI downscale unit 612 sets the first DNN with the DNN setting information determined for AI downscaling of the original image 105, and uses the first DNN to display the first image 115 with a predetermined resolution and/or predetermined image quality. It can be obtained through When DNN setting information for AI downscaling of the original image 105 is obtained from among the plurality of DNN setting information, each layer in the first DNN can process the input data based on the information included in the DNN setting information. .

Below, a description will be given of how the AI downscale unit 612 determines the downscale target. For example, the downscale target may indicate how much the first image 115 whose resolution is reduced from the original image 105 should be obtained.

In one embodiment, the AI downscale unit 612 determines compression rate (e.g., resolution difference between the original image 105 and the first image 115, target bitrate), compression quality (e.g., bitrate The downscale target may be determined based on at least one of the type), compression history information, and the type of the original image 105.

In one example, the AI downscale unit 612 may determine a downscale target based on a compression rate or compression quality that is preset or input from the user.

As another example, the AI downscale unit 612 may determine the downscale target using compression history information stored in the AI encoding device 600. For example, according to the compression history information available to the AI encoding device 600, the user's preferred encoding quality or compression rate may be determined, and the downscale target may be determined according to the encoding quality determined based on the compression history information. You can. For example, the resolution and quality of the first image 115 may be determined according to the most frequently used encoding quality according to compression history information.

As another example, the AI downscale unit 612 determines the encoding quality that has been used more than a predetermined threshold (for example, the average quality of the encoding quality that has been used more than a predetermined threshold) according to compression history information. The downscale target may be determined based on .

As another example, the AI downscale unit 612 may determine the downscale target based on the resolution and type (eg, file format) of the original image 105.

In one embodiment, when the original image 105 consists of a plurality of frames, the AI downscale unit 612 may independently determine a downscale target for each predetermined number of frames, or may use a common downscale for all frames. You can also decide on a target.

In one example, the AI downscale unit 612 may divide the frames constituting the original image 105 into a predetermined number of groups and independently determine a downscale target for each group. The same or different downscale targets may be determined for each group. The number of frames included in groups may be the same or different for each group.

In another example, the AI downscale unit 612 may independently determine a downscale target for each frame constituting the original image 105. The same or different downscale targets may be determined for each frame.

Below, an exemplary structure of the first DNN 700, which is the basis for AI downscale, will be described.

FIG. 8 is an exemplary diagram showing a first DNN 700 for AI downscaling of the original image 105.

As shown in FIG. 8, the original image 105 is input to the first convolution layer 710. The first convolution layer 710 performs convolution processing on the original image 105 using 32 filter kernels with a size of 5 x 5. The 32 feature maps generated as a result of convolution processing are input to the first activation layer 720. The first activation layer 720 can provide non-linear characteristics to 32 feature maps.

The first activation layer 720 determines whether to transfer sample values of feature maps output from the first convolution layer 710 to the second convolution layer 730. For example, some sample values of the feature maps are activated by the first activation layer 720 and transmitted to the second convolution layer 730, and some sample values are activated by the first activation layer 720. It is deactivated and is not transmitted to the second convolution layer 730. Information represented by the feature maps output from the first convolutional layer 710 is emphasized by the first activation layer 720.

The output 725 of the first activation layer 720 is input to the second convolution layer 730. The second convolution layer 730 performs convolution processing on the input data using 32 filter kernels with a size of 5 x 5. The 32 feature maps output as a result of the convolution processing are input to the second activation layer 740, and the second activation layer 740 can provide non-linear characteristics to the 32 feature maps.

The output 745 of the second activation layer 740 is input to the third convolution layer 750. The third convolution layer 750 performs convolution processing on the input data using one filter kernel with a size of 5 x 5. As a result of convolution processing, one image may be output from the third convolution layer 750. The third convolution layer 750 is a layer for outputting the final image and obtains one output using one filter kernel. According to the example of the present disclosure, the third convolution layer 750 may output the first image 115 through the result of the convolution operation.

DNN setting information indicating the number of filter kernels, filter kernel parameters, etc. of the first convolution layer 710, second convolution layer 730, and third convolution layer 750 of the first DNN 700 is There may be multiple DNNs, and the plurality of DNN configuration information must be linked to the plurality of DNN configuration information of the second DNN. Linkage between the plurality of DNN configuration information of the first DNN and the plurality of DNN configuration information of the second DNN may be implemented through linked learning of the first DNN and the second DNN.

Figure 8 shows that the first DNN (700) includes three convolutional layers (710, 730, 750) and two activation layers (720, 740), but this is only an example and an implementation example Depending on this, the number of convolutional layers and activation layers can be changed in various ways. Additionally, depending on the implementation, the first DNN 700 may be implemented through a recurrent neural network (RNN). In this case, it means changing the CNN structure of the first DNN 700 according to the example of the present disclosure to the RNN structure.

In one embodiment, the AI downscale unit 612 may include at least one ALU for convolution operation and activation layer operation. ALU can be implemented as a processor. For the convolution operation, the ALU may include a multiplier that performs a multiplication operation between the sample values of the original image 105 or the feature map output from the previous layer and the sample values of the filter kernel, and an adder that adds the resultant values of the multiplication. there is. In addition, for the operation of the activation layer, the ALU uses a multiplier that multiplies the input sample value by a weight used in a predetermined sigmoid function, Tanh function, or ReLU function, and compares the multiplication result with a predetermined value to determine the input sample value. It may include a comparator that determines whether to pass on to the next layer.

Referring again to FIG. 7, the first encoder 614, which receives the first image 115 from the AI downscale unit 612, first encodes the first image 115 to produce the first image 115. The amount of information can be reduced. As a result of the first encoding by the first encoder 614, image data corresponding to the first image 115 may be obtained.

The data processing unit 632 processes at least one of AI data and image data so that it can be transmitted in a predetermined form. For example, when AI data and image data must be transmitted in the form of a bitstream, the data processing unit 632 processes the AI data so that the AI data is expressed in the form of a bitstream, and transmits one bitstream through the communication unit 634. Transmits AI data and video data in the form of As another example, the data processing unit 632 processes AI data so that the AI data is expressed in a bitstream form, and sends each of the bitstream corresponding to the AI data and the bitstream corresponding to the image data through the communication unit 634 to the communication unit 634. ) is transmitted through. As another example, the data processing unit 632 processes AI data so that the AI data is expressed as a frame or packet, and transmits image data in the form of a bitstream and AI data in the form of a frame or packet through the communication unit 634.

The communication unit 630 transmits AI encoded data obtained as a result of AI encoding through the network. AI encoded data obtained as a result of AI encoding includes image data and AI data. Video data and AI data can be transmitted through homogeneous or heterogeneous networks.

In one embodiment, the AI-encoded data obtained as a result of processing by the data processing unit 632 is magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and floptical disks. It may also be stored in a data storage medium including a magneto-optical medium such as.

Hereinafter, with reference to FIG. 9, a method of jointly training the first DNN 700 and the second DNN 300 will be described.

FIG. 9 is a diagram for explaining a method of training the first DNN 700 and the second DNN 300.

In one embodiment, the original image 105 encoded through the AI encoding process is restored as the third image 145 through the AI decoding process, and the third image 145 and the original image 105 obtained as a result of the AI decoding are In order to maintain similarity, correlation is required in the AI encoding process and AI decoding process. In other words, information lost in the AI encoding process must be able to be restored in the AI decoding process, and for this purpose, linked training of the first DNN (700) and the second DNN (300) is required.

For accurate AI decoding, it is ultimately necessary to reduce the quality loss information 830 corresponding to the comparison result between the third training image 804 and the original training image 801 shown in FIG. 9. Accordingly, the quality loss information 830 is used for training both the first DNN 700 and the second DNN 300.

First, the training process shown in Figure 9 will be described.

In Figure 9, the original training image 801 is an image subject to AI downscaling, and the first training image 802 is an image that is AI downscaled from the original training image 801. It's a video. Additionally, the third training image 804 is an AI upscaled image from the first training image 802.

The original training image 801 includes a still image or a video consisting of multiple frames. In one embodiment, the original training image 801 may include a still image or a luminance image extracted from a video consisting of multiple frames. Additionally, in one embodiment, the original training image 801 may include a still image or a patch image extracted from a video consisting of multiple frames. When the original training image 801 consists of a plurality of frames, the first training image 802, the second training image, and the third training image 804 also consist of a plurality of frames. When a plurality of frames of the original training image 801 are sequentially input to the first DNN 700, the first training image 802 and the second training image are generated through the first DNN 700 and the second DNN 300. And a plurality of frames of the third training image 804 may be sequentially acquired.

For linked training of the first DNN 700 and the second DNN 300, the original training image 801 is input to the first DNN 700. The original training image 801 input to the first DNN 700 is AI downscaled and output as the first training image 802, and the first training image 802 is input to the second DNN 300. As a result of AI upscaling of the first training image 802, the third training image 804 is output.

Referring to FIG. 9, a first training image 802 is being input to the second DNN 300. Depending on the implementation, the image obtained through the first encoding and first decoding processes of the first training image 802 A second training image may be input to the second DNN 300. To input the second training image into the second DNN, any one of the codecs among MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 can be used. Specifically, in the first encoding of the first training image 802 and the first decoding of the image data corresponding to the first training image 802, MPEG-2, H.264, MPEG-4, HEVC, and VC-1 , any one of VP8, VP9, and AV1 codecs can be used.

Referring to FIG. 9, separately from the first training image 802 being output through the first DNN 700, a legacy downscaled reduced training image 803 is obtained from the original training image 801. Here, the legacy downscale may include at least one of bilinear scale, bicubic scale, lanczos scale, and stair step scale.

In order to prevent the structural features of the first image 115 from deviating significantly based on the structural features of the original image 105, a reduced training image 803 that preserves the structural features of the original training image 801 is obtained. .

Before training begins, the first DNN 700 and the second DNN 300 may be set to predetermined DNN configuration information. As training progresses, structural loss information 810, complexity loss information 820, and quality loss information 830 may be determined.

Structural loss information 810 may be determined based on a comparison result between the reduced training image 803 and the first training image 802. In one example, the structural loss information 810 may correspond to the difference between the structural information of the reduced training image 803 and the structural information of the first training image 802. Structural information may include various features that can be extracted from an image, such as image luminance, contrast, and histogram. Structural loss information 810 indicates the extent to which structural information of the original training image 801 is maintained in the first training image 802. The smaller the structural loss information 810, the more similar the structural information of the first training image 802 is to that of the original training image 801.

Complexity loss information 820 may be determined based on the spatial complexity of the first training image 802. In one example, the total variance value of the first training image 802 may be used as the spatial complexity. Complexity loss information 820 is related to the bitrate of image data obtained by first encoding the first training image 802. It is defined that the smaller the complexity loss information 820 is, the smaller the bit rate of the image data is.

Quality loss information 830 may be determined based on a comparison result between the original training image 801 and the third training image 804. The quality loss information 830 includes the L1-norm value, L2-norm value, SSIM (Structural Similarity) value, and PSNR-HVS (Peak Signal-To) value for the difference between the original training image 801 and the third training image 804. -It may include at least one of a Noise Ratio-Human Vision System) value, a Multiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF) value, and a Video Multimethod Assessment Fusion (VMAF) value. Quality loss information 830 indicates how similar the third training image 804 is to the original training image 801. The smaller the quality loss information 830 is, the more similar the third training image 804 is to the original training image 801.

Referring to FIG. 9, structural loss information 810, complexity loss information 820, and quality loss information 830 are used for training of the first DNN 700, and quality loss information 830 is used in the training of the second DNN (700). 300) is used for training. That is, the quality loss information 830 is used for training both the first DNN 700 and the second DNN 300.

The first DNN 700 may update parameters so that the final loss information determined based on the structural loss information 810, complexity loss information 820, and quality loss information 830 is reduced or minimized. Additionally, the second DNN 300 may update parameters so that quality loss information 830 is reduced or minimized.

Final loss information for training of the first DNN 700 and the second DNN 300 can be determined as shown in Equation 1 below.

In Equation 1, LossDS represents the final loss information to be reduced or minimized for training of the first DNN (700), and LossUS represents the final loss information to be reduced or minimized for training of the second DNN (300). indicates. Additionally, a, b, c, and d may correspond to predetermined weights.

That is, the first DNN 700 updates parameters in a direction that reduces LossDS in Equation 1, and the second DNN 300 updates parameters in a direction that reduces LossUS. When the parameters of the first DNN 700 are updated according to the LossDS derived in the training process, the first training image 802 obtained based on the updated parameters is different from the first training image 802 in the previous training process. loses, and accordingly, the third training video 804 also becomes different from the third training video 804 from the previous training process. If the third training image 804 is different from the third training image 804 in the previous training process, quality loss information 830 is also newly determined, and the second DNN 300 updates the parameters accordingly. When the quality loss information 830 is newly determined, LossDS is also newly determined, so the first DNN 700 updates parameters according to the newly determined LossDS. That is, parameter update of the first DNN (700) causes parameter update of the second DNN (300), and parameter update of the second DNN (300) causes parameter update of the first DNN (700). In other words, the first DNN 700 and the second DNN 300 are trained in conjunction through sharing the quality loss information 830, so the parameters of the first DNN 700 and the parameters of the second DNN 300 are They can be optimized in relation to each other.

Referring to Equation 1, it can be seen that LossUS is determined based on quality loss information 830, but this is just one example, and LossUS includes at least one of structural loss information 810 and complexity loss information 820, It may also be determined based on quality loss information 830.

Previously, the AI upscale unit 234 of the AI decoding device 200 and the AI downscale unit 612 of the AI encoding device 600 were explained as storing a plurality of DNN setting information, and the AI upscale unit 234 ) and a method of training each of the plurality of DNN setting information stored in the AI downscale unit 612 will be described.

As described in relation to Equation 1, in the case of the first DNN 700, the degree of similarity between the structural information of the first training image 802 and the structural information of the original training image 801 (structural loss information 810) ), the bitrate of the image data obtained as a result of the first encoding of the first training image 802 (complexity loss information 820), and the difference between the third training image 804 and the original training image 801 (quality loss The parameters are updated taking the information 830 into consideration.

In detail, it is possible to obtain a first training image 802 that is similar to the structural information of the original training image 801 and has a low bitrate of the image data obtained when performing the first encoding, and at the same time, the first training image 802 is similar to the structural information of the original training image 801. Parameters of the first DNN 700 may be updated so that the second DNN 300, which AI upscales 802, can obtain a third training image 804 similar to the original training image 801.

By adjusting the weights of a, b, and c in Equation 1, the direction in which the parameters of the first DNN 700 are optimized becomes different. For example, when the weight of b is determined to be high, the parameters of the first DNN 700 may be updated with greater importance given to lowering the bit rate than to the quality of the third training image 804. In addition, when the weight of c is determined to be high, more importance is placed on increasing the quality of the third training image 804 than on increasing the bit rate or maintaining the structural information of the original training image 801. 1 Parameters of the DNN 700 may be updated.

Additionally, the direction in which the parameters of the first DNN 700 are optimized may vary depending on the type of codec used to first encode the first training image 802. This is because, depending on the type of codec, the second training image to be input to the second DNN 300 may vary.

That is, the parameters of the first DNN 700 and the parameters of the second DNN 300 are linked based on weight a, weight b, weight c, and the type of codec for the first encoding of the first training image 802. So it can be updated. Therefore, when weight a, weight b, and weight c are each set to predetermined values, and the type of codec is determined to be a predetermined type, when the first DNN (700) and the second DNN (300) are trained, they are linked to each other. The optimized parameters of the first DNN 700 and the parameters of the second DNN 300 may be determined.

Then, after changing the type of weight a, weight b, weight c, and codec, when the first DNN 700 and the second DNN 300 are trained, the parameters of the optimized first DNN 700 are linked to each other. and parameters of the second DNN 300 may be determined. In other words, if the first DNN 700 and the second DNN 300 are trained while changing the respective values of weight a, weight b, weight c, and codec type, the plurality of DNN settings information trained in conjunction with each other are converted into the first DNN 700 and the second DNN 300. This can be determined in the DNN 700 and the second DNN 300.

As previously described with reference to FIG. 5 , a plurality of DNN setting information of the first DNN 700 and the second DNN 300 may be mapped to the first image-related information. To establish this mapping relationship, the first training image 802 output from the first DNN 700 is first encoded with a specific codec according to a specific bit rate, and the bitstream obtained as a result of the first encoding is first decoded. The obtained second training image can be input to the second DNN (300). That is, after setting the environment so that the first training image 802 of a specific resolution is first encoded at a specific bit rate by a specific codec, by training the first DNN 700 and the second DNN 300, the first DNN settings mapped to the resolution of the training image 802, the type of codec used for the first encoding of the first training image 802, and the bit rate of the bitstream obtained as a result of the first encoding of the first training image 802. Information pairs can be determined. The resolution of the first training image 802, the type of codec used for the first encoding of the first training image 802, and the bit rate of the bitstream obtained according to the first encoding of the first training image 802 are varied. By changing this, the mapping relationship between the plurality of DNN setting information of the first DNN 700 and the second DNN 300 and the first image-related information can be determined.

FIG. 10 is a diagram illustrating the training process of the first DNN 700 and the second DNN 300 by the training device 1000.

Training of the first DNN 700 and the second DNN 300 described with reference to FIG. 9 may be performed by the training device 1000. The training device 1000 includes a first DNN 700 and a second DNN 300. The training device 1000 may be, for example, the AI encoding device 600 or a separate server. The DNN setting information of the second DNN 300 obtained as a result of training is stored in the AI decoding device 200.

Referring to FIG. 10, the training device 1000 initially sets DNN configuration information of the first DNN 700 and the second DNN 300 (S840, S845). As a result, the first DNN 700 and the second DNN 300 can operate according to predetermined DNN setting information. The DNN setting information includes at least the number of convolution layers included in the first DNN (700) and the second DNN (300), the number of filter kernels for each convolution layer, the size of the filter kernel for each convolution layer, and the parameters of each filter kernel. It can contain information about one thing.

The training device 1000 inputs the original training image 801 into the first DNN 700 (S850). The original training image 801 may include at least one frame constituting a still image or video.

The first DNN 700 processes the original training image 801 according to the initially set DNN setting information and outputs the first training image 802 that is AI downscaled from the original training image 801 (S855). Figure 10 shows that the first training image 802 output from the first DNN 700 is directly input to the second DNN 300, but the first training image 802 output from the first DNN 700 ) may be input into the second DNN (300) by the training device (1000). Additionally, the training device 1000 may first encode and first decode the first training image 802 using a predetermined codec, and then input the second training image into the second DNN 300.

The second DNN 300 processes the first training image 802 or the second training image according to the initially set DNN setting information, and generates a third AI upscaled image from the first training image 802 or the second training image. The training video 804 is output (S860).

The training device 1000 calculates complexity loss information 820 based on the first training image 802 (S865).

The training device 1000 compares the reduced training image 803 and the first training image 802 to calculate structural loss information 810 (S870).

The training device 1000 compares the original training image 801 and the third training image 804 and calculates quality loss information 830 (S875).

The first DNN 700 updates the initially set DNN configuration information through a back propagation process based on the final loss information (S880). The training apparatus 1000 may calculate final loss information for training of the first DNN 700 based on complexity loss information 820, structural loss information 810, and quality loss information 830.

The second DNN 300 updates the initially set DNN setting information through a reverse transcription process based on quality loss information or final loss information (S885). The training device 1000 may calculate final loss information for training the second DNN 300 based on the quality loss information 830.

Thereafter, the training device 1000, the first DNN 700, and the second DNN 300 update the DNN configuration information by repeating processes S850 to S885 until the final loss information is minimized. At this time, during each iteration process, the first DNN 700 and the second DNN operate according to the DNN configuration information updated in the previous process.

Table 1 below shows the effects of AI encoding and AI decoding of the original image 105 and of encoding and decoding the original image 105 with HEVC according to an embodiment of the present disclosure.

As can be seen in Table 1, although the subjective image quality when AI encoding and AI decoding content consisting of 300 frames of 8K resolution according to an embodiment of the present disclosure is higher than the subjective image quality when encoding and decoding with HEVC , it can be seen that the bit rate has decreased by more than 50%.

FIG. 11 is an exemplary diagram illustrating an apparatus 20 for AI downscaling of the original image 105 and an apparatus 40 for AI upscaling of the second image 135.

The device 20 receives the original image 105 and provides image data 25 and AI data 30 to the device 40 using the AI downscale unit 1124 and the transformation-based encoding unit 1126. do. In one embodiment, image data 25 corresponds to the image data in FIG. 1 and AI data 30 corresponds to AI data in FIG. 1 . Additionally, in one embodiment, the transform-based encoder 1126 corresponds to the first encoder 614 of FIG. 7, and the AI downscale unit 1124 corresponds to the AI downscale unit 612 of FIG. 7. .

The device 40 receives the AI data 30 and the image data 25, and acquires the third image 145 using the transformation-based decoding unit 1146 and the AI upscaling unit 1144. In one embodiment, the transform-based decoding unit 1146 corresponds to the first decoding unit 232 of FIG. 2, and the AI upscaling unit 1144 corresponds to the AI upscaling unit 234 of FIG. 2.

In one embodiment, device 20 includes a CPU, memory, and a computer program including instructions. Computer programs are stored in memory. In one embodiment, upon execution of a computer program by a CPU, device 20 performs the functions described with respect to FIG. 11 . In one embodiment, the functions described with respect to FIG. 11 are performed by dedicated hardware chips and/or CPUs.

In one embodiment, device 40 includes a CPU, memory, and a computer program including instructions. Computer programs are stored in memory. In one embodiment, upon execution of a computer program by a CPU, device 40 performs the functions described with respect to FIG. 11 . In one embodiment, the functions described with respect to FIG. 11 are performed by dedicated hardware chips and/or CPUs.

11, configuration control 1122 receives one or more input values 10. In one embodiment, one or more input values 10 include the target resolution difference for the AI downscale unit 1124 and the AI upscale unit 1144, the bit rate of the image data 25, and the bit rate of the image data 25. It may include at least one of a rate type (eg, variable bitrate type, constant bitrate type, or average bitrate type, etc.) and a codec type for the conversion-based encoder 1126. One or more input values 10 may be pre-stored in the device 20 or may include values input from a user.

The configuration control unit 1122 controls the operations of the AI downscale unit 1124 and the transform-based encoding unit 1126 based on the received input value 10. In one embodiment, the configuration control unit 1122 obtains DNN setting information for the AI downscale unit 1124 according to the received input value 10, and configures the AI downscale unit 1124 with the obtained DNN setting information. Set it. In one embodiment, the configuration control unit 1122 transmits the received input value 10 to the AI downscale unit 1124, and the AI downscale unit 1124 determines the original image based on the received input value 10. DNN setting information for AI downscaling of (105) can be obtained. In one embodiment, the configuration control unit 1122 provides additional information along with the input value 10, such as the color format to which AI downscale is applied (luminance component, chrominance component, red component, green component, or blue component, etc.). Information, HDR (high dynamic range) tone mapping information, etc. are provided to the AI downscale unit 1124, and the AI downscale unit 1124 sets the DNN considering the input value (10) and additional information. Information can also be obtained. In one embodiment, the configuration control unit 1122 transfers at least a portion of the received input value 10 to the transform-based encoding unit 1126 so that the transform-based encoding unit 1126 generates a bit rate of a specific value and a specific type of bit. The first image 115 is first encoded at a rate and a specific codec.

The AI downscale unit 1124 receives the original image 105 and performs the operation described in relation to at least one of FIGS. 1, 7, 8, 9, and 10 to obtain the first image 115. Perform.

In one embodiment, AI data 30 is provided to device 40. The AI data 30 may include at least one of resolution difference information between the original image 105 and the first image 115 and information related to the first image 115. Resolution difference information may be determined based on the target resolution difference of the input value 10, and information related to the first image 115 may be determined based on at least one of target bitrate, bitrate type, and codec type. In one embodiment, AI data 30 may include parameters used in the AI upscaling process. AI data may be provided to the device 40 from the AI downscale unit 1124.

The first image 105 is processed by the transform-based encoding unit 1126 to obtain image data 25, and the image data 25 is transmitted to the device 40. The transform-based encoder 1126 may process the first image 115 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, or AV1.

The configuration control unit 1142 controls the operation of the AI upscale unit 1144 based on the AI data 30. In one embodiment, the configuration control unit 1142 obtains DNN setting information for the AI upscale unit 1144 according to the received AI data 30, and configures the AI upscale unit 1144 with the obtained DNN setting information. Set it. In one embodiment, the configuration control unit 1142 transmits the received AI data 30 to the AI upscale unit 1144, and the AI upscale unit 1144 generates a second image based on the AI data 30. 135), DNN configuration information for AI upscaling can be obtained. In one embodiment, the configuration control unit 1142 may store additional information along with the AI data 30, such as the color format to which AI upscaling is applied (luminance component, chrominance component, red component, green component, or blue component, etc.). Information, HDR (high dynamic range) tone mapping information, etc. are provided to the AI upscale unit 1144, and the AI upscale unit 1144 obtains DNN setting information by considering the AI data 30 and additional information. It may be possible. In one embodiment, the AI upscale unit 1144 receives AI data 30 from the configuration control unit 1142 and receives at least one of prediction mode information, motion information, and quantization parameter information from the transform-based decoding unit 1146. DNN configuration information may be obtained based on AI data 30 and at least one of prediction mode information, motion information, and quantization parameter information.

The transformation-based decoder 1146 processes the image data 25 and restores the second image 135. The transformation-based decoder 1146 can process the video data 25 according to MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, or AV1.

The AI upscale unit 1144 obtains the third image 145 by AI upscaling the second image 135 provided from the transformation-based decoding unit 1146 based on the set DNN setting information.

The AI downscale unit 1124 may include a first DNN, and the AI upscale unit 1144 may include a second DNN. In one embodiment, DNN configuration information for the first DNN and the second DNN includes Trained according to the training method described in relation to FIGS. 9 and 10.

Meanwhile, the AI decoding device 200 shown in FIG. 2 receives broadcast (e.g., terrestrial broadcasting, cable broadcasting, satellite broadcasting) data or streaming content, AI decrypts the received broadcast data or streaming content, and AI The decoded video can be displayed or output externally for display. However, when receiving streaming content using a dedicated media streaming hub device (e.g., dedicated media streaming hub, e.g. firestick, Chromecast, etc.) specialized in UX (user experience) provided by the content creator, the streaming hub device The first decoding may be performed, and AI upscaling of the first decoded second image may be performed in a separate device connected to the streaming hub device.

Accordingly, there is a need for devices that can separately perform the first decoding 130 and AI upscaling 140, and a method for connecting the devices to each other and transmitting and receiving AI data required for AI upscaling.

Figure 12 is a diagram showing an AI decoding system according to an embodiment.

The AI decoding system 1000 according to an embodiment may include a decoding device 1100 and an AI upscaling device 1200.

The decoding device 1100 according to an embodiment may refer to a device that receives encoded data or an encoded signal from an external source, an external server, or an external device and decodes the encoded data or signal. The decoding device 1100 according to one embodiment may be implemented in the form of a set-top box, dongle, etc. However, it is not limited to this and can be implemented with various electronic devices capable of receiving multimedia data from external devices.

The decoding device 1100 according to an embodiment may receive AI encoded data and perform first decoding based on the AI encoded data. AI encoded data is data generated as a result of AI downscaling and first encoding of the original video, and may include video data and AI data. The decoding device 1100 may restore the second image corresponding to the first image through the first decoding of the image data. At this time, the first image may be an image in which the original image has been AI downscaled.

The first decoding according to an embodiment includes a process of entropy decoding image data to generate quantized residual data, a process of dequantizing the quantized residual data, a process of generating prediction data, and using the prediction data and residual data. A process of restoring the second image may be included. As such, the first decoding process is frequency conversion such as MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, and AV1 used in the first encoding process of the AI downscaled first image. It can be implemented through an image restoration method corresponding to one of the image compression methods using .

The decoding device 1100 according to an embodiment may transmit the restored second image and AI data included in the AI encoded data to the AI upscale device 1200. At this time, the decoding device 1100 may transmit the second image and AI data to the AI upscaling device 1200 through the input/output interface.

Additionally, the decoding device 1100 may further transmit first decoding-related information, such as mode information and quantization parameter information included in the image data, to the AI upscale device 1200 through an input/output interface.

For example, the decoding device 1100 and the AI upscale device 1200 may be connected with an HDMI cable or a DP (Display Port) cable, and the decoding device 1100 may transmit the second video and AI data through HDMI or DP. Can be transmitted to the AI upscale device 1200.

The AI upscale device 1200 according to an embodiment may use AI data received from the decoding device 1100 to AI upscale the second image. For example, the third image can be generated by AI upscaling the second image through the second DNN.

Additionally, the AI upscale device 1200 according to one embodiment may be implemented as an electronic device including a display. For example, the AI upscale device 1200 may be used in a TV, a mobile phone, a tablet PC, a digital camera, a camcorder, a laptop computer, a desktop, a compatible computer monitor, or a video projector. , can be implemented in various electronic devices such as digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), navigation, etc.

When the AI upscale device 1200 according to an embodiment includes a display, the AI upscale device 1200 may display a second image or a third image on the display.

Figure 13 is a diagram showing the configuration of a decoding device according to an embodiment.

Referring to FIG. 13, the decoding device 1100 according to an embodiment may include a receiving unit 1110, a first decoding unit 1120, and an input/output unit 1130.

The receiving unit 1110 according to one embodiment may receive AI encoded data generated as a result of AI encoding. The receiving unit 1110 may include a communication unit 1111, a parsing unit 1112, and an output unit 1113. The receiving unit 1110 receives and parses the AI encoded data generated as a result of AI encoding, separates the image data from the AI data, outputs the image data to the first decoder 1120, and outputs the AI data to the input/output unit ( 1130).

Specifically, the communication unit 1111 receives AI encoded data generated as a result of AI encoding through the network. AI encoded data generated as a result of AI encoding includes video data and AI data.

AI data according to one embodiment may be received by being included in a video file along with image data. When AI data is included in a video file, the AI data may be included in metadata in the header of the video file.

Alternatively, when AI-encoded video data is received as segments divided in preset time units, the AI data may be included in the metadata of the segments.

Alternatively, AI data may be encoded and received in a form included in a bitstream. Alternatively, AI data may be received as a separate file.

Figure 13 shows a case where AI data is received in the form of metadata.

AI encoded data can be divided into video data and AI data. For example, the parsing unit 1112 receives AI encoded data received through the communication unit 1111, parses it, and divides it into image data and AI data. For example, data received over a network may be formatted as an MP4 file format that follows the ISO Base Media File Format standard, which is widely used to store or transmit multimedia data. The MP4 file format consists of a plurality of boxes, and each box can have type information indicating what data it contains and size information indicating the size of the box. At this time, data received in the MP4 file format may be composed of a media data box in which actual media data including video data is stored and a metadata box in which metadata related to the media is stored. By parsing the box type within the received data, it is possible to distinguish whether the data is video data or AI data. In one example, the parsing unit 1112 identifies the box type of data in the MP4 file format received through the communication unit 1111, distinguishes image data and AI data, and transmits the data to the output unit 1113. 1113 transmits each classified data to the first decoding unit 1120 and the input/output unit 1130.

At this time, the image data included in the AI encoded data is image data generated through a certain codec (e.g., MPEG-2, H.264, MPEG-4, HEVC, VC-1, VP8, VP9, or AV1). It can also be confirmed that it is. In this case, the corresponding information can be transmitted to the first decoding unit 1120 through the output unit 1113 so that the video data can be processed with the confirmed codec.

AI-encoded data according to an embodiment may be obtained from a data storage medium including a hard disk, etc., and the decoding device 1100 according to an embodiment may be used to access the storage medium through an input/output interface such as a USB port. AI encoded data can be obtained from.

Additionally, the AI encoded data received from the communication unit 1111 may be stored in the memory, and the parsing unit 1112 may parse the AI encoded data obtained from the memory. However, it is not limited to this.

The first decoder 1120 restores the second image corresponding to the first image based on the image data. The second image generated by the first decoding unit 1120 is transmitted to the input/output unit. Depending on the implementation, first decoding-related information, such as mode information and quantization parameter information included in the image data, may be further transmitted to the input/output unit 1130.

The input/output unit 1130 can receive AI data from the output unit 1113.

The input/output unit 1130 may transmit data to or receive data from an external device through an input/output interface. For example, the input/output unit 1130 can transmit and receive video data, audio data, and additional data. Alternatively, a command can be requested or received from an external device, and a response message to the command can be transmitted. However, it is not limited to this.

Again, referring to FIG. 13, the input/output unit 1130 transmits the second image received from the first decoding unit 1120 and the AI data received from the output unit 1113 to the AI upscale device 1200. You can.

For example, the input/output unit 1130 may include HDMI, and the input/output unit 1130 may transmit the second image and AI data to the AI upscale device 1200 through HDMI.

Alternatively, the input/output unit 1130 may include a DP (Display Port), and the input/output unit 1130 may transmit the second image and AI data to the AI upscale device 1200 through the DP. there is.

Hereinafter, with reference to FIG. 14, the data structure of AI data included in the form of metadata will be described in detail.

Figure 14 shows the data structure of AI data included in the form of metadata according to an embodiment.

AI data according to one embodiment may be included in the metadata of the header or metadata of the segment of the video file. For example, when following the aforementioned MP4 file format, a video file or segment may be composed of a media data box containing actual media data and a metadata box containing metadata related to the media. AI data in the form of metadata, which will be described in FIG. 14, can be delivered in a metadata box.

Referring to FIG. 14, AI data according to one embodiment may include elements such as ai_codec_info (1300), ai_codec_applied_channel_info (1302), target_bitrate_info (1304), res_info (1306), ai_codec_DNN_info (1312), and ai_codec_supplementary_info (1314). You can. The arrangement order of each element shown in FIG. 14 is only an example, and a person skilled in the art may change the arrangement order of the elements.

In one embodiment, ai_codec_info 1300 indicates whether AI upscaling is applied to a low-resolution image such as the second image 135. When ai_codec_info (1300) indicates that AI upscaling is applied to the second image restored according to the image data, the data structure of the AI data includes elements for obtaining upscale DNN information used for AI upscaling.

ai_codec_applied_channel_info (1302) is channel information indicating the color channel to which AI upscaling is applied. The image can be expressed in RGB format, YUV format, YCbCr format, etc., and depending on the type of frame, it can indicate the color channel that requires AI upscaling among YCbCr color channels, RGB color channels, or YUV color channels.

target_bitrate_info (1304) is information indicating the bitrate of image data obtained as a result of the first encoding (614). The AI upscale unit 234 can obtain upscale DNN information suitable for the quality of the second video according to target_bitrate_info (1304).

res_info (1306) represents resolution information related to the resolution of an AI upscaled high-resolution image such as the third image (145). res_info (1306) may include pic_width_org_luma (1308) and pic_height_org_luma (1310). pic_width_org_luma(1308) and pic_height_org_luma(1310) are high-resolution image width information and high-resolution image height information that indicate the width and height of the high-resolution image, respectively. The AI upscale unit 234 may determine the AI upscale rate according to the resolution of the high-resolution image determined according to pic_width_org_luma (1308) and pic_height_org_luma (1310) and the resolution of the low-resolution image restored by the first decoder 232. .

ai_codec_DNN_info (1312) is information indicating mutually promised upscale DNN information used for AI upscale of the second image. The AI upscale unit 234 may determine one of a plurality of pre-stored DNN setting information as upscale DNN information according to ai_codec_applied_channel_info (1302), target_bitrate_info (1304), res_info (1306), etc. In addition, the AI upscale unit 234 additionally considers other characteristics of the video (video genre, maximum luminance, color gamut, etc.) and encoded codec information, and converts one of the plurality of pre-stored DNN setting information into upscale DNN information. You can decide.

As described above, the DNN information representing the DNN for upscaling may be expressed as an identifier indicating one of the DNN setting information pre-stored in the AI upscale unit, and may be expressed as an identifier indicating the number of convolutional layers constituting the DNN and the filter kernel for each convolutional layer. It may also include information about at least one of the number and parameters of each filter kernel.

ai_codec_supplementary_info (1314) represents additional information about AI upscale. ai_codec_supplementary_info (1314) may include information necessary for determining upscale DNN information applied to video. ai_codec_supplementary_info (1314) may include information about genre, HDR maximum illuminance, HDR color gamut, HDR PQ information, codec information, rate control type, etc.

Meanwhile, when AI data is included in the metadata of the segment, the AI data may further include dependent_ai_condition_info indicating dependent information.

dependent_ai_condition_info indicates whether the current segment inherits the AI data of the previous segment.

For example, when dependent_ai_condition_info indicates that the current segment succeeds the AI data of the previous segment, the metadata of the current segment does not include AI data corresponding to the above-described ai_codec_info (1300) to ai_codec_supplementary_info (1314). Instead, the AI data of the current segment is determined to be the same as the AI data of the previous segment.

Additionally, when dependent_ai_condition_info indicates that the current segment does not inherit the AI data of the previous segment, the metadata of the current segment includes AI data. Therefore, AI data related to the media data of the current segment can be obtained.

Meanwhile, the receiving unit 1110 and the AI decoding unit 1120 according to one embodiment are described as separate devices, but may be implemented through a single processor. In this case, it may be implemented with a separate dedicated processor, or it may be implemented through a combination of a general-purpose processor such as an AP, CPU, or GPU and S/W. Additionally, in the case of a dedicated processor, it may be implemented including a memory for implementing an embodiment of the present disclosure, or may be implemented including a memory processing unit for using an external memory.

Additionally, the receiving unit 1110 and the AI decoding unit 1120 may be composed of one or more processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of multiple general-purpose processors such as AP, CPU, and GPU and S/W.

FIG. 15 is a diagram illustrating a case where AI data is included in video data and received in the form of a bitstream, according to an embodiment.

The configuration of the communication unit 1111, output unit 1113, first decoding unit 1120, and input/output unit 1130 of FIG. 15 has been described in detail in FIG. 13, and thus the same description will be omitted.

Referring to FIG. 15, the communication unit 1111 according to an embodiment may receive a bitstream in which image data and AI data are encoded together. At this time, the AI data may be included in the bitstream in the form of a Supplemental Enhancement Information (SEI) message, which is information that can additionally improve the function of the codec used for first encoding/decoding. SEI messages can be transmitted in frame units.

If AI encoded data is received in the form of a bitstream in which video data and AI data are encoded together, the video data and AI data cannot be distinguished. Accordingly, the communication unit 1111 transmits the AI encoded data in the form of a bitstream to the output unit 1113, and the output unit 1113 transmits the AI encoded data in the form of a bitstream to the first decoder 1120.

The first decoding unit 1120 restores the second image corresponding to the first image based on the image data included in the bitstream received from the output unit 1113, and converts the second image to the input/output unit ( 1130).

Additionally, the first decoding unit 1120 separates the payload of the SEI message containing AI data from the bitstream and transmits it to the input/output unit 1130.

The input/output unit 1130 may transmit the payload (eg, AI data) of the second video and SEI message received from the first decoder 1120 to the AI upscale device 1200.

At this time, AI data may be included in the SEI message in the form of high-level syntax, as shown in FIG. 16.

Figure 16 shows an AI codec syntax table according to one embodiment.

Referring to FIG. 16, the AI codec syntax table may include an AI codec main syntax table (ai_codec_usage_main). The AI codec main syntax table includes elements related to upscale DNN information used for AI upscaling of the second image restored according to the image data. The AI codec main syntax table may include AI data applied to AI upscaling of all frames within the video file.

According to the AI codec main syntax table in Figure 16, ai_codec_info, ai_codec_applied_channel_info, target_bitrate, pic_width_org_luma, pic_height_org_luma, ai_codec_DNN_info, ai_codec_supplementary_info_flag Syntax elements such as are parsed.

ai_codec_info corresponds to ai_codec_info (1300) in FIG. 14 and indicates whether AI upscaling is allowed for the second image. If ai_codec_info indicates that AI upscale is allowed (if(ai_codec_info)), the syntax elements necessary to determine AI upscale DNN information are parsed.

ai_codec_applied_channel_info is channel information corresponding to ai_codec_applied_channel_info (1302) in FIG. 14. target_bitrate is target bitrate information corresponding to target_bitrate_info (1304) in FIG. 14. pic_width_org_luma and pic_height_org_luma are high-resolution image width information and high-resolution image height information corresponding to pic_width_org_luma (1308) and pic_height_org_luma (1310) in FIG. 14, respectively. ai_codec_DNN_info is DNN information corresponding to ai_codec_DNN_info (1312) in FIG. 14.

ai_codec_supplementary_info_flag is an additional information flag that indicates whether ai_codec_supplementary_info 1314 of FIG. 14 is included in the syntax table. If ai_codec_supplementary_info_flag indicates that auxiliary information used for AI upscaling is not parsed, additional auxiliary information is not obtained. However, if ai_codec_supplementary_info_flag indicates that auxiliary information used for AI upscaling is parsed (if(ai_codec_supplementary_info_flag)), additional auxiliary information is obtained.

Additional auxiliary information obtained may include ai_cdeco_DNNstruct_info, genre_info, hdr_max_luminance, hdr_color_gamut, hdr_pq_type, and rate_control_type. ai_codec_DNNstruct_info is information indicating the structure and parameters of new DNN setting information suitable for video, separately from the DNN setting information pre-stored in the AI upscale unit. For example, information about at least one of the number of convolutional layers, the number of filter kernels for each convolutional layer, and the parameters of each filter kernel may be included.

genre_info is the genre of the content of the video data, hdr_max_luminance is the HDR (High Dynamic Range) maximum luminance applied to high-resolution images, hdr_color_gamut is the HDR color gamut applied to high-resolution images, hdr_pq_type is HDR PQ (Perceptual Quantizer) information applied to high-resolution images, rate_control_type indicates the rate control type applied to the image data obtained as a result of the first encoding. Depending on the embodiment, a specific syntax element among the syntax elements corresponding to auxiliary information may be parsed.

Additionally, the AI codec syntax table according to one embodiment may include an AI codec frame syntax table (ai_codec_usage_frame) including AI data applied to the current frame.

Figure 17 is a block diagram showing the configuration of an AI upscale device according to an embodiment.

Referring to FIG. 17, the AI upscale device 1200 may include an input/output unit 1210, a data processing unit 1220, and an AI upscale unit 1230.

The input/output unit 1210 may receive the second image and AI data from the decoding device 1100. At this time, the input/output unit 1210 may include HDMI, DP, etc.

When the decoding device 1100 and the AI upscale device 1200 according to an embodiment are connected with an HDMI cable, the input/output unit 1210 can receive the second image and AI data through HDMI.

Alternatively, the input/output unit 1210 may receive the second image and AI data through DP when the decoding device 1100 and the AI upscale device 1200 according to an embodiment are connected with a DP cable. . However, it is not limited to this, and the second image and AI data can be received through various input/output interfaces. Additionally, the input/output unit 1210 may receive the second image and AI data through another input/output interface.

The input/output unit 1210 may transmit the second image and AI data to the AI upscale unit 1230. When AI data is transmitted, the AI upscale unit 1230 according to one embodiment may determine an upscale target of the second image based on at least one of difference information and first image related information included in the AI data. there is. For example, based on the AI data shown and described in FIGS. 14 and 16, the upscale target of the second image may be determined.

For example, the upscale target may indicate to what resolution the second image 135 should be upscaled. When the upscale target is determined, the AI upscale unit 1230 AI upscales the second image 135 through the second DNN to generate the third image 145 corresponding to the upscale target. Since the method of AI upscaling the second image 135 through the second DNN has been described in detail with reference to FIGS. 3 to 6, detailed description will be omitted.

The AI upscale unit 1230 according to one embodiment acquires new DNN setting information based on AI data, rather than the DNN setting information previously stored in the AI upscale device 1200, and acquires new DNN setting information. By setting the second DNN, the second image 135 can be AI upscaled.

Meanwhile, when only the second image is received by the input/output unit 1210 and no AI data is received, the input/output unit 1210 may transmit the second image to the AI upscale unit 1230. The AI upscale unit 1230 may generate the fourth image by AI upscaling the second image according to a preset method without using AI data. At this time, the image quality of the fourth image may be lower than that of the third image for which AI upscaling was performed using AI data.

Figure 18 is a diagram illustrating an example in which a decoding device and an AI upscale device transmit and receive data through HDMI, according to an embodiment.

The input/output unit 1130 of the decoding device 1100 and the input/output unit 1210 of the AI upscale device 1200 may be connected with an HDMI cable. When the input/output unit 1130 of the decoding device 1100 and the input/output unit 1210 of the AI upscale device 1200 are connected with an HDMI cable, four channels providing TMDS data channels and TMDS clock channels are connected. Pairing may be performed. The TMDS data channel includes three data transmission channels and can be used to carry video data, audio data and additional data. At this time, a packet structure is used to transmit audio data and additional data through the TMDS data channel.

Additionally, the input/output unit 1130 of the decoding device 1100 and the input/output unit 1210 of the AI upscale device 1200 may provide a display data channel (DDC). DDC is a protocol standard for digital information transmission between monitors and computer graphics adapters defined by VESA (Video Electronics Standard Association). DDC is used to exchange configuration and status information between one source device (e.g., decryption device) and one sink device (e.g., AI upscale device).

Referring to FIG. 18, the input/output unit 1130 of the decoding device 1100 may include an HDMI transmitter 1610, a VSIF structuring unit 1620, and an EDID acquisition unit 1630. Additionally, the input/output unit 1210 of the AI upscale device 1200 may include an HDMI receiver 1640 and an EDID storage unit 1650.

The EDID storage unit 1650 of the AI upscale device 1200 according to one embodiment may store EDID (Extended Display Identification Data) information. EDID information is a data structure containing various information about the AI upscale device, and can be transmitted to the decoding device 1100 through a DDC channel.

EDID information according to one embodiment may include information about the AI upscale capability of the AI upscale device 1200. For example, EDID information may include information about whether the AI upscale device 1200 can perform AI upscale. This will be explained in detail with reference to FIG. 19.

Figure 19 is a diagram showing HF-VSDB included in EDID information according to an embodiment.

EDID information may include an EDID extension block containing additional information. The EDID extension block may include an HDMI Forum (HF)-Vendor-Specific Data Block (VSDB) 1710. The HF-VSDB (1710) is a data block in which vendor-specific data can be defined, and the HF-VSDB (1710) can be used to define HDMI-specific data.

HF-VSDB 1710 according to one embodiment may include reserved fields 1720 and 1730. Information about the AI upscale capability of the AI upscale device 1200 may be described using at least one field among the reserved fields 1720 and 1730 of the HF-VSDB 1710. For example, if the AI upscale device can perform AI upscaling using 1 bit of the reserved field, set the bit value of the reserved field to 1 and allow the AI upscale device to perform AI upscaling. If there is none, the bit value of the reserved field can be set to 0. Or, conversely, if the AI upscale device can perform AI upscaling, set the bit value of the reserved field to 0, and if the AI upscale device cannot perform AI upscaling, set the bit value of the reserved field to 0. can also be set to 1.

Again, referring to FIG. 18, the EDID acquisition unit 1630 of the decoding device 1100 may receive EDID information of the AI upscale device 1200 through DDC. EDID information according to one embodiment may be transmitted as HF-VSDB, and the EDID acquisition unit 1630 uses the reserved field value of HF-VSDB to obtain information about the AI upscale capability of the AI upscale device. You can.

The EDID acquisition unit 1630 may determine whether to transmit AI data to the AI upscale device based on information about the AI upscale capability of the AI upscale device. For example, if the AI upscale device is capable of performing AI upscale, the EDID acquisition unit 1630 may operate in the VSIF structuring unit 1620 to structure AI data in the form of a VSIF packet. On the other hand, if the AI upscaling device cannot perform AI upscaling, the EDID acquisition unit 1630 may operate in the VSIF structuring unit 1620 so that AI data is not structured in the form of VSIF packets.

The VSIF structuring unit 1620 may structure AI data transmitted from the first decoding unit 1120 or the output unit 1113 in the form of a VSIF packet. The VSIF packet will be described with reference to FIG. 20.

Figure 20 is a diagram showing the header structure and content structure of a VSIF (Vendor Specific InfoFrame) packet according to an embodiment.

Referring to FIG. 20, a VSIF packet includes a VSIF packet header 1810 and VSIF packet content 1820. The header of a VSIF packet may consist of 3 bytes, the first byte (HB0) is a value indicating the packet type, the value of the VSIF packet is expressed as 0x81, the second byte (HB1) indicates version information, and the third byte is The lower 5 bits of the byte (HB2) indicate the content length of the VSIF packet in bytes.

The VSIF structuring unit 1620 according to one embodiment may structure AI data in the form of a VSIF packet. For example, the VSIF structuring unit 1620 may generate a VSIF packet to include AI data in the VSIF packet content 1820. The VSIF structuring unit 1620 configures the AI data shown and described in FIGS. 14 and 16 into reserved field values 1830 of packet byte 5 (PB5) and packet byte NV (PB(Nv)) included in the VSIF packet content. The VSIF packet content can be created to be described in the reserved field values 1840 of ). Alternatively, VSIF packet content can be created so that AI data is described in reserved field values of NV+k (k=1,2,...,n) packet bytes.

The VSIF structuring unit 1620 can determine packet bytes for describing AI data depending on the amount of AI data. If the amount of AI data is small, the AI data can be described using only the reserved field values 1830 of the 5th packet byte (PB5). On the other hand, if the amount of AI data is large, the AI data can be described using reserved field values of packet byte 5 (PB5) and packet byte NV (PB(Nv)). Alternatively, AI data can be described using reserved field values of the NV packet byte (PB(Nv)) and the NV+k (k=1,2,...,n) packet byte. However, it is not limited to this, and AI data can be structured in the form of a VSIF packet in various ways.

Figure 21 is a diagram illustrating an example in which AI data is defined in a VSIF packet according to an embodiment.

Referring to 1910 of FIG. 21, the VSIF structuring unit 1620 according to one embodiment can describe AI data using only the reserved field values of packet byte number 5 (PB5). The VSIF structuring unit 1620 can define ai_codec_available info using bit 1 of PB5. ai_codec_available_info indicates whether AI upscaling is allowed in the current frame. Additionally, ai_codec_DNN_info can be defined using at least one of bits 2 to 3 of PB5. ai_codec_DNN_info is DNN information indicating the upscaling DNN used to AI upscale the current frame. For example, DNN setting information for the upscale DNN, an identifier of the upscale DNN, an identifier of the value of the lookup table for the upscale DNN, etc. may be included.

Additionally, ai_codec_org_width may be defined using at least one of bits 4 to 5 of PB5, and ai_codec_org_height may be defined using at least one of bits 6 to 7 of PB5. ai_codec_org_width indicates the width of the original image 105 and the width of the third image 145. Additionally, ai_codec_org_height represents the height of the original image 105 and the height of the third image 145. ai_codec_org_height and ai_codec_org_width are used to determine the size of the upscale target.

Referring to 1920 of FIG. 21, the VSIF structuring unit 1620 according to an embodiment configures NV packet bytes (PB(Nv)) and NV+k (k=1,2,...,n) packet bytes. AI data can be described using reserved field values.

ai_codec_available_info can be defined using at least one of bits 0 to 3 of PB(Nv) (Nv packet byte).

Additionally, ai_codec_DNN_info can be defined using at least one of bits 4 to 7 of PB(Nv) (packet byte Nv).

In addition, ai_codec_org_width can be defined using at least one of bits 0 to 3 of PB(Nv+1) (packet byte Nv+1), and the ai_codec_org_height can be defined using at least one of bits 4 to 7.

Additionally, bitrate_info can be defined using at least one of bits 0 to 3 of PB(Nv+2) (packet byte Nv+2). bitrate_info is information indicating the quality of the restored second video.

Additionally, ai_codec_applied_channel_info can be defined using at least one of bits 4 to 7 of PB(Nv+2) (packet byte Nv+1). ai_codec_applied_channel_info is channel information indicating the color channel that requires AI upscaling. Depending on the type of frame, the color channel requiring AI upscaling may be indicated among YCbCr color channels, RGB color channels, or YUV color channels.

In addition, bits of the remaining packet bytes (e.g., bits included in PB(Nv+4) (packet byte Nv+4) to PB(Nv+n) (packet byte (Nv+n)) Using at least one of, ai_codec_supplementary_info may be defined. ai_codec_supplementary_info refers to auxiliary information used for AI upscaling. Auxiliary information may include the structure and parameters of new DNN setting information suitable for the current video, genre, color range, HDR maximum illuminance, HDR color gamut, HDR PQ information, codec information, RC type (Rate Control type), etc. .

Meanwhile, the structure of the VSIF packet shown in FIG. 21 is only an example and is not limited thereto. If necessary, the positions or sizes of fields in which the AI data included in the VSIF packet of FIG. 19 are defined may be changed, and the AI data described in FIGS. 14 and 16 may be further included in the VSIF packet.

Referring again to FIG. 18, the VSIF structuring unit 1620 according to one embodiment may generate a VSIF packet corresponding to each of a plurality of frames. For example, when AI data is received once for a plurality of frames, the VSIF structuring unit 1620 uses the AI data received once to generate a VSIF packet corresponding to each of the plurality of frames. can do. For example, VSIF packets corresponding to a plurality of frames may be generated based on the same AI data.

On the other hand, when AI data is received multiple times for a plurality of frames, the VSIF structuring unit 1620 may generate a new VSIF packet using the newly received AI data.

The VSIF structuring unit 1620 can transmit the generated VSIF packet to the HDMI transmitter 1610, and the HDMI transmitter 1610 can transmit the VSIF packet to the AI upscale device 1200 through a TMDS channel.

Additionally, the HDMI transmitter 1610 may transmit the second video received from the first decoder 1120 to the AI upscale device 1200 through a TMDS channel.

The HDMI receiver 1640 of the AI upscale device 1200 may receive structured AI data in the form of a second video and a VSIF packet through a TMDS channel.

The HDMI receiver 1640 of the AI upscale device 1200 according to an embodiment may check the header information of the HDMI packet, search for the VSIF packet, and then determine whether the VSIF packet includes AI data. there is.

For example, the HDMI receiver 1640 may determine whether the received HDMI packet is a VSIF packet by checking whether the first byte (HB0) indicating the packet type among the header information of the received HDMI packet is 0x81. Additionally, if the HDMI receiver 1640 determines that the HDMI packet is a VSIF packet, it can determine whether AI data is included in the VSIF packet content. For example, the HDMI receiver 1640 determines that the values of the bits included in PB(Nv) (packet byte Nv) to PB(Nv+n) (packet byte (Nv+n)) included in the VSIF packet content are If set, AI data can be obtained using bit values. For example, the data processing unit 1220 may obtain ai_codec_available_info using at least one of bits 0 to 3 of PB(Nv) (Nv packet byte) of the VSIF packet content, and PB(Nv) (Nv packet byte). ai_codec_DNN_info can be obtained using at least one of bits 4 to 7 of the packet byte).

Additionally, the HDMI receiver 1640 may obtain ai_codec_org_width using at least one of bits 0 to 3 of PB(Nv+1) (Nv+1 packet byte), and PB(Nv+1)(Nv ai_codec_org_height can be obtained using at least one of bits 4 to 7 of packet byte +1.

In addition, the HDMI receiver 1640 obtains bitrate_info using at least one of bits 0 to 3 of PB(Nv+2) (packet byte Nv+2), and receives bitrate_info from PB(Nv+2)(Nv+2 packet byte). ai_codec_applied_channel_info can be obtained using at least one of bits 4 to 7 of the first packet byte).

Additionally, the HDMI receiver 1640 receives bits of the remaining packet bytes (e.g., PB(Nv+4) (packet byte Nv+4) to PB(Nv+n)(packet byte (Nv+n)). ), ai_codec_supplementary_info can be obtained using at least one of the bits included in ).

The HDMI receiver 1640 can provide AI data obtained from VSIF packet content to the AI upscale unit 1230, and the second image can also be provided to the AI upscale unit 1230.

When the second image and AI data are transmitted from the HDMI receiver 1640, the AI upscale unit 1230 according to an embodiment provides a first image based on at least one of difference information and first image related information included in the AI data. 2 You can determine the upscale target of the video. For example, the upscale target may indicate to what resolution the second image 135 should be upscaled. When the upscale target is determined, the AI upscale unit 1230 AI upscales the second image 135 through the second DNN to generate the third image 145 corresponding to the upscale target. The method of AI upscaling the second image 135 through the second DNN has been described in detail with reference to FIGS. 3 to 6, so detailed description will be omitted.

Meanwhile, in FIGS. 18 to 21, the case where the decoding device and the AI upscale device are connected via an HDMI cable is shown and described, but the case is not limited thereto, and the decoding device and the AI upscale device according to an embodiment are DP Can be connected via cable. When the decoding device and the AI upscale device are connected through a DP cable, the decoding device can transmit the second video and AI data to the AI upscale device through DP, similar to the HDMI method.

Additionally, the decoding device according to one embodiment may transmit the second video and AI data to the AI upscale device through other input/output interfaces in addition to HDMI and DP.

Additionally, according to one embodiment, the decoding device 1100 may transmit the second image and AI data to the AI upscale device 1200 through different interfaces. For example, the second video can be transmitted through HDMI, and the AI data can be transmitted through DP. Alternatively, the second video can be transmitted through DP and the AI data can be transmitted through HDMI.

Figure 22 is a flowchart showing a method of operating a decoding device according to an embodiment.

Referring to FIG. 22, the decoding device 1100 according to an embodiment may receive AI encoded data (S2010).

For example, the decoding device 1100 receives AI encoded data generated as a result of AI encoding through a network. AI encoded data is data generated as a result of AI downscaling and first encoding of the original video and includes video data and AI data.

At this time, AI data according to one embodiment may be received by being included in a video file along with image data. When AI data is included in a video file, the AI data may be included in metadata in the header of the video file. Alternatively, when AI-encoded video data is received as segments divided in preset time units, the AI data may be included in the metadata of the segments. Alternatively, AI data may be encoded and received as included in a bitstream, or may be received as a separate file from the video data. However, it is not limited to this.

The decoding device 1100 can separate AI encoded data into image data and AI data (S2020).

According to one embodiment, when AI data is received in the form of header metadata or segment metadata of a video file, the decoding device 1100 can parse the AI encoded data and distinguish it into video data and AI data. . For example, the decoding device 1100 can read the box type of data received through the network and distinguish whether the data is video data or AI data.

When AI data according to one embodiment is received in the form of a bitstream, the decoding device 1100 may receive a bitstream in which image data and AI data are encoded together. At this time, AI data may be inserted in the form of a SEI (Supplemental Enhancement Information) message. The decoding device 1100 can distinguish the payload of an SEI message including video data and AI data from the bitstream.

The decoding device 1100 according to an embodiment may decode the second image based on the image data (S2030).

The decoding device 1100 according to one embodiment may transmit the second image and AI data to an external device through an input/output interface (S2040).

An external device according to an embodiment includes an AI upscale device 1200.

For example, the decoding device 1100 may transmit the second image and AI data to an external device through HDMI, DP, etc. When transmitting AI data through HDMI, the decoding device 1100 can transmit the AI data in the form of a VSIF (Vendor Specific InfoFrame) packet.

Additionally, the transmitted AI data includes information that allows AI upscaling of the second image. For example, AI data may include information indicating whether AI upscale is applied to the second image, information about a DNN for upscaling the second image, etc.

Figure 23 is a flowchart showing a method by which a decoding device transmits a second video and AI data through HDMI according to an embodiment.

Referring to FIG. 23, the decoding device 1100 and the AI upscale device 1200 according to an embodiment may be connected with an HDMI cable (S2110).

The decoding device 1100 may transmit an EDID information request to the AI upscale device 1200 through DDC (S2120). In response to the EDID information request from the decoding device 1100, the AI upscale device 1200 may transmit the EDID information stored in the EDID storage unit to the decoding device 1100 through DDC (S2130). At this time, the EDID information may include HF (HDMI Forum)-VSDB (Vendor-Specific Data Block), and HF-VSDB may include information about the AI upscale capability of the AI upscale device.

The decoding device 1100 may receive the received EDID information (eg, HF-VSDB) and obtain information about the AI upscale capability of the AI upscale device.

The decoding device 1100 may determine whether to transmit AI data to the AI upscale device based on information about the AI upscale capability of the AI upscale device (S2150). For example, if the AI upscale device cannot perform AI upscale, the decoding device 1100 does not structure the AI data in the form of a VSIF packet, but only the second image is sent to the AI upscale device through the TMDS channel. Can be transmitted (S2160).

If the AI upscaling device 1200 can perform AI upscaling, the decoding device 1100 can operate to structure AI data in the form of a VSIF packet (S2170).

The decoding device 1100 can define AI data using reserved field values included in the VSIF packet. Since the method of defining AI data in a VSIF packet has been described in detail with reference to FIGS. 20 and 21, detailed description will be omitted.

The decoding device 1100 may transmit AI data and the second image structured as a VSIF packet to the AI upscale device 1200 through the TMDS channel (S2180).

Figure 24 is a flowchart showing a method of operating an AI upscale device according to an embodiment.

Referring to FIG. 24, the AI upscale device 1200 according to an embodiment may receive the second image and AI data through the input/output interface (S2210).

For example, when the AI upscale device 1200 is connected to the decoding device 1100 with an HDMI cable, it can receive the second video and AI data through HDMI. Alternatively, when connected to the decoding device 1100 with a DP cable, the second image and AI data can be received through DP. However, it is not limited to this, and the second image and AI data can be received through various input/output interfaces. Additionally, the AI upscale device 1200 may receive the second image and AI data through another input/output interface.

The AI upscaling device 1200 may determine whether to perform AI upscaling on the second image based on whether AI data is received through the input/output interface. If AI data is not received, the second image may be output without performing AI upscaling on the second image.

The AI upscale device 1200 according to an embodiment may receive an HDMI packet from the decoding device 1100, check header information of the HDMI packet, and search for a VSIF packet. When the VSIF packet is searched, the AI upscale device 1200 may determine whether the VSIF packet includes AI data.

The AI data may include information indicating whether AI upscale is applied to the second image, information about a DNN for upscaling the second image, etc.

The AI upscale device 1200 may determine whether to perform AI upscaling on the second image based on information indicating whether AI upscaling is applied to the second image.

In addition, the AI upscale device 1200 acquires information about the DNN for upscaling the second image based on the AI data (S2220), and uses the second DNN determined according to the obtained information to create the second image. By AI upscaling the image, a third image can be generated (S2230).

Figure 25 is a block diagram showing the configuration of a decoding device according to an embodiment.

The decoding device 2300 of FIG. 25 is an embodiment of the decoding device 1100 of FIG. 12.

Referring to FIG. 25, the decoding device 2300 according to an embodiment may include a communication unit 2310, a processor 2320, a memory 2330, and an input/output unit 2040.

The communication unit 2310 of FIG. 25 may correspond to the communication unit 1111 of FIGS. 13 and 15, and the input/output unit 2340 of FIG. 25 may correspond to the input/output unit 1130 of FIGS. 13 and 15, respectively. . Therefore, the same contents as those described in FIGS. 13 and 15 will be omitted from FIG. 25.

The communication unit 2310 according to one embodiment may transmit and receive data or signals with an external device (eg, a server) under the control of the processor 2320. The processor 2320 can transmit/receive content to an external device connected through the communication unit 2310. The communication unit 2310 provides wireless LAN (2311 (e.g., Wi-Fi)), Bluetooth (2312), and wired Ethernet (2313) in response to the performance and structure of the decoding device (2300). ) may include one of the following. Additionally, the communication unit 2310 may include a combination of wireless LAN 2311, Bluetooth 2312, and wired Ethernet 2313.

The communication unit 2310 according to one embodiment receives AI encoded data generated as a result of AI encoding. AI encoded data is data generated as a result of AI downscaling and first encoding of the original video and includes video data and AI data.

The processor 2320 according to one embodiment may generally control the decoding device 2300. The processor 2320 according to one embodiment may execute one or more programs stored in the memory 2330.

The memory 2330 according to one embodiment may store various data, programs, or applications for driving and controlling the decoding device 2300. Additionally, the memory 2330 may store the received AI encoded data or the first decoded second image, according to one embodiment. A program stored in memory 2330 may include one or more instructions. A program (one or more instructions) or application stored in the memory 2330 may be executed by the processor 2320.

The processor 2320 according to one embodiment may include a Central Processing Unit (CPU) 2321, a Graphic Processing Unit (GPU) 2323, and a Video Processing Unit (VPU) 2322. Additionally, depending on the embodiment, the CPU 2321 may include a GPU 2323 or a VPU 2322. Alternatively, the CPU 2321 may be implemented in the form of a System On Chip (SoC) that integrates at least one of the GPU 2323 and the VPU 2322. Alternatively, the GPU 2323 and the VPU 2322 may be implemented in an integrated form.

The processor 2320 controls the overall operation of the decoding device 2300 and signal flow between internal components of the decoding device 2300, and performs the function of processing data. The processor 2320 can control the communication unit 2310 and the input/output unit 2330. The GPU 2323 can perform graphics processing and create a screen containing various objects such as icons, images, and text. The VPU 2322 may perform processing on the image data or video data received by the decoding device 2300, including decoding (e.g., first decoding), scaling, noise filtering, Various image processing such as frame rate conversion, resolution conversion, etc. can be performed.

The processor 2320 according to one embodiment includes the operations of the parsing unit 1112, the output unit 1113, and the first decoding unit 1120 shown and described in FIG. 13, and the parsing unit 1112 shown and described in FIG. 15. , at least one of the operations of the output unit 1113 and the first decoding unit 1120 can be performed or controlled to be performed. Alternatively, at least one of the operations of the VSIF structuring unit 1620 and the EDID parsing unit 1630 shown and described in FIG. 18 may be performed or controlled to be performed.

For example, the processor 2320 may divide the AI-encoded data received from the communication unit 2310 into image data and AI data. When AI data is received in the form of header metadata or segment metadata of a video file, the processor 2320 can parse the AI encoded data and distinguish it into video data and AI data. For example, when AI-encoded data according to an embodiment is configured in an MP4 file format, the processor 2320 parses the box type of the data configured in the received MP4 file format to determine whether the data is video data or AI data. can be distinguished.

Additionally, when AI data is received in the form of a bitstream, the AI data may be included in the bitstream in the form of an SEI (Supplemental Enhancement Information) message, and the processor 2320 may generate the SEI data including the image data and AI data from the bitstream. The payload of the message can be distinguished.

The processor 2320 may control the GPU 2323 or VPU 2322 to restore a second image corresponding to the first image based on image data.

The processor 2320 may transmit the second image and AI data to an external device through the input/output unit 2340. The input/output unit 2040 may transmit or receive video, audio, and additional information to the outside of the decoding device 2300 under the control of the processor 2320. The input/output unit 2340 may include a High-Definition Multimedia Interface port (HDMI) port 2341, a Display Port (DP) 2342, and a USB port 2343. . It will be easily understood by those skilled in the art that the configuration and operation of the input/output unit 2340 can be implemented in various ways according to embodiments of the present invention.

For example, when the decoding device and the AI upscale device are connected through HDMI 2341, the input/output unit 2320 can receive EDID information of the AI upscale device through the DDC channel. Additionally, the processor 2320 can parse the EDID information of the AI upscale device received through the DDC channel. EDID information may include information about the AI upscale capability of the AI upscale device.

The processor 2320 may determine whether to structure AI data in the form of a VSIF packet, based on information about the AI upscale capability of the AI upscale device. For example, the processor 2320 may control AI data to be structured in the form of a VSIF packet if the AI upscale device is capable of performing AI upscaling. If there is none, it can be controlled not to perform the operation of structuring AI data in the form of a VSIF packet.

The input/output unit 2340 can structure AI data in the form of a VSIF packet under the control of the processor 2320, and sends the AI data and the second image structured as a VSIF packet to an AI upscale device through a TMDS channel. Can be transmitted.

AI data according to one embodiment includes information that allows AI upscaling of the second image. For example, AI data may include information indicating whether AI upscale is applied to the second image, information about a DNN for upscaling the second image, etc.

Figure 26 is a block diagram showing the configuration of an AI upscale device according to an embodiment.

The AI up-scale device 2400 of FIG. 26 is an embodiment of the AI up-scale device 1200 of FIG. 12.

Referring to FIG. 26, the AI upscale device 2400 according to one embodiment may include an input/output unit 2410, a processor 2420, a memory 2430, and a display 2440.

The input/output unit 2410 of FIG. 26 may correspond to the input/output unit 1210 of FIG. 17 . Therefore, the same content as that described in FIG. 17 will be omitted from FIG. 26.

The input/output unit 2410 according to one embodiment may receive or transmit video, audio, and additional information from the outside of the AI upscale device 2400 under the control of the processor 2420. The input/output unit 2410 may include a High-Definition Multimedia Interface port (HDMI port) 2411, a Display Port (DP) 2412, and a USB port 2413. . It will be easily understood by those skilled in the art that the configuration and operation of the input/output unit 2410 can be implemented in various ways according to embodiments of the present invention.

For example, when the decoding device and the AI upscale device are connected through HDMI 2411, the input/output unit 2410 receives the EDID information read request through the DDC channel, and receives the EDID information of the AI upscale device. It can be transmitted to a decryption device. Additionally, the input/output unit 2410 may receive structured AI data in the form of a second image and a VSIF packet through a TMDS channel.

Alternatively, the input/output unit 2410 may receive the second image and AI data through DP. However, it is not limited to this, and the second image and AI data can be received through various input/output interfaces. Alternatively, the input/output unit 2410 may receive the second image and AI data through another input/output interface.

The processor 2420 according to one embodiment can generally control the AI upscale device 2400. The processor 2420 according to one embodiment may execute one or more programs stored in the memory 2430.

The memory 2430 according to one embodiment may store various data, programs, or applications for driving and controlling the AI upscale device 2400. For example, the memory 2430 may store EDID information of an AI upscale device. EDID information may include various information about the AI upscale device, and in particular, may include information about the AI upscale capability of the AI upscale device. A program stored in memory 2430 may include one or more instructions. A program (one or more instructions) or application stored in the memory 2430 may be executed by the processor 2420.

The processor 2420 according to one embodiment may include a Central Processing Unit (CPU) 2421, a Graphic Processing Unit (GPU) 2423, and a Video Processing Unit (VPU) 2422. Additionally, depending on the embodiment, the CPU 2421 may include a GPU 2423 or a VPU 2422. Alternatively, the CPU 2421 may be implemented in the form of a System On Chip (SoC) that integrates at least one of the GPU 2423 and the VPU 2422. Alternatively, the GPU 2423 and the VPU 2422 may be implemented in an integrated form. Alternatively, the processor 2420 may further include a Neural Processing Unit (NPU).

The processor 2420 controls the overall operation of the AI upscale device 2400 and signal flow between internal components of the AI upscale device 2400, and performs the function of processing data. The processor 2420 can control the input/output unit 2410 and the display 2440. The GPU 2423 can perform graphics processing and create a screen including various objects such as icons, images, and text. The VPU 2422 may perform processing on image data or video data received by the AI upscale device 2400, and may perform decoding (e.g., first decoding), scaling, and noise on the image data or video data. Various image processing such as filtering, frame rate conversion, resolution conversion, etc. can be performed. The processor 2420 according to one embodiment may perform or control to perform at least one of the operations of the AI upscale unit 1230 shown and described in FIG. 17.

For example, the processor 2420 may perform AI upscaling for the second image based on whether AI data is received from the input/output unit 2410.

The processor 2420 may check the header information of the HDMI packet received from the input/output unit 2410 and search for the VSIF packet. When the VSIF packet is retrieved, the processor 2420 may determine whether the VSIF packet includes AI data. The AI data may include information indicating whether AI upscale is applied to the second image, information about a DNN for upscaling the second image, etc.

Additionally, the processor 2420 may determine whether to perform AI upscaling on the second image based on information indicating whether AI upscaling is applied to the second image.

The processor 2420 acquires information about a DNN for upscaling the second image based on AI data, and AI upscales the second image using the second DNN determined according to the obtained information, A third image can be generated. The processor 2420 may control the NPU to AI upscale the second image using the determined second DNN.

If AI data is not received from the input/output unit 2410, the processor 2420 may generate a fourth image by AI upscaling the second image according to a preset method without using the AI data. . At this time, the image quality of the fourth image may be lower than that of the third image for which AI upscaling was performed using AI data.

The display 2440 generates a driving signal by converting image signals, data signals, OSD signals, and control signals processed by the processor 2420. The display 2440 may be implemented as a PDP, LCD, OLED, flexible display, etc., and may also be implemented as a 3D display. Additionally, the display 2440 can be configured as a touch screen and used as an input device in addition to an output device. The display 2440 may display a third image or a fourth image.

Meanwhile, the block diagram of the decoding device 2300 and the AI upscale device 2400 shown in FIGS. 25 and 26 are block diagrams for one embodiment. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the decoding device 2300 and AI upscale device 2400 that are actually implemented. That is, as needed, two or more components may be combined into one component, or one component may be subdivided into two or more components. In addition, the functions performed by each block are for explaining the embodiments, and the specific operations or devices do not limit the scope of the present invention.

Meanwhile, the above-described embodiments of the present disclosure can be written as a program that can be executed on a computer, and the written program can be stored in a medium.

The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

Meanwhile, the model related to the above-described DNN can be implemented as a software module. When implemented as a software module (e.g., a program module including instructions), the DNN model may be stored in a computer-readable recording medium.

Additionally, the DNN model may be integrated in the form of a hardware chip and become part of the AI upscale device 2100 described above. For example, a DNN model may be built on a dedicated hardware chip for artificial intelligence, or as part of an existing general-purpose processor (e.g., CPU or application processor) or dedicated graphics processor (e.g., GPU). It could be.

Additionally, DNN models may be provided in downloadable software form. A computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically through a manufacturer or electronic marketplace. For electronic distribution, at least a portion of the software program may be stored on a storage medium or created temporarily. In this case, the storage medium may be a storage medium of a manufacturer or an electronic market server, or a relay server.

Above, the technical idea of the present disclosure has been described in detail with preferred embodiments, but the technical idea of the present disclosure is not limited to the above embodiments, and those with ordinary knowledge in the field within the scope of the technical idea of the present disclosure Various modifications and changes are possible depending on the user.

Claims (16)

In the decryption device,
A communication unit that receives AI encoded data including video data and AI data;
Obtaining the image data corresponding to an encoding result for the first image from the AI encoding data,
From the AI encoded data, the AI data related to AI downscaling from the original image to the first image is obtained, and the AI data includes first neural network (NN) setting information used for AI downscaling. Contains an index indicating
one or more processors that decode the acquired image data to obtain a second image; and
Includes an input/output unit,
The one or more processors:
Controlling the input/output unit to receive additional information from an external device,
Based on the additional information, determine whether AI upscaling is possible by the external device,
If the AI upscaling is not possible, control the input/output unit to transmit the second image to the external device,
If the AI upscaling is possible, control the input/output unit to transmit the second image and the AI data to the external device,
The index included in the AI data is used to select one second NN setting information among a plurality of second NN setting information for AI upscaling,
The first image is obtained through a downscale NN consisting of the first NN setting information among a plurality of first NN setting information for AI downscaling,
The first NN setting information is selected from the plurality of first NN setting information based on at least one of compression rate, compression quality, compression history information, resolution of the original image, and type of the original image,
The AI upscaling is performed through an upscale NN composed of the selected second NN setting information,
The decoding device wherein the plurality of first NN configuration information and the plurality of second NN configuration information are obtained through joint training of the downscale NN and the upscale NN. According to paragraph 1,
The input/output unit includes HDMI (High Definition Multimedia Interface),
The processor,
A decoding device that transmits the second video and the AI data to the external device through the HDMI. According to paragraph 2,
The processor,
A decoding device that transmits the AI data in the form of a VSIF (Vendor Specific InfoFrame) packet. According to paragraph 1,
The input/output unit includes a DP (Display Port),
The processor is
A decoding device that transmits the second image and the AI data to the external device through the DP. According to paragraph 1,
The AI data includes first information indicating that the second image has been downscaled by the AI. In the method of operating the decoding device,
Receiving AI encoded data including image data and AI data;
Obtaining the image data corresponding to an encoding result for a first image from the AI encoded data;
Obtaining AI data related to AI downscaling from the original image to the first image from the AI encoded data, wherein the AI data includes first neural network (NN) setting information used for the AI downscaling. Contains an index representing ;
Decoding the acquired image data to obtain a second image;
Receiving additional information from an external device;
Based on the additional information, determining whether AI upscaling is possible by the external device; and
If the AI upscaling is not possible, transmit the second image to the external device,
If the AI upscaling is possible, transmitting the second image and the AI data to the external device,
The index included in the AI data is used to select one second NN setting information among a plurality of second NN setting information for AI upscaling,
The first image is obtained through a downscale NN consisting of the first NN setting information among a plurality of first NN setting information for AI downscaling,
The first NN setting information is selected from the plurality of first NN setting information based on at least one of compression rate, compression quality, compression history information, resolution of the original image, and type of the original image,
The AI upscaling is performed through an upscale NN composed of the selected second NN setting information,
A method of operating a decoding device, wherein the plurality of first NN setting information and the plurality of second NN setting information are obtained through linked training of the downscale NN and the upscale NN. According to clause 6,
The step of transmitting the second image and the AI data to the external device is,
A method of operating a decoding device, comprising transmitting the second video and the AI data to the external device through HDMI. In clause 7,
The step of transmitting the second image and the AI data to the external device is,
A method of operating a decoding device, including transmitting the AI data in the form of a VSIF (Vendor Specific InfoFrame) packet. According to clause 6,
The step of transmitting the second image and the AI data to the external device is,
A method of operating a decoding device, comprising transmitting the second image and the AI data to the external device through a DP (Display Port). According to clause 6,
The AI data includes first information indicating that the second image has been downscaled by the AI. In the AI upscale device,
display;
Additional information indicating whether AI upscaling is possible by the AI upscaling device is transmitted to the decoding device, and a second image corresponding to the first image obtained by AI downscaling the original image from the decoding device, and the AI downscaling are transmitted to the decoding device. An input/output unit for receiving AI data related to, the AI data including an index indicating first neural network (NN) setting information used for AI downscaling;
a memory storing one or more instructions; and
Comprising one or more processors executing the one or more instructions stored in the memory,
The one or more processors:
Based on the index included in the AI data, one second NN setting information among a plurality of second NN setting information for AI upscaling is selected, and an upscale NN composed of the selected second NN setting information is selected. Obtain a third image by performing AI upscaling on the received second image,
Providing the acquired third image to the display,
The first image is obtained through a downscale NN composed of the first NN setting information among a plurality of first NN setting information for AI downscaling,
The first NN setting information is selected from the plurality of first NN setting information based on at least one of compression rate, compression quality, compression history information, resolution of the original image, and type of the original image,
The plurality of first NN setting information and the plurality of second NN setting information are obtained through linked training of the downscale NN and the upscale NN. According to clause 11,
The input/output unit includes HDMI (High Definition Multimedia Interface),
The input/output unit is an AI upscale device that receives the AI data and the second image through the HDMI. According to clause 12,
The input/output unit,
An AI upscale device that receives the AI data in the form of a VSIF (Vendor Specific InfoFrame) packet. According to clause 11,
The input/output unit includes a DP (Display Port),
The input/output unit,
An AI upscale device that receives the AI data and the second image through the DP. According to clause 11,
The AI data includes first information indicating that the second image has been downscaled by the AI. According to clause 11,
The downscale NN and the upscale NN are jointly trained based on sharing loss information.

Images