KR102465841B1 - System for changing SDR content to HDR10+ content using deep learning - Google Patents

Abstract

The present invention relates to a system for changing SDR content to HDR10+ content using deep learning that converts SDR video content into HDR10+ video content using an artificial neural network learned through comparison between SDR original content generated for the same video content, HDR10+ original content, and content obtained by converting HDR10+ original content to SDR.

Description

System for changing SDR content to HDR10+ content using deep learning}

The present invention relates to a system for changing HDR10+ content of SDR content using deep learning, and more specifically, SDR original content generated for the same video content, HDR10+ original content, and HDR10+ original content converted to SDR. It relates to an HDR10+ content change system of SDR content using deep learning that converts SDR image content into HDR10+ image content using a learned artificial neural network.

Until just a few years ago, all video was encoded according to the so-called low dynamic range (LDR) philosophy, also called standard dynamic range (SDR). This means that no matter what scene is being captured, by standardized definition the maximum value of the code (typically 8-bit luma Y'=255; or 100% voltage to drive an analog display) is 100 nits by the standard convention. .

Recently, with the evolution of display apparatuses, high-quality devices with wider and richer contrast (HDR) and color expression (10-bit) are emerging. As part of an effort to effectively utilize these new display technologies, new technology standards for HDR (High Dynamic Range) image transmission are emerging, among which the basic standard was announced on August 27, 2015. HDR10 standard.

HDR10+ is a standard developed based on Samsung and Amazon video, and a standard that supplements the shortcomings of the existing HDR10. Recently, most broadcasters mainly produce and provide UHD SDR content, while VOD operating platform producers focus on HDR10+ content. There is a trend to expand the services produced and provided by

However, conventionally, there is a limit in that it is difficult for small and medium-sized companies to produce HDR10+ content due to expensive equipment and license cost problems.

On the other hand, the above-mentioned background art is technical information that the inventor possessed for the derivation of the present invention or acquired in the process of derivation of the present invention, and it cannot be said that it is necessarily a known technique disclosed to the general public before the filing of the present invention. .

Korean Patent Registration No. 10-1695348

One aspect of the present invention is to convert SDR video content into HDR10+ video content using an artificial neural network learned through comparison between SDR original content, HDR10+ original content, and HDR10+ original content generated for the same video content. HDR10+ content change system for SDR content using deep learning.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

HDR10+ content change system of SDR content using deep learning according to an embodiment of the present invention learns through comparison between SDR original content, HDR10+ original content, and HDR10+ original content generated for the same video content, and content converted to SDR Converts SDR video content to HDR10+ video content using an artificial neural network.

HDR10+ content change system of SDR content using the deep learning,

a first deep learning engine unit for generating a first deep learning algorithm for deriving a difference between the SDR original content for the video content for training and the content obtained by converting the HDR10+ original content into SDR;

a second deep learning engine unit for generating a second deep learning algorithm for deriving a difference between the SDR original content and the HDR10+ original content for the video content for training;

a third deep learning engine unit for generating a third deep learning algorithm for deriving a difference between the HDR10+ original content for the video content for training and the content obtained by converting the HDR10+ original content into SDR; and

and an integrated modeling unit for generating transformation modeling for converting SDR image content into HDR10+ image content based on the first deep learning algorithm, the second deep learning algorithm, and the third deep learning algorithm.

The first deep learning engine unit, the second deep learning engine unit, and the third deep learning engine unit derive a difference by comparing the same n-th frame of two image contents to be compared with each other,

The pixel value is characterized in that any one of a brightness value, a contrast value, a color depth value, and a color gamut value,

HDR10+ content change system of SDR content using the deep learning,

A fourth deep learning engine unit for deriving a difference between the HDR10+ image content converted through the integrated modeling unit and the HDR10+ original content for the learning image content, and correcting the transformation modeling based on the derived difference; further includes.

According to one aspect of the present invention described above, it is possible to convert the previously generated SDR image content to the HDR10+ standard using a software conversion tool using machine learning, so it is possible to generate high-quality content at a low cost without expensive equipment. can

1 and 2 are block diagrams showing a schematic configuration of an HDR10+ content change system for SDR content using deep learning according to an embodiment of the present invention.
3 and 4 are diagrams for explaining specific functions of the first deep learning engine unit shown in FIG. 2 .
FIG. 5 is a view for explaining a specific function of the second deep learning engine unit shown in FIG. 2 .
FIG. 6 is a view for explaining a specific function of the third deep learning engine unit shown in FIG. 2 .
7 is a view for explaining a specific function of the fourth deep learning engine unit shown in FIG. 2 .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0016] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description set forth below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scope equivalents to those claimed. Like reference numerals in the drawings refer to the same or similar functions throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 and 2 are conceptual diagrams illustrating a schematic configuration of an HDR10+ content change system for SDR content using deep learning according to an embodiment of the present invention.

The HDR10+ content change system of SDR content using deep learning according to the present invention uses an artificial neural network learned through comparison between SDR original content, HDR10+ original content, and HDR10+ original content created for the same video content. It aims to convert SDR video content to HDR10+ video content.

Specifically, the HDR10+ content change system of SDR content using deep learning according to an embodiment of the present invention is a first deep learning engine unit 100, a second deep learning engine unit 200, a third deep learning engine unit ( 300 ), an integrated modeling unit 400 , and a fourth deep learning engine unit 500 .

Here, the first deep learning engine unit 100 , the second deep learning engine unit 200 , and the third deep learning engine unit 300 use image content for learning input for learning of the artificial neural network to perform first deep learning An algorithm, a second deep learning algorithm, and a third deep learning algorithm are generated, respectively.

When selecting the reference content, the video content for learning may be composed of a data set that has the same category (movie, drama, sports, game, animation, music video, etc.) and is divided into SDR and HDR10+.

In addition, the HDR10+ content change system for SDR content according to the present invention generates a data set that captures a large amount of HDR10+ produced game, animation, and movie content, and generates an HDR10+ data set as an SDR data set using a loss function. Hereinafter, the SDR data set generated in this way will be defined and described as image content converted from HDR10+ image content to SDR.

For example, if the video content for learning is a drama, SDR video content (first learning video content) that is an image according to the SDR standard, HDR10+ video content (second learning video content) that is an image according to the HDR10+ standard Create and use a loss function to convert HDR10+ video content (video content for second learning) to SDR standard, which is an image converted from HDR10+ video content to SDR (video content for third training), as data for the corresponding drama video It can be set to three.

The integrated modeling unit 400 generates transformation modeling for converting SDR image content into HDR10+ image content based on the first deep learning algorithm, the second deep learning algorithm, and the third deep learning algorithm.

The fourth deep learning engine unit 500 may improve the reliability of transformation modeling by learning (correcting) transformation modeling using the HDR10+ image content converted by the integrated modeling unit 400 .

Hereinafter, each component constituting the HDR10+ content change system of SDR content using deep learning according to the present invention will be described in detail.

First, as shown in FIG. 3 , the first deep learning engine unit 100 generates a first deep learning algorithm for deriving a difference between the SDR original content for the video content for training and the content obtained by converting the HDR10+ original content into SDR. do.

The first deep learning engine unit 100 sets any one of brightness, contrast, color depth, and color gumut as a pixel value in units of pixels of the image, and centers the pixel at the same position of each content. It is set to a window of a predetermined size (eg 3x3) and analyzed using the values of the peripheral pixels, and then used to apply the values when changing HDR10+ content.

For example, if the central pixel value of the SDR content is 50 and the peripheral values are as follows as shown in FIG. can be converted together.

The learning model can be used for all pixel values (brightness, contrast, color depth, color gamut, etc.) to be compared do.

The first deep learning engine unit 100 compares the n-th frame of the original SDR content with the n-th frame of the content converted from the HDR10+ original content into SDR by using an artificial neural network to extract a difference.

Referring to FIG. 5 , the second deep learning engine unit 200 generates a second deep learning algorithm for deriving a difference between the SDR original content and the HDR10+ original content for the video content for training.

The second deep learning engine unit 200 sets any one of brightness, contrast, color depth, and color gumut as a pixel value in units of pixels of the image, and centers the pixel at the same position of each content. Set to a window of a predetermined size (eg 3x3) and use the value of the peripheral pixel to apply the value when the content is changed after analysis.

The second deep learning engine unit 200 compares the nth frame of the original SDR content and the nth frame of the HDR10+ original content using an artificial neural network to extract a difference.

Referring to FIG. 6 , the third deep learning engine unit 300 generates a third deep learning algorithm for deriving a difference between the HDR10+ original content for the video content for training and the content obtained by converting the HDR10+ original content into SDR.

The third deep learning engine unit 300 sets any one of brightness, contrast, color depth, and color gumut as a pixel value in units of pixels of the image, and centers the pixel at the same position of each content. Set to a window of a predetermined size (eg 3x3) and use the value of the peripheral pixel to apply the value when the content is changed after analysis.

The third deep learning engine unit 300 compares the nth frame of the HDR10+ original content with the nth frame of the HDR10+ original content converted to SDR by using an artificial neural network to extract a difference.

The integrated modeling unit 400 generates transformation modeling for converting SDR image content into HDR10+ image content based on the first deep learning algorithm, the second deep learning algorithm, and the third deep learning algorithm.

Through the conversion modeling built by the integrated modeling unit 400, the user can easily convert SDR video content into HDR10+ video content, and can generate HDR10+ video content inexpensively without using expensive equipment.

Referring to FIG. 7 , the fourth deep learning engine unit 500 converts the SDR original content through the integrated modeling unit 400 to learn (correct) the transformation modeling built by the integrated modeling unit 400 . A fourth deep learning algorithm is created that derives the difference between the HDR10+ video content and the HDR10+ original content collected when building the training data set.

The fourth deep learning engine unit 500 converts the nth frame of the HDR10+ image content in which the SDR original content is converted through the integrated modeling unit 400 and the nth frame of the HDR10+ original content collected when building the training data set. Compare and extract differences.

In one embodiment, the fourth deep learning engine unit 500 includes the nth frame of the HDR10+ image content converted through the integrated modeling unit 400 and the nth frame of the HDR10+ original content collected when building the training data set. A difference value is calculated by comparing pixel values of the same pixel, and a pixel is extracted with a calculated difference value equal to or greater than a preset reference value.

The fourth deep learning engine unit 500 searches for pixel coordinates of the extracted pixels and sets the extracted pixels as feature points when the calculated difference value is greater than or equal to a preset reference number or more. do.

The fourth deep learning engine unit 500 generates a feature figure connecting a plurality of set feature points, and extracts feature information indicating the characteristics of the generated feature figure. That is, the fourth deep learning engine unit 500 extracts information about the position and number of vertices of the feature figure, the length of each side, the angle centered on the vertex, the total area, etc. as feature information.

The fourth deep learning engine unit 500 extracts pixels set as feature points in the above-described step from the nth frame of the HDR10+ image content converted through the integrated modeling unit 400, and sets the extracted pixels as comparison points do. The fourth deep learning engine unit 500 generates a comparison figure connecting a plurality of set comparison points, and extracts comparison information indicating characteristics of the generated comparison figure. The fourth deep learning engine unit 500 corrects the transformation model based on the extracted feature information and comparison information.

On the other hand, the artificial neural network used in the first deep learning engine unit 100 , the second deep learning engine unit 200 , the third deep learning engine unit 300 , and the fourth deep learning engine unit 500 is shown in FIG. 8 . It may be in the form of a deep neural network composed of an input layer, a hidden layer, and an output layer as shown in, but is not limited thereto.

In addition, in the above description, extracting and deriving a difference means a difference between pixel values constituting the same nth frame of two image contents to be compared, and as described above, pixel values are brightness value, contrast value, and color depth. It is characterized in that it is at least one of a value and a color gamut value.

That is, by accumulating data on differences, information on similarities and differences between frames can be utilized for analysis through the accumulated data.

In some embodiments, the HDR10+ content change system of SDR content using deep learning according to the present invention is a method of converting SDR to HDR when designing a deep learning algorithm. Inverse tone mapping and a method of converting HDR to SDR Tone mapping can be used.

In an embodiment, the tone mapping method extracts a luminance signal from a data signal corresponding to an image frame to be displayed on a display panel, and based on the luminance signal extracted from the data signal, a full grayscale luminance average and a low grayscale luminance average of the image frame and calculating the high grayscale luminance average, and calculating the tone mapping function based on the full grayscale luminance average, the low grayscale luminance average, and the high grayscale luminance average of the image frame, thereby deriving a tone mapping curve (GTM).

When the data signal corresponding to the image frame is an RGB signal, the image-adaptive tone mapping method of FIG. 1 may convert the RGB signal into a YCbCr signal and extract a luminance signal, that is, a Y signal from the YCbCr signal.

Thereafter, the average of all grayscale luminances of the image frame may be calculated as the average of the pixel luminances of all pixels included in the display panel (ie, luminances to be realized by the corresponding pixel in the corresponding image frame). In this case, the pixels included in the display panel are divided into high grayscale luminance pixels having a pixel luminance greater than the full grayscale luminance average of the image frame and low grayscale luminance pixels having a pixel luminance lower than the full grayscale luminance average of the image frame. The average of the low grayscale luminance of the image frame is calculated as the average of the pixel luminances of the low grayscale luminance pixels among the pixels included in the display panel, and the high grayscale luminance average of the image frame is calculated as the high grayscale luminance average of the high grayscale luminance pixels among the pixels included in the display panel. After calculating as the average of the pixel luminances, the tone mapping curve GTM may be derived by calculating the tone mapping function based on the full grayscale luminance average, the low grayscale luminance average, and the high grayscale luminance average of the image frame.

Next, it is determined whether a scene change has occurred between the image frame and the previous image frame by comparing the data signal with the previous data signal corresponding to the previous image frame.

At this time, if it is determined that a scene change has not occurred, a final tone mapping curve is generated based on the previous tone mapping curve applied to the previous image frame and the tone mapping curve, and if it is determined that the scene change has occurred, the tone mapping curve is determined as the final tone mapping curve.

Finally, the tone mapping is performed by applying the final tone mapping curve to the image frame.

In addition, the inverse tone mapping method according to the present invention divides the low-contrast image into a plurality of sub-layer low-contrast images using at least one separation filter, and divides each sub-layer low-contrast image into a plurality of patches, After determining the image category of each patch, the step of generating the high-contrast image by converting each patch into a high-contrast image patch according to a transformation matrix learned for each image category may be performed.

Such technology may be implemented as an application or implemented in the form of program instructions that may be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer-readable recording medium are specially designed and configured for the present invention, and may be known and available to those skilled in the art of computer software.

Examples of the computer-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, and a magneto-optical medium such as a floppy disk. media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although the above has been described with reference to the embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the following claims. will be able

100: first deep learning engine unit
200: second deep learning engine unit
300: third deep learning engine unit
400: integrated modeling unit
500: fourth deep learning engine unit

Claims (3)

Deep learning that converts SDR video content into HDR10+ video content using an artificial neural network learned through comparison between SDR original content, HDR10+ original content, and HDR10+ original content created for the same video content In the HDR10+ content change system of SDR content,
HDR10+ content change system of SDR content using the deep learning,
a first deep learning engine unit for generating a first deep learning algorithm for deriving a difference between the SDR original content for the video content for training and the content obtained by converting the HDR10+ original content into SDR;
a second deep learning engine unit for generating a second deep learning algorithm for deriving a difference between the SDR original content and the HDR10+ original content for the video content for training;
a third deep learning engine unit for generating a third deep learning algorithm for deriving a difference between the HDR10+ original content for the video content for training and the content obtained by converting the HDR10+ original content into SDR; and
An integrated modeling unit for generating transformation modeling for converting SDR image content into HDR10+ image content based on the first deep learning algorithm, the second deep learning algorithm, and the third deep learning algorithm; including,
The first deep learning engine unit, the second deep learning engine unit, and the third deep learning engine unit derive a difference by comparing the same n-th frame of two image contents to be compared with each other,
The pixel value is characterized in that any one of a brightness value, a contrast value, a color depth value, and a color gamut value,
HDR10+ content change system of SDR content using the deep learning,
A fourth deep learning engine unit for deriving a difference between the HDR10+ image content converted through the integrated modeling unit and the HDR10+ original content for the learning image content, and correcting the transformation modeling based on the derived difference; further comprising,
The fourth deep learning engine unit,
The difference value is calculated by comparing the pixel values of the nth frame of the HDR10+ video content converted through the integrated modeling unit and the nth frame of the HDR10+ original content, and the calculated difference value is greater than or equal to a preset reference value. If more than the reference number of pixels is detected, the extracted pixel is set as a feature point,
HDR10+ content change of SDR content using deep learning, which generates a feature figure connecting a plurality of set feature points, extracts feature information indicating the characteristics of the generated feature figure, and corrects transformation modeling based on the extracted feature information system.
delete delete

Images