KR102602690B1 - Method and apparatus for adaptive encoding and decoding based on image quality - Google Patents

Abstract

A method and device for adaptive encoding and decoding based on picture quality are provided. The encoding device can determine the optimal frame per second (FPS) for the video and encode the video according to the determined FPS. Additionally, the encoding device can provide improved temporal scalability. The decoding device can select a frame to be played among the frames of the video according to the minimum required image quality satisfaction. Through frame selection, the decoding device can provide improved temporal scalability.

Description

Method and device for adaptive encoding and decoding based on image quality {METHOD AND APPARATUS FOR ADAPTIVE ENCODING AND DECODING BASED ON IMAGE QUALITY}

The following embodiments relate to a video decoding method, a decoding device, an encoding method, and an encoding device. More specifically, they relate to a method and device that adaptively performs encoding and decoding based on the image quality of the image.

Through the continued development of the information and communications industry, broadcasting services with HD (High Definition) resolution have spread globally. Through this proliferation, many users have become accustomed to high-resolution, high-definition images and/or videos.

In order to satisfy users' demand for high image quality, many organizations are accelerating the development of next-generation imaging devices. User interest in not only High Definition TV (HDTV) and Full HD (FHD) TV, but also Ultra High Definition (UHD) TV, which has a resolution more than four times that of FHD TV. has increased, and with this increase in interest, image encoding/decoding technology for images with higher resolution and image quality is required.

In new video services such as UHD, the need for high frame rate (HFR) video is increasing. For example, a high frame rate may be a frame refresh rate of 60 Frames Per Second (FPS) or higher.

However, in order to provide such HFR video, a problem may arise in that the amount of video data increases. Additionally, as the amount of video data increases, cost issues and technical problems may arise in the transmission and storage of video.

Fortunately, due to the nature of human vision, HFR is not required in all situations. For example, most people will not see a difference in image quality between a 30 FPS video and a 60 FPS video in a video with little or no motion. I don't feel any degradation.

In other words, depending on the content of the video, if the FPS is above a certain threshold value, even if the FPS is higher, depending on human cognitive characteristics, most people may hardly notice any difference in picture quality.

One embodiment may provide a method and apparatus for predicting the degree of picture quality degradation that occurs when converting HFR video to low frame rate video.

One embodiment may provide a method and device for reducing the bit rate required for encoding by predicting the degree of deterioration in image quality.

One embodiment is a method of minimizing the degradation of image quality by using information generated by predicting the degree of degradation of image quality when converting the frame rate of a video to a lower level through temporal scalability (TS), etc. and devices can be provided.

One embodiment may provide a method and device for minimizing image quality degradation by also considering information related to image quality degradation when applying temporal scalability.

On one side, determining whether to decode the frame based on frame selection information; and performing decoding of the frame when decoding of the frame is determined.

The decision may be performed for each frame of a plurality of frames.

The plurality of frames may be frames of a group of pictures (GOP).

The selection information may be related to the proportion of people who do not recognize the deterioration in the image quality of the video even if playback of the video including the frame is set to exclude the frame from decoding.

The selection information may be related to the proportion of people who do not recognize the deterioration in video quality even if the frames per second (FPS) of video playback including the frame are determined so that the frame is excluded from decoding. .

Whether or not to decode the frame may be determined based on comparison between the value of the selection information and the value of the reproduction information.

The playback information may be information related to playback of a video including the frame.

The playback information may be information related to frames per second (FPS) of playback of a video including the frame.

If the value of the selection information is greater than the value of the reproduction information, it may be determined that the decoding is not performed on the frame.

The playback information may be information related to playback of a video including the frame.

The selection information may be included in Supplemental Enhancement Information (SEI) for the frame.

The decision on whether to decode may be commonly applied to other frames having the same temporal identifier as the temporal identifier of the frame among a plurality of frames including the frame.

On the other hand, a control unit that determines whether to decode the frame based on frame selection information; and a decoding unit that performs decoding of the frame when decoding of the frame is determined.

In another aspect, generating selection information for a frame of a video; and performing encoding of the video based on the selection information.

The encoding of the video may include determining whether to encode the frame based on the selection information for the frame; And when the encoding of the frame is determined, it may include performing encoding of the frame.

The decision on whether to encode may be commonly applied to other frames having the same temporal identifier as that of the frame among a plurality of frames including the frame.

The selection information may be related to the percentage of people who do not notice a decrease in the picture quality of the video even if the frame is excluded from video encoding.

The ratio can be calculated by machine learning.

The ratio may be determined based on the feature vector of the frame.

The selection information may be plural.

The plurality of selection information may be calculated for each frame per second (FPS) of video playback.

The bitstream generated by encoding the video may include the selection information.

The selection information may be commonly applied to other frames having the same temporal identifier as that of the frame among a plurality of frames including the frame.

The selection information may be related to the proportion of people who do not recognize a decrease in the image quality of the played video even if playback of the video is set to exclude the frame from decoding.

A method and apparatus are provided for predicting how much the picture quality of a unit interval will deteriorate when the frame rate of the unit interval is lowered for each unit interval of HFR video.

A method of reducing the data rate of HFR video by reducing the frame rate of a unit section while maintaining the perceived image quality within the image quality difference range input by the user or within the predefined allowable image quality difference range, and A device is provided. For example, a predefined acceptable range of difference in picture quality may be a range in which 80% of people do not notice a difference in picture quality.

A method and apparatus are provided that consider image quality degradation prediction information that predicts how much image quality will deteriorate in the application of TS.

A method and apparatus are provided for applying TS only to a unit section corresponding to an image quality degradation range input by a user or a predefined image quality degradation range by considering image quality degradation prediction information.

1 is a block diagram showing the configuration of an encoding device to which the present invention is applied according to an embodiment.
Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment to which the present invention is applied.
Figure 3 explains the operation of SURP according to one embodiment.
Figure 4 shows image quality evaluation results according to an example.
Figure 5 explains the procedure for extracting a feature vector according to an embodiment.
Figure 6 shows a prediction model for spatial random measurement according to an example.
Figure 7 shows SRM according to an example.
Figure 8 shows SM according to one example.
Figure 9 shows SCM according to an example.
Figure 10 shows a VSM according to an example.
Figure 11 shows the SIM according to work.
Figure 12 shows the relationship between a frame used to generate a temporal prediction model and a prediction target frame according to an example.
FIG. 13A shows the first frame of three consecutive frames according to an example.
FIG. 13B shows the second frame among three consecutive frames according to an example.
FIG. 13C shows the third frame among three consecutive frames according to an example.
FIG. 13D shows a temporal randomness map (TRM) for three consecutive frames according to an example.
FIG. 13E shows a SpatioTemporal Influence Map (STIM) for three consecutive frames according to an example.
FIG. 13F shows a Weighted SpatioTemporal Influence Map (WSTIM) for three consecutive frames according to an example.
Figure 14 shows image quality satisfaction by FPS according to an example.
Figure 15 shows the optimal frame rate determined for each GOP according to an example.
Figure 16 is a flowchart of a GOP encoding method using an optimal frame rate according to an example.
17 is a flowchart of a method for determining an optimal frame rate according to an example.
Figure 18 shows a hierarchical structure of frames with temporal identifiers according to an example.
Figure 19 shows a message including image quality satisfaction according to an example.
Figure 20 shows a message including an index of an image quality satisfaction table according to an example.
Figure 21 shows a message including all picture quality satisfaction levels according to an example.
Figure 22 illustrates a method of generating image quality satisfaction information according to an example.
Figure 23a shows a configuration for maintaining image quality satisfaction of 75% or more according to an example.
Figure 23b shows a configuration for determining whether to change FPS based on 75% image quality satisfaction according to an example.
Figure 24 is a structural diagram of an encoding device according to an embodiment.
Figure 25 is a flowchart of an encoding method according to an embodiment.
Figure 26 is a flowchart of a method for encoding a video according to an example.
Figure 27 is a structural diagram of a decoding device according to an embodiment.
Figure 28 is a flowchart of a decryption method according to an embodiment.

For a detailed description of the exemplary embodiments described below, refer to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description that follows is not to be taken in a limiting sense, and the scope of the exemplary embodiments is limited only by the appended claims, together with all equivalents to what those claims assert if properly described.

Similar reference numbers in the drawings refer to identical or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

When a component is said to be “connected” or “connected” to another component, the two components may be directly connected or connected to each other, but It should be understood that other components may exist in the middle of the components. In addition, the description of “including” a specific configuration in the exemplary embodiments does not exclude configurations other than the specific configuration, and does not mean that additional configurations are included in the implementation of the exemplary embodiments or the technical idea of the exemplary embodiments. This means that it can be included in the scope.

Terms such as first and second may be used to describe various components, but the components should not be limited by the above terms. The above terms are used to distinguish one component from other components. For example, a first component may be named a second component without departing from the scope of rights, and similarly, the second component may also be named a first component.

Additionally, the components appearing in the embodiments are shown independently to represent different characteristic functions, and this does not mean that each component consists of separate hardware or a single software component. That is, each component is listed as a separate component for convenience of explanation. For example, at least two of the components may be combined into one component. Additionally, one component may be divided into multiple components. Integrated and separate embodiments of each of these components are also included in the scope of rights as long as they do not deviate from the essence.

Additionally, some components may not be essential components that perform essential functions, but may simply be optional components to improve performance. Embodiments may be implemented including only components essential to implement the essence of the embodiment, and structures excluding optional components, for example, components used only to improve performance, are also included in the scope of rights.

Hereinafter, embodiments will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the embodiments. In describing the embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description will be omitted.

Hereinafter, an image may refer to a single picture constituting a video, and may also represent the video itself. For example, “encoding and/or decoding of an image” may mean “encoding and/or decoding of a video,” and may mean “encoding and/or decoding of one of the images that constitute a video.” It may be possible.

Hereinafter, the numbers used in the examples are merely examples and may be replaced with values other than those specified in the examples.

In embodiments, information, operations, and functions that apply to encoding may also apply to decoding in a corresponding manner to encoding.

If the frame rate can be adaptively and automatically adjusted using human visual characteristics as described above, the amount of video data can be reduced without deteriorating image quality in terms of perceptual quality for a large number of people. there is. If human visual characteristics can be applied to HFR video and the video can be converted to the optimal screen refresh rate for each section, the increase in the amount of video data can be minimized. Therefore, cost problems and technical problems that arise during the transmission and storage process of HFR video can be solved.

Adaptive and automatic adjustment of frame rates requires that it be possible to accurately predict the difference in picture quality that will occur when changing video from HFR to a lower frame rate. This prediction technology can be usefully used during the video encoding preprocessing process, encoding process, transmission process, and/or decoding process.

However, using conventional technology, the difference in picture quality that occurs when converting HFR video to low frame rate video cannot be accurately predicted.

Additionally, a conventional technology related to image quality adjustment is Temporal Scalability (TS) technology. TS technology uniformly adjusts the frame rate of the video without considering how the video quality will change when TS is applied to the video. Therefore, when TS is applied to video, a problem arises in that the picture quality of the video may be significantly deteriorated.

In the following embodiments, 1) a method for predicting image quality satisfaction, 2) a method for changing video to an optimal low frame rate through prediction of image quality satisfaction, and 3) utilizing the information generated by predicting image quality satisfaction in TS. How to do this is explained.

1) How to predict picture quality satisfaction: A picture quality satisfaction predictor that predicts the degree of picture quality difference that occurs when changing HFR to a low frame rate is explained.

2) Method of changing video to an optimal low frame rate through prediction of picture quality satisfaction: An encoder that changes video of fixed HFR to an optimal low frame rate for each basic decision unit and encodes it using a picture quality satisfaction predictor is explained.

3) Method of utilizing information generated by prediction of picture quality satisfaction in TS: A video encoding stream generated by video encoding may include picture quality satisfaction information generated by a picture quality satisfaction predictor. Picture quality information can be used for TS during the transmission or decoding process of a video encoding stream. An improved TS that was previously not possible can be provided based on image quality satisfaction information. How the video quality will change when TS is applied can be considered based on picture quality satisfaction information. Through these considerations, an improved TS that minimizes image quality degradation can be provided.

First, terms used in the embodiments will be explained.

Unit: “Unit” may represent a unit of video encoding and decoding. The meanings of unit and block may be the same. Additionally, the terms “unit” and “block” may be used interchangeably.

A unit (or block) may be an MxN array of samples. M and N can each be positive integers. A unit can often refer to a two-dimensional array of samples. Samples can be pixels or pixel values.

In video encoding and decoding, a unit may be an area created by dividing one image. One image can be divided into multiple units. In encoding and decoding images, predefined processing may be performed on units depending on the type of unit. Depending on the function, the type of unit can be classified into macro unit (Macro Unit), coding unit (CU), prediction unit (PU), and transform unit (TU). One unit may be further divided into subunits having a smaller size compared to the unit.

Block division information may include information about the depth of the unit. Depth information may indicate the number and/or extent to which a unit is divided.

One unit may be hierarchically divided into a plurality of sub-units while having depth information based on a tree structure. In other words, a unit and a sub-unit created by division of the unit may respectively correspond to a node and a child node of the node. Each divided sub-unit may have depth information. Since the depth information of a unit indicates the number and/or extent to which the unit is divided, the division information of the sub-unit may include information about the size of the sub-unit.

In a tree structure, the highest node may correspond to the first undivided unit. The highest node may be referred to as the root node. Additionally, the highest node may have the minimum depth value. At this time, the highest node may have a depth of level 0.

A node with a depth of level 1 may represent a unit created as the original unit is divided once. A node with a depth of level 2 may represent a unit created by dividing the original unit twice.

A node with a depth of level n may represent a unit created as the original unit is divided n times.

A leaf node may be the lowest node and may be a node that cannot be further divided. The depth of the leaf node may be at the maximum level. For example, the predefined value of the maximum level may be 3.

Transform Unit: A transform unit may be a basic unit in residual signal coding and/or residual signal decoding, such as transform, inverse transform, quantization, inverse quantization, transform coefficient coding, and transform coefficient decoding. . One transformation unit may be divided into multiple transformation units with smaller sizes.

Prediction Unit: A prediction unit may be a basic unit in performing prediction or compensation. A prediction unit may be divided into multiple partitions. Multiple partitions may also be the basic unit in performing prediction or compensation. A partition created by dividing a prediction unit may also be a prediction unit.

Reconstructed Neighbor Unit: The reconstructed neighbor unit may be a unit that has already been encoded or decoded and restored around the encoding target unit or the decoding target unit. The restored neighboring unit may be a spatially adjacent unit or a temporally adjacent unit to the target unit.

Prediction unit partition: Prediction unit partition may refer to the form in which the prediction unit is divided.

Parameter Set: The parameter set may correspond to header information among the structures in the bitstream. For example, the parameter set may include a sequence parameter set, a picture parameter set, an adaptation parameter set, etc.

Rate distortion optimization: The encoding device uses rate distortion optimization to provide high coding efficiency using a combination of coding unit size, prediction mode, prediction unit size, motion information, and transformation unit size. You can.

The rate distortion optimization method can calculate the rate distortion cost of each combination to select the optimal combination among the above combinations. The rate distortion cost can be calculated using Equation 1 below. In general, the combination that minimizes the rate distortion cost can be selected as the optimal combination in the rate distortion optimization method.

D may indicate distortion. D may be the mean square error of the difference values between the original transform coefficients and the restored transform coefficients within the transform block.

R can represent a rate. R can represent the bit rate using related context information.

λ may represent a Lagrangian multiplier. R may include not only encoding parameter information such as prediction mode, motion information, and coded block flag, but also bits generated by encoding of transform coefficients.

The encoding device performs processes such as inter prediction and/or intra prediction, transformation, quantization, entropy encoding, inverse quantization, and inverse transformation to calculate accurate D and R, and these processes can significantly increase the complexity of the encoding device. there is.

Reference picture: A reference picture may be an image used for inter prediction or motion compensation. A reference picture may be a picture that includes a reference unit referenced by the target unit for inter prediction or motion compensation. The meanings of a picture and a video may be the same. Additionally, the terms “picture” and “image” may be used interchangeably.

Reference picture list: The reference picture list may be a list containing reference pictures used for inter prediction or motion compensation. Types of reference picture lists may include List Combined (LC), List 0 (L0), and List 1 (L1).

Motion Vector (MV): A motion vector may be a two-dimensional vector used in inter prediction. For example, MV can be expressed in a form such as (mv _x , mv _y ). mv _x can represent the horizontal component, and mv _y can represent the vertical component.

MV may indicate an offset between a target picture and a reference picture.

Search range: The search range may be a two-dimensional area where a search for MV is performed during inter prediction. For example, the size of the search area may be MxN. M and N can each be positive integers.

Basic decision unit: The basic decision unit may represent a basic unit that is the target of processing in an embodiment. The basic decision unit may be a plurality of frames or a group of pictures (GOP).

Group of Pictures (GOP): The meaning of GOP in the embodiment may be the same as the meaning of GOP generally used in video encoding. In other words, the start frame of the GOP may be an I frame, and frames other than the start frame of the GOP may be frames that directly reference the I frame or frames that indirectly reference the I frame.

Frame rate: Frame rate can indicate the temporal resolution of the video. Frame rate may mean how many screen(s) are displayed per second.

Frames Per Second (FPS): FPS may be a unit representing frame rate. For example, “30 FPS” may indicate that 30 frames are played per second.

High Frame Rate (HFR): Can indicate a frame rate higher than a predetermined reference value. For example, according to Korean and American standards, a video with an FPS of 60 or more may be an HFR video, and according to European standards, a video with an FPS of 50 or more may be an HFR video. In embodiments, video at 60 FPS is illustrated as HFR video.

Optimal Frame Rate: The optimal frame rate may be a frame rate determined depending on the content of the video. The optimal frame rate can be determined for each basic decision unit. Even if the frame rate of the video or basic decision unit is increased beyond the optimal frame rate, people may not notice a difference in picture quality due to the increase in frame rate. In other words, the optimal frame rate may be the minimum frame rate at which people do not notice a decrease in video quality compared to HFR. Additionally, the optimal frame rate may be the FPS just before picture quality degradation occurs when measuring the picture quality difference between the original HFR video and the video of the reduced FPS while gradually reducing the FPS of the HFR video. Here, the occurrence of image quality degradation may mean that a difference in image quality is recognized by a percentage set by the user or by more than a preset percentage of people.

Homogeneous video: A homogeneous video may be a video in which all basic decision units that make up the video have the same or similar characteristics.

Perception rate of image quality degradation: The perception rate of image quality decline represents the proportion of people who can recognize the deterioration in image quality when the frame rate of HFR video is reduced for each basic decision unit. Typically, FPS can be changed in steps of 1/2. The unit of recognition rate of image quality degradation may be %.

Image quality degradation recognition rate predictor: The image quality degradation recognition rate predictor may be a predictor that predicts the image quality degradation recognition rate.

Satisfied with picture quality (satisfied user ratio): Satisfaction with picture quality may be the ratio of people who are satisfied with the picture quality of the video with a reduced frame rate compared to the picture quality of the HFR video when the FPS of the HFR video is reduced for each basic decision unit. Alternatively, satisfaction with picture quality may be the ratio of people who do not recognize the difference between the picture quality of HFR video and the picture quality of video with reduced FPS when the FPS of HFR media is reduced for each basic decision unit.

Typically, FPS can be changed in steps of 1/2. Here, the ratio of people may be expressed as a percentage of the value of 1a/b. a may be the number of people who evaluated that there was a difference between the image quality of the HFR video and the image quality of the reduced FPS video through subjective image quality evaluation. b may be the total number of people. For example, for the 60 FPS video and the 30 FPS video, a total of 5 people made subjective picture quality evaluations, and 3 out of 5 people gave the same picture quality score to the 60 FPS video and the 30 FPS video, and the remaining If two people gave a lower picture quality score to the 30 FPS video compared to the 60 FPS video, the satisfaction with picture quality could be 12/5 = 60%. At this time, the recognition rate of image quality deterioration may be 40%. In other words, the sum of image quality satisfaction and perceived image quality deterioration may be 100%.

Satisfied User Ratio Predictor (SURP): SURP may be a predictor that predicts satisfaction with image quality. The configuration and operation of the picture quality degradation perception rate predictor and SURP may be similar to each other, and the two may simply differ in the output value.

Optional information: In an embodiment, the image quality information may be a perceived rate of image quality degradation or satisfaction with image quality. The term “selection information” may be used with the same meaning as the term “perceived image quality information,” and both terms may be used interchangeably.

Basic operations of encoding and decoding devices

1 is a block diagram showing the configuration of an encoding device to which the present invention is applied according to an embodiment.

The encoding device 100 may be a video encoding device or an image encoding device. A video may contain one or more images. The encoding device 100 may sequentially encode one or more images of a video according to time.

Referring to FIG. 1, the encoding device 100 includes an inter prediction unit 110, an intra prediction unit 120, a switch 115, a subtractor 125, a transform unit 130, a quantization unit 140, and an entropy encoding unit. It may include a unit 150, an inverse quantization unit 160, an inverse transform unit 170, an adder 175, a filter unit 180, and a reference picture buffer 190.

The encoding device 100 may perform encoding on an input image in intra mode and/or inter mode. The input image may be referred to as the current image that is currently the target of encoding.

Additionally, the encoding device 100 can generate a bitstream including encoding information through encoding of an input image and output the generated bitstream.

If intra mode is used, switch 115 can be toggled to intra. When inter mode is used, switch 115 can be switched to inter.

The encoding device 100 may generate a prediction block for an input block of an input image. Additionally, the encoding device 100 may encode the residual of the input block and the prediction block after the prediction block is generated. The input block may be referred to as the current block that is the target of current encoding.

When the prediction mode is intra mode, the intra prediction unit 120 may use the pixel value of an already encoded block surrounding the current block as a reference pixel. The intra prediction unit 120 may perform spatial prediction for the current block using a reference pixel and generate prediction samples for the current block through spatial prediction.

The inter prediction unit 110 may include a motion prediction unit and a motion compensation unit.

When the prediction mode is inter mode, the motion prediction unit can search for the region that best matches the current block from the reference image during the motion prediction process and can derive motion vectors for the current block and the searched region. The reference image may be stored in the reference picture buffer 190, and may be stored in the reference picture buffer 190 when encoding and/or decoding of the reference image is processed.

The motion compensation unit may generate a prediction block by performing motion compensation using a motion vector. Here, the motion vector may be a two-dimensional vector used for inter prediction. Additionally, the motion vector may indicate an offset between the current image and the reference image.

The subtractor 125 may generate a residual block, which is the difference between the input block and the prediction block. A residual block may also be referred to as a residual signal.

The transform unit 130 may generate a transform coefficient by performing transformation on the residual block and output the generated transform coefficient. Here, the transformation coefficient may be a coefficient value generated by performing transformation on the residual block. When the transform skip mode is applied, the transform unit 130 may skip transforming the residual block.

A quantized transform coefficient level can be generated by applying quantization to the transform coefficient. Hereinafter, in embodiments, the quantized transform coefficient level may also be referred to as a transform coefficient.

The quantization unit 140 may generate a quantized transform coefficient level by quantizing the transform coefficient according to the quantization parameter. The quantization unit 140 may output the generated quantized transform coefficient level. At this time, the quantization unit 140 may quantize the transform coefficient using a quantization matrix.

The entropy encoding unit 150 may generate a bitstream by performing entropy encoding according to a probability distribution based on the values calculated by the quantization unit 140 and/or the encoding parameter values calculated during the encoding process. . The entropy encoding unit 150 may output the generated bitstream.

The entropy encoding unit 150 may perform entropy encoding on information for decoding the image in addition to pixel information of the image. For example, information for decoding an image may include syntax elements, etc.

Encoding parameters may be information required for encoding and/or decoding. The encoding parameter may include information that is encoded in the encoding device and transmitted to the decoding device, and may include information that can be inferred during the encoding or decoding process. For example, information transmitted to the decoding device includes a syntax element.

For example, coding parameters include prediction mode, motion vector, reference picture index, coding block pattern, presence or absence of residual signal, transform coefficient, quantized transform coefficient, quantization parameter, block size, block partition. It may contain values such as information or statistics. Prediction mode may refer to intra prediction mode or inter prediction mode.

The residual signal may refer to the difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming the difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming and quantizing the difference between the original signal and the predicted signal. The residual block may be a residual signal in block units.

When entropy coding is applied, a small number of bits may be assigned to symbols with a high probability of occurrence, and a large number of bits may be assigned to symbols with a low probability of occurrence. As symbols are expressed through this allocation, the size of the bitstring for the symbols that are the target of encoding can be reduced. Therefore, the compression performance of video encoding can be improved through entropy coding.

Additionally, for entropy coding, coding methods such as exponential golomb, ContextAdaptive Variable Length Coding (CAVLC), and ContextAdaptive Binary Arithmetic Coding (CABAC) can be used. . For example, the entropy encoding unit 150 may perform entropy encoding using a Variable Lenghth Coding/Code (VLC) table. For example, the entropy encoding unit 150 may derive a binarization method for the target symbol. Additionally, the entrope encoder 150 can derive a probability model of the target symbol/bin. The entropy encoding unit 150 may perform entropy encoding using a derived binarization method or a probability model.

Since encoding through inter prediction is performed by the encoding device 100, the encoded current image can be used as a reference image for other image(s) to be processed later. Accordingly, the encoding device 100 can decode the current encoded image again and store the decoded image as a reference image. For decoding, inverse quantization and inverse transformation of the encoded current image may be processed.

The quantized coefficient may be inversely quantized in the inverse quantization unit 160. It may be inversely transformed in the inverse transformation unit 170. The inverse-quantized and inverse-transformed coefficients can be combined with the prediction block through the adder 175. A reconstructed block can be generated by summing the inverse-quantized and inverse-transformed coefficients and the prediction block.

The restored block may pass through the filter unit 180. The filter unit 180 may apply at least one of a deblocking filter, Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF) to the restored block or restored picture. The filter unit 180 may also be referred to as an adaptive inloop filter.

The deblocking filter can remove block distortion occurring at the boundaries between blocks. SAO can add an appropriate offset value to the pixel value to compensate for coding errors. ALF can perform filtering based on a comparison value between the restored image and the original image. The restored block that has passed through the filter unit 180 may be stored in the reference picture buffer 190.

Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment to which the present invention is applied.

The decoding device 200 may be a video decoding device or an image decoding device.

Referring to FIG. 2, the decoding device 200 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra prediction unit 240, an inter prediction unit 250, and an adder 255. , may include a filter unit 260 and a reference picture buffer 270.

The decoding device 200 may receive the bitstream output from the encoding device 100. The decoding device 200 may perform intra-mode and/or inter-mode decoding on the bitstream. Additionally, the decoding device 200 can generate a restored image through decoding and output the generated restored image.

For example, switching to intra mode or inter mode according to the prediction mode used for decoding can be accomplished by a switch. If the prediction mode used for decoding is intra mode, the switch may be switched to intra. If the prediction mode used for decoding is inter mode, the switch may be switched to inter.

The decoding device 200 can obtain a reconstructed residual block from the input bitstream and generate a prediction block. Once the restored residual block and the prediction block are obtained, the decoding device 200 may generate a restored block by adding the restored residual block and the prediction block.

The entropy decoding unit 210 may generate symbols by performing entropy decoding on the bitstream based on a probability distribution. Generated symbols may include symbols in the form of quantized coefficients. Here, the entropy decoding method may be similar to the entropy encoding method described above. For example, the entropy decoding method may be the reverse process of the entropy encoding method described above.

The quantized coefficient may be inverse quantized in the inverse quantization unit 220. Additionally, the inverse quantized coefficient may be inversely transformed in the inverse transform unit 230. As a result of inverse quantization and inverse transformation of the quantized coefficients, a restored residual block may be generated. At this time, the inverse quantization unit 220 may apply a quantization matrix to the quantized coefficient.

When the intra mode is used, the intra prediction unit 240 may generate a prediction block by performing spatial prediction using pixel values of already encoded blocks surrounding the current block.

The inter prediction unit 250 may include a motion compensation unit. When inter mode is used, the motion compensation unit can generate a prediction block by performing motion compensation using a motion vector and a reference image. The reference image may be stored in the reference picture buffer 270.

The restored residual block and prediction block can be added through an adder 255. The adder 255 may generate a restored block by adding the restored residual block and the prediction block.

The restored block may pass through the filter unit 260. The filter unit 260 may apply at least one of a deblocking filter, SAO, and ALF to a reconstructed block or a reconstructed picture. The filter unit 260 may output a restored image. The reconstructed image can be stored in the reference picture buffer 270 and used for inter prediction.

Satisfied User Ratio Predictor (SURP)

Figure 3 explains the operation of SURP according to one embodiment.

The input of SURP 300 may be the basic decision unit. As shown in FIG. 3, the basic decision unit may be GOP. The output of SURP 300 may be the predicted picture quality satisfaction for the GOP of the changed FPS.

For example, the input GOP may be a GOP with 8 frames at 60 FPS.

To calculate the predicted picture quality satisfaction, SURP 300 may change the FPS of the GOP. For example, SURP 300 may change the GOP of X FPS to the GOP of Y FPS. For example, Y may be aX, and a may be 1/2. Here, “1/2” is just an exemplary value, and a may have a value of 1/2 ⁿ , such as 1/4, 1/8, and 1/16. n may be an integer of 1 or more.

X may represent the FPS of the GOP before change. Y may represent the FPS of the GOP after change. When X=2 ⁿ , Y can be one of X/2, X/2 ² , ... X/2 ⁿ . For example, when the value of X is 60, the value of Y may be 30. That is, GOP can be converted from 60 FPS to 30 FPS.

Predicted image quality satisfaction may be a predicted value for the proportion of people who do not recognize the difference in image quality between the image quality of the GOP before the change and the image quality of the GOP after the change, assuming that the FPS of the GOP has been changed from 60 to 30. Alternatively, the predicted picture quality satisfaction may be a predicted value for the ratio of people who are satisfied with the picture quality of the GOP after the change compared to the picture quality of the GOP before the change, under the assumption that the FPS of the GOP has been changed from 60 to 30.

SURP (300) can generate predicted image quality satisfaction through machine learning.

Hereinafter, the GOP before the FPS change may be referred to as the source GOP. Additionally, the GOP after changing the FPS may be referred to as a target GOP. Additionally, the video before the FPS change may be referred to as the source video. Additionally, the video after changing the FPS may be referred to as a target video.

Considerations for generating predicted image quality satisfaction through machine learning.

In an embodiment, the SURP 300 may use machine learning to measure how much image quality is degraded due to the change in FPS when changing the FPS for each GOP.

For machine learning, training data is required. In other words, ground truth for machine learning may be required. In the embodiment, one of the purposes is to maintain human perception image quality, so the human To obtain data on perceived image quality, a method of obtaining data through a subjective image quality evaluation experiment can be used.

Here, acquiring data on perceived image quality through subjective image quality evaluation may be just an example. Instead of directly measuring perceived image quality through subjective image quality evaluation, SURP 300 can measure perceived image quality using a measure that models perceived image quality. For example, the measure may be Structural SIMilarity (SSIM) or Video Quality Metric (VQM).

Typically, the Mean Opinion Score (MOS) value is mainly used as the value of perceived image quality that is finally obtained as subjective image quality. When applying MOS to the subjective image quality evaluation of the embodiment, problems such as 1) to 3) below may occur.

1) First, in the conventional method in which users assign scores, the standards for scores may be different depending on the user. Additionally, in the evaluation process of assigning scores, it may sometimes occur that the same person gives different scores to the same video.

2) Second, because the MOS values of the videos are different from each other, a problem may arise that it is difficult to use the different MOS values directly for machine learning. To solve this problem, SURP 300 can use relative perceived image quality information instead of MOS. For example, there is Difference MOS (DMOS) as relative perceived image quality information. If the difference between the MOS value of the source video and the MOS value of the target video is used for subjective picture quality evaluation, the results of this subjective picture quality evaluation can be used for machine learning. Such subjective picture quality evaluation may require picture quality evaluation of the source video in all evaluation items such as DoubleStimulus Continuous QualityScale (DSCQS) and Double Stimulus Impairment Scale (DSIS). Therefore, this image quality evaluation may cause the problem of increasing the time for the image quality evaluation experiment.

3) Third, the conventional image quality measurement method recommends that measurements be made for more than 8 seconds for each video sequence. In other words, the results of picture quality measurement can be viewed as sequence level results. If machine learning to determine the basic decision unit is performed using these results, good performance may not be obtained.

Accordingly, in one embodiment, a subjective image quality evaluation method for obtaining ground truth suitable for machine learning and conditions of a video sequence used for image quality evaluation are proposed.

machine learning

SURP (300) can determine the optimal FPS for each basic decision unit. In order to determine the FPS for each basic decision unit, ground truth data for this decision may need to be obtained.

In the following description, for convenience, the GOP has been considered to be the basic decision unit, and the size of the GOP has been assumed to be 8. That is, one GOP may contain 8 frames. These assumptions about the size of the basic decision unit and GOP are illustrative only.

The following explanations related to acquisition of ground truth and subjective image quality evaluation can be applied in the same or similar manner to other types of basic crystal units and basic crystal units of different sizes.

Subjective image quality evaluation can only be performed at the sequence level. On the other hand, for machine learning, evaluation results for GOP units are required. To overcome these differences, the length of the video sequence used for picture quality evaluation can be adjusted to ensure that all GOPs in the video sequence contain only the same or similar content characteristics. That is to say, the characteristics of all GOPs in a video may be similar to each other. In an embodiment, a video in which the characteristics of all GOPs of a video are similar to each other may be referred to as a homogeneous video.

The length of a homogeneous video may be limited to a predefined value. For example, the length of a homogeneous video can be adjusted to not exceed 5 seconds.

Since homogenous video composed of GOPs with similar characteristics is used, it may be possible for the image quality evaluation results obtained at the sequence level to be used as image quality information in units of GOPs. Additionally, SURP (300) can improve its performance through machine learning using this image quality information.

Figure 4 shows image quality evaluation results according to an example.

As an example of subjective image quality evaluation, the source video may be an HFR video at 60 FPS and may be a video with HD resolution. The target video may be a video at 30 FPS.

SURP (300) can generate a target video by lowering the temporal resolution of the source video to 1/n. n may be 2. Alternatively, n may be 2 ^m , and m may be an integer of 1 or more.

SURP 300 can generate a target video by applying frame drop to the source video. This may be removing a predefined frame from the frame-dropped source video. For example, frame dropping may be removing even-numbered frame(s) among the frames of the source video.

As a subjective image quality evaluation method, a pairwise comparison method can be used.

The pairwise comparison method may be a method of showing two videos to the evaluator for each evaluation question and asking which of the two videos has better picture quality. The evaluator can select the video with better quality among the two videos for each evaluation question. Alternatively, the pairwise comparison method may query for each evaluation question whether the first video has better picture quality, the second video has better picture quality, or the picture quality of the two videos is the same. That is, the evaluator can select one of three items for each evaluation question. In the example, a method using three items was illustrated.

The method of selecting one of three items for each evaluation question can lead to more accurate picture quality evaluation results when 1) the picture quality of the two videos is similar and 2) a relatively small number of evaluators participate in the experiment.

In evaluating picture quality, the order in which high FPS videos and low FPS videos are shown for each evaluation question can be randomly adjusted. This random adjustment may prevent the evaluator from having a preconceived idea about which of the two videos is the video with the higher FPS.

Additionally, if the length of the video sequence is less than a predetermined standard value, two videos for each evaluation question may be shown to the evaluator twice or more than twice. For example, video A and video B may be shown to the evaluator in the following order: A, B, A, B.

In Figure 4, a bar on the x-axis represents a video that is the subject of image quality evaluation. The string below the x-axis represents the name of the video. The y-axis represents the percentages selected by the raters of the three items. The three items are “the quality of the video with high quality (i.e., 60 FPS) is better,” “the quality of both videos is the same,” and “the quality of the video with low quality (i.e., 30 FPS) is better.” The selected ratio of each item may represent the ratio of evaluators who selected each item among all evaluators. The unit of ratio may be percent (%).

Among the items, if “the picture quality of both videos is the same” and “the picture quality of the low-quality video is better” is selected, it can be considered that the picture quality is maintained even after conversion of the video. In other words, selecting “the quality of both videos is the same” and “the quality of the low-quality video is better” among the items may mean that the quality of the converted video also satisfies the evaluator. In other words, among the evaluators, only the one who selected the item “the picture quality of the high-definition video is better” can be considered to have felt the difference between the picture qualities of the two videos.

Therefore, the sum of the proportion selecting “the quality of both videos is the same” and the proportion selecting “the quality of the lower quality video is better” is the “picture quality satisfaction” which represents the proportion of people who do not recognize the difference between the quality of the two videos. "It can be used as. The unit of “picture quality satisfaction” may be percentage (%).

In Figure 4, 17 video sequences are sorted in descending order of the percentage of people who do not recognize differences in picture quality. According to Figure 4, for the 7 video sequences, more than 75% of the participants did not notice the difference between the quality of the high-definition video and the low-definition video.

Similar to what was described above, various target videos can be generated from the HFR's source video. For example, the FPS of the target videos may be 1/4 and 1/8 of the FPS of the source video, etc. Satisfaction with picture quality can be measured for each of these target videos and also for the source video.

The results of this subjective image quality evaluation can be used as ground truth for machine learning. The quality of Ground Truth's data can improve as the number of evaluators increases. Additionally, the quality of ground truth data can be improved as the number of video sequences subject to evaluation increases.

When DSCQS, rather than pairwise comparison, is used for subjective image quality evaluation: 1) DSCQS's "opinion score of high quality video is higher than that of low quality video" means "high quality video" of pairwise comparison; You can respond to “The video quality is better.” 2) DSCQS's "if the opinion score of the high-quality video is the same as that of the low-quality video" can correspond to "the quality of both videos is the same" in the pairwise comparison. 3) DSCQS's "if the opinion score of the high-quality video is lower than that of the low-quality video" can correspond to "the low-quality video has better quality" of the pairwise comparison. Through this correspondence relationship, even when DSCQS is used, image quality satisfaction can be measured in the same way as when the pairwise comparison described above is used.

Feature vector

SURP 300 may be a predictor using machine learning. As an example of machine learning, SURP 300 may use a Support Vector Machine (SVM).

In order to predict the deterioration of video quality when the FPS of the video changes, features that affect video quality must be extracted as feature vectors. Below, various feature vector extraction methods that reflect human visual characteristics such as spatial masking effect, temporal masking effect, salient area, and contrast sensitivity are described.

SVM can generally use a feature vector that better represents the information of the input frame required for prediction, rather than the entire input frame. In other words, the prediction performance of SVM can be influenced by what information is used as a feature vector.

In an embodiment, so that the feature vector reflects the characteristics of perceived image quality, the feature vector may be designed to take into account whether a portion of the video has a small effect on the perceived image quality or a large effect on the perceived image quality.

For example, parts of the video that have little impact on perceived quality may be parts with a large spatial masking effect and parts with a large temporal masking effect. The part of the video that has a large impact on perceived image quality may be the part with high contrast sensitivity (CS). Additionally, visual saliency (VS) can also have a significant impact on perceived image quality. Therefore, VS can also be considered in designing feature vectors.

Figure 5 explains a procedure for extracting a feature vector according to an embodiment.

At step 510, SURP 300 may generate a spatial randomness map (SRM) for each frame.

SRM can be used to reflect the spatial masking effect in the feature vector. Areas that are not well recognized visually may be areas with high randomness. Therefore, the spatial masking area in the frame can be determined through calculation of spatial randomness (SR).

The characteristics of the SR area may be different from the characteristics of other surrounding areas. This difference may indicate that it is difficult to predict the area of the SR from information on the surrounding area. Therefore, to measure SR, the central area Y can be predicted from the peripheral area X as shown in FIG. 6.

Figure 6 shows a prediction model for spatial random measurement according to an example.

In Figure 6, it is shown that the central Y pixel is predicted from the surrounding X pixels.

Refer again to Figure 5.

The optimal prediction for Y can be expressed as Equation 2 below.

u can represent a spatial location. H may be a transformation matrix that provides optimal prediction of the Y value from surrounding X values. When the optimization method of minimum average error is used, H can be expressed as Equation 3 below.

R _xy may be the crosscorrelation matrix of X and Y.

R _x may be an autocorrelation matrix for X itself.

Using the approximated pseudoinverse matrix technique, the inverse matrix of R _x can be obtained according to Equation 4 below.

According to Equation 2, Equation 3, and Equation 4 described above, Equation 5 below can be established.

SURP 300 can obtain SRM for each pixel of the frame based on Equation 5.

Figure 7 shows SRM according to an example.

In FIG. 7, a bright pixel may represent a pixel that is not well predicted from surrounding pixels. In other words, bright pixels may indicate areas of high spatial randomness, i.e. areas with a large spatial masking effect.

Refer again to Figure 5.

In step 515, SURP 300 may generate an edge map (EM) for each frame.

SURP 300 may generate an edge map using a Sobel edge operation for each frame.

At step 520, SURP 300 may generate a smoothness map (SM) for each frame.

SM can be used to determine whether the background of the frame is a flat area.

SURP (300) can calculate the smoothness value of each block for blocks of SRM. The flatness value of each block can be calculated by Equation 6 below.

N _lc may be the number of pixels within the block with spatial randomness lower than the reference value. Here, spatial randomness lower than the reference value may mean low complexity.

W _b ² may represent the area of the block, that is, the number of pixels within the block. For example, W _b may be 32. The value of W _b is 32, which may indicate that a 32x32 block is used.

SURP 300 can generate SM by setting the values of all pixels inside each block to the flatness value of the block.

Figure 8 shows SM according to one example.

In SM, the same pixel value can be set for each block.

In SM, the brighter the blocks, the flatter the background can be. That is, bright blocks can represent a flat background. The edges of bright blocks can be better perceived compared to the edges of dark blocks.

Refer again to Figure 5.

At step 525, SURP 300 may generate a spatial contrast map (SCM) for each frame.

Stimulus sensitivity can be related to spatial contrast. SCM can be used to reflect spatial contrast in feature vectors.

SCM can be obtained by measuring edges and flatness. Sensitive responses to stimuli may occur in smooth background areas. Here, the stimulus may mean an edge. On the other hand, edges in background areas with low flatness may have low sensitivity to stimulation due to the masking effect. SCM can be a map to reflect these characteristics.

SCM may be a product of pixel values of corresponding pixels of SM and EM. SURP 300 may set the product of pixel values of pixels at the same location in SM and EM as the pixel value of SIM.

SURP (300) can generate SIM according to Equation 7 below.

Y may be a pixel at the same location in each of SCM, EM, and SM.

Figure 9 shows SCM according to an example.

In Figure 9, bright pixels may indicate a strong edge and a highly flat background. That is, a bright pixel may be a pixel representing a strong edge, or a pixel representing a highly flat background.

Refer again to Figure 5.

At step 530, SURP 300 may generate a Visual Saliency Map (VSM) for each frame.

Due to the characteristics of human vision, stimulation of areas of interest to humans may have a greater impact than stimulation of areas of interest to humans. Information on visual attention may be used so that these visual features are reflected in the feature vector.

As an example of VSM, the graphic based visual saliency method proposed by Harel and Perona can be used.

Figure 10 shows a VSM according to an example.

In FIG. 10, a bright area may represent an area of interest to a person when viewing an image of a frame. In other words, a bright area may be an area of high attention. A dark area may represent an area that a person is not interested in when viewing the image in the frame. In other words, a dark area may be an area of low attention.

Refer again to Figure 5.

At step 535, SURP 300 may generate a spatial influence map (SIM) for each frame.

SIM may be a map that reflects not only surrounding information of precognition, which is a visual stimulus, but also whether a pixel corresponds to a visual saliency area.

SIM may be a product of pixel values of corresponding pixels of SCM and VSM. SURP 300 may set the product of pixel values of pixels at the same location in the SCM and VSM as the pixel value of the SCM.

SURP (300) can generate SCM according to Equation 8 below.

Y may be a pixel at the same location in each of SIM, SCM, and VSM.

Figure 11 shows the SIM according to work.

Refer again to Figure 5.

SURP 300 may acquire a feature vector based on the spatial and temporal characteristics of the frames of the basic decision unit. In the above-described steps 510, 515, 520, 525, 530, and 535, the process of acquiring a feature vector by reflecting the spatial characteristics of the frame was explained. In the steps 540, 545, 550, and 555 below, the process of acquiring a feature vector by reflecting the temporal characteristics of the frame is explained.

An area with a high temporal masking effect may be an area with irregular movement or an area with sudden movement. That is, an area with high randomness may be an area with a high temporal masking effect, and an area with a high temporal masking effect can be determined by detecting an area with high randomness in the frame.

By acquiring a feature vector based on temporal characteristics, the SURP 300 can reflect human visual characteristics that react sensitively to irregular or sudden movements in the acquisition of the feature vector.

As described above, spatial randomness can be computed by a model predicting from surrounding pixels. Similar to this calculation method, temporal randomness can be calculated through a model that predicts the current frame from previous frames.

To calculate temporal randomness, first the input interval of frames of the GOP can be divided into two intervals.

The section can be expressed as Equation 9 below.

Y may represent a section. k may represent the start frame of the section. l may indicate that it consists of the last frame of the section. In other words, Y _k ^l may indicate that the section consists of the kth frame to the lth frame.

When Equation 9 is stored as a matrix, the columns of the matrix may each correspond to frames. In other words, storing the section as a matrix can be accomplished through steps 1) and 2) below.

1) The frames in the section can be arranged in the form of a one-dimensional row vector. For example, the second row of a frame can be concatenated to the first row of a frame. Rows of a frame can be chained sequentially.

2) Row vectors can be transposed in the form of column vectors. A matrix consisting of column vectors of frames may be stored.

When the length of each section is d, the two sections can be expressed as Y _k+d ^l and Y _k ^{l d} .

At step 540, SURP 300 may generate a temporal prediction model (TPM).

In FIG. 12 below, a case where the GOP size is 8 and the value of d is 4 is illustrated.

Figure 12 shows the relationship between a frame used to generate a temporal prediction model and a prediction target frame according to an example.

In FIG. 12, “frame 0” may represent the first frame, and “frame 1” may represent the second frame. In other words, “frame n” may be the n+1th frame.

According to FIG. 12, “Frame 0” to “Frame 3” may be the first section, and “Frame 4” to “Frame 7” may be the second section.

Frames in the first section can be used for TPM, and frames in the second section can be predicted by TPM.

Refer again to Figure 5.

TPM can be created using the first section of GOP. For example, for a GOP of size 8, the TPM may be generated using the first frame, second frame, third frame, and fourth frame.

SURP (300) can perform temporal prediction based on Equation 10 below.

Here, A may be a transformation matrix that provides the optimal prediction.

T may be a prediction matrix.

A can be calculated as Equation 11 below using pseudo inverse.

However, since each matrix in Equation 11 is very large, application of Equation 11 may be practically impossible.

Matrix A can be calculated more easily by using the state sequence expression in Equation 12 below.

Here, Y may be the state matrix of X.

C may be an encoding matrix.

W may be a bias matrix.

Equation 13 below can represent singular value decomposition for Y. At this time, W can be assumed to be a zero matrix.

Comparing Equation 12 and Equation 13, SURP (300) can derive an optimal state vector according to Equation 14 below.

SURP can obtain temporal randomness, which is a temporal prediction error, according to Equation 15 below.

13A to 13C show three consecutive frames according to an example.

FIG. 13A shows the first frame of three consecutive frames according to an example.

FIG. 13B shows the second frame among three consecutive frames according to an example.

FIG. 13C shows the third frame among three consecutive frames according to an example.

Next, maps for three frames are shown.

FIG. 13D shows a temporal randomness map (TRM) for three consecutive frames according to an example.

FIG. 13E shows a SpatioTemporal Influence Map (STIM) for three consecutive frames according to an example.

FIG. 13F shows a Weighted SpatioTemporal Influence Map (WSTIM) for three consecutive frames according to an example.

TRM, STIM and WSTIM are described in detail below.

Refer again to Figure 5.

At step 545, SURP 300 may generate a Temporal Randomness Map (TRM).

TRM may be a map for the fourth frame, fifth frame, sixth frame, and seventh frame.

In TRM, bright pixels may indicate areas of large temporal randomness. An area with large temporal randomness may be an area with little visual recognition effect.

At step 550, SURP 300 may generate a SpatioTemporal Influence Map (STIM).

STIM may be a map for the fourth frame, fifth frame, sixth frame, and eighth frame.

When a person watches a video, the person can perceive both spatial and temporal visual stimuli depending on the person's visual characteristics. Therefore, spatial characteristics and temporal characteristics need to be simultaneously reflected in the feature vector.

To reflect these spatial and temporal characteristics, STIM can be defined as shown in Equation 16 below.

Y can represent a pixel at a specific location.

STIM may be a product of pixel values of corresponding pixels of SIM and TRM. SURP 300 may set the product of pixel values of pixels at the same location in SIM and TRM as the pixel value of STIM.

At step 555, SURP 300 may generate a Weighted SpatioTemporal Influence Map (WSTIM).

WSTIM may be a map for the fourth frame, fifth frame, sixth frame, and eighth frame.

WSTIM can emphasize relatively large stimuli from the overall stimuli.

SURP 300 may set the pixel value of the SIM divided by the average value of the pixels of the SIM as the pixel value of the WSTIM.

WSTIM can be effective when there is stimulation of fast movement of small objects within the frames.

In step 560, SURP 300 may perform the function of a feature representor. SURP 300 may output at least one result among TRIM, STIM, and WSTIM.

SURP 300 may calculate the average of at least one of TRIM, STIM, and WSTIM for each block of blocks of a predefined size in the frame. SURP 300 can output the averages of blocks in descending sorted order. For example, the predefined size may be 64x64.

The maps described above may be expressed in the form of a matrix representing data such as an image. A feature descriptor process is required to express the above-described maps in the form of vectors suitable for feature vectors.

SURP 300 may divide each map into blocks of a predefined size. For example, if the video is HD video, the block size may be 64x64. SURP 300 can calculate the average value of the pixels of each block for each block. Additionally, in order to avoid dependency on location, the SURP 300 may sort the blocks of the map in order from blocks with a large average value to blocks with a small average value.

Through this process, each map can be expressed in the form of a one-dimensional array such as a histogram.

If the video is HD video, the size of the one-dimensional array may be 507.

A one-dimensional array may be an array sorted in order of the average value of the block. Therefore, in general, large values can be placed at the front of a one-dimensional array, and smaller values can be placed towards the back. Therefore, even if only the first part of the values of the one-dimensional array are used as the feature vector, the difference in performance may not be large compared to if all of the values of the one-dimensional array are used. According to this characteristic, SURP 300 can use only the first part with large values as a feature vector, instead of using all of the values of the one-dimensional array as a feature vector. Here, the portion may be a predefined length or a predefined ratio. In a one-dimensional array, larger values can and should be at the front of the array. As an example, SURP 300 can use only the first 100 values of a one-dimensional array.

Training of the model

SURP (300) can use SVM. The machine learning of SURP (300) may be tasks 1) to 3) below.

1) SURP 300 can extract feature vectors from GOP.

2) SURP 300 can perform output prediction using the extracted feature vector as an input to SVM.

3) SURP (300) can generate an SVM so that the value output by output prediction has a value as close as possible to the ground truth.

Below, an example of learning a model to predict the degradation of picture quality when a 60 FPS video is changed to a 30 FPS video is described.

First, ground truth has already been obtained through subjective image quality evaluation of the homogeneous video dataset.

Next, SURP (300) can obtain each of TRM, STIM, and WSTIM for a video of 60 FPS. Additionally, SURP (300) can obtain each of TRM, STIM, and WSTIM under the internal assumption that the FPS of the video is 30. can be obtained. In addition, SURP (300) can convert the TRM, STIM, and WSTIM of the 60 FPS video into a one-dimensional array through a feature descriptor, and the SURP (300) can convert the TRM, STIM, and WSTIM of the 30 FPS video into a feature descriptor. It can be converted to a one-dimensional array.

Next, SURP 300 can perform training on SVM so that the feature vector of the 60 FPS video and the feature vector of the 30 FPS video have a difference corresponding to the ground truth.

Next, when learning is completed, SURP (300) can extract the feature vector of the 60 FPS GOP and the feature vector of the 30 FPS GOP, respectively, using the 60 FPS GOP, without further changing the internals of the SVM. there is.

SURP (300) can predict the degree of difference in picture quality between the picture quality of the 60 FPS GOP and the picture quality of the 30 FPS GOP using the feature vector of the 60 FPS GOP and the feature vector of the 30 FPS GOP, and the predicted picture quality difference Based on the degree of quality, the final image quality satisfaction level can be generated and output.

Here, 60 FPS and 30 FPS are merely examples. SURP 300 can generate and output image quality satisfaction for conversion between different FPS, similar to the generation of image quality satisfaction for 60 FPS and 30 FPS described above. For example, FPS can be changed from 60 to 15. Alternatively, FPS can be changed from 120 to 60.

To generate a feature vector of SVM, only one map among the above-described SIM, TRM, STIM, and WSTIM can be used, and two or more maps can be used simultaneously.

Additionally, multiple SVMs can be combined in a cascade. For example, first, SVM using TRM as a feature vector may be applied, and then SVM using the results of STM and the immediately preceding SVM as a feature vector may be applied. Additionally, finally, an SVM that uses the results of WSTIM and the previous SVM as a feature vector can be applied.

Determination of optimal frame rate

Below, a method of determining the optimal frame rate for each basic decision unit using SURP (300) and performing optimal encoding of the video using the determined optimal frame rate will be described.

SURP (300) can predict the minimum FPS of the basic decision unit that satisfies the conditions of minimum picture quality satisfaction or predefined picture quality satisfaction for the basic decision unit of the video, and the basic decision unit converted to minimum picture quality satisfaction. can be provided. SURP (300) can encode the converted basic decision unit with minimum image quality satisfaction.

Figure 14 shows image quality satisfaction by FPS according to an example.

As described above, the SURP 300 can predict the image quality satisfaction of a GOP converted to a specific FPS and output the predicted image quality satisfaction. By applying the above-described function, the SURP 300 can predict and output a plurality of picture quality satisfaction values of a plurality of FPS for each of the GOPs of the video.

By predicting a plurality of picture quality satisfaction levels of a plurality of FPSs for a GOP, a plurality of predicted picture quality satisfaction levels can be used for encoding.

In Figure 14, the first horizontal line represents multiple GOPs of the video. The first vertical line represents the converted FPS of the GOP. Additionally, for each GOP, the % satisfaction with picture quality in the converted FPS is shown.

In Figure 14, the source video may be 60 FPS.

SURP 300 may sequentially perform machine learning on multiple FPSs of the target video. First, when the FPS of the video is converted to 30, SURP (300) can perform machine learning. Additionally, when the FPS of the video is converted to 15, SURP (300) can perform machine learning. Additionally, when the FPS of the video is converted to 7.5, SURP (300) can perform machine learning. In other words, SURP 300 can perform machine learning on each of the FPS of the target video.

In a machine learning state, the SURP 300 can sequentially predict the image quality satisfaction of the GOP of the video for each FPS of the plurality of FPS. For example, when the FPS of the target video is 30, the image quality satisfaction of the three GOPs of the target video may be 80%, 90%, and 50%. When the FPS of the target video is 15, the image quality satisfaction of the three GOPs may be 70%, 60%, and 45%.

Through this prediction, SURP 300 can calculate GOP picture quality satisfaction levels for a plurality of FPS.

Additionally, the SURP 300 may consider that the image quality satisfaction of the GOP of the source video is 100%.

In Figure 14, the last row may represent a requirement. Requirements may indicate required picture quality or minimum picture quality satisfaction. In other words, the last row may indicate that the image quality satisfaction level should be 75% or higher.

Figure 15 shows the optimal frame rate determined for each GOP according to an example.

SURP (300) can determine the optimal FPS that meets the requirements for each GOP. Here, the optimal FPP may be the FPS of the GOP that is greater than the requirement value and has the lowest image quality satisfaction. That being said, the optimal FPS may be the lowest FPS that meets your requirements. If, among the FPS of the target video generated by conversion, there is no FPS that satisfies the picture quality satisfaction exceeding the requirements, the optimal FPS may be the FPS of the source video.

In Figure 15, the optimal frame rate determined for each of the plurality of GOPs is shown.

For example, according to FIG. 14, the image quality satisfaction rates for the plurality of FPSs of the first GOP may be 80%, 70%, and 50%. Among these, the minimum satisfactory image quality that exceeds the requirements may be 80 at 30 FPS. Accordingly, the optimal frame rate of the first GOP may be 30 FPS. The image quality satisfaction ratings for the plurality of FPSs of the second GOP may be 90%, 80%, and 50%. Among these, the minimum satisfactory image quality that exceeds the requirements may be 80 at 15 FPS. Accordingly, the optimal frame rate of the second GOP may be 15 FPS. The image quality satisfaction ratings for the plurality of FPSs of the third GOP may be 50%, 45%, and 40%. Since none of the picture quality satisfactions exceed the requirements, the third GOP cannot be converted. Therefore, the optimal frame rate of the third GOP may be 60 FPS, which is the FPS of the source video.

Each GOP of the plurality of GOPs may be converted at the optimal frame rate of each GOP, and the converted GOPs of the video may be encoded.

Figure 16 is a flowchart of a GOP encoding method using an optimal frame rate according to an example.

First, the input video is illustrated to be 60 FPS. GOPs of the input video may be processed by steps 1610, 1620, 1630, 1640, 1650, and 1660 sequentially.

In step 1610, the SURP 300 can convert the 60 FPS GOP into the 30 FPS GOP and calculate the image quality satisfaction of the 30 FPS GOP.

In step 1620, SURP 300 can convert the 60 FPS GOP into a 15 FPS GOP and calculate the image quality satisfaction of the 15 FPS GOP.

In step 1630, the SURP 300 can convert the 60 FPS GOP into the 7.5 FPS GOP and calculate the image quality satisfaction of the 7.5 FPS GOP.

At step 1640, SURP 300 may determine the optimal frame rate.

SURP 300 can receive the required image quality. The required image quality may represent the minimum image quality satisfaction described above.

SURP 300 can select the optimal GOP that satisfies the minimum picture quality satisfaction among GOPs changed to a predefined FPS. Here, the optimal GOP may be the GOP with the lowest FPS.

Alternatively, SURP 300 may determine the optimal frame rate that satisfies the minimum picture quality satisfaction among the FPS of the changed GOPs. The SURP 300 may select the GOP with the smallest image quality satisfaction or the FPS of the GOP that is greater than or equal to the minimum image quality satisfaction among the image quality satisfactions of the changed GOPs. The optimal frame rate may be the selected FPS or the FPS of the selected GOP.

At step 1650, SURP 300 may select the FPS of the optimal frame rate GOP. SURP (300) can convert the FPS of the GOP to the optimal frame rate. Alternatively, SURP 300 may select a GOP of the optimal frame rate among the GOP of the input video and the GOPs changed by steps 1610, 1620, and 1630.

In step 1660, SURP 300 may perform encoding of the GOP selected in step 1650.

17 is a flowchart of a method for determining an optimal frame rate according to an example.

Step 1640 described above with reference to FIG. 16 may include steps 1710, 1720, and 1730 below.

The SURP 300 may sequentially check whether the GOP satisfies the required picture quality for the changed GOPs, in the order from the high FPS GOP to the low FPS GOP. SURP 300 may select the GOP immediately preceding the GOP that does not meet the required image quality.

If the first GOP among the changed GOPs does not meet the required picture quality, the GOP of the source video may be selected. In other words, if none of the changed GOPs meets the required picture quality, the GOP of the source video may be selected.

If the last GOP among the changed GOPs does not meet the required image quality, the last GOP may be selected. In other words, if all GOPs among the changed GOPs meet the required picture quality, the GOP with the lowest FPS can be selected.

In step 1710, SURP 300 may check whether the GOP of 30 FPS meets the required picture quality. If the GOP of 30 FPS does not meet the required picture quality, SURP (300) can select 60 FPS as the optimal frame rate and provide the GOP of 60 FPS. Step 1720 may be performed if the GOP of 30 FPS meets the required picture quality.

In step 1720, SURP 300 may check whether the GOP of 15 FPS satisfies the required picture quality. If the GOP of 15 FPS does not meet the required picture quality, the SURP 300 can select 30 FPS as the optimal frame rate and provide the GOP of 30 FPS. Step 1730 may be performed if the GOP of 15 FPS satisfies the required picture quality.

In step 1730, SURP 300 may check whether the GOP of 7.5 FPS satisfies the required picture quality. If the GOP of 7.5 FPS does not meet the required picture quality, SURP (300) can select 15 FPS as the optimal frame rate and provide a GOP of 15 FPS. If the GOP of 7.5 FPS satisfies the required image quality, the SURP 300 can select 7.5 FPS as the optimal frame rate and provide the GOP of 7.5 FPS.

Figure 18 shows a hierarchical structure of frames with temporal identifiers according to an example.

In Figure 18, nine frames are shown. The nine frames may be “Frame 0” to “Frame 8.”

Each frame may have a temporal identifier (TI). Additionally, each frame may be an I frame, P frame, or B frame.

As described above, when a GOP is converted into a plurality of FPS through the SURP 300, the image quality satisfaction of the plurality of converted GOPs can be predicted. If the bitstream generated by video encoding includes picture quality satisfaction information related to picture quality satisfaction, an improved bitstream supporting optional temporal scalability (TS) can be provided while maintaining picture quality satisfaction. there is.

In the High Efficiency Video Coding (HEVC) standard, the TI of each frame can be transmitted through a Network Abstraction Layer (NAL) header. For example, TI information “nuh_temporal_id_plus1” in the NAL header may indicate the TI of the frame.

A TS that provides picture quality satisfaction information can be provided through temporal identifier information in the NAL header.

When the size of the GOP is 8 and the hierarchical prediction structure of FIG. 18 is used, if odd-numbered frames with a TI value of 3 are removed from the GOP, the FPS of the GOP may be 1/2. In other words, the FPS of GOP can be changed from 60 to 30.

Additionally, if “Frame 2” and “Frame 6” with a TI value of 2 are removed, the FPS of the GOP may be 1/4. In other words, the FPS of GOP can be changed from 60 to 15.

In provision of TS, the video may have a fixed FPS if only TI is used. For example, in the playback of a 60 FPS video, if a frame with a TI of 3 is excluded from playback, a fixed 30 FPS may be applied to the entire video. In this way, when TS, which only considers TI, is used, if a fast-moving object exists in the middle of the video, a significant deterioration in image quality that the viewer can perceive for the fast-moving object may occur.

Improved temporal scalability

Below, an encoding method and a decoding method that provide improved temporal scalability using image quality satisfaction information generated by the SURP 300 will be described.

Using the SURP 300 described above, TS can be applied for each GOP. In other words, by using the SURP 300 described above, GOP picture quality satisfaction levels can be predicted for each FPS of a plurality of FPS. By using GOP picture quality predictions, TS can be provided while maintaining picture quality.

SURP 300 may determine, for each TI, image quality satisfaction information corresponding to each TI. Image quality satisfaction information may indicate image quality satisfaction. Here, picture quality satisfaction information corresponding to a specific TI may indicate picture quality satisfaction with respect to picture quality when frames of more than TI(s) of a specific TI are removed from encoding, decoding, or playback.

For example, “frame x” may be included in or excluded from playback depending on what FPS the GOP is being played at. Depending on the FPS, “frame x” may be excluded from decoding or playback by the TS. In this case, the maximum FPS among the FPSs in which “frame Satisfaction with image quality may be z.

Additionally, in GOP, frames with the same TI may have the same picture quality satisfaction information in common. Therefore, depending on the required image quality satisfaction, the decoding device 2700, which will be described later, can adaptively select whether to reproduce frames with a specific TI.

For example, if the required picture quality satisfaction is Z, it may indicate that frames of TI whose corresponding picture quality satisfaction is Z or less should be included in playback. Additionally, this decision may indicate that when the required image quality satisfaction is Z, frames of TI corresponding to an image quality satisfaction greater than Z may be excluded from playback.

Picture quality satisfaction corresponding to TI may be included in the SEI message of the access unit of the frame and may be included in the NAL unit header. At this time, the SEI message or NAL unit header may directly include the value of picture quality satisfaction information. Alternatively, the SEI message or NAL unit header may include the index value of the picture quality satisfaction table.

In GOP, frames with the same TI may have the same picture quality satisfaction information. Therefore, in the GOP, for each TI, picture quality satisfaction information can be transmitted from the encoding device 2400 to the decoding device 2700 through SEI only in the first frame of each TI. Alternatively, in the first frame of the GOP, picture quality satisfaction information for all TIs may be transmitted at once.

Figure 19 shows a message including image quality satisfaction according to an example.

In FIG. 19, the SEI message of the access unit of the frame may include picture quality satisfaction information. In other words, picture quality satisfaction information can be provided through a Supplemental Enhancement Information (SEI) message of the access unit of the frame.

In FIG. 19, “SEI_Temporal_ID_SURP” may represent data used to provide image quality satisfaction information. “Surp_value” may indicate image quality satisfaction information.

For example, a value of “Surp_value” of 70 may indicate that the image quality satisfaction is “70%”.

Figure 20 shows a message including an index of an image quality satisfaction table according to an example.

In FIG. 20, the SEI message of the access unit of the frame may include picture quality satisfaction information. In other words, picture quality satisfaction information can be provided through a Supplemental Enhancement Information (SEI) message of the access unit of the frame.

In FIG. 20, “SEI_Temporal_ID_SURP” may represent data used to provide image quality satisfaction information. “Surp_value_idx” may be an index of the picture quality satisfaction table.

Picture quality satisfaction values used in video or GOP may be predefined as a table in the picture quality satisfaction table. For example, the picture quality satisfaction table may have values of {90, 60, 30}. These values may indicate that image quality satisfaction levels of 90, 60, and 30 are used. If the value of "Surp_value_idx" is 0, it may indicate that the value of index 0 in the image quality satisfaction table, that is, 90 or 90%, is the value of image quality satisfaction. If the value of "Surp_value_idx" is 1, it may indicate that the value of index 1 in the image quality satisfaction table, that is, 60 or 60%, is the value of image quality satisfaction.

Figure 21 shows a message including all picture quality satisfaction levels according to an example.

In FIG. 21, one SEI message may include picture quality satisfaction information of all TIs of all frames of the GOP.

In FIG. 21, “Max_Temporl_ID” may indicate the maximum number of TIs. “Temoporal ID[i]” may indicate the index of TI whose value is i or i1. Alternatively, “Temoporal ID[i]” may indicate image quality satisfaction information of the ith TI.

The image quality satisfaction information may represent the image quality satisfaction value itself or an index of an image quality satisfaction table. In FIG. 21, “Temoporal ID[i]” is illustrated as indicating an index of an image quality satisfaction table.

Figure 22 illustrates a method of generating image quality satisfaction information according to an example.

Figure 22 is a flowchart of a GOP encoding method including picture quality satisfaction information according to an example.

First, the input video is illustrated to be 60 FPS. GOPs of input video may be processed sequentially in steps 2210, 2220, 2230, 2240, and 2250.

In step 2210, the SURP 300 can convert the 60 FPS GOP to the 30 FPS GOP and calculate the image quality satisfaction of the 30 FPS GOP.

In step 2220, the SURP 300 can convert the 60 FPS GOP into the 15 FPS GOP and calculate the image quality satisfaction of the 15 FPS GOP.

In step 2230, the SURP 300 can convert the 60 FPS GOP into the 7.5 FPS GOP and calculate the image quality satisfaction of the 7.5 FPS GOP.

In step 2240, encoding of picture quality satisfaction information for frames of the SURP 300 GOP may be performed.

In step 2250, SURP 300 may perform encoding of the GOP. Encoding of multiple frames of GOPs of SURP 300 can be performed.

In the embodiment of Figure 22, all frames of GOPs may be encoded. Which frame to reproduce among all frames can be determined at the encoding stage according to the minimum image quality satisfaction.

Figure 23a shows a configuration for maintaining image quality satisfaction of 75% or more according to an example.

In Figure 23A, a square may represent a frame. In FIG. 23A, the first GOP may include “Frame 0” to “Frame 7.” The second GOP may include “Frame 8” to “Frame 15.” “TI” inside the rectangle may represent the value of the temporal identifier of the frame. “I,” “P,” and “B” inside the square may indicate the type of frame. The numbers above or below the square can indicate the level of satisfaction with the picture quality of the frame.

The SURP 300 can convert each GOP of video to the minimum FPS to satisfy minimum picture quality satisfaction, and encode each converted GOP. In this case, video encoded at the minimum bit rate required to meet minimum picture quality satisfaction can be transmitted. This method can be advantageous in terms of transmission.

In Figure 23a, a case in which TS is applied to ensure image quality satisfaction of at least 75% is illustrated.

In FIG. 23A, frames within the dotted line may represent frames that are not encoded or transmitted. In Figure 23A, in the first GOP, frames may be transmitted at 30 FPS such that a minimum picture quality satisfaction of 75% is met. Additionally, in the second GOP, frames may be transmitted at 15 FPS so that a minimum image quality satisfaction of 75% is met.

SURP 300 may include image quality satisfaction information of TIs in the bitstream. The decoding device 2700 may determine the minimum FPS for satisfying the minimum image quality satisfaction by referring to the image quality satisfaction information, and may receive frames required for the determined FPS from the encoding device. The decoder can reproduce the received frames.

The decoding device 2700 may determine the minimum FPS for satisfying the minimum picture quality satisfaction for the GOP by referring to the picture quality satisfaction information. The decoding device 2700 may perform decoding of frames required for the determined FPS. Here, the frames may be frames of a GOP. Frames required for the determined FPS can be received from the encoding device. The decoder can reproduce the received frames.

The decoding device 2700 may selectively obtain from the encoding device 2400 only the frames determined to be decoded among video or GOP frames through TS.

The decoding device 2700 may selectively decode frames having a picture quality satisfaction level below the minimum picture quality satisfaction level among the frames of the GOP. By excluding frames with a picture quality satisfaction greater than the minimum picture quality satisfaction from decoding, the decoding device 2700 can efficiently transmit and play video without degrading the picture quality perceived by the viewer.

Figure 23b shows a configuration for determining whether to change FPS based on 75% image quality satisfaction according to an example.

In Figure 23b, a square may represent a frame. In FIG. 23B, the first GOP may include “Frame 0” to “Frame 7.” The second GOP may include “Frame 8” to “Frame 15.” “TI” inside the rectangle may represent the value of the temporal identifier of the frame. “I,” “P,” and “B” inside the square may indicate the type of frame. The numbers above or below the square can indicate the level of satisfaction with the picture quality of the frame.

SURP (300) basically changes the FPS of the GOP to 30, but if the image quality satisfaction is less than 75%, the FPS of the source video can be maintained without changing the FPS of the GOP. In other words, SURP 300 can adaptively use TS based on minimum image quality satisfaction. This function may be especially required in environments where image quality is important.

In Figure 23b, the first GOP can be converted to 30 FPS and encoded and transmitted, and the second GOP can be encoded and transmitted while maintaining 60 FPS.

Figure 24 is a structural diagram of an encoding device according to an embodiment.

The encoding device 2400 may include a control unit 2410, an encoding unit 2420, and a communication unit 2430.

The control unit 2410 may correspond to the SURP 300 described above. For example, the functions of the SURP 300 described above may be performed by the control unit 2410. Alternatively, the control unit 2410 may perform SURP (300).

The control unit 2410 can generate selection information about frames of a video.

The selection information may correspond to the above-described image quality satisfaction information. Alternatively, the selection information may include image quality satisfaction information.

Selection information may be related to the percentage of people who do not notice a decrease in video quality even if a frame is excluded from the encoding of the video.

Alternatively, the selection information may be related to the percentage of people who do not notice a decrease in the image quality of the played video even if the playback of the video is set to exclude frames from decoding.

The selection information may be plural. A plurality of selection information may be calculated for each FPS of video or basic decision unit playback. The control unit 2410 may calculate selection information for a plurality of FPS.

The percentage of people who do not recognize the decrease in video quality can be calculated by machine learning of the control unit 2410. As described above, the proportion of people who do not recognize the deterioration in video quality can be determined based on the frame or the feature vector of the basic decision unit including the frame.

The selection information may be commonly applied to other frames having the same TI as the TI of the frame among a plurality of frames including the frame. The plurality of frames may be frames of the basic decision unit. Alternatively, the plurality of frames may be frames of a GOP.

The control unit 2410 may perform encoding of the video based on the generated selection information.

For example, the control unit 2410 can perform encoding of selection information and include the encoded selection information in the bitstream of the encoded video.

For example, the control unit 2410 may select a frame to be encoded among frames of a video based on selection information.

The encoder 2420 may correspond to the above-described encoding device 100. For example, the above-described function of the encoding device 100 may be performed by the encoding unit 2420. Alternatively, the encoder 2420 may include the encoder 100.

The encoder 2420 can encode frames of a video.

For example, the encoder 2420 may encode all frames of a video.

For example, the encoder 2420 may encode a frame selected by the control unit 2410.

The communication unit 2430 may transmit the generated bitstream to the decoding device 2700.

The bitstream may include encoded video information and may include encoded selection information.

The functions and operations of the control unit 2410, the encoder 2420, and the communication unit 2430 are described in detail below.

Figure 25 is a flowchart of an encoding method according to an embodiment.

In step 2510, the controller 2410 may generate selection information for a frame of a video.

In step 2520, the controller 2410 may encode the video based on the selection information.

The encoder 2420 can encode frames of a video. The encoder 2420 can generate a bitstream by encoding frames of a video. A bitstream generated by encoding a video may include selection information.

In step 2530, the communication unit 2430 may transmit a bitstream including information about the encoded video to the decoding device 2700.

Figure 26 is a flowchart of a method for encoding a video according to an example.

Step 2520 described above with reference to FIG. 25 may include steps 2610 and 2620 below.

In step 2610, the control unit 2410 may determine whether to encode the frame based on selection information about the frame.

In step 2620, when the encoding of the frame is determined, the encoder 2420 may perform encoding of the frame.

When the encoding of the frame is determined, the control unit 2410 may request the encoding unit 2420 to encode the frame.

The determination of whether to encode a frame may be commonly applied to other frames having the same TI as the TI of the frame among a plurality of frames including the frame. Here, the plurality of frames may be frames of the basic decision unit. Alternatively, the plurality of frames may be frames of a GOP.

Figure 27 is a structural diagram of a decoding device according to an embodiment.

The decoding device 2700 may include a control unit 2710, a decoding unit 2720, and a communication unit 2730.

The communication unit 2730 may receive a bitstream from the encoding device 2400.

The bitstream may include information about encoded video. Information on the encoded video may include information on the encoded frame. The bitstream may include frame selection information.

The control unit 2710 may determine whether to decode a frame based on frame selection information.

The control unit 2710 may perform functions applicable to decoding among the functions of the SURP 300 described above. Alternatively, the control unit 2710 may include at least a portion of the SURP (300).

The decoding unit 2720 may correspond to the decoding device 200 described above. For example, the functions of the above-described decoding device 200 may be performed by the decoding unit 2720. Alternatively, the decoding unit 2720 may include the decoding device 200.

If decoding of the frame is determined, the decoder 2720 may perform decoding of the frame.

For example, the decoder 2720 may perform decoding of all frames of a video.

For example, the decoding unit 2720 may perform decoding of a frame selected by the control unit 2710.

The functions and operations of the control unit 2710, decoding unit 2720, and communication unit 2730 are described in detail below.

Figure 28 is a flowchart of a decryption method according to an embodiment.

In step 2810, the communication unit 2730 may receive a bit stream. The communication unit 2730 can receive encoded frame information.

In step 2820, the control unit 2710 may determine whether to decode the frame based on frame selection information.

The above determination may be performed for each frame of a plurality of frames of video. Here, the plurality of frames may be frames of the basic decision unit. Alternatively, the plurality of frames may be frames of a GOP.

The selection information may correspond to the above-described image quality satisfaction information. Alternatively, the selection information may include image quality satisfaction information.

The plurality of frames image quality satisfaction information selection information may be related to the proportion of people who do not recognize the deterioration in the image quality of the video even if playback of the video including the frame is determined so that the frame is excluded from decoding.

Alternatively, the selection information may be related to the proportion of people who do not recognize the deterioration in the image quality of the video even if the FPS of playback of the video including the frame is determined so that the frame is excluded from decoding.

Selection information may be included within the SEI for the frame.

Whether or not to decode a frame may be determined based on comparison between the value of selection information and the value of reproduction information.

Playback information may be information related to playback of a video including frames. Playback information may correspond to minimum image quality satisfaction. Alternatively, the reproduction information may include the minimum image quality satisfaction level.

The playback information may be information related to the FPS of playback of a video including frames.

For example, if the value of the selection information is greater than the value of the reproduction information, the control unit 2710 may decide not to perform decoding on the frame. Alternatively, if the value of the selection information is greater than that of the reproduction information, the control unit 2710 may determine whether to perform decoding on the frame.

The decision may be commonly applied to other frames having the same TI as the TI of the frame among a plurality of frames including the frame. The plurality of frames may be frames of a basic prediction unit. Alternatively, the plurality of frames may be frames of a GOP.

Decoding device 2700: A plurality of frames. The random decoding device 2700 determines a random 0 to be decoded among all frames of the video.

In step 2830, when decoding of the frame is determined, the decoder 2720 may perform decoding of the frame for which decoding has been determined.

Decryption device (2700)

In the above-described embodiments, the methods are described based on flowcharts as a series of steps or units, but the present invention is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. You can. Additionally, a person of ordinary skill in the art will recognize that the steps shown in the flowchart are not exclusive and that other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

Embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CDROMs and DVDs, magnetooptical media such as floptical disks, and ROM. It includes specially configured hardware devices to store and execute program instructions, such as RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

Claims (20)

determining whether to decode the frame based on perceived image quality information about the perceived image quality of the frame; and
If decoding of the frame is determined, performing decoding of the frame
Including,
The perceived picture quality information is related to the proportion of people who do not recognize the deterioration in picture quality of the video even if playback of the video including the frame is set to exclude the frame from decoding,
A video decoding method where the ratio is determined based on machine learning. According to paragraph 1,
The decision is performed for each frame of the plurality of frames,
The plurality of frames are frames of a group of pictures (GOP),
The machine learning is a video decoding method that uses subjective image quality evaluation using homogeneous video composed of GOPs with similar characteristics. determining whether to decode the frame based on perceived image quality information about the perceived image quality of the frame; and
If decoding of the frame is determined, performing decoding of the frame
Including,
The perceived picture quality information is related to the proportion of people who do not recognize the deterioration in picture quality of the video even if playback of the video including the frame is set to exclude the frame from decoding,
The ratio is determined based on the feature vector of the frame,
A video decoding method in which the feature vector is extracted by a feature vector extraction method that reflects human visual characteristics. According to paragraph 3,
A video decoding method wherein the human visual characteristics are spatial masking effect, temporal masking effect, salient area, or contrast sensitivity. According to paragraph 1,
A video decoding method in which the perceived image quality is measured using a measure that models the perceived image quality. According to clause 5,
A video decoding method where the measure is structural similarity (SSIM) or video quality metric (VQM). According to paragraph 1,
If the value of the perceived picture quality information is greater than the value of the reproduction information, it is determined that the decoding is not performed on the frame,
The playback information is information related to playback of the video including the frame. According to paragraph 1,
The perceived image quality information is related to the proportion of people who do not recognize the deterioration in the image quality of the video even if the frames per second (FPS) of the video including the frame are determined so that the frame is excluded from decoding,
The machine learning is a video decoding method used to measure the degree of image quality degradation caused by changes in the FPS. According to paragraph 1,
A video decoding method in which the determination of whether to decode a video is commonly applied to other frames having the same temporal identifier as that of the frame among a plurality of frames including the frame. a control unit that determines whether or not to decode the frame based on perceived image quality information about the perceived image quality of the frame; and
A decoding unit that performs decoding of the frame when decoding of the frame is determined.
Including,
The perceived picture quality information is related to the proportion of people who do not recognize the deterioration in picture quality of the video even if playback of the video including the frame is set to exclude the frame from decoding,
The ratio is determined based on machine learning or feature vectors of the frame,
The perceived image quality is measured using a measure that models the perceived image quality,
The measure is Structural SIMilarity (SSIM) or Video Quality Metric (VQM),
A video decoding device in which the feature vector is extracted by a feature vector extraction method that reflects human visual characteristics. Generating perceived image quality information for a frame of a video; and
Performing encoding of the video based on the perceived image quality information
Including,
The perceived picture quality information is related to the proportion of people who do not recognize the deterioration in picture quality of the video even if playback of the video including the frame is set to exclude the frame from decoding,
Video encoding method where the ratio is determined based on machine learning According to clause 11,
The step of encoding the video is,
determining whether to encode the frame based on the perceived picture quality information for the frame; and
When encoding of the frame is determined, performing encoding of the frame
A video encoding method including. According to clause 12,
A video encoding method in which the determination of whether to encode a video is commonly applied to other frames having the same temporal identifier as that of the frame among a plurality of frames including the frame. According to clause 11,
A video encoding method in which the perceived image quality is measured using a measure that models the perceived image quality. According to clause 14,
A video encoding method where the measure is structural similarity (SSIM) or video quality metric (VQM). Generating perceived image quality information for a frame of a video; and
Performing encoding of the video based on the perceived image quality information
Including,
The perceived picture quality information is related to the proportion of people who do not recognize the deterioration in picture quality of the video even if playback of the video including the frame is set to exclude the frame from decoding,
The ratio is determined based on the feature vector of the frame,
A video encoding method in which the feature vector is extracted by a feature vector extraction method that reflects human visual characteristics. According to clause 11,
The perceived image quality information is plural,
A video encoding method in which the plurality of perceived image quality information is calculated for each frame per second (FPS) of video playback. According to clause 11,
A video encoding method wherein the bitstream generated by encoding the video includes the perceived image quality information. According to clause 11,
A video encoding method in which the perceived image quality information is commonly applied to other frames having the same temporal identifier as that of the frame among a plurality of frames including the frame. According to clause 16,
A video encoding method in which the human visual characteristics are spatial masking effect, temporal masking effect, salient area, or contrast sensitivity.

Images