KR20170042235A - Method and apparatus for adaptive encoding and decoding based on image quality - Google Patents

Abstract

A method and apparatus for adaptive encoding and decoding based on image quality are provided. An encoding apparatus can determine an optimal frame per second (FPS) for a video and can encode the video according to the determined FPS. Further, the encoding apparatus can provide improved temporal scalability. The decoding apparatus can select a frame to be reproduced among the frames of the video according to a required minimum image quality satisfaction. Through the selection of the frame, the decoding apparatus can provide improved temporal scalability. So, the level of the deterioration of the image quality can be predicted.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for adaptive encoding and decoding based on picture quality,

The present invention relates to a decoding method, a decoding apparatus, a coding method, and an encoding apparatus, and more particularly, to a method and apparatus for adaptively performing encoding and decoding based on image quality of an image.

With the continuous development of the information and telecommunication industry, broadcasting service with HD (High Definition) resolution spread worldwide. With this proliferation, many users become accustomed to high resolution and high quality images and / or video.

In order to meet the users' demand for high image quality, many organizations are spurring development on next generation image devices. In addition to High Definition TV (HDTV) and Full HD (FHD) TVs, users' interest in ultra high definition (UHD) TVs with more than four times the resolution of FHD TVs As the interest increases, there is a need for image encoding / decoding techniques for images with higher resolution and image quality.

For new video services such as UHD, the need for high frame rate (HFR) video is increasing. For example, the high frame rate may be a frame refresh rate of more than a frame per second (FPS).

However, in order to provide such HFR video, there is a problem that the amount of video data increases. In addition, as the amount of video data increases, problems of cost and technical problems may occur in transmission and storage of video.

Fortunately, due to the nature of human vision, HFR is not required in all situations, for example, for most people, the video quality difference or picture quality between 30 FPS video and 60 FPS video, I do not feel a drop.

That is, depending on the content of the video, even if the FPS is higher than a certain threshold value, even if the FPS is higher, most people may hardly notice a difference in image quality depending on the human cognitive characteristic.

One embodiment can provide a method and apparatus for predicting the degree of image quality degradation that occurs when converting HFR video to low frame rate video.

One embodiment can provide a method and apparatus for reducing the bit rate required for encoding through prediction of the degree of degradation in image quality.

In one embodiment, when the frame rate of a moving image is lowered through temporal scalability (TS), a method of minimizing the deterioration of image quality by using information generated through prediction of the degree of degradation of image quality And apparatus.

One embodiment of the present invention can provide a method and apparatus for minimizing image quality deterioration by considering information relating to image quality degradation in applying temporal scalability.

The method comprising the steps of: determining, on a side, whether to decode the frame based on frame selection information; And performing decoding of the frame when decoding of the frame is determined.

The determination may be performed for each frame of the plurality of frames.

The plurality of frames may be frames of a group of pictures (GOP).

The selection information may be related to a ratio of a person who is not aware of a degradation in the picture quality of the moving picture even though the reproduction of the moving picture including the frame is set so that the frame is excluded from decoding.

The selection information may be related to a ratio of persons who do not perceive degradation in the picture quality of the moving picture even if frames per second (FPS) of the moving picture reproduction including the frame is determined so that the frame is excluded from decoding .

The decoding of the frame may be determined based on a comparison between the value of the selection information and the value of the reproduction information.

The reproduction information may be information related to reproduction of a moving picture including the frame.

The playback information may be information related to frames per second (FPS) of playback of a moving image including the frame.

If the value of the selection information is larger than the value of the reproduction information, it can be determined that the decoding is not performed for the frame.

The reproduction information may be information related to reproduction of a moving picture including the frame.

The selection information may be included in Supplemental Enhancement Information (SEI) for the frame.

The determination of the decryption may be commonly applied to other frames having the same temporal identifier as the temporal identifier of the frame among the plurality of frames including the frame.

A control unit for performing a determination as to whether to decode the frame based on the selection information of the frame, on the other side; And a decoding unit that decodes the frame when decoding of the frame is determined.

Generating a selection information for a frame of the moving picture; And encoding the moving picture based on the selection information.

Wherein the encoding of the moving picture comprises: determining whether to encode the frame based on the selection information for the frame; And performing encoding of the frame when the encoding of the frame is determined.

The determination as to whether to perform the encoding may be commonly applied to other frames having the same temporal identifier as the temporal identifier of the frame among the plurality of frames including the frame.

The selection information may be related to a ratio of persons who are not aware of the degradation of the picture quality of the moving picture even if the frame is excluded from the coding of the video.

The ratio can be calculated by machine learning.

The ratio can be determined based on the feature vector of the frame.

The selection information may be plural.

The plurality of selection information may be calculated for each frames per second (FPS) of reproduction of a moving image.

The bit stream generated by encoding the moving picture may include the selection information.

The selection information may be commonly applied to other frames having the same temporal identifier as the temporal identifier of the frame among a plurality of frames including the frame.

The selection information may be related to a ratio of persons who are not aware of the degradation of the picture quality of the moving picture to be played even if the moving picture is set to be reproduced so that the frame is excluded from decoding.

There is provided a method and apparatus for predicting how much deteriorated picture quality of a unit section when a frame rate of a unit section is lowered for each unit interval of video of the HFR.

A method of reducing the data rate of a video by reducing the frame rate of the unit section while maintaining the quality of the video of the HFR per unit section, the image quality difference range input by the user or within the predetermined allowable image quality difference range, and Device is provided. For example, a predetermined acceptable range of image quality differences may be such that 80% of people do not feel the image quality difference.

There is provided a method and an apparatus for considering image quality degradation prediction information that predicts how much image quality degrades in application of TS.

There is provided a method and an apparatus for applying a TS only to a unit section corresponding to an image quality degradation range input by a user or a predetermined image quality degradation range by considering image quality degradation prediction information.

1 is a block diagram illustrating a configuration of an encoding apparatus to which the present invention is applied.
2 is a block diagram illustrating a configuration of a decoding apparatus to which the present invention is applied.
Figure 3 illustrates the operation of SURP according to one embodiment.
4 shows a picture quality evaluation result according to an example.
Figure 5 illustrates the procedure for extracting a feature vector according to one embodiment.
6 shows a prediction model for spatial arbitrary measurement according to an example.
7 shows an SRM according to an example.
Figure 8 shows an SM according to an example.
9 shows an SCM according to an example.
10 shows a VSM according to an example.
Fig. 11 shows a SIM according to Fig.
FIG. 12 shows a relationship between a frame used for generation of the temporal prediction model and a prediction target frame according to an example.
13A shows a first one of three consecutive frames according to an example.
13B shows a second one of three consecutive frames according to an example.
13C shows a third frame of three consecutive frames according to an example.
13D shows a Temporal Randomness Map (TRM) for three consecutive frames according to an example.
FIG. 13E shows a Spatio Temporal Influence Map (STIM) for three consecutive frames according to an example.
FIG. 13F shows a weighted Spatio Temporal Influence Map (WSTIM) for three consecutive frames according to an example.
FIG. 14 shows image quality satisfaction by FPS according to an example.
FIG. 15 shows an optimum frame rate determined for each GOP according to an example.
16 is a flowchart of a method of encoding a GOP using an optimum frame rate according to an example.
17 is a flowchart of a method of determining an optimal frame rate according to an example.
FIG. 18 shows a hierarchical structure of frames having temporal identifiers according to an example.
FIG. 19 shows a message including the image quality satisfaction according to an example.
FIG. 20 shows a message including an index of a picture quality satisfaction table according to an example.
FIG. 21 shows a message including the entirety of image quality satisfaction according to an example.
FIG. 22 shows a method of generating image quality satisfaction information according to an example.
FIG. 23A shows a configuration for maintaining image quality satisfaction of 75% or more according to an example.
FIG. 23B shows a configuration for determining whether to change the FPS based on 75% picture quality satisfaction according to an example.
24 is a structural diagram of an encoding apparatus according to an embodiment.
25 is a flowchart of a coding method according to an embodiment.
26 is a flowchart of a method of performing motion picture coding according to an example.
27 is a structural diagram of a decoding apparatus according to an embodiment.
28 is a flowchart of a decoding method according to an embodiment.

The following detailed description of exemplary embodiments refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the location or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the embodiments. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the exemplary embodiments is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained.

In the drawings, like reference numerals refer to the same or similar functions throughout the several views. The shape and size of the elements in the figures may be exaggerated for clarity.

When it is mentioned that a component is "connected" or "connected" to another component, the two components may be directly connected or connected to each other, It is to be understood that other components may be present in the middle of the components. Also, in the exemplary embodiments, the description of "comprising" a specific configuration does not exclude a configuration other than the specific configuration, and the additional configuration is not limited to the implementation of the exemplary embodiments or the technical idea of the exemplary embodiments. Range. ≪ / RTI >

The terms first and second, etc. may be used to describe various components, but the components should not be limited by the terms above. The above terms are used to distinguish one component from another. For example, without departing from the scope of the right, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

In addition, the components shown in the embodiments are shown independently to represent different characteristic functions, which does not mean that each component is composed of separate hardware or one software constituent unit. That is, each component is listed as each component for convenience of explanation. For example, at least two of the components may be combined into a single component. Also, one component can be divided into a plurality of components. The integrated embodiments and the separate embodiments of each of these components are also included in the scope of the right without departing from the essence.

Also, some components are not essential components to perform essential functions, but may be optional components only to improve performance. Embodiments may be implemented only with components that are essential to implementing the essentials of the embodiments, and structures within which the optional components are excluded, such as, for example, components used only for performance enhancement, are also included in the scope of the right.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to facilitate embodiments of the present invention by those skilled in the art. In the following description of the embodiments, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear.

Hereinafter, an image may refer to one picture constituting a video, and may also represent the video itself. For example, "encoding and / or decoding of an image" may mean "encoding and / or decoding of video ", which means" encoding and / or decoding of one of the images constituting a video " It is possible.

In the following, the numbers used in the embodiments are merely examples, and may be replaced with values other than the values specified in the embodiments.

In an embodiment, the information, operations, and functions applied to the encoding may also be applied to decoding in a manner corresponding to the encoding.

If the frame rate can be adaptively and automatically adjusted using the human visual characteristics as described above, the data amount of the video can be reduced without degrading the picture quality in terms of perceptual quality for a large number of people have. If the human visual characteristics are applied to the HFR video and the video can be converted to the optimal screen refresh rate for each section, the increase of the data amount of the video can be minimized. Accordingly, a cost problem and a technical problem that occur in transmission and storage of HFR video can be solved.

For adaptive and automatic adjustment of the frame rate, it is necessary to be able to predict exactly how much the image quality difference occurs when changing the video from HFR to a low frame rate. This prediction technique can be usefully used in a preprocessing process, a coding process, a transmission process, and / or a decoding process of a video.

However, according to the conventional technique, the image quality difference which occurs when the video of the HFR is converted at a low frame rate can not be accurately predicted.

In addition, there is a temporal scalability (TS) technique as a conventional technique related to adjustment of image quality. The TS technique uniformly adjusts the frame rate of the video without considering how the quality of the video changes when TS is applied to the video. Therefore, when TS is applied to video, there occurs a problem that the video quality may be significantly deteriorated.

In the following embodiments, a method of predicting image quality satisfaction, a method of changing video to an optimal low frame rate through prediction of image quality satisfaction, and a method of using information generated by predicting image quality satisfaction in TS Is explained.

1) Method of predicting image quality satisfaction: A picture quality predictor that predicts the degree of image quality difference that occurs when changing HFR to a low frame rate is described.

2) A method for changing video to an optimal low frame rate through prediction of image quality satisfaction: An encoder that changes the video of a fixed HFR to an optimal low frame rate and encodes the fixed HFR video by using a picture quality satisfaction predictor is explained.

3) Method of utilizing information generated by predicting image quality satisfaction in TS: A video encoded stream generated by encoding a video may include image quality satisfaction information generated by a picture quality satisfaction predictor. The quality information may be used for TS during the transmission or decryption of the video encoding stream. An improved TS that has not been possible in the past due to image quality satisfaction information can be provided. It can be considered how the image quality of the video changes when the TS is applied by the image quality satisfaction information. This consideration can provide an improved TS that minimizes degradation in image quality.

First, terms used in the embodiments will be described.

Unit: A "unit" can represent a unit of encoding and decoding of an image. The meanings of units and blocks may be the same. In addition, the terms "unit" and "block"

 The unit (or block) may be an MxN array of samples. M and N may be positive integers, respectively. A unit can often be an array of two-dimensional samples. The sample may be a pixel or a pixel value.

 In coding and decoding of an image, a unit may be an area generated by division of one image. One image may be divided into a plurality of units. In the coding and decoding of the image, predetermined processing on the unit may be performed depending on the type of the unit. Depending on the function, the type of the unit can be classified into a macro unit, a coding unit (CU), a prediction unit (PU), and a transform unit (TU). One unit may be further subdivided into smaller units having a smaller size than the unit.

 The block partitioning information may include information on the depth of the unit. The depth information may indicate the number and / or the number of times the unit is divided.

 One unit can be divided hierarchically into a plurality of subunits having depth information based on a tree structure. That is to say, the unit and the lower unit generated by the division of the unit can correspond to the node and the child node of the node, respectively. Each divided subunit may have depth information. Since the depth information of the unit indicates the number and / or degree of division of the unit, the division information of the lower unit may include information on the size of the lower unit.

 In the tree structure, the top node can correspond to the first unit that has not been partitioned. The superordinate node may be referred to as a root node. Also, the uppermost node may have a minimum depth value. At this time, the uppermost node can have a level 0 depth.

 A node with a depth of level 1 can represent a unit created as the first unit is once partitioned. A node with a depth of level 2 may represent a unit created as the first unit is divided twice.

 A node with a depth of level n may represent a unit created as the first unit is divided n times.

 The leaf node may be the lowest node, and may be a node that can not be further divided. The depth of the leaf node may be the maximum level. For example, the default value of the maximum level may be three.

Transform Unit: A transform unit may be a base unit in residual signal coding and / or residual signal decoding such as transform, inverse transform, quantization, inverse quantization, transform coefficient coding, and transform coefficient decoding . One conversion unit can be divided into a plurality of conversion units having a smaller size.

Prediction Unit: A prediction unit may be a basic unit in performing prediction or compensation. The prediction unit may be partitioned into a plurality of partitions. Multiple partitions may also be a base unit in performing prediction or compensation. The partition generated by the division of the prediction unit may also be a prediction unit.

Reconstructed Neighbor Unit: The reconstructed neighboring unit may be a unit that has already been encoded or decoded around the encoding target unit or the target unit to be decoded. The restored neighboring unit may be a spatial adjacent unit or a temporally adjacent unit for the target unit.

Predictive unit partition: The predictive unit partition may mean a type in which the predictive unit is divided.

Parameter Set: A parameter set may correspond to header information among structures in a bitstream. For example, the parameter set may include a sequence parameter set, a picture parameter set, and an adaptation parameter set.

Ratedistortion optimization: The encoding apparatus uses rate distortion optimization to provide a high coding efficiency using a combination of the size of the coding unit, the prediction mode, the size of the prediction unit, the motion information, and the size of the conversion unit .

 The rate distortion optimization scheme may calculate the ratedistortion cost of each combination to select the optimal combination from among the combinations above. The rate distortion cost can be calculated using Equation (1) below. In general, the combination in which the rate distortion cost is minimized can be selected as the optimal combination in the rate distortion optimization method.

D can represent distortion. D may be the mean square error of the original transform coefficients within the transform block and the difference values between the restored transform coefficients.

R can represent the rate. R can represent the bit rate using related context information.

lambda can represent a Lagrangian multiplier. R may include not only encoding parameter information such as a prediction mode, motion information, and coded block flag, but also bits generated by encoding the transform coefficients.

The encoding apparatus performs inter prediction and / or intra prediction, conversion, quantization, entropy encoding, inverse quantization, and inverse transformation to calculate the correct D and R, and this process can greatly increase the complexity in the encoding apparatus have.

Reference picture: The reference picture may be an image used for inter prediction or motion compensation. The reference picture may be a picture including a reference unit referred to by the target unit for inter prediction or motion compensation. The meanings of pictures and images may be the same. In addition, the terms "picture" and "image" may be used interchangeably.

Reference picture list: The reference picture list may be a list including reference pictures used for inter prediction or motion compensation. The type of the reference picture list may be a list combination (LC), a list 0 (L0), and a list 1 (L1).

Motion Vector (MV): A motion vector may be a two-dimensional vector used in inter prediction. For example, MV can be expressed in the form (mv _x , mv _y ). mv _x may represent a horizontal component, and mv _y may represent a vertical component.

 The MV may indicate an offset between the target picture and the reference picture.

Search range: The search area may be a two-dimensional area where an MV search is performed during inter prediction. For example, the size of the search area may be MxN. M and N may be positive integers, respectively.

Basic decision unit: The basic decision unit may represent a basic unit to be processed in the embodiment. The basic decision unit may be a plurality of frames and may be a group of pictures (GOP).

Group of Picture (GOP): The meaning of a GOP in the embodiment may be the same as the meaning of a GOP generally used in video coding. That is to say, the start frame of the GOP may be an eye (I) frame, and other frames than the start frame of the GOP may be a frame directly referring to the I frame or a frame indirectly referring to the I frame.

Frame rate: The frame rate can represent the temporal resolution of the video. The frame rate may indicate how many screen (s) are displayed per second.

Frape Per Second (FPS): FPS can be a unit of frame rate. For example, "30 FPS" may indicate that 30 frames per second are played.

High Frame Rate (HFR): It can indicate a frame rate higher than a predetermined reference value. For example, video with a FPS of 60 or higher may be HFR video, and video with a FPS higher than 50 on a European basis may be HFR video. In the embodiments, video of 60 FPS is exemplified as HFR video.

Optimal Frame Rate: The optimal frame rate may be a frame rate determined by the content of the video. The optimal frame rate can be determined for each basic decision unit. Even if the frame rate of video or basic decision unit is increased beyond the optimum frame rate, people may not feel the difference in image quality due to the increase of the frame rate. That is to say, the optimal frame rate can be the minimum frame rate that prevents people from feeling degraded in video quality compared to HFR. Also, the optimal preliminary rate may be the FPS immediately before the image quality degradation occurs when the image quality difference between the original HFR video and the reduced FPS video is measured while gradually reducing the FPS of the HFR video. Here, the occurrence of image quality degradation may mean that the ratio set by the user or a predetermined ratio or more of people recognize the image quality difference.

Homogenous video: Homogeneous video can be a video with all the basic decision units that make up the video have the same or similar characteristics.

Perceived degradation of image quality: Perceived degradation of image quality indicates the percentage of people who can perceive degradation in image quality when the frame rate of HFR video is decreased per basic decision unit. Typically, the FPS may be changed stepwise by a factor of two. The unit of the perceived degradation of image quality may be%.

Picture quality degradation awareness predictor: The picture quality degradation awareness predictor can be a predictor to predict the picture quality degradation awareness.

Satisfied user ratio: Image quality satisfaction may be the ratio of people who are satisfied with the video quality of HFR video with reduced frame rate compared to the HFR video quality when the FPS of HFR video is decreased according to the basic decision unit. Alternatively, the image quality satisfaction may be a ratio of persons who do not perceive the difference between the image quality of the HFR video and the image quality of the video of the reduced FPS when the FPS of the HFR media is decreased by the basic decision unit.

Typically, the FPS may be changed stepwise by a factor of two. Here, the ratio of people can be expressed as a percentage of the value of 1a / b. a may be the number of people who evaluated that there is a difference between the image quality of the HFR video and the video of the reduced FPS through a subjective image quality assessment. b can be the total number of people. For example, for a video of 60 FPS and a video of 30 FPS, 5 people gave a subjective image quality evaluation, 3 of 5 gave the same image quality score to 60 FPS video and 30 FPS video, If two people gave a lower image quality score to 30 FPS video compared to 60 FPS video, the image quality could be 12/5 = 60%. At this time, the image quality degradation rate may be 40%. That is, the sum of the image quality satisfaction and the image quality degradation perception rate may be 100%.

Satisfied User Ratio Predictor (SURP): SURP can be a predictor of image quality satisfaction. The configuration and operation of the image quality degradation awareness predictor and the SURP may be similar to each other, and both can only be subtracted in the output value.

Selection information: In an embodiment, the image quality information may be an image quality degradation rate or image quality satisfaction. The term "selection information" may be used interchangeably with the term "cognitive image quality information ", and both terms may be used interchangeably.

Basic operations of the encoding apparatus and the decoding apparatus

1 is a block diagram illustrating a configuration of an encoding apparatus to which the present invention is applied.

The encoding apparatus 100 may be a video encoding apparatus or an image encoding apparatus. The video may include one or more images. The encoding apparatus 100 may sequentially encode one or more images of the video according to time.

1, an encoding apparatus 100 includes an inter prediction unit 110, an intra prediction unit 120, a switch 115, a subtractor 125, a transform unit 130, a quantization unit 140, An inverse quantization unit 160, an inverse transform unit 170, an adder 175, a filter unit 180, and a reference picture buffer 190. [

The encoding apparatus 100 may perform encoding in an intra mode and / or an inter mode on an input image. The input image can be referred to as the current image which is the object of the current encoding.

Also, the encoding apparatus 100 can generate a bitstream including encoding information through encoding of the input image, and output the generated bitstream.

When the intra mode is used, the switch 115 can be switched to intra. When the inter mode is used, the switch 115 can be switched to the inter.

The encoding apparatus 100 may generate a prediction block for an input block of an input image. Also, after the prediction block is generated, the encoding device 100 may code the residual of the input block and the prediction block. The input block may be referred to as the current block which is the current encoding target.

When the prediction mode is the intra mode, the intraprediction unit 120 can use the pixel value of the already coded block around the current block as a reference pixel. The intra predictor 120 can perform spatial prediction of a current block using a reference pixel and generate prediction samples of a current block through spatial prediction.

The inter prediction unit 110 may include a motion prediction unit and a motion compensation unit.

When the prediction mode is the inter mode, the motion prediction unit can search the reference image for the best match with the current block in the motion estimation process, and derive the motion vector for the current block and the searched area. The reference picture may be stored in the reference picture buffer 190 and may be stored in the reference picture buffer 190 when the coding and / or decoding of the reference picture is processed.

The motion compensation unit may generate a prediction block by performing motion compensation using a motion vector. Here, the motion vector may be a two-dimensional vector used for inter prediction. The motion vector may also indicate an offset between the current image and the reference image.

The subtractor 125 may generate a residual block that is a difference between the input block and the prediction block. The residual block may be referred to as a residual signal.

The transforming unit 130 may perform a transform on the residual block to generate a transform coefficient, and output the generated transform coefficient. Here, the transform coefficient may be a coefficient value generated by performing a transform on the residual block. When the transform skip mode is applied, the transforming unit 130 may omit the transform for the residual block.

A quantized transform coefficient level can be generated by applying quantization to the transform coefficients. Hereinafter, in the embodiments, the quantized transform coefficient level may also be referred to as a transform coefficient.

The quantization unit 140 may generate a quantized transform coefficient level by quantizing the transform coefficient according to the quantization parameter. The quantization unit 140 may output the generated quantized transform coefficient levels. At this time, the quantization unit 140 can quantize the transform coefficient using the quantization matrix.

The entropy encoding unit 150 can generate a bitstream by performing entropy encoding according to the probability distribution based on the values calculated by the quantization unit 140 and / or the encoding parameter values calculated in the encoding process . The entropy encoding unit 150 may output the generated bitstream.

The entropy encoding unit 150 may perform entropy encoding on information for decoding an image in addition to information on pixels of the image. For example, the information for decoding the image may include a syntax element or the like.

The encoding parameters may be information required for encoding and / or decoding. The encoding parameter may include information that is encoded in the encoding apparatus and transferred to the decoding apparatus, and may include information that can be inferred in the encoding or decoding process. For example, as information transmitted to the decoding apparatus, there is a syntax element.

For example, the coding parameters include a prediction mode, a motion vector, a reference picture index, a coding block pattern, a residual signal presence, a transformation coefficient, a quantized transform coefficient, a quantization parameter, a block size, Information, or the like. The prediction mode may indicate an intra prediction mode or an inter prediction mode.

The residual signal may mean a difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming and quantizing the difference between the original signal and the prediction signal. The residual block may be a residual signal on a block basis.

When entropy coding is applied, a small number of bits can be assigned to a symbol having a high probability of occurrence, and a large number of bits can be assigned to a symbol having a low probability of occurrence. As the symbol is represented through this allocation, the size of the bit string for the symbols to be encoded can be reduced. Therefore, the compression performance of the image encoding can be improved through the entropy encoding.

In addition, for entropy encoding, encoding methods such as exponential golomb, context adaptive variable length coding (CAVLC), and context adaptive binary arithmetic coding (CABAC) may be used . For example, the entropy encoding unit 150 may perform entropy encoding using a Variable Length Coding / Code (VLC) table. For example, the entropy encoding unit 150 may derive a binarization method for a target symbol. In addition, the entropy encoding unit 150 may derive a probability model of a target symbol / bin. The entropy encoding unit 150 may perform entropy encoding using the derived binarization method or probability model.

Since encoding is performed through inter prediction by the encoding apparatus 100, the encoded current image can be used as a reference image for another image (s) to be processed later. Accordingly, the encoding apparatus 100 can decode the encoded current image again, and store the decoded image as a reference image. The inverse quantization and inverse transform of the current encoded image for decoding can be processed.

The quantized coefficients may be inversely quantized in the inverse quantization unit 160. And may be inversely transformed in the inverse transform unit 170. [ The dequantized and inverse transformed coefficients may be combined with a prediction block via an adder 175. A reconstructed block may be generated by summing the dequantized and inverse transformed coefficients and the prediction block.

The restoration block may pass through the filter unit 180. The filter unit 180 may apply at least one of a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to a reconstructed block or a reconstructed picture. The filter unit 180 may be referred to as an adaptive inloop filter.

The deblocking filter can remove block distortion occurring at the boundary between the blocks. SAO may add a proper offset value to the pixel value to compensate for coding errors. The ALF can perform filtering based on the comparison between the restored image and the original image. The reconstructed block having passed through the filter unit 180 may be stored in the reference picture buffer 190.

2 is a block diagram illustrating a configuration of a decoding apparatus to which the present invention is applied.

The decoding apparatus 200 may be a video decoding apparatus or an image decoding apparatus.

2, the decoding apparatus 200 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra prediction unit 240, an inter prediction unit 250, an adder 255, A filter unit 260, and a reference picture buffer 270.

The decoding apparatus 200 can receive the bit stream output from the encoding apparatus 100. [ The decoding apparatus 200 may perform decoding of an intra mode and / or an inter mode with respect to a bit stream. Also, the decoding apparatus 200 can generate a reconstructed image through decoding and output the generated reconstructed image.

For example, switching to an intra mode or an inter mode according to a prediction mode used for decoding may be performed by a switch. When the prediction mode used for decoding is the intra mode, the switch can be switched to intra. When the prediction mode used for decoding is the inter mode, the switch can be switched to the inter.

The decoding apparatus 200 can obtain a reconstructed residual block from the input bitstream and generate a prediction block. Once the reconstructed residual block and prediction block are obtained, the decoding apparatus 200 can generate a reconstruction block by adding the reconstructed residual block and the prediction block.

The entropy decoding unit 210 may generate the symbols by performing entropy decoding on the bitstream based on the probability distribution. The generated symbols may include symbols in the form of quantized coefficients. Here, the entropy decoding method may be similar to the above-described entropy encoding method. For example, the entropy decoding method may be the inverse of the above-described entropy encoding method.

The quantized coefficients may be inversely quantized in the inverse quantization unit 220. Also, the inverse quantized coefficient may be inversely transformed by the inverse transform unit 230. As a result that the quantized coefficients are inversely quantized and inversely transformed, reconstructed residual blocks can be generated. At this time, the inverse quantization unit 220 may apply the quantization matrix to the quantized coefficients.

When the intra mode is used, the intra prediction unit 240 can generate a prediction block by performing spatial prediction using the pixel value of the already coded block around the current block.

The inter prediction unit 250 may include a motion compensation unit. When the inter mode is used, the motion compensation unit can generate a prediction block by performing motion compensation using a motion vector and a reference image. The reference picture may be stored in the reference picture buffer 270.

The reconstructed residual block and the prediction block may be added through an adder 255. The adder 255 may generate the restored block by adding the restored residual block and the predicted block.

The restoration block may pass through the filter unit 260. The filter unit 260 may apply at least one of a deblocking filter, SAO, and ALF to a restoration block or a restoration picture. The filter unit 260 may output a restored image. The restored image is stored in the reference picture buffer 270 and can be used for inter prediction.

Satisfied User Ratio Predictor (SURP)

Figure 3 illustrates the operation of SURP according to one embodiment.

The input of the SURP 300 may be a basic decision unit. As shown in FIG. 3, the basic decision unit may be a GOP. The output of the SURP 300 may be a prediction picture quality satisfaction for the GOP of the changed FPS.

For example, the input GOP may be a GOP having eight frames of 60 FPS.

To calculate the predicted image quality satisfaction, the SURP 300 may change the FPS of the GOP. For example, the SURP 300 may change the GOP of X FPS to the GOP of Y FPS. For example, Y may be aX and a may be 1/2. Here, "1/2" is only an exemplary value, and a may have a value of 1/2 ⁿ , such as 1/4, 1/8 and 1/16. n may be an integer of 1 or more.

X can represent the FPS of the GOP before the change. Y can indicate the FPS of the changed GOP. When X = 2 ⁿ , Y may be one of X / 2, X / 2 ² , ... X / 2 ⁿ . For example, when the value of X is 60, the value of Y may be 30. That is, the GOP can be converted from 60 FPS to 30 FPS.

The predicted image quality satisfaction may be a predicted value of the percentage of people who do not recognize the image quality difference between the image quality of the GOP before the change and the image quality of the changed GOP on the assumption that the FPS of the GOP has been changed from 60 to 30. [ Alternatively, the predicted image quality satisfaction may be a predicted value of the ratio of people who are satisfied with the image quality of the changed GOP as compared with the image quality of the GOP before the change, on the assumption that the FPS of the GOP is changed from 60 to 30.

The SURP 300 can generate predicted image quality satisfaction through machine learning.

Hereinafter, the GOP before the FPS change can be referred to as a source GOP. In addition, the GOP after the FPS change can be referred to as a target GOP. In addition, the video before the change of the FPS can be referred to as a source video. In addition, the video after the FPS change can be referred to as a target video.

Considerations for the generation of predicted image quality satisfaction through machine learning.

In an embodiment, SURP 300 may use machine learning to measure how much image degradation occurs by changing the FPS when changing the FPS per GOP.

For machine learning, training data is required, that is, a ground truth for machine learning may be required. In the embodiment, since maintaining the human perception quality is one of the goals, A method of acquiring data through a subjective image quality evaluation experiment may be used to acquire data on the perceived quality of the image.

Here, obtaining the data on the perceived image quality through the subjective image quality evaluation may be only one embodiment. Instead of directly measuring the cognitive quality through the subjective quality assessment, the SURP 300 can measure the cognitive quality using a measure that models the cognitive quality. For example, the sensor may be a Structural SIMilarity (SSIM) or a Video Quality Metric (VQM).

Typically, the Mean Opinion Score (MOS) value is mainly used as the value of the perceived image quality finally obtained as the subjective image quality. In applying the MOS to the subjective image quality evaluation of the embodiment, the following problems 1) to 3) may occur.

1) Firstly, in the manner in which a conventional user gives a score, the criterion of the score according to the user may be different from each other. In addition, even in the evaluation process of assigning scores, it is often possible that the same person gives different scores to the same video.

2) Secondly, since the MOS values of videos are different from each other, there may arise a problem that MOS values different from each other are difficult to be directly used for machine learning. To solve this problem, the SURP 300 may use relative cognitive image quality information instead of MOS. For example, there is a Difference MOS (DMOS) as relative cognitive quality information. If the difference between the MOS value of the source video and the MOS value of the target video is used for subjective image quality evaluation, the result of such subjective image quality evaluation can be utilized for machine learning. For such subjective image quality assessment, an evaluation of the quality of the source video may be required for all evaluation items, such as the DoubleStimulus Continuous Quality Scale (DSCQS) and the Double Stimulus Impairment Scale (DSIS). Therefore, there may arise a problem that the time for the experiment of image quality evaluation is increased by such image quality evaluation.

3) Third, the conventional picture quality measurement method recommends that a measurement of 8 seconds or more is performed for each video sequence. That is, the result of the image quality measurement can be regarded as a result of the sequence level. If machine learning for the determination of the basic decision unit is made using these results, good performance may not be obtained.

Therefore, in one embodiment, a subjective picture quality evaluation method for obtaining a ground truth suitable for machine learning and a condition of a video sequence used for picture quality evaluation are proposed.

Machine learning

The SURP 300 can determine an optimal FPS for each basic decision unit. In order for the FPS to be determined by the basic decision unit, the ground truth data for such a decision may have to be obtained.

In the following description, a GOP is regarded as a basic decision unit for convenience, and the size of the GOP is regarded as 8. That is to say, one GOP may contain 8 frames. The assumptions about the size of these basic decision units and GOPs are merely exemplary.

The following description related to the acquisition and subjective image quality evaluation of the ground truth can be applied to the basic decision units of different kinds and basic decision units of different sizes in the same or similar manner.

Subjective quality assessment can only be performed at the sequence level. On the other hand, evaluation of the unit of GOP is required for machine learning. To overcome this difference, the length of the video sequence used for image quality evaluation may be adjusted to ensure that all GOPs of the video sequence contain only the same or similar content characteristics. That is to say, the characteristics of all the GOPs of the video may be similar to each other. In an embodiment, a video similar in character to all the GOPs of the video may be referred to as homogeneous video.

The length of the homogeneous video can be limited to a predetermined value. For example, the length of the homogeneous video may be adjusted to not exceed 5 seconds.

Since homogeneous video composed of GOPs of these similar characteristics is used, it may be possible that the image quality evaluation result obtained at the sequence level is used as the image quality information of the unit of the GOP. In addition, the SURP 300 can improve the performance of the SURP 300 through machine learning using the image quality information.

4 shows a picture quality evaluation result according to an example.

As an example of a subjective picture quality assessment, the source video may be a 60 FPS HFR video, or may be HD video. The target video may be 30 FPS video.

The SURP 300 may generate the target video by lowering the temporal resolution of the source video to 1 / n. n may be 2. Alternatively, n may be 2 ^m , and m may be an integer of 1 or more.

The SURP 300 may generate a target video by applying a frame drop to the source video. It may be to remove the default frame from the frame dropped source video. For example, the frame drop may be to remove the even-numbered frame (s) of the frames of the source video.

 As a subjective image quality evaluation method, a pairwise comparison method can be used.

The fair-wise comparison method can be a method of showing two videos to the evaluator for each evaluation item, and querying which one of the two videos has better image quality. The evaluator can select a better video among the two videos for each evaluation item. Alternatively, the fairness comparison scheme may query whether the quality of the first video of the two videos is better, the quality of the second video is better, or the quality of the two videos is the same, for each evaluation item. That is to say, the evaluator can select one of three items for each evaluation item. In the embodiment, a method of using three items has been exemplified.

The way in which one of the three items is selected for each evaluation item is as follows: 1) the quality of two videos is similar; and 2) a relatively small evaluator participates in the experiment.

In the picture quality evaluation, the displayed order of the high FPS video and low FPS video for each evaluation item can be adjusted randomly. This random adjustment can prevent the evaluator from having a preconceived notion of which of the two videos is a larger FPS video.

Also, if the length of the video sequence is less than or equal to a predetermined reference value, two videos may be shown to the evaluator two times or two or more times for each evaluation item. For example, video A and video B can be viewed by the evaluator in the order A, B, A,

In Fig. 4, a bar on the x-axis represents video that is the object of image quality evaluation. The string under the x-axis represents the name of the video. The y-axis represents the ratios selected for the evaluator of the three items. The three items are "better picture quality of high picture quality (ie, 60 FPS)", "better picture quality of both videos" and "lower picture quality (ie, 30 FPS)". The selected percentage of each item can represent the percentage of the evaluators who selected each item among the total evaluators. The unit of rate can be a percentage (%).

Of the items, "the quality of both videos is the same" and "the image quality of the low-quality video is better" can be regarded as the image quality is maintained even after the video is converted. That is, among the items, "the picture quality of both videos is the same" and "the picture quality of the low-quality video is better" is selected, which means that the picture quality of the converted video also satisfies the evaluator. That is to say, only the evaluator who has selected the item of "the image quality of the high-quality video is better" among the evaluators may be considered to have felt the difference between the image qualities of the two videos.

Therefore, the sum of the ratios of "the video quality of both videos is the same" and "the image quality of the lower quality video is better" is selected as the ratio of the persons who do not recognize the difference between the image qualities of the two videos. " The unit of "image quality satisfaction" may be a percentage (%).

In FIG. 4, 17 video sequences are arranged in descending order of the percentage of people who do not recognize the image quality difference. According to Fig. 4, for seven video sequences, more than 75% of participants do not feel the difference between the image quality of high-definition video and low-definition video.

Similar to the above, various target videos can be generated from the source video of the HFR. For example, the FPS of the target videos may be 1/4 and 1/8 of the FPS of the source video. Quality of video quality can also be measured for each of these target videos and source video.

The result of this subjective image quality evaluation can be used as a ground trace for machine learning. The quality of the ground truth data can be improved as the number of evaluators increases. Further, the quality of the data of the ground truth can be improved as the number of video sequences to be evaluated increases.

When DSCQS is used for subjective image quality evaluation rather than fair-wise comparison, 1) DSCQS "opinion score of high-quality video is higher than opinion score of low-quality video" means "fairness comparison" The quality of the picture quality video is better ". 2) DSCQS 'opinion score of high-quality video is equal to opinion score of low-quality video' can correspond to 'image quality of both videos is equal to pairwise comparison'. 3) DSCQS's "opinion score of high quality video is lower than opinion score of low quality video" can correspond to "better quality of low quality video" of the pairwise comparison. Through this correspondence relationship, even when the DSCQS is used, the image quality satisfaction can be measured in the same manner as in the case where the above-described pairwise comparison is used.

Feature vector

The SURP 300 may be a predictor using machine learning. As an example of the machine learning, the SURP 300 can use a support vector machine (SVM).

In order to predict the deterioration of the video quality when the FPS of the video is changed, features that affect the video quality should be extracted as the feature vector. Various feature vector extraction methods that reflect human visual characteristics such as spatial masking effect, temporal masking effect, salient region, and contrast sensitivity are described below.

The SVM can generally use a feature vector that better represents the information of the input frame required for prediction, rather than the entirety of the input frame. That is, the prediction performance of the SVM can be influenced by what information is used as a feature vector.

In an embodiment, in order for the feature vector to reflect the quality of the cognitive quality, the feature vector may be designed to consider whether the portion of the video has a small impact on the cognitive quality or a large impact on the cognitive quality.

For example, a part of the video which has little influence on the picture quality can be a part having a large spatial masking effect and a part having a large temporal masking effect. A large part of the video that affects the image quality may be a part with high Contrast Sensitivity (CS). In addition, Visual Saliency (VS) can also have a significant impact on perceived quality. Therefore, VS in the design of the feature vector can also be considered.

FIG. 5 illustrates a procedure of extraction of a feature vector according to an embodiment.

At step 510, the SURP 300 may generate a Spatial Randomness Map (SRM) for each frame.

SRM can be used to reflect the spatial masking effect to the feature vector. The area that is not visually perceived can be a region with high randomness. Thus, through the calculation of the spatial randomness (SR), the spatial masking region in the frame can be determined.

The characteristics of the region of the SR may be different from those of the surrounding regions. This disparity may indicate that the area of the SR is hard to predict from the information of the surrounding area. Therefore, in order to measure the SR, the central region Y can be predicted from the peripheral region X as shown in Fig.

6 shows a prediction model for spatial arbitrary measurement according to an example.

In FIG. 6, it is shown that the center Y pixel is predicted from the surrounding X pixels.

See FIG. 5 again.

The optimal prediction for Y can be expressed as Equation (2) below.

u can represent a spatial position. H may be a transformation matrix that provides an optimal prediction of the Y value from the surrounding X values. If an optimization method of minimum mean error is used, H can be expressed as Equation 3 below.

R _xy may be a crosscorrelation matrix of X and Y.

R _x may be an auto correlation matrix for X itself.

Using an approximated pseudoinverse matrix technique, the inverse of R _x can be obtained according to Equation (4) below.

The following equation (5) can be established according to the above-mentioned expressions (2), (3) and (4).

The SURP 300 may obtain the SRM for each pixel of the frame based on Equation (5).

7 shows an SRM according to an example.

In Figure 7, a bright pixel may represent a pixel that is not predictable from surrounding pixels. That is to say, a bright pixel can represent a region having a high spatial randomness, that is, a region having a large spatial masking effect.

See FIG. 5 again.

In step 515, the SURP 300 may generate an edge map (EM) for each frame.

SURP 300 may generate an edge map using a sobel edge operation for each frame.

In step 520, the SURP 300 may generate a smoothness map (SM) for each frame.

The SM may be used to determine whether the background of the frame is a flat area.

The SURP 300 can calculate the flatness value of each block over the SRM blocks. The flatness value of each block can be calculated by Equation (6) below.

N _tc may be the number of pixels having a spatial randomness lower than the reference value in the block. Here, spatial randomness lower than the reference value may mean low complexity.

W _b ² can represent the area of the block, i.e., the number of pixels in the block. For example, W _b may be 32. A value of W _b of 32 may indicate that a 32x32 block is used.

The SURP 300 can generate the SM by setting the values of all the pixels inside each block to the flatness values of the blocks.

Figure 8 shows an SM according to an example.

In the SM, the same pixel value can be set for each block.

In SM, the brighter the block, the smoother the background. That is to say, a bright block can represent a flat background. The edge of the bright block can be perceived better than the edge of the dark block.

See FIG. 5 again.

In step 525, the SURP 300 may generate a spatial contrast map (SCM) for each frame.

Stimulus sensitivity can be related to spatial contrast. SCM can be used to reflect the spatial contrast in the feature vector.

The SCM can be obtained by measurement of edge and flatness. Sensitive responses to stimuli can occur in the smooth background area. Here, the stimulus can mean an edge. On the other hand, edges in background areas with low flatness may have low sensitivity to stimulation due to masking effects. The SCM may be a map to reflect these characteristics.

The SCM may be the product of pixel values of corresponding pixels of SM and EM. The SURP 300 may set the product of the pixel values of the pixels at the same position of SM and EM to the pixel value of the SCM.

The SURP 300 may generate the SCM according to Equation (7) below.

Y may be a pixel at the same position in each of SCM, EM and SM.

9 shows an SCM according to an example.

In Fig. 9, a bright pixel may exhibit a strong edge and a flat background. That is to say, a bright pixel may be a pixel representing a strong edge or a pixel representing a high flat background.

See FIG. 5 again.

In step 530, the SURP 300 may generate a visual saliency map (VSM) for each frame.

In terms of human visual characteristics, stimulation of areas of human interest may have a greater impact than stimulation of areas of no human interest. Information of visual attention can be used so that such a visual feature is reflected in the feature vector.

As an example of a VSM, a graphic based visual saliency method proposed by Harel and Perona may be used.

10 shows a VSM according to an example.

In Fig. 10, a bright area may indicate an area of interest when a person views an image of the frame. That is to say, the bright region may be a region of high attention. The dark area can indicate an area that is not of interest when a person views the image of the frame. That is to say, the dark area may be a low-notice area.

See FIG. 5 again.

At step 535, the SURP 300 may generate a Spatial Influence Map (SIM) for each frame.

The SIM may be a map that reflects whether or not the pixel corresponds to a visual saliency region, as well as predictive peripherals that are visual stimuli.

The SIM may be the product of the pixel values of the corresponding pixels of the SCM and the VSM. The SURP 300 may set the product of the pixel values of the pixels at the same position of the SCM and the VSM to the pixel value of the SCM.

The SURP 300 may generate the SCM according to Equation (8) below.

Y may be a pixel at the same position in each of the SIM, SCM, and VSM.

Fig. 11 shows a SIM according to Fig.

See FIG. 5 again.

The SURP 300 can acquire the feature vector based on the spatial characteristics and the temporal characteristics of the frames of the basic decision unit. In the above-described steps 510, 515, 520, 525, 530, and 535, the process of acquiring the feature vector by reflecting the spatial characteristics of the frame has been described. In the following steps 540, 545, 550, and 555, a process of acquiring a feature vector by reflecting temporal characteristics of a frame will be described.

The region having a high temporal masking effect may be a region having irregular motion or a region having sudden motion. That is, an arbitrarily high region may be an area having a high temporal masking effect, and an area having a high temporal masking effect may be determined by detecting a high arbitrary region in the frame.

By acquiring a feature vector based on temporal characteristics, the SURP 300 can reflect the human visual characteristics that are sensitive to irregular motions or sudden motions to the acquisition of feature vectors.

As described above, spatial randomness can be calculated by a model that predicts from neighboring pixels. Similar to this calculation scheme, the temporal arbitrary can be calculated through a model that predicts the current frame from previous frames.

In order to calculate the temporal randomness, the input interval of the frames of the GOP may be divided into two intervals.

The interval can be expressed as Equation (9) below.

Y may represent a section. k may represent the start frame of the interval. l may represent the last frame of the interval. That is to say, Y _k ^l may indicate that the interval consists of the k th frame to the l th frame.

When equation (9) is stored as a matrix, the columns of the matrix may correspond to the frames, respectively. That is to say, storing the interval as a matrix can be made by the following steps 1) and 2).

1) -th frame can be arranged in the form of a one-dimensional row vector. For example, the second row of the frame may be concatenated to the first row of the frame. The rows of the frame may be sequentially concatenated.

2) The row vector may be transposed in the form of a column vector. A matrix of column vectors of frames may be stored.

When the length of each interval is d, the two intervals can be expressed as Y _{k + d} ^l and Y _k ^{l d} .

In step 540, the SURP 300 may generate a Temporal Prediction Model (TPM).

12 illustrates a case in which the size of the GOP is 8 and the value of d is 4 in the following FIG. 12.

FIG. 12 shows a relationship between a frame used for generation of the temporal prediction model and a prediction target frame according to an example.

In Fig. 12, "frame 0" may represent the first frame and "frame 1" may represent the second frame. That is to say, "frame n" may be the (n + 1) -th frame.

According to Fig. 12, "frame 0" to "frame 3" may be the first section, and "frame 4" to "frame 7" may be the second section.

The frames of the first interval may be used for the TPM, and the frames of the second interval may be predicted by the TPM.

See FIG. 5 again.

The TPM can be generated using the first section of the GOP. For example, for a GOP of size 8, the TPM may be generated using the first frame, the second frame, the third frame, and the fourth frame.

The SURP 300 may perform temporal prediction based on Equation (10) below.

Here, A may be a transformation matrix providing optimal prediction.

T may be a prediction matrix.

A can be calculated as shown in Equation (11) using a pseudo inverse.

However, since each matrix of Equation (11) is very large, the application of Equation (11) may be practically impossible.

The matrix A can be calculated more easily by using the state sequence representation of Equation (12) below.

Here, Y may be a state matrix of X.

C may be an encoding matrix.

W may be a bias matrix.

Equation (13) below can represent a singular value decomposition for Y. At this time, W can be assumed to be a zero matrix.

Comparing Equations (12) and (13), the SURP 300 can derive an optimal state vector according to Equation (14) below.

SURP can obtain a temporal randomness which is a temporal prediction error according to the following equation (15).

Figures 13A-13C illustrate three frames in accordance with a contiguous example.

13A shows a first one of three consecutive frames according to an example.

13B shows a second one of three consecutive frames according to an example.

13C shows a third frame of three consecutive frames according to an example.

Next, maps for three frames are shown.

13D shows a Temporal Randomness Map (TRM) for three consecutive frames according to an example.

FIG. 13E shows a Spatio Temporal Influence Map (STIM) for three consecutive frames according to an example.

FIG. 13F shows a weighted Spatio Temporal Influence Map (WSTIM) for three consecutive frames according to an example.

TRM, STIM and WSTIM are described in detail below.

See FIG. 5 again.

In step 545, the SURP 300 may generate a Temporal Randomness Map (TRM).

TRM may be a map for the fourth frame, the fifth frame, the sixth frame and the seventh frame.

In TRM, bright pixels can represent regions with large temporal randomness. A region having a large temporal randomness may be a region having a small visual cognitive effect.

In step 550, the SURP 300 may generate a Spatio Temporal Influence Map (STIM).

The STIM may be a map for the fourth frame, the fifth frame, the sixth frame and the eighth frame.

When a person views a video, one can perceive spatial visual stimuli and temporal visual stimuli together, depending on the visual characteristics of the person. Therefore, it is also necessary that the spatial characteristic and the temporal characteristic are simultaneously reflected in the feature vector.

In order to rebound the spatial and temporal characteristics, an STIM may be defined as shown in Equation (16) below.

Y can represent a pixel at a specific position.

The STIM may be the product of pixel values of corresponding pixels of SIM and TRM. The SURP 300 may set the product of the pixel values of the pixels at the same position of the SIM and the TRM to the pixel value of the STIM.

In step 555, the SURP 300 may generate a Weighted Spatio Temporal Influence Map (WSTIM).

The WSTIM may be a map for the fourth frame, the fifth frame, the sixth frame and the eighth frame.

WSTIM can emphasize relatively large stimuli in the whole stimulus.

The SURP 300 may set a value obtained by dividing the value of the SIM pixel by the average value of the pixels of the SIM into the pixel value of the WSTIM.

The WSTIM may be effective when there is stimulation of fast movement of small objects within the frames.

In step 560, the SURP 300 may perform the function of a feature representor. The SURP 300 may output the result of at least one of TRIM, STIM, and WSTIM.

The SURP 300 may calculate an average of at least one of TRIM, STIM, and WSTIM for each block of blocks of a predetermined size of the frame. The SURP 300 may output the averages of blocks in descending order. For example, the default size may be 64x64.

The aforementioned maps may be expressed in the form of a matrix representing data such as an image. There is a need for a process of a characteristic presenter that expresses the aforementioned maps in the form of a vector suitable for the feature vector.

The SURP 300 may divide each map of maps into blocks of predetermined size. For example, if the video is HD video, the size of the block may be 64x64. The SURP 300 can calculate an average value of pixels of each block for each block. In addition, the SURP 300 may arrange the blocks of the map in order from the block having the largest average value to the block having the smallest value, so as not to have dependency on the position.

Through this process, each map can be expressed in the form of a one-dimensional array such as a histogram.

If the video is HD video, the size of the one-dimensional array may be 507.

The one-dimensional array may be an array arranged in the order of average values of the blocks. Thus, in general, large values may be placed in the front portion of the one-dimensional array, and smaller values may be placed in the backward direction. Therefore, even if only the preceding part of the values of the one-dimensional array is used as the feature vector, the performance difference may not be large compared to that of all the values of the one-dimensional array used. According to this characteristic, the SURP 300 can use only the previous part having large values as a feature vector, instead of using the entire values of the one-dimensional array as feature vectors. Here, a portion may be a predetermined length or a predetermined ratio. In a one-dimensional array, large values may or may not be in the front of the array. For example, the SURP 300 may use only the first 100 of the values of the one-dimensional array.

Model learning

The SURP 300 can use the SVM. The machine learning of the SURP 300 may be the operations of 1) to 3) below.

1) The SURP 300 can extract a feature vector from a GOP.

2) The SURP 300 can perform the output prediction using the extracted feature vector as the input of the SVM.

3) The SURP 300 may generate the SVM such that the value output by the output prediction has a value as close as possible to the ground truth.

Below is an example of learning a model to predict degradation in image quality when 60 FPS video is changed to 30 FPS video.

First, the ground truth was already obtained through the subjective image quality evaluation of the data set of the homogeneous video.

Next, the SURP 300 may obtain TRM, STIM, and WSTIM for 60 FPS video, respectively. Also, the SURP 300 may determine each of TRM, STIM and WSTIM based on an internal assumption that the FPS of the video is 30 Can be obtained. In addition, the SURP 300 can convert TRM, STIM, and WSTIM of 60 FPS video into a one-dimensional array through the feature presenters, and the SURP 300 can convert TRM, STIM, and WSTIM of 30 FPS video into feature descriptors Can be converted to a one-dimensional array.

Next, the SURP 300 may perform training on the SVM such that the feature vectors of 60 FPS video feature vectors and 30 FPS video feature differences corresponding to the ground truce.

Next, when the learning is completed, the SURP 300 can extract the feature vectors of the 60 FPS GOP and the 30 FPS GOP using the GOP of 60 FPS, without further changing the interior of the SVM have.

SURP (300) can predict the image quality difference between the image quality of 60 FPS GOP and 30 FPS GOP by using the characteristic vector of 60 FPS GOP feature vector and 30 FPS GOP feature image, and predicted image quality difference The image quality satisfaction can be finally generated and output based on the degree of the image quality.

Here, 60 FPS and 30 FPS are merely illustrative. SURP 300 can generate and output picture quality satisfaction for conversions between different FPSs, similar to the above-described generation of picture quality satisfaction at 60 FPS and 30 FPS. For example, the FPS may be changed from 60 to 15. Alternatively, the FPS may be changed from 120 to 60.

In order to generate the feature vector of the SVM, only one of the SIM, TRM, STM and RSTM described above may be used, and two or more maps may be used at the same time.

In addition, multiple SVMs may be combined into a cascade. For example, an SVM using a TRM as a feature vector may be applied first, and an SVM using a result of an STM and a previous SVM as a feature vector may be applied next. Finally, an SVM using the RSTM and the result of the immediately preceding SVM as a feature vector can be applied.

Determining the Optimum Frame Rate

Hereinafter, a method for determining the optimum frame rate for each basic decision unit using the SURP 300 and performing optimal encoding for video using the determined optimal frame rate will be described.

The SURP 300 can predict the minimum FPS of the basic decision unit that satisfies the condition of the minimum image quality satisfaction or the predetermined image quality satisfaction with respect to the basic decision unit of the video, Can be provided. The SURP 300 can encode the basic decision unit converted to the minimum picture quality satisfaction.

FIG. 14 shows image quality satisfaction by FPS according to an example.

As described above, the SURP 300 can predict the image quality satisfaction of the GOP converted into the specific FPS and output the predicted image quality satisfaction. Applying the functions described above, the SURP 300 can predict and output a plurality of image quality satisfactions of a plurality of FPSs for each of the GOPs of the video.

By predicting a plurality of image quality satisfactions of a plurality of FPSs for a GOP, a plurality of predicted image quality levels can be used for encoding.

In Fig. 14, the first line of the horizontal represents a plurality of GOPs of the video. The first vertical line indicates the converted FPS of the GOP. Also, for each GOP, the image quality satisfaction in the converted FPS is shown in%.

In Fig. 14, the source video may be 60 FPS.

The SURP 300 may sequentially perform machine learning on a plurality of FPSs of the target video. First, when the FPS of the video is converted to 30, the SURP 300 can perform the machine learning. Also, for the case where the FPS of the video is converted to 15, the SURP 300 can perform machine learning. Further, for the case where the FPS of the video is converted to 7.5, the SURP 300 can perform machine learning. That is, the SURP 300 may perform machine learning for each of the FPSs of the target video.

With machine learning enabled, the SURP 300 can sequentially estimate the image quality of a GOP of a video for each FPS of a plurality of FPSs. For example, when the FPS of the target video is 30, the image quality satisfaction of the three GOPs of the target video may be 80%, 90%, and 50%. When the FPS of the target video is 15, the image quality satisfaction of the three GOPs may be 70%, 60% and 45%.

Through this prediction, the SURP 300 can calculate the image quality satisfaction of a GOP for a plurality of FPSs.

Also, the SURP 300 can regard the image quality satisfaction of the GOP of the source video as 100%.

In FIG. 14, the last row may indicate a requirement. The requirement may indicate required pixture quality or minimum image quality satisfaction. That is to say, the last row can indicate that image quality satisfaction should be above 75%.

FIG. 15 shows an optimum frame rate determined for each GOP according to an example.

The SURP 300 can determine an optimal FPS that satisfies the requirement for each GOP. Here, the optimal FPP may be the FPS of the GOP having the lowest image quality satisfaction, above the value of the requirement. That is to say, the optimal FPS may be the lowest FPS that meets the requirement. If there is no FPS satisfying the picture quality satisfaction among the FPSs of the target video generated by the conversion, the optimal FPS may be the FPS of the source video.

In Fig. 15, the optimal frame rate determined for each of a plurality of GOPs is shown.

For example, according to FIG. 14, the image quality satisfaction for a plurality of FPSs of the first GOP may be 80%, 70% and 50%. Of these, the minimum requirement for image quality is 80 or 80 FPS at 30 FPS. Thus, the optimal frame rate of the first GOP may be 30 FPS. The image quality satisfaction for a plurality of FPSs of the second GOP may be 90%, 80% and 50%. Of these, the minimum requirement for image quality is 80 or 80 FPS at 15 FPS. Thus, the optimal frame rate of the second GOP may be 15 FPS. The image quality satisfaction for the plurality of FPSs of the third GOP may be 50%, 45% and 40%. The third GOP can not be transformed because there is no more than the requirement among the picture quality satisfacies. Thus, the optimal frame rate of the third GOP may be 60 FPS, which is the FPS of the source video.

Each GOP of a plurality of GOPs can be converted to an optimal frame rate of each GOP, and the converted GOPs of the video can be encoded.

16 is a flowchart of a method of encoding a GOP using an optimum frame rate according to an example.

First, input video is illustrated as being 60 FPS. The GOPs of the input video may be processed by steps 1610, 1620, 1630, 1640, 1650 and 1660 sequentially.

In step 1610, the SURP 300 may convert a 60 FPS GOP to a 30 FPS GOP and calculate a picture quality rating of a 30 FPS GOP.

In step 1620, the SURP 300 may convert a 60 FPS GOP to a 15 FPS GOP and calculate a picture quality rating of a 15 FPS GOP.

In step 1630, the SURP 300 may convert a GOP of 60 FPS to a GOP of 7.5 FPS and calculate a picture quality of a GOP of 7.5 FPS.

In step 1640, the SURP 300 may determine an optimal frame rate.

The SURP 300 can receive the requested image quality. The required image quality may indicate the minimum image quality satisfaction described above.

The SURP 300 can select an optimal GOP satisfying the minimum image quality satisfaction among the GOPs changed to the predetermined FPS. Here, the optimal GOP may be the GOP having the lowest FPS.

Alternatively, the SURP 300 may determine an optimal frame rate that satisfies the minimum image quality satisfaction among the FPSs of the modified GOPs. The SURP 300 can select the GOP having the minimum image quality satisfaction or the FPS of the GOP above the minimum image quality satisfaction among the image quality satisfaction degrees of the changed GOPs. The optimal frame rate may be the FPS of the selected FPS or the selected GOP.

In step 1650, the SURP 300 may select the FPS of the GOP of the optimal frame rate. The SURP 300 can convert the FPS of the GOP to an optimal frame rate. Alternatively, the SURP 300 may select the GOP of the input video and the GOP of the optimal frame rate among the changed GOPs by the steps 1610, 1620 and 1630.

At step 1660, the SURP 300 may perform the encoding of the GOP selected at step 1650. [

17 is a flowchart of a method of determining an optimal frame rate according to an example.

Step 1640 described above with reference to FIG. 16 may include the following steps 1710, 1720, and 1730.

The SURP 300 can check whether the GOP satisfies the requested image quality sequentially, in the order of the GOP of the high FPS to the GOP of the low FPS, for the changed GOPs. The SURP 300 can select the immediately preceding GOP of the GOP that does not satisfy the required image quality.

If the first GOP of the modified GOPs does not satisfy the required picture quality, the GOP of the source video can be selected. That is, the GOP of the source video can be selected if any of the changed GOPs does not satisfy the required image quality.

If the last GOP of the modified GOPs does not satisfy the requested image quality, the last GOP may be selected. That is, the GOP of the lowest FPS can be selected if all the GOPs among the changed GOPs satisfy the required image quality.

In step 1710, the SURP 300 may check whether a 30 FPS GOP meets the required image quality. If a 30 FPS GOP does not meet the required picture quality, the SURP 300 can select 60 FPS at the optimal frame rate and can provide a GOP of 60 FPS. Step 1720 can be performed if the GOP of 30 FPS satisfies the required image quality.

In step 1720, the SURP 300 may check whether a 15 FPS GOP meets the required image quality. If a 15 FPS GOP does not meet the required picture quality, the SURP 300 can select 30 FPS at the optimal frame rate and can provide a GOP of 30 FPS. Step 1730 may be performed if the GOP of 15 FPS satisfies the required image quality.

In step 1730, the SURP 300 may check whether a GOP of 7.5 FPS meets the required image quality. If a GOP of 7.5 FPS does not meet the required picture quality, the SURP (300) can select 15 FPS at the optimal frame rate and can provide a GOP of 15 FPS. When a GOP of 7.5 FPS meets the required picture quality, the SURP (300) can select 7.5 FPS at the optimal frame rate and can provide a GOP of 7.5 FPS.

FIG. 18 shows a hierarchical structure of frames having temporal identifiers according to an example.

In Fig. 18, nine frames are shown. The nine frames may be "frame 0" to "frame 8 ".

Each frame may have a Temporal Identifier (TI). Further, each frame may be an I frame, a P frame, or a B frame.

As described above, when the GOP is converted into a plurality of FPSs through the SURP 300, the image quality satisfaction of the plurality of converted GOPs can be predicted. When the bit stream generated by encoding the video includes the image quality satisfaction information related to the image quality satisfaction, an improved bit stream can be provided that supports the optional temporal scalability (TS) while maintaining image quality satisfaction have.

In the High Efficiency Video Coding (HEVC) standard, the TI of each frame can be transmitted through a Network Abstraction Layer (NAL) header. For example, TI information "nuh_temporal_id_plus1" of the NAL header can indicate the TI of a frame.

Through the temporal identifier information of the NAL header, a TS providing the picture quality satisfaction information can be provided.

When the size of the GOP is 8, when the hierarchical prediction structure of FIG. 18 is used, if the odd numbered frames having a TI value of 3 are removed from the GOP, the FPS of the GOP may be 1/2. That is, the FPS of the GOP may be changed from 60 to 30.

In addition, if "frame 2" and "frame 6" with a TI value of 2 are removed, the FPS of the GOP may be 1/4. That is, the FPS of the GOP may be changed from 60 to 15.

In the provision of TS, video may have a fixed FPS if only TI is used. For example, in the playback of 60 FPS video, if a frame with TI of 3 is excluded from playback, a fixed 30 FPS may be applied to the entire video. Thus, when a TS considering only TI is used, if there is a fast moving object in the middle of the video, a large quality deterioration that a viewer can perceive about a fast moving object may occur.

Improved temporal scalability

Hereinafter, an encoding method and a decoding method for providing an improved temporal scaler rule using the image quality satisfaction information generated by the SURP 300 will be described.

With the above-described SURP 300, a TS can be applied to each GOP. That is to say, using the above-described SURP 300, the image quality satisfaction of the GOP can be predicted for each FPS of a plurality of FPSs. Using the picture quality predictions of GOP, TS can be provided while maintaining image quality.

The SURP 300 may determine, for each of the TIs, image quality satisfaction information corresponding to each TI. The image quality satisfaction information can indicate the image quality satisfaction. Here, the image quality satisfaction information corresponding to a specific TI may indicate the image quality satisfaction with the image quality when the frames of the specific TI or more TI (s) are removed from the encoding, decoding, or reproduction.

For example, "frame x" may be included in playback or excluded from playback depending on what FPS the GOP is played back in. Depending on the FPS, "frame x" may be excluded from decryption or playback by the TS. In this case, if the maximum FPS of the FPSs in which "frame x" is not included in the reproduction is y, the image quality satisfaction when the FPS is y is z, and the TI of "frame x & The image quality satisfaction may be z.

Further, in a GOP, frames having the same TI can have the same image quality satisfaction information in common. Accordingly, depending on the required image quality satisfaction, the decoding apparatus 2700 to be described later can adaptively select whether to reproduce frames having a specific TI.

For example, the required image quality satisfaction Z may indicate that frames of TI with a corresponding image quality satisfaction of Z or below should be included in the playback. This determination may also indicate that frames of TI corresponding to image quality satisfaction greater than Z may be excluded from playback when the required image quality satisfaction is Z. [

The image quality satisfaction corresponding to the TI may be included in the SEI message of the access unit of the frame and may be included in the NAL unit header. At this time, the SEI message or the NAL unit header may directly include the value of the image quality satisfaction information. Alternatively, the SEI message or NAL unit header may include the value of the index of the picture quality satisfaction table.

In a GOP, frames having the same TI may have the same picture quality satisfaction information. Therefore, in the GOP, picture quality satisfaction information can be transmitted from the encoding device 2400 to the decoding device 2700 through the SEI only in the first frame of each TI for each TI. Alternatively, in the first frame of the GOP, image quality satisfaction information for all TIs may be transmitted at one time.

FIG. 19 shows a message including the image quality satisfaction according to an example.

In Fig. 19, the SEI message of an access unit of a frame may include image quality satisfaction information. That is to say, the image quality satisfaction information can be provided through Supplemental Enhancement Information (SEI) message of the access unit of the frame.

In Fig. 19, "SEI_Temporal_ID_SURP" may indicate data used to provide image quality satisfaction information. "Surp_value" may indicate image quality satisfaction information.

For example, a value of "Surp_value" of 70 indicates that the image quality satisfaction is "70%".

FIG. 20 shows a message including an index of a picture quality satisfaction table according to an example.

In Fig. 19, the SEI message of an access unit of a frame may include image quality satisfaction information. That is to say, the image quality satisfaction information can be provided through Supplemental Enhancement Information (SEI) message of the access unit of the frame.

In Fig. 19, "SEI_Temporal_ID_SURP" may indicate data used to provide image quality satisfaction information. "Surp_value_idx" may be an index of the image quality satisfaction table.

The values of the picture quality satisfaction used in the video or GOP may be predefined as a table in the picture quality satisfaction table. For example, the image quality satisfaction table may have values of {90, 60, 30}. These values may indicate that image quality levels of 90, 60, and 30 are used. If the value of "Surp_value_idx" is 0, it can indicate that the value of the index 0 in the image quality satisfaction table, that is, 90 or 90% is the image quality satisfaction value. If the value of "Surp_value_idx" is 1, it can indicate that the value of the index 0 in the image quality satisfaction table, that is, 60 or 60% is the image quality satisfaction value.

FIG. 21 shows a message including the entirety of image quality satisfaction according to an example.

In FIG. 21, one SEI message may include image quality satisfaction information of all TIs of all frames of a GOP.

In FIG. 21, "Max_Temporl_ID" may represent the maximum number of TIs. The "Temoporal ID [i]" may represent the index of TI whose value is i or i1. Alternatively, "Temoporal ID [i]" may indicate the image quality satisfaction information of the i-th TI.

The image quality satisfaction information may indicate the value of the image quality satisfaction itself or the index of the image quality satisfaction table. In FIG. 21, it is illustrated that "Temoporal ID [i]" represents the index of the image quality satisfaction table.

FIG. 22 shows a method of generating image quality satisfaction information according to an example.

22 is a flowchart of a method of encoding a GOP including image quality satisfaction information according to an example.

First, input video is illustrated as being 60 FPS. The GOPs of the input video may be processed sequentially by steps 2210, 2220, 2230, 2240 and 2250.

In step 2210, the SURP 300 may convert a 60 FPS GOP to a 30 FPS GOP and calculate a picture quality rating of a 30 FPS GOP.

In step 2220, the SURP 300 may convert a 60 FPS GOP to a 15 FPS GOP and calculate a picture quality rating of a 15 FPS GOP.

In step 2230, the SURP 300 may convert a GOP of 60 FPS to a GOP of 7.5 FPS and calculate a picture quality rating of a GOP of 7.5 FPS.

In step 2240, the SURP 300 may perform encoding of image quality satisfaction information for the frames of the GOP.

In step 1650, the SURP 300 may perform the encoding of the GOP. It is possible to perform encoding of a plurality of frames of the GOPs of the SURP 300.

In the embodiment of FIG. 22, all frames of GOPs can be encoded. Which frame to reproduce from among all the frames can be determined at the encoding stage according to the minimum image quality satisfaction.

FIG. 23A shows a configuration for maintaining image quality satisfaction of 75% or more according to an example.

In Figure 23A, the rectangle may represent a frame. In Figure 23A, the first GOP may include "frame 0" through "frame 7 ". The second GOP may include "frame 8" through "frame 15 ". The "TI" inside the rectangle may represent the value of the temporal identifier of the frame. The "I "," P ", and "B" inside the rectangle can indicate the type of frame. The number above or below the rectangle can indicate the image quality satisfaction of the frame.

The SURP 300 can convert each GOP of the GOPs of the video into a minimum FPS to satisfy the minimum image quality satisfaction, and can encode each converted GOP. In this case, the encoded video at the minimum bit rate required to meet the minimum picture quality satisfaction may be transmitted. This scheme can be advantageous in terms of transmission.

In Fig. 23A, a case in which TS is applied so as to ensure at least 75% of image quality satisfaction is exemplified.

In Fig. 23A, frames in the dashed line may indicate frames that are not encoded or transmitted. In Fig. 23A, in the first GOP, frames may be transmitted at 30 FPS so that a minimum picture quality satisfaction of 75% is satisfied. Also, in the second GOP, frames may be transmitted at 15 FPS so that a minimum picture quality satisfaction of 75% is satisfied.

The SURP 300 may include the image quality satisfaction information of the TIs in the bitstream. The decoding apparatus 2700 can determine the minimum FPS for satisfying the minimum picture quality satisfaction with reference to the picture quality satisfaction information and can receive the frames required for the determined FPS from the encoding apparatus. The decoder may play the received frames.

The decoding apparatus 2700 can determine the minimum FPS for satisfying the minimum picture quality satisfaction with respect to the GOP by referring to the picture quality satisfaction information. Decryption unit 2700 may perform decryption of the frames required for the determined FPS. Here, the frames may be frames of a GOP. And receive frames required for the determined FPS from the encoder. The decoder may play the received frames.

The decoding apparatus 2700 can selectively obtain only a frame determined to perform decoding among the video or GOP frames from the encoding apparatus 2500 through the TS.

The decoding apparatus 2700 can selectively decode a frame having the following picture quality satisfaction level of the minimum picture quality satisfaction among the frames of the GOP. By excluding a frame having a picture quality greater than the minimum picture quality satisfaction from decoding, the decoding apparatus 2700 can efficiently perform video transmission and reproduction without deteriorating the image quality recognized by the viewer.

FIG. 23B shows a configuration for determining whether to change the FPS based on 75% picture quality satisfaction according to an example.

In Fig. 23B, the rectangle may represent a frame. In Figure 23B, the first GOP may include "frame 0" through "frame 7 ". The second GOP may include "frame 8" through "frame 15 ". The "TI" inside the rectangle may represent the value of the temporal identifier of the frame. The "I "," P ", and "B" inside the rectangle can indicate the type of frame. The number above or below the rectangle can indicate the image quality satisfaction of the frame.

The SURP 300 basically changes the FPS of the GOP to 30, but if the image quality satisfaction is less than 75%, the FPS of the source video can be maintained without changing the FPS of the GOP. That is to say, the SURP 300 can adaptively use the TS based on the minimum image quality satisfaction. Such a function may be particularly required in an environment in which image quality is important.

In FIG. 24B, the first GOP can be encoded and transmitted at 30 FPS, and the second GOP can be encoded and transmitted at 60 FPS.

24 is a structural diagram of an encoding apparatus according to an embodiment.

The encoding device 2400 may include a control unit 2410, an encoding unit 2420, and a communication unit 2430.

The control unit 2410 may correspond to the SURP 300 described above. For example, the function of the above-described SURP 300 can be performed by the control unit 2410. [ Alternatively, the control unit 2410 may perform the SURP 300.

The control unit 2410 may generate selection information on a moving image frame.

The selection information may correspond to the above-described image quality satisfaction information. Alternatively, the selection information may include image quality satisfaction information.

The selection information may be related to the percentage of people who are not aware of the degradation of the video quality even if the frame is excluded from the encoding of the video.

Alternatively, the selection information may be related to a ratio of persons who do not perceive the deterioration of the picture quality of the moving picture to be reproduced even if the reproduction of the moving picture is set so that the frame is excluded from the decoding.

The selection information may be plural. The plurality of selection information may be calculated for each FPS of the reproduction of the moving image or the basic determination unit. The control unit 2410 may calculate selection information of a plurality of FPSs.

The ratio of persons who are not aware of the deterioration of the picture quality of the moving picture can be calculated by the machine learning of the control unit 2410. [ As described above, the ratio of persons who are not aware of the degradation of the picture quality of the moving picture can be determined based on the characteristic vector of the basic determination unit including the frame or the frame.

The selection information can be commonly applied to other frames having the same TI as the TI of the frame among the plurality of frames including the frame. The plurality of frames may be frames of a basic determination unit. Alternatively, the plurality of frames may be frames of a GOP.

The control unit 2410 may encode the moving picture based on the generated selection information.

For example, the control unit 2410 may encode the selection information and may include the encoded selection information in the bitstream of the encoded moving image.

For example, the control unit 2410 can select a frame to be encoded out of frames of the moving image based on the selection information.

The encoding unit 2420 may correspond to the encoding apparatus 100 described above. For example, the above-described function of the encoding apparatus 100 can be performed by the encoding unit 2420. [ Alternatively, the encoding unit 2420 may include an encoding device 100. [

The encoding unit 2420 can perform encoding of a moving image frame.

For example, the encoding unit 2420 may encode the entire frames of the moving picture.

For example, the encoding unit 2420 can perform encoding of a frame selected by the control unit 2410. [

The communication unit 2430 can transmit the generated bit stream to the decoding device 2700. [

The bitstream may include information on the encoded moving picture, and may include encoded selection information.

Functions and operations of the control unit 2410, the encoding unit 2420, and the communication unit 2430 will be described in detail below.

25 is a flowchart of a coding method according to an embodiment.

In step 2510, the control unit 2410 may generate selection information on the frame of the moving picture.

In step 2520, the control unit 2410 may perform encoding of the moving picture based on the selection information.

The encoding unit 2420 can perform encoding of a moving image frame. The encoding unit 2420 can generate a bit stream by performing encoding of a moving picture frame. The bit stream generated by the encoding of the moving picture may include selection information.

In step 2530, the communication unit 2430 may transmit the bitstream including the information of the encoded moving picture to the decoding apparatus 2700. [

26 is a flowchart of a method of performing motion picture coding according to an example.

The above-described step 2520 with reference to FIG. 25 may include the following steps 2610 and 2620.

In step 2610, the control unit 2410 may determine whether to encode the frame based on the selection information for the frame.

In step 2620, when the encoding of the frame is determined, the encoding unit 2420 can perform encoding of the frame.

If the encoding of the frame is determined, the control unit 2410 can request the encoding unit 2420 to encode the frame.

The determination as to whether or not the frame is coded can be commonly applied to other frames having the same TI as the TI of the frame among the plurality of frames including the frame. Here, the plurality of frames may be frames of a basic determination unit. Alternatively, the plurality of frames may be frames of a GOP.

27 is a structural diagram of a decoding apparatus according to an embodiment.

The decryption apparatus 2700 may include a control unit 2710, a decryption unit 2720, and a communication unit 2730.

The communication unit 2730 can receive the bit stream from the encoding device 2400. [

The bitstream may include information of the encoded moving picture. The information of the encoded moving picture may include information of the encoded frame. The bitstream may include frame selection information.

The controller 2710 may determine whether to decode the frame based on the frame selection information.

The control unit 2710 may perform a function applicable to decryption among the functions of the SURP 300 described above. Alternatively, the control unit 2710 may include at least a portion of the SURP 300.

The decoding unit 2720 may correspond to the decoding apparatus 200 described above. For example, the function of the above-described decryption apparatus 200 may be performed by the decryption unit 2720. [ Alternatively, the decryption unit 2720 may include a decryption apparatus 200.

When decryption of the frame is determined, the decryption unit 2720 can decrypt the frame.

For example, the decoding unit 2720 can perform decoding of the entire frames of the moving picture.

For example, the decoding unit 2720 can perform encoding of the frame selected by the control unit 2710. [

Functions and operations of the control unit 2710, the decryption unit 2720, and the communication unit 2730 will be described in detail below.

28 is a flowchart of a decoding method according to an embodiment.

In step 2810, the communication unit 2730 can receive the bit stream. The communication unit 2730 can receive the information of the encoded frame.

In step 2820, the control unit 2710 may determine whether to decode the frame based on the frame selection information.

The determination may be performed for each frame of a plurality of frames of video. Here, the plurality of frames may be frames of a basic determination unit. Alternatively, the plurality of frames may be frames of a GOP.

The selection information may correspond to the above-described image quality satisfaction information. Alternatively, the selection information may include image quality satisfaction information.

The plurality of frames of image quality satisfaction information selection information may be related to a ratio of persons who are not aware of the degradation of the image quality of the moving image even if the reproduction of the moving image including the frame is determined so that the frame is excluded from the decoding.

Alternatively, the selection information may be related to a ratio of persons who do not perceive degradation in the picture quality of the moving picture even though the FPS of the reproduction of the moving picture including the frame is determined so that the frame is excluded from the decoding.

The selection information may be included in the SEI for the frame.

Whether or not the frame is decoded can be determined based on a comparison between the value of the selection information and the value of the reproduction information.

The reproduction information may be information related to the reproduction of the moving picture including the frame. The reproduction information can correspond to the minimum image quality satisfaction. Alternatively, the playback information may include a minimum image quality satisfaction.

The playback information may be information related to the FPS of playback of the moving image including the frame.

For example, if the value of the selection information is larger than the value of the reproduction information, the control unit 2710 can determine not to perform decoding on the frame. Alternatively, if the value of the selection information is equal to or larger than the reproduction information, the control unit 2710 can determine to decode the frame.

The determination can be commonly applied to other frames having the same TI as the TI of the frame among the plurality of frames including the frame. The plurality of frames may be frames of a basic prediction unit. Alternatively, the plurality of frames may be frames of a GOP.

Decoding Apparatus 2700 Multiple Frames The arbitrary decoding apparatus 2700 decodes arbitrary frames 0,

In step 2830, when decryption of the frame is determined, the decryption unit 2720 can decrypt the frame with respect to the decrypted frame.

Decryption device 2700

In the above-described embodiments, although the methods are described on the basis of a flowchart as a series of steps or units, the present invention is not limited to the order of the steps, and some steps may occur in different orders or simultaneously . It will also be understood by those skilled in the art that the steps depicted in the flowchart illustrations are not exclusive and that other steps may be included or that one or more steps in the flowchart may be deleted without affecting the scope of the invention You will understand.

The embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specially designed and constructed for the present invention or may be those known and used by those skilled in the computer software arts. Examples of the computer-readable recording medium include magnetic media such as a hard disk, a floppy disk and a magnetic tape, optical recording media such as CDROM and DVD, magnetooptical media such as a floptical disk, , RAM, flash memory, and the like, which are specifically configured to store and execute program instructions. Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.

Therefore, the spirit of the present invention should not be construed as being limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the following claims, I will say.

Claims (20)

Determining whether to decode the frame based on selection information of the frame; And
Performing decoding of the frame if decoding of the frame is determined
And decoding the moving picture. The method according to claim 1,
The determination is performed for each frame of a plurality of frames,
Wherein the plurality of frames are frames of a group of pictures (GOP). The method according to claim 1,
Wherein the selection information is related to a ratio of a person who is not aware of a degradation in the picture quality of the moving picture even though the reproduction of the moving picture including the frame is set so that the frame is excluded from the decoding. The method according to claim 1,
Wherein the selection information includes at least one of a moving picture decoding method and a moving picture decoding method related to a ratio of a person who does not perceive degradation of a picture quality of a moving picture even if frames per second (FPS) of the moving picture reproduction including the frame is determined so that the frame is excluded from decoding . The method according to claim 1,
Whether or not the frame is decoded is determined based on a comparison between the value of the selection information and the value of the reproduction information,
Wherein the reproduction information is information related to reproduction of a moving picture including the frame. 6. The method of claim 5,
Wherein the reproduction information is information related to frames per second (FPS) of reproduction of a moving picture including the frame. The method according to claim 1,
It is determined that the decoding is not performed on the frame if the value of the selection information is larger than the value of the reproduction information,
Wherein the reproduction information is information related to reproduction of a moving picture including the frame. The method according to claim 1,
Wherein the selection information is contained in Supplemental Enhancement Information (SEI) for the frame. The method according to claim 1,
Wherein the determination as to whether or not the decoding is to be performed is commonly applied to another frame having a temporal identifier identical to a temporal identifier of the frame among a plurality of frames including the frame. A controller for determining whether to decode the frame based on frame selection information; And
A decoding unit for decoding the frame when decoding of the frame is determined,
And the moving picture decoding apparatus. Generating selection information for a frame of a moving picture; And
Performing encoding of the moving picture based on the selection information
And a moving picture coding method. 12. The method of claim 11,
The encoding of the moving picture may include:
Determining whether to encode the frame based on the selection information for the frame; And
Performing encoding of the frame if encoding of the frame is determined;
And a moving picture coding method. 13. The method of claim 12,
Wherein the determination as to whether or not the encoding is performed is commonly applied to another frame having a temporal identifier identical to a temporal identifier of the frame among a plurality of frames including the frame. 12. The method of claim 11,
Wherein the selection information is related to a ratio of persons who are not aware of a decrease in picture quality of the moving picture even though the frame is excluded from coding of the video. 15. The method of claim 14,
Wherein the ratio is calculated by machine learning. 15. The method of claim 14,
Wherein the ratio is determined based on a feature vector of a frame. 12. The method of claim 11,
The selection information is plural,
Wherein the plurality of selection information is calculated for frames per second (frames per second) of reproduction of moving pictures. 12. The method of claim 11,
Wherein the bitstream generated by encoding the moving picture includes the selection information. 12. The method of claim 11,
Wherein the selection information is commonly applied to another frame having a temporal identifier equal to a temporal identifier of the frame among a plurality of frames including the frame. 12. The method of claim 11,
Wherein the selection information is related to a ratio of a person who does not perceive degradation in the picture quality of the moving picture to be reproduced even if the moving picture is set to be reproduced so that the frame is excluded from decoding.

Images