JP2020156044A - Video image encoding device, video image decoding device and program - Google Patents

Abstract

To perform inter-screen prediction while compensating a blur due to encoding without increasing types of super-resolution images.SOLUTION: A high-frequency power calculation unit 10 of a video image encoding device 1 calculates high-frequency power p of an I frame. Super-resolution calculation means 12 calculates a super-resolution α on the basis of the high-frequency power p, an encoding order number n and a QP off-set value Qoff. Super-resolution processing means 13 performs super-resolution processing using the resolution α for a local decoding frame K, and generates a super-resolution frame C. Inter-screen prediction means 15 performs inter-screen prediction and calculates prediction images KY and CY, and prediction errors KE and CE for each area on the basis of a standard picture being a current local decoding frame K(P, B), and reference pictures being a past local decoding frame K and a super-resolution frame C. Selection means 16 selects a prediction image, in which RD cost of the prediction errors KE and CE, of prediction images KY and CY for each area, and outputs an inter-screen prediction result.SELECTED DRAWING: Figure 3

Description

The present invention relates to a moving image coding device that encodes a moving image, a moving image decoding device that decodes a coded signal, and a program thereof, and in particular, blurring due to coding in order to improve interscreen prediction efficiency. It relates to a technique for performing inter-screen prediction while compensating for super-resolution.

Conventionally, H.265 / HEVC (High Efficiency Video Coding) is known as a standard of a moving image compression method for a high-resolution 4K or 8K moving image (see, for example, Non-Patent Document 1). This H.265 / HEVC standard is based on motion compensation and orthogonal conversion techniques that predict motion using encoded frames and orthogonally transform and encode the signal of the predicted residuals.

FIG. 9 is a block diagram showing a configuration example of a conventional moving image coding device, and shows a configuration on the coding side according to the H.265 / HEVC standard. The moving image coding device 100 includes a subtraction unit 110, an orthogonal conversion unit 111, a quantization unit 112, an inverse quantization unit 113, an inverse orthogonal conversion unit 114, an addition unit 115, an in-screen prediction unit 116, and an in-loop filter 117. It includes an inter-screen prediction unit 118, a switch 119, and an entropy coding unit 120. The inter-screen prediction unit 118 includes inter-screen prediction means 121 and a frame memory 122.

The moving image coding device 100 is a device that performs coding processing on each frame of an input moving image to generate and output a coded signal.

The frame of the input moving image is subjected to subtraction processing by the subtraction unit 110 with the prediction image generated by the in-screen prediction unit 116 or the inter-screen prediction unit 118. As a result, a residual image is generated.

The residual image is subjected to orthogonal conversion processing by the orthogonal conversion unit 111, quantization processing by the quantization unit 112, and entropy coding by the entropy coding unit 120, and is output as a coded signal.

On the other hand, the quantization index sequence subjected to the quantization processing by the quantization unit 112 is subjected to the inverse quantization processing by the inverse quantization unit 113 and the inverse orthogonal conversion processing by the inverse orthogonal conversion unit 114. As a result, a decoding residual image is generated.

The decoded residual image is subjected to addition processing with the predicted image by the addition unit 115. The image of the addition result by the addition unit 115 is subjected to the in-screen prediction processing by the in-screen prediction unit 116. Further, the image of the addition result is filtered by the in-loop filter 117 and stored in the frame memory 122 of the inter-screen prediction unit 118 as a local decoding frame.

The local decoding frame stored in the frame memory 122 is read out as a past local decoding frame by the inter-screen prediction means 121. Then, the inter-screen prediction processing is performed using the current local decoding frame (frame of the input moving image) as a reference picture and the past local decoding frame as a reference picture.

One of the image generated by the in-screen prediction process by the in-screen prediction unit 116 and the image generated by the in-screen prediction process by the inter-screen prediction means 121 is selected by the switch 119 and used as the prediction image. It is input to the subtraction unit 110 and the addition unit 115.

In the moving image coding technology based on such motion compensation and orthogonal conversion, a technology for improving the coding efficiency has been proposed. For example, the image is reduced, then encoded and transmitted, and after decoding, the super-resolution is restored, and the optimum folding frequency is set according to the complexity of the image and transmitted to obtain super-resolution image quality. There is a technique for enhancing (see, for example, Patent Document 1).

Japanese Patent No. 6344800

Ei Okubo, Teruhiko Suzuki, Masayuki Takamura, Ken Nakajo, "Impress Standard Textbook Series H.265 / HEVC Textbook", Impress Japan

In the conventional moving image coding device 100 shown in FIG. 9, the in-screen prediction unit 116 performs in-screen prediction processing for the first I frame among the I, P, and B frames of the input moving image. In this in-screen prediction process, the I-frame is divided into block regions, orthogonal conversion is performed by the orthogonal conversion unit 111, and the quantization matrix is multiplied by the quantization unit 112 in order to reduce the high frequency power. Orthogonal. This results in lossy compression, but can reduce a large amount of information.

For the subsequent P and B frames, the inter-screen prediction processing is performed by the inter-screen prediction unit 118 using the local decoding frames obtained by the past in-screen prediction processing and the inter-screen prediction processing. , In-screen prediction processing is performed by the in-screen prediction unit 116 in a part of the frame.

In such a coding process, basically, the later the coding order is, the larger the amount of blurring due to coding tends to be. Therefore, in the conventional inter-screen prediction processing, there is a problem that the coding efficiency is lowered and the coded image quality is easily deteriorated.

Then, in order to compensate for this blurring, a method of performing inter-screen prediction using a super-resolution frame is assumed. However, this method has a problem that the circuit scale and the processing time increase because a large number of super-resolution frames with super-resolution must be prepared.

Therefore, the present invention has been made to solve the above problems, and an object of the present invention is to perform inter-screen prediction while compensating for blurring caused by coding without increasing the types of super-resolution images. It is an object of the present invention to provide a possible moving image encoding device, moving image decoding device and program.

In order to solve the above-mentioned problems, the moving image coding apparatus according to claim 1 subtracts a predicted image from the input moving image to generate a residual image, and orthogonally transforms, quantizes, and entropy code the residual image. In addition to generating and outputting a coded signal, the quantization index sequence generated by the quantization is subjected to dequantization and inverse orthogonal conversion to generate a decoding residual image, and the decoding residual image is generated. The predicted image is added to to generate an image after addition, the added image is filtered to generate a local decoding frame, and interscreen prediction is performed using the local decoding frame to generate the predicted image. A super-resolution frame is generated based on the local decoding frame, and the predicted image of the local decoding frame and the super-resolution frame of the super-resolution frame are described for each predetermined region by the inter-screen prediction. An inter-screen prediction unit that generates a prediction image and selects any one of the above-mentioned prediction images is provided, and the inter-screen prediction unit calculates the super-resolution to be used when generating the super-resolution frame. A super-resolution processing means for generating the super-resolution frame by performing super-resolution processing using the super-resolution calculated by the super-resolution calculation means on the local decoding frame. The screen-to-screen prediction is performed using the current local decoding frame as a reference picture, the past local decoding frame, and the past super-resolution frame generated by the super-resolution processing means as a reference picture, and the predetermined area. Each time, the inter-screen prediction means for generating the predicted image of the local decoding frame and the predicted image of the super-resolution frame, and the predicted image and the super-resolution of the local decoding frame for each predetermined region. The super-resolution calculation means includes a selection means for selecting one of the predicted images of the frame, and the super-resolution calculation means assigns a coding order number from the I frame included in the frame of the input moving image in the coding order. As a number, the super-resolution is calculated based on the coding sequence number, the QP offset value of the frame of the input moving image, or the coding sequence number and the QP offset value. To do.

Further, the moving image coding device according to claim 2 is the moving image coding device according to claim 1, further, the orthogonal conversion, the quantization, and the inverse quantization with respect to the I frame of the input moving image. A first high-frequency power calculation unit that calculates the high-frequency power based on the decoding frame in which the inverse orthogonal conversion is performed is provided, and the super-resolution calculation means is the high-frequency power calculated by the first high-frequency power calculation unit. , Or, the super-resolution is calculated based on the high-frequency power and the coding sequence number, or the high-frequency power and the QP offset value, or the high-frequency power, the coding sequence number, and the QP offset value. , Characterized by.

Further, the moving image coding device according to claim 3 is the moving image coding device according to claim 1, wherein the added image of the I frame of the input moving image is further filtered. A second high-frequency power calculation unit that calculates high-frequency power based on a local decoding frame is provided, and the super-resolution calculation means is the high-frequency power calculated by the second high-frequency power calculation unit, or the high-frequency power and the above. It is characterized in that the super-resolution is calculated based on the coding sequence number or the high frequency power and the QP offset value, or the high frequency power, the coding sequence number and the QP offset value.

Further, the moving image decoding device according to claim 4 inputs a moving image encoded signal, performs entropy decoding, inverse quantization, and inverse orthogonal conversion on the encoded signal to generate a decoding residual image, and the decoding is performed. The predicted image is added to the residual image to generate the added image, and the added image is filtered to generate a decoded frame, thereby restoring the original moving image and using the decoded frame. In the moving image decoding device that performs inter-screen prediction and generates the predicted image, the coded signal includes one type of the predicted image selected from the predicted image of the local decoding frame and the predicted image of the super-resolution frame, and When a super-resolution parameter for generating the super-resolution frame is included, the decoded frame is subjected to super-resolution processing using the super-resolution included in the parameter, and the super-resolution frame is performed. To generate the predicted image of any one of the decoded frame and the super-resolution frame according to the type of the predicted image included in the parameter for each predetermined region by the inter-screen prediction. It is characterized by having an interval prediction unit.

Further, the program of claim 5 is characterized in that the computer functions as the moving image coding device according to any one of claims 1 to 3.

The program of claim 6 is characterized in that the computer functions as the moving image decoding device according to claim 4.

As described above, according to the present invention, it is possible to perform inter-screen prediction while compensating for blurring caused by coding without increasing the types of super-resolution images.

It is a block diagram which shows the structural example of the moving image coding apparatus by embodiment of this invention. It is a figure explaining the processing example of the high frequency power calculation part. It is a block diagram which shows the structural example of the inter-screen prediction part provided in the moving image coding apparatus. It is a flowchart which shows the processing example of the interscreen prediction unit provided in the moving image coding apparatus. It is a figure explaining the example of the reference picture set information and QP offset value Q _off used by the super-resolution calculation means. It is a figure explaining the processing example (step S404) of the super-resolution processing means. It is a block diagram which shows the structural example of the moving image decoding apparatus by embodiment of this invention. It is a block diagram which shows the structural example of the interscreen prediction part provided in the moving image decoding apparatus. It is a block diagram which shows the structural example of the conventional moving image coding apparatus.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. The present invention is super-resolution used in inter-screen prediction processing based on any one of the frequency spectrum power of the local decoding frame of the I-frame, the coding order number from the I-frame, and the QP offset value Q _off. It is characterized by calculating the super-resolution of the resolution frame.

As a result, blur compensation can be performed using one super-resolution frame. Therefore, it is possible to perform inter-screen prediction while compensating for blurring caused by coding without increasing the types of super-resolution images.

Further, the decoding side generates a super-resolution frame using the super-resolution sent as a parameter from the coding side, and performs inter-screen prediction using the super-resolution frame. This makes it possible to perform inter-screen prediction while compensating for blurring caused by coding without increasing the types of super-resolution images.

[Video encoding device]
First, a moving image coding device according to an embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of a moving image coding device according to an embodiment of the present invention. The moving image coding device 1 includes a subtraction unit 110, an orthogonal conversion unit 111, a quantization unit 112, an inverse quantization unit 113, an inverse orthogonal conversion unit 114, an addition unit 115, an in-screen prediction unit 116, and a high frequency power calculation unit 10. , An in-loop filter 117, an inter-screen prediction unit 11, a switch 119, and an entropy coding unit 120. Note that FIG. 1 shows only the components related to the present invention, and the components not related to the present invention are omitted.

The subtraction unit 110 inputs a frame (I frame, P frame, and B frame) of the input moving image, and inputs a predicted image of the frame from the switch 119. Then, the subtraction unit 110 subtracts the predicted image from the frame of the input moving image, generates a residual image of the subtraction result, and outputs the residual image to the orthogonal conversion unit 111.

The orthogonal conversion unit 111 inputs a residual image from the subtraction unit 110, performs orthogonal conversion on the residual image, and generates a conversion coefficient sequence. Then, the orthogonal conversion unit 111 outputs the conversion coefficient sequence to the quantization unit 112.

The quantization unit 112 inputs a conversion coefficient sequence from the orthogonal conversion unit 111, performs quantization on the conversion coefficient sequence, and generates a quantization index sequence. Then, the quantization unit 112 outputs the quantization index sequence to the inverse quantization unit 113 and the entropy coding unit 120.

The inverse quantization unit 113 inputs the quantization index string from the quantization unit 112 and performs the reverse processing of the quantization unit 112 to inversely quantize the quantization index sequence and generate a conversion coefficient sequence. Then, the inverse quantization unit 113 outputs the conversion coefficient sequence to the inverse orthogonal conversion unit 114.

The inverse orthogonal conversion unit 114 inputs the conversion coefficient sequence from the inverse quantization unit 113 and performs the reverse processing of the orthogonal conversion unit 111 to perform inverse orthogonal conversion of the conversion coefficient sequence and generate a decoding residual image. Then, the inverse orthogonal conversion unit 114 outputs the decoding residual image to the addition unit 115.

The addition unit 115 inputs the decoding residual image from the inverse orthogonal conversion unit 114, and inputs the predicted image from the switch 119. Then, the adding unit 115 adds the decoding residual image to the predicted image, and outputs the added image (added image) to the in-screen prediction unit 116, the high frequency power calculation unit 10, and the in-loop filter 117.

The in-screen prediction unit 116 inputs the added image from the adding unit 115, performs in-screen prediction processing on the added image for each predetermined area, generates an in-screen prediction result, and generates an in-screen prediction result for each predetermined area. Is output to the switch 119.

The high-frequency power calculation unit 10 inputs an I-frame decoded image (I-frame decoded image) of the added image from the addition unit 115, calculates the high-frequency power p based on the I-frame decoded image, and calculates the I-frame decoded image. High-frequency power p is output to the inter-screen prediction unit 11. The details of the high frequency power calculation unit 10 will be described later.

Here, in the I-frame decoded image, the processing of the orthogonal conversion unit 111, the processing of the quantization unit 112, the processing of the inverse quantization unit 113, and the processing of the inverse orthogonal conversion unit 114 are performed on the I frame of the input moving image. It is a decoded image obtained by this.

The in-loop filter 117 inputs the added image from the adding unit 115, filters the added image, and generates a local decoding frame K. Then, the in-loop filter 117 outputs the local decoding frame K to the inter-screen prediction unit 11.

As a result, when the added image is an I frame, a local decoding frame K of the I frame is generated, and when the added image is a P frame, a local decoding frame K of the P frame is generated, and the added image is a B frame. In this case, the local decoding frame K of the B frame is generated.

The inter-screen prediction unit 11 inputs the high-frequency power p of the I-frame decoded image from the high-frequency power calculation unit 10, and inputs the local decoding frame K from the in-loop filter 117. Further, the inter-screen prediction unit 11 inputs preset reference picture offset information and QP (Quality of Picture) offset value Q _off . Details of the reference picture offset information and the QP offset value Q _off will be described later.

The inter-screen prediction unit 11 calculates the super-resolution α based on the high-frequency power p, the reference picture offset information, and the QP offset value, and performs super-resolution processing on the local decoding frame K using the super-resolution α to super-resolution. The resolution frame C is generated.

As a result, when the local decoding frame K input to the inter-screen prediction unit 11 is the local decoding frame K of the I frame, the super-resolution frame C of the I frame is generated, and when the local decoding frame K of the P frame is P. The super-resolution frame C of the frame is generated. Further, in the case of the local decoding frame K of the B frame, the super-resolution frame C of the B frame is generated.

The inter-screen prediction unit 11 uses a reference picture which is the current local decoding frame K (P and B frames of the input moving image) and a reference picture of the local decoding frame K and the super-resolution frame C for each predetermined area. , Perform inter-screen prediction. The inter-screen prediction unit 11 obtains prediction images KY, CY, prediction errors KE, CE, motion vectors, and the like for each of the local decoding frame K and the super-resolution frame C for each predetermined area.

As a result, when the inter-screen prediction processing of the P frame is performed, the prediction image KY and the prediction error KE for the local decoding frame K of the P frame, and the prediction image CY and the prediction error for the super-resolution frame C of the P frame. CE is obtained. Further, when the inter-screen prediction processing of the B frame is performed, the prediction image KY and the prediction error KE for the local decoding frame K of the B frame, and the prediction image CY and the prediction error CE for the super-resolution frame C of the B frame are performed. Is obtained.

The inter-screen prediction unit 11 selects the smallest (smaller) prediction image of the prediction errors KE and CE for each predetermined area, and uses the predicted image selected for each predetermined area as the inter-screen prediction result for each predetermined area. Output to switch 119.

Further, the inter-screen prediction unit 11 sets the super-resolution α calculated for each frame as a parameter, and in the process of selecting the predicted image for each predetermined area, the type and calculation of the predicted image actually selected for each predetermined area. Set the motion vector etc. as parameters. The type of the predicted image indicates any one of a decoding frame and a super-resolution frame. Then, the inter-screen prediction unit 11 outputs the parameter to the entropy encoding unit 120. The details of the inter-screen prediction unit 11 will be described later.

The switch 119 inputs the in-screen prediction result for each predetermined area from the in-screen prediction unit 116, and also inputs the in-screen prediction result for each predetermined area from the inter-screen prediction unit 11, selects one of them, and predicts the image. Is output to the subtraction unit 110 and the addition unit 115.

The entropy encoding unit 120 inputs the quantization index sequence from the quantization unit 112, inputs the parameters from the inter-screen prediction unit 11, and further inputs the parameters from the orthogonal conversion unit 111 and the like. Then, the entropy coding unit 120 performs entropy coding on the quantization index sequence and the parameter, and generates a coded signal. The entropy coding unit 120 outputs the coded signal to the moving image decoding device 2 described later.

[High frequency power calculation unit 10]
Next, the high frequency power calculation unit 10 shown in FIG. 1 will be described in detail. FIG. 2 is a diagram showing a processing example of the high frequency power calculation unit 10.

The high-frequency power calculation unit 10 inputs an I-frame decoded image from the addition unit 115 (step S201), and decomposes the I-frame decoded image into first-order wavelets (step S202).

The high-frequency power calculation unit 10 uses RMS (Root Mean Square: root mean square) for the image after first-order wavelet decomposition using the element values (pixel values) of all phase positions in the horizontal, vertical, and diagonal high-frequency bands. ) Calculate the value (step S203).

The high-frequency power calculation unit 10 normalizes the RMS value in the high-frequency band of the I-frame decoded image, obtains the high-frequency power p of the I-frame decoded image, and outputs the high-frequency power p of the I-frame decoded image to the inter-screen prediction unit 11 ( Step S204).

In this way, the high-frequency power calculation unit 10 calculates the high-frequency power p from the I-frame decoded image. The high-frequency power p is a power value indicating the degree of deterioration of the I-frame decoded image, and the larger the power value, the smaller the deterioration, and the smaller the power value, the larger the deterioration. Further, the high frequency power p is used in the super-resolution calculation means 12 described later when calculating the super-resolution α for all the frames constituting the GOP (Group Of Picture).

[Inter-screen prediction unit 11]
Next, the inter-screen prediction unit 11 shown in FIG. 1 will be described in detail. FIG. 3 is a block diagram showing a configuration example of the inter-screen prediction unit 11 provided in the moving image coding device 1, and FIG. 4 is a processing example of the inter-screen prediction unit 11 provided in the moving image coding device 1. It is a flowchart which shows.

The inter-screen prediction unit 11 includes a super-resolution calculation means 12, a super-resolution processing means 13, a frame memory 14, an inter-screen prediction means 15, a selection means 16, and a parameter processing means 17. Note that FIG. 3 shows only the components related to the present invention, and the components not related to the present invention are omitted.

The inter-screen prediction unit 11 inputs the high-frequency power p of the I-frame decoded image from the high-frequency power calculation unit 10 (step S401). Further, the inter-screen prediction unit 11 inputs the local decoding frame K (I, P, B) from the in-loop filter 117, and stores the local decoding frame K in the frame memory 14 (step S402).

The local decoding frame K (I, P, B) indicates a local decoding frame K of the I frame, a local decoding frame K of the P frame, and a local decoding frame K of the B frame. As a result, the frame memory 14 stores the local decoding frame K of the I frame, the local decoding frame K of the P frame, and the local decoding frame K of the B frame.

The super-resolution calculation means 12 inputs preset reference picture offset information and QP offset value Q _off , and in a process described later, super-resolution calculation means based on the high frequency power p, reference picture offset information and QP offset value Q _off. The resolution α is calculated (step S403). Then, the super-resolution calculation means 12 outputs the super-resolution α to the super-resolution processing means 13.

The super-resolution processing means 13 inputs the super-resolution α from the super-resolution calculation means 12, and also inputs the local decoding frame K (I, P, B). Then, the super-resolution processing means 13 performs super-resolution processing using the super-resolution α on the local decoding frame K (I, P, B) to perform super-resolution frame C (I, P, B). Generate and store the super-resolution frame C (I, P, B) in the frame memory 14 (step S404).

The super-resolution frame C (I, P, B) indicates an I-frame super-resolution frame C, a P-frame super-resolution frame C, and a B-frame super-resolution frame C. As a result, in the frame memory 14, in addition to the local decoding frame K (I, B, P), the super-resolution frame C of the I frame, the super-resolution frame C of the P frame, and the super-resolution frame C of the B frame Is stored.

The inter-screen prediction means 15 converts the local decoding frame K (I, P, B) and the super-resolution frame C (I, P, B) stored in the frame memory 14 into the past local decoding frames K (I, P, B). B) and the super-resolution frame C (I, P, B) are used, and these are read out as reference pictures (step S405). Further, the inter-screen prediction means 15 inputs the current local decoding frame K (P and B frames of the input moving image).

The inter-screen prediction means 15 uses the current local decoding frame K as a reference picture, and uses the reference picture and the reference picture for each predetermined area, for example, by a block matching method, to predict images KY, CY (P, B) and Prediction errors KE and CE (P, B) are generated (step S406). Then, the inter-screen prediction means 15 outputs the prediction images KY, CY (P, B) and the prediction errors KE, CE (P, B) to the selection means 16.

The predetermined area is, for example, a slice area, a CTU (Coding Tree Unit) area, and a CU (Coding Unit) area.

The predicted images KY and CY (P) indicate the predicted image KY of the local decoding frame K of the P frame and the predicted image CY of the super-resolution frame C of the P frame. The predicted images KY and CY (B) show the predicted image KY of the local decoding frame K of the B frame and the predicted image CY of the super-resolution frame C of the B frame. Further, the prediction errors KE and CE (P) indicate the prediction error KE of the local decoding frame K of the P frame and the prediction error CE of the super-resolution frame C of the P frame. The prediction errors KE and CE (B) indicate the prediction error KE of the local decoding frame K of the B frame and the prediction error CE of the super-resolution frame C of the B frame.

The selection means 16 inputs the prediction images KY, CY (P, B) and the prediction errors KE, CE (P, B) from the inter-screen prediction means 15. Then, the selection means 16 selects the predicted image (smaller predicted image) having the smallest RD cost of the prediction errors KE and CE from the predicted images KY and CY for each predetermined region (step S407). The selection means 16 outputs the predicted image selected for each predetermined area to the switch 119 as an inter-screen prediction result for each predetermined area (step S408).

The parameter processing means 17 sets the super-resolution α calculated for each frame by the super-resolution calculation means 12 as a parameter. Further, the parameter processing means 17 sets the type of the predicted image selected for each predetermined area by the selection means 16, the motion vector calculated by the inter-screen prediction means 15, and the like as parameters. Then, the parameter processing means 17 outputs the parameter to the entropy coding unit 120.

When the inter-screen prediction processing of the P frame is performed in this way, the prediction image KY and the prediction error KE for the local decoding frame K of the P frame, and the prediction image CY and the prediction for the super-resolution frame C of the P frame are performed. The error CE is obtained. Then, among the predicted images KY and CY, the predicted image having the smallest prediction error KE and CE RD cost is selected and output as an interscreen prediction result. The same applies to the B frame.

[Super resolution calculation means 12]
Next, the process of the super-resolution calculation means 12 shown in FIG. 3 (step S403 in FIG. 4) will be described in detail. As described above, the super-resolution calculation means 12 calculates the super-resolution α based on the high-frequency power p, the reference picture offset information, and the QP offset value Q _off .

FIG. 5 is a diagram illustrating an example of reference picture set information and QP offset value Q _off used by the super-resolution calculation means 12. The reference picture set information is composed of reference picture information used for inter-screen prediction processing in GOP. The reference picture set information includes information on the frame input order number (input order number), the frame coding order number (coding order number) n, the frame type, and the reference relationship.

The input sequence number indicates the order of the frames in the GOP, and the coding sequence number n indicates the order of the frames in which the coding process is performed. The larger the coding sequence number n, the farther away from the I frame.

The QP offset value Q _off is a value representing the image quality of the frame, and is set for each frame. As QP offset value Q _off is large, the image quality is poor, as the QP offset value Q _off is small, indicating that the image quality is good. In this way, the reference picture set information and the QP offset value Q _off are preset for each frame constituting the GOP.

With reference to FIG. 5, for example, in the reference picture set information, the frame of the input sequence number = 0 and the coded sequence number n = 0 is an I frame. It is shown that this I frame is referred to by the B frame of the input sequence number = 1,2,4,8,16 (coding sequence number n = 5,4,5,3,2,1). The QP offset value Q _off of this I frame is 0.

The super-resolution calculation means 12 calculates (sets) a super-resolution α having a smaller value as the high-frequency power p is larger, and calculates a super-resolution α having a larger value as the high-frequency power p is smaller. This is because when the high frequency power p is large, the degree of blurring of the frame due to coding is small, and when the high frequency power p is small, the degree of blurring is large. This high frequency power p is used when calculating the super-resolution α of each frame constituting the GOP.

The super-resolution calculation means 12 calculates the super-resolution α having a larger value as the coding sequence number n is larger, and calculates the super-resolution α having a smaller value as the coding sequence number n is smaller. This is because when the coding sequence number n is large, the degree of blurring of the frame due to coding is large, and when the coding sequence number n is small, the degree of blurring is small.

The super-resolution calculation means 12 calculates the super-resolution α of a large value as the QP offset value Q _off is larger (the lower the image quality), and the smaller the QP offset value Q _off is (the better the image quality). , Calculate the super-resolution α with a small value. This is because when the QP offset value Q _off is large, the degree of blurring of the frame due to coding is large, and when the QP offset value Q _off is small, the degree of blurring is small.

Specifically, the super-resolution calculation means 12 uses the following formula using the high-frequency power p, the coding sequence number n, the QP offset value Q _off, etc., to obtain the super-resolution α _n for the frame of the coding sequence number _n. Is calculated.
[Number 1]
α _n = a (1 / p) (n / n _max ) (Q _off / Q _off-max ) ・ ・ ・ (1)

Here, a is a parameter, n _max is the period of the I frame which is an intra frame, that is, the maximum value of the coding sequence number n, and Q _off-max is the maximum value of the QP offset value Q _off. .. The super-resolution α _n is a real number of 1 or more.

With reference to FIG. 5, for example, for the super-resolution α ₃ having a coding sequence number n = 3, substituting n _max = 16, Q _off = 4, and Q _off-max = 6 with respect to the above equation (1). It is calculated.

As described above, when the high frequency power p is large, the coding sequence number n is small, or the QP offset value Q _off is small, the super-resolution calculation means 12 considers that the blurring due to coding is small, and the value is small. The super-resolution α is calculated. On the other hand, when the high frequency power p is small, the coding sequence number n is large, or the QP offset value Q _off is large, the super-resolution α having a large value is calculated assuming that the blurring due to coding is large.

[Super-resolution processing means 13]
Next, the processing of the super-resolution processing means 13 shown in FIG. 3 (step S404 in FIG. 4) will be described in detail. As described above, the super-resolution processing means 13 performs super-resolution processing using the super-resolution α on the local decoding frame K (I, P, B), and super-resolution frame C (I, P, B). ) Is generated.

FIG. 6 is a diagram illustrating a processing example of the super-resolution processing means 13, and shows the details of step S404 of FIG.

The super-resolution processing means 13 inputs the local decoding frame K (I, P, B) from the in-loop filter 117 (step S601), and decomposes the local decoding frame K, for example, into the first-order wavelet (step S602).

The super-resolution processing means 13 multiplies the element values (pixel values) of all phase positions in the horizontal, vertical, and diagonal high-frequency bands of the image after the first-order wavelet decomposition by the super-resolution α (step S603). ).

The super-resolution processing means 13 reconstructs the first-order wavelet after multiplication after the first-order wavelet decomposition (step S604). Then, the super-resolution processing means 13 generates the super-resolution frame C (I, P, B) and stores the super-resolution frame C (I, P, B) in the frame memory 14 (step S605).

In this way, the super-resolution processing means 13 generates an I-frame super-resolution frame C for the I-frame local decoding frame K, and generates a P-frame super-resolution frame C for the P-frame local decoding frame K. Will be done. Further, the super-resolution frame C of the B frame is generated for the local decoding frame K of the B frame.

As described above, according to the moving image coding device 1 of the embodiment of the present invention, the high-frequency power calculation unit 10 decomposes the I-frame decoded image into the first-order wavelet, and performs all in the horizontal, vertical, and diagonal high-frequency bands. The RMS value is calculated using the element value of the phase position of. Then, the high-frequency power calculation unit 10 normalizes the RMS value to obtain the high-frequency power p of the I-frame decoded image.

The super-resolution calculation means 12 of the inter-screen prediction unit 11 uses the super-resolution α according to the above equation (1) based on the high-frequency power p, the coding sequence number n included in the reference picture offset information, and the QP offset value Q _off. Is calculated.

The super-resolution processing means 13 of the inter-screen prediction unit 11 performs super-resolution processing using super-resolution α on the local decoding frame K for each local decoding frame K (I, P, B) to achieve super-resolution. Image frames C (I, P, B) are generated and stored in the frame memory 14.

The inter-screen prediction means 15 of the inter-screen prediction unit 11 performs inter-screen prediction for each of the P frame and the B frame. Specifically, the inter-screen prediction means 15 includes a reference picture which is the current local decoding frame K (P, B), the past local decoding frame K (I, P, B) read from the frame memory 14, and super-resolution. Based on the reference picture which is the resolution frame C (I, P, B), inter-screen prediction is performed for each predetermined area, and the prediction images KY and CY and the prediction errors KE and CE are obtained.

The selection means 16 of the inter-screen prediction unit 11 selects one of the prediction images KY and CY for each predetermined area, and in this case, the prediction image (the smaller one) that minimizes the RD cost of the prediction errors KE and CE. Prediction image) is selected and this is output as an inter-screen prediction result.

As described above, as the reference picture used for the inter-screen prediction processing, in addition to the local decoding frame K, one super-resolution frame C corresponding to the type (I, P, B) of the local decoding frame K is generated. This super-resolution frame C is generated based on the super-resolution α calculated from the high-frequency power p, the coding sequence number n, and the QP offset value Q _off .

Here, when the high-frequency power p is large, the coding sequence number n is small, or the QP offset value Q _off is small, it is assumed that the blurring due to coding is small, and a super-resolution using a small value super-resolution α is used. The image frame C is generated. On the other hand, when the high frequency power p is small, the coding sequence number n is large, or the QP offset value Q _off is large, it is considered that the blurring due to coding is large, and the super-resolution using a large value super-resolution α is used. Frame C is generated.

In this way, it is possible to perform inter-screen prediction while compensating for blurring caused by coding without increasing the types of super-resolution images.

[Video decoding device]
Next, the moving image decoding device according to the embodiment of the present invention will be described. FIG. 7 is a block diagram showing a configuration example of a moving image decoding device according to an embodiment of the present invention. The moving image decoding device 2 includes an entropy decoding unit 20, an inverse quantization unit 21, an inverse orthogonal conversion unit 22, an addition unit 23, an in-screen prediction unit 24, an in-loop filter 25, an inter-screen prediction unit 26, and a switch 27. ing. Note that FIG. 7 shows only the components related to the present invention, and the components not related to the present invention are omitted.

The entropy decoding unit 20 inputs the coded signal output from the moving image coding device 1 shown in FIG. 1 and performs the reverse processing of the entropy coding unit 120 shown in FIG. 1, thereby performing the reverse processing of the coded signal. Is entropy-decoded to generate a quantized index string and parameters.

The entropy decoding unit 20 outputs the quantization index sequence to the inverse quantization unit 21. Further, the entropy decoding unit 20 outputs a parameter including the super-resolution α for each frame, the type of the predicted image selected for each predetermined area, and the calculated motion vector to the inter-screen prediction unit 26, and other parameters. Is output to the inverse quantization unit 21 and the like.

The inverse quantization unit 21 inputs the quantization index sequence from the entropy decoding unit 20 and performs the reverse processing of the quantization unit 112 shown in FIG. 1 to inversely quantize the quantization index sequence and convert coefficient sequence. To generate. Then, the inverse quantization unit 21 outputs the conversion coefficient sequence to the inverse orthogonal conversion unit 22.

The inverse orthogonal conversion unit 22 inputs the conversion coefficient sequence from the inverse quantization unit 21 and performs the reverse processing of the orthogonal conversion unit 111 shown in FIG. 1 to perform the inverse orthogonal conversion of the conversion coefficient sequence and the decoding residual. Generate an image. Then, the inverse orthogonal conversion unit 22 outputs the decoding residual image to the addition unit 23.

The addition unit 23 inputs the decoding residual image from the inverse orthogonal conversion unit 22, and also inputs the predicted image from the switch 27. Then, the adding unit 23 adds the decoding residual image to the predicted image, and outputs the added image (added image) to the in-screen prediction unit 24 and the in-loop filter 25.

The in-screen prediction unit 24 inputs the added image from the adding unit 23 and performs the same processing as the in-screen prediction unit 116 shown in FIG. 1, so that the in-screen prediction processing is performed on the added image for each predetermined area. Is performed, and the in-screen prediction result for each predetermined area is output to the switch 27. Further, the in-screen prediction unit 24 outputs the added image as a decoded image. As a result, the original moving image is restored.

The in-loop filter 25 inputs the added image from the adding unit 23 and performs the same processing as the in-loop filter 117 shown in FIG. 1 to filter the added image and generate a decoding frame K'. To do. Then, the in-loop filter 25 outputs the decoding frame K'to the inter-screen prediction unit 26.

The inter-screen prediction unit 26 inputs the decoding frame K'from the in-loop filter 25, and from the entropy decoding unit 20, parameters such as the super-resolution α and the type and motion vector of the prediction image selected for each predetermined area are input. input.

The inter-screen prediction unit 26 predicts one of the predicted image KY'of the decoding frame K'and the predicted image CY' of the super-resolution frame C'by performing a predetermined inter-screen prediction based on the parameters. Generate an image. Then, the inter-screen prediction unit 26 outputs this to the switch 27 as an inter-screen prediction result for each predetermined area. Further, the inter-screen prediction unit 26 outputs the decoding frame K'as a decoded image. As a result, the original moving image is restored. The details of the inter-screen prediction unit 26 will be described later.

The switch 27 inputs the in-screen prediction result for each predetermined area from the in-screen prediction unit 24, and also inputs the in-screen prediction result for each predetermined area from the inter-screen prediction unit 26, selects one of them, and predicts the image. Is output to the addition unit 23.

[Inter-screen prediction unit 26]
Next, the inter-screen prediction unit 26 shown in FIG. 7 will be described in detail. FIG. 8 is a block diagram showing a configuration example of the inter-screen prediction unit 26 provided in the moving image decoding device 2. The inter-screen prediction unit 26 includes a super-resolution processing means 30, a frame memory 31, an inter-screen prediction means 32, and a parameter processing means 33. Note that FIG. 8 shows only the constituent parts related to the present invention, and the unrelated constituent parts are omitted.

The inter-screen prediction unit 26 inputs the decoding frame K'(I, P, B) from the in-loop filter 25, and stores the decoding frame K'in the frame memory 31.

The parameter processing means 33 of the inter-screen prediction unit 26 inputs parameters such as the super-resolution α for each frame, the type of the predicted image selected for each predetermined area, and the calculated motion vector from the entropy decoding unit 20. The parameter processing means 33 outputs the super-resolution α to the super-resolution processing means 30, and outputs the type of the predicted image, the motion vector, and the like to the inter-screen prediction means 32. As described above, the type of the predicted image indicates any one of the decoding frame and the super-resolution frame.

The super-resolution processing means 30 inputs the super-resolution α from the parameter processing means 33, and for the decoding frame K'(I, P, B), similarly to the super-resolution processing means 13 shown in FIG. Super-resolution processing using the resolution α is performed to generate a super-resolution frame C'(I, P, B). The super-resolution processing means 30 stores the super-resolution frame C'in the frame memory 31.

The super-resolution frame C'(I, P, B) indicates an I-frame super-resolution frame C', a P-frame super-resolution frame C', and a B-frame super-resolution frame C'. As a result, in the frame memory 31, in addition to the decoding frame K'(I, B, P), the super-resolution frame C'of the I frame, the super-resolution frame C'of the P frame, and the super-resolution of the B frame Frame C'is stored.

The inter-screen prediction means 32 inputs the type of the prediction image, the motion vector, and the like from the parameter processing means 33, and when the type of the prediction image indicates a decoding frame, the decoding frame K'stored in the frame memory 31 is stored in the past. Read as the decoding frame K'of.

Further, when the type of the predicted image indicates a super-resolution frame, the inter-screen prediction means 32 reads out the super-resolution frame C'stored in the frame memory 31 as the past super-resolution frame C'.

The inter-screen prediction means 32 performs inter-screen prediction based on the past decoding frame K'or the past super-resolution frame C'and the motion vector, and obtains a predicted image for each predetermined area. The predetermined region is, for example, a slice region, a CTU region, and a CU region, as in FIG.

The inter-screen prediction means 32 outputs a prediction image for each predetermined area to the switch 27 as an inter-screen prediction result.

Further, a reading unit (not shown) reads the decoded frame K'from the frame memory 31 and outputs this as a decoded image.

As described above, according to the moving image decoding device 2 of the embodiment of the present invention, the super-resolution processing means 30 of the inter-screen prediction unit 26 is transmitted from the moving image coding device 1 based on the decoding frame K'. The super-resolution frame C'is generated using the super-resolution α included in the parameters. Then, the super-resolution processing means 30 stores the super-resolution frame C'in the frame memory 31.

The inter-screen prediction means 32 reads the decoding frame K'or the super-resolution frame C'as a past frame from the frame memory 31 according to the type of the prediction image included in the parameter. Then, the inter-screen prediction means 32 performs inter-screen prediction based on the read past frame and the motion vector included in the parameter, obtains a prediction image for each predetermined area, and outputs this as an inter-screen prediction result. ..

In this way, the moving image decoding device 2 follows the super-resolution α included in the parameters transmitted from the moving image coding device 1, and is equivalent to the super-resolution frame C generated by the moving image coding device 1. The resolution frame C'is generated. Further, the moving image decoding device 2 performs inter-screen prediction processing using the decoding frame K'or the super-resolution frame C'according to the type of the predicted image included in the parameters transmitted from the moving image coding device 1. .. Then, a predicted image of the same type as the predicted image selected by the moving image coding device 1 is generated, and a decoded image is generated.

In this way, it is possible to perform inter-screen prediction while compensating for blurring caused by coding without increasing the types of super-resolution images.

Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the technical idea.

For example, in the above embodiment, the super-resolution calculation means 12 of the inter-screen prediction unit 11 of the moving image coding device 1 is based on the high-frequency power p, the coding sequence number n, and the QP offset value Q _off. To calculate the super-resolution α in.

On the other hand, the super-resolution calculation means 12 may calculate the super-resolution α based on any one of the data of the high frequency power p, the coding sequence number n, and the QP offset value Q _off . Further, the super-resolution calculation means 12 may calculate the super-resolution α based on two data of the high frequency power p, the coding sequence number n, and the QP offset value Q _off .

For example, the super-resolution calculation means 12 calculates the super-resolution α based on the coding sequence number n and the QP offset value Q _off by the following formula.
[Number 2]
α _n = a (n / n _max ) (Q _off / Q _off-max ) ・ ・ ・ (2)
In this case, the moving image coding device 1 does not need to include the high frequency power calculation unit 10 in the configuration shown in FIG.

Further, in the above embodiment, the high frequency power calculation unit 10 inputs an I-frame decoded image from the addition unit 115, and calculates the high frequency power p based on the I-frame decoded image. This I-frame decoded image is obtained by performing the processing of the orthogonal conversion unit 111, the processing of the quantization unit 112, the processing of the inverse quantization unit 113, and the processing of the inverse orthogonal conversion unit 114 on the I frame of the input moving image. This is the obtained decoded image.

On the other hand, the high frequency power calculation unit 10 inputs the local decoding frame of the I frame from the in-loop filter 117, and based on the local decoding frame of the I frame filtered by the in-loop filter 117, the high frequency power p. May be calculated.

As the hardware configuration of the moving image coding device 1 and the moving image decoding device 2 according to the embodiment of the present invention, a normal computer can be used. The moving image coding device 1 and the moving image decoding device 2 are composed of a computer provided with a volatile storage medium such as a CPU and RAM, a non-volatile storage medium such as a ROM, and an interface.

Subtraction unit 110, orthogonal conversion unit 111, quantization unit 112, inverse quantization unit 113, inverse orthogonal conversion unit 114, addition unit 115, in-screen prediction unit 116, high frequency power calculation unit 10 provided in the moving image coding device 1. , The in-loop filter 117, the inter-screen prediction unit 11, the switch 119, and the entropy coding unit 120 are each realized by causing the CPU to execute a program describing these functions.

Further, the entropy decoding unit 20, the inverse quantization unit 21, the inverse orthogonal conversion unit 22, the addition unit 23, the in-screen prediction unit 24, the in-loop filter 25, the inter-screen prediction unit 26, and the switch 27 provided in the moving image decoding device 2 are provided. Is also realized by causing the CPU to execute a program that describes these functions.

These programs are stored in the storage medium, read by the CPU, and executed. In addition, these programs can be stored and distributed in storage media such as magnetic disks (floppy (registered trademark) disks, hard disks, etc.), optical disks (CD-ROM, DVD, etc.), semiconductor memories, etc., and can be distributed via a network. You can also send and receive.

1,100 Video coding device 2 Video decoding device 10 High-frequency power calculation unit 11,26,118 Inter-screen prediction unit 12 Super-resolution calculation means 13,30 Super-resolution processing means 14, 31, 122 Frame memory 15, 32 , 121 Inter-screen prediction means 16 Selection means 17, 33 Parameter processing means 20 Entropy decoding unit 21, 113 Inverse quantization unit 22, 114 Inverse orthogonal conversion unit 23, 115 Addition unit 24, 116 In-screen prediction unit 25, 117 In-loop Filter 27, 119 Switch 110 Subtraction unit 111 Orthogonal conversion unit 112 Quantization unit 120 Entropy coding unit p High-resolution power α Super-resolution n Coding sequence number n _max I-frame period (maximum value of coding sequence number n)
Q _off QP offset value Q _off-max QP offset value Q _off maximum value K Local decoding frame K'Decoding frame C, C'Super-resolution frame KY Local decoding frame K prediction image CY Super-resolution frame C prediction image KE Local decoding frame K prediction error CE Super-resolution frame C prediction error

Claims (6)

A residual image is generated by subtracting a predicted image from the input moving image, orthogonal conversion, quantization and entropy coding are performed on the residual image, a coded signal is generated and output, and the quantization is performed. The generated quantization index sequence is subjected to inverse quantization and inverse orthogonal conversion to generate a decoded residual image, and the predicted image is added to the decoded residual image to generate an added image, and the added image is generated. To generate a local decoding frame, perform inter-screen prediction using the local decoding frame, and generate the predicted image in the moving image coding device.
A super-resolution frame is generated based on the local decoding frame, and the prediction image of the local decoding frame and the prediction image of the super-resolution frame are generated for each predetermined region by the inter-screen prediction. An inter-screen prediction unit for selecting any one of the prediction images is provided.
The inter-screen prediction unit
A super-resolution calculation means for calculating the super-resolution used when generating the super-resolution frame, and
A super-resolution processing means that performs super-resolution processing using the super-resolution calculated by the super-resolution calculation means on the local decoding frame to generate the super-resolution frame, and
The screen-to-screen prediction is performed using the current local decoding frame as a reference picture and the past local decoding frame and the past super-resolution frame generated by the super-resolution processing means as a reference picture, and for each of the predetermined regions. In addition, the inter-screen prediction means for generating the prediction image of the local decoding frame and the prediction image of the super-resolution frame,
For each predetermined region, a selection means for selecting either one of the predicted image of the local decoding frame and the predicted image of the super-resolution frame is provided.
The super-resolution calculation means
The coding order number from the I frame included in the frame of the input moving image is used as the coding order number, or the QP offset value of the frame of the input moving image, or the coding. A moving image coding device characterized in that the super-resolution is calculated based on a sequence number and the QP offset value. In the moving image coding apparatus according to claim 1,
Further, a first high-frequency power calculation unit that calculates high-frequency power based on a decoding frame in which the orthogonal conversion, the quantization, the inverse quantization, and the inverse orthogonal conversion are performed on the I frame of the input moving image. With
The super-resolution calculation means
The high frequency power calculated by the first high frequency power calculation unit, or the high frequency power and the coding sequence number, or the high frequency power and the QP offset value, or the high frequency power, the coding sequence number and A moving image coding device characterized in that the super-resolution is calculated based on the QP offset value. In the moving image coding apparatus according to claim 1,
Further, a second high frequency power calculation unit that calculates high frequency power based on the local decoding frame obtained by applying the filter processing to the added image for the I frame of the input moving image is provided.
The super-resolution calculation means
The high frequency power calculated by the second high frequency power calculation unit, or the high frequency power and the coding sequence number, or the high frequency power and the QP offset value, or the high frequency power, the coding sequence number and A moving image coding device characterized in that the super-resolution is calculated based on the QP offset value. After inputting the encoded signal of the moving image, entropy decoding, inverse quantization and inverse orthogonal conversion are performed on the encoded signal to generate the decoded residual image, and the predicted image is added to the decoded residual image and added. By generating an image and filtering the added image to generate a decoded frame, the original moving image is restored, interscreen prediction is performed using the decoded frame, and the predicted image is generated. In the moving image decoding device
The coded signal includes one type of the predicted image selected from the predicted image of the locally decoded frame and the predicted image of the super-resolution frame, and a super-resolution parameter for generating the super-resolution frame. If so,
The decoding frame is subjected to super-resolution processing using the super-resolution included in the parameter to generate a super-resolution frame, and the prediction included in the parameter is included in each predetermined region by the inter-screen prediction. A moving image decoding device including an inter-screen prediction unit that generates the predicted image of any one of the decoding frame and the super-resolution frame according to the type of image. A program for causing a computer to function as the moving image coding device according to any one of claims 1 to 3. A program for causing a computer to function as the moving image decoding device according to claim 4.

Images