JP2008011009A - Video signal encoder, video signal decoder, video signal encoding program, and video signal decoding program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem of a conventional video coding system that a local difference of image quality is caused because blocks subjected to filter processing and blocks not subjected to filter processing are intermingled in L frames in the prediction processing. <P>SOLUTION: Since a filtering section 411D applies filtering to a signal resulting from mixing a video signal obtained by applying resolution interpolation to a base layer local decode signal with a signal subjected to the ME/MC processing in response to respective weight W values to produce H frames, and update processing is applied to a signal resulting from weighting the H frames by the weight W to produce the L frames, only components in the temporal direction are reflected on the L frames. Blocks subjected to the filter processing by the filtering section 411D exist in the L frames at all times. Thus, band division in the temporal direction can be realized while predicting between resolutions. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video signal encoding device, a video signal decoding device, a video signal encoding program, and a video signal decoding program, and in particular, a video signal encoding device and a video signal decoding having scalability in spatial resolution and frame rate, respectively. The present invention relates to an encoding device, a video signal encoding program, and a video signal decoding program encoding.

  Conventionally, many methods for realizing spatial resolution and SNR (Signal to Noise Ratio) scalability in video coding have been proposed, and these have been put to practical use in various fields. However, there is no breakthrough in frame rate scalability, and in order to realize a low-rate video signal, processing such as thinning out frames for convenience is required in addition to encoding processing. It was. On the other hand, in recent years, a method for realizing the scalability of the frame rate with high coding efficiency has been proposed (for example, see Patent Document 1).

  In this Patent Document 1, “motion compensation time direction filtering (hereinafter referred to as MCTF; Motion Compensation Temporal Filtering)” has been proposed. This is a method of performing band division using a lifting wavelet transform in the time direction after performing motion estimation (ME) and motion compensation (MC) like H.264 / AVC.

  FIG. 11 shows a 5/3 lifting configuration as an example of MCTF. An example of the MCTF procedure flow is shown in FIG. An example of the MCTF, which is an example of this conventional video signal encoding method, will be described with reference to FIGS. 11 and 12. First, the original video signal 201 shown in FIG. Is stored for processing (step S501 in FIG. 12). Here, as shown in FIG. 11, the video signal 201 is composed of eight consecutive frames of frame S (n−2) to frame S (n + 5).

  In the original video signal 201 of FIG. 11, when the S (n) frame is a processing target frame (even number), the S (n−1) frame and the S (n + 1) which are odd frames before and after the S (n) frame. ) ME / MC is performed so that the frame matches the phase of the S (n) frame (step S502 in FIG. 12).

Then, the ME / MC-processed S (n−1) frame, S (n + 1) frame, and S (n) frame are filtered (step S503 in FIG. 12). The process of step S202 and step S203 is called a prediction process (prediction process) here. A frame obtained as a result of the prediction process is an H ₁ (n) frame 202 (divided into high frequency components). This process is similarly performed for other consecutive even frames in the GOP (steps S204, S202, and S203).

Thus, as indicated by 202 in FIG. _11, in addition to _H 1 (n) _{frame, H 1 (n-2)} , H 1 (n + 2), H 1 (n + 4) frame is generated. In these H ₁ (n−2), H ₁ (n), H ₁ (n + 2), and H ₁ (n + 4) frames, the original video signal 201 of 1 GOP is divided into two divided frequency bands. The H component frame (hereinafter also referred to as the H frame) of the L component frame in the low frequency division frequency band and the H component frame in the high frequency division frequency band obtained in the above.

Next, when an S (n−1) frame is a processing target frame (odd number), a pre-predicted frame (even number) H ₁ (n) frame and H ₁ before and after the S (n−1) frame. H ₁ (n) obtained by performing inverse MC on the (n−2) frame so as to match the phase of the S (n−1) frame (step S505 in FIG. 12) and performing inverse MC as indicated by 203 in FIG. Filter processing is performed on the frame, the H ₁ (n−2) frame, and the S (n−1) frame (step S506 in FIG. 12). This filter processing is referred to as update processing (update processing) here.

The L ₁ (n−1) frame 203 obtained as a result of this update process is a low-frequency component in the time direction. In addition to the S (n−1) frame, the same processing is performed for other odd frames, ie, the S (n + 1) frame and the S (n + 3) frame (steps S507, S205, and S206 in FIG. 12). .

As a result, as indicated by 203 in FIG. 11, L ₁ (n + 1) frames, L ₁ (n + 3) frames, and L ₁ (n + 5) frames, which are low-frequency components in the time direction, are also generated. In these L ₁ (n−1), L ₁ (n + 1), L ₁ (n + 3), and L ₁ (n + 5) frames, the original video signal 201 of 1 GOP is divided into two divided frequency bands. The L component frame (hereinafter also referred to as L frame) of the L component frame in the low frequency division frequency band and the H component frame in the high frequency division frequency band obtained in the above.

  The above processing is repeated until it is determined in step S208 that there is one remaining L frame or the MCTF is terminated halfway (steps S502 to S507 in FIG. 12), and the frequency band is divided into low frequencies. By extracting the signal, a video signal having a low frame rate is generated.

That is, in the second MCTF, the pre-ready is performed using 2 frames out of 4 L frames 203 (L ₁ (n−1), L ₁ (n + 1), L ₁ (n + 3), L ₁ (n + 5)) as a unit. 11 generates two H frames H ₂ (n−1) and H ₂ (n + 3) indicated by 204 in FIG. 11, and two L frames L ₂ (n + 1) indicated by 205 in FIG. 11 by update processing. , L ₂ (n + 5). Furthermore, in the third MCTF, two L frames 205 (L ₂ (n + 1), L ₂ (n + 5)) are used, and an H frame H ₃ (n−1) indicated by 206 in FIG. ₃ (n + 5) is generated respectively.

  Here, when MCTF is repeatedly performed, as can be seen from FIG. 11, in the second MCTF, the prediction processing target frame changes from an even frame to an odd frame, the update processing target frame changes from an odd frame to an even frame, Note that the target frame changes each time the number of MCTFs is repeated.

In this way, finally, as indicated by 301 in FIG. 13, one L frame L ₃ (n + 5) and seven H frames H ₁ (n−2), H ₂ (n−1), H By obtaining a coefficient frame sequence for 1 GOP consisting of ₁ (n), H ₃ (n + 1), H ₁ (n + 2), H ₂ (n + 3), and H ₁ (n + 4), inter-frame coding of 1 GOP is completed. To do.

  When a high-rate video signal is reproduced on the decoding side, the reverse update process and the reverse prediction process are repeated in the reverse procedure of MCTF to synthesize the high-frequency signal with the low-frequency signal.

  Further, Patent Document 1 discloses a method of performing band division using, for example, a wavelet or DCT not only in the time direction but also in the spatial direction. As a result, on the decoding side, it is possible to select and receive an arbitrary resolution and frame rate in accordance with the performance of the decoder and display. In addition, the signal of each band in the method of Patent Document 1 is encoded independently without using the correlation of each band.

  On the other hand, the current high-performance encoding method is H.264. An encoding method that realizes encoding with higher efficiency and scalability functions of spatial resolution, frame rate, and SNR by combining the above MCTF with an extension of H.264 / AVC is known (for example, Non-Patent Document 1). Unlike the encoding apparatus described in Patent Document 1, the encoding apparatus described in Non-Patent Document 1 expresses high encoding efficiency by performing encoding using the correlation between space-time resolution. .

  FIG. 14 is a block diagram illustrating an example of the encoding unit 10 and the decoding unit 30 described in Non-Patent Document 1. In the figure, an original video signal is input to the encoding unit 10, and the bit stream generated by the encoding unit 10 is transmitted to the decoding unit 30 via the communication line or the medium 20. The decoding unit 30 extracts necessary information from the supplied bit stream and outputs a decoded video signal having a spatial resolution, a frame rate, and an SNR suitable for the performance of a display or the like.

  The encoding unit 10 includes a time-space decimation unit 11, a base layer encoding unit 12, a base layer reconstructing unit 13, an enhancement layer encoding unit 14, and a multiplexing unit 15.

  The space-time decimation unit 11 has a function of receiving an original video signal as an input and decimation (reduction) the input video signal to a desired spatial resolution. Here, as a spatial resolution decimation method, a method using a spatial filter, wavelet transform, or the like is used. In addition, the space-time decimation unit 11 has a function of performing time resolution decimation on a signal subjected to spatial resolution decimation to a desired spatial resolution to a desired frame rate and outputting it to the base layer encoding unit 12. Here, MCTF, simple frame thinning, or the like is used as a method of temporal resolution decimation.

  The base layer encoding unit 12 has a function of receiving the output signal of the space-time decimation unit 11 as an input, encoding (encoding) the input signal, generating a bit stream, and outputting the bit stream to the multiplexing unit 15. Here, H.264 or the like is used as the encoding method. In addition, the base layer encoding unit 12 has a function of outputting a signal that has been processed up to orthogonal transformation / quantization in H.264 or the like to the base layer restructuring unit 13.

  The base layer reconstructing unit 13 has a function of accepting, as an input, a signal obtained by performing processing up to quantization in a standard such as H.264 in the base layer encoding unit 12, and reconstructing the input signal to locally It has a function of generating a local decode signal which is a decoded image signal and outputting it to the enhancement layer encoding unit 14.

  The enhancement layer encoding unit 14 realizes time-space-SNR scalability by using a function of accepting an original video signal and a local decode signal input from the base layer reconstructing unit 13 as inputs, and these input signals. And a function of outputting the encoded signal (bit stream) obtained thereby to the multiplexing unit 15. Details will be described later.

  The multiplexing unit 15 receives each bit stream output from the base layer encoding unit 12 and the enhancement layer encoding unit 14 as an input and multiplexes to generate one bit stream. It has a function of outputting to a line or media 20.

  The decoding unit 30 includes an extract unit 31, a base layer decoding unit 32, and an enhancement layer decoding unit 33. The extract unit 31 cuts out and divides the necessary bit stream from the entire bit stream according to the function of accepting the bit stream as input and the performance of the decoding unit 30 or the display, and divides each of them into the base layer decoding unit 32 And a function of outputting to the enhancement layer decoding unit 33.

  The base layer decoding unit 32 receives a base layer bit stream extracted by the extract unit 31 as an input, decodes the input bit stream, and outputs a decoded video signal to the enhancement layer decoding unit 33 as necessary. A function of outputting to a display or the like. Here, an H.264 decoder or the like is used for decoding.

  The enhancement layer decoding unit 33 uses the function of receiving the bit stream obtained from the extract unit 31 and the decoded video signal output from the base layer decoding unit 32 as input, and the input signal to match the performance of the display or the like. And a function of decoding a video signal that realizes a spatial resolution, a frame rate, and an SNR. The decoded video signal is output to a display or the like. Details of the enhancement layer decoding unit 33 will be described later.

  Next, a procedure for scalable encoding of a video signal using the configuration example of the encoding unit shown in FIG. 14 will be described with reference to the flowchart of FIG. First, the encoding unit 10 performs a spatial resolution decimation on the original video signal in the space-time decimation unit 11 (step S601 in FIG. 15). Here, decimation is performed so that the spatial resolution of the signal after decimation becomes the lowest resolution in resolution scalability. Next, decimation in the time direction is performed (step S602 in FIG. 15). Here, similarly to the spatial resolution, temporal resolution decimation is performed so that the frame rate has the lowest frame rate scalability.

  Next, the signal obtained by decimating the spatial resolution and the frame rate is encoded using the base layer encoding unit 12 to generate a bit stream (step S603 in FIG. 15). Subsequently, the generated bit stream is sent to the multiplexing unit 15, and the orthogonal transform coefficient quantized data obtained in the encoding process is sent to the base layer reconstructing unit 13.

  Subsequently, the orthogonal transform coefficient quantized data obtained in the base layer encoding process is reconstructed in the base layer reconstructing unit 13 to generate a local decoded signal (step S604 in FIG. 15). Then, the generated local decode signal is sent to the enhancement layer encoding unit 14.

  Subsequently, using the original video signal and the generated base layer local decode signal, the enhancement layer encoding unit 14 encodes a signal for realizing time-space-SNR scalability (step S605 in FIG. 15). . Details will be described later. The bit stream generated by encoding is sent to the multiplexing unit 15.

  Then, the respective bit streams obtained from the base layer encoding unit 12 and the enhancement layer encoding unit 14 are multiplexed by the multiplexing unit 15 to generate one bit stream (step S606 in FIG. 15).

  Next, a procedure for obtaining a decoded video signal by decoding a scalable bit stream using the configuration example of the decoding unit 30 shown in FIG. 14 will be described with reference to the flowchart of FIG. When the bit stream is received from the communication line 20 or the media 20 of FIG. 14 using the extract unit 31, the extract unit 31 analyzes the received bit stream and needs to match the performance of the decoding unit 30 and the display. Extract code data. And it divides | segments into the data corresponding to each of the base layer decoding part 32 and the enhancement layer decoding part 33, and outputs them (step S701 of FIG. 16).

  Subsequently, the data corresponding to the base layer divided by the extract unit 31 is decoded by the base layer decoding unit 32 (step S702 in FIG. 16). The decoded base layer decoded video signal is output to the enhancement layer decoding unit 33 and, if necessary, output to a display or the like.

  Subsequently, the data corresponding to the enhancement layer divided by the extract unit 31 and the base layer decoded video signal decoded by the base layer decoding unit 32 are decoded by the enhancement layer decoding unit 33 (step S703 in FIG. 16). Then, the decoded decoded video signal is output to a display or the like. Details of the enhancement layer decoding procedure will be described later.

  FIG. 17 shows a block diagram of an example of the enhancement layer encoding unit 14 of FIG. As shown in FIG. 17, the enhancement layer encoding unit 14 includes an MCTF-ILP unit 141, an orthogonal transform / quantization unit 142, and an entropy encoding unit 143.

  An MCTF-ILP (MCTF-Inter Layer Prediction) unit 141 generates a signal that realizes time-space scalability using an input signal and a function of receiving an original video signal and a base layer local decode signal as inputs. , And a function of outputting the video information to the orthogonal transform / quantization unit 142 and the motion information to the entropy encoding unit 143. Details of the operation will be described later.

  The orthogonal transform / quantization unit 142 has a function of receiving video information as input, performing orthogonal transform and quantization, and outputting the quantized image to the entropy encoding unit 143. Typical examples of the orthogonal transform include DCT, Hadamard transform, and wavelet transform. The entropy encoding unit 143 has a function of accepting quantized video information and motion information as inputs, entropy encoding each of them, and outputting them to the outside of the enhancement layer encoding unit 14. Here, the entropy coding is devised so that SNR scalability can be realized.

  Next, a procedure for encoding an enhancement layer using the configuration example of the enhancement layer encoding unit 14 shown in FIG. 17 will be described with reference to the flowchart of FIG. Using the original video signal and the base layer local decode signal, the MCTF-ILP unit 141 generates a signal that realizes time-space scalability (step S801 in FIG. 18). Details of the procedure will be described later. The video information obtained as a result is sent to the orthogonal transform / quantization unit 142. The motion information is sent to the entropy encoding unit 143.

  The video information signal realizing the time-space scalability generated in the MCTF-ILP unit 141 is quantized after the orthogonal transformation by the orthogonal transformation / quantization unit 142 (step S802 in FIG. 18). The orthogonally transformed and quantized signal is sent to the entropy coding unit 143.

  The motion information obtained by the MCTF-ILP unit 141 and the video information output from the orthogonal transform / quantization unit 142 are entropy encoded by the entropy encoding unit 143 to generate a bit stream (step S803 in FIG. 18). ).

  Next, the enhancement layer decoding unit 33 in FIG. 14 will be described. FIG. 19 is a block diagram illustrating an example of the enhancement layer decoding unit 33. In FIG. 19, the enhancement layer decoding unit 33 includes an entropy decoding unit 331, an inverse orthogonal transform / inverse quantization unit 332, and an inverse MCTF-ILP unit 333.

  The entropy decoding unit 331 has a function of receiving and decoding a bit stream as input and outputting video information to an inverse quantization / inverse orthogonal transform unit 332 and motion information to an inverse MCTF-ILP unit 333.

  The inverse quantization / inverse orthogonal transform unit 332 has a function of receiving decoded video information as an input, performing inverse quantization, performing inverse orthogonal transform, and outputting the result to the inverse MCTF-ILP unit 333. The inverse MCTF-ILP unit 333 accepts an inverse orthogonal transformed video information signal, a base layer local decode signal, and decoded motion information as inputs, and uses the input signals to perform band synthesis and spatial resolution in the time direction. The video signal is decoded by performing reverse prediction, and output to the outside of the enhancement layer decoding unit 33. Details of the operation will be described later.

  Next, a procedure for decoding the enhancement layer using the configuration example of the enhancement layer decoding unit 33 shown in FIG. 19 will be described with reference to the flowchart of FIG. The enhancement layer decoding unit 33 decodes the input bit stream in the entropy decoding unit 331 (step S1001 in FIG. 20), and sends the video information obtained as a result of the decoding to the inverse quantization / inverse orthogonal transform unit 332 for decoding. The motion information thus transmitted is sent to the inverse MCTF-ILP unit 333.

  The decoded video information is inversely quantized using the inverse quantization / inverse orthogonal transform unit 332 and then inversely orthogonal transformed (step S1002 in FIG. 20). The video information obtained as a result is sent to the inverse MCTF-ILP unit 333. The inverse MCTF-ILP unit 333 performs inverse prediction between temporal band synthesis and spatial resolution using the video information obtained by inverse orthogonal transformation, the local decode signal of the base layer, and the decoded motion information. Is decoded (step S1003 in FIG. 20). Details of the procedure will be described later.

  Next, an operation example and a processing procedure example of the MCTF-ILP unit 141 in FIG. 17 will be described with reference to FIGS. 21 and 22. First, the original video signal 1104 shown in FIG. 21 is accumulated for processing, for example, in units of 8 frames (GOP) (step S1201 in FIG. 22). Here, as shown in FIG. 21, the video signal 1104 is composed of eight consecutive frames of frame S (n−2) to frame S (n + 5). Further, the frame rate of the base layer is ½ of the original video signal. As can be seen from FIG. 21, the MCTF-ILP unit 141 is an extension of ILP (Inter Layer Prediction, inter-resolution prediction) to MCTF.

  First, the procedure of the prediction process will be described. The base layer local decode signal 1103 shown in FIG. 21 is interpolated (enlarged) so as to have the same spatial resolution as the original video signal 1104 (1102 in FIG. 21, step S1202 in FIG. 22). When the S (n) frame in FIG. 21 is a processing target frame (even number), the S (n−1) frame and the S (n + 1) frame before and after the frame are matched with the phase of the S (n) frame. Each performs ME / MC (step S1203 in FIG. 22).

  At this time, unlike the conventional MCTF, a signal obtained by interpolating S ′ (n) in the local decode signal of the base layer indicated by 1103 in FIG. 21 can be selectively used as the third reference frame. That is, it is possible to adaptively adopt the one with the better coding efficiency of the prediction between the band division accompanied with the temporal motion compensation for each block and the spatial resolution. Here, when prediction between resolutions is used, the vector of the block is set to 0 so that the decoding side can determine.

Filtering is performed on the S (n−1) frame, the S (n + 1) frame, and the S (n) frame that have been MCed by selectively using the base layer information as described above, and H ₁ (n) shown in FIG. A frame is generated (step S1204 in FIG. 22). The processes in steps S1203 and S1204 are also performed for other even frames in the GOP (step S1205 in FIG. 22).

Next, the update processing procedure will be described. When the S (n−1) frame in FIG. 21 is a processing target frame (odd number), the pre-predicted H ₁ (n) frame and the H ₁ (n−2) frame before and after the S (n−1) frame are S (n−1) frames. The inverse MC is performed to match the phase (step S1206 in FIG. 22), and the H ₁ (n) frame, the H ₁ (n−2) frame, and the S (n−1) frame subjected to the inverse MC are filtered. (Step S1208 in FIG. 22).

  Here, since the block for which the inter-resolution prediction is performed using the base layer information in the MC in the prediction process is not the H component in the time direction, the block obtained by performing the inverse MC in step S1206 is the block in which the base layer is adopted. If it is determined in step S1207, the filtering process in step S1208 (processing for generating the L component in the time direction) is skipped. Subsequently, it is determined whether or not the update for the even frame in the GOP has been completed (step S1209 in FIG. 22). If not, the update process in steps S1206 to S1208 is performed for other odd frames. .

  The above processing is repeated in step S1210 until the number of remaining L frames is one, or until the MCTF is terminated halfway (steps S1202 to S1209 in FIG. 22), and the band is divided in the time direction to obtain a low frequency band. By extracting the signal, a video signal having a low frame rate is generated. Band division in the time direction can be repeated as described above. On the other hand, prediction between resolutions can be used when the frame rate of the base layer matches.

  Next, an operation example and a processing procedure example of the inverse MCTF-ILP unit 333 in FIG. 19 will be described with reference to FIGS. 23 and 24. In the example of FIG. 23, the GOP size of the original video signal is “8”, the MCTF-ILP on the encoding side is first-order decomposition as indicated by 1304 in FIG. That is, a signal encoded under 1/2 of the above condition is restored. First, the original video signal 1304 shown in FIG. 23 is stored for processing, for example, in units of 8 frames (GOP) (step S1401 in FIG. 24).

Next, the base layer decoded signal 1303 shown in FIG. 23 is interpolated in advance as shown by 1302 so as to have the same spatial resolution as the enhancement layer signal 1304 (step S1402 in FIG. 24). Subsequently, reverse update processing is performed. First, when restoring the S (n-1) frame of FIG. 23, the inverse MC is performed so that the H ₁ (n-2) frame and the H ₁ (n) frame are matched with the phase of the S (n-1) frame. (Step S1403 in FIG. 24), the H ₁ (n-2) frame, the H ₁ (n) frame, and the L ₁ (n-1) frame subjected to inverse MC are filtered (Step S1405 in FIG. 24).

  Here, for the inverse MC, the motion information decoded by the entropy decoding unit 331 (step S1001 in FIG. 20) is used, and the filtering process is not performed on the block whose vector is “0” (step S1404 in FIG. 24). . Similar processing is performed for other L frames of the same temporal level in the GOP (step S1406 in FIG. 24).

Next, the procedure of reverse prediction processing will be described. When the S (n) frame in FIG. 23 is restored, the reversely updated S (n−1) frame and S (n + 1) frame before and after the S (n) frame to be processed are replaced with the H ₁ (n) frame. MC is performed to match the phase (step S1407 in FIG. 24). Here, a signal obtained by interpolating the base layer decoded signal S ′ (n) is applied to a block whose vector is “0”.

Then, the S (n-1) frame, the S (n + 1) frame, and the H ₁ (n) frame that have been MCed are filtered to obtain an S (n) frame (step S1408 in FIG. 24). Similar processing is performed for other H frames of the same temporal level in the GOP (step S1409 in FIG. 24).

  When MCTF-ILP of a plurality of hierarchies is performed on the encoding side, the reverse update process and the reverse prediction process are further repeated to obtain a decoded video signal (step S1410 in FIG. 24).

JP-T-2004-524730 Diego Santa-Cruz, Davide Maestroni, Francesco Ziliani, Julien Reichel and Stefano Tubaro, “Improved Scalable MCTF Video Codec Using A H.264 / AVC Base Layer”, Picture Coding Symposium 2004, Dec. 2004.

The video signal coding method described in Non-Patent Document 1 is superior in that time-space scalability is realized with high coding efficiency by selectively using MCTF which is band division in the time direction and prediction ILP between resolutions. It is an encoding method. However, this encoding method has a problem that flicker occurs when the decoded video is subjectively evaluated. This is pointed out as “a flash of artifacts can appear” in Non-Patent Document 1, and is a problem requiring improvement.
In the video signal encoding method described in Non-Patent Document 1, a block that uses a base layer local decode signal in a prediction process (prediction process) is not subjected to a filter process in an update process (update process). For this reason, even though the L frame is subjected to filtering processing and the time frequency is high (if the filtering processing is not performed, there may be a problem in the quality of the video signal) Blocks that have been drawn are mixed, resulting in a local difference in image quality.

  In a moving part, this problem is likely to occur because a base layer local decoded signal having a stronger correlation in principle is frequently used. In particular, in MCTF, since time band division is repeated, deterioration caused by thinning out frames without being subjected to filter processing on L frames propagates to the next MC and time band division processing. It becomes difficult to maintain the accuracy of subsequent processing. Therefore, the adaptive processing in MCTF needs to be processed based on the signal processing theory (the signal is not broken) rather than the simple space-time adaptive processing in the conventional video coding.

  The present invention has been made in view of the above points, and is a video signal code that solves the conflicting phenomenon between encoding and signal processing that occurs in Non-Patent Document 1, and can improve both encoding efficiency and decoded video quality. It is an object to provide an encoding device, a video signal decoding device, a video signal encoding program, and a video signal decoding program.

  In order to achieve the above object, a video signal encoding apparatus according to a first aspect of the present invention performs a compensation time direction filtering process on an input video signal, and performs subband division into two divided frequency bands. A first code obtained by generating a frame sequence signal composed of an L frame in the frequency band and an H frame in the high frequency division frequency band, and performing orthogonal transform / quantization and entropy encoding on the frame sequence signal A video signal encoding apparatus that outputs a video signal together with a second encoded signal obtained by encoding a video signal having a resolution lower than that of the input video signal, the input video signal being space-time reduced by a predetermined ratio, Reduction means (11) for generating a low-resolution video signal having a spatial-temporal resolution different from the input video signal, and encoding means (12) for encoding the low-resolution video signal to generate a second encoded signal , Second mark A local decoding means (13) for locally decoding the encoded signal to generate a local decoded signal, and an expanding means (4117) for expanding the spatial resolution of the local decoded signal at a ratio opposite to that of the reducing means to generate a high spatial resolution video signal (4117). ) And a motion indicating the relative positional relationship between each block corresponding to the reference frame and the other frame by estimating the motion of the other frame using one of two different frames existing in the time direction of the input video signal as a reference frame. A motion estimation / compensation unit (4118) for acquiring information and compensating for the motion of the other frame based on the motion information; and an input video signal motion-compensated by the motion estimation / compensation unit is weighted with a first weight The first weighting means (4119, 411A) that performs the processing, and the high spatial resolution video signal generated by the enlargement means from the local decoded signal is weighted by the second weight calculated using the first weight. The second weighting means (4119, 411B) and the first processing means (411C) for generating an H frame by combining and filtering the signals weighted by the first and second weighting means. , 411D), a third weighting means (4119, 4112) for weighting the H frame generated by the first processing means with the first weight, and the H frame weighted by the third weighting means The second processing means (4113) for applying the motion information acquired by the motion estimation / compensation means in the reverse direction to perform motion compensation and then filtering to generate an L frame, and the first and second processing means Frame sequence signal generation means (4115) for generating and outputting a frame sequence signal by synthesizing the generated H frame and L frame, respectively.

  In order to achieve the above object, the video signal encoding program according to the second invention performs a compensation time direction filtering process on the input video signal to perform subband division into two divided frequency bands. A first frame obtained by generating a frame sequence signal composed of an L frame in the side division frequency band and an H frame in the high side division frequency band, and performing orthogonal transform / quantization and entropy coding on the frame sequence signal A video signal encoding program for causing a computer to execute an operation of outputting the encoded signal together with a second encoded signal obtained by encoding a video signal having a resolution lower than that of the input video signal. It is made to function as each means of the structure of the video signal encoding device of this invention.

  In order to achieve the above object, the video signal decoding device of the third invention is generated by the video signal encoding program output from the video signal encoding device of the first invention or by the video signal encoding program of the second invention. A video signal decoding apparatus that receives and decodes a signal composed of first and second encoded signals, motion information, and weight information indicating first to third weights. Receiving means (31) for receiving and separating the two encoded signals, and entropy decoding the first encoded signal from the receiving means, and then performing inverse quantization and inverse orthogonal transform to perform H frame and L frame A first decoding means (511, 512) for decoding the image, a second decoding means (32) for decoding the second encoded signal from the receiving means to generate a low-resolution video signal, and a second decoding means The spatial resolution of the decoded low-resolution video signal is expanded to achieve high spatial resolution. A first weighting means (5133) for weighting the H frame from the first decoding means with a first weight obtained from the received weight information; a first weighting means (5133) for generating a video signal; First processing means for generating a first decoded signal by performing motion compensation on the H frame or decoded video signal weighted by the weighting means by applying the motion information received by the receiving means in the reverse direction and performing motion compensation. (5134) and second weighting means for weighting the first decoded signal with the first weight obtained from the weight information received by the receiving means after applying motion compensation to the motion information received by the receiving means (5136, 5138), a third weighting means (5139) for weighting the high spatial resolution video signal from the enlarging means with the second weight obtained from the weight information received by the receiving means, Second processing means (513A) that combines the weighted signals by the third weighting means and performs filtering on the combined signal and the H frame received and separated by the receiving means to generate a second decoded signal. , 513B) and decoded video signal generation means (513C) for generating a decoded video signal by synthesizing the first decoded signal and the second decoded signal.

  In order to achieve the above object, the video signal decoding program of the fourth invention is output by the video signal encoding device of the first invention or generated by the video signal encoding program of the second invention. A video signal decoding program for receiving a signal composed of first and second encoded signals, motion information, and weight information indicating first to third weights, and decoding the signal by a computer, Is made to function as each means of the structure of the video signal decoding device of the third invention.

  An object of the present invention is to selectively use base layer local decoding in the prediction process (prediction process) of Non-Patent Document 1, and to determine whether or not to perform a filter process in the update process (update process). The problem is to solve the above-mentioned problem caused by switching. In order to achieve this object, in the first or second invention, the input video signal motion-compensated by the motion estimation / compensation means is weighted by the first weight, and the spatial resolution is expanded from the local decoded signal. Is weighted with the second weight calculated from the first weight, and the weighted signal is synthesized and filtered to generate the H frame as the first processing means. And the H frame generated by the first processing means is weighted by the same first weight as the motion compensated input video signal, and the H processing is inversely compensated for the H frame by the second processing means. Since the L frame is generated after filtering, the first processing means always uses the motion compensated input video signal and the local decoded signal. Produced by between resolution enlarged on the basis of a high spatial resolution image signal are weighted synthesized signal can be generated H frame, it can produce L-frame which reflects only a component in the time direction by the second processing means. That is, band division in the time direction is realized while predicting between resolutions.

  That is, in the present invention, there is a block in which the L frame obtained by the second processing means is always subjected to filter processing. Therefore, in the conventional apparatus described in Non-Patent Document 1, the base layer local decoded signal is used in the prediction process. The adopted block is not subjected to the filtering process in the update process. Therefore, if the filtering process is not applied to the block subjected to the filtering process on the L frame, there may be a problem in the quality of the video signal, but the filtering process is not performed. There has been a problem that a block in which frames are thinned out is mixed and a difference in local image quality occurs, but this problem can be solved.

  Furthermore, in MCTF, time band division is repeatedly performed, so that degradation caused by thinning out frames without performing filter processing on L frames propagates to the next MC and time band division processing. The problem that it is difficult to maintain the accuracy of the subsequent processing can be solved, and the coding efficiency can be improved because the degree of freedom of space-time component prediction and band division increases.

  According to the present invention, an H frame is generated on the basis of a signal obtained by weighting and combining an input video signal and a local decoded signal that are always motion-compensated by the first processing means, and a time direction component is produced by the second processing means. By generating an L frame that reflects only the L frame obtained by the second processing means, there is always a block that has been subjected to the filter processing. Therefore, the L frame in the conventional apparatus described in Non-Patent Document 1 There is a mixture of blocks that have been subjected to filter processing and blocks in which frames are thinned out without being subjected to filter processing, although there may be a problem in the quality of the video signal if filter processing is not performed. This can solve the problem of differences in image quality, which improves flicker in the decoded video and improves the quality of the decoded video. It can be.

  In addition, according to the present invention, since MCTF repeatedly performs time band division, degradation caused by thinning out frames without performing filter processing on L frames may cause subsequent MC and time band division processing. It is possible to solve the problem that it becomes difficult to maintain the accuracy of the subsequent processing because of propagation, and the coding efficiency can be improved because the degree of freedom of space-time component prediction and band division increases. .

  Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of a video signal encoding apparatus and video signal decoding apparatus according to the present invention. In the figure, the same components as those in FIG. 14 are denoted by the same reference numerals, and the description thereof is omitted. In FIG. 1, an encoding unit 40, which is an embodiment of a video signal encoding device according to the present invention, includes a time-space decimation unit 11, a base layer encoding unit 12, a base layer reconstructing unit 13, and an enhancement layer encoding unit. 41 and the multiplexing unit 15. The decoding unit 50 as an embodiment of the video signal decoding apparatus according to the present invention includes an extract unit 31, a base layer decoding unit 32, and an enhancement layer decoding unit 51.

  Similarly to FIG. 14, in this embodiment, the original video signal is input to the encoding unit 40, and the bit stream generated by the encoding unit 40 is transmitted to the decoding unit 50 via the communication line or the medium 20. Is done. The decoding unit 50 extracts necessary information from the supplied bit stream and outputs a decoded video signal having a spatial resolution, a frame rate, and an SNR suitable for the performance of a display or the like.

  Here, the present invention aims to improve the adverse effects caused by the selective use of MCTF and ILP in Non-Patent Document 1, and in this embodiment, an enhancement layer including the MCTF-ILP part of Non-Patent Document 1 is used. The encoding unit 14 is changed to an enhancement layer encoding unit 41 having a configuration described later, and the enhancement layer decoding unit 33 including the inverse MCTF-ILP unit is changed to an enhancement layer decoding unit 51 having a configuration described later. Other components are the same as the conventional configuration of FIG. The embodiment of FIG. 1 realizes spatial resolution scalability with two layers as in the example of FIG. 14, but it may be multi-layered.

  FIG. 2 shows a block diagram of an embodiment of the enhancement layer encoding unit 41 in FIG. In FIG. 2, the enhancement layer encoding unit 41 includes an MCTF-ILP unit 411, an orthogonal transform / quantization unit 412, and an entropy encoding unit 413. The MCTF-ILP unit 411 receives the original video signal and the base layer local decode signal (local decoded signal) from the base layer reconstruct unit 13 as inputs, and realizes space-time scalability using these signals. A function of generating a signal to be generated. Details of the operation will be described later. The MCTF-ILP unit 411 has a function of outputting video information to the orthogonal transform / quantization unit 412 and outputting weight information and motion information to the entropy encoding unit 413, respectively.

  The orthogonal transform / quantization unit 412 has a function of receiving the video information output from the MCTF-ILP unit 411 as an input and quantizing the signal after orthogonal transform. Typical examples of the orthogonal transform include DCT, Hadamard transform, and wavelet transform. The orthogonal transform / quantization unit 412 has a function of outputting a signal quantized after the orthogonal transform to the entropy coding unit 413.

  The entropy encoding unit 413 obtains the video information output from the orthogonal transform / quantization unit 412 and the weight information and motion information output from the MCTF-ILP unit 411, and entropy the acquired input information. A function of generating a bit stream by encoding and outputting the bit stream to the outside of the enhancement layer encoder unit 41; Here, the entropy coding can adopt a configuration and procedure that can realize SNR scalability as in the conventional method.

  A procedure for encoding an enhancement layer using the enhancement layer encoding unit 41 having the configuration shown in FIG. 2 will be described with reference to the flowchart of FIG. Using the original video signal and the base layer local decoded signal, the MCTF-ILP unit 411 generates a signal that realizes time-space scalability (step S101 in FIG. 3). Details of the procedure will be described later. The video information obtained as a result is sent to the orthogonal transform / quantization unit 412. The weight information and motion information are sent to the entropy coding unit 413.

  The video information signal realizing the time-space scalability generated in the MCTF-ILP unit 411 is orthogonally transformed by the orthogonal transformation / quantization unit 412, and the obtained orthogonal transformation coefficient is quantized (step S102 in FIG. 3). ). The orthogonally transformed and quantized signal is sent to the entropy coding unit 413. The weight information and motion information obtained by the MCTF-ILP unit 411 and the video information output from the orthogonal transform / quantization unit 412 are entropy-coded by the entropy coding unit 413 to generate a bit stream (see FIG. 3). Step S103).

  Next, the configuration and operation of the MCTF-ILP unit 411 shown in FIG. 2 in the enhancement layer encoding unit 41 will be described in detail. FIG. 4 shows a configuration diagram of an embodiment of the MCTF-ILP unit 411. In the figure, an MCTF-ILP unit 411 includes an input switching unit 4110, a prediction processing unit 4111, an update weighting unit 4112, an update processing unit 4113, a frame management unit 4114, a frame rearranging unit 4115, and a motion information updating unit 4116. Composed.

  Prediction processing unit 4111 receives a signal input from input switching unit 4110, a base layer local decoded signal output from base layer reconstructing unit 13, and frame management information output from frame management unit 4114. It has the function to acquire as. The internal configuration and operation of the prediction processing unit 4111 will be described later. Also, the prediction processing unit 4111 has a function of outputting a signal of the prediction processing result (prediction processing result) to the frame rearranging unit 4115 and the update weighting unit 4112, the weight information for the update weighting unit 4112, and the motion information update. A function to output to the outside of the unit 4116 and the MCTF-ILP unit 411, and a function to output motion information (eg, motion vector, macroblock type, etc.) to the update processing unit 4113 and the motion information update unit 4116.

  The update processing unit 4113 receives a signal input from the input switching unit 4110, a signal output from the update weighting unit 4112, frame management information output from the frame management unit 4114, and output from the prediction processing unit 4111. A function of acquiring motion information as input, a function of performing inverse MC and filtering using the input signal and information, and a function of outputting the processed signal to the frame rearranging unit 4115.

  The frame management unit 4114 has a function of managing a frame in the MCTF-ILP unit 411, and a function of outputting the management information to the input switch 4110, the prediction processing unit 4111, the update processing unit 4113, and the frame rearrangement unit 4115. Have

  The frame rearrangement unit 4115 receives the signal after the prediction process (after the prediction process) output from the prediction processing unit 4111, the signal after the update process (after the update process) output from the update processing unit 4113, and the frame management. A function for acquiring the frame management information output from the unit 4114, a function for appropriately rearranging the frames for which the MCTF-ILP processing has been completed based on each acquired information, and the processing result of the rearrangement as MCTF- A function of outputting to the outside of the ILP unit 411.

  The weighting unit for update 4112 acquires the signal of the prediction process result (H frame) and weight information output from the prediction processing unit 4111 and the acquired signal of the prediction process result (H frame). It has a function of performing weighting processing based on the weight information and a function of outputting a signal of the weighting processing result to the update processing unit 4113.

  The motion information update unit 4116 has a function of acquiring the motion information and weight information output from the prediction processing unit 4111, a function of updating the motion information according to the content of the acquired weight information, and the update result as MCTF- A function of outputting to the outside of the ILP unit 411.

  The input switching unit 4110 has a function of acquiring a video signal, a signal (L frame) of an update processing result output from the update processing unit 4113, and frame management information output from the frame management unit 4114, and the acquired frame It has a function of selecting either a video signal acquired according to management information or a signal (L frame) of an update processing result and outputting it to a prediction processing unit 4111 and an update processing unit 4113.

  Next, a detailed configuration example of the prediction processing unit 4111 will be described. 4, the prediction processing unit 4111 includes a resolution interpolation unit 4117, an ME / MC unit 4118, a weight instruction unit 4119, a prediction weighting unit 411A, an ILP weighting unit 411B, a signal synthesis unit 411C, A filtering unit 411D is included.

  The resolution interpolation unit 4117 interpolates (enlarges; the base layer local decode signal and the frame management information output from the frame management unit 4114) and the base layer local decode signal to the same spatial resolution as the video signal. Interpolation). Here, any interpolation method may be used, but it is desirable to use a filter corresponding to the filter used when decimating (reducing) the spatial resolution of the base layer. Further, the resolution interpolation unit 4117 has a function of outputting the signal resulting from the interpolation to the ILP weighting unit 411B.

  The ME / MC unit 4118 has a function of acquiring a signal input from the input switcher 4110 and frame management information output from the frame management unit 4114, and, for example, a conventional MPEG (Moving Picture Experts Group) series, H.264, A function for performing motion estimation (ME) and motion compensation (MC) such as MCTF, and a signal obtained as a result of ME / MC are output to the weighting unit 411A for prediction, and motion information is output to the outside of the prediction unit 4111. It has the function to do.

  The weight instruction unit 4119 has a function of acquiring a prediction processing result signal (H frame) output from the filtering unit 411D and frame management information output from the frame management unit 4114, and a weight W based on the acquired information. It has a function of calculating, supplying it to the weighting unit for prediction 411A, the weighting unit for ILP 411B, the weighting unit for update 4112, and the motion information update unit 4116 and outputting it to the outside of the MCTF-ILP unit 411. Note that the input to the weight instruction unit 4119 may accept an instruction from the outside.

  The weighting unit for prediction 411A obtains a signal obtained by ME / MC from the ME / MC unit 4118 and a weight W output from the weight instruction unit 4119, and weights the weight W on the signal subjected to ME / MC, and performs signal synthesis. Part 411C. The ILP weighting unit 411B obtains the signal output from the resolution interpolation unit 4117 and the weight W from the weight instruction unit 4119, the function of calculating (1-W), and the calculation result as the resolution interpolation. The weighted signal is weighted and output to the signal synthesizer 411C.

  The signal synthesis unit 411C has a function of acquiring signals output from the weighting unit for prediction 411A and the weighting unit for ILP 411B, frame management information output from the frame management unit 4114, a weighting unit for prediction 411A, For example, the signals acquired from the ILP weighting unit 411B are added and combined, for example, to create a final reference frame for prediction processing, and a function to output the resulting signal to the filtering unit 411D .

  The filtering unit 411D has a function of acquiring a signal output from the signal combining unit 411C, a signal output from the input switching unit 4110, and frame information output from the frame management unit 4114, and a reference frame obtained by the signal combining unit 411C. And a function of performing filter processing on the input signal to generate an H frame and outputting it to the outside of the prediction processing unit 4111 and to the weight instruction unit 4119.

  That is, the MCTF-ILP unit 411 converts the signal obtained by performing ME / MC in the ME / MC unit 4118 and the base layer local decoded signal to the signal obtained by interpolating in the spatial resolution / interpolation unit 4117 with respect to the original video signal. Weighting is performed by the weighting unit for prediction 411A and the weighting unit for ILP 411B. Then, each signal is synthesized by the signal synthesis unit 411C to generate a reference frame. The reference frame and the output from the input switch 4110 are filtered by the filtering unit 411D to obtain a signal (H frame) as a result of the prediction process. In the filter processing in the update processing unit 4113, the H frame weighted in the same way as the prediction weighting unit 411A in the prediction unit 4111 is used without skipping, and only the component in the time direction of the H frame is reflected in the L frame. In other words, time-division band division is realized while prediction between spatial resolutions.

  Next, a procedure for performing MCTF-ILP processing by the MCTF-ILP unit 411 having the configuration shown in FIG. 4 will be described with reference to the flowchart of FIG. In order to perform the series of processing shown in FIG. 5 in units of 1 GOP as in Non-Patent Document 1, video frames for 1 GOP are accumulated (step S201 in FIG. 5). Note that the series of processing shown in FIG. 5 may be performed in units of a plurality of GOPs, and processing across a plurality of GOPs may be performed. Next, the base layer local decoded signal from the base layer reconstructing unit 13 in FIG. 1 is converted into a spatial resolution interpolator at a resolution opposite to the decimation ratio of the space-time decimation unit 11 in the resolution interpolation unit 4117 in FIG. It is porated and converted into a video signal having the same high spatial resolution as the input video signal of the MCTF-ILP unit 411 (step S202 in FIG. 5).

  Next, the prediction processing unit 4111 performs a prediction process. In the prediction process, initially, a high spatial resolution video signal input from the outside is input to the prediction processing unit 4111 through the input switch 4110, and the first band division process by MCTF is performed. In the prediction processing unit 4111, the ME / MC unit 4118 uses the ME / MC unit 4118 before and after the processing target frame (even number) and the frames before and after the processing target frame (even number) that are output signals from the input switching unit 4110 (odd number). ) Is performed (step S203 in FIG. 5).

  That is, the ME / MC unit 4118 first estimates the motion of the previous or subsequent frame using the processing target frame as a reference frame, and shows a motion vector indicating the relative positional relationship between corresponding blocks of the reference frame and the previous or subsequent frame. Motion compensation (MC) processing for performing motion estimation (ME) processing for acquiring information (hereinafter also referred to as motion information), and subsequently compensating for motion of a frame before or after the reference frame based on the motion vector information I do. This ME / MC processing is performed for each of the frames before and after the reference frame.

  In addition, after the H frame of the high frequency side divided frequency band is first output from the two divided frequency bands obtained by subband division from the filtering unit 411D described later, the prediction processing unit 4111 subsequently performs the second time in the same manner. Subsequent band division processing by MCTF is performed, but since the second and subsequent band division processing by MCTF is performed on the L frame as described with reference to FIG. 11 and the like, the low frequency side output from the update processing unit 4113 The L frame of the divided frequency band is selected by the input switch 4110 and input to the prediction processing unit 4111.

  The signal subjected to the ME / MC processing in the ME / MC unit 4118 is weighted with a value of W (0 <W <1) in the weighting unit for prediction 411A in accordance with an instruction from the weight instruction unit 4119 (step of FIG. 5). S204). Here, the weight instruction unit 4119 feeds back the signal (H frame) after the prediction process output from the filtering unit 411D so as to reduce the output signal of the signal synthesis unit 411C so as to obtain the optimum W. However, there are various other rules for calculating the weight W in the weight instruction unit 4119. For example, an instruction may be given from the outside.

  On the other hand, a high spatial resolution video signal obtained by performing resolution interpolation on the base layer local decode signal in the resolution interpolation unit 4117 is transmitted to the ILP weighting unit 411B according to the instruction from the weight instruction unit 4119 ( 1-W) is weighted (step S205 in FIG. 5).

  Subsequently, signals weighted by the weighting unit for prediction 411A and the weighting unit for ILP 411B are combined by the signal combining unit 411C to generate a final reference frame for the prediction process (step S206 in FIG. 5). .

  Subsequently, the filtering unit 411D performs a filtering process on the signal selected by the input switching unit 4110 and the reference frame from the signal synthesis unit 411C (step S207 in FIG. 5). As a result, an H frame can be obtained from the filtering unit 411D as a signal of a prediction processing result in the MCTF-ILP unit 411.

  Subsequently, it is determined whether or not the prediction processes in steps S203 to S207 have been performed on all the even frames in the GOP (step S208 in FIG. 5), and the prediction processes in steps S203 to S207 are performed until all the frames are completed. Repeated. Thus, the first and subsequent MCTF band division processing by the prediction processing unit 4111 ends.

  Subsequently, an update process is performed. In the update process, in order to reflect only the high frequency component in the time direction in the signal of the prediction process result in the update process, the signal (H frame) of the prediction process result output from the filtering unit 411D is weighted for update. The unit 4112 weights W (step S209 in FIG. 5).

  Next, the signal (H frame) of the prediction processing result of the frame (even number) before and after the processing target frame (odd number) weighted by W in the update weighting unit 4112 is obtained from the prediction processing unit 4111. Using the information, the update processing unit 4113 performs inverse MC (step S210 in FIG. 5). Subsequently, the update processing unit 4113 performs a filtering process on the inverted MC signal and the signal selected by the input switching unit 4110 (step S211 in FIG. 5). As a result, an L frame subjected to the update process is generated.

  Subsequently, it is determined whether or not the update process has been completed for even frames in the same GOP (step S212 in FIG. 5), and the update processes in steps S209 to S211 are performed for other frames until the update process is completed. .

  When it is determined in step S212 that the update process has been completed, the first band division process by MCTF is completed, and then the input switching unit 4110 is switched to select the L frame from the update processing unit 4113. Thereafter, the band splitting process by the MCTF after the second time in the above steps S203 to S212 is performed, and the above steps S203 and after until the remaining L frame becomes one or the MCTF is ended halfway. This process is repeated, and the band division in the time direction is repeated as in the conventional MCTF (step S213 in FIG. 5). Finally, the frame rearrangement unit 4115 rearranges the frames in a desired frame order, and outputs them to the outside of the MCTF-ILP unit 411. By using the above method, a signal subjected to MCTF-ILP processing can be generated.

  Next, handling of control information (motion information, weight information) will be described. Basically, the motion information obtained by the ME / MC unit 4118 and the weight W calculated by the weight instruction unit 4119 are directly output to the outside of the MCTF-ILP unit 411 and are entropy encoded.

  Thus, according to the present embodiment, the signal (H frame) after the prediction process output from filtering section 411D is fed back to weight instruction section 4119 so that the output signal of signal combining section 411C becomes small. The optimum weight W is obtained, and the signal input to the filtering unit 411D is weighted according to the value of the weight W or the value of (1-W), and the prediction process output from the filtering unit 411D is performed. Since the signal (H frame) is also weighted by the weight W, the video signal obtained by the resolution interpolation of the base layer local decode signal and the ME / MC-processed signal have the weight W value, respectively. Filtering by the filtering unit 411D is performed on the mixed signal accordingly. Since an H frame is generated and an L frame is generated by performing an update process on the signal weighted by the weight W to the H frame, only the time direction component can be reflected in the L frame (L frame There is always a block that has been filtered by the filtering unit 411D).

  For this reason, it is possible to realize band division in the time direction while predicting between resolutions, and as a result, the block that uses the base layer local decode signal in the prediction process is not subjected to the filter process in the update process. , A block in which the L frame is subjected to filter processing and a block in which the frame is thinned out without being subjected to the filter processing although there may be a problem in the quality of the video signal if the filter processing is not performed. In addition, the problem of Non-Patent Document 1 in which a local difference in image quality occurs can be solved, and the flicker in the decoded video is improved. Also, in MCTF, time band division is repeatedly performed, so that degradation caused by thinning out frames without being subjected to filter processing on L frames propagates to the next MC and time band division processing. The problem that it is difficult to maintain the accuracy of the subsequent processing can be solved, and the coding efficiency can be improved because the degree of freedom of space-time component prediction and band division increases. In addition, since the configuration and procedure of the MCTF-ILP unit 411 according to the present embodiment are all linear processing, reconfiguration on the decoding side is possible as long as the influence of quantization can be ignored.

  As an application example of this embodiment, for example, the weight W may be set to “0”. This case is equivalent to the case of selecting the base layer information in Non-Patent Document 1, and the problem of Non-Patent Document 1 remains, but the motion information update unit 4116 in FIG. 4 sets the vector to “0”. This method can be employed when the entropy is reduced. Furthermore, if the value of the weight W is limited to “0” or “1” and the weight W is not encoded, it is completely equivalent to the non-patent document 1, and compatibility can be maintained.

  Next, the enhancement layer decoding unit 51 in the decoding unit 50 of FIG. 1 will be described. FIG. 6 shows a block diagram of an embodiment of the enhancement layer decoding unit 51. As shown in the figure, the enhancement layer decoding unit 51 includes an entropy decoding unit 511, an inverse quantization / inverse orthogonal transform unit 512, and an inverse MCTF-ILP unit 513. The entropy decoding unit 511 acquires and analyzes a bitstream, performs entropy decoding, decodes video information, weight information, and motion information, and an inverse quantization / inverse orthogonal transform unit for the decoded video information And the function of outputting the weight information and the motion information to the inverse MCTF-ILP unit 513. Here, when the input bit storm corresponds to SNR scalability, a configuration and a procedure for decoding the bit storm may be provided as in the conventional method.

  The inverse quantization / inverse orthogonal transform unit 512 acquires the decoded video information output from the entropy decoding unit 511, performs inverse quantization on the decoded video information, and then performs inverse orthogonal transform, and inverse quantization and inverse orthogonal A function of outputting the converted signal to the inverse MCTF-ILP unit 513; The inverse MCTF-ILP unit 513 includes a signal obtained by inverse quantization and inverse orthogonal transform in the inverse quantization / inverse orthogonal transform unit 512, decoded weight information and motion information output from the entropy decoding unit 511, A function for obtaining a signal obtained by decoding the base layer in one base layer decoding unit 32, a function for obtaining a decoded video signal by performing band synthesis in the time direction and inverse prediction between resolutions, and an enhancement layer for the decoded video signal. A function of outputting to the outside of the decoding unit 51.

  Next, the procedure for decoding the enhancement layer by the enhancement layer decoding unit 51 having the configuration shown in FIG. 6 will be described with reference to the flowchart of FIG. The enhancement layer decoding unit 51 decodes the input bit stream in the entropy decoding unit 511 (step S301 in FIG. 7). The video information obtained as a result of decoding is sent to the inverse quantization / inverse orthogonal transform unit 512, and the decoded weight information and motion information are sent to the inverse MCTF-ILP unit 513.

  The decoded video information is inversely quantized by the inverse quantization / inverse orthogonal transform unit 512 and then inversely orthogonal transformed (step S302 in FIG. 7). The signal obtained as a result is sent to the inverse MCTF-ILP unit 513. The inverse MCTF-ILP unit 513 performs temporal direction band synthesis and inverse prediction between resolutions using the video information obtained by inverse orthogonal transformation, the base layer decoded signal, and the decoded weight information and motion information, and outputs the decoded video signal. Is obtained (step S303 in FIG. 7). Details of the procedure will be described later.

  Next, the configuration and operation of the inverse MCTF-ILP unit 513 in the enhancement layer decoding unit 51 shown in FIG. 6 will be described in detail. FIG. 8 shows a configuration diagram of an embodiment of the inverse MCTF-ILP unit 513. As shown in the figure, the inverse MCTF-ILP unit 513 includes a frame management unit 5130, a frame division unit 5131, an input switch 5132, a reverse update weighting unit 5133, a reverse update processing unit 5134, a reverse prediction processing unit 5135, and The frame rearrangement unit 513C is configured.

  The frame management unit 5130 manages the frame in the inverse MCTF-ILP unit 513, and the frame management information of the frame management unit 5131, the input switching unit 5132, the reverse update processing unit 5134, the reverse prediction processing unit 5135, and the frame A function of outputting to the rearrangement unit 513C.

  The frame division unit 5131 divides the acquired video information into an L frame and an H frame and acquires the video information to be restored and the frame management information output from the frame management unit 5130, and the L frame is input to the input switch 5132. And a function of outputting the H frame to the reverse update weighting unit 5133 and the reverse prediction processing unit 5135.

  The input switching unit 5132 acquires a signal output from the frame dividing unit 5131, a signal output from the reverse prediction processing unit 5135, and a frame management information output from the frame management unit 5130, according to the frame management information. A function of selecting either an output signal from the input frame dividing unit 5131 or a signal output from the reverse prediction processing unit 5135 and outputting the selected signal to the reverse update processing unit 5134;

  The inverse update weighting unit 5133 weights the H frame based on the weight information W based on the function of acquiring the H frame output from the frame dividing unit 5131 and the weight information decoded by the entropy decoding unit 511, And a function of outputting a signal obtained as a result to the reverse update processing unit 5134.

  The inverse update processing unit 5134 receives the signal (H frame) output from the inverse update weighting unit 5133, the signal output from the input switch 5132, the motion information decoded by the entropy decoding unit 511, and the frame management unit 5130. The function to acquire the output frame management information and the inverse MC based on the acquired motion information (the motion information is applied in the reverse direction to perform motion compensation), and the L frame obtained by further filtering And a function of outputting to the reverse prediction processing unit 5135 and the frame rearranging unit 513C.

  The reverse prediction processing unit 5135 is decoded by the frame management information output from the frame management unit 5130, the H frame output from the frame division unit 5131, the L frame output from the reverse update processing unit 5134, and the entropy decoding unit 511. A function of acquiring the motion information and weight information and the base layer decode signal, and a function of outputting the reverse prediction processing result to the frame rearrangement unit 513C. The internal configuration and operation of the reverse prediction processing unit 5135 will be described later.

  The frame rearranging unit 513C uses the L information restored by the reverse update processing unit 5134, the H frame restored by the reverse prediction processing, and the frame management information output from the frame management unit 5130, and the frame information. The L frame restored by the reverse update processing unit 5134 and the H frame restored by the reverse prediction processing 2107 are rearranged in a desired frame order and output to the input switching unit 5132 and the outside.

  Next, a detailed configuration example of the reverse prediction unit 5135 will be described. As shown in FIG. 8, the inverse prediction processing unit 5135 includes an MC unit 5136, a resolution interpolation unit 5137, an inverse prediction weighting unit 5138, an inverse ILP weighting unit 5139, a signal / synthesis unit 513A, and a filtering unit 513B. Consists of

  The MC unit 5136 performs a function of acquiring the L frame output from the inverse update processing unit 5134, the motion information and the frame information output from the frame management unit 5130, and motion compensation (MC) of the L frame based on the motion information. And a function of outputting a signal obtained as a result to the weighting unit 5138 for reverse prediction.

  The resolution interpolation unit 5137 interpolates the base layer decoded signal and the frame management information output from the frame management unit 5130 and the acquired base layer decoded signal to the same spatial resolution as the video information to be restored. It has the function to do. Here, any interpolation method may be used, but it is preferable to use a filter corresponding to the filter used in the MCTF-ILP unit 411 of the encoding unit 40. Further, the resolution interpolation unit 5137 has a function of outputting a signal resulting from the interpolation to the inverse ILP weighting unit 5138.

  The weighting unit for inverse prediction 5138 weights the signal output from the MC unit 5136 based on the function of acquiring the signal and weight information output from the MC unit 5136 and the acquired weight information W, and performs signal synthesis. A function of outputting to the unit 513A. The inverse ILP weighting unit 5139 calculates (1-W) based on the function of acquiring the signal and weight information output from the resolution interpolation unit 5137 and the acquired weight information W, and the resolution interpolation unit A function of weighting the signal output from 5137 and outputting the result to the signal synthesis unit 513A.

  The signal synthesis unit 513A adds the respective signals output from the inverse prediction weighting unit 5138 and the inverse ILP weighting unit 5139, and the obtained signals, for example, and synthesizes them to obtain an inverse MCTF-ILP. For generating a reference frame for and for outputting to the filtering unit 513B.

  The filtering unit 513B has a function of acquiring the H frame output from the frame dividing unit 5131, the reference frame output from the signal combining unit 513A, and the frame management information output from the frame management unit 5130, and the acquired H frame and reference The filter processing is performed on the frames to generate a band synthesis in the time direction and a reversely predicted (restored) signal between the resolutions and output to the frame rearranging unit 513C.

  Next, a procedure for performing the inverse MCTF-ILP process by the inverse MCTF-ILP unit 513 having the configuration shown in FIG. 8 will be described with reference to the flowchart of FIG. In order to perform the series of processing shown in FIG. 9 in units of 1 GOP, video frames for 1 GOP are accumulated (step S401 in FIG. 9). Note that the series of processing shown in FIG. 9 may be performed in units of a plurality of GOPs, and processing that spans a plurality of GOPs may be performed. Subsequently, the base layer decoded signal from the base layer decoding unit 32 in FIG. 1 is interpolated in advance in the resolution interpolation unit 5137 in FIG. 8 (step S402 in FIG. 9).

  Next, reverse update processing is performed. In the reverse update process, L frames of the input video information to be restored are initially input to the reverse update processing unit 5134 through the input switch 5132. Here, since the second and subsequent band synthesis processes are performed on the restored frame (output of the reverse prediction process result), the restored frame is output from the input switching unit 5132.

  The H-frame signal used for the reverse update process is weighted by the weight W in the reverse update weighting unit 5133 (step S403 in FIG. 9). The H frame signal weighted with W before and after the L frame to be processed is inverse-MCed using the motion information in the inverse update processing unit 5134 (step S404 in FIG. 9). Further, the inverse update processing unit 5134 performs a filter process on the signal that is inverse MC to the signal output from the input switching unit 5132 (step S405 in FIG. 9). As a result, a restored frame subjected to the reverse update process is generated. Subsequently, it is determined whether or not all reverse updates have been completed for L frames at the same level in the GOP (step S406 in FIG. 9), and the processes of steps S403 to S405 are repeated until all the reverse updates are completed.

  Next, reverse prediction processing is performed. In this reverse prediction process, the MC unit 5136 MCs the L frame signal restored by reverse updating before and after the H frame to be processed using the motion information (step S407 in FIG. 9). Then, the weighted information W is weighted by the inverse prediction weighting unit 5138 for the MC signal (step S408 in FIG. 9). On the other hand, for the base layer decoded signal subjected to resolution interpolation by the resolution interpolation unit 5137, the inverse ILP weighting unit 5139 calculates a weight (1-W) value and weights it (FIG. 9). Step S409).

  The signals weighted by the inverse prediction weighting unit 5138 and the inverse ILP weighting unit 5139 are supplied to the signal synthesis unit 513A and synthesized, and generated as a final reference frame for the prediction process (see FIG. 9 step S410). The filtering unit 5138 performs a filtering process on the reference frame output from the signal combining unit 513A and the H frame output from the frame dividing unit 5131 (step S411 in FIG. 9). As a result, the signal of the reverse prediction processing result in the reverse MCTF-ILP unit 513 can be obtained.

  Subsequently, it is determined whether or not the reverse prediction process for the H frame at the same level in the GOP has been completed (step S412 in FIG. 9). If not, the processes in steps S407 to S411 are repeated until the process is completed. Is called. Note that band synthesis in the time direction is repeatedly performed according to the number of band resolutions in the time direction of the encoding unit 40 and the environment of the decoding unit 50 (step S413 in FIG. 9). In other words, the operation after step S403 is repeated until the all-level inverse MCTF-ILP is completed or until the inverse MCTF-ILP is terminated halfway.

  Finally, the frame rearrangement unit 513C rearranges the frames in a desired frame order and outputs the result to the outside of the inverse MCTF-ILP unit 513. The above is an example of the configuration and procedure of the enhancement layer encoding unit 41 including the MCTF-ILP unit 411 and the enhancement layer decoding unit 51 including the inverse MCTF-ILP unit 513 in the present embodiment. By implementing these, it is possible to realize a scalable coding scheme in which quality is improved with higher coding efficiency than in the prior art.

  Next, embodiments of the encoding program and decoding program of the present invention will be described. FIG. 10 shows a block diagram of an example of an information processing apparatus 100 provided with an embodiment of an encoding program and a decoding program according to the present invention. As shown in the figure, the information processing apparatus 100 includes an external storage device 101, a temporary storage device 102, a communication device 103, an input device 104, an output device 105, and a central processing control device 106, and is a central computer that is a computer. The processing control device 106 realizes the functions of the encoding and decoding device by a program.

  Here, the above program may be read from the recording medium by the input device 104 and taken into the central processing control device 106, or may be received by the communication device 103 via the network and taken into the central processing control device 106. Also good.

  The central processing control device 106 uses the above encoding program to generate a space-time decimation unit 107 corresponding to the space-time decimation unit 11 of the encoding unit 40, a base layer encoding unit 108 corresponding to the base layer encoding unit 12, a base layer The base layer reconstructing unit 109 corresponding to the reconstructing unit 13 and the enhancement layer encoding unit 110 corresponding to the enhancement layer encoding unit 41 are executed, and it corresponds to the extract unit 31 of the decoding unit 50 by a decoding program. The extraction unit 113, the base layer decoding unit 113 corresponding to the base layer decoding unit 32, and the enhancement decoding unit 114 corresponding to the enhancement layer decoding unit 51 are executed.

It is a block diagram of one embodiment of a video signal encoding device and a video signal decoding device of the present invention. It is a block diagram of one Embodiment of the enhancement layer encoding part of the principal part of the video signal encoding apparatus of this invention. It is a flowchart for operation | movement description of FIG. It is a block diagram of one Embodiment of the MCTF-ILP part in FIG. 5 is a flowchart for explaining the operation of FIG. It is a block diagram of one Embodiment of the enhancement layer decoding part of the principal part of the video signal decoding apparatus of this invention. It is a flowchart for operation | movement description of FIG. It is a block diagram of one Embodiment of the reverse MCTF-ILP part in FIG. It is a flowchart for operation | movement description of FIG. It is a block diagram of an example of the information processing apparatus which performs the encoding and decoding program of this invention. It is a figure which shows an example of MCTF. It is a flow chart showing an example of MCTF. It is a figure which shows an example of the signal after MCTF process. It is a block diagram of an example of the conventional video signal encoding device and video signal decoding device. It is a flowchart for operation | movement description of the conventional encoding part. It is a flowchart for operation | movement description of the conventional decoding part. It is a block diagram of an example of the enhancement layer encoding part of the principal part of the conventional video signal encoding apparatus. 18 is a flowchart for explaining the operation of FIG. It is a block diagram of an example of the enhancement layer encoding part of the principal part of the conventional video signal decoding apparatus. 20 is a flowchart for explaining the operation of FIG. It is a figure explaining the operation | movement of an example of the MCTF-ILP part in FIG. It is a flowchart for operation | movement description of the conventional MCTF-ILP part. It is a figure explaining operation | movement of an example of the conventional reverse MCTF-ILP part. It is a flowchart for operation | movement description of the conventional reverse MCTF-ILP part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Space-time decimation part 12 Base layer encoding part 13 Base layer restructuring part 15 Multiplexing part 20 Communication line or media 31 Extraction part 32 Base layer decoding part 40 Encoding part 41 Enhancement layer encoding part 50 Decoding part 51 Enhancement Layer decoding unit 100 Information processing device 106 Central processing control device 110 Enhancement layer encoding unit 114 Enhancement layer decoding unit 411 MCTF-ILP unit 412 Orthogonal transformation / quantization unit 413 Entropy encoding unit 511 Entropy decoding unit 512 Inverse quantization / inverse Orthogonal transformation unit 513 Inverse MCTF-ILP unit 4110, 5132 Input switching unit 4111 Prediction unit 4112 Update weight 4114 5130 Frame management unit 4115, 513C Frame rearrangement unit 4116 Motion information update unit 4117, 5137 Resolution interpolation unit 4118 ME / MC unit 4119 Weight instruction unit 411A Prediction weighting unit 411B For ILP Weighting unit 411C Signal combining unit 411D, 513B Filtering unit 5131 Frame division unit 5133 Reverse update weighting unit 5134 Reverse update processing unit 5135 Reverse prediction processing unit 5136 MC unit 5138 Reverse prediction weighting unit 5139 Reverse ILP weighting unit 513A Signal・ Synthesizer

Claims (4)

A frame sequence composed of an L frame in a low frequency division frequency band and an H frame in a high frequency division frequency band, which are subband divided into two division frequency bands by performing compensation time direction filtering on the input video signal A first encoded signal obtained by generating a signal and performing orthogonal transform / quantization and entropy encoding on the frame sequence signal is encoded with a video signal having a lower resolution than the input video signal. A video signal encoding device for outputting together with the encoded signal of 2;
Reduction means for space-time reduction of the input video signal by a predetermined ratio to generate a low-resolution video signal having a space-time resolution different from the input video signal;
Encoding means for encoding the low resolution video signal to generate the second encoded signal;
Local decoding means for locally decoding the second encoded signal to generate a local decoded signal;
Expansion means for expanding the spatial resolution of the local decoded signal at a ratio opposite to that of the reduction means, and generating a high spatial resolution video signal;
Correspondence between the reference frame and the other frame is estimated by estimating a motion of another frame among the plurality of frames using one of the different frames existing in the time direction of the input video signal as a reference frame. Motion estimation / compensation means for acquiring motion information indicating a relative positional relationship of each block and compensating for motion of the other frame based on the motion information;
First weighting means for weighting the input video signal motion-compensated by the motion estimation / compensation means with a first weight;
Second weighting means for weighting the high spatial resolution video signal generated by the enlarging means from the local decoded signal with a second weight calculated using the first weight;
First processing means for generating the H frame by combining and filtering the respective signals weighted by the first and second weighting means;
Third weighting means for weighting the H frame generated by the first processing means with the first weight;
Applying the motion information acquired by the motion estimation / compensation means in the reverse direction to the H frame weighted by the third weighting means to perform motion compensation and then filtering to generate the L frame. Two processing means;
Frame sequence signal generating means for generating and outputting the frame sequence signal by combining the H frame and the L frame generated by the first and second processing units, respectively;
Frame sequence signal encoding means for performing orthogonal transform / quantization and entropy encoding on the frame sequence signal generated by the frame sequence signal generation means to generate the first encoded signal. A video signal encoding device. A frame sequence composed of an L frame in a low frequency division frequency band and an H frame in a high frequency division frequency band, which are subband divided into two division frequency bands by performing compensation time direction filtering on the input video signal A first encoded signal obtained by generating a signal and performing orthogonal transform / quantization and entropy encoding on the frame sequence signal is encoded with a video signal having a lower resolution than the input video signal. A video signal encoding program for causing a computer to execute an operation to be output together with the encoded signal of 2;
The computer,
Reduction means for space-time reduction of the input video signal by a predetermined ratio to generate a low-resolution video signal having a space-time resolution different from the input video signal;
Encoding means for encoding the low resolution video signal to generate the second encoded signal;
Local decoding means for locally decoding the encoded signal to generate a local decoded signal;
Expansion means for expanding the spatial resolution of the local decoded signal at a ratio opposite to that of the reduction means, and generating a high spatial resolution video signal;
The movement of the other frames of the plurality of frames is estimated using one of a plurality of different frames existing in the time direction of the input video signal as a reference frame, and the correspondence between the reference frame and the other frames is estimated. Motion estimation / compensation means for acquiring motion information indicating a relative positional relationship of each block and compensating for the motion of the other frame based on the motion information;
First weighting means for weighting the input video signal motion-compensated by the motion estimation / compensation means with a first weight;
Second weighting means for weighting the high spatial resolution video signal generated by the enlarging means from the local decoded signal with a second weight calculated using the first weight;
First processing means for generating the H frame by combining and filtering the respective signals weighted by the first and second weighting means;
Third weighting means for weighting the H frame generated by the first processing means with the first weight;
Applying the motion information acquired by the motion estimation / compensation means in the reverse direction to the H frame weighted by the third weighting means to perform motion compensation and then filtering to generate the L frame. Two processing means;
Frame sequence signal generating means for generating and outputting the frame sequence signal by synthesizing the H frame and the L frame generated by the first and second processing units, respectively;
The frame sequence signal generated by the frame sequence signal generation means is subjected to orthogonal transform / quantization and entropy encoding to function as a frame sequence signal encoding means for generating the first encoded signal. A video signal encoding program. The first and second encoded signals, the motion information, and the first to second signals output from the video signal encoding device according to claim 1 or generated by the video signal encoding program according to claim 2. A video signal decoding apparatus for receiving and decoding a signal including weight information indicating a third weight,
Receiving means for receiving and separating the first and second encoded signals, respectively;
First decoding means for decoding the H frame and the L frame by performing entropy decoding on the first encoded signal from the receiving means and then performing inverse quantization and inverse orthogonal transform;
Second decoding means for decoding the second encoded signal from the receiving means to generate the low resolution video signal;
Expansion means for enlarging a spatial resolution of the low resolution video signal decoded by the second decoding means to generate a high spatial resolution video signal;
First weighting means for weighting the H frame from the first decoding means with the first weight obtained from the received weight information;
The H frame or decoded video signal weighted by the first weighting unit is applied to the motion information received by the receiving unit in the reverse direction to perform motion compensation and then filtered to obtain a first decoded signal. First processing means to generate;
A second weighting that weights the first decoded signal with the first weight obtained from the weight information received by the receiving means after applying motion compensation to the motion information received by the receiving means; Means,
Third weighting means for weighting the high spatial resolution video signal from the enlarging means with a second weight obtained from the weight information received by the receiving means;
The second weighted signal is synthesized by the second and third weighting means, and the second decoded signal is generated by performing filtering on the synthesized signal and the H frame received and separated by the receiving means. Two processing means;
A video signal decoding apparatus comprising: a decoded video signal generation unit configured to combine the first decoded signal and the second decoded signal to generate the decoded video signal. The first and second encoded signals, the motion information, and the first to second signals output by the video signal encoding device according to claim 1 or generated by the video signal encoding program according to claim 2. A video signal decoding program for receiving a signal composed of weight information indicating a third weight and decoding the signal by a computer,
The computer,
Receiving means for receiving and separating the first and second encoded signals, respectively;
First decoding means for decoding the H frame and the L frame by performing entropy decoding on the first encoded signal from the receiving means and then performing inverse quantization and inverse orthogonal transform;
Second decoding means for decoding the second encoded signal from the receiving means to generate the low resolution video signal;
Expansion means for enlarging a spatial resolution of the low resolution video signal decoded by the second decoding means to generate a high spatial resolution video signal;
First weighting means for weighting the H frame from the first decoding means with the first weight obtained from the received weight information;
The H frame or decoded video signal weighted by the first weighting unit is applied to the motion information received by the receiving unit in the reverse direction to perform motion compensation and then filtered to obtain a first decoded signal. First processing means to generate;
A second weighting that weights the first decoded signal with the first weight obtained from the weight information received by the receiving means after applying motion compensation to the motion information received by the receiving means; Means,
Third weighting means for weighting the high spatial resolution video signal from the enlarging means with a second weight obtained from the weight information received by the receiving means;
The second weighted signal is synthesized by the second and third weighting means, and the second decoded signal is generated by filtering the synthesized signal and the H frame received and separated by the receiving means. Two processing means;
Decoded video signal generation means for generating the decoded video signal by combining the first decoded signal and the second decoded signal;
And a video signal decoding program characterized by the above.

Images