JP2007096541A - Coding method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving image coding technique of reducing the coding amount due to motion vector information. <P>SOLUTION: In the coding method of obtaining multiple layers with different frame rates by performing motion compensation temporal filtering for a moving image in a recursive manner, the motion vector precision used for motion compensation prediction in each layer by each layer can be varied. The layer and the motion vector precision can be associated with each other in advance or the layer and the motion vector precision can be associated with each other by the prescribed number of pictures. Correlation information thus established is included in the coded data of the moving image. As the frame rate of the layer decreases, it is preferable to reduce the motion vector precision. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an encoding method for encoding a moving image.

  Broadband networks are rapidly developing, and there are high expectations for services that use high-quality moving images. In addition, a large-capacity recording medium such as a DVD is used, and a user group who enjoys high-quality images is expanding. There is compression coding as an indispensable technique for transmitting moving images via a communication line or storing them in a recording medium. As an international standard for moving image compression coding technology, the MPEG4 standard and H.264 standard. There is a H.264 / AVC standard. In addition, there is a next-generation image compression technique such as SVC (Scalable Video Coding) in which one stream includes a high-quality stream and a low-quality stream.

  H. In the H.264 / AVC standard, in order to perform more detailed prediction in motion compensation, the block size of motion compensation can be made variable, and the pixel accuracy of motion compensation can be reduced to ¼ pixel accuracy. The amount of code related to the motion vector increases. In SVC, which is a next-generation image compression technology, MCTF (Motion Compensated Temporal Filtering) technology is being studied in order to improve temporal scalability. This is a combination of subband division in the time axis direction and motion compensation. Since hierarchical motion compensation is performed, information on motion vectors becomes very large. As described above, the recent video compression coding technology tends to increase the data amount of the entire video stream due to an increase in the amount of information related to motion vectors, and there is a further demand for a technology for reducing the amount of codes resulting from motion vector information. ing.

Patent Document 1 discloses an encoding method for switching the encoding accuracy of a motion vector for each block. As a result, the amount of motion vector coding in low-rate coding can be reduced.
JP 2004-48522 A

  In a frame with many high-frequency components and strong correlation with the reference frame, the prediction error can be reduced by increasing the accuracy of the motion vector and performing high-precision motion compensation, but the motion of the object moving in the frame For frames that are weakly correlated with the reference frame due to high speed and frames that have few high-frequency components, reducing the prediction error does not contribute even if the accuracy of motion compensation is increased. It will be useless.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a moving image encoding technique capable of reducing the amount of code caused by motion vector information.

  In order to solve the above-described problem, an aspect of the present invention provides an encoding method for generating encoded data having a plurality of layers having scalability from a moving image, for each layer, for motion compensation prediction within the layer. It is characterized in that the accuracy of the motion vector used can be changed.

  According to this aspect, by using the optimal motion vector accuracy for each layer, it is possible to improve the compression efficiency of moving images by suppressing the amount of code of useless motion vectors that does not lead to a reduction in prediction errors. Scalability types include temporal scalability and spatial scalability.

  A plurality of layers having different frame rates may be obtained by recursively performing motion compensation time filtering on a moving image. Further, the above method can be applied to an encoding method for obtaining a plurality of layers having different frame rates by performing motion compensation time filtering on a moving image according to the MCTF technique. According to this, since the code amount of motion vector information can be reduced in MCTF in which motion vector information is obtained for each layer, the compression efficiency of moving images is improved.

  The hierarchy and the accuracy of the motion vector may be associated in advance, and the correspondence information may be included in the encoded data of the moving image. Thereby, the accuracy of the motion vector used for motion compensation prediction in each layer can be determined for each encoded data stream.

The hierarchy and the accuracy of the motion vector are associated with each fixed number of pictures, and the correspondence information may be included in the encoded data of the moving image. This makes it possible to determine the accuracy of the motion vector used for motion compensation prediction in each layer for each fixed unit picture such as GOP.
A “picture” is a unit of encoding including a frame, a field, a VOP (Video Object Plane), and the like.

  The hierarchy and the accuracy of the motion vector are associated in advance, and the accuracy of the motion vector in each hierarchy may be determined according to this association. This eliminates the need to include correspondence information between the hierarchy and the motion vector accuracy in the encoded data.

  You may make it change the precision of a motion vector in steps every time it passes through a hierarchy. Further, the accuracy of the motion vector may be coarsened as the frame rate of the hierarchy is lowered. In general, if the frame rate decreases and the correlation between frames decreases, even if the accuracy of the motion vector is reduced, it is considered that the influence on the prediction error is small, so this can reduce the code amount of the motion vector information, The compression efficiency of moving images is improved.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the encoding amount resulting from motion vector information can be reduced in the encoding of a moving image.

  FIG. 1 is a configuration diagram of an encoding apparatus 100 according to an embodiment. These configurations can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and in software, it is realized by a program having an image encoding function loaded in the memory. Here, functional blocks realized by the cooperation are depicted. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The encoding apparatus 100 according to the present embodiment includes an ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) which is an international standardization organization, and an ITU-T (International Telecommunication Union-Telecommunication Standardization) which is an international standard organization related to telecommunications. H., the latest video compression coding standard jointly standardized by Sector). H.264 / AVC (official recommendation names in both organizations are MPEG-4 Part 10: Advanced Video Coding and H.264 respectively).

  The image acquisition unit 10 of the encoding device 100 receives a GOP (Group of Pictures) of the input image and stores each frame in a dedicated area of the image holding unit 60. The image acquisition unit 10 may divide each frame into macro blocks as necessary.

  The MCTF processing unit 20 performs motion compensation time filtering according to the MCTF technique. The MCTF processing unit 20 obtains a motion vector from the frame stored in the image holding unit 60, and performs temporal filtering using the motion vector. Temporal filtering is performed using a Haar wavelet transform, and as a result, the temporal filtering is decomposed into a plurality of layers having different frame rates including a high frequency frame H and a low frequency frame L in each layer. The decomposed high-frequency frame and low-frequency frame are stored in a dedicated region of the image holding unit 60 for each layer, and motion vectors are also stored in a dedicated region of the motion vector holding unit 70 for each layer. Details of the MCTF processing unit 20 will be described later.

  When the processing in the MCTF processing unit 20 is completed, the high frequency frames of all layers and the low frequency frame of the final layer in the image holding unit 60 are sent to the image encoding unit 80. Also, the motion vectors of all layers in the motion vector holding unit 70 are sent to the motion vector encoding unit 90.

  The image encoding unit 80 performs encoding after performing spatial filtering using wavelet transform on the frame supplied from the image holding unit 60. The encoded frame is sent to the multiplexing unit 92. The motion vector encoding unit 90 performs encoding on the motion vector supplied from the motion vector holding unit 70 and supplies the encoded motion vector to the multiplexing unit 92. Since the encoding method is known, detailed description is omitted.

  The multiplexing unit 92 multiplexes the encoded frame information given from the image coding unit 80 and the coded motion vector information given from the motion vector coding unit 90 to generate a coded stream. To do.

Subsequently, a time filtering process according to the MCTF technique will be described with reference to FIGS. 2 and 3.
The MCTF processing unit 20 sequentially acquires two consecutive frames in one GOP and generates a high frequency frame and a low frequency frame. The two frames will be referred to as “frame A” and “frame B” in time order.

The MCTF processing unit 20 detects a motion vector MV from the frame A and the frame B. In FIG. 2 and FIG. 3, motion vectors are detected in units of frames for the sake of simplicity. However, motion vectors may be detected in units of macroblocks, or blocks (8 × 8 pixels or 4 × 4). The motion vector may be detected in units of pixels.
Next, an image (hereinafter referred to as “frame A ′”) in which frame A is motion-compensated with a motion vector MV is generated.
The low frequency frame L is defined as an average value of the frames A ′ and B as shown in FIG.
L = 1/2 · (A ′ + B) (1)

Next, an image (hereinafter referred to as “frame B ′”) in which the frame B is motion-compensated with the inversion value −MV of the motion vector MV is generated.
The high frequency frame H is defined as a difference between the frame A and the frame B ′ as shown in FIG.
H = A−B ′ (2)

Equation (2) is transformed.
A = B '+ H (3)
If motion compensation is performed by the motion vector MV on both the right side and the left side, the following equation is established. “H ′” represents an image obtained by motion compensation of the high frequency frame H with the motion vector MV.
A '= B + H' (4)
Substituting equation (4) into equation (2) gives the following equation.
L = 1/2 · (A '+ B)
= 1/2 ・ (B + H '+ B)
= B + 1/2 · H '(5)
That is, the low-frequency frame L can be generated by adding the pixel values of the frame B and the pixel values of the high-frequency frame H ′ that are halved.

  By repeating the same operation as described above with the generated low-frequency frame L as a new frame A and frame B, a high-frequency frame, a low-frequency frame, and a motion vector of the next hierarchy are generated. This operation is recursively repeated until one low frequency frame is generated. Therefore, the number of layers obtained is determined by the number of frames included in the GOP. For example, if the GOP contains 8 frames, the first operation generates four high frequency frames and four low frequency frames (layer 2), and the second operation generates two high frequency frames and two low frequency frames. A region frame is generated (layer 1), and one high-frequency frame and one high region frame are generated by the third operation (layer 0).

  FIG. 4 shows the configuration of the MCTF processing unit 20. The frame A and the frame B stored in the image holding unit 60 are input to the motion vector detection unit 21. As described above, frame A and frame B are frames constituting the GOP in layer 2, but note that in layer 1 and later, the low-frequency frame L generated in the immediately preceding layer becomes frame A and frame B. To do.

  The motion vector accuracy determination unit 28 determines the accuracy of the motion vector used for motion compensation prediction, that is, the minimum pixel, and provides the motion vector detection unit 21 with the accuracy. As will be described later, in the present embodiment, the accuracy of the motion vector can be determined for each layer. Accordingly, the motion vector accuracy determination unit 28 determines which layer of frame motion compensation is being performed at that time, and determines the accuracy of the motion vector.

  The motion vector detection unit 21 searches the frame A for a prediction area with the smallest error for each macroblock in the frame B, and obtains a motion vector MV indicating a deviation from the macroblock to the prediction area. At this time, the motion vector MV is obtained with the accuracy given from the motion vector accuracy determination unit 28. The motion vector MV is stored in the motion vector holding unit 70 and supplied to the motion compensation units 22 and 24.

  The motion compensation unit 22 performs motion compensation for each macroblock by using (−MV) obtained by inverting the motion vector MV output from the motion vector detection unit 21 with respect to the frame B, and generates a frame B ′. .

  The image composition unit 23 adds the pixels of the frame A and the frame B ′ output from the motion compensation unit 22 to generate a high frequency frame H. The high frequency frame H is stored in the image holding unit 60 and supplied to the motion compensation unit 24. The motion compensation unit 24 performs motion compensation for each macroblock using the motion vector MV for the high frequency frame H, and obtains a frame H ′. The obtained frame H ′ is multiplied by ½ by the processing block 25 and supplied to the image composition unit 26.

  The image composition unit 26 adds the pixels of the frame B and the frame H ′ to generate the low-frequency frame L. The generated low frequency frame L is stored in the image holding unit 60.

  FIG. 5 is a diagram illustrating an image and a motion vector output in each layer when the GOP is configured with 8 frames. FIG. 6 is a flowchart showing an encoding method according to the MCTF technique. A specific example will be described with reference to FIGS.

Hereinafter, the high-frequency frame of layer n is expressed as H _n , the low-frequency frame is _expressed as L _n , and the motion vector is expressed as MV _n . In the example of FIG. 5, among the frames 101 to 108 in the GOP, the frames 101, 103, 105, and 107 become the frame A, and the frames 102, 104, 106, and 108 become the frame B.

First, the image acquisition unit 10 receives the frames A and B and stores them in the image holding unit 60 (S10). At this time, the image acquisition unit 10 may divide the frame into macro blocks. Subsequently, the MCTF processing unit 20 reads out the frame A and the frame B from the image holding unit 60, and executes the first time filtering process (S12). The generated high frequency frame H ₂ and low frequency frame L ₂ are stored in the image holding unit 60, and the motion vector MV ₂ is stored in the motion vector holding unit 70 (S14). When the processing of the frame 101-108 is finished, MCTF processor 20, the image storing unit 60 reads out the low band frames _{L 2,} executes the second time temporal filtering process (S16). The generated high frequency frame H ₁ and low frequency frame L ₁ are stored in the image holding unit 60, and the motion vector MV ₁ is stored in the motion vector holding unit 70 (S18). Subsequently, MCTF processor 20 reads from the image holding unit 60 the two low-pass frames _{L 1,} executes the third time temporal filtering process (S20). The generated high frequency frame H ₀ and low frequency frame L ₀ are stored in the image holding unit 60, and the motion vector MV ₀ is stored in the motion vector holding unit 70 (S22).

The high frequency frames H _{0 to} H ₂ and the low frequency frame L ₀ are encoded by the image encoding unit 80 (S24), and the motion vectors MV _{0 to} MV ₂ are encoded by the motion vector encoding unit 90 (S26). ). The encoded frame and motion vector are multiplexed by the multiplexing unit 92 and output as an encoded stream (S28).

  Since the high frequency frame H is a difference between frames, the amount of data at the time of encoding decreases. Further, as can be seen from FIG. 5, the number of low-frequency frames L decreases to ½ each time a time filtering process is performed, but the low-frequency frames L are average values between frames of higher layers. Therefore, a frame sequence in which the image quality and the resolution are not deteriorated can be obtained. As one specific example, if the original moving image is 60 fps, the frame rate decreases every time the layer passes, such as 30 fps in layer 2, 15 fps in layer 1, and 7.5 fps in layer 0. Therefore, moving images having different frame rates can be transmitted in one bit stream.

  The decoding device that has received the encoded stream executes decoding processing in order from the lower layer. If only the lower layer is decoded, a moving image with a low frame rate is obtained, and a moving image with an increased frame rate is obtained as the upper layer is decoded. Thus, temporal scalability can be achieved by temporal filtering according to MCTF technology.

In the present embodiment, the motion vector accuracy determination unit 28 can change the accuracy of the motion vector used for motion compensation prediction in each layer for each layer. The association between the hierarchy and the motion vector accuracy may be determined in advance as an encoding standard, or may be arbitrarily set. For example, when the motion vector accuracy is set for each layer, as shown in FIG. 7, the accuracy data of each motion vector is stored in the header of each layer in the encoded stream. When the association between the layer and the motion vector accuracy is determined as an encoding standard, it is not necessary to include the association information in the encoded stream.
You may make it determine matching with a hierarchy and a motion vector precision for every encoding stream. In this case, the association information is stored in the header of the entire encoded stream. Further, the association between the hierarchy and the motion vector accuracy may be determined for each predetermined number of pictures such as GOP. In this case, the association information is stored in the GOP header or the like.

  FIG. 8 shows an example of the correspondence between the frame rate of the hierarchy and the motion vector accuracy. In this example, when the frame rate of the hierarchy is 30 to 60 fps, 1/4 pixel accuracy is used, when it is 15 to 30 fps, 1/2 pixel accuracy is used, and when it is 15 fps or less, 1 pixel accuracy is set. The motion vector accuracy determination unit 28 described above provides the motion vector detection unit 21 with the motion vector accuracy according to the table of FIG. 8 according to the frame rate of the layer that is the target of the motion compensation process. The motion vector accuracy determination unit 28 determines the motion vector accuracy of each layer so that the code amount of the difference image is minimized, instead of determining the motion vector accuracy of each layer according to a predetermined table as shown in FIG. You may decide. Alternatively, the motion vector accuracy determination unit 28 may receive a motion vector accuracy command for each layer from the outside when executing the encoding.

  As shown in FIG. 8, it is preferable that the accuracy of the motion vector becomes coarser as the frame rate of the hierarchy decreases. This is due to the following reason. That is, in general, when the frame rate is lowered, that is, when the temporal distance between frames is increased, the correlation between the frames is reduced. It cannot be said. In other words, if the difference value of the difference image can be further reduced by searching with high accuracy, the code amount can be reduced as a result even if the number of bits required to encode the motion vector increases. Therefore, as the frame rate of the hierarchy is lowered, moving picture coding efficiency is improved by reducing the accuracy of the motion vector and reducing the code amount of the motion vector. In addition, when it leads to reduction of the code amount of a moving image, you may coarsen the precision of a motion vector, so that the frame rate of a hierarchy becomes high.

  FIG. 9 is a configuration diagram of the decoding device 300 according to the embodiment. The encoded stream is input to the stream analysis unit 310 of the decoding device 300. The stream analysis unit 310 extracts a data portion corresponding to a necessary hierarchy, and further separates the decoded data of the frame from the decoded data of the motion vector. The frame data is provided to the image decoding unit 320, and the motion vector data is provided to the motion vector decoding unit 330. If the encoded stream includes motion vector accuracy data, the accuracy data is separated and supplied to the motion vector decoding unit 330.

The image decoding unit 320 performs entropy decoding and inverse wavelet transform, and generates a low-frequency frame L _{0 in the} lowest hierarchy and all high-frequency frames H _{0 to} H ₂ . The frame decoded by the image decoding unit 320 is stored in a dedicated area of the image holding unit 350.

The motion vector decoding unit 330 uses the motion vector accuracy data to decode the motion vector information, and then calculates the motion vector MV ₀ in the lowest hierarchy and the motion vectors MV ₁ and MV ₂ in the higher hierarchy. The motion vector decoded by the motion vector decoding unit 330 is stored in a dedicated area of the motion vector holding unit 360.

  The image synthesis unit 370 synthesizes frames in the reverse procedure of the above-described MCTF processing. The synthesized frame is output to the outside, and when a higher layer frame is required, the synthesized frame is stored in the image holding unit 350 for later processing.

  Each time the image composition unit performs composition processing, it is possible to reproduce a moving image having a high frame rate, and finally, a moving image having the same frame rate as the input image can be obtained.

  As described above, according to the encoding apparatus 100 of the present embodiment, when encoding a motion vector, the motion vector information is encoded by using a motion vector with appropriate accuracy for each layer of temporal scalability. The amount can be reduced. In the hierarchical encoding of moving images, the amount of code of the motion vector itself increases, so it is necessary to efficiently encode the motion vector. However, according to the present embodiment, the amount of code of the entire moving image stream And the compression efficiency can be increased.

  In the present embodiment, attention is paid to the correlation between the MCTF hierarchy and the accuracy of the motion vector. In general, in a frame that has many high-frequency components and a strong correlation with the reference frame, the prediction error can be reduced by performing high-precision motion compensation with improved motion vector accuracy. For frames with weak correlation with the reference frame due to fast motion and frames with few high-frequency components, increasing the accuracy of motion compensation does not contribute to reducing the prediction error. Information is wasted. On the other hand, in the present embodiment, encoding is performed using the optimal motion vector accuracy for each layer, and the compression amount of moving images is improved by suppressing the amount of code of useless motion vectors that does not reduce the prediction error. Let

  In addition, since the accuracy of the motion vector is the same for each layer, for example, when the motion vector is changed for each macroblock, the amount of computation required for encoding increases although the code amount of the motion vector decreases. On the other hand, in the present embodiment, the code amount of the motion vector can be reduced without increasing the calculation amount.

  In particular, in encoding a moving image by temporal filtering according to the MCTF technique, a motion vector must be encoded in each layer, and the amount of code of motion vector information increases, so this embodiment is effective.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  In the above description, a motion vector in the case of performing MCTF processing by Haar wavelet transform that generates one low-frequency frame from two consecutive frames has been described as an example. However, the present invention provides one from five consecutive frames. The present invention can also be applied to a motion vector in the case of performing MCTF processing by 5/3 wavelet transform in which a low-frequency frame is generated and one high-frequency frame is generated from three consecutive frames.

  In the above description, the encoding device 100 and the decoding device 300 are H.264. H.264 / AVC is compliant with video coding and decoding, but the present invention can also be applied to other systems that perform hierarchical video coding and decoding with temporal scalability. .

  In the above description, encoding of a moving image having temporal scalability has been described. However, encoding of a motion vector according to the present invention is performed when hierarchical moving image encoding and decoding having spatial scalability are performed. It can also be applied to.

It is a block diagram of the encoding apparatus which concerns on embodiment. It is a figure which shows the production | generation method of a low-pass frame. It is a figure which shows the production | generation method of a high region frame. It is a block diagram of a MCTF processing part. It is a figure which shows the image and motion vector which are output in each hierarchy. It is a flowchart which shows the encoding method according to MCTF technique. It is a figure which shows a mode that the precision data of a motion vector are stored for every hierarchy. It is a table which shows an example of matching of the frame rate of a hierarchy, and motion vector accuracy. It is a block diagram of the decoding apparatus which concerns on embodiment.

Explanation of symbols

  10 image acquisition unit, 20 MCTF processing unit, 21 motion vector detection unit, 28 motion vector accuracy determination unit, 60 image holding unit, 70 motion vector holding unit, 80 image encoding unit, 90 motion vector encoding unit, 92 multiplexing Unit, 100 encoding device, 300 decoding device, 310 stream analysis unit, 320 image decoding unit, 330 motion vector decoding unit, 350 image holding unit, 360 motion vector holding unit, 370 image synthesizing unit.

Claims (7)

In an encoding method for generating encoded data having a plurality of layers having scalability from a moving image,
An encoding method characterized in that the accuracy of a motion vector used for motion compensation prediction in each layer can be changed for each layer. In an encoding method for recursively performing motion compensation time filtering on a moving image to obtain a plurality of layers having different frame rates,
An encoding method characterized in that the accuracy of a motion vector used for motion compensation prediction in each layer can be changed for each layer.   The encoding method according to claim 1, wherein the hierarchy and the accuracy of the motion vector are associated in advance, and the corresponding information is included in encoded data of a moving image.   The encoding according to claim 1 or 2, wherein the hierarchy and the accuracy of the motion vector are associated with each other for a certain number of pictures, and the correspondence information is included in the encoded data of the moving image. Method.   The encoding method according to claim 1 or 2, wherein the hierarchy and the accuracy of the motion vector are associated in advance, and the accuracy of the motion vector in each layer is determined according to the association.   6. The encoding method according to claim 1, wherein the accuracy of the motion vector changes step by step every time a hierarchy is passed.   The encoding method according to claim 2, wherein the accuracy of the motion vector is coarsened as the frame rate of the hierarchy is lowered.

Images