JP2007068218A - Method of video-encoding, method of video-decoding, video-encoding apparatus, video-decoding apparatus, video-encoding program, and video-decoding program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To appropriately control not only the reference frame, but also a frame that is waiting for the output in backward interframe prediction. <P>SOLUTION: A video processing system 100 is provided with a video-encoding apparatus 10 and a video-decoding apparatus 20. The video-encoding apparatus 10 performs backward interframe prediction, from a temporally subsequent frame and outputs information, indicating to the effect that an option to eliminate use of a decoded image of the temporally subsequent frame has been chosen. The video decoding apparatus 20 eliminates the need for the decoded image of the frame, on the basis of this information, in conjunction with input of the information for eliminating use of the decoded image of the temporally subsequent frame. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a moving image encoding method, a moving image decoding method, a moving image encoding device, a moving image decoding device, a moving image processing system, a moving image encoding program, and a moving image decoding program.

  2. Description of the Related Art Conventionally, a moving image signal encoding technique is used to transmit a moving image signal and to store and reproduce the moving image signal. Examples of such techniques include ITU-T Recommendation H.263 (hereinafter referred to as “H.263”) and ISO / IEC International Standard 14496-2 (MPEG-4 Visual, hereinafter referred to as “MPEG-4”). International standardized video coding schemes such as are known.

  Newer encoding schemes include video encoding schemes planned for joint international standardization between ITU-T and ISO / IEC, ITU-T Recommendation H.264, ISO / IEC International Standard 14496-10 ( Joint Final Committee Draft of Joint Video Specification (hereinafter referred to as “H.26L”) is known. For general encoding techniques used in these video encoding systems, refer to Non-Patent Document 1 shown below, for example.

  A moving image signal is composed of a sequence of images (frames) that change little by little over time. For this reason, in general, in these video encoding systems, inter-frame prediction is performed on a frame (current frame) input as an encoding target with another frame (reference frame). Reduce temporal redundancy in moving picture signals.

  In this case, the inter-frame prediction is performed between the reference frame and the reference frame having a smaller change from the current frame, so that the redundancy can be greatly reduced and the encoding efficiency can be increased. For this reason, as a reference frame, not only a temporally preceding frame but also a temporally subsequent frame may be used. The case where the previous frame is used is referred to as forward prediction, and the case where the subsequent frame is used is referred to as backward prediction (see FIG. 1). Moreover, the case where both predictions are arbitrarily selected at this time, or the case where it uses simultaneously is described as bidirectional | two-way prediction.

  In general, when such bi-directional prediction is used, one of the temporally preceding frames is a reference frame for forward prediction, and one of the temporally subsequent frames is a reference frame for backward prediction. Are stored in the frame buffer in advance prior to the current frame.

  For example, in MPEG-4 decoding, when a current frame is decoded by bidirectional inter-frame prediction, first, prior to decoding of the current frame, one of the frames temporally prior to the current frame and a time Therefore, one of the later frames is decoded as a frame decoded by intra-frame prediction without using inter-frame prediction or a frame decoded by forward inter-frame prediction, and these are stored in the frame buffer as reference frames. . Thereafter, the current frame is decoded by bidirectional prediction using these two retained frames (see FIG. 2A).

  Therefore, in this case, the order of decoding times of the reference frame and the current frame that are temporally later is reversed from the order of the output times of the respective decoded images. Note that information indicating the output time is associated with each frame, and the temporal order of each frame can be known according to this information. For this reason, the decoded images are output in the correct order (see FIG. 2B). In MPEG-4, the output time is described as an absolute value.

  In recent video coding schemes, in the inter-frame prediction described above, not only one reference frame in each of the forward and reverse directions is used so that prediction from a frame with a smaller change from the current frame is possible. There are some that can be used (see FIG. 3).

  For example, in decoding of H.26L, when a plurality of reference frames are held in a frame buffer within a range up to a predetermined upper limit of the number of reference frames, and inter-frame prediction is performed, it is optimal from among them. Anything is arbitrarily designated and used. In this case, when decoding the current frame as a bidirectional prediction frame, first, the reference frame is decoded prior to decoding of the current frame. A plurality of temporally subsequent frames are decoded and held in the frame buffer as reference frames. The current frame can be predicted by arbitrarily specifying a frame to be used for prediction from those frames (see FIG. 4A).

  Therefore, in this case, the order of decoding times of a plurality of reference frames that are temporally later and the current frame is reversed from the order of output times. Each of these frames is associated with information indicating the output time or information indicating the output order, and the temporal context of each frame can be known according to this information. Therefore, the decoded images are output in the correct order (see FIG. 4B). The output time is often described as an absolute value. The output order is used when the frame interval is constant.

  As described above, also in backward prediction, when a plurality of reference frames are used, the frame held in the frame buffer is not necessarily used for backward prediction in the current frame and thereafter. This example will be described with reference to the prediction structure shown in FIG. The current frame F1 is predicted backward from the temporally subsequent reference frame F2, F2 is predicted backward from F3, F3 is backward predicted from F4, and F4 is temporally predicted from the previous reference frame F0. It shall be. In such prediction, for example, the change between the previous reference frame F0 and the current frame F1 is large while the change between F1 and the subsequent reference frames F2, F3, and F4 is small. And it is performed when a change becomes comparatively small between F0 and F3, and an efficient prediction can be performed.

  At this time, since the current frame F1 is predicted only from the reference frame F2 that is temporally later, F3 and F4 are frames that are not used for inter-frame prediction at the time of decoding the current frame F1. However, since F3 and F4 are frames later in time than the current frame F1, it is necessary to keep them until they are output as decoded images at the respective output times.

  Thus, when holding a temporally subsequent frame in the frame buffer for backward prediction, such a frame is used as a reference frame in inter-frame prediction after the current frame, and is used as a reference frame. There are two types of things that cannot be done. Hereinafter, a frame that is not used as a reference frame in this way but has not reached the output time and is held in the frame buffer is referred to as an “output waiting frame”.

  In order to explain the difference between the frames, a schematic diagram of the configuration of the moving picture decoding apparatus is shown in FIGS. 6 (a) and 6 (b). As shown in FIG. 6A, the decoding apparatus 1 includes a frame buffer 3 that holds a reference frame, and outputs the reference frame to the decoding processing unit 2 when performing inter-frame prediction. At this time, when a plurality of reference frames are used in the backward prediction as described above, not only the reference frames but also the output waiting frames are held in the frame buffer, which is logically shown in FIG. As shown in FIG. 4, an area for storing a frame that is continuously held as a reference frame and is also output to the decoding processing unit 2, and a frame that is not output to the decoding processing unit 2 as an output waiting frame. Thus, there is an area for storing a frame that is held only until it is output as a decoded image at the output time.

  By the way, when a plurality of reference frames are used, for example, when there is a frame having a unique feature in a certain moving image and a large change from another frame, this is held as a reference frame. However, efficient prediction cannot be expected. Therefore, there is a case where inter-frame prediction can be performed more efficiently if it is possible to hold other frames in the corresponding frame buffer without holding such frames as reference frames. Or, conversely, a frame having a representative characteristic in a certain moving image and having a small change from another frame is held in the frame buffer for a long time as a reference frame regardless of the temporal distance from the current frame. Therefore, it can be expected that inter-frame prediction can be efficiently performed in many frames.

  In order to realize such a reference frame sorting operation, it is conceivable to notify reference frame sorting information with encoded data. For example, in H.26L, a memory management control (MMCO) command is defined. In the MMCO command, for example, a Reset command that can instruct all reference frames held in the frame buffer as unnecessary is defined, and a frame that is arbitrarily held as a reference frame in the frame buffer as necessary. Can be ordered.

  In addition, in order to perform random access on a moving image, when attempting to start decoding from the middle of encoded data, the frame to start decoding is a frame that does not use inter-frame prediction from other frames. A frame encoded by prediction, and frames subsequent to that frame are also not stored in the frame buffer prior to the decoding of the frame that starts decoding. All the reference frames that have been specified must be indicated as unnecessary.

For example, in H.26L, an Instantaneous Decoder Refresh (IDR) picture is defined to clearly indicate such a state. In the IDR picture, all the reference frames before the IDR picture is decoded are unnecessary, and the inter-frame prediction in the subsequent frames is not performed from the frames before the IDR picture. As a result, when decoding is performed in the middle of encoded data, such as random access, decoding can be performed from the IDR picture without causing the problem of the presence of reference frames in inter-frame prediction.
International standard video coding basic technology (Fumitaka Ono, Yusuke Watanabe, Corona, published on March 20, 1998)

  As described above, when a plurality of reference frames are used in backward prediction inter-frame prediction, a plurality of reference frames can be efficiently handled by providing means for controlling the holding of the reference frames in the frame buffer. On the other hand, in such a frame buffer, there are not only reference frames used for inter-frame prediction but also output waiting frames held until the output time, but in the conventional means for controlling the holding of reference frames However, the control of the output waiting frame is not taken into consideration. Therefore, there is a problem that an output waiting frame that is not scheduled to be output occupies the frame buffer for a long time.

For this reason, when the control of the output waiting frame cannot be performed appropriately, there may occur a problem that the output of the decoded image cannot be handled uniquely.
For example, it is assumed that switching from the moving image composed of the encoded data F00 to F04 in FIG. 7A to the moving image composed of the encoded data F10 to F13 in FIG. This is because when the encoded data stored for editing a moving image is switched in units of frames, or in broadcast-type moving image communication, from any encoded data corresponding to one channel to another channel This may occur when switching to other encoded data corresponding to.

  In this case, F10 needs to be an IDR picture, so that all the reference frames held in the frame buffer prior to decoding of F10 are unnecessary, and decoding of frames after F10 can be performed without any problem. However, since it is a reference frame that is unnecessary in this case, if F03 and F04 are not reference frames but output waiting frames, the handling is not unique. In other words, when switching to F10 is performed, whether F03 and F04 are frames that are to be continuously held and output in the frame buffer, or are frames that should be handled as unnecessary prior to decoding of F10. Not clear. Therefore, which frame is output as a decoded image in this case is not unique.

  Therefore, an object of the present invention is to clearly define the handling of frames held in the frame buffer when using a plurality of reference frames in backward inter-frame prediction so that not only the reference frames but also the output waiting frames are properly defined. Is to control.

  In order to solve the above problems, a video encoding method according to the present invention is a video encoding method in which a video encoding device performs backward inter-frame prediction from a temporally subsequent frame, Information indicating that it is selected that the decoded image of the later frame in time is unnecessary is output.

In the moving image encoding method according to the present invention, preferably, the information is information for giving an instruction to make all frames unnecessary for the decoded image of the temporally subsequent frame.
Preferably, in the moving image encoding method according to the present invention, the information includes an instruction that eliminates all of the frames that are not used as reference frames for backward prediction with respect to the decoded image of the temporally subsequent frame. Information to do.
In the moving image encoding method according to the present invention, preferably, the information includes information indicating an output time related to a decoded image of a frame that is unnecessary.

  The moving image decoding method according to the present invention is a moving image decoding method in which the moving image decoding apparatus performs backward inter-frame prediction from a temporally subsequent frame, and the decoded image of the temporally subsequent frame is Along with the input of information for making it unnecessary, the decoded image of the frame is not output based on this information.

In the moving picture decoding method according to the present invention, preferably, the information is information for instructing all frames to be unnecessary regarding the decoded picture of the temporally subsequent frame.
In the moving picture decoding method according to the present invention, preferably, the information instructs to make all frames that are not used as reference frames for backward prediction unnecessary for the decoded picture of the temporally subsequent frame. Information.
In the moving image decoding method according to the present invention, preferably, the information includes information indicating an output time relating to a decoded image of a frame that is unnecessary.

  The moving picture coding apparatus according to the present invention is a moving picture coding apparatus that performs backward inter-frame prediction from a temporally subsequent frame, and does not require a decoded image of the temporally subsequent frame. Outputs information indicating that is selected.

In the video encoding device according to the present invention, preferably, the information is information for instructing all frames to be unnecessary with respect to the decoded image of the temporally subsequent frame.
Preferably, in the moving picture encoding device according to the present invention, the information is an instruction to eliminate all frames that are not used as reference frames for backward prediction with respect to a decoded image of the temporally subsequent frame. Information to do.
In the video encoding apparatus according to the present invention, preferably, the information includes information indicating an output time related to a decoded image of an unnecessary frame.

  The moving picture decoding apparatus according to the present invention is a moving picture decoding apparatus that performs reverse inter-frame prediction from a temporally subsequent frame, and information for making the decoded image of the temporally subsequent frame unnecessary Is input, the decoded image of the frame is not output based on this information.

In the moving picture decoding apparatus according to the present invention, preferably, the information is information for instructing all frames to be unnecessary with respect to the decoded picture of the temporally subsequent frame.
In the moving picture decoding apparatus according to the present invention, preferably, the information instructs to make all frames that are not used as reference frames for backward prediction unnecessary for the decoded picture of the temporally subsequent frame. Information.
In the moving image decoding apparatus according to the present invention, preferably, the information includes information indicating an output time related to a decoded image of an unnecessary frame.

  The moving picture coding program according to the present invention eliminates the need for the decoded picture of the temporally subsequent frame in the moving picture coding apparatus as a computer that performs backward inter-frame prediction from the temporally subsequent frame. This realizes a function of outputting information indicating that is selected.

In the moving picture coding program according to the present invention, preferably, the information is information for giving an instruction to make all frames unnecessary for the decoded image of the temporally subsequent frame.
In the moving picture encoding program according to the present invention, preferably, the information is an instruction to eliminate all frames that are not used as reference frames for backward prediction with respect to the decoded image of the temporally subsequent frame. Information to do.
In the moving image encoding program according to the present invention, preferably, the information includes information indicating an output time related to a decoded image of an unnecessary frame.

  A moving picture decoding program according to the present invention eliminates the need for a decoded picture of a temporally subsequent frame in a moving picture decoding apparatus as a computer that performs backward inter-frame prediction from a temporally subsequent frame. Along with the input of information, the function of not outputting the decoded image of the frame is realized based on this information.

In the moving picture decoding program according to the present invention, preferably, the information is information for instructing all frames to be unnecessary with respect to the decoded picture of the temporally subsequent frame.
In the moving picture decoding program according to the present invention, preferably, the information instructs to make all frames that are not used as reference frames for backward prediction unnecessary for the decoded picture of the temporally subsequent frame. Information.
In the moving image decoding program according to the present invention, preferably, the information includes information indicating an output time related to a decoded image of a frame that is unnecessary.

  According to these inventions, in encoding, information indicating that a frame waiting for output of a decoded image is unnecessary is output, and in decoding, this information is input, and all output waiting frames are not output. As a result, in the decoding of the encoded data after editing or in the decoding after random access on the encoded data, unnecessary decoded image output occurs, or conversely, the necessary decoded image output is deleted. To prevent that. As a result, an appropriate decoded image output can be obtained.

  The video encoding device includes the above-described video encoding device and the video decoding device described above, and the video decoding device constructs a video processing system that decodes the data encoded by the video encoding device. However, the same effect can be obtained.

  The moving image encoding apparatus according to the present invention is encoded by an input unit that inputs an image to be encoded, an encoding unit that encodes the image and generates encoded data, and the encoding unit. An image storage means for storing the reproduced image; and a buffer management means for managing the image stored in the image storage means. The buffer management means is encoded without referring to another image. When an image (IDR image) is encoded, a flag (no_output_of_prior_pics_flag) indicating whether or not all the images already stored in the image storage means become unnecessary is output together with the encoded data. For example, the moving image encoding apparatus outputs a flag “1” together with the encoded data when all the images are unnecessary, and a flag “0” otherwise.

  Preferably, in the moving image encoding apparatus according to the present invention, the encoding unit performs backward inter-frame prediction from a temporally subsequent frame, and the buffer management unit encodes without referring to another image. When an image to be converted (IDR image) is encoded, the decoded image of the temporally subsequent frame already stored in the image storage means is deleted.

  The moving image decoding apparatus according to the present invention includes an input means for inputting image data including encoded data of an encoded image and an image output instruction flag added to the encoded data, and the encoded data Decoding means for decoding and generating a reproduction image, image storage means for storing the reproduction image, and buffer management means for managing the reproduction image stored in the image storage means, the buffer management means, All images stored in the image storage unit are deleted according to the image output instruction flag corresponding to an image (IDR image) encoded without referring to the image stored in the image storage unit.

  Preferably, in the moving image encoding apparatus according to the present invention, when the image output instruction flag is “0”, all the reference images in the buffer are unnecessary, and when the image output instruction flag is “1”, The reference image and the output waiting image are deleted.

  The moving image encoding method according to the present invention includes an input step in which a moving image encoding device inputs an image to be encoded, and the moving image encoding device encodes the image, and stores encoded data. An encoding step to generate; an image storage step in which the moving image encoding device stores an image encoded and reproduced in the encoding step in an image storage means; and the moving image encoding device A buffer management step for managing an image stored in the image storage means. In the buffer management step, the moving image encoding device encodes an image to be encoded without referring to another image. At this time, a flag indicating whether or not all the images already stored in the image storage means become unnecessary is output together with the encoded data.

  Preferably, in the video encoding method according to the present invention, in the encoding step, the video encoding device performs backward inter-frame prediction from a temporally subsequent frame, and in the buffer management step, the buffer management step When encoding an image to be encoded without referring to another image, the moving image encoding device deletes a decoded image of a temporally subsequent frame already stored in the image storage unit.

  In the moving picture decoding method according to the present invention, the moving picture decoding apparatus inputs image data including encoded data of an encoded image and an image output instruction flag added to the encoded data; A decoding step in which the moving image decoding apparatus decodes the encoded data to generate a reproduction image; an image storage step in which the moving image decoding apparatus stores the reproduction image in an image storage unit; and the moving image And a buffer management step for managing a playback image stored in the image storage means. In the buffer management step, the moving picture decoding device refers to an image stored in the image storage means. All the images stored in the image storage means are erased according to the image output instruction flag corresponding to the image to be encoded.

  Preferably, in the video decoding method according to the present invention, when the video output instruction flag is “0”, the video decoding apparatus makes all the reference images in the buffer unnecessary, and when the flag is “1”. Deletes all the reference images and output-waiting images in the buffer.

  A moving image encoding program according to the present invention includes a function of inputting an image to be encoded, a function of encoding the image and generating encoded data, and storing the encoded and reproduced image as an image. A function stored in the image storage unit, a function for managing the image stored in the image storage unit, and an image stored in the image storage unit when encoding an image encoded without referring to another image. The moving picture encoding apparatus realizes a function of outputting a flag indicating whether or not all the images that have been made unnecessary together with the encoded data.

  Preferably, in the moving image encoding program, the function of performing backward inter-frame prediction from a temporally subsequent frame and the image storage when encoding an image to be encoded without referring to another image And a function of erasing a decoded image of a temporally subsequent frame already stored in the means.

  A moving image decoding program according to the present invention includes a function of inputting image data including encoded data of an encoded image and an image output instruction flag added to the encoded data, and decoding the encoded data A function for generating a reproduction image, a function for storing the reproduction image in the image storage means, a function for managing the reproduction image stored in the image storage means, and an image stored in the image storage means In accordance with the image output instruction flag corresponding to the image to be encoded without referring to the image, the moving image decoding apparatus is realized with a function of erasing all the images stored in the image storage means.

  Preferably, in the moving picture decoding program according to the present invention, when the image output instruction flag is “0”, all the reference images in the buffer are unnecessary, and when it is “1”, all the reference images in the buffer are not used. A function of deleting the reference image and the output waiting image is further realized.

  According to the present invention, when a plurality of reference frames are used in backward inter-frame prediction, the handling of the frames held in the frame buffer is clearly defined, and not only the reference frames but also the output waiting frames are appropriately controlled. be able to. Further, since the output waiting frame that has not been output is deleted from the frame buffer, the frame buffer can be used without waste.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
As a premise for explanation, encoding and decoding shall be realized based on H.26L, and parts not particularly referred to for operation in moving picture encoding shall be in accordance with the operation of H.26L. .

  First, the configuration of the moving image processing system according to the present invention will be described. FIG. 8 is a block diagram illustrating a functional configuration of the moving image processing system 100. As shown in FIG. 8, the moving image processing system 100 includes a moving image encoding device 10 and a moving image decoding device 20. The moving image encoding device 10 and the moving image decoding device 20 perform backward inter-frame prediction from a temporally subsequent frame.

Although detailed processing contents will be described later, the moving image encoding apparatus 10 includes a selection information output unit 11, and the selection information output unit 11 has been selected to eliminate the need for a decoded image of a later frame in time. Information indicating that is output to the video decoding device 20.
The moving image decoding apparatus 20 includes a decoded image disabling unit 21, and the decoded image disabling unit 21 receives information from the moving image encoding apparatus 10 for making a decoded image of a later frame unnecessary. Accordingly, the decoded image of the frame is not output based on this information.

  The information is, for example, information for instructing all frames to be unnecessary with respect to a decoded image of a later frame. Further, for example, this information is information for instructing that all frames that are not used as reference frames for backward prediction are unnecessary with respect to a decoded image of a temporally subsequent frame. Further, for example, this information includes information indicating an output time regarding a decoded image of a frame that is unnecessary.

  Next, the video encoding method and video decoding method according to the present invention will be described. In this embodiment, information that can be instructed to be unnecessary for a frame held in the frame buffer is defined. At this time, the reference frame and the output waiting frame are clearly defined.

  Here, it is defined in NAL unitsyntax that notifies information about NAL (Network Abstraction Layer), which is a unit for transmitting encoded data in H.26L. Specifically, it is defined as the type notified by the NAL unit type included therein.

  Here, for the conventional IDR picture, it is instructed that only all reference frames held in the frame buffer are unnecessary. In other words, the output waiting frame held in the frame buffer is defined as being unnecessary depending on the IDR picture.

  Next, an Instant Buffer Refresh picture (hereinafter referred to as “IBR picture”) is defined as a new picture different from the IDR picture. An IBR picture is defined to indicate that all output-waiting frames are not required in addition to the IDR picture definition that indicates that only all reference frames held in the frame buffer are not required. . That is, if the current frame is an IBR picture, all reference frames held in the frame buffer and all output waiting frames are unnecessary before decoding of the current frame.

  In encoding, the moving image encoding apparatus 10 instructs such an IBR picture in the following case. For example, when an IDR picture is to be designated as a random access point, the previous frame waiting for output is not retained regardless of the delay caused by backward prediction, and the frame is designated as an IBR picture. be able to.

  In decoding, if the current frame is an IBR picture, the moving picture decoding apparatus 20 eliminates all reference frames and output waiting frames included in the frame buffer prior to decoding of the current frame. Are not used for inter-frame prediction and decoded image output.

In an IBR picture, not all output waiting frames are required, but unnecessary output waiting frames may be limited and made unnecessary.
In view of this, it is possible to make an unnecessary determination using the decoded image output time associated with the IBR picture, and to make only the output waiting frame having an output time longer than this output time unnecessary.

  Alternatively, as a separate notification of the output time, only an output waiting frame having an output time larger than the notified time is made unnecessary, and an output waiting frame that becomes unnecessary is output as a decoded image associated with the frame. It may be specified by time.

  In this case, a syntax for notifying the output time is required. For example, in the NAL unit syntax, the syntax that exists only when the NAL unit type indicates an IBR picture is used. It may be a syntax for notifying time together with a picture. For example, “latest_output_time” to be notified following the NAL unit type when the NAL unit type indicates an IBR picture is defined. Here, latest_output_time indicates the maximum delay time in units of 90 kHz, assuming that the same unit as the time unit used for other syntaxes indicating time in H.26L is used. In addition, a numerical value represented by the time unit is encoded with a 32-bit unsigned fixed-length code and transmitted.

  In decoding, when the latest_output_time is notified, the video decoding device 20 does not need only a frame whose decoded image output time is longer than latest_output_time among the output waiting frames held in the frame buffer.

  In addition, when the latest_output_time is used as described above, a flag may be further notified before that, and a syntax in which the latest_output_time exists may be used only when there is an instruction by the flag. In this case, when latest_output_time is omitted, it can be defined that all output waiting frames are unnecessary.

  In this embodiment, the IBR picture is defined as a picture different from the IDR picture, but the difference between the two is the difference in the handling of the output waiting frame held in the frame buffer. Instead of defining a new picture, a flag (no_output_of_prior_pics_flag) accompanying the IDR picture that indicates the handling of an output waiting frame may be defined, and the same definition as the IBR picture may be given to the IDR picture.

  That is, as in the definition described above, in the IDR picture, it is instructed that all the reference frames held in the frame buffer are unnecessary, and the output waiting frame is defined as not unnecessary. At this time, the flag associated with this IDR picture is notified separately, and when there is an instruction by the flag (no_output_of_prior_pics_flag = 1), the frame in the IDR picture is the same as the above IBR picture. In addition to all the reference frames held in the buffer, it is instructed that all output waiting frames are unnecessary. By such notification, it is possible to give an instruction to make an output waiting frame unnecessary without defining a new picture.

This embodiment will be further described with reference to FIGS.
FIG. 10 is a block diagram showing a configuration of the moving picture coding apparatus 10 according to the present embodiment. A condition for encoding video is input from the input terminal 47. In general, an encoding condition is selected or input by a keyboard. Examples of the encoding condition include the size, frame rate, and bit rate of an image to be encoded. Furthermore, the prediction reference structure of the video and the capacity of the buffer 44 are mentioned.

  The predictive reference structure of video is, for example, the timing at which an input image is encoded as an IDR image, the image that is referred to by the predictive encoded image, and the like. The capacity of the buffer 44 is, for example, the number of images that are temporarily stored and used as reference images. These conditions may be set to change with time. The input encoding condition is stored in the controller 48.

  When the encoding process starts, the controller 48 outputs the encoding condition to the encoder 43, and the encoding condition is set. On the other hand, when an image to be encoded is input from the input terminal 41, the image is output to the encoder 43 via the frame memory 42 and then encoded. Since the order of the images is changed when the backward prediction is performed, the input image is temporarily stored in the frame memory 42.

  The encoder 43 is H.264. The image is encoded based on the 26L algorithm. The encoded image is output from the output terminal 46 after being multiplexed with other related information via the multiplexer 45. The image used for prediction is reproduced by the encoder 43 and then stored in the buffer 44 as a reference image for encoding the next image.

  FIG. 11 is a flowchart for explaining a process in which the controller 48 determines no_output_of_prior_pics_flag. This process is executed for each image constituting the video to be encoded. First, in S1, the encoding condition of the image to be encoded (whether it should be encoded as an IDR image) and whether the reproduced image already stored in the buffer 44 is used as a reference image or a display image. Information about is entered. Such information is separately managed by the controller 48.

  In S2, it is determined whether the image to be encoded is an IDR image. If the result of determination is that the image is not an IDR image (S2; NO), this process ends (S3). If it is an IDR image (S2; YES), the process proceeds to S4. In S4, it is determined whether or not an image already stored in the buffer 44 is used as a reference image or a display image.

  As a result of the determination, when it is used as a reference image or a display image (S4; YES), no_output_of_prior_pics_flag is set to “0” (S5). On the other hand, when it is not used as a reference image or a display image (S4; NO), no_output_of_prior_pics_flag is set to “1” (S6). The no_output_of_prior_pics_flag set in this way is output to the multiplexer 45 via the bus L8, added to the encoded data of the corresponding image, and sent from the output terminal 46 (S7). If no_output_of_prior_pics_flag is set to “1” in S6, the controller 48 outputs a command for erasing all the images stored in the buffer 44 via L7.

  FIG. 12 is a block diagram showing a configuration of the video decoding device 20 in the present embodiment. Data to be decoded is input from the input terminal 51. This data is image data of each frame encoded by the moving image encoding device 10. When this data is an IDR image, no_output_of_prior_pics_flag is multiplexed. The input data is stored in the input buffer 52. At a predetermined time, one frame of data is input from the input buffer 52 to the decoder 53 in accordance with an instruction from the controller 56. Decoding according to the 26L algorithm is started.

  The decoded image is temporarily stored in the output buffer 54. The image stored in the output buffer 54 is fed back to the decoder 53 via the bus M5 and then used as a reference image for decoding the next image. On the other hand, data of no_output_of_prior_pics_flag added to the IDR image decoded by the decoder 53 is input to the controller 56.

  The controller 56 controls the output buffer 54 with reference to the value of “no_output_of_prior_pics_flag” (“0” or “1”). That is, when no_output_of_prior_pics_flag is “1”, both the reference image and the display waiting image become unnecessary, and the controller 56 passes all the images stored in the output buffer 54 via the bus M8. Output a command to delete. On the other hand, when no_output_of_prior_pics_flag is “0”, since the image is an IDR image, the controller 56 makes all images (reference images) used for reference unnecessary and deletes them from the output buffer 54. . However, the display waiting image is not deleted. The display waiting image is displayed on the display at that time.

  Further, in the present embodiment, an example in which the present invention is realized based on H.26L has been described. However, a moving picture encoding method to which the present invention can be applied is not limited to H.26L, and a reverse frame. There are various video coding schemes using inter prediction.

  Furthermore, in this embodiment, the NAL unit type notified in the NAL unit syntax is added as a syntax for notifying that an output waiting frame is unnecessary, and the output time is notified. In this case, a syntax with a fixed-length code is added. Of course, the definition and syntax for notifying this are not limited to these. A variable-length code may be used instead of the fixed-length code for notifying the output time, and the unit of the output time is not 90 kHz. For example, in H.26L, video usability information (Video Usability Information, VUI), and may be based on a time unit defined by num_units_in_tick and time_scale.

  In addition to NAL unit syntax, notification can be made by various syntaxes that can notify information to be applied in units of frames. For example, in H.26L, a syntax may be added to a supplemental enhancement information message.

  Alternatively, a memory management control processing (MMCO) command defined for controlling the reference frame in H.26L may be extended. In this case, since the reset command defined as the MMCO command defines the same operation as the IDR picture in the above-mentioned NAL unit type, for example, as in the case of the definition in the NAL unit type, for example, an output waiting frame A reset (Reordering Buffer Reset) command can be defined. The definition of the operation by this command may be the same as the definition of the operation in the IBR picture in the NAL unit type.

  In the case of using another moving image encoding method, a syntax for notifying information applied to each frame in the encoding method can be used. Moreover, it is good also as notifying outside the encoding data by a moving image encoding system like ITU-T Recommendation H.245 utilized for the notification of control information in the communication using H.263.

Finally, with reference to FIG. 9, a moving picture encoding program and a moving picture decoding program according to the present invention will be described.
As shown in FIG. 9, the moving image processing program 310 is stored in a program storage area 30 a formed in the recording medium 30. The moving image processing program 310 can be executed by the moving image processing system 100 shown in FIG. 8, and includes a main module 311 that supervises moving image processing, a moving image encoding program 312 described later, and a moving image decoding program described later. 313.

  The moving image encoding program 312 includes a selection information output module 312a. The function realized by operating the selection information output module 312a is the same as the function of the selection information output unit 11 of the video encoding device 10.

  The moving image decoding program 313 includes a decoded image unnecessary module 313a. The functions realized by operating the decoded image unnecessary module 313a are the same as the functions of the decoded image unnecessary unit 21 of the moving image decoding apparatus 20, respectively.

  Further, the moving image processing program 310 may be configured such that a part or all of the moving image processing program 310 is received and recorded by another device via a transmission medium such as a communication line. Conversely, the moving image processing program 310 may be transmitted via a transmission medium and installed in another device.

It is a figure for demonstrating the forward direction prediction and reverse direction prediction in inter-frame prediction. FIG. 2A is a diagram illustrating an example in which the current frame is decoded by bidirectional prediction using two frames. FIG. 2B is a diagram illustrating an example in which the order of decoding times of a reference frame and a current frame that are later in time is reversed from the order of output times of the decoded images. It is a figure for demonstrating the forward direction prediction and reverse direction prediction in case multiple reference frames are used. FIG. 4A is a diagram illustrating an example in which inter-frame prediction of a plurality of reference frames is performed within a range up to a predetermined reference frame number upper limit. FIG. 4B is a diagram illustrating an example in which the order of decoding times of a plurality of reference frames that are temporally later and the current frame is reversed from the order of output times of the decoded images. It is a figure which shows the prediction structure in case the flame | frame hold | maintained at a frame buffer is not used for the reverse direction prediction after the present frame. FIG. 6A is a diagram schematically showing the configuration of the video decoding device. FIG. 6B is a diagram schematically showing the configuration of the frame buffer. FIG. 7A is a diagram illustrating an example of a moving image including encoded data before switching. FIG. 7B is a diagram showing a moving image after switching composed of encoded data. It is a block diagram which shows the functional structure of the moving image processing system which concerns on this invention. It is a figure which shows the structure of the moving image processing program which concerns on this invention. It is a block diagram which shows the structure of the moving image encoder which concerns on this invention. It is a flowchart for demonstrating the process in which a controller determines no_output_of_prior_pics_flag. It is a block diagram which shows the structure of the moving image decoding apparatus which concerns on this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Moving image encoding device, 11 ... Selection information output part, 20 ... Moving image decoding apparatus, 21 ... Decoded image unnecessary part, 310 ... Moving image processing program, 312 ... Selection information output module, 313 ... Decoded image unnecessary Module, 41, 47 ... input terminal, 42 ... frame memory, 43 ... encoder, 44 ... buffer, 45 ... multiplexer, 46 ... output terminal, 48 ... controller, 51 ... input terminal, 52 ... input buffer, 53 ... Decoder, 54 ... output buffer, 55 ... output terminal, 56 ... controller, 100 ... moving image processing system

Claims (12)

The video encoding device is a video encoding method for performing backward interframe prediction from a temporally subsequent frame,
A moving picture coding method characterized by outputting information indicating that it is selected that the decoded picture of the later frame is unnecessary. The video decoding device is a video decoding method in which backward inter-frame prediction from a temporally subsequent frame is performed,
A moving picture decoding method characterized by not outputting a decoded image of the frame based on this information when information for making a decoded image of a later frame unnecessary is input. A video encoding device that performs backward inter-frame prediction from a temporally subsequent frame,
A moving picture coding apparatus that outputs information indicating that it is selected that a decoded image of a later frame in time is unnecessary.   4. The moving picture encoding apparatus according to claim 3, wherein the information is information for instructing that all the frames are unnecessary with respect to the decoded image of the temporally subsequent frame.   4. The information according to claim 3, wherein the information is an instruction for instructing that all the frames that are not used as reference frames for backward prediction are unnecessary with respect to the decoded image of the temporally subsequent frame. The moving image encoding apparatus described.   4. The moving picture encoding apparatus according to claim 3, wherein the information includes information indicating an output time relating to a decoded image of a frame that is unnecessary. A video decoding device that performs backward inter-frame prediction from a temporally subsequent frame,
A moving picture decoding apparatus characterized by not outputting a decoded image of the frame on the basis of the input of information for making a decoded image of a later frame unnecessary.   8. The moving picture decoding apparatus according to claim 7, wherein the information is information for instructing that all frames are unnecessary with respect to the decoded image of the temporally subsequent frame.   8. The information according to claim 7, wherein the information is an instruction to make all the frames that are not used as reference frames for backward prediction unnecessary for the decoded image of the temporally subsequent frame. The moving picture decoding apparatus described.   The moving picture decoding apparatus according to claim 7, wherein the information includes information indicating an output time related to a decoded picture of a frame that is unnecessary. In a video encoding device as a computer that performs backward interframe prediction from a temporally subsequent frame,
A moving picture coding program that realizes a function of outputting information indicating that it is selected that the decoded picture of a later frame in time is unnecessary. In a video decoding device as a computer that performs backward inter-frame prediction from a later frame in time,
A moving picture decoding characterized by realizing a function of not outputting a decoded image of the frame based on this information in response to the input of information for making the decoded image of the later frame unnecessary program.

Images