JP3663559B2 - Video coding method for real-time backward playback - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a moving image coding method, and more particularly to a moving image coding method capable of generating a backward reproduction bit stream in real time.
[0002]
For example, VOD (Video on Demand) is an application that records moving image data on a storage medium such as a hard disk, an MO (magneto-optical disk), or a CD-ROM, and accesses the medium directly or via a communication path. Has been developed.
[0003]
As a moving image encoding method in such a case, it is required to realize a moving image encoding method capable of generating a bitstream for backward reproduction in real time.
[0004]
[Prior art]
Since moving image data has an enormous amount of information, generally, a method of recording information on a recording medium by compressing the amount of information by high-efficiency encoding is employed. Typical examples of high-efficiency encoding methods in this case are MPEG-1 (ISO / IEC 11172) and MPEG-2 (ISO / IEC 13818). These encoding methods are inter-frame encoding (P image or B image) using motion compensation in which intra-frame encoding (I image) is periodically inserted.
[0005]
Here, an I picture (Intra coded picture) is an intra-frame coded picture, which is an picture that is coded using only information of the picture. P picture (Predictive coded picture) is a prediction coding picture, and means a picture coded from a past reference picture using motion compensation prediction. A B picture (Bidirectionally predictive coded picture) is a bidirectional predictive coded picture, and refers to an picture that is coded from a past reference picture and / or a future reference picture using motion compensated prediction.
[0006]
FIG. 8 shows a configuration example of a forward reproduction encoder, which is represented by MPEG-1 or MPEG-2, and is an interframe coding (P (Image or B image).
[0007]
In FIG. 8, 1 is a frame rearranger, 2 is a discrete cosine transformer (DCT), 3 is a quantizer (Q), 4 is a variable length encoder (VLC), and 5 is an inverse quantizer (IQ). , 6 is an inverse discrete cosine transformer (IDCT), 7 is a frame memory (or field memory) (FM), 8 is a motion detector (ME), 9 is a motion compensator (variable delay) (MC), and 10 is a subtraction. And 11 is an adder.
[0008]
For the image input, the frames are rearranged in the encoding order in the frame rearrangement unit 1 having a frame memory of 1 GOP or more and a function of changing the frame order for each frame of 1 GOP. Here, GOP (Group of Picture) indicates an image group as a unit of processing, and an I image is arranged at the head. On the other hand, the frame memory 7 stores images encoded in the past. The motion detector 8 detects a motion vector by taking the difference between the rearranged image and the image in the frame memory 7.
[0009]
The motion compensator 9 performs motion compensation on the output image of the frame memory 7 using a motion vector, and generates a motion compensated reproduced image. The subtracter 10 obtains the difference between the rearranged image and the motion compensated reproduced image, performs the discrete cosine transform in the discrete cosine transformer 2, and performs the quantization in the quantizer 3. Further, the variable length encoder 4 encodes the quantization result into a variable length code together with the motion vector detected by the motion detector 8 and outputs it as a bit stream.
[0010]
On the other hand, the quantization result is inversely quantized by the inverse quantizer 5 and the result of inverse discrete cosine transform by the inverse discrete cosine transformer 6 is added to the motion compensated reproduced image by the adder 11. A locally decoded image is generated and stored in the frame memory 7. Two frame memories 7 are provided corresponding to the I image and the P image.
[0011]
FIG. 9 illustrates the relationship between reference images used in predictive coding, and illustrates a prediction reference screen for I images, P images, and B images. As shown in the figure, 1 GOP is composed of 9 frames, and I, P, and B image frames are arranged in sequence as I, B, B, B, P, B, B, B, P. It shall be encoded. That is, in this case, 1 GOP is encoded in about 0.3 seconds.
[0012]
In this case, the I image is first intra-frame encoded, and then the P image is encoded by (one-way) predictive encoding. Next, the B image between the I image and the P image is sequentially encoded by bidirectional predictive encoding. Further, the next P image is encoded by predictive encoding from the P image. Next, the B image between the P image and the P image is sequentially encoded by bidirectional predictive encoding.
[0013]
10A and 10B illustrate encoding for generating a forward reproduction bitstream, where FIG. 10A shows the order of input images, and FIG. 10B shows the order of encoding and the output order of bitstreams. . In (a) and (b), the upper numbers indicate the frame numbers in the input order, and the lower I, P, and B indicate the distinction between the I image, the P image, and the B image, respectively. In addition, / indicates a GOP break.
[0014]
As shown in FIG. 10A, the order of the input image at the point A in FIG. 8 includes two frames of B images between the I image and the P image and between the P image and the P image. When this is encoded, it is encoded at the point C in FIG. 8 in the order shown in FIG. 10B, and is output as a bit stream to the point D in FIG. In this case, the I image is first encoded, and then the B image and the next B image are encoded using the I image and the P image.
[0015]
Next, the P image is encoded using the I image, and the B image and the next B image are encoded using the I image and the P image. Further, the next P image is encoded using the P image, and the B image and the next B image are encoded using the two P images.
[0016]
In this way, encoding for generating a normal forward reproduction bitstream can be performed. The output bit stream is output in the order of I, B, B, P, B, B, P, B, B for each GOP. In a later-stage decoder (not shown), when a bit stream arranged in such an order is received, decoding can be performed and the original image can be reproduced in the forward direction.
[0017]
[Problems to be solved by the invention]
On the other hand, from the user's point of view, there is a demand not only for forward playback but also for backward playback as in the case of VCR (VTR). However, in general, in order to perform backward reproduction from a bit stream for forward reproduction, it is necessary to have a frame memory for a number of frames on the receiving side (decoding side). Will become expensive. Therefore, it is necessary to generate a reverse bit stream on the transmission side (encoder).
[0018]
Conventionally, on the transmission side, for example, in order to generate a bitstream for backward playback of a program for one hour, the program is once recorded as a forward image using a VCR or the like, and a frame is created by editing the VCR. In this case, the rearranged order must be rearranged, and the rearranged frames must be input to the image encoder to create a bit stream for reverse playback. For this reason, it is impossible to create a bit stream for reverse playback unless editing work is indispensable and one program has been recorded.
[0019]
On the other hand, there is a demand to generate data for reverse playback in real time. For example, when broadcasting live programs such as sports on a hard disk, There is a case where it is desired to perform backward reproduction. In this case, it is necessary to create data for backward reproduction in real time. However, it is impossible to generate a bit stream in real time with the above-described method in which encoding is performed in the reverse direction after editing with the VCR.
[0020]
The present invention is intended to solve such a problem of the prior art, and in an image encoding system that generates a normal reproduction bitstream in real time in a moving image encoding system, It is an object of the present invention to provide a real-time backward reproduction moving image coding method capable of generating a reverse reproduction bitstream in real time by adding a few functions.
[0021]
[Means for Solving the Problems]
In the present invention, for example, a frame memory of 1 GOP (Group of Picture) or more representing an image group and a function of changing the frame order are provided in the previous stage of the image encoder. The frames whose order has been changed in this way are input to the image encoder. In the image encoder, a closed GOP (in the order of input, in the encoding of the B image arranged immediately before the I image, only the I image is used as the reference image, and the I image or P immediately before the B image is used. An image is a GOP when not used as a reference image), and a bit stream for reverse reproduction is generated in real time by rearranging the output bit stream in reverse order of output in units of GOP. (Aspect 1 of the present invention).
[0022]
In this case, an n-times bit stream can be generated by inputting an image every n frames (aspect 2 of the present invention).
[0023]
At this time, if the B image is not used for encoding, the configuration of the image encoder can be simplified (aspect 3 of the present invention).
[0024]
In contrast to the scheme of aspect 1, a frame memory for backward encoding is provided in the encoding loop of the image encoder, and the I image is accumulated, thereby being closed.
Even if encoding is performed without limitation of GOP, a bit stream for reverse reproduction can be generated (aspect 4 of the present invention).
[0025]
In the mode 4, an n-times bit stream can be generated by inputting an image every n frames (aspect 5 of the present invention).
[0026]
Hereinafter, specific means for solving the problems of the present invention will be described.
[0027]
(1) I-frame coding for performing intra-frame coding periodically in the order of input on framed image data, and P-picture coding for predictive coding from a past reference image in the middle of this I-picture coding; The input frames are rearranged and encoded so that the B image encoding that performs bidirectional predictive encoding from the past reference image and the future reference image is performed in a predetermined order, and is performed first after the I image encoding. B image encoding is performed using only this I image as a reference image, and the encoding result is sequentially encoded in a variable length to output a bitstream. A reverse frame rearranging unit 12 is provided for accumulating frames equal to or more than GOP indicating the number of frames in the image encoding cycle and switching the order of input frames in the reverse direction for each GOP. In addition, a bit stream rearranging unit 13 that sequentially accumulates and accumulates the bit stream output of the image encoder 20 for each GOP and stores it in the memory is provided at the subsequent stage of the image encoder 20, and the contents of this memory are stored from the latest GOP. By playing back sequentially, the backward reproduction bit stream is output in real time.
[0028]
(2) In the case of (1), by inputting image data at n (n is an arbitrary integer) frame interval, an n-times speed reverse reproduction bit stream is output.
[0029]
(3) In the case of (1) or (2), by performing encoding only by I image encoding and P image encoding, a reverse reproduction bit stream or an n-times reverse reproduction bit stream is output. .
[0030]
(4) For framed image data, I image coding is performed for intra-frame coding periodically in the order of input, and P image coding for predictive coding from past reference images in the middle of this I image coding; B image coding for bi-directional predictive coding from past reference images and future reference images is performed in a predetermined order, and P image coding is performed immediately after I image coding, and then B image coding is performed. The input frames are rearranged and encoded so as to perform B image encoding using the I image of the previous I image encoding cycle held in the memory 7B at the end of the I image encoding cycle as one reference image. For the image encoder 20 that sequentially encodes the encoding results and outputs a bit stream, frames equal to or more than the GOP indicating the number of frames in the I image encoding period are stored in the previous stage of the image encoder 20. The reverse frame rearrangement unit 12 is provided for multiplying the input frames in the reverse direction for each GOP, and the order of the frames in the bitstream output of the image encoder 20 is provided at the subsequent stage of the image encoder 20. After rearranging in a predetermined order, a bit stream rearranging unit 13 is provided that sequentially accumulates each GOP and stores it in the memory, and reproduces the contents of this memory sequentially from the latest GOP. Output reverse-direction playback bitstream in real time.
[0031]
(5) In the case of (4), by inputting image data at n (n is an arbitrary integer) frame interval, an n-times speed reverse reproduction bit stream is output.
[0032]
(6) In the case of (1) to (5), an image field is used instead of the framed image data, and processing for each field is performed instead of processing for each frame.
[0033]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows an embodiment (1) of the present invention, and shows a reverse reproduction encoder according to aspect 1 of the present invention. The same forward reproduction encoder as in FIG. 8 is shown by the image encoder 20, 12 is a reverse frame rearranger, and 13 is a bitstream rearranger.
[0034]
The major difference between the embodiment shown in FIG. 1 and the conventional example shown in FIG. 8 is that it has a reverse-order frame rearranger 12 and a bitstream rearranger 13, and the encoding is a Closed GOP. Is to process. The reverse order frame rearrangement unit 12 has a function of accumulating frames that are equal to or more than 1 GOP (number of frames from the I image to the next I image) of the image data input in the forward direction and switching the frame order to the reverse order. And is provided on the input side of the image encoder 20.
[0035]
On the other hand, the bitstream rearranging unit 13 sequentially accumulates the bitstreams output from the image encoder 20 while folding each GOP and stores the bitstreams in the memory.
[0036]
FIG. 2 shows the encoding of the reverse reproduction bit stream generation in the case of the embodiment (1), where (a) is the order of the input images, and (b) is from the reverse frame rearranger. The output order, (c) shows the encoding order and the bitstream output order, respectively. In (a), (b), and (c), the upper number indicates the frame number in the input order, and the lower I, P, and B indicate the distinction between the I image, the P image, and the B image, respectively. In addition, / indicates a GOP break.
[0037]
By rearranging the frame order of the input image shown in FIG. 2A at point A in FIG. 1 in reverse order every GOP by the reverse frame rearranging unit 12, at point B in FIG. The output order shown in) is obtained. This is rearranged by the frame rearranger 1 in the same encoding order as in the prior art, and the frames in the order shown in FIG. 2 (c) are output to the point C in FIG. This is further encoded and output as a bit stream to point D in FIG.
[0038]
The encoding in this case is performed as follows. For example, 16I is intra-frame encoded, then 18B and 17B are encoded from 16I using motion compensated prediction (due to the Closed GOP limitation), then 13P is encoded from 16I using motion compensated prediction, and then 16I And 13P are bi-directionally predictively encoded using motion compensated prediction, then 10P is encoded using motion compensated prediction from 13P, and then 12B and 13P are encoded using motion compensated prediction. 11B is bi-directional predictively encoded.
[0039]
The encoded bit stream is accumulated and stored in a memory (not shown) by the bit stream rearranging unit 13 while being folded back for each GOP. That is, as shown in FIG. 2 (c), a bit stream of 7 frames to 2 frames corresponding to the bit stream of the first GOP is stacked at the bottom, and a bit stream of 16 frames to 11 frames is stacked thereon, Further, output data is created in such a manner that bit streams from 25 frames to 20 frames are stacked thereon. By reading the created data sequentially from the top in order from the latest GOP, a bit stream for backward reproduction can be obtained in real time.
[0040]
The embodiment (2) of the present invention corresponds to the aspect (2) of the present invention. In the configuration of FIG. 1, by inputting an image every n (n is an arbitrary integer) frame, the reverse reproduction at the n-times speed is performed. A bitstream can be generated.
[0041]
FIG. 3 shows an embodiment (3) of the present invention and shows a backward reproduction encoder according to aspect 3 of the present invention. The same elements as those in FIG. 1 are denoted by the same numbers, and the frame rearranger 1 is omitted. The frame memory 7A is composed of one surface.
[0042]
In the backward reproduction encoder shown in FIG. 1, encoding is performed using a B image, and therefore a frame rearranger 1 is required. If a B image is not used, a frame is used. Therefore, the frame rearranging unit 1 is not necessary, and the reverse frame rearranging unit 12 only needs to perform reverse rearrangement in units of GOPs. Further, since prediction is not performed from both directions, it is sufficient to have only one surface as the frame memory 7A.
[0043]
FIG. 4 shows coding for generating a backward reproduction bit stream in the case of the embodiment (3), where (a) is the order of the input images, and (b) is the coding order and the bit stream. The output order is shown respectively. In (a) and (b), the upper numbers indicate the frame numbers in the input order, and the lower I and P indicate the distinction between the I image and the P image, respectively. In addition, / indicates a GOP break.
[0044]
The input order in this case is as shown in FIG. 3 by rearranging the frame order of the input image shown in FIG. 4A at point A in FIG. At the point, the output order shown in FIG. 4 (b) is obtained. This is encoded and output as a bit stream to point D in FIG.
[0045]
In this case, for example, the first 18I is subjected to intra-frame encoding, and then the motion compensation prediction is performed by sequentially using 17P to 10P and the previous frame as a past reference image.
[0046]
The encoded bit stream is sequentially stacked and stored in a memory (not shown) by the bit stream rearranging unit 13 while being folded back for each GOP. That is, as shown in FIG. 4 (b), a bit stream of 9 frames to 1 frame corresponding to the bit stream of the first GOP is stacked at the bottom, and a bit stream of 18 frames to 10 frames is stacked thereon, Furthermore, output data is created in such a manner that bit streams from 27 frames to 19 frames are stacked thereon. By reading the created data sequentially from the top in order from the latest GOP, a bit stream for backward reproduction can be obtained in real time.
[0047]
FIG. 5 shows an embodiment (4) of the present invention and shows a backward reproduction encoder according to aspect 4 of the present invention. The same components as those in FIG. 1 are denoted by the same numbers, and the frame memory 7B is composed of four surfaces.
[0048]
The major difference between the embodiment shown in FIG. 5 and the embodiment shown in FIG. 1 is that the frame memory has two extra frames in the encoding loop. The I image for encoding for backward reproduction is held. In the case of the embodiment (4), there is no restriction that it must be encoded with a Closed GOP as in the case of the embodiment (1) or (2). Further, the rearrangement method in the reverse order frame rearranger and the bitstream rearranger is also different from those in the embodiments (1) and (2).
[0049]
FIG. 6 shows the encoding of the reverse reproduction bit stream generation in the case of the embodiment (4), where (a) is the order of the input images, and (b) is from the reverse frame rearranger. The output order, (c) shows the encoding order and bitstream output order, and (d) shows the rearranged bitstream output order. In (a), (b), (c), (d), the upper number indicates the frame number in the input order, and the lower I, P, and B indicate the distinction between the I image, the P image, and the B image, respectively. ing. In (c), the third row shows the stored contents of one of the added frame memories, and the fourth row shows the stored contents of the other added frame memory. Further, / indicates a GOP break, and% indicates a break for explanation.
[0050]
By rearranging the frame order of the input image shown in FIG. 6A at point A in FIG. 5 for each segment by the reverse frame rearranging unit 12, at point B in FIG. The output order shown in FIG. This is rearranged by the frame rearranger 1 in the order of encoding, and the frame in the order shown in FIG. 6 (c) is output to the point C in FIG. Output as a bitstream. At this time, the encoding result of the I image is alternately stored in the frame memory and held until the end of the next break. Further, this is rearranged by the bitstream rearranger 13 in the same manner as in FIG. 2 (c), and is output for each GOP.
[0051]
The encoding in this case is performed as follows. For example, 16I is intra-frame encoded and stored in one side of the frame memory 7B, then 13P is encoded using motion compensated prediction from 16I, and then 15B and 14B are encoded using motion compensated prediction from 16P and 13P. Next, 10P is encoded from 13P using motion compensated prediction, then 12B and 11B are bidirectionally encoded from 13P and 10P using motion compensated prediction, and then 13P and the previous encoding are performed as frame memory. The bi-predictive coding of 9B and 8B is performed using motion compensated prediction from 7I stored in.
[0052]
The encoded bitstream is rearranged by the bitstream rearranger 13 as shown in FIG. 6 (d), and then sequentially stacked and stored in a memory (not shown) for each GOP. That is, as shown in FIG. 6 (d), a bit stream of 7 frames to 2 frames corresponding to the bit stream of the first GOP is stacked at the bottom, and a bit stream of 16 frames to 11 frames is stacked thereon, Further, output data is created in such a manner that bit streams from 25 frames to 20 frames are stacked thereon. By reading the created data sequentially from the top in order from the latest GOP, a bit stream for backward reproduction can be obtained in real time.
[0053]
The embodiment (5) of the present invention corresponds to the aspect (5) of the present invention. In the configuration of FIG. 5, by inputting an image every n (n is an arbitrary integer) frame, the reverse reproduction for n-times speed is performed. A bitstream can be generated.
[0054]
FIG. 7 shows an application example of the present invention, and shows a system configuration example for simultaneously generating a forward reproduction bit stream and a backward reproduction bit stream.
[0055]
In FIG. 7, reference numeral 100 denotes a forward reproduction encoder, which has, for example, the same configuration as the conventional forward reproduction encoder shown in FIG. Reference numeral 200 denotes a backward reproduction encoder, which has the same configuration as that of the backward reproduction encoder of the present invention as shown in FIGS.
[0056]
According to the configuration of FIG. 7, the forward reproduction bit stream and the backward reproduction bit stream can be simultaneously obtained in real time.
[0057]
In the above description, image processing is performed in units of frames. However, the present invention is not limited to this, and frames may be replaced with fields. Each embodiment has been described as having a hardware configuration, but the method of the present invention can also be applied to an encoder that performs processing only by software.
[0058]
In addition, when a plurality of I images are included in one GOP, the above GOP definition may be divided into an image group including one or more I images, and may not necessarily match the definition of MPEG or the like. .
[0059]
【The invention's effect】
As described above, according to the present invention, in the moving image coding system, real-time encoding is performed by adding some functions to the moving image encoder that generates a normal reproduction bit stream in real time. Thus, it is possible to realize a moving image encoder capable of generating a backward reproduction bit stream.
[Brief description of the drawings]
FIG. 1 is a diagram showing an embodiment (1) of the present invention.
FIGS. 2A and 2B are diagrams illustrating encoding for generating a backward reproduction bit stream in the case of the embodiment (1), where FIG. 2A is a sequence of input images, and FIG. 2B is an output from a reverse frame rearranger. The order, (c) shows the order of encoding and the output order of the bitstream, respectively.
FIG. 3 is a diagram showing an embodiment (3) of the present invention.
FIGS. 4A and 4B are diagrams illustrating encoding for generating a backward reproduction bit stream in the case of the embodiment (3), in which FIG. 4A is an input image order, and FIG. 4B is an encoding order and bit stream order; Each output order is shown.
FIG. 5 is a diagram showing an embodiment (4) of the present invention.
FIGS. 6A and 6B are diagrams illustrating encoding for generating a backward reproduction bit stream in the case of the embodiment (4), where FIG. 6A is an order of input images, and FIG. 6B is an output from a reverse frame rearranger. (C) shows the encoding order and bitstream output order, and (d) shows the rearranged bitstream output order.
FIG. 7 is a diagram showing an application example of the present invention.
FIG. 8 is a diagram illustrating a configuration example of a forward reproduction encoder.
FIG. 9 is a diagram for explaining a relationship between reference images used in predictive coding.
FIGS. 10A and 10B are diagrams illustrating encoding for generating a forward reproduction bitstream, where FIG. 10A shows the order of input images, and FIG. 10B shows the order of encoding and the output order of bitstreams;
[Explanation of symbols]
12 reverse order frame rearrangement unit 13 bit stream rearrangement unit 20 image encoder

Claims (6)

For framed image data, I image coding is performed to periodically perform intra-frame coding in the order of input, and P image coding for predictive coding from a past reference image in the middle of the I image coding, and past reference The B image first encoded after the I image encoding is performed by rearranging and encoding the input frames so that the B image encoding that performs bidirectional predictive encoding from the image and the future reference image is performed in a predetermined order. For an image encoder that performs encoding only with the I image as a reference image, sequentially encodes the encoding result, and outputs a bit stream.
A reverse frame rearranger is provided in the preceding stage of the image encoder to accumulate frames equal to or more than GOP indicating the number of frames in the I image encoding period and to change the order of input frames in the reverse direction for each GOP. ,
A bitstream rearranger that sequentially accumulates and accumulates the bitstream output of the image encoder for each GOP and stores it in a memory is provided at a subsequent stage of the image encoder.
A video encoding method for real-time reverse reproduction, wherein a bit stream for reverse reproduction is output in real time by reproducing the contents of the memory sequentially from the latest GOP. The moving image coding system for real-time backward reproduction according to claim 1, wherein an image data is input at n (n is an arbitrary integer) frame interval, thereby outputting an n-times reverse reproduction bitstream. A video encoding method for real-time reverse playback characterized by 3. The moving image coding system for real-time reverse reproduction according to claim 1 or 2, wherein the bit stream for reverse reproduction or the reverse reproduction of n-times speed is obtained by encoding only by I image encoding and P image encoding. A video encoding method for real-time backward reproduction, characterized in that a bit stream for video is output. For framed image data, I image coding is performed to periodically perform intra-frame coding in the order of input, and P image coding for predictive coding from a past reference image in the middle of the I image coding, and past reference B image coding that performs bidirectional predictive coding from an image and a future reference image is performed in a predetermined order, and P image coding is performed immediately after the I image coding, and then B image coding is performed. The input frames are rearranged and encoded so that the B image encoding is performed using the I image of the previous I image encoding cycle held in the memory at the end of the image encoding cycle as one reference image, and the encoding result is For an image encoder that sequentially outputs a bit stream by variable length encoding,
A reverse frame rearranger is provided in the preceding stage of the image encoder to accumulate frames equal to or more than GOP indicating the number of frames in the I image encoding period and to change the order of input frames in the reverse direction for each GOP. ,
Bits that are rearranged in the subsequent stage of the image encoder so that the order of the frames in the bitstream output of the image encoder is in the predetermined order, and then are sequentially stacked and accumulated in the memory while being folded back for each GOP A stream sorter,
A video encoding method for real-time reverse reproduction, wherein a bit stream for reverse reproduction is output in real time by reproducing the contents of the memory sequentially from the latest GOP. 5. The moving image coding system for real-time reverse reproduction according to claim 4, wherein an n-times reverse reproduction bitstream is output by inputting image data at n (n is an arbitrary integer) frame interval. A video encoding method for real-time reverse playback characterized by 6. The moving image coding system for real-time backward reproduction according to claim 1, wherein an image field is used instead of framed image data, and each field is replaced with a process for each frame. A video encoding method for real-time backward reproduction, characterized by performing processing.

Images