JP3461280B2 - Moving image editing apparatus and moving image editing method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture editing apparatus and a moving picture editing method for re-editing a plurality of coded moving picture data into one moving picture data.

[0002]

2. Description of the Related Art The international standard MPEG2 (M
moving Picture Experts Gro
The digital moving picture coding technology represented by up phase 2) is currently attracting attention as a technology having a possibility of replacing the current analog moving picture media such as television broadcasting and video.

For moving picture compression in MPEG2, a method called interframe prediction, which removes redundancy in the time direction and reduces the amount of information, is adopted. An example of an MPEG2 encoding device is shown in FIG.

Removal of the redundancy in the time direction is executed by the motion detecting section 40 and the motion compensating section 39 shown in FIG. In the motion detecting unit 40, as shown in FIG. 8, a block (macro block) obtained by dividing the screen into 16 × 16 pixels from the image previously encoded stored in the reference image memory 41.
The motion between two frames is detected by searching for similar portions in units. This motion information (motion vector) is transmitted to the outside as a bit stream and is also output to the motion compensation unit 39.

In the motion compensator 39, a 16 × 16 pixel area similar to a macroblock to be encoded from a motion vector is read from the reference image memory 41 and used as a predicted image. The subtraction unit 31 obtains the difference between the macroblock to be encoded of the input image signal rearranged in the encoding screen order by the frame rearrangement unit 30 and the predicted image (obtains a residual signal), and As a result, the amount of information possessed by the relevant macroblock to be encoded is greatly suppressed. Next, information compression in the spatial direction is performed on the suppressed macroblock.

First, the discrete cosine transform (DCT) unit 32
The residual signal is mapped to the frequency space by the quantizer 33.
To quantize with a predetermined roughness. Next, this is subjected to variable length coding by the variable length coding unit (VLC) 34, whereby the originally existing moving image information is compressed to several tenths. This information is multiplexed with the above-described motion vector in the multiplexing unit 35 and output as a bit stream.

The output from the quantizer 33 is decoded by the inverse quantizer 36, the inverse DCT unit 37, and the adder 38, and then recorded in the reference image memory 41 as a reference image. The motion detection processing has the largest amount of processing in such processing. As a typical motion detection method, the pixel value of the macroblock to be encoded and the area within the search range is set to 0.5.
There is a method in which the sum of squares of differences in pixel values is calculated while shifting in pixel units, and the one having the smallest sum of squares is used as a motion vector. It is known that such a motion detection method generally requires a huge amount of calculation, and the amount of calculation per unit time required when performing encoding in real time is several hundred GOPS (Giga Operation).
It also becomes a per second). Therefore, special hardware such as a dedicated processor is required to perform real-time encoding processing.

On the other hand, with the recent improvement in the processing capability of personal computers (hereinafter referred to as PCs), digital moving image processing / editing processing has become possible on an individual level. In the processing and editing of moving images by a PC, special parts such as scene change, fade-in / fade-out, and cross-fade are extracted from a plurality of pre-encoded moving image data (bitstreams) by extracting a portion corresponding to a certain period. One bitstream (program) that connects to other bitstreams using effects
The work such as putting together is typical.

Regarding the editing of a moving image in MPEG2, for example, a method of connecting two moving image bit streams by a scene change as disclosed in Japanese Patent Laid-Open No. 8-251582 has been proposed. This is to connect the bit stream A and the bit stream B without overlapping in time, and by adjusting the buffer capacity of the virtual decoder for connection, the calculation amount required for connection does not matter.

[0010]

However, when attempting to realize crossfade, which is one of the special effects frequently used when joining two scenes,
It is necessary to newly generate a crossfade image from two different moving image bitstreams.

In this case, as shown in FIG. 9, the decoding process of the two moving image bit streams A and B by the decoding units 20 and 21, the synthesis process of the decoded images A and B by the synthesis unit 22, A process of re-encoding the cross-fade image obtained by combining by the encoding unit 23 is required.

As described above, since the motion detection processing is required for the encoding, the amount of calculation required for the re-encoding becomes enormous. Therefore, in a system that is configured only by a general-purpose processor without using additional hardware, this processing takes time, which causes deterioration of the user interface.

The present invention has been made in view of the above circumstances, and an object thereof is re-encoding when two types of moving picture bit streams are connected by crossfade without adding special hardware. An object of the present invention is to provide a moving image editing apparatus and a moving image editing method that reduce the processing time in the conversion processing and do not impair the operability of the user.

[0014]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention uses a crossfade to cross two types of first and second moving picture bit streams encoded using interframe prediction. A moving image editing apparatus having a function of partially overlapping and combining, the means comprising: decoding means for decoding each of the first and second bitstreams corresponding to a period of combining by the crossfade; Means for creating a crossfade image from the two moving images, and analyzing the first and second bitstreams during the period of combining with the crossfade,
Means for extracting respective coding parameters, code amount prediction means for predicting the code amount when the crossfade image is coded using the coding parameters for the first and second bit streams, and the prediction Selecting means for encoding one of the first and second bitstreams based on the result and re-encoding the crossfade image using the selected encoding parameter. Characterize.

In this moving image editing apparatus, first, the crossfade part of each of the two types of bitstreams to be combined is decoded, and a crossfade image is generated by superimposing the decoded images. .
Then, of the coding parameters represented by the motion vectors extracted from each of the two types of bitstreams, the coding amount when the crossfade image is decoded using the coding parameter of one bitstream, and the other And the code amount when the cross-fade image is decoded using the encoding parameter of the bit stream. Then, the cross-fade image is re-encoded using the encoding parameter having the smaller predicted code amount.

As described above, the coding parameters represented by the motion vectors recorded in the two types of bit streams to be connected are obtained by the motion detection process originally required for the coding process of the crossfade image. It is possible to significantly reduce the amount of calculation required for the re-encoding processing of the cross-fade portion by using the encoding parameter instead of the encoding parameter.

Further, when determining which of the two coding parameters obtained from the two bit streams is used, the code amount generated when the coding parameter is used for coding the cross-fade part is predicted. However, it is possible to suppress a decrease in coding efficiency due to an erroneous selection of a coding parameter by selecting a smaller coding parameter and re-coding.

Further, according to the present invention, for each image region to be encoded of the cross-fade image, the code amount when the encoding parameters of the first and second bit streams corresponding to the image region are used, respectively. And the coding parameter of one of the first and second bitstreams is dynamically selected for each image area of the crossfade image to be coded. As a result, it becomes possible to apply the optimum coding parameter to each macroblock of the crossfade image, and it is possible to further improve the coding efficiency.

[0019]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention makes it possible to reduce the amount of calculation required for encoding when re-encoding a cross-fade image obtained by decoding and combining two moving image bit streams. It is a thing. Before describing the specific configuration of the present invention, the encoding structure of MPEG2 will be described first.

In MPEG2, the picture to be coded is classified into I picture, P picture and B picture according to its type. I-pictures are compressed using only spatial redundancy. That is, no motion detection is performed. The P picture is encoded by using the motion from the past picture on the time axis. B-pictures are coded by interpolation using the motion from both past and future pictures on the time axis. A diagram showing this prediction structure is shown in FIG. In FIG. 3, a group of pictures from two B pictures immediately before an I picture to a P picture immediately before the next I picture is called a Group Of Picture (GOP).

The moving image editing process of this embodiment will be described below. In this embodiment, as shown in FIG.
Editing is performed in P units. Here, an example of creating a new bitstream C from the original bitstream A (GOP1a to GOP16a) and bitstream B (GOP6b to GOP21b) is shown.

In this example, the first 5 GOPs of bitstream C are bitstreams A (GOP1a-GOP5).
Use a) as is. The last 5 in bitstream C
GOP is a bitstream B (GOP17b to GOP2
1b) is used as is. For 11 GOPs on the way, bit stream A (GOP6a to GOP16a)
And the bit stream B (GOP6b to GOP16b) are decoded, and a composite that is a cross-fade image is re-encoded.

Here, a typical crossfade image is
Bitstream A faded out (GOP6a
To GOP16a) and the image of the bitstream B (GOP6b to GOP16b) to be faded in, but the pixel value of the composite image does not exceed a predetermined upper limit value. If so, bit stream A (GOP6a to GOP16a)
Image while fading out the image of bitstream B
The images of (GOP6b to GOP16b) are cut in so as to overlap them, or vice versa.
It is also possible to perform processing such as cutting out images 6a to GOP 16a).

FIG. 1 shows the detailed configuration of the cross-fade image re-encoding processing unit of this embodiment. As shown in FIG. 1, the cross-fade image re-encoding processing unit has a storage device 1 for storing the bit stream A and a storage device 2 for storing the bit stream B as storage devices.
A storage device 3 in which the encoding parameter A in the crossfade period of the bitstream A is recorded, a storage device 4 in which the encoding parameter A in the crossfade period of the bitstream B is recorded, and a crossfade of the bitstream A. The storage device 5 in which the decoded image A for the period is recorded, the storage device 6 in which the decoded image B for the crossfade period of the bitstream B is recorded, and the moving image bitstream C after editing are recorded. A storage device 14 is provided. The storage devices 1, 2, 3, 4, 5, 6, 14 may be configured by using different areas of the same storage device.

The cross-fade image re-encoding processing unit includes a decoder 7 for decoding the bit streams A and B.
, Two generated code amount predictors 8 and 9, and a decoded image A,
A crossfade image generator 10 for generating a crossfade image by synthesizing B, and a comparator 11 for comparing the predicted generated code amount and selecting the coding parameter of either bitstream A or B are selected. An encoder 12 that encodes a crossfade image using an encoding parameter, and a multiplexer that multiplexes the bitstream A, the encoded crossfade image, and the bitstream B to generate an edited moving image bitstream C. And a rectifier 13.

These decoder 7, generated code amount predictor 8,
9, the comparator 11, the crossfade image generator 10, the encoder 12, and the multiplexer 13 may be realized by hardware or software.

Next, referring to the flowchart of FIG.
The procedure of moving image joining processing by crossfade will be described. First, of the bit stream A recorded in the storage device 1, a portion from which a crossfade image is to be created is sent to the decoder 7. Then, in the decoder 7, the bit stream A is decoded for each macroblock, and the decoded image information is sent to the storage device 5. At the same time, each of the motion vector, the coding type, etc. used in the decoding process. Information on the coding parameter for the macroblock is sent to the storage device 3 (step S101). The image and the motion vector created by the decoder 7 are temporarily stored in the storage device 5 and the storage device 3, respectively.
Remembered in.

Similarly, the part of the bit stream B recorded in the storage device 2 from which a crossfade image is to be created is sent to the decoder 7.
The decoder 7 sends the decoded image information to the storage device 6 and at the same time sends the coding parameter information for each macroblock such as the motion vector and coding type used for the decoding to the storage device 4 (step S102). ). The image and the motion vector created by the decoder 7 are once stored in the storage device 6 and the storage device 4, respectively.

The crossfade image generator 10 creates a crossfade image from the decoded moving images recorded in the storage devices 5 and 6 (step S103). The time from the start to the end of the crossfade is T, and the elapsed time from the crossfade start time is t (0 ≦ t ≦
T) and the pixel values of the macroblocks of the bit streams A and B decoded at time t are Pa and Pb, respectively.
Then, when crossfading is performed in which the image of the bitstream A is faded out while the image of the bitstream B is faded in, the crossfade image Pc at the time t is created by applying, for example, the following equation. can do.

Pc = Pa × (T−t) / T + Pb × (t / T) In the generated code amount predictors 8 and 9, the motion vectors of the bit stream A and the bit stream B are used for encoding the crossfade image. The generated code amount at this time is predicted in macroblock units (steps S104 and S10).
5).

Now, a picture αPA + (1-α) PB of a macroblock in which the component of the image PA obtained by decoding the bitstream A is α and the component of the image PB obtained by decoding the bitstream B is 1-α , The motion vector M of the corresponding macroblock position contained in the bitstream A
If VA is coded, the code amount generated when the picture is coded is the code amount generated when PA is coded by MVA and the code amount generated when PB is coded by MVA. Can be approximated by weighted addition.

Since it is expected that the motion vector does not match at all in the code amount generated when PB is encoded by MVA, when the macroblock at the same position on the closest I picture in the bit stream B is encoded. It is approximated by the generated code amount.

FIG. 5 shows that the pixel value of an image is α
FIG. 6 is a diagram showing the relationship between α and the amount of code generated when a DCT coefficient is coded in the case of multiplying by (0 ≦ α ≦ 1). From this figure, it can be seen that the relationship between α and the generated code amount can be approximated by a substantially straight line.

From the above, the predicted generated code amount BA when the macroblock of the cross-fade image is decoded using the motion vector MVA of the bit stream A is
For example, it can be expressed by the following formula.

BA = BA0 + BAdct × (T−t) /
T + BBIdct × t / T where BA0 is the bit amount of the header part (including information such as a motion vector and coding mode) in the macroblock in the bitstream A, and BAdct is D in the macroblock in the bitstream A.
The bit amount of the CT coefficient, BBIdct, represents the bit amount of the DCT coefficient in the macro block at the same position as the macro block on the I picture closest to the macro block in time in the bit stream B.

Similarly, for the predicted generated code amount BB of the macroblock when the motion vector MVB of the bit stream B is used, the following equation can be used, for example. BB = BB0 + BBdct x t / T + BAIdct x (T
-T) / T Here, BB0 is the bit amount of the header part in the macroblock in the bitstream B, and BBdct is D in the macroblock in the bitstream B.
The bit amount of the CT coefficient, BAIdct, represents the bit amount of the DCT coefficient in the macro block at the same position as the macro block of the I picture closest in time in the bit stream A.

In the comparator 11, the generated code amount predictors 8, 9
The predicted generated code amounts obtained from the above are compared, and the motion vector on the side with the smaller predicted generated code amount, that is, MVA is selected if BA <BB, and MVB is selected if BA> BB, from the storage devices 3 and 4, and the encoder is selected. 12 (step S106).

The encoder 12 compares the crossfade image obtained from the crossfade image generator 10 with the comparator 11.
The inter-frame coding is performed for each macroblock using the motion vector sent from (steps S107 and S10).
8).

The encoded bit stream is combined with the bit streams stored in the storage devices 1 and 2 by the multiplexer 13, and one bit stream connected from the bit stream A to the bit stream B by crossfading. Is stored in the storage device 14 as the above, and the target connection by the crossfading of the bitstream A and the bitstream B is realized.

In this way, until the coding of all the cross-fade images is completed, the coding parameters of either one of the two corresponding macro blocks of the bit streams A and B are set in macro block units of the cross-fade images. While being dynamically selected, step S104-
The process of S108 is repeatedly executed (step S10).
9).

FIG. 6 shows an example of a concrete configuration of the encoder 12 shown in FIG. As shown in the figure, the encoder includes a frame order rearranging unit 50, a subtracting unit 51, a DCT.
Transform unit 52, quantization unit 53, variable length coding unit (VLC)
54, multiplexing section 55, inverse quantization section 56, inverse DCT conversion section 57, addition section 58, motion compensation section 59, reference image memory 6
0, which are respectively the frame order rearranging unit 30, the subtracting unit 31, and the DCT converting unit 3 of FIG.
2, a quantization unit 33, a variable length coding unit (VLC) 34, a multiplexing unit 35, an inverse quantization unit 36, an inverse DCT conversion unit 37, an addition unit 38, a motion compensation unit 39, and a reference image memory 41. Therefore, the function corresponding to the motion detection unit 40 in FIG. 7 is omitted.

That is, in this embodiment, the motion compensation unit 5
Since the encoding parameter of the bit stream A or B is given to 9 as it is, it is not necessary to newly perform the motion detection process at the time of encoding. Therefore, as compared with the encoder shown in FIG. 7, the motion detector is omitted.

Therefore, the amount of calculation required by the encoder 12 in FIG. 1 can be suppressed to about twice the amount of calculation required by the decoder 7 for decoding a normal MPEG2 bit stream. Considering that the decoder 7 decodes two bit streams, in the present system the encoder 1
It is possible to reduce the amount of calculation required by 2 for encoding to the extent of the amount of calculation required for the decoder 7 to decode two bit streams.

That is, when the editing process of the present embodiment is configured by a PC capable of decoding MPEG2 in real time by software, even if the connection process by crossfade is realized by software without additional hardware, The stress caused to the user due to the waiting time can be extremely small.

As described above, according to the present embodiment, the coding parameters represented by the motion vectors recorded in the two types of bit streams A and B to be connected are originally subjected to the coding process of the crossfade image. It is possible to significantly reduce the amount of calculation required for the re-encoding process of the cross-fade part by using the parameter instead of the encoding parameter obtained by the motion detection process required for the above.

Further, when determining which of the two coding parameters obtained from the two bit streams is used, the generated code amount when the coding parameter is used for coding the crossfade part is predicted. , By re-encoding with the lesser encoding parameter,
It is possible to suppress a decrease in coding efficiency due to an erroneous selection of a coding parameter. In particular, high coding efficiency can be obtained by dynamically selecting the optimum coding parameter for each macroblock of the crossfade image.

In the above description, the case of predicting the generated code amount for the cross-fade image created by superimposing the image of the bit stream A to be faded out and the image of the bit stream B to be faded in will be described. However, since the prediction of the generated code amount can be obtained based on the mixing ratio of the bit streams A and B with respect to the passage of time, for example, while fading out the image of the bit stream A, overlapping the image of the bit stream B with it. The present embodiment also applies to a cross-fade image obtained by combining processing such as cutting in as described above, or vice versa, while cutting out the image of bitstream A while fading in the image of bitstream B. You can apply the editing method by That.

[0048]

As described above, according to the present invention,
Re-encoding of cross-fade image parts by reusing encoding parameters such as motion vectors in originally encoded bit streams in a moving image editing apparatus that connects inter-frame encoded bit streams by crossfading By performing, it is possible to significantly reduce the amount of calculation required for re-encoding.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of a moving image editing apparatus according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a procedure for combining two bitstreams by crossfading in the same embodiment.

FIG. 3 is a diagram showing an example of a prediction structure of a picture of an MPEG2 bitstream to be edited in the embodiment.

FIG. 4 is an exemplary view showing an example of a bitstream after cross-fade connection obtained by the editing process of the embodiment.

FIG. 5 is a diagram showing a relationship between a dynamic range of an edited image and a generated code amount in the same embodiment.

FIG. 6 is a diagram showing an example of a configuration of a moving image encoder applied to the embodiment.

FIG. 7 is a diagram showing a configuration of a conventional moving image encoder.

FIG. 8 is a diagram for explaining the principle of motion detection processing in a conventional moving image encoder.

FIG. 9 is a diagram for explaining a moving image variant operation according to a conventional crossfade connection.

[Explanation of symbols]

1, 2, 3, 4, 5, 6, 14, ... Storage device 7 ... Decoder 8, 9 ... Generated code amount predictor 10 ... Crossfade image generator 11 ... Comparator 12 ... Encoder 13 ... Multiplexer 20, 21 ... Decoder 22 ... Crossfade image synthesizer 23 ... Encoder 30, 50 ... Frame order rearranging unit 31, 51 ... Subtraction unit 32, 52 ... Discrete cosine transform unit 33, 53 ... Quantizer 34, 54 ... Variable length coding unit 35, 55 ... Multiplexing unit 36, 56 ... Inverse quantizer 37, 57 ... Inverse discrete cosine transform unit 38, 58 ... Adder 39, 59 ... Motion compensation unit 40 ... Motion detector 41, 60 ... Reference image memory

─────────────────────────────────────────────────── ─── Continuation of the front page (58) Fields surveyed (Int.Cl. ⁷ , DB name) H04N 5/91-5/956 H04N 7/32

Claims (6)

(57) [Claims] 1. A moving image editing apparatus having a function of partially superimposing and combining two types of first and second moving image bit streams encoded using interframe prediction by crossfading. , The first corresponding to the period of combining with the crossfade
And a means for decoding each of the second bitstreams, a means for creating a crossfade image from the two moving images obtained by the decoding, and the first and second bits in the period of combining by the crossfade A means for analyzing a stream and extracting respective coding parameters, and a code amount prediction for predicting a code amount when the crossfade image is coded using the coding parameters of the first and second bit streams, respectively. Means for selecting an encoding parameter of one of the first and second bit streams based on the prediction result,
And a means for re-encoding the crossfade image using the selected encoding parameter. 2. The code amount predicting means, for each image area to be encoded of the crossfade image,
A code amount when the encoding parameters of the first and second bit streams corresponding to the image region are used is predicted, and the first and second image regions are encoded for each image region of the crossfade image. 2. The moving image editing apparatus according to claim 1, wherein any one of the encoding parameters of the bitstreams is dynamically selected. 3. The moving image editing apparatus according to claim 1, wherein the encoding parameter is motion vector information. 4. A moving picture editing method for partially overlapping and combining two kinds of first and second moving picture bit streams coded using inter-frame prediction by cross fading, wherein The first corresponding to the period of fading
And a second bitstream are respectively decoded, a crossfade image is created from the two moving images obtained by the decoding, and the first and second bitstreams are coded in the period of combining with the crossfade. A parameter is used to predict the code amount when the crossfade image is coded, and one of the first and second bitstreams is selected as the coding parameter based on the prediction result, and the selected parameter is selected. A moving image editing method comprising re-encoding the crossfade image using an encoding parameter to combine the first and second types of moving image bitstreams by crossfade. 5. For each image area to be encoded of the cross-fade image, a code amount when using encoding parameters of each of the first and second bit streams corresponding to the image area is predicted, 5. The moving image editing method according to claim 4, wherein the encoding parameter of one of the first and second bitstreams is dynamically selected for each image area of the crossfade image to be encoded. . 6. The moving image editing method according to claim 4, wherein the encoding parameter is motion vector information.