JPH11243525A - Dynamic image editor and dynamic image editing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce an operation amount required to apply re-coding to a cross fade image in the case of using a motion between frames to connect two bit streams that are inter-frame coded by means of the cross fade image by adopting the picture experts group (MPEG)2 or the like. SOLUTION: Generated code amount prediction devices 8, 9 predict respectively a code amount in the case of decoding a cross fade image, by using coding parameters of one bit stream and of decoding the cross fade image through the use of coding parameters of other bit stream among the coding parameters represented by motion vectors which are extracted from the two kinds of the bit streams A, B going to be connected. Then re-coding is applied to the cross fade image by using the coding parameter whose predicted code amount is smaller. Thus, the need for applying motion detection processing to the cross fade image is eliminated, and the operation amount is reduced.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture editing apparatus and a moving picture editing method in which a plurality of coded moving picture data are re-edited into one moving picture data.

[0002]

2. Description of the Related Art The international standard MPEG2 (M
oving Picture Experts Gro
2. Description of the Related Art Digital video coding technology represented by up phase 2) is currently receiving attention as a technology that has the potential to replace current analog video media such as television broadcasts and videos.

[0003] Moving picture compression in MPEG2 employs a method called inter-frame prediction, which removes redundancy in the time direction and reduces the amount of information. FIG. 7 shows an example of an MPEG2 encoding apparatus.

The removal of the redundancy in the time direction is executed by the motion detecting section 40 and the motion compensating section 39 in FIG. As shown in FIG. 8, the motion detecting unit 40 divides the screen into 16 × 16 pixels from a previously coded image stored in the reference image memory 41 (macro block).
A motion between two frames is detected by searching for a similar part in units. The motion information (motion vector) is sent to the outside as a bit stream, and is also output to the motion compensation unit 39.

[0005] The motion compensating section 39 reads from the reference image memory 41 an area of 16 × 16 pixels similar to the macroblock to be coded from the motion vector, and sets it as a predicted image. The subtraction unit 31 obtains a difference between the macroblock to be coded of the input image signal rearranged in the coding screen order by the frame rearrangement unit 30 and the predicted image (obtains a residual signal). As a result, the information amount of the macroblock to be coded is greatly reduced. Next, information compression in the spatial direction is performed on the suppressed macroblock.

First, a discrete cosine transform (DCT) unit 32
The residual signal is mapped to the frequency space by the
To quantize with a predetermined roughness. Next, this is subjected to variable-length coding by a variable-length coding unit (VLC) 34, so that the original moving picture information is compressed to several tenths. This information is multiplexed with the motion vector in the multiplexing unit 35, and is output as a bit stream.

The output from the quantization unit 33 is decoded via an inverse quantization unit 36, an inverse DCT unit 37, and an addition unit 38, and then recorded in a reference image memory 41 as a reference image. Among these processes, the motion detection process has the largest processing amount. A typical motion detection method includes a macroblock to be coded and a pixel value of an area within a search range of 0.5.
There is a method in which the sum of squares of the difference between pixel values is calculated while shifting in pixel units, and the one having the smallest value of the sum of squares is used as a motion vector. It is known that such a motion detection method generally requires a huge amount of calculation, and the amount of calculation per unit time required for performing encoding in real time is several hundred GOPS (Giga Operation).
per second). Therefore, special hardware such as a dedicated processor is required to perform the real-time encoding process.

On the other hand, with the improvement in the processing capability of personal computers (hereinafter, referred to as PCs) in recent years, it has become possible to process and edit digital moving images on an individual level. The processing and editing processing of moving images by a PC extracts a portion corresponding to a certain period from a plurality of moving image data (bit streams) encoded in advance, and performs a special change such as a scene change or a fade-in / fade-out or a cross-fade. Connect with other bitstreams using effects, one bitstream (program)
The work of putting them together in a representative way is typical.

With regard to editing of a moving image in MPEG2, for example, a method of connecting two moving image bit streams by a scene change has been proposed as disclosed in Japanese Patent Application Laid-Open No. Hei 8-251582. This is to connect the bit stream A and the bit stream B without overlapping temporally, and by adjusting the buffer capacity of the virtual decoder for connection, the amount of computation required for connection does not matter.

[0010]

However, when trying to realize crossfade, which is one of the special effects frequently used when joining two scenes,
It is necessary to newly generate a cross fade image from two different video bit streams.

In this case, as shown in FIG. 9, the decoding processes of the two video bit streams A and B by the decoding units 20 and 21 and the synthesis process of the decoded images A and B by the synthesis unit 22 are performed. A process of re-encoding the cross-fade image obtained by the synthesis by the encoding unit 23 is required.

As described above, since the motion detection processing is required for encoding, the amount of calculation required for re-encoding is enormous. Therefore, in a system composed of only a general-purpose processor without using additional hardware, this processing takes a long time, and causes a reduction in the user interface.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and has as its object to re-encode two types of moving image bit streams when they are connected by cross-fading without adding special hardware. An object of the present invention is to provide a moving image editing apparatus and a moving image editing method that reduce the processing time in the conversion processing and do not impair the operability of the user.

[0014]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention provides a method for cross-fading a first and a second type of moving image bit stream encoded using inter-frame prediction. A moving image editing apparatus having a function of partially overlapping and combining, wherein: means for decoding each of the first and second bit streams corresponding to a period to be combined by the crossfade; Means for creating a cross-fade image from the two moving images, and analyzing the first and second bit streams during the period of combining with the cross-fade,
Means for extracting the respective coding parameters; code amount predicting means for predicting the code amount when the cross-fade image is coded using the respective coding parameters of the first and second bit streams; Means for selecting one of the encoding parameters of the first and second bit streams based on the result, and re-encoding the cross-fade image using the selected encoding parameter. Features.

In this moving picture editing apparatus, first, a cross-fade portion of each of two types of bit streams to be combined is decoded, and a cross-fade image is generated by superimposing the decoded images. .
Then, among the encoding parameters represented by the motion vectors extracted from each of the two types of bit streams, the code amount when the cross-fade image is decoded using the encoding parameter of one bit stream, and the other And the amount of code when the cross-fade image is decoded using the coding parameter of the bit stream of. Then, re-encoding of the cross-fade image is performed using the encoding parameter with the smaller predicted code amount.

As described above, the coding parameters represented by the motion vectors recorded in the two types of bit streams to be connected are obtained by the motion detection processing which is originally required for the coding processing of the cross-fade image. By using the coding parameters instead of the coding parameters to be used, it is possible to greatly reduce the amount of calculation required for the recoding process of the cross-fade portion.

Further, when judging which of the two coding parameters obtained from the two bit streams is to be used, the code amount generated when the coding parameter is used for coding the cross-fade portion is estimated. Then, by selecting the smaller encoding parameter and re-encoding, it is possible to suppress a decrease in encoding efficiency due to erroneous selection of the encoding parameter.

Further, according to the present invention, for each image area to be encoded of the cross-fade image, a code amount when each of the encoding parameters of the first and second bit streams corresponding to the image area is used. And dynamically selecting one of the encoding parameters of the first and second bit streams for each image region to be encoded of the cross-fade image. As a result, it is possible to apply an optimal encoding parameter to each macroblock of the cross-fade image, and it is possible to further enhance encoding efficiency.

[0019]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention makes it possible to reduce the amount of calculation required for encoding when a cross-fade image obtained by decoding and combining two moving image bit streams is re-encoded. Things. Before describing the specific configuration of the present invention, the encoding structure of MPEG2 will be described first.

In MPEG2, pictures to be coded are classified into I pictures, P pictures and B pictures according to their types. I-pictures are compressed using only spatial redundancy. That is, no motion detection is performed. The P picture is encoded using a motion from a past picture on the time axis. The B picture is encoded by interpolation using both motions from both past and future pictures on the time axis. FIG. 3 shows this prediction structure. In FIG. 3, a group of pictures from two B pictures immediately before an I picture to a P picture immediately before a next I picture is called a Group Of Picture (GOP).

Hereinafter, the moving image editing process according to the present embodiment will be described. In the present embodiment, as shown in FIG.
Editing is performed in P units. Here, an example is shown in which a new bit stream C is created from the original bit stream A (GOP1a to GOP16a) and the bitstream B (GOP6b to GOP21b).

In this example, the first 5 GOPs of bit stream C are bit stream A (GOP1a to GOP5
Use a) as is. Last 5 of bitstream C
The GOP is a bit stream B (GOP17b to GOP2).
1b) is used as is. Bit stream A (GOP6a to GOP16a) for 11 GOPs on the way
And the bit stream B (GOP6b to GOP16b) are decoded, and a result obtained by synthesizing the decoded image into a cross-fade image is obtained by re-encoding.

Here, a typical crossfade image is
Bit stream A (GOP6a) to be faded out
To GOP16a) and the image of the bit stream B (GOP6b to GOP16b) to be faded in, but the pixel value of the composite image does not exceed a predetermined upper limit. If so, the bit stream A (GOP6a to GOP16a)
Bit stream B while fading out the image of
The image of (GOP6b to GOP16b) is cut in so as to overlap the image, or conversely, the image of bitstream B (GOP6b to GOP16b) is faded in while the bitstream A (GOP
Processing such as cutting out images of 6a to GOP16a) is also possible.

FIG. 1 shows a detailed configuration of the cross-fade image re-encoding processing unit of the present embodiment. As shown in FIG. 1, the cross-fade image re-encoding processing unit includes, as storage devices, a storage device 1 that stores a bit stream A and a storage device 2 that stores a bit stream B.
A storage device 3 in which the coding parameter A is recorded during the cross-fade period of the bit stream A; a storage device 4 in which the coding parameter A is recorded during the cross-fade period of the bit stream B; A storage device 5 for storing the decoded image A for the period, a storage device 6 for storing the decoded image B for the cross-fade period of the bit stream B, and a moving image bit stream C after editing. A storage device 14 is provided. Note that these storage devices 1, 2, 3, 4, 5, 6, and 14 may be configured using different areas of the same storage device.

The cross-fade image re-encoding unit includes a decoder 7 for decoding the bit streams A and B.
And two generated code amount predictors 8 and 9 and decoded images A and
A cross-fade image creator 10 for generating a cross-fade image by synthesizing B, a comparator 11 for comparing the predicted generated code amount and selecting one of the bit stream A and B encoding parameters, and An encoder 12 that encodes a cross-fade image by using an encoding parameter; and a multiplex that multiplexes a bit stream A, an encoded cross-fade image, and a bit stream B to generate an edited video bit stream C. And a converter 13.

These decoder 7, generated code amount predictor 8,
9, the comparator 11, the cross-fade image creator 10, the encoder 12, and the multiplexer 13 may be realized by hardware or software.

Next, referring to the flowchart of FIG.
A description will be given of a procedure of processing for joining moving images by crossfade. First, of the bit stream A recorded in the storage device 1, a portion from which a cross-fade image is to be created is sent to the decoder 7. Then, in the decoder 7, the bit stream A is decoded for each macroblock, and the decoded image information is sent to the storage device 5. At the same time, each of the motion vector used in the decoding process, the encoding type, etc. Information on the coding parameters for the macroblock is sent to the storage device 3 (step S101). The image and the motion vector created by the decoder 7 are temporarily stored in the storage devices 5 and 3 respectively.
Is stored.

Similarly, a portion of the bit stream B recorded in the storage device 2 from which a cross-fade image is to be created is sent to the decoder 7.
The decoder 7 sends the decoded image information to the storage device 6 and, at the same time, sends to the storage device 4 information on the coding parameters for each macroblock such as the motion vector and the coding type used for decoding (step S102). ). The image and the motion vector created by the decoder 7 are temporarily stored in the storage devices 6 and 4, respectively.

The crossfade image creator 10 creates a crossfade image from the decoded moving images recorded in the storage devices 5 and 6 (step S103). The time from the start of the crossfade to the end thereof is T, and the elapsed time from the crossfade start time is t (0 ≦ t ≦
T), and the pixel values of the macroblocks of the bit streams A and B decoded at time t are Pa and Pb, respectively.
Then, when a cross-fade in which the image of the bit stream B is faded in while the image of the bit stream A is faded out, the cross-fade image Pc at the time t is created by applying, for example, the following equation: can do.

Pc = Pa × (T−t) / T + Pb × (t / T) In the generated code amount predictors 8 and 9, the motion vectors of the bit stream A and the bit stream B are used for coding a cross-fade image. The generated code amount at the time is predicted in macroblock units (steps S104 and S10).
5).

Now, a picture αPA + (1-α) PB of a macroblock in which the component of the image PA obtained by decoding the bit stream A is α and the component of the image PB obtained by decoding the bit stream B is 1-α. , The motion vector M at the corresponding macroblock position contained in the bit stream A
If the picture is coded by VA, the code amount generated when the picture is coded is the code amount generated when PA is coded by MVA and the code amount generated when PB is coded by MVA. Can be approximated by weighted addition.

Since the amount of code generated when PB is encoded by MVA is expected to have no motion vector at all, when encoding a macroblock at the same position on the nearest I picture in bit stream B, Approximate by the generated code amount.

FIG. 5 shows the pixel value of a certain image as α.
FIG. 9 is a diagram illustrating a relationship between α and a code amount generated when encoding a DCT coefficient when multiplying by (0 ≦ α ≦ 1). From this figure, it can be seen that the relationship between α and the generated code amount can be approximated by a substantially straight line.

From the above, the predicted generated code amount BA when a macroblock of a cross-fade image is decoded using the motion vector MVA of the bit stream A is:
For example, it can be represented by the following equation.

BA = BA0 + BAdct × (Tt) /
T + BBIdct × t / T where BA0 is the bit amount of the header portion (including information such as a motion vector and a coding mode) in the macroblock in the bit stream A, and BAdct is D in the macroblock in the bit stream A.
The bit amount of the CT coefficient, BBIdct, represents the bit amount of the DCT coefficient in the macroblock at the same position as the macroblock on the I picture closest to the macroblock in the bit stream B.

Similarly, the following formula can be used for the predicted code amount BB of the macroblock when the motion vector MVB of the bit stream B is used. BB = BB0 + BBdct × t / T + BAidct × (T
−t) / T where BB0 is the bit amount of the header part in the macroblock in the bitstream B, and BBdct is D in the macroblock in the bitstream B.
The bit amount of the CT coefficient, BAIdct, represents the bit amount of the DCT coefficient in the macroblock at the same position as the macroblock of the I picture closest in time in the bit stream A.

In the comparator 11, generated code amount predictors 8, 9
Are calculated from the storage devices 3 and 4, and the motion vector on the side where the predicted code amount is small, that is, MVA if BA <BB, and MVB if BA> BB are selected from the storage devices 3 and 4. 12 (step S106).

The encoder 12 converts the cross-fade image obtained from the cross-fade image generator 10 into a comparator 11
(Steps S107, S10)
8).

The coded bit stream is combined with the bit streams stored in the storage devices 1 and 2 by the multiplexer 13 to form one bit stream connected from the bit stream A to the bit stream B by cross-fading. Is stored in the storage device 14, and the intended connection between the bit stream A and the bit stream B is realized by cross-fading.

In this manner, until coding of all the cross-fade images is completed, one of the coding parameters of one of the two corresponding macro blocks of the bit streams A and B is used in units of macro blocks of the cross-fade image. Are dynamically selected, and steps S104-
The processing of S108 is repeatedly executed (Step S10
9).

FIG. 6 shows an example of a specific configuration of the encoder 12 shown in FIG. The encoder includes a frame order rearranging unit 50, a subtracting unit 51, a DCT
Conversion unit 52, quantization unit 53, variable length coding unit (VLC)
54, multiplexing section 55, inverse quantization section 56, inverse DCT transform section 57, adder section 58, motion compensation section 59, reference image memory 6
0, which are respectively composed of the frame order rearranging unit 30, the subtracting unit 31, and the DCT converting unit 3 shown in FIG.
2. Corresponds to a quantization unit 33, a variable length coding unit (VLC) 34, a multiplexing unit 35, an inverse quantization unit 36, an inverse DCT transformation unit 37, an addition unit 38, a motion compensation unit 39, and a reference image memory 41. In this case, a function corresponding to the motion detection unit 40 in FIG. 7 is omitted.

That is, in this embodiment, the motion compensator 5
Since the encoding parameter of the bit stream A or B is given to 9 as it is, there is no need to perform a new motion detection process at the time of encoding. Therefore, as compared with the encoder shown in FIG. 7, the motion detector is omitted.

For this reason, the amount of calculation required by the encoder 12 in FIG. 1 can be suppressed to about twice the amount of calculation required by the decoder 7 for decoding an ordinary MPEG2 bit stream. Given that the decoder 7 decodes two bit streams, the encoder 1 in the present system
The amount of operation required for encoding 2 can be reduced to about the amount of operation required for the decoder 7 to decode two bit streams.

That is, when the editing process of this embodiment is configured by a PC capable of decoding MPEG2 in real time by software, even if the connection process by crossfade is realized by software without additional hardware, The stress on the user due to the waiting time can be extremely small.

As described above, according to the present embodiment, the encoding parameters typified by the motion vectors recorded in the two types of bit streams A and B to be connected are originally encoded in the cross-fade image encoding process. By using the parameters instead of the coding parameters obtained by the motion detection processing required for, it is possible to greatly reduce the amount of calculation required for the re-coding processing of the cross-fade portion.

When judging which of the two encoding parameters obtained from the two bit streams is to be used, the amount of generated code when the encoding parameter is used for encoding the cross-fade portion is estimated. , By re-encoding using the lesser encoding parameter,
It is possible to suppress a decrease in coding efficiency due to erroneous selection of a coding parameter. In particular, high encoding efficiency can be obtained by dynamically selecting the optimal encoding parameter for each macroblock of the cross-fade image.

In the above description, a case is described in which the generated code amount of a cross-fade image created by superimposing a fade-out bit stream A image and a fade-in bit stream B image is superimposed. However, since the prediction of the generated code amount can be obtained based on the mixing ratio of the bit streams A and B with the passage of time, for example, the image of the bit stream B overlaps the image of the bit stream A while fading out the image of the bit stream A. The cross-fade image obtained by the synthesizing process in which the image of the bit stream A is cut out in the middle while the image of the bit stream B is faded in while the image of the bit stream B is faded in, You can apply the editing method by That.

[0048]

As described above, according to the present invention,
In a moving image editing apparatus for connecting inter-coded bit streams by cross-fading, re-encoding of a cross-fade image portion by reusing coding parameters such as motion vectors in the originally coded bit stream. , It is possible to greatly reduce the amount of calculation required for re-encoding.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a moving image editing apparatus according to an embodiment of the present invention.

FIG. 2 is an exemplary flowchart showing a procedure when two bit streams are combined by crossfading in the embodiment.

FIG. 3 is an exemplary view showing an example of a prediction structure of a picture of an MPEG2 bit stream to be edited in the embodiment.

FIG. 4 is an exemplary view showing an example of a bit stream after cross-fade connection obtained by the editing process of the embodiment.

FIG. 5 is a diagram showing a relationship between a dynamic range of an edited image and a generated code amount in the embodiment.

FIG. 6 is an exemplary view showing an example of the configuration of a video encoder applied to the embodiment.

FIG. 7 is a diagram showing a configuration of a conventional video encoder.

FIG. 8 is a diagram for explaining the principle of motion detection processing in a conventional video encoder.

FIG. 9 is a view for explaining a moving image variation operation based on a conventional cross-fade connection.

[Explanation of symbols]

 1, 2, 3, 4, 5, 6, 14 ... storage device 7 ... decoder 8, 9 ... generated code amount predictor 10 ... cross-fade image creator 11 ... comparator 12 ... encoder 13 ... multiplexing Decoders 20 and 21 Decoders 22 Crossfade image synthesizers 23 Encoders 30 and 50 Frame order rearranging units 31 and 51 Subtracting units 32 and 52 Discrete cosine transform units 33 and 53 Quantizing units 34, 54 ... variable length coding unit 35, 55 ... multiplexing unit 36, 56 ... inverse quantization unit 37, 57 ... inverse discrete cosine transform unit 38, 58 ... addition unit 39, 59 ... motion compensation unit 40 ... motion detection Units 41, 60: Reference image memory

Claims (6)

[Claims] 1. A video editing apparatus having a function of partially overlapping and combining first and second types of video bit streams coded using inter-frame prediction by crossfading. The first period corresponding to the period of combining with the cross fade
Means for decoding each of the first and second bit streams, means for creating a cross-fade image from two moving images obtained by decoding, and the first and second bits during a period of combining by the cross-fade. Means for analyzing a stream and extracting respective coding parameters; code amount prediction for predicting a code amount when the cross-fade image is coded using the respective coding parameters of the first and second bit streams Means for selecting one of the first and second bitstream encoding parameters based on the prediction result;
Means for re-encoding the cross-fade image using the selected encoding parameter. 2. The method according to claim 1, wherein the code amount predicting unit comprises:
Predicting the code amount when using the respective coding parameters of the first and second bit streams corresponding to the image area, and calculating the first and second code areas for each of the coding target image areas of the cross-fade image. The moving picture editing apparatus according to claim 1, wherein one of the encoding parameters of the bit stream is dynamically selected. 3. The moving picture editing apparatus according to claim 1, wherein the encoding parameter is motion vector information. 4. A moving image editing method in which first and second two types of moving image bit streams encoded using inter-frame prediction are partially overlapped and combined by crossfading, The first corresponding to the time period of the fading
And a second bit stream are respectively decoded, and a cross-fade image is created from the two moving images obtained by the decoding, and each of the first and second bit streams is encoded during a period in which the cross-fade is combined. A code amount when the cross-fade image is coded is predicted using parameters, and one of the first and second bit streams is selected based on the prediction result, and the selected coding parameter is selected. A moving picture editing method, characterized in that the first and second moving picture bit streams are combined by cross-fading by re-encoding the cross-fade picture using coding parameters. 5. For each image region to be encoded of the cross-fade image, predict a code amount when using each of the encoding parameters of the first and second bit streams corresponding to the image region; 5. The moving image editing method according to claim 4, wherein one of the first and second bitstream encoding parameters is dynamically selected for each image region to be encoded of the cross-fade image. . 6. The moving picture editing method according to claim 4, wherein said encoding parameter is motion vector information.