JPH07177514A - Dynamic image compressor - Google Patents

Abstract

PURPOSE:To accurately reproduce a frame missing at a receiver side. CONSTITUTION:The compressor is made up of coder 1 consisting of a motion detection section 11 obtaining a motion vector from frame-thinned image data, a frame thinning processing section 12 implementing thinning processing of a transmission frame number at a sender side, an adder 13, a DCT arithmetic section 14, a quantization section 15, an entropy coding section 16, a buffer 17 and a motion inter-frame predict section 18 or the like, and up of a decoder 2 consisting of a buffer 23, an entropy decoding section 24, an inverse quantization section 25, an IDCT arithmetic section 26, an adder 27, a prediction device 28, and a dynamic image interpolation section 29 or the like using a motion vector to interpolate an interpolation frame and providing the result to an external device as a picture output. The dynamic image interpolation section 29 approximates a motion to be a curve based on a motion vector obtained from an adjacent transmission frame and re-compose an interpolation frame to be interpolated based on the ratio of the distance between adjacent transmission frames.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture interpolating device used for moving picture compression processing and the like, and more particularly to a moving picture interpolating device capable of accurately reproducing a missing frame on the receiving side.

[0002]

2. Description of the Related Art A moving image interpolating device is considered to be used mainly in a video telephone device, a video conference and the like.

A conventional moving picture processing apparatus uses a moving picture interpolating apparatus for reproducing a missing frame on the receiving side.
In a device using the conventional frame interpolation method, the motion is detected for each received frame, the ratio of the distance between the frame to be interpolated and the frames before and after the received frame is calculated, and the motion is divided based on the ratio, The interpolated frame to be interpolated was reconstructed based on

An example thereof is shown in FIG. FIG. 5 shows a transmitted frame (adjacent transmission frame) P and a transmitted frame N.
FIG. 10 is a diagram showing a conventional frame interpolation method when there are two interpolation frames M between and. In FIG. 5, arrows indicate the motion vector MV, and BP, BN, BM indicate blocks on each frame. Further, the distance between the transmitted frame P and the transmitted frame N is set to 1, and the transmitted frame P and the transmitted frame N are set.
The distance from the side interpolation frame M is a.

In FIG. 5, the transmitted frame is divided into blocks, and between the transmitted frame P and the transmitted frame N, the most similar block on the transmitted frame N is obtained in each block of the transmitted frame P (see FIG. In 5, the amount of movement of the block BP and the block BN) is set as MV and is set as a motion vector. Then, the block BM of the interpolation frame M to be interpolated from the two blocks associated with the motion vectors on these transmitted frames P and N is obtained by the ratio of the distance between the transmitted frame P and the transmitted frame N. .

[0006]

However, in such a conventional moving image processing apparatus, since the detection of the motion vector is the interpolation method based on the linear approximation, it is possible to cope with the linear movement of the moving object. Although it is possible, there is a problem that it cannot cope with curvilinear motion such as rotational motion and parabolic motion. If it is not possible to cope with such curvilinear motions and rotary motions, the frame missing on the receiving side cannot be accurately reproduced.

Therefore, an object of the present invention is to provide a moving picture interpolating device which can accurately reproduce a missing frame on the receiving side.

[0008]

The invention according to claim 1 is
In order to achieve the above object, in a moving image interpolating device for predictively interpolating a frame image between each frame from moving image information including a plurality of frame image information and motion information between each frame, the motion information between each frame is nonlinearly approximated. Then, the frame image between each frame is predicted and interpolated.

According to a second aspect of the present invention, in a moving image interpolating device for predictively interpolating a frame image between respective frames from moving image information consisting of a plurality of frame image information, motion information of the respective interframe images is detected, The motion information is nonlinearly approximated to predictively interpolate the frame images between the frames.

For example, as described in claim 3, the motion information is a movement amount corresponding to a block of the predetermined frame image information between each frame, which corresponds to the most similar block among other frame image information. It may be.

The interpolation may be a predictive interpolation based on a ratio of temporal distances between frames, as described in claim 4, for example.

[0012]

According to the first aspect of the present invention, for example, when moving image information with a reduced number of frames is transmitted from the transmitting side when compressing a low-rate moving image in a videophone or the like, the moving image is received at the receiving side. As the information is decoded, the motion is detected from the thinned and transmitted frame, and the motion is nonlinearly approximated. Then, the thinned frames are reconstructed on the approximated curve.

Therefore, it is possible to cope with curvilinear motion, rotary motion, etc., and it becomes possible to accurately reproduce the missing frame on the receiving side.

According to the present invention, the motion vector is detected from the frame thinned out by the motion detecting means and sent, and the motion is nonlinearly approximated based on the motion vector detected by the interpolating means. And the decimated frames are reconstructed by non-linearly approximating the motion. In this case, the interpolating means obtains a curve passing through three points of the start point and the end point of the motion vector detected by the motion detecting means, and a curve passing through the three points (from the ratio of the distance between the frame to be interpolated and the transmitted frame). For example, a block of a frame to be interpolated on a quadratic curve may be interpolated.

Therefore, it is possible to cope with curvilinear motion, rotary motion, etc., and it becomes possible to accurately reproduce the missing frame on the receiving side.

[0016]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings.

1 to 4 are views showing an embodiment of a moving image interpolation apparatus according to the present invention.

First, the structure will be described. FIG. 1 is a block diagram of a moving image interpolating device. In this figure, the moving image interpolating device comprises an encoder 1 on the transmitting side and a decoder 2 on the receiving side.

In FIG. 1, the encoder 1 of the moving image processing apparatus includes a motion detector 11, a frame thinning processor 12, an adder 13, a DCT calculator 14, a quantizer 15, an entropy encoder 16, and a buffer. 17 and the motion inter-frame prediction unit 18 and the like.

Image data is input to the encoder 1 from an image memory (not shown). The image memory stores image data to be compressed, and the frame thinning processing unit 1
Output to 2.

The image data read out from the image memory is input to the frame thinning processing unit 12, and the frame thinning processing unit 12 determines the number of transmission frames at the transmitting side when compressing a low-rate moving image in a video telephone device or the like. The thinning process is performed. As for the thinned frames, the missing frames are accurately reproduced by the moving image interpolating unit 29 on the receiving side, which will be described later.

The image data thinned out by the frame thinning processing unit 12 is input to the adder 13, and the adder 13 outputs the image data thinned out by the frame thinning processing unit 12 and the motion inter-frame prediction unit 18. The prediction results of the motion compensation inter-frame prediction process are added, and the addition result is output to the DCT calculation unit 14.

The DCT calculation unit 14 performs a DCT calculation on the image data of the addition result of the adder 13, and outputs the calculation result to the quantization unit 15.

The quantizer 15 quantizes the data calculated by the DCT calculator 14 according to the quantization width determined by the controller within a certain error range according to the value of the quantization table ROM, and the inter-frame predictor 18 and Output to the entropy coding unit 16. A quantization table ROM (not shown) stores in advance a quantization table having quantization matrix values of different sizes for each DCT coefficient.

The entropy coding unit 16 performs entropy coding on the output of the quantization unit 15 according to the value of the entropy coding table ROM, and writes the generated code amount of each block in the data buffer 17.

At the time of encoding, the entropy encoder 16 zigzag-scans the DCT coefficients for each block in order from the energy-concentrated coefficient to convert the DCT coefficient into a one-dimensional array, and the last AC coefficient in the block is 0. When
EOB is added next to the code for the final effective coefficient.

Further, the motion detecting section 1 together with this image data
The motion vector from 1 is transmitted in a predetermined format.

The buffer 17 smoothes the fluctuating information generation to a constant rate, outputs it to the controller, and transmits the image data to the decoder 2 on the receiving side, which will be described later, and other moving image processing apparatus through the transmission path. .

The motion detection unit 11 obtains a motion vector from the image data thinned out by the frame thinning processing unit 12 and outputs it to the entropy coding unit 16 and a predictor 22 which will be described later.

The inter-motion frame prediction unit 18 performs motion compensation prediction based on the motion vector detected by the motion detection unit 11 based on a periodic intra-frame coded frame. The inverse quantization unit 19, IDCT operation unit (IDC
T) 20, an adder 21 and a predictor 22.

The inverse quantization unit 19 inversely quantizes the image data quantized by the quantization unit 15, restores the image data before quantization, and outputs it to the IDCT operation unit 20.

The IDCT operation unit 20 performs inverse D on the image data returned to the state before quantization by the inverse quantization unit 19.
It performs a CT (IDCT) operation and outputs it to the adder 21.

The adder 21 adds the prediction result of the motion compensation interframe processing to the image data returned to the state before the DCT processing by the IDCT calculation unit 20 and outputs the result to the predictor 22.

A switch (not shown) is provided at the input / output stage of the predictor 22 to switch the signal path according to the image mode and the prediction mode from the controller.
For example, the switch is turned on / off in the prediction mode in response to it, connecting the predictor 22 to the adder 13.

The predictor 22 performs motion compensation prediction based on the motion vector detected by the motion detector 11.

Here, the encoder 1 is controlled by a controller (not shown), image data from the image memory, information and difference values from the buffer 17 are input to the controller, and the controller determines the image mode and prediction. The mode and various control signals are output to control the entire system. The motion detector 11 may be configured as a part of the function of this controller.

Further, the moving image processing apparatus of this embodiment is provided with a decoder 2 which performs an operation reverse to that of the encoder 1, and includes a buffer 23, an entropy decoding unit 24, an inverse quantization unit 25, IDCT operation unit 26, adder 27, predictor 2
8 and a moving picture interpolating section 29.

Compressed data is input to the buffer 23 from the encoder 1 and the like, and the buffer 23 smoothes the fluctuating information generation at a constant rate and outputs the image data to the storage device.

The entropy decoding unit 24 uses the buffer 2
The image data to be decoded inputted from 3 is decoded by performing a process reverse to the process of the entropy coding unit 16 and output to the dequantization unit 25, and at the same time a motion vector is extracted from a predetermined format and output. .

The dequantization unit 25 dequantizes the image data output from the entropy decoding unit 24 according to the determined quantization width, and outputs it to the IDCT operation unit 26.

The IDCT operation unit 26 performs an inverse DCT operation on the data inversely quantized by the inverse quantization unit 25, and outputs the operation result to the adder 27.

The adder 27 adds the prediction result to the output of the IDCT calculator 26 and outputs it to the predictor 28 and the like. A switch (not shown) is provided at the input / output stage of the predictor 28, and like the switch of the encoder 1, the signal path is switched according to the image mode and prediction mode from the controller.

The predictor 28 includes an entropy decoding unit 24.
Motion-compensated prediction is performed using the motion vector calculated in. In the motion compensation prediction in the predictor 28, a motion vector described later is used.

The moving picture interpolating section 29 interpolates the interpolated frame based on the motion vector calculated by the entropy decoding section 24 and outputs it as an image output to the outside. The moving picture interpolation unit 29 approximates the motion to a curve from the motion vector obtained from the adjacent transmission frames, and reconstructs the interpolation frame to be interpolated from the ratio of the distances between the adjacent transmission frames.

Next, the operation of this embodiment will be described.

2 to 4 are diagrams for explaining the frame interpolation operation of the moving image processing apparatus of this embodiment, and FIG. 2 is an interpolation frame between the transmitted frame P and the transmitted frame N. M1 and M2 are transmitted frame N and transmitted frame A
FIG. 7 is a diagram showing a frame interpolation method when there are two interpolation frames M3 and M4 between and.

In FIG. 2, MVP represents a motion vector from the transmitted frame P to the transmitted frame N, MVN represents a motion vector from the transmitted frame N to the transmitted frame A, and BVN.
P, BN, BA, and BMi indicate blocks on each frame. Further, the distance between the transmitted frame P and the transmitted frame N and the distance between the transmitted frame N and the transmitted frame A are respectively set to 1, and the distance between the transmitted frame P and the interpolation frame M2 is set to a, and the transmitted frame is set. The distance between N and the interpolation frame M3 is s, the distance between the transmitted frame P and the interpolation frame M1 is b, and the distance between the transmitted frame N and the interpolation frame M4 is t.

In FIG. 2, for the block BN of the transmitted frame N, motion vectors MVP and MVN based on similar blocks BP and BA are transmitted between the adjacent transmitted frame P and the transmitted frame A.

The frame interpolation process in this case is as follows. First, a curve passing through the three points of the start point and the end point of these motion vectors MVP and MVN is obtained, and a block is interpolated on the curve from the ratio of the distance between the frame to be interpolated and the transmitted frame. As the block, in FIG. 2, BM1, BM2,
BM3 and BM4 correspond. Further, the block to be interpolated may be, for example, a block in which BM1 and BM2 have an average of each pixel of BP and BN as a pixel.

Next, how to obtain the curve will be described. For example, the horizontal movement is as shown in FIG. 3 when viewed from above. When a block is interpolated on a curve as shown in FIG. 3, a two-dimensional orthogonal coordinate system is set from the motion vectors MVP and MVN with the block BN of the transmitted frame as the origin, as shown in FIG.

Specifically, each coordinate is represented by P (xP,
VP), M1 (xM1, VM1), M2 (xM2, VM2), M3
(XM3, VM3), M4 (xM4, VM4), A (xA, VA)
If a curve passing through A, P, O (origin) is obtained, xP, xM1 to xM4, xA are known from the ratio of the distances between frames, VP is the horizontal component of MVP, and VA is the horizontal component of MVN. Since it is a component, it is sufficient to obtain VM1 to VM4.

Since this curve passes through 3 points, 2
It is represented as a quadratic curve. For example, the quadratic curve is y = ax
If expressed by the equation 1 of the curve 2 + bx + c, it passes through A, P, and O, so that each of them is substituted into the equation 1 of the curve to solve the simultaneous linear equations, and the coefficients a, b, and c are obtained, VM1, VM
2, VM3, VM4 can be obtained. Further, even when the rotational movement is assumed, VM1, VM2, VM3, and VM4 can be obtained in the same manner from the equation 2 in which x2 + y2 + ax + by + c = 0.

Further, with respect to the movement in the vertical direction, the movement can be set on the curve by a similar method.

The position of the block to be interpolated is determined from these horizontal and vertical components, and the block is interpolated to form an interpolated frame.

The above series of operations is performed by the moving picture interpolating section 29 of FIG. 1 as post-processing after the information source sent from the transmission path is decoded by the decoding section 24.

That is, in the example using the predictive coding method based on the motion-compensated DCT shown in FIG.
The input image is subjected to frame thinning to be entropy coded, and the decoder 2 reproduces the decoded image on the receiving side. Also, it is encoded together with the image data by the encoder 1,
At the time of decoding, the motion vector as the motion information decoded is used (MVP and MVN in FIG. 2 correspond to this) to compose the interpolated image according to the above-described method, and output the image.

In the example using the predictive coding method based on the motion-compensated DCT shown in FIG. 3, the input side is subjected to block conversion into a predetermined format on the transmitting side, and inter-frame / intra-frame prediction is carried out according to the block conversion to perform entropy coding. The receiving side also decodes according to the format obtained by block conversion to obtain an output image.

As described above, the moving image processing apparatus according to the present embodiment has the motion detecting unit 11 for obtaining a motion vector from the image data subjected to frame thinning, and the frame thinning process for thinning the number of transmission frames on the transmitting side. Part 1
2, an adder 13, a DCT calculator 14, a quantizer 15, an entropy encoder 16, a buffer 17, an inter-motion prediction unit 18, and the like, and an encoder 1 and a buffer 2.
3, entropy decoding unit 24, inverse quantization unit 25, ID
The decoder 2 is composed of a CT calculator 26, an adder 27, a predictor 28, a motion vector interpolation unit 29 that interpolates an interpolation frame by using a motion vector, and outputs the result as an image output to the outside. The reference numeral 29 approximates the motion to a curve from the motion vector obtained from the adjacent transmission frames, and performs the process of reconstructing the interpolation frame to be interpolated from the ratio of the distances between the adjacent transmission frames. Of course, it is possible to cope with a curvilinear motion, a rotational motion and the like, and it is possible to accurately reproduce a missing frame on the receiving side.

Although the present embodiment is an example of frame interpolation when there are two interpolation frames M3 and M4 between the transmitted frame N and the transmitted frame A, the motion is approximated to a curve. Any frame interpolation can be applied as long as it reconstructs the decimated frames. Needless to say, if the motion is approximate to a straight line, the linear interpolation using the motion vector is performed as in the conventional example.

In this embodiment, the case where the motion vector is transmitted from the transmitting side has been described, but the present invention is not limited to this, and the motion vector may be detected at the receiving side, for example. In this case, with reference to FIG. 2, the most similar block within the set range between the adjacent transmitted frame P and the transmitted frame A is obtained for the block BN of the transmitted frame N. In FIG. 2, the block BP and the block BA correspond to this. Then, the movement amount is detected with respect to these blocks BP and BA. These movement amounts are used as the motion vectors MVP and MVN in FIG.

The relationship between the transmitted frame and the interpolated frame shown in this embodiment is an example, and it goes without saying that the number of frames, the position, the direction of movement, etc. are not limited to those in the above-mentioned embodiments.

Further, in this embodiment, the moving picture interpolating apparatus is applied to the compression of a low rate moving picture in a videophone apparatus or the like, but if it is an apparatus which accurately reproduces a missing frame on the receiving side. It is needless to say that the present invention can be applied to all devices, for example, a moving image interpolation device based on the MPEG algorithm.

Further, it goes without saying that the number and types of circuits and members constituting the above moving picture interpolating device are not limited to those in the above-mentioned embodiment, but may be realized by software (for example, C language). .

[0064]

According to the invention as set forth in claim 1, since the sent motion is approximated to a curve and the motion is approximated to the curve, the respective frames are reconstructed.
It is possible to cope with a curvilinear motion, a rotary motion, etc., and it becomes possible to accurately reproduce a missing frame on the receiving side.

According to the invention described in claims 2, 3 and 4,
Since the motion is approximated to a curve based on the detected motion vector, and the motion is approximated to the curve to reconstruct each frame, it is possible to cope with a curvilinear motion or a rotational motion. It becomes possible to accurately reproduce the frame that is missing on the side.

[Brief description of drawings]

FIG. 1 is a diagram showing a block configuration of a first embodiment of a moving image interpolation apparatus according to the present invention.

FIG. 2 is a diagram for explaining a frame interpolation operation of the moving image interpolation apparatus of the same embodiment.

FIG. 3 is a diagram for explaining a frame interpolation operation of the moving image interpolation apparatus of the same embodiment.

FIG. 4 is a diagram for explaining a frame interpolation operation of the moving image interpolation apparatus of the same embodiment.

FIG. 5 is a diagram showing a frame interpolation operation of a conventional moving image interpolation apparatus.

[Explanation of symbols]

 1 Encoder 2 Decoder 11 Motion Detection Unit 12 Frame Decimation Processing Unit 13, 21, 27 Adder 14 DCT Calculation Unit 15 Quantization Unit 16 Entropy Encoding Unit 17, 23 Buffer 18 Motion Interframe Prediction Unit 19 Inverse Quantization Part 20, 26 IDCT calculation part (IDCT) 22, 28 Predictor 24 Entropy decoding part 25 Inverse quantization part 29 Video interpolation part

Claims (4)

[Claims] 1. A moving image interpolating device for predictively interpolating a frame image between respective frames from moving image information comprising motion information between respective frames together with a plurality of frame image information, wherein the motion information between respective frames is nonlinearly approximated. A moving image interpolating device, which predictively interpolates a frame image between the frames. 2. A moving image interpolating device for predictively interpolating a frame image between each frame from moving image information consisting of a plurality of frame image information, detecting motion information of each inter-frame image and nonlinearly approximating this motion information. Then, the moving image interpolating device is characterized by predictively interpolating the frame images between the respective frames. 3. The motion information is a movement amount corresponding to a block that is the most similar among other frame image information with respect to a block of predetermined frame image information between each frame. The moving image interpolation apparatus according to claim 2. 4. The moving image interpolating apparatus according to claim 1, wherein the interpolation is predictive interpolation based on a ratio of temporal distances between frames.