JPH0937258A - Hierarchical warp type motion compensation predict method and its device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To avoid reduction in the compression rate while suppressing a heavy load. SOLUTION: A motion detector 103 divides image data of a current frame outputted from an image input device 101 and image data of a preceding frame stored in a frame memory 110 into blocks with cross reference to each other. Then while moving a noted moving point on a preceding frame and its coordinate moving point, an area deformed by the moving is compensated and the compensated area is compared with an area of a corresponding current frame to obtain an error and a coordinate minimizing the error is set to each moving point of the preceding frame. In this case, when the entire error between frames is not within a permissible range, a moving is added newly to reduce the block size and an error minimum position of the moving point added newly is continuously detected.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to warp-type motion compensation that performs motion compensation prediction by deforming (distorting) an image.

[0002]

2. Description of the Related Art In recent years, coding using interframe correlation (interframe coding) has been widely performed. In the case of inter-frame prediction or the like, the coding efficiency is often extremely reduced in the case of a moving image, and therefore a method called motion compensation prediction is widely used for coding a moving image.

FIG. 8 is a diagram showing an example of image shift by general motion compensation prediction. In FIG. 8, (a) is the image of the previous frame, and (b) is the image of the current frame. Motion compensation prediction, which is generally widely used, shifts the image of the previous frame entirely in the vertical and horizontal directions so that the prediction error from the image of the current frame is minimized. FIG. 8 shows a case where the image of the previous frame is shifted to the upper right, that is, from the coordinate position of the point P1 to the coordinate position of the point P2. The shift amount from the point P1 to the point P2 is the motion vector amount. The image of FIG. 8A and the image of FIG. 8B are different only in the eye part except the position of the object (king) in the image. Therefore, when this motion-compensated prediction is performed, the image of the current frame can be obtained from the image of the previous frame only by sending the error (difference) between the pixel values of the eye part and its periphery and the information on the above-mentioned motion vector amount. . If the two images match only by shifting the image, the current frame image can be obtained from the motion vector amount and the previous frame image.

The motion-compensated prediction described above can achieve very efficient coding when an object such as a rigid body is moving between frames. However, when the object is deformed between frames such as enlargement, reduction, or change in shape, the generated code amount becomes large, which causes a problem that efficient encoding cannot be performed. Warp-type motion-compensated prediction, which can avoid this inconvenience and can cope with various movements of an object, has recently attracted attention.

In the warp type motion compensation prediction, two images are divided into blocks each having the same number and the same shape.
Basically, the shape of a block of one image is fixed, and the block of the other image is deformed (including moved) so as to best match the fixed block. FIG. 9 is a diagram for explaining this warp type motion compensation prediction. FIG.
In (a), the image of the current frame is shown, and (b) is the image of the previous frame. Also, the broken lines in the figure represent the shapes of the blocks, and these blocks are shown at their corner points (A
The shape and the position are controlled by the position of the point (vertex) at which the broken lines such as a and a intersect.

Normally, in the warp type motion compensation prediction, as shown in FIG. 9 (a), the image of the current frame (and the previous frame) is divided into blocks, and the state when the image is projected on the corresponding block of the current frame ( The block of the previous frame shown in FIG. 8B is transformed so that the (compensated block) is the best match.

There are many image transformation methods, and it is not realistic to perform a very complicated process for finding a transformation method suitable for the transformation for each target image. Therefore, in the warp type motion compensation prediction, a method of dividing an image into blocks and transforming the blocks into processing units is generally adopted so that various images can be flexibly and easily dealt with. . In addition, before the block is deformed, a general shift of an image by detecting a motion vector is also generally performed.

In the above-mentioned transformation, when the vertices of a block including the object (king) to be transformed on the current frame are A, B, C, D, E, and F, these vertices A,
Vertex a on the previous frame corresponding to B, C, D, E, F,
It corresponds to detecting (setting) each coordinate position of b, c, d, e, f (in this way, the vertices of the current frame are represented by capital letters and the corresponding vertices of the previous frame are represented by lowercase letters). In this way, the projection of the block, which is performed after newly setting the coordinate position at each vertex of the previous frame, changes the shape from the block of the previous frame, which is the shape of the corresponding block of the current frame (this is the original (Equal to the shape of), which is equivalent to the motion compensation (correction) for the block in the warp type. The block projection is usually performed by moving the coordinate position of a point (pixel) in the block of the current frame, and after finding the coordinate position on the block of the previous frame corresponding to the coordinate position, It is performed by calculating the pixel value.

By the projection of this block, for example, the block indicated by the vertex abcd in the previous frame is projected onto the block indicated by the vertex ABCD in the current frame.
As a result, the pixels in the corresponding blocks between the previous frame and the current frame have a one-to-one correspondence, and it is possible to perform comparison for each block. If the projection of the block is expressed by another word, the deformed block is returned to the original shape by the shape as it is, which corresponds to the figure conversion.

In the warp-type motion-compensated prediction, the coordinate position of each apex at which the sum of error between blocks, that is, the difference between pixel values of corresponding pixels in the block is minimized is detected and set. In encoding image data using warp-type motion-compensated prediction, the current frame and the previous frame after projecting each block are associated with each other on the basis of the block, and the difference in pixel value of each corresponding pixel is encoded. To be done. At this time, information about the motion such as the motion vector amount and the displacement of each variation point from the reference position (the coordinate position initially set) is also encoded.

In warp-type motion-compensated prediction, the image is divided into blocks as described above, but at each vertex (a or b)
It is very difficult to detect the corresponding coordinate position of (). Therefore, the coordinate position of each vertex (a, b, etc.) is detected while moving the vertex in the vicinity of a certain position as the reference position (since this movement is performed, the vertex is hereinafter referred to as a variation point. ), Each time the movement is performed, the blocks are projected (corrected), the corresponding blocks between the frames are compared, and the coordinate position with the smallest error among the comparison results is detected (extracted). So to speak, trial and error methods are used.

The coordinate position detection and projection (figure conversion) of the fluctuation point will be further described with reference to FIGS. FIG. 10 is a diagram illustrating an actual state of the variation point. In FIG. 10, reference characters A to L respectively indicate the changing points (coordinate positions) on the current frame, and reference characters a to l respectively indicate the changing points (coordinate positions) on the previous frame.

First, as shown in FIG. 10 (a), the image is divided into blocks, and the motion vector between the previous frame and the current frame is detected based on this block. Determine the approximate position. This position serves as a reference position for moving the fluctuation point.

After the reference position is determined, the block deformed by the movement is projected for each variation point of the previous frame while sequentially moving around the reference position.
Contrast with the corresponding block in the current frame.

FIG. 11 is a diagram showing an example of a change in the shape of a block due to the movement of the variation point. As shown in FIG. 11, when the fluctuation point b is moved from b ′ to b ″, the fluctuation point b and the fluctuation points a, c, d, e, f, j,
The shape of the four blocks formed by k, l changes. Therefore, when the variation point b is moved, an octagon ajkklcfed and an octagon AJKL consisting of four blocks are moved.
Calculate the error between CFED.

One block is represented by a plurality of fluctuation points, and the corresponding coordinate position in the block is changed between the current frame and the previous frame simply by moving one of the fluctuation points. Therefore, the coordinate position that minimizes the error between frames (hereinafter, referred to as the minimum error position) is repeatedly detected for each variation point. In FIGS. 10 to 12, “′” is the minimum error position detected first, and “″” is the coordinate position detected second (this will be expressed in the same manner hereinafter). In the example of FIG. 11, the first error minimum position is detected for each variation point, the second minimum coordinate position is detected for the variation point a, and then the second variation coordinate point is detected for the variation point b. This shows a state in which the minimum coordinate position is being detected. Empirically, when the detection of the minimum error position is repeated twice for each variation point, the variation of the minimum error position converges to a practically sufficient level.

FIG. 12 is a diagram for explaining a projection method in warp type motion compensation prediction. In this projection, as described above, the shape of the block deformed by moving the fluctuation point is changed to the original shape (the corresponding block of the current frame) while maintaining the number of pixels inside and the relative pixel arrangement. Equivalent to the shape of). There are several possible projection methods, but affine transformation is often used.

It is known that this affine transformation converts a straight line into a straight line and preserves the ratio of points on the straight line. Points (pixels) in the block ABED of FIG.
P is a line AB, B indicating the area (boundary) of the block
The ratios when the coordinate positions are projected on E, AD, and DE are all 1: 3, but when affine transformation is used, this ratio is preserved even in the previous frame in which the block is deformed. That is, as shown in FIG. 6B, the position of the point (pixel) p on the previous frame corresponding to the point (pixel) P on the current frame is the line a "b" indicating the area (boundary) of the block,
The ratios when the coordinate positions are projected on b ″ e ′, d′ e ′, and a ″ d ′ are all 1: 3. This also applies to other points (pixels).

In this way, the coordinate position in the block of the previous frame corresponding to the coordinate position in the block of the current frame is obtained. In the current frame, one pixel corresponds to each coordinate position, but in the previous frame, in order to deform the block, the corresponding coordinate position of the previous frame obtained from the coordinate position of the current frame is expressed by an integer value. It usually does not match the coordinate position of the previous frame. Therefore, interpolation calculation is performed using the values of the pixels around the obtained coordinate position, and the pixel value at this coordinate position is calculated. This is the same at other obtained coordinate positions (pixels) in the block.

In this way, the coordinate positions in the block of the previous frame corresponding to the coordinate positions in the block of the current frame are calculated, and the pixel value of each of the calculated coordinate positions is calculated, thereby transforming. The projection of the block of the previous frame to the original shape, that is, the motion compensation (correction) of the block is performed.

[0021]

In the conventional warp type motion compensation prediction, the block size for dividing an image is fixed. Therefore, even if the minimum error position of each variation point is accurately detected, the generated code amount (compression rate) varies depending on the magnitude of the image change between frames.

When the generated code amount is small, the problem does not usually occur, but when the generated code amount is large, it takes a long time to transfer the image data for one frame, and the update interval of the image is extended to extend the object on the image. The movement of is unnatural.

In the warp type motion compensation prediction, as described above, the minimum error position of each fluctuation point is detected by a trial and error method, and the minimum error position is detected every time the fluctuation point is moved. Since the regions that are deformed by the movement are compared between images, many calculations must be performed before the error minimum position of one variation point is detected. The larger the number of fluctuation points, that is, the smaller the block size,
Since the motion compensation prediction can be performed with high accuracy, the generated code amount tends to be reduced. However, the smaller the block size, the more the amount of calculation increases according to the increased number of fluctuation points, and the heavier the load becomes.Therefore, it is necessary to keep the generated code amount low. Setting is not realistic. When the load becomes heavy, it is necessary to prepare a system capable of high-speed arithmetic processing, which leads to an increase in cost and an increase in size.

The object of the present invention is to reduce the weight of the load while
This is to avoid a reduction in compression rate.

[0025]

The hierarchical warp type motion compensation prediction method according to the first aspect of the present invention has the following processing steps.

First, in the first processing step, the variation points indicating the boundaries of the block areas are arranged on the two images in correspondence with each other according to the designated block size. In the second processing step, compensation is performed by obtaining the state in which the block area deformed by moving the variation point arranged on one of the two images is projected onto its original shape. In the third processing step, the error of the image information between the block area compensated in the second processing step and the corresponding block area on the other image of the two images is calculated. In the last fourth processing step, the minimum error position where the error calculated in the third processing step is the minimum is set for each variation point.

The hierarchical warp type motion compensation prediction method of the second aspect further includes the following fifth to seventh processing steps in addition to the above processing steps. First, in the fifth processing step, one image obtained by setting the minimum error position at each fluctuation point in the fourth processing step and applying the second processing step to each block region The error of the image information obtained by comparing with is calculated. In the sixth processing step, it is judged whether or not the error of the image information between the two images calculated in the fifth processing step is within a predetermined allowable range.
In the seventh processing step, if the error in the image information between the two images is not within a predetermined allowable range as a result of the determination processing in the sixth processing step, a block smaller than the block size currently specified The size is newly designated and the setting of the minimum error position of the variation point is repeated.

In the hierarchical warp type motion compensation prediction method of the second aspect, when the block size is newly designated in the sixth processing step, the minimum error position is set in the first processing step. The block size is changed by adding a new variation point to the existing variation point, and in the fourth processing step, the minimum error position is set only for the newly added variation point. Is desirable.

The warp type motion compensation prediction apparatus of the first aspect of the present invention comprises the following means. First, the variation point arranging unit arranges the variation points indicating the boundaries of the block areas on the two images in correspondence with each other according to the designated block size. The compensating means performs compensation by obtaining a state in which the block area deformed by moving the fluctuation point arranged by the fluctuation point arranging means on one of the two images is projected onto its original shape. . The inter-region error calculation means calculates the error of the image information between the block area compensated by the compensation means and the corresponding block area on the other image of the two images. The position setting means sets an error minimum position at which the error calculated by the inter-region error calculating means becomes minimum for each variation point.

The hierarchical warp type motion compensation prediction apparatus of the second aspect further comprises the following means in addition to the above means. The inter-image error calculating means compares one image with the other image obtained by the position setting means setting the minimum error position at each fluctuation point and the compensating means compensating each block area from that state. The error of the image information between the two images obtained by doing is calculated. The control means determines whether or not the error calculated by the inter-image error calculation means is within a predetermined allowable range, and if it is determined that the error is not within the allowable range, a block size smaller than the currently specified block size is set. Control is performed to newly specify and repeat the setting of the minimum error position of the variation point.

In the hierarchical warp type motion compensation predicting apparatus of the second aspect, when the control means newly specifies the block size, the variation point allocating means determines the variation point at which the minimum error position is set. , The block size is changed by newly adding and arranging the changing point, and the compensating means focuses only on the newly arranged changing point and moves the block,
It is preferable that the position setting means only sets the minimum error position of the newly arranged fluctuation point.

As the block size decreases, the amount of calculation required for prediction increases and the load increases, but on the other hand, more accurate prediction can be performed and the compression rate improves. By preparing a plurality of block sizes (hierarchizing the block sizes), it becomes possible to control the weight of the load and the compression rate.

If the block size is successively set to be small so that the error of the image information between the compensated one image and the other image falls within the allowable range, a high compression rate can be maintained. At the same time, it becomes possible to minimize the amount of calculation required for prediction.

The change of the block size corresponds to the change of the number of fluctuation points indicating the boundary of the block area. The change to reduce the block size will increase the number of fluctuation points that have been placed until then. This change to reduce the block size is performed by adding a new variation point to the variation point that has already been placed, so that it can be used for the variation point for which the minimum error position has already been set. It will be possible. If the minimum error position that has already been set is used, more accurate prediction can be performed simply by setting the minimum error position of the newly added fluctuation point, so it is newly executed when the block size is changed. The amount of calculation to be performed is reduced, and the load is further reduced.

[0035]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below.
A detailed description will be given with reference to the drawings. FIG. 1 is a block diagram of a system adopting an embodiment according to the present invention. This system is used for video telephones, video conferences, and the like. The systems used for these handle audio, but here, description of parts related to audio is omitted, and only parts related to image processing are described.

In FIG. 1, an image input device 101 is composed of, for example, a CCD (Charge Coupled Device) and its drive circuit, and captures image data. The image data captured by the image input device 101 is the subtractor 10
2 and the motion detector 103.

The subtractor 102 subtracts the image data output from the image input device 101 from the image data of the previous frame after the warp type motion compensation is performed as described later, and the subtraction is performed for each pixel data. Calculate the difference. The quantizer 104 quantizes the difference using a quantization table prepared in advance, for example, after performing DCT conversion to convert the difference into a DCT coefficient.

The variable-length encoder 105 divides the DCT coefficient, which has been quantized by the quantizer 104, into a DC component and an AC component, and encodes each group using the Huffman encoding method. . When this encoding is completed, the encoded image data (compressed data) generated by this
Is output to the buffer 106 at any time. This buffer 106
Is for absorbing fluctuations in the generated code amount (compressed data amount) for each image (frame) and the encoding speed (encoding can be performed at high speed if the images between frames are similar). In addition to the DCT coefficients quantized by the quantizer 104, the variable-length encoder 105 also receives the quantization table used when the image data is quantized, the Huffman code table, and the motion detector 103. Information on the movement of the object on the image is encoded by a predetermined method and output to the buffer 106.

The network interface section 107 is an ISDN.
(Integrated Service Digital Network), LAN (Lo
data is exchanged with a communication network such as a cal area network, the compressed data stored in the buffer 106 is sent to the communication network, the compressed data is received from the communication network, and the compressed data is stored in the buffer 112.

The inverse quantizer 108 decodes the image data output from the quantizer 104, that is, the quantized DCT coefficient into the original image data (difference), and outputs it to the adder 109. The adder 109 adds the image data output from the dequantizer 108 and the image data output from the motion compensator 111, which will be described later, and outputs the result to the frame
Output to the memory 110.

When the image data of the next frame is output from the image input device 101, the motion detector 103 refers to the image data of the immediately preceding frame stored in the frame memory 110 to detect the object in the image. The motion such as the overall movement and its deformation is detected, and the detection result is output to the motion compensator 111 and the variable length encoder 105. As described above, the variable length encoder 105 includes the motion detector 103.
Encodes the motion information sent by. The other motion compensator 111 compensates the image data stored in the frame memory 110 according to the motion information sent from the motion detector 103, and the subtractor 102 and the adder 109.
Output to

At this time, the image data output from the motion compensator 111 is obtained by compensating the image data of the previous frame by reflecting the motion of the image (object) detected by the motion detector 103. The same thing is generated on the receiving side by compensating the image data of the previous frame according to the transmitted motion information. As a result, the subtractor 1
02 outputs the difference between the image data of the current frame input from the image input apparatus 101 and the image data of the previous frame compensated by the motion information on the receiving side.

The other adder 109 is a motion compensator 111.
The image data input from the above is added to the difference output of the subtractor 102 input via the quantizer 104 and the inverse quantizer 108. Therefore, the same data as the final image data restored on the receiving side is stored in the frame memory 110.

As described above, in the transmission of image data, in order to grasp the state of the restored image data on the receiving side, the image of the previous frame for which motion compensation has been performed while performing feedback for restoring the encoded image data The difference from the data and the motion-compensated contents (motion information) are encoded and sent to the receiving side. On the other hand, when the compressed data (encoded image data) is received, the decompression (decoding) is performed by the procedure reverse to the compression of the image data.

The compressed data received by the network interface unit 107 from the communication network is temporarily stored in the buffer 112. The variable length decoder 113 restores the compressed data to the original DCT coefficient by the data representing the type of the Huffman code table sent together as the compressed data, and outputs it to the inverse quantizer 114. The dequantizer 108 is basically the same as the dequantizer 108, and based on the data indicating the type of the quantization table sent from the transmitting side,
Restore image data (difference). The image data restored by the inverse quantizer 114 is sent to the adder 115, where it is added to the image data output by the motion compensator 117.

The addition result of the adder 115 is output to the frame memory 118 and the image display device 116. The image display device 116 is, for example, a CRT (Cathode Ray Tub).
The image data input from the adder 115 is displayed on the display screen to reproduce the image.

The motion compensator 117 is a variable length decoder 11
3 according to the motion information sent from the frame memory 11
The image data of the previous frame stored in 8 is corrected. That is, on the transmitting side, the motion compensator 111 uses the subtractor 1
The image data of the frame memory 110 corrected according to the movement of the object on the image detected by the motion detector 103, which is output to 02, is reproduced. The addition of this image data and the image data of the difference output from the inverse quantizer 114 is performed by the adder 1
The image data of the current frame stored in the frame memory (corresponding to the frame memory 110 in FIG. 1) on the transmitting side is reproduced by the receiving side by causing the receiving side 15 to perform the processing. This reproduced image data is used as the image display device 1.
16 and is stored in the frame memory 118 and used to reproduce the image data of the next frame.

The above is the configuration of the present system and its general operation. Next, with reference to FIG. 2 to FIG. 7, the motion detection of the object between frames according to the present embodiment will be described in detail. In this motion detection, the motion detector 103
The image data stored in the memory 110 is compared with the image data input from the image input device 101. FIG. 2 is an operation flowchart of the motion detection process according to the present embodiment, which is executed by the motion detector 103.

First, in step 201, a variable grid_ is substituted with a value representing a block size for dividing an image.
Substitute 0 for no. In this embodiment, the block size is changed according to the magnitude of the error between the frames. This change in the block size corresponds to the change in the number of changing points arranged on the image.

FIG. 3 is a diagram for explaining the reference position of the variation point (coordinate position initially set) set for each layer. In FIG. 3, A _n , B _n , and C _n (n is an integer) indicate a changing point (coordinate position) on the image (screen) that is newly set (added) according to the value assigned to the variable grid_no. ing. Here, the broken lines in FIGS. 3A to 3C are shown to represent the positional relationship of the variation points added for each layer.

The value of the variable grid_no in FIG. 3A is 0.
In the case of, the broken line shows the shape of the block. When 1 is assigned to the variable grid_no, each variation point B shown in FIG. 3B is newly added to each variation point A shown in FIG. As a result, if the block A ₀ A ₃ A ₂ A ₁₁ in FIG.
_{Since 0} is newly set, this block A ₀ A ₃ A ₂
A ₁₁ is a triangle A ₂ A ₃ B ₀ , A ₂ B ₀ A ₁₁ , A ₁₁ B ₀ A
₀ and B ₀ A ₃ A ₀ are divided into four. As is apparent from this, when 1 is assigned to the variable grid_no, it is possible to perform the correction with higher accuracy than when the value of the variable grid_no is 0.

On the other hand, when 2 is assigned to the variable grid_no, the variation points A and B shown in FIGS. 3A and 3B are obtained.
In addition, each variation point C shown in FIG. 7C is newly added and set. Therefore, when all the changing points set when the value of the variable grid_no is 2 are shown in FIG.
As shown in (d).

When the value of the variable grid_no becomes 2, the
The block formed by the change points that have been set so far
KU is further divided into four. For example, the block of FIG.
A_ThreeB₁A₀B₀Is a triangle A_ThreeC_FourB₀, B₀C
_FourA₀, A_ThreeB₁C_Four, And C_FourB ₁A₀Divided into 4
You. As is clear from this, the variable grid_no
When 2 is substituted into, the value of the variable grid_no is
It is possible to perform correction with higher accuracy than in the case of 1.
is there.

As described above, in this embodiment, the variable gr
A value of 0 to 2 is assigned to id_no, and as the value assigned to it increases, the block size decreases and is changed stepwise. In this way, the provision of a plurality of block sizes is defined as block size hierarchization, and the warp type motion compensation prediction in which the block sizes are hierarchized is defined as hierarchical warp type motion compensation prediction. Also,
For each layer, for example, the value of the variable grid_no is 0
In this case, the value of the variable grid_no will be used for the expression such as hierarchy 0.

Returning to the description of FIG. 2, the processing of step 202 and subsequent steps following step 201 will be described. In step 202, by moving the currently set variation point on the previous frame and comparing it with the image of the current frame, the minimum error position of each variation point on the previous frame is detected and detected respectively. An error calculation process for calculating an error between frames when the minimum error position is set at each variation point is executed. The error between frames at this time is the total error between the current frame and the previous frame that is compensated by setting the minimum error position detected at the variation point (hereinafter, unless otherwise specified, the frame In the case of describing as the error between the frames, it means the error between the previous frame and the current frame after compensation). Then, the process proceeds to step 203. Details of this error calculation processing will be described later.

In step 203, the above step 202
Whether the inter-frame error obtained by executing the error process calculation process of is not sufficiently small, and the variable grid_no
It is determined whether the value of is 2. Whether or not the error between frames is sufficiently small is determined by whether or not the error is within a predetermined allowable range. If the error between the frames is within the allowable range or the value of the variable grid_no is 2, the determination is YES and the series of processes is ended.
Otherwise, the determination is no and step 20.
The process shifts to the process of No. 4.

In step 204, the variable grid_no is changed.
Increments the value of. When the value of the variable grid_no is incremented, the process returns to the error calculation process of step 202. In this case, in the error calculation process of step 202, the minimum error position of the variable point newly added by incrementing the value of the variable grid_no is detected, and the error between frames is calculated.

In this way, the block size is sequentially changed, that is, the hierarchy is changed so that the error between frames falls within the allowable range. As a result, it is possible to suppress a decrease in the compression rate for each frame.

In the above-described motion detection processing, when the images differ greatly between frames, the finally set block size is small and the number of fluctuation points is large, so the amount of calculation executed in the motion detection processing is large. . However, when the images are similar between frames, that is, when the block size is large and the error between them is small, the block size is not changed and the number of fluctuation points remains small, so that the calculation amount also remains small. Therefore, by changing the hierarchy as needed according to the magnitude of the error between the frames, it is possible to reduce the amount of calculation in units of frames as a whole. That is, it is possible to prevent the compression rate from decreasing while suppressing the magnitude of the load.

Next, the error calculation processing in step 202 will be described with reference to the operation flowchart shown in FIG. First, in step 401, the variable grid
The coordinate position (reference position) initially set at the newly added variation point in the hierarchy indicated by the value of _no is calculated. In this step 401, when the value of the variable grid_no is 0, the overall movement of the object between the previous frame and the current frame is detected. This movement is detected by moving (shifting) an object on the screen (mainly a person from the use of this system).
The amount, that is, the amount of motion vector is detected.

The motion vector amount is detected by, for example, defining a macroblock of interest (aggregation of a plurality of blocks shown in FIG. 3A) in the previous frame, and moving the macroblock in the xy direction while moving the macroblock on the current frame. This is done by finding the best matching movement amount by comparing with the area corresponding to that. The amount of movement of the macro block is detected within a range of ± 15 pixels in both the xy directions with respect to the original position, although it depends on the image transfer rate. By detecting the motion vector amount, the position of the block on the previous frame corresponding to each block on the current frame, that is, the correspondence of the position (coordinates) between the frames of each variation point shown in FIG. .

On the other hand, when the value of the variable grid_no is not 0, the reference position of each fluctuation point is compensated for by the compensation according to the detected motion vector amount at the coordinate position which is initially set for each fluctuation point to be added. To calculate.

In step 402 following step 401, each time the newly added fluctuation point is moved on the previous frame, the area deformed by the movement of the fluctuation point is compared with the corresponding area of the current frame. A minimum error position detection process for detecting the minimum error position of each newly added fluctuation point on the previous frame is executed. After this minimum error position detection process is completed, the process proceeds to step 403. Details of the minimum error position detection processing will be described later.

In step 403, the above step 402
When the coordinate position of each fluctuation point is set to the minimum error position detected in step 3, the error between the frames of each block deformed by the movement of each fluctuation point is summed. The error between the frames of the block at this time is an error obtained when the minimum error positions are set at all the fluctuation points indicating the boundaries of the block, though it is slightly different depending on the block compensation (projection) method. The sum of the errors is the error between the frames. After calculating the error between the frames, the series of processes is terminated and the process proceeds to step 203 in FIG.

Next, the minimum error position detection processing in step 402 will be described with reference to the operation flowchart shown in FIG. 5 and the explanatory diagrams shown in FIGS. 6 and 7.
This minimum error position detection processing is performed by the variable g as described above.
Move the variable point on the previous frame that is newly added (corresponding to the addition of the variable point even when 0 is substituted) with the change of the value of rid_no around the reference position calculated in step 401 of FIG. In this process, the minimum error position is detected for each fluctuation point by comparing the regions deformed by the movement of the fluctuation point between frames.

First, in step 501, a variable point to be aimed at is set from the newly added variable points. In the following step 502, the focused variation point is moved in a predetermined order. The movement of the fluctuation point is within a range of ± 3 pixels in the xy direction with the reference position as the center. After the movement of the change point is completed, step 503
Move to the processing of.

In step 503, the coordinate position of the pixel located around the moved variation point of the previous frame, which corresponds to the position of the current frame, is calculated, and the pixel value thereof is calculated. In the following step 504, the pixel value of the coordinate position (pixel) located around the variation point obtained by executing the process of step 503 is compared with the pixel value of the corresponding coordinate position (pixel) of the current frame. The error is calculated. The error here is a value obtained by accumulating the differences in pixel value for each pixel obtained by comparing all the pixels in the area deformed by the movement of the fluctuation point.

Here, the above steps 503 and 504.
The process will be further described. FIG. 6 is a diagram for explaining a region in which the coordinate position changes with the movement of the variation point, and the region is indicated by diagonal lines for each value of the variable grid_no, that is, for each layer. The variation point of interest is a in FIG.
₀ , b ₀ in the figure (b), c ₄ in the figure (c),
These fluctuation points are indicated by ○.

As shown in FIG. 6A, the variable grid_
When the value of no is 0, when the coordinate position of the variation point a ₀ is moved, the area (octagon) surrounded by the variation points a ₂ a ₃ a ₄ a ₁ a ₈ a ₉ a ₀ a ₁ is deformed. This area is compared between frames every time the variation point a ₀ is moved, and a value obtained by taking the difference between the pixel values for each pixel is calculated as an error. The coordinate position where the calculated error is the smallest is detected as the minimum error position.

When the value of the variable grid_no is 1,
As described above, the variation point is added in the block formed when the value of the variable grid_no is 0. Therefore, as shown in FIG. 6B, the region (block) a ₂ a ₃ a ₀ a ₁ is deformed by moving the coordinate position of the variation point b ₀ .

When the value of the variable grid_no is 2, the variable grid_no is the same as when the value of the variable grid_no is 1.
A variation point is added in the block formed when the value of _no is 1. Therefore, as shown in FIG. 6C, the region b ₀ a ₃ b ₁ a ₀ is deformed by the movement of the variation point c ₄ .

When the value of the variable grid_no is 0,
All the changing points set at that time are focused, and the minimum coordinate position is detected and set, respectively. However, in other cases, that is, when the values of the variable grid_no are 1 and 2, only the minimum error position of the newly added fluctuation point is detected and set. For example, among the variation points shown in FIG. 6C, only the variation point c ₄ is focused on, and the minimum error position is detected and set.

When the values of the variable grid_no are 1 and 2, the coordinate position of the variable point newly added is within the block whose shape has been determined by then. Even if the variation point added in the block is moved, the boundary of the block is not deformed, but only the inside is changed. As can be seen from the above, in the layering according to the present embodiment, after roughly compensating a large-sized block first, according to the error between the frames, the error is kept within the allowable range within the block. Are sequentially subdivided to compensate. As a result, even if the hierarchy is changed, the minimum error position of each fluctuation point is detected only once.

As described above, the block size is reduced without overlapping the detection of the minimum error position of the variation point with the change of the hierarchy, and when the error between the frames falls within the allowable range, the block is subdivided. End the transformation. By avoiding unnecessary change of the block size and avoiding duplication of detection of the minimum error position of the variation point, it is possible to maintain a high compression rate while further suppressing the calculation amount. It is also possible to suppress the fluctuation of the compression rate itself.

Corresponding coordinate positions between frames change as the blocks are deformed. In the warp type motion compensation prediction, as described above, while moving the coordinate position on the current frame, the coordinate position on the previous frame after deforming the block corresponding to the coordinate position is calculated, and the calculated coordinate is calculated. A method of calculating the pixel value of the position is widely used, and this embodiment also employs this method. Although there are some known methods for calculating the coordinate position on the previous frame corresponding to the coordinate position on the current frame, a triangular calculation method (affine transformation) is used in this embodiment.

The coordinate position is usually expressed by xy coordinates. The xy coordinates correspond to the arrangement of pixels, that is, one pixel corresponds to each coordinate position on the xy coordinates one to one. The upper left corner of the image (screen) is the coordinate position (0,0). In the triangular calculation method, the block is divided into triangles, and the coordinate position inside thereof is treated as a vector. The coordinate conversion by this method will be described with reference to FIG. 7.

FIG. 7A is a block of the previous frame, which is shown in FIG.
(B) is a block of the current frame corresponding to the block
It is. Also, in FIG. 7, the letters with numbers are
Represents a vector of change points and follows the
The letters should be uppercase in the current frame and lowercase in the previous frame.
ing. Here, the triangle A₀A_TenA_Two(A₀a
_Tena _Two) The internal position vector conversion will be used as an example.
You.

As described above, in warp type motion compensation prediction, the coordinate transformation is assumed to be an affine transformation. Therefore, the relationship with the position vector x in the triangle a ₀ a ₁₀ a ₂ corresponding to the arbitrary position vector X in the triangle A ₀ A ₁₀ A ₂ can be written as x = TX + k. Here, T is a parameter and k is a movement amount (constant). Since the positions of the fluctuation points between frames correspond to each other, the relationship between them can also be written as follows.

[0079]

[Formula 1] a ₀ = TA ₀ + k ··· 1 formula a ₁₀ = TA ₁₀ + k ··· 2 formulas a ₂ = TA ₂ + k ··· 3 formulas 1 −2 formulas, 2 −3 formulas From the formula, a ₀ −a ₁₀ = T (A ₀ −A ₁₀ ) ··· 4 formula a ₁₀ −a ₂ = T (A ₁₀ −A ₂ ) ·· 5 formula Further, 4 formula and 5 From the equation, (a ₀ −a ₁₀ , a ₁₀ −a ₂ ) = T (A ₀ −A ₁₀ , A ₁₀ −
A ₂ ) Therefore, T = (a ₀ −a ₁₀ , a ₁₀ −a ₂ ) (A ₀ −A ₁₀ , A ₁₀ −
A ₂ ) ⁻¹ One k is obtained by substituting T into equation 1 and k = a ₀ − (a ₀ −a ₁₀ , a ₁₀ −a ₂ ) (A ₀ −A ₁₀ ,
A ₁₀ −A ₂ ) ⁻¹ A ₀ Therefore, the relationship between the position vectors x and X in the example of FIG. 7 is as follows: x = (a ₀ −a ₁₀ , a ₁₀ −a ₂ ) (A ₀ −A ₁₀ , A ₁₀ −
^{_{A 2) -1 X + a 0}} - (a 0 -a 10, a 10 -a 2) (A 0
-A _10, A ₁₀ -A ₂₎ is ^-1 A _0.

In this way, the position vector x on the previous frame corresponding to the position vector X on the current frame is obtained. Normally, unlike the position vector X represented by an integer, both the xy components of the obtained position vector x are real numbers instead of integers. Therefore, according to the value of the decimal part of each component of the position vector (coordinate position) x, interpolation calculation of the pixel value is performed between the peripheral pixels of the position vector x, and the calculation result is the position vector (coordinate position). Let x be the pixel value.

The coordinate position of the position vector x is (x, y),
Each coordinate position around the coordinate position (x, y) is (x₀, Y
₀), (X₀+1, y₀), (X₀, Y₀+1), and
(X ₀+1, y₀+1), and the pixel value at each coordinate position is
It is shown by adding p to the coordinate position. As the relationship of each coordinate position
Is x₀≦ x ≦ x₀+1, y₀≤ y ≤ y₀At +1
You. Interpolation calculation under such an assumption is as follows.
Like.

[0082]

[Number 2] _{t 0 = (x 0 + 1} -x) · p (x 0, y 0) + (x-x 0) · p (x 0 + 1, y 0) t 1 = (x 0 + 1-x) _{· (x 0, y 0 +1} ) + (x-x 0) · p (x 0 + 1, y 0 +1) p (x, y) = (y 0 + 1-y) · t 0 + (y-y 0 )
T _{1 In} this way, the coordinate position on the previous frame after the block corresponding to the coordinate position on the current frame is deformed is calculated, and the pixel value corresponding to the coordinate position is calculated by the interpolation calculation.

The coordinate position on the previous frame and its pixel value are calculated in units of the area deformed by the moved variation point. When the pixel value at each coordinate position in this area on the previous frame is calculated, the difference in pixel value between frames is taken for each coordinate position and accumulated. The accumulated value of this difference is the error between the frames of the area. By calculating this error, the process of step 504 ends, and step 505
Move to the processing of.

In step 505 following step 504, it is determined whether or not the movement of the fluctuation point of interest is completed. As described above, in the present embodiment, the movement of the variation point is limited within the preset range, and when the variation point is moved to all coordinate positions within the range, the determination is Y.
It becomes ES and shifts to the processing of step 506. On the other hand, if not, the determination is no and step 50.
It returns to the process of 2.

In step 506, the coordinate position where the error is minimized while moving the variation point (minimum error position) is searched, and the retrieved coordinate position is set as the coordinate position of this variation point. After that, in step 507, it is determined whether or not the minimum error positions have been set at all the current fluctuation points. When the variation point for setting the coordinate position still remains, the determination becomes NO and the process returns to the process of step 501. After the variation point of interest is newly set here, the subsequent processes are performed. When the coordinate positions are redefined for all the variation points, the determination is YES and the series of processes is ended, and the process proceeds to step 203 in FIG.

As described above, the focus point is moved at any time, and the focus point is changed as soon as the movement is completed, whereby the coordinate position where the error is minimized is set to each focus point. It will be. The coordinate position (minimum error position) newly set at each variation point is sent to the receiving side in the form of a displacement vector from the first coordinate position (reference position), for example. On the one receiving side, after shifting the image of the previous frame according to the motion vector amount sent from the transmitting side,
Each block is compensated by correcting the coordinate position of each variation point according to the displacement vector of each variation point.

In the warp-type motion compensation prediction, the minimum error position of each variation point is repeatedly detected, but this repetition is not performed in this embodiment. This is mainly because the block size is made smaller as necessary, so that the generated code amount can be efficiently reduced without performing such repetition, and a reduction in compression rate can be avoided. This is because it is considered that it will be done. However, when it is desired to reduce the generated code amount more surely, the detection of the minimum error position of each fluctuation point is repeated, for example, when the error between frames is out of the allowable range even if the block size is minimized. You may do it.

Further, in this embodiment, when the total error between frames is not within the allowable range, the hierarchy is changed to reduce the block size as a whole, but some frames (images) are It may be divided into parts and the hierarchy may be changed for each part. This makes it possible to reduce the error by setting the block size to a small value only in the part where the error between frames is large,
Unnecessary calculation can be avoided for other small errors. Therefore, it is possible to further reduce the load for avoiding the reduction of the compression rate.

By the way, the block size has a disadvantage that the calculation amount increases as the size decreases, but the generated code amount decreases. The present invention can control both the calculation amount and the generated code amount (compression amount) in motion compensation prediction by layering block sizes. That is, when the emphasis is placed on the reduction of the generated code amount, the block size is set small,
When focusing on high-speed processing speed, a large block size may be set, and the block size can be flexibly applied according to the content to be focused on.

In recent years, due to the rapid improvement in the performance of the CPU, it is conceivable that the DSP can be made unnecessary and the system can be simplified by, for example, causing the CPU to execute the processing that was previously executed by the dedicated DSP. Is coming. Since the present invention controls the processing load in motion compensation prediction, it can also contribute to simplification of such a system.

[0091]

As described above, according to the present invention, since the block size is hierarchized, it is possible to control both the weight of the load and the compression rate when performing the warp type motion compensation prediction.

Further, since the block sizes are successively set to be small so that the error of the image information between the one image and the other image subjected to the compensation falls within the allowable range, it is possible to maintain a high compression rate. In addition, the amount of calculation required for prediction can be suppressed.

Furthermore, since the size of the block size is changed by newly adding a changing point to the already arranged changing point, the changing point is already set for the changing point for which the minimum error position has been set. Can be used. As a result, it is possible to reduce the amount of calculation that is newly executed in accordance with the change in the block size, and it is possible to further reduce the load.

[Brief description of drawings]

FIG. 1 is a block diagram of a system that employs an embodiment according to the present invention.

FIG. 2 is an operation flowchart of motion detection processing.

FIG. 3 is a diagram illustrating a reference position of a variation point set for each layer.

FIG. 4 is an operation flowchart of error calculation processing.

FIG. 5 is an operation flowchart of minimum error position detection processing.

FIG. 6 is a diagram illustrating an area in which a coordinate position changes as a variation point moves.

FIG. 7 is a diagram for explaining coordinate conversion.

[Fig. 8] Fig. 8 is a diagram illustrating an example of image shift based on general motion compensation prediction.

[Fig. 9] Fig. 9 is a diagram for describing warp-type motion-compensated prediction.

FIG. 10 is a diagram illustrating a state of an actual fluctuation point.

FIG. 11 is a diagram showing an example of a change in the shape of a block due to the movement of a change point.

FIG. 12 is a diagram illustrating a projection method in warp type motion compensation prediction.

[Explanation of symbols]

 101 image input device 103 motion detector 110 frame memory 111 motion compensator

Claims (6)

[Claims] 1. Fluctuations arranged on one of the two images by arranging fluctuation points indicating boundaries of block areas in correspondence with each other according to a designated block size. Compensation is performed by obtaining the state in which the block area deformed by moving the points is projected to its original shape, and the compensated block area and the corresponding block area on the other image of the two images. And a minimum error position at which the calculated error is minimized is set for each variation point. A hierarchical warp-type motion compensation prediction method. 2. Fluctuations arranged on one of the two images are arranged so that fluctuation points indicating boundaries of block areas are respectively arranged on the two images in correspondence with each other according to a designated block size. Compensation is performed by obtaining the state in which the block area deformed by moving the points is projected to its original shape, and the compensated block area and the corresponding block area on the other image of the two images. Error in the image information between and is set, the minimum error position where the calculated error is the minimum is set for each variation point, and the minimum error position is set for each variation point to compensate each block area. The above-mentioned one image obtained by performing
An error of the image information obtained by comparing with the other image is calculated, and it is determined whether the calculated error of the image information between the two images is within a predetermined allowable range. If the error in the image information between the two is not within a predetermined allowable range, a block size smaller than the block size currently being specified is newly specified, and the setting of the minimum error position of the fluctuation point is repeated. Hierarchical warp type motion compensated prediction method. 3. When the block size is newly designated, the block size is changed by newly adding a variation point to a variation point where the minimum error position is set, and the minimum error The hierarchical warp type motion compensation prediction method according to claim 2, wherein the position is set only for the newly added variation point. 4. A variable point arranging means for arranging variable points indicating boundaries of block areas in correspondence with each other on the two images according to a designated block size, and on one image of the two images. Compensating means for performing compensation by obtaining a state in which the block area deformed by moving the variation point arranged by the variation point arranging means is projected to its original shape, and the block compensated by the compensating means. An inter-region error calculating means for calculating an error in image information between a region and a corresponding block region on the other of the two images, and an error minimum for minimizing an error calculated by the inter-region error calculating means. A hierarchical warp-type motion-compensated prediction apparatus, comprising: position setting means for setting a position for each variation point. 5. The one image obtained by the position setting means setting the minimum error position at each variation point and the compensating means compensating each block area from that state, the other image. 2 obtained by comparing with
An inter-image error calculating means for calculating an error in image information between two images; and determining whether the error calculated by the inter-image error calculating means is within a predetermined allowable range, and determining that the error is not within the allowable range. In this case, a block size smaller than the currently specified block size is newly designated, and control means for executing control to repeat setting of the error minimum position of the variation point is further provided. The hierarchical warp type motion compensation prediction device according to claim 4. 6. When the control unit newly designates a block size, the variation point placement unit adds a new variation point to the variation point where the minimum error position is set, and arranges it. The block size is changed with, the compensating unit focuses on only the newly arranged fluctuation point to perform movement and compensation, and the position setting unit minimizes the error of the newly arranged fluctuation point. The hierarchical warp type motion compensation prediction device according to claim 5, wherein only the position is set.