JP2853974B2 - Hierarchical motion vector detection method and apparatus - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of detecting a "motion vector", which is one of the core technologies of moving image compression, and more particularly to a method of detecting a block by applying a block matching method to hierarchical image information and an apparatus therefor. Regarding improvement.

[0002]

[Background explanation]

“Moving Image Compression” In general, the amount of information of a moving image is much larger than that of a still image, and a compression technique that is more efficient than a still image is required for use in information transmission media and storage media. For example, Moving Picture Experts Groove (MPEG), one of the International Standards Committees on Image Compression,
up), the video encoding / decoding scheme (M
PEG1 and MPEG2; hereinafter collectively referred to as “MP
EG ") is the preferred compression technique for such applications. In the “temporal redundancy and prediction error” still image, compression is realized mainly using “spatial redundancy”. On the other hand, spatial redundancy is also used in video compression, but the use of "temporal redundancy" becomes more important. Temporal redundancy refers to the property that, out of a large number of frames constituting a moving image, frames that are temporally close to each other become very similar images. For example, in an animation, a smooth movement without a sense of incongruity is produced by gradually changing a continuous picture using this property. In this case, the temporally adjacent frames often have little or no difference. Thus, transmitting or storing only "different portions" (called prediction errors) of frames that are close in time (in other words, not transmitting or storing the same portion) results in a significantly higher compression ratio Can be obtained. So-called "inter-frame predictive coding" is performed to obtain the "inter-frame predictive coding" prediction error. this is,
This is a method in which image information of a frame to be compressed is predicted from another frame. In MPEG, three methods of “forward prediction”, “reverse prediction”, and “bidirectional prediction” are performed. Forward prediction is a temporally close "past frame"
And the prediction error between the "current frame" to be compressed and the backward prediction is to obtain the prediction error between the "future frame" that is close in time and the "current frame" to be compressed. Further, bidirectional prediction is to obtain a prediction error between an average value of “past frames” and “future frames” and “current frame”.

[0003] In MPEG, three kinds of picture types such as I, P and B are defined. I-pictures do not perform inter-frame prediction (no need for reference frames)
, A P picture is an image generated by forward prediction (using one past frame as a reference frame), and a B picture is obtained by forward prediction, backward prediction or bidirectional prediction (past and future). These are images that are generated (using two frames as reference frames), and are Intra-Picture, Predictive-Picture, and Bidirection, respectively.
ally Predictive-Picture. "Motion vector and motion compensation"
For example, considering a running vehicle, only the vehicle moves in parallel between frames that are temporally adjacent to each other, and the background hardly changes. For this reason, when an image of a moving part (vehicle) is viewed as a pixel value, that is, information of points (pixels) constituting the image represented by numerical values, the temporally adjacent frames (for convenience) are displayed. Only the position information (coordinate information) differs between the n-th frame and the (n-1) -th frame, and the other information (for example, color information and luminance information) hardly changes. Therefore, if "motion vectors", which are linear movement information, are given to the pixels of the moving portion (vehicle) of the (n-1) th frame, only the motion vector information is transmitted, and there is almost no need to transmit the prediction error information. A method of shifting an image signal using a motion vector is called “motion compensation”. Compared to simply using inter-frame prediction coding, the amount of transmission information can be greatly reduced, and the compression efficiency can be further increased. “Basic Principle of Motion Vector Detection by Block Matching Method” The basic principle of motion vector detection will be described with reference to FIGS. In FIG. 3, reference numeral 100 denotes a current frame (corresponding to the above-described n frame).
00 is divided into blocks of a predetermined size (for example, 16 × 16 pixels). Here, one of the blocks 101 is represented as a motion vector detection target block (hereinafter, “target block”). In FIG.
Reference numeral 102 denotes a temporally previous frame (corresponding to the above-described n-1 frame). In this frame 102, an area 103 for motion vector search (hereinafter, "search area") is set. The size of the search area 103 is larger than the target block 101, for example, the target block 101
Has a size of 47 × 47 pixels obtained by adding 16 pixels in the vertical and horizontal directions and adding 15 pixels in the + direction.

Now, in the search area 103, while shifting the block of interest 101 in the horizontal / vertical direction by a predetermined number of pixels (generally one pixel), the difference in pixel value between the overlapping pixels is sequentially obtained. At a stage where the inside of the region 103 has been completed, a partial region 104 in which the value representing the sum of the difference values of all the pixels (for example, the sum of the absolute values of the difference values or the sum of the squares of the difference values) is found is found. This partial area 10
Reference numeral 4 denotes an area having the highest correlation with the block of interest 101 of n frames, and has the same size as the block of interest 101 (here, 16 × 16 pixels). Therefore,
Since the two blocks 101 and 104 are matched, the partial area block 1 is located from the center of the block 101 of interest.
A vector 105 heading toward the center of the block 04 may be obtained, and this vector 105 may be set as the “motion vector” of the block 101 of interest. "Detection Accuracy of Block Matching Method" The block matching method basically aims at detecting a motion amount for realizing reduction of inter-frame prediction error entropy.
For this reason, with respect to the detection accuracy, the frequency of occurrence of detection errors is high, and the present situation is that accurate detection of the amount of movement in accordance with the original movement of the subject has not been achieved.

[0005]

2. Description of the Related Art Transactions of the Institute of Electronics, Information and Communication Engineers (D-II Vol.
J72-D-II No. 3 pp. 395-403 March 1989) describes a "motion vector detection method" in which a block matching method is applied to hierarchical pixel information. FIG.
It is a schematic diagram of the hierarchy pixel information described in the said paper. In FIG. 5, “0”, “h”, and “h + 1” are three representative hierarchical images, where 0 is the lowest image, h is an arbitrary middle image, and h + 1 is one of the images h. It is an image of one upper layer. Assuming that the image 0 is the original image, the pixel density of the image h is set to be lower than the pixel density of the image 0, and the pixel density of the image h + 1 is set to be further lower. According to a preferred example, the pixel density of image h + 1 is １ ／ of the pixel density of image h. This corresponds to an image obtained by halving the frequency band and the sampling frequency of the image h.

In such a hierarchical image, since the spatial resolution of the image of the upper layer is deteriorated compared to the lower layer, the upper layer and the lower layer have contradictory properties. That is, in the upper hierarchy, the movement of the subject can be captured globally (in other words, in the upper hierarchy, the movement of the subject cannot be captured in detail), while, in the lower hierarchy, the movement of the subject can be captured in detail. (In other words, the lower layer can only capture the movement of the subject locally).

Therefore, if the motion vector detected in the upper hierarchy is used as the initial deviation vector and the motion vector in the lower hierarchy is sequentially detected, the motion of the subject can be detected efficiently and with high accuracy. . In the block matching search in the hierarchy of the image h (hereinafter, the “h-th hierarchy”), the dependent block 200 (hereinafter, the “parent block”) in the hierarchy of the image h + 1 immediately above (hereinafter, the “h + 1-th hierarchy”)
Of the detection result V _{h + 1} (2V _{h + 1} ) is used as the initial displacement vector of the block of interest 201 in the h-th hierarchy. For example, matching evaluation function is defined by the following equation _{"S (u h ', v h} ') " it is detected by searching V _h 'having the smallest. At this time, the motion vector V _h of the block of interest 201 is V _h = V _h ′ + 2V, as shown in FIG.
given by _{h + 1} . The matching search range is, for example, 5 × 5 around the detection result 2V _{h + 1} = (2u _{h + 1} , 2v _{h + 1} ) of the parent block 200 in consideration of the correspondence between the layers of the hierarchical image. Set to a range.

[0008]

(Equation 1)

However, initial deviation point: 2V _{h + 1} = (2u _{h + 1} , 2v _{h + 1} ) Search point: V _h ′ = (u _h ′, v _h ′) Image h of previous frame: f _{t -1, h} (x, y) Image h of current frame: f _{t, h} (x, y) Detected motion vector: V _h = (u _h , v _h ) = (2u _{h + 1} + u _h ′, 2v _{h + 1} + v _h ′) In the hierarchical image of FIG. 5 described above, the resolution of the image h + 1 is １ ／ compared to the resolution of the image h. For this reason, the detection result V _{h + 1 of the} parent block 200 used as the initial deviation vector
Also, since only a half of the resolution of the detection object of a motion vector V _h, the detection result of the parent block 200 is doubled, after adjusting the resolution of the image h, it must be set as initial vector There is. For example, if V _hmax = 0, the motion vector V _h detected by matching of each layer is expressed by the following equation (see FIG. 6B).

[0010]

(Equation 2)

[0011]

By the way, in the technique of the above-mentioned paper, the number of pixels (the number of matching pixels) of the blocks in each layer is set so that the block size is subdivided substantially as the layers are lowered. Are all the same. However, optimizing the bottom block size may result in the top block size becoming too large,
There is a problem that the number of pictures having different movements contained in one block in the uppermost layer increases, resulting in a decrease in search accuracy. On the other hand, when the block size of the uppermost layer is optimized, the block size of the lowermost layer may become too small, and there is a problem that the mismatch in the lowermost layer occurs frequently and the search accuracy is also lowered.

[0012]

Therefore, the present invention can suppress both the increase in the number of pictures with different motions contained in one block on the upper layer side and the mismatch on the lower layer side, and apply the block matching method to the hierarchical image information. An object of the present invention is to improve the accuracy of motion vector search when applied.

[0013]

According to a first aspect of the present invention, there is provided a method according to the first aspect, wherein a lowermost first hierarchical image having a resolution equal to or close to that of an original image and a resolution stepwise with respect to the resolution of the first hierarchical image. And the motion vector of the parent block detected in the upper arbitrary layer is used as the initial deviation vector, and the target block of the next lower layer of the arbitrary layer is used. In the hierarchical motion vector detection method for detecting the motion vector of the parent block, when the layer to which the parent block belongs is other than a predetermined layer, the number of pixels of both blocks is the same, while when the layer is a predetermined layer, Is characterized in that the size ratio of each block to the belonging hierarchical image is the same.

According to a second aspect of the present invention, there is provided a first-layer memory for storing a lowest-order first-layer image having the same or close resolution as an original image, and a step-by-step process for the resolution of the first-layer image. The second to m-th hierarchical memories for storing the higher-order second to m-th hierarchical images whose resolution is reduced, and the motion vector of the parent block detected in the upper arbitrary layer as an initial displacement vector, A motion vector detecting means for detecting a motion vector of a block of interest of a layer one level below, and when the parent block belongs to a layer other than a predetermined layer, the number of pixels of both blocks is made the same. In the case of the hierarchy, the number of block pixels / size ratio setting means for equalizing the size ratio of both blocks to the respective belonging hierarchical images is provided.

[0015]

According to the present invention, the block setting method is appropriately switched at a predetermined hierarchical level, so that the number of pictures having different motions contained in one block on the upper layer side is increased, and the mismatch on the lower layer side is reduced. Are suppressed together.

[0016]

Embodiments of the present invention will be described below with reference to the drawings. 1 and 2 are principle configuration diagrams of an embodiment of a hierarchical motion vector detecting device according to the present invention. Note that the following description is an example in which the present invention is applied to a three-layer hierarchical image, but is not limited to this number of layers.

In FIG. 1, reference numerals 10 to 30 denote motion vector detecting units provided for each layer. The motion vector detecting unit 30 on the lower side of the drawing shows an image of the first layer (lowest layer) of a three-layer type image. , The motion vector detecting unit 20 at the center of the drawing corresponds to the image of the second layer (intermediate layer image) of the same image, and the motion vector detecting unit 10 at the upper side of the drawing corresponds to the third layer of the same image. It corresponds to the eye image (top layer image).
Hereinafter, for convenience of explanation, the motion vector detection unit 1 at the top of the drawing
0 is the “top layer motion vector detection unit”, the motion vector detection unit 20 at the center of the drawing is the “middle layer motion vector detection unit”,
Further, the motion vector detecting unit 30 on the lower side of the drawing is referred to as a “lowest layer motion vector detecting unit”.

The configuration of each unit is the same for the middle layer motion vector detection unit 20 and the lowest layer motion vector detection unit 30 and partially different for the top layer motion vector detection unit 10. That is, the uppermost layer motion vector detecting unit 10 includes two image memories 10a and 10b and a motion vector / evaluation value memory 1
0c, motion vector evaluator 10d and block setting device 1
0h, while the middle layer motion vector detecting section 20 (the lowest layer motion vector detecting section 30) includes two image memories 20a, 20b (30a, 30b), a motion vector / evaluation value memory 20c (30c), Motion vector evaluator 20
In addition to d (30d) and the block setting unit 20h (30h), a motion vector decompressor 20e (30e) is included in the configuration.

The motion vector detector 10 (20 or 3)
0) included in the two image memories 10a and 10b (20
a, 20b or 30a, 30b) for storing an image signal of the current frame (hereinafter, simply referred to as “current frame”), and the other for an image of a reference frame (for example, a temporally previous frame). Signal (hereinafter simply referred to as "reference frame")
), And is not particularly limited, but in the figure,
The left image memory 10a (20a or 30a) is used for storing the current frame, and the right image memory 10b (20b or 30b) is used for storing the reference frame.

Here, the image memories 10a, 10b, 20
The resolution (pixel density) of a, 20b, 30a, 30b is
It gradually decreases as going from the lower side to the upper side. That is, the images stored in the image memories 30a and 30b of the lowest-layer motion vector detection unit 30 are “high-resolution images”, and are stored in the image memories 20a and 20b of the middle-layer motion vector detection unit 20. The image is a “medium resolution image”, and the image stored in the image memories 10 a and 10 b of the top layer motion vector detection unit 10 is a “low resolution image”.

Therefore, the image memories 10a and 10b for storing the high-resolution images function as the "first hierarchical memory" described in the gist of the present invention, and the remaining image memories 20a and 20a for storing the medium-resolution images and the low-resolution images, respectively. 20b, 30a,
30b has a function as a "second to m-th hierarchical memory" described in the gist of the invention. The pixel density of each image is
It may be set so as to gradually decrease in the order of “high”, “medium”, and “low”. However, it is desirable to change at a constant magnification in terms of design easiness. For example, the pixel density of the high-resolution image is set to the same (or close) resolution as that of the original image, and the pixels of the high-resolution image are thinned out by 1/4 to obtain a medium-resolution image. What is thinned out by 1/4 may be used as the low resolution image.
In this case, the pixel density of the low resolution image is 1/16 that of the high resolution image.

Next, the function of each unit other than the image memory will be described. First, the motion vector evaluator 10d of the uppermost layer motion vector detection unit 10 calculates a target block image of the current frame read from one image memory 10a and a search area image read from the other image memory 10b. , A known method (refer to the block matching method described above)
The detected motion vector and the evaluation value are written into the motion vector / evaluation value memory 10c.

The motion vector estimator 20d of the middle layer motion vector detection section 20 (or the motion vector estimator 30d of the lowest layer motion vector detection section 30) outputs the current frame read from one image memory 20a (30a). The motion vector estimator 10d of the uppermost layer is different from the motion vector estimator 10d of the uppermost layer in that the motion vector of the target block is detected by a known method based on the image of the target block and the image of the search area read from the other image memory 20b (30b). Similar but first
The difference is that the motion vector of the parent block in the next higher layer is used as the initial displacement vector.

That is, the middle layer motion vector detecting section 20
The motion vector estimator 20d (or the motion vector estimator 30d of the lowest layer motion vector detection unit 30) has a parent of the next higher hierarchy that has been subjected to a predetermined decompression process by the motion vector decompressor 20e (30e). The block motion vectors 10f (20f)... 10f ′ or 20f ′ are input with the decompressed motion vectors..., And the motion vector evaluator 20d (30d) outputs the motion vector 10f ′ (20f ′) of the parent block. ), A search area (x) slightly larger than the area corresponding to the vector indicated by the vector is set, an image in the area is read from the image memory 20b (30b), and the vector having the highest evaluation in the area (x) is set. The difference is that Vx is found and the vector Vx and the evaluation value x of the search result are written to the motion vector / evaluation value memory 20c (30c). The search area x can be reduced to a necessary minimum size, and unnecessary search processing can be avoided. Accordingly, the motion vector estimators 20d and 30d of the middle layer and the lowest layer use the motion vector of the parent block detected in the higher layer as the initial displacement vector and calculate the motion vector of the target block of the next lower layer. As a “motion vector detecting means”.

Second, peripheral blocks in the same hierarchy
In that the motion vector is referred to. That is,
Already written in the vector / evaluation value memory 20c (30c).
The motion vector of the neighboring block
At least one read as 20g (30g)
Corresponding to the reference motion vector 20g (30g).
Search area (Y_i) (I is the identification number of the reference motion vector
) Is read out from the image memory 20b (30b).
You. Then, the area (Y_i)
And a motion vector Vy as a result of the search._i
And evaluation value y_iAnd the motion vector / evaluation value memory 20c
Write to (30c). Finally, the motion vector of the parent block
Evaluation value x when referring to the
Evaluation value y when referring to vector_i(Eg two
Y when reference is made to peripheral blocks₁, Y _Two) And the most
Also, the vector Vz with high evaluation is assigned to the block of interest of the hierarchical image.
Determined as a motion vector 20f (30f)
Output to the next lower layer,
Write to the evaluation value memory 20c (30c).

Here, the points of this embodiment will be described. One of the points of this embodiment is that the motion vector estimator 10d of the top layer and the motion vector estimator 20d of the middle layer
Is to set the number of pixels (the number of matching pixels) of the block to be evaluated to be the same. Such setting is performed by the block setting unit 10h of the top motion vector detecting unit 10 and the block setting unit 20h of the middle motion vector detecting unit 20. Another point of the present embodiment is that the size of the block evaluated by the motion vector estimator 30d in the lowermost layer is set to a predetermined size ratio (for the image stored in the image memory 30a in the same layer). The point is that it is set to be A) for convenience. Such setting is performed by the block setting unit 30h of the lowest motion vector detecting unit 30. Therefore, the block setting units 10h and 20 of each hierarchy
h and 30h have a function as “block pixel number / size ratio setting means” described in the gist of the invention.

Next, with reference to FIG. 2, a description will be given of the block setting of this embodiment. In FIG. 2, h _th represents a part of the image of the middle layer, h _{th + 1} represents a part of the image of the upper layer, and h _th−1 represents a part of the image of the lower layer. h _{th + 1} corresponds to the image in the uppermost layer image memory 10a in FIG. 1, h _th corresponds to the image in the middle layer image memory 20a in FIG. 1, and furthermore, h _th-1 Corresponds to the image in the image memory 30a of the lowest layer in FIG. The squares typically represent blocks set in each hierarchical image, 40 is a block of an upper layer image, 50 to 53 are blocks of a middle layer image, and 54 is a block of a lower layer image. The upper layer image block 40 and the middle layer image blocks 50 to 53 have the same number of pixels in the block, while the middle layer image block (for example, 50) and the lower layer image block 54 are Each belonging hierarchical image (block 50 is a middle layer image, block 54
Have the same size ratio with respect to the lower layer image. That is, the size ratio of the block 50 to the middle layer image is “A”, and similarly, the block 5 to the lower layer image is
The size ratio of No. 4 is also “A”.

That is, according to the block setting of FIG.
At a predetermined layer (middle layer), the setting can be switched between a block setting mainly based on the number of matching pixels (upper side) and a block setting mainly based on the size ratio (lower side). Therefore, while the number of matching pixels on the upper layer side (h _th , h _{th + 1} ) including the predetermined layer is set to an appropriate amount effective for texture suppression, the lower layer side (h _th , h _th) including the predetermined layer is set. _th-1 ) block size can be set to an appropriate size that is effective in suppressing mismatching, and the uniqueness of avoiding a decrease in the accuracy of motion vector search when the block matching method is applied to hierarchical image information can be avoided. The effect can be obtained.

Further, on the upper layer side including the predetermined layer,
Using the vector obtained in one block 40 of h _{th + 1} as an initial displacement vector, four blocks 50 to 53 of h _th
Are collectively searched, and since the search range can be read collectively, the effect of reducing the memory access amount can be expected, and the inconvenience that the block size of the uppermost layer becomes too large can be solved.

In the above embodiment, the block setting method is switched in the middle layer, but may be performed in the upper layer. Further, in the embodiment, an example is shown in which the present invention is applied to three-layer hierarchical image information, but the present invention is not limited to this. It may be a multilayer of four or more layers. When applied to a multi-layer image having four or more layers, a predetermined layer is not always an intermediate layer image.

[0031]

According to the present invention, the block setting method can be switched at a predetermined hierarchical level. Specifically, the upper layer can switch to a setting method effective for suppressing an increase in the number of patterns having different motions contained in one block, and the lower layer can switch to a setting method effective for suppressing mismatch. . Therefore, it is possible to suppress both the increase in the number of patterns with different motions contained in one block on the upper layer side and the mismatch on the lower layer side, and to search for a motion vector when the block matching method is applied to the hierarchical image information. Accuracy can be improved.

[Brief description of the drawings]

FIG. 1 is an overall conceptual diagram of a principal part of an embodiment.

FIG. 2 is a conceptual diagram of a block setting in each layer according to an embodiment.

FIG. 3 is a basic principle diagram (part 1) of motion vector detection.

FIG. 4 is a basic principle diagram (part 2) of motion vector detection.

FIG. 5 is a schematic diagram of a conventional hierarchical image.

FIG. 6 is a conceptual diagram of motion vector detection in a hierarchical image.

[Explanation of symbols]

10a, 10b: Image memory (first hierarchical memory) 20a, 20b, 30a, 30b: Image memory (second to second memory)
20d, 30d: motion vector evaluator (motion vector detection means, motion vector reference means, second motion vector reference means) 10h, 20h, 30h: block setting device (number of block pixels / size ratio setting means) )

──────────────────────────────────────────────────の Continued on the front page (72) Inventor Koichi Oyama 4-36-19 Yoyogi, Shibuya-ku, Tokyo Inside Graphics Communication Laboratories Co., Ltd. (72) Inventor Maki Sato, Shibuya-ku, Tokyo 4-36-19 Yoyogi Inside Graphics Communication Laboratories Co., Ltd. (72) Inventor Yasuhiro Yamada 4-36-19 Yoyogi Shibuya-ku, Tokyo Graphics Communication Laboratories Co., Ltd. (56) References JP-A-3-4686 (JP, A) JP-A-3-247190 (JP, A) JP-A-6-78292 (JP, A) JP-A-7-107486 (JP, A) (58) Fields surveyed (Int.Cl. ⁶ , DB name) H04N ^7/ 24-7/68

Claims (2)

(57) [Claims] 1. A lower-order first hierarchical image having the same or close resolution as an original image, and higher-order second to m-th hierarchical images whose resolution gradually decreases with respect to the resolution of the first hierarchical image. And using the motion vector of the parent block detected in the arbitrary layer on the upper side as an initial deviation vector, and detecting the motion vector of the block of interest one layer below the arbitrary layer in the hierarchical motion vector detection method. When the parent block belongs to a layer other than the predetermined layer, the number of pixels of both blocks is the same. On the other hand, when the parent layer belongs to the predetermined layer, the size ratio of both blocks to the respective layer images is the same. A hierarchical motion vector detection method, characterized in that: 2. A first-layer memory for storing a lowest-order first-layer image having a resolution that is the same as or close to that of an original image, and an upper-layer memory whose resolution gradually decreases with respect to the resolution of the first-layer image. A second to m-th hierarchical memories for storing second to m-th hierarchical images, and a motion vector of a parent block detected in an upper hierarchical layer as an initial deviation vector, and a motion vector of a hierarchy one level lower than the arbitrary hierarchical layer. A motion vector detecting means for detecting a motion vector of the block of interest; and, when the layer to which the parent block belongs is other than a predetermined layer, the number of pixels in both blocks is the same. And a block pixel number / size ratio setting unit for setting the size ratio of each block to the belonging hierarchical image to be the same.