JPH08265772A - Method for detecting hierarchical motion vector and device therefor - Google Patents

Abstract

PURPOSE: To suppress both increase of the number of the patterns of different motions provided in one block on a high order layer side and mismatching on a low order layer side and to improve the accuracy of motion vector searching in the case of applying a block matching method in hierarchical picture information. CONSTITUTION: In this method for using the first hierarchy pictures of a lowest order provided with resolution identical or close to source pictures and second to (m)th hierarchy pictures on a high order side whose resolution is lowered stepwise from the resolution of the first hierarchy pictures and detecting the motion vector of the block under consideration of a hierarchy one order lower than an optional hierarchy on the high order side with the motion vetor of a master block detected in the optional hierarchy on the high order side as an initial deflection vector, the picture element numbers of both blocks are made to be identical when the belonging hierarchy of the master block is other than a prescribed hierarchy and size ratios to the respective belonging hierarchy pictures of both blocks are made to be identical when it is the prescribed hierarchy.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a "motion vector" detecting method, which is one of the core techniques for moving image compression, and more particularly to a detecting method and apparatus for applying a block matching method to hierarchical image information. Regarding improvement.

[0002]

[Background explanation]

"Moving image compression" In general, the amount of information of a moving image is much larger than that of a still image, and a compression technique that is more efficient than a still image is required when used as an information transmission medium or a storage medium. For example, MPEG (Moving Picture Experts Groove), which is one of the international standardization committees for image compression.
video coding / decoding method (M
PEG1 and MPEG2; hereinafter collectively referred to as "MP
EG ") is the preferred compression technique for these applications. "Temporal Redundancy and Prediction Error" In still images, compression is mainly achieved by using "spatial redundancy". On the other hand, even in video compression, spatial redundancy is used, but the use of "temporal redundancy" becomes more important. Temporal redundancy refers to the property that, among a large number of frames that make up a moving image, frames that are close in time have very similar images. For example, animation makes use of this property to change a series of pictures little by little to create a smooth motion without discomfort. In this case, frames that are temporally adjacent to each other often have little difference to some extent. Therefore, if only the "different parts" of frames that are close in time (called prediction errors) are transmitted or stored (in other words, the same parts are not transmitted or stored), the result is a considerably higher compression rate. Can be obtained. "Inter-frame predictive coding" So-called "inter-frame predictive coding" is performed to obtain a prediction error. this is,
This is a method of predicting image information of a frame to be compressed from another frame. In MPEG, three types of "forward prediction", "reverse prediction" and "bidirectional prediction" are performed. Forward prediction is “past frame” that is close in time
And getting the prediction error between the "current frame" that you want to compress, and backward prediction that you get the prediction error between the "future frame" and the "current frame" that you want to compress. Moreover, bidirectional prediction is to obtain the prediction error between the average value of "past frame" and "future frame" and "current frame".

In MPEG, three types of picture types such as I, P and B are defined. I-picture does not perform inter-frame prediction (no reference frame is required)
, A P picture is generated by forward prediction (using one past frame as a reference frame), and a B picture is generated by forward prediction, backward prediction, or bidirectional prediction (past and future predictions). Images generated using two frames as reference frames), Intra-Picture, Predictive-Picture, and Bidirection, respectively.
It is an acronym for ally Predictive-Picture. "Motion vector and motion compensation" Motion part in video,
For example, when considering a rushing vehicle, only the vehicle moves in parallel between adjacent frames in time, and the background hardly changes. Therefore, when an image of a moving portion (vehicle) is viewed as a pixel value ... Information indicating points (pixels) forming the image by a numerical value .. The only difference between the n frame and the n-1 frame is position information (coordinate information), and other information (for example, color information or luminance information) is hardly changed. Therefore, if a "motion vector", which is linear movement information, is given to the pixels of the moving portion (vehicle) of the n-1 frame, it is unnecessary to send only the motion vector information and the prediction error information. A method of shifting an image signal using a motion vector is called "motion compensation". Compared to simply using interframe predictive coding, the amount of transmission information can be greatly reduced, and the compression efficiency can be further improved. [Basic Principle of Motion Vector Detection by Block Matching Method] The basic principle of motion vector detection will be described with reference to FIGS. In FIG. 3, 100 is the current frame (corresponding to the n frame described above), and this frame 1
00 is divided into blocks of a predetermined size (for example, 16 × 16 pixels). Here, one of the blocks 101 is represented as a motion vector detection target block (hereinafter, “block of interest”). In FIG.
Reference numeral 102 denotes a temporally previous frame (corresponding to the n-1 frame described above), and an area (hereinafter, “search area”) 103 for motion vector search is set in this frame 102. The size of the search area 103 is larger than that of the target block 101.
It has a size of 47 × 47 pixels in which 16 pixels are added in the negative and horizontal directions and 15 pixels are added in the positive direction.

Now, in the search area 103, when the target block 101 is shifted in the horizontal / vertical direction by a predetermined number of pixels (generally, one pixel), the pixel value differences between the overlapping pixels are sequentially obtained. At the stage where the inside of the area 103 is circled, the partial area 104 in which the value representing the sum of the difference values of all pixels (for example, the sum of the absolute values of the difference values or the sum of the squares of the difference values) is minimum is found. This partial area 10
A region 4 has the highest correlation with the target block 101 of n frames, and its size is the same as that of the target block 101 (here, 16 × 16 pixels). Therefore,
Since the two blocks 101 and 104 are matched, the partial area block 1 from the center of the target block 101
A vector 105 directed to the center of 04 is obtained, and this vector 105 may be set as the “motion vector” of the block 101 of interest. "Detection accuracy of block matching method" The block matching method basically aims at motion amount detection for reducing interframe prediction error entropy.
Therefore, in terms of detection accuracy, the frequency of detection errors is high, and it is the current situation that the amount of motion is not accurately detected according to the original motion of the subject.

[0005]

2. Description of the Related Art IEICE Transactions (D-II Vol.
J72-D-II No.3 pp.395-403 March 1989) describes a "motion vector detection method" in which a block matching method is applied to hierarchical pixel information. Figure 5
It is a schematic diagram of hierarchical pixel information described in the above paper. In FIG. 5, “0”, “h”, and “h + 1” are three representative hierarchical images, 0 is the bottom layer image, h is an arbitrary middle layer image, and h + 1 is one of the images h. It is an image of the upper layer. When the image 0 is the original image, the pixel density of the image h is lower than the pixel density of the image 0, and the pixel density of the image h + 1 is set to be further lower. According to a preferred example, the pixel density of image h + 1 is 1/2 of the pixel density of image h. This corresponds to an image obtained by halving the frequency band and the sampling frequency of the image h.

[0006] In such a hierarchical image, the spatial resolution of the image in the upper layer is deteriorated as compared with that in the lower layer, so that the upper layer and the lower layer have contradictory properties. That is, the movement of the subject can be comprehensively captured in the upper layer (in other words, the movement of the subject cannot be captured in detail in the upper layer), while the movement of the subject can be captured in detail in the lower layer. (In other words, the motion of the subject can only be locally captured in the lower layer).

Therefore, if the motion vector detected in the upper layer is used as the initial displacement vector and the motion vector in the lower layer is sequentially detected, the motion of the subject can be detected efficiently and with high accuracy. . The block matching search in the hierarchy of the image h (hereinafter “h-th hierarchy”) is performed by searching for the dependent block 200 (hereinafter “parent block”) of the next higher hierarchy of the image h + 1 (hereinafter “h + 1 hierarchy”).
(2V _{h + 1} ) obtained by doubling the detection result V _{h + 1} of (2V _{h + 1} ) is performed as the initial displacement vector of the target block 201 in the h-th layer. For example, the matching evaluation function “S (u _h ′, v _h ′)” defined by the following equation is searched for and detected for V _h ′. At this time, the motion vector V _h of the block of interest 201 is V _h = V _h ′ +2 V as shown in FIG.
Given by _{h + 1} . Further, the matching search range is, for example, 5 × 5 with the detection result 2V _{h + 1} = (2u _{h + 1} , 2v _{h + 1} ) of the parent block 200 as the center, in consideration of the correspondence between layers of the hierarchical image. Set to range.

[0008]

[Equation 1]

However, initial deviation point: 2V _{h + 1} = (2u _{h + 1} , 2v _{h + 1} ) Search point: V _h ′ = (u _h ′, v _h ′) Image h: f _{t of the} previous frame _{-1, h} (x, y) Current frame image h: f _{t, h} (x, y) Detected motion vector: V _h = (u _h , v _h ) = (2u _{h + 1} + u _h ′, 2v _{h +1} + v _h ′) In the hierarchical image shown in FIG. 5, the resolution of the image h + 1 is 1/2 that of the image h. Therefore, the detection result V _{h + 1 of the} parent block 200 used as the initial displacement vector
However, since the resolution is only 1/2 of the motion vector V _h to be detected, it is necessary to double the detection result of the parent block 200 and match it with the resolution of the image h, and then set this as the initial displacement vector. There is. For example, when V _hmax = 0 is set, the motion vector V _h detected by the matching of each layer becomes as shown in the following expression (see FIG. 6B).

[0010]

[Equation 2]

[0011]

By the way, in the technique of the above-mentioned paper, the number of pixels (the number of matching pixels) of the blocks of each layer is so divided that the block size is subdivided substantially as the layer is lowered. Are all the same. However, if the block size of the bottom layer is optimized, the block size of the top layer may become too large,
As a result of the increase in the number of pictures with different movements included in one block in the uppermost layer, there is a problem in that the search accuracy decreases. On the other hand, if the block size of the uppermost layer is optimized, the block size of the lowermost layer may become too small this time, and there are many mismatches in the lowermost layer, which also lowers the search accuracy.

[0012]

[Object] Therefore, the present invention can suppress both an increase in the number of pictures having different movements included in one block on the upper layer side and a mismatch on the lower layer side, and a block matching method is applied to hierarchical image information. The purpose is to improve the accuracy of motion vector search when applied.

[0013]

According to a first aspect of the present invention, a lowermost first hierarchical image having a resolution equal to or close to that of an original image and a resolution of the first hierarchical image are gradually increased. And the second to m-th layer images on the upper side, the motion vector of the parent block detected on the arbitrary layer on the upper side is used as the initial displacement vector, and the block of interest of the layer one lower than the arbitrary layer. In the hierarchical motion vector detection method for detecting the motion vector of the block, if the parent block belongs to a layer other than a predetermined layer, the number of pixels of both blocks is the same, while if the layer is a predetermined layer, both pixels have the same number of pixels. The size ratio of each block to each belonging hierarchy image is the same.

According to a second aspect of the present invention, there is provided a first hierarchy memory for storing the lowest first hierarchy image having a resolution equal to or close to that of the original image, and the resolution of the first hierarchy image is stepwise. The second to mth layer memories for storing the images of the second to mth layers on the upper side where the resolution is lowered, and the motion vector of the parent block detected in the arbitrary layer on the upper side are used as the initial displacement vector A motion vector detecting means for detecting a motion vector of a block of interest in the next lower layer, and if the parent block belongs to a layer other than a predetermined layer, the numbers of pixels of both blocks are made the same, In the case of the hierarchy, the block pixel number / size ratio setting means for making the size ratio of both blocks to each belonging hierarchy image the same is provided.

[0015]

According to the present invention, the block setting method is appropriately switched with a predetermined layer as a boundary, and an increase in the number of pictures having different movements contained in one block on the upper layer side and a mismatch on the lower layer side. Are both suppressed.

[0016]

Embodiments of the present invention will be described below with reference to the drawings. 1 and 2 are principle configuration diagrams of an embodiment of a hierarchical motion vector detection device according to the present invention. Note that the following description is an example in which the image is applied to a hierarchical image of three layers, but the number of layers is not limited.

In FIG. 1, 10 to 30 are motion vector detection units provided for each layer, and the motion vector detection unit 30 on the lower side of the drawing is the first layer image (the lowest layer) of the three layered images. Image), the motion vector detection unit 20 in the center of the drawing corresponds to the second layer image (intermediate layer image) of the same image, and the motion vector detection unit 10 on the upper side of the drawing corresponds to the third layer of the same image. It corresponds to the eye image (top layer image).
Hereinafter, for convenience of description, the motion vector detection unit 1 on the upper side of the drawing
0 is the “uppermost layer motion vector detecting unit”, the motion vector detecting unit 20 at the center of the drawing is the “middle layer motion vector detecting unit”,
Further, the motion vector detecting unit 30 on the lower side of the drawing is referred to as a "lowermost layer motion vector detecting unit".

Regarding the configuration of each unit, the middle layer motion vector detecting unit 20 and the lowermost layer motion vector detecting unit 30 are the same, and the uppermost layer motion vector detecting unit 10 is partially different. That is, the uppermost layer motion vector detection unit 10 includes two image memories 10a and 10b and a motion vector / evaluation value memory 1
0c, motion vector evaluator 10d and block setter 1
0h, on the other hand, the middle layer motion vector detection unit 20 (lowermost layer motion vector detection unit 30) includes two image memories 20a, 20b (30a, 30b), a motion vector / evaluation value memory 20c (30c), Motion vector evaluator 20
In addition to d (30d) and block setter 20h (30h), a motion vector decompressor 20e (30e) is included in the configuration.

The motion vector detector 10 (20 or 3)
0) two image memories 10a, 10b (20)
a, 20b or 30a, 30b) is for storing the image signal of the current frame (hereinafter, simply referred to as "current frame"), and the other is the image of the reference frame (for example, the frame immediately preceding in time). Signal (hereinafter simply referred to as "reference frame")
It is for storing, and there is no particular limitation, but in the figure,
The left image memory 10a (20a or 30a) is used for storing the current frame, and the right image memory 10b (20b or 30b) is used for storing the reference frame.

Here, the image memories 10a, 10b, 20
The resolution (pixel density) of a, 20b, 30a, 30b is
It gradually decreases from the lower side to the upper side. That is, the images stored in the image memories 30a and 30b of the lowermost layer motion vector detection unit 30 are “high resolution images”, and are also stored in the image memories 20a and 20b of the middle layer motion vector detection unit 20. The image is a "medium resolution image", and the images stored in the image memories 10a and 10b of the uppermost layer motion vector detection unit 10 are "low resolution images".

Therefore, the image memories 10a and 10b for storing high resolution images function as the "first layer memory" described in the gist of the invention, and the remaining image memories 20a for storing medium resolution images and low resolution images, 20b, 30a,
30b has a function as a "second to m-th hierarchical memory" described in the gist of the invention. The pixel density of each image is
It may be set so as to gradually decrease in the order of “high”, “medium”, and “low”, but it is desirable to change at a constant magnification from the viewpoint of design ease. For example, the pixel density of the high-resolution image is set to the same (or close to) the resolution of the original image, the pixels of this high-resolution image are thinned out by 1/4, and the medium-resolution image is obtained. A low resolution image may be obtained by thinning out 1/4.
In this case, the pixel density of the low resolution image is 1/16 of that of the high resolution image.

Next, the function of each unit other than the image memory will be described. First, the motion vector evaluator 10d of the uppermost layer motion vector detection unit 10 is based on the image of the target block of the current frame read from one image memory 10a and the image of the search area read from the other image memory 10b. , A known method (see the block matching method described above)
The motion vector of the block of interest is detected by, and the detected motion vector and evaluation value are written in the motion vector / evaluation value memory 10c.

The motion vector evaluator 20d of the middle layer motion vector detecting unit 20 (or the motion vector evaluator 30d of the lowermost layer motion vector detecting unit 30) of the current frame read from one image memory 20a (30a). Based on the image of the block of interest and the image of the search area read from the other image memory 20b (30b), the motion vector of the block of interest is detected by a known method, and the motion vector evaluator 10d of the uppermost layer is used. Similar, but first
The difference is that the motion vector of the parent block in the next higher layer is used as the initial displacement vector.

That is, the middle layer motion vector detecting section 20.
The motion vector evaluator 20d (or the motion vector evaluator 30d of the lowest layer motion vector detection unit 30) of FIG. The motion vector 10f (20f) ... 10f 'or 20f' of the block is inputted as the motion vector after the expansion processing, and the motion vector evaluator 20d (30d) determines that the motion vector 10f '(20f' of the parent block). ), A search area (x) that is slightly larger than the area corresponding to the vector indicated by () is set, the image in that area is read from the image memory 20b (30b), and the vector with the highest evaluation in the area (x) is set. The difference is that Vx is searched for, and the vector Vx of the search result and the evaluation value x are written in the motion vector / evaluation value memory 20c (30c). The search area x can be minimized in size, and useless search processing can be avoided. Therefore, the motion vector evaluators 20d and 30d of the middle layer and the lowest layer use the motion vector of the parent block detected in the upper layer as the initial displacement vector to calculate the motion vector of the target block in the lower layer. It has a function as a "motion vector detecting means" for detecting.

Secondly, peripheral blocks in the same hierarchy
The difference is that the motion vector of is referred to. That is, motion
Already written in the vector / evaluation value memory 20c (30c).
The reference vector is used as the reference
Read at least one as a vector 20g (30g)
Corresponding to this reference motion vector 20g (30g)
Search area (Y_i) (I is the identification number of the reference motion vector
No.) image from the image memory 20b (30b)
You Then, the area (Y_i) Of the highest-rated vector
Search for V le, and the motion vector Vy of the search result_i
And evaluation value y_iAnd motion vector / evaluation value memory 20c
Write to (30c). Finally, move the parent block
Evaluation value x when referencing toll and movement of surrounding blocks
Evaluation value y when the vector is referenced_i(Eg two
Y when the peripheral block is referenced₁, Y ₂) And most
The vector Vz, which is highly evaluated, is the block of interest for the hierarchical image.
Confirmed as motion vector 20f (30f)
Then, the motion vector /
Write to the evaluation value memory 20c (30c).

Here, the points of this embodiment will be described. One of the points of this embodiment is that the motion vector evaluator 10d for the uppermost layer and the motion vector evaluator 20d for the middle layer are provided.
The point is that the number of pixels (the number of matching pixels) of the blocks evaluated in step 1 is set to be the same. Such setting is performed by the block setter 10h of the uppermost layer motion vector detection unit 10 and the block setter 20h of the middle layer motion vector detection unit 20. Further, another one of the points of the present embodiment is that the size of the block evaluated by the motion vector evaluator 30d in the lowermost layer is a predetermined size ratio () to the image stored in the image memory 30a in the same layer. For convenience, the point is to set A). Such setting is performed by the block setter 30h of the lowermost layer motion vector detection unit 30. Therefore, the block setters 10h and 20 of each layer
h and 30h have a function as "block pixel number / size ratio setting means" described in the gist of the invention.

Next, the block setting of this embodiment will be described with reference to FIG. In FIG. 2, h _th represents a part of the middle layer image, h _{th + 1} represents a part of the upper layer image, and h _th−1 represents a part of the lower layer image. h _{th + 1} corresponds to the image in the image memory 10a in the uppermost layer in FIG. 1, h _th corresponds to the image in the image memory 20a in the middle layer in FIG. 1, and h _th-1 Corresponds to the image in the image memory 30a of the lowest layer in FIG. The squares representatively represent blocks set in each hierarchical image, 40 is a block of an upper layer image, 50 to 53 are blocks of an intermediate layer image, and 54 is a block of a lower layer image. The block 40 of the upper layer image and the blocks 50 to 53 of the middle layer image have the same number of pixels in the block, while the block of the middle layer image (for example, 50) and the block 54 of the lower layer image are Each belonging hierarchy image (block 50 is the middle layer image, block 54
Have the same size ratio to the lower layer image). That is, the size ratio of the block 50 to the middle layer image is “A”, and similarly, the size ratio of the block 50 to the lower layer image is 5.
The size ratio of 4 is also “A”.

That is, according to the block setting of FIG.
With a predetermined layer (middle layer) as a boundary, it is possible to switch between a block setting mainly on the matching pixel number (upper side) and a block setting mainly on the size ratio (lower side). Therefore, the number of matching pixels on the upper layer side (h _th , h _{th + 1} ) including the predetermined layer is set to an appropriate amount effective for texture suppression, while the lower layer side (h _th , h _th) including the predetermined layer is set. _{The th-1} ) block size can be set to an appropriate size that is effective in suppressing mismatches, and it is possible to avoid the loss of accuracy in motion vector search when applying the block matching method to hierarchical image information. The effect can be obtained.

Moreover, on the upper layer side including a predetermined layer,
With the vector obtained in one block 40 of h _{th + 1} as the initial displacement vector, four blocks 50 to 53 of h _th
Since the search is performed collectively, the search range can be read out collectively, so that the effect of reducing the memory access amount can be expected and the disadvantage that the block size of the uppermost layer becomes too large can be solved.

In the above embodiments, the block setting method is switched in the middle layer, but it may be in the upper layer. Further, although the embodiment shows an example in which it is applied to three-layered hierarchical image information, the present invention is not limited to this. It may be a multilayer of four or more layers. When applied to a multi-layer image having four or more layers, the predetermined layer does not always become an intermediate layer image.

[0031]

According to the present invention, it is possible to switch the block setting method at the boundary of a predetermined layer. Specifically, the upper layer side can be switched to a setting method effective for suppressing an increase in the number of pictures having different movements contained in one block, and the lower layer side can be switched to a setting method effective for suppressing a mismatch. . Therefore, it is possible to suppress both the increase in the number of pictures with different motions included in one block on the upper layer side and the mismatch on the lower layer side, and to search the motion vector when applying the block matching method to the hierarchical image information. The accuracy of can be improved.

[Brief description of drawings]

FIG. 1 is a conceptual overall configuration diagram of an embodiment.

FIG. 2 is a conceptual diagram of block setting in each layer of an embodiment.

FIG. 3 is a basic principle diagram (1) of motion vector detection.

FIG. 4 is a basic principle diagram (part 2) of motion vector detection.

FIG. 5 is a schematic diagram of a conventional hierarchical image.

FIG. 6 is a conceptual diagram of motion vector detection in a hierarchical image.

[Explanation of symbols]

10a, 10b: image memory (first layer memory) 20a, 20b, 30a, 30b: image memory (second to second)
M-th layer memory) 20d, 30d: motion vector evaluator (motion vector detection means, motion vector reference means, second motion vector reference means) 10h, 20h, 30h: block setter (block pixel number / size ratio setting means) )

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Koichi Oyama 4-36-19 Yoyogi, Shibuya-ku, Tokyo Within Graphics Communications Laboratories, Inc. (72) Inventor Maki Sato 4-yoyogi, Shibuya-ku, Tokyo 36-19 No. 19 in Graphics Communication Laboratories, Inc. (72) Inventor Yasuhiro Yamada No. 36-19, Yoyogi, Shibuya-ku, Tokyo Within 36-19, Graphics Communication Laboratories, Inc.

Claims (2)

[Claims] 1. A lowermost first hierarchical image having a resolution equal to or close to that of an original image, and upper second to mth hierarchical images whose resolution is gradually reduced with respect to the resolution of the first hierarchical image. And a motion vector of a parent block detected in the upper arbitrary layer is used as an initial displacement vector to detect a motion vector of a target block in the lower hierarchical layer of the arbitrary hierarchical layer. If the hierarchy to which the parent block belongs is other than a predetermined hierarchy, the number of pixels of both blocks is the same, while if the hierarchy is a predetermined hierarchy, the size ratio of both blocks to each belonging hierarchy image is the same. A hierarchical motion vector detection method characterized by the above. 2. A first layer memory for storing a lowermost first layer image having a resolution equal to or close to that of an original image, and an upper side memory whose resolution gradually decreases with respect to the resolution of the first layer image. Second to mth layer memories for storing the second to mth layer images, and a motion vector of the parent block detected in the upper arbitrary layer as an initial displacement vector. When the motion vector detecting means for detecting the motion vector of the block of interest and the hierarchy to which the parent block belongs are other than a predetermined hierarchy, the number of pixels of both blocks is made the same, while in the case of a predetermined hierarchy, both are Block pixel number / size ratio setting means for making the size ratio of each block to each belonging hierarchical image the same, and a hierarchical motion vector detecting apparatus.