JPH06133298A - Motion vector detector - Google Patents

Abstract

PURPOSE:To enhance the detection accuracy of a motion vector. CONSTITUTION:A motion vector detection circuit 13 detects the distribution of motion vectors. A microcomputer 14 divides areas of a picture based on the distribution of the motion vectors detected by the motion vector detection circuit 13. Then the appearance area is discriminated based on the result of area division and the direction of the motion vector at the moment and the motion vector detected from the appearance area is eliminated. Then the motion quantity of the picture to be corrected is calculated. A memory read control circuit 15 controls a read position of the memory 16 so as to correct the final motion vector calculated by the microcomputer 14. A read luminance signal and a read chrominance signal are converted into an analog signal by a D/A converter 17 and the signal is outputted from a video signal output terminal 18.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion vector detecting device used for a moving image signal coding device, an image touch correcting device, and the like.

[0002]

2. Description of the Related Art Conventionally, as a motion vector detecting method required for an image coding apparatus and an image touch correction apparatus, a spatiotemporal gradient method described in US Pat. No. 3,890,462, Japanese Patent Publication No. 60-46878, or the like, or There are a correlation method and a block matching method (template matching method) based on a correlation calculation. For the spatiotemporal gradient method, see BKP Horn et al., Artificial Intelligence 17, p. 185-203 (1.
981), and regarding the block matching method, Morio Onoue et al., Information Processing Vol. 17,
No. 7, p. 634-640 July 1976, discussed in detail.

The spatio-temporal gradient method is a method in which the amount of motion of an image is represented by d / Δ based on a brightness difference d between frames (or fields) and a brightness difference Δ between pixels in a screen. this is,
The signal obtained from the camera is the time average of the field cycle, and the larger the motion amount of the image is, the more the edge becomes dull and the brightness difference Δ between pixels becomes smaller. It is normalized by the signal Δ.

On the other hand, in the block matching method, an input image signal is divided into blocks of an appropriate size (for example, 8 pixels × 8 lines), and the difference from the pixels in a certain range of the previous frame (or field) is divided in block units. The block of the previous frame (or field) is calculated and the sum of the absolute values of the differences is minimized. The relative shift of the block represents the motion vector of the block.

Both methods are premised on the existence of corresponding points between images for detecting image movement.

[0006]

However, in the above-described conventional motion vector detecting method, when a region that does not exist in the immediately preceding screen newly appears in the screen at a certain moment, the time-series consecutive images are interleaved. In principle, since there is no corresponding point, it is not possible to detect a correct motion vector in principle, and an incorrect motion vector detected from the newly appearing area causes the encoding accuracy of the encoding device or the correction of the image blur correction device to be corrected. There is a problem that the accuracy deteriorates.

It is an object of the present invention to reduce the influence of a motion vector detection error due to the absence of corresponding points and to improve the motion vector detection accuracy.

[0008]

A motion vector detecting apparatus according to the present invention divides an image into a plurality of small areas and obtains a motion vector for each of the small areas. It doesn't exist on the previous screen,
The present invention is characterized by having a specifying means for specifying a newly appearing area on the current screen and an evaluating means for excluding or lightly evaluating a motion vector detected from the area specified by the specifying means as an invalid vector.

Some of the specifying means use spatial distribution information of motion vectors to specify a newly appearing area. As the spatial distribution information of motion vectors, a set area of motion vectors having the same size and size in the same direction is extracted from the screen, and the area behind the extracted set area is newly created. It is specified as a region that appears in. In this case, the area behind the moving direction of the motion vector aggregate area in the same direction and of the same size is determined by the size of the motion vector, or is determined by using the image density. , Some are detected by a signal from a sensor attached to an image input device.

Further, the motion vector detecting device of the present invention is
In a motion vector detection device that divides an image into a plurality of small areas and obtains a motion vector for each of the small areas, a new area that does not exist on the screen that is one screen before the current screen and newly appears on the current screen. The present invention is characterized by having a specifying unit for specifying and a detecting unit for searching a corresponding point of the area specified by the specifying unit by tracing back two or more screens in the past and detecting an approximate motion vector.

Some of the specifying means use spatial distribution information of motion vectors to specify a newly appearing area. As the spatial distribution information of motion vectors, a set area of motion vectors having the same size and size in the same direction is extracted from the screen, and the area behind the extracted set area is newly created. It is specified as a region that appears in. In this case, the area behind the moving direction of the motion vector aggregate area in the same direction and of the same size is determined by the size of the motion vector, or is determined by using the image density. , Some are detected by a signal from a sensor attached to an image input device. The detection means specifies the size of the object that has hidden the newly appearing area and the size of the motion vector of the object in order to specify a past screen for searching for a corresponding point of the newly appearing area. Is used.

[0012]

Operation: The specifying unit detects a region that does not exist on the screen one screen before the current screen and newly appears on the current screen (hereinafter referred to as an appearance region). By eliminating or lightly evaluating a low motion vector, only a highly reliable motion vector is detected, so that the influence of a motion vector detection error due to the absence of corresponding points can be reduced.

Further, the appearance means is detected by the specifying means, and the corresponding means of the appearance area is searched back by the detection means to the past two screens or more to detect an approximate motion vector and detected from the appearance area. Since the motion vector detection accuracy can be improved by replacing the motion vector with low reliability with a modern motion vector, it is possible to reduce the influence of the motion vector detection error due to the lack of corresponding points. .

[0014]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the construction of a video camera incorporating an image blur correction device to which the first embodiment of the motion vector detection device of the present invention is applied. This video camera includes a taking lens 2 for taking an image of a subject 1, an image sensor 3 including, for example, a two-dimensional CCD, and a sample hold (hereinafter, referred to as S / H) for holding an output signal from the image sensor 3.
The circuit 4 and an automatic gain control circuit (hereinafter, referred to as an AGC circuit) 5 for adjusting the gain of the output of the S / H circuit 4
An A / D converter 6 for A / D converting the output of the AGC circuit 5, a delay circuit 7 for delaying the color difference line sequential signal from the image sensor 3 by two horizontal scanning periods, and a color signal (C) generation circuit. 8, a low-pass filter (hereinafter, referred to as LPF) 10 for removing a color signal mixed in the luminance signal, an enhancer 11 that emphasizes a high frequency component of the luminance signal, a gamma corrector 12, and a motion vector detection circuit 13 As a specifying means and an evaluation means, a microcomputer 14 for determining an appearance area, a memory read control circuit 15, a field memory (or frame memory) 16, and a D / A converter for D / A converting a luminance signal and a color signal. It is composed of an A converter 17 and an output terminal 18 for a luminance signal (Y) and a color signal (C). Elements after the LPF 10 form a luminance signal processing unit. The motion vector detection circuit 13 may be configured by a circuit based on the spatiotemporal gradient method or a circuit based on a matching calculation, but in the present embodiment, it is necessary to be a detection circuit capable of real-time processing. .

Next, the operation of the video camera shown in FIG. 1 will be described. The subject 1 is imaged and photoelectrically converted on the image sensor 3 by the taking lens 2. The S / H circuit 4 holds the output signal of the image sensor 3, and subsequently the AGC circuit 5 automatically controls the gain of the signal. The A / D converter 6 A / D converts the output signal of the AGC circuit 5. The delay circuit 7 includes the image sensor 3
Color difference line sequential signal from 1H delay signal and (OH + 2H)
Separated into delayed signals, the luminance signal processing units 10 to 18 respectively
And the color signal generation circuit 8 is sent. The color signal generation circuit 8 generates a color signal and stores it in the memory 16.

On the other hand, the signal sent to the luminance signal processing units 10 to 18 is first input to the LPF 10. LPF10 is
The luminance component is separated by removing the carrier component from the color difference sequential signal. The enhancer 11 responds to the output of the LPF 10 by
Processing for enhancing high-frequency components such as edges of a subject for improving image quality is performed. Normally, the second derivative of the video signal is added to the original signal. The gamma correction circuit 12 prevents saturation of the output of the enhancer 11 in the highlight portion and widens the dynamic range. The motion vector detection circuit 13 detects a motion vector distribution (optical flow) from the output from the gamma corrector 12. The memory 16 is a delay circuit that delays the luminance signal Y and the chrominance signal C by a predetermined time (one field time in this embodiment), stores the video signal one field before and calculates the spatiotemporal gradient method with the current field. , Or enable correlation calculation. Microcomputer 1
Reference numeral 4 divides the area of the image from the optical flow detected by the motion vector detection circuit 13. Then, the appearance region is judged from the result of the region division and the direction of the motion vector at that moment, and the motion vector detected from the appearance region is removed or lightly weighted. After that, the microcomputer 14 extracts the block outputting the motion vector due to the camera shake to be corrected, and calculates the motion amount of the image to be corrected at that moment. The memory read control circuit 15 controls the read position of the memory 16 so as to cancel or correct the final motion vector calculated by the microcomputer 14. The read luminance signal and chrominance signal are converted into analog signals by the D / A converter 17 and output from the video signal output terminal 18.

The appearance of the appearance area in various shooting situations will be described with reference to FIGS. FIG. 2 is a diagram showing a state in which a rectangular object crosses the drawing. Figure 2
As shown in (a), the object 21 at the right end of the screen at time T _i-2 has time T _i−1 , T _i , and T _{i −1} as shown in FIGS. 2 (b), (c), and (d) in order. As it goes to T _{i + 1} , it is moving to the left. The appearance area 22 in FIG. 2B is an area that is hidden and invisible one screen before (FIG. 2A). The length of the object 21 in the traveling direction is d, and the moving speed is w ₁
And The length W ₂ of the appearance area in the traveling direction is equal to the length w ₁ of the motion vector.

FIG. 3 is a diagram showing the state of the screen when the camera is continuously moved in the fixed direction at the same speed (so-called panning or tilting operation). Figure 3 (b),
As shown in (c) and (d), an appearance area 31 that does not exist in the previous screen (FIG. 3A) appears at the left end of the screen.

FIG. 4 shows a case where a round object in the center of the screen is tracked.
Shows the state of the screen during tracking shooting as object 41
It is a figure. The tracking subject 41 and the background 42 are relatively different.
4 (a), (b), (c),
As shown in (d), time T _i-2 , T_i-1 , T_i , T
_{i + 1} Appear from behind the tracking subject 41 as
Area 43 has appeared. At the same time, you can
The appearance area 44 appears according to the swing of the melody.

Next, a method of evaluating the reliability of the motion vector performed in the microcomputer 14 will be described in detail with reference to FIG.

FIG. 5 is a diagram showing the flow from the input of the image signal to the determination of the appearance area at that moment. FIG. 5A is a diagram showing an input image, which is one frame when the automobile as the subject 51 is being tracked and photographed. In this embodiment, the input image is divided into 12 vertical blocks and 16 horizontal blocks, and a motion vector is detected for each block. FIG. 5B shows the result of detecting the motion vector for each block. Such a motion vector distribution diagram is generally called an optical flow diagram. In FIG. 5B, from the set distribution of motion vectors having the same size in the same direction, it can be seen that there are two different moving regions, the central part and the peripheral part of the screen. Then, a motion vector with low reliability accuracy due to a corresponding point error is detected behind these two areas in the traveling direction. In this embodiment, these two areas are extracted as the spatial distribution information of the motion vector. FIG. 5C is a diagram showing a result of extracting appearance regions from the optical flow diagram of FIG. 5B. The microcomputer 14 removes the motion vector detected from the extracted regions 52 and 53, or evaluates a relatively light weight compared with the others, thereby suppressing the adverse effect on the subsequent processing. The appearance area 52,
The size of 53 directly depends on the moving speed of the object that has hidden the appearance regions 52 and 53 on the immediately previous screen. Subject 5
If 1 is moving slowly, appearance areas 52, 5
3 is small, the accuracy degradation of the motion vector due to the appearance regions 52 and 53 is not a problem, but when the subject 51 is moving at high speed, the appearance regions 52 and 53 become large, so the motion vectors due to the appearance regions 52 and 53 are large. The influence of the accuracy deterioration of is increased. In the example of FIG. 5C, the amount of movement of the subject 51 between consecutive images is just equal to the width of the unit block in which the motion vector is detected, so the widths of the appearance regions 52 and 53 are also determined to be the width of one block. To be done. When the appearance area extends over a block that is a unit area for detecting a motion vector, the boundary between the moving object and the appearance area is not clear. In this case, the weighting may be performed so that the motion vector detected from the appearance area is evaluated to be relatively low, and the weighting is performed so as to descend from the head part to the rear part of the moving body. The microcomputer 14 further selects a region for which image stabilization is desired (image stabilization region) from the distribution of motion vectors remaining after removing the appearance region, and calculates a correction vector from the motion vector in the image stabilization region. As a method of extracting the image stabilization area from the motion vector, there is a time direction addition method of motion vectors for each block (see Japanese Patent Laid-Open No. 2-117276).

The memory read control circuit 15 determines the image read position from the memory 16 using the correction vector data obtained by the microcomputer 14,
An image in which the unpleasant camera shake component is corrected is output from the output terminal 18.

In the first embodiment, the size of the appearance area is determined according to the magnitude of the motion vector of the object that hides the appearance area. You may use the density difference between. The principle will be described below with reference to FIG.

FIG. 6A shows that the object edge 61 is moving to the right. In the figure, A is the rising point (start point) of the edge 61 one screen before, B is the falling point (end point) of the edge 61 one screen before, C is the rising point (start point) of the edge 61 of the current screen, and D is Edge 61 of current screen
Is the falling point (end point) of. Edges 61 model the boundaries of different objects in the image. Figure 6
FIG. 6B shows the density difference between the two edges 61 shown in FIG. The density difference when the image is translated by a small amount corresponds to the first-order differentiation of the image. Therefore, the density difference greatly changes at a high frequency component such as the edge 61, and the edge can be easily detected from the density difference. You can Therefore, the appearance region 62 can be specified by obtaining a region where the density difference between the images is large behind the moving body that is revealed by the motion vector. In FIGS. 6B and 6D, the hatched area between ACs is the appearance area 62.
The motion vector detected from is invalidated. Between BCs is an area that becomes an appearance area 62 when the amount of movement of the edge 61 increases. As shown in FIG. 6B, when the point C is behind the point B with respect to the moving body, no new area appears, but as shown in FIG. Becomes larger and C point is ahead of B point, BC
The space also becomes the appearance area 62. Therefore, the motion vector detected from the appearance area 62 between BCs is lightly evaluated to suppress the error of the motion vector. In the present embodiment, the object boundary is detected using the density difference (time direction) between images that are continuous in time series, but the density difference (spatial direction) of adjacent pixels in one image is determined. The boundary may be detected and obtained. FIGS. 7A to 7D show the state of the first-order differentiation of the edge for each screen when the edge 71 is moving to the right, and the view is the same as in FIG. In this case, the appearance area 72 is always between the rising points A and C of the edge 71.

In each of the above-described embodiments, the size of the appearance area is calculated by a method based on image data processing. However, the width of the appearance area due to the swing of the camera as shown in FIG. Alternatively, it may be obtained using a signal from a sensor (not shown) attached to the camera. The amount of movement of the camera detected by this sensor may be converted into the amount of movement on the imaging surface according to the focal length of the taking lens.

Although a correct motion vector cannot be detected from the appearance area, it is possible to approximate the motion vector by searching the corresponding points of the newly appearing area retrospectively in the past two or more screens. The principle will be described with reference to FIG.

FIG. 8 shows a round object 81 moving parallel to the left in the screen. The appearance area 82 at time T _{i + 1} shown in FIG. 8D has a corresponding point 83 in the screen at time T _i-2 , which is three screens backward shown in FIG. 8A.
The amount of movement for three screens can be calculated. By dividing the movement amount for the three screens by 3, the movement amount of the appearance area 82 at time T _{i + 1} can be approximately obtained. The number of images traced back is roughly determined by the moving speed of the moving body recognized from the motion vector and the size of the subject 81. If the size of the subject 81 in the traveling direction is d, and the size of the motion vector and the length of the appearance region 82 in the traveling direction are w, the corresponding points of the appearance region at time T _{i + 1} are expressed by the following equation. Go back in time.

T = (d + w) / w For example, in the case of the relationship of d = 2w, T = 3 and 3
You can see that you can go back to the screen. However, in this embodiment, it is necessary to store a plurality of past images.

[0030]

As described above, according to the present invention, it is possible to easily determine a newly appearing region in the current screen instead of the previous screen from the motion vector distribution detected in the image, and make an error detected in the appearing region. Motion vectors can be eliminated. Further, the motion vector can be approximately calculated by finding the corresponding points of the appearance area two or more screens in the past. As a result, there is an effect of improving the accuracy of a moving picture coding apparatus, an image blur correction apparatus, etc., which has been significantly deteriorated in accuracy by a motion vector with low reliability due to appearance / concealment of an image.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a motion vector detection device of the present invention.

2A to 2D are diagrams showing appearance of appearance areas.

FIG. 3A to FIG. 3D are diagrams showing appearances during a panning operation.

FIGS. 4A to 4D are diagrams showing appearances during a tracking shooting operation.

5A to 5C are diagrams showing a flow from extraction of an appearance area from an input image.

6A to 6D are diagrams showing edge movements and their density differences (time direction).

7A to 7D are diagrams showing edge movements and their density differences (spatial direction).

FIG. 8 is an explanatory diagram of a method of retroactively searching for corresponding points.

[Explanation of symbols]

 1 subject 2 photographing lens 3 image sensor 4 S / H circuit 5 AGC circuit 6 A / D converter 7 delay circuit 8 color signal generation circuit 10 low-pass filter 11 enhancer circuit 12 gamma correction circuit 13 motion vector detection circuit 14 microcomputer 15 memory Read control circuit 16 Field memory 17 A / D converter 18 Video signal output terminal

Claims (13)

[Claims] 1. An image is divided into a plurality of small areas,
In the motion vector detecting device for obtaining the motion vector for each small area, a specifying means for specifying a newly appearing area on the current screen that does not exist on the screen one screen before the current screen, and is specified by the specifying means. And a motion vector detected from the region, which is excluded or lightly evaluated as an invalid vector, and a motion vector detecting device. 2. The motion vector detecting device according to claim 1, wherein the specifying means uses spatial distribution information of the motion vector to specify a newly appearing area. 3. The specifying means extracts, as the spatial distribution information of the motion vector, a set area of motion vectors in the same direction and having a similar size from the screen, and sets in the moving direction of the extracted set area. The motion vector detecting device according to claim 2, wherein a region behind the region is specified as a newly appearing region. 4. The specifying means determines a region behind the movement direction of a motion vector aggregation region in the same direction and having a similar size according to the size of the motion vector. Item 5. The motion vector detection device according to item 3. 5. The identifying means determines a region behind the moving direction of a set region of motion vectors having the same size and the same size by using the image density. 3. The motion vector detection device described in 3. 6. The signal from a sensor attached to a device for inputting an image of an area behind a moving area of a set area of motion vectors having the same size and the same size. The motion vector detecting device according to claim 3, wherein the motion vector detecting device detects the motion vector. 7. The image is divided into a plurality of small areas,
In the motion vector detecting device for obtaining the motion vector for each small area, a specifying means for specifying a newly appearing area on the current screen that does not exist on the screen one screen before the current screen, and is specified by the specifying means. A motion vector detection device, comprising: a detection unit that searches for a corresponding point in an area that is two or more screens in the past and detects an approximate motion vector. 8. The motion vector detecting device according to claim 1, wherein the specifying means uses spatial distribution information of the motion vector to specify a newly appearing area. 9. The specifying means extracts, as spatial distribution information of motion vectors, motion vector aggregate areas in the same direction and having a similar size from the screen, and sets the extracted movement directions of the extracted aggregate areas. 9. The motion vector detecting device according to claim 8, wherein the area behind the area is specified as a newly appearing area. 10. The specifying means determines a region behind a moving direction of a motion vector aggregation region in the same direction and having a similar size according to the size of the motion vector. Item 9. The motion vector detecting device according to item 9. 11. The specifying means determines a region behind a moving direction of a set region of motion vectors having the same size and the same size by using the density of the image. 9. The motion vector detection device according to item 9. 12. The specifying means is a signal from a sensor attached to a device for inputting an image, in an area behind the movement area of a set area of motion vectors having the same size and the same size. The detection is performed by
The motion vector detection device described. 13. The detection means specifies the size of an object that has concealed the newly appearing region and the size of the object in order to identify a past screen for searching for a corresponding point of the newly appearing region. The motion vector detecting device according to claim 7, wherein the size of the motion vector is used.