JPH11331828A - Automatic tracking device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To track a motion of an object in a camera optical axis direction efficiently and automatically in the case of conducting tracking through the use of block matching. SOLUTION: The automatic tracking device is provided with a main camera 1 and a sub camera 2 that catches a motion of a tracking object in a depth direction, an optical flow arithmetic section 4 acquires an optical flow indicating the motion of the tracking object in the depth direction from an image of the sub camera 2, a control section 5 calculates divergence from the optical flow, an image tracking arithmetic device 3 magnifies/reduces a template image in use based on the divergence to control a magnification and a panning/tilt angle of a zoom lens of the camera 2.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an automatic tracking device for automatically tracking an object to be tracked by a main camera, and more particularly, to tracking movement of the object to be tracked in the camera optical axis direction when performing tracking using block matching. The present invention relates to an automatic tracking device that tracks efficiently.

[0002]

2. Description of the Related Art Conventionally, in the fields of camera control in movie / sports broadcasting, control of mobile robots / autonomous vehicles, recognition of gestures / facial expressions / gaze, monitoring, automation of observation, etc., moving objects on a screen are imaged. 2. Description of the Related Art There is known a technology of automatically tracking using processing. For example, Japanese Unexamined Utility Model Publication No. 58-178773 discloses a television tracking device configured to control the zoom mechanism so that the size of a target on a monitor television is always constant, so that the approach or the distance of the target can be tracked. An apparatus is disclosed. Japanese Patent Application Laid-Open No. Hei 3-174671 discloses an optical automatic configuration in which a target object is automatically tracked even if the size or direction of a moving object changes by performing a correlation process based on a Fourier transform. A tracking device is disclosed.

However, the former uses a contrast difference between a target and the surrounding scenery, and thus has a problem that it has only a tracking ability to track a white target on a black background. In addition, the latter adopts a method of batch processing images from the camera, so that window processing for tracking cannot be performed, and when there are a plurality of moving objects, it becomes unclear which one to track. There is.

[0004] For this reason, "Journal of the Robotics Society of Japan Vol.13
In No.1, P134-140, 1995 "Development of real-time vision system based on local correlation calculation", automatic tracking is performed without causing the above problem by performing correlation calculation based on block matching. Specifically, a correlation operation is performed densely between the image of the tracking target and the local image area that is currently gazing, to capture the change in the movement of the tracking target object, By reflecting it on the point where it is, robust and high-speed tracking is realized.

[0005]

However, since this tracking device performs a correlation operation for capturing a motion on a fixed template image, it can cope with up, down, left, and right movements, but on the other hand, forward and backward (in the depth direction). There is a problem that cannot track the movement. That is, the block matching employed by this tracking device is effective in terms of the robustness and speed of its tracking ability, but has the problem that it cannot track the movement of the target in the camera optical axis direction in this block matching. .

[0006] As a tracking device that follows a movement in the depth direction, Japanese Patent Laid-Open No. 24577/1990 discloses that a distant moving object moving in parallel with the optical axis of an image pickup apparatus is tracked on a reproduction screen. A moving object tracking device that can perform the moving object tracking is disclosed. However, since no technique for tracking a moving object that moves on a plane perpendicular to the optical axis of the imaging device has been disclosed, such a problem has not been practically solved.

Therefore, the present invention solves the above problems,
It is an object of the present invention to provide an automatic tracking device that can efficiently and efficiently track the movement of a target in the camera optical axis direction when performing tracking using block matching.

[0008]

In order to achieve the above object, a first aspect of the present invention is to provide at least a main camera for tracking an object to be tracked by a local correlation method, a zoom lens and a pan-tilt mechanism, A first camera that calculates the amount of movement of the tracking target object by matching the image information from the main camera with the template image, and a sub camera that captures the tracking target object captured by the sub camera. Second calculating means for calculating an optical flow indicating a movement in the depth direction, and a pan-tilt operation angle and a zoom lens of the sub-camera based on a calculation result by the second calculating means, and tracking used by the first calculating means. Control means for controlling the size of the template image of the object. Therefore, when tracking using block matching is performed, the movement of the target in the camera optical axis direction can be efficiently and automatically tracked.

Further, the second invention is characterized in that the control means controls the magnification of the zoom lens of the sub-camera to be the maximum magnification at which the object to be tracked falls within the viewing angle of the sub-camera. Therefore, an optical flow for detecting the movement of the tracking target in the camera optical axis direction can be obtained with high accuracy.

A third invention is characterized in that the second calculating means calculates a change in the size of the image of the tracking target object based on a local correlation operation. Therefore, it is possible to obtain an optical flow at high speed and follow a quick movement in the optical axis direction of the camera.

In a fourth aspect of the present invention, the second calculating means calculates the reliability of the motion vector of the tracked object based on a local correlation operation, and calculates a low reliability from a flow vector having a high reliability. It is characterized in that a flow vector is obtained by complementation. For this reason, an optical flow can be accurately obtained, and automatic tracking that is resistant to noise can be performed.

According to a fifth aspect of the present invention, the control means controls the magnification of the zoom lens of the sub camera using the divergence of the optical flow calculated by the second calculating means. Therefore, higher-speed automatic tracking using divergence with a small calculation amount can be performed.

[0013]

Embodiments of the present invention will be described below with reference to the drawings. First, the concept of the automatic tracking device used in the present embodiment will be described. The automatic tracking device used in the present embodiment is provided with a sub camera 2 in addition to the main camera 1 as shown in FIG.
, The movement of the tracking target in the depth direction is detected, and the size of the template image of the tracking target tracked by the main camera 1 is changed based on the detected change amount, thereby following the movement of the tracking target in the depth direction. I can do it.

That is, when the main camera 1 tracks an object to be tracked, the template image of the object to be tracked is shifted and matched with the data in the image memory to determine the moving direction and amount of movement of the object to be tracked. If the size of the tracking target fluctuates due to the movement of the object in the depth direction, the template image cannot be used as it is. Therefore, in this automatic tracking device, the size of the template image is enlarged or reduced in response to the movement of the tracking target in the depth direction. In this automatic tracking device, the motion of the tracking target is calculated as an optical flow, and the flow vector field is analyzed to determine whether the tracking target approaches or leaves the camera.

In this automatic tracking device, since the optical flow, which is the movement of the tracking target in the depth direction, is calculated using the sub-camera 2, it is difficult to capture the tracking target on the entire screen of the sub-camera 2 in terms of accuracy. desirable. For this reason, the zoom magnification and the pan / tilt movement of the sub camera 2 are adjusted in conjunction with the movement of the tracking target object in the depth direction, so that the tracking target object enters the field of view of the sub camera 2 having a narrow field of view. Therefore, if the tracking target moves farther or closer in the depth direction, it can be regarded as a change in the optical flow of the entire image.

As described above, the optical flow is always estimated in the image from the sub-camera 2, and the divergence of the obtained flow vector field (div = du / d)
x + dv / dy). When the absolute value of the divergence (div) exceeds a certain threshold value, it is determined that the tracking target has moved in the depth direction,
The template image is enlarged or reduced at a magnification proportional to the value of the divergence (div). By changing the template image in this way, the main camera 1 can track the object including the movement in the depth direction.

Next, the configuration of the automatic tracking device used in this embodiment will be described. FIG. 1 is a diagram showing an overall configuration of an automatic tracking device used in the present embodiment. As shown in FIG. 1, the automatic tracking device includes a main camera 1, a sub camera 2, an image tracking calculation device 3, an optical flow calculation unit 4, and a control unit 5.

The main camera 1 is a CCD camera module having a relatively wide viewing angle for tracking an object to be imaged by a local correlation method. The sub camera 2 is capable of performing a pan / tilt operation by a pan / tilt table 7 and is scaled by an electric signal. This is a CCD camera having a zoom lens 6 whose rate changes. The reason for providing the sub camera 2 in addition to the main camera 1 is that the main camera 1
This is because the image information in the window is small and the optical flow, which is the movement information of the tracking target, cannot be obtained with high accuracy. That is, since the optical flow is obtained by calculation in the tracking window in which the tracking target exists, the accuracy is improved if the image information in the window is large. For this reason, this automatic tracking device is provided with an auxiliary camera (sub camera 2) that largely captures only the tracking target. The main camera 1
Is input to the image tracking arithmetic processing unit 3, and the video signal from the sub camera 2 is
Is input to A signal line for controlling the zoom lens 6 of the sub camera 2 is connected to the control unit 5, and a control signal of a pan / tilt table 7 on which the sub camera 2 is mounted is also connected to the control unit 5.

The image tracking operation unit 3 is a processing unit for calculating the amount of movement of the object based on image information from the main camera 1, and as shown in FIG. 2, an image memory 3a, a correlation operation processing unit 3b and Consists of a template image 3c. The optical flow calculation unit 4 is a processing unit that calculates an optical flow including a change in the size of the target image, and includes an image memory 4a and a correlation calculation processing unit 4b as shown in FIG. As described above, the image tracking operation processing device 3 and the optical flow operation unit 4 each include the correlation operation unit 3b and the correlation operation unit 4b having the same function.

The correlation calculators 3b and 4b receive the reference template image and its coordinate values, and obtain the amount of deviation between the candidate image located near the coordinates and the template image as shown in FIG. For example, for a template image of 8 × 8 pixels, a correlation is made between the candidate image and the template image for any shift amount in a window of 24 × 24 pixels, and an image having the highest matching degree is searched for. Find the shift amount. And
In the image tracking calculation processing device 3, the shift amount obtained by the correlation calculation unit 3b is regarded as the movement amount of the tracking target. In the optical flow calculation unit 4, the correlation calculation unit 4b sets the template image as the image at the time of the immediately preceding processing. The obtained shift amount is regarded as a movement vector.

The optical flow calculation unit 4
For example, an improved correlation method disclosed in Japanese Patent Application Laid-Open No. 9-73540 can be used. In this case, the reliability of the motion vector of the tracking target is calculated based on the local correlation operation, and the reliability is calculated. A flow vector with low reliability is obtained from a flow vector with high reliability by interpolation. Using this improved correlation method, with a slight speed reduction,
Since a motion vector with low reliability and a lot of noise components can be complemented by a motion vector with high reliability, an optical flow can be accurately obtained, and a tracking process that is strong against noise can be performed.

The control unit 5 controls the pan / tilt operation angle of the sub camera 2 and the zoom lens 6 based on the outputs of the image tracking calculation processing unit 3 and the optical flow calculation unit 4 and the template of the tracking target tracked by the main camera 1. The control unit controls the size of the image, and includes a pan / tilt mechanism control unit 5a, a divergence calculation unit 5b, a zoom lens control unit 5c, and a template image enlargement / reduction processing unit 5d, as shown in FIG.

Here, the pan / tilt mechanism control unit 5a is a processing unit that controls the pan / tilt table 7 of the sub camera 2 based on data from the image tracking calculation processing unit 3. The divergence calculation unit 5b is a processing unit that calculates divergence, that is, div = du / dx + dv / dy (scalar amount) based on the flow vector field data (u, v) from the optical flow calculation unit 4. The calculation result is output to the zoom lens control unit 5c and the template image enlargement / reduction processing unit 5d. The zoom lens control unit 5c determines the optimum magnification of the zoom lens 6 of the sub camera 2 using the divergence, and the template image enlargement / reduction processing unit 5d responds to the movement of the tracking target in the depth direction. In order to cope with the change in the size, the template image 3c is enlarged or reduced.

More specifically, when the main camera 1 tracks an object to be tracked, the image tracking arithmetic processing unit 3 determines the current coordinates of the object to be tracked, the moving direction of the object in a unit time of the tracking arithmetic processing, and The moving amount is sent to the control unit 5, and the control unit 5
Moves the pan-tilt table 7 by an amount corresponding to the amount of movement so that the sub camera 2 always keeps the tracking target within its field of view. Therefore, as shown in FIG. 6, the main camera 1 tracks the tracking target with a wide viewing angle, and the sub camera 2 can always obtain the tracking target with the full viewing angle.

Next, the case where the tracked object moves in the depth direction, that is, the case where the tracked object approaches or moves away from the camera will be described. As described above, the sub-camera 2 can always catch the tracking target in the entire viewing angle by the zoom lens control unit 5c of the control unit 5, so that the movement of the tracking target in the depth direction can be grasped sensitively.

More specifically, the movement of the tracked object in the depth direction can be regarded as a change in the flow vector field obtained from the optical flow calculation unit 4.
The divergence calculation unit 5b calculates the divergence (div) of the vector field from the change of the flow vector field, and detects the amount of movement in the depth direction. Since the tracking target object shown in the main camera 1 has changed in size in proportion to the divergence (div), the template image enlargement / reduction processing unit 5d determines whether the tracking target object is to be compared with the tracking target object. The template image 3c is enlarged or reduced at a magnification proportional to the divergence (div). Further, since the size of the image captured by the sub camera 2 also changes in response to the divergence (div), the zoom lens control unit 5c changes the magnification of the zoom lens 6 based on the divergence (div).

As described above, in the present embodiment, the change in the size of the tracking object due to the movement of the tracking object in the depth direction is absorbed by the enlargement / reduction of the template image 3c and the change in the magnification of the zoom lens 6. Therefore, both the main camera 1 and the sub camera 2 can appropriately capture the tracking target, and three-dimensional tracking can be performed.

[0028]

As described in detail above, the first aspect of the present invention provides a sub-camera in addition to the main camera, and performs pan-tilt of the sub-camera based on the movement of the tracking target in the depth direction captured by the sub-camera. Since the configuration is such that the operation angle, the zoom lens, and the size of the template image of the tracking target are controlled, the following effects can be obtained. 1) When performing tracking using block matching, it is possible to automatically and efficiently track the movement of the tracking target in the optical axis direction of the camera. 2) Regarding the movement in the depth direction, which has been regarded as a weak point in the past,
By adding a sub-camera capable of obtaining an optical flow and estimating the movement in the depth direction and reflecting the movement in the enlargement / reduction ratio of the template image, accurate three-dimensional tracking becomes possible.

In the second invention, the magnification of the zoom lens of the sub camera is controlled so as to be the maximum magnification at which the object to be tracked falls within the viewing angle of the sub camera, so that the following effects can be obtained. 1) The optical flow for detecting the movement of the tracking target in the camera optical axis direction can be accurately obtained. 2) Since the motion information of the tracking target in the camera optical axis direction is obtained in the entire field of view of the sub camera, the motion information is accurately obtained, and the motion information is reflected in the enlargement / reduction ratio of the template image, thereby enabling more accurate tracking. .

Further, since the third invention is configured to calculate the change in the size of the image of the object to be tracked based on the local correlation operation, the optical flow can be obtained at high speed, and the movement can be quickly performed in the camera optical axis direction. Can also follow.

According to a fourth aspect, the reliability of the motion vector of the tracking object is calculated based on a local correlation operation,
Since low-reliability flow vectors are obtained by complementation from high-reliability flow vectors, motion vectors with low reliability, that is, low-reliability, that is, motion vectors with many noise components are also highly reliable , The optical flow is accurately obtained, and tracking processing resistant to noise can be performed.

The fifth aspect of the present invention is configured to control the magnification of the zoom lens of the sub camera using the divergence of the optical flow, so that the following effects can be obtained. 1) Since the movement in the optical axis direction is obtained based on the divergence with a small amount of calculation, the movement in the optical axis direction of the camera can be obtained at higher speed, and the tracking processing can be performed at a higher speed. 2) A tracking process that is resistant to noise and that can cope with the movement of the target object in the camera optical axis direction can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall configuration of an automatic tracking device used in the present embodiment.

FIG. 2 is a block diagram illustrating a configuration of an image tracking calculation processing device illustrated in FIG. 1;

FIG. 3 is a block diagram illustrating a configuration of an optical flow calculation unit illustrated in FIG. 1;

FIG. 4 is an explanatory diagram illustrating a process of a correlation operation processing unit illustrated in FIGS. 2 and 3;

FIG. 5 is a block diagram showing a configuration of a control unit shown in FIG.

FIG. 6 is an explanatory diagram showing images of an object tracked by a main camera and a sub camera.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Main camera 2 ... Sub camera 3 ... Image tracking calculation device 4 ... Optical flow calculation part 5 ... Control part 6 ... Zoom lens 7 ... Pan-tilt stand

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification symbol FI G05D 3/12 B60R 21/00 620Z H04N 5/232 B62D 1/24 // B60R 21/00 620 G02B 7/11 K B62D 1 / 24 G03B 3/00 A 57/032 B62D 57/02 E G06T 7/20 G06F 15/70 410

Claims (5)

[Claims] 1. A main camera that tracks a tracking target by a local correlation method, a sub camera that has at least a zoom lens and a pan-tilt mechanism, and captures the tracking target as an enlarged image, and image information from the main camera. A first method of calculating the movement amount of the tracking target object by matching with a template image
Calculating means for calculating an optical flow indicative of the movement of the tracking target object in the depth direction captured by the sub-camera; and pan-tilt of the sub-camera based on the calculation result by the second calculating means. Control means for controlling the operating angle, the zoom lens, and the size of the template image of the tracking target used by the first calculating means. 2. The automatic tracking apparatus according to claim 1, wherein the control unit controls the magnification of the zoom lens of the sub-camera to be a maximum magnification at which the tracking target falls within the viewing angle of the sub-camera. apparatus. 3. The automatic tracking apparatus according to claim 1, wherein the second calculating means calculates a change in the size of the image of the tracking target based on a local correlation operation. 4. The second calculating means calculates the reliability of the motion vector of the tracking target based on a local correlation operation, and obtains a low-reliability flow vector from a high-reliability flow vector by complementation. The automatic tracking device according to claim 1 or 2, wherein 5. The automatic control system according to claim 3, wherein the control unit controls the magnification of the zoom lens of the sub camera using the divergence of the optical flow calculated by the second calculation unit. Tracking device.