JP2992065B2 - Image tracking device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION [Outline] A target is extracted by a correlation operation between an image acquired by an image sensor and a target reference image, and a drive signal to a gimbal mechanism for turning a visual axis of the image sensor is generated. A first frame memory for storing an actual image signal from an image sensor, a second frame memory for storing a reference image in advance, and a search reference point. And a correlation value between the actual image and the reference image is calculated for each pixel between the real image and the reference image, and a point having the largest correlation value among the correlation values is set as a next search reference point. By sequentially repeating this calculation, the correlation calculation unit that outputs a tracking command to the imaging sensor when the correlation value of the new search reference point has a higher value than the correlation value of the adjacent point. Constitute.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image tracking device, and more particularly to a gimbal mechanism that extracts a target by correlating an image acquired by an image sensor and a target reference image and turns the visual axis of the image sensor. The present invention relates to an image tracking device that generates a drive signal of the following.

[Conventional technology]

FIG. 1 shows such an image tracking device in principle, 1 is an image sensor, 2 is a first frame memory for storing the actual image signal from the image sensor 1, and 3 is a reference image stored in advance. The second frame memory and 4 compare the correlation values between the real image from the first frame memory 1 and the reference image from the second frame memory 3 and determine the maximum correlation value among them. A correlation calculation unit that gives a tracking command to the image sensor 1 with the point having the target position.

In such a configuration, in the correlation operation unit 4, for example, as shown in FIG.
In the whole area of one frame of the real image composed of × C pixels, the following is shown for each pixel between the reference image composed of l × c (L>l;C> c) pixels as shown in FIG. The correlation operation shown in the equation is performed.

That is, in this equation, the reference image R shown in FIG.
(X, y) is shifted by X and Y in the x and y directions while performing the convolution operation with the corresponding real image f (x, y) in FIG. (Denominator) is obtained.

In this way, an image correlation operation is performed between the entire real image and the reference image to generate a correlation image as shown in FIG.
A maximum value is detected in the correlation image, and a tracking command is issued to the image sensor 1 so that the image position where the correlation value is maximum is set as a target position and the position is tracked for each frame image.

[Problems to be solved by the invention]

In such a conventional image tracking device, when the above-described correlation operation is performed for the entire real image, the number of product-sum operations required for the correlation tracking is as follows: operation amount = 3 · c · l (C−c) (L− l) For example, when C = L = 256 and c = l = 32, 147
× 2 ²⁰ ≒ 1.5 × 10 ⁶ , which is a huge amount of computation.

Therefore, there is a problem in that high-speed hardware for parallel processing, pipeline processing, and the like that can realize real-time processing used for correlation calculation becomes large-scale.

Therefore, the present invention relates to an image tracking device that generates a drive signal to a gimbal mechanism that extracts a target by correlating an image acquired by an image sensor with a target reference image and turns the visual axis of the image sensor. The purpose of the present invention is to reduce the amount of correlation calculation as much as possible.

[Means and actions for solving the problem]

The basic configuration of the image tracking apparatus according to the present invention is the same as that of the conventional example as shown in FIG. 1, but the difference is that in the operation of the correlation operation unit 4, the search reference point and its adjacent point are different. A correlation value is calculated for each pixel between the actual image and the reference image, and the correlation values are compared with each other, and the point having the largest correlation value is set as the next search reference point, and this calculation is sequentially repeated. Thus, the tracking command is output to the image sensor 1 when the correlation value of the new search reference point reaches a value higher than the correlation value of the adjacent point.

That is, in the present invention, in order to search for the maximum correlation value, instead of taking the correlation value between the whole area of the real image (FIG. 5 (a)) and the reference image (FIG. 5 (b)) as in the related art, Fig. 2 (a)
As shown in (1), the target position in the previously sampled frame can be considered to be present in the hill portion of the correlation distribution, and the current target position is to be found at the shortest distance from the previous target point.

In order to do this, the correlation calculator 4 calculates the correlation value g ₀ (x ₀ , y ₀ ) between the real image and the reference image for the search reference point for each pixel as shown in FIG. It is determined by the following equation, and a point adjacent to this search reference point (FIG. 2 (b)
In the example of (4), there are four adjacent points, but the present invention is not limited to this.) The correlation value g between the real image and the reference image
₁ (x ₁ , y ₁ ) to g ₄ (x ₄ , y ₄ ) are obtained, and a point having the largest value is searched for.

Then, a similar correlation value search is performed using the largest point among the correlation values as the next search reference point.

By sequentially searching for the search reference point in this way, a search path reaching the target position is obtained as shown in FIG. 2 (a).

When the correlation value g ₀ (x ₀ , y ₀ ) of the search reference point is larger than any of the correlation values g ₁ (x ₁ , y ₁ ) to g ₄ (x ₄ , y ₄ ) of the adjacent point 2 (a), the correlation calculation unit 4 gives the imaging sensor 1 a command to track the target position in the current frame image at this time. .

Therefore, since the image correlation calculation is limited to the search reference point and its adjacent points, the amount of calculation can be greatly reduced, the processing speed can be improved, and the apparatus scale can be reduced.

〔Example〕

FIG. 3 shows an embodiment of the image tracking device according to the present invention shown in FIG. 1. In this embodiment, the output signal of the image sensor 1 is sampled by the A / D converter 5 for pixel sampling. One frame is stored in the first frame memory 2 as a digital signal. The correlation calculator 4 calculates a correlation value between pixel data of an actual image in the frame memory 2 and pixel data of a reference image in the frame memory 3.
Of the correlation values calculated by the correlation processor 41, the correlation value register 42 storing the maximum correlation value at each comparison time point, and the correlation value calculated by the correlation processor 41 and accumulated in the correlation value register 42. A correlation value comparator 43 that compares the correlation value and outputs a comparison result, and a maximum value position determiner 44 that determines a maximum correlation value position from the comparison result from the correlation value comparator 43 and outputs target coordinates.
And a tracking command calculator 45 that calculates a tracking command according to the target coordinates from the determiner 44 and provides the tracking command to the visual axis turning mechanism 6, and further searches according to the comparison result from the correlation value comparator 43. A search path generator 46 for moving a point, and an address generator 47 for giving a read address to each of the frame memories 2 and 3 according to a search direction for moving the search point from the generator 46. I have. Note that the correlation value register
Reference numeral 42 denotes a correlation value processor 41 based on the comparison result of the correlation value comparator 43.
Is stored. The visual axis turning mechanism 6 that has received the tracking command from the tracking command calculator 45 gives a visual axis direction for correcting the target direction of the image sensor 1.

Hereinafter, the operation of this embodiment will be described with reference to the calculation process shown in FIG. FIG. 4 corresponds to a software program when the correlation operation section 4 is constituted by a microprocessor.

First, the target position (appropriately set at first) obtained in the previous frame sampling is supplied from the search path generator 46 to the address generator 47 as a search start point, and the address of the target position is transmitted from the address generator 47 to the frame. It is provided to the memories 2 and 3 (step S1 in FIG. 4).

At this point, an actual image is obtained in a form in which the visual axis direction of the image sensor 1 is corrected, and this is sampled by the A / D converter 5 and pixel data for one frame is stored in the memory 2.

Accordingly, the pixel data at the search start point is read out from the frame memories 2 and 3 to which the read address is given from the address generator 47, and the correlation processor 41 uses the correlation value g _{0 at the} search start point based on these pixel data. (X ₀ , y ₀ ) (S
2).

Then, the calculated correlation value _{g 0 (x 0, y 0} ) and stored in the correlation value register 42 to be compared to a reference correlation value g _r of the search points (the S3).

Thereafter, in order to obtain a correlation value of, for example, four adjacent points (corresponding to FIG. 2 (b)) around the search start point, n = 0 is set (S4), and this n is set to "1". The value is incremented (S5), and initially n = 1.

Then, it is determined whether or not n has been calculated for all adjacent points around the search start point (n = 4?) (S6).
These steps S4 to S6 can be performed in the correlation value comparator 43 for each comparison.

Since the correlation value calculation has not yet been completed because n = 1 at this time, the correlation value comparator 43 instructs the correlation processing unit 41 to calculate the correlation value g ₁ (x ₁ , y ₁ ) for n = 1. (Same
7).

The correlation value comparator 43, further judges whether the correlation value g ₁ exceeds the reference correlation value g _r stored in the register 42 to be compared (the S8).

This result, g ₁ <g correlation value has been stored in the correlation value register 42 if _r g _r will be greater than the correlation value g ₁ of the first neighboring point, the correlation value of the correlation value register 42 Without updating
Returning to step S5, n is incremented by "1" and n
= Is a 2, g _1> Since g correlation value if _r g _r will be less than the correlation value g ₁ of the first neighboring point, the correlation value comparator 43
Instructs the correlation value register 42 to set g _r = g _1, and the correlation value register 42 updates the reference correlation value from g ₀ to g ₁ (S9), returns to step S5, and sets n to “ Increment by 1 "to make n = 2.

In this manner, repeatedly executes the above steps until n Correlation value comparator 43 is 4, n = reference correlation values stored in the correlation value register 42 when it becomes 4 g _r
Is equal to or the same as the correlation value g ₀ of the search starting point (the S10).

As a result, when a g _r ≠ g _0, the search path generator 46
So is set as search points once the correlation values g ₀ pixel coordinates of g _r stored in the correlation value register 42 for further movement of the search points in one frame transmitted to the address generator 47 (the S11) and return to step S4. . In this case, the search start point is set in steps S1 to S3, but in step S1
In FIG. 1, since the search within one frame is progressing, it is simply referred to as a search point, and both constitute a search reference point.

On the other hand, when g _r = g ₀ , the correlation value g ₀ of the search point (or search start point) is the correlation value g ₁ (x ₁ , y ₁ ) to g of all the adjacent points.
₄ (x ₄ , y ₄ ), and this search point becomes the target point shown in FIG. 2A, so that the maximum value decision unit 44 receives the comparison result of the correlation value comparator 43 and With the search point as the maximum value position, the target coordinates are instructed to the tracking command calculator 45.

In this way, the target position shown in FIG. 2A can be searched.

〔The invention's effect〕

As described above, according to the present invention, the pixel correlation value between the search reference point and its adjacent point is calculated between the real image and the reference image, the correlation values are compared with each other, and the maximum value is calculated. The point having the correlation value is set as the next search reference point, and this operation is sequentially repeated, so that the tracking command is issued when the correlation value of the new search reference point reaches a value higher than the correlation value of the adjacent point. , The image correlation processing can be limited to the correlation value maximum value search path and its adjacent points, and the amount of calculation is greatly reduced, so that small-scale hardware can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the principle configuration of an image tracking apparatus according to the present invention and a conventional example, FIG. 2 is a diagram for explaining the operation principle of a maximum value search method according to the present invention, and FIG. Block diagram showing an embodiment of the image tracking device according to the present invention, FIG. 4 is a flowchart for explaining the operation of the embodiment of the present invention, FIG. 5 is a diagram showing a real image and a reference image, FIG. 6 is a diagram showing a distribution of correlation values. In FIG. 1, 1... An imaging sensor, 2... A first frame memory, 3... A second frame memory, 4. In the drawings, the same reference numerals indicate the same or corresponding parts.

──────────────────────────────────────────────────続 き Continuation of the front page (72) Inventor Hiroyuki Fujiwara 1015 Kamiodanaka, Nakahara-ku, Kawasaki City, Kanagawa Prefecture Inside Fujitsu Limited (56) References JP-A-60-21472 (JP, A) (58) Fields investigated (Int.Cl. ⁶ , DB name) H04N 5/225-5/232

Claims (1)

(57) [Claims] 1. A first frame memory (2) for storing an actual image signal from an image sensor (1), and a second frame memory (3) for storing a reference image in advance.
And a correlation value between the search reference point and its adjacent point is calculated for each pixel between the real image and the reference image, and the correlation values are compared with each other. By sequentially repeating this calculation, a tracking command is issued to the image sensor (1) when the correlation value of the new search reference point reaches a value higher than the correlation value of the adjacent point. An image tracking device, comprising: a correlation operation unit (4) that outputs the following.