JP2000259838A - Image tracking device and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To be hardly affected by the background change of a tracking object and to perform stable tracking even in the case that the tracking object is partially hidden. SOLUTION: In this device for setting an image area to be tracked (tracking block) on the basis of image data inputted from an image pickup device 31, performing correlation computation with the image area as an object and performing an image tracking processing on the basis of the result of the correlation computation, a control part 32 divides the image area to be tracked into a plurality of small blocks. For each divided small block, a correlation computation part 27 performs the correlation computation and obtains correlation values. The control part 32 evaluates the obtained correlation values for the respective small blocks and uses only the correlation value of a valid small block for the correlation value calculation of the entire image area to be tracked.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image tracking apparatus and a recording medium used for tracking a target moving object using an image of a video camera, for example, for monitoring traffic on a road. About.

[0002]

2. Description of the Related Art A conventional example will be described below. FIG. 12 is an explanatory diagram (part 1) of a conventional example. FIG. 12A is an explanatory diagram of a motion vector measurement by a correlation operation, and FIG. 12B is an explanatory diagram of tracking of a moving object. FIG. 13 is an explanatory view of a conventional example (part 2), and FIG.
FIG. 4 is an explanatory view of a conventional example (part 3), FIG. 15 is an explanatory view of a conventional example (part 4), and FIG. 16 is an explanatory view of a conventional example (part 5). Hereinafter, a conventional example will be described with reference to FIGS.

Conventionally, there has been known an image tracking device for tracking a target moving object based on image data taken from a video camera. In this image tracking device, an image tracking process based on pattern matching using a correlation operation has been performed. The principle will be described with reference to FIGS. In FIG. 12, a frame f is an image stored in the frame memory at a certain time, and a frame f + 1 is an image stored in the frame memory at the next time.

As shown in FIG. 12A, in the correlation calculation, a rectangular area 1 to be tracked (image area of frame f)
Is extracted as a reference block (hereinafter also referred to as a “template”), and the next candidate block 3 (frame f + 1
Is calculated.

In this case, two-dimensional scanning is performed while moving the candidate block 3 one pixel at a time, and a position having the highest correlation (= the smallest correlation value) is obtained. A motion vector 5 indicating where the moving object 4 such as the helicopter shown in FIG. 12B has moved in the next frame is obtained. The following equation is used for calculating the correlation value.

[0006]

(Equation 1)

In the above equation, the reference block (template) 1 is an image of 8 × 8 pixels, and the total sum of the absolute values of the density differences of the pixels in the reference block 1 is obtained. Where R
(X, y) is the image data of the reference block 1 and S (x, y) is the image data of the search block 2.

In this case, the two-dimensional coordinates X and Y are set to + in the right direction and the downward direction in the figure, and − in the upward direction and the left direction.
The search block 2 divides the candidate block 3 from the end of the candidate block 3 by 7 pixels to the right in the drawing and to the left by −
The size is set so that it can be moved by 8 pixels, by -8 pixels in the upward direction, and by 7 pixels in the downward direction.

This technique is based on the SAD (Sum of Absolute Di
fference), and the D (x,
y) is called a correlation value (or distortion), and the smaller the correlation value is, the higher the correlation is and the position where the correlation value is the smallest is calculated to calculate the motion vector 5. When the motion vector 5 is obtained, the moving amount of the image to be tracked can be known. Therefore, by repeatedly obtaining the motion vector 5, a specific moving object can be tracked.

At this time, the position of the search block 2 is moved in accordance with the obtained motion vector 5, as shown in FIG. In the above example, the tracking of an area of 8 × 8 pixels is performed, but by combining this local correlation operation, a correlation operation of an image of a wide area can be performed. This is called cumulative correlation, and its formula is as follows:

[0011]

(Equation 2)

FIG. 13 shows the correlation calculation result of 8 × 8 pixels as 4 points.
The principle of a method of performing a correlation operation on a region of 16 × 16 pixels by accumulating the times will be described. As shown, the 16 × 16 pixel area is divided into 8 × 8 pixel B1, B2, B3, B
The area is divided into four areas consisting of four areas. Then, a correlation operation is performed on each of the divided regions B1, B2, B3, and B4 of 8 × 8 pixels to create a correlation value map.

For example, the correlation value map of the area B1 = S1,
Assuming that the correlation value map of the area B2 = S2, the correlation value map of the area B3 = S3, and the correlation value map of the area B4 = S4, the entire correlation value map S is S = S1 + S2 + S3 + S4.

However, in the above-described method, when the background of the tracking image changes or when a part of the object being tracked is hidden, the correlation deteriorates (the correlation value increases), and stable tracking can be performed. May disappear. Figure 1 shows this situation.
It is shown in FIG. In FIG. 14, for example, when the tracked object is an automobile 7, the automobile 7 runs and a part of the automobile 7 is hidden by a tree 8 (partially hidden), or a mountain 9
When the tree 8 or the tree 8 appears (background change), the correlation becomes worse (the correlation value increases) due to the tree 8 or the mountain 9, and stable tracking may not be performed.

When the moving object being tracked is rotated (for example, when the helicopter is tracked and the helicopter is rotated in the air as shown in FIG. 12B),
The shape of the tracked object may change significantly on the image.
In such a case, a method of responding by sequentially updating the template has been conventionally proposed. However, when a background or a partial occlusion occurs, tracking may not be able to continue. This is shown in FIG.

In the example shown in FIG. 15, the updated template includes the changed background, and the tracking block remains in place because the correlation of the background portion is high even though the tracking target has moved. It is. In this example, the first template includes the car 7 and the car 7
Has no background (no background).

It is assumed that the car 7 travels from such a state, the background changes over time, and trees 8 and mountains 9 appear in the background.
In such a case, the template is updated to the image of the automobile 7 with the tree 8 and the mountain 9 as a background, and the tracking is continued. But,
At this time, the correlation becomes high in the background of the tree 8 and the mountain 9.
When the car 7 travels and another background appears, this image is ignored because it has low correlation. Therefore, the tracking block stays there.

Further, when a plurality of objects having similar shapes pass each other, the tracking may shift to another object. This is shown in FIG. The example of FIG. 16 shows a state in which two vehicles 7 pass each other. In this case, if the two vehicles 7 have similar shapes, the tracking target may move to another vehicle 7 that is not the tracking target. Therefore, tracking of the tracking object becomes impossible.

[0019]

The above-mentioned prior art has the following problems.

(1): When the background of the tracking image changes,
When a part of the object being tracked is hidden, the correlation value becomes worse, and stable tracking may not be performed.

(2): When the moving object being tracked rotates, the shape of the tracked object may change greatly on the image. In such a case, a method of responding by sequentially updating the template has been conventionally proposed. However, when a background or a partial occlusion occurs, tracking may not be able to continue.

(3): If the updated template contains a changed background, the tracking target has moved,
The correlation of the background part becomes high, and the tracking block may stay in place.

(4): When a plurality of objects having similar shapes pass each other, tracking may shift to another object. In such a case, tracking of the tracking object may not be possible.

SUMMARY OF THE INVENTION The present invention solves such a conventional problem, and makes it possible to perform stable tracking even when a part of the tracked object is hidden by making the tracked object less affected by a background change. With the goal. Another object of the present invention is to make it possible to continue tracing when a partially hidden object occurs in a tracked object and to perform tracking while coping with both shape change and partially obscured. Furthermore, by taking into account the motion prediction,
An object of the present invention is to enable tracking to be continued even when two objects pass each other.

[0025]

FIG. 1 is a diagram for explaining the principle of the present invention, FIG. 1A is a diagram showing the configuration of an apparatus, and FIG. In FIG. 1, reference numeral 32 denotes a control unit (for example, CP
U), 21 denotes a memory, 27 denotes a correlation operation unit, 12 denotes a display device, and 31 denotes an imaging correlation (for example, a video camera). The present invention has the following configuration to achieve the above object.

(1): An image area (tracking block) to be tracked is set from input image data (data input from the image pickup apparatus 31 to the main body of the image tracking apparatus and stored in the memory 21), and this image area is targeted. In an image tracking device that performs a correlation operation and performs an image tracking process based on a result of the correlation operation, an image area dividing unit (a part of the control unit 32) that divides the image area to be tracked into a plurality of small blocks; Correlation calculation means (correlation calculation unit 27) for calculating a correlation value by performing a correlation calculation for each divided small block, and evaluating the obtained correlation value for each small block to obtain an effective small block (FIG. 1B). The correlation value processing means (a part of the control unit 32) is used for calculating the correlation value of only the entire image area to be tracked using only the correlation value (excluding the thick frame block shown in FIG. 3).

(2): In the image tracking device of the above (1),
The correlation value processing means, based on the correlation value of the small block having the highest correlation among the divided small blocks, if the difference from the correlation value is within a preset threshold, (A part of the control unit 32).

(3): In the image tracking device of the above (1),
The correlation value processing means calculates in advance the average luminance of the entire image area to be tracked and the average luminance of each of the divided small blocks, and only the small blocks having an average luminance higher than the average luminance of the entire image area are valid. An effective block determination unit (a part of the control unit 32) for determining a small block is provided.

(4): In the image tracking device of (1),
A storage unit (memory 21) for storing the effective small block arrangement pattern and the number of blocks at the start of tracking, and comparing the storage data of the storage unit with the arrangement pattern of the image area to be tracked and the number of blocks. If the layout pattern at the start of tracking is maintained, the template obtained from the image area (tracking block) to be tracked is updated,
When the arrangement pattern at the start of tracking changes, the template control unit (control unit 3) stops updating the template.
2).

(5): In the image tracking device of the above (1),
A correlation value storage unit (memory 21) for storing history data of correlation values for a plurality of past frames; and obtaining an average correlation value for a plurality of past frames from the history data; Value accumulating means (control unit 3) for accumulating the correlation value obtained in the current frame as the predicted value of the motion of
2).

(6): An image area to be tracked is set from the input image data, a correlation operation is performed on the image area, and the image is tracked by an image tracking apparatus that performs an image tracking process based on the result of the correlation operation. Image area dividing means for dividing the image area into a plurality of small blocks, correlation calculating means for performing a correlation operation for each divided small block to obtain a correlation value, and evaluating the calculated correlation value for each small block, A computer-readable recording medium in which a program for realizing a function of a correlation value processing unit that uses only a correlation value of an effective small block for calculating a correlation value of the entire image area to be tracked is recorded.

(Operation) The operation of the present invention based on the above configuration will be described.

(A): In the above (1), when performing the image tracking process, the image area dividing means divides the image area to be tracked into a plurality of small blocks, and the correlation calculating means performs the correlation calculation for each of the divided small blocks. The correlation value processing means evaluates the correlation value for each small block (valid / invalid evaluation), excludes invalid small blocks, and sets only the correlation value of valid small blocks. Is used for calculating the correlation value of the entire image area to be tracked.

In this way, it is possible to reduce the influence of the change of the background image on the tracking and to cope with the case where a part of the tracking object is hidden. That is, in the conventional apparatus, when the background of the tracking image changes or when a part of the object being tracked is hidden, the correlation value becomes worse, and stable tracking may not be performed. Then, such a conventional disadvantage is solved, and stable image tracking processing can always be performed.

(B): In the above (2), the effective block determining means sets a difference between the divided block and the correlation value in advance based on the correlation value of the small block having the highest correlation among the divided small blocks. If it is within the threshold value, it is determined that the block is a valid small block, and the others are determined to be invalid small blocks.

In this way, the validity / invalidity of a small block can be easily and reliably determined. Then, the influence of the change of the background image on tracking is reduced, and stable image tracking processing can be performed even when a part of the tracking object is hidden.

(C): In the above (3), the effective block determination means calculates in advance the average luminance of the entire image area to be tracked and the average luminance of each of the divided small blocks, and calculates the average luminance of the entire image area. Only small blocks having a higher average luminance are determined as valid small blocks.

In this way, when performing the image tracking process based on the image input from the infrared imaging camera, the validity / invalidity of the small block can be reliably determined. That is, when a moving object is tracked using an image input from an infrared imaging camera, the object to be tracked is often a heat source such as a vehicle or a person, and thus the brightness of the image of the tracked object is higher than that of the background. Taking advantage of this, when first setting a small block, effective blocks are narrowed down in advance. In this way, the validity / invalidity of the small block can be reliably determined, and the influence of the background change can be reduced in the image tracking process.

(D): In the above (4), under the control of the control unit 32, the effective small block arrangement pattern and the number of blocks at the start of tracking are stored in the storage means. Then, the template control means compares the stored data of the storage means with the arrangement pattern and the number of blocks of the image area to be tracked, and if the layout pattern at the time of starting the tracking is maintained, the image area to be tracked When the arrangement pattern at the start of tracking changes, the update of the template is stopped.

In this way, it is possible to achieve both the tracking of the shape change of the tracking object by updating the template and the coping with the partial hiding of the object.

(E): In the above (5), under the control of the control unit 32, the correlation value storage means stores correlation values for a plurality of past frames as history data. The correlation value accumulating means obtains an average correlation value of a plurality of past frames (for example, several frames) from the history data, and obtains the average correlation value as a predicted value of the movement of the tracking object in the current frame. Accumulates in the correlation value. In this way, tracking can be continued stably due to passing or the like.

(F): In the above (6), the control unit 32 (for example, CPU) reads and executes a program on the recording medium to divide an image area to be tracked into a plurality of small blocks; A procedure for calculating a correlation value by performing a correlation operation for each of the divided small blocks, and evaluating the correlation value for each of the obtained small blocks to determine only a correlation value of an effective small block and a correlation of the entire image area to be tracked. And the procedure used for value calculation. In this way, it is possible to reduce the influence on the tracking due to the change of the background image, and to cope with a case where a part of the tracking object is hidden.

[0043]

Embodiments of the present invention will be described below in detail with reference to the drawings.

§1: Description of Image Tracking Apparatus FIG. 2 is an apparatus configuration diagram. In the present embodiment, an image tracking process described below is performed using the apparatus shown in FIG. In this example, an image tracking device is configured using various computers such as a personal computer and a workstation. A computer main body 11, a display device 12, a keyboard 13, and a removable disk drive connected to the computer main body 11 are provided. (Hereinafter "RD
D) 14, a hard disk drive (hereinafter “HD”).
D), a video camera 16, an infrared imaging camera 17, and the like.

The computer main body 11 has:
A CPU 20 for performing various controls in the computer, and a memory 21 for writing and reading various data
And a ROM 22 for storing programs and various data
And a video board 23 for performing image correlation calculation.

The video board 23 is commercially available for realizing the SAD method, and is an interface control unit (hereinafter referred to as an I / O controller) for controlling an interface with the video camera 16 and the infrared imaging camera 17. F
Control unit 25), a frame memory 26 for storing an input image for each frame, a correlation operation unit 27 for performing a correlation operation, a command shared memory 28, a data shared memory 29, and the like. It is provided.

The command shared memory 28 and the data shared memory 29 are memories shared by the CPU 20 and the correlation operation unit 27, and exchange commands and data between them. The I / F control unit 25 controls an interface with the video camera 16 and the infrared imaging camera 17, controls an input image input from the camera, and stores an image in a frame unit into a frame memory 26.
Are stored sequentially.

In this case, if the output image of the camera is analog image data, the input image data is converted to digital data by an ADC (analog / digital converter), and the converted digital data is stored in a frame memory 26 in frame units. Are stored sequentially. If the output of the camera is digital image data, the digital image data from the camera is converted and sequentially stored in the frame memory 26 in frame units.

The program executed by the CPU 20 and data such as various parameters are stored in advance in the storage medium of the HDD 15 or the ROM 22, and the CPU 20 reads out necessary data and performs an image tracking process. In the process of performing this processing, the CPU 20 uses the memory 21 as a work memory.

The image tracking process is performed under the control of the CPU 20, and the outline thereof is as follows. Input image data from the video camera 16 or the infrared imaging camera 17 is transmitted to an I / F
The control unit 25 controls the frame memory 2 in frame units.
6 sequentially. Then, the correlation operation unit 27 performs a correlation operation in accordance with an instruction command of the CPU 20, and the CPU 20 acquires data of the operation result, stores the data in the memory 21, and performs the processing.

In this case, when the CPU 20 stores a command instructed to the correlation operation unit 27 in the command shared memory 28, the command is read out by the correlation operation unit 27. Then, the correlation operation unit 27 reads the corresponding image from the frame memory 26 according to the read command,
Perform correlation calculation. The data obtained as a result of performing the correlation operation is stored in the data shared memory 29 by the correlation operation unit 27, and this data is read out by the CPU 20.
20 obtains the data of the correlation operation result. When performing the above processing, the CPU 20 stores various data (image data, correlation value data, and the like acquired from the frame memory 26) in the memory 21.
Store in and work.

Although the correlation operation section 27 is an operation section having a hardware configuration, the present invention is not limited to such an example.
May be realized by a program executed by the software (correlation calculation unit having a software configuration).

§2: Overview of Image Tracking Process—See FIG. 3 FIG. 3 is a schematic explanatory diagram of the image tracking process. This example is 3
This is an example using a tracking block of 2 × 32 pixels. In this example, a region of 32 × 32 pixels of the frame f is acquired as a template and compared with the frame f + 1. In this case, for example, the automobile 7 is present in the area and the frame f
At +1 a part of the car 7 is shaded by a tree 8. The outline of the processing for such an example will be described below.

In the present invention, it is hardly affected by the background change.
Even if a part of a tracking object is hidden, tracking can be stably continued. In addition, while responding to changes in the shape of the tracked object by updating the template normally, if partial occlusion occurs, updating the template is temporarily stopped,
It enables tracking while coping with both shape change and partial occlusion without breaking tracking. Further, by taking into account the motion prediction, tracking can be continued even when two objects pass each other.

Therefore, as shown in FIG.
The pixel tracking block is divided into several small blocks 35 (8 × 8 pixels in this example), and each of the small blocks 35
To calculate a correlation value. When all the correlation values for each of the small blocks 35 are accumulated, the correlation value becomes equal to the correlation value of the entire tracking block. At this time, the correlation value of the small block 35 having a low correlation is excluded from the accumulation target, thereby achieving stable tracking.

If there is a change in the background (see the conventional example) or a partial occlusion (for example, in the frame f + 1 in FIG. 3, a part of the car 7 is hidden by the tree 8), the background change or Since the correlation value of a small block with occlusion becomes worse, do not accumulate the correlation value of that small block,
Eliminate the effects of background changes and hiding.

That is, in the example of FIG. 3, a small block (four small blocks shown by thick lines) in which a part of the automobile 7 is hidden by the tree 8 is set as an exclusion area 36 to be excluded from the accumulation of the correlation values. Then, the correlation values of the remaining small blocks (12 small blocks in this example) except for the four small blocks set as the exclusion area 36 are accumulated.

§3: Description of Image Tracking Process—See FIGS. 4 to 7 FIG. 4 is a flowchart (part 1) of the image tracking process, and FIG. 5 is a flowchart (part 2) of the image tracking process. Also,
FIG. 6 is an explanatory diagram of an image tracking process (part 1), FIG. 7 is an explanatory diagram of an image tracking process (part 2), and A to H are explanatory diagrams of each process. Hereinafter, the image tracking processing will be described with reference to FIGS. 6 and 7 based on the processing flowcharts of FIGS. In addition, S1 to S16 indicate each processing step.

First, the CPU 20 sets a tracking block as shown in FIG. 6A and obtains a template (S
1). In this case, the tracking block is, for example, 32 × 32
This is a pixel area. Next, the CPU 20 divides the set tracking block into a plurality of small blocks (for example, 8 × 8 pixels) as shown in FIG. 6B (S2). And FIG.
As shown in FIG. 3C, the correlation calculation unit 27 performs a correlation calculation for each of the divided small blocks (1 to n) based on the control of the CPU 20 (S3), and the CPU 20 determines the correlation values d1 to dn for each of the small blocks. Get.

Next, the CPU 20 detects the minimum correlation value Di (i = 1 to n) for each small block as shown in FIG. 6D (S4), and as shown in FIG. A minimum correlation value (hereinafter, referred to as “min”) in the small block is detected (S
5). Next, the CPU 20 sets an arbitrary parameter i to i =
1 (S6), and as shown in FIG.
ind) <It is determined whether or not a condition of a threshold Th is satisfied (S7).

As a result, if the condition of (Di-mind) <threshold Th is satisfied, the correlation value is accumulated (S
8) It is determined whether i ≦ n (S9). as a result,
If the condition of i ≦ n is satisfied, the parameter i is incremented to i = i + 1 (S10), and the processing is repeated from the processing of S7. If the condition of (Di-mind) <threshold Th is not satisfied in the process of S7, the process of S9 is performed.

When the processing is performed as described above and the condition of S9 is no longer satisfied, that is, when i> n, the CPU 20 calculates a predicted value as shown in FIG. A map is created (S11). Then, the prediction value is determined (S12). As a result, if the predicted value is valid, the predicted value is accumulated (S13).

That is, the average correlation value per small block is calculated from the accumulation result of the correlation values of the effective blocks and the number of the effective blocks, and the average correlation value for the past several frames (for a plurality of frames) is stored. Then, by dividing by the number of frames, a predicted value is calculated and accumulated in the correlation value of the current frame, thereby realizing stable tracking due to passing or the like.

At this time, the prediction value is multiplied by a weight coefficient k so that the prediction ratio can be adjusted. Note that St is a correlation value map calculated for the current frame, Sf is a prediction correlation value map calculated from the correlation of past frames, and S is S =
It is a correlation value map obtained by (St + k · Sf) / (k + 1) (details will be described later).

Thereafter, the CPU 20 calculates the motion vector of the tracking block as shown in H of FIG. 7 (S1).
4), the position of the tracking block is updated (S15). However, if the predicted value is invalid in the process of S12, the S
14 is performed. In this way, the process from S2 is repeated until all the processes are completed (S16), and when all the processes are completed, this process is completed.

In the above processing, S11 to S13
If i ≧ n in the process of S9 without performing the process of
14 can also be performed.

§4: Detailed description of the image tracking process—see FIGS. 8 to 11 FIG. 8 is a specific example of the image tracking process, FIG. 9 is a diagram illustrating the processing of the arrangement pattern and the number of blocks, and FIG. And FIG. 11 is an explanatory diagram of processing of a predicted value. Hereinafter, the image tracking process will be described in more detail with reference to FIGS.

(1): The image tracking process is as described above, but the determination of the small block to be excluded from the process of accumulating the correlation value is based on the small block having the highest correlation (= the smallest correlation value). And do it. If the difference from the correlation value of the small block having the highest correlation is less than a preset threshold Th, the target is to be accumulated, and if the difference is equal to or greater than the threshold Th, it is excluded from the accumulation.

(2): Infrared imaging camera 17 (see FIG. 2)
When tracking a moving object using an image input from a computer, the object to be tracked is often a heat source such as a vehicle or a person. When setting a small block for the first time, effective blocks are narrowed down in advance. With this processing, it is possible to make the influence of the change in the background less likely.

For example, as shown in FIG.
U20 calculates in advance the average luminance of the entire tracking block and the average luminance of the small blocks, and makes only the small blocks having an average luminance higher than the average luminance of the entire block the accumulation target. Usually, in order to follow the change in the shape of the tracking object, a method of sequentially updating the template is adopted, but when hiding occurs, the template is updated in a state where the hiding has occurred, so the tracking is stuck. (Tracking may fail).

On the other hand, the number and the arrangement pattern of the effective blocks set by the above method are controlled by the CPU 20.
When the number or pattern is changed, the CPU 20 determines that partial occlusion has occurred, and stops updating the template. By performing such control, it is possible to achieve both the tracking of the shape change of the tracking object by updating the template and the coping with the partial hiding of the object. This situation is shown in FIGS.

The tracking block shown in FIG. 8A is, for example, 32 × 32 pixels, and the small block is 8 × 8 pixels. Then, the average luminance of the tracking block and the average luminance of the small block are calculated, and the small blocks satisfying the condition of (average luminance of the tracking block) <(average luminance of the small block) (the range indicated by the thick black line in the figure) Only the small blocks within the block). Since the luminance appears as a density on the image, the average luminance is the same as the average density on the image.

Then, as shown in FIG. 8B, the arrangement pattern of the blocks to be accumulated and the accumulated number of blocks (the number of blocks = 6 in this example) are stored in the memory 21 under the control of the CPU 20 (FIG. 8B). The storage method will be described later). Thereafter, in the next frame, the CPU 20 compares the data of the previous frame (the arrangement pattern and the data of the number of accumulated blocks), and updates the template if the arrangement and the number of blocks to be accumulated do not change. When the arrangement of the blocks to be accumulated and the number of blocks are changed, processing for stopping the update of the template is performed.

For example, in a certain frame f, the one having the arrangement pattern and the number of blocks (= 6) shown in FIG. 8B is changed to the next frame f + 1 as shown in FIG. Let's say In this case, the car 7 to be tracked
Are hidden by the shadow of the tree 8, the arrangement pattern and the number of blocks to be accumulated (= 3) are changed. In this case, the updating of the template is stopped.

That is, in order to follow the change in the shape of the tracking object, a method of sequentially updating the template is usually employed. However, even if the tracking target object is partially hidden, the template is updated. Tracking may get stuck (cannot be tracked).

However, in the above method, the set number of effective blocks and the arrangement pattern are stored in the memory 21, and when the stored arrangement pattern or the number of blocks is changed, it is determined that partial occlusion has occurred. However, by stopping the update of the template, it is possible to achieve both the tracking of the shape change of the tracking object by the template update and the correspondence to the partial hiding of the object.

(3): The storage processing of the arrangement pattern and the number of blocks to be accumulated described in (2) and the comparison processing will be described with reference to FIG. When storing the arrangement pattern,
A bit string in which the accumulation-target small block is set to “1” and other non-accumulation target small blocks are set to “0” is created and stored as arrangement pattern data (for example, stored in the memory 21). ).

When the arrangement patterns are compared between the frames, it may be determined whether or not they match by simply comparing the stored bit strings. For example, FIG. 9 shows an example in which a tracking target block of 32 × 32 pixels is divided into 4 × 4 small blocks. When the number of divisions is increased, it is only necessary to simply increase the width of the bit string to be stored.

As shown in FIG. 9D, the tracking block is divided into 4 × 4 small blocks, and row 1, row 2, row 3 and row 4 are each expanded in the direction of the arrow shown in FIG. To In this manner, the small blocks of the tracking block are arranged in a line, and for each small block, 1 (the small block to be accumulated =
1) or 0 (small block other than accumulation target = 0) is inserted to create a bit string of an arrangement pattern.

For example, in the case of the arrangement pattern shown in FIG. 9E (the same as the example of FIG. 8), as shown in FIG. 9F, the black portion (small black block) is 1 (accumulated). Target), and all other parts are set to 0 (non-cumulative target).
In this case, the bit string of the arrangement pattern is “000001”.
1011110000 ".

Therefore, this bit string is stored in the memory 21. For example, if the bit string is expressed in hexadecimal, it is 0x06F0, so this is substituted for a variable (patt
ern = 0x06F0) It may be stored in the memory 21. And
When comparing arrangement patterns between frames, data (pattern = 0x06F0) stored in the memory 21 may be used.

(4): The process of accumulating the predicted values is as follows. In the tracking process, the CPU 20 calculates an average correlation value per small block from the accumulation result of the correlation values of the effective blocks (accumulation target small blocks) and the number of effective blocks, and calculates the average correlation value for the past several frames (a plurality of Frame)
The prediction value is calculated by storing them in the memory 21 and dividing them by the number of frames, and accumulating them in the correlation value of the current frame.

In this way, stable tracking can be realized even when the moving objects pass each other. On this occasion,
The prediction value is multiplied by a weighting factor so that the prediction ratio can be adjusted.

For example, as shown in FIG. 10, the correlation map calculated for the current frame is St, and the correlation map calculated for the past frame is St-1 and St-, respectively.
2, St-3... St-n. In this case, St-
1 is a correlation value map of a frame before St, and St-
2 is a correlation value map of a frame preceding St-1;
t-3 is a correlation value map of the frame immediately before St-2.

The correlation value maps St-1, St-2, St-3,..., St-n calculated in such past frames.
Are sequentially stored in the memory 21, and these are accumulated to perform an averaging process to create a predicted correlation value map Sf. Then, the correlation value map S calculated for the current frame
The correlation value map Sf of the prediction is added to t to calculate the correlation value map S. In this case, the weight count is k, and S =
The correlation value map S is calculated by the equation (St + k · Sf) / (k + 1).

In this way, when the number of valid blocks (small blocks to be accumulated) is small, it is possible to perform an adaptive operation such as increasing the prediction ratio. In addition, when accumulating the predicted value, it is determined whether the predicted value is valid,
Accumulate only when valid. The method of determining the predicted value is shown in FIG.

In FIG. 11A, Cc is a central correlation value, Cs is a correlation value at a position having the highest correlation, and Th is a threshold value. The absolute value (Cc-Cs) of the difference between the central correlation value Cc on the correlation value map and the correlation value Cs at the position with the highest correlation is calculated, and if the absolute value (Cc-Cs) exceeds the preset threshold Th, (Cc -Cs ≧ Th) valid, if not exceeded (Cc-
Cs <Th) It is determined to be invalid.

In the case of an oscillating movement of the tracked object, the correlation value indicates a value with no average change. In such a case, it is determined that the predicted value is not valid. For example, as shown in FIG. 11B, when tracking a ship floating between the waves, the ship that is the tracking object vibrates up and down in accordance with the up and down of the wave.

In this case, the predicted value has an adverse effect on tracking. In the case of such an oscillating motion, each correlation value on the correlation value map becomes average, and it can be determined by eliminating the difference between Cc and Cs.

§7: Description of Recording Medium and Program The processing performed by the image tracking device is the CP in the image tracking device.
The U20 executes by executing a program stored in the ROM 22 or the disk of the HDD 15, but is not limited to such an example, and can be realized as follows. In this case, the program stored in the disk of the HDD 15 is recorded (stored) as follows.

: Program stored on a removable disk (program data created by another device)
Is read by the RDD 14 and stored in the recording medium (hard disk) of the HDD 15.

: Magneto-optical disk or CD-ROM
The data stored in a storage medium such as is read by a drive device (such as a CD-ROM drive) provided in the computer main body 11 and stored in a recording medium (hard disk) of the HDD 15.

The computer main unit 11 receives data such as a program transmitted from another device via a communication line such as a LAN via a communication control unit, and transmits the data to the H
It is stored in the recording medium (hard disk) of the DD15.

[0094]

As described above, the present invention has the following effects.

(1): When performing the image tracking process, the tracking object is hardly affected by the background change, and stable tracking can be performed even when the tracking object is partially hidden.

(2): Even if the tracked object is partially hidden, the tracking can be continued. Further, it is possible to perform tracking while coping with both the shape change and partial occlusion of the tracked object.

(3): Tracking can be continued even when two objects pass each other by taking into account the motion prediction data of the tracked object.

(4): When performing the image tracking process, the image area dividing means divides the image area to be tracked into a plurality of small blocks, and the correlation calculating means performs a correlation calculation for each of the divided small blocks to calculate a correlation value. The correlation value processing means evaluates the calculated correlation value for each small block, excludes invalid small blocks, and uses only the correlation values of valid small blocks in calculating the correlation value of the entire image area to be tracked. Used.

In this way, it is possible to reduce the influence of the change of the background image on the tracking and to cope with the case where a part of the tracking object is hidden. That is, in the conventional apparatus, when the background of the tracking image changes or when a part of the object being tracked is hidden, the correlation value becomes worse, and stable tracking may not be performed. Then, such a conventional disadvantage is solved, and stable image tracking processing can always be performed.

(5): The effective block judging means uses the correlation value of the small block having the highest correlation among the divided small blocks as a reference if the difference from the correlation value is within a predetermined threshold value. , Is determined to be a valid small block, and the others are determined to be invalid small blocks.

In this way, the validity / invalidity of a small block can be easily and reliably determined. Then, the influence of the change of the background image on tracking is reduced, and stable image tracking processing can be performed even when a part of the tracking object is hidden.

(6) The effective block determination means calculates in advance the average luminance of the entire image area to be tracked and the average luminance of each of the divided small blocks, and has an average luminance higher than the average luminance of the entire image area. Only small blocks are determined to be valid small blocks.

Thus, when performing the image tracking process based on the image input from the infrared imaging camera, the validity / invalidity of the small block can be reliably determined. That is, when a moving object is tracked using an image input from an infrared imaging camera, the object to be tracked is often a heat source such as a vehicle or a person, and thus the brightness of the image of the tracked object is higher than that of the background. Taking advantage of this, when first setting a small block, effective blocks are narrowed down in advance. In this way, the validity / invalidity of the small block can be reliably determined, and the influence of the background change can be reduced in the image tracking process.

(7): The effective small block arrangement pattern and the number of blocks at the start of tracking are stored in the storage means.
The template control unit compares the stored data of the storage unit with the arrangement pattern and the number of blocks of the image area to be tracked, and acquires the data from the image area to be tracked when the arrangement pattern at the time of starting the tracking is maintained. The updated template is updated, and if the arrangement pattern at the start of tracking changes, the updating of the template is stopped. With this configuration, it is possible to achieve both the tracking of the shape change of the tracking object by updating the template and the response to the partial hiding of the object.

(8): Correlation value storage means stores history values of correlation values for a plurality of past frames, and correlation value accumulation means stores the correlation values for a plurality of past frames (for example,
The average correlation value of several frames is calculated, and the average correlation value is accumulated as the predicted value of the motion of the tracking object in the correlation value obtained in the current frame. In this way, tracking can be continued stably due to passing or the like.

(9): A procedure for dividing an image area to be tracked into a plurality of small blocks by reading and executing a program on a recording medium, and a procedure for obtaining a correlation value by performing a correlation operation for each of the divided small blocks. And a procedure of evaluating the obtained correlation value of each small block and using only the correlation value of the effective small block for calculating the correlation value of the entire image area to be tracked. In this way, it is possible to reduce the influence on the tracking due to the change of the background image, and to cope with a case where a part of the tracking object is hidden.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a device configuration diagram according to an embodiment of the present invention.

FIG. 3 is a schematic explanatory diagram of an image tracking process according to the embodiment of the present invention.

FIG. 4 is a flowchart (part 1) of an image tracking process according to the embodiment of the present invention.

FIG. 5 is a flowchart (part 2) of an image tracking process according to the embodiment of the present invention.

FIG. 6 is an explanatory diagram (1) of an image tracking process according to the embodiment of the present invention.

FIG. 7 is an explanatory diagram (part 2) of an image tracking process according to the embodiment of the present invention.

FIG. 8 is a specific example of an image tracking process according to the embodiment of the present invention.

FIG. 9 is an explanatory diagram of processing of an arrangement pattern and the number of blocks according to the embodiment of the present invention.

FIG. 10 is a diagram illustrating a process of accumulating predicted values according to the embodiment of the present invention.

FIG. 11 is an explanatory diagram of processing of a predicted value according to the embodiment of the present invention.

FIG. 12 is an explanatory diagram (part 1) of a conventional example.

FIG. 13 is an explanatory diagram (part 2) of a conventional example.

FIG. 14 is an explanatory diagram (part 3) of a conventional example.

FIG. 15 is an explanatory view (No. 4) of a conventional example.

FIG. 16 is an explanatory view (No. 5) of a conventional example.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Reference block (template) 2 Search block 3 Candidate block 4 Moving object 5 Motion vector 7 Car 8 Tree 9 Mountain 11 Computer main body 12 Display device 13 Keyboard 14 Removable disk drive (RDD) 15 Hard disk device (HDD) 16 Video camera 17 Infrared ray Imaging camera 20 CPU (central processing unit) 21 memory 22 ROM (read only memory) 23 video board 25 interface control unit (I / F control unit) 26 frame memory 27 correlation operation unit 28 shared memory for command 29 shared memory for data 31 Imaging device 32 Control unit 35 Small block 36 Exclusion area

Continued on front page F term (reference) 5B057 BA02 CA08 CA12 CA16 CC02 DA07 DB02 DB09 DC30 DC34 5H303 BB02 CC02 EE03 EE09 FF14 GG14 HH01 JJ02 LL03 5L096 BA02 BA04 CA02 FA32 FA34 GA19 HA03 HA05 JA09 KA03

Claims (6)

[Claims] 1. An image tracking apparatus for setting an image area to be tracked from input image data, performing a correlation operation on the image area, and performing an image tracking process based on a result of the correlation operation. Image area dividing means for dividing an area into a plurality of small blocks, correlation calculating means for performing a correlation operation for each divided small block to obtain a correlation value, and evaluating the obtained correlation value for each small block to be effective An image tracking device comprising: a correlation value processing unit that uses only the correlation value of a small block for calculating the correlation value of the entire image area to be tracked. 2. The method according to claim 1, wherein the correlation value processing unit determines, based on a correlation value of a small block having the highest correlation among the divided small blocks, a difference from the correlation value within a predetermined threshold value. 2. The image tracking apparatus according to claim 1, further comprising an effective block determining unit that determines that the block is an effective small block. 3. The correlation value processing means calculates in advance the average luminance of the entire image area to be tracked and the average luminance of each of the divided small blocks, and calculates the average luminance of the image area having a higher average luminance than the average luminance of the entire image area. 2. The image tracking apparatus according to claim 1, further comprising an effective block determination unit that determines only a block as an effective small block. 4. A storage means for storing an arrangement pattern and the number of blocks of effective small blocks at the start of tracking, and comparing data stored in the storage means with an arrangement pattern and the number of blocks of an image area to be tracked. A template control unit that updates the template acquired from the image area to be tracked when the arrangement pattern at the start of tracking is maintained, and stops updating the template when the arrangement pattern at the start of tracking changes. The image tracking device according to claim 1, wherein: 5. A correlation value storage means for storing history data of correlation values of a plurality of past frames, obtaining an average correlation value of a plurality of past frames from the history data, and tracking the average correlation value. 2. The image tracking apparatus according to claim 1, further comprising a correlation value accumulating means for accumulating a correlation value obtained in the current frame as a predicted value of the motion of the object. 6. An image tracking device that sets an image area to be tracked from input image data, performs a correlation operation on the image area, and performs an image tracking process based on a result of the correlation operation. Means for dividing the image into a plurality of small blocks, correlation calculating means for performing a correlation operation on each of the divided small blocks to obtain a correlation value, and evaluating the obtained correlation value for each small block to obtain an effective A computer-readable recording medium in which a program for realizing a function of a correlation value processing means for using only a correlation value of a small block for calculating a correlation value of the entire image area to be tracked is recorded.