JPH1186004A - Moving body tracking device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the real-time performance and tracking performance by finding color information, differential information, and target position information based upon the prediction of movement based upon movement in the past in the form of two-dimensional maps and detecting the target position according to the map obtained by integrating them. SOLUTION: This device consists of a color space distortion map generation part 10, a differentiating process part 20, a differential space distortion map generation part 30, an evaluated value map generation part 40, a minimum position detection part 50,a movement prediction value and map generation part 60, and a color template update part 70. Further, a correlator 80 is provided and shared by the color space distortion map generation part 10, differentiating process part 20, and differential space distortion map generation part 30. Then, the color information, the differential information, and the target position information based upon the prediction of the movement based upon the movement in the past are found in the form of the two-dimensional maps, which are integrated to detect the target position on the basis of the integrated map.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of inputting an original image composed of a plurality of frames which are successively formed by photographing a scene including one or more movement values, and tracking the image displayed in the original image. The present invention relates to a moving object tracking device that tracks a moving object.

[0002]

2. Description of the Related Art For example, on a moving image obtained by photographing a state where a soccer game is being played in a wide field, a specific player specified from among a plurality of players who are playing the soccer game is written in a literal time. The advent of devices that can be tracked is desired. 2. Description of the Related Art Conventionally, there are already some types of devices that display the result of tracking a moving object to be tracked (target) such as the above-mentioned player in a moving image. However, these devices are devices that process captured moving images offline with software, and the processing time is extremely long, requiring real-time capability of capturing moving images and tracking specific players on the spot. It is useless at all. Also, in a scene where a specific player is tracked in a soccer game as described above, the player may run, stop, change direction, or even fall, and may have a large size on the captured image. Because of the deformation, in the conventional device, even in the offline processing, the tracking of the target player is immediately impossible, and the operator must manually connect the section where the automatic tracking cannot be performed, and sufficient automation can be realized. There is another problem.

In recent years, as a moving object tracking method that attaches importance to real-time properties, a method using template matching based on a correlation operation has been developed. In this method, the amount of calculation is reduced by limiting the search area for tracking to a local area on the image, and processing can be parallelized on the calculation principle, so that real-time processing is possible.
However, since this method only creates a template in advance and performs a correlation operation with the template created in advance, a target that greatly deforms on an image like the player in the sucker game described above is used. Has the disadvantage that the target is immediately lost, and although the real-time property is improved, the tracking performance is not satisfactory.

[0004] As for what to adopt as information for tracking, it is suggested that a motion vector be obtained using color information (Japanese Patent Laid-Open No. Hei 8-130675).
(See Japanese Patent Application Laid-Open No. 60-209), it is better to perform a correlation operation using a differential image obtained by spatially differentiating the image instead of an image used for ordinary observation.
Various proposals have been made in the past, such as a proposal that the next movement should be predicted from information on past movements of a tracked target (see Japanese Patent Application Laid-Open No. 60-195682). However, none of the proposals satisfies the requirement to achieve both real-time performance and practically sufficient tracking performance.

[0005] In view of the above circumstances, it is an object of the present invention to provide a mobile object tracking device that has improved real-time performance and tracking performance.

[0006]

A moving object tracking apparatus according to the present invention, which achieves the above object, inputs an original image consisting of a plurality of consecutive frames in which a scene including one or more moving objects is photographed. In a moving object tracking device that tracks a moving object to be tracked selected from the moving objects displayed in the original image, the moving object tracking device is compared with the moving object to be tracked displayed in a color separation image constituting the original image. Color template and
A first correlation value distribution operation unit for obtaining a distribution of a first correlation value with the color separation image, a differentiation processing unit for obtaining a differential image by spatially differentiating the original image, and a tracking target displayed on the differential image A second correlation value distribution calculating unit for calculating a distribution of a second correlation value between the differential template and the differential image, and a tracking object based on a past movement of the tracking mobile object Predicted value distribution calculating unit for obtaining a distribution of motion predicted values representing the probability that a moving object exists, and integrating the first correlation value, the second correlation value, and the motion prediction value into an evaluation value Calculating the distribution of the evaluation value representing the position of the moving object to be tracked. A position detection unit that detects the position of the body And it features.

Here, the "color-separated image forming the original image" referred to in the present invention is a black-and-white image having only lightness information, since there is only an image for one color. When the original image is composed of, for example, R (red), G (green), and B (blue) images, the images of the respective colors are color separation images. In the present invention, when there are a plurality of color separation images, only one color separation image may be employed, or each of the plurality of color separation images may be employed. When using color separation images for multiple colors, the color template
A plurality of colors are prepared for comparison with each color separation image.

In addition, the first correlation value distribution calculator, the second
The first correlation value and the second correlation value calculated by the correlation value distribution calculation unit need not be the correlation values calculated by the correlation calculation in a mathematical sense, but the degree of approximation between the template and the partial area. Any value may be used as an index. Here, a value indicating the degree of approximation is referred to as a “correlation value”, and a calculation for obtaining the correlation value is referred to as a “correlation calculation”.

In the first correlation value distribution calculator, for example, a color template and a color separation image
The operation of associating the first correlation value obtained by obtaining the first correlation value representing the degree of approximation of the size of the color template with a partial region having the same size as one pixel in the partial region,
By sequentially changing the cutout position of a partial region from the color separation image while sequentially changing the position, the distribution of the first correlation value within a predetermined search region including the entire region or the partial region of the original image is obtained. In the second correlation value distribution calculating section, for example, a second correlation value representing the degree of approximation between the differential template and a partial region cut out from the differential image and having the same dimensions as the dimensions of the differential template is obtained. By sequentially executing the operation of associating the second correlation value obtained as described above with one pixel in the partial area while sequentially changing the cutout position of the partial area from the differential image, A second correlation value distribution is determined.

Further, the search area according to the present invention may be the whole area of the original image or a partial area. However, the size of the original image and the size and movement of In consideration of the speed and the like, it is preferable to limit the region to a local region as much as possible from the viewpoint of reducing the amount of calculation. As described above, conventionally, there have been proposals for adopting color information for tracking a target, proposals for differentiating images, and proposals for predicting the movement of a target based on past movements of the target. . However, it is important how these are combined to ensure real-time performance and improve tracking performance.

According to the present invention, a distribution of a first correlation value (hereinafter referred to as a "color space distortion map") is calculated by a correlation operation between a color separation image and a color template.
), And a second correlation value distribution (hereinafter, this second correlation value distribution may be referred to as a “differential space distortion map”) by calculating a correlation between the differential image and the differential template. ), And in the present invention, two distortion maps are obtained based on the correlation operation as described above, and furthermore, the motion of the target is predicted based on the past motion of the target. A distribution (hereinafter, this distribution of motion prediction values is sometimes referred to as a “motion prediction value map”) is obtained. Is one of the features of the present invention.
Three distributions (maps)
The integration of the three distributions, that is, the integration of the tracking results by the three methods becomes easy, and the automatic tracking with high tracking performance by integrating the tracking results by the three methods can be performed while ensuring the real-time property.

Here, in the moving object tracking apparatus of the present invention, the evaluation value distribution calculating section includes a reliability evaluating section for evaluating the reliability of the distribution of the first correlation value, and determines that the reliability is low. It is preferable that the distribution of the evaluation values is obtained after excluding the distribution of the obtained first correlation values from the processing for obtaining the distribution of the evaluation values. Alternatively, the evaluation value distribution calculation unit includes a reliability evaluation unit that evaluates both the reliability of the first correlation value distribution and the reliability of the second correlation value distribution, and is determined to have low reliability. Calculating the distribution of the evaluation values after excluding the distribution of the first correlation values and the distribution of the second correlation values determined to have low reliability from the processing for obtaining the distribution of the evaluation values. Is also good.

A distribution (distortion map) of a correlation value in which the position of the target is not well captured due to deformation of the target or the like may be obtained. Therefore, for example, the reliability of the distortion map is evaluated by a method such as threshold processing, and if the distortion map with low reliability is excluded from integration for obtaining the 'distribution of evaluation values' for comprehensive evaluation, A highly-reliable 'evaluation value distribution' is required, and the tracking performance is further improved.

The probability that a distortion map with low reliability is required is much higher in the color space distortion map than in the differential space distortion map. Therefore, the reliability evaluation section described above uses the color space distortion map and the differential space Although it is more preferable to evaluate the reliability of both of the distortion maps, when there is a time limitation or a limitation on the circuit scale, the reliability of only the color space distortion map may be evaluated.

Further, in the moving object tracking device of the present invention,
The color template includes a temporary template that is sequentially updated, and an image area in the color separation image that includes the position of the tracked moving object detected by the position detection unit is included in the temporary template. It is preferable to include a color template update unit that updates the temporary template by integrating the color templates.

As described above, the color space distortion map is inferior in reliability to the differential space distortion map. This means that it is difficult to catch the target on the color separation image. Therefore, one of the color templates
By preparing a temporary template and updating the temporary template according to the situation at that time, a color template suitable for the current situation is constructed, and the reliability of the color space distortion map can be improved. it can.

Here, as a configuration for updating the temporary template, the evaluation value distribution calculation unit integrates the first correlation value and the second correlation value for each pixel and integrates each pixel. A first integrated processing unit for obtaining an integrated correlation value distribution by obtaining a correlation value, an integrated correlation value obtained by the first integrated processing unit, and a motion prediction value calculation unit A second integrated processing unit that obtains an evaluation value distribution by integrating the motion prediction value for each pixel and obtains an evaluation value for each pixel, wherein the template updating unit includes the integrated correlation value An approximate object detection unit that detects the existence position of an object approximated to the moving object to be tracked based on the distribution of the plurality of positions while allowing detection of a plurality of positions; A first image area including the position of the tracking mobile unit and a temporary image First approximation degree representing the degree of approximation of the plate, and in the first image area, detected by the approximation object detecting unit, the including the presence position of the approximate object 2
A second degree of approximation representing the degree of approximation to the image area of the first image is obtained. If the first degree of approximation indicates that the degree of approximation is greater than the second degree of approximation, By integrating the area into a temporary template,
It is preferable to have an evaluation / integration unit that updates the temporary template.

The template must be capable of accurately detecting the target. For this purpose, it is desired that the template reacts strongly to the target and hardly reacts to a non-target. Therefore, by adopting the above configuration, the temporary template is updated using the image area only when an image area similar to the previous temporary template is obtained,
A temporary template that matches the current state of the target and that is more reliably distinguished from other moving objects is generated.

Further, the first correlation value distribution operation unit and the second correlation value distribution operation unit are shared by the first correlation value distribution operation unit and the second correlation value distribution operation unit of the present invention. A correlator for determining a correlation value obtained by integrating the absolute value of a difference between pixel values assigned to mutually corresponding pixels of two images or image regions over the two images or image regions, respectively, provided for each of them. In another preferred embodiment, the differential processing unit causes the correlator to execute a part of differential operations for differentiating an original image to obtain a differential image.

As one preferable embodiment of the correlation calculation algorithm in the first correlation value distribution calculation unit and the second correlation value distribution calculation unit in the mobile unit tracking apparatus of the present invention, 2
It is possible to employ a correlation calculation algorithm for obtaining a correlation value obtained by integrating the absolute value of the difference between pixel values assigned to mutually corresponding pixels in one image or image region over the two images or image regions. When such a correlation calculation algorithm is employed, the multiplication in the correlation calculation in the mathematical sense is replaced with the subtraction, and the correlation value can be obtained very quickly.

When such a correlator is provided, it is possible to cause the correlator to perform a part of the differential operation for obtaining a differential image from an original image, as described later. In the differential processing unit of the present invention, when performing the differential operation, a unique module that executes the differential operation may be configured, or the above-described correlator may be shared. Sharing the correlator reduces the circuit scale accordingly.

Also, in the moving object tracking device of the present invention,
The motion prediction value distribution calculation unit obtains a predicted motion vector that predicts the motion of the tracked moving object from the current time to the next time, and the predicted motion vector represents the tracked movement at the next time. It is preferable to associate a motion prediction value representing the maximum probability at the predicted body presence position and the probability of decreasing with distance from the predicted body presence position with each pixel in the search area.

In predicting the movement of a target from the past movement of the target, the moving object tracking apparatus of the present invention does not calculate only the motion vector as in the prior art, but distributes the motion prediction value (motion prediction value). There is one characteristic in obtaining the map), but again, the target is most likely to move to the target movement predicted position indicated by the predicted motion vector. Therefore, a predicted motion vector is calculated, and a predicted motion value map is created that indicates the maximum probability of the target moving to the target movement predicted position represented by the predicted motion vector and the probability of the target moving to that position, and the probability of decreasing as the distance from the target increases. By doing so, more reliable tracking becomes possible.

Here, when obtaining the predicted motion vector, the motion predicted value distribution calculation unit calculates the predicted motion vector obtained at the past time, which indicates the predicted position of the moving object to be tracked at the present time, and the position. Based on a motion vector representing the actual motion of the tracked moving body determined from the difference between the positions of the existing positions of the tracked moving body detected at both the past time and the current time by the detection unit, It is preferable to obtain a predicted motion vector that predicts the motion of the tracked moving object from the current time to the next time.

The predicted motion vector obtained by such an algorithm has the meaning of the culmination of the trajectory actually passed by the past target. Therefore, as described above, by calculating the next predicted motion vector based on the predicted motion vector that is the culmination of the past trajectory of the target and the motion vector representing the latest motion of the target, the past motion of the target is obtained. In consideration of the above, a more accurate predicted motion vector is obtained, and the tracking performance of the target is improved.

In obtaining a motion predicted value map after obtaining a predicted motion vector, the motion predicted value distribution calculating section calculates a distance of a distance around the predicted position of the tracked moving object represented by the predicted motion vector. The motion prediction value as a function may be associated with each pixel in the search area. However, the motion prediction value distribution calculation unit may be configured to center the predicted motion position of the tracked moving object represented by the predicted motion vector. The motion prediction value as a function of these two distances having different weights for the distance in the direction of the predicted motion vector and the distance in the direction intersecting the direction is assigned to each pixel in the search area. It is more preferable that they are associated.

Normally, a target moving at a certain speed is likely to be moving in the same direction at the next point in time. For example, in such a situation, more stable tracking can be performed by creating an ellipsoidal motion predicted value map having a major axis in the direction of the predicted motion vector. However, in the moving object tracking device of the present invention, it is not always necessary to have a long axis in the direction of the predicted motion vector. For example, when the moving object tracking device of the present invention is applied to a target that moves with many curves. Preferably has a long axis in a direction intersecting the direction of the motion prediction vector, or applies the moving object tracking device of the present invention to a target in which a motion that cannot be said to be either straight or curved is predicted. In such a case, an isotropic motion prediction value map having no major axis or minor axis in any direction may be employed.

Further, in obtaining a motion predicted value map, the motion predicted value distribution calculation unit associates a motion predicted value as a function with the size of the predicted motion vector as a variable with each pixel in the search area. This is also a preferred embodiment. For example, when targeting a certain player in a soccer game, if the target moves fast, it is highly likely that the movement continues even at the next point in time, and if the target moves slowly, it shifts to another movement at the next point in time The possibilities increase. Therefore, by creating a motion prediction value map that follows the motion of the target in consideration of the size of the predicted motion vector, the target can be tracked more stably.

In the present invention, it goes without saying that a motion prediction value map may be created in consideration of both the direction of the predicted motion vector and the size of the predicted motion vector. Further, in the moving object tracking apparatus of the present invention, the second correlation value distribution calculating section includes a predetermined position on the differential image which is the same as a predetermined position on the original image designated prior to tracking of the tracked moving object. Including, a predetermined image area in the differential image,
It may be set as a differential template.

When a target is specified on a certain initial frame on the original image prior to tracking, the image area including the target in the differential image is set as it is as a differential template. The target can be tracked using the derived derivative template. Alternatively, in the moving object tracking apparatus according to the present invention, the second correlation value distribution calculating unit may include a predetermined position on the differential image that is the same as a predetermined position on the original image specified prior to tracking of the tracked moving object. In the derivative image, including the position,
Within the image area of a size larger than the size of the derivative template, the same size as the size of the derivative template, which has a high degree of approximation with the common template that is commonly set for the moving body that can be designated as the tracked moving body, It is preferable that a partial area is detected and the partial area is set as a differential template.

When a target is specified on the original image, the target often moves, especially on a moving image, and there are situations where it is difficult to specify the target accurately. In such a case, a common template including the target and reacting to any moving object similar to the target is prepared, and the vicinity of the target is specified in order to specify a target. Then, the target is searched from a wide area including the specified point, and the searched target is cut out to set a differential template. With this configuration, a correct differential template is set even if there is an error in the designated position of the target.

Here, when the common template is prepared as described above and the target position is designated so as to allow an error, the second correlation value distribution calculator further includes a differential template. At one or more times after the time when the is set for the first time, integrating the image area including the position of the moving object to be tracked, which is detected by the position detection unit, in the differential image into the differential template By
It is further preferable that the apparatus further includes a derivative template updating unit that operates at a predetermined initial stage after the tracking of the tracked moving object is started, which updates the derivative template.

As described above, when a partial area of the differential image is cut out using the common template and the cut-out partial area is set as a differential template, the common template is not only a target but also a considerably wide area. Since the region is created so as to respond to a moving body or the like, there is a possibility that the partial region cut out as described above may include some errors. Therefore, when the differential template updating unit is provided and the differential template is updated in the initial stage after the tracking is started, a more accurate differential template is set, and the target can be tracked more stably. .

On the other hand, with respect to the color template, the color template includes a fixed template, and the first correlation value distribution calculating unit determines whether or not the differential template is set or finally updated at the time of It is preferable that an image area in the color-separated image that includes the position of the tracked moving object detected by the position detection unit is set as a fixed template.

When a partial area to be cut out as a differential template on the differential image is determined, the same partial area on the color separation image as the partial area on the differential image can be used as a color template. The case where the color template includes the temporary template that is sequentially updated has been described above. However, in the case where the color template includes a fixed template that is set once and is used fixedly,
An accurate color template can be set by cutting out a partial area on the color separation image, which is the same as a partial area on the differential image, which is cut out as a differential template and using it as a fixed template.

In setting a color template,
Rather than detecting a partial region set as a color template on a color-separated image by setting a common template to act on the color-separated image, it is generally more preferable to detect on a differential image as described above. High detection accuracy can be obtained. Further, the moving object tracking apparatus of the present invention includes an input means for sequentially and sequentially inputting an original image composed of a plurality of frames obtained by photographing a scene including the moving object, and a position of the moving object between the plurality of frames. Moving object tracking means for tracking different moving objects and sequentially outputting the position of the moving object in the frame at the continuous input time, and moving object tracking of the original image input by the input means It may be configured to include a mark generating means for displaying a mark at the position of the moving body tracked by the means.

By generating such a mark, it is possible to obtain a moving image in which the presence or movement of a certain moving object is noticed.

[0038]

Embodiments of the present invention will be described below. FIG. 1 is a schematic diagram of an automatic tracking system including one embodiment of a moving object tracking device according to the present invention. In the automatic tracking system shown in FIG. 1, a moving object tracking device as one embodiment of the present invention is realized inside a computer system 1.

The video camera 2 is configured to be driven by a servo system 3 to change its direction up, down, left, and right. An image obtained by shooting with the video camera 2 is taken into the computer system 1, and inside the computer system 1, a certain moving object (target) shown in the image is detected, and the target is set to the video camera 2. The vertical and horizontal directions of the video camera 2 are changed by controlling the servo system 3 so that the video system 2 is always included in the image obtained by the photographing.

The image obtained by the video camera 2 is taken into the computer system 1 and transmitted to another device, for example, a broadcasting device (not shown), together with the target position information on the image output from the computer system 1. The broadcast equipment performs image processing such as displaying the target in a special color so as to be conspicuous, and finally broadcasts it by radio waves or records it for later broadcast.

Here, the computer system 1 shown in FIG.
And a CRT display unit 104. The main unit 101 includes a CPU for executing programs, a hard disk for storing programs and data, and a computer system for fetching images from the video camera 2 therein. I / O board for outputting a signal to be transmitted from 1 to the servo system 3 and other devices, a floppy disk drive device which is loaded with a floppy disk and accesses the loaded floppy disk, and a device unique to the computer system 1 A board or the like on which a correlator described later is mounted is built in.

FIG. 2 is a functional block diagram of a moving object tracking apparatus as one embodiment of the present invention constructed in the computer system shown in FIG. The original image input to this moving object tracking device is a moving image composed of a series of a large number of frames that are successively consecutive. The following processing is performed for each frame, and the position of a specified target is determined for each frame. Each time, which results in automatic tracking of that target. However, in the following description, in order to avoid the complexity of the description, when there is no need to express explicitly, an image for each frame and a moving image as a whole are simply referred to as an original image without discrimination. The same applies to a decomposed image constituting the original image and a differential image obtained by differentiating the original image.

Here, the operation of each part will be described later.
The configuration of the moving object tracking device shown in FIG. 2 will be described. The moving object tracking apparatus shown in FIG. 2 includes a color space distortion map creating unit 10, a differentiation processing unit 20, a differential space distortion map creating unit 30, and an evaluation value map creating unit 4.
0, minimum position detection unit 50, motion prediction value, map creation unit 6
0 and a color template updating unit 70.

Here, the color space distortion map creating section 10 corresponds to an example of a first correlation value distribution calculating section according to the present invention, and the differential processing section 20 corresponds to an example of a differential processing section according to the present invention. The differential space distortion map creation unit 30 corresponds to the second correlation value distribution calculation unit according to the present invention. The evaluation value map creation unit 40 corresponds to an example of the evaluation value distribution calculation unit according to the present invention, and the minimum value position detection unit 50
Corresponds to an example of a position detecting unit according to the present invention, and the motion predicted value map creating unit 60 corresponds to an example of a motion predicted value distribution calculating unit according to the present invention. Further, the color template updating unit 70 corresponds to an example of the color template updating unit according to the present invention. Hereinafter, the internal configuration of each unit will be described.

Each of the various templates and maps described below has a form of a kind of image having a two-dimensional spread. The moving object tracking device shown in FIG. 2 includes a correlator 80, and the correlator 80 includes a color space distortion map creating unit 10,
It is shared by the differential processing unit 20 and the differential space distortion map creating unit 30.

In this correlator 80, when data corresponding to each pixel of two input images (including a partial area of an image and a template) is A _xy and B _xy , the data for each pixel Is obtained by integrating the absolute value of the difference between all the pixels of the two images. That is,

[0047]

(Equation 1)

Is performed. Here, this operation is referred to as SAD (Sum of Absolute Diff
The value d obtained by this SAD processing is called a distortion value. This SAD processing is an example of a correlation operation, and the distortion value corresponds to an example of the correlation value according to the present invention. This distortion value d means that the smaller the value, the greater the degree of approximation of the two images { _Axy } and { _Bxy }.

Color Space Distortion Map Creation Unit 10
Is set with two types of color templates named a fixed template and a temporary plate. SAD processing is performed between the color separation image forming the image and each of the fixed template and the temporary plate to create a color space distortion map. This color space distortion map is an example of the distribution of the “first correlation value” according to the present invention. Details will be described later. In the following description of each part, only the outline is described here, and the detailed description will be given later.

In the differentiation processing section 20, the input original image is spatially differentiated to create a differentiated image. The differential processing unit 20 is provided with a differential operation unit 21, and the differential operation unit 21 generates a differential image while causing the correlator 80 to perform a part of the differential operation. In the differential space distortion map creating section 30, a differential template for creating a differential space distortion map and a common template for setting a differential template are set.
In the calculation unit 31 constituting the above, the SAD process between the input differential image and the differential template is performed using the correlator 80, and is an example of the “distribution of the second correlation value” according to the present invention. A differential space distortion map is created. In addition, when the differential template is set, the arithmetic unit 31 performs SAD processing between the input differential image and the common template using the correlator 80, and should set the differential template in the differential image. A partial region is detected, the partial region is cut out, and the cut out partial region is set as a differential template.
The differential space distortion map creating unit 30
Is provided with a differential template updating unit 32,
The set differential template, the differential image, and the output from the minimum value position detecting unit 50 are input to the differential template updating unit 32.
In 2, based on the output from the minimum value position detection unit 50,
In the differential image, a partial area in which the initial target after the start of tracking is projected is cut out, and the cut out partial area is merged (integrated) with the differential template to create a new differential template, The previously set differential template is updated to the newly created differential template. This derivative template updating unit 3
Reference numeral 2 denotes an example of a differential template updating unit according to the present invention. The processing of updating the differential template by the differential template updating unit 32 is performed only at an initial stage of starting tracking of the target, and thereafter, the differential template is updated. Instead, the derivative template remains fixed and is used to create a differential spatial distortion map.

The evaluation value map creator 40 is composed of a reliability evaluator 41, a weighted adder 42, and another weighted adder 43, each of which is
This corresponds to an example of each of the reliability evaluation unit, the first integrated processing unit, and the second integrated processing unit according to the present invention. In the reliability evaluation section 41, the color space distortion map creation section 10
And the reliability of the differential space distortion map created by the differential space distortion map creation unit 30 are evaluated, and the distortion maps are determined to have reached the predetermined reliability, respectively. (A generic term for color space distortion map and differential space distortion map is
Here, this is referred to as a distortion map).

Normally, both the color space distortion map and the differential space distortion map that have reached the predetermined reliability via the reliability evaluation unit 41 are input to the weighting addition unit 42, and the two distortion maps are input. Are weighted and added for each pixel corresponding to each other to create an integrated distortion map. This integrated distortion map is an example of the “distribution of integrated correlation values” in the present invention. Note that the weighting addition section 42
May be fixedly determined in advance, or may be configured to be switchable according to the type and nature of the input original image. The integrated distortion map created by the weighting and adding unit 42 is added to another weighting and adding unit 43.
Is input to the color template updating unit 70. The color template updating unit 70 will be described later, and here, another weighting and adding unit 43 will be described.
In addition to the integrated distortion map, the motion estimation value map created by the motion estimation value map creation unit 60 is also input to the weighting addition unit 43.
Then, the integrated distortion map and the motion prediction value map are weighted and added for each corresponding pixel to create an evaluation value map. This evaluation value map is an example of the “evaluation value distribution” according to the present invention. Each weight for the integrated distortion map and the motion prediction value map in the weighting and adding unit 43 may be fixedly determined in advance, similarly to the weight in the weighting and adding unit 42, and may be different depending on the type or property of the input original image. It may be configured to be switchable in response to the request.

Here, each pixel value forming the distortion map is the above-mentioned distortion value d,
The smaller the value is, the larger the degree of approximation to the template is.
The values of the respective pixel values constituting the motion prediction value map created by the above, that is, the motion prediction values, are determined such that the smaller the value is, the higher the possibility that the target exists at the position of the pixel is. Therefore, since the evaluation value map created by the evaluation value map creation unit 40 is obtained by weighting and adding the two distortion maps and one motion prediction value map for each pixel, the minimum value in the evaluation value map is used. The value position indicates the position where the target exists.

The weighting and adding unit 43 creates
That is, the evaluation value map created by the evaluation value map creation unit 40 is input to the minimum value position detection unit 50. In the minimum value position detection unit 50, a pixel having the minimum evaluation value is detected from the input evaluation value map, and the position information of the pixel is output as target position information representing the target position.

The target position information is, for example, as shown in FIG.
Is used to control the camera orientation and to specially process the target. The motion predicted value map creator 60 includes a next frame predicted motion vector calculator 61 and a prediction function generator 62. The target position information obtained by the minimum value position detection unit 50 is input to the next frame prediction motion vector calculation unit 61, and the next frame prediction motion vector calculation unit 61 sets the target position information in the next frame. A predicted motion vector that predicts in which direction and how much the object has moved from the target position represented by is calculated. The obtained predicted motion vector is input to the prediction function generation unit 62 and the prediction function generation unit 6
In step 2, a prediction function that probabilistically predicts to which position the target will move in the next frame is generated around the position indicated by the predicted motion vector, and the motion prediction value map for each pixel is generated according to the prediction function. A set of motion prediction values is determined. The smaller the value of the motion prediction value constituting the motion prediction value map, the higher the possibility that the target has moved there in the next frame. The motion prediction value having the smallest value is assigned to the pixel specified by the calculated motion vector. This motion predicted value map corresponds to an example of “distribution of motion predicted values” in the present invention.

As described above, the motion predicted value map created by the motion predicted value map creating section 60 is input to the weighted adding section 43 and combined with the integrated distortion map generated by the other weighted adding section 42. Are weighted and added. The color template update unit 70 includes an approximate object detection unit 71 and an evaluation / integration unit 72. These approximate object detection unit 71 and evaluation / integration unit 72
This corresponds to an example of each of the approximate object detection unit and the evaluation / integration unit according to the present invention. Further, in the present embodiment, the approximate object detection unit 71 includes the minimum value position detection unit 71a and the
b.

The integrated distortion map generated by the weighting and adding section 42 constituting the evaluation value map creating section 40 is input to the weighting and adding section 43 and also to the color template updating section 70 as described above. You. The integrated distortion map input to the color template updating unit 70 is input to a local minimum position detecting unit 71a that further configures the approximate object detecting unit 71 included in the color template updating unit 70.

The minimum value position detecting section 71a detects the position of the minimum value in the integrated distortion map. It is presumed that the position of the minimum value is where an approximate object close to the target exists. The position of the minimum value is not limited to one, and a plurality of positions can be detected. Although the true target itself can be detected by the local minimum position detecting section 71a, the position of the true target is detected by the minimum position detecting section 50, and its target position information is input to the evaluation / integration section 72, and the true target is detected. And the approximate object are distinguished from each other, so that there is no problem even if the true target is detected by the local minimum position detecting unit 71a.

The interference object monitoring unit 71b approximates the local minimum position detected by the local minimum position detection unit 71a to the target based on the state of the change of the distortion value near the local minimum position of the color space distortion map. It is determined whether or not an object (here, this approximate object is also referred to as an interference object in the sense of an object that interferes with the target (disturbs the target position detection)) is present.

The evaluation / integration section 72 includes the interference object monitoring section 7
1b, the position information of the interfering object detected by step 1b and the position information of the target at which the minimum value position is detected are input, and the temporary template, the target image, and the image of the interfering object that are currently set are comprehensively taken into account. Is updated or not, and when a determination to update the temporary template is made,
The current target image cut out from the color separation image is merged (integrated) with the currently set temporary template, whereby the temporary template is updated to the merged temporary template.

The general description of each unit constituting the moving object tracking device shown in FIG. 2 has been completed, and the specific operation of each unit will be described below. FIG. 3 is an explanatory diagram of a process of creating a color space distortion map in the color space distortion map creating unit 10. The method of setting the fixed template and the temporary template that constitute the color template will be described later, and here, it is assumed that both the fixed template and the temporary template have already been set.

In this embodiment, the input original image is R
(Red), G (green), B (blue)
That is, it is composed of three types of color separation images, and both the fixed template and the temporary template
3 corresponding to each of the three types of color separation images of R, G, and B
Set by type. Therefore, here, at an intermediate stage, three color space distortion maps are created by SAD processing between three types of color separation images and a fixed template set for each color, and further, three types of color separation images and each color Three color space distortion maps are created by SAD processing with a separately set temporary template. Thereafter, the six color space distortion maps created in this way are averaged for each pixel and one color space distortion map is created. It is integrated into the spatial distortion map. The color space distortion map creation unit 10 outputs the color space distortion map integrated as described above and inputs the evaluation value map creation unit 40. However, in the description of FIG. 3, typically, the SAD of one color separation image and one template
The processing will be described.

Next, the motion prediction value map forming unit 60 will be described.
The prediction obtained by the frame prediction motion vector calculation unit 61
The target moves to the next frame based on the motion vector information.
Around the position that would have moved before
A search area is set in the color separation image, and the search area
Each distortion value is obtained for each pixel.
In order to perform this operation, the search area
The cutout area D shown in FIG.
Each partial area P from the cutout area D₁ , P _Two ,…,
P_nAre sequentially cut out. Here, first, the cutout area
Set the top and leftmost corner of D as one vertex
Partial area P of the same shape as the plate₁ Is cut out and its
Cut out partial area P₁ And the template
The SAD processing is performed by the function 80, that is, the equation (1) is used.
Is performed, and one distortion value d is obtained.
The calculated distortion value is extracted
Part area P₁ Is associated with the center pixel. Below
Indicates the value obtained by SAD processing for convenience of explanation.
It is simply called SAD processing, including the process of associating elements.
I do.

Next, a partial area P ₂ shifted by one pixel in the x direction from the partial area P _{1 at the} corner is cut out, and the SAD is placed between the newly cut out partial area P ₂ and the template.
The processing is executed. This operation is repeated, and when reaching the rightmost side of the cutout area D, it returns to the leftmost. This time, a partial area shifted by one pixel in the y direction is cut out, and the same SAD as described above is performed.
The processing is executed, and furthermore, a partial area sequentially shifted by one pixel in the x direction is cut out, and the SAD processing is executed.
This is further repeated, and finally, the lowermost and rightmost area _Pn of the cutout area D is cut out, and the SAD
The processing is executed.

By a series of these SAD processes, a two-dimensional map of distortion values corresponding to each pixel of the search area P, that is, a color space distortion map is created. As described above, the color space distortion map creation unit 10 once creates six color space distortion maps as described above,
The six color space distortion maps are averaged for each pixel and integrated into one color space distortion map. However, as will be described later, there is also a situation where a fixed template is not used and only a temporary template is used. In this case, three color space distortion maps created using only the temporary template are integrated into one color space distortion map. Is done.

FIG. 4 is an explanatory diagram showing the process of the differential processing in the differential processing section 20. The original image is input to the differential processing unit 20, and in the present embodiment, first, color information is discarded from three (R, G, B) color separation images constituting the original image, and A black-and-white original image having only brightness (luminance, density) information is created, and a spatial differentiation is performed on the black-and-white original image to generate a differential image corresponding to the black-and-white original image. In the following, it is not specified that the image is a monochrome image.

FIG. 4A is a diagram showing a case where y is differentiated in the y direction.
FIG. 4B is a diagram illustrating an example of an x-direction differential filter that differentiates an image in the x-direction. These differential filters are overlapped with a partial region of the original image, and the pixel values of the pixels of the overlapped original image and the pixel values of the pixels of the filter are multiplied with each other by overlapping the pixels.
The multiplication results are added over the entire area of the superimposed region, and an operation of associating the addition result with the pixel at the center of the superimposed region is performed while sequentially changing the superimposed region. The original image is differentiated in the y direction when using the y direction differential filter, and differentiated in the x direction when using the x direction differential filter.

The differential processing unit 20 shown in FIG. 2 performs the same operation while using the correlator 80. That is, here, the differential filter shown in FIGS. 4A and 4B is not used, and two templates shown in FIGS. 4C and 4D are used and described with reference to FIG. An SAD process (including a process of associating a value with a central pixel) is performed. After that, the differential operation unit 21 obtains a y-direction differential image obtained by differentiating the original image in the y direction and an x-direction differential image obtained by differentiating the original image in the x direction by the operation described below. That is, the pixel value of each pixel (x, y) of the image obtained by the SAD processing when the template 1 shown in FIG. 4C is used is represented by F _s (x, y).
The pixel value of each pixel (x, y) of the image obtained by the SAD processing using the template 2 shown in (D) is represented by F _ss
(X, y), the pixel value of each pixel (x, y) of the y-direction differential image is ΔP _y (x, y), and each pixel (x, y) of the x-direction differential image
Assuming that the pixel value of y) is ΔP _x (x, y), the differential control unit 21 calculates ΔP _y (x, y) = F _s (x, y + 1) −F _s (x, y−1). (2) ΔPx (x, y) = F _ss (x + 1, y) −F _ss (x−1, y) (3) is calculated. These operations are equivalent to the differential operations using the differential filters shown in FIGS.

In the differential operation section 21, these two differential images {ΔP _y (x, y)} and ｛ΔP _x (x,
y) The larger pixel value of the pixel values of 採用 is adopted,
One differential image is created. In the differential space distortion map creating unit 30, except for the difference that the cutout region D shown in FIG. 3 is a cutout region set in the differential image and the template shown in FIG. SAD processing similar to that in step 10 is executed to create a differential space distortion map. However, since one type of differential image is created in the differential processing unit 20, only one type of differential space distortion map is created. In the differential space distortion map creation unit 30, the SAD
A common template may be used as a template for executing a process.

Note that the color space distortion
Similar to the description of the map creation unit 10, the differentiation template and
It is assumed that the common template has already been set. Figure
5 is a reliability evaluation unit constituting the evaluation value map creation unit 40
FIG. 41 is an explanatory diagram of the operation of the controller 41. The reliability evaluation unit 41
As described above, the color space distortion map creating unit 10
Color space distortion map created in
This is D₁And differential space distortion
Differential space distortion created by map creation unit 30
Map (this is D_TwoIs entered)
However, in the reliability evaluation section 41, these input 2
Distortion maps D₁, D_TwoIs referenced
Step 5_1)), those two distortion maps
Step D ₁, D_TwoEach minimum value D_1min, D_2minSought
(Step 5_2)). Incidentally, in FIG.
Distortion map D₁, D_TwoOr their minimum
D_{1m in}, D_2minIs required for each frame.
That is, to make it clear that it is a function of time t, D₁
(T), D_Two(T), D_1min(T), D_2min(T) and table
It is written.

Next, in step 5_3, the minimum values D _1min and D _2min are compared with preset threshold values TH1 and TH2. Here, D _1min >
TH1 means that the reliability of the color space distortion map D ₁ is low.
That _2min> is a TH2, it means that there is low reliability of the differential space distortion map D _2.

As a result of the comparison in step 5_3, D
_1min> TH1 and,, D _2min> TH2 (low distortion map D _1, D ₂ both reliability both) and when it is determined, the process proceeds to step 5_4, which is tracking the subsequent target are abandoned. Step 5 Results of the comparison in 3, D _1min ≦ TH1, and D _2min> TH2 (of the two distortion map, only the differential space distortion map D ₂ unreliable) and when it is determined in step 5 Proceed to 5, the differential space distortion map D ₂ is ignored, only the color space distortion map D _1, is input to the weighted addition unit 42. In this case, the differential space distortion map D ₂ In the weighted addition unit 42 is not input, therefore the weighted sum is not performed, it is input color space distortion map D _1, as integrated distortion map from the weighted addition unit 42 Is output.

Step 5 As a result of the comparison in FIG.
_1minThe following> TH1 and D_2min≤ TH2 (two
Of the distortion map, the color space distortion map
Top D ₁ Only low reliability).
Top 5 Proceed to Step 6 to get the color space distortion map D₁ 
Is ignored, and the differential space distortion map D_Two of
, And is input to the weighting and adding unit 42. Weight adder
At 42, the weighted addition is not performed, and the input differential
Distortion map D₁ Is integrated
It becomes a torsion map and its weighted addition unit 42
Is output.

Step 5 If it is determined that D _1min ≤ TH1 and D _2min ≤ TH2 as a result of the comparison in step 3, Proceeding to 7, the two distortion maps D ₁ and D ₂ are input to the weighting and adding unit 42, and the two distortion maps D ₁ and D ₂ are weighted and added to each other for each pixel. 6, two distortion map D _1, each of the D ₂ minimum D _1min,
FIG. 7 is a diagram showing a temporal change of D _{2min and} a temporal change of whether or not two distortion maps D ₁ and D ₂ are used (highly reliable) in subsequent processing.

FIG. 6A is a diagram showing a temporal change of the minimum value D _1min of the color space distortion map D ₁ , and the difference between the time domain B and the time domain C exceeds the threshold value TH 1. I have. FIG. 6B is a diagram showing a temporal change of the minimum value D _2min of the differential space distortion map D ₂ , between the time domain A and the time domain B, and the time domain C
Exceeds the threshold value TH2.

The situation shown in FIGS. 6A and 6B
In the case of, hatching is applied as shown in FIG.
In the time domain
Step D ₁ , D_Two Is ignored (toward the weighting addition unit 42).
Is not output). Here,
Color space distortion map and differential space distortion
Distortion of both sides by treating both
An example of evaluating the reliability of the
However, comparing the two distortion maps,
Color space distortion maps are less reliable,
Therefore, the reliability evaluation unit 41 shown in FIG.
Differential space distortion
The weighting and adding unit 42 unconditionally
May be output.

FIG. 7 is an explanatory diagram of the operation of the two weighting and adding units 42 and 43 constituting the evaluation value map creating unit 40. Here, it is assumed that both the color space distortion map D ₁ and the differential space distortion map D ₂ are output from the reliability evaluation unit 41 and input to the weighting addition unit 42. In FIG. 7, the color space distortion map D ₁ is represented by the color space distortion map D.
Set of pixel values D _1Xy of each pixel _{1 (x, y) {D} 1xy}
, And the differential spatial distortion map D ₂ is similarly represented by {D _2XY }, and the motion predicted value map created by the motion predicted value map creating section 60 is similarly represented by {E _XY }.

Here, the two weighting addition units 42 and 43
In the weighting addition unit 42 on the front stage, the two distortion maps {D _1xy } and {D _2XY } are weighted and added for each corresponding pixel, and an integrated distortion map {F _XY } is created. That is, in the weighting addition unit 42, for each pixel, F _XY = a × D _1xy + (1−a) × D _2XY (4) where a is a positive value of 1 or less, and a, 1 -A represents each weight. Is calculated.

The other one of the two weighting addition units 42 and 43 has a weighting addition unit 4
2, the integrated distortion map {F _xy } obtained by the operation shown in the equation (4) and the motion prediction map {E _xy } created by the motion prediction value map creation unit 60 are input, and the weighting addition unit At 43, the two maps {F _xy }, {E _xy } are weighted and added for each corresponding pixel, and an evaluation value map {G _xy } is created. That is, in the weighting addition unit 43, for each pixel, G _xy = b × F _1xy + (1−b) × E _xy (5) where b is a positive value of 1 or less, and b, 1 -B represents each weight. Is calculated. The evaluation value map {G _xy } obtained in this way is input to the minimum value position detection unit 50.

In the minimum value position detecting section 50, the position of the pixel having the minimum value G _min in the pixel value G _xy of each pixel constituting the input evaluation value map {G _xy } is obtained. Here, only the operation of detecting the minimum value G _min in the set {G _xy } of the pixel values G _xy and finding the pixel having the minimum value G _min is omitted, and further description with reference to the drawings is omitted. I do. FIG.
8 is an explanatory diagram of the operation of the motion prediction value map creating unit 60.

The next-frame predicted motion vector calculation section 61 constituting the predicted motion value map creating section 60 performs the predicted motion prediction based on the predicted motion of the target between the current frame t and the next frame t + 1 as described below. Vector m _t
Is required. The predicted motion vector obtained in the same manner as relates the previous frame t-1 and m _t-1, between the previous frame t-1 in the current frame t, when the motion vector representing the actual movement of the target was set to v _t, between the current frame t and the next frame t + 1, the predicted motion vector m _t predicted the motion of the target is obtained by calculation of the _{m t = (v t +4 ·} m t-1) / 5 ...... (6) . Here, a motion vector v _t representing the actual motion of the target between the previous frame and the current frame is similarly calculated for the current frame from the target position obtained by the minimum value position detection unit 50 for the previous frame. It is a vector headed for the determined target position. Here, the weights of v _t and m _t−1 are respectively 1 /
Although it is set to 5, 4/5, this is an example, and another weight may be adopted.

As can be seen from equation (6), the next motion prediction vector is obtained based on the past motion vector and the past prediction motion vector, and the past prediction motion vector further includes the past information. Therefore, by using the equation (6), a predicted motion vector as a compilation of a series of past information is obtained. The predicted motion vector m _t obtained according to the equation (6) in the next frame predicted motion vector calculation unit 61 is input to the prediction function generation unit 62. In the present embodiment, the prediction function generator 62 calculates the motion prediction function P _{P (x, y)} defined by the following equation (7 _).
, A motion prediction value corresponding to each pixel in the search area is obtained.

[0083] _{P P (x, y) =} (x 2 + y 2) · (1 + α · | m t |) ...... (7) However, α is a weight having a positive value. This equation (7) is
Basically it has a parabolic shape, the magnitude of the prediction motion vector m _t | m _t | higher the increase, a parabolic shape that rises steeply. However, (7) the vertex of the parabola shape of the formula was predicted by the prediction motion vector m _t,
The target position in the next frame (predicted presence position in the present invention) is made to coincide with the target position.

In this way, the motion prediction values P _{P (x, y)} are obtained corresponding to each pixel of the search area according to the equation (7), and the motion prediction values as a set of the motion prediction values are obtained. A map is created. Incidentally, there where a function of the distance around the predicted target position, and the magnitude of the prediction motion vector m _t | m _x | example has been described for obtaining the motion prediction value that varies with further prediction the direction of the motion vector m _t, in a direction perpendicular thereto, may be changed parabolic rising steepness is a function indicating a motion prediction value.

FIG. 9 is a diagram for explaining the operation of the approximate object detecting section 71 constituting the color template updating section 70. Here, the three (R, G, B) color separation images described above are simply referred to as color separation images without distinction, and the temporary templates corresponding to the respective color separation images are simply referred to as temporary templates without distinction. I do. FIG. 9A shows a target position (a position indicated by oblique lines in FIG. 9A) detected by the minimum value position detection unit 50 in the color separation image and the same dimensions as the template centered on the target position. It is a figure which shows the approximate oblique line A of.

FIG. 9B shows a color template updating unit 70.
The minimum value position detected by the minimum value position detection unit 71a (the position indicated by oblique lines in FIG. 9B) and the image areas A and
It is a figure showing B, C, and D. In the example shown here, the minimum value position detection unit 71a detects four minimum value positions including the target position detected by the minimum value position detection unit 50.

FIG. 9B shows the integrated distortion map created by the weighting and adding section 42 in the description of the interference object monitoring section 71b described next, and the evaluation / integrating section 72 described next. Means a color-separated image. However, the illustration of the image areas A, B, C and D has no meaning on the integrated distortion map. The interfering object monitoring unit 71b applies the filter shown in FIG. 9C to each minimum value position on the integrated distortion map for the four minimum value positions detected by the minimum value position detection unit 71a. The way this filter works is shown in FIG.
Since the operation is the same as that of the differential filter described with reference to FIG.

The filter shown in FIG. 9C corresponds to FIG.
As shown in FIG. 9D, the pixel value at the minimum value position (FIG. 9)
(D) schematically shows the distance (depth) between a pixel value of a pixel around the minimum value position (indicated by X in FIG. 9D). A filter to be extracted. This depth is compared with a predetermined threshold, and when the depth is deeper than the depth indicated by the threshold, the interference at the minimum value position interferes with the target. It is determined that an object exists. Here, it is determined that an interference object exists at each minimum value position in the image areas B and C shown in FIG. 9B, and that no interference object exists at the minimum value position in the image area D. And

FIG. 10 is a diagram for explaining the operation of the evaluation / integration section 72 constituting the color template update section 70. Here, as shown in FIGS. 9A and 9B, an image area A having the same size as the template on the color separation image centered on the target position detected by the minimum value position detection unit 50 is approximated. Approximate object (interfering object) detected by the object detecting unit 71
Image areas B and C having the same dimensions as the template on the color separation image, centered on the local minimum position where it is determined that
The calculation shown in the above-described equation (1) is executed between each of them, and the respective distortion values d ₁ and d ₂ are obtained.
These distortion values d ₁ and d ₂ correspond to an example of the second approximation according to the present invention. Also, like this,
Even between the temporary template and the image area A and the operation is executed as shown in the aforementioned equation (1) is the distortion value d ₃ is determined. This distortion value d ₃ corresponds to an example of the first approximation according to the present invention. Next, the distortion values determined in this way are compared with each other, and the distortion value d ₃ determined between the temporary template and the image area A is compared with the image area A,
Distortion values d ₁ and d obtained between the respective image regions B and C determined to have an interference object
₂ (smaller value indicates higher degree of approximation as described above), the pixel value of each pixel of the temporary template and the pixel value of each pixel of the image area A The value is averaged for each corresponding pixel to obtain a new temporary template, and the previous temporary template is updated with the newly obtained temporary template. The temporary templates are sequentially updated in this manner. Here, the temporary template is updated only when d ₃ <d ₁ , d _2. Therefore, the updated temporary template reacts strongly to the target and the degree of reaction to the interfering object is reduced. The object can be stably tracked separately from the object.

Next, a method of setting a differential template and a fixed template, and a method of initial setting of a temporary template will be described. FIG. 11 is an explanatory diagram illustrating a first example of a method of setting each template. An original image including a moving object to be designated as a target is displayed on the CRT display unit 104 shown in FIG. 1, and the mouse 103 is operated to indicate a desired moving object in the original image. Then,
The position information on the pointed original image is input to the moving object tracking device shown in FIG. 2 together with the original image, and the arithmetic unit 11 constituting the color space distortion map creating unit 10 executes the mouse An image area centered on the position pointed by 103 and having the same size as the template is cut out, and the cut out image area is set as a fixed template and also initialized as a temporary template. At the same time, the calculation unit 31 constituting the differential space distortion map creation unit 30 cuts out an image area of the same size as the template centered on the position indicated by the mouse 103 on the differential image. Is set as a differential template.

FIG. 12 is an explanatory diagram showing a second example of the setting method of each template. A common template is set in the mobile object tracking device shown in FIG. This common template is a template set so as to detect any moving object that may be specified as a target. This common template is manually set in advance. Here, this common template is used.

An original image including a moving object to be designated as a target is displayed on the CRT display unit 104 shown in FIG. 1, and a desired moving object is designated by operating the mouse 103. At this time, a moving image is displayed on the CRT display unit 104, and a moving object to be specified as a target may be moving, and the position of the target is not always specified accurately. Therefore, in the example shown here, the arithmetic unit 31 configuring the differential space distortion map creating unit 30 has a region centered on the designated position on the differential image and larger than the template (in FIG. Area) is set, and within that area, SAD processing is performed with the common template,
The minimum value position is detected, an image area centered at the minimum value position and having the same size as the template is cut out, and the cut out image area is set as a differential template. The arithmetic unit 11 constituting the color space distortion map creating unit 10 receives information on the minimum value position from the arithmetic unit 31 constituting the differential space distortion map creating unit 30, and receives the minimum value position on the color separation image. , An image area of the same size as the template is cut out, and the cut out image area is set as a fixed template and is initialized as a temporary template.

FIG. 13 is an explanatory diagram showing a third example of the setting method of each template. The processing described with reference to FIG. 13 is processing subsequent to the processing described with reference to FIG. However, the fixed template is not yet set at the stage of the processing described with reference to FIG. 12, and only the differential template and the temporary template are set. Under such circumstances, the tracking of the target is started. However,
In the color space distortion map creation unit 10, since no fixed template has been set yet, only the temporary template is used.

In this manner, when the target position is detected by the minimum value position detecting section 50, the target position information is transferred to the differential space distortion map forming section 30.
Is input to the differential template updating unit 32 constituting
It is assumed that hatched positions in FIG. 13 are target positions detected by the minimum value detection unit 50. In the differential template updating unit 32, the minimum value detecting unit 5 on the differential image
An image area of the same size as the template centered on the target position detected by 0 is cut out, and the cut-out image area and the differential template already set are averaged for each corresponding pixel and newly obtained. Is obtained, and the previously set differential template is updated with the obtained new differential template. The temporary template is updated by the color template updating unit 70. The above updating process is repeated for a preset number of frames, and thereafter, the differential template updating unit 32 stops its operation, so that the differential template is not updated thereafter and is fixed at the current differential template. . Regarding the temporary template, the color template updating unit 70
, The update process is continued.

Further, regarding the fixed template, immediately before the updating operation is stopped by the differential template updating unit 32,
The target position information detected by the minimum value position detecting unit 50 is transmitted from the differential template updating unit 32 to the calculating unit 11 constituting the color space distortion map creating unit 10, and the calculating unit 11 Then, an image area centered on the target position at that time and having the same size as the template is cut out, and the cut out image area is set as a fixed template.

The mobile object tracking device shown in FIG. 2 is realized inside the computer system shown in FIG. 1, and is configured by software except for the correlator 80. A considerable real-time property is secured. Further, the moving object tracking device shown in FIG. 2 can be implemented by hardware. If the device shown in FIG. 2 is constituted by hardware, the real-time property can be further improved. In addition, the moving object tracking device shown in FIG. 2 captures various information of the target and integrates them in the form of a two-dimensional map to detect the target position. It is possible to track the target with a practically accurate accuracy.

FIG. 14 is a block diagram showing another embodiment of the moving object tracking apparatus according to the present invention. Camera 100
The image photographed by the moving object tracking device 200 is sequentially captured in real time by the input means 201 of the moving object tracking device 200 and tracked by the moving object tracking means 202. The contents of the moving object tracking means correspond to the entire apparatus shown in FIG. 2 and are as described above. The mark generating means 203 converts a vertically long elliptical mark created in advance into a moving object tracking means 202.
Is formed at the position of the moving body calculated by Mark generating means 2
02 is combined with the image captured by the camera 100 by the telop display device 300, and the broadcast device 4
Broadcast at 00.

The image broadcasted by the broadcasting device 400 is
An image is received by a receiver (not shown) (for example, a television), and an image is displayed on the display screen. On the displayed image, a moving object is marked with a vertically long elliptical mark. Even if the moving object moves, the moving object can always be noticed without being lost.

[0099]

As described above, according to the present invention,
A mobile object tracking device with improved real-time performance and tracking performance is realized.

[Brief description of the drawings]

FIG. 1 is a schematic diagram of an automatic tracking system including an embodiment of a moving object tracking device according to the present invention.

FIG. 2 is a functional configuration diagram of a mobile object tracking device as one embodiment of the present invention constructed in the computer system shown in FIG. 1;

FIG. 3 is an explanatory diagram of a process of creating a color space distortion map.

FIG. 4 is a diagram showing a differential filter and a template.

FIG. 5 is an operation explanatory diagram of a reliability evaluation unit.

[6] Two distortion map D _1, D ₂ of the minimum value D _1min, temporal changes in the D _2min, and uses two distortion map D _1, D ₂ in the subsequent processing (reliable It is a figure which shows the time change of whether or not.

FIG. 7 is an explanatory diagram of the operation of two weighting addition units.

FIG. 8 is an explanatory diagram of an operation of a motion predicted value map creating unit.

FIG. 9 is a diagram illustrating the operation of an approximate object detection unit.

FIG. 10 is an operation explanatory diagram of an evaluation / integration unit.

FIG. 11 is an explanatory diagram showing a first example of a setting method of each template.

FIG. 12 is an explanatory diagram showing a second example of a setting method of each template.

FIG. 13 is an explanatory diagram showing a third example of a setting method of each template.

FIG. 14 is a block diagram showing another embodiment of the moving object tracking device of the present invention.

[Description of Signs] 1 Computer system 2 Video camera 3 Servo system 10 Color space distortion map creation unit 11 Operation unit 20 Differential processing unit 21 Differential operation unit 30 Differential space distortion map creation unit 31 Operation unit 40 Evaluation value map creation unit 41 Reliability Sex evaluation sections 42, 43 weighting addition section 50 minimum value position detection section 60 motion prediction value map creation section 61 next frame prediction motion vector calculation section 62 prediction function generation section 70 color template update section 71 approximate object detection section 71a detection of minimum value position Unit 71b Interference object monitoring unit 72 Evaluation / integration unit 100 Camera 101 Main unit 102 Keyboard 103 Mouse 104 CRT display unit 200 Moving object tracking device 201 Input means 202 Moving object tracking means 203 Mark generation means 204 Telop display device 20 Broadcast equipment

Claims (16)

[Claims] 1. An original image comprising a plurality of consecutive frames, which is obtained by capturing a scene including one or more moving objects, is input, and an object selected from the moving objects reflected in the original image is input. In a moving object tracking apparatus for tracking a tracking moving object, a first correlation value between a color template to be compared with a tracked moving object projected on a color separation image constituting the original image and the color separation image A first correlation value distribution calculating unit for obtaining a distribution of the differential image; a differential processing unit for obtaining a differential image by spatially differentiating the original image; and a tracking target moving object displayed on the differential image. A differential template, a second correlation value distribution calculation unit for obtaining a distribution of a second correlation value with the differential image, and the tracked moving object is present based on past movement of the tracked moving object. Motion prediction value representing probability And a motion prediction value distribution calculation unit that calculates the first correlation value, the second correlation value, and the motion prediction value to obtain an evaluation value, thereby indicating the position of the moving object to be tracked. An evaluation value distribution calculating unit that obtains an evaluation value distribution; and a position detecting unit that detects an existing position of the tracked moving object based on the evaluation value distribution obtained by the evaluation value distribution calculating unit. A moving object tracking device characterized by the above-mentioned. 2. The evaluation value distribution calculation unit includes a reliability evaluation unit that evaluates reliability of the distribution of the first correlation value, and calculates a distribution of the first correlation value determined to be low in reliability. 2. The moving object tracking apparatus according to claim 1, wherein the evaluation value distribution is obtained after excluding the evaluation value distribution from the processing for obtaining the evaluation value distribution. 3. The evaluation value distribution calculation unit includes a reliability evaluation unit that evaluates the reliability of the distribution of the first correlation value and the reliability of the distribution of the second correlation value, and the reliability is low. After excluding the distribution of the first correlation value determined to be and the distribution of the second correlation value determined to be low in reliability from the process of obtaining the distribution of the evaluation values, the distribution of the evaluation values is 2. The moving object tracking device according to claim 1, wherein the moving object tracking device is obtained. 4. An image in which the color template includes a sequentially updated temporary template, and includes an existing position of the tracked moving object detected by the position detection unit in the color separation image. 2. The moving object tracking apparatus according to claim 1, further comprising a color template updating unit that updates the temporary template by integrating an area into the temporary template. 5. The integrated correlation value calculation unit calculates an integrated correlation value for each pixel by integrating the first correlation value and the second correlation value for each pixel. A first integrated processing unit for obtaining a value distribution, and an integrated correlation value obtained by the first integrated processing unit and a motion prediction value obtained by the motion predicted value distribution calculation unit for each pixel. And a second integrated processing unit that obtains an evaluation value for each pixel to obtain a distribution of the evaluation values, wherein the template updating unit performs, based on the distribution of the integrated correlation values, An approximate object detection unit that detects the existence position of an approximate object that approximates the tracked moving object by allowing detection of a plurality of positions; and in the color-separated image, A first image area including the position of the moving object, and an approximation of the temporary plate. A first degree of approximation representing a degree, and a second degree of approximation between the first image area and a second image area including the position of the approximate object detected by the approximate object detection unit. Find the degree of approximation,
When the first degree of approximation indicates that the degree of approximation is greater than the second degree of approximation, the first image area is integrated with the temporary template to evaluate the updating of the temporary template. The moving object tracking device according to claim 4, further comprising: an integration unit. 6. The first correlation value distribution calculator and the second correlation value distribution calculator, which are shared by the first correlation value distribution calculator and the second correlation value distribution calculator. A correlator for determining a correlation value obtained by integrating an absolute value of a difference between pixel values assigned to pixels corresponding to each other between two images or image areas over the two images or image areas; 2. The moving body according to claim 1, wherein the differential processing unit causes the correlator to execute a part of differential operations for differentiating the original image to obtain a differential image. Tracking device. 7. The motion prediction value distribution calculating unit obtains a motion vector predicting the motion of the moving object to be tracked from the current time to the next time, and calculates the next time point represented by the motion vector predictor. The motion prediction value representing the maximum probability of the predicted position of the tracked moving object in the above, and the probability of decreasing as the distance from the predicted position increases, is associated with each pixel in the search area. The moving object tracking device according to claim 1. 8. A motion vector predictor calculated by the motion detector, the motion vector predictor being calculated at a past time, the motion vector indicating a predicted position of the moving object to be tracked at the present time. Based on the motion vector representing the actual movement of the tracked moving object determined from the difference between the positions of the tracked moving object detected both at the current time and the current position, 8. The moving object tracking device according to claim 7, wherein a motion vector predicting a motion from a current time to a next time is obtained. 9. The motion prediction value distribution calculation unit calculates a motion prediction value as a function of a distance centered on a predicted position of the tracked moving object represented by the predicted motion vector for each pixel in the search area. 8. The moving object tracking device according to claim 7, wherein 10. The motion prediction value distribution calculation unit, wherein the predicted motion vector represents a center of a predicted position of a tracked moving object, and a weight for a distance in a direction of the predicted motion vector and a direction intersecting the direction. The moving object tracking according to claim 7, wherein a motion prediction value as a function of the two distances having different weights from the distance is associated with each pixel in the search area. apparatus. 11. The motion prediction value calculation unit associates a motion prediction value as a function with the magnitude of the prediction motion vector as a variable with each pixel in the search area. The mobile object tracking device according to claim 7. 12. The differential image, wherein the second correlation value calculating section includes a predetermined position on a differential image, which is the same as a predetermined position on an original image designated prior to tracking of the tracked moving object. 2. The moving object tracking device according to claim 1, wherein a predetermined image area in the moving object is set as the differential template. 13. The differential correlation value calculating section, wherein the second correlation value distribution calculating section includes a predetermined position on a differential image which is the same as a predetermined position on an original image designated prior to the tracking of the moving object to be tracked. In the image, within the image area of a size larger than the size of the derivative template, the derivative having a high degree of approximation with a common template commonly set for a moving body that can be designated as a tracked moving body. 2. The moving object tracking apparatus according to claim 1, wherein a partial area having the same size as the template is detected, and the partial area is set as the differential template. 14. The method according to claim 1, wherein the second correlation value distribution calculation unit detects the differential template in the differential image at one or more times after the time when the differential template is set for the first time. In addition, by integrating the image area including the position of the target to be tracked into the differential template,
14. The moving object tracking apparatus according to claim 13, further comprising a differential template updating unit that operates at a predetermined initial stage after starting tracking of the tracked moving object, for updating the differential template. 15. The color template includes a fixed template which is fixed, and wherein the first correlation value distribution calculating section sets the color template when the differential template is set or finally updated. In the decomposition image,
15. The image processing apparatus according to claim 12, wherein an image area including the position of the tracked moving object detected by the position detection unit is set as the fixed template. Mobile tracking device. 16. An input means for sequentially and sequentially inputting an original image composed of a plurality of frames obtained by photographing a scene including a moving body, tracking a moving body having a different position of the moving body between the plurality of frames. Moving body tracking means for sequentially outputting the position of the moving body in the frame at the continuous input time; and a moving body tracked by the moving body tracking means of the original image input by the input means. A moving object tracking device comprising a mark generating means for displaying a mark at a position of the moving object.