JP3174869B2 - Target object tracking device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention performs a residual operation between input image data obtained by an image pickup means such as a camera and reference image data stored in a storage means to obtain a field of view of the image pickup means. The present invention relates to a target object tracking device that tracks a predetermined target object in an area.

[0002]

2. Description of the Related Art A target object tracking apparatus of this type is disclosed in, for example, Japanese Patent Publication No. 60-33350 and Japanese Patent Publication No.
There is one disclosed in Japanese Patent No. 9076. These obtain a correlation coefficient between a reference screen stored in advance and an input screen obtained by A / D-converting an image signal supplied from the imaging apparatus, and detect a coordinate giving a maximum value of the correlation coefficient to obtain a predetermined value. Track the target object. At this time, for the purpose of realizing the tracking operation in real time, an area for correlation calculation is provided in the input screen, and the correlation coefficient calculation with the reference screen is performed in this area.

[0003]

However, in the above-described conventional target object tracking apparatus, the area for correlation calculation is set centering on the position of the target object obtained by the previous tracking operation. Or fixed within the input screen, a fast moving target object such as a sports scene may fall out of the correlation calculation area, making it difficult to track its position. .
In a sports scene or the like, when the shape of the target object changes (for example, the front direction becomes horizontal) or the luminance of the target object changes (for example, the target object moves from the sun to the shade), the reference image stored in advance. In some cases, it may not be possible to track the position of the target object depending on the correlation calculation.

It is an object of the present invention to provide a target object tracking device capable of quickly and reliably tracking the position of a fast moving target object.

[0005]

According to a first aspect of the present invention, there is provided an imaging apparatus for measuring an object field and outputting input image data having a plurality of color components. Means 101, storage means 102 for storing reference image data serving as image data of a target object to be tracked based on the output of the photographing means 101, and calculation for calculating the minimum residual amount of the input image data and the reference image data Means 10
3 and a target object tracking device including a position detection unit 104 for detecting the position of the target object based on the minimum residual amount. Then, an adjacent area setting means 1 for setting an adjacent area adjacent to the area including the reference image data in the object scene
07, the residual value of the reference image data and the residual value of the input image data included in the adjacent area are calculated for each of a plurality of color components, and the residual value color detecting means 109 for detecting the color having the maximum minimum residual value is detected. And a color selecting means 105 for selecting a color detected by the residual value color detecting means 109. The calculating means 103 calculates the minimum residual amount for the color component selected by the color selecting means 105. Thereby, the above-described object is achieved. According to a second aspect of the present invention, in the target object tracking apparatus according to the first aspect, the adjacent area setting means 107 includes:
An adjacent area is set in the object scene by shifting an area including the reference image data corresponding to a predetermined number of pixels. According to a third aspect of the present invention, in the target object tracking apparatus according to the second aspect, the adjacent area setting means 107 includes:
An adjacent area is set in the field by shifting the area including the reference image data up, down, left and right, and left, right, up and down by one pixel.

[0006]

The color selecting means (105) calculates a residual value calculated for each of a plurality of color components with respect to reference image data and input image data included in an area adjacent to the area including the reference image data. The color having the largest minimum value is selected, and the calculation means 10
3 calculates the minimum residual amount for the color component selected by the color selection means 105. When one color is selected from a plurality of existing colors by the color selection unit 105, the calculation procedure in the calculation unit 103 is compared with the case where the calculation unit 103 calculates the minimum residual amount for all the color components. Omission and shortening of calculation time can be achieved.

[0007]

【Example】

(Embodiment) FIG. 2 is a schematic diagram showing a camera in which an embodiment of the target object tracking apparatus according to the present invention is incorporated. In the figure, reference numeral 1 denotes a single-lens reflex camera (hereinafter simply referred to as a camera) applied to silver halide photography. In this camera 1, light from a subject passes through a photographing lens 2 in front of the camera 1 and is reflected by a mirror 3 The light is reflected upward and forms an image on the screen 4. The photographer observes the subject image formed on the screen 4 through the prism 5 and the eyepiece 9. A subject image on the screen 4 is formed on an image sensor 7 by a re-imaging lens 6, and an output from the image sensor 7 is processed by an arithmetic unit 8 including a microcomputer or the like. The arithmetic unit 8 includes a storage element (not shown), in which a reference image serving as a template is stored as digital data.

FIG. 3 is a diagram showing details of the image sensor 7. As shown in FIG. 3, the image sensor 7 includes a plurality of pixels 1 (18 columns × 12 rows in the illustrated example) arranged in a matrix.
0, an image equivalent to the object scene image observed by the photographer is formed on the image sensor 7. In the figure, A is a visual field region corresponding to the entire object field observed by the photographer (that is, the entire image sensor 7), B is a tracking region indicating a target object detection position, and in this embodiment, the tracking region B has four columns. It is composed of × 4 rows of pixels. Each pixel 10 is further divided into three fine pixels 11a to 11c as shown in FIG.
Each of the fine pixels 11a to 11c is provided with an RGB primary color filter (not shown) as shown in the figure, so that an RGB output of a subject image can be obtained.

Next, the operation of the camera 1 of the present embodiment will be described with reference to FIGS. 5 to 6, FIGS. 9 to 10, and FIGS. (1) Color Selection Process The program shown in the flowchart of FIG. 5 is a process in which a photographer observes a subject image and captures a target subject in the tracking area B, for example, half-presses a release button (not shown). The operation is started by the photographer instructing the subject to be captured in the above operation. At the start of the program,
It is assumed that the tracking area B is located substantially at the center of the visual field area A as shown in FIG.

First, in step S 1, the RGB output of each pixel in the tracking area B is obtained from the image sensor 7. Tracking area B
Are stored in a temporary storage element (not shown) in the arithmetic unit 8. In the subroutine SR1, a color selection process is performed to select any one of the RGB outputs in the tracking area B obtained in step S1.

The details of the color selection process are shown in the flowchart of FIG. In step S101, the RGB outputs in the tracking area B obtained in step S1 are added for each color.
In step S102, the maximum value of the addition result obtained in step S101 is obtained, and the color (R,
G or B) is detected and selected. Hereinafter, the selected color is indicated by α. Thereafter, the program returns to the main routine of FIG.

In step S2, the RGB in the tracking area B
Out of the outputs, the output of the color α selected in the subroutine SR1 is set as B _ij (i, j = 1 to 4), and stored in a storage element (not shown) in the arithmetic unit 8 in a matrix form. This value B _ij is
This is reference image data including a target subject (target object) whose position is to be tracked.

(2) Calculation Area Setting In step S 3, an output of the color α of each pixel in the visual field area A is obtained from the image sensor 7. The output of the color α in the visual field area A is stored in a temporary storage unit (not shown) in the arithmetic unit 8. In step S4, a velocity vector and an acceleration vector of the tracking area B are calculated. As shown in FIG. 11, the tracking area B moves within the visual field area A with the movement of the target subject, and its position is set each time the target subject is detected in step S9 described later. The velocity vector V (v _x , v _y )
It is defined by the number of pixels in the row direction (hereinafter also referred to as Y direction) and the column direction (hereinafter also referred to as X direction) in which the tracking area B has moved during the target subject detection interval in step S9. Further, the acceleration vector AC (a _x, a _y), the velocity vector V ₁ adjacent to each other, V ₂ row and column directions of the amount of change (see Fig. _{11) (a x = v 1x} -v 2x, a y = _{V1y- v2y} )
Is defined by

In step S5, the absolute values | v _1x -v _2x |, | v _1y -v of the amounts of change in the components of the velocity vector V calculated in step S4 (ie, the components of the acceleration vector).
_2y | it is determined whether predetermined thresholds T ₁ smaller than, the determination is affirmative the program proceeds to step S6, the determination is negative the process proceeds to step S7. Thresholds T ₁ is for determining whether the target object is moving at approximately constant velocity, is appropriately set according to the measurement error of the velocity vector. In step S6, the target object because the amount of change in the velocity vector V _1, V ₂ are thresholds T ₁ following a small value is assumed to be moved also have approximately the same velocity vector next, V ₀ as shown in FIG. 11 = V ₁
In addition, the velocity vector V ₀ is calculated from the current position of the tracking area B.
The calculation area C (the calculation area will be described later) is set at a position shifted by only this. After this, the program proceeds to step SR
Move to 2.

Next, in step S7, whether the absolute value is greater than the threshold T ₂ for the predetermined is calculated acceleration vector AC in step S4, it is determined if the decision is affirmative the program to step S8 The process proceeds to a subroutine SR2 if the determination is negative. The threshold T ₂ is
This is for determining whether or not the speed of the target subject is rapidly changing, and is appropriately set in accordance with the size of the calculation area. In step S8, the movement of the target subject from the value of the acceleration vector AC is the threshold value T ₂ or more large value is indeterminate (Shirezu be speed is accelerated more or might suddenly stop, and), V ₀ = V ₁
In addition, the velocity vector V ₀ is calculated from the current position of the tracking area B.
Calculation area C larger than the normal calculation area
(For example, twice or more for both rows and columns).

(3) Minimum residual amount calculation In the subroutine SR2, a minimum residual amount calculation is performed. Details of the minimum residual amount calculation are shown in the flowchart of FIG. First, in step S201, initialization is performed.
Specifically, a detection area is set at the upper left corner in the calculation area C shown in FIG. 12, and 1 is substituted for the counter n. FIG. 12 is a diagram showing a positional relationship among a tracking area, a calculation area, and a detection area. In FIG. 12, C is a calculation area indicating an area where a target subject is searched. In the illustrated example, the calculation area C is set in an area that is larger by one pixel in the row direction and the column direction than the tracking area B with respect to the tracking area B in the illustrated example. Not limited.
D is a detection area indicating an area where a residual calculation is performed with respect to the tracking area B, and is composed of 4 columns × 4 rows of pixels equal to the tracking area B.

In step S203, step S201
The output D _ij (i, j = i) of the color α in the detection area D set by
1 to 4) are obtained from the output of the visual field area A obtained in step S3. FIG. 1 shows the positional relationship between output B _ij (which is also reference image data) and D _ij of color α in tracking area B and detection area D when the area shown in FIG. 12 is set.
3 is shown.

In step S204, reference image data B
The residual amount of _ij and the output D _ij of the detection area D is calculated.
The residual amount is the sum of the differences between the outputs in the corresponding (i.e., common subscript) pixel units, that is, ΣΣW _ij | B _ij −D _ij |
(Where ΣΣ represents the sum of i = 1 to 4 and j = 1 to 4). W _ij is a weight coefficient, and the amount of information decreases when the absolute difference is calculated, and the amount of information further decreases when the sum is calculated. This is for preventing the difference amount from being minimized (called pseudo matching). In consideration of the fact that the closer to the center of the reference image data B _ij , the higher the probability of giving important data of the target subject is, the weighting factor W _ij is set larger for pixels closer to the center, and the value is determined empirically. . As an example, if both i and j are 2 or 3, W _ij = 2, otherwise W _ij = 1. Residual amount is stored in the storage element in the arithmetic unit 8 together with the operation number n in the operation region C as a variable Sum _n.

In step S205, the detection area D is moved for the next calculation of the residual amount, and the counter n is incremented by one. That is, the detection area D is shown in FIGS.
When the detection area D is already located at the right end of the calculation area C, the detection area D is shifted down by one pixel and at the left end of the calculation area C. In this way, the detection area D sequentially moves from left to right and top to bottom in FIGS. 12 and 13 for each pixel. The output B _ij of the tracking area B and the color α in the detection area D in a state where the detection area D is shifted to the right by one pixel from the position shown in FIG.
FIG. 14 shows the positional relationship of D _ij .

In step S206, it is determined whether or not the value of the counter n has reached the final value (n = 10 in the illustrated example). If the determination is affirmative, the program proceeds to step S207.
The process returns to step S203 if the determination is negative, and the above-described processing is repeated. Thus, the detection area D
Are repeated, and the residual calculation is repeated with reference image data to calculate nine residual amounts. FIG. 15 shows the positional relationship between the outputs B _ij and D _ij of the color α in the tracking area B and the detection area D at the end of the residual amount calculation.

In step S207, Sum _n (n = 1)
To 9) for detecting the Sum _n giving the minimum value (minimum residual amount) of the. In step S208, it is determined that the target object is present in the detection region D that gives the minimum value of Sum _n, calculates the position of the detection area D. After this, Figure 5
Return to the main routine.

(4) Movement of tracking area, next calculation area setting In step S9, the tracking area B is moved to the position of the detection area D including the target object calculated by the subroutine SR2. Thereby, tracking of the moving target object can be performed. Further, in the present embodiment, as shown in FIG. 11, the position of the past tracking area B is required for the calculation of the velocity vector V and the acceleration vector AC. Is stored within.

In the subroutine SR3, the next calculation area is set. This is because, as shown in FIG. 16, assuming that the pixel width in the column direction is S _x and the pixel width in the row direction is S _y with respect to the pixel width indicating how many pixels the calculation area C is larger than the tracking area B, the residual error The relationship n = (2S _x +1) × (2S _y +1) exists between the number of times n and S _x , S _y, and when the calculation area C is enlarged so that the target object is not lost, the residual amount becomes Since the number of calculations n increases and the time required to calculate the target subject position becomes longer, and as a result the target subject may be lost, the target subject may be lost. Therefore, based on the minimum residual amount calculated in the subroutine SR2, To set the optimal size of the calculation area C.

The details of the next calculation area setting are shown in the flowchart of FIG. First, in step S301, the minimum residual amount obtained in the subroutine SR2 is determined whether the threshold T ₃ is greater than that, if the decision is affirmative the program proceeds to step S302, if the decision is negative step Move to S305. Step S302, S3
In 03, assuming operation target object is large because the minimum residual amount is larger value than the threshold value T _3, the work to extend the operation region C is carried out. That is, the pixel widths S _x and S _y in the column direction and the row direction of the calculation area C are (minimum residual amount /
T ₃ + constant). The term (minimum residual amount / T ₃₎ than that in step S301 the minimum residual amount> T ₃ becomes determined are taking one or more values, increases in proportion to the increase of the minimum residual amount. Then in step S304, in order to stabilize the size of the extended operation region C, increasing the threshold value T ₃ by a predetermined amount. On the other hand, the step in S305,306, assuming operation target object is small because the minimum residual quantity is small the value of the threshold T ₃ below, the column and row directions of the pixel width S _x of the calculation region C, S _y
Are reduced by one pixel. Then in step S307, it provided the operations after the target object to reduce the threshold T ₃ by a predetermined amount. Thereafter, the program returns to the main routine.

[0025] (5) In the reference image updating step S10, the minimum residual amount obtained in the subroutine SR2 is determined whether larger than the threshold value T ₃ described above, the determination is affirmative the program returns to step S3 Are repeated. On the other hand, if the determination is negative, the program moves to step S11.

In step S11, the reference image is updated. In the reference image update, every time an image is input in order to cope with a continuous change of the target subject (for example, from a frontal face to a side face), a part of the reference image data is updated to include new data. In this step, a value given by the following equation B _ij = (1−k) B _ij + kD _ij (i, j = 1 to 4) is substituted into the reference image data B _ij . Here, it is an empirical value given by 0 ≦ k ≦ 1, usually k = about 0.1 to 0.2. Also,
D _ij is, Sum _n in the subroutine SR2 is data of the detection area D of the minimum value. However, when the movement of the target subject is intense, or when an obstacle passes in front of the target subject, data other than the original target subject is captured when the reference image is updated.
As shown in step S10 above, the minimum residual amount is equal to the threshold T
If the value is greater than ₃ , update processing is prohibited. Thereafter, the program returns to step S3 and repeats the above processing.

With the above operation, the position of the target object imaged by the camera 1 in the finder screen can be tracked. Here, in the present embodiment, the output for one color α is selected from the RGB outputs from the image sensor 7 and the residual calculation is performed, so that the calculation time for target object detection can be shortened as a whole, As a result, the position of a fast-moving target subject such as a sports scene can be quickly tracked. In addition, by selecting the color α at which the maximum output is obtained, a characteristic color can be selected for detecting the target object, and the position detection can be reliably performed by compensating for a decrease in the information amount due to the color output selection. Furthermore, since the reference image data is partially updated each time the position is detected, the target object can be reliably captured even if there is a continuous change in the target object or a change in the luminance of the target object. Detection becomes possible. Moreover, in the present embodiment, the calculation area C is provided in the visual field area A corresponding to the image sensor 7,
The position of the calculation region C is predicted by measuring the speed and acceleration of the target subject, and the size of the calculation region C is optimally controlled by the size of the minimum residual amount, as in the conventional example described above. A situation in which the target object is lost can be prevented, and the target object can be reliably tracked. This eliminates the need to obtain the residual amount for the entire screen, and enables the position of even a fast-moving target subject such as a sports scene to be reliably and quickly tracked.

(Modification) As for the color selection, not only the method of selecting the color α that provides the maximum output as shown in the above-described embodiment, but also other suitable methods can be adopted. FIG. 7 is a flowchart illustrating another example of the color selection process. Step S4
01, the RG in the tracking area B obtained in step S1
The B output is added for each color. In step S402, as shown in FIG. 17, the tracking area B is vertically and horizontally adjacent to each other, and in the same manner as the tracking area B, there are four adjacent areas B _{1 to} B ₄ composed of 4 columns × 4 rows of pixels. The respective RGB outputs are added for each color. In step S403, it obtains a difference between the addition result of the addition result and the respective adjacent regions B ₁ .about.B ₄ in the tracking area B obtained in step S401 and S402 for each color, the color of its maximum value is obtained ( R,
G or B) is detected and selected. Thereafter, the program returns to the main routine of FIG.

FIG. 8 is a flowchart showing another example of the color selection process. In step S501, similarly to the tracking area B, eight detection areas and tracking areas which are composed of pixels of 4 columns × 4 rows and which are respectively shifted from the tracking area B by one pixel vertically, horizontally, and diagonally left and right and up and down. Are obtained for each color. Step S
In 502, the minimum value of the eight residual amounts obtained in step S501 is obtained for each color. In step S503, from the minimum value of the residual amount obtained in step S502, a color (one of R, G, and B) having the maximum value is detected and selected. Thereafter, the program returns to the main routine of FIG.

The details of the target object tracking apparatus of the present invention are not limited to the above-described embodiment, and various modifications are possible. As an example, in one embodiment, an example in which the present invention is applied to a single-lens reflex camera has been described. However, the present invention is also applicable to a moving image capturing unit such as a video camera. In addition, the reference image data serving as a template is not limited to the one captured before the target object tracking as in the above-described embodiment, and if the shape or the like of the target object can be known in advance, the reference image data corresponding to the target object is used. It may be stored in advance. Similarly, the input image data is not limited to data input in real time, and may be data stored in advance. Further, in the above-described embodiment, the calculation area is set using both the speed and the acceleration. However, only one of them may be set. When the acceleration is used, the zero acceleration means that the target object is moving at a constant speed. Therefore, if the acceleration and the initial speed are detected, both the position and the size of the calculation area can be set.

[0031]

As described above in detail, according to the first aspect of the present invention, the residual values of the reference image data and the input image data included in the adjacent area adjacent to the area including the reference image data are calculated. A color having a maximum value of the calculated residual value is selected for each of the plurality of color components, and the residual amount is calculated for the selected color component. Compared with the case of calculating the difference amount, it is possible to omit the residual calculation procedure and shorten the calculation time. This makes it possible to quickly track the position of a fast-moving target object such as a sports scene. In particular, since a characteristic color can be selected for detecting a target object, reliable position detection can be performed by compensating for a decrease in the amount of information due to color output selection.

[Brief description of the drawings]

FIG. 1 is a diagram corresponding to claims of the present invention.

FIG. 2 is a schematic diagram showing an embodiment of a camera to which the target object tracking device of the present invention is applied.

FIG. 3 is a diagram illustrating details of an image sensor.

FIG. 4 is a diagram showing details of a pixel.

FIG. 5 is a flowchart for explaining the operation of one embodiment.

FIG. 6 is a flowchart illustrating an example of a color selection process.

FIG. 7 is a flowchart illustrating another example of the color selection process.

FIG. 8 is a flowchart illustrating another example of the color selection process.

FIG. 9 is a flowchart illustrating details of a minimum residual amount calculation.

FIG. 10 is a flowchart showing details of next calculation area setting.

FIG. 11 is a diagram showing a definition of a velocity vector and prediction of a subject position based on the velocity.

FIG. 12 is a diagram showing a positional relationship when a tracking area, a calculation area, and a detection area are initially set.

FIG. 13 is a diagram showing a positional relationship between color output in a tracking area and a detection area at the time of initial setting.

FIG. 14 is a diagram illustrating a positional relationship between color outputs in a tracking area and a detection area during a residual calculation.

FIG. 15 is a diagram illustrating a positional relationship between color outputs in a tracking area and a detection area at the end of residual calculation.

FIG. 16 is a diagram for explaining a calculation area setting procedure.

FIG. 17 is a diagram illustrating a positional relationship between a detection area and an adjacent area in a color selection process 2;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Camera 7 Image sensor 8 Arithmetic unit 10 pixel

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-4-117089 (JP, A) JP-A-2-123877 (JP, A) JP-A-62-206980 (JP, A) JP-A-62-206980 237591 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 7/20 G02B 7/28 H04N 7/18

Claims (3)

(57) [Claims] 1. An image pickup means for measuring an object scene and outputting input image data having a plurality of color components, and reference image data serving as image data of a target object to be tracked based on an output of the image pickup means. Storage means for storing a minimum residual amount of the input image data and the reference image data; and a position detecting means for detecting a position of the target object based on the minimum residual amount. In the target object tracking device, an adjacent area setting means for setting an adjacent area adjacent to an area including the reference image data in the object scene; and the input image data included in the reference image data and the adjacent area. A residual value of each of the plurality of color components, and a residual value color detecting means for detecting a color having a maximum minimum value of the residual value; and a color detected by the residual value color detecting means. And a color selection means for selecting, said arithmetic means, a target object tracking apparatus characterized by computing the minimum residual amount for the components of the color selected by the color selection means. 2. The target object tracking device according to claim 1, wherein the adjacent area setting means shifts the area including the reference image data by a predetermined number of pixels to shift the adjacent area. A target object tracking device, which is set within a scene. 3. The target object tracking device according to claim 2, wherein the adjacent area setting means shifts the area including the reference image data up, down, left, right, and left, right, up, down, and up by one pixel. A target object tracking device, wherein an area is set within the object scene.