JP5365327B2 - Subject tracking device and camera - Google Patents

Description

  The present invention relates to a subject tracking device and a camera.

  The following automatic tracking devices are known. This automatic tracking device calculates the variance value of the similarity map in the search image, and determines whether or not to update the template based on the degree of variance (for example, Patent Document 1).

JP-A-9-265538

  However, according to the conventional automatic tracking device, the template is not updated when the variance value is a predetermined value or more, and there is a possibility that tracking of the subject cannot be continued when the subject changes.

The subject tracking device according to claim 1, a similarity calculation unit that calculates a similarity value between a template image and a target image within a search frame set in the input image, and a variance of similarity values in the input image A calculation means for calculating a value or a standard deviation value, a specification means for specifying a similarity value satisfying a predetermined condition for the similarity between the template image and the target image in the input image, and a specific similarity value Accumulated for each frame, based on the accumulated specific similarity value, it is judged that the similarity between the template image and the target image has decreased continuously for a predetermined frame, and the variance value or standard deviation value of the similarity value If There is less than a predetermined threshold value, a determination unit that it is necessary to update the template image, when the determination means determines that it is necessary to update the template image Characterized by comprising a updating means for updating the template images.
The subject tracking device according to claim 2, a similarity calculation unit that calculates a similarity value between a template image and a target image in a search frame set in the input image, and a variance of the similarity value in the input image A calculation means for calculating a value or a standard deviation value, a specification means for specifying a similarity value satisfying a predetermined condition for the similarity between the template image and the target image in the input image, and a specific similarity value It is accumulated for each frame, and based on the accumulated specific similarity value, it is determined that the similarity of the current frame is decreased at a certain rate or more with respect to the average value of the similarity of a plurality of past frames, and A determination unit that determines that the template image needs to be updated when the variance value or the standard deviation value of the similarity value is equal to or less than a predetermined threshold, and the determination unit needs to update the template image When it is determined that, characterized in that it comprises an update means for updating the template images.
A camera according to the present invention includes an imaging unit that captures an image of a subject and obtains the image, and any one of the subject tracking devices described above.

  According to the present invention, subject tracking can be continued even when the subject changes.

1 is a block diagram illustrating a configuration of an embodiment of a camera 100. FIG. It is a flowchart figure which shows the flow of the update process of a template image. It is a figure which shows the relationship between the variance value of a similarity value in a similarity map, and the raise degree of a minimum similarity value.

  FIG. 1 is a block diagram illustrating a configuration of an embodiment of a camera according to the present embodiment. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various input members operated by the user, such as a power button, a release button, a zoom button, a cross key, an enter button, a play button, and a delete button.

  The lens 102 is composed of a plurality of optical lenses, but is representatively represented by one lens in FIG. The image sensor 103 is an image sensor such as a CCD or a CMOS, for example, and captures a subject image formed by the lens 102. Then, an image signal obtained by imaging is output to the control device 104.

  The control device 104 generates image data in a predetermined image format, for example, JPEG format (hereinafter referred to as “main image data”) based on the image signal input from the image sensor 103. Further, the control device 104 generates display image data, for example, thumbnail image data, based on the generated image data. The control device 104 generates an image file that includes the generated main image data and thumbnail image data, and further includes header information, and outputs the image file to the memory card slot 105.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is written and recorded on the memory card. The memory card slot 105 reads an image file stored in the memory card based on an instruction from the control device 104.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . Further, when the user sets the mode of the camera 100 to the shooting mode, the control device 104 outputs image data for display of images acquired from the image sensor 103 in time series to the monitor 106. As a result, a through image is displayed on the monitor 106.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the camera 100. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the CPU to develop a program when the program is executed or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  In the present embodiment, the control device 104 performs a template matching process using a template image prepared in advance for each frame of a through image input from the image sensor 103, thereby resembling a template image from within the frame. The image area to be specified is specified. Then, the control device 104 performs subject tracking processing by tracking the specified region between frames.

  Specifically, the control device 104 performs a matching operation between a template image prepared in advance and each frame input from the image sensor 103 in time series. For example, when input of a through image from the image sensor 103 is started, the control device 104 sets a search frame having the same size as the template image at a predetermined position in the frame for each frame (frame). The target image B within the set search frame is set as a template matching target.

  That is, the control device 104 calculates the similarity between the target image and the template image at each search frame position while moving the search frame within the frame. Then, the control device 104 specifies the position where the similarity between the template image and the target image in the frame is the highest as the subject position.

  The control device 104 performs subject tracking for a through image by specifying a subject position in each frame input in time series from the image sensor 103 and tracking the subject position between frames.

  A method for calculating the degree of similarity between the template image and the target image is well known and will not be described in detail. However, in general, similarities such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference) are used. A degree calculation method is used.

For example, when SAD (residual sequential test) is used as the similarity calculation method, the control device 104 calculates the similarity value r by the residual sum shown in the following equation (1), and calculates the similarity Template matching is performed based on the value r. The similarity value r calculated by the following equation (1) indicates that the smaller the value is, the higher the similarity between the template image A and the target image B is, and the larger the value is, the higher the template image A and the target image B are. It shows that the degree of similarity is low.

In Expression (1), A represents a template image, and B represents a target image. M represents the pixel position in the horizontal direction in the template image A, and n represents the pixel position in the vertical direction in the template image A. For example, the pixel of the upper left corner in the template image A is expressed as A _11, the pixel of the upper left corner of the target image B is expressed as B _11.

  As described above, when subject tracking is performed by calculating the similarity between the template image and the target image, the shape of the subject to be tracked changes from moment to moment, so the initial template image prepared in advance is used as it is. As the similarity value is calculated, there is a possibility that the similarity gradually decreases and eventually the subject tracking cannot be continued. Therefore, in the present embodiment, in order to solve such a problem, the control device 104 observes a change in the similarity value between the template image and the target image in time series, and based on the change, the template image. The update timing is determined. Then, the control device 104 updates the template image by taking in the target image specified as the subject position in the current frame at the update timing of the template image and using it as a new template image.

  FIG. 2 is a flowchart showing a flow of template image update processing in the present embodiment. The processing illustrated in FIG. 2 is executed by the control device 104 as a program that is activated when input of a through image from the image sensor 103 is started.

  In step S1, the control device 104 reads image information, that is, one frame of the through image, and proceeds to step S2. In step S2, as described above, the control device 104 calculates the similarity between the target image and the template image at each search frame position while moving the search frame within the frame. Then, the control device 104 generates a similarity map in which the similarity value at each search position is two-dimensionally expressed. Then, it progresses to step S3.

  In step S3, the control device 104 detects the minimum similarity value from the similarity map generated in step S3. The search position where the minimum similarity value detected here is calculated is the search position with the highest similarity between the template image and the target image, that is, the subject position. The control device 104 records the minimum similarity value detected here in the SDRAM. As a result, the minimum similarity value detected for each frame can be accumulated.

In step S <b> 3, the control device 104 calculates a variance value of similarity values in the similarity map (hereinafter, referred to as “similarity map variance value”) by the following equation (2). In the following equation (2), x _i represents the similarity value of each search position in the similarity map, and x _ave represents the average value of the similarity values in the entire similarity map.

Thereafter, the process proceeds to step S4, where the control device 104 reads out the minimum similarity value stored in the SDRAM, and increases the minimum similarity value in the current frame with respect to the past frame (hereinafter referred to as “increase degree of minimum similarity value”). "). For example, in the present embodiment, the control device 104 increases the minimum similarity value of the current frame (nth frame) with respect to the average value of the minimum similarity values for the past n−1 frames by the following equation (3). The rate R _n is calculated, and this is set as the increase degree of the minimum similarity value. In the following formula (3), i is a frame number for identifying each of the past frames, and takes a range of 1 ≦ i ≦ n−1. P _i is the minimum similarity value of the i-th frame, and P _n represents the minimum similarity value of the current frame (n-th frame).

Further, in step S4, the control device 104 calculates the continuously increasing frame number N _n indicating how many frames the increase in the minimum similarity value continues continuously. Specifically, the control device 104 compares the minimum similarity value calculated in the previous frame with the minimum similarity value calculated in the current frame, and the minimum similarity of the current frame is more than the minimum similarity value of the previous frame. If the degree value is larger, 1 is added to the continuously rising frame number N _n . On the other hand, if the minimum similarity value of the previous frame is larger than the minimum similarity value of the current frame, the number of consecutively rising frames N _n is reset to zero. As a result, the continuously rising frame number N _n can be calculated.

  Thereafter, the process proceeds to step S5, and the control device 104 determines that the increase degree of the minimum similarity value calculated in step S4 is not less than a predetermined threshold, for example, not less than 1.5, and the number of continuously rising frames calculated in step S4 is predetermined. It is determined whether the number is 3 or more, for example. If a negative determination is made in step S5, the process proceeds to step S8 to be described later. If an affirmative determination is made in step S5, the process proceeds to step S6. In step S6, the control device 104 determines whether or not the variance value of the similarity map calculated in step S3 is equal to or less than a predetermined threshold, for example, equal to or less than the average value of the variance values of the similarity map in the past frame. If a negative determination is made in step S6, the process proceeds to step S8 described later. If an affirmative determination is made in step S6, the process proceeds to step S7.

  In step S7, as described above, the control device 104 updates the template image using the target image specified as the subject position in the current frame as a new template image. In other words, in the present embodiment, the control device 104 determines that the increase degree of the minimum similarity value is equal to or greater than the threshold value (the similarity degree of the current frame decreases at a certain rate or more with respect to the similarity value of the past frame). The number of consecutively rising frames is equal to or greater than a predetermined number (similarity decreases continuously for a predetermined frame), and the variance value of the similarity map is equal to or less than a threshold value (variation in similarity value of the similarity map) Is determined to be the update timing of the template image, and the template image is updated. Hereinafter, the reason for determining the timing as the update timing of the template image will be described.

  First, the relationship between the variance value of the similarity map and the increase degree of the minimum similarity value will be described with reference to FIG. FIG. 3 shows the time series change 3a of the minimum similarity value and the time series change 3b of the variance value of the similarity map when template matching is performed without updating the template image. First, paying attention to the time series change 3a of the minimum similarity value, when template matching is continued without updating the template image, the minimum similarity value between the template image and the target image is changed as the subject changes. It can be seen that it increases, that is, the similarity gradually decreases. For example, in FIG. 3, the minimum similarity value gradually increases from the start of template matching toward time point A.

  If the template matching is continued thereafter, the minimum similarity value further increases, and as a result of the correlation between the template image and the target image being lost at a certain point in time, the template image and the target image at other pseudo subject positions The similarity is minimized, and erroneous matching occurs. For example, in FIG. 3, as a result of calculating the minimum similarity value at the pseudo subject position different from the original subject position at the time point B, the minimum similarity value that has continued to rise is reduced again.

  Next, paying attention to the time series change 3b of the variance value of the similarity map, the variance value of the similarity value increases from time A to time B. This indicates that the variation in the similarity value of the similarity map is large, and that the reliability of the subject detected using the current template image is gradually decreasing. .

  As described above, it is necessary to update the template image at least at a time point before time point B at which erroneous matching occurs, that is, at a time point when the reliability of the subject detected using the current template image is maintained to some extent. I understand that. Here, if the template image is updated frequently with the update timing of the template image as the timing before the time point A, the template can be updated with a margin until the occurrence of erroneous matching. In this case, however, the template image is frequently updated. The template image is updated, and the processing speed is reduced.

  Therefore, in the present embodiment, in order to prevent erroneous matching and suppress the decrease in processing speed as much as possible, in order to set the time between time A and time B as the template image update timing, step S5 in FIG. The update timing is determined based on the conditions in step S6.

  Specifically, in step S5, a determination condition is set so that the template image is updated when the minimum similarity value continuously increases at a certain degree of increase, and the template image update timing is set. After time A. In step S6, determination conditions are set so that the template image is updated while the reliability of the subject detected using the current template image is maintained to some extent, and the update timing of the template image is set. It is set to be before time B.

  In the present embodiment, even if the condition of step S5 is satisfied, the template image is not updated if the condition of step S6 is not satisfied. This is because when the variance value of the similarity map is large, the reliability of the subject detected using the current template image is low, and therefore the target image at this time is updated as the template image. This is because there is a possibility that a subject different from the original may be a tracking target.

  After updating the template image in step S7, the process proceeds to step S8, and the control device 104 determines whether the processing for all the frames of the through image has been completed, that is, whether the input of the through image from the image sensor 103 has been completed. Determine whether. If a negative determination is made in step S8, the process returns to step S1, and the above processing is repeated for the next input frame. On the other hand, when an affirmative determination is made in step S8, the process ends.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 104 calculates a similarity value between the template image and the target image, and generates a similarity map based on the calculation result. In addition, the control device 104 detects the minimum similarity value, calculates the variance value of the similarity value, calculates the increase degree of the minimum similarity degree, and calculates the number of continuously rising frames. Then, the control device 104 determines whether or not the template image needs to be updated based on the variance value of the similarity value, the increase rate of the minimum similarity, the number of continuously rising frames, and the variance value of the similarity value. When it is determined that the update is necessary, the template image is updated using the target image for which the minimum similarity is calculated in the current frame. Accordingly, it is possible to determine the optimal update timing of the template image, and to continue subject tracking even when the subject changes.

(2) In the control device 104, the degree of increase in the minimum similarity value is equal to or greater than a predetermined threshold, the number of continuously rising frames is equal to or greater than a predetermined number, and the variance value of the similarity map is equal to or smaller than the predetermined threshold. In this case, it was determined that the template image needs to be updated. This makes it possible to update the template image so as to prevent erroneous matching and suppress the decrease in processing speed as much as possible.

-Modification-
The camera according to the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the case where one template image is used for template matching has been described. However, the present invention can also be applied to a case where template matching is performed using a plurality of template images.

(2) In the above-described embodiment, the control device 104 increases the minimum similarity value by a certain amount or more based on the increase degree (increase rate R _n ) of the minimum similarity value and the number of continuously rising frames N _n. An example has been described in which the time points of continuous ascents are specified. However, the control device 104 continues the minimum similarity value continuously with a certain degree of increase based on one of the increase degree of the minimum similarity value (increase rate R _n ) and the number of continuously rising frames N _n. You may make it identify the time of rising.

(3) In the above-described embodiment, the example has been described in which the control device 104 identifies the point in time when the reliability of the subject in the target image as the tracking target subject has decreased based on the variance value of the similarity map. However, the control device 104 may specify a point in time when the reliability of the subject in the target image as the tracking target subject decreases based on the standard deviation value that is the square root of the variance value of the similarity map.

(4) In the above-described embodiment, the description has been given of the example in which the control device 104 generates the similarity map in which the similarity value at each search position is two-dimensionally expressed. However, among the similarity values at each search position, search positions whose similarity values are far away from other search positions are considered to be affected by noise, and may be excluded from the similarity map. .

(5) In the embodiment described above, in step S6 of FIG. 2, the control device 104 determines whether or not the variance value of the similarity map in the current frame is equal to or less than the average value of the variance values of the similarity map in the past frame. An example of determining whether has been described. However, the threshold value used for comparison with the variance value may be a fixed value set in advance.

(6) In the above-described embodiment, the control device 104 has described an example in which subject tracking is performed on a through image. However, when the camera 100 has a moving image shooting function, the control device 104 may perform subject tracking between frames of a captured moving image instead of the through image.

(7) In the above-described embodiment, an example has been described in which the control device 104 provided by the camera 100 performs template matching processing to perform subject tracking. However, it is also possible to record a program for executing the template matching process in another terminal such as a personal computer and execute the process on those terminals. In this case, if moving image data captured by a camera is taken into the terminal side and processing is performed on the data, subject tracking can be performed between frames of the moving image. The present invention can also be applied to a camera-equipped mobile phone.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

100 Camera, 101 Operation member, 102 Lens, 103 Image sensor, 104 Control device, 105 Memory card slot, 106 Monitor

Claims (3)

Similarity calculation means for calculating a similarity value between the template image and the target image in the search frame set in the input image;
Calculating means for calculating a variance value or a standard deviation value of the similarity value in the input image;
A specifying means for specifying, as a specific similarity value, the similarity value in which the similarity between the template image and the target image satisfies a predetermined condition in the input image;
The specific similarity value is accumulated for each frame, and based on the accumulated specific similarity value, it is determined that the similarity between the template image and the target image continuously decreases for a predetermined frame, and A determination unit that determines that the template image needs to be updated when the variance value or the standard deviation value of the similarity value is equal to or less than a predetermined threshold ;
Wherein when the determination means determines that it is necessary to update the template image, the object tracking apparatus characterized by comprising updating means for updating the template images. Similarity calculation means for calculating a similarity value between the template image and the target image in the search frame set in the input image;
Calculating means for calculating a variance value or a standard deviation value of the similarity value in the input image;
A specifying means for specifying, as a specific similarity value, the similarity value in which the similarity between the template image and the target image satisfies a predetermined condition in the input image;
The specific similarity value is accumulated for each frame, and based on the accumulated specific similarity value, the similarity of the current frame is decreased by a certain ratio or more with respect to the average value of the similarity of a plurality of past frames. Determining means for determining that the template image needs to be updated when the variance value or the standard deviation value of the similarity value is equal to or less than a predetermined threshold value ;
Wherein when the determination means determines that it is necessary to update the template image, the object tracking apparatus characterized by comprising updating means for updating the template images. Imaging means for capturing a subject image and acquiring the image;
A camera comprising the subject tracking device according to claim 1.

Images