JP2011090488A - Object tracking device, object tracking program, and camera - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an object tracking device for tracking an object between frames, and for specifying the position of the object with high accuracy. <P>SOLUTION: A controller 104 specifies a transition destination of an object based on time-sequentially input inter-frame information by motion estimation, and calculates first similarity between each pixel in the frame and a template for tracking an object by template matching, calculates second similarity by weighting the first similarity of each pixel according to a distance from the transition destination of the object specified by motion estimation, and specifies the position of the object in the frame based on the second similarity. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a subject tracking device, a subject tracking program, and a camera.

  The following tracking devices are known. The tracking device tracks the subject by predicting the subject position of the next frame from the motion information of the subject in the past (for example, Patent Document 1).

JP 2007-42072 A

  However, in the conventional tracking device, when the correlation between adjacent frames is poor at a low frame rate, the prediction accuracy of the subject position is lowered, and the tracking performance may be lowered.

The subject tracking device according to the present invention performs transition matching with a transition destination specifying unit that specifies a transition destination of a subject based on information between frames input in time series, and performs pixel matching for each pixel in the frame and subject tracking Calculated by the first similarity calculating means according to the distance from the transition destination of the subject specified by the transition destination specifying means and the first similarity calculating means for calculating the first similarity with the template. A second similarity calculation unit that calculates the second similarity by weighting the first similarity of each pixel, and the second similarity calculated by the second similarity calculation unit And a subject position specifying means for specifying the subject position in the frame.
In the present invention, the transition destination specifying means may specify the transition destination of the subject by calculating a motion vector indicating the motion of the subject between frames.
You may make it further provide the template update part which updates the template for a motion search for calculating a motion vector using the image in the to-be-photographed object position vicinity area | region specified by the object position specifying means.
The transition destination specifying means generates a plurality of images with different resolutions based on the same frame, and performs a template matching process using a motion search template in order from the low resolution images to calculate a motion vector. May be.
The subject tracking program according to the present invention performs a transition destination identification procedure for identifying a transition destination of a subject on the basis of information between frames input in time series to a computer, performs template matching, and sets each pixel in the frame. In accordance with the first similarity calculation procedure for calculating the first similarity with the template for tracking the subject and the distance from the transition destination of the subject specified in the transition destination specification procedure, the first similarity calculation procedure A second similarity calculation procedure for calculating a second similarity by weighting the first similarity of each calculated pixel, and a second similarity calculated by the second similarity calculation procedure And a subject position specifying procedure for specifying a subject position in the frame based on the degree.
A camera according to the present invention includes any one of the subject tracking devices described above.

  According to the present invention, the subject position can be specified with high accuracy.

1 is a block diagram illustrating a configuration of an embodiment of a camera 100. FIG. It is a flowchart figure which shows the flow of a subject tracking process. It is a figure which shows the example of a detection of a motion vector. It is the figure which showed the method of the motion search process typically. It is the figure which showed the image from a motion search process to calculation of a 2nd similarity.

  FIG. 1 is a block diagram illustrating a configuration of an embodiment of a camera according to the present embodiment. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various input members operated by the user, such as a power button, a release button, a zoom button, a cross key, an enter button, a play button, and a delete button.

  The lens 102 is composed of a plurality of optical lenses, but is representatively represented by one lens in FIG. The image sensor 103 is an image sensor such as a CCD or a CMOS, for example, and captures a subject image formed by the lens 102. Then, an image signal obtained by imaging is output to the control device 104.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the camera 100. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the CPU to develop a program when the program is executed or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  The control device 104 generates image data in a predetermined image format, for example, JPEG format (hereinafter referred to as “main image data”) based on the image signal input from the image sensor 103. Further, the control device 104 generates display image data, for example, thumbnail image data, based on the generated main image data. The control device 104 generates an image file that includes the generated main image data and thumbnail image data, and further includes header information, and outputs the image file to the memory card slot 105.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is written and recorded on the memory card. The memory card slot 105 reads an image file stored in the memory card based on an instruction from the control device 104.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . Further, when the user sets the mode of the camera 100 to the shooting mode, the control device 104 outputs the display image data generated based on the image signal acquired in time series from the image sensor 103 to the monitor 106. As a result, a through image is displayed on the monitor 106.

  In the camera 100 according to the present embodiment, the control device 104 performs a template matching process using a template image prepared in advance for each frame (frame image) of a through image input from the image sensor 103. An image region similar to the template image is specified from within the frame. Then, the control device 104 performs subject tracking by tracking the specified region between frames.

  In the present embodiment, when performing template matching, the control device 104 uses two templates, a motion search template for searching for a motion between frames of a subject and a subject tracking template, to perform subject tracking. I do. The details of the subject tracking process in the present embodiment will be described below with reference to the flowchart shown in FIG. 2 is executed by the control device 104 as a program that is activated when input of an image signal from the image sensor 103 is started.

  In step S1, the control device 104 takes in a frame image input from the image sensor 103, and proceeds to step S2. In step S2, the control device 104 acquires a template for motion search and a template for subject tracking from the frame image captured in step S1 for an image within a predetermined range. Note that the motion search template and the subject tracking template are obtained by cutting out an image within the range specified by the user in the frame image captured in step S1. Further, the initial motion search template and the subject tracking template acquired in step S2 may be the same image or different images.

  Thereafter, the process proceeds to step S3, and the control device 104 performs a motion search process using the motion search template acquired in step S2 for the frame image captured in step S1. For example, the control device 104 specifies the transition destination of the subject based on information between frames input in time series. Specifically, motion search processing using a template for motion search is performed on a frame image by a method such as template matching (block matching) or gradient method.

  By this motion search process, a motion vector indicating the transition of the subject between frames is detected. For example, when the subject 2a in the frame image of the i-th frame shown in FIG. 3A moves to the position shown in the frame image of the i + 1-th frame shown in FIG. A motion vector 2b indicating the motion of the subject is detected.

In the motion search process according to the present embodiment, as shown in FIG. 4, the control device 104 performs the motion search process by detecting a motion vector through a hierarchical template matching process. Specifically, the control device 104 creates an image having a hierarchical structure as shown in FIG. 4 in arbitrary adjacent frames, for example, i frame and i + 1 frame. There are two or more hierarchies. In the example shown in FIG. 4, the i frame has hierarchies 4a to 4c and the i + 1 frame has hierarchies 4d to 4f. Further, the size ratio of one side of the Nth hierarchical image from the high resolution side (4c side, 4f side) is 1 / ^2N .

In this case, it is assumed that the template is extracted from the i frame. Specifically, in the i frame, the template is extracted at each layer as indicated by the dotted frame. Using each of these templates, template matching processing is performed in order from the lowest resolution image, that is, the image 4f in the i + 1 frame. Here, the coordinate obtained by doubling the coordinate value matched in the L-1 hierarchy is set as the search initial value in the L hierarchy. This can be expressed by the following equation (1), where Xend is the matching coordinate and Xini is the initial search coordinate.

  By performing such hierarchical template matching, even if the search area is reduced in each hierarchy, a large movement of the subject can be detected accurately and at high speed. Then, it progresses to step S4.

  In step S <b> 4, the control device 104 performs template matching for the frame image using the subject tracking template, and calculates the similarity between the pixel of the frame image and the subject tracking template. In the present embodiment, the similarity calculated for each pixel in step S4 is referred to as a first similarity. Note that a template matching method using a subject tracking template is well known and will not be described in detail. Generally, a method such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference) is used. Is used.

Thereafter, the process proceeds to step S5, where the control device 104 performs the second weighting by weighting the first similarity of each pixel according to the distance from the transition destination of the main subject obtained by the motion search process in step S3. Calculate similarity. Specifically, the control device 104 calculates the second similarity by the following equation (2).
2nd similarity = 1st similarity x weight based on distance information (2)

In Expression (2), weighting by distance information includes weighting using a Gaussian function represented by the following Expression (3) and a linear function represented by the following Expression (4).

  Here, an image from the motion search process in step S3 to the calculation of the second similarity in step S5 will be described with reference to FIG. As shown in FIG. 5A, the motion vector 5b of the subject 5a to be tracked is calculated by the motion search process in step S3. In the example shown in FIG. 5A, the subject position of the subject 5a in the i frame is within the frame indicated by the dotted line, whereas in the i + 1 frame, the motion vector is moved into the frame indicated by the solid line. 5b is calculated.

  FIG. 5B shows a weighted image when weighting the first similarity of each pixel according to the distance from the transition destination 5c of the main subject obtained by the motion search process in step S3 in step S5. FIG. The transition destination 5c of the main subject shown in FIG. 5B corresponds to the position of the subject 5a in the i + 1 frame shown in FIG. 5A, and the pixel closer to the center of the transition destination 5c has a higher weight. The weight is lowered as the distance from the center increases. That is, in the example shown in FIG. 5B, the darker the color, the higher the weight, and the lower the color, the lower the weight.

  FIG. 5C is a diagram showing an image of the first similarity calculated in step S4. In this example, in addition to the first similarity component 5e corresponding to the subject 5a to be tracked in FIG. 5A, the first similarity component 5f corresponding to the other subject 5c and the first similarity corresponding to the subject 5d. The degree component 5g is calculated.

  FIG. 5D shows an image of the second similarity calculated by weighting the first similarity component according to the distance from the transition destination of the main subject obtained by the motion search process in step S5. FIG. In the example shown in FIG. 5D, the first similarity component 5e corresponding to the tracking target subject 5a whose distance from the transition destination 5c of the main subject is close based on the weighting shown in FIG. 5B. As a result, a second similarity component 5h corresponding to the subject 5a is calculated.

  Thereafter, the process proceeds to step S6, and the control device 104 identifies and extracts the position where the second similarity value calculated in step S5 is minimum in the frame image, that is, the position where the second similarity is the highest. The control device 104 performs subject tracking between frames by specifying the extracted position as the subject position to be tracked in the frame image. Thereafter, the process proceeds to step S7.

  In step S7, the control device 104 acquires a new motion search template from the vicinity region of the position having the highest second similarity extracted in step S6 for the motion search process for the next frame image, and moves the motion. The search template is updated, and the process proceeds to step S8.

  In step S8, the control device 104 determines whether or not the highest second similarity specified in step S6, that is, the best similarity is greater than a predetermined threshold value. When an affirmative determination is made in step S8, the process proceeds to step S9, and the control device 104 newly starts from the neighboring region of the position having the highest second similarity extracted in step S6 for the template matching process for the next frame image. The subject tracking template is acquired, the subject tracking template is updated, and the process proceeds to step S10. On the other hand, if a negative determination is made in step S8, the process proceeds directly to step S10.

  As described above, in this embodiment, the motion search template is updated every frame, whereas the subject tracking template is not always updated every frame, and satisfies the condition of step S8. Update only when As a result, the motion search template can always be a template that reflects the latest shape of the subject, so that the motion vector can be calculated by capturing the latest shape of the subject. On the other hand, since the subject tracking template is updated only when the similarity is high, the template can be prevented from being updated in the case of incorrect matching, and the template matching can be continued with high accuracy.

  In step S10, the control device 104 determines whether or not to end the process. For example, the control device 104 determines to end the process when the input of the frame image from the image sensor 103 is stopped. If a negative determination is made in step S10, the process returns to step S3 and is repeated. On the other hand, if an affirmative determination is made in step S10, the process ends.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 104 performs a motion search process, specifies a subject transition destination based on information between frames input in time series, performs template matching, and tracks each pixel in the frame and subject tracking A first similarity with the template is calculated. Then, according to the distance from the transition destination of the subject specified by the motion search process, the first similarity of each pixel is weighted to calculate the second similarity, and the second similarity is calculated. Based on this, the subject position in the frame is specified. Thus, the subject position can be specified with high accuracy by specifying the subject position based on the second similarity obtained by weighting the first similarity according to the distance of the subject from the transition destination. it can.

(2) The control device 104 specifies the transition destination of the subject by calculating a motion vector indicating the motion of the subject between frames. Thereby, the transition destination of the subject can be specified with high accuracy.

(3) The control device 104 obtains a new motion search template from the vicinity region of the position having the highest second similarity, and updates the template for motion search. As a result, the motion search template can always be a template that reflects the latest shape of the subject, so that the motion vector can be calculated by capturing the latest shape of the subject.

(4) In the motion search process, the control device 104 performs the motion search process by detecting a motion vector by a hierarchical template matching process. Thereby, even if the search area is reduced in each hierarchy, a large movement of the subject can be detected accurately and at high speed.

-Modification-
The camera of the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the control device 104 has described an example in which subject tracking is performed on a through image. However, when the camera 100 has a moving image shooting function, the control device 104 may perform subject tracking between frames of a captured moving image instead of the through image.

(2) In the above-described embodiment, the example in which the control device 104 provided by the camera 100 performs template matching processing to perform subject tracking has been described. However, it is also possible to record a program for executing the template matching process in another terminal such as a personal computer and execute the process on those terminals. In this case, if moving image data captured by a camera is taken into a terminal functioning as a subject tracking device and processing is performed on the terminal, subject tracking can be performed between frames of a moving image. The present invention can also be applied to a camera-equipped mobile phone.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

100 Camera, 101 Operation member, 102 Lens, 103 Image sensor, 104 Control device, 105 Memory card slot, 106 Monitor

Claims (6)

A transition destination specifying means for specifying a transition destination of a subject based on information between frames input in time series;
First similarity calculation means for performing template matching to calculate a first similarity between each pixel in the frame and a template for subject tracking;
According to the distance from the transition destination of the subject specified by the transition destination specifying means, weighting is performed on the first similarity of each pixel calculated by the first similarity calculation means, A second similarity calculating means for calculating a second similarity;
A subject tracking device comprising: subject position specifying means for specifying a subject position in a frame based on the second similarity calculated by the second similarity calculating means. The subject tracking device according to claim 1,
The subject tracking device characterized in that the transition destination specifying means specifies a transition destination of the subject by calculating a motion vector indicating the motion of the subject between frames. The subject tracking device according to claim 2,
A subject tracking apparatus, further comprising a template update unit that updates a motion search template for calculating the motion vector using an image in a region near the subject position specified by the subject position specifying unit. The subject tracking device according to claim 3,
The transition destination specifying unit generates a plurality of images with different resolutions based on the same frame, and performs the template matching process using the motion search template in order from the low resolution images to calculate the motion vector. An object tracking device characterized by: On the computer,
Based on information between frames input in time series, a transition destination specifying procedure for specifying a transition destination of a subject,
A first similarity calculation procedure for performing a template matching to calculate a first similarity between each pixel in the frame and a subject tracking template;
The first similarity of each pixel calculated in the first similarity calculation procedure is weighted according to the distance from the transition destination of the subject specified in the transition destination specification procedure, and a second A second similarity calculation procedure for calculating the similarity of
A subject tracking program for executing a subject position specifying procedure for specifying a subject position in a frame based on the second similarity calculated in the second similarity calculation procedure.   A camera comprising the subject tracking device according to claim 1.

Images