JP5526588B2 - Subject tracking program and subject tracking device - Google Patents

Description

  The present invention relates to a subject tracking program and a subject tracking device.

  2. Description of the Related Art A subject tracking device is known in which template matching is performed on a plurality of consecutively photographed images using a preset template image to search for a subject position similar to the template image (for example, a patent) Reference 1). In this subject tracking method, a search area to be matched with the template image is defined in the image, and template matching is performed while sequentially shifting the area to be compared with the template image in the search area, and the area most similar to the template image is tracked. This is an area where the target subject exists.

Prior art documents related to the invention of this application include the following.
Japanese Patent Laid-Open No. 06-083963

In the conventional subject tracking device described above, if the search area is widened, the matching calculation time becomes long, and it becomes easy to cause false matching that misidentifies a subject different from the subject to be tracked. Therefore, the size of the search area is changed according to the similarity of the template matching results.
However, depending on the size of the search area, there is a problem that the calculation time for template matching becomes long.

The invention of claim 1, each of the plurality of images continuously captured, a search area for searching a subject similar to setting boss was a template image for comparison with the template image in the search area A region setting procedure for setting the comparison target region, and similarity calculation for calculating the similarity by comparing the image of the comparison target region with the template image while moving the comparison target region within the search region A subject tracking program for causing a computer to execute a procedure and a position specifying procedure for specifying the position of the comparison target area where the maximum value of the similarity is calculated in the image as a position where the subject to be tracked exists a is the area setting step, when representing the number of operations of the similarity in function of the size of the size and the search area of the comparison regions, wherein As the number of arithmetic operations similarity score is not a maximum, and sets the size of the search area to the size of the comparison area.

  According to the present invention, the size of the search area with respect to the size of the comparison target area has a predetermined relationship, and the number of times of the template matching calculation increases rapidly, thereby preventing the template matching calculation time from becoming long. it can.

The figure which shows the structure of one embodiment The figure which shows the relationship between the captured image I, the template image A, the comparison object area image B, and the search area S Figure with square search area and comparison area set The figure which shows the relation of the number of operations of template matching to the size of the comparison target area when the square search area and the comparison target area are set Diagram when the shapes of the search area and comparison area are generalized The figure which shows the relation of the number of operations of template matching to the size of the comparison area when the shape of the search area and the comparison area is generalized The figure which shows the relation of the number of operations of template matching to the size of the comparison area when the shape of the search area and the comparison area is generalized The flowchart which shows the subject tracking control program of one embodiment The figure which shows the subject tracking result by one Embodiment

  An embodiment in which the subject tracking program and the subject tracking device of the present invention are applied to a digital camera will be described. The subject tracking program and the subject tracking device of the present invention can be applied to, for example, a digital video camera, various imaging devices, and an image processing device other than a digital camera.

  FIG. 1 shows the configuration of an embodiment. The digital camera 1 includes a lens 2, an image sensor 3, a control device 4, an operation member 5, a monitor 6, and the like. The image sensor 3 is composed of a CCD, a CMOS or the like, and converts the subject image formed by the lens 2 into an image signal. The control device 4 includes a CPU 4a, a memory 4b, and the like. The control device 4 performs various kinds of arithmetic control and sequence control of the camera, and executes a subject tracking control program to be described later. Is identified. The operation member 5 includes a switch that is turned on when the shutter button is half-pressed, a switch that is turned on when the shutter button is fully pressed, and a direction key that designates a tracking target subject by positioning the cursor on the subject to be tracked on the image. Yes. The monitor 6 displays captured images and various types of shooting information.

  A search area setting method in template matching according to an embodiment will be described with reference to FIG. In FIG. 2, I is an image captured by the image sensor 3. A is a subject image to be tracked, that is, a template image. The template image A can be set by a method other than the method in which the photographer manually designates an arbitrary tracking target subject using the operation member 5 described above and the image of the designated portion is used as the template image A. There is a method in which the face area of a person is recognized, the camera automatically sets a tracking target subject, and the image of the set face area is used as a template image A.

  B is an image of a comparison target area on which the template matching operation is performed on the captured image I in comparison with the template image A. Template matching is performed while shifting the comparison target area on the captured image I by one pixel or by a predetermined pixel. Do. In this embodiment, the size of the comparison target region image B is the same as the size of the template image A, and both are p horizontal pixels and q vertical pixels.

  S is a search area for performing template matching with the template image A while shifting the comparison target area by a predetermined pixel. This search area S is an area narrower than the captured image I. The method for setting the size of the search area S will be described later in detail.

In template matching, the residual r between the template image A and the comparison target region image B shown in the equation (1) is calculated, and the smaller the residual r, the higher the similarity between both images.
r = ΣΣ | Apq−Bpq | (1)
In the equation (1), ΣΣ represents a sum operation in units of horizontal (1 to p) pixels and vertical (1 to q) pixels of the template image A and the comparison target region image B. That is, the sum of absolute values of differences between the template image A and the comparison target region image B for each pixel is defined as a residual r. In this embodiment, the reciprocal (1 / r) of the residual r is defined as the similarity, and the smaller the residual r, that is, the higher the similarity (1 / r), the more the comparison target region image B becomes the template. Judged to be similar to image A. Note that if the template image A and the comparison target image B completely match, that is, if A = B, the similarity (1 / r) = 1.

  Here, the size of the search area S will be considered. As described above, if the search area S is widened, the calculation time in template matching becomes long, and it becomes easy to cause pseudo matching that misidentifies a subject different from the subject to be tracked. Therefore, the size of the search region S is basically changed according to the similarity (1 / r) of the template matching result.

  When the similarity (1 / r) is low, there is a high possibility that the tracking target subject that matches the template image A is not near the current comparison target region. prevent. Conversely, if the similarity is high, there is a high possibility that the tracking target subject is at or near the current comparison target region, so that the search region S is narrowed to prevent pseudo matching.

Here, when the similarity (1 / r) is normalized from 0 (unmatched) to 1 (completely matched), the size of the search area S is expressed by the following equation.
(Size of search area S) = k · (1− (1 / r)) · (Size of template image A) (2)
In equation (2), (1 / r) is the similarity obtained from the residual r, and k is a constant set according to the purpose of photographing the image and the photographing conditions. In this embodiment, the size of the template image A is equal to the size of the comparison target region image B.

Next, the relationship between the size of the comparison target region (same as the size of the template image A in this embodiment), the size of the search region S, and the number of template matching operations will be considered. First, as shown in FIG. 3, the above relationship is derived when both the search area S and the comparison target area are square. The length of one side of the search area S is m (pixel), and the length of one side of the comparison target area is n (
Pixel). In this case, the number F of template matching operations is expressed by the following equation.
F = d (m−n + 1) ² n ² (3)
In equation (3), d is a pixel thinning rate. Further, (m−n + 1) ² represents the number of times the position of the comparison target region is shifted within the search region S, and n ² represents the number of loops for performing the template matching operation between the template image A and the search target region image B. Represent.

It is reasonable to fix the template size n to track the subject and to use the search area size m as a variable to change the search area size according to the similarity. However, looking at equation (3), under the constraint that m ≧ n, the number of operations F is a quadratic equation of m with the minimum value n ² times when the template size n and the search area size m are equal. Therefore, the value of F will diverge. In other words, it can be concluded that the minimum number of operations F is equal to the template size n and the search area size m, but if this is done, the search process becomes impossible.
Conversely, the search area size m is fixed, and the template size n is used as a variable to obtain a location where the number of operations is maximized as a quaternary expression. When the equation (3) is first-order differentiated with respect to n, the following equation (4) is obtained.
F (n) ′ = 2nd (2n ² −3 mn + m ² ) (4)
In the equation (4), the position where F (n) ′ = 0, that is, the extreme value is
n = 0, m / 2, m (5)
It is. The maximum value increases when n = m / 2, that is, when the length m (pixel) of one side of the search region S is close to twice the length n (pixel) of one side of the comparison target region. Therefore, it is necessary to avoid that the length m (pixel) of one side of the search region S is twice the length n (pixel) of one side of the comparison target region or a value close thereto.

  FIG. 4 shows the number of operations F (vertical axis of the graph) for the length n (pixel) of one side of the comparison target region (horizontal axis of the graph) when the length of one side of the search region S is 10 (pixels). It is a graph which shows the relationship. FIG. 4A shows a relationship when no pixel thinning is performed (d = 1), that is, when template matching calculation is performed using all pixels. When n = 5, i.e., n = m / 2, the number of operations F shows a maximum value. On the other hand, FIG. 4B shows the relationship when the pixel thinning rate d is 0.5, that is, one pixel is extracted for every two pixels and the template matching calculation is performed. Even in this case, the number of operations F is reduced, but the number of operations F is maximized when n = 5, that is, n = m / 2.

Next, a case where the shapes of the comparison target region and the search region S are generalized will be considered. Here, as shown in FIG. 5, the comparison target area is horizontal n (pixels) and vertical αn (pixels), and the search area S is horizontal m (pixels) and vertical βm (pixels). In this case, the number F of template matching operations is expressed by the following equation (6).
F (n) = dα (m−n + 1) (βm−αn + 1) n ² (6)
The position where F (n) ′ = 0 is obtained by first-order differentiation of equation (6) with respect to n, that is, the extreme value is
n = 0, {3 (α + β) ± (9α ² -14αβ + 9β ² ) ^1/2 } m / 8α (7)
It becomes. When the length n (pixel) of one side of the comparison target area and the length m (pixel) of one side of the search area S are in the relationship shown in the expression (7) except when n = 0, the calculation is performed. It shows that the number of times F becomes maximum.

  Incidentally, when α = β, that is, when the comparison target area and the search area S are similar, Expression (7) is equal to Expression (5). This is because if the comparison target area and the search area S are similar, the length of one side of the search area S is twice the length of one side of the comparison target area corresponding to one side of the search area S. It shows that the number of operations sometimes becomes maximal.

  FIG. 6 shows the number of calculations when the shapes of the comparison target region and the search region S are generalized. In this example, as shown in FIG. 7, the aspect ratio of the comparison target area is 1.2: 1, the aspect ratio of the search area S is 0.9: 1, and the length of one side of the search area S is 10 ( In the case of (pixel), the number of times of calculation F becomes maximum when the length n of the horizontal side of the comparison target region is n = 4.20 (pixels).

  In this way, when determining the size of the search area S, if it is determined so that the number of calculations F with respect to the size of the comparison target area does not become maximal, the subject position to be tracked can be calculated with a small number of calculations. it can.

  FIG. 8 is a flowchart illustrating a subject tracking control program according to an embodiment. The operation of the embodiment will be described with reference to this flowchart. The CPU 4a of the control device 4 operates the direction key of the operation member 5 to designate the subject to be tracked, and when the half-press switch is turned on by half-pressing the shutter button, the image of a predetermined size at the designated position is tracked. The template image of the target subject is stored in the memory 4b, and execution of the subject tracking control program is started. Here, the tracking target subject manually designated by the photographer will be described as an example, but the same applies to the case of the tracking target subject automatically set by the camera by face recognition or the like.

  In step 1, a search area S in the captured image I is set (see FIG. 2). Immediately after starting the subject tracking operation, a preset initial value is set as the size of the search area S. Here, the initial value of the search area S is set to a size obtained by multiplying the size of the template image A by a predetermined value. In the subsequent step 2, a comparison target region for performing a template matching operation is set in the search region S in comparison with the template image A (see FIG. 2). First, a comparison target area is set, for example, in the upper left corner of the search area S, and the comparison target area is sequentially shifted in the horizontal direction and the vertical direction in units of predetermined pixels every time the template matching calculation is performed.

  In step 3, the template matching calculation shown in the above equation (1) is performed between the template image A and the comparison target region image B to calculate the similarity (1 / r). In this embodiment, an example in which the residual r is calculated by the equation (1) to obtain the similarity (1 / r) is shown. However, the similarity between the template image A and the comparison target region image B is calculated as follows. The calculation method is not limited to the calculation method of the embodiment, and a known calculation method such as a calculation method using normalized correlation can be employed.

  In step 4, the comparison target area is moved in the search area S to check whether or not the subject tracking has been completed. In step 5, the comparison target region is shifted in the search region S, and the process returns to step 3 to perform a template matching operation between the new comparison target region image B and the template image A. On the other hand, when the subject tracking in the search area S is completed, the process proceeds to step 6, and the comparison target that obtains the maximum similarity among the similarities between the template image A and the comparison target area image B calculated so far. An area is specified as a subject position to be tracked.

  In step 7, it is determined whether or not the half-press switch of the operation member 5 is kept on, that is, whether or not the half-press operation of the shutter button is continued. If no push operation has been performed, the execution of the subject tracking control program is terminated.

  When the half-press operation of the shutter button is continued, the process returns to step 1 to reset the search area S. In this case, the size of the search area S is basically the size corresponding to the maximum similarity (1 / r) shown in the above equation (2), but the size of the search area S corresponding to the maximum similarity is shown. Search area in which the number of times of template matching with respect to the size of the comparison target area does not become a maximum when the relation is as shown in the above formula (5) or (7) with respect to the size of the comparison target area Change to the size of S.

  After resetting the size of the search region S based on the maximum similarity between the template image A and the comparison target region image B in the previous template matching and the number of template matching operations, the process proceeds to step 2 and the above-described processing is performed. repeat.

  FIG. 9 is a diagram illustrating a subject tracking result according to the embodiment. As a result of subject tracking using the image of the child's face shown in FIG. 9A as the template image A, as shown in FIG. Is close to the position at the time of initial template image A acquisition, the similarity between the image B in the comparison target region (indicated by a white solid line rectangular frame in the figure) and the template image A is high, so the size of the template image A A search area S (indicated by a white broken-line rectangular frame in the figure) that is slightly larger than that is set. Here, the size of the set search area S is such that the relationship between the size of the search area S and the comparison target area excludes the condition shown in the above formula (5) or (7). As a result, when the movement of the tracking target subject is small, the number of computations in template matching is reduced, that is, the computation time is reduced to improve subject tracking responsiveness, and pseudo matching is performed to a subject different from the tracking target subject. Can be prevented.

  On the other hand, as shown in FIG. 9 (c), when the child's face of the tracking target subject moves away from the initial position because the child suddenly lowered his / her posture, the comparison target region (white solid line in the figure) Since the similarity between the image B (shown by a rectangular frame) and the template image A is low, a search area S (shown by a white broken-line rectangular frame in the figure) wider than that in the case of FIG. 9B is set. Even in this case, the size of the set search region S is such that the relationship between the size of the search region S and the comparison target region excludes the condition shown in the above formula (5) or (7). Thereby, even when the movement of the tracking target subject is large, the tracking target subject can be reliably captured in the search region S.

  In the above-described embodiment, the embodiment in which the size of the template image A and the size of the comparison target region image B are made equal is shown. However, the sizes of both images may be different. For example, when the comparison target area image B is larger than the template image A, the pixels of the comparison target area image B are thinned out and extracted as many as the number of pixels of the template image A, or adjacent pixels in the comparison target area image B What is necessary is just to compare by averaging, reproducing | regenerating the image same as the pixel number of the template image A, etc. FIG.

  In the above-described embodiments and their modifications, all combinations of the embodiments or the embodiments and the modifications are possible.

  According to the above-described embodiment and its modifications, the following operational effects can be achieved. First, a search area for searching for a subject similar to a preset template image and a comparison target area for comparison with the template image are set in each of a plurality of images captured continuously. Then, while moving the comparison target region in the search region, the image of the comparison target region is compared with the template image to calculate the similarity, and the position of the comparison target region in which the maximum value of the similarity is calculated in the image Since the size of the search area is set so that the size of the search area does not have a predetermined relationship with the size of the comparison target area when specifying the position where the subject to be tracked exists The size of the search area with respect to the size of the target area has a predetermined relationship, the number of template matching operations increases rapidly, and the template matching operation time increases. It is possible to prevent the.

  Further, according to an embodiment, when the number of similarity calculations is expressed as a function of the size of the comparison target area and the size of the search area, there is no predetermined relationship that maximizes the number of similarity calculations. Since the size of the search area is set in this way, the relationship between the size of the comparison target area and the size of the search area can be easily derived, and the relationship between the number of template matching operations can be easily derived. An increase can be reliably prevented.

  Furthermore, according to an embodiment, as the maximum value of the similarity is lower, a larger search area is set for the next image. Therefore, when the movement of the tracking target subject is small, the template matching is performed. It is possible to prevent pseudo-matching with a subject different from the tracking target subject while reducing the number of computations, that is, reducing the computation time to improve subject tracking responsiveness. On the other hand, even if the movement of the tracking target subject is large, the tracking target subject can be reliably captured in the search area.

4; Control device, 4a; CPU

Claims (3)

In each of a plurality of images continuously captured, set a search area for searching a subject similar to setting boss was the template image and the comparison regions to be compared with the template image in the search area Region setting procedure to be performed,
A similarity calculation procedure for calculating the similarity by comparing the image of the comparison target area and the template image while moving the comparison target area in the search area;
A subject tracking program for causing a computer to execute a position specifying procedure for specifying a position of the comparison target region where the maximum value of the similarity is calculated in the image as a position where a subject to be tracked exists. ,
In the region setting procedure, when the number of similarity calculations is expressed as a function of the size of the comparison target region and the size of the search region, the comparison target is not maximized. An object tracking program, wherein the size of the search area is set with respect to the area size. The subject tracking program according to claim 1 ,
The subject tracking program is characterized in that the region setting procedure sets a larger search region for the next image as the maximum value of the similarity is lower. A subject tracking apparatus comprising execution means for executing the subject tracking program according to claim 1 .

Images