JP6685823B2 - Subject tracking device, control method thereof, imaging device, and program - Google Patents

Description

  The present invention relates to a technique of subject tracking processing used in an imaging device or the like.

  A technique of extracting a specific subject image from a moving image and tracking the subject is used for identifying a face area or a human body area of a subject person in the image. For example, it can be used in many fields such as teleconferencing, man-machine interface, security, monitor system for tracking arbitrary objects, image compression and so on.

  There is a technique of extracting and tracking an image of a subject in a captured image designated by a user, and optimizing a focus state and an exposure state of the subject (see Patent Document 1). In a general template matching technique, a template image is registered with reference to an area arbitrarily designated in the image by a user using an input interface such as a touch panel. A process of estimating a region having the highest similarity or the lowest difference with the template image in the image and tracking a specific subject is performed. In the method of using the similarity of pixel patterns as an evaluation measure, if the pixel patterns of the partial areas of the tracking target and other subjects (background, etc.) are similar, there is a possibility that tracking will be performed on the wrong subject. . Further, in the method of using the similarity of the color histogram as an evaluation measure, if the color ratios of the partial areas of the tracking target and other subjects are similar, there is a possibility that tracking may be performed on the wrong subject. .

  Techniques have been proposed that use distance information in order to accurately track an object. In Patent Document 2, by using distance information of a tracking target, a region of another subject (shielded subject) existing between the imaging device and the subject of the tracking target is detected so as not to focus on the shielded subject. The technology is disclosed.

JP 2001-60269 A JP, 2014-202875, A JP 2002-251380 A

  In the conventional technique, there are shooting scenes in which it is difficult to track an object, and there are cases where the accuracy of object tracking cannot be sufficiently obtained. For example, assume a scene in which a subject (hereinafter, referred to as a similar subject) having a pixel pattern, a color histogram, and the like that are similar to a subject to be tracked (hereinafter, referred to as a main subject) exists. When the difference in distance between the main subject and the similar subject in the optical axis direction (shooting direction) of the imaging device is small, it is difficult to differentiate the main subject and the similar subject based on the distance information. Further, it is assumed that the main subject moves in the optical axis direction of the image pickup device while the main subject is shielded by similar subjects and invisible when viewed from the image pickup device. There is a possibility that it may be difficult to detect the transfer of the tracking target or to track the main subject that reappears.

  It is an object of the present invention to improve the accuracy of subject tracking by using distance information related to a captured image and detection information of a position change in the depth direction of a subject.

An apparatus according to an embodiment of the present invention is an object tracking apparatus that acquires image data and distance information related to the image data to perform tracking processing of an object in an image, and acquires the image data and distance information. Acquiring means, first detecting means for detecting an image of a subject from the image data and outputting information of a plurality of first candidate regions related to the subject region, the distance information and the depth of the subject in the image. A second detection unit that outputs information on a second candidate region related to the subject region based on a change in position in the direction, and information on the first and second candidate regions are acquired, and the plurality of information items are acquired by the second candidate region. Determining unit that determines a region narrowed down from the first candidate region as a subject region used for subject tracking. The first detection unit obtains the image data and detects a subject, and obtains the image data and detection information of the subject detection unit to perform matching processing to obtain the first candidate region. And a matching unit that outputs a plurality of evaluation values and area information as information. The second detection unit obtains the distance information and calculates a position change vector indicating a position change of the subject in the image in the depth direction, and the distance information regarding the plurality of detected subject regions. And a comparison means for comparing the position change vectors and outputting the information of the second candidate area.

  According to the present invention, the accuracy of subject tracking can be improved by using the distance information related to the captured image and the detection information of the position change of the subject in the depth direction.

It is an external view which illustrates the imaging device which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the imaging device which concerns on embodiment of this invention. It is a schematic diagram which shows the relationship between the pupil of an imaging optical system, and the photoelectric conversion part of an image sensor. 3 is an overall flowchart including subject tracking processing according to the first embodiment of the present invention. FIG. 3 is a block diagram showing a subject tracking processing unit in the first embodiment of the present invention. It is a figure which shows a matching process typically. It is a figure which shows the time change of the image surface conversion value of a subject's image surface position and a focus lens position. It is a figure which shows a to-be-photographed area | region and a focus detection area. It is a flowchart which shows a subject tracking process. It is a flow chart which shows the detection processing of the 2nd subject candidate field. It is a figure which shows subject tracking when two subjects exist in the same distance. It is a figure which shows the time change of the defocus amount and optical axis vector amount of two subjects. It is a figure which shows subject tracking when two subjects pass each other. It is a figure which shows the time change of the defocus amount and optical axis vector amount of two subjects. It is a block diagram which shows the subject tracking process part in 2nd Embodiment of this invention. It is a figure which shows the focus detection area | region in 2nd Embodiment of this invention.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The image pickup apparatus of each embodiment has a configuration capable of focus detection by an image pickup surface phase difference detection method, and has distance information related to image data and optical axis vector information (information of an optical axis component of a position change vector) described later. Is used to calculate the subject tracking. The configurations described in the following embodiments are merely examples, and the present invention is not limited to the configurations described in the embodiments.

[First Embodiment]
An image pickup apparatus according to this embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a perspective view showing the outer appearance of the image pickup apparatus 101. The optical axis direction of the imaging device 101 is a depth direction 103 that is a normal line direction to the imaging surface 102 in the imaging device 101. FIG. 2 is a block diagram showing a configuration example of the image pickup apparatus 200. The imaging device 200 has a function of recording image data (moving image or still image data) of a subject imaged by an imaging element via an imaging optical system on a recording medium. The recording medium is various media such as a tape, a solid-state memory, an optical disk and a magnetic disk. The imaging device 200 is a digital still camera, a video camera, or the like, but is not limited to these.

  The taking lens 201 is a lens unit including a fixed lens 202, a diaphragm 203, and a focus lens 204. The diaphragm control unit 211 drives the diaphragm 203 to adjust the aperture diameter of the diaphragm 203 to adjust the light amount at the time of shooting. The focus control unit 212 determines the drive amount of the focus lens 204 based on the defocus amount of the photographing lens 201. The focus adjustment state is controlled by moving the focus lens 204 which is the focus adjustment lens. The lens control unit 213 controls the movement of the focus lens 204 via the focus control unit 212, and automatic focus adjustment control is realized. Although the focus lens 204 is simply shown as a single lens in FIG. 2, it is usually composed of a plurality of lenses. The aperture controller 211 and the focus controller 212 are controlled by the lens controller 213. As will be described in detail later, the focus lens 204 is drive-controlled so that the main subject is always focused.

  The field light that has entered through the optical member that constitutes the taking lens 201 forms an image on the light receiving surface of the image sensor 221. The image sensor 221 is a photoelectric conversion element that photoelectrically converts a subject image (optical image) into signal charges, and is configured by a CCD (charge coupled device) image sensor or a CMOS (complementary metal oxide semiconductor) image sensor. The signal charge accumulated in each photoelectric conversion unit of the image sensor 221 is sequentially read as a voltage signal corresponding to the signal charge by the drive pulse output from the timing generator 222.

  The image sensor 221 includes a plurality of photoelectric conversion units that share one microlens, and generates a plurality of parallax images to enable focus detection by an imaging plane phase difference detection method. The plurality of photoelectric conversion units are, for example, a first photoelectric conversion unit and a second photoelectric conversion unit, and can acquire a pair of parallax image data. The image sensor 221 will be described later with reference to FIG.

  The imaging signal processing unit 223 acquires and processes the output signal of the imaging device 221, and stores the image signal in an SDRAM (Synchronous Dynamic Random Access Memory) 261 via the bus 231. The focus detection signal processing unit 224 acquires and processes the focus detection signal output from the image sensor 221. The focus detection method of the imaging plane phase difference detection method performed by the focus detection signal processing unit 224 will be described later.

  The image signal stored in the SDRAM 261 is read by the display control unit 241 via the bus 231, and the display unit 242 displays an image according to the image signal. In addition, in the operation mode in which the image signal is recorded, the recording medium control unit 251 performs control to read the image signal from the SDRAM 261 and record it on the recording medium 252.

  The camera control unit 225 includes a main CPU (central processing unit) and controls the respective units of the camera system. A ROM (Read Only Memory) 262 stores a control program executed by the camera control unit 225 and various data necessary for control. The flash ROM 263 stores various setting information regarding camera operation such as user setting information.

  The camera control unit 225 converts the defocus amount output from the focus detection signal processing unit 224 into a focus lens drive amount and transmits the focus lens drive amount to the lens control unit 213. The lens control unit 213 includes a sub CPU and instructs the focus control unit 212 to control the movement of the focus lens 204. Further, the camera control unit 225 receives an operation instruction given by the user using the operation unit and controls the operation of each unit. For example, the camera control unit 225 sets the accumulation time of the image sensor 221, the set value of the gain when the image sensor 221 outputs to the image signal processing unit 223, and the setting of the timing generator 222 based on the size of the pixel signal. Determine the value. The pixel signal is a pixel signal related to the image data temporarily stored in the SDRAM 261.

  The subject tracking unit 271 executes a tracking process of the main subject. The subject tracking unit 271 performs calculation using distance information and an optical axis component of a position change vector (hereinafter referred to as an optical axis vector) that can be calculated from a parallax image in subject tracking. In this embodiment, the distance information is used to improve the tracking accuracy, but the focus detection method of the imaging plane phase difference detection method is used to improve the accuracy. That is, since the subject tracking unit 271 can output the tracking result of the subject for each frame, the process of acquiring the parallax image from the same imaging plane is performed so that the distance information for each frame can be calculated similarly. . The tracking result of the subject tracking unit 271 is transmitted to each module via the bus 231. The tracking result of the subject tracking unit 271 is transmitted to the focus control unit 212 via the camera control unit 225 and the lens control unit 213, and automatic focus detection and focus adjustment control for the main subject region in the captured image are realized. The main subject area is an image area of the main subject to be tracked. Further, the aperture control unit 211 performs exposure control using the brightness value of a specific subject area. Regarding the display of the main subject tracked by the subject tracking unit 271, the display unit 242 displays the image region of the main subject in a rectangular frame or the like on the screen. Note that the captured image data stored in the SDRAM 261 is sent to the subject tracking unit 271 via the bus 231. The subject tracking unit 271 tracks the main subject using a plurality of images at different shooting times, and extracts a partial area indicating the main subject as a tracking result.

  Next, the relationship between the pupil of the image pickup optical system and the photoelectric conversion unit of the image pickup device will be described with reference to FIG. The image sensor illustrated in FIG. 3 includes m × n sensor units arranged two-dimensionally. A cross section 301 in FIG. 3 shows a part of the image sensor 221. A microlens 303 and two photoelectric conversion units (304, 305) are arranged in each sensor unit 302. It is possible to obtain each output signal of the first photoelectric conversion unit 304 and the second photoelectric conversion unit 305 and generate an image signal used for automatic focus adjustment by the imaging plane phase difference detection method.

  FIG. 3 shows the correspondence between different regions (307, 308) of the pupil 306 of the imaging optical system and the photoelectric conversion units. The first pupil partial region 307 and the second pupil partial region 308 are located with the optical axis sandwiched therebetween. The light flux passing through each region is received by each of the two photoelectric conversion units (304, 305) via the microlens 303 arranged in each sensor unit 302 about the axis in the depth direction 103, that is, the optical axis. . The light flux that has passed through the first pupil partial region 307 is received by the first photoelectric conversion unit 304 via the microlens 303. The light flux that has passed through the second pupil partial region 308 is received by the second photoelectric conversion unit 305 via the microlens 303. An image pickup signal and a focus detection signal can be acquired by the two photoelectric conversion units provided in each sensor unit. That is, the data of the captured image is acquired by adding the outputs of the two photoelectric conversion units (304, 305). The image pickup signal processing unit 223 of FIG. 2 prepares the image pickup signal as an image signal (image data). Further, by handling the outputs of the two photoelectric conversion units (304, 305) respectively, two images (parallax images) with different viewpoints can be acquired. The first parallax image is obtained from the output of the first photoelectric conversion unit 304, and the second parallax image is obtained from the output of the second photoelectric conversion unit 305. The focus detection signal processing unit 224 in FIG. 2 performs focus detection calculation using the signals of the pair of parallax images.

  In the present embodiment, an image of an image signal acquired by adding the outputs of the two photoelectric conversion units is called an A + B image. The images of the image signals respectively acquired from the outputs of the two photoelectric conversion units are called A image and B image. The method of generating the phase difference signal is not limited to the method of this embodiment, and other methods may be used. For example, the signal of the A image or the B image obtained by the output of one photoelectric conversion unit is subtracted from the signal of the A + B image obtained by the addition output of the two photoelectric conversion units, and the signal of the B image or the A image of the other is subtracted. A signal can be generated.

  Here, the focus detection method of the imaging plane phase difference detection method performed by the focus detection signal processing unit 224 will be described. In the imaging plane phase difference detection, a pair of image signals for focus detection, for example, an A image signal and a B image signal, are acquired from the image sensor 221 for the set focus detection area. Next, the focus detection signal processing unit 224 calculates the correlation amount between the acquired image signals. The calculation for obtaining the correlation amount is performed for each scanning line in the focus detection area. The focus detection signal processing unit 224 calculates a correlation change amount from the correlation amount, and calculates a shift amount between the two images based on the correlation change amount. A defocus amount can be converted by multiplying the shift amount of these two images by a predetermined conversion coefficient. At this time, the camera control unit 225 acquires shooting parameter information used for automatic focus detection. The shooting parameters are stored in the memory of the camera body or the lens. The photographing parameters are information such as diaphragm information of the diaphragm 203 in the photographing lens 201 and sensor gain of the image sensor 221 in the camera body. Regardless of the configuration of the present embodiment, necessary information may be appropriately acquired according to the configuration of the camera. The camera control unit 225 provides necessary information so that the process related to the signal generation for focus detection and the region for focus detection can be set based on the shooting parameter. The focus detection area in this embodiment is set by equally dividing the entire screen in two dimensions.

  Next, the operation of the image pickup apparatus will be described with reference to FIGS. 4 to 14. First, the flow of the entire processing including the subject tracking processing will be described with reference to FIG. The following processing is realized according to a program executed by the CPU of the camera control unit 225.

  The image pickup device performs exposure, and the image pickup element 221 converts the light image into an electric signal (S401). The processes S402 and S403 and the processes S404 and S405 are executed as parallel processes. The imaging signal processing unit 223 converts the signal of the captured image (A + B image) read from the image sensor 221 into an image signal (S402). The image signal of the acquired A + B image is stored in the SDRAM 261 (S403). On the other hand, each signal of the parallax image (A image, B image) acquired by the image sensor 221 is input to the focus detection signal processing unit 224 (S404). The focus detection signal processing unit 224 detects the shift amount between the A image and the B image and calculates the defocus amount (S405).

  After the processes S403 and S405, the process proceeds to the determination process S406. The camera control unit 225 determines the mode. If the user has selected the tracking mode (YES in S406), the process proceeds to step S407, and if the tracking mode has not been selected (NO in S406), the process proceeds to determination process S408. The tracking mode is a mode in which the imaging device performs the tracking process of the main subject. The subject tracking unit 271 performs subject tracking processing based on the captured image and the defocus information (S407). In the determination processing S408, the camera control unit 225 determines ON / OFF of the power source. When the user turns off the power, the operation of the system ends (YES in S408). If the power-off operation is not performed (NO in S408), the process returns to step S401 and the exposure is performed.

  Subsequently, the subject tracking process will be described with reference to FIG. FIG. 5 is a block diagram illustrating subject tracking according to this embodiment. The subject tracking unit 271 includes a first processing unit that processes a captured image (A + B image) and a second processing unit that processes defocus information. The first processing unit includes a subject detection unit 501, a matching unit 502, and a subject region determination unit 503. The subject detection unit 501 and the matching unit 502 extract the first subject candidate region based on the captured images (A + B images) sequentially supplied from the SDRAM 261. The second processing unit also includes an optical axis vector calculation unit 504 and a distance / vector comparison unit 505. The second processing unit uses the defocus information calculated by the focus detection signal processing unit 224 based on the parallax image (A image, B image) to improve the second subject candidate region in order to improve the subject tracking performance. To extract. The subject area determination unit 503 tracks the main subject based on the first and second subject candidate areas. As a result of the tracking, a partial area indicating the specific subject in the image is extracted.

  In the present embodiment, the subject tracking process using the defocus information calculated by the focus detection signal processing unit 224 will be described as an example, but the present invention is applicable to various embodiments as information corresponding to the depth of the subject in the image. It can be applied in. That is, the information (depth information) indicated by the data corresponding to the depth of the subject directly represents the subject distance from the image pickup device to the subject in the image, or the subject distance and depth of the subject in the image. Any information that represents a relative relationship may be used. The details of each processing unit will be described below.

  The subject detection unit 501 in FIG. 5 acquires a captured image (A + B image), detects a target subject, and sets the target as a tracking target for subject tracking. The data of the captured image is output from the captured signal processing unit 223 and stored in the SDRAM 261. The subject detection unit 501 performs face detection, for example, and identifies a human face area as a subject area in the image. A known face detection method is used as the detection method. There are a method of using knowledge about the face (skin color information, parts of eyes, nose, mouth, etc.) and a method of configuring a discriminator for face detection by a learning algorithm represented by a neural network. In face detection, face recognition is generally performed by combining these in order to improve the recognition rate. Specifically, there is a method of detecting a face using the wavelet transform and the image feature amount described in Patent Document 3. The present invention is not limited to this, and the user (operator) may use an input interface unit including a touch panel, buttons, and the like to specify an arbitrary subject image included in the image as a tracking target. In that case, the subject detection unit 501 detects the subject region in the captured image based on the position information designated by the user's operation, and outputs the detection information to the matching unit 502. The subject region specified by the detection is set as the image region of the main subject.

  The matching unit 502 acquires the detection information of the subject detection unit 501 and registers the detected subject region as a template. The matching unit 502 performs matching processing between the template and the partial areas of the image sequentially supplied from the SDRAM 261, and outputs a plurality of evaluation values and area information as information on the first subject candidate area. Although there are various types of matching methods, in the present embodiment, the template matching method based on the degree of difference between pixel patterns is applied. Details of the template matching method will be described with reference to FIG.

FIG. 6A shows an example of a subject model (template) in template matching. The matching unit 502 treats the pixel pattern of the image 601 showing the target area of the main subject as a feature amount. FIG. 6A shows an example in which the characteristic amount 602 of the image 601 is expressed in a matrix, and the luminance signal of the pixel data is used as the characteristic amount. The feature amount is expressed as T (i, j), the coordinates in the template region are expressed as (i, j), the number of horizontal pixels is W, and the number of vertical pixels is H. The feature amount T (i, j) is expressed by the following equation.

FIG. 6B exemplifies image information when searching for a main subject, and shows an image 603 in a range in which matching processing is performed. The coordinates in the search image are represented by (x, y). A partial area 604 for obtaining the matching evaluation value is shown by a rectangular frame. As for the feature amount 605 of the partial area 604, the brightness signal of the image data is used as the feature amount as in the template of FIG. The feature amount is expressed as S (i, j), the coordinates in the partial region 604 are expressed as (i, j), the number of horizontal pixels is W, and the number of vertical pixels is H. The feature amount S (i, j) is expressed by the following equation.

As a calculation method for evaluating the similarity between the template area and the partial area 604, there is a method of using a difference absolute sum, that is, a so-called SAD (Sum of Absolute Difference) value. The SAD value (denoted as V (x, y)) is calculated by the following formula.

  The matching unit 502 calculates the SAD value V (x, y) while shifting the partial area 604 by one pixel in order from the upper left of the image 603 in the search range. The coordinates (x, y) at which the calculated SAD value V (x, y) exhibits the minimum value indicate the position most similar to the template. That is, the position where the SAD value V (x, y) exhibits the minimum value is the position where the main subject is likely to exist in the search image.

  Although the example in which the one-dimensional information of the luminance signal is used as the feature amount has been described, three-dimensional information such as a signal of lightness, hue, and saturation may be treated as the feature amount. Further, although the SAD value has been described as the calculation method of the matching evaluation value, a different calculation method such as normalized cross-correlation (NCC) may be used. Application of the present invention is not limited to template matching, and other matching methods such as histogram matching based on histogram similarity may be used.

  The subject region determination unit 503 in FIG. 5 acquires a plurality of evaluation values and region information from the matching unit 502, and determines the region having the smallest evaluation value from the candidate regions of the main subject as the subject region. However, the evaluation value of the similar subject area may be small in a scene in which a similar subject having a pixel pattern or a color histogram similar to that of the main subject is near the main subject. As a countermeasure, there is a method of distinguishing the main subject and the similar subject by using the defocus amount output from the focus detection signal processing unit 224. However, with this method, the main subject and the similar subject are close to each other in the optical axis direction, or the main subject passes the similar subject and moves in the optical axis direction while temporarily disappearing from the imaging surface. Therefore, it is difficult to distinguish the main subject and the similar subject. That is, there is a limit to the discrimination between the main subject and the similar subject only with the defocus information. Therefore, in the present embodiment, the tracking performance can be further improved by referring to the optical axis vector of the subject in addition to the defocus information.

  The optical axis vector calculation unit 504 will be described with reference to FIG. 7. The optical axis vector refers to the optical axis component of the position change vector of the subject image existing in each focus detection area on the imaging surface. FIG. 7 exemplifies a temporal change of a value obtained by converting an image plane position and a focus lens position where an object is in focus in an arbitrary focus detection area into an image plane value. The horizontal axis represents the time axis, and the vertical axis represents the image plane value. At each of the times t1 and t2, the focus lens position is represented by a black circle symbol, and the position at which the subject is in focus is represented by a black rectangle symbol.

なお、フォーカスレンズ２０４の駆動量については、当該駆動量をデフォーカス量に変換する換算係数と、フォーカスレンズ２０４の駆動量１パルスあたりの繰り出し量を乗算することで像面値Lに変換することができる。フォーカスレンズ駆動量からデフォーカス量への換算係数や１パルスあたりの繰り出し量の情報はＲＯＭ２６２に格納されているので、カメラ制御部２２５は必要に応じてバス２３１を介して取得できる。また、焦点検出用信号処理部２２４が算出したデフォーカス量はフォーカスレンズ２０４の駆動量へ変換することができる。 An arbitrary focus detection area is a c-th focus detection area in the x direction and an r-th focus detection area in the y direction, and its position is represented by (c, r). The past (time t1) defocus amount in an arbitrary focus detection area is expressed as Dpre (c, r), and the current (time t2) defocus amount in the focus detection area is expressed as Dcur (c, r). write. The optical axis vector is represented by v (c, r), and the value obtained by converting the drive amount of the focus lens 204 into the image plane value is represented by L. From FIG. 7, the optical axis vector v (c, r) is calculated by the following formula.

It should be noted that the drive amount of the focus lens 204 is converted into the image plane value L by multiplying the conversion coefficient for converting the drive amount into the defocus amount by the extension amount per pulse of the drive amount of the focus lens 204. You can Since the conversion coefficient from the focus lens drive amount to the defocus amount and the information about the amount of extension per pulse are stored in the ROM 262, the camera control unit 225 can obtain it via the bus 231 as necessary. The defocus amount calculated by the focus detection signal processing unit 224 can be converted into the drive amount of the focus lens 204.

  A method of determining the drive amount of the focus lens 204 will be described with reference to FIG. FIG. 8 shows an example of an image for explaining the tracking of the subject 801. The subject area 802 in the image is the subject area determined by the subject area determination unit 503 one frame before, and the center of gravity thereof is indicated by a point 803. The rectangular area is a focus detection area 804 including the center of gravity 803. The camera control unit 225 determines the drive amount of the focus lens 204 when tracking the subject 801 by reflecting the defocus amount calculated in the focus detection area 804 including the center of gravity 803 of the subject area 802. At this time, since the defocus amount for the main subject continues to be stored in the SDRAM 261, the next subject position can be predicted by calculating the approximate curve based on the time tendency of the defocus amount for a certain period of time. The drive amount of the focus lens 204 can be determined based on the predicted position.

  By the way, in the moving image shooting mode and the like, the number of parallax image samples for focus detection per unit time is large. With respect to such a mode, since the absolute amount of the optical axis vector is small, there is a possibility of being affected by noise due to focus detection error. Therefore, in the present embodiment, the optical axis vector calculated by the optical axis vector calculation unit 504 is stored in the SDRAM 261 and is updated as a moving average value over several frames. This makes it possible to calculate an optical axis vector with reduced noise. Note that the noise reduction method is not limited to the moving average method, and there is a method of performing low-pass filter processing on the defocus amount stored in the SDRAM 261 during several frames. As described above, various methods of smoothing the change in the defocus amount and then calculating the optical axis vector may be used.

The distance / vector comparison unit 505 in FIG. 5 determines whether the defocus amount and the optical axis vector of the subject stored in the SDRAM 261 and the defocus amount and the optical axis vector in each focus detection area match. . Hereinafter, the threshold value for determination will be described. The defocus amount and the threshold value of the optical axis vector are determined based on the plurality of evaluation values output from the matching unit 502 and the variance value of the center point coordinates of each area, and are dynamically set according to the tracking reliability. It Specifically, of the N evaluation values output from the matching unit 502, the i-th evaluation value corresponding to the coordinate is written as Vi, and the minimum evaluation value of the N evaluation values is Vmin. write. Based on the center coordinates (denoted as Ci) of the target area, the evaluation value barycentric coordinates (denoted as G) are calculated from the following equation.

The evaluation value Vi is normalized by dividing by the minimum evaluation value Vmin. Further, as described above, the evaluation value Vi means the difference from the template, and thus the smaller the value, the higher the similarity to the template. Therefore, in order to calculate the evaluation value barycentric coordinate G weighted with the evaluation value Vi, the central coordinate Ci is divided by the normalized evaluation value Vi / Vmin, and the sum is divided by N to average. Done. As shown in the following formula, the Euclidean distance between the evaluation value barycentric coordinate G and the center coordinate Ci of each area is weighted by each evaluation value Vi, and a calculation is performed to obtain a total value, which is a tracking reliability evaluation value (denoted as R). ) Is calculated.

  Since the tracking reliability evaluation value R is a variance value, the smaller the value, the smaller the variation in the coordinates of the first subject candidate area and the higher the tracking reliability. Therefore, when the tracking reliability evaluation value R is small, regions having high evaluation values are densely arranged, and it is unlikely that the similar subject is mistaken for the main subject. Therefore, the distance / vector comparison unit 505 changes the threshold for determination, widens the allowable range for determining the degree of coincidence between the defocus amount and the optical axis vector, and reduces the influence of the defocus amount and the optical axis vector. By doing so, the influence of erroneous distance measurement can be reduced. On the other hand, when the tracking reliability evaluation value R is large, the areas having a high evaluation value are scattered, and the possibility of erroneous tracking of a similar subject increases. Therefore, the distance / vector comparison unit 505 changes the threshold for determination, narrows the allowable range for determining the degree of coincidence between the defocus amount and the optical axis vector amount, and increases the influence of the defocus amount and the optical axis vector. . Although the method of weighting the evaluation value centroid coordinate G and the tracking reliability evaluation value R with the evaluation value Vi has been described, the present invention is not limited to this method. A method without weighting or another method may be used.

  Next, the subject tracking processing performed by the subject tracking unit 271 of FIG. 5 will be described in detail with reference to the flowcharts of FIGS. 9 and 10. First, the outline of subject tracking will be described with reference to FIG. In determination processing S901, the subject tracking unit 271 determines whether or not there is a template image. Since the template image does not exist in the initial frame from the time when the user selects the tracking mode, in this case (NO in S901), the process proceeds to step S902. If the template image already exists, the parallel processing from step S906 onward and step S909 onward is executed.

  When it is determined that the template image needs to be registered, the data of the captured image (A + B image) of the SDRAM 261 is input to the subject tracking unit 271 (S902). The subject detection unit 501 performs subject detection from the feature amount on the image based on the input captured image (S903). The detected subject area is registered as a template (S904). In the subject tracking process in the initial frame, only template registration is performed. Proceeding to the determination processing S905, the termination determination of the tracking mode is performed. Here, when the tracking mode ends (YES in S905), the subject tracking mode ends. If the end of the tracking mode is not determined, the process returns to the determination process S901.

  Since the template image exists in the second and subsequent frames after the subject tracking is started, the process proceeds to step S906 and step S909. The matching unit 502 performs matching processing on the captured image (A + B image) input in processing S906 based on the template (S907), and detects the first subject candidate area (S908).

  On the other hand, in step S909, defocus information is input from the focus detection signal processing unit 224 to the subject tracking unit 271. The optical axis vector calculation unit 504 calculates the optical axis vector, and the distance / vector comparison unit 505 detects the second subject candidate area (S910). Details of the method of detecting the second subject candidate area will be described later.

  After steps S908 and S910, the process proceeds to step S911, and the first subject candidate area is narrowed down from the second subject candidate area. In order to add distance information to the object tracking, of the center points of the first object candidate area (output of S908) based on template matching, the second object candidate area (of S910) detected by the distance / vector comparison unit 505 is detected. Points existing in (output) are extracted. The smaller the evaluation value of the first subject candidate area, the higher the degree of similarity with the template. Therefore, in step S912, the subject area determination unit 503 determines the subject candidate area having the smallest evaluation value among the first subject candidate areas remaining after the narrowing down as the subject area.

  By the way, the subject area determination unit 503 cannot always determine the subject area. For example, if the shape or color distribution of the subject changes significantly and the evaluation value output from the matching unit 502 is smaller than a predetermined value, the subject area cannot be determined. Further, in the processing S911, if there is no matching area between the first subject candidate area and the second subject candidate area and the first subject candidate area is lost, the subject area cannot be determined. In this way, the state in which the subject area cannot be determined is defined as the LOST state. The state in which the subject area determination unit 503 can determine the subject area is defined as the FIND state. In the determination processing S913, the FIND / LOST state is determined, and the processing branches to S914 and S917, respectively.

  In the FIND state, the process proceeds to step S914, and the template image of the subject is updated to the latest subject image. After that, it is necessary to determine the drive amount of the focus lens 204 in order to continue focusing on the subject. The drive amount of the focus lens 204 is determined based on the defocus amount of the gravity center position in the main subject area determined by the subject area determination unit 503 (S915). At this time, the value obtained by converting the drive amount of the focus lens 204 into the image plane value is the same as the magnitude of the optical axis vector of the subject. Therefore, the drive amount of the focus lens 204 is stored in the SDRAM 261 as the optical axis vector of the subject (S916). Then, the process proceeds to the determination process S905. On the other hand, when the determination result in the determination processing S913 is the LOST state, the main subject area is not determined and the defocus amount for the main subject is unknown. Therefore, the drive of the focus lens 204 is stopped (S917). It is determined whether the LOST state has continued for a predetermined time (a threshold time for determination) or more (S918). If the LOST state continues for a predetermined time or longer (YES in S918), the subject template is deleted (S919). After deleting the template, the process returns to the determination process S901, and the template is registered again according to the determination result (NO) (S904). When the LOST state is less than the predetermined time (NO in S918), the template is held and the process proceeds to the determination process S905. In this embodiment, the LOST state for a predetermined time or longer is adopted as the judgment condition in the judgment processing S918, but it may be changed to other conditions.

  Subsequently, a method of determining the second subject candidate area in the distance / vector comparison unit 505 will be described with reference to the flowchart of FIG. The distance / vector comparison unit 505 performs a calculation process on each of the set focus detection areas, and determines whether each focus detection area is the main subject area or another area. First, the optical axis vector calculation unit 504 calculates the optical axis vector based on the input defocus amount (S1001). After this, it is determined whether the state one frame before was the FIND state or the LOST state (S1002). When the state one frame before is the FIND state (YES in S1002), the process proceeds to the determination process S1003, and when the state one frame before is the LOST state (NO in S1002), the process proceeds to the determination process S1006. In the determination processing S1003, it is determined whether the calculated defocus amount is within a predetermined range. If the defocus amount is within the predetermined range (YES in S1003), the process proceeds to determination process S1004, and if the defocus amount is not within the predetermined range (NO in S1003), the process proceeds to process S1008.

  In the determination processing S1004, the optical axis vector calculated in the processing S1001 is compared with the optical axis vector of the main subject stored in the SDRAM 261 in the processing S916 of FIG. If it is determined that the directions and sizes of these optical axis vectors are the same, the process proceeds to step S1005, and if it is determined that the directions or the sizes are not the same, the process proceeds to step S1008. The coincidence of the optical axis vectors means that the directions of the optical axis vectors are the same, and the difference in size between the optical axis vector calculated in step S1001 and the optical axis vector of the main subject stored in the SDRAM 261 is predetermined. Is less than or equal to the threshold value of. That is, the optical axis vector in the determination processing S1004 means the velocity of the subject in the optical axis direction corresponding to the focus detection area. Therefore, while the tracking of the same subject continues, the directions of the optical axis vectors match. Therefore, the difference in the magnitude of the optical axis vector is small.

  In step S1005, the distance / vector comparison unit 505 regards the focus detection area that is the determination target as the main subject area and sets it as the second subject candidate area. If either one of the conditions is not satisfied in the determination processes S1003 and S1004, the distance / vector comparison unit 505 regards the focus detection area to be determined as an object area other than the main object (S1008). As described above, since the imaging device always keeps the focus on the main subject, when it is determined that the defocus amount is out of the predetermined range in the determination process S1003, it is considered that the main subject is not captured. Also, when it is determined in the determination processing S1004 that the directions or sizes of the optical axis vectors do not match, it is considered that the main subject is not captured.

  On the other hand, when the determination result of the determination process S1002 is the LOST state, the determination processes S1006 and S1007 are executed. The processing contents of the determination processes S1006 and S1007 are the same as those of the determination processes S1003 and S1004, respectively. In the determination processing S1006, when the calculated defocus amount is small and is within the predetermined range, the processing proceeds to processing S1005. If the defocus amount is out of the predetermined range, the process proceeds to a determination process S1007, and the direction and size of the optical axis vector in the focus detection area to be determined and the optical axis vector of the main subject stored in the SDRAM 261 are compared. . If the directions and sizes of the optical axis vectors match, the process proceeds to step S1005, and if the directions and sizes of the optical axis vectors do not match, the process proceeds to step S1008. After the processes S1005 and S1008, the process proceeds to a determination process S1009, and it is determined whether or not the process is completed for all the set focus detection areas. If all focus detection areas have been completed, the series of processes is ended, but if not completed, the process returns to step S1001 to continue the process.

  Next, a specific example will be described with reference to FIGS. 11 to 14. 11 and 12 are explanatory diagrams regarding the method of determining the second subject candidate region when the state one frame before is the FIND state (YES in the determination process S1002 in FIG. 10). FIG. 11 illustrates a scene in which there are two similar dogs (subjects). The dog on the right side is the main subject 1101, and the dog on the left side is the similar subject 1102.

  FIG. 11A illustrates a scene in which a similar subject 1102 staying at the same place on the left side of the main subject 1101 moving from the back to the front of the screen is photographed. The focus detection area 1103 for the main subject 1101 and the focus detection area 1104 for the similar subject are shown by rectangular frames. In FIG. 11B, the center point 1105 of the first subject candidate area output from the matching unit 502 is indicated by a black triangle symbol.

  FIG. 12A illustrates a temporal change of the defocus amount in each of the focus detection areas 1103 and 1104 with respect to the positions of the main subject 1101 and the similar subject 1102. The horizontal axis represents the time axis and the vertical axis represents the defocus amount. FIG. 12B exemplifies a temporal change of each optical axis vector in the focus detection areas 1103 and 1104. The horizontal axis represents the time axis and the vertical axis represents the optical axis vector. Regarding the direction of the optical axis, when viewed from the imaging device, the back direction is the positive direction and the front direction is the negative direction. The main subject 1101 is indicated by a circular symbol, and the similar subject 1102 is indicated by a rectangular symbol.

  In FIG. 12A, the defocus amount regarding the similar subject 1102 staying at the same place changes with time. The reason is that the focus lens 204 is moving according to the main subject 1101. Two dashed-dotted lines in FIG. 12A indicate the defocus amount threshold 1201. When the defocus amount is within the range between the two alternate long and short dash lines, it can be regarded as the main subject, but T1 indicates a period in which the defocus amount for the similar subject falls within the range. In the period T1, since the defocus areas relating to the main subject and the similar subject are within the threshold 1201, it is impossible to distinguish them. Therefore, the two can be distinguished from each other based on the temporal tendency of the optical axis vector shown in FIG. 12B, that is, the difference between the optical axis vectors of the main subject and the similar subject in the period T1. In the similar subject 1102 that does not move in the optical axis direction as shown in FIG. 12B, the optical axis vector amount maintains around zero. On the other hand, in the main subject 1101 that moves toward the front side of the optical axis, the optical axis vector amount maintains a non-zero value V1. Therefore, it is possible to distinguish between the similar subject and the main subject. In FIG. 11B, the second subject candidate region 1106 obtained by extracting the focus detection region in which the optical axis vector amount maintains V1 is shown by a rectangular frame.

  In the above processing, the main subject and the similar subject are accurately identified by grinding the first subject candidate region and the second subject candidate region based on the time change of the defocus amount and the time change of the optical axis vector for a plurality of subjects. Can be distinguished. That is, accurate tracking of the main subject can be realized by discriminating the main subject and the similar subject, which are difficult to be distinguished only by the temporal change of the defocus amount, based on the temporal change of the optical axis vector.

  Subsequently, the subject area determination in the case where the state one frame before is the LOST state (NO in determination processing S1002 in FIG. 10) will be described with reference to the specific examples in FIGS. 13 and 14. FIG. 13 illustrates a scene in which the main subject 1301 is moving from the back to the front, and the similar subject 1302 crosses from the right side to the left side of the screen. 13A shows an image taken at time t2, and FIG. 13B shows an image taken at time t3, starting from the shooting start time t1. FIG. 13C shows an image taken at time t4. Let “t1 <t2 <t3 <t4”.

  In FIG. 13A, the area 1303 of the main subject 1301 is determined and is in the FIND state. A focus detection area 1304 for the main subject and a focus detection area 1305 for the similar subject 1302 are shown. In FIG. 13B, the similar subject 1302 overlaps the main subject 1301 and the main subject 1301 is hidden, so that it cannot be seen on the shooting screen. Therefore, the LOST state is set. In FIG. 13C, the main subject 1301 appears again.

  FIG. 14A illustrates a change over time in the defocus amount in the focus detection areas 1304 and 1305 for each subject. FIG. 14B exemplifies a temporal change of each optical axis vector in the focus detection areas 1304 and 1305. The setting of each axis is the same as in FIG. The periods T2, T3, and T4 correspond to the scenes in FIGS. 13A to 13C. That is, the shooting time t2 in FIG. 13A is the time within the period T2, and the shooting time t3 in FIG. 13B is the time within the period T3. The shooting time t4 in FIG. 13C is a time within the period T4.

  In the period T2, the change in the defocus amount and the change in the optical axis vector are the same as in the scene in the FIND state shown in FIG. However, in the period T3, the main subject 1301 is hidden by the similar subject 1302 that crosses the front side. Therefore, it becomes impossible to calculate both the defocus amount and the optical axis vector amount for the main subject 1301. Further, since the drive of the focus lens 204 is stopped in the LOST state, the defocus amount for the similar subject 1302 that has not moved in the optical axis direction becomes a constant value.

  In the period T4, the main subject 1301 appears again on the shooting screen, and the defocus amount 1401 and the optical axis vector amount 1402 are acquired. That is, the defocus amount and the optical axis vector amount in the focus detection area 1306 in FIG. 13C are acquired, and these are the results that are supposed to exist when the main object that was once lost can be accurately tracked when it reappears. is there. In the period T4, the focus lens 204 is stopped and the main subject 1301 moves forward, so that the absolute amount of defocus increases. Therefore, it is difficult to regard the focus detection area 1306 as the area of the main subject only with the defocus amount 1401. On the other hand, in the optical axis vector amount shown in FIG. 14B, it can be seen that the optical axis vector of the main subject is the same in both direction and size (V2) in the periods T2 and T4. In this way, in the LOST state in the periods T3 and T4, the second subject candidate region can be determined again by continuing to search for the region where the optical axis vector amount is V2 for all the focus detection regions. That is, in a scene in which the main subject is hidden by a similar subject, it is possible to identify the main subject, which is difficult to identify only by the temporal change of the defocus amount, based on the temporal change of the optical axis vector. The processing after the second subject candidate area is determined is the same as the processing in the FIND state shown in FIGS. 11 and 12, and thus detailed description thereof will be omitted. Note that the graphs in FIGS. 12B and 14B show a situation in which the main subject is moving at a constant velocity. Not limited to this, it is also possible to deal with the acceleration motion of the main subject by generating and predicting an approximate expression from the tendency of the temporal change of the optical axis vector.

  According to the present embodiment, the tracking accuracy can be improved by using the distance information of the entire imaging surface and the optical axis component (optical axis vector information) of the position change vector as the information for tracking. Pixel pattern or color histogram is similar to the main subject When the main subject approaches the similar subject in the optical axis direction, or when the main subject is moving in the optical axis direction, the similar subject crosses in front of you. Even in this case, the tracking process of the main subject can be accurately performed.

[Second Embodiment]
The second embodiment of the present invention will be described below. First, the difference between the present embodiment and the first embodiment will be described. The second embodiment aims to reduce the amount of calculation and improve the processing efficiency as compared with the first embodiment. In the first embodiment, the entire image pickup surface is divided into a plurality of small areas for focus detection, but in the present embodiment, the focus detection area is determined based on the coordinates of the first subject candidate area output from the matching unit 502. decide. By using this method, the number of focus detection regions is reduced when the first subject candidate regions are concentrated in one place or a narrow range, and the amount of calculation processing can be reduced. Since the configuration of the image pickup apparatus in this embodiment is the same as the configuration in FIG. 2, the reference numerals already used are used, and the detailed description thereof is omitted.

  FIG. 15 is a block diagram showing a configuration example of the subject tracking unit 271 of this embodiment. In the first embodiment, the plurality of evaluation values and the area information output from the matching unit 502 are input only to the subject area determination unit 503. In this embodiment, a plurality of evaluation values and area information are also input to the camera control unit 225. The focus detection signal processing unit 224 sets the focus detection region based on these pieces of information, and outputs the calculated defocus information to the optical axis vector calculation unit 504. The processes after the optical axis vector calculation unit 504 are the same as those in the first embodiment, and thus the description thereof will be omitted.

  A method of setting the focus detection area based on the output of the matching unit 502 will be described with reference to FIG. FIG. 16 shows an example image of a scene in which two subjects are photographed. The focus detection signal processing unit 224 calculates center point groups 1601 and 1604 respectively indicating the center coordinates of each region based on the plurality of region information output from the matching unit 502. The center point group 1601 shows the center coordinates of a plurality of regions corresponding to the image of the first subject, and the center point group 1604 shows the center coordinates of the plurality of regions corresponding to the image of the second subject. The focus detection area is set based on the central point group. Specifically, as shown in FIG. 16, small areas are arranged in advance on the entire screen, and an area including the center point group 1601 for each small area is a temporary focus detection area 1602 (see a solid thick line frame). Is set. The reason for setting the provisional focus detection area 1602 at this stage is that the size of the subject is not taken into consideration. Therefore, the focus detection area 1603 including a small area is set in consideration of the area information of each center point of the subject. For example, since one of the center point groups 1604 has the area information of the area 1605, a part of the two small areas 1606 and 1607 located above the provisional focus detection area (see the solid bold frame) is the area. Included in 1605. Therefore, the two small areas 1606 and 1607 are added to the focus detection area 1603. In this way, the focus detection signal processing unit 224 performs a correlation operation on the small area existing in the wider focus detection area 1603 to calculate the defocus amount.

  In the present embodiment, when the central point group is dense as shown in FIG. 16, the number of the focal point detection areas 1603 is minimized by setting the focal point detection areas from the area information of the central points. be able to. As a result, it is not necessary to perform the correlation calculation for the areas other than the focus detection area 1603, so the calculation load can be reduced. According to this embodiment, the processing efficiency of focus detection can be improved.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

200 ... image pickup device 204 ... focus lens 212 ... focus control unit 213 ... lens control unit 221 ... image pickup device 224 ... focus detection signal processing unit 225 ... camera control unit 271 ‥‥‥ Subject tracking unit

Claims (14)

A subject tracking device for performing a tracking process of a subject in an image by acquiring image data and distance information related to the image data,
Acquisition means for acquiring the image data and distance information,
First detection means for detecting an image of a subject from the image data and outputting information on a plurality of first candidate regions related to the subject region;
Second detection means for outputting information on a second candidate region related to the subject region based on the distance information and the position change of the subject in the depth direction in the image;
Determining means for acquiring information on the first and second candidate regions and determining a region narrowed down from the plurality of first candidate regions by the second candidate region as a subject region used for subject tracking. ,
The first detection means is
Subject detecting means for obtaining the image data to detect a subject,
A matching unit that acquires the image data and the detection information of the subject detection unit, performs matching processing, and outputs a plurality of evaluation values and region information as information of the first candidate region;
The second detection means is
Calculating means for acquiring the distance information and calculating a position change vector indicating a position change of the subject in the image in the depth direction;
And a comparing unit that compares the distance information and the position change vector relating to the plurality of detected subject areas and outputs the information of the second candidate area . The acquisition unit acquires the distance information from data of a plurality of parallax images having parallax and outputs the distance information to the second detection unit,
The comparing means stores the distance information and the position change vector corresponding to the subject area in the past captured image stored in the storage means, and the distance information and the position change vector corresponding to the subject area in the current captured image. comparing the subject tracking apparatus according to claim 1, the difference between the difference and the positional change vector of the distance information of a subject area less than the threshold value, respectively, and determines the second candidate region. Signal processing means for generating a signal for focus detection from the data of the parallax image and calculating a defocus amount related to the imaging optical system;
A control unit that acquires a defocus amount from the signal processing unit and controls focus adjustment of the imaging optical system,
The subject tracking apparatus according to claim 2 , wherein the second detection unit acquires a defocus amount from the signal processing unit as the distance information. The signal processing means calculates a defocus amount in a plurality of focus detection areas set in a captured image, and the control means performs drive control of a focus lens of the imaging optical system,
4. The subject tracking apparatus according to claim 3 , wherein the calculation unit calculates the position change vector from the defocus amount calculated in the past and the current by the signal processing unit and the drive amount of the focus lens. In the first state in which the subject is being identified, the comparison unit has a defocus amount equal to or less than a threshold value, and a position change vector corresponding to a subject region in a past captured image, and a current position change vector. The subject tracking device according to claim 3 or 4 , wherein a subject region having a difference from a position change vector corresponding to the subject region in the captured image of 1 is a threshold value or less is determined as the second candidate region. In the second state in which the subject is not specified, the comparison unit has the defocus amount equal to or less than a threshold value, or a position change vector corresponding to a subject region in a past captured image, and a current position change vector. The subject tracking device according to claim 3 or 4 , wherein a subject region having a difference from a position change vector corresponding to the subject region in the captured image of 1 is a threshold value or less is determined as the second candidate region. The comparing means may obtain a plurality of the position change vectors stored in the storage means, and perform a process of identifying a position change vector corresponding to a current subject region based on a tendency of the position change vectors with respect to time change. The subject tracking device according to any one of claims 2 to 6 , which is characterized in that. The comparison means acquires the plurality of evaluation values and area information by the matching means, calculates a reliability evaluation value of subject tracking, and uses the reliability evaluation value for determining the difference in the distance information and the difference in the position change vector, respectively. The subject tracking device according to claim 2 , wherein the threshold value is changed to a value corresponding to the reliability evaluation value. Wherein, according to claim 4, characterized in that to determine the driving amount of defocus amounts obtained by the focus lens in the focus detection area centroid of the object region determined by the determining means is positioned Subject tracking device. Imaging apparatus including a subject tracking apparatus according to any one of claims 1 9. A plurality of microlenses, an imaging device having a plurality of photoelectric conversion units corresponding to each microlens,
The signals output by the plurality of photoelectric conversion units corresponding to the plurality of focus detection areas set in the captured image are acquired, the defocus amount is calculated from the displacement amount of the plurality of images, and the focus is calculated from the defocus amount. The image pickup apparatus according to claim 10 , further comprising a focus adjustment control unit that calculates a drive amount of the lens and performs focus adjustment by controlling the drive of the focus lens. The focus adjustment control means calculates a defocus amount in the focus detection area set corresponding to the first candidate area among the plurality of focus detection areas, and does not correspond to the first candidate area. The image pickup apparatus according to claim 11 , wherein focus detection calculation processing is not performed on the focus detection area. A control method executed by a subject tracking device that obtains image data and distance information related to the image data to perform a tracking process of a subject in an image,
A first detection unit acquires the image data, detects a subject, performs matching processing based on the image data and detection information of the subject, and obtains a plurality of evaluation values as information of a first candidate region related to the subject region. Outputting the area information,
The second detecting means acquires the distance information, calculates a position change vector indicating a position change of the subject in the image in the depth direction, and calculates the distance information and the position change vector relating to the plurality of detected subject regions. A step of comparing and outputting information of the second candidate area related to the object area;
Determining means acquires information on the first and second candidate areas, and determines an area narrowed down from the plurality of first candidate areas by the second candidate area as a subject area to be used for subject tracking; A method of controlling a subject tracking device, comprising: A program that causes a computer of a subject tracking device to execute each step according to claim 13 .

Images