JP6833483B2 - Subject tracking device, its control method, control program, and imaging device - Google Patents

Description

The present invention relates to a subject tracking device, a control method thereof, a control program, and an imaging device.

Generally, in an imaging device such as a digital camera, important processing such as exposure determination and focusing is automated. Further, if an imaging device equipped with an anti-vibration control device for preventing image blurring due to camera shake or the like is used, it is possible to almost eliminate factors that induce a user's shooting error during shooting.

On the other hand, when shooting a moving subject or when shooting at a telephoto where the focal length becomes large, there are the following problems.

When shooting a moving subject, if the subject deviates from the shooting screen, a special technique of the user is required in order to accurately track the subject by user operation. Further, when taking a picture with an image pickup device having a telephoto lens, the influence of image blurring due to camera shake becomes large. Therefore, it is difficult for the user to keep the subject in the center of the shooting screen. Then, even if the user operates the image pickup device to return the subject to the shooting screen, the amount of camera shake caused by the operation is corrected by the image pickup device. As a result, it becomes difficult to finely adjust the subject to the shooting screen or the center of the shooting screen due to the influence of the vibration isolation control.

In order to deal with such a problem, for example, there is an imaging device provided with a tracking control device that automatically tracks a subject by moving a part of the optical system in a direction intersecting the optical axis (see Patent Document 1). .. Furthermore, an imaging device that extracts a target subject from the image obtained as a result of shooting and tracks the subject with a rotating pan head or the like so that the position of the center of gravity of the subject is near the center of the shooting screen. Yes (see Patent Document 2).

JP-A-2010-93362 Japanese Unexamined Patent Publication No. 7-2267873

However, in the image pickup apparatus described in Patent Document 2, the estimation accuracy of the position of the center of gravity extracted from the captured image affects the tracking control of the rotating pan head. In the following description, extracting a specific subject area from the sequentially supplied captured images is called tracking, and a part of the optical system or a moving part such as a rotating pan head is controlled based on the subject position based on the tracking result. Doing is called tracking.

In tracking a subject, if there is an area similar to the tracking target in the captured image, an erroneous area may be tracked. Then, if tracking is performed based on an erroneous tracking result, the user's shooting will be greatly hindered.

Therefore, an object of the present invention is to provide a subject tracking device, a control method thereof, a control program, and an imaging device capable of preventing a tracking malfunction caused by an error in tracking the subject.

To achieve the above object, the subject tracking apparatus according to the present invention is a subject tracking apparatus in shooting the image, and the object tracking means for determining the confidence level indicating the position and subject likeness of the subject in the captured image, the captured image by controlling the shifting means for shifting the position of the subject in the position of the subject in the captured image, and a Ru control means is moved so that the difference becomes smaller between the target position in the captured image Yes, and the control means to change the state of tracking of the object by changing the control gain in the control of the shift means on the basis of the reliability.

The imaging device according to the present invention is characterized by having an imaging means for obtaining a captured image according to an optical image imaged via an imaging optical system, and the subject tracking device described above.

The control method according to the present invention is a control method of a subject tracking device that tracks a subject by controlling a shift means for shifting the position of the subject in a continuously obtained captured image. The difference between the subject tracking step for obtaining the reliability indicating the position of the subject and the subject-likeness in the image and the position of the subject in the captured image by controlling the shift means is reduced from the target position in the captured image. have a, and Ru control step moves, in said control step changes the control gain in the control of the shifting means and changing the state of tracking of the object based on the reliability.

The control program according to the present invention is a control program used in a subject tracking device that tracks a subject by controlling a shift means that shifts the position of the subject in a continuously obtained captured image, and is used in the subject tracking device. A computer provided with a subject tracking step for obtaining a reliability indicating the position and subject-likeness of the subject in the captured image in the captured image, and controlling the shift means to position the subject in the captured image. wherein the Ru control step moves so that the difference becomes smaller between the target position in the captured image, is executed, in the control step, the change of the control gain in the control of the shift means on the basis of the reliability characterized that you change the state of tracking of the object.

According to the present invention, it is possible to prevent a tracking malfunction caused by an error in tracking a subject.

It is a figure which shows typically the appearance of an example of the image pickup apparatus provided with the subject tracking apparatus according to 1st Embodiment of this invention. It is a figure which shows an example of the structure of the camera shown in FIG. It is a figure for demonstrating the tracking of the subject detected by the camera shown in FIG. 2, (A) is the figure which shows the photographed image before the start of tracking of a subject, (B) is the figure which shows the photographed image after the start of tracking of a subject. It is a figure which shows. It is a flowchart for demonstrating an example of the subject tracking process performed by the subject tracking unit shown in FIG. It is a figure for demonstrating the template matching processing performed by the subject tracking part shown in FIG. It is a block diagram for demonstrating an example of the tracking amount calculation part shown in FIG. It is a flowchart for demonstrating an example of subject tracking performed by the camera shown in FIG. It is a figure which shows an example about the structure of the camera which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the notification method in the camera shown in FIG.

The following is a drawing of an example of a subject tracking device capable of reducing a tracking malfunction caused by an error in tracking a subject according to the embodiment of the present invention, a control method thereof, a control program, and an imaging device including the subject tracking device. Will be described with reference to. An example in which the present invention is applied to a digital camera as an imaging device will be described in both the first and second embodiments described below, but the present invention is not limited to digital cameras, but digital video cameras and surveillance. It can also be applied to imaging devices such as cameras, Web cameras, and mobile phones. Further, the present invention can be applied to both interchangeable lens type and integrated lens type cameras.

[First Embodiment]
FIG. 1 is a diagram schematically showing the appearance of an example of an imaging device including a subject tracking device according to the first embodiment of the present invention.

The illustrated imaging device is, for example, a digital camera (hereinafter, simply referred to as a camera) 101, and a release button 104 is arranged on the upper surface of the camera housing. Here, the axis extending in the direction of the side surface of the camera 101 is the X axis, and the axis extending in the direction of the upper surface is the Y axis. Further, the axis extending in the front direction of the camera 101 is defined as the Z axis. Then, the rotation 103Pp around the X axis is defined as the pitch (PITCH), and the rotation 103y around the Y axis is defined as the yaw (YAW).

FIG. 2 is a diagram showing an example of the configuration of the camera shown in FIG.

With reference to FIGS. 1 and 2, the correction lens 114 and the image sensor 106 are positioned on the optical axis 102 of the image pickup optical system. The image pickup device 106 generates an electric signal (analog signal) corresponding to the optical image formed through the image pickup optical system, A / D-converts the analog signal, and outputs the captured image. Although not shown, the imaging optical system is provided with a zoom lens and a focus lens. The angular velocity meter 103 detects the angular velocity shake (that is, the shake of the camera 101) at the pitch 103p and the yaw 103y, and sends the angular velocity shake signal to the CPU 105. The CPU 105 constitutes the subject tracking device of the present embodiment.

In the CPU 105, the runout correction angle calculation unit 108 obtains the runout correction angle based on the angle runout signal. For example, the runout correction angle calculation unit 108 cuts the DC component added as detection noise in the angle runout signal, performs integration processing, and outputs an angle signal indicating the angle of the camera 101. For example, an HPF (high-pass filter) is used to cut the DC component. Then, the angle signal is sent to the sensitivity adjusting unit 109.

The zoom and focus position detection unit 107 sends a zoom position and a focus position indicating the positions of the zoom lens and the focus lens to the sensitivity adjustment unit 109. The sensitivity adjustment unit 109 obtains the focal length and the shooting magnification based on the zoom position and the focus position. Then, the sensitivity adjusting unit 109 amplifies the angle signal according to the focal length and the photographing magnification to set the shake correction target value. The sensitivity adjustment unit 109 sends a runout correction target value as a runout correction amount to the drive control unit 113.

Since the sensitivity of the camera image plane to the shake correction stroke of the correction lens 114 changes depending on the position of the focus lens and the zoom lens, the shake correction target value is obtained here based on the zoom position and the focus position. ..

The correction lens 114 described above is used to shift the subject to be tracked (the subject to be tracked) in the image obtained as a result of shooting. As will be described later, the drive control unit 113 drives and controls the correction lens 114 to track the subject. Further, the drive control unit 113 drives the correction lens 114 in a direction different from the optical axis (for example, a direction in which it intersects) to correct image blur (optical vibration isolation).

In the example shown in FIG. 2, optical vibration isolation is performed using the correction lens 114, but as an image blur correction method, a method of moving the image sensor 106 in a plane orthogonal to the optical axis to perform image blur correction. May be used. Further, the cutout position of the image may be changed for each photographing frame which is the output of the image sensor 106, and electronic vibration isolation which reduces the influence of the shake may be used. Then, the above-mentioned plurality of image blur correction methods may be used in combination.

The subject tracking unit 110 receives the captured image output from the image sensor 106 and extracts the position of the subject region in the captured image. In the present embodiment, the position of the center of gravity of the subject area is used as the position of the subject. Further, the subject tracking unit 110 obtains the reliability (likelihood) indicating the subject-likeness, and sends the position of the center of gravity and the likelihood to the tracking amount calculation unit 111. The tracking amount calculation unit 111 calculates the tracking correction amount, which is the control amount used for tracking the subject by the correction lens 114, based on the position of the center of gravity of the subject area. Further, the tracking amount calculation unit 111 changes the tracking correction amount based on the likelihood.

The adder 112 adds the runout correction amount output from the sensitivity adjustment unit 109 and the tracking correction amount output from the tracking amount calculation unit 111, and sends the addition result to the drive control unit 113. The drive control unit 113 obtains the drive amount of the correction lens 114 based on the addition result which is the output of the adder 112, drives the correction lens 114 based on the drive amount, and performs tracking of the subject and correction of image blur. Do.

FIG. 3 is a diagram for explaining tracking of a subject detected by the camera shown in FIG. FIG. 3A is a diagram showing a captured image before the start of tracking of the subject, and FIG. 3B is a diagram showing a captured image after the start of tracking of the subject.

In the captured image 301a shown in FIG. 3A, the subject 302a exists at a position away from the image center 304. Here, the position of the center of gravity of the subject 302a (the position of the center of gravity of the subject) is indicated by reference numeral 303a. The CPU 105 gradually brings the subject center of gravity position 303a closer to the image center 304 by tracking (also referred to as tracking control), and finally makes the image center 304 and the subject center of gravity position 303a substantially coincide with each other. As a result, as shown in FIG. 3B, the subject center of gravity position 303a of the subject 302a that has been successfully tracked coincides with the image center 304.

Here, the subject tracking method in the subject tracking unit 110 will be described. In the illustrated example, when tracking a subject, a method of estimating a region with a low degree of difference by collating with a captured image while shifting the template using a partial image showing the target subject as a template (template matching). Is used. Further, in order to correspond to the scale conversion in the time direction for the target subject, a method of extracting the characteristic color of the subject and estimating the subject area based on the distribution state of the characteristic color of the captured image is also used. Then, the template is updated based on the estimated subject area.

FIG. 4 is a flowchart for explaining an example of the subject tracking process performed by the subject tracking unit 110 shown in FIG.

When the subject tracking process is started, the subject tracking unit 110 reads the captured image from the image sensor 106 (step S401). Then, the subject tracking unit 110 determines whether or not a subject that is a tracking target designated in advance exists in the captured image (step S402). If the subject to be tracked does not exist (NO in step S402), the subject tracking unit 110 detects the subject in the captured image in order to determine the subject to be tracked (step S403).

In the process of step S403, the subject tracking unit 110 detects the subject based on the user's instruction or automatically detects the subject. When detecting a subject based on a user's instruction, the user instructs the position of the subject in a captured image by an input interface having a touch panel, buttons, and the like. Then, the subject tracking unit 110 detects the subject based on the position instructed by the user and extracts the subject area.

On the other hand, when automatically detecting a subject, the subject tracking unit 110 uses face detection. In face detection, for example, a method that uses knowledge about the face (skin color information, parts such as eyes, nose, and mouth) and a method that constructs a classifier for face detection by a learning algorithm represented by a neural net, etc. There is. Further, in face detection, in order to improve the detection rate, face detection is generally performed by combining the above methods. As for face detection, for example, a method of performing face detection using the wavelet transform and the image feature amount described in JP-A-2002-251380 is known.

Subsequently, the subject tracking unit 110 extracts the feature amount of the subject to be tracked from the subject area (step S404). Here, since the tracking process is performed using template matching, the image pattern of the subject area is extracted as a feature amount. Further, since the subject area is estimated based on the distribution of the feature colors, the color histogram Hin of the subject area is maintained. After that, the subject tracking unit 110 ends the subject tracking process and waits until the sampling cycle of the next shooting.

On the other hand, when the subject to be tracked exists (YES in step S402), the subject tracking unit 110 performs the template matching process using the template (step S405).

FIG. 5 is a diagram for explaining the template matching process performed by the subject tracking unit 110 shown in FIG. FIG. 5A is a diagram showing an example of a subject model (template) in the template matching process, and FIG. 5B is a diagram showing an example of an image for searching a subject to be tracked.

Here, the pixel pattern of the partial image (template) 501 showing the subject to be tracked is used as the feature amount. Then, the brightness of the pixel data is used as the feature amount 502 of the template 501. The feature amount T (i, j) is expressed by the following equation (1), where the coordinates in the template are (i, j), the number of horizontal pixels is W, and the number of vertical pixels is H.

In FIG. 5B, the coordinates in the search image 503, which is the range in which the matching process is performed, are defined as (x, y). In the search image 503, a partial area 504 for obtaining an evaluation value of template matching is set. Then, the brightness of the captured image is used for the feature amount 505 of the partial region 504 as in the template 501. The feature amount S (i, j) is expressed by the following equation (2), where the coordinates in the partial region 504 are (i, j), the number of horizontal pixels is W, and the number of vertical pixels is H.

When evaluating the similarity between the template 501 and the partial region 504, the absolute difference difference, the so-called SAD (Sum of Absolute Evaluation) value, is used. Then, the SAD value is obtained by the following equation (3).

The SAD value V (x, y) is obtained while shifting the partial region 504 by one pixel in order from the upper left of the search image 503. The coordinates (x, y) where the SAD value V (x, y) indicates the minimum value indicates the position most similar to the template 501. That is, the position showing the minimum value is the position in the search image 503 where there is a high possibility that the target subject to be tracked exists.

Here, an example in which one-dimensional information such as luminance (luminance signal) is used as the feature amount has been described, but three-dimensional information such as brightness, hue, and saturation may be used as the feature amount. In addition, although the SAD value is used when obtaining the evaluation value of template matching, a method such as normalized cross-correlation, so-called NCC (Normalized Correlation Coaffiency) may be used.

With reference to FIG. 4 again, the subject tracking unit 110 estimates the subject area based on the position estimated by the template matching process (step S406). Here, the subject tracking unit 110 uses the information represented by the following equation (4) based on the color histogram Hin of the subject region obtained in step S404 and the color histogram Hout of all or part of the captured image at the current time. Obtain the quantity I (a).

The amount of information I (a) indicates the probability of occurrence of the subject region with respect to the entire or part of the captured image in each bin of the color histogram. The subject tracking unit 110 adapts the amount of information I (a) to each pixel of the captured image at the current time to generate a map showing the possibility that the subject exists in the search image 503. Then, the subject tracking unit 110 estimates the subject area based on the map and outputs the position of the center of gravity of the estimated subject area.

Subsequently, the subject tracking unit 110 calculates the likelihood of subject tracking (step S407). Factors that hinder the certainty of subject tracking include, for example, changes in the subject, existence of similar subjects, and accumulation of tracking errors. The subject tracking unit 110 calculates the likelihood by multiplying these factors by the evaluation values obtained from the template matching process and the estimation of the subject area.

The larger the minimum value of the SAD value V (x, y) obtained by the above equation (3), the larger the change in the subject. Therefore, the subject tracking unit 110 reduces the likelihood as the minimum value increases. When the SAD value similar to the minimum value of the SAD value V (x, y) obtained by the formula (3) exists at a position separated from the estimated position of the subject by a predetermined threshold value or more, a similar subject exists. Probability is high. Therefore, the subject tracking unit 110 has a degree of similarity between the SAD value existing at a position separated by a predetermined threshold value or more from the estimated position of the subject and the minimum value of the SAD value V (x, y) obtained by the equation (3). The higher the value, the lower the likelihood.

Regarding the amount of information I (a) indicating the color feature color of the subject obtained by the equation (4), the smaller the entropy Ein, which is the average value (expected value) in the subject area, the larger the change in the subject. The entropy Ein is expressed by the following equation (5).

The subject tracking unit 110 reduces the likelihood as the entropy Ein becomes smaller. Further, with respect to the amount of information I (a), which is the characteristic color of the subject represented by the equation (4), the larger the entropy Eout, which is the average value (expected value) outside the subject area, the higher the possibility that a similar object exists. The entropy Eout is expressed by the following equation (6).

The subject tracking unit 110 reduces the likelihood as the entropy Eout increases. Furthermore, once the certainty of subject tracking is low, the reliability of subsequent tracking is also low. Therefore, the subject tracking unit 110 adds the likelihood of the history when calculating the likelihood. For example, the average value of the likelihoods of a predetermined period is used as the likelihood of the current frame. In this way, the subject tracking unit 110 calculates the likelihood of subject tracking.

Subsequently, the subject tracking unit 110 updates the subject feature amount (step S408). Here, the subject tracking unit 110 updates the template based on the subject area estimated in the process of step S406 to correspond to the scale change of the subject. After that, the subject tracking unit 110 ends the subject tracking process and waits until the sampling cycle of the next shooting.

FIG. 6 is a block diagram for explaining an example of the tracking amount calculation unit 111 shown in FIG.

The tracking amount calculation unit 111 obtains the tracking correction amount in each of the vertical direction and the horizontal direction of the image, and here, the calculation of the tracking correction amount in one direction will be described. The tracking amount calculation unit 111 calculates a count value for tracking the subject based on the difference between the position of the subject (that is, the position of the center of gravity) and the position of the center of the image (the target position of the subject). The tracking amount calculation unit 111 adds the count value for each sampling time and controls the position of the subject to move to the target position. Then, the tracking amount calculation unit 111 changes the magnitude of the count value to change the degree of tracking based on the likelihood (subject likelihood) that is the output of the subject tracking unit 110.

The subtractor 604 subtracts the coordinates of the image center position acquired by the image center position acquisition unit 602 from the coordinates of the subject position acquired by the subject position acquisition unit 601. As a result, the distance (center deviation amount) between the image center position and the center of gravity position of the subject on the image is calculated. This amount of center deviation is signed data when the center of the image is set to zero. The output of the subtractor 604 is input to the count value table 605, and the count value corresponding to the magnitude of the difference distance between the position of the center of gravity of the subject and the center of the image is output from the count value table 605. The subject position acquisition unit 601 may acquire the subject position on the coordinates centered on the image center position (both x-coordinate and y-coordinate are zero).

The tracking amount calculation unit 111 sets the count value to zero when the center deviation amount is equal to or less than the threshold value Z or greater than or equal to the threshold value −Z. As a result, a dead zone region in which tracking is not performed is set within a range of ± Z from the center of the image. The count value table 605 is a table in which the count value increases as the center shift amount increases, and the sign of the count value is matched with the sign of the center shift amount.

The output of the count value table 605 is input to the variable gain device 606. The gain amount calculation unit 607 calculates the gain (control gain) related to the count value based on the subject likelihood acquired by the subject likelihood acquisition unit 603. Here, when the subject likelihood is high, the gain amount calculation unit 607 considers that there is a high possibility that the correct subject can be tracked, and increases the gain in order to improve the tracking responsiveness. On the other hand, if the subject likelihood is low, there is a possibility that the wrong subject has been tracked. Then, tracking a subject that is an erroneous tracking target greatly hinders the user's shooting. Therefore, the gain amount calculation unit 607 reduces the gain when the subject likelihood is low. At this time, if the subject likelihood is equal to or less than the threshold value, the gain may be set to 0 and the tracking control may be stopped. Further, the tracking control may be stopped by turning off the tracking switch 608 instead of setting the gain to zero. Then, the gain amount calculation unit 607 sets the calculated gain (gain amount) as the variable gain Cg in the variable gain device 606.

The output of the variable gain device 606 is input to the signal selection unit 609. The down count value of the down count value setting unit 611 and the setting of the tracking switch 608 are input to the signal selection unit 609. The signal selection unit 609 selects the output of the variable gain device 606 when the tracking switch 608 is on. On the other hand, when the tracking switch 608 is off, the signal selection unit 609 selects the output of the down count value setting unit 611. Then, the output of the signal selection unit 609 is input to the adder 610.

The down count value setting unit 611 sets the down count value. The tracking amount previous sampling value is input from the sampling unit 613 to the down count value setting unit 611. When the tracking amount previous sampling value has a positive sign, the down count value setting unit 611 sets the down count value to a negative value. On the other hand, when the tracking amount previous sampling value has a negative sign, the down count value setting unit 611 sets the down count value to positive. As a result, the down count value setting unit 611 reduces the absolute value of the tracking correction amount. The down count value setting unit 611 sets the down count value to zero when the tracking amount previous sampling value is within 0 ± predetermined range. Further, the tracking amount previous sampling value indicates the tracking correction amount up to the previous sampling.

The adder 610 adds the output of the signal selection unit 609 and the tracking amount previous sampling value. When a negative down count value is added to the tracking amount previous sampling value, the absolute value of the tracking correction amount becomes small. The output of the adder 610 is input to the upper / lower limit value setting unit 612. The upper / lower limit value setting unit 612 sets the tracking correction amount to an amount that is less than a predetermined upper limit value and exceeds a predetermined lower limit value. The output of the upper / lower limit value setting unit 612 is input to the sampling unit 613 and the LPF 614. The LPF614 sends a tracking correction amount that cuts high frequency noise to the correction lens amount conversion unit 615. The correction lens amount conversion unit 615 converts the tracking correction amount into a signal form for tracking by the correction lens 114, and outputs the final tracking correction amount.

In this way, the tracking amount calculation unit 111 obtains a count value for each sampling according to the difference between the image center position and the subject position, adds it to the tracking correction amount, and gradually brings the subject position closer to the image center. Perform tracking.

In the above description, the gain amount calculation unit 607 sets the gain based on the subject likelihood, but the gain amount calculation unit 607 sets the gain based on the position of the center of gravity of the captured image in addition to the subject likelihood. The gain may be set.

As described above, the subject tracking unit 110 estimates the target subject area from the captured image. Therefore, it is necessary to perform tracking so that the target subject fits in the captured image. When the target subject exists at the edge of the captured image, there is a high possibility that the subject will deviate from the captured image after the next frame. Therefore, it is important to move the subject to the center of the image (target position) by tracking.

On the other hand, when the target subject exists near the center of the image (target position), it is unlikely that the subject will deviate from the captured image after the next frame. Therefore, the importance of tracking decreases. Therefore, the lower the subject likelihood, the smaller the tracking degree (that is, the tracking state), and the closer the subject center of gravity position is to the image center (target position), the smaller the tracking degree is controlled.

FIG. 7 is a flowchart for explaining an example of subject tracking performed by the camera shown in FIG. The process according to the illustrated flowchart starts when the main power supply of the camera 101 is turned on, and is performed at a predetermined sampling cycle.

First, the CPU 105 determines whether or not the anti-vibration SW (not shown) is ON (step S701). When the anti-vibration SW is ON (YES in step S701), the CPU 105 captures the output of the angular velocity meter 103 (step S702). Then, the CPU 105 determines whether or not the camera 101 is in a state where the shake correction is possible (step S703). Here, the CPU 105 determines that the shake correction is not possible when the camera 101 is supplied with power and the output of the angular velocity meter 103 is stable. On the other hand, when the camera 101 is in a state after the output of the angular velocity meter 103 is stable, the CPU 105 determines that the shake correction is possible. As a result, runout correction is not performed when the output of the angular velocity meter 103 is unstable immediately after the power is supplied.

When the camera 101 is in a state where the shake correction is possible (YES in step S703), the CPU 105 outputs the angular velocity meter 103 by the shake correction angle calculation unit 108 and the sensitivity adjustment unit 109 as described above. The amount of runout correction is obtained based on (step S704). On the other hand, when the camera 101 is not in a state where the shake correction is possible (NO in step S703), the CPU 105 sets the shake correction amount to zero (step S705). If the anti-vibration SW is OFF (NO in step S701), the CPU 105 proceeds to the process of step S705.

After the process of step S704 or S705, the CPU 105 determines whether or not the tracking SW (not shown) is ON (step S706). When the tracking SW is ON (YES in step S706), the CPU 105 determines whether or not there is a subject to be tracked in the captured images continuously obtained from the image sensor 106 (step S707). When the subject to be tracked exists (YES in step S707), the CPU 105 estimates the position of the center of gravity of the subject region in the captured image (step S708).

Subsequently, the CPU 105 obtains the subject likelihood related to the position of the center of gravity as described above (step S709). As described above, the processing of steps S708 and S709 is performed by the subject tracking unit 110. Then, the CPU 105 obtains the tracking correction amount based on the position of the center of gravity of the subject and the likelihood of the subject by the tracking amount calculation unit 111 (step S710).

If there is no subject to be tracked (NO in step S707), the CPU 105 sets the tracking correction amount to zero (step S711). If the tracking SW is OFF (NO in step S706), the CPU 105 proceeds to the process of step S711.

After the process of step S710 or S711, the CPU 105 adds the runout correction amount and the tracking correction amount to calculate the lens drive amount (step S712). Then, the CPU 105 drives and controls the correction lens 114 based on the lens drive amount by the drive control unit 113 (step sS713). After that, the CPU 105 finishes the subject tracking process and waits until the next sampling cycle.

In this way, in the first embodiment, the correction lens 114 is driven and controlled based on the position of the center of gravity of the subject obtained by tracking the subject and its likelihood. As a result, it is possible to prevent a malfunction of tracking and perform highly responsive tracking.

[Second Embodiment]
In the above-described first embodiment, an imaging device including a subject tracking device that changes the tracking degree based on the subject likelihood has been described. On the other hand, in the second embodiment, the tracking degree is changed based on the subject likelihood, and when the tracking control is difficult, the photographer is notified (warning) that the tracking control is difficult. An imaging device including a subject tracking device instructed in the above will be described. Whether or not tracking control is difficult is determined based on the subject likelihood. Since the point of changing the tracking degree based on the subject likelihood is the same as that of the first embodiment, the description thereof will be omitted. Further, similarly to the first embodiment, a mode in which the subject tracking device is applied to the camera will be described.

FIG. 8 is a diagram showing an example of the configuration of the camera according to the second embodiment of the present invention. In the second embodiment, the CPU 105 as the subject tracking device is based on the determination results of the tracking state determination unit 801 and the tracking state determination unit 801 that determine whether or not the tracking control is difficult based on the subject likelihood. It differs from the first embodiment in that it includes a notification instruction unit 802 that instructs the photographer to notify the photographer based on the above. Further, the camera 101 of the second embodiment includes a display unit (FIGS. 9A to 9D) for displaying an image and a display control unit 803 for controlling an image displayed on the display unit. The display control unit 803 receives the notification instruction from the notification instruction unit 802 and notifies the photographer that the tracking control is difficult.

The subject tracking unit 110 acquires the subject likelihood for the set subject to be tracked, as in the first embodiment. Then, the acquired subject likelihood is sent to the tracking state determination unit 801. The tracking state determination unit 801 determines whether or not the subject likelihood is equal to or less than the threshold value based on the subject likelihood received from the subject tracking unit 110. Then, when the received subject likelihood is equal to or less than the threshold value, the tracking state determination unit 801 determines that the tracking control is difficult because the subject likelihood is low. On the other hand, when the received subject likelihood is larger than the threshold value, the tracking state determination unit 801 determines that the subject likelihood is higher than the tracking controllable degree. The determination result by the tracking state determination unit 801 is sent to the notification instruction unit 802. When the tracking state determination unit 801 determines that the subject likelihood is equal to or less than the threshold value, the notification instruction unit 802 instructs the display control unit 803 to notify the photographer that the tracking control is difficult. send. When the display control unit 803 receives the notification instruction to the photographer from the notification instruction unit 802, the display control unit 803 changes the content of the image displayed on the display unit and notifies the photographer that the tracking control is difficult.

Next, with reference to FIG. 9, a method of notifying the photographer that the tracking control is difficult will be described. As shown in FIGS. 9 (A) to 9 (D), the camera 101 is provided with a liquid crystal display 901 that functions as a display unit on the back surface, and displays an image captured by the camera 101 before shooting by the live view display. can do. Further, the camera 101 has a touch panel 902 arranged under the liquid crystal display 901, and can acquire the coordinate position of the subject 903 designated by the touch operation and set the subject to be tracked. The subject 903 designated by the touch operation is indicated by the subject frame 904. When the display control unit 803 receives the notification instruction from the notification instruction unit 802, the display control unit 803 changes the display of the tracking icon 905 displayed on the liquid crystal display 901 to notify the photographer that the tracking control is difficult. An example of the change of the tracking icon 905 with the change of the tracking control state will be described.

In FIG. 9A, when the subject 903 to be tracked is designated by the touch operation, the tracking control is performed, and the screen of the liquid crystal display 901 (hereinafter simply referred to as “simply”) so that it can be seen that the tracking control is being performed. The tracking icon 905 is displayed on the screen). While the subject is not tracked (the subject is not specified), the tracking icon 905 is not displayed on the screen.

FIG. 9B is a diagram showing a state after the subject is designated. The amount of runout correction and the amount of tracking correction when the state of the camera changes in the order of FIG. 9 (A), FIG. 9 (B), FIG. 9 (C), and FIG. 9 (D) with the passage of time are added. The output of the adder 112 is shown in FIG. 9 (E). Further, the change in the subject position is shown in FIG. 9 (F), the change in the subject likelihood is shown in FIG. 9 (G), and the display state of the tracking icon 905 is shown in FIG. 9 (H). The tracking icon 905 is not displayed on the screen until the timing T11 when the subject is designated by the touch operation as shown in FIG. 9A, and the subject to be tracked is as shown in FIG. 9B. If specified, the tracking icon 905 is displayed on the screen. After that, even if the designated subject 903 is far from the center of the image, the tracking control is performed based on the tracking correction amount calculated by the tracking amount calculation unit 111, so that the subject 903 is as shown in FIG. 9B. Is returned to the center of the image. In the figure, the timing when the subject is specified and the position is detected is shown as T11, and the timing when the subject is returned to the center of the image by the tracking control is shown as T12.

After that, when the subject 903 moves further as shown in FIG. 9C, the tracking correction amount is further increased to perform tracking control. When a plurality of similar subjects appear on the screen as shown in FIG. 9D, the subject likelihood is set low. In the present embodiment, when the subject likelihood is equal to or less than the threshold Th1, the tracking gain is set to zero and the tracking control is not performed. At this time, in order to notify the photographer that tracking is not performed any more, the tracking icon 905 is grayed out and the photographer is notified of the non-tracking state (see FIG. 9H).

As described above, the subject tracking device according to the second embodiment sends an instruction to change the display of the tracking icon 905 when it is determined that the subject likelihood is low and the tracking control cannot be performed. As a result, the photographer can framing the camera by himself / herself to move the subject to the center of the image. It should be noted that the threshold value for determining whether or not to perform tracking control and the threshold value for notifying the photographer that the tracking control is difficult may be the same or different. If the thresholds match, the photographer can be notified that tracking control is not performed. When the threshold value for determining whether or not to perform tracking control is smaller, it is possible to notify the photographer that the subject likelihood is low while performing tracking control, and to prompt the photographer to frame before the tracking control is stopped. Therefore, the tracking control can be performed more smoothly than when the tracking control is stopped and then restarted.

In the second embodiment, the photographer is notified that the tracking control is difficult by changing the color of the tracking icon, but the notification method is not limited to this. In addition to the tracking icon, an icon indicating that the subject likelihood is low may be displayed on the liquid crystal display 901, or an LED lamp is provided on the camera 101 to indicate the lighting state (lighting, turning off, blinking, etc.) of the LED lamp as the subject. It may be changed when the likelihood is high (when tracking control is possible).

In the above-described embodiment, a case where a correction lens is used as a shake correction member and the correction lens is moved in a plane intersecting (for example, orthogonal to) the optical axis has been described. On the other hand, the present invention is not limited to optical vibration isolation, and includes subject tracking that moves the image sensor in a plane orthogonal to the optical axis, subject tracking that changes the cutout position in each of the photographing frames output by the image sensor, and an image sensor. It can also be applied to subject tracking that rotationally drives a lens barrel including a group of photographing lenses, subject tracking that combines a rotating pan head that pans and tilts another camera, or a combination of the above-mentioned plurality of subject tracking.

As is clear from the above description, in the example shown in FIG. 2, the correction lens 114 and the CPU 105 function as shift means, and the CPU 105, the angular velocity meter 103, and the zoom and focus position detection unit 107 function as subject tracking means. Further, the CPU 105 functions as a control means. The subject tracking device is composed of at least the correction lens 114, the CPU 105, the angular velocity meter 103, and the zoom and focus position detection unit 107.

Although the present invention has been described above based on the embodiments, the present invention is not limited to these embodiments, and various embodiments within the scope of the gist of the present invention are also included in the present invention. ..

For example, the function of the above embodiment may be used as a control method, and the subject tracking device may be made to execute this control method. Further, a program having the functions of the above-described embodiment may be used as a control program, and the control program may be executed by a computer provided in the subject tracking device. The control program is recorded on a computer-readable recording medium, for example.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

103 Angular velocity meter 107 Zoom and focus position detection unit 108 Shake correction angle calculation unit 109 Sensitivity adjustment unit 110 Subject tracking unit 111 Tracking amount calculation unit 113 Drive control unit 114 Correction lens

Claims (10)

It is a subject tracking device
In shooting images, and the object tracking means for determining the confidence level indicating the position and subject likeness of the subject in the captured image,
The captured image by controlling the shifting means for shifting the position of the subject in the position of the subject in the captured image, and a Ru control means is moved so that the difference becomes smaller between the target position in the captured image Yes, and
The control means is a subject tracking device, characterized in that the control gain in the control of the shift means is changed based on the reliability to change the tracking state of the subject. The subject tracking means uses a partial area indicating the subject in the captured image as a template, and collates the template with a partial region of the captured image obtained after the captured image obtained as the partial image used as the template. The subject tracking device according to claim 1, wherein the evaluation value is obtained, and the reliability is obtained according to the distribution of the evaluation value. The subject tracking means estimates the subject area based on the distribution of the characteristic colors of the subject in the captured image, and obtains the reliability based on the distribution of the characteristic colors inside and outside the subject area. The subject tracking device according to claim 1. Any one of claims 1 to 3, wherein the subject tracking means obtains the reliability of the subject in the currently captured image by referring to the history of the reliability when obtaining the reliability. The subject tracking device described in the section. The subject tracking device according to any one of claims 1 to 4, wherein the control means lowers the tracking state of the subject as the reliability is lower. The subject tracking device according to claim 5 , wherein the control means lowers the tracking state of the subject as the position of the subject in the captured image is closer to the target position. An imaging means for obtaining a captured image according to an optical image imaged via an imaging optical system,
The subject tracking device according to any one of claims 1 to 6.
An imaging device characterized by having. It said shifting means, the imaging apparatus according to claim 7, wherein the obtaining Bei correction lens driven in a direction crossing the optical axis of the imaging optical system provided in the imaging optical system. A control method for a subject tracking device that tracks a subject by controlling a shift means for shifting the position of the subject in continuously obtained captured images.
In the captured image, a subject tracking step for obtaining reliability indicating the position and subject-likeness of the subject in the captured image, and
The position of the subject in the photographed image by controlling said shifting means, have a, and Ru control step moves so that the difference becomes smaller between the target position in the captured image,
A control method characterized in that, in the control step, the tracking state of the subject is changed by changing the control gain in the control of the shift means based on the reliability. A control program used in a subject tracking device that tracks a subject by controlling a shift means that shifts the position of the subject in continuously obtained captured images.
In the computer provided in the subject tracking device,
In the captured image, a subject tracking step for obtaining reliability indicating the position and subject-likeness of the subject in the captured image, and
Wherein the position of the subject in the captured image by controlling the shifting means, to execute the, and Ru control step moves so that the difference becomes smaller between the target position in the captured image,
In the control step, the control program which is characterized that you change the state of tracking of the object by changing the control gain in the control of the shift means on the basis of the reliability.

Images