JP2023103964A - Image blur correction apparatus, control method therefor, imaging apparatus, program, and storage medium - Google Patents

Abstract

To provide an image blur correction apparatus capable of performing object blur correction without impairing quality of an image even when parts of an object to be detected are switched.SOLUTION: Included are: object detection means configured to detect an object for each part from an input image and output a position of the part of the object detected; selection means configured to select a part of the object from the part of one or more objects detected by the object detection means on the basis of a predetermined priority; and object blur correction amount calculation means configured to calculate an object blur correction amount for correcting object blur on the basis of a difference between a position of the part of the object selected by the selection means and a predetermined target position in the image. The object blur correction amount calculation means, when the part of the object selected by the selection means is switched from a first part to a second part, calculates the object blur correction amount on the basis of a difference between the position of the selected part of the object and the target position, a position of the second part in the image, and a position of the second part in the image acquired before the switching.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image blur correction technique in imaging devices such as digital cameras and digital videos.

In an imaging device such as a digital camera, in addition to "image blurring" caused by camera shake of the user holding the camera body, "subject blurring" caused by a change in the position of a subject such as a person is corrected. ing.

"Image blurring" due to camera shake can be detected by an angular velocity sensor attached to an imaging device or by obtaining a motion vector of a stationary object (background) in a captured image. On the other hand, "subject blur" can be detected by performing subject detection and measuring the position of the subject.

In subject detection, it is also possible to detect each part of the subject (eg, head, face, eyes, nose, mouth, body, arms, legs, etc.). Depending on the shooting conditions, a certain part of the subject may be detected, but other parts may not be detected. For example, the subject is far away and the face is too small to be detected, but the body is too large to be detected. Conversely, there are cases where the face can be detected from the color of the skin, but the body cannot be detected because the color of the clothes blends with the background. In this way, the part of the subject that can be detected may change depending on the shooting conditions.

Therefore, for example, Japanese Patent Application Laid-Open No. 2002-200001 discloses a technique for tracking a subject by switching to detecting parts other than the face when the face cannot be detected.

Japanese Patent No. 5771461

Japanese Patent Laid-Open No. 2002-200001 only tracks the position of the subject, and does not take into consideration subject shake correction that brings the position of the subject closer to a predetermined target position in the image.

In subject shake correction, control is performed so that the position of the part of the subject stably stays near a predetermined position in the image. At that time, when the position of the detected part of the subject changes, the difference in the position of the part before and after the switch is reflected in the control, and even though the subject does not move, the position of the subject changes within the screen. may be lost.

For example, when performing control so that the face of a person stays in the center of the image, it is assumed that the face cannot be detected in the middle and switching is made to detect the body. In this case, the state in which the face is positioned in the center changes to the state in which the body is positioned in the center of the image, resulting in an unnatural change in the positions of the face and the body, which impairs the quality of the image.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems, and its object is to provide an image blur correction device capable of subject blur correction without impairing image quality even when the part of the subject to be detected changes. to provide.

An image blur correction apparatus according to the present invention includes subject detection means for detecting each part of a subject from an input image and outputting the position of the detected part of the subject, and one or more detected by the subject detection means. selection means for selecting a part of the subject based on a predetermined priority from the parts of the subject, and based on the difference between the position of the part of the subject selected by the selection means and a predetermined target position in the image and subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur, wherein the subject blur correction amount calculation means determines whether the part of the subject selected by the selection means is from the first part. When switching to the second part, the difference between the position of the selected part of the subject and the target position, the position of the second part in the image, and the second part in the image acquired before switching The subject blur correction amount is calculated based on the position of .

According to the present invention, even when the part of the subject to be detected is switched, it is possible to perform subject shake correction without impairing the image quality.

1 is a block diagram showing the configuration of an imaging device according to a first embodiment; FIG. 4 is a flowchart for explaining processing performed in the first embodiment; 4A and 4B are diagrams showing examples of subject detection results; FIG. 4A and 4B are diagrams for explaining a method of calculating a subject blur correction amount; FIG. FIG. 10 is a diagram for explaining an example in which a part selected in subject detection is switched; 4A and 4B are diagrams for explaining suppression processing of subject blur correction; FIG. FIG. 2 is a block diagram showing the configuration of an imaging device according to a second embodiment; FIG. 9 is a flowchart for explaining processing performed in the second embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
<Description of block diagram>
FIG. 1 is a diagram showing the configuration of an imaging device 100 that is a first embodiment of an image blur correction device of the present invention.

In FIG. 1, an imaging apparatus 100 includes an imaging optical system 101, an imaging element 102, a development processing unit 103, an object detection unit 104, an object part selection unit 105, an object shake correction amount calculation unit 106, a control unit 110, a ROM 112, and a RAM 114. Prepare.

In this embodiment, the imaging optical system 101 includes a correction lens that is displaced in a direction different from the optical axis of the imaging optical system 101 to move the subject image on the imaging plane and correct image blur. Further, the imaging device 102 is configured to be able to correct image blur by displacing in a direction perpendicular to the optical axis.

In addition, although FIG. 1 shows the imaging apparatus 100 as an imaging apparatus in which a lens and a camera body are integrated, the present invention can also be applied to a lens-interchangeable imaging apparatus.

In FIG. 1, a subject image formed by a photographing optical system 101 is converted into an analog image signal by an image sensor 102 . The analog image signal is A/D converted in the development processing unit 103, then subjected to development processing such as white balance correction, color (luminance/color difference signal) conversion, and γ correction, and output as image data.

A subject detection unit 104 detects each part of a specific subject from the input image data from the development processing unit 103, and outputs the positions of the detected parts of the subject. Parts include, for example, the head, face, eyes, nose, mouth, body, arms, and legs.

The subject part selection unit 105 selects a part of the subject based on a predetermined priority from one or more parts of the subject detected by the subject detection unit 104 .

Subject blur correction amount calculation section 106 calculates subject blur correction amount based on the difference between the position of the part of the subject selected by subject part selection section 105 and a predetermined target position in the image.

Then, at least one of the correction lens in the photographing optical system 101 and the image sensor 102, which is optical image blur correction means, is displaced based on the subject blur correction amount calculated by the subject blur correction amount calculation unit 106. FIG. As a result, optical image blur correction can be performed so that the position of the desired part of the subject is brought closer to the target position.

A control unit 110 such as a microcomputer develops a program stored in a ROM 112 in a RAM 114 and executes the program, thereby controlling the imaging apparatus 100 as a whole.

The present invention is not limited to optical image blur correction, and electronic image blur correction may be performed based on the subject blur correction amount. The electronic image blur correction is an operation of correcting image blur by changing the clipping position of a smaller display image area from the entire image area based on the subject blur correction amount.

<Description of flow chart>
Next, the image blur correction operation in the imaging apparatus 100 configured as described above will be described using the flowchart shown in FIG. The operation of the flowchart in FIG. 2 is realized by the control unit 110 developing the program stored in the ROM 112 in the RAM 114 and executing the program.

In FIG. 2, in step S201, the control unit 110 uses the subject detection unit 104 to detect a specific subject for each part from the input image from the development processing unit 103. FIG. A person's face is a representative part of a subject.

In this case, a known face detection method may be used as the detection method. Known techniques for face detection include a method that uses knowledge about faces (skin color information, parts such as eyes, nose, mouth, etc.), and a method that configures a classifier for face detection using a learning algorithm represented by a neural network. and so on. In order to improve the detection accuracy, it is common to perform face detection by combining them. A similar method can be used to detect objects other than human faces.

FIG. 3 is a diagram showing an example of subject detection results when the subject is a person. Rectangles 301 to 310 indicate detected regions. Specifically, rectangle 301 is the face or head, rectangle 302 is the right eye, rectangle 303 is the left eye, rectangle 304 is the nose, rectangle 305 is the mouth, rectangle 306 is the right arm, rectangle 307 is the left arm, rectangle 308 is the body, rectangle 309. represents the right leg and rectangle 310 represents the left leg.

Below, the representative point included in each rectangle is treated as the position of the part of the subject. As the representative point, for example, barycentric coordinates, which are the coordinates of the barycenter of each rectangle, are used. Here, the detected parts are represented by rectangles for the sake of convenience, but they are not limited to rectangles and may be represented by arbitrary shapes.

Note that the type of subject is not limited to people as in the example of FIG. 3, and may be animals such as dogs, cats, birds, and horses. It can also be applied to general objects other than animals. The fineness of the detection part is not limited to the example shown in FIG. 3, and may be finer or coarser.

If there are multiple subjects in the image, one subject that is considered to be the most important is selected, and the parts of that subject are output. As a selection method, for example, a method of preferentially selecting an image having a larger size, a position closer to the center of the image, and a higher reliability of detection can be considered. Alternatively, the user may be allowed to select from a plurality of candidates. As described above, the position of the part of the subject can be obtained.

In step S202, the control unit 110 uses the subject part selection unit 105 to select a part of the subject from the one or more parts of the subject detected in step S201 based on a predetermined priority.

For example, when subject detection results are used for autofocus (AF), the predetermined priority is generally to give higher priority to small organs such as eyes and pupils than to the face.

On the other hand, when subject detection results are used for subject blur correction in this embodiment, it is necessary to set priorities in consideration of the stability of detection, stability of shape, and stability of position. .

Detection stability represents how stably a site can be detected. For example, since the eyes are smaller than the head and body and difficult to detect, the stability of detection is low.

Shape stability represents how stable the shape of a site is. For example, human arms and legs, bird wings, and the like change their shape as they expand and contract, so their shape stability is low. Also, in the case of the eyes, the shape changes depending on the degree of opening, although the change is not as large as that of the arms and legs. It can be said that the shape of the head is more stable than the arms, legs, and eyes.

Positional stability represents how stable the position of the part of the subject is when the subject changes orientation. For example, in the case of the eyes, positional stability is low because the positions of the parts of the subject also change when the direction of the subject changes from front to side to back. On the other hand, in the case of the head, even if the orientation of the subject changes from the front, side, or back, the positions of the parts of the subject do not change significantly, and it can be said that the position is highly stable.

In order to stably perform subject blur correction, it is desirable that these are highly stable. In this way, since the viewpoint of priority is different from that in the case of AF, it is preferable to be able to select the part of the subject to be used for subject blur correction independently of the part of the subject to be used for AF. .

Specifically, it is preferable to select the face or the head as the part of the subject with priority over organs included in the face such as the eyes, pupils, nose, and mouth. In addition, from the viewpoint of the image, the face or the head is generally more focused than the body. Therefore, it is preferable to correct blurring of the face or the head with priority over the body as the part of the subject.

In step S203, the control unit 110 uses the subject blur correction amount calculation unit 106 to perform subject blur correction based on the difference between the position of the part of the subject selected in step S202 and a predetermined target position in the image. Calculate quantity. A method of calculating the subject blur correction amount will be described with reference to FIG.

A black square 401 indicates a predetermined target position in the image and can be set at any position. In the example of FIG. 4, it is set at the center of the image. A rectangle 402 indicated by a dotted line indicates the head selected in step S202, and a black circle 403 indicates the position of its center of gravity. An arrow 404 indicates the difference between the coordinates 401 and 403, and this difference is used as the subject shake correction amount.

Note that the position of the detected part of the subject generally has an error due to the effects of image noise and accuracy of subject detection. Therefore, it is desirable to apply a low-pass filter (LPF) to the position of the detected part of the subject or the subject blur correction amount, and use the low-frequency component as the subject blur correction amount.

In step S204, control unit 110 determines whether the part of the subject selected in step S202 is the same as the part selected in the previous frame. Control unit 110 advances the process to step S205 if different from the part selected in the previous frame, and advances the process to step S206 if the same.

FIG. 5 is a diagram showing an example in which the selected site is different from the previous frame. 5A shows the previous frame, FIG. 5B shows the current frame, and FIG. 5C shows the next frame. In contrast to FIG. 5(a), FIG. 5(b) and FIG. 5(c) show the subject moving away and becoming smaller. A black square 501 indicates the target position of the subject, which is the center of the image in this example.

The subject's head 502 is selected in FIG. 5(a), and the body 503 is selected in FIGS. 5(b) and 5(c). As described in step S202, when the head can be detected in FIGS. 5B and 5C, the head is preferentially selected. However, in FIGS. 5B and 5C, the head is too small to be detected, and only the body is detected, so the body is selected.

In FIG. 5A , subject shake correction is performed so that the coordinates of the center of gravity of the head 502 overlap the target position 501 .

In FIG. 5B, the trunk 502 is selected as the part. At the time when FIG. 5B is obtained, subject shake correction is performed so that the coordinates of the center of gravity of the head 501 still overlap the target position 501 . A black circle 504 indicates the coordinates of the center of gravity of the trunk, and an arrow 505 indicates the subject shake correction amount for moving the coordinates of the center of gravity of the trunk to the target position 501 .

FIG. 5C shows a frame acquired as a result of subject blur correction performed according to an arrow 505 (subject blur correction amount) so that the coordinates 504 of the center of gravity of the body overlap with the target position 501 (image center). That is, as the body part of the subject to be detected is switched, the head of the subject is positioned near the target position 501 at the center of the image (FIGS. 5A and 5B), and the body of the subject is positioned at the center of the image. change to the state in That is, although the position of the subject on the images is almost the same between FIGS. By correcting the blurring of the subject by using the above method, the position of the subject moves unnaturally by the change in the coordinates of the center of gravity of the part of the subject.

Therefore, in order to suppress unnatural movement of the position of the subject due to the switching of the part of the subject, when the part selected in step S202 changes, the object shake correction amount suppression process shown in the next step S205 is performed. I do.

In step S205, the control unit 110 uses the object shake correction amount calculation unit 106, based on the amount of displacement of the position of the object selected by the object part selection unit 105 when the part of the object changes, calculated in step S203. suppresses the amount of subject blur correction applied.

Specifically, a value obtained by subtracting the barycentric coordinates of the head 502 in FIG. 5A from the barycentric coordinates of the body 503 in FIG. 5B is subtracted from the subject shake correction amount 505 as an offset. By doing so, the amount of change in the coordinates of the center of gravity of the part of the subject is canceled from the subject shake correction amount, and unnatural movement of the position of the subject is eliminated.

The processing for suppressing the amount of subject blur correction will be described in detail with reference to FIG. The horizontal axis in FIG. 6 represents time, and the vertical axis represents the vertical coordinates of the position of the part of the subject selected in step S202. In this example, the head, which is the part with the highest priority, is initially selected, and at time t1, the head is switched to the body, and at time t2, the body is switched back to the head.

In FIG. 6 , a dashed line 601 represents time-series data of the positions of parts of the subject before the suppression process, and a dotted line 602 represents the time-series data of the positions of the parts of the subject after the suppression process. A solid line 603 is time-series data obtained by performing correction processing, which will be described later, on the dotted line 602 . Here, in FIG. 5, the downward direction of the image is defined as the direction in which the vertical coordinate increases. Since the head is higher than the body in FIG. 5, the vertical coordinate is smaller for the head and larger for the body in FIG.

At times t1 and t2, the vertical coordinate is greatly displaced due to the switching of the parts. Let 604 be the amount of displacement of the position of the part of the subject at t1, and 605 be the amount of displacement of the position of the part of the subject at t2. If these displacement amounts are directly reflected in the subject blur correction amount, the position of the subject will change unnaturally just by changing the part to be selected.

Therefore, a displacement amount 604 is subtracted as an offset from the position 601 of the part of the subject before suppression processing after time t1, and a displacement amount 606, which is an amount obtained by adding the displacement amounts 604 and 605 after time t2, is used as an offset. By subtracting, the position 602 of the part of the subject after the suppression process can be obtained.

In this way, the subject blur correction amount calculation unit 106 subtracts from the subject blur correction amount, as an offset, the amount of displacement of the position of the subject when the part of the subject selected by the subject part selection unit 105 is switched. Further, the subject blur correction amount calculation unit 106 adds an offset to the current offset each time the part of the subject selected by the subject part selection unit 105 is switched, and subtracts the added offset from the subject blur correction amount. Subtract. As a result, it is possible to prevent the position of the subject from moving unnaturally due to the switching of the part of the subject.

Here, since the head is selected again after time t2, it is ideal that the dashed line 602 after suppression processing matches the dashed line 601 before suppression processing, but they do not match. This is because the offset at the time t1 and the offset at the time t2 are not canceled and an offset error occurs. Possible causes of the offset error include the motion of the subject when the body part of the subject is switched, the subject detection error, and the like.

Therefore, after time t2 when the head is detected again, the added offset (displacement amount 604+605=displacement amount 606) is gradually brought closer to zero. We approach the dashed line 601 and obtain the solid line 603 .

That is, when the subject part selection unit 105 selects the part of the subject with the highest priority, the subject blur correction amount calculation unit 106 gradually reduces the added offset to zero, thereby correcting the subject blur by the offset. Quantity deviations can be corrected.

In step S206, the control unit 110 performs image blur correction based on the subject blur correction amount calculated in step S203 or step S205. As described above, the correcting lens in the photographing optical system 101, which is optical image blur correction means, the image sensor 102, and electronic image blur correction means (not shown) can be used as the image blur correction means.

In the final step S207, the control unit 110 determines whether or not the processing has been completed up to the final frame, and if the processing has not been completed up to the final frame, the processing returns to step S201. When the processing is completed up to the final frame, the operation of this flow ends.

By the above processing, even when the part of the subject to be detected is switched, subject shake correction can be performed without impairing the image quality.

It should be noted that the processes of steps S201 to S207 can all be performed in real time during image capturing, but they do not necessarily have to be performed during image capturing, and may be performed during image reproduction.

For example, by recording an image at the time of shooting in a recording unit (not shown) and performing the processing of steps S201 to S207 on the image recorded by the electronic image blur correction, subject blur correction can be performed when the image is reproduced. It can be carried out.

In addition to the image, information on the position of the part of the subject obtained by the subject detection unit 104 may be recorded in addition to the image at the time of photographing, so that subject detection does not have to be performed when the image is reproduced.

(Modification of the first embodiment)
In the subject blur correction amount suppression processing in the above-described embodiment, the coordinates of the center of gravity of the body 503 in FIG. A value obtained by subtracting the coordinates of the center of gravity of the head 502 in 5(a) is subtracted from the subject shake correction amount 505 as an offset. However, the offset calculation method is not limited to this.

As the offset, it is preferable to use the difference between the barycentric coordinates of the head after the selected part of the subject is switched and the barycentric coordinates of the body after the selected part of the subject is switched. We do not know the coordinates of the center of gravity of the head. Therefore, in the above-described embodiment, the change in the barycentric coordinates of the head before and after switching is considered to be small, and in the above-described embodiment, the barycentric coordinates of the head 502 before switching are subtracted from the barycentric coordinates of the body 503 after switching. The value obtained was used as an offset. However, if the coordinates of the center of gravity of the head 502 before switching are deviated from the target position, the deviation is subject to subject blur correction. changes. Therefore, the subject blur correction amount corrected before and after switching is added to the value obtained by subtracting the coordinates of the center of gravity of the head 502 before switching from the coordinates of the center of gravity of the body 503 after switching (displacement amount 604 described above). may However, subject shake correction has been performed at the time of FIG. A value obtained by subtracting the coordinates of the center of gravity of the head 502 before switching from the coordinates of the center of gravity of 503 may be used as the offset.

Further, if the body 503 can be detected before the selected parts of the subject are switched, the coordinates of the center of gravity of the body 503 in FIG. can be used as an offset. As described above, if there is a change in the size of the subject between FIG. 5A and FIG. When obtained, the offset also includes an error corresponding to the variation in the size of the subject. However, the change in the size of the object that occurs in a short period of time before and after switching is not large. Therefore, the offset may be obtained from the coordinates of the center of gravity of the head 502 and the coordinates of the center of gravity of the body 503 before switching. In addition, in order to obtain the offset more accurately, the body size before and after switching is compared to obtain the change in the size of the subject that occurs before and after switching, and the offset is obtained by taking this into consideration. may

(Second embodiment)
In the first embodiment, only subject blurring is treated as an object of image blurring correction, but in the present embodiment, image blurring correction is performed in consideration of camera shake as well.

FIG. 7 is a diagram showing the configuration of an imaging device 700 according to the second embodiment of the invention. In FIG. 7, the same reference numerals as in FIG. 1 denote the same components as in FIG. 1, and the description thereof will be omitted. The imaging apparatus of this embodiment has a camera shake detection unit 701, a camera shake correction amount calculation unit 702, and an image blur correction amount synthesizing unit 703 in addition to the configuration shown in FIG. In this embodiment, only portions that perform processing different from that in the first embodiment will be described.

A camera shake detection unit 701 detects camera shake applied to the imaging device 700 . A shake correction amount calculation unit 702 calculates an image shake correction amount (shake correction amount) for correcting image blur caused by shake detected by the shake detection unit 701 .

The image blur correction amount synthesizing unit 703 synthesizes the subject blur correction amount output from the subject blur correction amount calculation unit 106 and the image blur correction amount (shake correction amount) output from the camera shake correction amount calculation unit 702, A final image blur correction amount (composite image blur correction amount) is generated.

FIG. 8 is a flow chart showing an image blur correction operation in the second embodiment. 8, steps common to those shown in FIG. 2 are assigned the same reference numerals as in FIG. 2, and descriptions thereof are omitted. The operation of the flowchart of FIG. 8 is realized by the control unit 110 developing the program stored in the ROM 112 in the RAM 114 and executing the program.

In step S204, control unit 110 determines whether the part of the subject selected in step S202 is the same as the part selected in the previous frame. Control unit 110 advances the process to step S205 if different from the part selected in the previous frame, and advances the process to step S801 if the same. After step S205 is completed, control unit 110 advances the process to step S801.

In step S<b>801 , the control unit 110 detects camera shake applied to the imaging device 700 using the camera shake detection unit 701 . A gyro sensor, for example, can be used as the camera shake detection unit 701 . However, the detection method is not limited to this. Note that the timing of executing step S801 is arbitrary as long as it is before S802, and may be executed before or after S201 to S205.

In step S802, the camera shake correction amount calculation unit 702 calculates a camera shake correction amount for correcting image blur caused by camera shake detected in step S801. For example, the shake correction amount can be calculated by integrating the shake angular velocity signal obtained by the gyro sensor and inverting the sign.

In step S803, the control unit 110 uses the image blur correction amount synthesizing unit 703 to synthesize the subject blur correction amount obtained in step S204 or step S205 and the camera shake correction amount obtained in step S802. An image blur correction amount is calculated.

As a synthesizing method, for example, there is a method of synthesizing by dividing by frequency. Specifically, a low-pass filter (LPF) is applied to the subject shake correction amount to extract low-frequency components. On the other hand, a high-pass filter (HPF) is applied to the shake correction amount to extract high frequency components. The low frequency component of the subject blur correction amount obtained in this way and the high frequency component of the camera shake correction amount are added and synthesized.

That is, the final image blur correction amount is such that the low frequency corrects the subject blur component and the high frequency corrects the camera shake component.

Here, it is preferable that the cutoff frequencies of the LPF and HPF are substantially matched and dynamically changed according to the reliability of object detection. For example, if the reliability of subject detection is high, increase the cutoff frequency to strengthen the subject blur correction, and if the reliability of subject detection is low, decrease the cutoff frequency to strengthen the camera shake correction. .

In step S804, the control unit 110 performs image blur correction based on the final image blur correction amount obtained in step S803. As described above, the correcting lens in the photographing optical system 101, which is optical image blur correction means, the image sensor 102, and electronic image blur correction means (not shown) can be used as blur correction means.

By the above processing, even when the part of the subject to be detected is switched, subject shake correction and image blur correction due to camera shake can be performed without impairing the image quality.

(Third Embodiment)
In both the first embodiment and the second embodiment, a configuration has been described in which an offset is acquired in accordance with the switching of the subject part, and the acquired offset is subtracted from the subject shake correction amount. Effects similar to these can also be obtained by changing the target position according to the switching of the subject part. In the present embodiment, a form will be described in which the target position is changed according to the switching of the subject part.

This embodiment differs from the first embodiment in the process of suppressing the subject blurring correction amount shown in step S205, but the rest of the configuration is the same as that of the first embodiment, so the description is omitted. In this embodiment, the target position is changed as suppression processing. The target position is determined based on the position of the subject part selected after switching. The part of the subject selected after switching refers to the body 503 in FIGS. 5(a) and 5(b). Hereinafter, for the sake of explanation, the part of the subject selected before the switching occurs (in the case of FIGS. 5A and 5B, the head 502) is the first part, and the part selected after the switching occurs. The part of the subject (body 503 in FIGS. 5A and 5B) is sometimes called a second part.

The target position after change will be explained. In the present embodiment, for example, the subject blur correction amount after switching is calculated using the barycentric coordinates (position of black circle 504) of the second part (torso 503) in the frame after switching shown in FIG. 5B as the target position. do. FIG. 5B is a frame image obtained as a result of subject blur correction using the coordinates of the center of gravity of the first part (head 502). located nearby. Therefore, the coordinates of the center of gravity of the body 503 in FIG. 5B are set as the target position, and the coordinates of the center of gravity of the body 503 are set to the target position (that is, the coordinates of the center of gravity of the body 503 in FIG. 5B) in subsequent frames. By performing subject blur correction on the first image, subject blur correction control can be performed so that the head 502 is positioned near the center of the image even if the head 502 is not selected. This makes it possible to reduce abrupt changes in the subject position that accompany the switching of the subject part.

As for the target position, the coordinates of the center of gravity of the second part (torso 503) in the frame before switching shown in FIG. 5A may be used as the target position. In this case, similarly to the modified example of the first embodiment, it is affected by the change in the size of the object that occurs between FIGS. 5(a) and 5(b). However, since it can be assumed that the change in the size of the subject that occurs in a short period of time is not large, there is little possibility that the composition will change significantly even if the position of the center of gravity of the body 503 in the frame before switching is set as the target position.

Also, the target position is set based on the difference in the center-of-gravity coordinates between the first part (head 502) and the second part (body 503) in the frame before switching shown in FIG. 5(a). may When the head 502 is displaced from the target position in the frame before switching, if the position of the body 503 in the frame before switching is set as the target position, subject shake correction is performed so as to maintain the displaced state. By setting the position obtained by adding the difference between the head 502 and the body 503 in the frame before switching to the target position before switching (the center of the image in FIG. 5A) as the target position, the head 502 is positioned at the center of the image. The position of the torso 503 in the case of doing so can be set as the target position. However, similarly to the case where the coordinates of the center of gravity of the body 503 in the frame before switching are set as the target position, it is affected by the change in the size of the object that occurs between FIGS. 5A and 5B.

Further, regardless of which position of the second part (torso 503) is used before switching or after switching, the coordinates of the center of gravity of the torso may not only be used as the target position, but various adjustments may be made. For example, when the subject suddenly moves at the same time when the subject part is switched, there is a possibility that the barycentric coordinates of the selected subject part are positioned at the edge of the image. In this case, if the barycentric coordinates themselves are set as the target position, subject blur correction control is performed so that the subject is positioned at the edge of the image. A position closer to the center of the image than the barycentric coordinates may be set as the target position by, for example, limiting the coordinates that can be set as the position. Further, if the size of the second part before and after switching can be detected, the target position may be a position adjusted with respect to the coordinates of the center of gravity of the second part before switching in consideration of the change in size. . For example, when the subject part is switched from the face to the body due to the reduction in the size of the subject, the target position may be a position where the coordinates of the center of gravity of the body before the change are moved to the center by the amount of the change in size.

The disclosure of this specification includes the following image blur correction device and its control method, imaging device, program, and storage medium.

(Item 1)
subject detection means for detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
selection means for selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected by the subject detection means;
subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected by the selection means and a predetermined target position in the image; prepared,
The subject blur correction amount calculation means includes:
When the part of the subject selected by the selection means is switched from the first part to the second part, the difference between the position of the selected part of the subject and the target position, and the position of the second part in the image. The image blur correction device, wherein the subject blur correction amount is calculated based on the position and the position of the second part in the image acquired before switching.

(Item 2)
When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. 3. The method according to item 1, wherein the subject blur correction amount is calculated based on the position of the first part in the image acquired before switching and the position of the second part after switching. image stabilization device.

(Item 3)
When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. and calculating the subject blur correction amount based on a difference between the position of the first part in the image acquired before switching and the position of the second part in the image acquired after switching. The image blur correction device according to item 2, characterized by:

(Item 4)
When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. and calculating the subject blur correction amount based on the position of the first part in the image acquired before switching and the position of the second part in the image acquired before switching. The image blur correction device according to item 1.

(Item 5)
When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. and calculating the subject blur correction amount based on a difference between the position of the first part in the image acquired before switching and the position of the second part in the image acquired before switching. The image blur correction device according to item 4, characterized by:

(Item 6)
The subject blur correction amount calculation means calculates the position of the first part and the position of the second part from a provisional subject blur correction amount based on the difference between the position of the second part and the target position. 6. The image blur correction device according to item 3 or 5, wherein the subject blur correction amount is calculated by subtracting the difference as an offset.

(Item 7)
Item 7. The image according to item 6, wherein the subject blur correction amount calculation means adds the offset each time a part of the selected subject changes, and subtracts the added offset from the provisional subject blur correction amount. blur correction device.

(Item 8)
Item 8. The image blur correction apparatus according to item 7, wherein the subject blur correction amount calculation means gradually approaches the added offset to zero when a part of the subject with the highest priority is selected. .

(Item 9)
When the part of the subject selected by the selection means is switched from the first part to the second part, the position of the second part in the image before switching and the position of the first part in the image before switching 9. The image blur correction device according to item 8, wherein the target position is changed based on:

(Item 10)
subject detection means for detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
selection means for selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected by the subject detection means;
subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected by the selection means and a predetermined target position in the image; prepared,
An image blur correction device, wherein when the part of the subject selected by the selection means is switched from the first part to the second part, the target position is changed based on the position of the second part.

(Item 11)
changing the target position based on the position of the second part in the image acquired after switching when the part of the subject selected by the selecting means is switched from the first part to the second part; 11. The image blur correction device according to item 10.

(Item 12)
changing the target position based on the position of the second part in the image acquired before switching when the part of the subject selected by the selecting means is switched from the first part to the second part; 11. The image blur correction device according to item 10.

(Item 13)
camera shake correction amount calculation means for calculating a camera shake correction amount for correcting image blur caused by shake applied to an imaging device; the subject blur correction amount; and the camera shake correction amount calculated by the camera shake correction amount calculation means. 13. The image blur correction device according to any one of items 1 to 12, further comprising combining means for generating a combined image blur correction amount by combining the above.

(Item 14)
14. The image blur correction apparatus according to item 13, wherein the synthesizing unit adds and synthesizes the low frequency component of the subject blur correction amount and the high frequency component of the camera shake correction amount.

(Item 15)
15. The method according to any one of items 1 to 14, wherein the selection means can select a part of the subject to be used for correcting blurring of the subject independently of a part of the subject to be used for autofocus. Image blur correction device.

(Item 16)
16. The image blur correction device according to any one of items 1 to 15, wherein the selection means selects the face or the head with priority over the organs included in the face as the part of the subject.

(Item 17)
17. The image blur correction device according to any one of items 1 to 16, wherein the selection means selects the face or the head with priority over the body as the part of the subject.

(Item 18)
18. The image blur correction device according to any one of items 1 to 17, further comprising image blur correction means for correcting image blur based on the subject blur correction amount.

(Item 19)
19. An image blur correction apparatus according to item 18, wherein the image blur correction means has an imaging device that corrects image blur by being displaced in a direction perpendicular to the optical axis of the photographing optical system.

(Item 20)
19. The image blur correction device according to item 18, wherein the image blur correction means has a correcting lens that corrects image blur by being displaced in a direction different from the optical axis of the photographing optical system.

(Item 21)
imaging means for imaging a subject;
an image blur correction device according to any one of items 1 to 20;
An imaging device comprising:

(Item 22)
a subject detection step of detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
a selection step of selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected in the subject detection step;
a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image; have
When the part of the subject selected in the selection step switches from the first part to the second part,
In the subject blur correction amount calculation step,
Based on the difference between the position of the part of the selected subject and the target position, the position of the second part in the image, and the position of the first part in the image acquired before switching, A control method for an image blur correction device, comprising: calculating the subject blur correction amount.

(Item 23)
a subject detection step of detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
a selection step of selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected in the subject detection step;
a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image; have
Control of an image blur correction device characterized by changing the target position based on the position of the second part when the part of the subject selected in the selection step is switched from the first part to the second part. Method.

(Item 24)
A program for causing a computer to execute each step of the control method according to item 22 or 23.

(Item 25)
A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to item 22 or 23.

(Other embodiments)
In addition, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads the program. It can also be realized by executing processing. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: Imaging Apparatus 101: Optical System 102: Imaging Device 103: Development Processing Unit 104: Subject Detection Unit 105: Subject Part Selection Unit 106: Subject Blur Correction Amount Calculation Unit 701: Camera Shake Detection Unit 702 ; camera shake correction amount calculation unit 703: image blur correction amount synthesis unit

Claims (25)

subject detection means for detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
selection means for selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected by the subject detection means;
subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected by the selection means and a predetermined target position in the image; prepared,
The subject blur correction amount calculation means includes:
When the part of the subject selected by the selection means is switched from the first part to the second part, the difference between the position of the selected part of the subject and the target position, and the position of the second part in the image. The image blur correction device, wherein the subject blur correction amount is calculated based on the position and the position of the second part in the image acquired before switching. When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. and calculating the subject blur correction amount based on the position of the first part in the image acquired before switching and the position of the second part after switching. The image blur correction device described. When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. and calculating the subject blur correction amount based on a difference between the position of the first part in the image acquired before switching and the position of the second part in the image acquired after switching. 3. The image blur correction device according to claim 2, characterized by: When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. and calculating the subject blur correction amount based on the position of the first part in the image acquired before switching and the position of the second part in the image acquired before switching. 2. The image blur correction device according to claim 1. When the part of the subject selected by the selecting means is switched from the first part to the second part, the subject shake correction amount calculating means calculates the difference between the position of the selected part of the subject and the target position. and calculating the subject blur correction amount based on a difference between the position of the first part in the image acquired before switching and the position of the second part in the image acquired before switching. 5. The image blur correction device according to claim 4, characterized by: The subject blur correction amount calculation means calculates the position of the first part and the position of the second part from a provisional subject blur correction amount based on the difference between the position of the second part and the target position. 4. The image blur correction device according to claim 3, wherein the subject blur correction amount is calculated by subtracting the difference as an offset. 7. The subject blur correction amount calculation means according to claim 6, wherein each time a part of the selected subject changes, the offset is added, and the added offset is subtracted from the provisional subject blur correction amount. Image blur correction device. 8. The image blur correction according to claim 7, wherein said subject blur correction amount calculation means gradually brings said added offset closer to zero when a part of the subject with the highest priority is selected. Device. When the part of the subject selected by the selection means is switched from the first part to the second part, the position of the second part in the image before switching and the position of the first part in the image before switching 9. The image blur correction device according to claim 8, wherein the target position is changed based on and. subject detection means for detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
selection means for selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected by the subject detection means;
subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected by the selection means and a predetermined target position in the image; prepared,
An image blur correction device, wherein when the part of the subject selected by the selection means is switched from the first part to the second part, the target position is changed based on the position of the second part. changing the target position based on the position of the second part in the image acquired after switching when the part of the subject selected by the selecting means is switched from the first part to the second part; 11. The image blur correction device according to claim 10. changing the target position based on the position of the second part in the image acquired before switching when the part of the subject selected by the selecting means is switched from the first part to the second part; 11. The image blur correction device according to claim 10. camera shake correction amount calculation means for calculating a camera shake correction amount for correcting image blur caused by shake applied to an imaging device; the subject blur correction amount; and the camera shake correction amount calculated by the camera shake correction amount calculation means. 2. The image blur correction apparatus according to claim 1, further comprising synthesizing means for synthesizing the and to generate a synthetic image blur correction amount. 14. The image blur correction apparatus according to claim 13, wherein said synthesizing means adds and synthesizes a low frequency component of said subject blur correction amount and a high frequency component of said camera shake correction amount. 2. The image blur correction device according to claim 1, wherein said selection means can select a part of the subject to be used for correcting blurring of the subject independently of a part of the subject to be used for autofocus. 2. The image blur correction device according to claim 1, wherein said selection means selects the face or the head with priority over the organs included in the face as the part of the subject. 2. The image blur correction device according to claim 1, wherein the selection means selects the face or the head with priority over the body as the part of the subject. 2. The image blur correction device according to claim 1, further comprising image blur correction means for correcting image blur based on said subject blur correction amount. 19. The image blur correction apparatus according to claim 18, wherein said image blur correction means has an imaging device that corrects image blur by being displaced in a direction perpendicular to the optical axis of the imaging optical system. 19. The image blur correction device according to claim 18, wherein said image blur correction means has a correction lens that corrects image blur by being displaced in a direction different from the optical axis of the photographing optical system. imaging means for imaging a subject;
an image blur correction device according to any one of claims 1 to 20;
An imaging device comprising: a subject detection step of detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
a selection step of selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected in the subject detection step;
a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image; have
When the part of the subject selected in the selection step switches from the first part to the second part,
In the subject blur correction amount calculation step,
Based on the difference between the position of the part of the selected subject and the target position, the position of the second part in the image, and the position of the first part in the image acquired before switching, A control method for an image blur correction device, comprising: calculating the subject blur correction amount. a subject detection step of detecting each part of the subject from the input image and outputting the position of the detected part of the subject;
a selection step of selecting a part of the subject based on a predetermined priority from one or more parts of the subject detected in the subject detection step;
a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on the difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image; have
Control of an image blur correction device characterized by changing the target position based on the position of the second part when the part of the subject selected in the selection step is switched from the first part to the second part. Method. A program for causing a computer to execute each step of the control method according to claim 22 or 23. 24. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 22 or 23.

Images