JP4274233B2 - Imaging apparatus, image processing apparatus, image processing method therefor, and program causing computer to execute the method - Google Patents

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus that corrects blurring of a captured image, an image processing apparatus, an image processing method therefor, and a program that causes a computer to execute the method.

  2. Description of the Related Art Conventionally, imaging devices such as digital still cameras and digital video cameras, and imaging devices such as moving image and still image cameras having the functions of both have rapidly spread, and the performance of these imaging devices has been increasing. When shooting with these imaging devices held by hand, two types of blurring, namely blurring of the imaging device itself and blurring of the subject, mainly occur.

  As a method for detecting these shakes, (1) a shake detection method using a vibration sensor, speed sensor, acceleration sensor, gravity sensor, etc., and (2) a detection method such as a motion vector detection method using image processing are mainly known. It has been.

  Using the blur component detected by these detection methods, the blur in the shot image can be corrected by moving the position of the shot image in a direction to cancel the blur component. In addition, as a blur correction method, (3) electronic image blur correction processing such as shift of the read address of the image sensor, image memory write or read address, and (4) shift or tilt of the blur correction lens, Two types of correction methods for optical blur correction processing such as deformation and tilt of a blur correction prism and shift of an image sensor are mainly known.

For example, the image obtained by the image sensor based on the sampled camera shake detection information is sampled by sampling the camera shake detection information supplied from the camera shake sensor at the sample timing corresponding to the shutter speed among the sampling signals supplied from the timing generator. An imaging device that corrects camera shake by controlling a signal readout position has been proposed (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 05-316404 (FIG. 1)

  In the above-described conventional technology, the blur correction process is uniformly performed regardless of the subject included in the captured image. For this reason, it is possible to correct blur such as camera shake as the entire captured image.

  However, in these correction processes, since the background and the person included in the photographed image are treated uniformly, the blur component remains in the person, the face, eyes, nose, mouth, etc. that the photographer aimed at as the subject. It is possible to become. Therefore, it is considered that a face with higher sharpness can be photographed if the blur of the entire photographed image can be corrected and the blur of a person's face or the like included in the photographed image can be appropriately corrected.

  Accordingly, an object of the present invention is to appropriately correct blur included in an image.

The present invention has been made in order to solve the above problems, and a first aspect of the present invention is an imaging unit that captures an image of a subject, and a face detection unit that detects a face from the image captured by the imaging unit. And a face movement amount that is a blur amount of the face area detected by the face detection unit and a background movement amount that is a blur amount of an area other than the face region in the image captured by the imaging unit. A moving amount detecting unit; and a blur correcting unit that corrects a blur of an image captured by the imaging unit based on a face moving amount and a background moving amount detected by the moving amount detecting unit. An imaging apparatus, an image processing method thereof, and a program for causing a computer to execute the method. Thus, when a face is detected from the captured image, a face movement amount that is a blur amount of the face area in the captured image and a background movement amount that is a blur amount of an area other than the face area are detected, and the detected face is detected. This brings about the effect of correcting the blur of the captured image based on the movement amount and the background movement amount.

  Further, in the first aspect, a correction amount calculating means for calculating a correction amount of an image blur to be corrected by the blur correction means based on the face movement amount and the background movement amount detected by the movement amount detection means. Furthermore, it can comprise. Accordingly, there is an effect of calculating a blur correction amount of the captured image based on the face movement amount and the background movement amount detected from the captured image. In this case, the correction amount calculation means can calculate the correction amount based on the face movement amount and background movement amount detected by the movement amount detection means and a predetermined weighting coefficient. This brings about the effect that the amount of blur correction of the captured image is calculated based on the face movement amount and background movement amount detected from the captured image and the predetermined weighting coefficient. In this case, the correction amount calculation means uses the weighting coefficient value related to the face movement amount detected by the movement amount detection means rather than the weighting coefficient value related to the background movement amount detected by the movement amount detection means. The correction amount can be calculated by increasing it. Thereby, the weighting coefficient value related to the face movement amount is made larger than the weighting coefficient value related to the background movement amount, and the blur correction amount of the captured image is calculated.

  In the first aspect, the face detection unit detects various information about the face detected from the image captured by the imaging unit, and the correction amount calculation unit is detected by the face detection unit. The weighting coefficient can be calculated based on various information regarding the face. Thereby, various information regarding the face detected from the captured image is detected, and the weighting coefficient is calculated based on the various information regarding the detected face. In this case, the various information regarding the face includes the area of the face area, the coordinates of the face area, the value of the face-likeness, the degree of face orientation, the degree of inclination of the face, the degree of facial expression laughing, and the expression is serious. It may be information on any one of the degree, the degree of closing the eyes, or information combining these. As a result, the area of the face area, the coordinates of the face area, the value of facialness, the degree of face orientation, the degree of inclination of the face, the degree of facial expression smiling, the degree of seriousness of facial expression, the degree of closed eyes This brings about the effect that the weighting coefficient is calculated based on any one piece of information or information obtained by combining these pieces of information.

  In the first aspect, the correction amount calculating means can calculate the correction amount by increasing the value of the weighting coefficient in the vicinity of the face area detected by the face detecting means. This brings about the effect of increasing the value of the weighting coefficient in the vicinity of the face area detected from the photographed image and calculating the blur correction amount of the captured image.

  In the first aspect, the movement amount detection unit divides the image captured by the imaging unit into a plurality of regions so that the face detected by the face detection unit is included in a predetermined region. The face movement amount can be detected from the area including the face, and the background movement amount can be detected from the area other than the area including the face. Thus, the captured image is divided into a plurality of regions so that the face detected from the captured image is included in the predetermined region, and the face movement amount is detected from the region including the face and the region including the face is included. The effect of detecting the amount of background movement from the other area is brought about.

  Further, in the first aspect, the image processing apparatus further includes vibration detection means for detecting a shake amount due to vibration of the imaging device, and the shake correction means includes the face movement amount and the background movement amount detected by the movement amount detection means. And the amount of blur detected by the vibration detecting means can correct the blur of the image captured by the imaging means. This brings about the effect | action of correcting the blurring of a captured image based on the detected face movement amount and background movement amount, and the blurring amount due to the vibration of the imaging device.

  In the first aspect, the camera control value detection unit further detects a degree of focus or exposure of the face based on various information about the face detected by the face detection unit. And the correction amount calculation means calculates the correction amount related to the face based on the degree of focus or the degree of exposure of the face detected by the camera control value detection means. it can. This brings about the effect that the correction amount related to the face is calculated based on the degree of focus or the degree of exposure of the face detected from the captured image.

  In the first aspect, the image forming apparatus may further include setting means for setting whether or not to detect a face from the image picked up by the image pickup means. Thereby, whether or not to detect a face from the captured image is set, and the presence or absence of blur correction of the captured image in consideration of the face movement amount is selected.

  In the first aspect, the specific person storage means for storing specific person information relating to the face of the specific person, and the face detected by the face detection means are stored in the specific person storage means. Specific person determining means for determining whether or not the face matches the face, and the correction amount calculating means includes a specific person whose face detected by the face detecting means is stored in the specific person storing means. When it is determined by the specific person determination means that the face matches, the correction amount can be calculated by changing the weighting coefficient according to whether or not the area is the face of the specific person. As a result, when it is determined that the face detected from the captured image matches the face of the specific person, the weighting coefficient is changed depending on whether or not the face area of the specific person is, and blurring of the captured image is performed. The correction amount is calculated.

  In the first aspect, the correction amount calculation unit calculates a first average value that is an average value of the face movement amount detected by the movement amount detection unit, and calculates the first average value based on the first average value. Based on the second average value by calculating a correction amount related to the face area detected by the face detection means and calculating a second average value that is an average value of the background movement amounts detected by the movement amount detection means. Thus, it is possible to calculate a correction amount relating to an area other than the face area detected by the face detecting means. Thus, the average value of the detected face movement amount is calculated, the correction amount related to the face area is calculated based on the average value, and the average value of the detected background movement amount is calculated and the average value The correction amount relating to the area other than the face area is calculated based on the above.

  In the first aspect, the correction amount calculating means calculates an average value of face movement amounts for each detected face area when the face detecting means detects a plurality of faces. The correction amount can be calculated based on the average value. Thereby, when a plurality of faces are detected, the average value of the face movement amount is calculated for each face region, and the correction amount is calculated for each face region based on the average value. .

  Further, in this first aspect, the blur correction unit causes the blur of the image so that the face area detected by the face detection unit moves closer to the center of the image in the image captured by the imaging unit. Can be corrected. This brings about the effect of correcting the blur of the captured image so that the face area detected in the captured image moves closer to the center of the image.

  Further, in this first aspect, the movement amount detection unit is configured to detect the face in an image captured by the face detection unit in an image captured by the imaging unit and an image captured before a predetermined time of the image. Based on the high correlation with the face detected by the means, it is possible to detect the face movement amount related to the face area. Accordingly, the face movement amount of the face area is detected based on the high degree of correlation between the face detected in the captured image and the face detected in the image captured a predetermined time before the image. Bring.

  According to a second aspect of the present invention, there is provided an image input means for inputting an image, a face detection means for detecting a face from the image input by the image input means, and the image in the image input by the image input means. A movement amount detection means for detecting a face movement amount that is a blur amount of a face area detected by the face detection means and a background movement amount that is a blur amount of an area other than the face area; and the movement amount detection means. An image processing apparatus comprising: a blur correction unit that corrects a blur of an image input by the image input unit based on the detected face movement amount and background movement amount; A program that causes a computer to execute a method. Thus, when a face is detected from the input image, a face movement amount that is a blur amount of the face area in the input image and a background movement amount that is a blur amount of an area other than the face area are detected, and the detected face is detected. This provides an effect of correcting the blurring of the input image based on the movement amount and the background movement amount.

  According to the present invention, it is possible to achieve an excellent effect that blur included in an image can be corrected appropriately.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a functional configuration example of an imaging apparatus 100 according to an embodiment of the present invention. Examples of the imaging device 100 include an imaging device such as a digital still camera or a digital video camera, or an imaging device such as a camera attached to a terminal device such as a mobile phone or a personal computer.

  The imaging apparatus 100 includes a control unit 110, an imaging unit 120, a signal processing unit 130, an image storage unit 140, an image compression / decompression unit 150, a user interface 160, an input / output terminal 170, a storage medium 180, A bus 190, a vibration sensor 200, a memory 210, a face detection unit 300, a motion vector detection unit 400, and a shake correction unit 410 are provided. Note that transmission / reception of image signals between the blocks is performed by direct exchange or exchange via the image storage unit 140 or the bus 190.

  The control unit 110 is a control unit that controls each unit of the imaging apparatus 100 based on various control programs stored in a memory (not shown).

  The imaging unit 120 includes an optical system 121 and an imaging element 122, and photoelectrically converts incident light from a subject into an electrical signal and outputs the photoelectrically converted electrical signal to the signal processing unit 130. The optical system 121 includes a lens group such as a focus lens, a zoom lens, a blur correction lens, and a blur correction prism, and outputs incident light from a subject input through these lens groups to the image sensor 122. It is. The image sensor 122 photoelectrically converts incident light output from the optical system 121 and outputs the photoelectrically converted electric signal to the signal processing unit 130. In the imaging unit 120, the detected blur can be optically corrected by moving or tilting the blur correction lens, deforming or tilting the blur correction prism, moving the image sensor 122, or the like. These are executed based on control from the control unit 110.

  The signal processing unit 130 performs various types of signal processing on the electrical signal output from the image sensor 122, and the image data subjected to the signal processing is stored in the image storage unit 140, the image compression / decompression unit 150, the user interface 160, and the face. This is output to the detection unit 300. Signal processing in the signal processing unit 130 includes signal processing such as noise reduction processing, level correction processing, A / D conversion processing, and color correction processing. In addition, the signal processing unit 130 performs various image processing on the image input from each unit based on an instruction from the control unit 110.

  The image storage unit 140 stores image data to be processed in the imaging apparatus 100.

  The image compression / decompression unit 150 compresses or decompresses various input image data according to each image processing. For example, the image data subjected to the compression processing by the image compression / decompression unit 150 is output to the storage medium 180 and recorded on the storage medium 180. Further, the image data subjected to the decompression process by the image compression / decompression unit 150 is output to the image storage unit 140, the display unit 161, and the face detection unit 300. As a compression format, for example, a JPEG (Joint Photographic Experts Group) format can be adopted.

  The user interface 160 includes a display unit 161 and a selection receiving unit 162, and provides an interface for a user who uses the imaging apparatus 100.

  The display unit 161 is a display unit that displays an image corresponding to the image data output from the signal processing unit 130 or the image compression / decompression unit 150. The display unit 161 displays, for example, a captured image that is an image of a subject captured by the imaging device 100.

  The selection receiving unit 162 converts selection information input by the user into an electric signal and outputs the converted electric signal to the control unit 110. For example, when the face detection unit 300 detects a face from the captured image output from the imaging unit 120, the imaging apparatus 100 executes a shake correction process based on the face detected for the captured image. As described above, when a captured image includes a face, it is possible to set (ON setting) to execute the shake correction process based on the face, and to not execute the shake correction process based on the face. (OFF setting) can be performed. The ON / OFF setting is performed in the selection receiving unit 162.

  In addition, the user interface 160 may be configured such that the display unit 161 and the selection receiving unit 162 are integrated as a touch panel, for example. The display unit 161 is a liquid crystal display (LCD), and the selection receiving unit 162 is a hard key. You may make it comprise both separately.

  The input / output terminal 170 outputs the image data output from the image compression / decompression unit 150 to an external device such as an external storage medium, and outputs the image data input from the external storage medium to the image compression / decompression unit 150. It is.

  The storage medium 180 is an image storage medium that stores the image data output from the image compression / decompression unit 150 and outputs the stored image data to the image compression / decompression unit 150. Examples of the image storage medium include an image storage medium such as a magnetic disk, an optical disk, a semiconductor storage medium, and a magnetic tape. The image storage medium is at least one of an externally removable storage medium and a built-in storage medium.

  The bus 190 is a shared bus for transmitting image data.

  The vibration sensor 200 detects vibration of the imaging apparatus 100 and detects a blur component that does not depend on an image (that is, a blur component of the main body of the imaging apparatus 100), and outputs various detected information to the control unit 110. . The vibration sensor 200 can be realized by, for example, a gyro sensor, a speed sensor, or an acceleration sensor.

  The memory 210 is a volatile / nonvolatile storage medium that stores various types of information.

  The face detection unit 300 detects a human face included in an image corresponding to input image data. Details of the face detection unit 300 will be described with reference to FIG.

  The motion vector detection unit 400 detects a motion vector of each block divided for an image corresponding to input image data by image processing, and outputs a value corresponding to the detected motion vector to the control unit 110. is there.

  The blur correction unit 410 moves the position of the image corresponding to the input image data based on the blur correction amount calculated based on the motion vector detected by the motion vector detection unit 400, and blurs the image. Is to correct. Note that the blur correction unit 410 includes an electronic blur correction unit and an optical blur correction unit. Details thereof will be described with reference to FIG.

  Note that an image whose blur is corrected by the blur correction unit 410 is output to the image compression / decompression unit 150. Then, the image compression / decompression unit 150 performs image compression processing on the image whose blurring has been corrected, and records the image subjected to the image compression processing in the storage medium 180. In addition, an image whose blur is corrected is displayed on the display unit 161.

  FIG. 2 is a block diagram illustrating a functional configuration example of the face detection unit 300.

  The face detection unit 300 includes a control unit 310, an image input unit 320, an image enlargement / reduction unit 330, an image holding unit 340, a reference data holding unit 350, a determination unit 360, a detection result output unit 370, an image And an output unit 380.

  The control unit 310 controls each unit of the face detection unit 300 in accordance with instructions from the control unit 110.

  When image data is input from any of the signal processing unit 130, the image storage unit 140, and the image compression / decompression unit 150, the image input unit 320 outputs the input image data to the image enlargement / reduction unit 330. .

  The image enlargement / reduction unit 330 performs an enlargement or reduction process on an image suitable for face detection on the image corresponding to the image data output from the image input unit 320. The image enlargement ratio or reduction ratio is determined based on an instruction from the control unit 110.

  The image holding unit 340 holds an image that has been enlarged or reduced by the image enlargement / reduction unit 330.

  The reference data holding unit 350 holds face reference data used for face detection, and outputs the held face reference data to the determination unit 360. Here, the reference face data is, for example, a face image itself, a feature database as a human face, a feature database related to a specific person's face, or the like. In general, the feature data can hold many face databases with a relatively small storage capacity than the face image itself.

  The determination unit 360 determines whether or not a face is included in the image corresponding to the image data held in the image holding unit 340, and outputs the determination result to the detection result output unit 370. Specifically, the determination unit 360 partially extracts an image corresponding to the image data stored in the image storage unit 340 with a certain window size, and stores the extracted image and the reference data storage unit 350. Whether or not the extracted image is a face image is determined based on the high correlation with the face data. These determinations are repeatedly executed to determine whether or not a face is included in the image corresponding to the image data held in the image holding unit 340.

  Further, the determination unit 360 is included in the image corresponding to the image data held in the image holding unit 340 based on the high degree of correlation with the face data held in the reference data holding unit 350. Various information about the face is extracted, and the extracted various information is output to the detection result output unit 370 as a face detection result.

  When the detection result output unit 370 receives the determination result and the face detection result output from the determination unit 360, the detection result output unit 370 outputs the received determination result and face detection result to the control unit 110.

  The image output unit 380 outputs the image held in the image holding unit 340 to the image storage unit 140.

  Here, the value of the face detection result output by the face detection unit 300 will be described.

As face detection result values output by the face detection unit 300, there are evaluation values related to face areas such as (1) to (6) shown below.
(1) Face size (distance from camera)
(2) Face position (distance from image center)
(3) Face-like value (4) Lateral degree of face (front direction, up / down / left / right direction)
(5) Degree of face inclination (the neck is bent)
(6) Up / Down / Left / Right

In addition, as face detection result values other than (1) to (6), there are evaluation values related to facial expressions, face states, etc., such as (7) to (9) shown below.
(7) Degree of smile (8) Degree of serious face (serious face) (9) Degree of eyes closed (winning)

  A weighting coefficient or the like for a motion vector can be calculated using an evaluation value or the like related to these face regions.

  FIG. 3 is a block diagram illustrating a functional configuration example related to image blur correction in the imaging apparatus 100 according to the embodiment of the present invention.

  The imaging apparatus 100 includes an image input unit 420, a face detection unit 300, a motion vector detection unit 400, a shake correction unit 410, an image output unit 430, a correction amount calculation unit 440, and a selection reception unit 162. .

  The image input unit 420 inputs an image, and outputs the input image to the face detection unit 300, the motion vector detection unit 400, and the shake correction unit 410.

  The face detection unit 300 is the same as the face detection unit 300 shown in FIGS. 1 and 2, detects a face from the image output from the image input unit 420, and uses the detection result as the motion vector detection unit 400. And the correction amount calculation unit 440. Further, the face detection unit 300 executes the face detection process when the ON setting is output from the selection receiving unit 162 for the shake correction process based on the face included in the photographed image, and when the OFF setting is output. No face detection process is executed.

  The motion vector detection unit 400 detects a motion vector for the image output from the image input unit 420 by image processing, and outputs the detected motion vector to the correction amount calculation unit 440. The motion vector detected by the motion vector detection unit 400 will be described in detail with reference to FIGS.

  The correction amount calculation unit 440 calculates a shake correction amount by applying a weighting process to the motion vector output from the motion vector detection unit 400 based on various information about the face detected by the face detection unit 300, and the like. The correction amount is output to the blur correction unit 410. Note that the blur correction amount calculated by the correction amount calculation unit 440 will be described in detail with reference to FIGS.

  The blur correction unit 410 moves the position of the image output by the image input unit 420 based on the blur correction amount output by the correction amount calculation unit 440 and corrects the blur of the image.

  The image output unit 430 outputs an image whose shake has been corrected by the shake correction unit 410. The image output unit 430 corresponds to, for example, the display unit 161 that displays a captured image. For example, it corresponds to an input / output terminal 170 that outputs image data corresponding to a captured image to another external apparatus that displays the captured image. Note that image data corresponding to a captured image can be output to an image storage medium such as the storage medium 180 and the captured image can be stored in the external device.

  The selection receiving unit 162 is the same as the selection receiving unit 162 shown in FIG. 1, and outputs selection information related to ON setting or OFF setting related to face blur correction input by the user to the face detection unit 300. .

  Next, as one of motion vector detection methods, a motion vector detection method using block matching image processing will be described with reference to the drawings.

  FIG. 4 is a diagram illustrating an example of image division in the case where the motion vector detection unit 400 detects a motion vector from a captured image by block matching image processing.

  As shown in FIG. 4, the motion vector detection method based on block matching image processing divides a temporally continuous frame image (image 500) constituting a moving image into small blocks, which are similar to the divided blocks. This is a method of searching where the current image block is in the previous frame and detecting a motion vector in each block divided based on the search result.

  Specifically, when the motion vector detection unit 400 detects a motion vector from the image 500 by block matching image processing, the image 500 is divided into a plurality of regions. For example, the simplest image division is a division that equally divides the image vertically and horizontally, and the image 500 can be divided into 16 as shown in FIG. In addition, target blocks are arranged in each region in the divided image, and a search range is set to the maximum amount of motion assumed for each arranged target block. Then, a motion vector is calculated by the target block within the set search range. For example, as shown in FIG. 4, target blocks 501, 503 to 517 are arranged in each region obtained by dividing an image 500 into 16 parts, and a search range is provided for each target block. The target block is an area for searching for a search target within the search range. For example, the target block 501 searches the search range 502.

  FIG. 5 is a diagram illustrating an example of the relationship between the previous frame 540 and the current frame 550 when the motion vector detection unit 400 detects a motion vector from a captured image by block matching image processing.

  The previous frame 540 and the current frame 550 are the current frame and the previous frame of a predetermined time before the temporally continuous frame images constituting the moving image. Note that a target block 541 is disposed in the previous frame 540. In the current frame 550, the target block 551 is projected to the same coordinates where the target block 541 is arranged, and a search range 552 is set around the target block 551. It should be noted that the previous frame 540 and the current frame 550 shown in FIG. 5 and the target blocks, search range, and the like included in these are enlarged for explanation. Further, each search range may be set overlappingly.

  Subsequently, the target block 551 is moved in the search range 552 set in the current frame 550. For example, the areas 551a to 551c and the like are moved in the search range 552. Then, in the search range 552, an area having the highest correlation with the target block 541 included in the previous frame 540 is searched. For example, a region having the highest correlation with the target block 541 in the search range 552 is set as a region 553.

  As a result of this search, when a region having the highest correlation with the target block 541 is detected in the search range 552, the target block 541 included in the previous frame 540 is detected in the search range 552 of the current frame 550. It is estimated that the position has moved to the position of the area 553. Then, based on the positional relationship between the target block 551 and the region 553 in the search range 552 set in the current frame 550, a motion vector 554 is obtained. In this way, in the motion vector detection method using block matching image processing, one motion vector is calculated for one target block. That is, correlation determination (matching determination) processing between frames is performed for each divided block, and a motion vector is obtained for each block.

  As shown in FIG. 4, for one image 500, a motion vector in each target block is detected, a predetermined averaging process is performed on the detected motion vector, and one global in one image 500 is detected. Simple motion vectors are detected. In addition, the motion vector detected in each target block is weighted, and a local motion vector for each target block is detected.

  Here, as shown in FIG. 4, a case where the face area 603 of the face 602 is detected by the face detection unit 300 when the face 602 of the person 601 is included in the image 500 will be considered. For example, as illustrated in FIG. 4, when a face area 603 detected by the face detection unit 300 is included in the target block 511, a motion vector for the target block 511 including the face area 603 is detected. As described above, since one motion vector can be detected for the face area 603, for example, even when the human 601 is moving, an appropriate motion vector for the face area 603 is detected. Can do.

  On the other hand, as shown in FIG. 6A, when the face 605 of the person 604 is included in the image 560, the face area 606 of the face 605 is detected by the face detection unit 300, and the image 560 becomes 16 Consider the case where it is divided. In this case, the face area 606 detected by the face detection unit 300 is not included in one target block, but is included in a plurality of target blocks 563 to 566. Then, motion vectors for each of the plurality of target blocks 563 to 566 including the face area 606 are detected. For this reason, for example, when the person 604 is moving, an appropriate motion vector may not be detected. Therefore, in the embodiment of the present invention, when the motion vector detection unit 400 detects a motion vector from the captured image, the captured image is divided based on the face area detected by the face detection unit 300 to detect the motion vector. To do.

  FIG. 6 is a diagram illustrating an example of image division when the motion vector detection unit 400 detects a motion vector from a captured image by block matching image processing.

  FIG. 6A is a diagram illustrating an example of image division in the case where the image 560 is divided into 16 as in the case of the image 500 illustrated in FIG. 4. As described above, when the image 560 including the person 604 is divided without considering the face area 606 detected by the face detection unit 300, the face area 606 is included in the plurality of target blocks 563 to 566. Will be.

  FIG. 6B is a diagram illustrating an image division example when the image 560 is divided into nine based on the face region 606 detected by the face detection unit 300. As described above, by dividing the image 560 including the human 604 so that the face area 606 detected by the face detection unit 300 is included in one target block 573, one face area 606 is obtained. A motion vector can be detected.

  As described above, in the image 560 shown in FIG. 6A, the face area intersects with the boundary of image division, and therefore one motion vector cannot be detected for one face area. On the other hand, in the image 560 shown in FIG. 6B, since the boundary of the image division and the face area do not intersect, one motion vector can be detected for one face area. This is preferable as a method for setting the boundary. As described above, by moving the boundary of the image division to a position where any target block includes the face area, it is possible to reliably detect an optimal motion vector for the face area. When the target block is arranged, other arrangement methods other than equally dividing the image can be used as long as the target block is arranged so as to surround the detected face.

  As described above, in the embodiment of the present invention, the coordinates of the face area output by the face detection unit 300 are used in the process of detecting the motion vector from the captured image.

  In order to intensively correct the blur component (blur amount) in the face area, a relatively large weighting process is executed on the target block including the face area to increase the contribution to the calculation of the entire motion vector. As shown below, this weighting process can be precisely set according to the size of the face area, the certainty of the face, the degree of frontal orientation, and the like.

  FIG. 7 is a diagram illustrating an example of a motion vector detected from the image 580 by the motion vector detection unit 400.

  In the image 580, as in the image 500 shown in FIG. 4, the image 580 is divided into 16 and the motion vector in each block is detected. Note that the image 580 includes humans 607 and 610, and the face area 609 of the face 608 and the face area 612 of the face 611 are detected by the face detection unit 300. For this reason, each target block is arranged so as to include each detected face area 609 and 612. Arrows 581v, 584v, etc. show examples of motion vectors detected in the target blocks 581 to 597. As shown in FIG. 7, a motion vector is detected in each target block.

  Next, a weighting coefficient calculation method for weighting the detected motion vector will be described with reference to the drawings.

  First, a case where a weighting coefficient is calculated based on various types of information related to the detected face will be described.

  FIG. 8 is a diagram illustrating an image 701 including human faces 613, 615, and 617. As described above, when a motion vector is detected in a captured image, the detected motion vector is not used as it is as a blur correction amount, and weighting processing is performed according to the subject, and the motion after the weighting processing is performed. Perform blur correction using vectors. For example, in the case of a captured image including a human face, the weighting coefficient is calculated according to whether or not the target block in the captured image includes a face area. When the target block includes a face area, a larger weighting coefficient is calculated for the target block in order to intensively correct image blurring in the vicinity of the face area.

For example, a case where the number of target blocks in a captured image is N will be described. When the target block in the captured image does not include a face area, a predetermined weighting coefficient wn ′ = w0 (n = 0 to N−1) is calculated. However, the initial value w0 of this weighting coefficient is w0 = 1 / N. On the other hand, when the target block in the captured image includes a face area, a weighting coefficient is calculated using a plurality of weighting factors as shown in the following (Equation 1). E01 to E05 are weighting coefficients for the respective face evaluation values.
wn ′ = w0 × {(E01 × face area)
+ (E02 x face area coordinates)
+ (E03 x facial value)
+ (E04 x degree of face orientation)
+ (E05 × degree of face inclination)} (Formula 1)

  Specifically, for the face area included in the target block, the weight is increased as the area of the face area is wider, the weight is increased as the coordinates of the face area are closer to the center, and the weight is increased as the face-likeness value is higher. The weight is increased as the degree of frontal orientation of the face is higher and the weight is increased as the face inclination is smaller.

  When the weighting coefficients w0 ′ to w (N−1) ′ are calculated from the N target blocks, normalization processing is performed so that the sum of the weighting coefficients becomes 1.

  For example, as shown in FIG. 8, in an image 701 including a plurality of human faces 613, 615, and 617, the human face 613 included in the face area 614 has a large area of the face area, and its coordinates are The weighting coefficient is high because it is closer to the center, has a high degree of face orientation, and has a small face inclination. The human face 617 included in the face area 618 has a low weighting coefficient because the face area has a relatively small area, its coordinates are not close to the center, and the face inclination is relatively large. Furthermore, the human face 615 included in the face area 616 has a relatively low weight area because the area of the face area is relatively small, the coordinates are not close to the center, and the degree of frontal orientation of the face is low.

  Next, a case where the weighting coefficient is calculated based on the degree of focus and exposure of the detected face will be described.

  FIGS. 9 to 12 are diagrams illustrating examples of captured images including a plurality of human faces.

  As shown in FIG. 9, for example, in a captured image including a plurality of human faces with different distances from the imaging device 100, weighting coefficients can be calculated based on various types of face information, The weighting coefficient can be calculated based on the degree of focus and exposure of the face. Also, the weighting coefficient calculated based on various face information can be corrected based on the degree of focus and exposure of each face.

  FIG. 9A is a diagram illustrating an image 702 including a plurality of human faces 619, 621, and 623 having different distances from the imaging device 100. In the image 702, face areas 620, 622, and 624 including the faces 619, 621, and 623 are detected.

  As a method of detecting the degree of focus, an AF detection frame is provided in the detected face area, a high frequency component is obtained inside the frame, and the higher the high frequency component integrated value, the more the focus is considered. There is a way. For example, in the image 702 illustrated in FIG. 9A, the face area that is most focused is set as the face area 622. In this case, the weight of the weighting coefficient of the target block related to the face area 622 in the image 702 is increased.

  FIG. 9B is a diagram illustrating an image 703 including a plurality of human faces 625, 627, and 629 having different distances from the imaging device 100. In the image 703, face areas 626, 628, and 630 including the faces 625, 627, and 629 are detected.

  As a method for detecting the degree of exposure, there is a method in which an appropriate exposure can be set in advance, and the closer the value is, the more the exposure is considered. For example, in the image 703 shown in FIG. 9B, the most exposed face area is set as the face area 626. In this case, the weight of the weighting coefficient of the target block related to the face area 626 in the image 703 is increased.

  FIG. 10 is a diagram showing an image 704 (so-called group photo) including many human faces. In the image 704, all the other face areas are detected together with the face area 633 including the face 632 of the human 631.

  As shown in FIG. 10, in an image 704 like a so-called group photo, there are many face areas having a relatively small area, and a face area having a large area does not appear. In many cases, many of the faces present in the image 704 are facing the front. For this reason, when it is determined that the captured image is a so-called group photo image, the weighting for each face region can be made uniform, and the weighting coefficient can be set lower. Thereby, it can correct | amend to the weighting suitable for the image of what is called a group photograph.

  FIG. 11 is a diagram showing an image 705 in which other persons 638 and 640 are included in the background together with persons 634 and 636 as main subjects.

  As shown in FIG. 11, in the image 705 in which other people 638 and 640 exist in the background in addition to the main subjects 634 and 636, the areas of the face regions of the main subjects 634 and 636 are compared. In many cases, the face area is present near the center of the image. On the other hand, for the other persons 638 and 640 in the background, the area of the face area is relatively small, and the face area often exists near the periphery of the image. In many cases, the person who is the main subject has a high probability of facing the front and the other persons in the background have a low probability of facing the front. For this reason, when it is determined that the person is a person in the background in consideration of these, the weighting coefficient for the face area of the person in the background can be set lower. Thereby, it can correct | amend to the weighting suitable for the person of a background.

  FIG. 12 is a diagram illustrating an image 706 including a plurality of human faces 642, 644, 646.

  As shown in FIG. 12, for example, when there are a plurality of face areas 643, 645, and 647 in the image 706, there is a face area recognized as a specific person (known person) from the plurality of face areas. If so, a weighting coefficient larger than that of the unknown face area is given. As a result, it is possible to realize the optimum blur correction for the specific person.

  For example, as illustrated in FIG. 12, when a plurality of face areas 643, 645, and 647 exist in the image 706, the face area of the specific person is set as the face area 643. In this case, with respect to the weighting coefficient of the target block related to the face area 643, the largest weighting coefficient is given to the specific person in preference to the area of the face area, the degree of frontal orientation, and the like.

  Note that the specific person is a person specified by the user of the imaging apparatus 100. Various data relating to the face area of the person is stored in the imaging apparatus 100 as a specific person database. The various data includes, for example, the face contour, the shape, characteristics, position, etc. of the face components (eyes, nose, eyebrows, mouth, ears, hair, beard, glasses, etc.). In addition, specific person data may be given to the imaging apparatus 100 from the outside. The specific person database related to the face of the specific person is compared with various data in the face area detected from the photographed image, and the correlation value is calculated to determine whether or not the same person. If there is, it is determined that the person is a specific person. Regarding the recognition of a specific person using such a face image, a technique described in Japanese Patent Publication No. 6-7388 is well known.

  As described above, when one or more faces are detected from a captured image, it is possible to express numerically which face is an important subject and how the face detection results It is possible to accurately reflect the calculation of the weighting coefficient of the vector.

  Next, the operation of the imaging apparatus 100 according to the embodiment of the present invention will be described with reference to the drawings.

  FIG. 13 is a flowchart illustrating a processing procedure of blur correction amount calculation processing by the imaging apparatus 100.

  First, a captured image that is an image captured by the image capturing unit 120 in the image capturing apparatus 100 is input (step S901). When a captured image is input from the imaging unit 120, it is determined whether the face detection setting is turned on (step S902). If the face detection setting is OFF (step S902), since it is not necessary to perform face-based blur correction, the process proceeds to step S912.

  If the face detection setting is ON (step S902), the face detection unit 300 detects a face from the input image and calculates various information of each detected face (step S903).

  Subsequently, it is determined whether or not a face is detected from the input image (step S904). If no face is detected from the input image (step S904), it is not necessary to perform blur correction based on the face, and the process proceeds to step S912.

  If a face is detected from the input image (step S904), the input image is divided into a plurality of blocks so as to include each detected face region (step S905). For this division, as shown in FIG. 6B, the boundary of each block is set so that the face area is not divided. For example, assume that the input image is divided into N blocks. If a plurality of faces are detected from the input image, one or more faces that are considered to be main are selected based on the evaluation value of each detected face, and each face of the selected face is selected. The input image may be divided into a plurality of blocks so as to include the region.

  Subsequently, one target block is extracted from the plurality of target blocks (step S906). Subsequently, a motion vector relating to the extracted target block is detected and stored (step S907). Subsequently, it is determined whether or not a face is included in the extracted target block (step S908). If a face is included in the extracted target block (step S908), a weighting coefficient is calculated based on various types of face information included in the target block (step S909). For example, when the face is included in the target block, the weighting coefficient in the target block is calculated using (Equation 1) described above (step S909). In this way, when a face is included in the target block, a relatively large weighting coefficient is calculated for the target block, and blurring of an image near the face area of the face included in the target block is performed. It can be corrected intensively. In the case where the weighting coefficient in the target block is calculated in step S909, the weighting coefficient calculation method described with reference to FIGS. 9 to 12 may be used.

  On the other hand, if the extracted target block does not contain a face (step S908), a predetermined weighting coefficient is calculated for the target block (step S910). For example, when there are N target blocks and wn ′ = w0 = 1 / N is calculated as the weighting coefficient when the initial value of the weighting coefficient determined in advance is w0 (n = 0 to N). -1).

  Subsequently, it is determined whether all target blocks of the plurality of target blocks have been extracted (step S911). If all the target blocks have not been extracted (step S911), the process proceeds to step S906, and the process of calculating the weighting coefficient for each target block is repeated (steps S906 to S910).

  If all target blocks have been extracted (step S911), normalization processing is performed so that the sum of weighting coefficients w0 ′ to w (N−1) ′ in the calculated N target blocks is 1. Is executed. Subsequently, a weighting process is performed on the detected motion vector in each target block using the weighting coefficients w0 ′ to w (N−1) ′ in each target block that has been subjected to normalization processing, and a frame related to each target block The motion vector at is calculated (step S915). Subsequently, the calculated motion vector of each frame is output to the blur correction unit 410 as a correction amount (step S916).

  If the face detection setting is ON (step S902) or if no face is detected from the input image (step S904), the input image is divided into a plurality of target blocks (step S912). For example, the input image is equally divided into N target blocks.

  Subsequently, one target block is sequentially extracted from the plurality of target blocks, and motion vectors relating to the extracted target blocks are sequentially detected and stored (step S913). Subsequently, a predetermined weighting coefficient is calculated for each target block (step S914). For example, in the case where there are N target blocks, wn ′ = w0 = 1 / N is calculated as the weighting coefficient when the initial value of the weighting coefficient determined in advance is w0. Then, it progresses to step S915 and the motion vector in the frame which concerns on each target block is calculated.

  As described above, the motion vector detected in each frame related to the N target blocks is subjected to the weighting process using the weighting coefficient in consideration of the face area included in each target block, and finally 1 per frame. One motion vector is calculated. This motion vector is output to the shake correction unit 410 and corrected in the direction opposite to the shake. As a result, a good image with sufficiently small blurring can be obtained for the output image. In particular, the blurring is suppressed on the face portion of the person rather than the background, and the main subject can be photographed sharply.

  Next, the blur correction unit 410 that corrects the blur of the captured image based on the blur correction amount calculated by the correction amount calculation unit 440 will be described with reference to FIG.

  As described above, the shake correction unit 410 includes an electronic shake correction unit and an optical shake correction unit. The electronic blur correction unit corrects blur by shifting the address of the image sensor 122 to perform writing or reading operation, or blurs by shifting the address of the image storage unit 140 and performing writing or reading operation. It is a means to correct. The optical blur correction unit is a unit that corrects blur by movement or tilt of the blur correction lens, a unit that corrects blur by deformation or tilt of the blur correction prism, or a blur by moving the image sensor 122. It is a means to correct.

  The electronic blur correction unit and the optical blur correction unit calculate one motion vector from a plurality of local motion vectors obtained for one image, and correct the blur of the image based on the vector. can do. In addition, the electronic blur correction unit can perform local correction from a plurality of local motion vectors obtained for one image. The blur correction unit 410 includes at least an electronic blur correction unit.

  FIG. 14 is a diagram illustrating motion vectors 751 to 759 calculated from each block divided into nine in an image 707 including a person 648. In the image 707, motion vectors 751 to 758 are motion vectors (global vectors) of blocks determined as background portions in the image 707, and are the same motion vectors. On the other hand, since the motion vector 759 is a motion vector (local vector) in a block including the face area 650 of the face 649 of the human 648 that is a moving subject, the motion vector is different from the motion vectors 751 to 758. Note that the motion vector 759 is an example of a face movement amount that is a blur amount of a face area, and the motion vectors 751 to 758 are examples of a background movement amount that is a blur amount of an area other than the face area.

  For the image 707 shown in FIG. 14, the electronic shake correction unit of the shake correction unit 410 performs the same local correction because the blocks related to the motion vectors 751 to 758 are the same motion vector. On the other hand, regarding the block related to the motion vector 759, since the motion vector 759 is a motion vector different from the motion vectors 751 to 758, local correction according to the motion vector 759 is performed.

  As a result, a sharp image with less blur can be taken with respect to the human face as the subject. Particularly in the vicinity of a human face, blurring can be sufficiently suppressed, and a main subject can be photographed with high image quality. In addition, it is possible to effectively achieve blur correction by automatically focusing on a person's face without being affected by background blur or movement of an object on the background. The embodiment of the present invention is not limited to a specific means for the shake correction means, and can be applied to other shake correction means regardless of electronic correction or optical correction means. it can.

  Next, a case where blurring in a captured image is corrected using a blur component (blur amount) 201 in the main body of the imaging apparatus 100 output from the vibration sensor 200 will be described with reference to FIG.

  FIG. 15 shows the subject based on the blur component (blur amount) 201 of the image pickup apparatus 100 output from the vibration sensor 200 and the blur component (blur amount) 401 in each part of the image output from the motion vector detection unit 400. It is a figure which shows an example of the extraction method in the case of extracting this blur component.

  As shown in FIG. 15, when the blur component 201 of the main body of the imaging device 100 output by the vibration sensor 200 is subtracted from the blur component 401 in each part of the image output by the motion vector detection unit 400, the blur component 402 of the moving subject is obtained. Can be calculated. That is, since the blur component (blur component due to camera shake) of the main body of the imaging apparatus 100 can be removed from the blur component in each part of the captured image, the blur component of the moving subject can be calculated. The calculated blur component can be used for blur component detection in the face area.

  FIG. 16 illustrates an image based on the motion vector in each part of the image 708 detected by the motion vector detection unit 400 and the motion vector of the human 651 that is a moving subject included in the image 708 detected by the motion vector detection unit 400. 708 is a diagram illustrating an example of an extraction method when a blur component of a background portion other than a moving subject included in 708 is extracted.

  As shown in FIG. 16, when the motion vector related to the region 651a estimated as the face region and the body portion of the human 651 that is the moving subject is subtracted from the motion vector of each portion of the image 708, the moving subject is affected. It is possible to extract a motion vector of only a pure background.

  Thus, by using the vibration sensor 200, it is possible to detect a blur component that does not depend on image processing (that is, a blur component of the main body of the imaging device 100). In addition, the blur component detected by the vibration sensor 200 is subtracted from the motion vector detected by the motion vector detection unit 400 in each part of the image, and weighting processing is performed using a value obtained by subtraction, thereby blurring the moving subject. A correction amount can be calculated. As described above, the blur correction amount can be calculated using the value of the motion vector detected by the motion vector detection unit 400 and the value of the blur component detected by the vibration sensor 200.

  Next, blur correction for correcting the blur of each part of the captured image based on the face area detected by the face detection unit 300 and the motion vector detected by the motion vector detection unit 400 will be described with reference to the drawings. In the above description, when the motion vector detection unit 400 detects a motion vector and the face detection unit 300 detects a face, the captured image is divided into a plurality of regions so as to include the detected face. The example in which the motion vector in each divided area is obtained has been described.

  Here, when the motion vector detection unit 400 detects a motion vector, the captured image is divided into a plurality of regions regardless of the face included in the captured image, and a motion vector in each divided region is obtained. An example of blur correction will be described. In other words, the captured image is divided into a plurality of regions, and the motion vector detected in each of the divided regions is divided into a background blur vector (global vector) and a blur vector (local vector) of a moving subject including a face region. ) And the blur amount is detected and corrected locally.

  FIG. 17 is a diagram illustrating motion vectors 759 to 774 calculated from each block divided into 16 in an image 711 including the human 652 and the human 655. In the image 711, motion vectors 759 to 768 are motion vectors (global vectors) of blocks determined as background portions in the image 711, and are the same motion vectors. The motion vectors 769 and 770 are motion vectors (local vectors) in a block including a part of the face area 654 of the face 653 of the human 652 that is a moving subject. The motion vector 771 is a motion vector (local vector) in a block including a part of the torso portion of the human 652 that is a moving subject. Also, the motion vectors 772 and 773 are motion vectors (local vectors) in a block including a part of the face area 657 of the face 656 of the human 655 that is a moving subject. The motion vector 774 is a motion vector (local vector) in a block including a part of the torso portion of the human 655 that is a moving subject.

  The electronic shake correction means included in the shake correction unit 410 can perform local correction from a plurality of local motion vectors obtained for one image.

  With respect to the image 711 shown in FIG. 17, the electronic shake correction unit of the shake correction unit 410 performs the same local correction because the blocks related to the motion vectors 759 to 768 have the same motion vector. On the other hand, for the blocks related to the motion vectors 769 to 774, local correction according to the motion vectors 769 to 774 is performed. That is, when the face 656 or the torso of the human 655 that is a moving subject is moving, the shake can be corrected in accordance with the movement.

  In this way, it is determined whether or not there is at least a partial area of the face for each divided block, and the determination result is stored in correspondence with the block, and the average vector of each of the face area and the background area is stored. Is calculated, the calculated average vector is stored, and a blur correction amount is calculated based on these average vectors. Note that a moving subject area estimated from the face area (for example, the torso part or arm / leg part of the person) is also determined as a part of the face, and together with the face area or a moving subject area other than the face area and face. An average vector may be calculated for each of the above. As a result, a sharp image with less blur can be taken with respect to the human face as the subject. Particularly in the vicinity of a human face, blurring can be sufficiently suppressed, and a main subject can be photographed with high image quality. In addition, it is possible to effectively achieve blur correction by automatically focusing on a person's face without being affected by background blur or movement of an object on the background.

  Next, the operation of the blur correction method shown in FIG. 17 will be described with reference to FIG.

  FIG. 18 is a flowchart illustrating a processing procedure of blur correction amount calculation processing by the imaging apparatus 100.

  First, a captured image that is an image captured by the image capturing unit 120 in the image capturing apparatus 100 is input (step S921). When a captured image is input from the imaging unit 120, it is determined whether the face detection setting is turned on (step S922). If the face detection setting is OFF (step S922), it is not necessary to perform face-based blur correction, the blur correction amount is calculated by normal blur correction processing, and the calculated blur correction amount is input to the blur correction unit 410. Output (step 938).

  If the face detection setting is ON (step S922), the face detection unit 300 detects a face from the input image and calculates various information of each detected face (step S923).

  Subsequently, the input image is divided into a plurality of regions (blocks) (step S924). For example, assume that the input image is divided into N blocks.

  Subsequently, one block is extracted from the plurality of areas (step S925), and it is determined whether or not a face is included in the extracted block (step S926). If the extracted block includes a face (step S926), the motion vector in the extracted block is detected, and the detected motion vector is stored in the face array (step S927). Subsequently, “face” is stored as face / background determination information corresponding to the extracted block (step S928). When a plurality of faces are detected, each motion vector is stored in the face array for each face, and information on each face is stored as face / background determination information corresponding to the extracted block. The

  On the other hand, if the extracted block does not contain a face (step S926), the motion vector in the extracted block is detected and the detected motion vector is stored in the background array (step S929). . Subsequently, “background” is stored as face / background determination information corresponding to the extracted block (step S930).

  Subsequently, it is determined whether or not all the blocks of the plurality of blocks have been extracted (step S931). If all the blocks have not been extracted (step S931), the process proceeds to step S925, and the motion vector detection process for each block is repeated (steps S925 to S930).

  If all the blocks have been extracted (step S931), an average vector for the face area is calculated based on the motion vector stored in the face array and the motion vector stored in the background array. Based on the above, an average vector related to the background region is calculated (step S932). When a plurality of faces are detected, an average vector related to the face area for each face is calculated.

  Subsequently, one block is extracted from the plurality of areas (step S933), and it is determined whether or not “face” is stored in the face / background determination information corresponding to the extracted block (step S934). ). If “face” is stored in the face / background determination information corresponding to the extracted block (step S934), the blur correction amount corresponding to the extracted block is calculated based on the average vector regarding the face area. (Step S935). On the other hand, if “background” is stored in the face / background determination information corresponding to the extracted block (step S934), the blur correction amount corresponding to the extracted block is calculated based on the average vector regarding the background region. (Step S936).

  Subsequently, it is determined whether or not all the blocks of the plurality of blocks have been extracted (step S937). If all the blocks have not been extracted (step S937), the process proceeds to step S933, and the correction amount calculation process for each block is repeated (steps S933 to S936).

  If all the blocks have been extracted (step S937), the calculated correction amount of each block is output to the blur correction unit 410 (step S938).

  As described above, a motion vector is detected for all N blocks, an average vector corresponding to the face area and the background area is calculated based on the motion vector in each block, and based on the calculated average vector A correction amount corresponding to the face area and background area in each block is calculated. This correction amount is given to the blur correction unit 410, and correction is performed in the direction opposite to the blur, so that a good image with sufficiently small blur is obtained in the output image. In particular, the blurring is suppressed on the face portion of the person rather than the background, and the main subject can be photographed sharply.

  Next, by calculating the correction amount based on the local motion vector for the moving subject area including the face area so as to be closer to the center of the captured image, the image of the person is always moved near the center of the image. The correction will be described.

  FIG. 19 is a diagram illustrating an outline in the case where calculation is performed so that the moving subject region including the face region is closer to the center of the captured image.

  As shown in FIG. 19A, when the global correction amount (correction vector) calculated for the image 712 is the vector a, the blur correction process is performed on the face area included in the image 712 by the vector a. Is called.

  For example, as shown in FIG. 19B, when the human face 658 included in the image 714 does not exist near the center of the image, the human face 658 is moved near the center of the image 714 together with the blur correction. Can do.

For example, when the global correction amount calculated for the image 714 is the vector a, the vector b for moving to the center of the image is obtained, the vector b is multiplied by the weighting coefficient W6, and the combination with the vector a is local Correction is performed (Formula 2). The vector b is a vector calculated from the difference between the face coordinates and the image center coordinates. Moreover, w6 is a weighting coefficient and can use the value of 0.1-0.5, for example.
Correction amount = vector a + w6 × vector b (Expression 2)

  Thereby, both the blur correction of the face area and the framing correction can be performed simultaneously.

  In addition, the face area and the person area estimated from the face area can be corrected with local vectors, and other background parts can be corrected with global (global) vectors. Each can perform the optimal blur correction.

  In addition, by calculating a local correction vector for the face area and the moving subject area estimated from the face area so as to be closer to the center of the image, blur correction is performed such that the image of the person moves to the center of the image. And more appropriate framing can be achieved.

  Next, as one of motion vector detection methods, an example of a motion vector detection method other than the above-described motion vector detection method based on block matching image processing will be described with reference to the drawings.

  FIG. 20 is a diagram illustrating an example of a motion vector detection method when a motion vector is detected from a captured image based on the face area detected by the face detection unit 300.

  As shown in FIG. 20, the motion vector detection method based on the face area detected by the face detection unit 300 is a temporally continuous previous frame image (image 715) and current frame image (image 716) constituting a moving image. It is a method of detecting based on the face area in The current frame image (image 716) is a frame after a predetermined time t has elapsed from the previous frame image (image 715). Further, both the vibration sensor 200 and the motion vector detection unit based on the face area detected by the face detection unit 300 are used as the shake detection unit (movement amount calculation unit).

  FIG. 20A is a diagram showing a person 660, its face 661, and a face area 662 of the face 661 detected by the face detection unit 300 in the previous frame image (image 715).

  FIG. 20B shows the human 660 and its face 661, the face area 663 of the face 661 detected by the face detection unit 300, and the hand movement vector 680 detected by the vibration sensor 200 in the current frame image (image 716). And a motion vector 681 detected based on the face area detected by the face detection unit 300. As described above, FIG. 20B shows a case where the camera shake vector 681 and the motion vector 681 based on the face area are combined.

For example, when a face area is detected by the face detection unit 300 in the current frame image (image 716), the coordinates of the face area, the face-likeness value, and the face orientation value for one or more detected faces. A comprehensive evaluation value En is obtained based on various information such as. For example, a comprehensive evaluation value En is obtained using the following (Expression 3). Note that w1 to w5 are weighting coefficients of the respective face evaluation values.
En = w1 × face area area + w2 × face area coordinates + w3 × face-like value + w4 × frontal degree + w5 × face inclination degree (Equation 3)

  Note that the comprehensive evaluation value En may be obtained using information other than the various types of information related to the face shown in (Expression 3), or may be used individually without integrating the evaluation values. Moreover, the comprehensive evaluation value En shown by (Formula 3) is an example, and you may make it obtain | require the comprehensive evaluation value En based on other various information.

  In addition, a wider search range including the vicinity of the face area detected in the current frame image (image 716) in the previous frame image (image 715) is set. In the set search range, a face having a high correlation with the evaluation value related to the face area detected by the face detection unit 300 is searched. As a result of the search, if faces having similar face evaluation values exist in both the previous frame and the current frame, the vector between them is determined as a motion vector (face region vector).

In addition, a final correction vector is obtained by using the following (Equation 4) using a camera shake vector that does not depend on the image detected by the vibration sensor 200. Note that w7 and w8 are respective weighting coefficients. Note that w8 may be the same value as w6 described above.
Correction vector = correction vector for camera shake components
+ W7 × face area motion vector
+ W8 x vector from face to center of screen ... (Formula 4)

  When a plurality of face areas are detected, the correction vector calculated using (Equation 4) may be integrated into one correction vector for each frame, and is optimal for each face area. A correction vector may be applied locally to correct blur.

  Next, the operation of the shake correction method using the motion vector based on the face area will be described with reference to FIG.

  FIG. 21 is a flowchart illustrating a processing procedure of blur correction amount calculation processing by the imaging apparatus 100.

  First, a captured image that is an image captured by the image capturing unit 120 in the image capturing apparatus 100 is input (step S941). When a captured image is input from the imaging unit 120, it is determined whether the face detection setting is turned on (step S942). If the face detection setting is OFF (step S942), based on the camera shake vector detected by the vibration sensor 200, a correction amount 3 that is a shake correction amount is calculated (step S950), and the process proceeds to step S951.

  If the face detection setting is ON (step S942), the face detection unit 300 detects a face from the input image and calculates various information of each detected face (step S943).

  Subsequently, a comprehensive evaluation value En is obtained based on various information of each face detected by the face detection unit 300. For example, a comprehensive evaluation value En is obtained using the following (Equation 3) (step S944). Subsequently, a wide search range including the vicinity of the face area detected in the current frame image is set in the temporally continuous previous frame image constituting the moving image, and the face detection unit 300 detects the search range in this search range. It is determined whether or not there is a face highly correlated with the evaluation value related to the face area (step S945). If there is no face highly correlated with the evaluation value related to the face area in this search range (step S945), a motion vector based on the face area cannot be detected, and the process proceeds to step S950.

  On the other hand, in the search range, if there is a face highly correlated with the evaluation value related to the face area (step S945), each face area determined to be a highly correlated face existing in both the previous frame and the current frame. Based on, a vector between them is determined as a motion vector (a motion vector of the face area) (step S946).

Subsequently, based on the camera shake vector detected by the vibration sensor 200, a correction amount 1 that is a shake correction amount is calculated (step S947), and based on the calculated correction amount 1 and the detected motion vector of the face area. Then, the correction amount 2 is calculated using the following (Equation 5) (step S948).
Correction amount 2 = correction amount 1
+ W7 × Face area motion vector (Equation 5)

Subsequently, the correction amount 3 is calculated using the calculated correction amount 2 and the following (Equation 6) (step S949).
Correction amount 3 = correction amount 2
+ W8 × vector from the face to the center of the screen (Equation 6)

  In steps S948 and S949, the value of (Equation 4) shown above is obtained. W7 and W8 are the same as the weighting coefficients shown above.

  Subsequently, based on the correction amount 3 calculated in step S949 or S950, the blur correction amount of the entire captured image is calculated, and the calculated blur correction amount is output to the blur correction unit 410 (step S951).

  As a result, it is possible to simultaneously realize the framing correction such that the face moves toward the center of the image together with the blur correction process for the face area, so that the blur of the face area is suppressed and the framing balance is better. A preferable output image can be obtained. Note that the numerical values and expressions shown in (Expression 1) and the like are examples, and other numerical values and expressions may be used.

  As described above, according to the embodiment of the present invention, a sharp image with little blur can be taken. Particularly in the vicinity of a human face, blurring can be appropriately corrected, so that a main subject can be photographed with high image quality.

  In addition, since it is possible to reduce the influence of background blur and background object movement, it is possible to effectively achieve blur correction with emphasis on the face of a person.

  Furthermore, it is possible to execute a blur correction amount weighting process based on the coordinates of the face region, the position in the image, the area, the value of the face-likeness, the degree of frontal orientation, the degree of inclination of the face, etc. The amount of blur correction can be calculated.

  In addition, it is possible to correct blur entirely using a single vector on the entire screen, and to correct blur locally using a plurality of vectors in the screen. Corrections can be made.

  Furthermore, in the case of local blur correction, a person as a main subject can be more appropriately framed by calculating a correction vector that moves the face region as close to the center of the image as possible.

  Further, when only the electronic shake detection means and the shake correction means are provided in the image pickup apparatus as the shake detection means and the shake correction means, it is possible to reduce the size, cost and power consumption of the image pickup apparatus. can do.

  Furthermore, by discriminating an image of a group of people (a so-called group photo), a weighting process suitable for a so-called group photo can be executed, so that an appropriate group photo can be taken.

  Further, since the influence of the person in the background can be reduced by discriminating between the person of the main subject and the person mixed in the background, it is possible to realize the optimum blur correction for the person of the main subject.

  In addition, when there are multiple faces on the screen, the most suitable blur correction can be realized by selecting the face with the most appropriate focus and exposure and correcting the blur in that area. You can take pictures.

  In addition, when a face of a specific person is detected from the captured image, it is possible to realize blur correction optimum for the specific person by changing the weighting coefficient related to the specific person to a larger value.

  Furthermore, by subtracting the blur component in the vicinity of the face area from the result of motion vector detection, the blur component of the entire image that is not attracted by the motion of the moving subject can be extracted.

  Further, instead of the conventionally known block matching type motion vector detection method, a method of detecting a motion vector by detecting a movement vector of a face region may be used. As a result, similar to the block matching motion vector detection method, it is possible to achieve optimal local correction for each face area, and even if there are multiple faces, each face is properly shaken. Can be corrected.

  Thus, according to the embodiment of the present invention, it is possible to appropriately correct blur included in an image.

  Note that the present embodiment can be applied to an image processing apparatus such as a personal computer that inputs an image output from a digital still camera or the like and displays the input image on a display or the like.

  The embodiment of the present invention is an example for embodying the present invention and has a corresponding relationship with the invention-specific matters in the claims as shown below, but is not limited thereto. However, various modifications can be made without departing from the scope of the present invention.

  That is, in claim 1, the imaging unit corresponds to the imaging unit 120, for example.

  In claim 1, claim 5, and claim 17, the face detection unit corresponds to the face detection unit 300.

  Further, in claims 1, 8, 16, and 17, the movement amount detection unit corresponds to, for example, the motion vector detection unit 400.

  Further, in claims 1, 9, 15, and 17, the shake correction unit corresponds to, for example, the shake correction unit 410.

  Further, in claims 2 to 7, 10, and 12 to 14, the correction amount calculation means corresponds to, for example, the correction amount calculation unit 440.

  Further, in claim 9, the vibration detecting means corresponds to the vibration sensor 200, for example.

  In claim 10, camera control value detection means corresponds to the control unit 110, for example.

  In claim 11, the setting means corresponds to, for example, the selection receiving unit 162.

  Further, in claim 12, the specific person storage means corresponds to the memory 210. The specific person determination unit corresponds to the control unit 110, for example.

  The image input means corresponds to the signal processing unit 130, for example.

  Further, in claim 18 or claim 19, the image input procedure corresponds to step S <b> 901. The face detection procedure corresponds to step S903. The movement amount detection procedure corresponds to step S907. The blur correction procedure corresponds to step S916.

  The processing procedure described in the embodiment of the present invention may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program May be taken as

It is a block diagram which shows the function structural example of the imaging device 100 in embodiment of this invention. 3 is a block diagram illustrating an exemplary functional configuration of a face detection unit 300. FIG. It is a block diagram which shows the function structural example regarding the blurring correction of the image in the imaging device 100 in embodiment of this invention. It is a figure which shows an example of an image division in case the motion vector detection part 400 detects a motion vector from a picked-up image by the image processing of a block matching system. It is a figure which shows an example of the relationship between the front frame 540 and the present frame 550 in case the motion vector detection part 400 detects a motion vector from a picked-up image by the image processing of a block matching system. It is a figure which shows the example of an image division in case the motion vector detection part 400 detects a motion vector from a picked-up image by the image processing of a block matching system. 6 is a diagram illustrating an example of a motion vector detected from an image 580 by a motion vector detection unit 400. FIG. It is a figure which shows the image 701 in which the human face 613, 615, 617 is contained. It is a figure which shows the example of the picked-up image containing the several human face. It is a figure which shows the example of the picked-up image containing the several human face. It is a figure which shows the example of the picked-up image containing the several human face. It is a figure which shows the example of the picked-up image containing the several human face. 12 is a flowchart illustrating a processing procedure of blur correction amount calculation processing by the imaging apparatus. It is a figure which shows the motion vector 751 thru | or 759 calculated from each block divided into 9 in the image 707 in which the person 648 is included. It is a figure which shows an example of the extraction method which extracts the blur component of a to-be-photographed object based on the blur component 201 of the imaging device 100 main body, and the blur component 401 in each part of an image. 10 is a diagram illustrating an example of an extraction method for extracting a blur component of a background portion included in an image 708 based on a motion vector in each portion of the image 708 and a motion vector of a human 651 included in the image 708. FIG. It is a figure which shows the motion vector 759 thru | or 774 calculated from each block divided | segmented into 16 in the image 711 containing the human 652 and the human 655. FIG. 12 is a flowchart illustrating a processing procedure of blur correction amount calculation processing by the imaging apparatus. It is a figure which shows the outline in the case of calculating so that the moving subject area | region including a face area | region may become near the center of a picked-up image. It is a figure which shows an example of the motion vector detection method in the case of detecting a motion vector from a picked-up image based on the face area | region which the face detection part 300 detected. 12 is a flowchart illustrating a processing procedure of blur correction amount calculation processing by the imaging apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image pick-up device 110 Control part 120 Image pick-up part 121 Optical system 122 Image pick-up element 130 Signal processing part 140 Image memory | storage part 150 Image compression / expansion part 160 User interface 161 Display part 162 Selection reception part 170 Input / output terminal 180 Storage medium 190 Bus 200 Vibration sensor 210 memory 300 face detection unit 310 control unit 320 image input unit 330 image enlargement / reduction unit 340 image holding unit 350 reference data holding unit 360 determination unit 370 detection result output unit 380 image output unit 400 motion vector detection unit 410 motion correction unit 420 image Input unit 430 Image output unit 440 Correction amount calculation unit

Claims (19)

Imaging means for capturing an image of a subject;
Face detection means for detecting a face from an image captured by the imaging means;
A movement amount for detecting a face movement amount that is a blur amount of the face area detected by the face detection unit and a background movement amount that is a blur amount of an area other than the face region in the image captured by the imaging unit. Detection means;
An image pickup apparatus comprising: a shake correction unit that corrects a shake of an image captured by the image pickup unit based on a face movement amount and a background movement amount detected by the movement amount detection unit.   The image processing apparatus further comprises a correction amount calculation unit that calculates a correction amount of an image blur corrected by the blur correction unit based on the face movement amount and the background movement amount detected by the movement amount detection unit. Item 2. The imaging device according to Item 1.   The imaging apparatus according to claim 2, wherein the correction amount calculation unit calculates the correction amount based on a face movement amount and a background movement amount detected by the movement amount detection unit and a predetermined weighting coefficient. .   The correction amount calculation means increases the value of the weighting coefficient related to the face movement amount detected by the movement amount detection means to be larger than the weighting coefficient value related to the background movement amount detected by the movement amount detection means. The imaging apparatus according to claim 3, wherein a correction amount is calculated. The face detection means detects various information related to the face detected from the image picked up by the image pickup means,
The imaging apparatus according to claim 3, wherein the correction amount calculating unit calculates the weighting coefficient based on various types of information related to the face detected by the face detecting unit.   The various information about the face includes the area of the face area, the coordinates of the face area, the value of the face-likeness, the degree of face orientation, the degree of inclination of the face, the degree of facial expression smiling, the degree of serious expression, The imaging apparatus according to claim 5, wherein the information is any one information of a degree of closing or information obtained by combining them.   The imaging apparatus according to claim 5, wherein the correction amount calculating unit calculates the correction amount by increasing a value of a weighting coefficient in the vicinity of the face area detected by the face detecting unit. The movement amount detection unit divides the image captured by the imaging unit into a plurality of regions so that the face detected by the face detection unit is included in a predetermined region, and moves the face from the region including the face. The imaging apparatus according to claim 1, wherein the background movement amount is detected from a region other than the region including the face by detecting the amount. Further comprising vibration detecting means for detecting a blur amount due to vibration of the imaging device;
The blur correction unit corrects the blur of the image captured by the imaging unit based on the face movement amount and the background movement amount detected by the movement amount detection unit and the blur amount detected by the vibration detection unit. The imaging apparatus according to claim 1. Further comprising camera control value detection means for detecting a degree of focus or exposure of the face based on various information about the face detected by the face detection means;
The correction amount calculation unit calculates a correction amount related to the face based on a degree of focus or exposure of the face detected by the camera control value detection unit. The imaging device according to claim 5.   The imaging apparatus according to claim 1, further comprising setting means for setting whether or not to detect a face from an image captured by the imaging means. Specific person storage means for storing specific person information relating to the face of a specific person;
A specific person determination means for determining whether or not the face detected by the face detection means matches the face of the specific person stored in the specific person storage means;
The correction amount calculation means, when the specific person determination means determines that the face detected by the face detection means matches the face of the specific person stored in the specific person storage means, The imaging apparatus according to claim 3, wherein the correction amount is calculated by changing the weighting coefficient according to whether or not the region is a face region.   The correction amount calculation means calculates a first average value that is an average value of the face movement amounts detected by the movement amount detection means, and based on the first average value, detects the face detected by the face detection means. A correction amount related to the area is calculated, a second average value that is an average value of the background movement amount detected by the movement amount detection unit is calculated, and the face detection unit detects the second average value based on the second average value. The imaging apparatus according to claim 2, wherein a correction amount relating to an area other than the face area is calculated.   When a plurality of faces are detected by the face detection unit, the correction amount calculation unit calculates an average value of face movement amounts for each detected face area, and calculates the correction amount based on the average value. The imaging device according to claim 13, wherein the imaging device is calculated.   The blur correction unit corrects the blur of the image so that the face area detected by the face detection unit moves closer to the center of the image in the image captured by the imaging unit. Item 2. The imaging device according to Item 1.   The movement amount detecting means correlates a face detected by the face detecting means in an image picked up by the image pickup means and a face detected by the face detecting means in an image picked up a predetermined time before the image. The image pickup apparatus according to claim 1, wherein a face movement amount related to the face area is detected based on the height of the characteristic. An image input means for inputting an image;
Face detection means for detecting a face from the image input by the image input means;
Movement for detecting a face movement amount that is a blur amount of the face area detected by the face detection means and a background movement amount that is a blur amount of an area other than the face area in the image input by the image input means. A quantity detection means;
An image processing apparatus comprising: a blur correction unit that corrects a blur of an image input by the image input unit based on a face movement amount and a background movement amount detected by the movement amount detection unit. Image input procedure for inputting images,
A face detection procedure for detecting a face from the image input in the image input procedure;
Movement that detects a face movement amount that is a blur amount of the face area detected by the face detection procedure and a background movement amount that is a blur amount of an area other than the face area in the image input by the image input procedure. A quantity detection procedure;
An image processing method comprising: a blur correction procedure for correcting blur of an image input in the image input procedure based on a face movement amount and a background movement amount detected in the movement amount detection procedure. Image input procedure for inputting images,
A face detection procedure for detecting a face from the image input in the image input procedure;
Movement that detects a face movement amount that is a blur amount of the face area detected by the face detection procedure and a background movement amount that is a blur amount of an area other than the face area in the image input by the image input procedure. A quantity detection procedure;
A program for causing a computer to execute a blur correction procedure for correcting blur of an image input in the image input procedure based on a face movement amount and a background movement amount detected in the movement amount detection procedure.

Images