JP2007081682A - Image processor, image processing method, and executable program by information processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve problems that, when calculating deviation between image signals, an image sensor is affected by the movement or the like of hands and feet when the deviation of a composition is calculated on the basis of a person, and when calculating the deviation of the composition on the basis of a background while avoiding the person, accuracy becomes poor since a blurred image is targeted when the depth of field is shallow. <P>SOLUTION: A specific region is detected from the image signal by executing matching with reference information. The image signal is divided into a plurality of regions. Different weighting is set to information (a motion vector) showing positional deviation for each region corresponding to the detection result of a specific region. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program executable by an information processing apparatus for calculating a motion vector, which is a composition shift between a plurality of images.

  Many of the current cameras have automated operations important for shooting such as exposure determination and focus adjustment, and even a person unskilled in camera operation can easily obtain an appropriate image. In addition to this, a device for preventing camera shake applied to the camera has been developed. As a method for preventing this camera shake, for example, there is an optical camera shake correction. In this method, vibration due to camera shake is detected by a sensor that detects acceleration, angular acceleration, angular velocity, angular displacement, and the like, and the correction lens is displaced according to the detected value to correct the optical axis (for example, patents). Reference 1).

  In addition, as a difference from optical camera shake correction, there is image synthesis camera shake correction. This compensates for lack of exposure while correcting the difference in composition for each image by repeating shooting with a short exposure time so as not to cause camera shake and combining the positions of a plurality of shot images. (For example, refer to Patent Document 2).

In this image composition type camera shake correction, if an attempt is made to correct a composition shift with a person as a reference, it is difficult to detect the reference person with high accuracy due to the influence of limb movements and the like. Therefore, a method of calculating a motion vector using a background that avoids a region where a person is located has been proposed (see, for example, Patent Document 3).
JP 05-066452 A JP 2002-064743 A JP 2004-219765 A

  Unlike the optical camera shake correction, the image synthesis type camera shake correction is suitable for downsizing because it is not necessary to provide a correction lens and its driving mechanism. However, the following problems may occur in the image synthesis method of camera shake correction.

  For example, when the aperture is opened and the depth of field is set shallow, the amount of blur greatly differs between the main subject and the background. For this reason, when the motion vector is calculated using the background, the alignment calculation is performed using the image information of the blurred area, so that only a result with poor accuracy can be obtained.

  The present invention has been made in view of the above problems, and it is an object of the present invention to appropriately calculate a composition shift between image signals of main subjects by accurately detecting main subjects.

  Under such an object, the first invention provides a displacement detection means for detecting displacement between a plurality of image signals, and a specific region for detecting a specific region from the image signal according to a comparison result with reference information. Detecting means, and the deviation detecting means sets and outputs different weights for the information indicating the positional deviation for each area of the image signal according to the detection result of the specific area by the specific area detecting means. An image processing apparatus is provided.

  Similarly, under such an object, the second invention provides a person according to a comparison result between a deviation detecting means for detecting a positional deviation between a plurality of image signals and information indicating the shape of a person's face from the image signals. A face area detecting means for detecting an area where the face of the image is present, and the deviation detecting means includes information indicating a positional deviation of the area where the human face of the image signal is present and a human face of the image signal is present. It is an object of the present invention to provide an image processing apparatus that outputs any one of information indicating a positional deviation of an area different from the area.

  Similarly, under such an object, the third invention detects a specific region from the image signal in accordance with a result of comparison with reference information, and a displacement detection step for detecting a displacement between a plurality of image signals. Specific area detection step, and in the deviation detection step, according to the detection result of the specific area in the specific area detection step, different weights are set for the information indicating the positional deviation for each area of the image signal. An image processing method characterized by output is provided.

  Similarly, under such an object, the fourth invention provides a person according to a result of comparison between a displacement detection step for detecting a positional deviation between a plurality of image signals and information indicating the shape of a person's face from the image signals. A face area detecting step for detecting an area where the face of the image is present, and in the deviation detecting step, information indicating the positional deviation of the area where the human face of the image signal is present and the human face of the image signal exist It is an object of the present invention to provide an image processing method characterized by outputting any information indicating a positional deviation of an area different from the area.

  Similarly, under such an object, the fifth invention provides a program executable by an information processing apparatus characterized by having a program code for realizing the third invention or the fourth invention. Is.

  According to the present invention, it is possible to detect a specific target with high accuracy by performing a comparison process with reference information. Therefore, it is possible to appropriately calculate a composition shift between image signals centered on the specific target.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
The image processing apparatus according to the first embodiment of the present invention will be described with reference to FIGS.

  Here, as an example of an image processing apparatus having a subject detection function, a digital camera having a function of detecting a human face will be described.

  The image processing apparatus may have any configuration as long as it can input a plurality of continuous image data. For example, a digital camera, a digital video camera, or a camera-equipped mobile phone including an image sensor such as a CCD or CMOS sensor and a processor that performs digital signal processing based on an output signal of the image sensor is an example of an image processing apparatus. It is done. An example of an image processing apparatus is a computer that performs digital signal processing based on image data received from outside by wireless communication or wired communication.

  The subject detection function may be any function that detects a predetermined subject by detecting and determining a specific shape, a specific luminance distribution, a specific hue distribution, and the like. The subject of the subject may be something other than a human face, such as an animal such as a dog, cat, or bird, or a vehicle such as a car, airplane, or train.

  Various methods are known for such a subject detection function.

  For example, Japanese Patent Laid-Open No. 05-197793 discloses a technique for detecting a subject from a matching result between shape data of each part of a face and an input image. In this technique, shape data such as iris, mouth and nose are prepared, and two irises are first obtained. Subsequently, when obtaining the mouth, nose, etc., the search area for the facial parts such as the mouth, nose, etc. is limited based on the position of the iris. In other words, this algorithm does not detect the facial parts that make up the face such as the iris (eye), mouth, and nose in parallel, but first finds the iris (eye) and uses the result to It detects a facial part called the nose.

  As other matching processing, for example, for image data, the method used in the image information extraction apparatus described in Japanese Patent Laid-Open No. 9-130714 can be used. In this apparatus, a template image having a size corresponding to the subject distance is generated. Then, using this, while calculating the normalized correlation coefficient at each location while scanning the screen, the similarity distribution between the local portion of the input image and the model data is calculated. In addition, an algorithm based on a spatial arrangement relationship of local features described in Japanese Patent No. 3078166, an algorithm based on a convolutional neural network described in Japanese Patent Laid-Open No. 2002-8032, or the like may be used.

  In this embodiment, an edge is extracted using luminance information of image data, and pattern matching is performed by comparing indexes such as the extracted edge shape, position in the image, and hue with a database stored in advance. A method for detecting a desired subject is used.

  FIG. 1 shows a block diagram of an imaging apparatus 100 in the present embodiment.

  In FIG. 1, reference numeral 101 denotes a photographic lens composed of a plurality of lenses including a zoom lens and a focus lens, 102 denotes a diaphragm for adjusting the amount of light flux that has passed through the photographic lens, and 103 denotes light flux that has passed through the photographic lens. This is a shutter for blocking. Reference numeral 104 denotes an image sensor such as a CMOS sensor or a CCD. The diaphragm 102 and the shutter 103 are disposed between the photographing lens 101 and the image sensor 104.

  Reference numeral 105 denotes image sensor driving means for controlling the charge accumulation operation and reset operation of the image sensor. Reference numeral 106 denotes an AF drive motor that drives a focus lens included in the photographing lens 101 in the optical axis direction to change the focal position. Reference numeral 107 denotes shutter driving means for driving the shutter 103, reference numeral 108 denotes aperture driving means for driving the iris 102 to adjust the aperture diameter, and reference numeral 109 denotes focus control means for controlling the driving amount and driving direction of the AF driving motor 106. It is.

  Reference numeral 110 denotes analog / digital (hereinafter referred to as A / D) conversion means. Reference numeral 111 denotes signal processing for an image signal output from the A / D conversion means 110 or an image signal stored in the memory 112 described later. Signal processing means to be applied. The analog signal output from the image sensor 104 is converted into a digital signal by the A / D conversion unit 110 and input to the signal processing unit 111. The signal processing circuit 111 forms a luminance signal and a color signal from the digital signal input from the A / D conversion means 110, and forms a display image signal, a recording image signal, and a face detection image signal. A memory 112 temporarily stores a face detection image signal and a display image signal. 113 is a control means, and each of the shutter drive means 107, the aperture drive means 108, the focus control means 109, the signal processing means 111, and the face detection means 114 described later operates based on a control command from the control means 113. .

  Reference numeral 114 denotes a face detection unit that analyzes an image signal for face detection formed by the signal processing unit 111 and stored in the memory 112 to detect a region where a human face exists. As will be described later, the face detection unit 114 includes a first reference level and a second reference level, which is stricter than the first reference level, as a reference level for determining a face. Reference numeral 115 denotes a deviation detection means, which performs a correlation operation between a plurality of image signals and calculates a relative positional deviation amount (motion vector) between the image signals. A method of calculating the motion vector of the deviation detecting unit 115 will be described later. A coordinate conversion unit 116 performs image conversion of each image signal in accordance with the motion vector calculated by the deviation detection unit 115. Reference numeral 117 denotes image storage means for storing the image signal whose coordinates have been converted by the coordinate conversion means 116, and reference numeral 118 denotes image composition means for synthesizing the image signals stored in the image storage means 117.

  The shift detection unit 115, the coordinate conversion unit 116, the image storage unit 117, and the image synthesis unit 118 do not operate when the possibility of blurring is not high, and the camera operates in the image synthesis mode only when the possibility of blurring is high. It does not matter if this is set. A specific example is a case where the shutter speed is longer than a predetermined value and the camera is not fixed to a tripod or the like. Note that the image composition mode may be manually set by the user.

  Reference numeral 119 denotes a display unit made up of an LCD (Liquid Crystal Display) or the like, and displays a subject image by inputting a display image signal formed by the signal processing unit 111. Various parameters relating to shooting such as shutter speed, aperture value, sensitivity, and shooting mode can be displayed superimposed on the subject image. The display unit 115 is also used to reproduce an image that has already been shot and stored in the camera. The signal processing unit 111 receives information related to the face area detected by the face detection unit 113, and superimposes a frame indicating the position and size of the face on the display image signal. The display means 119 receives and displays this image signal, so that the user of the camera can recognize the face detection result. Reference numeral 120 denotes a recording unit that records a recording image signal formed by the signal processing unit 111. Reference numeral 121 denotes an operating means for setting a switch SW1 for instructing a photometric operation and a focus adjustment operation for photographing, a switch SW2 for starting photographing, a photographing mode, an image composition mode, or a panning mode to be described later. The operation member is included.

  When the image composition mode is set and the release button is pressed halfway by the photographer and the switch SW1 is turned on, the imaging apparatus performs a focus adjustment operation and a photometric operation for the actual photographing. The control means 113 determines the aperture diameter of the aperture 102 and the shutter speed (exposure time) of the shutter 103 according to the photometric result at this time. If the subject brightness is low, the aperture is fully open and the exposure time is also long. If the exposure time is longer, the possibility of camera shake is higher. Therefore, when the exposure time is longer than a predetermined value, an image composition mode is set in which exposure in a shorter time is continuously performed a plurality of times.

  In the image composition mode, the image signal obtained by one exposure is in a state where the luminance is insufficient, but the influence of camera shake can be reduced. Further, if a plurality of image signals obtained in succession are combined, the luminance of each image signal is added, so that the luminance of the combined image signal can be set to an approximately appropriate value. However, even if the blur of each image signal is small, the composition of each image signal changes due to camera shake during continuous shooting. Therefore, if the image signals are synthesized as they are, the composition deviation between the image signals is accumulated. End up.

  Therefore, a plurality of continuously exposed image signals are input to the face detection unit 114 and the shift detection unit 115 via the signal processing unit 111, respectively, and the relative positional shift amounts between the plurality of image signals, that is, A motion vector is calculated.

  The shift detection unit 115 divides the plurality of image signals sent from the signal processing unit 111 into a plurality of blocks, as shown in FIG. FIG. 2 is a diagram for explaining a motion vector of an image signal. In this embodiment, an example is shown in which the long side direction of one image signal is divided into eight and the short side direction is divided into six and divided into a total of 48 blocks. Of course, the number of divisions of the block is arbitrary, and other numbers and other shapes may be used.

  In FIG. 2, for ease of explanation, A column, B column, C column, D column, E column, F column, G column, H column are shown from the left, and 1 row, 2 rows, 3 rows, 4 rows from the top. Lines 5, 5 and 6 are assumed. And the block in x column y row is called xy. That is, the upper left block is A1, and the lower right block is H6.

  The shift detection unit 115 performs a two-dimensional correlation calculation on the blocks at the same position between a plurality of image signals, and calculates a motion vector that is a position shift amount for each block. The calculated motion vectors of the 48 blocks are weighted and then a histogram is obtained. The mode value is used as the motion vector of the entire image signal.

  In the present embodiment, the face detecting unit 114 detects a block in which a human face exists from the image signal, and reflects the position of the block in which the face is detected in the weighting. Specifically, the motion vector of the entire image is obtained by making the appearance frequency of the motion vector of the block in which the face is detected higher than other regions.

  Specifically, the motion vector of the block in which the face is detected is made larger than that of other regions, and the motion vector of the entire image is obtained. Alternatively, the motion vector of the entire image is obtained using only the motion vector of the block in which the face is detected.

  FIG. 3 shows the relationship between the position where the person's face is detected and the position of the block reflecting the weight. In FIG. 3, the ellipse indicates the outline of the face of the person, and a total of 10 blocks of blocks C4, C5, D4, D5, F3, F4, F5, G3, G4, and G5 are completely superimposed on the face area. ing. In the present embodiment, the motion vectors of these 10 blocks are weighted more than all other blocks, and the motion vector of the entire image is determined from the motion vector histogram. In the example shown in FIG. 3, a block that is completely overlapped with the face area is set as a target for increasing the weight, but a block in which the face area is overlapped by a predetermined ratio or more may be set as the same target. Also, the distance to the face and the position where the face exists may be taken into account when weighting. For example, a method is conceivable in which the weight of the block superimposed on the face area close to the center is made larger than the weight of the block superimposed on the surrounding face area.

  The coordinate conversion unit 116 performs coordinate conversion on each image signal in accordance with the motion vector of the entire image calculated by the shift detection unit 115 in order to cancel the shift between the image signals. The image signal transformed by the coordinate transformation unit 116 is stored in the image storage unit 117, and the stored image signal is synthesized into one image by the image synthesis unit 118 and stored in the memory 112. In the image signal synthesized to cancel out the shift between the image signals, an area that overlaps with all the image signals before the synthesis and an area that overlaps only a part of the image signals before the synthesis occur. Therefore, the image synthesizing unit 118 cuts a region that is superimposed only on a part of the image signal before synthesis, and generates an image signal that is synthesized only on the region that is superimposed on all the image signals before synthesis. Then, a diffusion complement process is performed on the synthesized image signal to generate an image signal having the same size as the original frame. The image signal is recorded in the recording unit 120 via the memory 112, and the display unit 119 displays the combined image signal using the display image signal generated by thinning out the image signal.

  Next, motion vector calculation processing according to the present embodiment will be described in detail with reference to the flowchart shown in FIG.

  When the camera is activated by operating the power switch of the camera, the process starts from step S1. In step S <b> 1, the control unit 113 initializes control parameters of the shutter driving unit 107, the aperture driving unit 108, and the focus control unit 109. Then, the output signal of the image sensor 104 is input to the signal processing circuit 111 via the A / D conversion unit 110, and the signal processing circuit 111 generates an image signal for display and inputs it to the display unit 119. By periodically repeating such an imaging operation, the camera operator can monitor the state of the subject in real time by observing the image displayed on the display means 119.

  The control unit 113 causes the face detection unit 114 to set a second reference level that is stricter than the first reference level as an initial value of the reference level for determining the face. Here, the standard level for determining a face is a level for determining to what degree the input image signal matches the face template information stored as the reference database. That is.

  As a method for providing a difference in the reference level, the following example can be considered. For example, when the first reference level is used, if there is a region whose degree of coincidence with the template information satisfies a predetermined condition in the shape of the face, the region is determined as a face. On the other hand, when the second reference level is used, if there is a region that satisfies a predetermined condition in all of the face color and face size in addition to the face shape, the degree of coincidence with the template information The area is determined as a face.

  As another example, when the second reference level is set, there is a method of narrowing a predetermined numerical range recognized as a detection target as compared with the case where the first reference level is set. For example, when the first reference level is set, it is set as one of the conditions necessary for determining as a face that an iris pair exists within a range of 5.5 cm to 7.5 cm. On the other hand, when the second reference level is set, it is determined as a face that there is an iris pair in a range of 6.0 cm to 7.0 cm whose distance from each other is narrower than the first reference level. One of the conditions necessary to do this. Another method may be used as long as a difference can be provided in the reference value for recognizing the face between the first reference level and the second reference level.

  When the above preparation is completed, the control unit 113 determines the shooting mode in step S2. This determines whether the shooting mode of the camera is set to an image synthesis mode in which a plurality of images are shot by the operation unit 121 and then set to another shooting mode. If the shooting mode is set to the image composition mode, the process proceeds to step S3, and if it is set to another shooting mode, the process proceeds to step S22.

  First, the case where it is determined in step S2 that the other shooting mode is set will be described. In step S22, the control unit 113 determines whether the switch SW1 included in the operation unit 121 is turned on. If the switch SW1 is turned on, the control unit 113 proceeds to step S23, and if not, waits until it is turned on.

  In step S23, in response to the switch SW1 being turned on, the signal processing unit 111 generates an image signal for face detection, and the face detection unit 114 performs face detection on the image signal for face detection. Then, the control unit 113 determines a focus adjustment area and a photometry area according to the face detection result, and performs a focus adjustment operation and a photometry operation. When face detection fails, the focus adjustment operation and the photometry operation are performed using a method of focusing on a close subject or a method of obtaining subject luminance by increasing the weight near the center of the screen.

  In step S24, the control means 113 determines whether or not the switch SW2 included in the operation means 121 is turned on. If it is turned on, the process proceeds to step S25, and if not, the process returns to step S22.

  In step S25, based on the photometric information obtained in step S23, the aperture value and shutter speed are set to perform shooting, and the obtained image is stored in the memory 112. The signal processing unit 111 converts the combined image signal stored in the memory 112 into a display image signal or a storage image signal. In response to the converted image signal, the display unit 119 displays the combined image signal, and the recording unit 120 records the combined image signal.

  Next, a case where it is determined in step S2 that the image composition mode is set will be described.

  In step S3, the control unit 113 causes the face detection unit 114 to set the first reference level as the reference level for determining the face. The reference level for determining the face is changed even when the subject and the camera move relatively during panning, etc., and the face detection is difficult compared to the situation where the subject is stationary. This is to maintain a high probability of successful detection. For this reason, in step S3, a first reference level that is more gradual than the second reference level is set.

  In step S4, the control unit 113 determines whether the switch SW1 included in the operation unit 121 is turned on. If the switch SW1 is turned on, the process proceeds to step S5, and if not, it waits until it is turned on.

  In step S5, in response to the switch SW1 being turned on, the signal processing unit 111 generates an image signal for face detection, and the face detection unit 114 performs face detection on the image signal for face detection. . Then, the control unit 113 determines a focus adjustment area and a photometry area according to the face detection result, and performs a focus adjustment operation and a photometry operation. At this time, the coordinate information of the area where the detected face exists is stored in the memory 112. When face detection fails, the focus adjustment operation and the photometry operation are performed using a method of focusing on a close subject or a method of obtaining subject luminance by increasing the weight near the center of the screen.

  Note that it is assumed that the image composition mode is automatically set according to the luminance of the subject, and it is possible to determine whether the image composition mode is set in step S2 after receiving the photometric result in step S5. Absent. In this case, the position of step S2 is changed between step S5 and step S6, and if the image composition mode is not set, the process proceeds to step S22.

  In step S <b> 6, the control unit 113 determines whether or not the operation unit 121 is set to a panning mode in which the camera itself is expected to move or the camera itself changes direction. If the panning mode is set, the panning flag flg stored in the memory in the control means 113 is set to 1 (step S7). Otherwise, the flag flg is set to 0 (step S8).

  The value of the flag flg may be set according to the output of a sensor that detects acceleration, angular acceleration, angular velocity, angular displacement, etc. (not shown) incorporated in the camera, even if the panning mode is not set by the operating means. I do not care. In this case, the flag flg is set to 1 when it is detected by a sensor (not shown) that the camera is moving in a certain direction.

  In step S9, the control means 113 determines whether or not the switch SW2 included in the operation means 121 is turned on. If it is turned on, the process proceeds to step S10, and if not, the process returns to step S4.

In step S10, continuous shooting is performed based on the photometric information obtained in step S5. The number of consecutive shots n is T _S as the original exposure time required for shooting one shot in the scene, and T _{M as} the exposure time per shot when shooting multiple shots.
T _S = T _M × n
Should be satisfied. As a specific example, a method may be considered in which “1 / focal length”, which is a measure of the exposure time without camera shake, is determined as TM and the number of shots n is determined based on this.

  In step S11, the shift detection unit 115 divides each image signal continuously shot into a plurality of blocks as shown in FIG. For all the blocks of the image signal, two-dimensional correlation calculation between the image signals is performed on the same block. As a result, a shift amount for each block, that is, a motion vector is obtained and stored in the buffer memory.

  In step S <b> 12, the control unit 113 determines whether the image composition mode set by the operation unit 121 is a person priority mode in which the person who is the main subject is preferentially combined. If the person priority mode is set, the process proceeds to step S13, and if not, the process proceeds to step S18.

  In step S13, the control unit 113 causes the face detection unit 114 to set the second reference level as a reference level for determining the face. Although it takes more time than the face detection to determine the focus adjustment area and photometry area, the face detection is performed on the captured image signal. Therefore, unlike the face detection in the shooting operation, effects such as release time lag occur. do not do. By making the reference level stricter, face detection with higher accuracy than in step S5 can be performed.

  In step S14, the face detection unit 114 performs face detection on the first image signal continuously photographed in step S10. However, since the brightness of each image is low, an image signal for face detection with increased brightness is generated by performing digital gain processing or the like, and face detection is performed on the image signal for face detection. If face detection cannot be performed from the first image signal, face detection is performed on the second and third image signals. If face detection fails for all image signals, the face detection result in step S5 may be substituted.

  In the present embodiment, different template information for face detection is provided for each individual registered in advance, and not only face detection but also individual recognition can be performed from an image signal. The region detected as a face is further matched with template information prepared for each individual. If the degree of coincidence is the highest and a predetermined threshold is satisfied, it is determined that the person is registered as the template information. The individual template information is stored in advance in association with information related to the priority order of the individual. This is because it is almost impossible to correct the positional deviation so as to be optimal for all of the faces of a plurality of persons. In addition, by assigning priorities in advance, it is possible to give priority to the correction of the positional deviation of the subject intended by the user.

An example will be described in which there is a face of Mr. A registered with a high priority (for example, first place) and a face of Mr. B with no priority registered as shown in FIG. The upper left coordinates of the rectangular area superimposed on Mr. A's face are (x _A1 , y _A1 ), and the lower right coordinates are (x _A2 , y _A2 ). The upper left coordinates of the rectangular area superimposed on Mr. B's face are (x _B1 , y _B1 ), and the lower right coordinates are (x _B2 , y _B2 ). When the reliability of the face detection result of Mr. _A is R _A and the reliability of the face detection result of Mr. _B is R _B , the following information is stored in the memory 112 for each of the face detection results of Mr. A and Mr. B. Of course, the information stored in this information differs depending on the number of people detected and the individual.
Mr. A {(x _A1 , y _A1 ), (x _A2 , y _A2 ), R _A , 1}
Mr. B {(x _B1 , y _B1 ), (x _B2 , y _B2 ), R _B , −}
Here, the reliability of the face detection result will be described.

  Several calculation methods are conceivable for the reliability of the face detection result. For example, there are several parameters such as template matching result, color (skin color if face), motion detection (moving if face (person)), and there is a method of setting reliability by combining these parameters . Other methods include setting the reliability according to the degree of matching with the template, and setting the reliability according to the resolution and area of the face detection area (faces captured in high-resolution images, large images (Face is more reliable). Japanese Unexamined Patent Application Publication No. 2004-206665 discloses a method of detecting a skin color area and determining that the face area detected from the skin color area is highly reliable. Japanese Patent Laid-Open No. 2003-266348 discloses a method in which the number of matched pixels is set to nmatch and the reliability of nmatch / n with respect to the number of pixels n of the template.

  In the present embodiment, a value normalized to 0 to 1 is set as a value indicating the reliability of the face detection result obtained by various methods as described above. The higher the reliability, the closer to 1.

  If face detection fails in step S14 and the area where the face exists cannot be detected, the process proceeds to step S19 described later (step S15).

In step S <b> 16, the deviation detection unit 115 calculates the weight W for the blocks included in the range of the coordinate information of the face detection result stored in the memory 112. In calculating the weight W, the reliability R (0 ≦ R ≦ 1) of the face detection result and the priority order regarding the registered face are taken into consideration. As an example of a specific calculation method of W, a weight w _f (f is a priority) for each registered priority order of a face and a weight w ₀ for a general face that is not registered are stored in the camera. is doing. W ₁ ≧ w ₂ ≧ ・ ・ ・ ≧ w ₀ ≧ 1 so that the higher the priority, the higher the weight.
Is satisfied. At this time, the weight W of each face block is
W = {(w _f −1) × R} +1
If the registered priority order and the reliability at the time of face detection are higher, the value of the weight W becomes larger. In the example shown in FIG. 3 in which the faces of Mr. A and Mr. B are detected, {(w ₁ -1) for a total of 6 blocks F3 to F5 and G3 to G5 which are blocks from which the face of Mr. A is detected. _A weight of × R _A } +1 is given. Further, a weight of {(w ₀ −1) × R _B } +1 is given to a total of four blocks C4 to C5 and D4 to D5, which are blocks in which Mr. B's face is detected. Note that if the weight W of the unregistered face is set to be 0, the user does not intend the user, such as a passing person, who happens to be captured when calculating the positional deviation. You can ignore people.

  In this embodiment, weight W is given using both the registered priority and the reliability at the time of face detection, but only one of the registered priority and the reliability at the time of face detection is used. The weight W may be given.

  In step S17, a motion vector histogram is created for each image signal based on the motion vector for each block calculated in step S11 and stored in the buffer memory and the weight calculated in step S16. For blocks where no face is detected, 1 is added to each motion vector calculated for each block, whereas for blocks where a face is detected, only W (W> 1) is added. Create appearance frequency as a histogram. Then, the process proceeds to step S20.

  Returning to step S12, if the normal synthesis mode has been set, the value of the flag flg set in step S7 is detected in step S18. When the flag flg is set to 1, it indicates that the camera is set to the panning mode or the camera has moved in a certain direction at the time of shooting. For this reason, it is considered that a preferable result can be obtained when image synthesis is performed with the subject as the center. Accordingly, if the flag flg is set to 1, the process proceeds to step S13, and the process proceeds to a process of performing image synthesis using the face detection result.

  On the other hand, when the flag flg is set to 0, the process proceeds to step S19, where 1 is added to each motion vector calculated for each block without weighting for each block, and the motion vector for each image signal is added. Create a histogram for. If face detection fails in step S14, the process of step S19 is performed through step S15. Then, the process proceeds to step S20.

  In step S20, the motion vector that appears most frequently in the histogram created in step S17 or step S19 is extracted and used as the motion vector of the image signal.

  Based on this result, the coordinate conversion means 116 performs coordinate conversion and stores it in the image storage means 117 in step S21. By performing coordinate conversion based on the motion vector of the entire image obtained in step S20, the main positional deviation between consecutively captured image signals is offset. The image synthesizing unit 118 reads out a series of image signals whose coordinates have been converted from the image storage unit 117, and generates an image signal in which only a region that overlaps only all the image signals is synthesized. In this embodiment, the image synthesizing unit 118 cuts without synthesizing a region where only a part of the image signal is superimposed, but also synthesizes a region where only a part of the image signal is superimposed, and the insufficient luminance is converted into a digital gain. You may make it compensate by. The image synthesizing unit 118 stores the synthesized image signal in the memory 112 after making the size of the original frame by diffusion complement processing. The signal processing unit 111 converts the combined image signal stored in the memory 112 into a display image signal or a storage image signal. In response to the converted image signal, the display unit 119 displays the combined image signal, and the recording unit 120 records the combined image signal.

  As described above, in this embodiment, the motion vector is obtained by increasing the weighting of the region where the human face exists. Therefore, it is possible to obtain a motion vector, which is a shift between a plurality of image signals, with high accuracy by giving priority to a person without being affected by the movement of limbs.

(Second Embodiment)
In the first embodiment described above, the motion vectors are calculated for all the blocks obtained by previously dividing the captured image, then the face is detected, and the block from which the face is detected is weighted, and then the image signal The overall motion vector was calculated. In the present embodiment, the face is detected first, and the motion vector of the entire image signal is calculated using only the block in which the face is detected.

  A block diagram of a camera which is an imaging apparatus in the present embodiment is the same as the block diagram of the first embodiment.

  The motion vector calculation processing of the present embodiment will be described in detail with reference to the flowchart shown in FIG. 5, focusing on processing different from that of the first embodiment. In the flowchart shown in FIG. 5, the same step numbers as those in FIG. 4 are attached to the processes common to the flowchart of the first embodiment. The flowchart shown in FIG. 5 performs the same processing up to step S10 of the flowchart shown in FIG.

  If continuous shooting is performed in step S10 based on the photometric information obtained in step S5, the process proceeds to step S12. Unlike the first embodiment, two-dimensional correlation calculation is not performed on all the blocks after continuous shooting.

  In step S <b> 12, the control unit 113 determines whether the image composition mode set by the operation unit 121 is a person priority mode in which the person who is the main subject is preferentially combined. If the person priority mode is set, the process proceeds to step S13, and if not, the process proceeds to step S18.

  In step S13, the control unit 113 causes the face detection unit 114 to set the second reference level as a reference level for determining the face.

  In step S14, the face detection unit 114 performs face detection on the first image signal continuously captured in step S10. However, since the brightness of each image is low, an image signal for face detection with increased brightness is generated by performing digital gain processing or the like, and face detection is performed on the image signal for face detection. If face detection cannot be performed from the first image signal, face detection is performed on the second and third image signals. If face detection fails for all image signals, the face detection result in step S5 may be substituted.

  Also in this embodiment, different template information for face detection is provided for each individual registered in advance, and not only face detection but also individual recognition can be performed from an image signal.

Assume that face detection is performed on the image signal of FIG. 3 and the same face detection result and personal recognition result as in the first embodiment are obtained. Is stored in the memory 112.
Mr. A {(x _A1 , y _A1 ), (x _A2 , y _A2 ), R _A , 1}
Mr. B {(x _B1 , y _B1 ), (x _B2 , y _B2 ), R _B , −}
Next, in step S31, the shift detection unit 114 divides each image signal continuously shot into a plurality of blocks. Two-dimensional correlation calculation between image signals is performed on the same block only for blocks included in the range of coordinate information stored in the memory 112 of these image signals. As a result, the shift amount for each block in which the face is detected, that is, the motion vector is obtained and stored in the buffer memory.

  In step S32, a weight W is calculated for the block in which the face is detected, taking into account the reliability R (0 ≦ R ≦ 1) of the face detection result and the priority order for the registered face. Based on this weighting W, a motion vector histogram of a block in which a face is detected is created for each image signal. In step S20, the motion vector that appears most frequently in the histogram is extracted and used as the motion vector of the image signal.

  Returning to step S12, if the normal composition mode is set, the process proceeds to step S18, and the value of the flag flg stored in step S7 or step S8 is detected. When the flag flg is set to 1, it indicates that the camera is set to the panning mode or the camera has moved in a certain direction at the time of shooting. For this reason, it is considered that a preferable result can be obtained when image synthesis is performed with the subject as the center. Accordingly, if the flag flg is set to 1, the process proceeds to step S13, and the process proceeds to a process of performing image synthesis using the face detection result.

  On the other hand, if the flag flg is set to 0, the process proceeds to step S33. In step S <b> 33, the deviation detection unit 115 divides each continuously captured image signal into a plurality of blocks. For all the blocks of the image signal, two-dimensional correlation calculation between the image signals is performed on the same block. As a result, a shift amount for each block, that is, a motion vector is obtained and stored in the buffer memory.

  In step S34, 1 is added to each motion vector calculated for each block, and a motion vector histogram is created for each image signal. In step S20, the motion vector that appears most frequently in the histogram is extracted and used as the motion vector of the image signal.

  In step S21, as in the first embodiment, coordinate conversion is performed based on the motion vector of the entire image obtained in step S20, thereby canceling main deviations between continuously captured image signals. Synthesize the image signal.

  In the second embodiment, in the person priority mode, unlike the first embodiment, the motion vector is calculated only from the area where the face is detected. Time is shortened.

  In the first and second embodiments, the image signal is divided into a plurality of blocks, and the correlation calculation is performed on the blocks included in the range of the coordinate information stored in the memory 112. The present invention is not limited to this method. For example, the image signal is not divided into a plurality of blocks, and the correlation calculation is performed on the area corresponding to the range of the coordinate information stored in the memory 112. Also good. Alternatively, a region along the face shape may be extracted and correlation calculation may be performed on the region.

(Other embodiments)
In the first embodiment and the second embodiment, the configuration based on still images has been described. However, the present invention is not limited to this, and can be applied to moving images. As for moving image blur correction, for example, Japanese Patent Application Laid-Open No. 11-187303 discloses a technique for detecting a positional deviation between a specific image and a continuous image and reproducing the corrected image. Has been. By applying the present invention to this technique, it becomes possible to increase the accuracy of motion vector calculation and shorten the processing time, and therefore it is effective for reproducing moving images. In addition, as described in the first and second embodiments, the motion vector calculated from a certain area of the main subject (person's face) is not given priority, and there is no main subject. It is desirable to give priority to the area (background).

  That is, the process in step S16 in FIG. 4 may be replaced with a process in which the weight of the area where the face is detected is made smaller than the other areas.

  Alternatively, the processing in steps S31 and S32 in FIG. 5 may be replaced with processing for calculating a motion vector only for an area where no face is detected and creating a histogram.

  In the above-described embodiment, the motion vector is calculated inside the camera, but the present invention is not limited to this. The present invention can also be applied to an image processing apparatus such as a PC that has received a plurality of images from a camera or a network via wired communication or wireless communication, and motion vector calculation processing can be performed.

  The object of the present invention can also be achieved as follows. First, a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following can be achieved. That is, based on the instruction of the read program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. Is the case. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered. Also, a computer network such as a LAN (Local Area Network) or a WAN (Wide Area Network) can be used to supply the program code.

1 is a block diagram of a camera that is an image processing apparatus according to first and second embodiments of the present invention. FIG. It is a figure for demonstrating the motion vector of an image signal. It is a figure which shows the relationship between the position where the face of a person was detected, and the position of the block which reflects weighting. It is a flowchart showing operation | movement of the camera in the 1st Embodiment of this invention. It is a flowchart showing operation | movement of the camera in the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Imaging device 101 Shooting lens 102 Diaphragm 103 Shutter 104 Image pick-up element 105 Image pick-up element drive means 106 AF drive motor 107 Shutter drive means 108 Aperture drive means 109 Focus control circuit 110 A / D conversion means 111 Signal processing circuit 112 Memory 113 Control means 114 Face detection means 115 Deviation detection means 116 Coordinate conversion means 117 Image storage means 118 Image composition means 119 Display means 120 Recording means 121 Operation means

Claims (21)

A deviation detecting means for detecting a positional deviation between a plurality of image signals;
From the image signal, according to the comparison result with the reference information, having a specific area detection means for detecting a specific area,
The deviation detecting means sets and outputs different weights for information indicating positional deviation for each area of the image signal in accordance with the detection result of the specific area by the specific area detecting means. Processing equipment.   The deviation detection means sets and outputs different weights for information indicating positional deviation for each area of the plurality of image signals according to the reliability of the detection result of the specific area of the specific area detection means. The image processing apparatus according to claim 1.   3. The image processing according to claim 2, wherein the shift detection unit sets and outputs a larger weight to information indicating the shift in a region where the reliability of the detection result of the specific region detection unit is higher. apparatus.   The specific area detection means determines the type of the detected specific area, and sets and outputs different weights for information indicating positional deviation for each area of the plurality of image signals according to the determined type. The image processing apparatus according to claim 1, wherein:   The specific area is a person's face, and the specific area detecting unit detects an area where the person's face exists according to a comparison result with reference information indicating a shape of the person's face. The image processing apparatus according to claim 1.   The specific area is a person's face, and the specific area detecting means determines an individual from the detected person's face according to a comparison result with reference information indicating the shape of the individual's face, and determines the determined individual. 5. The image processing apparatus according to claim 4, wherein different weights are set for the information indicating the positional deviation and output.   7. The displacement detection unit according to claim 1, wherein the displacement detection unit calculates a displacement of the entire image signal using information indicating a displacement in which different weights are set for each region of the image signal. An image processing apparatus according to 1. A deviation detecting means for detecting a positional deviation between a plurality of image signals;
A face area detecting means for detecting an area in which a person's face exists in accordance with a comparison result with information indicating the shape of the person's face from an image signal;
The deviation detecting means outputs either information indicating a positional deviation of an area where the human face of the image signal is present or information indicating a positional deviation of an area different from the area where the human face of the image signal is present. An image processing apparatus.   The image processing apparatus according to claim 7, wherein an area having a higher reliability of the detection result of the face area detecting unit sets and outputs a larger weight for information indicating positional deviation.   8. The face area detection unit determines an individual from the detected person's face, and sets and outputs different weights for information indicating positional deviation according to the determined individual. The image processing apparatus described.   The deviation detecting means uses either information indicating a positional deviation of an area where the human face of the image signal is present or information indicating a positional deviation of an area different from the area where the human face of the image signal is present. The image processing apparatus according to claim 8, wherein a position shift of the entire image signal is calculated.   The image processing apparatus is a camera provided with an image sensor, and the specific area detecting means detects a specific area for focus adjustment or photometry at the time of shooting, and an image signal obtained by shooting. The image processing apparatus according to claim 1, wherein a detection criterion for detecting a specific area for performing signal processing on the image processing apparatus is different.   The image processing apparatus is a camera including an image sensor, and the deviation detection unit sets and outputs different weights for information indicating positional deviation according to information on movement and orientation of the image processing apparatus itself. An image processing apparatus according to claim 1, wherein: A displacement detection step for detecting a displacement between a plurality of image signals;
From the image signal, according to the comparison result with the reference information, having a specific area detection step of detecting a specific area,
According to the detection result of the specific area in the specific area detection step, the deviation detection step sets and outputs different weights for information indicating positional deviation for each area of the image signal. Image processing method.   In the shift detection step, different weights are set and output for information indicating the shift for each region of the plurality of image signals according to the reliability of the detection result of the specific region in the specific region detection step. The image processing method according to claim 14.   In the specific area detecting step, the type of the detected specific area is determined, and according to the determined type, different weights are set and output for the information indicating the positional deviation for each area of the plurality of image signals. The image processing apparatus according to claim 14, wherein:   The specific area is a person's face, and the specific area detecting step detects an area where the person's face exists according to a comparison result with reference information indicating a shape of the person's face. The image processing method according to claim 14.   The specific area is a person's face, and in the specific area detection step, an individual is determined from the detected person's face according to a comparison result with reference information indicating the shape of the individual's face, and the determined individual The image processing method according to claim 16, wherein different weights are set for the information indicating the positional deviation and output accordingly.   19. The displacement detection step according to claim 14, wherein the displacement of the entire image signal is calculated using information indicating a displacement in which different weights are set for each region of the image signal. An image processing apparatus according to 1. A displacement detection step for detecting a displacement between a plurality of image signals;
A face region detection step of detecting a region where the human face exists according to a comparison result with information indicating the shape of the human face from the image signal;
In the deviation detection step, either information indicating a positional deviation of an area where the human face of the image signal exists or information indicating an positional deviation of an area different from the area where the human face of the image signal exists is output. An image processing method.   21. A program executable by an information processing apparatus, comprising program code for realizing the image processing method according to claim 14.

Images