JP2010072559A - Imaging apparatus and object area extracting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To correctly select a focusing state related to an object from focusing states detected in a focus detection position corresponding to the area of the object. <P>SOLUTION: The imaging apparatus includes: an obtaining means 23 which obtains image information in a screen where the focus detection position for detecting the focusing state of an focusing optical system is set; detecting means 15 and 24d which estimate the area of the object from the image information and detect the focusing state in the focus detection position corresponding to the area of the object; and an eliminating means 24d which eliminates from the area of the object the part of the focus detection position where the focusing state which is not in a predetermined range of the focusing states is detected. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an object region extraction method.

The skin color area is detected by image analysis and face area candidates are extracted, the distance information of the shooting optical system is acquired, the maximum size of the face that is actually shot is estimated, and the skin color that exceeds this maximum size An imaging device is known in which a true face area is extracted by excluding it from a face area candidate because the area is not a face area (see, for example, Patent Document 1).
JP 2004-185555 A

However, in the conventional imaging apparatus, when the focus of the photographing optical system is adjusted based on the defocus amount detected in the focus detection area in the extracted face area, and the person's face area is focused, the following problems occur. is there.
For example, in a shooting scene where there is an obstacle in front of a person and the person overlaps the obstacle, if the focus detection area is on the boundary between the person and the obstacle, the near priority AF There is a problem of focusing on an obstacle.

(1) The invention of claim 1 is an acquisition means for acquiring image information within a screen in which a focus detection position for detecting a focus adjustment state of the imaging optical system is set; A detection means for estimating a region and detecting a focus adjustment state at a focus detection position corresponding to the region of the object; and a portion of the focus detection position where a focus adjustment state that is not within a predetermined range is detected. And an exclusion device that excludes from the region of the object.
(2) According to a second aspect of the present invention, in the imaging apparatus according to the first aspect, the exclusion unit determines the predetermined range based on at least one of a variance and an average of a plurality of focus adjustment states.
(3) The invention according to claim 3 is the imaging device according to claim 1 or 2, wherein the subject is closest to the focus detection position corresponding to the area of the object after being excluded by the exclusion means. The focus adjustment of the imaging optical system is performed based on the focus adjustment state detected at the focus detection position.
(4) According to the invention of claim 4, in the imaging device according to any one of claims 1 to 3, the detection means estimates at least one allowable range of the size and shape of the object, and the allowable range The focus adjustment state is detected at the focus detection position corresponding to the region of the object inside.
(5) According to a fifth aspect of the present invention, in the imaging apparatus according to any one of the first to fourth aspects, the detection means is based on a shift amount of a pair of images by a pair of light beams passing through the imaging optical system. To detect the focus adjustment state.
(6) The invention according to claim 6 is the imaging apparatus according to any one of claims 1 to 5, wherein the detecting means is configured to detect the object based on at least one of color information and luminance information of the image information. Estimate the region.
(7) According to the invention of claim 7, in the imaging device according to any one of claims 1 to 6, the detection means estimates a region of the object based on information on a distance from the object.
(8) In the invention according to claim 8, in the imaging apparatus according to any one of claims 1 to 7, the detection unit estimates the area of the object based on the image magnification of the object.
(9) The invention of claim 9 sets the focus detection position in the screen imaged by the imaging optical system, estimates the area of the object from the image information in the screen, and sets the area of the object. The focus adjustment state of the imaging optical system is detected at a plurality of corresponding focus detection positions, and the portion of the focus detection position where the focus adjustment state that is not within the predetermined range is detected from the region of the object. This is an object region extraction method to be excluded.

  According to the present invention, the focus adjustment state related to the object can be correctly selected from the focus adjustment states detected at the focus detection position corresponding to the region of the object.

  An embodiment in which the image pickup apparatus of the present invention is applied to an automatic focus adjustment (AF) single-lens reflex digital still camera will be described. FIG. 1 is a cross-sectional view illustrating a configuration of an imaging apparatus according to an embodiment. In FIG. 1, illustration and description are omitted for devices other than the apparatus and apparatus related to the imaging apparatus of the present invention. In a camera according to an embodiment, a lens barrel 2 is detachably attached to a camera body 1, and the lens barrel 2 can be replaced with a lens barrel incorporating various interchangeable lenses.

  The camera body 1 includes a main mirror 11, a sub mirror 12, a shutter 13, an image sensor 14, a focus detection device 15, a diffusion screen 16, a condenser lens 17, a penta roof prism 18, an eyepiece lens 19, a photometric optical element 20, a diffractive optical element 21, A photometric lens 22, a photometric sensor 23, a body drive control device 24, and the like are provided. The image sensor 14 is composed of a CCD, a CMOS, or the like, and converts the subject image formed by the photographing lens 31 into an electric signal and outputs it. The focus detection device 15 detects the focus adjustment state of the photographic lens 31, that is, the defocus amount. The body drive control device 24 includes a microcomputer (not shown), a ROM, a RAM, an A / D converter, and the like, and performs various camera calculations, sequence control, image sensor drive control, and the like.

  On the other hand, the lens barrel 2 includes a photographing lens 31 (including a zooming lens and a focusing lens), a diaphragm 32, a lens drive control device 33, and the like. The lens drive control device 33 includes a microcomputer (not shown), ROM, RAM, a lens drive actuator, an aperture drive actuator, and the like, and performs focus adjustment of the taking lens 31, aperture adjustment of the aperture 32, and the like. The body drive control device 24 of the camera body 1 and the lens drive control device 33 of the lens barrel 2 are electrically connected via an electrical contact (not shown) provided on the mount portion of the lens barrel 2. And exchange information.

  At the time of non-photographing, as shown by a broken line in FIG. 1, the main mirror 11 and the sub mirror 12 are placed in the photographing optical path, and part of the subject light transmitted through the photographing lens 31 is reflected by the main mirror 11 and is reflected on the diffusion screen 16. The object image is formed by being guided. This subject image is guided to the photographer's eyes through the condenser lens 17, the penta roof prism 18, and the eyepiece lens 19, and is visually recognized by the photographer. The subject image on the diffusion screen 16 is guided to the photometric sensor 23 through the condenser lens 17, the penta roof prism 18, the photometric optical element 20, the diffractive optical element 21, and the photometric lens 22, and is captured by the photometric sensor 23.

  On the other hand, the remaining part of the subject light is transmitted through the central transmission part of the main mirror 11, reflected by the sub-mirror 12, and guided to the focus detection device 15. The focus amount is detected. This focus detection device 15 is a pupil detection phase difference detection type focus detection device, and detects the defocus amount in each of a plurality of focus detection areas in the photographing screen.

  At the time of shooting, as shown by the solid line in the figure, the main mirror 11 and the sub mirror 12 are retracted from the shooting optical path, and the subject image is shot by the image sensor 14. The subject image signal output from the image sensor 14 is processed by an image processing device (not shown), and the subject image is recorded in a recording device such as a memory card (not shown).

  FIG. 2 is a control block diagram illustrating operations of photometry, person area detection, and autofocus (AF) according to an embodiment. The body drive control device 24 includes a photometry sensor control unit 24b, a photometry calculation unit 24c, a face area detection unit 24d, an autofocus unit 24e, an AWB (auto white balance) control unit 24f, according to a software form of a microcomputer (not shown). The exposure control unit 24g and the like are configured, and an A / D converter 24a is provided.

  When performing photometry of the object scene, the photometry sensor control unit 24b sets the charge accumulation time and the amplifier gain of the photometry sensor 23 so that the maximum luminance of the object scene becomes the target photometry value. Feedback control. The AWB control unit 24f calculates an AWB control parameter based on the photometric value of the photometric sensor 23, and performs AWB control. The photometric calculation unit 24c performs photometric calculation based on the photometric value obtained by the photometric sensor 23, and calculates a shutter speed and an aperture value corresponding to the subject brightness. The exposure control unit 24g controls the shutter 13 and the aperture 32 based on the shutter speed and the aperture value calculated by the photometry calculation unit 24c.

  In this embodiment, a captured image by the photometric sensor 23 is used instead of a captured image by the image sensor 14 (see FIG. 1) for detecting a human region (for example, a human face region) in the image. The signal of the subject image picked up by the photometric sensor 23 is converted into a digital signal by the A / D converter 24a of the body drive control device 24 and then input to the person region extracting unit 24d. When detecting the person area, the photometric sensor control unit 24b sets the charge accumulation time and the amplifier gain of the photometric sensor 23 so as to obtain the average luminance of the object scene, and performs feedback control of the photometric sensor 23. Thereby, for example, the face of a person becomes sufficiently bright even under backlight.

  The person area extraction unit 24d detects color information and luminance information of the captured image by the photometric sensor 23, and extracts person area candidates (for example, human face area candidates) in the image based on these information. In addition, the person area extraction unit 24d receives the distance information (focal length, shooting distance (subject distance)) and distance error (focal length and focus distance) from the lens driving control device 33 through the autofocus unit 24e from the lens driving control device 33 (see FIG. 1). Error in the shooting distance)) and a defocus amount from the focus detection device 15 to calculate the image magnification (shooting magnification) based on the distance information and correct the image magnification based on the defocus amount. Then, based on the corrected image magnification, it is determined whether the person area candidate is a true person area. If a plurality of AF areas correspond to the person area when extracting the person area, the variance of the defocus amount is calculated and the AF area far from the average value of the defocus amount is calculated from the person area. Perform exclusion process. Then, the defocus amount of the focus detection area corresponding to the area determined as the true person area is output to the AF unit 24e.

  The AF unit 24e controls the lens drive control unit 33 based on the defocus amount of the focus detection area corresponding to the true human area, and the lens drive control unit 33 drives the focusing lens of the photographing lens 1 to adjust the focus. Do.

The principle of correcting the image magnification by the defocus amount will be described with reference to FIG. FIG. 3A is a diagram illustrating an outline of the imaging optical system of the imaging apparatus according to the embodiment. The subject distance (shooting distance) when the photographing lens 31 (see FIG. 1) is focused on the subject. Indicates the focal length. The lens drive control device 33 (see FIGS. 1 and 2) detects the position of a zooming lens (not shown) of the photographing lens 31 by a sensor (not shown) to acquire a focal length, and is provided in the lens barrel 2. The state of the distance ring (not shown) is detected by an encoder to obtain the subject distance. The body drive control device 24 obtains these distance information (focal length and subject distance (shooting distance)) from the lens drive control device 33, and calculates the shooting magnification M by the following equation.
M = (focal length) / (subject distance) (1)

  Here, the subject distance (photographing distance) is strictly the distance from the object plane (subject plane) to the front focal point of the photographing optical system, but in this specification, it will be described simply as the subject distance (photographing distance).

FIG. 3B is a diagram illustrating an outline of the focus detection optical system of the imaging apparatus according to the embodiment, and illustrates the subject distance and the focal length when the photographing lens is not in focus with respect to the subject. A pair of focus detection light fluxes from the subject that has passed through the photographic lens 31 passes through a separator lens (not shown) of the focus detection device 15 and then forms an image on the pair of line sensors 15a and 15b. The phase difference between the pair of images is detected by 15b to calculate the defocus amount. In this embodiment, the deviation α of the image magnification M due to the defocus amount is calculated by the following equation, and the deviation α is subtracted from the image magnification M to calculate an accurate image magnification M ′ when out of focus. .
α = (defocus amount) / (focal length) (2),
M ′ = M−α (3)

  FIG. 4 is a diagram in which an AF area is superimposed on an image captured by the photometric sensor 23 and shows a case where a wall A is on the left side of the person A. With reference to this figure, a method of correcting the image magnification when there are a plurality of AF areas in the person area will be described. When performing close-priority AF control, if there is a wall A or the like in front of a person, lens driving is performed to focus on the closest wall. When the focus position falls within a predetermined threshold value, a person area extraction process is performed and a defocus amount is detected. At this time, since there is a distance difference between the wall A and the person A, the defocus amount of the person A is larger than the defocus amount of the wall A.

  In the person area, AF areas 1, 2, 5, and 7 exist. In this case, the average value of the defocus amounts of the AF areas 1, 2, 5, and 7 is obtained, the defocus amount closest to the average value is adopted, and an accurate image magnification is obtained by the above equations (2) to (3). Suppose you want. In the case of FIG. 4, since the person A and the wall A do not overlap, the defocus amount in the AF area of the person area is almost the same value.

FIG. 5 shows an example of a captured image when there is a wall (obstacle) in front of the person to be imaged. As shown in FIG. 5A, when a person area extraction process is performed in a shooting scene in the case where a wall B serving as an obstacle is in front of the person A on the left side, the person information is extracted based on the color information and the luminance information. A person region as shown in (b) (see (b) hatched part in the center of the figure) is extracted. In this state, the focal length and subject distance of the photographic lens 31 are obtained, and the image magnification is calculated based on the information. Next, the defocus amounts defocus [m] (m = 1, 5, 7) of the AF areas 1, 5, and 7 included in the extracted person area and their average value defAve are detected, and the image obtained from the lens information. Correction of the image magnification corresponding to the deviation due to the defocus amount from the magnification is performed. Further, the variance V of the defocus amount is calculated by the following equation.
V = Σ (defocus [m] −defAve) 2 / n (4)
In equation (4), n is the number of AF areas included in the extracted person area, and n = 3.

Then, in the AF areas 1, 5, and 7, the AF area indicating the defocus amount that is greater than or less than the variance V is excluded from the person region with the defocus amount average value defAve as the center. It is preferable to use variance for the exclusion determination because it is less affected by the unevenness of the human face. Specifically, an AF area with a defocus amount that satisfies the exclusion determination condition expressed by the following equation is excluded from the person region.
defocus [m] ≦ defAve−√V or defAve + √V ≦ defocus [m] (5)
As shown in FIG. 6, the photometric sensor 23 has a plurality of pixels arranged on a two-dimensional plane, and the four pixels of the photometric sensor 23 are associated with each AF area in advance. When the AF area 1 among the AF areas 1, 5, and 7 included in the person region shown in FIG. 5B satisfies the exclusion determination condition of the above expression (5), as shown in FIG. Four pixels corresponding to the AF area 1 are excluded from the person area to determine a true person area. Note that the accuracy of region extraction may be set by multiplying (defAve−√V) or (defAve + √V) in the above equation (5) by a coefficient.

  FIG. 7 is a flowchart illustrating an imaging operation according to an embodiment. The imaging operation of one embodiment will be described with reference to this flowchart. The body drive control device 24 starts an imaging operation when a half-press switch (not shown) of the release button is turned on in step 11. Note that in step 21, if there is no release half-press operation for a predetermined time after the imaging apparatus is turned on, the imaging operation is terminated.

  In step 12, the photometry processing subroutine shown in FIG. 8 is executed, and exposure control and person detection processing based on the photometry results are executed. Details of this photometry processing will be described later. In step 13, AF processing is performed, and focus adjustment of a focusing lens (not shown) of the photographing lens 31 is performed based on a defocus amount detection result by the focus detection device 15. In step 14, a release button full-press switch (not shown) is turned on, and when a full-press operation is detected, the process proceeds to step 15 where the main mirror 11 and the sub mirror 12 are mirrored up. In step 16, the image sensor 14 is initialized, and in step 17, charge accumulation and accumulated data are swept out. After mirroring down at step 18 after the completion of imaging, the captured image is processed at step 19, and the captured image is recorded on a recording medium (not shown) at step 20.

  FIG. 8 is a flowchart showing a photometric processing subroutine. In step 101, the photometric sensor 23 performs imaging for photometry, that is, charge accumulation and readout. In step 102, it is determined whether or not the maximum value of the output of the photometric sensor 23 has reached a predetermined level. If the predetermined level has not been reached, the process returns to step 101, where charge accumulation time and amplifier gain are reset, and charge accumulation and readout are performed again. If the maximum photometric value has reached a predetermined level, the process proceeds to step 103 to perform photometric calculation. In this photometric calculation, the luminance of the subject is calculated based on the output of the photometric sensor 23, and an appropriate exposure value is calculated.

  As described above, in this embodiment, in order to detect a person area from a captured image, the charge accumulation time of the photometric sensor 23 and the gain of the output amplifier are set so that the average luminance of the object scene is obtained, and the photometric sensor 23 is feedback controlled. In step 104, in order to detect a person area based on the photometric result in step 101, a charge accumulation time and an amplifier gain are set such that the object scene has an average luminance, and charge accumulation and readout are performed. In step 105, a person area detection processing subroutine shown in FIG. 8 is executed to specify a person area in the captured image.

  FIG. 9 is a flowchart showing a person area detection processing subroutine. In step 201, white balance correction is performed on the output data of the photometric sensor 23. In step 202, an area that is considered to be a person (for example, a human face area) is extracted based on the color information and luminance information of the captured image. In the next step 203, grouping processing is performed on the person area. This grouping process is performed in order to individually classify areas that are considered to be persons when there are a plurality of persons. In step 204, it is determined whether or not the defocus amount as a result of the AF process (see step 13 in FIG. 7) has become smaller than a predetermined threshold value. That is, when the subject is in focus to some extent, the master data of the area that seems to be a person is created.

  If the defocus amount is smaller than the predetermined threshold value, the process proceeds to step 205, and the defocus amounts of all AF areas are acquired. As described in step 11 of FIG. 7, since the AE process and the AF process become a loop process after the release button is pressed halfway, the AF process performed at the closest timing or obtained immediately before is acquired. Read the defocus amount. In step 206, the image magnification M is calculated by the above equation (1) based on the distance information (focal length, subject distance, distance error) obtained from the lens drive control device 33. In step 207, the person area is associated with the AF area, and the image magnification M is corrected by the defocus amount detected in the AF area corresponding to the person area according to the expressions (2) and (3). Then, the corrected image magnification M ′ is obtained. If a plurality of AF areas correspond to the person area, the image magnification M is corrected using the average value of the defocus amounts to obtain a corrected image magnification M ′.

  In step 208, the variance V of the defocus amount of the person area is calculated by the above equation (4). In the following step 209, the size and face width of the person are determined. Estimating the size and shape of the human area, for example, the human face based on the corrected image magnification M ′ for the human area, and calculating the allowable range of the human face size and shape based on the distance error information, It is determined whether the size and shape of the person's face is within an allowable range. Then, the person area that is not within the allowable range is deleted.

  In step 210, the reliability of the person region extraction result is calculated. In this embodiment, the reliability of the person region extraction is, for example, whether or not the same similar color as the person is scattered as a background, and the reliability is lower as the number of similar colors is larger. Further, when the luminance difference exceeds the threshold value, the reliability is low. Alternatively, it is assumed that the reliability is low when the size of the person area exceeds a threshold value. In step 211, as described above, the AF area indicating the defocus amount that is greater than or less than the variance V is excluded from the person region with the defocus amount average value defAve as the center. Specifically, an AF area with a defocus amount that satisfies the exclusion determination condition shown in the above equation (5) is excluded from the person region.

  In the AF process in step 13 of FIG. 7 after the person area detection process, according to the defocus amount of the AF area indicating that the subject is closest to the AF area corresponding to the true person area with high reliability. The focusing lens of the photographic lens 31 is driven to focus on the true person area.

  It is excluded that the AF area exclusion determination process from the human area in step 211 is performed after the determination of the person size, face width, etc. in step 209 and the calculation of the reliability of the human area extraction result in step 210. This is because, even when the number of AF areas is small before, accurate determination of the size and face width of the person and the reliability of the person area are accurately calculated before the AF area exclusion process. Note that AF area exclusion determination processing from the person area may be performed after determining the size and face width of the person, and then the reliability of the extraction result of the person area may be calculated.

  In the focus detection described above, an example in which it is determined whether to exclude the AF area from the person area based on the defocus amount variance V has been described, but instead of the variance V, an average value of the defocus amount is used. It may be determined whether or not to exclude the AF area from the person area based on the deviation from the average value. That is, an AF area in which a defocus amount whose deviation from the average value exceeds a predetermined value is detected is excluded from the person area.

  In the above-described embodiments and their modifications, all combinations of the embodiments or the embodiments and the modifications are possible.

  According to the above-described embodiment and its modifications, the following operational effects can be achieved. First, an image formed by the photographic lens 31 is picked up, and the image based on information on the image, that is, color information and luminance information, and information on the distance of the photographic lens 31, that is, focal length, photographing distance, distance error, etc. The human face area is estimated from the image area, and the defocus amount of the photographic lens 31 is detected in a plurality of AF areas in the screen of the photographic lens 31, and detected in the AF area corresponding to the human face area. It is determined whether or not the defocus amount is a defocus amount related to a person's face, and an AF area portion determined not to be a defocus amount related to the person's face is excluded from the person's face region. Then, the focus of the photographic lens 31 is adjusted based on the defocus amount detected in the AF area corresponding to the area of the true person's face, and the person's face is correctly focused. According to the embodiment, the defocus amount related to the person's face can be correctly selected from the defocus amounts detected in the AF area corresponding to the person's face area, and the photographing lens 31 can be Can be accurately focused on.

1 is a cross-sectional view illustrating a configuration of an imaging apparatus according to an embodiment. Control block diagram showing operations of photometry, person area detection and autofocus according to one embodiment The figure explaining the principle which corrects image magnification by defocus amount Figure showing the AF area superimposed on the image taken by the photometric sensor The figure which shows the example of a picked-up image when there exists a wall (obstacle) in front of the person of photography object The figure which shows arrangement of the imaging screen and AF area of the photometric sensor The flowchart which shows the imaging operation of one embodiment Flow chart showing photometric processing subroutine Flowchart showing a person area detection processing subroutine

Explanation of symbols

15; focus detection device, 23; photometric sensor, 24; body drive control device, 31; photographing lens

Claims (9)

An acquisition means for acquiring image information in a screen in which a focus detection position for detecting a focus adjustment state of the imaging optical system is set;
Detecting means for estimating an object region from the image information and detecting the focus adjustment state at the focus detection position corresponding to the object region;
An imaging apparatus comprising: an exclusion unit that excludes a portion of the focus detection position where the focus adjustment state that is not within a predetermined range from the focus adjustment state is detected from the region of the object. The imaging device according to claim 1,
The imaging apparatus according to claim 1, wherein the exclusion unit determines the predetermined range based on at least one of a variance and an average of the plurality of focus adjustment states. In the imaging device according to claim 1 or 2,
The imaging optics based on the focus adjustment state detected at the focus detection position where the subject is closest to the focus detection position corresponding to the region of the object after being excluded by the exclusion means An image pickup apparatus that performs focus adjustment of a system. In the imaging device according to any one of claims 1 to 3,
The detecting means estimates an allowable range of at least one of the size and shape of the object, and detects the focus adjustment state at the focus detection position corresponding to the area of the object within the allowable range. An imaging apparatus characterized by the above. In the imaging device according to any one of claims 1 to 4,
The image pickup apparatus, wherein the detection unit detects the focus adjustment state based on a shift amount of a pair of images by a pair of light beams passing through the imaging optical system. In the imaging device according to any one of claims 1 to 5,
The image pickup apparatus, wherein the detection unit estimates a region of the object based on at least one of color information and luminance information of the image information. In the imaging device according to any one of claims 1 to 6,
The image pickup apparatus, wherein the detection unit estimates a region of the object based on information on a distance to the object. In the imaging device according to any one of claims 1 to 7,
The image pickup apparatus, wherein the detection unit estimates a region of the object based on an image magnification of the object. Set the focus detection position in the screen imaged by the imaging optical system,
Estimate the area of the object from the image information in the screen,
Detecting a focus adjustment state of the imaging optical system at a plurality of the focus detection positions corresponding to the region of the object;
An object region extraction method comprising: excluding, from the region of the object, a portion of the focus detection position where the focus adjustment state that is not within a predetermined range is detected.

Images