JP2019008830A - Subject detection device, imaging device, and program - Google Patents

Abstract

To extract an accurate specific area according to a subject.SOLUTION: A subject detection device includes: a binarization processing unit that creates a plurality of different binarized images by using an image of a processing object; an evaluation value calculation unit that calculates evaluation values indicating likelihood of a subject respectively for pixel areas by using intensity of pixels corresponding to the image areas acquired from the images before binarization processing with respect to the pixel areas extracted from the respective binarized images; and an extraction unit that extracts the pixel areas corresponding to a specific area of the image of the processing object by using the evaluation values.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a subject detection device, an imaging device, and a program.

  Conventionally, various techniques relating to focus adjustment have been considered. For example, the invention of Patent Document 1 discloses an imaging apparatus that performs focus adjustment using a subject extraction technique. In the invention of Patent Document 1, it is possible to optimally perform automatic focus adjustment on a subject even when the subject moves in the camera direction by deforming the AF distance measurement frame in accordance with the amount of change in the subject shape. Yes.

JP 2009-069748 A

  By the way, when extracting a subject, when a specific region (for example, a subject region, a region of interest, etc.) targeted by the user is extracted, the region that is the background may also be extracted.

  An object detection apparatus according to one aspect includes a binarization processing unit that generates a plurality of different binarized images using an image to be processed, and binarization of a pixel area extracted from each binarized image An evaluation value calculation unit that calculates an evaluation value indicating the subjectivity of the pixel region using the intensity of the pixel corresponding to the pixel region acquired from the image before processing, and the processing target using the evaluation value An extraction unit that extracts the pixel region corresponding to the specific region of the image.

  An imaging device according to one aspect includes an imaging unit that captures an image of a subject and a subject detection device according to one aspect.

  The program according to one aspect generates a plurality of different binarized images using an image to be processed, and the pixel area extracted from each binarized image is acquired from the image before binarization processing. Using the intensity of the pixel corresponding to the pixel area, an evaluation value indicating the subjectivity of the pixel area is calculated, and using the evaluation value, the pixel area corresponding to a specific area of the processing target image is extracted. Cause the computer to execute the process.

  According to the present invention, it is possible to accurately extract a specific area according to a subject.

It is a functional block diagram which shows the structural example of an imaging device. It is a functional block diagram which shows the structural example of a to-be-photographed object detection part. It is a figure which shows an example of distribution of the pixel value of the difference image of Y component. It is a flowchart which shows the process in still image shooting mode. It is a flowchart which shows the process which concerns on an object detection process. It is a flowchart which shows the process which narrows down a difference image. It is a figure which shows an example of a through image. It is a figure which shows an example of the difference image of Y component, Cb component, and Cr component obtained from the through image shown in FIG. It is a figure which shows an example of the binarized image obtained from the difference image shown in FIG. It is a flowchart which shows the process which narrows down a mask. It is a figure which shows an example of a through image. It is a figure which shows an example of the binarized image obtained from the through image shown in FIG. It is a flowchart which shows the process which extracts the mask corresponding to a to-be-photographed object. It is a figure which shows an example of a through image. It is a figure which shows an example of the binarized image obtained from the through image shown in FIG. It is a figure which shows an example of a through image. It is a figure which shows an example of the binarized image obtained from the through image shown in FIG. It is a figure which shows an example of a through image. It is a figure which shows an example of the binarized image obtained from the through image shown in FIG. It is a figure which shows the example which mapped the position of the mask each extracted from the some binarized image. It is a figure which shows the average intensity | strength of each mask shown in FIG. (A) It is a figure which shows the upper 2 masks finally extracted by embodiment. (B) It is a figure which shows the upper two masks finally extracted based only on the shape and position of a mask. It is a functional block diagram which shows the structural example of an image processing apparatus.

<First Embodiment>
Embodiments of the present invention will be described below. FIG. 1 is a functional block diagram illustrating an example of an imaging apparatus according to the present invention. In addition to the digital camera, examples of the imaging device 10 include portable terminal devices such as a mobile phone and a tablet PC having a camera function.

  The imaging apparatus 10 includes an imaging optical system 15, an optical system drive unit 16, an imaging unit 17, a focus adjustment unit 18, a control unit 19, a first memory 20, a second memory 21, a media I / F 22, a display unit 23, and an operation unit. 24. Here, the optical system drive unit 16, the imaging unit 17, the focus adjustment unit 18, the first memory 20, the second memory 21, the media I / F 22, the display unit 23, and the operation unit 24 are connected to the control unit 19. .

  The imaging optical system 15 has a plurality of lenses including, for example, a zoom lens and a focus lens. For simplicity, the imaging optical system 15 is shown as a single lens in FIG. The lens positions of the zoom lens and the focus lens included in the imaging optical system 15 are adjusted in the optical axis direction by the optical system driving unit 16, respectively. The imaging optical system 15 may be provided in a lens barrel that is detachable from the apparatus main body of the imaging apparatus 10 as represented by a viewfinder camera such as a single-lens reflex camera, or a compact camera. As represented, it may be provided in a lens barrel that is provided integrally with the apparatus main body of the imaging apparatus 10.

  The imaging unit 17 is a module that captures (captures) an image of a subject by a light beam incident through the imaging optical system 15. The imaging unit 17 may include, for example, an imaging element that converts an optical image formed on the photoelectric conversion surface by the imaging optical system 15 into a digital signal within the imaging element and outputs the digital signal. Further, for example, the imaging unit 17 includes an imaging device and an A / D conversion unit, and the imaging device converts an optical image formed on the photoelectric conversion surface by the imaging optical system 15 into an electrical signal and converts the image into an A / D conversion unit. The A / D converter may be configured to digitize the electrical signal converted by the image sensor and output it as a digital signal. Here, the above-described imaging element is configured by a photoelectric conversion element such as a complementary metal oxide semiconductor (CMOS), for example. For example, an RGB color filter is arranged in a pixel of the image sensor according to a known Bayer array, and an image signal corresponding to each color can be output by color separation by the color filter. Note that the imaging device may be configured to perform so-called image clipping, in which a subject image is converted into an electrical signal for a partial region of the photoelectric conversion surface.

  Here, the imaging unit 17 captures a still image for recording accompanying recording in the nonvolatile storage medium 35 in accordance with an imaging instruction from the operation unit 24. At this time, the imaging unit 17 can also shoot continuous still images for recording. In addition, the imaging unit 17 captures images for observation (through images) at predetermined time intervals during standby for capturing. The through image has a lower resolution (image size) due to thinning than a still image for recording. The through-image data acquired in time series is used for various arithmetic processes by the control unit 19 and moving image display (live view display) on the display unit 23. Note that an image acquired by the imaging unit 17 may be referred to as a captured image.

  The focus adjustment unit 18 uses the focus detection area information set in the imaging range of the imaging unit 17 to perform automatic focus adjustment (Auto Focus) of the focus lens.

  For example, the focus adjustment unit 18 may perform AF by well-known contrast detection using a through image. Alternatively, the focus adjustment unit 18 may execute AF based on the image shift amount of the subject image divided into pupils by a known phase difference detection method. In the case of using the phase difference detection method, the focus adjustment unit 18 uses a part of the light beam incident from the imaging optical system 15 and performs focus detection independently from the imaging unit 17 (for example, mounted on a single-lens reflex camera). Focus detection module). Alternatively, a light receiving element for focus detection may be disposed on the light receiving surface of the imaging unit 17 so that the focus adjusting unit 18 performs phase difference detection AF on the imaging surface.

  The control unit 19 is a processor that comprehensively controls the operation of the imaging apparatus 10. For example, the control unit 19 performs shooting control of a captured image by the imaging unit 17, automatic exposure control, image display control by the display unit 23, and image recording by the first memory 20 and the media I / F 22. Take control. The control unit 19 has an image processing unit 31. The image processing unit 31 performs image processing such as color interpolation, gradation conversion, white balance correction, edge enhancement, and noise removal on the data of the through image and the captured image. The image processing unit 31 includes a function of a subject detection unit 32 as an example of a subject detection device in addition to the above-described image processing function. In the first embodiment, the image processing unit 31 having the function of the subject detection unit 32 is used. However, the subject detection unit 32 can be provided as a function different from that of the image processing unit 31.

  The subject detection unit 32 specifies the position, shape, and size of the subject included in the image using the image captured by the imaging unit 17. For example, the subject detection unit 32 specifies the position, shape, and size of the subject included in the image using a color through image.

  Here, each functional block of the control unit 19 shown in FIG. 1 can be realized by an arbitrary processor, memory, or other electronic circuit in hardware, and is realized by a program loaded in the memory in software. The image processing unit 31 and the subject detection unit 32 described above are program modules processed by the control unit 19. However, at least one of the image processing unit 31 and the subject detection unit 32 may be an ASIC (Application Specific Integrated Circuit) or the like.

  The first memory 20 is a buffer memory that temporarily stores captured image data in the pre-process and post-process of image processing. For example, the first memory 20 is a volatile memory such as an SDRAM. The second memory 21 is a memory that stores a program processed by the control unit 19 and various data used in the program. For example, the second memory 21 is a nonvolatile memory such as a flash memory.

  The media I / F 22 has a connector for connecting a nonvolatile storage medium 35. Then, the media I / F 22 executes writing / reading of data (recording image) with respect to the storage medium 35 connected to the connector. The storage medium 35 is, for example, a hard disk or a memory card incorporating a semiconductor memory. Note that when the media I / F 22 optically reads or writes data to the storage medium 35, the storage medium 35 is an optical disk.

  Next, the configuration of the subject detection unit 32 will be described with reference to FIG. The subject detection unit 32 includes a color space conversion unit 41, a resolution conversion unit 42, a difference image generation unit 43, an image determination unit 44, a binarization processing unit 45, a mask narrowing unit 46, an evaluation value calculation unit 47, and a mask extraction unit. 48 is provided. Here, each functional block of the subject detection unit 32 may be a program module processed by the control unit 19 or may be an ASIC or the like.

  The color space conversion unit 41 performs a color space conversion process on the input through image data. The image data input to the control unit 19 is, for example, data represented in an RGB color space. Therefore, the color space conversion unit 41 converts input image data from data represented in the RGB color space to data represented in the YCbCr color space.

  The resolution conversion unit 42 performs resolution conversion processing on the data of the through image that has been subjected to the color space conversion processing. By this resolution conversion processing, the input through image has a lower resolution than the original resolution, in other words, the image size of the through image is reduced.

  In the first embodiment, the subject detection unit 32 includes a color space conversion unit 41 and a resolution conversion unit 42. However, data of a through image that has been subjected to the color space conversion process and the resolution conversion process is the subject detection unit. In the case where it is assumed that the data is input to 32, the configuration of the color space conversion unit 41 and the resolution conversion unit 42 can be omitted. Further, if it is assumed that the color space conversion process and the resolution conversion process are executed together as a pre-process in the subject detection process in one process, the configuration of the color space conversion unit 41 and the resolution conversion unit 42 is one configuration. (As a configuration of the image conversion unit).

  The difference image generation unit 43 calculates the average value of the pixel values of the through image represented in the YCbCr color space for each of the Y component, the Cb component, and the Cr component. The difference image generation unit 43 generates a reference density image for each component of the Y component, the Cb component, and the Cr component by using the average value of the pixel values calculated for each component of the Y component, the Cb component, and the Cr component. To do. Here, the reference density image for each component of the Y component, the Cb component, and the Cr component has the same resolution as the resolution of the through image that has been subjected to the resolution conversion process, and the pixel value of each pixel is the same. It is.

  Here, it is known that the pixel values of the Cb component and the Cr component among the Y component, the Cb component, and the Cr component are smaller than the pixel value of the Y component. That is, among the component masks obtained by the binarization process, the Y component mask has a high evaluation value. Here, the Y component mask having a high evaluation value has many areas with high luminance such as an area where light is reflected and an area which is a background. As a result, an area where there is no subject is specified as the subject area. Therefore, in order to appropriately calculate the evaluation value of the mask generated from the binarized image of each component, the pixel values of the Cb component and the Cr component are multiplied in advance by a predetermined coefficient (for example, 3). It is also possible to keep it.

  The difference image generation unit 43 generates two difference images for each of the Y component, the Cb component, and the Cr component by using the through image subjected to the resolution conversion process and the reference density image. These two difference images are composed of a positive difference image and a negative difference image. Here, the positive difference image is an image representing a distribution of pixels having pixel values exceeding the pixel value of the reference density image in the through image subjected to the resolution conversion process, and the negative difference image is a resolution In the through image that has been subjected to the conversion process, the image represents a distribution of pixels having pixel values that are less than the pixel value of the reference density image.

  When generating the positive difference image of the Y component, the difference image generation unit 43 sets the pixel value of the pixel whose difference value exceeds 0 as the difference value, and sets the pixel value of the pixel whose difference value is 0 or less. Set to 0. Alternatively, the difference image generation unit 43 may set the pixel value of a pixel having a difference value of 0 or more as the difference value, and the pixel having the difference value less than 0 may have no difference value. In any case, in the positive difference image of the Y component, the gradation is maintained as it is in the pixel in which the pixel value of the Y component image exceeds the pixel value of the reference density image of the Y component. Here, in the positive difference image of the Y component, the magnitude of the pixel value of the pixel in which the pixel value of the Y component image exceeds the pixel value of the Y component reference density image is different from the Y component reference density image. Indicates the degree.

  Further, when generating the negative difference image of the Y component, the difference image generation unit 43 sets the pixel value of a pixel having a difference value of 0 or more to 0, and the pixel value of a pixel having a difference value of less than 0 Is set to the absolute value of the difference value. Alternatively, the difference image generation unit 43 sets the difference value to 0 for the pixel value of the pixel whose difference value exceeds 0, and sets the pixel whose difference value is less than 0 to the absolute value of the difference value. In any case, in the negative difference image of the Y component, the gradation is maintained as it is in the pixel in which the pixel value of the Y component is less than the pixel value of the reference density image of the Y component. Here, in the negative difference image, the magnitude of the pixel value at which the pixel value of the Y component image is less than the pixel value of the Y component reference density image indicates the degree of deviation from the reference density image.

  The difference image generation unit 43 generates the positive difference image and the negative difference image of the Y component by the method described above, and at the same time, the positive difference image and the negative difference image of the Cb component, and the positive difference image of the Cr component and A negative difference image is generated. Therefore, the difference image generation unit 43 generates a total of six difference images.

  In the above description, the difference image generation unit 43 generates negative difference images of the Y component, the Cb component, and the Cr component, respectively, using the result of subtracting the reference density image of the corresponding component from the image of each component. However, it is not necessary to limit to this, and after subtracting the corresponding component image from the reference density image, the pixel value of the pixel whose difference value exceeds 0 is set as the difference value, and the pixel value for which the difference value is 0 or less is set. A negative difference image may be generated by setting the pixel value to 0.

  The image determination unit 44 generates a histogram for each of the generated six difference images. Then, the image determination unit 44 obtains a standard deviation σ and a peak / peak value (peak to peak) pp of the histogram for each generated histogram. Here, the standard deviation σ and the peak-to-peak value pp of the histogram serve as indices indicating the variation (or spread) of the pixel value distribution in the histogram. As described above, in the case of a positive difference image in which the pixel value of a pixel whose difference value exceeds 0 is set as the difference value, and the pixel value of the pixel whose difference value is 0 or less is set to 0, the peak / peak value For example, pp is a width from a pixel value having the highest frequency to a pixel value having the second highest frequency. In the case of a positive difference image in which a pixel value of a pixel having a difference value of 0 or more is set as a difference value and no difference value is set for a pixel having a difference value of less than 0, the peak / peak value is, for example, a histogram From 0 to the maximum value.

  Note that the above-described peak / peak value pp is an example, and any index may be used as long as it is an index indicating variations in the distribution of pixel values and frequencies in the histogram. For example, instead of the difference image, a histogram is generated for each component of the through image represented in the YCbCr color space, and the peak / peak value pp is set based on the value having the highest frequency in the histogram for each component. It is also possible to ask for it.

  The image determination unit 44 compares the standard deviation σ generated for each histogram with the threshold Th1. The image determination unit 44 also compares the peak / peak value pp for each histogram with the threshold Th2. Here, the threshold value Th1 and the threshold value Th2 are threshold values for determining variation in the distribution of pixel values of pixels included in the difference image. Further, the threshold value Th1 and the threshold value Th2 are different threshold values and are determined independently.

  For example, consider a case where a through image is acquired with a plurality of flowers in the shooting range. In this case, in the captured through image, the flower region occupies most of the image. Therefore, in the difference image of the color difference component, the frequency of the color difference component based on the color of the flower is high, and the variation in the value of the color difference component is small.

  For example, if the plurality of flowers are yellow flowers, the standard deviation σ and the peak / peak value pp in the histogram generated from the negative difference image of the Cb component are the standard deviation σ <threshold Th1 and the peak / peak value pp < The threshold value Th2. On the other hand, the standard deviation σ and the peak / peak value pp in the positive difference image of the Cb component and the histogram of the positive and negative difference images of the Cr component are standard deviation σ <threshold Th1, and peak / peak value pp <threshold Th2. It will not be. In such a case, the image determination unit 44 determines that there is no object that can be specified as a subject only for the negative difference image of the Cb component, and uses the negative difference image of the Cb component when specifying the subject. Exclude from the difference image.

  Thus, when the color of the flower is another single color, the standard deviation σ <threshold Th1 and the peak-peak value pp <Th2 are satisfied in either of the Cb component and Cr component difference images. Therefore, the image determination unit 44 excludes the difference image of the color difference component satisfying the standard deviation σ <threshold Th1 and the peak / peak value pp <Th2 from the difference image used when specifying the subject. Here, a plurality of flower images are described as examples. However, in the case of an image in which a single color component occupies most of the through image, such as a forest image, any color difference component is used. In the difference image, it is easy to make the above-described determination. Therefore, the image determination unit 44 excludes the Cb component or Cr component difference image satisfying the standard deviation σ <threshold Th1 and the peak-peak value pp <threshold Th2 from the difference image used when the subject is specified.

  On the other hand, in the through image of the imaging range including different subjects and background parts in addition to a plurality of flowers, the area occupied by the plurality of flowers is the same size or smaller than the area of the different subjects and the background part area. It becomes the size, that is, the proportion of the imaging range is reduced. In this case, in the histogram of the difference image of the color difference component, the frequency of the pixel value of the specific color is high, but the variation of the pixel value is large. In such an image, the value of the standard deviation σ is small, but the peak / peak value pp tends to be large. Such a difference image is determined as standard deviation σ <threshold Th1, and peak-peak value pp ≧ threshold Th2, or standard deviation σ ≧ threshold Th1, and peak-peak value pp <threshold Th2. Therefore, the image determination unit 44 retains the corresponding difference image without excluding it from the difference image used when specifying the subject.

  Further, in the case of a through image without a protruding color difference component, the histogram of the difference image tends to satisfy standard deviation σ ≧ threshold Th1 and peak-peak value pp ≧ threshold Th2. Also in this case, the image determination unit 44 holds the corresponding difference image without excluding it from the difference image used when specifying the subject.

  In addition, although it demonstrated demonstrating the difference image of a color difference component, the said determination is performed also about the difference image of Y component other than the difference image of Cb component and Cr component.

  Here, when the above determination is performed, all the difference images of the generated six difference images may be excluded from the difference images used when the subject is specified. In this case, the image determination unit 44 holds the six differential images once excluded as the differential images used when specifying the subject.

  The binarization processing unit 45 performs binarization processing that binarizes the held difference image with a predetermined threshold. Hereinafter, the binarization process according to the type of difference image to be held will be described.

When binarization processing is performed on a positive difference image of the Y component, the binarization processing unit 45 adds coefficients K ₁ and K ₂ (K ₁ <K) to the standard deviation σ obtained from the histogram of the positive difference image of the Y component. The values multiplied by K ₂ ) are set as threshold values K ₁ σ and K ₂ σ. After setting the threshold values K ₁ σ and K ₂ σ, the binarization processing unit 45 performs binarization processing on the positive difference image of the Y component using the threshold value K ₁ σ. Next, the binarization processing unit 45 performs binarization processing using the positive difference image of the Y component and the threshold value K ₂ σ. That is, the binarization processing unit 45 generates two binarized images from the positive difference image of the Y component.

When binarization processing is performed on the negative difference image of the Y component, the binarization processing unit 45 multiplies the standard deviation σ obtained from the negative difference image of the Y component by the coefficient K ₃ to the threshold value K _3. Set as σ. After setting the threshold value K ₃ σ, the binarization processing unit 45 performs binarization processing using the negative difference image of the Y component and the threshold value K ₃ σ. That is, the binarization processing unit 45 generates one binarized image from the negative difference image of the Y component. Here, the above-described coefficient K ₁ , coefficient K ₂ , and coefficient K ₃ are values set in accordance with shooting conditions such as a shooting scene.

When performing binarization processing on a positive difference image of the Cb component, the binarization processing unit 45 sets a threshold value L ₁ and a threshold value L ₂ (L ₁ <L ₂ ) as threshold values. Binarization processing unit 45, a threshold L ₁ and threshold L ₂ that is set, performs binarization processing using a positive difference image Cb component. That is, the binarization processing unit 45 generates two binarized images from the positive difference image of the Cb component.

Also, when performing binarization processing for the negative difference image Cb component, binarization processing unit 45, as a threshold value, it sets the threshold value L _3. Binarization processing unit 45, a threshold L ₃ that is set, performs binarization processing using a positive difference image Cb component. That is, the binarization processing unit 45 generates one binarized image from the negative difference image of the Cb component. Here, the threshold value L ₁ , the threshold value L ₂ , and the threshold value L ₃ described above are values that are set according to shooting conditions such as a shooting scene.

When the binarization process is performed on the positive difference image of the Cr component, the binarization processing unit 45 uses the threshold value M ₁ and the threshold value M ₂ (M ₁ as threshold values set for the positive difference image of the Cr component. <M ₂ ) is set. Binarization processing unit 45, a threshold M ₁ and threshold M ₂ that is set, performs binarization processing using a positive difference image Cr component. That is, the binarization processing unit 45 generates two binarized images from the positive difference image of the Cr component.

Also, when performing binarization processing for the negative difference image Cr component, the binarization processing unit 45 sets the threshold value M ₃ as the threshold. Binarization processing unit 45 performs a binarization process using the threshold M ₃ that is set. That is, the binarization processing unit 45 generates one binarized image from the negative difference image of the Cr component. Here, the above-described threshold value M ₁ , threshold value M ₂ , and threshold value M ₃ are values set in accordance with shooting conditions such as a shooting scene.

  In the first embodiment, for a positive difference image of each component, two binary images are respectively used using two threshold values, and for a negative difference image, one threshold value is used. Although a binarized image is generated, the number of threshold values to be used and the number of binarized images to be generated are not limited to the above, and may be set as appropriate.

  Hereinafter, pixels that are equal to or higher than the threshold when binarization processing is performed on each difference image are white pixels, and pixels that are less than the threshold are black pixels. The binarization processing unit 45 executes a labeling process for obtaining a group of pixels on the binarized image generated by performing the binarization process on each difference image. After performing the labeling process, the binarization processing unit 45 extracts a group of white pixels as a mask (island area).

  The mask narrowing unit 46 narrows down the masks used as subject candidates among the masks extracted from the binarized image. Hereinafter, a mask used as a subject candidate is referred to as a subject candidate mask.

  Among the masks extracted from the binarized image, there is a mask in which the area ratio of the mask to the binarized image is 1% or less. The mask narrowing unit 46 determines whether the area ratio of the mask to the binarized image is 1% or less with respect to the mask extracted from the binarized image. The mask narrowing unit 46 determines that a mask whose area ratio of the mask to the binarized image is 1% or less is noise, and excludes it from the subject candidate mask.

  Among the masks extracted from the binarized image, there is a mask having an area ratio of the mask to the binarized image of 60% or more. The mask narrowing unit 46 determines whether the area ratio of the mask to the binarized image is 60% or more with respect to the mask extracted from the binarized image. The mask narrowing unit 46 determines that the mask whose ratio of the mask area to the binarized image is 60% or more is the background, and excludes it from the subject candidate mask.

  Among the masks extracted from the binarized image, there is a mask in which the area ratio of the mask to the rectangular area including the mask is a predetermined ratio (for example, 0.2) or less. The mask narrowing unit 46 determines whether the area ratio of the mask to the rectangular area including the mask is equal to or less than a predetermined ratio with respect to the mask extracted from the binarized image. The mask narrowing unit 46 determines that the mask whose area ratio of the mask to the rectangular area including the mask is equal to or less than a predetermined ratio is a mask with a low filling rate, and excludes the mask from the subject candidate masks.

  Among the masks extracted from the binarized image, among the four sides corresponding to the outer periphery of the binarized image, the masks on adjacent two sides and the aspect ratio of the rectangular area including the mask are predetermined. Some masks are not included in the range (for example, 0.2 or more and less than 5). Therefore, when these masks are present in the extracted masks, the mask narrowing unit 46 may exclude these masks from the subject candidate masks as masks that are not suitable as subjects.

  In addition, the mask narrowing unit 46 counts the number of masks extracted from the binarized image for each binarized image. The mask narrowing unit 46 determines whether or not the number of masks extracted from one binarized image is equal to or greater than the threshold Th3. When it is determined that the number of masks extracted from the same binarized image is equal to or greater than the threshold Th3, the mask narrowing unit 46 obtains the average intensity of the entire target mask. Here, the average intensity of the entire mask includes an average value (average pixel value) of pixel values of pixels corresponding to all masks in the difference image. The mask narrowing unit 46 determines that the subject is not included in the original difference image when the obtained average intensity of the entire mask is equal to or less than the threshold Th4, and all the extracted from the corresponding difference image Are excluded from the subject candidate masks. On the other hand, when the average intensity of the entire mask exceeds the threshold Th4, the mask narrowing unit 46 determines that the subject is included in the original difference image. In this case, the target mask is held as a mask for the subject candidate.

  The mask narrowing unit 46 determines whether or not the number of masks extracted from the same binarized image is equal to or greater than the threshold value Th3 and the average intensity of the entire mask is equal to or less than the threshold value Th4. However, when it is determined whether or not the number of masks extracted from the binarized image is equal to or greater than the threshold value Th3, and the number of masks extracted from the binarized image is equal to or greater than the threshold value Th3, the corresponding binary value is determined. It is also possible to exclude the mask extracted from the converted image from the subject candidate mask.

  The evaluation value calculation unit 47 calculates an evaluation value for each of the subject candidate masks narrowed down by the mask narrowing unit 46. For example, the evaluation value is a value obtained from the mask area, the moment of inertia with respect to the mask, and the average strength of the mask. Here, the average intensity of the mask includes an average value of pixel values of pixels corresponding to the mask in the difference image.

  First, the evaluation value calculation unit 47 calculates the moment of inertia of each mask. The moment of inertia is calculated by the sum of the square of the pixel distance from the center of gravity of the mask × (0 or 1). Here, the center of gravity of the mask is the center of gravity of the subject candidate mask extracted from the same binarized image. Note that it is also possible to obtain the moment of inertia based on the center of the binarized image instead of obtaining the moment of inertia based on the center of gravity of the mask.

  Next, the evaluation value calculation unit 47 obtains the mask area and the average mask intensity, respectively. Note that the strength of the mask can be obtained from the pixel value of the pixel corresponding to the mask in the difference image from which the mask is extracted. The average intensity of the mask is an average pixel value of each pixel corresponding to the mask in the difference image from which the mask is extracted.

  Finally, an evaluation value for the mask is obtained using the following equation (1). Hereinafter, the evaluation value for the mask is referred to as Ev.

Ev = Ar ^ α / MOI + Av / β (1)
Here, “Ar” represents the area of the mask, “MOI” represents the moment of inertia of the mask, and “Av” represents the average intensity of the mask. “Α” and “β” are coefficients as tuning parameters. The coefficient α is set to about 1.5 to 2, and the coefficient β is set to about 100. However, the coefficient α and the coefficient β are not limited to the above example.

  The evaluation value Ev increases as the mask area Ar increases, and decreases as the mask inertia moment MOI increases. The evaluation value Ev becomes larger as the average intensity of the mask increases.

  The mask extraction unit 48 extracts a mask as a subject from the subject candidate masks. First, the mask extraction unit 48 ranks the subject candidate masks using the evaluation value Ev obtained for each of the subject candidate masks. The mask extraction unit 48 selects the top five masks from the ranked subject candidate masks as the final candidate masks. The mask extraction unit 48 performs the following processing on the mask selected as the final candidate.

  The mask extraction unit 48 determines whether there is a mask extracted based on the difference image of the Cb component or the Cr component among the top five masks that are final candidates. If there is a mask extracted based on the difference image of the Cb component or Cr component, the mask extracting unit 48 obtains the average intensity of the corresponding mask. The mask extraction unit 48 determines whether or not the obtained average mask intensity is equal to or less than the threshold Th5. When the average intensity of the mask is equal to or less than the threshold Th5, the mask extraction unit 48 excludes the corresponding mask from the final candidate mask. For example, the color difference component of the background region included in the through image includes a lot of yellow components and cyan components. Therefore, the threshold value Th5 in this determination is a value that is set to exclude the mask of the background portion that contains a lot of yellow and cyan components from the subject candidate mask. Here, it is assumed that the first extraction determination is to determine whether the mask is extracted based on the difference image of the Cb component or the Cr component and the average intensity of the mask is equal to or less than the threshold Th5.

  Next, the mask extraction unit 48 has a mask extracted based on the difference image of the Y component below the mask extracted based on the difference image of the Cb component or the Cr component among the masks as final candidates. It is determined whether or not. If there is a mask extracted based on the difference image of the Y component below the mask extracted based on the difference image of the Cb component and the Cr component, the mask extraction unit 48 calculates the average intensity of the corresponding mask. Ask. The mask extraction unit 48 determines whether or not the obtained average mask intensity is equal to or less than the threshold Th6. This threshold Th6 is a value set to exclude a mask such as a black area such as a shadow or a white area where a color is missing from the subject candidate mask. Here, the determination as to whether or not there is a mask extracted based on the difference image of the Y component below the mask extracted based on the difference image of the Cb component and the Cr component is a second extraction determination. Here, the value of the threshold Th6 may be different depending on the type of the difference image, or may be a fixed value regardless of the type of the difference image.

  Next, the mask extraction unit 48 includes a mask extracted based on a positive difference image of the Y component and a mask extracted based on a negative difference image of the Y component among the masks as final candidates. It is determined whether or not. In this determination, when there is a mask extracted based on the positive difference image of the Y component and a mask extracted based on the negative difference image of the Y component, the mask extraction unit 48 selects the higher rank of these masks. And the lower mask is excluded from the subject candidate masks. When a high brightness (highlight) area and a low brightness (shadow) area are mixed in the image, specifically, when shooting in fine weather, the brightness of the subject is high. The shade brightness of is low. Further, the evaluation value Ev of the mask corresponding to these is higher than the evaluation value Ev of the mask corresponding to the shadow of the subject. Therefore, when the masks corresponding to these regions are the final candidate masks, the mask corresponding to the subject is left and the mask corresponding to the shadow of the subject is excluded. Here, the determination as to whether or not the mask extracted based on the positive difference image of the Y component and the negative difference image of the Y component is in the final candidate mask is a third extraction determination.

  After the first to third extraction determinations described above, the mask extraction unit 48 determines whether there is a mask in the inclusion relationship among the final candidate masks. If there is a mask having an inclusion relationship, the mask extraction unit 48 performs processing for integrating the masks having the inclusion relationship. The determination as to whether there is a mask in an inclusive relationship is the fourth extraction determination.

As described above, the binarization processing unit 45 performs binarization processing using two different thresholds for each of the positive difference images of the Y component, the Cb component, and the Cr component. FIG. 3 shows an example of a positive difference image of the threshold Y component. In the positive difference image P1 of the Y component shown in FIG. 3, a region A1 shown in FIG. 3 shows a region where the pixel value of the included pixel is higher than the threshold value K ₁ σ. When the binarization process is performed on the positive difference image P1 of the Y component using the threshold value K ₁ σ, the area A1 becomes a white pixel area, that is, a mask. In addition, a region A2 illustrated in FIG. 3 is a region of pixels that are included in the region A1 and have pixel values higher than the pixel values of other pixels included in the region A1. When the binarization process is performed on the positive difference image P1 of the Y component using the threshold value K ₂ σ, the area A2 becomes a white pixel area, that is, a mask. Here, the region A2 is a region that is extracted as a mask in the binarization process for the positive difference image P1 of the Y component using the threshold value K ₁ σ. Therefore, it can be said that the region A1 and the region A2 are in an inclusive relationship. In other words, an inclusive relationship means that when a binarization process using different threshold values using the same difference image is executed, a mask extracted from one binarized image is changed from the other binarized image. It is included in the extracted mask. When a mask having such an inclusion relationship is included in the final candidate mask, the mask extracting unit 48 integrates these masks. Here, integrating the masks means that, among the masks in an inclusive relationship, the other mask including one mask is retained and the one mask is excluded. In FIG. 3, the mask extraction unit 48 excludes the mask shown in the region A2 from the masks shown in the region A1 and the region A2, and holds the mask shown in the region A1. Although the mask generated from the positive difference image of the Y component has been described, the same applies to the mask generated from the positive difference image of the Cb component and the Cr component.

  After performing the first to fourth extraction determinations described above, if the number of masks to be retained exceeds three, the mask extraction unit 48 sets the top third mask among the masks to be used as a subject. Extract as a corresponding mask. On the other hand, when the number of held masks is three or less, all the held masks are extracted as masks corresponding to the subject. The mask extraction unit 48 identifies an area corresponding to the extracted mask as a subject area for the through image. Then, the mask extraction unit 48 outputs the subject area data indicating the identified subject area. On the other hand, when all the masks are excluded in the plurality of determinations described above, in other words, when there is no mask to be held, the mask extraction unit 48 assumes that there is no mask corresponding to the subject.

  Here, the imaging apparatus 10 according to the first embodiment displays, as an operation mode, a still image shooting mode for shooting a still image, a moving image shooting mode for shooting a moving image, and the acquired still image or moving image as a display unit 23. It has a playback mode to play back with. Hereinafter, the flow of processing in the still image shooting mode will be described with reference to the flowchart of FIG.

  Step S101 is processing for acquiring a through image. The control unit 19 drives the imaging unit 17 to capture a through image. A through image is acquired by driving the imaging unit 17. Note that through image data acquired by driving the imaging unit 17 is input from the imaging unit 17 to the control unit 19. The control unit 19 performs image processing on the through image data from the imaging unit 17. Here, the control unit 19 can also output the through image data subjected to the image processing to the display unit 23 and cause the display unit 23 to display the acquired through image.

  Step S102 is subject detection processing. The control unit 19 performs subject detection processing using the through image acquired in step S101. Thereby, the region of the subject within the imaging range of the imaging unit 17 is detected.

  Step S103 is AF control. The control unit 19 executes AF control in which the subject area detected in step S102 is set as the focus detection area. If the subject area is not obtained in the subject detection process in step S102, the control unit 19 executes AF control in which the central area of the imaging range is set as the focus detection area. Here, examples of the central area of the imaging range include an area set for landscape photography. The area set for landscape photography is an area having an area of 60% of the area of the imaging range. When executing the process of step S103, the control unit 19 may perform an AE calculation based on the set focus detection area.

  Step S104 is processing to determine whether or not there is a shooting instruction. When an instruction to shoot a still image for recording is given by operating the operation unit 24, the control unit 19 sets the result of the determination process in step S104 to Yes. In this case, the process proceeds to step S105. On the other hand, when an instruction to shoot a still image for recording is not made by operating the operation unit 24, the control unit 19 sets the result of the determination process in step S104 to No. In this case, the process proceeds to step S106.

  Step S105 is an imaging process. The control unit 19 drives the imaging unit 17 to execute a recording still image imaging process. The control unit 19 performs predetermined image processing on still image data for recording. The control unit 19 writes the still image data for recording subjected to image processing into the storage medium 35 via the media I / F 22. When this process ends, the process proceeds to step S106.

  Step S106 is processing to determine whether or not to end shooting. When an instruction to end the recording of the still image for recording is given by operating the operation unit 24, the control unit 19 sets the result of the determination process in step S106 to Yes. In this case, the process of the flowchart in FIG. 4 ends.

  On the other hand, if the instruction to end the recording of the still image for recording is not made by operating the operation unit 24, the control unit 19 sets the result of the determination process in step S106 to No. In this case, the process returns to step S101. Therefore, when the determination process in step S106 is No, the processes from step S101 to step S104 are repeatedly executed. When these processes are repeatedly executed, the imaging process in step S105 is executed as necessary. .

  Next, the subject detection process shown in step S102 of the flowchart of FIG. 4 will be described based on the flowchart of FIG.

  Step S201 is a color space conversion process. The control unit 19 performs color space conversion processing on the through image that has been subjected to image processing. As a result, the through image represented in the RGB color space is converted into a through image represented in the YCbCr color space.

  Step S202 is resolution conversion processing. The control unit 19 performs resolution conversion processing on the through image that has been subjected to the color space conversion processing in step S201. Thereby, the through image is converted to a resolution lower than the original resolution. That is, a through image with a reduced image size is generated.

  Step S203 is processing for generating a difference image. The control unit 19 generates a reference density image for each component of the Y component, the Cb component, and the Cr component, using the through image that has been subjected to the resolution conversion processing in step S202. And the control part 19 produces | generates a positive difference image and a negative difference image for every component of Y component, Cb component, and Cr component using the through image and reference density image which were subjected to resolution conversion processing To do. Note that the positive difference image and the negative difference image generated by each component of YCbCr are held in the first memory 20 or the second memory 21 under the control of the control unit 19.

  Step S204 is processing to narrow down the difference image. The control unit 19 generates a histogram from each of a total of six difference images of the positive difference image and the negative difference image of each component generated in step S203. The control unit 19 obtains the standard deviation σ and the peak / peak value using the generated histogram. Using these values, the control unit 19 narrows down the difference image used when specifying the subject. The process of step S204 will be described later.

  Step S205 is a binarization process. The control unit 19 performs binarization processing for each of the difference images narrowed down by the processing in step S204. After generating the binarized image, the control unit 19 performs a labeling process on the generated binarized image. By performing this labeling process, the control unit 19 extracts a mask from the binarized image.

  Step S206 is a process of narrowing down the mask. The control unit 19 narrows down subject candidate masks from the masks extracted from the binarized image. The process of step S206 will be described later.

  Step S207 is processing for calculating the evaluation value Ev. The control unit 19 calculates the evaluation value Ev for the mask narrowed down in step S206 using the above-described equation (1).

  Step S208 is processing for extracting a mask corresponding to the subject. The control unit 19 ranks the subject candidate masks based on the evaluation value Ev of each mask obtained in step S207. The control unit 19 selects, as the final candidate mask, the top five masks among the ranked subject candidate masks. Then, the first to fourth extraction determinations described above are performed on the final candidate mask. Then, the mask held after the extraction determination is extracted as a mask corresponding to the subject. Details of the processing in step S208 will be described later. When the process of step S208 ends, the subject detection process shown in FIG. 5 ends.

  Here, when a mask is extracted by the subject detection process described above, the control unit 19 sets a region corresponding to the extracted mask as a region of the subject. As a result, when the through image is displayed on the display unit 23, a rectangular frame (subject frame) including the region of the subject is displayed superimposed on the through image. Therefore, when the mask is extracted when the processing of step S208 is performed, the photographer recognizes the position and size of the subject included in the imaging range by displaying the subject frame superimposed on the through image. can do.

  Next, the process of narrowing down the difference image shown in step S204 in FIG. 5 will be described using the flowchart in FIG.

  Step S301 is processing for generating a histogram. By performing the process of step S203 in FIG. 5 described above, the control unit 19 generates a total of six difference images, that is, a positive difference image and a negative difference image for each of the Y component, the Cb component, and the Cr component. . The control unit 19 generates a histogram for each of these six difference images.

  Step S302 is processing for calculating the standard deviation σ and the peak / peak value pp. The control unit 19 calculates the standard deviation σ and the peak / peak value pp for each histogram of each difference image generated in step S302. By the processing in step S302, an index indicating variation in the distribution of pixel values in each histogram is calculated.

  Step S303 is processing for comparing the standard deviation σ with the threshold Th1. The control unit 19 compares the threshold value σ for each histogram obtained in step S302 with the threshold value Th1.

  Step S304 is processing for comparing the peak / peak value pp with the threshold Th2. The control unit 19 compares the peak / peak value pp for each histogram obtained in step S303 with the threshold Th2.

  Step S305 is processing to determine whether there is a difference image to be excluded. The control unit 19 determines whether there is a difference image to be excluded among the generated six difference images, using the results of the determination processing in step S303 and step S304, respectively. If there is a difference image satisfying, for example, standard deviation σ <threshold Th <b> 1 and peak / peak value pp <Th <b> 2 among the total of six difference images, the control unit 19 sets the difference image to exclude this difference image. In this case, the control unit 19 sets Yes as the result of the determination process in step S305, and sets it as step S306. On the other hand, if none of the total of six difference images satisfies the standard deviation σ <threshold Th1, and the peak / peak value pp <Th2, the control unit 19 sets the result of the determination process in step S305 to No. . In this case, it is determined that there is no difference image to be excluded. Therefore, when the result of the determination process in step S305 is No, the control unit 19 determines to use a total of six difference images as they are, and ends the process of the flowchart illustrated in FIG.

  Step S306 is processing to exclude the difference image. In step S305, it is determined that there is a difference image to be excluded. Therefore, the control unit 19 excludes the target difference image. The difference image is narrowed down by performing the process of step S306. After executing the process of step S306, the control unit ends the process of the flowchart shown in FIG.

  FIG. 7 shows an example of a through image obtained when a bird is within the imaging range, and FIG. 8 shows an example of positive and negative difference images of the Y component, Cb component, and Cr component obtained from the through image shown in FIG. Indicates. Here, FIG. 8A is a positive difference image of the Y component, and FIG. 8B is a negative difference image of the Y component. FIG. 8C shows a positive difference image of the Cb component, and FIG. 8D shows a negative difference image of the Cb component. FIG. 8E shows a positive difference image of the Cr component, and FIG. 8F shows a negative difference image of the Cr component.

  When the processing of step S301 and step S302 described above is executed, the control unit 19 generates a histogram for each of the generated six difference images, and the standard deviation σ and the peak / peak value for each histogram. pp is calculated. Here, the standard deviation σ and peak / peak value pp of the histogram based on the difference image shown in FIG. 8A are standard deviation σ <threshold Th1, and peak / peak value pp <threshold Th2. Therefore, the control unit 19 determines that there is no object corresponding to the subject in the difference image of FIG. 8A, and among the generated six difference images, the difference image shown in FIG. Are excluded from the difference image used when the subject is specified. Therefore, the difference images used when specifying the subject are narrowed down to a total of five difference images shown in FIGS. 8B to 8F.

  Then, when the processing of step S205 is executed, the control unit 19 executes binarization processing for each of a total of five difference images from FIG. 8B to FIG. 8F. By this binarization processing, a total of seven binarized images from FIG. 9C to FIG. 9I are generated. In FIG. 9, the threshold when performing binarization processing on the positive difference image of each component is 2σ and 3σ, and the threshold when performing binarization processing on the negative difference image of each component Is set to 2σ.

  In FIG. 9, two binarized images obtained from the positive difference image of the Y component are illustrated for convenience as FIGS. 9A and 9B, but the difference of FIG. When the image is excluded, the binarized image shown in FIGS. 9A and 9B is not generated. Note that the gray and white regions shown in the binarized images in FIGS. 9C to 9I indicate the masks extracted from each binarized image.

  When the process of step S206 is executed, the control unit 19 performs a process of narrowing down the mask using the masks in the binarized images of FIGS. 9C to 9I. Then, the evaluation value Ev is calculated using the narrowed-down mask, and a mask corresponding to the subject is extracted. FIG. 9 shows a case where a mask corresponding to the subject is not extracted. That is, when there is no mask corresponding to the subject, the frame 51 used for landscape photography is superimposed on the through image P2 and displayed on the display unit 23.

  In this way, in the above-described processing for narrowing down the difference image, if the through image is obtained when taking a landscape image, the pixel value of each pixel of the through image is an approximate value, that is, the contrast in the image is small. As a result, any one of the histograms based on the difference image has a small standard deviation σ and peak / peak value pp. On the other hand, in the case of a through image obtained when a person is photographed, the contrast in the background area is small, but the contrast with the person area is large. As a result, in the histogram based on the difference image, the standard deviation σ and the peak / peak value pp are large. Therefore, when the above-described standard deviation σ is compared with the threshold value Th1, or the peak-to-peak value pp is compared with the threshold value Th2, a difference image that is assumed to have no subject can be excluded. As a result, it is possible to reduce a processing load related to processing to be described later and to shorten a processing time. Further, by excluding the difference image that is assumed that the subject does not exist before the binarization process, the extraction accuracy of the mask corresponding to the subject is improved, and the specific region (for example, the subject region or the attention region) is accurately detected. Etc.) can be extracted.

  Note that the above-described processing for narrowing down the difference image is particularly useful when photographing in the close-up mode (macro mode). Therefore, the processing for narrowing the difference image may be executed in conjunction with the setting of the close-up mode (macro mode), or the processing for narrowing the difference image in conjunction with AF information, AE information, subject analysis information, or the like. May be.

  Next, the process of narrowing down the mask in step S206 shown in the flowchart of FIG. 5 will be described based on the flowchart of FIG.

  Step S401 is processing to exclude a mask recognized as noise. After obtaining the area of the mask extracted from the binarized image, the control unit 19 obtains the area ratio between the area of the binarized image and the area of the mask. The control unit 19 refers to the area ratio between the area of the binarized image and the area of the mask, and determines whether or not there is a mask in which the area ratio between the area of the binarized image and the area of the mask is 1% or less. Determine. If there is a mask in which the area ratio between the area of the binarized image and the area of the mask is 1% or less, the control unit 19 excludes the mask from the subject candidate mask as noise.

  Step S402 is processing to exclude a mask recognized as a background. The control unit 19 obtains an area ratio between the area of the binarized image and the area of the mask for each mask by the process of step S401. The control unit 19 refers to the area ratio between the area of the binarized image and the area of the mask, and determines whether there is a mask having an area ratio of 60% or more. If there is a mask in which the area ratio between the area of the binarized image and the area of the mask is 60% or more, the control unit 19 excludes the mask from the subject candidate mask as the background.

  Step S403 is processing for removing a mask with a low filling rate. The control unit 19 determines whether or not the ratio of the mask area to the area of the rectangular region including the mask is a predetermined threshold value (for example, 0.2) or less. In this determination, if there is a mask in which the ratio of the mask area to the area of the rectangular area including the mask is equal to or less than a predetermined threshold, the control unit 19 excludes the corresponding mask from the subject candidate mask.

  Step S404 is processing to count the number of masks in the binarized image. The control unit 19 counts the number of masks extracted from the binarized image.

  Step S405 is processing for determining whether or not the number of masks is equal to or greater than the threshold value Th3. When the counted number of masks is equal to or greater than the threshold Th3, the control unit 19 sets the result of the determination process in step S405 to Yes. In this case, the process proceeds to step S406. On the other hand, if the counted number of masks is less than the threshold Th3, the process proceeds to step S409.

  Step S406 is processing for calculating the average intensity of the entire mask. The control unit 19 reads the pixel value of the pixel corresponding to each mask from the difference image that is the basis of the binarized image, and calculates the average intensity of the entire mask.

  Step S407 is processing for determining whether or not the average intensity of the entire mask is equal to or less than the threshold Th4. When the average intensity of the entire mask obtained in step S406 is equal to or less than the threshold Th4, the control unit 19 sets the result of the determination process in step S407 to Yes. In this case, the process proceeds to step S408. On the other hand, when the average intensity of the entire mask obtained in step S406 exceeds the threshold Th4, the control unit 19 sets the result of the determination process in step S407 to No. In this case, the process proceeds to step S409.

  Step S408 is processing to exclude the target mask. When the determination process in step S407 is Yes, the control unit 19 determines that the mask extracted from the binarized image does not include a mask, and all the masks extracted from the binarized image. Are excluded from the subject candidate mask.

  Step S409 is processing for determining whether or not the processing has been executed for all the binarized images. When the processes from step S405 to step S408 are performed on all the binarized images, the control unit 19 sets the result of the determination process in step S409 to Yes. Thereby, the process of the flowchart of FIG. 10 is completed. On the other hand, when the processes from step S404 to step S408 have not been performed on all the binarized images, the control unit 19 sets the result of the determination process in step S409 to No. In this case, the process returns to step S404. Therefore, the processing from step S404 to step S408 is executed for all binarized images.

  FIG. 11 shows a through image P3 when the giraffe is within the imaging range. Here, in the binarized images of FIGS. 12A to 12I, regions represented by gray and white are extracted as masks. In FIG. 12, the threshold when binarization processing is performed on the positive difference image of each component is 2σ and 3σ, and the binarization processing is performed on the negative difference image of each component. The case where the threshold is set to 2σ is shown.

  The controller 19 narrows down subject candidate masks from the masks extracted from these binarized images. Here, by performing the processing of step S402 and step S403, for example, the masks of FIGS. 12B, 12D, and 12E are all excluded from the subject candidate masks.

  Here, a large number of masks are extracted from the binarized images of FIG. 12C, FIG. 12F, and FIG. Therefore, if the control part 19 performs the determination process of step S405 with respect to these binarized images, the result of the determination process will be Yes. Further, the result of the determination process in step S407 is also Ys for the average brightness of the entire mask extracted from these binarized images. Therefore, all the masks extracted from each of the binarized images of FIGS. 12C, 12F, and 12I are excluded. Therefore, when the mask narrowing process is executed, the mask shown in the white area in FIG. 12A, the two masks shown in the white area in FIG. 12G, and the mask shown in the white area in FIG. Are held as masks of subject candidates. The control unit 19 obtains an evaluation value Ev for these subject candidate masks, and ranks the subject candidate masks using the obtained evaluation value Ev. After this ranking, first to fourth extraction determinations are made, and for example, two masks shown in the white area in FIG. 12G are extracted as masks corresponding to the subject. Accordingly, when the through image of FIG. 11 is displayed on the display unit 23, the subject frames 52 and 53 are displayed in a superimposed manner.

  In this way, when a large number of masks are extracted from the binarized image, by determining the average intensity of the entire mask, whether the area where a large number of islands are extracted is due to the background texture, It is possible to determine whether the texture of the subject itself. In other words, since the average intensity of the entire mask is low for a background texture, it can be determined that the extracted mask is an area corresponding to the background and no area corresponding to the subject exists. On the other hand, if the texture of the subject itself, the average intensity of the entire mask is high. Therefore, it can be determined that the extracted mask has a mask corresponding to the subject. In this way, by performing the above determination, unnecessary masks can be excluded from subject candidate masks, and subject candidate masks can be appropriately narrowed down.

  Next, the flow of processing for extracting a mask corresponding to the subject in step S208 shown in the flowchart of FIG. 5 will be described based on the flowchart of FIG.

  Step S501 is processing for obtaining a final candidate mask. The control unit 19 ranks the subject candidate masks using the evaluation value Ev for the subject candidate masks obtained in step S207. Then, the control unit 19 selects the top five masks among the ranked masks as final candidate masks.

  Step S502 is processing for determining whether or not there is a mask extracted based on the Cr component difference image or the Cb component difference image. If there is a mask extracted based on the difference image of the Cr component and the Cb component among the final candidate masks, the control unit 19 sets the result of the determination process in step S502 to Yes. In this case, the process proceeds to step S503. On the other hand, when there is no mask extracted based on the difference image of the Cr component and the Cb component among the top five masks, the control unit 19 sets the result of the determination process in step S502 to No. In this case, the process proceeds to step S505.

  Step S503 is processing for determining whether or not there is a mask whose average intensity is equal to or less than the threshold Th5. The control unit 19 obtains the average intensity of the mask extracted for each mask based on the difference image of the Cr component or the Cb component. The control unit 19 compares the obtained average intensity of the mask with the threshold Th5. When any of the average intensities of the masks extracted based on the difference image of the Cr component or the Cb component is equal to or less than the threshold Th5, the control unit 19 sets the result of the determination process in step S503 to Yes. In this case, the process proceeds to step S504. On the other hand, when all the average intensities of the masks extracted based on the difference image of the Cr component or Cb component exceed the threshold Th5, the control unit 19 sets the result of the determination process in step S503 to No. In this case, the process proceeds to step S505. The processes in step S502 and step S503 are processes related to the first extraction determination.

  Step S504 is processing to exclude the target mask. In the determination process in step S503, the control unit 19 determines that there is a mask whose average intensity is equal to or less than the threshold Th5. Therefore, the control unit 19 determines that the mask whose average intensity is equal to or less than the threshold Th5 is a mask obtained from the background portion, and excludes the corresponding mask from the final candidate mask.

  Step S505 is a process of determining whether or not there is a mask extracted based on the Y component difference image below the mask extracted based on the Cb component or Cr component difference image. If the mask is not excluded by the process of step S504, the target mask is all the final candidate masks. On the other hand, when a mask is excluded by the process of step S504, a mask other than the excluded mask among the final candidate masks corresponds.

  The control unit 19 reads information indicating the history of processing in the subject detection processing from the first memory 20. The control unit 19 specifies whether the target mask is extracted from the read information based on any one of the Cb component, Cr component, and Y component difference images. After this specification, the control unit 19 determines whether there is a mask extracted based on the Y component difference image below the mask extracted based on the Cb component or Cr component difference image among the target masks. Determine whether or not. When there is a mask extracted based on the difference image of the Y component below the mask extracted based on the difference image of the Cb component or the Cr component, the control unit 19 sets the result of the determination process in step S505 to Yes. . In this case, the process proceeds to step S506. On the other hand, when there is no mask extracted based on the difference image of the Y component below the mask extracted based on the difference image of the Cb component or Cr component, the control unit 19 determines the result of the determination process in step S505. Is No. In this case, the process proceeds to step S508.

  Step S506 is processing for determining whether there is a mask having an average intensity equal to or less than the threshold Th6. The control unit 19 obtains the average intensity of the mask with respect to the mask extracted based on the Y component difference image below the mask extracted based on the Cb component or Cr component difference image. Then, the control unit 19 compares the obtained average intensity of the mask with the threshold Th6. If there is a mask whose average intensity is equal to or less than the threshold Th6 among the target masks, the control unit 19 sets the result of the determination process in step S506 to Yes. In this case, the process proceeds to step S507. When the average intensity of all the target masks exceeds the threshold Th6, the control unit 19 sets the result of the determination process in step S506 to No. In this case, the process proceeds to step S508. The processes in step S505 and step S506 are processes related to the second extraction determination.

  Step S507 is processing to exclude the target mask. The control unit 19 excludes the mask whose average intensity is equal to or less than the threshold Th6 from the final candidate mask.

  Step S508 is processing for determining whether there is a mask extracted based on a positive difference image of Y component and a mask extracted based on a negative difference image of Y component. The target mask is a mask obtained by removing the masks excluded by the process of either step S504 or step S507 from among the final candidate masks. Note that if neither the processing in step S504 nor step S507 is performed, all of the final candidate masks are the target masks.

  If the target mask includes a mask extracted based on the positive difference image of the Y component and a mask extracted based on the negative difference image of the Y component, the control unit 19 determines in step S508. The processing result is set to Yes. In this case, the process proceeds to step S509. On the other hand, if the target mask has only one of the mask extracted based on the positive difference image of the Y component or the mask extracted based on the negative difference image of the Y component, If there is no mask of both the mask extracted based on the positive difference image and the mask extracted based on the negative difference image of the Y component, the control unit 19 sets the result of the determination process in step S508 to No. And go to step S510. Note that the processing in step S508 is processing related to the third extraction determination.

  Step S509 is processing to exclude the lower mask. In step S508, it is determined that the target mask includes both a mask extracted based on a positive difference image of Y component and a mask extracted based on a negative difference image of Y component. Yes. Therefore, the control unit 19 holds a mask having a higher rank as a final candidate mask among these masks, and excludes a mask having a lower rank from the final candidate mask.

  Step S510 is processing for determining whether there is a mask having an inclusion relationship among masks generated from positive difference images of the same component. The target mask is a mask obtained by removing the masks excluded by any one of the processes in step S504, step S507, or step S509 from among the final candidate masks. If none of the processes in step S504, step S507, or step S509 is performed, all of the final candidate masks are the target masks. As described above, in the binarization process for the positive difference image of the Y component, the Cb component, and the Cr component, two different threshold values are used. Therefore, a mask extracted by a binarization process using a threshold for a positive difference image of the same component is included in a mask extracted by a binarization process using a threshold. Refers to the case. If there is a mask in the inclusion relationship among the target masks, the control unit 19 sets the result of the determination process in step S510 to Yes. In this case, the process proceeds to step S511. On the other hand, when there is no mask in the inclusion relationship among the target masks, the control unit 19 sets the result of the determination process in step S510 to No. In this case, the process proceeds to step S512.

  Step S511 is processing for integrating masks in an inclusive relationship. For example, when two masks are in an inclusive relationship, the control unit 19 integrates the masks. By executing the processing in step S511, the control unit 19 can narrow down the final candidate mask.

  Step S512 is processing for determining whether or not the number of masks to be held exceeds three. The control unit 19 performs the first to fourth extraction determinations on the final candidate mask by performing the processing from step S502 to step S511. For example, when no final candidate masks are excluded or only one mask is excluded, the number of masks held as final candidates exceeds three. Therefore, the control unit 19 sets the result of the determination process in step S512 to Yes. In this case, the process proceeds to step S513.

On the other hand, when two or more masks are excluded from the final candidate masks, the number of masks to be held is three or less. Therefore, the control unit 19 sets the result of the determination process in step S512 to No. In this case, the process proceeds to step S514.
Step S513 is processing to extract the top three masks. The control unit 19 extracts, from among the masks held as final candidates, the mask with the third highest evaluation value Ev as the mask corresponding to the subject area. When the process of step S513 is executed, the control unit 19 ends the process of the flowchart in FIG.

  Step S514 is processing to determine whether or not there is a held mask. By performing the first to fourth extraction determinations described above, all the masks that are final candidates may be excluded. In such a case, the control unit 19 sets the result of the determination process in step S514 to Yes. That is, in such a case, the control unit 19 determines that there is no mask corresponding to the subject area, and ends the process of the flowchart in FIG.

  On the other hand, when 2 to 4 masks are excluded from the final candidate masks when the first to fourth extraction determinations are performed, the mask held by the control unit 19 is any of 1 to 3 masks. It is. Therefore, the control unit 19 sets the result of the determination process in step S514 to No. In this case, the process proceeds to step S515.

  Step S515 is processing to extract the held mask. The control unit 19 extracts all the masks held as final candidate masks as masks corresponding to the subject area. When the process of step S515 is executed, the control unit 19 ends the process of the flowchart in FIG.

  Hereinafter, the threshold when performing binarization processing on the positive difference image of each component is 2σ and 3σ, and the threshold when performing binarization processing on the negative difference image of each component is 2σ, The case where each is set will be described.

  FIG. 14 shows a through image when a red flower is placed in the imaging range. As described above, when the through image P4 is acquired, the control unit 19 generates a total of nine binarized images of the Y component, the Cb component, and the Cr component. FIG. 15A to FIG. 15I show an example of a binarized image of Y component, Cb component, and Cr component obtained from the through image P3. In FIG. 15, “(positive)” described below each threshold is a binarized image generated based on a positive difference image, and “(negative)” described below the threshold is negative. It shows that it is a binarized image generated based on the difference image.

  FIG. 15A to FIG. 15I are binarized images of a Y component, a Cb component, and a Cr component, and regions represented by gray and white in each binarized image are extracted as a mask. By narrowing down the masks that are subject candidates from the extracted masks of the respective binarized images, the masks expressed in gray are excluded, and the binary values shown in FIGS. 15A and 15D to 15H are used. Masks indicated by white areas in the digitized image are narrowed down as subject candidate masks. As described above, when the subject candidate masks are narrowed down, the control unit 19 obtains the evaluation value Ev for each of the subject candidate masks. Here, the “x” mark shown in FIGS. 15A, 15D to 15H is the center of gravity of the mask, which serves as a reference when obtaining the moment of inertia of the mask. After obtaining the evaluation value Ev for each mask, the control unit 19 ranks the masks using the obtained evaluation value Ev. Here, the mask shown as the white area in FIG. 15A, the mask shown as the white area in FIG. 15D, the mask shown as the white area in FIG. 15F, and the white area in FIG. The mask shown in the region and the mask shown as the white region in FIG. 15H are the final candidate masks.

  The control unit 19 performs the first extraction determination on the final candidate mask. If there is a mask extracted based on the difference image between the Cb component and the Cr component in the first extraction determination, it is determined whether the average intensity of the mask is equal to or less than the threshold Th5. In FIG. 15, the mask shown as the white region in FIG. 15D and the mask shown as the white region in FIG. 15F are masks extracted based on the difference image of the Cb component. Further, the mask shown as the white area in FIG. 15G and the mask shown as the white area in FIG. 15H are masks extracted based on the difference image of the Cr component. In this first extraction determination, for example, the mask shown as the white area in FIG. 15D and the mask shown as the white area in FIG. 15F are excluded from the final candidate mask.

  Next, the control unit 19 performs a second extraction determination on the target mask. In this second extraction determination, first, if there is a Y component mask in the target mask, it is determined whether or not the Y component mask is located below the Cb component or Cr component mask. The For example, if the mask shown as the white area in FIG. 15A is lower in rank than the mask shown as the white area in FIG. 15G and the mask shown as the white area in FIG. The control unit 19 determines whether or not the average intensity of the mask shown as the white region in FIG. 15A is equal to or less than the threshold Th6. In this determination, when it is determined that the average intensity of the mask shown as the white area in FIG. 15A is equal to or less than the threshold Th6, the control unit 19 performs the mask shown as the white area in FIG. Are excluded from the final candidate mask. That is, the mask shown as the white region in FIG. 15G and the mask shown as the white region in FIG. 15H are held as final candidate masks.

  Next, the control unit 19 performs a third extraction determination. Here, the masks held as final candidate masks are a mask shown as a white region in FIG. 15G and a mask shown as a white region in FIG. These masks are extracted based on the difference image of the Cr component, and are not extracted based on the positive difference image and the negative difference image of the Y component. Therefore, no mask is excluded from the final candidate mask by the third extraction determination.

  Finally, the control unit 19 performs a fourth extraction determination. In the example of FIG. 15, the masks retained after the third extraction determination is performed are a mask shown as a white region in FIG. 15G and a mask shown as a white region in FIG. These masks are respectively extracted from the binarized images generated by using different threshold values for the positive difference image of the Cr component. Further, the mask shown as the white region in FIG. 15G includes the mask shown as the white region in FIG. Therefore, the control unit 19 determines that the mask shown as the white region in FIG. 15G and the mask shown as the white region in FIG. Then, the control unit 19 integrates the mask shown as the white region in FIG. 15G and the mask shown as the white region in FIG. 15H into the mask shown as the white region in FIG. Here, as a result of performing the first to fourth extraction determinations, the masks retained as the final candidate masks are only the masks shown as white regions in FIG. The control unit 19 extracts the mask shown as the white area in FIG. 15G as a mask corresponding to the subject.

  Here, the mask shown as a white region in FIG. 15G corresponds to a flower region in the through image. Accordingly, as shown in FIG. 14A, when the through image is displayed on the display unit 23, the subject frame 55 is superimposed on the flower area.

  For example, in the conventional method, if the mask shown as the white region in FIG. 15A and the mask shown as the white region in FIG. The area corresponding to the mask shown as the white area in FIG. 15A and the area corresponding to the mask shown as the white area in FIG. Note that a region corresponding to a mask shown as a white region in FIG. 15A is a region where light is reflected. Further, the area corresponding to the mask shown as the white area in FIG. 15G is a flower area. Therefore, as shown in FIG. 14B, when the through image is displayed on the display unit 23, the subject frames 55 and 56 are superimposed on the flower area and the area where the light is reflected. . However, the region where the light is reflected is a region with high luminance and is not a region where the subject is located. In other words, according to the conventional method, even if there is no subject, if the luminance is high, it is extracted as a mask corresponding to the subject region.

  However, in the first embodiment, by the ranking of the evaluation values Ev described above, the mask extracted based on the luminance component difference image lower than the mask extracted based on the Cb component or Cr component difference image. Is located, the mask extracted based on the luminance component difference image is excluded based on the average intensity of the mask. Therefore, even if an area with high luminance is extracted as a mask, if it is determined that the rank is lower than that of the chromaticity component mask, the mask is easily excluded. Therefore, it is possible to prevent a mask that has no subject and is simply a high luminance region from being identified as a region corresponding to the subject.

  FIG. 16 is a through image P5 in which an amusement park vehicle is within the imaging range. The binarized images of the Y component, Cb component, and Cr component in this case are shown in FIGS. 17 (a) to 17 (i). Here, in the binarized images of FIGS. 17A to 17I, regions represented by gray and white are extracted as masks. The subject candidate masks are narrowed down from the masks extracted from the respective binary images. Note that the two masks shown as the white area in FIG. 17F, the mask shown as the white area in FIG. 17G, and the mask shown as the white area in FIG. It is.

  The control unit 19 calculates evaluation values Ev for these subject candidate masks, and ranks the masks using the calculated evaluation values Ev. In this case, since the subject candidate masks are the above-described four masks, these four masks are the final candidate masks.

  The control unit 19 performs the first extraction determination on the four masks that are final candidates. Here, the two masks shown as the white area in FIG. 17F are masks extracted based on the negative difference image of the Cb component, and the mask and the figure shown as the white area in FIG. The mask shown as a white area 17 (h) is a mask extracted based on a positive difference image of the Cr component. In this example, the two masks shown as white regions in FIG. 17F are equal to or less than the threshold Th5, so these masks are excluded from the final candidate masks. Accordingly, the mask shown as the white region in FIG. 17G and the mask shown as the white region in FIG. 17F are held as the final candidate mask. Thereafter, the second extraction determination and the third extraction determination are performed. Since there is no corresponding mask in these determinations, the mask narrowed down by the first extraction determination is held as it is as the final candidate mask.

  Finally, the control unit 19 performs the fourth extraction determination on the final candidate mask. The above-described mask shown as the white region in FIG. 17G and the mask shown as the white region in FIG. 17H are respectively extracted by binarization processing using different thresholds for the difference image of the Cr component. Is a mask. Therefore, these two masks are in an inclusive relationship. Therefore, the control unit 19 performs processing for integrating the masks. As a result, the mask shown as the white area in FIG. 17G is held as the final candidate mask. In this case, the control unit 19 extracts the mask shown as the white region in FIG. 17G as a mask corresponding to the subject region. Here, the mask shown as the white area in FIG. 17G corresponds to the area of the vehicle. Therefore, when the through image P5 is displayed on the display unit 23, the frame 58 is superimposed on the vehicle area in the through image P5 (see FIG. 16A).

  Here, in the conventional method, among the masks of the subject candidates, if the ranking is based on the evaluation value Ev and the top three masks, the masks are determined to be masks corresponding to the subject. That is, the two masks shown as white areas in FIG. 17F are determined to be masks corresponding to the subject area. Here, the two masks shown as white areas in FIG. 17F are each a tree area. Therefore, when the through image P5 is displayed on the display unit 23, the subject frame 58 is superimposed and displayed on the vehicle area, and the subject frames 59 and 60 are also superimposed and displayed on the tree area. (See FIG. 16 (b)).

  However, in the first embodiment, if there is a mask having a low average intensity in the first extraction determination, it is determined that the mask is a background portion and is excluded from the final candidate mask. Therefore, even when an area without a subject is extracted as a mask, the mask can be reliably excluded, and only a mask corresponding to the subject can be extracted.

  FIG. 18 is a through image P6 in which a cat is taken as a subject within the imaging range. Here, in the binarized images shown in FIGS. 19A to 19I, regions represented by gray and white are extracted as masks. In this example, subject candidate masks are narrowed down from masks extracted from the respective binarized images. 19A, the mask shown in the white area, the mask shown in the white area in FIG. 19B, the mask shown in the white area in FIG. 19C, and the white area in FIG. 19D. The mask becomes a subject candidate mask.

  In this example, when the masks of the subject candidates are ranked, the mask shown in the white area in FIG. 19A, the mask shown in the white area in FIG. 19B, and the white area in FIG. And the mask shown in the white area in FIG. 19D. In this case, the masks that are subject candidates are the above-described four masks. Therefore, these four masks are selected as final candidate masks. The control unit 19 performs a first extraction determination on these masks. Here, the mask shown in the white area in FIG. 19D is a mask extracted based on the difference image of the Cb component. Therefore, if the average intensity of the mask shown in the white region in FIG. 19D is equal to or less than the threshold Th3 by the first extraction determination, the control unit 19 uses the mask shown in the white region in FIG. 19D. , Exclude from the final candidate mask. Next, the control unit 19 performs a second extraction determination. In this case, since there is no mask extracted based on the difference image of the Y component below the mask extracted based on the difference image of the Cb component and the Cr component in the final candidate mask, No mask is excluded from the mask.

  Here, the mask shown in the white region in FIG. 19A and the mask shown in the white region in FIG. 19B are masks extracted based on the positive difference image of the Y component, and FIG. The mask shown in the white area of c) is a mask extracted based on the negative difference image of the Y component. Therefore, when the control unit 19 performs the third extraction determination, any of these masks is excluded from the final candidate mask. Here, the mask shown in the white area in FIG. 19A and the mask shown in the white area in FIG. 19B are ranked in the order of the evaluation value Ev over the mask shown in the white area in FIG. Is expensive. Therefore, the control unit 19 excludes the mask shown in the white area in FIG. 19C from the final candidate mask.

  For example, when a mask with high luminance is higher than a mask with low luminance, the mask with low luminance often corresponds to a black portion of the subject and becomes an unnecessary mask. Therefore, when both the mask extracted from the positive binary image of the Y component and the mask extracted from the negative binary image of the Y component are the top five masks, The control unit 19 holds a mask that is positioned higher among the mask extracted from the positive binary image of the Y component and the mask extracted from the negative binary image of the Y component, Excludes masks located below.

  If the mask extracted from the negative binary image of the Y component is positioned higher than the mask extracted from the positive binary image of the Y component, the mask is displayed in the through image. It can be assumed that the subject corresponding to the color of the subject itself is dark, and the area corresponding to the mask extracted from the positive binary image of the Y component is often the background. Therefore, when both the mask extracted from the positive binary image of the Y component and the mask extracted from the negative binary image of the Y component are the top five masks, The control unit 19 holds the mask positioned at the upper level and excludes the mask positioned at the lower level.

  Finally, the control unit 19 performs a fourth extraction determination. Here, the mask shown in the white area in FIG. 19A and the mask shown in the white area in FIG. 19B are in an inclusive relationship. Therefore, the control unit 19 integrates these masks. As a result, the mask shown in the white area in FIG. 19A is extracted as a mask corresponding to the object area.

  The mask shown in the white area | region of Fig.19 (a) is the area | region of a torso of a cat and a leg | foot. Accordingly, when the through image P6 is displayed on the display unit 23, the subject frame 61 is superimposed and displayed on the cat's torso and foot regions. As described above, when both the high luminance region and the low luminance region are extracted as masks, the region corresponding to the subject is appropriately identified by selecting only the high evaluation mask. It becomes possible.

  Hereinafter, the operational effects of the above embodiment will be described. The control unit 19 of the above embodiment calculates the mask evaluation value Ev using the value of the average intensity Av of the mask (S207). Then, the control unit 19 extracts a mask corresponding to the subject by ranking the subject candidate masks based on the mask evaluation value Ev (S208).

  FIG. 20 shows an example in which the positions of the masks extracted from a plurality of binarized images are mapped. FIG. 21 shows the average intensity of each mask shown in FIG.

  FIG. 20 shows a state in which the masks 1 to 4 are acquired as the top four masks in step S501. In FIG. 20, the mask 4 is located farther from the center of the screen than the masks 1 to 3. In FIG. 20, the masks 1 and 4 have an elongated shape, and the moment of inertia is larger than that of the masks 2 and 3. On the other hand, as shown in FIG. 21, the average intensity of each mask decreases in the order of mask 4, mask 3, mask 2, and mask 1. In FIG. 21, the difference in average intensity between the masks 1 and 2 is smaller than the difference in average intensity between the masks 3 and 2, and the average intensity of the masks 3 and 4 is sufficiently higher than the average intensity of the masks 1 and 2. large.

  FIG. 22A shows the top two masks finally extracted in the above embodiment. FIG. 22B shows, as a comparative example, the upper two masks that are finally extracted based only on the shape and position of the mask.

  In the case of the comparative example shown in FIG. 22B, the average intensity of the mask is not taken into consideration, so the masks 2 and 3 that are close to the center of the screen and have a uniform shape (small moment of inertia) are extracted. However, if a mask is extracted by paying attention only to the shape, if there is a region different from the surrounding in the vicinity of the center of the screen, that region is easily extracted. Therefore, there is a possibility that a target mask that is not conspicuous and does not match the subject (for example, a mask corresponding to the background partially exposed only in the center of the screen) may be erroneously extracted.

  On the other hand, in the case of the above-described embodiment shown in FIG. 22A, the evaluation value is calculated in consideration of the average intensity of the mask, so that the masks 3 and 4 having the high average intensity of the mask are extracted. As described above, in the above embodiment, the mask intensity (saliency of the mask itself) in each difference image is reflected at the time of mask extraction, so that the mask extraction is performed under conditions closer to human senses. Is possible.

<Modification 1>
In the above embodiment, the evaluation value calculation unit 47 may weight the evaluation value of the mask according to the magnitude of the average intensity of the mask. The weighting process according to the first modification may be performed, for example, when the evaluation value is calculated in S207. When the mask corresponding to the subject is extracted in step S208, the weighted upper mask is extracted. You can do it.

For example, the evaluation value calculator 47, the mask average intensity Av mask is a threshold gamma ₁ or weighted so large value of Av of formula (1), and the average intensity Av threshold gamma less than ₁ mask Is weighted so that the Av value in the equation (1) becomes small. In this case, the average intensity of the mask becomes relatively higher than the threshold value gamma ₁ or more to become a mask evaluation value is less than the threshold value gamma ₁ of the mask, a high mask average strength tends to be preferentially extracted.

The evaluation value calculation unit 47, when the difference in average intensity between the mask exceeds the threshold value gamma ₂ may provide a difference in the weighted average intensity Av mask. For example, the evaluation value calculator 47, a high average intensity than the mask as a reference, the value of Av of formula (1) is increased to mask the difference between the average intensity of the reference and made a mask exceeds a threshold gamma ₂ Weight as follows. Alternatively, the evaluation value calculator 47, the average intensity is lower than the mask as a reference, the mask the difference between the average intensity of the reference and made a mask exceeds a threshold value gamma ₂ is the value of Av of formula (1) becomes smaller Weight as follows. In these cases, the evaluation value of the mask having an extremely high average intensity compared to the reference mask becomes relatively high, and a mask having a high average intensity is easily extracted.

Here, for example, the evaluation value calculation unit 47 may perform the above weighting by multiplying the Av value by a weighting coefficient. The threshold values γ ₁ and γ ₂ may be different values for each component of YCbCr. Further, the threshold values γ ₁ and γ ₂ may be different values for a mask extracted from a positive difference image and a mask extracted from a negative difference image even with the same component.

  For example, the evaluation value calculation unit 47 reduces the weighting of the Y component based on the average intensity of the mask as compared with the CbCr component or does not perform the weighting itself. In the positive difference image of the Y component, for example, the pixel value is large in a highlight area that is overexposed, and in the negative difference image of the Y component, for example, the pixel value is large in a shadow area that is blackened. The highlight area and the shadow area do not have characteristics as a subject. Therefore, by suppressing the weighting of the Y component by the average intensity of the mask, it is possible to make it difficult to extract the mask corresponding to the highlight area or the shadow area.

<Modification 2>
In the above embodiment, the evaluation value calculation unit 47 may calculate the evaluation value of the mask based on the average intensity of the mask without using the area of the mask and the moment of inertia of the mask.

  In the above embodiment, the evaluation value calculation unit 47 separately calculates the first evaluation value based on the average intensity of the mask and the second evaluation value based on the mask area and the moment of inertia of the mask. Also good. Then, after extracting the masks up to a predetermined rank with the first evaluation value, the mask extraction unit 48 extracts the mask of the final subject from the masks up to the predetermined rank using the second evaluation value. Also good.

<Supplementary items of the first embodiment>
In the first embodiment, an average value of pixel values of each pixel of a through image subjected to resolution conversion processing is calculated, and a reference density image is generated using the calculated average value of pixel values. There is no need to be limited, and it is also possible to generate a reference density image using the median value of each pixel value of the through image.

  In the first embodiment, the binary image of each component is generated from the difference image of the Y component, Cb component, and Cr component. However, the binary of each component is generated from the image of the Y component, Cb component, and Cr component. It is also possible to generate a digitized image. In this case, three or more binarized images may be generated using three or more different thresholds for each component image. In this case, a histogram is generated for each image of the Y component, the Cb component, and the Cr component, and the standard deviation σ and the peak / peak value pp in the generated histogram are obtained, and the standard deviation σ and the peak / peak value pp are obtained. It is sufficient to determine whether or not to use an image for narrowing down subject candidates.

  In the first embodiment, whether or not to generate a histogram based on the generated difference image and use the standard deviation σ and peak / peak value pp in the generated histogram as a difference image used when narrowing down subject candidates is determined. However, the present invention is not limited to this, and it is also possible to perform the above determination after generating a binarized image from the difference image.

  In the first embodiment, it is determined whether or not the number of masks extracted from the binarized image is equal to or greater than the threshold Th3 and the average intensity of the entire mask is equal to or less than the threshold Th4, and when these conditions are satisfied, All masks extracted from the corresponding binarized image are excluded from the subject candidate masks. However, in this determination, all of the masks extracted from all the binarized images may be excluded from the subject candidate masks. Therefore, in such a case, the binarization process may be performed by increasing the number of threshold values.

  In the first embodiment, the number of masks is counted for each of all binarized images, and when the number of masks extracted from the binarized image is equal to or greater than the threshold Th3, the average intensity of the entire mask is the threshold Th4. It is determined whether or not: However, it is not necessary to use all the binarized images. For example, the above determination can be performed only on the binarized image based on the difference image having a small standard deviation σ in the histogram generated from the difference image. . That is, when the standard deviation σ in the difference image described above is small, it can be determined that the pixel value in the difference image is a pixel value close to the average, that is, there is no conspicuous object in the difference image. Therefore, it is effective to perform the above determination on a binarized image obtained from a difference image having a small standard deviation σ.

  In addition, this determination is performed on the binarized image, but the determination is not limited to this, and the binarized image may be divided into a plurality of regions, and may be performed on each of the divided regions. Is possible.

  In the first embodiment, the standard deviation σ is obtained when a positive difference image and a negative difference image of the Y component, the Cb component, and the Cr component are generated. Therefore, it is possible to select a difference image used when generating a binarized image from the value of the standard deviation σ obtained when the difference image is generated.

  In the first embodiment, the threshold value for the number of extracted masks is the threshold value Th3, the threshold value for the average intensity of the entire mask is the threshold value Th4, and these threshold values are fixed. These threshold values are the Y component, the Cb component, and the threshold value. It is good also as a different value for every Cr component. In this case, for example, the threshold Th3 and the threshold Th4 for each of the Y component, the Cb component, and the Cr component can be individually set based on the shooting scene and the image configuration in the through image.

  In the first embodiment, of the final candidate masks, masks in an inclusive relationship are integrated. In this integration, when two masks are in an inclusive relationship, the other mask included in one mask is excluded from the final candidate mask, and one mask is excluded from the final candidate mask. However, both of the two masks in the inclusive relationship are held as final candidate masks, and when the subject frame is displayed, the subject frame is displayed only for the subject area corresponding to one mask. It is also possible.

  In the first embodiment, the masks in the inclusion relationship among the final candidate masks are integrated. However, the present invention is not limited to this. For example, when the mask is extracted from the binarized image, the mask is in the inclusion relationship. After performing the process of integrating the masks, it is also possible to perform a process of narrowing down the subject candidate masks.

  In the first embodiment, the process for narrowing down the difference image (step S204 in the flowchart shown in FIG. 5 and the process in the flowchart shown in FIG. 6) and the process for narrowing down the mask (step S206 in the flowchart shown in FIG. 5 and the flowchart shown in FIG. 10). Process). However, at least one of the process of narrowing down the difference image and the process of narrowing down the mask may be performed, or both processes can be omitted.

  In the first embodiment, first to fourth extraction determinations are performed as processing for extracting a mask corresponding to a subject (step S208 in the flowchart shown in FIG. 5 and processing in the flowchart shown in FIG. 13). However, it is also possible to perform only at least one determination process among the first to fourth extraction determinations. Further, without performing these extraction determinations, a mask having the highest evaluation value Ev among the final candidate masks may be extracted as a mask corresponding to the subject.

  In the first embodiment, the mask narrowing unit 46 may narrow down subject masks using the following criteria.

  For example, the mask narrowing unit 46 sets a rectangular footprint area whose area is 60% of the entire image and whose center of gravity coincides with the center of the image, and a mask that does not belong to 50% or more of the footprint area is set as a subject. You may exclude from a candidate.

Further, for example, the mask narrowing unit 46 obtains a mask motion vector between continuously acquired through images, and among masks excluded from subject candidates under other conditions, a mask having a motion vector is a subject candidate. And masks without motion vectors may be excluded from subject candidates as they are.
Second Embodiment
FIG. 23 is a diagram illustrating a configuration example of the image processing apparatus according to the second embodiment. The image processing device 90 of the second embodiment functions as a subject detection device of the present invention. A specific example of the image processing apparatus 90 is a computer.

  23 includes a data reading unit 91, a storage device 92, a CPU 93, a memory 94, an output I / F 95, and a bus 96. The data reading unit 91, the storage device 92, the CPU 93, the memory 94, and the input / output I / F 95 are connected to each other via a bus 96. The image processing apparatus 90 is connected to an input device 97 such as a keyboard and a mouse and a monitor 98 via an input / output I / F 95. The input / output I / F 95 receives various inputs from the input device 97 and outputs display data to the monitor 98.

  The data reading unit 91 is used when reading image data or a program from the outside. The data reading unit 91 communicates with a reading device (such as a reading device such as an optical disk, a magnetic disk, or a magneto-optical disk) that acquires data from, for example, a removable storage medium 99 or an external device in accordance with a known communication standard. Communication device (USB interface, wired or wireless LAN module, etc.) FIG. 23 shows a case where the data reading unit 91 is a reading device that acquires data from a removable storage medium 99.

  The storage device 92 is composed of a storage medium such as a hard disk or a nonvolatile semiconductor memory. The storage device 92 stores the program and various data necessary for executing the program. The storage device 92 can store image data read by the data reading unit 91 and the like.

  The CPU 93 is a processor that comprehensively controls each unit of the image processing apparatus 90. The CPU 93 has the function of the image processing unit 31 when the program is executed. As one of the functions of the image processing unit 31, a subject detection unit 32 is provided. Here, since the image processing unit 31 and the subject detection unit 32 have the same functions as those in the first embodiment, the same reference numerals as those in the first embodiment are given. That is, the subject detection unit 32 has the same configuration as that of the first embodiment (color space conversion unit 41, resolution conversion unit 42, difference image generation unit 43, image determination unit 44, binarization processing unit 45, mask narrowing unit 46). , An evaluation value calculation unit 47 and a mask extraction unit 48). In the second embodiment, the subject detection unit 32 is provided as one function of the image processing unit 31, and the CPU 93 executes a program different from the program for executing the function of the image processing unit 31. It is good also as a form which has the function of the to-be-photographed object detection part 32. FIG.

  The memory 94 temporarily stores various calculation results when the CPU 93 executes the program. The memory 94 is, for example, a volatile SDRAM.

  In the image processing apparatus 90 according to the second embodiment, when data of an image serving as an input image is acquired from the data reading unit 91 or the storage device 92, the CPU 93 performs subject detection processing illustrated in FIGS. 4, 5, 10, and 13. Execute. In the image processing apparatus 90 according to the second embodiment, the same effect as the subject detection process according to the first embodiment can be obtained.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 19 ... Control part, 31 ... Image processing part, 32 ... Subject detection part, 41 ... Color space conversion part, 42 ... Resolution conversion part, 43 ... Difference image generation part, 44 ... Image determination part, 45 ... Binarization processing unit, 46... Mask narrowing unit, 47 .. evaluation value calculation unit, 48... Mask extraction unit, 90.

Claims (17)

A binarization processing unit that generates a plurality of different binarized images using an image to be processed;
For each pixel region extracted from each binarized image, an evaluation value indicating the subjectness of the pixel region is calculated using the pixel intensity corresponding to the pixel region acquired from the image before binarization processing. An evaluation value calculation unit to perform,
An object detection apparatus comprising: an extraction unit that extracts the pixel area corresponding to a specific area of the processing target image using the evaluation value. The subject detection apparatus according to claim 1,
A first generation unit that generates a reference density image using an average value of pixel values in the processing target image;
Using a difference between the image to be processed and the reference density image, a first difference image indicating a degree of deviation of the pixel value from the reference density image at a location where the pixel value exceeds the reference density image; A second generation unit that generates a second difference image indicating a degree of deviation of the pixel value from the reference density image at a location where the pixel value is lower than the reference density image;
The binarization processing unit binarizes the first difference image and the second difference image to generate a plurality of binarized images. The subject detection apparatus according to claim 2,
The subject detection apparatus, wherein the evaluation value calculation unit calculates the evaluation value by using the degree of divergence acquired from the first difference image or the second difference image. In the subject detection apparatus according to claim 2 or 3,
The first generation unit generates the reference density image corresponding to each component of the color space,
The second generation unit generates the first difference image and the second difference image with each component of a color space,
The subject detection apparatus, wherein the binarization processing unit generates a plurality of different binarized images for each component of a color space. The subject detection apparatus according to any one of claims 1 to 4,
The evaluation value calculation unit calculates the evaluation value by further using the area of the pixel region and the moment of inertia of the pixel region. The subject detection apparatus according to any one of claims 1 to 5,
The evaluation value calculation unit weights the evaluation value so that the evaluation value becomes high when the intensity of the pixel corresponding to the pixel area is equal to or greater than a threshold value. The subject detection apparatus according to any one of claims 1 to 5,
The evaluation value calculation unit is configured so that when the difference in pixel intensity between the two pixel areas is equal to or greater than a threshold, the subjectivity is higher with respect to the pixel area having the higher pixel intensity. A subject detection apparatus that weights evaluation values. In the photographic subject detection device according to any one of claims 1 to 7,
The subject detection apparatus further comprising a display unit indicating a corresponding position of the pixel region extracted by the extraction unit. An imaging unit that captures an image of a subject;
The subject detection device according to any one of claims 1 to 8,
An imaging apparatus comprising: Generate multiple different binarized images using the image to be processed,
For each pixel region extracted from each binarized image, an evaluation value indicating the subjectness of the pixel region is calculated using the pixel intensity corresponding to the pixel region acquired from the image before binarization processing. And
A program that causes a computer to execute a process of extracting the pixel area corresponding to a specific area of the image to be processed using the evaluation value. The program according to claim 10, wherein
A reference density image is generated using an average value of pixel values in the processing target image,
Using a difference between the image to be processed and the reference density image, a first difference image indicating a degree of deviation of the pixel value from the reference density image at a location where the pixel value exceeds the reference density image; Generating a second difference image indicating a degree of deviation of the pixel value from the reference density image at a location where the pixel value is lower than the reference density image;
A program for causing a computer to execute a process of binarizing the first difference image and the second difference image to generate a plurality of binarized images. The program according to claim 11,
A program that causes a computer to execute a process of calculating the evaluation value using the degree of divergence acquired from the first difference image or the second difference image. In the program according to claim 11 or 12,
Generating the reference density image corresponding to each component of the color space,
Generating the first difference image and the second difference image with each component of the color space;
A program that causes a computer to execute a process of generating a plurality of different binarized images for each component of a color space. In the program according to any one of claims 10 to 13,
A program for causing a computer to execute processing for calculating the evaluation value by further using the area of the pixel region and the moment of inertia of the pixel region. The program according to any one of claims 10 to 14,
A program for causing a computer to execute a process of weighting the evaluation value so that the subjectness is high when the intensity of the pixel corresponding to the pixel area is equal to or greater than a threshold value. The program according to any one of claims 10 to 14,
A process of weighting the evaluation value so that the subjectivity is higher with respect to the pixel area having the higher pixel intensity when the difference in intensity between the two pixel areas is equal to or greater than a threshold value. A program that causes a computer to execute. The program according to any one of claims 10 to 16,
A program for causing a computer to execute processing for causing a display device to display a corresponding position of the extracted pixel area.

Images