JP6011146B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, various techniques relating to subject extraction have been considered. For example, in the invention of Patent Document 1, an imaging device that automatically adjusts an automatic focus on a subject even when the subject moves in the camera direction by deforming an AF distance measurement frame according to the amount of change in the subject shape. It is disclosed.

JP 2009-069748 A

  By the way, in subject extraction, erroneous detection may occur in subject extraction due to the environment at the time of capturing an image to be processed. For example, a shadow portion may be extracted as a subject area with respect to an image to be processed taken in a situation such as outdoors in fine weather. Originally, the shadow is attached to the object, and it is not preferable that the shadow itself is extracted as the subject area.

  The present invention has been made in view of the above points, and an object of the present invention is to accurately extract a subject area even when an image to be processed is affected by a shadow.

An image processing apparatus according to the present invention includes an acquisition unit that acquires image data to be processed, an area detection unit that calculates a feature amount of an image indicated by the image data, and detects a subject region determined based on the feature amount; , An evaluation value calculation unit that calculates an evaluation value indicating the influence of a shadow in the image, and a control that determines a detection method of the subject region by the region detection unit based on the evaluation value calculated by the evaluation value calculation unit comprising a part, the said area detection unit, said as a feature amount, calculates a characteristic quantity relating to at least the luminance based on the feature quantity relating to the luminance, and detects an object area based on at least the low luminance area, the control The unit determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold, and when the influence of the shadow is large, the low luminance region Changing the method of detecting a subject region based.

Further, the area detection unit detects a subject area based on a spatial continuous area of pixels in which the result of comparing the feature value related to the brightness with a predetermined threshold is true, and the control unit detects the influence of the shadow. If large, the threshold is set smaller than the influence of the shadow is not large, it may be performed discovery of the subject region based on the spatial continuity area by the area detection unit.

  Further, the area detection unit calculates a plurality of feature amounts as the feature amount, detects a plurality of subject regions based on the plurality of feature amounts, and the control unit determines the plurality of feature amounts based on the evaluation value. The priority for the subject area may be determined, and the main subject area may be determined based on the determined priority.

  In addition, the area detection unit calculates at least a luminance-related feature amount as the feature amount, and detects a subject region based on at least a low-luminance region based on the luminance-related feature amount. By comparing the evaluation value and a predetermined threshold, it is determined whether the influence of the shadow is large, and when the influence of the shadow is large, the priority regarding the subject area based on the low-luminance area is The main subject area may be determined by setting a lower value than when the influence of the shadow is not large.

  In addition, the area detection unit calculates at least a luminance-related feature amount as the feature amount, and detects a subject region based on at least a low-luminance region based on the luminance-related feature amount. By comparing the evaluation value with a predetermined threshold value, it is determined whether or not the influence of the shadow is large. When the influence of the shadow is large, the subject area based on the low-luminance area is different from other subject areas. It is further determined whether or not the subject area based on the low brightness area is close to other subject areas, and the priority regarding the subject area based on the low brightness area is set as the shadow. The main subject area may be determined by setting it lower than the case where the influence of is not large.

  Further, the area detection unit calculates at least a luminance-related feature quantity as the feature quantity, and detects a plurality of subject areas including at least a subject area based on the low-luminance area based on the feature quantity related to the brightness, The control unit determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold, and when the influence of the shadow is large, the subject area based on the low luminance area Is further close to other subject areas, and when the subject area based on the low brightness area is close to other subject areas, the plurality of areas detected by the area detection unit Of the subject areas, the main subject area may be determined from the subject areas excluding the subject area based on the low luminance area.

Further, the evaluation value calculation unit calculates the evaluation value based on at least one of information related to brightness of the image and information related to contrast, and the control unit is configured such that the brightness is brighter than a predetermined reference. In other cases, when the brightness is brighter than a predetermined reference and the contrast is higher than a predetermined reference, it may be determined that the influence of the shadow is large.

An image pickup apparatus according to the present invention detects an object region determined based on an image pickup unit that picks up an image by an optical system and generates image data, calculates a feature amount of an image indicated by the image data, and An area detection unit; an evaluation value calculation unit that calculates an evaluation value indicating an influence of a shadow in the image; and a method of detecting the subject area in the region detection unit based on the evaluation value calculated by the evaluation value calculation unit And a control unit that determines at least a luminance-related feature amount as the feature amount, and detects a subject region based on at least a low-luminance region based on the feature amount related to the luminance. Then, the control unit determines whether the influence of the shadow is large by comparing the evaluation value and a predetermined threshold, and when the influence of the shadow is large, Serial to change the method of detecting a subject region based on the low luminance region.

An image processing program according to the present invention is an area for detecting an object region determined based on an acquisition procedure for acquiring image data to be processed, a feature amount of an image indicated by the image data, and a computer. Based on a detection procedure, an evaluation value calculation procedure for calculating an evaluation value indicating the influence of a shadow in the image, and the evaluation value calculated in the evaluation value calculation procedure, a method for detecting the subject region in the region detection procedure is provided. A control procedure to determine, and the region detection procedure calculates, as the feature amount, a feature amount related to at least luminance, and based on the feature amount related to luminance, an object region based on at least a low luminance region is calculated. A detection procedure, wherein the control procedure compares the evaluation value with a predetermined threshold value to determine the influence of the shadow. It determines whether large, when the influence of the shadow is large, includes a procedure for changing the detection method of the subject region based on the low-luminance region.

  According to the present invention, it is possible to accurately extract a subject area even when an image to be processed is affected by a shadow.

2 is a block diagram illustrating configurations of a lens barrel 10, an imaging device 20, and a storage medium 40. FIG. It is a flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is another flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is a figure explaining mask extraction processing (I). It is another figure explaining a mask extraction process (I). It is another figure explaining a mask extraction process (I). It is another figure explaining a mask extraction process (I). It is a figure explaining mask extraction processing (II). It is another figure explaining mask extraction processing (II).

  Hereinafter, embodiments will be described in detail with reference to the drawings.

  In the present embodiment, a description will be given by taking as an example an apparatus including a lens barrel 10, an imaging device 20, and a recording medium 40 as shown in FIG.

  The imaging device 20 captures an optical image incident from the lens barrel 10. The obtained image is stored in the storage medium 40 as a still image or a moving image.

  The lens barrel 10 includes a focus adjustment lens (hereinafter referred to as an “AF (Auto Focus) lens”) 11, a lens driving unit 12, an AF encoder 13, and a lens barrel control unit 14. The lens barrel 10 may be detachably connected to the imaging device 20 or may be integrated with the imaging device 20.

  The imaging device 20 includes an imaging unit 21, an image processing device 22, a display unit 23, a buffer memory unit 24, a storage unit 25, a CPU 26, an operation unit 27, and a communication unit 28. The imaging unit 21 includes an imaging element 29 and an A / D (Analog / Digital) conversion unit 30. The imaging unit 21 is controlled by the CPU 26 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.).

  In the lens barrel 10, the AF lens 11 is driven by the lens driving unit 12 and guides an optical image to the light receiving surface (photoelectric conversion surface) of the imaging element 29 of the imaging device 20. The AF encoder 13 detects the movement of the AF lens 11 and outputs a signal corresponding to the movement amount of the AF lens 11 to the lens barrel control unit 14. Here, the signal corresponding to the movement amount of the AF lens 11 may be, for example, a sine wave signal whose phase changes according to the movement amount of the AF lens 11.

  The lens barrel control unit 14 controls the lens driving unit 12 in accordance with a drive control signal input from the CPU 26 of the imaging device 20. Here, the drive control signal is a control signal for driving the AF lens 11 in the optical axis direction. The lens barrel control unit 14 changes, for example, the number of steps of the pulse voltage output to the lens driving unit 12 according to the drive control signal. Further, the lens barrel control unit 14 outputs the position (focus position) of the AF lens 11 in the lens barrel 10 to the CPU 26 of the imaging device 20 based on a signal corresponding to the movement amount of the AF lens 11. Here, the lens barrel control unit 14 integrates, for example, signals according to the movement amount of the AF lens 11 according to the movement direction of the AF lens 11, thereby moving the AF lens 11 in the lens barrel 10 ( Position) may be calculated. The lens driving unit 12 drives the AF lens 11 according to the control of the lens barrel control unit 14 and moves the AF lens 11 in the optical axis direction within the lens barrel 10.

  In the imaging device 20, the imaging element 29 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 10 (optical system) into an electric signal, and supplies the electric signal to the A / D conversion unit 30. Output. The imaging element 29 is configured by a photoelectric conversion element such as a CMOS (Complementary Metal Oxide Semiconductor), for example. Further, the image sensor 29 may convert an optical image into an electric signal for a partial region of the photoelectric conversion surface (image cutout). Further, the image sensor 29 outputs an image obtained when a photographing instruction from the user is received via the operation unit 27 to the storage medium 40 via the A / D conversion unit 30 and the communication unit 28. On the other hand, the image pickup device 29 converts the continuously obtained image into a through image and converts it into a buffer memory unit 24 and the display unit 23 in the state before accepting a shooting instruction from the user via the operation unit 27. The data is output via the unit 30.

  The A / D conversion unit 30 digitizes the electrical signal converted by the image sensor 29 and outputs an image, which is a digital signal, to the buffer memory unit 24 and the like.

  The image processing device 22 performs image processing on the image temporarily stored in the buffer memory unit 24 based on the image processing conditions stored in the storage unit 25. The image after image processing is stored in the storage medium 40 via the communication unit 28. Further, the image processing device 22 performs mask extraction processing on the image temporarily stored in the buffer memory unit 24 (details will be described later). The extracted mask information is output to the CPU 26 and stored in the storage unit 25, the storage medium 40, and the like.

  The display unit 23 is, for example, a liquid crystal display, and displays an image generated by the imaging unit 21, an operation screen, and the like. The buffer memory unit 24 temporarily stores the image generated by the imaging unit 21. The storage unit 25 stores imaging conditions, determination conditions referred to by the CPU 26 in various determinations, and the like.

  The CPU 26 appropriately acquires necessary information from the image processing unit 22, the storage unit 25, and the like, and comprehensively controls each unit in the imaging device 20 based on the acquired information. Control by the CPU 26 includes focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, flash emission amount change setting, subject tracking setting, and various shooting mode settings. , Various image processing settings, various display settings, brightness optimization settings linked to zoom magnification, and the like. In addition, the CPU 26 monitors the operation state of the operation unit 27 and outputs image data to the display unit 23.

  The operation unit 27 includes, for example, a power switch, a shutter button, a multi-selector (cross key), or other operation keys. The operation unit 27 receives a user operation input when operated by the user, and outputs a signal corresponding to the operation input to the CPU 26. Output to.

  The communication unit 28 is connected to a removable storage medium 40 such as a card memory, and writes, reads, or deletes information (image data, area information, etc.) to the storage medium 40.

  The storage medium 40 is a storage unit that is detachably connected to the imaging device 20 and stores information (image data, area information, and the like). Note that the storage medium 40 may be integrated with the imaging device 20.

  The imaging device 20 includes an automatic detection mode for automatically detecting the main subject area in addition to the normal mode for detecting the main subject area based on the focus adjustment information at the time of shooting. The automatic detection mode is a mode for automatically and continuously detecting a main subject area based on a through image for composition confirmation. This automatic detection mode may be set by a user operation via the operation unit 27, or may be automatically set by the CPU.

  Hereinafter, the operation of the CPU 26 when the automatic detection mode is executed will be described with reference to the flowchart of FIG.

  In step S101, the CPU 26 starts photometry of the object scene. Metering of the object scene is performed in the same manner as in the known art. For example, the CPU 26 may perform photometry based on a through image generated by the imaging unit 21 or may perform photometry based on an output of a photometric sensor (not shown). The CPU 26 performs photometry at a predetermined time interval and records the acquired BV value, which is a luminance value, in the storage unit 25 as needed.

  In step S102, the CPU 26 starts measuring the contrast. The contrast is measured in the same manner as in the known technique. For example, the CPU 26 may perform contrast measurement based on a through image generated by the imaging unit 21 or may perform contrast measurement based on an output from another sensor or the like. The CPU 26 measures the contrast at a predetermined time interval and records the acquired contrast value in the storage unit 25 as needed.

  In step S103, the CPU 26 controls the imaging unit 21 to start acquiring a through image. The acquired image information of the through image is temporarily stored in the buffer memory unit 24. This through image is continuously generated at a predetermined time interval. The acquisition of the through image by the CPU 26 is sequentially performed sequentially in time.

  In step S104, the CPU 26 controls the image processing device 22 to perform normal image processing. Normal image processing includes white balance adjustment, interpolation processing, color tone correction processing, gradation conversion processing, and the like. Since the specific method of each process is the same as that of a well-known technique, description is abbreviate | omitted. The image processing device 22 acquires the image data of the target image from the buffer memory unit 24, performs image processing, and then outputs the image data to the buffer memory unit 24 again.

  In step S105, the CPU 26 determines whether there is an influence of a shadow. If the CPU 26 determines that there is an influence of a shadow, it proceeds to step S107 described later. On the other hand, when determining that there is no influence of the shadow, the CPU 26 proceeds to step S106.

  The determination as to whether there is an influence of a shadow is made based on at least one of the result of photometry started in step S101 and the result of contrast measurement started in step S102.

  The case where there is an influence of a shadow is a case where shooting is performed in a situation such as outdoors in a fine weather, or a case where a subject is irradiated with special illumination in an indoor studio or the like. In such a case, there is a possibility that a shadow portion is extracted as a subject area in the image to be processed. The CPU 26 analyzes the brightness of the object scene, the scene of the object scene, and the like based on at least one of the photometry result started in step S101 and the contrast measurement result started in step S102. It is determined whether or not there is an influence of a shadow.

  Specifically, for example, it may be determined that there is an influence of a shadow when the BV value is larger than a predetermined threshold (that is, bright), the BV value is larger than the predetermined threshold, and the contrast value May be determined to have an influence of a shadow when the value is larger than a predetermined threshold (that is, the contrast is high). Further, an evaluation value indicating the influence of the shadow is calculated based on at least one of the photometry result started in step S101 and the contrast measurement result started in step S102, and the evaluation value and a predetermined threshold value are calculated. By comparing, it may be determined whether there is an influence of a shadow.

  In step S106, the CPU 26 controls the image processing device 22 to perform mask extraction processing (I). The mask extraction process is a technique for calculating a feature amount in an image and detecting a subject area based on the feature amount. For example, the feature extraction in the image is obtained, and the mask extraction is performed by obtaining a continuous region having the same feature quantity. The mask extraction process (I) is a normal mask extraction process for a general image that is not affected by a shadow.

  The mask extraction process (I) will be described with reference to the schematic diagrams of FIGS.

  The image processing device 22 reads out the image that has been subjected to the normal image processing in step S104 from the buffer memory unit 24, appropriately performs resizing processing and color space conversion processing, and generates a YUV (for example, YCbCr) image. Next, the image processing device 22 creates a mask image based on each of the Y image, Cb image, and Cr image of the generated YUV image.

  As shown in FIG. 3, the image processing device 22 creates a Y mask M [Y], a Cb mask M [Cb], and a Cr mask M [Cr] from each of the Y image, the Cb image, and the Cr image. For example, the image processing device 22 generates a binarized image in a predetermined range (for example, Yave + K · σ) with respect to the average pixel value Yave at the four pixels at the center of the Y image, for example. Then, a labeling process is performed on the generated Y-binarized image, and unnecessary masks are eliminated to create a Y mask M [Y]. Note that K described above is a predetermined coefficient (for example, K = 0.6), and σ is a deviation from Yave. Since the binarization process and the labeling process are the same as those in the known technique, the description thereof is omitted. The image processing apparatus 22 performs the same processing to create a Cb mask M [Cb] from the Cb image, and creates a Cr mask M [Cr] from the Cr image.

  Further, as shown in FIG. 3, the image processing apparatus 22 creates a center mask M [a] from the created Y mask M [Y], Cb mask M [Cb], and Cr mask M [Cr]. The center mask M [a] is a mask formed of an overlapping region where the Y mask M [Y], the Cb mask M [Cb], and the Cr mask M [Cr] overlap. Note that when the center mask M [a] is created, the overlapping region may be obtained by limiting the region to the central region, for example. Or you may limit to the mask containing a center. In the subsequent processing, only the finally created center mask M [a] is used.

  Further, as shown in FIG. 4, the image processing device 22 creates three types of luminance masks from the Y image (however, four or more types of masks may be created from the Y image). The image processing device 22 uses each pixel value of the Y image as an input value and each pixel value after correction (for example, gamma correction) as an output value. Then, binarized images of three sections are generated according to the output value. As shown in FIG. 4, a Y highlight binarized image is generated from the highlight portion of the Y image, and a Y intermediate or higher binarized image is generated from the middle portion or more of the Y image. A Y shadow binary image is generated from the portion. Then, the image processing apparatus 22 performs a labeling process on each of the generated three types of binarized images, and eliminates unnecessary masks, whereby a Y highlight mask M [Y1], a Y intermediate or higher mask M Each brightness mask of [Y2] and Y shadow mask M [Y3] is created.

  Further, as shown in FIG. 5, the image processing device 22 creates three types of color masks from each of the Cb image and the Cr image. The image processing device 22 generates a binarized image of three predetermined sections according to the output value, similarly to the example of FIG. 4 described above, for the Cb image. As shown in FIG. 5, from the Cb image, the Cb blue side binarized image centered on the blue side, the Cb intermediate binarized image centered on the intermediate component, and the Cb yellow side binary centered on the yellow side. A digitized image is generated. Similarly, from the Cr image, a Cr red side binarized image centered on the red side, a Cr intermediate binarized image centered on the intermediate component, and a Cr green side binarized image centered on the green side are generated. Is done. Then, the image processing device 22 performs a labeling process on each of the generated six types of binarized images, and eliminates unnecessary masks, whereby the Cb blue mask M [Cb1] and the Cb intermediate mask M [ Six color masks are created: Cb2], Cb yellow side mask M [Cb3], Cr red side mask M [Cr1], Cr intermediate mask M [Cr2], and Cr green side mask M [Cr3].

  Further, as shown in FIG. 6, the image processing apparatus 22 creates three types of pure color masks from the Cb image and the Cr image. The pure color mask is a mask obtained by extracting an “absolutely pure subject” having an arbitrary fixed area. Since the creation of the pure color mask is performed in the same manner as in the known technique, description thereof is omitted. As shown in FIG. 6, the image processing apparatus 22 creates pure color masks of a pure color red mask M [R], a pure color yellow mask M [Ye], and a pure color purple mask M [P] from the Cb image and the Cr image. .

  Through the processing described above, the image processing apparatus 22 has one type of mask (center mask M [a]) shown in FIG. 3, three types of luminance masks shown in FIG. 4, and six types shown in FIG. A total of 13 types of masks are prepared, including the color masks shown in FIG.

  Next, the image processing apparatus 22 excludes unnecessary masks from the created 13 types of masks, and determines priorities for the remaining masks. The priority indicates a priority when determining a mask that is actually used to extract a subject area from 13 types of masks.

  In the mask extraction process (I) that is a normal mask extraction process for a general image, for example, as shown in FIG. 7, three levels of priority are determined in advance. The first priority is three kinds of pure color masks, the second priority is the Cb blue side mask M [Cb1] and the Cb yellow side mask M [Cb3] of the color masks, and the third priority is others. It is a mask. Note that the priority shown in FIG. 7 is an example, and any number of stages may be used as long as there are a plurality of stages. Further, the assignment of each mask to each priority in FIG. 7 is an example. As shown in FIG. 7, the priority is determined so that the priority of the mask related to the color information is relatively high. Then, the image processing apparatus 22 obtains a mask having a certain evaluation value or higher while giving priority to each mask having a high priority in accordance with these priorities, and uses the mask as a mask for extracting a subject area. Note that a method for selecting a mask for extracting a subject area may be any method such as a known technique. Finally, the image processing apparatus 22 extracts a subject area based on the selected mask, and proceeds to step S108 described later.

  In step S107, the CPU 26 controls the image processing device 22 to perform mask extraction processing (II). The mask extraction process (II) is a mask extraction process for an image affected by a shadow. The CPU 26 controls the image processing device 22 to perform at least one of the following (a) to (c) in the mask extraction process (II). Only differences from the mask extraction process (I) described in step S106 will be described below.

  (A) Change in the method of creating the Y shadow mask M [Y3].

  The image processing apparatus 22 creates the 13 types of masks described in step S106. However, the image processing apparatus 22 changes the conditions from the mask extraction process (I) when creating the Y shadow mask M [Y3] described in FIG. As described above, the mask extraction process (II) is a mask extraction process for an image affected by a shadow. The Y shadow mask M [Y3] is a mask for detecting a subject area based on a low-brightness area (a spatial continuous area of pixels for which a result compared with a predetermined threshold is true), and detects a shadow portion as a subject area. It is a mask that is highly likely to be Therefore, the method of creating the Y shadow mask M [Y3] is changed so as to prevent the Y shadow mask M [Y3] from detecting a shadow and to extract the black side as the color.

  In the mask extraction process (II), a threshold for generating the Y shadow binary image shown in FIG. 4 is set smaller than that in the mask extraction process (I). That is, as compared with the case of the mask extraction process (I), by shifting the threshold value to the shadow side, it becomes difficult to pick up a thin shadow part and only a black object can be picked up. Note that the threshold setting may be changed according to the degree of influence of the shadow. That is, the stepwise setting may be performed so that the above-described threshold value decreases as the influence of the shadow increases. Moreover, it is good also as a structure which changes the threshold value mentioned above to the predetermined | prescribed 2nd threshold value which is a smaller value instead of setting the threshold value mentioned above small.

  (B) Changing the priority of the Y shadow mask M [Y3].

  The image processing apparatus 22 creates the 13 types of masks described in step S106. However, unlike step S106, the image processing device 22 makes the priority of the Y shadow mask M [Y3] lower (lower) than in the mask extraction process (I). For example, the priority is 4th lower than the 3rd priority described with reference to FIG. The priority is determined in consideration of the fact that the target image is an image having an influence of a shadow, and a shadow portion is detected as a subject area by the Y shadow mask M [Y3]. This is to suppress the possibility of being lost. Then, as in step S106, the image processing apparatus 22 obtains a mask having a certain evaluation value or higher while giving priority to each mask having a high priority, and selects it as a mask for extracting a subject area.

  (C) Change based on the relative position of the Y shadow mask M [Y3] and other masks.

  The image processing apparatus 22 creates the 13 types of masks described in step S106. Thereafter, the image processing apparatus 22 confirms the relative positions of the Y shadow mask M [Y3] and other masks.

  The case where the target image is the image shown in FIG. 8 will be described as an example. In the example of FIG. 8, a cat T1 that is a main subject and a cat shadow T2 are included. Examples of 13 types of masks created for this image are shown in FIG. As shown in FIG. 9, the shadow T2 is extracted from the Y shadow mask M [Y3]. Therefore, first, the image processing apparatus 22 confirms the relative positions of the Y shadow mask M [Y3] and other masks. A dotted line in each mask in FIG. 9 indicates a region corresponding to the region extracted by the Y shadow mask M [Y3]. In the example of FIG. 9, three types of masks, that is, a Cb yellow side mask M [Cb3], a Cr red side mask M [Cr1], and a Cr intermediate mask M [Cr2] are close to the Y shadow mask M [Y3]. . Whether or not the Y shadow mask M [Y3] and the other masks are close to each other is based on whether the closest distance between the edges of both masks is within a predetermined number of pixels (for example, within two pixels). It is determined by whether or not. In such a case, the image processing apparatus 22 excludes the Y shadow mask M [Y3] from the subsequent processing. That is, a mask for extracting the above-described subject region is selected from 12 types of masks excluding the Y shadow mask M [Y3] among the 13 types of masks described in step S106.

  The mask may be further excluded based on the relative position between the Y shadow mask M [Y3] and the other masks. In the example of FIG. 9, for example, a Cb blue mask M [Cb1], a Cb intermediate that is included in the Y shadow mask M [Y3] or substantially overlaps with the Y shadow mask M [Y3]. The five types of masks M [Cb2], Cr green side mask M [Cr3], and center mask M [a] may be excluded from the subsequent processes in the same manner as the Y shadow mask M [Y3] described above. .

  The reason why some masks are excluded in this way is that the target image is an image having an influence of a shadow, and by excluding a mask that is considered to have a large influence of a shadow, This is to prevent the shadow portion from being detected as the subject area. Then, as in step S106, the image processing apparatus 22 obtains a mask having a certain evaluation value or higher while giving priority to each mask having a high priority, and selects it as a mask for extracting a subject area.

  Furthermore, the priority may be changed based on the relative position between the Y shadow mask M [Y3] and the other masks. For example, when the Y shadow mask M [Y3] and other masks are close to each other, the priority of the Y shadow mask M [Y3] is set to the mask extraction process (as in the case (2) described above). It may be lower than in the case of I). In addition, the Cb blue mask M [Cb1], the Cb intermediate mask M [Cb2], the Cr green mask M [Cr3], and the center mask M [a] which are substantially overlapping with the Y shadow mask M [Y3]. Also for these five types of masks, the priority may be relatively lower than in the mask extraction process (I).

  In each mask extraction process described in step S106 and step S107, the image processing apparatus 22 may extract only one subject area or a plurality of subject areas. The change in the mask extraction process (II) described in step S107 is an example. In any case, the mask creation method and the mask selection method for extracting the subject region may be changed depending on whether the target image is an image having an influence of a shadow. Further, for example, the above points may be changed more finely according to the degree of the influence of the shadow described above.

  In step S108, the CPU 26 records the result of the mask extraction process in step SS106 or step S107 as subject information in the buffer memory unit 24, the storage unit 25, or the like. The subject information includes information such as the extraction conditions (color classification, mask classification, etc.) of the mask extraction process, the position of the mask, the size and shape of the mask, the position of the extracted subject area, and the size and shape of the subject area.

  In step S109, the CPU 26 determines whether or not a shooting instruction has been issued. If the CPU 26 determines that a shooting instruction has been given, the process proceeds to step S110. On the other hand, if it is determined that the shooting instruction is not performed, the CPU 26 returns to step S104 and performs the processing from step S104 on the image of the next frame. The shooting instruction is given by a user operation via the shutter button of the operation unit 27.

  In step S110, the CPU 26 controls each unit to perform shooting. At this time, the CPU 26 performs shooting based on the subject information recorded in step S108. The CPU 26 performs 3A processing of focus adjustment (AF) setting processing, exposure adjustment (AE) setting processing, and white balance adjustment processing (AWB) based on the subject information recorded in step S108, and the image processing device 22. The conditions of various image processing in are determined.

  In the focus adjustment (AF) setting process, the CPU 26 shapes the extracted subject area and executes a contrast scan. Then, the CPU 26 causes the AF lens 11 to focus based on the result of the contrast scan (focus drive).

  In addition, when a shooting instruction has been issued in a state where mask extraction processing and subject information recording have not been completed, the CPU 26 performs normal 3A processing and performs shooting based on the result of this 3A processing. do it. Alternatively, reception of user operations relating to shooting instructions may be prohibited until mask extraction processing and recording of subject information are completed.

  In step S <b> 111, the CPU 26 records an image generated by imaging on the storage medium 40 via the communication unit 28 and ends a series of processes.

  As described above, according to the present embodiment, the image data to be processed is acquired, the feature amount of the image indicated by the image data is calculated, and the subject region determined based on the feature amount is detected. Further, an evaluation value indicating the influence of the shadow in the acquired image is calculated, and a subject area detection method is determined based on the calculated evaluation value. Therefore, even if the image to be processed is affected by a shadow, the subject area can be accurately extracted without erroneously recognizing the shadow area.

  In particular, for a through image for composition confirmation, since the display time generally increases, such a device has a great effect.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  In the above-described embodiment, an example in which the detection method of the subject area is determined according to whether or not the image to be processed has an influence of a shadow has been described, but the present invention is not limited to this example. For example, a process (mask extraction process (II)) when there is an influence of a shadow may be executed based on a user operation via the operation unit 27. Also, depending on the type of so-called scene mode in which shooting conditions etc. are preset according to the scene, such as landscape mode and night view mode, a process (mask extraction process (II)) when there is an influence of shadow is executed. Also good.

  In the above embodiment, when the photometry is started in step S101 in FIG. 2, if the change in BV value is large and unstable, the determination process in step S105 among the processes in and after step S104 is performed. It may not be performed correctly. Therefore, the configuration may be such that the BV value is monitored after the photometry is started in step S101, and the processing after step S104 is performed after confirming that the BV value has stabilized to some extent.

  In the above-described embodiment, an example in which a series of processing is performed based on a through image for composition confirmation has been described, but the present invention is not limited to this example. For example, the present invention can be similarly applied to a case where a live view image for composition confirmation generated in a single-lens reflex camera or the like is targeted. Further, the present invention can be similarly applied to a case where a moving image recorded on the recording medium 40 or the like is targeted.

  In the above embodiment, an example in which a series of processing is performed for all frames has been described, but the present invention is not limited to this example. For example, a plurality of images generated intermittently in time may be targeted. Specifically, a plurality of images subjected to frame thinning as appropriate may be targeted. By performing such processing, the processing load can be reduced.

  Further, the image processing described in the above-described embodiment may be realized by software by a “computer system” including a computer and an image processing program. In this case, a part or all of the processing of the flowchart described in the embodiment may be configured to be executed by the computer system. For example, part or all of the processing from step S104 to step S108 in FIG. 2 may be executed by a computer. By adopting such a configuration, it is possible to perform the same processing as in the above-described embodiment. In this case, the computer may acquire the BV value and contrast value used in the embodiment together with the image to be processed.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a www system is used. The computer-readable recording medium is a writable nonvolatile memory such as a flexible disk, a magneto-optical disk, a ROM, or a flash memory, a portable medium such as a CD-ROM, and a storage such as a hard disk built in the computer system. Refers to the device.

  Further, the computer-readable recording medium is a volatile memory (for example, DRAM (Dynamic Random Access) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory)) that holds a program for a certain period of time is also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 20 ... Imaging device, 21 ... Imaging part, 22 ... Image processing apparatus, 23 ... Display part, 26 ... CPU

Claims (9)

An acquisition unit for acquiring image data to be processed;
A region detecting unit that calculates a feature amount of an image indicated by the image data and detects a subject region determined based on the feature amount;
An evaluation value calculation unit for calculating an evaluation value indicating an influence of a shadow in the image;
A control unit that determines a detection method of the subject region by the region detection unit based on the evaluation value calculated by the evaluation value calculation unit ;
Equipped with a,
The region detection unit calculates at least a feature amount related to luminance as the feature amount, and detects a subject region based on at least a low luminance region based on the feature amount related to luminance.
The control unit determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold, and when the influence of the shadow is large, the subject area based on the low luminance area Image processing apparatus for changing the detection method . The image processing apparatus according to claim 1.
The region detection unit detects a subject region based on a spatial continuous region of pixels in which the result of comparing the feature value related to the luminance and a predetermined threshold is true,
When the influence of the shadow is large, the control unit sets the threshold value to be smaller than that when the influence of the shadow is not large, and the area detection unit detects the subject area based on the spatial continuous area. An image processing apparatus to perform . The image processing apparatus according to claim 1 or 2,
The region detection unit calculates a plurality of feature amounts as the feature amount, detects a plurality of subject regions based on the plurality of feature amounts,
The image processing apparatus , wherein the control unit determines priorities regarding the plurality of subject areas based on the evaluation values, and determines a main subject area based on the determined priorities . The image processing apparatus according to claim 3.
The region detection unit calculates at least a feature amount related to luminance as the feature amount, and detects a subject region based on at least a low luminance region based on the feature amount related to luminance.
The control unit determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold, and when the influence of the shadow is large, the subject area based on the low luminance area The image processing apparatus determines the main subject region by setting the priority for the lower than the case where the influence of the shadow is not large . The image processing apparatus according to claim 3 .
The region detection unit calculates at least a feature amount related to luminance as the feature amount, and detects a subject region based on at least a low luminance region based on the feature amount related to luminance.
The control unit determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold, and when the influence of the shadow is large, the subject area based on the low luminance area Is further adjacent to the other subject area, and if the subject area based on the low brightness area is close to the other subject area, the priority regarding the subject area based on the low brightness area is determined. An image processing apparatus that determines the main subject region by setting a degree lower than that when the influence of the shadow is not large . The image processing apparatus according to claim 1 or 2 ,
The region detection unit calculates a feature amount related to at least luminance as the feature amount, and detects a plurality of subject regions including a subject region based on at least a low luminance region based on the feature amount related to the brightness,
The control unit determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold, and when the influence of the shadow is large, the subject area based on the low luminance area Is further close to other subject areas, and when the subject area based on the low luminance area is close to other subject areas, the plurality of subjects detected by the area detection unit An image processing apparatus for determining a main subject area from among subject areas excluding a subject area based on the low luminance area . The image processing apparatus according to any one of claims 1 to 6 ,
The evaluation value calculation unit calculates the evaluation value based on at least one of information related to brightness of the image and information related to contrast,
The control unit determines that the influence of the shadow is large when the brightness is brighter than a predetermined reference or when the brightness is brighter than a predetermined reference and the contrast is higher than a predetermined reference. Processing equipment. An imaging unit that captures an image by an optical system and generates image data;
  A region detecting unit that calculates a feature amount of an image indicated by the image data and detects a subject region determined based on the feature amount;
  An evaluation value calculation unit for calculating an evaluation value indicating an influence of a shadow in the image;
  A control unit that determines a detection method of the subject region by the region detection unit based on the evaluation value calculated by the evaluation value calculation unit;
  With
  The region detection unit calculates at least a feature amount related to luminance as the feature amount, and detects a subject region based on at least a low luminance region based on the feature amount related to luminance.
  The control unit determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold, and when the influence of the shadow is large, the subject area based on the low luminance area Imaging device for changing the detection method.
On the computer,
  An acquisition procedure for acquiring image data to be processed;
  A region detection procedure for calculating a feature amount of an image indicated by the image data and detecting a subject region determined based on the feature amount;
  An evaluation value calculation procedure for calculating an evaluation value indicating an influence of a shadow in the image;
  A control procedure for determining a method of detecting the subject area in the area detection procedure based on the evaluation value calculated in the evaluation value calculation procedure;
  As well as
  The region detection procedure includes a procedure of calculating a feature amount related to at least luminance as the feature amount, and detecting a subject region based on at least a low luminance region based on the feature amount related to the luminance,
  The control procedure determines whether or not the influence of the shadow is large by comparing the evaluation value with a predetermined threshold value, and if the influence of the shadow is large, the subject area based on the low luminance area Image processing program including a procedure for changing the detection method.

Images