JP5609425B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program for identifying a region included in an image.

  An image processing apparatus that extracts a region from a captured image and identifies the extracted region based on a histogram of image feature values is disclosed (see Patent Document 1).

Japanese Patent Laid-Open No. 4-10079

  However, the image processing apparatus disclosed in Patent Document 1 cannot extract a region that is noticed by a person from a captured image, that is, cannot predict a region of a main subject.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an imaging apparatus, and an image processing program capable of predicting a region of a main subject from a captured image. .

  The present invention has been made to solve the above-described problem, and for each of a plurality of features in an image, a feature amount calculation unit that calculates a feature amount of the feature, and an area included in the image, An area determined for each feature quantity is detected from the image, and an area detection unit for setting an identifier and a score for comparing the areas detected based on the feature quantities of the different features are provided for each area. An image processing apparatus comprising: a score calculation unit to calculate; and a priority setting unit that sets a priority of the region in the image based on the score.

  In addition, the present invention provides an imaging unit that captures an image by an optical system and outputs the captured image, a feature amount calculation unit that calculates a feature amount of the feature for each of a plurality of features in the image, and the image An area that is included and that is determined for each feature quantity is detected from the image, and an area detection unit that sets an identifier is compared with the areas detected based on the feature quantities of different features An imaging apparatus comprising: a score calculation unit that calculates a score for each region; and a priority setting unit that sets a priority of the region in the image based on the score.

  In addition, the present invention also allows a computer to detect a feature amount of a feature for each of a plurality of features in an image and a region that is included in the image and that is determined for each feature amount from the image. , A procedure for setting an identifier, a score for comparing the areas detected based on the feature quantities of the different features, a score calculation unit that calculates for each area, and based on the scores, An image processing program for executing a priority setting unit that sets the priority of the area in the image.

  According to the present invention, the image processing apparatus can predict the area of the main subject from the captured image.

2 is a block diagram illustrating a configuration of a lens barrel 111, an imaging device 100 including an image processing device 140, and a storage medium 200. FIG. 4 is a flowchart illustrating an outline of image processing in the image processing apparatus 140. 12 is a flowchart illustrating an image processing procedure of the image processing apparatus 140 when framing a subject in a through image and tracking a subject image. 6 is a flowchart illustrating a procedure of image processing executed by the image processing apparatus 140 when the CPU 190 receives a shooting instruction via the operation unit 180. 14 is a flowchart illustrating a procedure of processing in which a region detection unit 142 sets a “motion label” in a moving subject image region in a through image. 10 is a flowchart illustrating a procedure of processing in which a region detection unit 142 sets a “motion label” when tracking a moving subject image in a through image. It is a flowchart which shows the procedure of the process which the area | region detection part 142 sets a "motion label", when snowfall, rainfall, a fountain, etc. are imaged in a through image. It is a flowchart which shows the procedure of the labeling process which labels the area | region detected for every feature-value of saturation S (C) and hue h. It is a histogram of saturation S (C) and its partial enlarged view. It is a flowchart which shows the procedure of the process which calculates a score for every area | region. It is a flowchart which shows the procedure of the process which focuses on the area | region which set the priority label (high priority and a score). It is the figure where the area | region 30, the contrast AF area | region 40, and the hue contrast were shown. It is the figure where the area | region 30 and the contrast AF area | region 40 were shown. FIG. 6 is a diagram for explaining processing for detecting an edge in a contrast AF area 40 set in a range including a boundary (outer periphery) of an area 30; The area 31 detected by the area detector 142, the contrast AF area 41 set in the area 31, the area 32 detected by the area detector 142, and the contrast AF area 42 set in the area 32 are shown. is there. 6 is a diagram illustrating an example of a region detected by a region detection unit 142, a score calculated by a score calculation unit 143, and a contrast AF region set by the region detection unit 142. FIG. It is a flowchart which shows the process sequence which detects an area | region based on a hue. It is the figure by which an example of the hue histogram was shown. It is a flowchart which shows the process sequence which detects an area | region based on the feature-value of brightness. It is a flowchart which shows the process sequence which detects an area | region based on the texture detected by an angle detection operator. It is a figure which shows the angle detection operator of 8 patterns for detecting an edge for every angle of 22.5 degrees. It is a figure which shows the example of the histogram which shows the feature-value (power value, (SIGMA) P) of an edge for every angle of an edge. It is a figure which shows the example of the area | region which the area | region detection part 142 detected based on the texture. It is a flowchart which shows the process sequence in which the area | region detection part 142 detects an area | region based on a DCT coefficient. It is a figure for demonstrating the process which the area | region detection part 142 detects an area | region based on a DCT coefficient. It is a figure for demonstrating the process in which the score calculation part 143 adds a score to the score of the area | region which the area | region detection part 142 detected based on the DCT coefficient of the high frequency band. It is a flowchart which shows the process sequence in which the area | region detection part 142 detects an area | region based on the feature-value of an edge. It is a flowchart which shows the procedure of the exposure (AE) process based on a label or a priority (score). It is a flowchart which shows the procedure of the white balance adjustment (AWB) process based on a label or a priority (score). 4 is a flowchart illustrating a basic image processing procedure including reproduction display of an image by the image processing apparatus 140. It is a flowchart which shows the procedure of the reproduction | regeneration display based on a label or a priority (score). It is a flowchart which shows the procedure of the reproduction | regeneration display based on a label or a priority (score). It is a flowchart which shows the procedure of the process which combines an area | region, and the process which divides | segments an area | region. It is a figure which shows the 1st area | region 50 detected based on the feature-value of a 1st feature, and the 2nd area | region 51 detected based on the feature-value of a 2nd feature. It is a figure which shows the example of a depth map. It is a figure for demonstrating the joining process and grouping process of the area | region which the area | region detection part 142 detected. It is a figure for demonstrating the division | segmentation process of the area | region which the area | region detection part 142 detected. It is a flowchart which shows the procedure of the process which the area | region detection part 142 detects the area | region which shows a background based on the projection of an area | region. It is a figure for demonstrating the process which detects the area | region which shows a background. It is a figure for demonstrating the process which detects the area | region which shows a background. It is a flowchart which shows the procedure of the focusing operation | movement in case a focus adjustment (AF) area | region is being fixed. It is a flowchart which shows the procedure of the focusing operation | movement in the case of imaging the night view P. It is a flowchart which shows the procedure of the focusing operation | movement with the case where face detection is possible and the case where face detection is impossible in the case of imaging the night view P.

[First Embodiment]
A first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating the configuration of a lens barrel 111, an imaging device 100 including an image processing device 140, and a storage medium 200. The imaging device 100 captures an optical image incident from the lens barrel 111 and stores the obtained image in the storage medium 200 as a still image or a moving image.

First, the configuration of the lens barrel 111 will be described.
The lens barrel 111 includes a focus adjustment lens (hereinafter referred to as “AF (Auto Focus) lens”) 112, a lens driving unit 116, an AF encoder 117, and a lens barrel control unit 118. The lens barrel 111 may be detachably connected to the imaging device 100 or may be integrated with the imaging device 100.

  The AF lens 112 is driven by a lens driving unit 116 and guides an optical image to a light receiving surface (photoelectric conversion surface) of an image sensor 119 of the image capturing unit 110 described later.

  The AF encoder 117 detects the movement of the AF lens 112 and outputs a signal corresponding to the movement amount of the AF lens 112 to the lens barrel control unit 118. Here, the signal corresponding to the movement amount of the AF lens 112 may be a sine wave signal whose phase changes according to the movement amount of the AF lens 112, for example.

  The lens barrel control unit 118 controls the lens driving unit 116 in accordance with a drive control signal input from the CPU 190 of the imaging apparatus 100. Here, the drive control signal is a control signal for driving the AF lens 112 in the optical axis direction. The lens barrel control unit 118 changes the number of steps of the pulse voltage output to the lens driving unit 116, for example, according to the drive control signal.

  Further, the lens barrel control unit 118 outputs the position (focus position) of the AF lens 112 in the lens barrel 111 to the CPU 190 based on a signal corresponding to the movement amount of the AF lens 112. Here, the lens barrel control unit 118 integrates, for example, signals according to the movement amount of the AF lens 112 according to the movement direction of the AF lens 112, thereby moving the AF lens 112 in the lens barrel 111 (the movement amount ( Position) may be calculated.

  The lens driving unit 116 drives the AF lens 112 in accordance with the control of the lens barrel control unit 118 and moves the AF lens 112 in the optical axis direction within the lens barrel 111.

Next, the configuration of the imaging device 100 will be described.
The imaging device 100 includes an imaging unit 110, an image processing device 140, a display unit 150, a buffer memory unit 130, an operation unit 180, a storage unit 160, a CPU 190, and a communication unit 170.

  The imaging unit 110 includes an imaging element 119 and an A / D (Analog / Digital) conversion unit 120. The imaging unit 110 is controlled by the CPU 190 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.).

  The image sensor 119 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 111 (optical system) into an electric signal, and outputs the electric signal to the A / D conversion unit 120. The image sensor 119 may be configured by, for example, a CMOS (Complementary Metal Oxide Semiconductor). Further, the image sensor 119 may convert an optical image into an electrical signal for a partial region of the photoelectric conversion surface (image clipping).

  Further, the image sensor 119 stores an image obtained when a shooting instruction from the user is received via the operation unit 180 in the storage medium 200 via the A / D conversion unit 120. On the other hand, the imaging device 119 performs A / D conversion on the buffer memory unit 130 and the display unit 150 as images that are continuously obtained in a state where a shooting instruction from the user is not received via the operation unit 180. Output via the unit 120.

  The A / D converter 120 digitizes the electrical signal converted by the image sensor 119 and outputs an image that is a digital signal to the buffer memory unit 130.

  The operation unit 180 includes, for example, a power switch, a shutter button, a multi-selector (cross key), or other operation keys. The operation unit 180 receives a user operation input when operated by the user, and outputs a signal corresponding to the operation input to the CPU 190. Output to.

  The image processing device 140 performs image processing on the image temporarily stored in the buffer memory unit 130 based on the image processing conditions stored in the storage unit 160. Then, the image-processed image is stored in the storage medium 200 via the communication unit 170. Further, the image processing apparatus 140 detects, from the image temporarily stored in the buffer memory unit 130, an area determined for each feature amount of the image feature and “area information (label attribute)” to be described later. . Then, the image processing apparatus 140 causes the storage unit 160 and the storage medium 200 to store the region determined for each feature amount of the image feature and the region information in association with each other.

  Here, the area information (label attribute) is, for example, the distribution (position, shape (moment of inertia)), area, number, subject distance (defocus amount), motion vector (difference data), hue in the image. , Saturation, brightness, texture (pattern), edge, AF contrast evaluation value, set flag, set label (identifier), calculated score, and set priority. These will be described later in the description of the image processing apparatus 140.

  Note that the feature amount may be, for example, a statistical feature amount or a dynamic feature amount.

  The display unit 150 is, for example, a liquid crystal display, and displays an image obtained by the imaging unit 110, an operation screen, and the like.

  The buffer memory unit 130 temporarily stores the image captured by the imaging unit 110.

  The storage unit 160 stores determination conditions referred to by the CPU 190 in scene determination, imaging conditions associated with each scene determined by scene determination, area information, and the like.

  The CPU 190 controls the imaging unit 110 according to the set imaging conditions (for example, aperture value, exposure value, etc.). In addition, the CPU 190 acquires area information from the storage unit 160 or the storage medium 200.

  Then, based on the area information, the CPU 190 performs focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, subject image (object) setting as pre-processing or post-processing. Controls tracking settings, night scene determination settings, color correction processing settings, subject image enlarged display settings, panning display settings, brightness optimization settings linked to zoom magnification, and the like.

  Further, the CPU 190 controls setting of a tone curve (gradation curve), whether or not to execute simultaneous shooting of a still image and a moving image, and the like based on area information.

  Further, the CPU 190 sets whether or not continuous shooting is performed as pre-processing or post-processing based on the region information, setting of switching of the subject image to be focused in continuous shooting, and continuous change of the focal length. It controls settings, settings for changing the amount of flash light emitted by the light emitting unit (not shown) of the image capturing apparatus 100, settings for changing the exposure amount, settings for whether to change the shutter speed, and the like.

  Further, the CPU 190 causes the image processing apparatus 140 to perform image processing as a still image or a moving image based on the “signal according to the operation input” input from the operation unit 180.

  The communication unit 170 is connected to a removable storage medium 200 such as a card memory, and writes, reads, or erases information (image data, area information, etc.) to the storage medium 200.

  The storage medium 200 is a storage unit that is detachably connected to the imaging apparatus 100 and stores information (image data, area information, and the like). Note that the storage medium 200 may be integrated with the imaging device 100.

Next, details of the image processing apparatus 140 will be described.
The image processing apparatus 140 includes a feature amount calculation unit 141, a region detection unit 142, a score calculation unit 143, and a priority setting unit 144.

  The feature amount calculation unit 141 acquires the image captured by the imaging unit 110 from the buffer memory unit 130. Then, the feature amount calculation unit 141 calculates the feature amount for each of a plurality of features (for example, hue, saturation, brightness, texture, edge, etc.) in the image acquired from the buffer memory unit 130, and calculates the calculated feature amount. Is output to the area detector 142.

  The region detection unit 142 acquires the feature amount calculated by the feature amount calculation unit 141. In addition, the area detection unit 142 acquires an image captured by the imaging unit 110 from the buffer memory unit 130. Then, the area detection unit 142 detects an area included in the acquired image and determined for each feature amount from the image. Here, the region detected by the region detection unit 142 may be, for example, a region of a subject image included in an image captured by the imaging unit 110, or may be a subject included in an image stored in the storage medium 200, for example. It may be an image area.

  Furthermore, the region detection unit 142 sets a label (identifier) for identifying a region included in the image detected for each feature amount (labeling process). Here, the label is an area (group) composed of pixels having feature quantities close to each other and grouped. Labeling is to sequentially detect and group areas adjacent to pixels whose feature amounts are close to each other (for example, corresponding to saturation h = 0 to 20 degrees). For example, it is common to detect an area where the luminance value (Y) = 128 ± 30 (labeling) and give a name (for example, label 1, label 2,...) To the detected label.

  Features for detecting a region include a motion vector, hue, saturation, brightness, texture edge, distance (depth, depth map, defocus amount), and contrast in an image (for example, contrast between adjacent pixels). )and so on. The area detection unit 142 detects an area (label) by labeling each one or more evaluation values as feature amounts.

  Also, for example, when the hue is a feature amount and a plurality of regions are detected based on the feature amount, the region detection unit 142 determines the position, shape, and degree of mass (moment of inertia) for these regions. , Size, heterogeneity, discomfort (degree of peculiarity in a certain evaluation value), etc. Moreover, the priority of each area | region is set based on these evaluation results. Here, the degree of heterogeneity is the degree of unique distribution in the histogram of feature values. For example, when an image in which a single red flower is blooming on a green background is evaluated based on the hue, the red hue is uniquely distributed in the hue histogram.

  Further, the region detection unit 142 may label the region detected for each feature amount based on the contrast value of the feature amount. Further, the label for identifying the region may be a label for identifying whether the region is a main subject image region, for example. Further, the region detection unit 142 divides the feature amount into several value ranges for the histogram distribution of each feature value, converts the image into a bitmap (BMP) for each value range, and labels the converted bitmap. To do.

  Here, the main subject image is, for example, a subject image with a short subject distance (captured on the near side) in a deep (perspective) image, a subject image that has been stationary for a certain period of time, or a hue that is flesh-colored. Subject images, subject images that do not protrude from the angle of view, subject images that have a unique distribution or range of features in the histogram of feature values, subject images that have a high degree of “mass” (small moment of inertia), etc. . Note that the condition as to whether the region is the main subject image region may be determined as appropriate.

  Then, the region detection unit 142 stores a label for identifying the region in the storage unit 160 and the storage medium 200 for each region as region information.

  The score calculation unit 143 calculates, for each label, a score for comparing regions labeled based on feature amounts of different features (for example, hue and saturation) (score calculation processing). Then, the score calculation unit 143 outputs the calculated score to the priority setting unit 144.

  The priority setting unit 144 sets “region priority” in the image based on the score calculated by the score calculation unit 143. The “region priority” is a region priority that a person will be more interested in. The priority setting unit 144 stores the set “region priority” in the storage unit 160 and the storage medium 200 for each region as region information.

  The priority of the area may be determined as follows. In the following, it is assumed that the priority of a region is higher as the number indicating the priority is smaller. Note that the priority shown below is an example.

Priority 1.
A moving area (when there are a plurality of moving areas in the image, an area that does not protrude from the angle of view or an area that is distributed in the center of the angle of view) Also, if the moving area is widely distributed in the image due to panning, the area is not moving.
These areas are set to priority “1”.
This is because a person pays attention to a moving area and an area having a small amount of change when panning.

Priority 2.
Areas related to angle of view (composition), depth (perspective), and subject distance (depth, depth). An area distributed near the center of the angle of view. When there is a depth in the image, among the regions distributed on the closest side, the region does not protrude from the angle of view and has an area within a predetermined range (not extremely large).
These areas are set to priority “2”.
This is because the main subject image area may exist in a wide range from close to infinity.

Priority 3.
Area detected based on face detection, skin color detection in the area where the face is detected, texture of animal hair, or plant texture.
These areas are set to priority “3”.
This is because when a person or animal is imaged, the person pays attention to the face.

Priority 4
A region having a characteristic amount that is unique (for example, a hue that is discontinuous in the hue circle). An area divided into a background area and other areas. Highly saturated area. An area where the brightness is significantly different from the surrounding area. An area where the hue (hue) is significantly different from the surrounding area. An area where the texture (pattern) is different from the surrounding area. An area where the spatial frequency (image frequency) is higher than the surrounding area. A region in which an image has a sense of incongruity when a certain amount of discontinuous distribution has a certain amount and a region is formed as a result of labeling based on the feature amount.
These areas are set to priority “4”.

Priority 5.
An area detected based on a specific hue or saturation.
These areas are set to the priority “5”.
This is because a person pays attention to red, yellow, and flesh-colored areas.

Priority 6.
Area detected based on brightness (brightness).
These areas are set to priority “6”.

Priority 7.
Area detected based on texture (pattern).
These areas are set to priority “7”.

Priority 8.
A region detected based on the spatial frequency (image frequency).
These areas are set to priority “8”.

Priority 9.
An area surrounded by edges extending vertically or horizontally and having a certain thickness or more.
These areas are set to the priority “9”.

Next, image processing in the image processing apparatus 140 will be described.
FIG. 2 is a flowchart showing an outline of image processing in the image processing apparatus 140. First, the image processing apparatus 140 acquires a through image. Here, the image processing apparatus 140 may acquire through images (through image sequences) that are continuous in time (step S1). Then, the image processing apparatus 140 performs primitive image processing on the through image (step S2). Here, the primitive image processing is, for example, simple exposure adjustment (simple AE), white balance adjustment (AWB), and noise reduction (NR).

  The feature amount calculation unit 141 calculates a feature amount for each feature from the features of the image (step S3). Then, the region detection unit 142 detects a region from the image for each feature amount (for example, when the feature is a hue, for each of green and red). Then, the region detection unit 142 detects the degree of difference (uncomfortable feeling) between the region and the surrounding region based on “correlation between feature amounts” or “relative distance between feature amounts between regions” or the like ( Step S4).

  Further, the region detection unit 142 performs labeling based on the feature amount calculated by the feature amount calculation unit 141. For example, the region detection unit 142 performs labeling using the feature amount of a region having a relative distance on the hue axis, and detects the degree of difference (uncomfortable feeling) between the region and the surrounding region, thereby capturing an image. Several red flowers are detected in the green grass field (step S4a). Further, the feature amount calculation unit 141 determines whether the label is valid (step S4b).

  Furthermore, the feature amount calculation unit 141 selects a feature amount to be used in the subsequent processing based on the degree of difference in the feature amount between the region and its surroundings. For example, the feature amount calculation unit 141 sets a feature amount having a large degree of difference between a region and its surroundings as a feature amount used in the subsequent processing. For example, when the “hue” is greatly different between the area and the surrounding area, the feature amount calculation unit 141 uses the “hue feature amount” in the subsequent processing (step S5).

  The region detection unit 142 sets a label for identifying the region in the region detected for each feature amount. Further, the region detection unit 142 stores the set label in the storage unit 160 for each region as region information (step S6).

  Further, the score calculation unit 143 calculates a score for each region detected by the region detection unit 142, and stores the score in the storage unit 160 and the storage medium 200 as region information (step S7). And the priority setting part 144 sets the priority of the area | region which the area | region detection part 142 detected based on the score calculated by the score calculation part 143, and memorize | stores it in the memory | storage part 160 and the storage medium 200 for every area | region ( Step S8).

  Furthermore, the area detection unit 142 shapes the detected area. For example, the region detection unit 142 shapes (expands) the region so that the contrast evaluation value of the focus color tone in the region is equal to or greater than a predetermined threshold (step S9). Then, the area detection unit 142 updates “area information” and stores it in the storage unit 160 and the storage medium 200 (step S10).

  The CPU 190 acquires area information (label) from the storage unit 160. Then, the CPU 190 sets the focus adjustment (AF), the exposure adjustment (AE), the white balance adjustment (AWB), and the subject image tracking based on the “label” set for the area by the area detection unit 142. Settings, shooting scene (for example, night scene) judgment settings, color correction processing settings, enlarged subject image display settings, panning display settings, brightness optimization settings linked to zoom magnification, Control etc.

  Further, the CPU 190 sets whether to execute tone curve (gradation curve) setting and simultaneous shooting (simultaneous recording) of a still image and a moving image based on the “label” set for the region by the region detection unit 142. Control etc.

  Further, the CPU 190 sets whether or not continuous shooting is performed based on the “label” set for the region by the region detection unit 142, setting for switching the subject image to be focused in continuous shooting, and continuous focal length. Settings for changing, settings for changing the amount of flash light emitted from the light emitting unit (not shown) of the imaging apparatus 100, settings for changing the exposure amount, setting for whether to change the shutter speed, and the like are controlled. .

  Further, the CPU 190 may acquire region information (region priority) from the storage unit 160 and control these processes based on the “region priority” set by the priority setting unit 144. (Step S11).

  When the CPU 190 receives a shooting instruction from the user via the operation unit 180, the CPU 190 causes the imaging unit 110 to capture an image (step S12). Further, the image processing apparatus 140 performs a subsequent process (image processing) based on the area information (step S13). Then, the region detection unit 142 stores the region information in the storage unit 160 and the storage medium 200 in association with the image (region) (step S14).

  FIG. 3 is a flowchart showing an image processing procedure of the image processing device 140 when the imaging device 100 frames a subject in a through image and tracks (tracks) the subject image. Note that if the image does not have a priority label and a label with a certain high score (it is difficult to determine which main subject is), the imaging apparatus 100 proceeds to 3A (AF, AWB, AE) processing. Alternatively, the imaging apparatus 100 employs the contrast AF area and the contrast AF area of the low-score label in combination, and adopts an area that has been imaged in the vicinity and an area having a high AF contrast evaluation value peak as a high-priority area. .

  First, the image processing apparatus 140 acquires a through image. Here, the image processing apparatus 140 may acquire through images (through image sequences) that are continuous in time (step Sa1). Then, the image processing apparatus 140 performs primitive image processing (for example, simple exposure adjustment) on the through image (step Sa2).

  The CPU 190 performs focus adjustment (AF). Here, the CPU 190 executes a contrast scan, and stores the lens position at which the AF contrast evaluation value shows a peak in the storage unit 160 for each part (block) of the image (step Sa3).

  Note that the CPU 190 may create a depth map indicating the distribution of the subject distance in the image based on the lens position at which the AF contrast evaluation value shows a peak. The depth map will be described later with reference to FIG.

  Then, the CPU 190 causes the imaging unit 110 to execute a debayer (color interpolation) process. In addition, the CPU 190 causes the image processing apparatus 140 to perform noise reduction processing (NR) and white balance adjustment (AWB) (step Sa4).

  The feature amount calculation unit 141 acquires an image captured by the imaging unit 110 from the buffer memory unit 130, and calculates a motion vector (motion amount and motion direction) as a feature amount in the acquired image (step Sa5). Then, the feature amount calculation unit 141 determines whether or not the motion amount is equal to or greater than a predetermined threshold value (step Sa6).

  If the amount of motion is equal to or greater than a predetermined threshold (step Sa6-YES), the region detection unit 142 sets a “motion label” in a region where the amount of motion is equal to or greater than the predetermined threshold (step Sa7). . Here, the motion label is a label indicating a region detected based on a motion vector. The process in which the area detection unit 142 sets the “motion label” will be described later with reference to FIGS. Further, the image processing apparatus 140 stores the image-processed image as a moving image in the storage medium 200 (step Sa8). Then, the processing of the image processing apparatus 140 proceeds to step Sa10.

  On the other hand, if the amount of motion is not equal to or greater than a predetermined threshold value in step Sa6 (step Sa6-NO), the region detection unit 142 detects a region for each feature amount and sets a label in the detected region (labeling process). ). Then, the region detection unit 142 stores the label (identifier) as region information in the storage unit 160 for each region.

  The score calculation unit 143 calculates, for each region, a score for comparing regions detected based on feature amounts of different features (for example, hue and saturation) (step Sa9). The process in which the score calculation unit 143 calculates the score for each area will be described later with reference to FIGS.

  The feature amount calculation unit 141 acquires the image captured by the imaging unit 110 from the buffer memory unit 130 (step Sa10). Then, the feature amount calculation unit 141 calculates a motion vector (motion amount and motion direction) in the acquired image (step Sa11).

  The feature amount calculation unit 141 determines whether there is a change in the image in a region other than the region detected by the region detection unit 142 based on the calculated motion vector. When there is a change in the image in a region other than the region detected by the region detection unit 142 (step Sa12—YES), the processing of the image processing device 140 returns to step Sa7. Based on the calculated motion vector, the feature amount calculation unit 141 further determines whether or not the range in which the image is moving is the entire image. Similarly, when the range in which the image is moving is the entire image (step Sa12—YES), the processing of the image processing device 140 returns to step Sa7.

  On the other hand, in step Sa12, when there is no change in the image in the region other than the region detected by the region detection unit 142 and the range in which the image is moving is not the entire region of the image (step Sa12-NO), the feature amount calculation is performed. The unit 141 detects a feature quantity other than a motion vector for each region detected by the region detection unit 142 and outputs the feature amount to the score calculation unit 143. And the score calculation part 143 calculates a score for every area | region which the area | region detection part 142 gave the label.

  Further, the priority setting unit 144 sets the priority of the area in the image based on the score calculated by the score calculation unit 143 (step Sa13). The process in which the priority setting unit 144 sets the priority for the area will be described later with reference to FIGS. Then, the priority setting unit 144 outputs the region priority to the CPU 190.

  The CPU 190 controls focus adjustment (AF) for the region detected by the region detection unit 142 based on the “label” attached to the region by the region detection unit 142. For example, the CPU 190 controls the focus adjustment so as to preferentially focus on an area with a “label identifying the main subject image area”.

  Further, the CPU 190 may control the focus adjustment (AF) for the region detected by the region detection unit 142 based on the “region priority” set by the priority setting unit 144. For example, the CPU 190 may control the focus adjustment so that the area set with the highest priority is set as the main subject image area and is focused on the area with priority (step Sa14). .

  The CPU 190 controls exposure adjustment (AE) for the region detected by the region detection unit 142 based on the label attached to the region by the region detection unit 142 based on the lightness gradation. For example, the CPU 190 controls the exposure adjustment so that the area with the “label identifying the area of the main subject image” has an optimum exposure amount.

  Further, the CPU 190 controls exposure adjustment (AE) for the area detected by the area detecting unit 142 based on the “area priority” set by the priority setting unit 144 based on the lightness gradation. May be. For example, the CPU 190 may control the exposure adjustment so that the area set with the highest priority is set as the main subject image area and the exposure amount is optimal for the area (step Sa15). .

  The CPU 190 adjusts the white balance (AWB) for the area detected by the area detection unit 142 based on the “label” attached to the area based on the “area information (for example, white)” by the area detection unit 142. To control. For example, the CPU 190 controls the white balance adjustment so that the optimum white balance is obtained for the region with the “label identifying the region of the main subject image”.

  In addition, the CPU 190 performs white balance on the area detected by the area detection unit 142 based on the “area priority” set by the priority setting unit 144 based on “area information (for example, white)”. Adjustment (AWB) may be controlled. For example, the CPU 190 may control the white balance adjustment so that the region with the highest priority is set as the main subject image region and the white balance is optimal for the region (step Sa16). ).

  The CPU 190 executes image scene determination for the area detected by the area detection unit 142 based on the “label” attached to the area by the area detection unit 142 based on the lightness gradation. For example, the CPU 190 captures the night scene based on the distribution of the point light source region, with the region with the “label identifying the region of the non-main subject image” as the region where the point light source is captured. It is determined whether it is an image.

  Further, the CPU 190 may determine the scene of the image based on the “region priority” set by the priority setting unit 144 based on the lightness gradation. For example, the CPU 190 determines whether or not the night scene is an image based on the distribution of the area of the point light source, with the area having the lowest priority set as the area where the point light source is imaged. (Step Sa17).

  Then, the processing of the image processing apparatus 140 proceeds to step Sa10 and step Sa18. In this way, the image processing apparatus 140 repeats the processing of Step Sa10 to Step Sa17.

  While the image processing apparatus 140 repeats the processing of Step Sa10 to Step Sa17, the CPU 190 causes the imaging unit 110 to execute a debayer (color interpolation) process. The CPU 190 also causes the image processing apparatus 140 to perform noise reduction processing (NR), white balance adjustment (AWB), tone curve (tone curve) setting, Retinex (tone correction) processing, color correction processing, rendering, and edge enhancement. The process is executed (step Sa18).

  The image processing apparatus 140 outputs, to the buffer memory unit 130, an image of a frame indicating a “tracking area” that is a range in which a calculation for tracking the detected area is executed (step Sa19). Further, the image processing apparatus 140 causes the display unit 150 to display an image obtained by superimposing (overlaying) an image of a frame indicating the tracking area. As described above, the image processing apparatus 140 repeats the processes of Step Sa18 to Step Sa20 in parallel with the processes shown in Steps Sa10 to 17.

In a loop in which the image processing apparatus 140 repeats the processing of step Sa10 to step Sa17, the CPU 190 receives a shooting instruction from the user via the operation unit 180.
FIG. 4 is a flowchart showing an image processing procedure executed by the image processing apparatus 140 when the CPU 190 receives a shooting instruction via the operation unit 180.

  When the CPU 190 receives (detects) a shooting instruction from the user via the operation unit 180, the CPU 190 outputs a signal indicating that the shooting instruction has been received to the image processing apparatus 140 (step Sb1). When a signal indicating that the photographing instruction has been received is acquired from the CPU 190 (step Sb1-YES), the processing of the image processing device 140 and the CPU 190 proceeds to step Sb2.

  The CPU 190 controls the exposure adjustment (AE, dimming) for the area detected by the area detecting unit 142 based on the “label” set in the area by the brightness gradation by the area detecting unit 142. Further, the CPU 190 may control the exposure adjustment for the region detected by the region detection unit 142 based on the “region priority” set by the priority setting unit 144 based on the lightness gradation. (Step Sb2).

  The CPU 190 causes the imaging unit 110 to capture (exposure) an optical image incident from the lens barrel 111 based on the “label” set in the region by the brightness gradation by the region detection unit 142. Further, the CPU 190 captures (exposures) an optical image incident from the lens barrel 111 on the imaging unit 110 based on the “region priority” set by the priority setting unit 144 based on the lightness gradation. (Step Sb3).

  Then, the CPU 190 causes the imaging unit 110 to execute a debayer (color interpolation) process. In addition, the CPU 190 causes the image processing apparatus 140 to perform noise reduction processing (NR) (step Sb4).

  Further, the CPU 190 performs white balance adjustment (AWB) on the area detected by the area detecting unit 142 based on the “label” set for the area by the “area information (hue etc.)”. Control. Further, the CPU 190 adjusts the white balance for the area detected by the area detection unit 142 based on the “area priority” set based on the “area information (hue, etc.)” by the priority setting unit 144. May be controlled (step Sb5).

  The image processing apparatus 140 sets a tone curve (gradation curve) for the image, and executes Retinex (gradation correction for each region) process (step Sb6). Then, the image processing apparatus 140 executes color correction processing and rendering (step Sb7). Further, the image processing apparatus 140 executes edge enhancement processing (step Sb8).

  The image processing apparatus 140 displays the processed image as a still image (freeze image) on the display unit 150 (step Sb9). Then, the image processing apparatus 140 converts the image into a predetermined image format (for example, JPEG format) (step Sb10). Then, the image processing apparatus 140 stores the image converted into the predetermined image format in the storage medium 200 (step Sb11).

  Furthermore, the region detection unit 142 stores the information on the region in the storage unit 160 and the storage medium 200. For example, the image processing apparatus 140 may store the region information in the storage unit 160 and the storage medium 200 as tag information of “Exchangeable image file format (Exif (registered trademark))” (step Sb12).

  Next, when there is a large motion in the image, that is, when the amount of motion is greater than or equal to the threshold (see step Sa6 in FIG. 3), the region detection unit 142 displays “motion label” Details of the process of setting (see step Sa7 in FIG. 3) will be described.

  FIG. 5 is a flowchart showing a processing procedure in which the area detection unit 142 sets a “motion label” in the area of the moving subject image in the through image. First, the feature amount calculation unit 141 forms a histogram of the motion amount (step Sc1). Then, based on the histogram, the region detection unit 142 determines whether or not the number of pixels whose motion amount (average) is equal to or less than a threshold value is equal to or greater than a predetermined number of pixels and whether the motion amount has a peak value. Is determined (peak value is detected) (step Sc2).

  In the through image, when the number of pixels whose motion amount (average) is equal to or less than the threshold is equal to or more than a predetermined number of pixels and the motion amount has a peak value (step Sc2-YES), the region detection unit 142 Performs pixel labeling based on the “motion amount” at the position of the image where the motion amount exhibits a peak value (step Sc3). Then, based on the “movement direction” at the position of the image where the amount of movement shows a peak value, the area detection unit 142 classifies the movement direction into eight angles by 45 degrees, for example. Labeling is executed (step Sc4).

  Furthermore, the region detection unit 142 detects a portion (positional product set) where the region detected based on the amount of motion and the region detected based on the direction of motion overlap, and calculates the area of the detected overlapping portion ( Step Sc5). Further, the region detection unit 142 removes an isolated region (noise) in the through image (step Sc6).

  In addition, the region detection unit 142 sets a motion label in the labeled region. Here, when there are a plurality of regions with motion labels, the priority setting unit 144 determines the hue, saturation, unique hue (for example, discontinuous hue in the hue circle), and unique lightness (for example, The priority of the region in the image is set on the basis of the brightness (which is significantly different from the surroundings) (step Sc7). Further, the region detection unit 142 stores the motion label in the storage unit 160 for each region as region information (step Sc8).

FIG. 6 is a flowchart showing a processing procedure in which the area detection unit 142 sets a “motion label” when tracking a moving subject image in a through image.
In the through image, when the number of pixels whose motion amount (average) is less than or equal to the threshold is smaller than the predetermined number of pixels or there is no peak value in the motion amount (step Sc2-NO in FIG. 5), the region The detection unit 142 determines whether or not the number of pixels whose motion amount (average) is equal to or greater than a threshold value is equal to or greater than a predetermined number of pixels, and whether or not there is a peak value on the side with a smaller motion amount in the motion amount histogram Is determined (peak value is detected) (step Sc9).

  When the number of pixels whose motion amount (average) is equal to or greater than the threshold is equal to or greater than a predetermined number of pixels, and there is a peak value on the side where the motion amount is small in the motion amount histogram (step Sc9-YES) The region detection unit 142 performs pixel labeling on a region where the amount of motion shows a peak value (step Sc10).

  Then, the region detection unit 142 labels the region labeled with pixels based on the amount of motion again based on the hue or brightness (step Sc11). Further, the region detection unit 142 sets a motion label in the region that has been labeled again (step Sc12). Furthermore, the image processing apparatus 140 stores the motion label in the storage unit 160 for each region as region information (step Sc13).

  FIG. 7 is a flowchart showing a procedure of processing in which the region detection unit 142 sets a “motion label” when snowing, raining, or a fountain is captured in the through image. When the number of pixels whose motion amount (average) is equal to or greater than the threshold is smaller than a predetermined number of pixels or there is no peak value on the side where the motion amount is small in the motion amount histogram (step Sc9-NO in FIG. 6) ) The region detection unit 142 determines whether or not the motion amount is divided into two and the motion direction is constant in the through image (step Sc14).

  In the through image, when the motion amount is divided into two and the motion direction is constant (step Sc14-YES), the region detection unit 142 determines whether or not pixels whose motion amount is greater than or equal to the threshold value are dispersed in the image. Determination is made (step Sc15). When pixels whose motion amount is greater than or equal to the threshold value are dispersed in the image (step Sc15-YES), the region detection unit 142 performs pixel labeling on the region that shows the peak value on the side with the smaller motion amount in the histogram. Execute (Step Sc16).

  Then, the region detection unit 142 calculates a hue (average hue) having a large area in the detected region (step Sc17). Further, the region detection unit 142 masks a part of the temporally continuous through image (through image sequence) with a motion vector to create a plurality of mask images (step Sc18). The region detection unit 142 masks a part of the through image (through image sequence) with a motion vector, thereby extracting, for example, a region other than raindrops into the mask image. Then, the region detection unit 142 combines the plurality of created mask images, and labels the combined mask images again based on the hue (step Sc19).

  The region detection unit 142 sets a label (hereinafter referred to as “normal label”) indicating that the amount of movement is a region (for example, a region other than raindrops) in the labeled region (step Sc20). And the area | region detection part 142 memorize | stores a normal label in the memory | storage part 160 for every area | region as area | region information (step Sc21).

  On the other hand, in the through image, when the motion amount is not divided into two or the motion direction is not constant (step Sc14-NO), the processing of the region detection unit 142 proceeds to step Sa8 in FIG. Similarly, when pixels whose motion amount is equal to or greater than the threshold value are not dispersed in the image (step Sc15—NO), the processing of the region detection unit 142 proceeds to step Sa8 in FIG.

  Next, when there is no large motion in the image, that is, when the amount of motion is not equal to or greater than the threshold (see step Sa6 in FIG. 3), the region detection unit 142 detects a region for each feature amount, Details of the process of setting the label (see step Sa9 in FIG. 3) will be described.

  FIG. 8 is a flowchart showing a labeling process procedure for setting a label in an area detected for each feature amount of saturation S (C) and hue h. Steps Sd3-8 and steps Sd10-13 are processed in parallel.

  The feature amount calculation unit 141 converts the RGB color space in the image into a Lab color space. The image processing apparatus 140 uses data expressed in the Lab color space in the subsequent processing (step Sd1). Then, the feature amount calculation unit 141 calculates the feature amount of the saturation S (C) and the hue h from the image (Step Sd2).

  First, the feature amount calculation unit 141 calculates a histogram of saturation S (C) (step Sd3). Then, the region detection unit 142 detects a region having a particularly high saturation. Here, the region detection unit 142 determines whether or not there is a peak in the high saturation region based on the distribution of the histogram (detects peaks and valleys). Furthermore, the region detection unit 142 determines the saturation specificity based on the difference between the peak value and the average saturation (step Sd4).

  FIG. 9 shows a histogram of saturation S (C) and a partially enlarged view thereof. The area detection unit 142 detects peaks S1 and valleys in the histogram.

  Returning to FIG. 8, the description of the procedure is continued. Further, the feature amount calculation unit 141 calculates the standard deviation σ of the peak waveform (step Sd5). Here, when the standard deviation σ is equal to or smaller than a predetermined threshold value (a sharp peak waveform), the region detection unit 142 performs a labeling process on a region indicating the peak waveform (for example, a range of 3σ). Execute (step Sd6).

  The region detection unit 142 creates a bitmap image of a portion where the region detected based on the saturation and the “region detected based on the hue” described later in steps Sd10 to 12 are combined (a combination of different features). Then, the score calculation unit 143 adds “2” to the score of the portion where the region detected based on the saturation and the region detected based on the hue overlap. For example, the score of a portion where the region detected based on the saturation and the region detected based on red (score = 3 points) overlap is “5 = 3 + 2” (step Sd7).

  Further, the region detection unit 142 displays a “saturation label” indicating that the region detected based on the saturation is in a portion where the region detected based on the saturation overlaps with the region detected based on the hue. Set (step Sd8). Then, the process of the area detection unit 142 proceeds to step Sd9.

  On the other hand, the area detection unit 142 detects an area based on the hue h after step Sd2. For this purpose, the feature amount calculation unit 141 calculates the hue h from the image. Then, the region detection unit 142 creates a plurality of bitmap images from one image for each hue calculated by the feature amount calculation unit 141 (step Sd10). Further, the region detection unit 142 performs a labeling process on the region detected based on the hue (step Sd11).

  Further, the score calculation unit 143 has a score of “3” for the red or yellow hue area, a score of “2” for the purple, cyan, or emerald hue area, and a score of “other” hue area. 1 "point is calculated (step Sd12).

  Further, the region detection unit 142 sets a “hue label” indicating that the region is detected based on the hue in the region detected based on the hue (step Sd13). Then, the processing of the area detection unit 142 proceeds to step Sd9 in FIG.

  The priority setting unit 144 sets the priority of the area in the image according to the score of the area. Then, the priority setting unit 144 stores the “region priority” set according to the score in the storage unit 160 for each region.

  Further, the area detection unit 142 sets a “priority label” indicating that the area has a high priority in the area having a high score set by the priority setting unit 144. Further, the region detection unit 142 sets “highest priority label” to a region having the highest priority among the regions for which motion labels are set (step Sd9). Then, the region detection unit 142 stores the set label (identifier) in the storage unit 160 for each region.

  The CPU 190 controls focus adjustment (AF) and the like for the area detected by the area detection unit 142 based on the “label” set by the area detection unit 142 (for example, step S11 in FIG. 2 and FIG. 3). Step Sa14). For example, the CPU 190 controls the focus adjustment (AF) so that the area where the “highest priority label” is set is set as the area of the most main subject image, and the area is preferentially focused.

  Further, the CPU 190 may control focus adjustment (AF) or the like for the area detected by the area detection unit 142 based on the “area priority” set by the priority setting unit 144. For example, the CPU 190 may control the focus adjustment so that the area set with the highest priority is set as the main subject image area and is focused on the area with priority.

Next, a procedure for the score calculation unit 143 to calculate a score for each region will be described.
FIG. 10 is a flowchart showing a procedure of a process for calculating a score for each area (for example, see steps Sd7 and S12 in FIG. 8).

  The region detection unit 142 sets a labeling condition according to the feature amount (step Se1). This condition may be determined as appropriate. For example, the condition may be that the region detection unit 142 sets a “highest priority label” to the region detected based on the skin hue. Then, the region detection unit 142 performs pixel labeling based on the labeling conditions (step Se2). Then, the region detection unit 142 excludes an isolated region (isolated point) in the through image from the target for calculating the score (step Se3).

  The score calculation unit 143 calculates the center of gravity of each area (step Se4). Then, the score calculation unit 143 calculates the distance from the center of the image (center of angle of view) to the center of gravity of the region for each region (step Se5). And the score calculation part 143 gives a score for every area | region according to the distance from the center (angle of view angle) of an image to the gravity center of an area | region. Here, the score calculation unit 143 assigns a higher score to the region closer to the center of the image (center of view angle) (for example, score = 2 points, 1.5 points, 1 point, 0.5 points). Order) (step Se6).

  Further, the score calculation unit 143 calculates the moment of inertia for each region in order to evaluate the degree of concentration of the region at its center of gravity (step Se7). The score calculation unit 143 excludes a region where the moment of inertia is greater than or equal to a predetermined threshold, that is, a region not concentrated on the center of gravity, from the subject to be scored (step Se8).

  The score calculation unit 143 gives a score to the area according to the moment of inertia of the area. The score calculation unit 143 assigns a high score to a region where the moment of inertia is small (concentrated at the center of gravity) (for example, score = 2 points, 1.5 points, 1 point, 0.5 points in this order) ( Step Se9).

  Furthermore, the score calculation unit 143 calculates the area (number of pixels) of the region detected by the region detection unit 142, and selects a region having an area equal to or larger than a predetermined ratio (for example, 70%) with respect to the area of the image. Then, it is excluded from the target to be scored (step Se10). This is because such a region may not be a main subject image region such as a region indicating a background.

  Then, the score calculation unit 143 gives a score to the area according to the area (number of pixels) of the area. For example, the grade may be four levels from “2 to 0.5”. Further, the score calculation unit 143 gives a higher score to a region closer to a predetermined area (step Se11). Further, the score calculation unit 143 stores the score in the storage unit 160 for each region as region information (step Se12).

  Next, a process procedure for focusing on an area where priority labels (high priority and score) are set will be described.

  FIG. 11 is a flowchart showing a processing procedure for focusing on an area where priority labels (high priority and score) are set. This process is executed, for example, at step Sa9 in FIG. 3 (more specifically, step Sd9 in FIG. 8) or step Sa14 in FIG.

  The area detection unit 142 widens the difference between the upper and lower thresholds of the hues grouped in the same label until a predetermined contrast value can be acquired, and performs pixel labeling based on the widened threshold (step Sg1). As a result, the area of the label is expanded.

  FIG. 12 shows a region 30, a contrast AF region 40 that is a region for contrast scanning, and a hue contrast of the region 30. FIG. 12A shows the region 30 and the contrast AF region 40 in step Sg1 of FIG. FIG. 12B shows the hue contrast in FIG. 12A, and the hue contrast of the region 30 in the contrast AF region 40 is shown. FIG. 12B shows that the hue range of the region 30 in the contrast AF region 40 is expanded by expanding the region 30.

  Returning to FIG. 11, the description of the procedure is continued. The region detection unit 142 detects a hue (such as a color channel) in which the contrast value is dominant (shows a peak value) in the hue contrast in which the hue range is expanded (step Sg2). Until the predetermined contrast value can be acquired, the processing of the region detection unit 142 repeats step Sg1 and step Sg2.

  FIG. 13 shows the region 30 and the contrast AF region 40 in step Sg2. FIG. 13 shows that the contrast AF area 40 is set in the area 30 where the edge power is strong.

  Returning to FIG. 11, the description of the procedure is continued. The region detection unit 142 detects a region having a strong edge power in the region 30 based on the texture (pattern) of the region 30 (step Sg3). Then, the region detection unit 142 detects the “edge hue” of the region 30 (step Sg4).

  The region detection unit 142 sets a “contrast AF region” that is a region for contrast scanning in a region where the edge power is strong. Here, even if the subject distance is unknown, the region detection unit 142 sets the contrast AF region if the priority label (high priority and score) is set (step Sg5). .

  The area detection unit 142 sets the exposure amount in the contrast scan based on the “area information” of the detected area and the “point light source flag” indicating that a point light source such as a night view is captured (step Sg6).

  Then, the region detection unit 142 determines whether or not the detected region has low contrast or low brightness (step Sg7). When the detected area is not low contrast or low brightness (step Sg7-NO), the CPU 190 executes a contrast scan in the contrast AF area (step Sg8).

  On the other hand, when the detected area has low contrast (low brightness) (step Sg7-YES), the area detection unit 142 searches for an edge near the boundary (outer periphery) of the detected area, and the hue of the searched edge ( Color configuration) is detected (step Sg9).

  The region detection unit 142 sets the contrast AF region 40 in a range including the boundary (outer periphery) of the region 30, and determines whether there is an edge (detects an edge) in the set contrast AF region 40 (step Sg10). ).

  FIG. 14 is a diagram for explaining processing for detecting an edge in the contrast AF area 40 set in a range including the boundary (outer periphery) of the area 30. FIG. 14 shows that the contrast AF area 40 is set in a range including the boundary (outer periphery) of the area 30.

  Returning to FIG. 11, the description of the procedure is continued. When there is no edge in the contrast AF area 40 (step Sg10-NO), the area detection unit 142 detects the edge in the contrast AF area 40 of the low-contrast area 30, so that the sensitivity for detecting the contrast of hue and luminance is set. increase. For example, the area detection unit 142 increases the sensitivity for detecting the hue and luminance contrast by extending the exposure time or changing the gamma table (step Sg11).

  In FIG. 15, the region 31 detected by the region detection unit 142, the contrast AF region 41 set in the region 31, the region 32 detected by the region detection unit 142, and the contrast AF region 42 set in the region 32 are shown. It is shown. Here, as an example, the hue and brightness of the region 31 are not low contrast. On the other hand, the hue and brightness of the region 32 are assumed to be low contrast.

  The area detection unit 142 sets a normal gamma table in the contrast AF area 41 in order to detect an edge in the contrast AF area 41 of the area 31. On the other hand, the area detection unit 142 sets a high gamma table with increased sensitivity in the contrast AF area 42 in order to detect edges in the contrast AF area 42 of the area 32 having low contrast (low luminance). As described above, the region detection unit 142 increases the sensitivity of detecting the contrast of hue and luminance for the region 32 having low contrast or low luminance.

  Returning to FIG. 11, the description of the procedure is continued. When there is an edge in the contrast AF area 40 (step Sg10-YES), the process of the area detection unit 142 proceeds to step Sg8. The CPU 190 executes a contrast scan based on the sampling frequency and the contrast value detected by the feature amount calculation unit 141 (step Sg8). When a plurality of labels are set in the contrast area, the CPU 190 prioritizes the height of the score (priority) over the height of the peak of the histogram and focuses on the area having a high score (priority) ( Step Sg12).

  FIG. 16 shows an example of the area detected by the area detector 142, the score calculated by the score calculator 143, and the contrast AF area set by the area detector 142. Here, the contrast AF area 43 is a contrast AF area set in the area 33 by the area detection unit 142. Also, as an example, the score of the region 33 is “20”, the score of the region 30 is “23”, the score of the region 31 is “15”, and the score of the region 32 is “14”. Further, it is assumed that a priority label is set in the area 30.

  In such a case, the CPU 190 may focus the area 30 in which the priority label is set as the main subject image area. Further, the CPU 190 may focus the area 30 having the highest score (priority) on the area 30 as the main subject image area. Further, the CPU 190 may continuously change the focused area in descending order of the score (priority) of the area.

  When the contrast cannot be obtained, the CPU 190 may set a contrast AF area circumscribing the label in addition to the contrast AF area set inside the label. In this case, the CPU 190 may use the contrast AF area set inside the label and the contrast AF area circumscribing the label to focus on the closest side.

  As described above, the image processing apparatus 140 calculates, from the image, the feature amount calculation unit 141 that detects the feature amount of the feature for each feature of the image, and the region that is included in the image and is determined for each feature amount. Based on the score, the area detector 142 for setting the label (identifier), the score calculator 143 for calculating the score for comparing the areas detected based on the feature quantities of different features, and the score A priority setting unit 144 that sets the priority of the area in the image.

  Thereby, the image processing apparatus 140 can predict the region of the main subject from the captured image. For example, the image processing apparatus 140 can identify a main subject image region and a non-main subject image region in a captured image. Further, the image processing apparatus 140 can improve the accuracy of AF, AE, AWB, tracking, scene determination, and color processing by detecting a region based on a plurality of features (for example, hue and saturation). it can. In addition, the image processing apparatus 140 can shorten the time lag (delay) until shooting by improving the tracking accuracy. Thereby, the image processing apparatus 140 can identify the region 30 having the highest score (priority) as the main subject image region.

  In addition, the image processing device 140 detects the sampling frequency at which contrast can be picked up by varying (expanding) the area of the contrast AF region and sweeping the sampling frequency, and at the sampling frequency optimum for detecting the contrast. , Detect the peak of contrast.

  As a result, even if the region has a low contrast, the image processing apparatus 140 increases the sensitivity for detecting at least one contrast of hue, luminance, RGB, and color difference (Cr, Cb), and executes a contrast scan. Therefore, the region in the captured image can be identified with high accuracy. For example, the image processing apparatus 140 can identify a main subject image region and a non-main subject image region in a captured image.

<Detection of areas based on unique hues>
The region detection unit 142 may detect a region based on a unique hue (for example, a hue that is discontinuous in the hue circle).

  FIG. 17 is a flowchart showing a processing procedure for detecting a region based on the hue. Steps Sd14-17 are processed in parallel with steps Sd3-8 of FIG. The feature amount calculation unit 141 creates a histogram of the hue h in the image after step Sd2 in FIG. 8 (step Sd14).

  The region detection unit 142 determines whether there is a peak in the histogram based on the histogram distribution. Further, the region detection unit 142 detects the continuity of hues in the hue histogram or hue circle, and determines whether there is a discontinuous hue in the hue histogram (step Sd15).

  FIG. 18 shows an example of a hue histogram. The region detection unit 142 detects a region showing a peak in the hue histogram and a region of a hue (discontinuous region) that is discontinuous in the hue histogram.

  Returning to FIG. 17, the description of the procedure is continued. The region detection unit 142 detects a region based on a hue that shows a peak in the hue histogram or a discontinuous hue (step Sd16). Furthermore, the area detection unit 142 labels the detected area (step Sd17).

  The region detection unit 142 sets a “single hue label” indicating that the region is detected based on the unique hue in the region detected based on the unique hue. Further, the score calculation unit 143 calculates the score of the region detected by the region detection unit 142 as “4” (step Sd18). Then, the processing of the area detection unit 142 proceeds to step Sd9 in FIG.

  As a result, the image processing apparatus 140 can detect a region based on a unique hue.

<Detection of regions based on unique brightness>
The region detection unit 142 may detect a region based on lightness that is unique (for example, lightness that is significantly different from the surroundings).

  FIG. 19 is a flowchart showing a processing procedure for detecting a region based on brightness. Here, steps Sd19 to 23 are processed in parallel with steps Sd2 to 8 of FIG.

  The feature amount calculation unit 141 creates a histogram of lightness L in the image after step Sd1 in FIG. 8 (step Sd19). The region detection unit 142 determines whether or not there is a peak in the histogram based on the histogram distribution (step Sd20).

  The region detection unit 142 detects a region based on the lightness indicating a peak in the lightness histogram or the lightness discontinuous due to the valley of the histogram (step Sd21). Then, the region detection unit 142 sets a “singular lightness label” in the region detected based on the lightness that is unique (step Sd22).

  Further, the region detection unit 142 sets a “lightness label” indicating that the region is detected based on the lightness in the region detected based on the lightness. Further, the score calculation unit 143 calculates the score of the area detected by the area detection unit 142 as “4” (step Sd23). Then, the processing of the area detection unit 142 proceeds to step Sd9 in FIG.

  Moreover, the area | region detection part 142 determines whether the difference of the lightness which shows a peak value, and the average value of lightness is more than a threshold value after step Sd20 (step Sd24). When the difference between the lightness indicating the peak value and the average value of lightness is not equal to or greater than the threshold value (step Sd24-NO), there is no difference in lightness, so the processing of the region detection unit 142 proceeds to step Sd21.

  On the other hand, when the difference between the lightness indicating the peak value and the average lightness value is equal to or greater than the threshold value (step Sd24-YES), since there is a difference in lightness, the area detection unit 142 captures the point light source. “Point light source flag” indicating that the image is captured (night scene is captured) is set to an image (region) (scene determination) (step Sd25).

  The region detection unit 142 detects a region based on the lightness indicating a peak in the lightness histogram or the lightness that becomes discontinuous. And the area | region detection part 142 labels the area | region detected based on the brightness (step Sd24). Then, the process of the region detection unit 142 proceeds to step Sd23.

  Thereby, the image processing apparatus 140 can detect the region based on the unique brightness.

<Region detection based on texture (pattern) detected by the angle detection operator>
The region detection unit 142 may detect a region based on a texture (pattern) detected by an angle detection operator (spatial filter).

  FIG. 20 is a flowchart showing a processing procedure for detecting a region based on the texture detected by the angle detection operator. Here, steps Sd26-31 are processed in parallel with steps Sd1-8 of FIG.

  The feature amount calculation unit 141 scans an image (for example, raster scan) by an angle detection operator, and calculates a feature amount of lightness L (luminance Y) or hue (color component RGB). Then, the feature amount calculation unit 141 divides the image into, for example, eight regions for each angle “22.5” degrees (step Sd26). Here, the angle detection operator is a spatial filter for detecting an edge in an image for each angle of the edge. FIG. 21 shows eight patterns of angle detection operators for detecting edges at an angle of “22.5” degrees.

  Returning to FIG. 20, the description of the procedure is continued. The feature amount calculation unit 141 creates a histogram indicating the feature amount (power value, ΣP) of the edge for each edge angle (step Sd27). Note that the feature quantity calculation unit 141 may create a histogram indicating the edge length for each edge angle.

  The region detection unit 142 determines whether or not there is a peak in the histogram based on the histogram distribution (step Sd28). FIG. 22 shows an example of a histogram indicating edge feature values (power values, ΣP) for each edge angle. Assume that the region detection unit 142 determines that there are peaks A and B of edge feature amounts in the histogram shown in FIG. Here, it is assumed that the feature amount (power value, ΣP) of an edge whose angle is “22.5” indicates the peak A. Further, it is assumed that the feature amount (power value, ΣP) of the edge whose angle is “135” degrees indicates the peak B.

  Returning to FIG. 20, the description of the procedure is continued. The area detection unit 142 creates a bitmap (power map) indicating the distribution (area) of edges at the angle for each angle at which the feature amount shows a peak (step Sd29). FIG. 23 shows an example of a region detected by the region detection unit 142 based on the texture (edge distribution). FIG. 23A shows an example of regions 1 to 5 where edges having an angle of “22.5” degrees are distributed. FIG. 23B shows an example of a region 6 in which edges having an angle of “135” degrees are distributed. For each angle at which the feature amount shows a peak, the region detection unit 142 creates a bitmap that indicates a region in which edges of the angle are distributed. Furthermore, the region detection unit 142 labels the region detected based on the texture. Also, the score calculation unit 143 calculates the score of the area detected by the area detection unit 142 based on the texture as “2” (step Sd30).

  In addition, the area detection unit 142 sets a “texture label” indicating that the area is detected based on the texture in the area detected based on the texture detected by the angle detection operator (step Sd31). Then, the processing of the area detection unit 142 proceeds to step Sd9 in FIG.

  Steps Sd32 and 33 are processed in parallel with steps Sd27-30. In the histogram (see FIG. 22), the region detection unit 142 creates a bitmap (power map) indicating a region in which the edge of the angle is distributed for each angle of the edge whose feature amount is equal to or greater than the threshold (step Sd32). ).

  Further, the region detection unit 142 thins the edge. Then, the area detection unit 142 labels the detected area based on an element (perspective element) indicating the depth of the image, that is, the spatial frequency, interval (span), or distribution of the thinned edge. For example, the region detection unit 142 may add a label to a region where the interval between edges is close. Also, the score calculation unit 143 calculates the score of the area detected by the area detection unit 142 by the angle detection operator as “1” (step Sd33). Then, the process of the area detection unit 142 proceeds to step Sd31.

  Thereby, the image processing apparatus 140 can detect the region based on the texture detected by the angle detection operator.

<Regarding Region Detection Based on Texture (Pattern) Detected by Discrete Cosine Transform (DCT) Coefficient>
The region detection unit 142 may detect a region based on a texture (pattern) detected by a discrete cosine transform (DCT) coefficient based on the JPEG standard.

  FIG. 24 is a flowchart showing a processing procedure in which the region detection unit 142 detects a region based on the DCT coefficient. Here, steps Sd34-42 are processed in parallel with steps Sd1-8 of FIG.

  The feature quantity calculation unit 141 executes DCT calculation processing on the image in units of DCT blocks (= 8 × 8 pixels) (step Sd34). Then, the feature amount calculation unit 141 quantizes the DCT coefficient based on a quantization table (for example, a quantization table compliant with the JPEG standard) (step Sd35).

  The feature amount calculation unit 141 calculates a representative value in the DCT block for each band predetermined for the DCT block. Here, the representative value in the DCT block is, for example, the square value of the DCT coefficient for each band or the absolute value of the DCT coefficient for each band. Then, the feature amount calculation unit 141 calculates a representative value in each DCT block for all DCT blocks included in the image (step Sd36).

  The representative value in the DCT block may be a value obtained by adding the square value of the DCT coefficient or the absolute value of the DCT coefficient over a plurality of bands.

  The region detection unit 142 creates a bitmap (power map) indicating the distribution of the DCT block for which the representative value is determined for each band predetermined for the DCT block (step Sd37). When creating a bitmap (power map), the region detection unit 142 may perform inverse discrete cosine transform for each DCT block.

  Then, the feature amount calculation unit 141 normalizes the representative value in the DCT block to the “0 to 255” range (step Sd38). Further, the feature amount calculation unit 141 sets a representative value equal to or less than a predetermined threshold (for example, a value “30”) to a value “0” (step Sd39).

  Further, the region detection unit 142 determines whether the representative value ranges (for example, three ranges of values “31 to 100”, values “101 to 170”, and values “171 to 255”) from each bitmap. A plurality of bitmaps are created (step Sd40). Then, the region detection unit 142 detects a region from a plurality of bitmaps, and labels the detected region (step Sd41).

  The region detection unit 142 determines whether labeling processing has been executed for all the created bitmaps (step Sd42). When there is a bitmap that does not label the region (step Sd42-NO), the processing of the region detection unit 142 returns to step Sd40.

  On the other hand, when the regions are labeled for all the bitmaps (step Sd42-YES), the region detection unit 142, when all the created bitmaps are overlapped, the portion where the regions overlap (positional product set). Calculate the area. And when the area of the part which area | regions overlap is more than a predetermined area, the area | region detection part 142 may combine those area | regions (step Sd43).

  Then, the score calculation unit 143 calculates the score of the area detected by the area detection unit 142 based on the DCT coefficient as “2” (step Sd44).

  Furthermore, the region detection unit 142 sets a “texture label” indicating that the region is detected based on the texture in the region detected based on the texture detected based on the DCT coefficient (step Sd45). Then, the processing of the area detection unit 142 proceeds to step Sd9 in FIG.

  FIG. 25 is a diagram for explaining processing in which the region detection unit 142 detects a region based on the DCT coefficient. Here, a plurality of DCT blocks 10a are arranged in one image. Each DCT block 10a has a predetermined band 11-n (n is an integer of 1 to 5) as an example. The feature amount calculation unit 141 calculates the representative value in the DCT block 10a for each band predetermined for the DCT block (see step Sd36 in FIG. 24). And the area | region detection part 142 performs inverse discrete cosine transformation for every DCT block 10a, and produces the DCT block 10b.

  The area detection unit 142 creates a bitmap 12-n indicating the distribution in the image of the DCT block 10b-n (predetermined band 11-n) for each predetermined band 11-n in the DCT block 10a. (See step Sd37 in FIG. 24). For example, the region detection unit 142 creates the bitmap 12-1 indicating the distribution in the image of the predetermined band 11-1 by arranging the DCT blocks 10b-1.

  Further, the region detection unit 142 creates a plurality of bitmaps from each bitmap for each representative value range (see step Sd40 in FIG. 24). Here, as an example, it is assumed that the bitmap 13 includes a region 16 in which the “representative value in the DCT block” of the predetermined band 11-1 is a value “31 to 100”. The bitmap 14 includes regions 17 and 18 in which “representative values in the DCT block” of the predetermined band 11-1 are values “101 to 170”. Further, it is assumed that the bitmap 15 includes regions 19 and 20 in which “representative values in the DCT block” of the predetermined band 11-1 are values “171 to 255”.

  Then, the region detection unit 142 creates a bitmap 21 by superimposing all the created bitmaps, and detects a portion where the regions overlap each other. In FIG. 25, as an example, a region 22 where the regions 18 and 20 overlap and a region 23 where the regions 16 and 17 overlap are detected in the bitmap 21.

  Further, the region detection unit 142 calculates the area of the portion where the regions 22 and 23 overlap (positional product set), and if the area of the overlapping portion is equal to or larger than a predetermined area, the region detection unit 142 determines that the regions 22 and 23 are the same. They may be combined (see step Sd43 in FIG. 24).

  The score calculation unit 143 may add the score to the score of the area detected by the area detection unit 142. FIG. 26 is a flowchart showing a process in which the score calculation unit 143 adds the score to the score of the region detected by the region detection unit 142 based on the DCT coefficient in the high frequency band. Here, steps Sd46 to Sd56 are processed in parallel with steps Sd1 to Sd8 of FIG.

  The feature amount calculation unit 141 executes DCT calculation processing on the image in units of DCT blocks (= 8 × 8 pixels) (step Sd46). And the feature-value calculation part 141 quantizes a DCT coefficient based on a quantization table (step Sd47).

  The feature amount calculation unit 141 selects a high frequency band from the DCT block (step Sd48). And the feature-value calculation part 141 calculates the representative value in the DCT block based on the DCT coefficient of the selected high frequency band. Here, the representative value in the DCT block is, for example, the square value of the DCT coefficient in the selected high frequency band or the absolute value of the DCT coefficient in the selected high frequency band. And the feature-value calculation part 141 calculates the representative value in a DCT block about all the DCT blocks contained in an image (step Sd49).

  The region detection unit 142 creates a bitmap (power map) indicating the distribution of DCT blocks (the distribution of selected high-frequency bands) for which representative values are determined for each band predetermined in the DCT block. Then, the feature amount calculation unit 141 normalizes the representative value in the DCT block to the “0 to 255” range (step Sd50). Further, the feature amount calculation unit 141 sets the representative value equal to or less than a predetermined threshold (for example, the value “30”) to the value “0” (step Sd51).

  Further, the region detection unit 142 determines whether the representative value ranges (for example, three ranges of values “31 to 100”, values “101 to 170”, and values “171 to 255”) from each bitmap. A plurality of bitmaps are created (step Sd52). Then, the region detection unit 142 detects a region from a plurality of bitmaps, and labels the detected region (step Sd53).

  The area detection unit 142 determines whether or not the labeling process has been executed for all created bitmaps (step Sd54). When there is a bitmap in which the region is not labeled (step Sd54-NO), the processing of the region detection unit 142 returns to step Sd52 in order to execute the labeling processing for all the created bitmaps.

  On the other hand, when the regions are labeled for all the bitmaps (step Sd54-YES), the region detection unit 142 determines in advance that the regions detected based on the high frequency band when all the created bitmaps are overlaid. The area of the portion (product set) that overlaps the detected region based on the other frequency band thus obtained is calculated. If the area of the overlapping area is equal to or larger than the predetermined area, the score calculation unit 143 adds “1” to the rating of the area including the overlapping area (step Sd55).

  The region detection unit 142 sets a “texture label” indicating a region detected based on the texture detected by the DCT coefficient in the region detected based on the texture detected by the DCT coefficient in the high frequency band (step Sd56). ). Then, the processing of the area detection unit 142 proceeds to step Sd9 in FIG.

  As described above, the image processing apparatus 140 can detect a region based on a texture (pattern) detected by a discrete cosine transform (DCT) coefficient.

<Regarding detection of regions based on feature amounts of edges extending vertically, horizontally, or diagonally>
The area detection unit 142 detects the captured pole (pillar), bar (bar), perspective (depth), and building area based on the feature amount of the edge extending vertically, horizontally, or diagonally. May be.

  FIG. 27 is a flowchart showing a processing procedure in which the region detection unit 142 detects a region based on the feature amount of the edge. Here, steps Sd27-62 are processed in parallel with steps Sd1-8 of FIG.

  The feature amount calculation unit 141 calculates the feature amount (intensity) of the edge (step Sd57). Then, the region detection unit 142 detects an edge whose thickness is equal to or greater than the threshold and an edge whose length is equal to or greater than the threshold (step Sd58). Furthermore, the region detection unit 142 sets an edge region whose thickness is equal to or greater than a predetermined threshold and whose length is equal to or greater than the predetermined threshold as a target to be labeled (step Sd59).

  The feature amount calculation unit 141 calculates the distance from the center of the image (view angle center) to the edge. Then, if the distance from the center of the image to the edge is equal to or greater than the threshold, the area detection unit 142 determines that the edge is not the main subject image area and excludes the edge area from the labeling target (step Sd60). ). Furthermore, the area detection unit 142 executes a labeling process based on the pixel values (lightness, hue, etc.) of the edge area that is the target of labeling. Then, the score calculation unit 143 calculates the score of the region detected by the region detection unit 142 based on the edge as “1” (step Sd61).

  The area detection unit 142 sets an “edge label” indicating the area detected based on the edge in the area detected based on the edge (step Sd62). Then, the processing of the area detection unit 142 proceeds to step Sd9 in FIG.

  Steps Sd63 and 64 are processed in parallel with steps Sd60 and 61. The region detection unit 142 detects an edge whose angle with respect to the horizontal direction of the image is in the range of “0 to 30” degrees and an edge whose angle with respect to the horizontal direction of the image is in the range of “150 to 180” degrees (step) Sd63).

  Furthermore, when the edge toward the center of the image is distributed discretely in the image, the region detection unit 142 turns on a “perspective flag” indicating that the image has a depth, and stores it as region information. The data is stored in the unit 160 (step Sd64).

  Then, the region detection unit 142 sets a “pole label” in a region in which edges having an angle in the range of “0 to 30” degrees with respect to the horizontal direction of the image are distributed. Further, the region detection unit 142 sets “bar label” in a region where edges having an angle in the range of “150 to 180” with respect to the horizontal direction of the image are distributed. Furthermore, the region detection unit 142 sets a “perspective label” to an region that is an edge toward the center of the image and in which an edge that is discretely distributed in the image is distributed (step Sd62). Since a label for which a perspective is set has a depth, a plurality of different focal positions exist within the same label. For this reason, control is necessary to focus on a label for which a perspective is set.

  For example, the area detection unit 142 may set a “building label” indicating that the area where the edge is distributed is an area where the building is imaged in an area where the edge is distributed.

  Thereby, the image processing apparatus 140 can detect a region indicating a pillar, a bar, a depth, or a building based on a feature amount of an edge extending vertically, horizontally, or obliquely.

<Exposure (AE) processing based on label or priority (score)>
The image processing apparatus 140 may execute the exposure process so that the exposure amount in the area where the priority label (high priority and score) is set is optimized.

  FIG. 28 is a flowchart showing the procedure of exposure (AE) processing based on the label or priority (score). This process is executed in, for example, step Sa9 in FIG. 3 (step Sd9 in FIG. 8) or step Sa15 in FIG.

  The feature amount calculation unit 141 creates a histogram for a region with high priority (for example, the top three scores) (step Sh1). Note that the feature amount calculation unit 141 may not create a histogram for a region where the “point light source flag” is on.

  And the feature-value calculation part 141 adds up the feature-value of an area | region with high priority (a high score), and produces a histogram (step Sh2). Further, the feature amount calculation unit 141 calculates an exposure amount at which the standard deviation value σ of the histogram is 30% of the full range of the histogram (Step Sh3).

  Based on the exposure amount calculated by the feature amount calculation unit 141, the CPU 190 sets the exposure amount to the imaging unit 110 so that the pixel value (for example, brightness) of the region in which the background is captured (region indicating the background) is not saturated. Adjust (step Sh4). In addition, the feature amount calculation unit 141 calculates parameters for region-specific (tone correction) processing such as Retinex so that the standard deviation value σ of the histogram is 30% of the full range of the histogram, thereby reducing the exposure shortage. Is compensated (step Sh5).

  Based on the “region priority” set by the priority setting unit 144, the CPU 190 controls the setting of the tone curve (tone curve) in the image processing device 140 so as to emphasize the contrast of the region with a high priority. To do. Further, the CPU 190 sets the tone curve (gradation curve) in the image processing apparatus 140 so as to emphasize the contrast of the area for which the priority label is set based on the “label” set for the area by the area detection unit 142. You may control (step Sh6).

  Steps Sh7-9 are processed in parallel with steps Sh2-6. The feature amount calculation unit 141 calculates the maximum and minimum brightness (luminance) in the histogram for the high priority region (part of the image) (may be 5% inside each end point of the histogram) (step Sh7). Further, the feature amount calculation unit 141 calculates the maximum and minimum brightness (luminance) in the histogram for the entire image (may be 5% inside each end point of the histogram) (step Sh8).

  When the difference between the maximum brightness in the histogram for the high-priority area (part of the image) and the minimum brightness in the histogram for the entire image area is equal to or greater than a predetermined threshold, The “high dynamic range flag” indicating that the brightness dynamic range is wide is turned on, and the high dynamic range flag is set for the image (region) as region information.

  Similarly, the area detection unit 142 determines that the difference between the minimum brightness in the histogram for the high priority area (part of the image) and the maximum brightness in the histogram for the entire image is equal to or greater than a predetermined threshold. In some cases, the “high dynamic range flag” indicating that the brightness dynamic range is wide is turned on, and the high dynamic range flag is set in the image (region) as region information (step Sh9).

  Thereby, the image processing apparatus 140 can perform an exposure (AE) process based on the label or the priority (score).

<About white balance adjustment (AWB) processing based on label or priority (score)>
The image processing apparatus 140 may execute the white balance adjustment process so that the white balance of the area where the priority label (high priority and score) is set is optimal.

  FIG. 29 is a flowchart showing the procedure of white balance adjustment (AWB) processing based on the label or priority (score). This process is executed in, for example, step Sa9 in FIG. 3 (step Sd9 in FIG. 8) or step Sa15 in FIG.

  The feature amount calculation unit 141 calculates a white point from the image. Here, the feature amount calculation unit 141 may detect a white spot from a region where the background is imaged (region indicating the background) (step Si1). The area where the background is imaged (area indicating the background) will be described later with reference to FIGS.

  Then, the feature amount calculation unit 141 determines whether or not there is a white spot in the image (step Si2). If the image has a white spot (step Si2-YES), the CPU 190 controls the white balance adjustment (AWB) process based on the white spot area (step Si3).

  On the other hand, when there is no white point in the image (step Si2-NO), the feature amount calculation unit 141 performs gamut mapping for the hue in the main subject image region based on the priority label or the region priority (score). (Color gamut coordinate conversion) is executed. And the feature-value calculation part 141 estimates the kind (for example, sunlight, a fluorescent lamp, etc.) of the light source of the imaged light by gamut mapping (step Si4).

  And the feature-value calculation part 141 determines whether the kind of light source was detected (step Si5). When the type of light source is detected (step Si5-YES), the CPU 190 controls the white balance adjustment process based on the type of light source detected by the feature amount calculation unit 141 (step Si3).

  On the other hand, when the type of light source is not detected (step Si5-NO), the feature amount calculation unit 141 selects a scene based on the label set for the region, the flag set for the region (image), and the distribution of the region. Determine and estimate the type of light source. For example, the feature amount calculation unit 141 has a “point light source flag” turned on and is set as a region (image), and a region (point light source) having significantly different brightness compared to the surroundings is widely distributed in the image. In this case, it may be determined that the scene is a night view (step Si6). Then, the CPU 190 controls the white balance adjustment processing of the image processing device 140 based on the type of light source estimated by the feature amount calculation unit 141 (step Si3).

  Thereby, the image processing apparatus 140 can perform white balance adjustment (AWB) based on the label or the priority (score).

<Regarding the processing to reproduce an image based on the label or priority (score)>
The image processing apparatus 140 may classify the detected area based on the label or priority (score), and may reproduce the image by a visually effective method according to the classification.

  FIG. 30 is a flowchart showing a basic image processing procedure including image reproduction display by the image processing apparatus 140. Here, steps S1 to S14 in FIG. 30 are the same steps as steps S1 to S14 in FIG. The image processing device 140 reproduces the image associated with the region information with an effective effect based on the region information, and displays the image on the display unit 150 (step S15).

  FIG. 31 is a flowchart showing the procedure of reproduction display based on the label or priority (score). This reproduction display process is executed in step S15 of FIG. First, the region detection unit 142 acquires region information from the storage unit 160 or the storage medium 200 (step Sf1).

  When the “motion amount” stored as the area information is equal to or greater than the threshold value, that is, when the image is a moving image, the image processing apparatus 140 reproduces the image as a moving image after step Sf1 ( Step Sf2). Then, the image processing apparatus 140 performs zoom reproduction centered on the area where the priority label is set, and causes the display unit 150 to display (Step Sf3). Further, the image processing apparatus 140 performs panning display centering on a region with one lower priority, performs zoom playback, and causes the display unit 150 to display (step Sf4).

  In addition, when the “motion amount” of the motion vector stored as the information of the area where the motion label is set is equal to or greater than the threshold value, when the moving speed of the area where the motion label is set is fast, the image processing device 140 After step Sf1, the image is played back slowly and displayed on the display unit 150 (step Sf5).

  In addition, when the amount of motion is bisected, the direction of motion is constant, and the amount of motion is greater than or equal to the threshold is dispersed in the image (see FIG. 7), that is, the foreground (for example, snowfall) If the image has been picked up and focused on the priority label, the image processing apparatus 140 reproduces the image by focusing on the foreground of the area (front pin reproduction) after step Sf1, and displays the image on the display unit 150. (Step Sf6). Then, the image processing apparatus 140 reproduces the image while changing the focus so as to gradually focus on the region, and displays the image on the display unit 150 (step Sf7). The image processing apparatus 140 changes the image synthesis ratio (step Sf8).

  Further, when the area for which “normal label” is set is reproduced and displayed (see step Sc20 in FIG. 7), that is, when the area other than the foreground such as snowfall is reproduced and displayed, the image processing apparatus 140 follows step Sf1. The image is reproduced (step Sf9). Then, the image processing apparatus 140 performs panning display of the areas in order of the set priority, performs zoom reproduction, and causes the display unit 150 to display (Step Sf10). Further, when the area for which “normal label” is set is an area detected based on the texture, the image processing apparatus 140 performs zoom reproduction with the edge strengthened and displays it on the display unit 150 (step Sf11).

  If the hue, high saturation, or high-score hue label is set in the region, the image processing apparatus 140 continuously sets the saturation of the region in which the hue label is set after step Sf1. It is reproduced while being raised and displayed on the display unit 150 (step Sf12). Further, when there is an opposite hue region in the hue circle, the image processing device 140 increases the saturation of the opposite hue region, reproduces the image, and displays the image on the display unit 150 (step Sf13).

  If the image has a wide gradation range (dynamic range), the image processing apparatus 140 combines the gradation range of the area where the priority label is set with the gradation area of the entire image after step Sf1. The images are continuously reproduced and displayed on the display unit 150. For example, the image processing device 140 changes the coefficient in the Retinex (gradation correction) process to reproduce the image by continuously increasing the gradation of the low-lightness area while suppressing overexposure and blackout, It is displayed on the display unit 150 (step Sf14). Further, the image processing apparatus 140 alternately displays images having different gradation ranges, continuously reproduces them with an afterimage effect, and displays them on the display unit 150 (step Sf15).

  FIG. 32 is a flowchart showing a playback display procedure based on the label or priority (score). This reproduction display process is executed in step S15 of FIG. Also, step Sf1 in FIG. 32 is the same as step Sf1 in FIG.

  When the perspective flag is set to ON, that is, when the image has a depth (perspective), the image processing apparatus 140 has a subject image with a short subject distance (captured on the closest side) after step Sf1. The image is reproduced while being focused on the area, and displayed on the display unit 150 (step Sf16). Then, the image processing device 140 reproduces the image while changing the focal length from the nearest to infinity, and displays the image on the display unit 150. In addition, the image processing apparatus 140 performs zoom reproduction of the area where the priority label is set, and displays the area on the display unit 150 (step Sf17).

  If the specific brightness label (see step Sd22 in FIG. 19) is set, the image processing apparatus 140 continuously changes the tone curve (gradation curve) after step Sf1, while The area where the specific brightness label is set is zoomed and reproduced and displayed on the display unit 150.

  For example, when the area where the specific brightness label is set is bright, the image processing apparatus 140 may execute the process of enhancing the contrast while suppressing the whiteout of the area. In addition, when the area where the specific brightness label is set is dark, the image processing apparatus 140 may set a high gamma table with increased sensitivity in the area (step Sf18).

  Further, when the point light source flag is set to ON, that is, for example, when the image is a point light source, an evening scene, or a night scene, the image processing apparatus 140 reduces the brightness of the image after step Sf1. To start playback. Next, the image processing device 140 increases the brightness, reproduces the image, and displays the image on the display unit 150 (step Sf19). Thereby, the image processing apparatus 140 can emphasize a point light source.

  If the image includes an edge region extending vertically, horizontally, or obliquely, the image processing apparatus 140 reproduces the edge region by zoom reproduction and panning display after step Sf1, and displays the display unit 150. To display. Then, the image processing apparatus 140 changes the lightness contrast according to the lightness of the region (step Sf20).

  As a result, the image processing apparatus 140 can reproduce the image associated with the area information with an effective presentation.

[Second Embodiment]
A second embodiment of the present invention will be described in detail with reference to the drawings. The second embodiment is different from the first embodiment in that the region detection unit 142 combines or divides regions detected based on feature amounts of different features. Only the differences from the first embodiment will be described below.

  The region detection unit 142 detects the first region detected based on the feature amount of the first feature (for example, saturation) and the second region detected based on the feature amount of the second feature (for example, texture). The area of the overlapping part with the region is calculated, and if the area of the overlapping part is greater than or equal to a predetermined area, the second region is combined with the first region.

  In addition, the region detection unit 142 determines the subject distance corresponding to the third region detected based on the feature amount of the third feature (for example, saturation) and the feature amount of the fourth feature (for example, texture). It is determined whether or not the subject distance corresponding to the fourth area detected based on the distance is the same. When the subject distance corresponding to the third region and the subject distance corresponding to the fourth region are the same distance, the region detection unit 142 combines the fourth region with the third region.

  Moreover, the area | region detection part 142 adds up the score for every area | region before combining, and uses the added score as the score of the area | region after combining, when combining an area | region. In addition, the region detection unit 142 divides the region into a plurality of small regions when the plurality of small regions included in the region correspond to different subject distances.

  Further, the region detection unit 142 combines or divides regions based on the depth map. Details of the depth map will be described later with reference to FIG.

  FIG. 33 is a flowchart showing a procedure of processing for combining regions and processing for dividing regions. The region detection unit 142 detects the first region detected based on the feature amount of the first feature (for example, saturation) and the second region detected based on the feature amount of the second feature (for example, texture). The area of the portion overlapping with the region is calculated. And the area | region detection part 142 couple | bonds a 2nd area | region with a 1st area | region, when the area of the overlapping part is more than a predetermined area (it makes the same label). Also, the score calculation unit 143 adds the scores of the combined regions, and updates the “region information” of the combined regions (step Sk1).

  FIG. 34 shows a first region 50 detected based on the feature amount of the first feature, and a second region 51 detected based on the feature amount of the second feature. In FIG. 34, the region detection unit 142 calculates the area of the overlapping portion (OR) between the first region 50 and the second region 51. Then, the area detection unit 142 couples the second area 51 to the first area 50 when the area of the overlapping portion is equal to or larger than a predetermined area.

  Returning to FIG. 33, the description of the procedure is continued. The area detection unit 142 detects an area where the subject distance is the same distance. Here, the region detection unit 142 may acquire a depth map from the storage unit 160 and detect a region where the subject distance is the same distance based on the depth map. Then, when regions having the same subject distance are close to each other, the region detection unit 142 combines the regions and sets a label on the combined region. Further, the score calculation unit 143 sets the square root of the sum of squares of the scores of the adjacent areas as a new score of the combined area, and “area information” of the combined area (see step Se12 in FIG. 10). Remember as.

  Further, the region detection unit 142 sets a label indicating the same group in the region having the same subject distance even when the regions having the same subject distance are not close to each other (grouping). For example, the area detection unit 142 labels the areas indicating the same group A, such as “group A-1” and “group A-2”, in areas where the subject distances are the same distance but are not close to each other. May be set. In addition, the region detection unit 142 gives priority to the region set with the “texture label” among the regions set with the label indicating the same group, and the region with the “texture label” set, Focus adjustment (AF) may be executed (step Sk2).

  Depth map will be described. FIG. 35 shows an example of the depth map. Here, the depth map is a map showing the distribution of the subject distance in the image. The depth map may be a map showing the distribution of the defocus amount of the subject image in the image.

  In the depth map 60, evaluation areas 61 to 71 for combining or dividing the areas detected by the area detection unit 142 are set as an example. Each evaluation area has a subject distance corresponding to the evaluation area. Here, the subject distance corresponding to the evaluation region is the subject distance of the subject imaged in the evaluation region.

  In the following, as an example, the evaluation areas 67 to 71 correspond to the subject distance “1 (closest)” according to the subject distance of the subject imaged in those evaluation areas, and the evaluation areas 65 and 66 are the same. The subject distance “2” corresponds to the evaluation area 64, the subject distance “3” corresponds to the evaluation area 64, and the evaluation area 61 to 63 similarly corresponds to the subject distance “4 (infinite)”. To do.

  In FIG. 35, the evaluation area whose corresponding subject distance is “1 (closest)” is shown in black. In addition, the evaluation area whose corresponding subject distance is “4 (infinite)” is shown in white. In addition, an evaluation area in which the corresponding subject distance is other than that is indicated by an intermediate color between white and black corresponding to the subject distance (grayed out).

  Here, the feature amount calculation unit 141 is based on, for example, the gradient of the color component amount (hue feature amount) of the edge of the region detected for each hue by the region detection unit 142 and the ratio or difference between the color component amounts. The subject distance may be detected. Here, the gradient of the color component amount refers to the position of the line indicating the color component amount in a diagram (graph) showing the position on the axis (for example, the horizontal axis) in the image and the color component amount that changes according to the position. It is a gradient. And the feature-value calculation part 141 may produce the depth map 60 based on the to-be-photographed object distance corresponding to the evaluation areas 61-71 calculated in this way.

  The feature amount calculation unit 141 may detect the subject distance based on, for example, the standard deviation value of the line spread function (LSF) of the edge of the region detected for each hue by the region detection unit 142. Here, the feature amount calculation unit 141 may detect the subject distance based on the fact that the standard deviation value of the edge line spread function is larger as the edge defocus amount of the detected region is larger. Then, the feature amount calculation unit 141 may create the depth map 60 based on the subject distance corresponding to the evaluation areas 61 to 71 detected in this way.

  The death map 60 is, for example, a subject distance detected by the phase difference detection method, a subject distance detected by the contrast AF method, or a subject distance detected by a time when a light ray or a sound wave reciprocates between the imaging device 100 and the subject. May be created based on The depth map 60 is not limited to the one created as described above.

  FIG. 36 shows regions 80 and 81 as diagrams for explaining the region combining processing and grouping processing detected by the region detecting unit 142 in step Sk2 of FIG. The regions 80 and 81 may be regions detected by the region detection unit 142 based on feature amounts of the same feature, or regions detected by the region detection unit 142 based on feature amounts of different features. Good.

  Here, as an example, it is assumed that the position of the region 80 in the image corresponds to the position of the evaluation region 67 in the depth map 60 (see FIG. 35). Further, it is assumed that the position of the area 81 in the image corresponds to the position of the evaluation area 69 in the depth map 60.

  In this case, since the same subject distance “1 (closest)” corresponds to the evaluation regions 67 and 69 (see FIG. 35), the region detection unit 142 determines that the regions 80 and 81 have the same subject distance. (See FIG. 36) is detected. Since the regions 80 and 81 are not close to each other, the region detection unit 142 sets labels indicating the same group in the regions 80 and 81 (grouping).

  As an example, it is assumed that the position of the area 80 in the image corresponds to the position of the evaluation area 67 in the depth map 60. Further, it is assumed that the position of the area 81 in the image corresponds to the position of the evaluation area 68 in the depth map 60.

  In this case, since the same subject distance “1 (closest)” corresponds to the evaluation regions 67 and 68 (see FIG. 35), the region detection unit 142 determines that the regions 80 and 81 have the same subject distance. (See FIG. 36) is detected. Since the regions 80 and 81 are close to each other, the region detection unit 142 combines the regions 80 and 81 and sets a label in the combined region.

  Returning to FIG. 33, the description of the procedure is continued. When a plurality of small regions included in the region detected by the region detection unit 142 correspond to different subject distances, the region detection unit 142 divides the region into small regions for each subject distance based on the depth map 60. (Split processing). Further, the score calculation unit 143 calculates the score of the small area divided by the area detection unit 142 (step Sk3).

  FIG. 37 shows a region 90 and small regions 91 to 93 included in the region 90 as a diagram for explaining the division processing of the region detected by the region detection unit 142 in step Sk3 of FIG. ing.

  Here, as an example, it is assumed that the position of the small area 91 in the image corresponds to the position of the evaluation area 64 in the depth map 60 (see FIG. 35). As an example, it is assumed that the position of the small area 92 in the image corresponds to the position of the evaluation area 69 in the depth map 60. As an example, it is assumed that the position of the small area 93 in the image corresponds to the position of the evaluation area 66 in the depth map 60.

  In this case, since different subject distances correspond to the evaluation regions 64, 69, and 66 (see FIG. 35), the region detection unit 142 performs sub-region 91 included in the region 91 for each subject distance. ˜93 (see FIG. 37) are detected. Since the small areas 91 to 93 correspond to different subject distances, the area detecting unit 142 divides the area 90 into small areas for each subject distance (evaluation areas 64, 69, and 66) based on the depth map 60. Divide into 91-93.

  Returning to FIG. 33, the description of the procedure is continued. The score calculation unit 143 adds points to the score of the area according to the subject distance based on the depth map. Here, the score calculation unit 143 adds a high score to the near side area (area indicating the foreground). On the other hand, the score calculation unit 143 adds a low rating to the infinite region (region indicating the background) (step Sk4).

  The score calculation unit 143 may add points to the score of the area according to the difference (relative distance) between the subject distance corresponding to the first evaluation area and the subject distance corresponding to the second evaluation area. .

  The area detection unit 142 detects an area indicating the background based on the projection of the area. And the score calculation part 143 deducts the score of the area | region which shows a background. Furthermore, the region detection unit 142 excludes the region indicating the background from the targets for which priority labels are set (step Sk5). Similarly, the region detection unit 142 may exclude a region indicating the foreground (for example, snowfall) from the targets for which priority labels are set.

  In step Sk5, a process in which the area detection unit 142 detects an area indicating the background based on the projection of the area will be described. FIG. 38 is a flowchart showing a procedure of processing in which the region detection unit 142 detects a region indicating the background based on the projection of the region.

  The region detection unit 142 divides the region in the image into a first group and a second group according to the size of the area. Then, the “projection on the X axis (X projection)” of the first group area includes the X projection of the second group area, and the “projection on the Y axis (Y projection)” of the first group area. However, when the Y projection of the second group region is included, the region detection unit 142 detects the first group region as the region in which the background is imaged (the region indicating the background), and sets the first group region as the first group region. “Background label” is set (step Sn1).

  FIG. 39 is a diagram for explaining the process of detecting the region indicating the background in step Sn1 of FIG. Here, it is assumed that the X axis is set in the vertical direction of the image and the Y axis is set in the horizontal direction of the image. Furthermore, it is assumed that the region detection unit 142 divides the region in the image into a first group region 95 and a second group region 94 according to the size of the area.

  Then, the X projection 96x of the first group region 95 includes the X projection 97x of the second group region 94, and the Y projection 96y of the first group region 95 is Y of the second group region 94. When the projection 97y is included, the area detection unit 142 sets a “background label” indicating that the area 95 indicates the background in the area 95 of the first group.

  Returning to FIG. 38, the description of the processing procedure for detecting the area indicating the background will be continued. In step Sn1, even when the “background label” is not set in the first group region 95, the area of the first group region 95 is larger than a predetermined area, and from the second group region 94. When the outward normal line takes an angle “180” degrees or more to the first group region 95, the region detection unit 142 sets a “background label” in the first group region 95 (step Sn2).

  FIG. 40 is a diagram for explaining the process of detecting the region indicating the background in step Sn2 of FIG. In FIG. 40, unlike FIG. 39, the Y projection 96y of the first group area 95 does not include the Y projection 97y of the second group area 94. Then, the area of the first group region 95 is larger than a predetermined area, and the normals 98 and 99 outward from the second group region 94 have an angle of 180 degrees or more with respect to the first group region 95. In such a case, the region detection unit 142 sets a “background label” in the region 95 of the first group.

  Returning to FIG. 33, the description of the procedure is continued. The image may include an area where the subject distance is unknown. Then, the region detection unit 142 determines whether or not a high score (for example, within the top three) is assigned to a region where the subject distance is unknown (step Sk6).

  When a high score is not given to an area where the subject distance is unknown (step Sk6-NO), the area detection unit 142 sets a “priority label” in the detected area according to the score. Here, the region detection unit 142 may set “priority labels” in a plurality of regions (step Sk7).

  On the other hand, when a high score is assigned to an area where the subject distance is unknown (step Sk6-YES), the area detection unit 142 sets a “priority label” to the area where the high score is assigned. This is because the subject distance may be unknown due to the low contrast despite the main subject image area.

  Also, the score calculation unit 143 adds the highest score among the scores assigned to the region on the closest side (for example, the subject distance “1 (closest)”) to the score of the region set with the “priority label”. And the process of the score calculation part 143 progresses to step Sk7 (step Sk8).

  Summarizing the above-described process for setting a label in a region and the process for calculating a score, the outline of these processes is as follows.

  First, the image processing apparatus 140 detects an area by labeling for each feature amount, and executes a labeling process and a score calculation process for the detected area. The area detection unit 142 detects an area determined for each feature amount from the image. And the area | region detection part 142 sets a motion label to the area | region with a motion, for example. Further, for example, the score calculation unit 143 sets a score for a region according to the distance from the center of the image (center of view angle) to the center of gravity of the region, the moment of inertia, and the area of the region (number of pixels).

  Next, the image processing apparatus 140 adds a score to the region based on the feature amount. For example, the score calculation unit 143 may include a region having a characteristic amount that is unique (for example, a hue that is discontinuous in the hue circle), a region having high saturation, a region having a significantly different hue (hue) compared to the surroundings, and Detected based on areas detected based on (specific hue), areas with significantly different brightness compared to surroundings, areas detected based on texture detected by an angle detection operator or DCT, and edges extending vertically or horizontally A score will be added to the score of the selected area.

  Next, the image processing apparatus 140 adds points to the score of the area based on the subject distance (depth, depth). For example, the score calculation unit 143 adds points to the score of the region according to the difference in the subject distance (relative distance).

  Furthermore, the image processing apparatus 140 combines, divides, or groups areas and sets a label for the area. For example, when the areas having the same subject distance are close to each other, the area detection unit 142 combines the areas and sets a label on the combined area. For example, the region detection unit 142 divides the region into small regions based on the depth map 60. Further, for example, the area detection unit 142 sets a label indicating the same group in an area where the subject distance is the same distance.

  Next, the image processing apparatus 140 adds points to the score of the combined, divided, or grouped area. For example, the score calculation unit 143 adds points to the score of the combined area. For example, the score calculation unit 143 deducts the score of the area indicating the foreground or background. Further, for example, the region detection unit 142 sets “priority label” in a region where the subject distance is unknown and a high score is given. And the score calculation part 143 adds the score attached to the area | region of the near side to the score of the area | region which set the "priority label".

  Next, the image processing apparatus 140 sets a “priority label”. For example, the area detection unit 142 sets a “priority label” in the detected area according to the score. Here, the area detection unit 142 may set “priority labels” in the order in which the scores are higher in the plurality of areas.

  As described above, the image processing apparatus 140 overlaps the first region detected based on the feature amount of the first feature and the second region detected based on the feature amount of the second feature. When the area of the overlapping portion is equal to or larger than a predetermined area, the second region is combined with the first region. As a result, the image processing apparatus 140 can identify an area in which the detected small areas are collected as a main subject image area.

  The image processing apparatus 140 also detects the distance (subject distance) corresponding to the third region detected based on the feature amount of the third feature and the fourth region detected based on the feature amount of the fourth feature. If the distance corresponding to is the same distance, the fourth area is combined with the third area. As a result, the image processing apparatus 140 can identify an area obtained by collecting the detected areas as a main subject image area.

  In addition, when combining the areas, the image processing apparatus 140 adds the scores for the respective areas before combining, and uses the combined score as the score of the combined area. Accordingly, the image processing apparatus 140 can identify the main subject image area based on the score of the combined area.

  The image processing apparatus 140 divides the region into a plurality of small regions when the plurality of small regions constituting the region correspond to different distances (subject distances). Thus, the image processing apparatus 140 can identify the main subject image area based on the plurality of small areas included in the detected area.

  Further, the image processing apparatus 140 combines or divides regions based on a depth map indicating a distribution of distances (subject distances) in the image. As a result, the image processing apparatus 140 can identify the main subject image area based on the acquired or created depth map.

[Third Embodiment]
A third embodiment of the present invention will be described in detail with reference to the drawings. In the third embodiment, when the focus adjustment (AF) area is fixed at the center of the angle of view (the AF area is at the center of the angle of view), even if the area of the subject image has low contrast, The image processing apparatus 140 is different from the first and second embodiments in that a main subject image area is identified and contrast is ensured. In the third embodiment, a method of increasing the contrast of the AF area is assumed on the assumption that there is a subject in the AF area set as a fixed point in the image. Here, the CPU 190 sets an area including the inside of the label or the edge of the label as a contrast AF area, and searches for an in-focus position by contrast scanning. Only the differences from the first and second embodiments will be described below.

  FIG. 41 is a flowchart showing the procedure of the focusing operation when the focus adjustment (AF) area is fixed. First, the CPU 190 sets a focus adjustment (AF) area at the center of the angle of view based on an operation input from the user (step Sp1). Then, the image processing device 140 acquires a through image. Here, the image processing apparatus 140 may acquire through images (through image sequences) that are continuous in time (step Sp2).

  The CPU 190 causes the imaging unit 110 to execute a debayer (color interpolation) process. Further, the CPU 190 causes the image processing apparatus 140 to execute noise reduction processing (NR) and white balance adjustment (AWB) (step Sp3). Then, the feature amount calculation unit 141 acquires the image captured by the image capturing unit 110 from the buffer memory unit 130, and calculates the label creation feature amount of the area of the subject image captured in the focus adjustment (AF) area ( Step Sp4).

  Further, the feature amount calculation unit 141 creates a histogram for each feature amount, and selects a feature amount (a distinguishable feature amount) to be used in subsequent processing (step Sp5). Then, the area detection unit 142 calculates a discontinuous area (number of divisions and division areas) of the histogram based on the peak or valley in the histogram of the selected feature amount (see FIG. 18) (step Sp6).

  The region detection unit 142 detects a region by labeling based on a feature amount (hue in FIG. 18) indicating a peak waveform or a discontinuous region in the histogram (step Sp7). The priority setting unit 144 sets priorities for the areas detected by the area detection unit 142 based on the scores calculated by the score calculation unit 143. Then, the region detection unit 142 selects a region (effective region) having a high priority (Step Sp8).

  In order to expand the selected region, the region detection unit 142 relaxes the condition (for example, lowers the threshold value of the feature amount for labeling) and executes the labeling process again. Here, when there are a plurality of selected regions, the region detection unit 142 expands each region (step Sp9). The feature amount calculation unit 141 creates a histogram for a region 5% inside the boundary (outer periphery) of the region expanded by the region detection unit 142. Furthermore, the region detection unit 142 selects a hue (color channel) that is advantageous for contrast detection based on the histogram. For example, the region detection unit 142 selects a hue (color channel) having a high contrast value as a hue (color channel) advantageous for detecting the contrast (step Sp10).

  The region detection unit 142 determines whether the hue and luminance contrast values of the selected region are low (low contrast), or whether the hue and luminance contrast values are lower (further low contrast) (Step S1). Sp11). For example, when the contrast value is lower than a predetermined first threshold, the region detection unit 142 may determine that the contrast value of the region is low contrast. For example, when the contrast value is lower than the second threshold value that is smaller than the first threshold value, the region detection unit 142 may determine that the contrast value of the region is further low contrast.

  When the hue and brightness contrast values of the selected area are low (step Sp11—low contrast), the area detection unit 142 expands the contrast AF area and sets a high gamma table (see FIG. 15) in the selected area. (Step Sp12).

  The region detection unit 142 sets a contrast AF region in a region including 5% inside from the boundary (outer periphery) of the region where the high gamma table is set (step Sp13). The CPU 190 executes a contrast scan for the hue selected by the area detection unit 142, and causes the storage unit 160 to store the lens position at which the contrast of the hue selected by the area detection unit 142 has peaked (step Sp14).

  The CPU 190 moves the AF lens 112 to the lens position where the contrast has shown a peak, thereby focusing on the imaged area in the contrast AF area. Here, when a plurality of “selected areas” are imaged in the contrast AF area, the CPU 190 focuses on an area in which the subject distance is the shortest (imaged closest) (step Sp15). .

  On the other hand, when the hue and brightness contrast values of the selected area are even lower (step Sp11-lower contrast), the area detection unit 142 detects an edge near the boundary (outer periphery) of the selected area (step Sp16). . Then, the region detection unit 142 searches for an edge near the boundary (outer periphery) of the selected region, and detects the hue (color configuration) of the edge (step Sp17).

  Furthermore, the area detection unit 142 searches for an edge in the area where the hue is detected, and expands the area where the hue of the edge is detected until the SN ratio can be secured (step Sp18). Note that the area detection unit 142 may increase the sensitivity for detecting the hue and brightness contrast by changing the aperture (F value) or changing to a high gamma table.

  Then, the region detection unit 142 sets a contrast AF region in the region where the hue is detected (Step Sp19). The process of the area detection unit 142 proceeds to step Sp14.

  When there are a plurality of labels having the same priority, a plurality of labels may be used for searching for the in-focus position.

The procedure of the focusing operation when the focus adjustment (AF) area is fixed may be as follows.
The feature amount calculation unit 141 calculates a feature amount optimal for label detection (step Sr1). The region detection unit 142 detects a label based on the feature amount calculated by the feature amount calculation unit 141 (step Sr2). The area detection unit 142 sets a label in the contrast AF area (step Sr3). The region detection unit 142 extracts an edge outside the set label (step Sr4). The area detection unit 142 extends the label until it includes the extracted edge (step Sr5). The area detection unit 142 sets a label in the contrast AF area (step Sr6). The area detector 142 has two gamma tables, one of which is a high gamma table (step Sr7). The CPU 190 drives the AF lens 112 to execute a contrast scan (step Sr8). The region detection unit 142 determines that the label distributed on the closest side, either on the inner side or the outer side of the label, is a focal point (step Sr9).

  As described above, the image processing apparatus 140 expands the contrast AF area and sets a high gamma table in the selected area when the hue and luminance contrast values of the area of the subject image are low. Further, when the hue and luminance contrast values of the subject image area are further lower, the image processing apparatus 140 expands the area where the edge hue is detected until the SN ratio can be secured.

  As a result, when the focus adjustment (AF) area is fixed at the center of the angle of view, the image processing apparatus 140 detects the hue and brightness contrast even if the object image area has a low contrast. Since the contrast scan is executed, the region in the captured image can be identified with high accuracy. For example, the image processing apparatus 140 can identify a main subject image region and a non-main subject image region in a captured image.

<Combination with subject recognition technology>
The image processing device 140 detects the position of a human face by face detection (subject recognition technology), estimates the position of the torso or hair based on the detected position of the face, and sets the label at the estimated position. Also good.

  When the evaluation value (high SN, high contrast) that is more advantageous than using the face label can be calculated by using the torso or hair label, the CPU 190 may replace the face label with the torso or hair. A contrast scan may be performed using the label. The same applies to the case of detecting an animal, and the region detection unit 142 may search for a label having the same hue or texture as the animal's face around the animal's face. The CPU 190 performs a contrast scan using the advantageous evaluation value in the searched label.

  The same concept can be applied when other subject recognition techniques are used. In this case, it is assumed that the contrast peak of the face label and the contrast peak of the label around the face are at the same level. After confirming that this premise holds, the CPU 190 executes a contrast scan using the contrast evaluation values of the labels around the face. Further, the CPU 190 omits the process of confirming that this assumption is satisfied, and based on a predetermined positional relationship (for example, the positional relationship between the face and the torso), the contrast evaluation value of the label around the face May be used to perform a contrast scan.

[Fourth Embodiment]
A fourth embodiment of the present invention will be described in detail with reference to the drawings. In the fourth embodiment, when a night scene portrait (night scene P) is captured, the image processing apparatus 140 selects a feature quantity (for example, a hue feature quantity) that is advantageous for detecting a contrast. Different from the third embodiment. Only the differences from the first to third embodiments will be described below.

  FIG. 42 is a flowchart showing the procedure of the focusing operation when shooting a night scene. First, the CPU 190 sets the imaging mode to the night view portrait mode based on the operation input from the user (step Sq1). As a result, the CPU 190 causes the light emitting unit (not shown) included in the imaging apparatus 100 to emit AF auxiliary light. When the light emitting unit (not shown) emits AF auxiliary light, the AF auxiliary light is irradiated to the subject (step Sq2). Then, the image processing device 140 acquires a through image. Here, the image processing apparatus 140 may acquire a through image (through image sequence) that is continuous in time (step Sq3).

  The CPU 190 causes the imaging unit 110 to execute a debayer (color interpolation) process. Further, the CPU 190 causes the image processing apparatus 140 to perform noise reduction processing (NR) and white balance adjustment (AWB) (step Sq4). Then, the feature amount calculation unit 141 acquires the image captured by the imaging unit 110 from the buffer memory unit 130, executes face detection processing, and calculates a region where the face is captured in the image (step Sq5).

  Here, the face hue when the AF auxiliary light is applied to the face is measured in advance, for example, before manufacture of the imaging apparatus 100 and is stored in advance in the storage unit 160 as a predicted value of the face hue. Shall. Then, the region detection unit 142 labels a region where the face is captured in the image based on the predicted value of the hue of the face when the AF auxiliary light is irradiated to the face. As a result, the CPU 190 calculates an approximate distance from the imaging apparatus 100 to the face based on the lightness of the label and the magnification position of the AF lens 112 (see FIG. 1), and drives the AF lens 112. . When a plurality of labels are present in the image, the CPU 190 drives the AF lens 112 to focus on the face of the label with the highest brightness. If the plurality of labels have the same brightness, the CPU 190 drives the AF lens 112 to focus on the label closest to the center of the angle of view (step Sq6).

  As a result, the CPU 190 can compare the contrast of each label while avoiding a blurry state where the face label is not focused at all. In addition, when there are a plurality of regions in which an image of a face is captured, the region detection unit 142 sets a priority label and a priority according to the score of each region.

  Also, prediction of the hue of the face when the AF assist light is applied to the face in the area where the face is imaged because light (mixed light) from multiple types of light sources is applied to the face. When there is no value, the feature amount calculation unit 141 detects a face area by face detection, and calculates an average hue in the detected face area (step Sq7). Then, the region detection unit 142 labels the region in which the face is captured in the image based on the average hue detected by the feature amount calculation unit 141 (step Sq8). Then, the area detection unit 142 sets a “face label” in the area where the face is imaged (step Sq9).

  The feature amount calculation unit 141 creates an RGB histogram for a region 5% inside the outer periphery of the region where the face is imaged. Furthermore, the region detection unit 142 selects a feature amount advantageous for detecting contrast based on the histogram (step Sq10). In the following, as an example, the region detection unit 142 will be described assuming that a hue (color channel) having a high contrast value is selected as a hue advantageous for contrast detection.

  The region detection unit 142 determines whether the RGB and luminance of the region where the face is imaged are low contrast (step Sq11). For example, when the contrast value is lower than a predetermined threshold value, the region detection unit 142 determines that the contrast value of the region is low contrast. On the other hand, when the contrast value is equal to or greater than a predetermined threshold, the region detection unit 142 determines that the contrast value of the region is not low contrast.

  When the contrast value of the RGB channel and the luminance of the region where the face is imaged is low contrast (step Sq11-YES), the region detection unit 142 selects the color channel of the face contour (step Sq12). Then, the CPU 190 executes a contrast scan based on the selected color channel (step Sq13).

  On the other hand, when the contrast value of the face label is not low contrast (step Sq11-NO), the region detection unit 142 sets a contrast AF region in a region including 5% inside from the outer periphery of the region where the face is captured ( Step Sq14). And the process of the area | region detection part 142 progresses to step Sq13.

<Focusing operation when face detection is possible and when face detection is not possible when shooting night scene P>
When face detection can be performed (when there is a light intensity sufficient for face detection), the region detection unit 142 labels the periphery of the face based on the detected face. The CPU 190 calculates an approximate distance from the detected face to the imaging apparatus 100 based on the detected face, and drives the AF lens 112.

  On the other hand, when face detection is not possible, for example, there is no steady light quantity that can detect the face, and the AF auxiliary light is applied to the face, so that the hue of the face is caused by the mixed light of the steady light and the AF auxiliary light. Is not specified, the region detection unit 142 detects the region irradiated with the AF auxiliary light based on the difference image between when the AF auxiliary light is emitted and when the AF auxiliary light is turned off. In addition, the region detection unit 142 labels a region where a face is captured in an image based on a predicted value of the hue of the face when AF assist light is irradiated to the face (face label creation).

  FIG. 43 is a flowchart showing the procedure of the focusing operation when the face can be detected and when the face cannot be detected when the night scene P is imaged. The CPU 190 emits AF auxiliary light (step St1). The area detection unit 142 searches for a specific hue (for example, the hue of the mixed light) (step St2). The region detection unit 142 determines whether there is a specific hue (step St3). If there is a specific hue (step St3-YES), the region detection unit 142 evaluates the contrast of the label (step St4). When the contrast is low, the region detection unit 142 sets a high gamma table. If the high gamma table is set and the contrast is still low, the area detection unit 142 expands the label. Here, the region detection unit 142 sets a normal gamma table that is not a high gamma table (step St5). The CPU 190 executes contrast scan (step St6).

  On the other hand, when there is no specific hue (step St3-NO), the area detection unit 142 determines whether face detection is possible (step St7). If face detection is possible (step St7—YES), the region detection unit 142 performs labeling based on the predicted value of the hue of the face (step St8). When face detection is possible (step St7-NO), the area detection unit 142 performs labeling based on the difference image between when the AF auxiliary light is emitted and when it is turned off (step St9).

  As described above, when the night scene P is imaged, the image processing apparatus 140 detects the hue of the face when the AF auxiliary light is applied to the face, or the average hue in the area where the face is imaged, and Based on the hue, the region is labeled and a color channel advantageous for contrast detection is selected.

  Thus, when the night scene P is imaged, even if the face area of the subject image has a low contrast, a color channel that is advantageous for contrast detection is selected and a contrast scan is performed. The region can be identified with high accuracy. For example, the image processing apparatus 140 can identify a “face region” of a main subject image and a non-main subject image region in a captured image.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  For example, the feature amount calculation unit 141 may detect the subject distance or the defocus amount based on the “half width” of the line spread function (LSF) of the edge of the region detected for each hue by the region detection unit 142. .

  Further, for example, the area detection unit 142 cuts out the area of the subject image (graph cut) based on the area information (for example, the position, shape, and subject distance of the area), and other areas of the cut out subject image are displayed. May be combined with the image.

  Further, for example, the CPU 190 may cause the imaging unit 110 to perform imaging (continuous shooting) while driving the AF lens 112 in only one direction (changing the focus position) (AF bracket). Here, the area of the subject image in which the “priority label” is set may be subjected to contrast scanning before the AF bracketing is executed. Further, the CPU 190 may perform contrast scanning on the area of the subject image while executing the AF bracket.

  Further, for example, the order of image recording and reproduction display may be rearranged according to the priority of the area included in the image captured by the AF bracket (continuous shooting). For example, the image processing apparatus 140 sets the image in which the region with the highest priority is captured as the first image that is temporally continuous, and the image in which the region with the lowest priority is captured as the last temporally continuous image. As an image, the order of the images may be rearranged to perform recording and reproduction display.

  Further, for example, depending on the operation mode or when the “perspective flag” is on, an instruction to switch the area of the subject image to be tracked is received while the CPU 190 is received via the operation unit 180 or every time it is received. The CPU 190 may switch the area of the subject image to be tracked according to the priority label or in descending order of priority (tracking control). Further, when the CPU 190 receives a shooting instruction via the operation unit 180, the CPU 190 may cause the imaging unit 110 to capture (exposure).

  Further, for example, the CPU 190 may execute focus adjustment such as a contrast AF method or a phase difference detection method as the initial AF in step Sa3 in FIG.

  Further, for example, the image processing apparatus 140 may use data expressed in the RGB color space in the subsequent processing.

  2, 3, 4, 5, 6, 7, 8, 10, 11, 17, 19, 20, 24, 26, 27, 28, 29, 30, 31, 32, 33, 38, 41, 42 , And 43, a program for realizing the procedure described above is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, thereby executing an execution process. You may go. Here, the “computer system” may include an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 100 ... Imaging device, 110 ... Imaging part, 111 ... Lens barrel, 140 ... Image processing apparatus, 141 ... Feature-value calculation part, 142 ... Area | region detection part, 143 ... Rating calculation part, 144 ... Priority setting part

Claims (8)

A feature amount calculation unit for calculating a feature amount of the feature for each of a plurality of features in the image;
An area detection unit that detects an area included in the image and is determined for each feature amount from the image, and sets an identifier;
A score calculation unit for calculating, for each region, a score for comparing the regions detected based on the feature quantities of the different features;
A priority setting unit that sets the priority of the region in the image based on the score;
An image processing apparatus comprising:   The region detection unit overlaps the first region detected based on the feature amount of the first feature and the second region detected based on the feature amount of the second feature. 2. The image processing according to claim 1, wherein the area of the portion is calculated, and the second region is combined with the first region when the area of the overlapping portion is equal to or larger than a predetermined area. apparatus.   The region detection unit detects a distance corresponding to the third region detected based on the feature amount of the third feature and a fourth region detected based on the feature amount of the fourth feature. 3. The image processing apparatus according to claim 1, wherein the fourth region is coupled to the third region when the distance corresponding to the same is the same distance. 4. The region detection unit, when combining the regions, adds the scores for each of the regions before combining, and sets the combined score as the score of the region after combining. The image processing apparatus according to claim 2 or claim 3. The image processing according to claim 3 , wherein the region detection unit divides the region into the plurality of small regions when the plurality of small regions constituting the region correspond to the different distances. apparatus. The region detection unit is based on a depth map indicating the distribution of the distance in the image.
The image processing apparatus according to claim 3 or claim 5, characterized in that coupling or dividing the region. An imaging unit that captures an image by an optical system and outputs the captured image;
A feature amount calculation unit that calculates a feature amount of the feature for each of a plurality of features in the image;
An area detection unit that detects an area included in the image and is determined for each feature amount from the image, and sets an identifier;
A score calculation unit for calculating, for each region, a score for comparing the regions detected based on the feature quantities of the different features;
A priority setting unit that sets the priority of the region in the image based on the score;
An imaging apparatus comprising: On the computer,
For each of a plurality of features in the image, a procedure for detecting a feature amount of the feature;
Detecting an area included in the image and determined for each feature amount from the image, and setting an identifier;
A score calculation unit for calculating, for each region, a score for comparing the regions detected based on the feature quantities of the different features;
A priority setting unit that sets the priority of the region in the image based on the score;
An image processing program for executing

Images