JP2017201818A - Area extraction device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an area extraction device capable of shortening processing time for focusing at a main subject area.SOLUTION: An area extraction device comprises: an area extraction unit which extracts a subject area on the basis of a feature quantity from a captured image; and an area processing unit which selects the subject area on the basis of positional relationship of the subject area extracted by the area extraction unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a region extraction device that extracts a region from an image.

  An image processing apparatus that extracts a region from a captured image and identifies the extracted region based on a histogram of image feature values is disclosed (see Patent Document 1).

Japanese Patent Laid-Open No. 4-10079

  However, the image processing apparatus disclosed in Patent Document 1 has a problem that the processing time for focusing on the area of the main subject is long.

  The present invention has been made in view of the above points, and an object of the present invention is to provide an area extraction device with a short processing time for focusing on the area of the main subject.

  The present invention has been made to solve the above-described problem, and an area extraction unit that extracts a subject area from a captured image based on a feature amount, and the subject area extracted by the area extraction unit An area processing unit that selects the subject area based on the positional relationship.

  According to the present invention, the area extracting apparatus can shorten the processing time for focusing on the area of the main subject.

It is a block diagram showing the structure of a lens barrel, an imaging device provided with a region extraction device, and a storage medium in the first embodiment of the present invention. It is a figure for demonstrating the method to integrate a to-be-photographed area | region in 1st Embodiment of this invention. It is a figure for demonstrating the method to integrate the to-be-photographed object area | region based on a color difference in the 1st Embodiment of this invention. FIG. 7 is a diagram for explaining a method of dividing a portion where a subject area based on luminance and a subject area based on color difference overlap from a subject area based on luminance in the first embodiment of the present invention. It is a figure for demonstrating the method to integrate and divide | segment a to-be-photographed area | region in 1st Embodiment of this invention. It is a flowchart showing the procedure of integrating and dividing | segmenting a to-be-photographed area | region in 1st Embodiment of this invention. It is a flowchart showing the procedure of the process until it makes it focus on an area | region in 1st Embodiment of this invention. It is a figure for demonstrating the example of the process which preferentially selects the to-be-photographed object area | region in the 2nd Embodiment of this invention.

[First Embodiment]
A first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating the configuration of a lens barrel, an imaging device including a region extraction device, and a storage medium. The imaging apparatus 100 captures an optical image incident from the lens barrel 111 and stores a still image format or moving image format image in the storage medium 200.

First, the configuration of the lens barrel 111 will be described.
The lens barrel 111 includes a focus adjustment lens (hereinafter referred to as “AF (Auto Focus) lens”) 112, a lens driving unit 116, an AF encoder 117, and a lens barrel control unit 118. The lens barrel 111 may be detachably connected to the imaging device 100 or may be integrated with the imaging device 100.

  The AF lens 112 is driven by a lens driving unit 116 and guides an optical image to a light receiving surface (photoelectric conversion surface) of an image sensor 119 of the image capturing unit 110 described later.

  The AF encoder 117 detects the movement of the AF lens 112 and outputs a signal corresponding to the movement amount of the AF lens 112 to the lens barrel control unit 118. Here, the signal corresponding to the movement amount of the AF lens 112 may be a sine wave signal whose phase changes according to the movement amount of the AF lens 112, for example.

  The lens barrel control unit 118 controls the lens driving unit 116 in accordance with a drive control signal input from the CPU 190 of the imaging apparatus 100. Here, the drive control signal is a control signal for driving the AF lens 112 in the optical axis direction. The lens barrel control unit 118 changes the number of steps of the pulse voltage output to the lens driving unit 116, for example, according to the drive control signal.

  Further, the lens barrel control unit 118 outputs the position (focus position) of the AF lens 112 in the lens barrel 111 to the CPU 190 based on a signal corresponding to the movement amount of the AF lens 112. Here, the lens barrel control unit 118 integrates, for example, signals according to the movement amount of the AF lens 112 according to the movement direction of the AF lens 112, thereby moving the AF lens 112 in the lens barrel 111 (the movement amount ( Position) may be calculated.

  The lens driving unit 116 drives the AF lens 112 in accordance with the control of the lens barrel control unit 118 and moves the AF lens 112 in the optical axis direction within the lens barrel 111.

Next, the configuration of the imaging device 100 will be described.
The imaging device 100 includes an imaging unit 110, a region extraction device 140, a display unit 150, a buffer memory unit 130, an operation unit 180, a storage unit 160, a CPU 190, and a communication unit 170.

  The imaging unit 110 includes an imaging element 119 and an A / D (Analog / Digital) conversion unit 120. The imaging unit 110 is controlled by the CPU 190 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.).

  The image sensor 119 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 111 (optical system) into an electric signal, and outputs the electric signal to the A / D conversion unit 120. The image sensor 119 may be configured by, for example, a CMOS (Complementary Metal Oxide Semiconductor). Further, the image sensor 119 may convert an optical image into an electrical signal for a partial region of the photoelectric conversion surface (image clipping).

  Further, the image sensor 119 stores an image obtained when a shooting instruction from the user is received via the operation unit 180 in the storage medium 200 via the A / D conversion unit 120. On the other hand, the imaging device 119 performs A / D conversion on the buffer memory unit 130 and the display unit 150 as images that are continuously obtained in a state where a shooting instruction from the user is not received via the operation unit 180. Output via the unit 120.

  The A / D converter 120 digitizes the electrical signal converted by the image sensor 119 and outputs an image that is a digital signal to the buffer memory unit 130.

  The operation unit 180 includes, for example, a power switch, a shutter button, a multi selector (cross key), or other operation keys. The operation unit 180 receives a user operation input when operated by the user, and a signal corresponding to the received operation input. Is output to the CPU 190.

  The area extraction device 140 performs image processing on the image temporarily stored in the buffer memory unit 130 based on the image processing conditions stored in the storage unit 160. The processed image is stored in the storage medium 200 via the communication unit 170. Further, the region extraction device 140 extracts a region determined by the feature amount of the image from the image temporarily stored in the buffer memory unit 130. Further, the region extraction device 140 stores information indicating the extracted region in the storage unit 160 or the like. Here, the information indicating the region includes, for example, information indicating the distribution of the region and information indicating the feature amount of the region. In addition, the region extraction device 140 sets a priority for focusing the subject region on the subject region extracted from the image temporarily stored in the buffer memory unit 130.

  The display unit 150 displays an image obtained by the imaging unit 110, an operation screen, and the like. The display unit 150 is, for example, a liquid crystal display.

  The buffer memory unit 130 temporarily stores the image captured by the imaging unit 110.

  The storage unit 160 stores determination conditions referred to by the CPU 190 in scene determination, imaging conditions associated with each scene determined by scene determination, information indicating a region, and the like.

  The CPU 190 controls the imaging unit 110 according to the set imaging conditions (for example, aperture value, exposure value). In addition, the CPU 190 acquires information indicating a region from the region extraction device 140. In addition, the CPU 190 (focus adjustment unit) outputs a drive control signal to the lens barrel control unit 118 of the lens barrel 111 based on the information indicating the region, so that the region extraction device 140 out of the regions extracted from the image. Adjust the focus to focus on the high priority area set by.

  Also, the CPU 190 performs focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, subject image as pre-processing or post-processing of image processing based on the information indicating the area. (Object) tracking setting, night scene judgment setting, color correction processing setting, subject image enlarged display setting, panning display setting, brightness optimization setting linked with zoom magnification, etc. To control. Further, the CPU 190 controls setting of a tone curve (gradation curve), whether or not to execute simultaneous shooting of a still image and a moving image, and the like based on information indicating a region.

  In addition, the CPU 190 sets whether to perform continuous shooting as pre-processing or post-processing based on information indicating the area, setting for switching the subject image to be focused in continuous shooting, and continuously changing the focal length. , Setting for changing the amount of flash light emitted from the light emitting unit (not shown) of the image capturing apparatus 100, setting for changing the exposure amount, setting for whether to change the shutter speed, and the like. In addition, the CPU 190 causes the region extraction device 140 to perform image processing on an image in a still image format or a moving image format based on the “signal according to the operation input” input from the operation unit 180.

  The communication unit 170 is connected to a removable storage medium 200 such as a card memory, and writes, reads, or deletes information (image data, information indicating an area, etc.) on the storage medium 200.

  The storage medium 200 is a storage unit that is detachably connected to the imaging apparatus 100, and stores information (image data, information indicating an area, and the like). Note that the storage medium 200 may be integrated with the imaging device 100.

Next, details of the region extraction device 140 will be described.
The region extraction device 140 includes a feature amount calculation unit 141, a region extraction unit 142, a region processing unit 143, and a priority setting unit 144.

  The feature amount calculation unit 141 acquires the image captured by the imaging unit 110 from the buffer memory unit 130. The feature amount calculation unit 141 also has a feature amount (for example, a feature amount based on luminance information) for each of a plurality of features (for example, hue, saturation, brightness, texture, edge) in the image acquired from the buffer memory unit 130. , The feature amount based on the color information) is calculated, and the calculated feature amount is output to the region extraction unit 142.

  The region extraction unit 142 acquires the feature amount calculated by the feature amount calculation unit 141. Further, the region extraction unit 142 acquires an image captured by the imaging unit 110 from the buffer memory unit 130. In addition, the region extraction unit 142 extracts a subject region that is included in the acquired image and is determined for each feature amount from the image. Here, the subject region extracted by the region extraction unit 142 is, for example, a region of a subject image included in an image captured by the imaging unit 110.

  The region extraction unit 142 extracts a subject region from the image based on the acquired feature amount (for example, luminance (Y), color information (Cr, Cb, RGB)). For example, the region extraction unit 142 extracts a subject region from an image binarized into an image feature amount within a certain range and an image feature amount outside the range. The area extraction unit 142 assigns a label to the subject area (labeling).

  Further, the region extraction unit 142 generates a first rectangular region circumscribing the subject region based on the first image feature amount and a rectangular region circumscribing the subject region based on the second image feature amount. In the following, it is assumed that the first image feature amount is a feature amount according to the color information. A rectangular area circumscribing the subject area based on the color difference is hereinafter referred to as a “color difference area”. The second image feature amount is assumed to be a feature amount corresponding to the luminance. The rectangular area circumscribing the subject area based on the luminance is hereinafter referred to as “luminance rectangular area”. Note that the shape of the area circumscribing the subject area (the circumscribed area) does not have to be a rectangle.

  The area processing unit 143 determines whether or not the subject area satisfies a predetermined condition. For example, when there are regions that are estimated to be noise (for example, regions having an area smaller than a predetermined threshold) in the image, the region processing unit 143 removes (adds) those regions from the image. Further, for example, if the image includes a region estimated to be the background (for example, a region having an area larger than a predetermined threshold), the region processing unit 143 removes the region from the image. The area processing unit 143 causes the storage unit 160 to store information representing the distribution of the subject area that remains without being removed.

  The area processing unit 143 selects a subject area based on the positional relationship of the subject areas extracted by the area extraction unit 142, and integrates or divides the selected subject areas. That is, the area processing unit 143 selects subject areas that are close in position, and integrates or divides the selected subject areas. Further, the area processing unit 143 selects a subject area based on the shape of the subject area, and integrates or divides the selected subject area. That is, the area processing unit 143 selects a subject area whose shape is approximate, and integrates or divides the selected subject area.

  The area processing unit 143 uses an area of the subject area extracted from the image based on the luminance as a numerator, and calculates an area ratio (filling rate) for each luminance rectangular area by using the area of the luminance rectangular area circumscribing the subject area as a denominator calculate. In addition, the region processing unit 143 excludes the subject region and the luminance rectangular region whose area ratio is less than a predetermined threshold from the objects to be divided or integrated. Similarly, the area processing unit 143 uses the area of the subject area extracted from the image based on the color difference as the numerator, and sets the area ratio (filling rate) using the area of the color difference rectangular area circumscribing the subject area as the denominator. Calculate for each region. In addition, the region processing unit 143 excludes the subject region and the color difference rectangular region whose area ratio is less than a predetermined threshold from the objects to be divided or integrated.

  The area processing unit 143 integrates subject areas based on color differences based on the coordinates of the corners of the color difference rectangular areas. Further, the area processing unit 143 integrates subject areas based on luminance based on the coordinates of the corners of the luminance rectangular area. FIG. 2 shows a diagram for explaining a method of integrating subject areas. Here, the subject areas inscribed in the rectangular areas are not shown.

  In FIG. 2A, the color difference rectangular area 610 is included in the color difference rectangular area 600, and the coordinate difference between the corners of the color difference rectangular area 600 and the color difference rectangular area 610 is the predetermined number of pixels M (M is 1 or an integer) or less. In this case, the area processing unit 143 integrates the color difference rectangular area 610 into the color difference rectangular area 600 having a large area. Similarly, the area processing unit 143 integrates subject areas (not shown) extracted from images based on luminance.

  Note that when the color difference rectangular area 600 and the color difference rectangular area 610 are integrated, the area processing unit 143 may integrate the areas so that the color difference rectangular area 600 and the color difference rectangular area 610 have the longest vertical and horizontal sides. The same applies to the luminance rectangular area.

  In FIG. 2B, a part of the color difference rectangular area 710 and the color difference rectangular area 710 overlap, and the coordinate difference between the corners of the color difference rectangular area 700 and the color difference rectangular area 710 is equal to or less than a predetermined number M of pixels. Suppose there is. In this case, the area processing unit 143 integrates the color difference rectangular area 710 into the color difference rectangular area 700 having a large area. Similarly, the area processing unit 143 integrates subject areas (not shown) extracted from images based on luminance.

  The area processing unit 143 integrates the subject area based on the luminance into the subject area based on the color difference based on the coordinates of the corners of the color difference rectangular area and the coordinates of the corners of the luminance rectangular area. For example, when the difference between the corner coordinates of the color difference rectangular area and the corner coordinates of the luminance rectangular area is within N (N is an integer equal to or greater than 1) pixels, the area processing unit 143 determines the subject area based on the luminance. Are integrated into the subject area based on the color difference.

  FIG. 3 is a diagram for explaining a method for integrating a subject area based on luminance into a subject area based on color difference. Here, the subject areas inscribed in the rectangular areas are not shown. The area processing unit 143 includes the luminance rectangular area 410 included in the color difference rectangular area 400, and the coordinate difference between the corners of the color difference rectangular area 400 and the luminance rectangular area 410 has a predetermined number of pixels L (L is It is determined whether or not it is an integer of 1 or more.

  In FIG. 3, it is assumed that the luminance difference rectangular area 410 is included in the color difference rectangular area 400 and that the coordinate difference between the corners of the color difference rectangular area 400 and the luminance rectangular area 410 is equal to or more than a predetermined number L of pixels. In this case, the area processing unit 143 integrates the luminance rectangular area 410 into the color difference rectangular area 400. Similarly, the area processing unit 143 integrates subject areas (not shown) extracted from images based on luminance. Similarly, the area processing unit 143 integrates a subject area (not shown) inscribed in the luminance rectangular area 410 into a subject area (not shown) inscribed in the color difference rectangular area 400.

  The area processing unit 143 is based on the luminance when the luminance rectangular area includes the color difference rectangular area and the coordinate difference between the corners of the luminance rectangular area and the color difference rectangular area is equal to or greater than the predetermined number L of pixels. A portion where the subject region and the subject region based on the color difference overlap is divided from the subject region based on the luminance.

  FIG. 4 shows a diagram for explaining a method of dividing a portion where a subject region based on luminance and a subject region based on color difference overlap from a subject region based on luminance. Here, the subject areas inscribed in the rectangular areas are not shown.

  The area processing unit 143 includes a luminance rectangular area 500 circumscribing a subject area (not shown) based on luminance, a color difference rectangular area 510 circumscribing a subject area (not shown) based on color difference, and a color difference rectangle. It is determined whether the difference between the coordinates of the corners of the area and the luminance rectangular area is equal to or greater than a predetermined number of pixels L.

  The region processing unit 143 divides a portion where a subject region based on luminance and a subject region based on color difference overlap from a subject region based on luminance. Accordingly, the area processing unit 143 divides the color difference rectangular area 510 from the luminance rectangular area 500 and transforms the luminance rectangular area 500 into the luminance rectangular area 500-1. The same applies to the luminance rectangular area 500-2 to the luminance rectangular area 500-4. Here, the area processing unit 143 may execute a process of removing the color difference rectangular area 540 from the luminance rectangular area 500 according to the processing capability.

  Returning to FIG. 1, the description of the region extraction device 140 will be continued. The priority setting unit 144 determines “region priority” in an image for a subject region that satisfies a predetermined condition. The “region priority” is a region priority that a person will be more interested in. The priority setting unit 144 stores the set “region priority” in the storage unit 145, the storage unit 160, and the storage medium 200 for each region as information indicating the region.

  The priority of the area may be determined as follows. In the following, it is assumed that the priority of a region is higher as the number indicating the priority is smaller. Moreover, the priority shown below is an example.

Priority 1.
A moving area (when there are a plurality of moving areas in the image, an area that does not protrude from the angle of view or an area that is distributed in the center of the angle of view). Also, if the moving area is widely distributed in the image due to panning, the area is not moving.
These areas are set to priority “1”.
This is because a person pays attention to a moving area and an area having a small amount of change when panning.

Priority 2.
Areas related to angle of view (composition), depth (perspective), and subject distance (depth, depth). An area distributed near the center of the angle of view. When there is a depth in the image, among the regions distributed on the closest side, the region does not protrude from the angle of view and has an area within a predetermined range (not extremely large).
These areas are set to priority “2”.
This is because the main subject image area may exist in a wide range from close to infinity.

Priority 3.
Regions extracted based on human face detection, animal (pet) face detection, skin color detection in the area where the face is detected, animal fur texture (pattern), or plant texture.
These areas are set to priority “3”.
This is because when a person or animal is imaged, the person pays attention to the face.

Priority 4.
A region having a characteristic amount that is unique (for example, a hue that is discontinuous in the hue circle). An area divided into a background area and other areas. Highly saturated area. An area where the brightness is significantly different from the surrounding area. An area where the hue (hue) is significantly different from the surrounding area. An area where the texture (pattern) is different from the surrounding area. An area where the spatial frequency (image frequency) is higher than the surrounding area. A region in which an image has a sense of incongruity when a certain amount of discontinuous distribution has a certain amount and a region is formed as a result of labeling based on the feature amount.
These areas are set to priority “4”.

Priority 5.
An area extracted based on a specific hue or saturation. Here, the specific hue is a highly saturated color such as yellow, purple, red, and skin color.
These areas are set to the priority “5”.
This is because a person pays attention to red, yellow, and flesh-colored areas.

Priority 6.
An area extracted based on brightness (brightness).
These areas are set to priority “6”.

Priority 7.
An area extracted based on a texture (pattern).
These areas are set to priority “7”.

Priority 8.
A region extracted based on the spatial frequency (image frequency).
These areas are set to priority “8”.

Priority 9.
An area surrounded by edges extending vertically or horizontally and having a certain thickness or more.
These areas are set to the priority “9”.

Next, the processing procedure of the region extraction apparatus will be described.
FIG. 5 shows a diagram for explaining a method of integrating and dividing the subject area. FIG. 6 is a flowchart showing a procedure for integrating and dividing the subject area.

  The area extraction unit 142 reads the cropped image (binarized image) 300Y, the cropped image 300Cr, and the cropped image 300Cb, which are the resized captured images, for each image feature amount via the buffer memory unit 130.

  The area extraction unit 142 extracts a plurality of subject areas based on the color difference (Cr) from the image 300Cr, and generates a color difference rectangular area circumscribing the extracted subject area in the image. Further, the area processing unit 143 calculates the ratio of the area of the color difference rectangular area and the area of the subject area included in the color difference rectangular area for each color difference rectangular area. The same applies to the color difference (Cb).

  Further, the region extraction unit 142 extracts a subject region based on the luminance (Y) from the image 300Y, and generates a luminance rectangular region that circumscribes the extracted subject region. Further, the area processing unit 143 calculates a ratio between the area of the luminance rectangular area and the area of the subject area included in the luminance rectangular area for each luminance rectangular area.

  The area extraction unit 142 generates a crop image for each fixed range in the luminance gradation range. In FIG. 5, it is assumed that the binarized image 300Y-1 having the gradation range with the highest brightness is generated from the crop image 300Y. Further, it is assumed that a binarized image 300Y-2 having the second highest luminance gradation range is generated from the crop image 300Y. Further, it is assumed that a binarized image 300Y-3 composed of the third highest luminance gradation range is generated from the crop image 300Y. Further, it is assumed that a binarized image 300Y-4 including the fourth highest luminance gradation range is generated from the crop image 300Y.

  Here, it is assumed that the subject region 310 based on luminance is extracted from the binarized image 300Y-1. Further, it is assumed that the subject area 320 based on luminance is extracted from the binarized image 300Y-2. The region extraction unit 142 generates a luminance rectangular region that circumscribes the subject region 310 in the binarized image 300Y-1. Further, the region extraction unit 142 generates a luminance rectangular region circumscribing the subject region 320 in the binarized image 300Y-2.

  Similarly for the color difference (Cr), in FIG. 5, it is assumed that binarized images 300Cr-1 to 300Cr-4 are generated from the crop image 300Cr. Here, it is assumed that the subject region 330 based on the color difference (Cr) is extracted from the binarized image 300Cr-2. Further, it is assumed that a subject area 340 based on the color difference (Cr) is extracted from the binarized image 300Cr-3. The area extraction unit 142 generates a color difference rectangular area circumscribing the subject area 330 in the binarized image 300Cr-2. In addition, the region extraction unit 142 generates a color difference rectangular region circumscribing the subject region 340 in the binarized image 300Cr-3.

  Similarly, regarding the color difference (Cb), in FIG. 5, it is assumed that binarized images 300Cb-1 to 300Cb-4 are generated from the cropped image 300Cb. Here, it is assumed that the subject region 350 based on the color difference (Cb) is extracted from the binarized image 300Cb-1. The area extraction unit 142 generates a color difference rectangular area circumscribing the subject area 350 in the binarized image 300Cb-1.

  The area processing unit 143 stores information indicating an object to be excluded in order to exclude the subject area and the luminance rectangular area whose area ratio is less than a predetermined threshold from the objects to be divided or integrated. Similarly, the area processing unit 143 stores information indicating an object to be excluded in order to exclude the subject area and the color difference rectangular area whose area ratio is less than a predetermined threshold from the objects to be divided or integrated (step S1). ).

  Based on the coordinates of the corner of the color difference rectangular area circumscribing the subject area 340 and the coordinates of the corner of the color difference rectangular area circumscribing the subject area 350 (see FIG. 2), the area processing unit 143 A subject area 370 obtained by integrating the areas 350 is extracted from the binarized image 300Cr-3 and the binarized image 300Cb-1 (step S2).

  Based on the coordinates of the corners of the luminance rectangular area circumscribing the subject area and the coordinates of the corners of the luminance rectangular area circumscribing the other subject areas (see FIG. 2), the area processing unit 143 Is extracted from the binarized image 300Y-1 (step S3).

  The area processing unit 143 coordinates the corners of the color difference rectangular area circumscribing the subject area 330 based on the color difference (Cr in FIG. 5) and the coordinates of the corners of the luminance rectangular area circumscribing the subject area 320 based on the luminance (Y). Based on the above, the subject area 320 based on the luminance is integrated with the subject area 330 based on the color difference, and the subject area 360 is extracted from the binarized image 300Y-2 and the binarized image 300Cr-2 (step S4).

  The area processing unit 143 includes a luminance rectangular area circumscribing the subject area based on luminance in the color difference rectangular area circumscribing the subject area based on color difference, and a coordinate difference between corners of the color difference rectangular area and the luminance rectangular area. Is determined for each subject area whether or not is equal to or greater than a predetermined number of pixels (step S5).

  When the determination condition is not satisfied in step S5 (step S5-No), the area processing unit 143 includes the color difference rectangular area circumscribing the subject area 370 based on the color difference in the luminance rectangular area circumscribing the subject area 310 based on the luminance. In addition, it is determined for each subject area whether the difference between the coordinates of the corners of the color difference rectangular area and the luminance rectangular area is equal to or greater than a predetermined number of pixels (step S6).

  On the other hand, when the determination condition is satisfied in step S5 (step S5-Yes), the area processing unit 143 selects a luminance rectangular area whose corner coordinate difference is greater than or equal to a predetermined number of pixels, out of the subject areas based on luminance. The inscribed subject area is integrated into the subject area based on the color difference (see FIG. 3) (step S7).

  When the determination condition is satisfied in step S6 (step S6-Yes), the region processing unit 143 divides a portion where the subject region 310 based on luminance and the subject region 370 based on color difference overlap from the subject region 310 based on luminance. (Step S8).

  Here, of the subject region 310 and the subject region 370, the region processing unit 143 determines, for example, the subject region 370 distributed at the center of the angle of view as the main subject region.

  FIG. 7 is a flowchart showing a processing procedure until focusing on an area. The feature amount calculation unit 141 captures a captured image (through image or through image sequence) from the buffer memory unit 130 (Step Sa1), and executes basic image processing on the captured image (Step S1). Sa2).

  The region extraction unit 142 reads a cropped image (64 × 40 pixels) obtained by resizing the captured image for each color difference via the buffer memory unit 130 (step Sa3). The region extraction unit 142 generates a binarized image based on a plurality of image feature amounts in the crop image (step Sa4). The region extraction unit 142 extracts a subject region from the binarized image and assigns a label to the extracted region (step Sa5).

  When there are regions that are estimated to be noise in the image, the region processing unit 143 removes these regions from the image (step Sa6). In addition, when the region estimated to be the background is in the image, the region processing unit 143 removes the region from the image (step Sa7).

  The priority setting unit 144 determines the “region priority” in the image for each subject region. The priority setting unit 144 divides the subject area based on a predetermined condition (step Sa8). Further, the priority setting unit 144 integrates the subject areas based on a predetermined condition (step Sa9). Further, the area extracting unit 142 shapes the subject area for which the priority setting unit 144 has determined the priority into a rectangle so as to fit the AF evaluation area (step Sa10). The CPU 190 executes a contrast scan (focus adjustment process) based on the contrast value of the AF evaluation area (step Sa11). Further, the CPU 190 gives priority to the subject area in which the subject that is in close proximity is imaged based on the result of the contrast scan, and focuses the subject area (step Sa12).

As described above, the region extraction device 140 extracts a subject region (for example, the subject region 310 to the subject region 350) from the captured image based on the feature amount (for example, color information (color difference, hue), luminance). The subject region is selected (for example, see FIG. 5) based on the positional relationship (for example, see FIGS. 2 to 4) of the subject region extracted by the region extraction unit 142 and the region extraction unit 142 An area processing unit 143.
With this configuration, the area extraction device 140 selects a subject area based on the positional relationship of the subject areas extracted by the area extraction unit 142. Thereby, the area extracting apparatus 140 can shorten the processing time for focusing on the area of the main subject. Note that the selected subject area or a frame indicating the position thereof may be displayed on the display unit 150, for example.

Further, the region processing unit 143 integrates or divides the selected subject region based on the positional relationship (for example, see FIGS. 2 to 4) (for example, see FIG. 5).
With this configuration, the region extraction device 140 integrates or divides the subject regions based on the positional relationship of the subject regions extracted by the region extraction unit 142. Thereby, the area extracting apparatus 140 can shorten the processing time for specifying the area of the main subject and focusing on the area of the main object.

In addition, the area processing unit 143 integrates or divides the subject area based on the shape of the subject area (see, for example, FIG. 5).
With this configuration, the region extraction device 140 integrates or divides the subject region based on the shape of the subject region. Thereby, the area extracting apparatus 140 can shorten the processing time for focusing on the area of the main subject.

In addition, the area extraction unit 142 generates a circumscribed area that circumscribes the subject area, and the area processing unit 143 integrates or divides the subject area based on the coordinates of the corners of the circumscribed area (for example, , See FIGS.
With this configuration, the area processing unit 143 integrates or divides the subject area based on the coordinates of the corners of the circumscribed area. Thereby, the area extracting apparatus 140 can shorten the processing time for focusing on the area of the main subject.

In addition, the area processing unit 143 integrates the subject areas when the coordinate difference between the corners of the circumscribed area is equal to or less than a predetermined number of pixels M (see, for example, FIG. 2).
With this configuration, the area processing unit 143 integrates the subject areas when the coordinate difference between the corners of the circumscribed area is equal to or less than a predetermined number of pixels. Thereby, the area extracting apparatus 140 can shorten the processing time for focusing on the area of the main subject.

Further, the area processing unit 143 has a corner of a first circumscribed area (for example, the color difference rectangular area 400) circumscribing the subject area extracted based on the first feature amount (for example, color information (color difference, hue)). Based on the coordinates and the coordinates of the corners of the second circumscribed area (for example, luminance rectangular area 410) circumscribing the subject area extracted based on the second characteristic amount (for example, luminance), the second characteristic amount The subject region extracted based on the first feature amount is integrated with the subject region extracted based on the first feature amount (see, for example, FIG. 3).
With this configuration, the area processing unit 143 extracts the subject area extracted based on the second feature amount based on the coordinates of the corners of the first circumscribed area and the coordinates of the corners of the second circumscribed area. The subject area extracted based on the first feature amount is integrated. Thereby, the area extracting apparatus 140 can shorten the time required for the process for focusing on the area of the main subject.

In addition, the area processing unit 143 includes the second circumscribed area (for example, the luminance rectangular area 410) in the first circumscribed area (for example, the color difference rectangular area 400), and the first circumscribed area and the When the coordinate difference between the corners of the second circumscribing area is equal to or greater than a predetermined number of pixels L, among the subject areas extracted based on the second feature amount, the pixel with a predetermined coordinate difference between the corners A subject region extracted based on the second feature amount inscribed in the second circumscribed region that is equal to or greater than a number L is changed to a subject region extracted based on the first feature amount inscribed in the first circumscribed region. Integrate (see, eg, FIG. 3).
With this configuration, the area processing unit 143 includes the second circumscribed area in the first circumscribed area, and a coordinate difference between corners of the first circumscribed area and the second circumscribed area is determined in advance. If the number of pixels is greater than or equal to L, the subject areas are integrated. Thereby, the area extracting apparatus 140 can shorten the time required for the process for focusing on the area of the main subject.

In addition, the area processing unit 143 adds the first circumscribed area (for example, the color difference rectangular area 510, the color difference rectangular area 520, the color difference rectangular area 530, and the color difference rectangular area 540) to the second circumscribed area (for example, the luminance rectangular area 500). And a subject extracted based on the second feature amount when a coordinate difference between corners of the second circumscribed area and the first circumscribed area is equal to or greater than a predetermined number of pixels L. A portion where a region (for example, the subject region 310) and a subject region (for example, the subject region 370) extracted based on the first feature amount overlap is extracted from the subject region extracted based on the second feature amount. Divide (see, for example, FIGS. 4 and 5).
With this configuration, the area extracting device 140 includes the first circumscribed area in the second circumscribed area, and a coordinate difference between corners of the second circumscribed area and the first circumscribed area is determined in advance. If the number of pixels is greater than or equal to L, the subject area is divided. Thereby, the area extracting apparatus 140 can shorten the processing time for extracting the main subject area from the image and focusing on the main subject area.

In addition, the imaging apparatus 100 includes a region extraction device 140, a CPU 190 that adjusts the focus so that the subject region extracted by the region extraction device is focused on a high priority region, and the high priority region. And an imaging unit 110 that captures the focused image.
With this configuration, the area extraction device 140 shortens the processing time for focusing on the area of the main subject. Thereby, the imaging device 100 can image the region of the main subject that the region extraction device 140 has focused in a short time.

[Second Embodiment]
The second embodiment is different from the first embodiment in that the subject area remaining after the integration is preferentially selected as a candidate for the main subject area. Only the differences from the first embodiment will be described below.

  The subject areas extracted based on different feature quantities are integrated (for example, see step Sa9 in FIG. 7). This means that the integrated subject areas have the same position and shape (for example, FIG. 2). That is). Therefore, there is a high possibility that the subject area remaining after the integration is the main subject area. On the other hand, a subject region that is not integrated with other subject regions is likely to be a region other than the main subject. Therefore, setting a subject area that is not integrated with another subject area as a candidate for the main subject area increases the possibility of focusing on an area other than the main subject.

  In the following, the description is continued assuming that the number Nrd of candidate areas of the main subject is predetermined as a value 3 as an example. Note that information indicating the candidate number Nrd of the main subject area may be stored in advance in the storage unit 160 (see FIG. 1), for example.

  When the extracted subject regions are integrated, the region processing unit 143 reduces the number Nrd of candidate main subject regions according to the number N of subject regions integrated with other subject regions. For example, the area processing unit 143 reduces the number Nrd of candidate areas of the main subject to the value “Nrd− (N−1)” according to the number N of the subject areas integrated with the other subject areas. A specific example will be described later with reference to FIG. Note that “Nrd− (N−1)” is an example of an expression indicating a reduction in the number Nrd of candidates for the main subject area.

  The area extraction unit 142 shapes the subject areas with the highest priority Nrd selected by the area processing unit 143 into a rectangle to match the AF evaluation area (see, for example, step Sa10 in FIG. 7). Since the number Nrd of candidates for the main subject area is reduced, the area extracting unit 142 can shape the subject areas with the highest priority Nrd in a short time. In addition, since the number Nrd of candidates for the main subject area is reduced, the CPU 190 gives priority to the subject area in which the closest subject is imaged from among the reduced Nrd subject areas. The area can be focused in a short time (see, for example, step Sa12 in FIG. 7).

  Further, the area processing unit 143 may increase the priority (order) of the subject areas remaining after integration according to the number N of other subject areas integrated with the remaining subject areas. Here, the area processing unit 143 adds a score corresponding to the number N of other subject areas integrated into the remaining subject areas to the priority evaluation points of the subject areas remaining after integration. The subject area remaining after the integration is added to the evaluation score of the priority, so that, for example, the priority is increased from priority 4 to priority 3, so that the priority is compared with other subject areas. May be a higher subject area.

  FIG. 8 is a diagram for explaining an example of processing for preferentially selecting a subject area remaining after integration. In FIG. 8, based on the luminance, for example, the subject region 310 is extracted from the binarized image 300Y-1, the subject region 320 is extracted from the binarized image 300Y-2, and the subject region 383 is binarized image. Suppose that it was extracted from 300Y-3.

  In FIG. 8, it is assumed that, as an example, the subject region 381 is extracted from the binarized image 300Cr-1 and the subject region 340 is extracted from the binarized image 300Cr-3 based on the color difference (Cr). In FIG. 8, it is assumed that the subject region 382 is extracted from the binarized image 300Cr-1 as an example based on the color difference (Cb).

  Hereinafter, in an initial state where each subject area is extracted, for example, the subject area 340 has a priority 1 (rank 1), the subject area 310 has a priority 2 (rank 2), and the subject area 381 has a priority 3 (rank 3). The description will be continued assuming that the subject area 382 has priority 4 (rank 4), the subject area 320 has priority 5 (rank 5), and the subject area 383 has priority 6 (rank 6).

  Further, it is assumed that the region processing unit 143 integrates the subject region 381 and the subject region 382 into the subject region 310 as an example of the extracted subject regions. Hereinafter, the subject region 310 that remains after the subject region 381 and the subject region 382 are integrated is referred to as a “subject region 310a”. Here, the priority of the subject area 310 a may remain the priority 2 of the subject area 310.

  The region processing unit 143 selects a subject region in which subject regions of a predetermined number (for example, three) or more are integrated regardless of the priority. In FIG. 8, the area processing unit 143 selects the subject area 310 a that remains after integration regardless of the priority.

  Further, the area processing unit 143 sets the number of candidate areas Nrd = 3 of the main subject to a value 2 (= Nrd− (N−1)) according to the number N = 2 of the subject areas integrated into the subject area 310a. To reduce. The area processing unit 143 selects other subject areas in descending order of priority until the total number including the subject area 310a reaches Nrd = 2. In FIG. 8, the area processing unit 143 selects a total of two subject areas, a subject area 310a selected regardless of priority and a subject area 340 selected according to priority (priority 1). In this way, the area processing unit 143 selects a subject area based on the positional relationship between the subject areas extracted by the area extracting unit 142.

  Note that the area processing unit 143 may select subject areas in order of priority from the highest priority until the total number of subject areas reaches Nrd = 2 for all subject areas. Further, the area processing unit 143 may select only the subject area 310a (priority level 2) remaining after integration as a candidate for the main subject area.

  Further, the area processing unit 143 increases the priority of the subject area 310a remaining after the integration according to the number of subject areas (the subject area 381 and the subject area 382 in FIG. 8) integrated with the subject area 310a. Also good. The area processing unit 143 adds, for example, two points according to the number of subject areas integrated into the subject area 310a (= 2) to the priority evaluation points of the subject area 310a.

  The area processing unit 143 increases the priority 2 of the subject area 310a to the priority 1 by adding two points to the priority evaluation points of the subject area 310a. Accordingly, the area processing unit 143 reduces the priority 1 of the subject area 340 to, for example, the priority 2. Further, the area processing unit 143 may similarly reduce the priority for other subject areas.

  The area processing unit 143 selects up to Nrd = 2 subject areas whose priorities have been changed in order of priority. In FIG. 8, the area processing unit 143 selects a subject area 310a (new priority level 1) and a subject area 340 (new priority level 2). In this way, the area processing unit 143 selects a subject area based on the positional relationship between the subject areas extracted by the area extracting unit 142.

  As described above, the area processing unit 143 integrates the subject areas (for example, the subject area 310a) in which the subject areas of a predetermined number (for example, three) or more are integrated when the subject areas are integrated. Select as a candidate for the main subject area. Thereby, the area extraction device 140 can select the area of the main subject reliably and shorten the processing time for focusing on the area of the main subject.

  When the subject areas are integrated, the area processing unit 143 reduces the number of candidate areas Nrd of the main subject area according to the number N of the subject areas integrated with other subject areas. Thereby, the area extraction device 140 shapes the rectangle to fit the AF evaluation area (see step Sa10 in FIG. 7), reduces the number of subject areas, and reduces the processing time for focusing on the main subject area. Can be shortened.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  Further, by recording a program for realizing the region extraction device and the imaging device described above on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium, Execution processing may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 100 ... Imaging device, 110 ... Imaging part, 111 ... Lens barrel, 140 ... Area extraction apparatus, 141 ... Feature-value calculation part, 142 ... Area extraction part, 143 ... Area processing part, 144 ... Priority setting part

Claims (1)

A region extraction unit that extracts a subject region based on a feature amount from a captured image;
An area processing unit that selects the subject area based on the positional relationship of the subject areas extracted by the area extraction unit;
An area extraction apparatus comprising:

Images