JP6316006B2 - SUBJECT SEARCH DEVICE, ITS CONTROL METHOD, CONTROL PROGRAM, AND IMAGING DEVICE - Google Patents

Description

  The present invention relates to a subject search device, a control method thereof, a control program, and an imaging device, and more particularly to a method for searching a subject region in an image obtained as a result of shooting.

  In general, in an imaging device such as a digital camera, an image obtained as a result of imaging is divided into color regions, and a subject region is divided into areas with respect to the color region. An area to be trimmed is determined based on the result of area division and distance information obtained by autofocus (hereinafter referred to as AF).

  For example, a luminance value is extracted as a feature amount from the input image, and a luminance score indicating evaluation based on the distribution of the luminance value in the subject area of the input image is obtained. In addition, there is an imaging apparatus configured to show the evaluation of the shooting state of an input image according to the luminance score value (see Patent Document 1).

  Furthermore, an area estimated to correspond to a human face is extracted from the image data. Then, the imaging apparatus obtains the complexity of the image based on the number of the face areas, the degree of variation of the size of the face area, or the degree of dispersion of the average density of the face area, and obtains the exposure amount according to the complexity. (See Patent Document 2).

JP 2010-287948 A JP-A-9-146194

  In the above-described imaging device, when the boundary can be defined by color in the image obtained as a result of imaging, or when there is a distance difference between the subject region and another region in the image, the subject region is recognized. it can. Then, it is possible to automatically determine the subject area as a trimming area.

  However, when the area cannot be divided by color in the image or when there is no distance difference in the image, and in a shooting scene where it is difficult to obtain the distance of the trimming area that is the subject area, the photographer intends An area that is not to be used may be a trimming area.

  Accordingly, an object of the present invention is to provide a subject search device, a control method thereof, a control program, and an imaging device that can accurately determine a trimming region that is a subject region regardless of a shooting scene.

In order to achieve the above object, a subject search device according to the present invention is a subject search device that searches for a subject in an image obtained by imaging the subject, and calculates a complexity of a feature in the image Search means for searching for a subject area indicating the subject area in the image, and when the complexity exceeds a predetermined complexity threshold , the image is divided into a plurality of blocks with a predetermined first number of blocks. Thus, the search means performs a first subject search process for searching for the subject region based on a grouped block obtained by grouping the blocks according to a representative value representing the block in each of the blocks. , when the complexity is less than the complexity threshold, system to perform the first object search process is different from the second subject search processing in the search means And having a means.

  An image pickup apparatus according to the present invention includes an image pickup unit that picks up a subject and obtains an image, and the subject search device.

The control method according to the present invention is a control method for a subject search apparatus that searches for a subject in an image obtained by imaging the subject, a calculation step for calculating the complexity of features in the image, A search step for searching for a subject area indicating a subject area, wherein in the search step, when the complexity exceeds a predetermined complexity threshold , the image is divided into a plurality of blocks with a predetermined first number of blocks. Performing a first subject search process for searching the subject region based on a grouped block obtained by grouping the blocks according to a representative value representing the block in each of the blocks , When the complexity is equal to or less than the complexity threshold value, a second subject search process different from the first subject search process is performed.

The control program according to the present invention is a control program used in a subject search device for searching for a subject in an image obtained by capturing an image of a subject. And a search step for searching for a subject region indicating the subject region in the image. In the search step, when the complexity exceeds a predetermined complexity threshold , the image is The subject area is searched based on a grouped block obtained by dividing the block into a plurality of blocks with a predetermined first number of blocks and grouping the blocks according to representative values representing the blocks in each of the blocks. first performs a subject search processing, the object complexity is the first of less than or equal to the complexity threshold for And performing body search process different from the second object search process.

  According to the present invention, since the search process when searching for a subject is changed according to the determination result of the complexity of the image, the trimming region that is the subject region can be accurately determined regardless of the shooting scene. Can do.

It is a block diagram which shows the structure about an example of an imaging device provided with the to-be-photographed object search apparatus by embodiment of this invention. 3 is a flowchart for explaining a photographing process in the camera shown in FIG. 1. It is a figure for demonstrating calculation of the complexity performed with the camera shown in FIG. 1, (a) is a figure which shows the image obtained as a result of imaging | photography, (b) divided the image shown in (a) into the area | region The figure which shows a state, (c) is a figure which shows the state which counts the area | region where an edge component is high. 2A and 2B are diagrams for explaining main subject search processing performed by the camera shown in FIG. 1, in which FIG. 1A is a diagram showing an image obtained as a result of shooting, and FIG. 2B is a diagram in which the image shown in FIG. FIG. 8C is a diagram illustrating an example of a state, FIG. 10C is a diagram illustrating another example of a state in which the image illustrated in FIG. 9A is divided into regions, and FIG.

  Hereinafter, an example of an imaging apparatus including a subject searching apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an example of an imaging apparatus including a subject search device according to an embodiment of the present invention.

  The illustrated imaging apparatus is, for example, a digital camera (hereinafter simply referred to as a camera), and includes an imaging element 101 such as a CCD or CMOS sensor. For example, an RGB color filter based on a Bayer array is arranged on the surface of the image pickup device 101, and color imaging can be performed.

  A photographing lens unit (hereinafter simply referred to as a photographing lens) 100 including a focus lens, a diaphragm, a shutter, and the like is disposed on the front side of the imaging element 101, and an optical image is connected to the imaging element 101 via the photographing lens. Image. The image sensor 101 generates an electrical signal (analog image signal) corresponding to the optical image, and outputs the analog image signal as a digital image signal by A / D conversion. The digital image signal is once recorded in the memory 102.

  The CPU 114 controls the entire camera. For example, as will be described later, the CPU 114 obtains a shutter speed and an aperture value at which an image obtained as a result of photographing becomes brighter, and obtains a driving amount of a focus lens for focusing on the subject.

  The exposure value (shutter speed and aperture value) obtained by the CPU 114 and the focus lens drive amount are sent to the control circuit 113, which controls the shutter, aperture, and focus lens according to the exposure value and the focus lens drive amount. Drive control.

  Furthermore, the control circuit 113 performs pixel readout control of the image sensor 101 under the control of the CPU 114, and causes the strobe 116 to emit light in synchronization with imaging as necessary.

  The white balance (WB) control unit 103 reads a digital image signal stored in the memory 102 and obtains a WB gain value according to the digital image signal. The WB control unit 103 corrects the WB gain for the digital image signal using the WB gain value, and outputs a WB corrected image signal.

  A color conversion matrix (MTX) circuit 104 multiplies the WB corrected image signal by a color gain to obtain color difference signals RY and BY in order to reproduce the WB corrected image signal with an optimum color. The low-pass filter (LPF) circuit 105 limits the bands of the color difference signals RY and BY and supplies them to a CSUP (Chroma Suppress) circuit 106.

  The CSUP circuit 106 suppresses the false color signal in the saturated portion in the image signal band-limited by the LPF circuit 105.

  On the other hand, the WB correction image signal is given from the WB control unit 103 to the luminance signal (Y) generation circuit 111. The Y generation circuit 111 generates a luminance signal Y from the WB corrected image signal. The edge enhancement circuit 112 performs edge enhancement processing on the luminance signal Y and outputs an edge enhanced luminance signal Y.

  The color difference signals RY and BY output from the CSUP circuit 106 and the edge enhanced luminance signal Y output from the edge enhancement circuit 112 are supplied to the RGB conversion circuit 107. The RGB conversion circuit 107 converts the color difference signals RY and BY and the edge enhanced luminance signal Y into RGB signals (color signals). Then, the gamma (γ) correction circuit 108 performs gradation correction on the RBG signal, and sends the RGB signal after gradation correction to the color luminance conversion circuit 109.

  The color luminance conversion circuit 109 is a combination of the RGB signal after gradation correction, the color information as the luminance signal (Y), the difference between the luminance signal and the blue component (U), and the difference between the luminance signal and the red component (V). Is converted into a YUV signal represented by

  The JPEG compression circuit 110 compresses the YUV signal and records the compressed YUV signal as image data on an external recording medium or an internal recording medium (not shown).

  The image processing circuit 115 reads out a digital image signal from the memory, performs predetermined image processing, and displays it as an image on a display unit (not shown).

  FIG. 2 is a flowchart for explaining a photographing process in the camera shown in FIG. Note that the processing according to the illustrated flowchart is performed under the control of the CPU 114.

  In the camera, when the start of shooting is instructed by an operation unit (not shown), under the control of the CPU 114, the image processing circuit 115 generates image data for live display according to the digital image signal obtained by the image sensor 101. The live display image (live image) is displayed on the display unit (step S201: live image acquisition).

  Subsequently, the CPU 114 calculates the complexity (live complexity) in the live image as described later (step S202). Then, the CPU 114 determines whether the complexity exceeds a predetermined complexity threshold and obtains a determination result. That is, the CPU 114 determines whether or not the live image is complicated (step S203).

  If the live image is complicated (YES in step S203), the CPU 114 performs a first main subject search process (standard), which will be described later, assuming that the shooting scene is complicated (step S204). In the process of step S204, the live image is divided and a main subject (that is, a subject region) is searched with a standard number of mesh divisions.

  On the other hand, if the live image is not complicated (NO in step S203), the CPU 114 performs a second main subject search process (details) described later, assuming that the shooting scene is not complicated (step S205). The processing in step S205 is used in the case of a shooting scene in which the complexity of the live image is low due to low complexity and it is difficult to recognize the subject in a normal main subject search. Here, a large number of mesh divisions are set to capture small features of the subject, and the main subject (that is, the subject region) is searched using the luminance signal Y.

  The processing in steps S201 to S205 shown in the figure is a shooting preparation sequence, and when the shooting preparation sequence ends, the CPU 114 shifts to a development sequence.

  Subsequent to step S204 or S205, the CPU 114 determines the complexity of the image in the same manner as in step S203 (step S206). If the image is not complicated (NO in step S206), CPU 114 determines whether or not the main subject has been searched in the first or second main subject search process (step S207).

  If it is determined that the main subject has been found (YES in step S297), CPU 114 performs bracket shooting (step S208). Here, for example, the CPU 114 performs AF bracket shooting for changing the focus position or AE bracket shooting for changing the exposure.

  If the image is complicated (YES in step S206), the CPU 114 proceeds to the process of step S208 and performs bracket shooting. Here, since the complexity of the image is high, there are subjects having various characteristics in the image. For this reason, it is possible to perform imaging by changing standard imaging parameters.

  If it is determined that the main subject cannot be searched (NO in step S297), the CPU 114 performs single shooting (single shooting) on the assumption that the image characteristics are poor and effective shooting by changing shooting parameters cannot be expected (step shooting). S209).

  Subsequent to the processing in step S208 described above, the CPU 114 calculates the complexity (still image complexity) for the still image (that is, the captured image) obtained by bracket shooting in the same manner as in step S202 (step S210). .

  When calculating the complexity when a plurality of still images are obtained by bracket shooting, the complexity is calculated for all the still images obtained by bracket shooting, and the average value is obtained. Alternatively, the lowest complexity among a plurality of complexity may be selected. Furthermore, the complexity may be obtained from the representative image in the still image obtained by bracket shooting.

  Here, if bracket shooting is AE bracket shooting, a still image with proper exposure is selected as a representative image. If bracket shooting is AF bracket shooting, a still image focused on the first focus position is selected as a representative image.

  Subsequently, the CPU 114 determines whether or not the still image (or representative image) is complicated according to the still image complexity (step S211). That is, the CPU 114 determines whether the still image complexity exceeds a predetermined threshold value.

  If the still image is complicated (YES in step S211), the CPU 114 performs a first main subject search process (step S212). On the other hand, if the still image is not complicated (NO in step S211), the CPU 114 performs a second main subject search process (step S213). Note that after the process of step S209, the CPU 114 proceeds to the process of step S213.

  After the process of step S212 or S213, the CPU 114 determines whether or not the main subject has been searched for in the still image (step S214).

  If it is determined that the main subject has been searched for in the still image (YES in step S214), the CPU 114 performs image processing for cutting out the main subject by the image processing circuit 115 and trimming a region including the main subject (step S215). Then, the CPU 114 ends the photographing process.

  On the other hand, if it is determined that the main subject cannot be searched for in the still image (NO in step S214), the CPU 114 does not perform trimming on the assumption that the feature of the still image is poor. Performs image composition processing with graphic elements. Then, the CPU 114 ends the photographing process.

  Note that the processing from step S206 to step S216 is a development sequence.

  FIG. 3 is a diagram for explaining the calculation of the complexity performed by the camera shown in FIG. FIG. 3A is a diagram showing an image (original image) obtained as a result of photographing, and FIG. 3B is a diagram showing a state where the image shown in FIG. 3A is divided into regions. FIG. 3C is a diagram showing a state in which regions with high edge components are counted.

  The illustrated image 301 has a gray tone as a whole, and a black and white logo (east side) is the main subject. The CPU 114 detects the luminance edge of the image 301 and extracts an edge component (high frequency component). Then, the CPU 114 performs mesh division with a predetermined number of mesh divisions on the image from which the edge component has been extracted to obtain an image 302. Here, the image 302 is divided into 30 rectangular blocks (areas) 304.

  Subsequently, the CPU 114 integrates the edge component for each of the blocks 304 in the image 302 to obtain an integral value. Then, the CPU 114 normalizes the integral value with the number of pixels of the block and obtains an edge evaluation value of the block.

  Next, the CPU 114 counts the number of blocks having a high edge evaluation value in accordance with the edge evaluation value of the block in the image 303 after calculating the edge evaluation value to obtain the count number. Here, the CPU 114 counts the number of blocks whose edge evaluation value exceeds a predetermined edge threshold.

  In the illustrated example, two blocks including a logo (east side) which is a main subject are blocks having a high edge evaluation value, and the CPU 114 counts the number of blocks having a high edge evaluation (here, the number of blocks having a high edge evaluation value ( That is, the count number) = 2) is defined as the complexity of the image 301.

  As described above, the CPU 114 determines whether the complexity exceeds a predetermined complexity threshold. That is, when the number of high edge evaluation blocks is small, the CPU 114 determines that the complexity of the image 301 is low and the image has poor features.

  FIG. 4 is a diagram for explaining a main subject search process performed by the camera shown in FIG. 4A is a diagram illustrating an image (original image) obtained as a result of photographing, and FIG. 4B is a diagram illustrating an example of a state where the image illustrated in FIG. 4A is divided into regions. is there. FIG. 4C is a diagram showing another example of a state where the image shown in FIG. 4A is divided into regions, and FIG. 4D is a diagram showing grouping of regions in FIG. 4C. is there.

  The illustrated image 401 has a gray tone as a whole, and a black and white logo (east side) is the main subject. The CPU 114 performs mesh division on the image 401 with a predetermined first mesh division number. Here, the image 402 is divided into 30 rectangular blocks (areas) 406.

  Next, the CPU 114 obtains a representative color for each of the blocks. In the illustrated example, the CPU 114 integrates the color signal values of the block to obtain an integrated value, and then normalizes the integrated value by the number of pixels to determine a representative color that is a representative value of the block.

  Furthermore, the CPU 114 groups the blocks that approximate the representative colors to obtain grouped blocks. In the image 402, the CPU 114 cannot perform grouping because the representative colors approximate in all blocks of the image. That is, the CPU 114 cannot search for a logo (east side) that is a main subject.

  The example shown in FIG. 4B is an example when the first main subject search process is performed. That is, when the complexity exceeds a predetermined complexity threshold, the main subject search process is performed by the division shown in FIG. On the other hand, since the complexity of the image 401 is low here (that is, the complexity is equal to or lower than the complexity threshold value), the main subject cannot be searched in the first main subject search process. Therefore, here, the second main subject search process shown in FIG. 4C is performed.

  The CPU 114 performs mesh division on the image 401 with a predetermined second mesh division number. Here, the image 402 is divided into 120 rectangular blocks (areas) 407.

  Next, the CPU 114 obtains a luminance evaluation value that is a representative value for each of the blocks. In the illustrated example, the CPU 114 integrates the luminance signal value of the block to obtain an integrated value, and then normalizes the integrated value with the number of pixels to determine the luminance evaluation value of the block.

  Further, the CPU 114 obtains a grouped block by grouping blocks having similar brightness evaluation values. In the state shown in FIG. 4D, the three blocks 405 whose luminance evaluation values approximate in the image 404 are grouped blocks, and the CPU 114 can search for a logo (east side) that is a main subject. That is, when the complexity of the image is low, the CPU 114 performs a second main subject search process to search for the main subject.

  By the way, in the second main subject search process shown in FIGS. 4C and 4D, the luminance evaluation value is obtained for each block as a representative value and grouping is performed. When grouping is performed using the luminance evaluation value, there are many small groups, which is not suitable for the main subject search. However, when the features in the image are poor (that is, when the complexity is low), if the number of mesh divisions is increased and block grouping is performed according to the luminance evaluation value, the main subject can be searched well. it can.

  As described above, in the embodiment of the present invention, it is difficult to divide the area by color, and the main subject in an image obtained in a shooting scene where it is difficult to obtain features, such as a case where the distance difference between subjects in the image is small. Can be trimmed or bracketed.

  In the above-described embodiment, when calculating the complexity, the entire image is handled uniformly. However, in the image, weights are given to the central portion and the peripheral portion where the main subject is likely to be located. The complexity may be obtained by changing the attachment.

  Furthermore, in the above-described embodiment, when the complexity is low and the main subject cannot be searched, the image composition process is performed. However, the process of transforming the image into a trapezoidal shape or the cross filter process is performed. It may be.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and the control method may be executed by the subject search device. Further, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the subject search device. The control program is recorded on a computer-readable recording medium, for example.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

DESCRIPTION OF SYMBOLS 101 Image pick-up element 103 WB control part 104 Color conversion MTX circuit 107 RGB conversion circuit 108 Gamma correction circuit 109 Color luminance conversion circuit 111 Y generation circuit 112 Edge emphasis circuit 113 Control circuit 114 CPU
115 Image processing circuit

Claims (10)

A subject search device for searching for a subject in an image obtained by imaging the subject,
Calculating means for calculating the complexity of features in the image;
Search means for searching a subject area indicating the area of the subject in the image;
When the complexity exceeds a predetermined complexity threshold , the image is divided into a plurality of blocks with a predetermined first number of blocks, and the blocks are grouped according to representative values representing the blocks in each of the blocks. the first object search process of searching for the subject region based on the grouping blocks obtained by performed the search means, wherein the complexity is not more than the complexity threshold, the first object search Control means for causing the search means to perform a second subject search process different from the process;
A subject search device characterized by comprising:   The calculation means divides the image into a plurality of blocks, extracts edge components of the blocks in each of the blocks, and counts the number of blocks whose edge evaluation value corresponding to the edge components exceeds a predetermined edge threshold value. The subject search apparatus according to claim 1, wherein the count is the complexity. Representative value used in the first object search process, according to claim 1, characterized in that the representative color obtained by normalizing the integral value obtained by integrating the color signal values for each of the blocks in the number of pixels Or the subject search device according to 2; The second subject search process divides the image into a plurality of blocks with a second number of blocks larger than the first number of blocks, and according to a representative value representing the block in each of the blocks object search device according to any one of claims 1 to 3, characterized in that a process of searching for the subject region based on the grouping blocks obtained by grouping the blocks. The representative value used in the second subject search process is a luminance evaluation value obtained by normalizing an integrated value obtained by integrating luminance signal values for each block by the number of pixels. 5. A subject search device according to 4 . Imaging means for capturing an image of a subject and obtaining an image;
The subject search device according to any one of claims 1 to 5 ,
An imaging device comprising: The imaging apparatus according to claim 6 , further comprising an image processing unit configured to perform predetermined image processing on the subject area when the subject area is searched for by the search unit included in the subject search apparatus. When the subject region is searched by the search means provided in the subject search device, an imaging apparatus according to claim 6 or 7 characterized by having a photographing control means for controlling to bracketing said imaging means . A method for controlling a subject search apparatus for searching for a subject in an image obtained by imaging a subject,
A calculating step for calculating the complexity of the feature in the image;
Searching for a subject area indicating the subject area in the image, and
In the searching step, when the complexity exceeds a predetermined complexity threshold value , the image is divided into a plurality of blocks with a predetermined first number of blocks, and a representative value representing a block in each of the blocks is determined. First subject search processing for searching for the subject region based on the grouped blocks obtained by grouping the blocks, and the first subject search processing when the complexity is equal to or less than the complexity threshold value. And performing a second subject search process different from the above. A control program used in a subject search apparatus for searching for a subject in an image obtained by imaging a subject,
A computer provided in the subject search device,
A calculating step for calculating the complexity of the feature in the image;
Searching for a subject area indicating the subject area in the image; and
In the searching step, when the complexity exceeds a predetermined complexity threshold value , the image is divided into a plurality of blocks with a predetermined first number of blocks, and a representative value representing a block in each of the blocks is determined. First subject search processing for searching for the subject region based on the grouped blocks obtained by grouping the blocks, and the first subject search processing when the complexity is equal to or less than the complexity threshold value. A control program for performing a second subject search process different from the above.