JP2012119994A - Image processing system, control method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a region concerning a desired subject as a main subject without being divided.SOLUTION: A control method of an image processing system comprises processes of: dividing each image imaged in succession into a plurality of blocks; generating a histogram of the image based on signal values of each block; dividing the generated histogram into a plurality of sections; setting regions so that adjacent blocks, among a plurality of blocks of the image, whose signal values belong to a same section become one region; calculating an evaluation value for each region; and determining whether a second region, having an evaluation value whose difference from the highest evaluation value is less than or equal to a prescribed threshold, exists or not. An image processing system determines a first region having the highest evaluation value as a main subject region if any second region does not exist and determines a region combining a first and a second regions as a main subject region if a second region exists.

Description

  The present invention relates to a technique for determining a main subject in continuously captured images.

  An imaging device such as a digital camera or a digital video camera can focus on a desired subject, and the subject captured within an imaging angle of view can be easily captured so that the subject is properly exposed. Some have a function of automatically recognizing the main subject.

  The subject captured within the imaging angle of view can be obtained by dividing the distribution derived from the histogram of hue and saturation in the captured image into sections and classifying the captured images for each section. It can be recognized as each area. Then, by calculating an evaluation value for each recognized subject, the region having the highest evaluation value can be determined as the main subject.

  In Patent Document 1, a histogram of a plurality of color components created for a captured image is divided into mountain-shaped distribution ranges, and the captured images are classified into regions belonging to the same combination of sections. A technique for recognizing a subject is disclosed.

JP 2007-110638 A

  However, when the histograms are grouped into mountain-shaped distribution ranges as in Patent Document 1, the main subject region may not be appropriately determined depending on the grouping classification method. For example, as shown in FIG. 5A, when two local minimum points in the histogram are set as boundaries for grouping, the entire region of the car can be appropriately recognized as the main subject. However, as shown in FIGS. 5 (b) and 5 (c), when three or four local minimum points are set as grouping boundaries, the area of the car to be recognized as the main subject is a different area. It will be classified.

  If the main subject area is not properly selected in this way, the shooting setting is performed so that only a part of the subject to be recognized as the main subject is in focus and proper exposure. An image shot with a desired shooting setting could not be obtained.

  In addition, it is not practical to store in advance a histogram reference that can appropriately classify the main subject region for every possible subject and shooting conditions such as lighting conditions, because this increases the circuit scale. .

  The present invention has been made in view of the above-described problems, and an object of the present invention is to recognize a region related to a desired subject as a main subject without being divided.

In order to achieve the above object, an image processing apparatus of the present invention comprises the following arrangement.
Based on the input means for sequentially acquiring images captured successively, the block dividing means for dividing the image acquired by the input means into a plurality of blocks, and the representative values of the blocks divided by the block dividing means Generating means for generating a histogram related to the color components, section dividing means for dividing the histogram generated by the generating means into a plurality of sections, and among the plurality of blocks, the representative values belong to the same section and are adjacent to each other. A setting means for setting a block as one area, an evaluation means for calculating an evaluation value for each of the areas set by the setting means, and a main subject area is determined based on the evaluation value calculated by the evaluation means Determining means, and the determining means has an evaluation value whose difference from the highest evaluation value among the evaluation values of the area is equal to or less than a predetermined threshold value. If the second area does not exist, the first area having the highest evaluation value is determined as the main subject area, and if the second area exists, the first area and the second area are determined. The combined area is determined as a main subject area.

  With such a configuration, according to the present invention, an area related to a desired subject can be recognized as a main subject without being divided.

1 is a block diagram showing a functional configuration of a digital camera according to an embodiment of the present invention. The figure for demonstrating the process performed in the area | region setting part which concerns on embodiment. Another figure for demonstrating the process performed in the area | region setting part which concerns on embodiment The figure for demonstrating the evaluation value of the area | region which concerns on embodiment The figure which showed the example of a division | segmentation of the to-be-photographed area | region by the section division of the histogram which concerns on embodiment Flowchart of main subject determination processing according to the embodiment The figure which showed the calculation example of the evaluation value by the area division of the histogram which concerns on embodiment

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example in which the present invention is applied to a digital camera capable of recognizing and tracking a main subject area as an example of an image processing apparatus will be described. However, the present invention can be applied to any device that can recognize the main subject region.

FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.
The control unit 101 is a CPU, for example, and controls the operation of each block included in the digital camera 100. For example, the control unit 101 controls the operation of each block by reading out an operation program of each block included in the digital camera 100 from the ROM 102, developing the program in the RAM 103, and executing the program. The ROM 102 is a rewritable non-volatile memory that stores parameters necessary for the operation of each block, shooting settings set by the user, and the like in addition to the operation program for each block provided in the digital camera 100. The RAM 103 is a volatile memory, and is a storage area used as an area for temporarily storing data output in the operation of each block, for example.

  In the present embodiment, the operation of each block included in the digital camera 100 is described as being controlled by the operation program of the block. It may be composed of a circuit that performs the above.

  The image pickup unit 105 is an image pickup device such as a CCD or a CMOS sensor, for example, and photoelectrically converts an optical image formed on the image pickup device via the optical system 104 to thereby convert an analog image signal to an A / D conversion unit described later. It outputs to 106. The imaging unit 105 can realize an electronic viewfinder by displaying analog image signals sequentially acquired by repeating photoelectric conversion as video on a display unit (not shown). In the present embodiment, subject detection processing or main subject region tracking processing is performed on the sequentially acquired analog image signals.

  The A / D conversion unit 106 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data. The image processing unit 107 is a block that applies predetermined image processing to input digital image data. In the present embodiment, the image processing unit 107 performs conversion processing from the RGB space to the HLS space for the digital image data output from the A / D conversion unit 106, and the hue (H) and saturation (S) of each pixel. ), To obtain the luminance (L) value.

  The area setting unit 108 divides the digital image data subjected to the HLS conversion by the image processing unit 107 into blocks of a predetermined size (block division) as shown in FIG. The average value (representative value) of the hues of the pixels inside is calculated. Then, a hue histogram as shown in FIG. 2B is generated from the hues of all blocks. In the hue histogram, the area setting unit 108 has a point where the color difference value is 0 as shown in FIG. 2C, or the ratio of the local maximum point with respect to the local minimum point is greater than or equal to a predetermined ratio. The hue is divided so as to be different sections with the minimum point as a boundary (section division). Then, the area setting unit 108 determines which section each block of the image belongs to which representative value of the block belongs, and sets blocks that belong to the same section and are adjacent to each other as one area. Specifically, when the image blocks are classified as shown in FIG. 3A, the region setting unit 108 sets a region composed of blocks in the same section as shown in FIG.

  In the present embodiment, a description will be given assuming that a histogram is generated for the hue of an image and section division is performed, but the present invention is not limited to this. For example, other color components such as saturation and luminance may be used, and the present invention may perform section division using a histogram generated for at least one color component.

The main subject determination unit 109 calculates an evaluation value for each region of the image set by the region setting unit 108, and determines at least the region having the highest evaluation value as the main subject. The evaluation value calculated for each region of the image is calculated using, for example, at least one of the following items.
・ Distance between the center of gravity of the area and the center position of the image ・ Size of the area ・ Aspect ratio of the circumscribed rectangle of the area (long side / short side)
Area peripheral length Area color information Area brightness information

In the present embodiment, values calculated from relational expressions (graphs) obtained experimentally for the distance between the center of gravity position of the region and the center position of the image, the size of the region, and the aspect ratio of the circumscribed rectangle of the region, respectively. Accumulated and used as an evaluation value. Specifically, with respect to the distance between the center of gravity position of the region and the center position of the image, as shown in FIG. The evaluation value S _dist is set high. The size of the region is such that the evaluation value S _size is increased as the region is larger than the amount shown in FIG. 4B, and the evaluation value is maximized for a region larger than the predetermined size. Set. The aspect ratio of the circumscribed rectangle of the area is set based on the shape of the circumscribed rectangle of the area where the object assumed as the subject is recognized. For example, in the case where two persons are photographed at a position close to the digital camera 100, a subject that is present behind the two persons and exists behind the two persons is shaped as a circumscribed rectangle of the area. Becomes portrait or landscape. In order to lower the evaluation value for an object that is difficult to be determined as the main subject as described above, the evaluation value S _aspect increases as the aspect ratio of the circumscribed rectangle of the region is closer to 1, as shown in FIG. Set to.

  In the present embodiment, in order to reduce the calculation amount, the control unit 101 may exclude an area such as a background from the evaluation value calculation target. Specifically, the control unit 101 determines that an area in contact with at least one of the upper side and the side edge of the image is not the background or the subject that is completely framed in and is evaluated. Exclude from value calculation.

The main subject determination unit 109 assigns numbers to regions in descending order of the evaluation value S (= S _dist × S _size × S _aspect ) of each region that can be the main subject. The main subject determining section 109 determines a region having at least the highest evaluation values S ₁ as the main subject.

  The operation input unit 111 is a user interface provided in the digital camera 100 such as a release button or a menu button, for example, analyzes a signal input by the user operating the user interface, and transmits the operation content to the control unit 101. To do. The recording medium 110 is a recording device that is detachably connected to the digital camera 100 such as a built-in memory included in the digital camera 100, a memory card, or an HDD. The recording medium 110 records an image photographed by the imaging unit 105 when the user gives a photographing instruction by operating a release button, for example.

(Main subject determination process)
A specific process of the main subject determination process of the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the control unit 101 reading, for example, a corresponding processing program stored in the ROM 102, developing the program in the RAM 103, and executing the program. The main subject determination process is started when, for example, the digital camera 100 is activated and through display, which is a video display on a display unit (not shown), is started and executed for each image obtained by imaging. It will be described as a thing.

  In step S <b> 601, the control unit 101 causes the region setting unit 108 to output the image captured by the imaging unit 105 and converted into the HLS space by the image processing unit 107, and causes the region setting unit 108 to block the image with a predetermined size. To divide.

  In step S602, the control unit 101 causes the region setting unit 108 to generate a hue histogram of the image based on the representative values of the plurality of blocks divided in step S601, and divides the distribution of the hue histogram into a plurality of sections. Further, the control unit 101 stores the generated hue histogram in the RAM 103.

  In step S <b> 603, the control unit 101 determines whether the hue histogram generated for the image captured immediately before the image currently being processed, that is, the image captured immediately before in time series order, is stored. In this case, the difference of the hue histogram between the images is calculated. The difference of the hue histogram is calculated by subtracting the distribution number for each section of the hue, and the control unit 101 stores the difference of the hue histogram in the RAM 103 in association with the image currently being processed.

  In step S <b> 604, the control unit 101 determines a plurality of blocks based on the hue histogram section determined in step S <b> 602 so that a plurality of blocks belong to the same section and adjacent blocks form one area. The setting unit 108 is caused to set a region for each subject in the image. In this step, the control unit 101 excludes the region that is difficult to become the main subject such as the background as described above from the target region in the following steps.

In step S <b> 605, the control unit 101 causes the main subject determination unit 109 to calculate an evaluation value for each region set by the region setting unit 108, and assigns a number to the region in descending order of the obtained evaluation value. At this time the main subject determining section 109 determines the region with the highest evaluation values S ₁ as the main subject region (first region).

  In step S <b> 606, the control unit 101 determines whether or not there is an area (second area) having an evaluation value that is equal to or less than a predetermined threshold among the areas in the image and the evaluation value of the main subject area. to decide. For example, as shown in FIGS. 7B and 7C, the area related to the subject to be recognized as the main subject is determined as the main subject area without being divided as shown in FIG. When the area divided into two is determined as the main subject area, the evaluation value of the area is low. At this time, the evaluation value of the area not determined as the main subject area among the areas related to the subject to be recognized as the main subject has a high similarity between the area and the main subject area. The difference between and becomes smaller. That is, even when the region related to the subject to be recognized as the main subject is divided, the difference between the evaluation values between the regions is determined so as to determine that each region is included in the region related to the subject. An appropriate threshold value is set in advance.

  The control unit 101 moves the process to S607 if there is an area in the image that has an evaluation value that is equal to or less than a predetermined threshold value in the image with respect to the evaluation value of the main subject area. If not, the control unit 101 determines in S605. The main subject area information is output to complete the main subject determination process.

  In step S <b> 607, the control unit 101 determines whether or not an area having an evaluation value that is equal to or less than a predetermined threshold value is different from the evaluation value of the main subject area detected in step S <b> 606. . The control unit 101 moves the process to S608 when an area having an evaluation value whose difference from the evaluation value of the main subject area is equal to or less than a predetermined threshold is adjacent to the main subject area, and when not adjacent to the main subject area. Information on the main subject area determined in S605 is output, and the process is completed.

  In step S <b> 608, the control unit 101 calculates the distance between the center of gravity of an area having a difference between the evaluation value of the main subject area and the evaluation value equal to or less than a predetermined threshold and the center of gravity of the main subject area. The distance between the centroids is stored in the RAM 103 in association with the image.

  In step S <b> 609, the control unit 101 determines whether the region having an evaluation value that is equal to or smaller than a predetermined threshold value is a part of the region related to the subject to be recognized as the main subject. The main subject determination unit 109 determines whether or not. Specifically, the control unit 101 reads out information on the difference of the hue histogram and information on the distance between the centroids stored in association with each of all the images captured in a predetermined period from the RAM 103, and stores the main subject. Input to the determination unit 109. First, the main subject determination unit 109 determines whether or not the difference of the hue histogram is less than or equal to a predetermined value in each section in all images in a predetermined period, that is, whether or not the hue histogram is stable during the predetermined period. Judging. Further, the main subject determination unit 109 determines the difference between the center of gravity of the main subject region and the region having an evaluation value that is equal to or less than a predetermined threshold value in all images in a predetermined period. It is determined whether the standard deviation of the distance is less than a predetermined value. That is, the main subject determination unit 109 can be regarded that there is no change in the positional relationship between the main subject region and the region having an evaluation value equal to or less than a predetermined threshold value during a predetermined period. Determine whether or not. If the main subject determination unit 109 determines that there is no change in the positional relationship and the hue histogram is stable in a predetermined period, the main subject determination unit 109 sets an evaluation value whose difference from the evaluation value is equal to or less than a predetermined threshold value. It is determined that the area that is included is part of the area that is recognized as the main subject. That is, it is determined that the area related to the subject to be recognized as the main subject is divided.

  If the main subject determining unit 109 determines that the area related to the subject to be recognized as the main subject is divided, the control unit 101 moves the process to S610, and otherwise moves the process to S611. In this step, if there is no information on the difference of the associated hue histogram or information on the distance between the centroids for all the images in the predetermined period, the process is not executed, and the main subject area determined in S605 And the main subject determination process may be completed.

  In step S <b> 610, the control unit 101 causes the main subject determination unit 109 to add a region that includes a main subject region and a region having an evaluation value that is equal to or less than a predetermined threshold value to the main subject determination unit 109. Let me decide as. Then, the control unit 101 outputs information on the newly determined main subject area, and completes the main subject determination process.

  In step S611, the control unit 101 causes the region setting unit 108 to classify the distribution of hue histograms into a plurality of new groups by a method different from the method performed in step S602. Specifically, the region setting unit 108 classifies the reference value of the ratio of the local maximum point that sandwiches the local minimum point with respect to the local minimum point set to determine the local minimum point as a boundary in the hue histogram in step S602. Classification is performed by setting a value higher than the reference value. That is, the region setting unit 108 increases the reference value for determining the minimum point as the boundary so that the number of groups into which the hue histogram is classified decreases. Then, the control unit 101 returns the process to S604 and again detects the main subject area.

In the present embodiment, when there is a second region that has an evaluation value that is equal to or less than a predetermined threshold value, the difference from the evaluation value of the main subject region (first region),
The hue histogram is stable for a predetermined period. The first area and the second area are adjacent. The distance between the centroid of the first area and the centroid of the second area is predetermined. Whether or not the first region and the second region are set as the main subject region is determined on the condition that the fluctuation is small, but the embodiment of the present invention is not limited to this. That is, for example, when only one person is known in advance as a subject, it will be easily understood that the above-described additional conditions need not be considered.

  As described above, the image processing apparatus according to the present embodiment can recognize a region related to a subject desired by a photographer as a main subject without being divided. Specifically, the image processing apparatus divides the image into a plurality of blocks for each of the continuously captured images, and generates an image histogram based on the signal value of each block. Then, the generated histogram is divided into a plurality of sections, and regions of the image are set so that a plurality of blocks of the image belong to a section having the same signal value and adjacent blocks become one region. To do. Further, an evaluation value is calculated for each of the areas, and it is determined whether or not there is a second area having an evaluation value whose difference from the highest evaluation value is equal to or less than a predetermined threshold value. When the second area does not exist, the image processing apparatus determines the first area having the highest evaluation value as the main subject area, and when the second area exists, the first area and the second area are determined. The area combined with the area is determined as the main subject area.

  In this way, even if the main subject is divided into different regions depending on the lighting conditions and the state of the subject, the region related to the subject that should be properly recognized as the main subject is set as the main subject region. Can be determined. That is, since the entire area of the subject that should be recognized as the main subject is recognized, it is possible to shoot with appropriate shooting settings.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

Input means for sequentially acquiring images captured successively;
Block dividing means for dividing the image acquired by the input means into a plurality of blocks;
Generating means for generating a histogram relating to color components of the image based on representative values of blocks divided by the block dividing means;
Section dividing means for dividing the histogram generated by the generating means into a plurality of sections;
Of the plurality of blocks, setting means for setting, as one area, blocks that belong to the section having the same representative value and are adjacent to each other;
Evaluation means for calculating an evaluation value for each of the areas set by the setting means;
Determining means for determining a main subject region based on the evaluation value calculated by the evaluation means,
The determination means has the highest evaluation value when there is no second region having an evaluation value equal to or lower than a predetermined threshold value with a difference from the highest evaluation value among the evaluation values of the region. A first area is determined as the main subject area, and when the second area is present, a combination of the first area and the second area is determined as the main subject area. A featured image processing apparatus.   The determining means determines, when the first area and the second area are adjacent to each other, an area that is a combination of the first area and the second area as the main subject area. The image processing apparatus according to claim 1.   The determination means includes the second region in all images captured in a predetermined period, and a standard deviation of a distance between the centroid of the first region and the centroid of the second region is a predetermined value. 3. The image processing apparatus according to claim 1, wherein an area obtained by combining the first area and the second area is determined as the main subject area when the following is true:   The determining means, when the difference between the histograms between consecutive images is not less than or equal to a predetermined value in each section in all images captured in the predetermined period, the first area and the second area The image processing apparatus according to claim 3, wherein the combined area is not determined as the main subject area.   The interval dividing means is a minimum point of the histogram, and the ratio of the maximum point sandwiching the minimum point with respect to the minimum point is a predetermined point or more, or the value of the histogram is 0 The image processing apparatus according to claim 3, wherein the image processing apparatus is divided so as to be different sections with a point to be a boundary as a boundary.   The section dividing means includes the second area in all the images captured in the predetermined period, and the area obtained by combining the first area and the second area is determined by the determining means. 6. The image processing apparatus according to claim 5, wherein when the main subject area is not determined, the histogram is divided by increasing a predetermined ratio for determining the minimum point as the boundary.   The evaluation value calculated for each of the regions is at least one of the distance, size, circumscribed rectangle aspect ratio, peripheral length, color information, and luminance information between the center of gravity and the center position of the image. The image processing apparatus according to claim 1, wherein the image processing apparatus is calculated using   The image processing apparatus according to claim 1, wherein the histogram is a histogram of at least one of hue, saturation, and luminance of the block. An input step in which the input means sequentially acquires images taken successively;
A block dividing unit that divides the image acquired in the input step into a plurality of blocks; and
A generating step of generating a histogram related to the color components of the image based on a representative value of the block divided in the block dividing step;
A section dividing step for dividing the histogram generated in the generating step into a plurality of sections;
A setting step in which a setting unit sets, as one area, blocks adjacent to and belonging to the same section of the plurality of blocks;
An evaluation unit calculates an evaluation value for each of the areas set in the setting step;
A determination unit that determines a main subject region based on the evaluation value calculated in the evaluation step;
In the determination step, the determination unit determines that the difference between the determination value and the highest evaluation value of the regions is the highest when there is no second region having an evaluation value equal to or lower than a predetermined threshold value. A first area having an evaluation value is determined as the main subject area, and when the second area exists, an area obtained by combining the first area and the second area is set as the main subject area. A control method for an image processing apparatus, characterized by:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 8.

Images