JP2013157934A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract an area of a main subject from an image.SOLUTION: An image processing device comprises: an extraction unit that extracts a plurality of subject areas included in an image; an evaluation value calculation unit that, for each subject area, calculates an area evaluation value on the basis of an area center in the subject area; and a selection unit that selects a subject area from the plurality of subject areas on the basis of the area evaluation values.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program for identifying a region included in an image.

  An image processing apparatus that extracts a region from a captured image and identifies the extracted region based on a histogram of image feature values is disclosed (see Patent Document 1).

Japanese Patent Laid-Open No. 4-10079

  However, the image processing device disclosed in Patent Document 1 cannot extract a region that is noticed by a person from a captured image, that is, a region of a main subject.

  The present invention has been made in view of such a situation, and provides an image processing device, an imaging device, and an image processing program that extract a region of a main subject from a captured image.

  In order to solve the above-described problems, the present invention provides an extraction unit that extracts a plurality of subject areas included in an image, and an evaluation value calculation that calculates a region evaluation value for each subject area based on the center of the subject area. And a selection unit that selects a subject region from a plurality of subject regions based on a region evaluation value.

  The present invention also provides an imaging unit that captures an image of a subject via a focusing lens and generates an image, an extraction unit that extracts a plurality of subject regions included in the image, and a region center in the subject region for each subject region. An evaluation value calculation unit that calculates an area evaluation value based on the image, a selection unit that selects a subject area from a plurality of subject areas based on the area evaluation value, and a subject area selected by the selection unit An imaging device comprising: a driving unit that drives a focusing lens.

  Further, the present invention provides a computer for extracting a plurality of subject areas included in an image, for each subject area, calculating an area evaluation value based on the center of the subject area, and from the plurality of subject areas. And a step of selecting a subject area based on the area evaluation value.

  As described above, according to the present invention, the region of the main subject can be extracted from the image.

It is a block diagram which shows the structural example of the camera system by one Embodiment of this invention. It is a figure which shows the example of the mask which the extraction part by one Embodiment of this invention extracts. It is a figure which shows the outline | summary of the evaluation value which the evaluation value calculation part by one Embodiment of this invention calculates. 6 is a flowchart illustrating an operation example of the image processing apparatus according to the embodiment of the present invention. It is a figure which shows the example of the total value of the primary evaluation value and secondary evaluation value by one Embodiment of this invention. 6 is a flowchart illustrating an operation example of the image processing apparatus according to the embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
<First Embodiment>
FIG. 1 is a block diagram showing a configuration of a camera system 1 according to the first embodiment of the present invention. The camera system 1 includes a lens barrel 111, an imaging device 100 including an image processing device 130, and a storage medium 190. Here, an optical image incident from the lens barrel 111 is picked up by the image pickup apparatus 100, and the obtained image is stored in the storage medium 190 as a still image or a moving image.

The lens barrel 111 includes a focusing lens (hereinafter referred to as an “AF (Auto Focus) lens”) 112, an AF encoder 113, a lens barrel control unit 114, and a lens driving unit 115. The lens barrel 111 may be detachably connected to the imaging device 100 or may be integrated with the imaging device 100.
The AF lens 112 is driven by the lens driving unit 115 and guides an optical image to the light receiving surface (photoelectric conversion surface) of the imaging element 116 of the imaging unit 110.

  The AF encoder 113 detects the movement of the AF lens 112 and outputs a signal corresponding to the movement amount of the AF lens 112 to the lens barrel control unit 114. The signal corresponding to the movement amount of the AF lens 112 is a sine wave signal whose phase changes according to the movement amount of the AF lens 112, for example.

The lens barrel control unit 114 controls the lens driving unit 115 in accordance with a drive control signal input from the CPU 170 of the imaging apparatus 100. Here, the drive control signal is a control signal for driving the AF lens 112 in the optical axis direction. The lens barrel control unit 114 changes, for example, the number of steps of the pulse voltage output to the lens driving unit 115 in accordance with the drive control signal.
Further, the lens barrel control unit 114 outputs the position (focus position) of the AF lens 112 in the lens barrel 111 to the CPU 170 based on a signal corresponding to the movement amount of the AF lens 112. Here, the lens barrel control unit 114 integrates, for example, a signal corresponding to the movement amount of the AF lens 112 according to the movement direction of the AF lens 112, thereby moving the AF lens 112 in the lens barrel 111 ( Position) can be calculated.
The lens driving unit 115 drives the AF lens 112 according to the control of the lens barrel control unit 114 and moves the AF lens 112 in the optical axis direction within the lens barrel 111.

The imaging device 100 includes an imaging unit 110, an operation unit 120, an image processing device 130, a display unit 140, a buffer memory unit 150, a storage unit 160, a CPU 170, and a communication unit 180.
The imaging unit 110 is controlled by the CPU 170 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.), and performs imaging to generate an image. The imaging unit 110 includes an imaging element 116 and an A / D (Analog / Digital) conversion unit 120.
The image sensor 116 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 111 (optical system) into an electrical signal, and outputs the electrical signal to the A / D conversion unit 117. The image sensor 116 is configured by, for example, a CMOS (Complementary Metal Oxide Semiconductor).

  Further, the image sensor 116 causes the storage medium 190 to store an image obtained when a shooting instruction from the user is received via the operation unit 120 via the A / D conversion unit 117. On the other hand, the image sensor 116 converts the continuously obtained image into a through image in a state where it does not accept a shooting instruction from the user via the operation unit 120, and performs A / D conversion on the buffer memory unit 150 and the display unit 140. The data is output via the unit 117.

The A / D conversion unit 117 digitizes the electrical signal converted by the image sensor 116 and outputs an image that is a digital signal to the buffer memory unit 150.
The operation unit 120 includes operation keys such as a power switch, a shutter button, and a multi-selector (cross key), receives an operation input from the user, and outputs a signal corresponding to the operation input to the CPU 170.

The image processing apparatus 130 performs image processing based on the image processing conditions stored in the storage unit 160. The image processing apparatus 130 includes an extraction unit 131, an evaluation value calculation unit 132, and a selection unit 133.
The extraction unit 131 reads an image temporarily stored in the buffer memory unit 150, and extracts a plurality of subject areas included in the read image. Here, the extraction unit 131 calculates the feature amount for each of a plurality of features in the image acquired from the buffer memory unit 150. The feature amount includes, for example, a motion vector, hue, saturation, brightness, texture, edge, distance (depth, depth map, defocus amount), contrast in an image (for example, contrast of adjacent pixels), and the like.

  Further, the extraction unit 131 performs a labeling process for setting a label (identifier) for identifying a region included in the image detected for each feature amount based on the one or more types of feature amounts. The label indicates a grouped region (set), which is a region composed of pixels whose feature amounts are close to each other. Labeling is to sequentially detect and group areas adjacent to pixels whose feature amounts are close to each other (for example, corresponding to saturation h = 0 to 20 degrees). For example, it is common to detect an area where the luminance value (Y) = 128 ± 30 (labeling) and name the detected area (for example, label 1, label 2,...).

  Here, the subject area extracted by labeling by the extraction unit 131 is referred to as a mask. When the extraction unit 131 detects a plurality of masks based on the feature amount, the extraction unit 131 calculates a primary evaluation value (view angle evaluation value) for each of the plurality of masks based on the view angle center in the image. The primary evaluation value is, for example, an evaluation value indicating the degree of mask clumping with respect to the center of the angle of view. Specifically, for example, the extraction unit 131 performs the primary evaluation value based on one or a combination of the position, shape, degree of mass (moment of inertia), size, filling rate, and the like with respect to the center of the angle of view in the image. Is calculated. The filling rate is, for example, the ratio of the area occupied by the mask to the circumscribed rectangle of the mask.

  Further, the extraction unit 131 extracts a plurality of masks based on the calculated primary evaluation value. For example, the extraction unit 131 removes a noise mask, a background mask, a mask at the edge of the screen, and the like. FIG. 2 is a diagram illustrating an example of a mask extracted by the extraction unit 131. The extraction unit 131 calculates an evaluation value of the mask for the code a3 that is the center of the angle of view of the image indicated by the code a1. Here, the extraction unit 131 removes a mask outside a certain center region with respect to the image, as indicated by reference numeral a2, and extracts a mask within the center region. The extraction unit 131 stores a label for identifying the extracted mask in the storage unit 160 and the storage medium 190 for each region.

  For each mask extracted by the extraction unit 131, the evaluation value calculation unit 132 calculates a secondary evaluation value (region evaluation value) based on the mask center that is the center of the region in the subject region indicated by the mask. The secondary evaluation value is, for example, an evaluation value indicating the degree of mask clumping with respect to the mask center. That is, the primary evaluation value calculated by the extraction unit 131 is the evaluation value of the mask with respect to the center of the view angle, whereas the secondary evaluation value calculated by the evaluation value calculation unit 132 is the evaluation value of the mask with respect to the mask center. . FIG. 3 is a diagram showing an outline of the evaluation value calculated by the evaluation value calculation unit 132. The evaluation value calculation unit 132 calculates a circumscribed rectangle of the mask, and uses the center of the circumscribed rectangle as the mask center, the position, shape, degree of blockiness (moment of inertia), size, filling rate, pure chromaticity, aspect of the mask. A secondary evaluation value is calculated based on one or a plurality of combinations such as a ratio. Alternatively, for example, a histogram of each pixel from the mask center can be calculated in the X direction and the Y direction, and the secondary evaluation value can be calculated according to the magnitude of the deviation. Here, an example in which the center of the circumscribed rectangle of the mask is the center of the mask will be described, but the center of gravity of the mask or the center of the inscribed rectangle of the mask may be used as the center of the mask.

The selection unit 133 selects a mask based on the secondary evaluation value from the plurality of masks for which the evaluation value calculation unit 132 has calculated the secondary evaluation value. For example, a mask having a high secondary evaluation value is selected. In addition, when the selection unit 133 selects a plurality of masks based on the secondary evaluation value, the selection unit 133 selects the closest mask from the plurality of masks.
As described above, in the present embodiment, not only the evaluation value (primary evaluation value) of the mask from the view angle center calculated by the extraction unit 131 but also the evaluation value of the mask from the mask center calculated by the evaluation value calculation unit 132. (Secondary evaluation value) is calculated, and a mask is selected based on the primary evaluation value and the secondary evaluation value. As a result, the mask to be selected is biased toward the center with only the primary evaluation value, so that the mask can be selected by calculating the secondary evaluation value for the mask in a certain central region. For example, in the case of the example shown in FIG. 2, the mask A and the mask B close to the center of the angle of view may be selected only by the primary evaluation value. On the other hand, if the mask is selected based on the secondary evaluation value for each mask of the mask A, mask B, mask C, and mask D within a certain central region, the mask C, the mask D, etc. are larger. A mask with more shape can be selected.

The display unit 140 is, for example, a liquid crystal display, and displays an image obtained by the imaging unit 110, an operation screen, and the like.
The buffer memory unit 150 temporarily stores the image captured by the imaging unit 110.
The storage unit 160 stores information such as images captured by the image capturing unit 110 and various conditions for operating each unit included in the image capturing apparatus 100.

The CPU 170 controls the imaging unit 110, the lens barrel 111, and the like according to the set imaging conditions (for example, aperture value, exposure value, etc.). Also, the CPU 170 performs focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, subject image (object) tracking setting, and night scene as pre-processing or post-processing. Control for determining whether or not, setting of color correction processing, setting of enlarged display of a subject image, setting of panning display, setting of optimization of brightness linked to zoom magnification, and the like.
The communication unit 180 is connected to a removable storage medium 190 such as a card memory, and performs writing, reading, or erasing of information (image data, area information, etc.) on the storage medium 190.
The storage medium 190 is a storage unit that is detachably connected to the imaging apparatus 100 and stores information (image data, area information, and the like). Note that the storage medium 190 may be integrated with the imaging device 100.

Next, an operation example of the imaging apparatus 100 according to the present embodiment will be described with reference to the drawings. FIG. 4 is a flowchart illustrating an operation example in which the image processing apparatus 130 selects a mask based on the secondary evaluation value after extracting the mask based on the primary evaluation value.
When an operation input is input from the user to the operation unit 120, the imaging unit 110 captures an image and stores the through image in the buffer memory unit 150. The image processing apparatus 130 reads through images stored in the buffer memory unit 150 (step S1) and performs basic image processing. Here, basic image processing includes, for example, simple exposure adjustment (simple AE), white balance adjustment (AWB), and noise reduction (NR).

  The extraction unit 131 calculates a feature amount for each feature of the image, performs labeling based on each feature amount (for example, green and red when the feature is a hue), and extracts a mask from the image (step S3). . The extraction unit 131 removes the noise mask and the background mask from the plurality of masks extracted for each feature amount (step S4). Further, the extraction unit 131 removes a certain mask from the plurality of masks extracted for each feature amount based on the screen position. For example, the mask at the edge of the screen is removed (step S5). The extraction unit 131 sets a label for identifying the mask in the mask detected for each feature amount, and causes the storage unit 160 to store the set label for each mask.

  Further, the extraction unit 131 calculates a primary evaluation value with respect to the center of the angle of view for each detected mask (step S6). Then, the evaluation value calculation unit 132 calculates circumscribed rectangles for N masks whose calculated primary evaluation values are, for example, the top N (N is a predetermined arbitrary integer) among the plurality of masks. (Step S7). Further, the evaluation value calculation unit 132 calculates a secondary evaluation value, which is an evaluation value from the circumscribed rectangle center, for each of the N masks (step S8). The selection unit 133 compares the secondary evaluation values for each mask calculated by the evaluation value calculation unit 132, and selects, for example, the top two masks (step S9). The CPU 170 shapes the mask area selected by the selection unit 133 into an AF rectangle (step S10), performs a contrast scan in the window set by controlling the lens barrel 111 (step S11), and gives priority to the closest distance. AF control is then performed (step S12). Thereby, the lens driving unit 115 drives the AF lens 112 so as to focus on the mask selected by the selection unit 133.

In the present embodiment, an example in which the extraction unit 131 extracts a mask in the center region with respect to the center of the angle of view has been described. You can also. The specific area is, for example, a central area that changes according to the zoom magnification. For example, if the zoom magnification is high, the specific area is reduced, and if the zoom magnification is low, the specific area is increased. Alternatively, the upper half area of the angle of view can be set as the specific area. In this case, for example, based on whether the image direction included in the EXIF information of the image indicates a vertical position or a horizontal position, the upper area toward the image direction can be set as the specific area. Or the input which selects the arbitrary area | regions from a user is received, and the selected area | region can also be made into a specific area | region. Also, the evaluation value of each mask can be calculated based on the center point of such a specific area. According to this, a mask in a certain region can be evaluated equivalently without depending on the evaluation value from the center of the angle of view.
Alternatively, the selection unit 133 may select a mask based on only the secondary evaluation value calculated by the evaluation value calculation unit 132 without calculating the primary evaluation value by the extraction unit 131. Or the extraction part 131 can also extract the mask contained in the whole view angle.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. In the first embodiment, after the extraction unit 131 performs determination based on the primary evaluation value and extracts a plurality of masks, the evaluation value calculation unit 132 calculates a secondary evaluation value for the plurality of masks, The example which selected the mask based on the evaluation value was shown. On the other hand, in this embodiment, a mask is selected based on the total value of the primary evaluation value calculated by the extraction unit 131 and the secondary evaluation value calculated by the evaluation value calculation unit 132. For example, FIG. 5 is a diagram illustrating an example of the total value of the primary evaluation value and the secondary evaluation value. This figure shows an example of calculating the total value based on the rank according to the magnitude, moment, and the like. Here, the rank of the primary evaluation value for the mask A is 1st, and the rank of the secondary evaluation value is 3rd. The rank of the primary evaluation value for the mask B is 2nd, and the rank of the secondary evaluation value is 4th. The rank of the primary evaluation value for the mask C is 4th, and the rank of the secondary evaluation value is 2nd. The rank of the primary evaluation value for the mask D is 3rd, and the rank of the secondary evaluation value is 1st. At this time, the selection unit 133 calculates the total value of the rank value of the secondary evaluation value and the value obtained by multiplying the rank of the secondary evaluation value by 0.5, and the total value is the lowest (the rank is high). Select a mask. As described above, the selection unit 133 selects a mask based on the total value of the primary evaluation value calculated by the extraction unit 131 and the secondary evaluation value calculated by the evaluation value calculation unit 132.

  FIG. 6 is a flowchart illustrating an operation example in which the image processing apparatus 130 selects a mask based on the total value of the primary evaluation value and the secondary evaluation value. Steps S21 to S26 are the same as steps S1 to S6. Then, the evaluation value calculation unit 132 calculates the center of gravity of the masks of the N masks whose calculated primary evaluation values are the top N among the plurality of masks (step S27). Further, the evaluation value calculation unit 132 calculates a secondary evaluation value that is an evaluation value from the center of gravity for each of the N masks (step S28). The selection unit 133 calculates the total value of the primary evaluation value calculated by the extraction unit 131 and the secondary evaluation value calculated by the evaluation value calculation unit 132, compares the total value calculated for each mask, and obtains the second highest rank. Is selected (step S29). Steps S30 to S32 are the same as the processes from step S10 to step S12.

  As described above, according to the present embodiment, for the mask extracted based on the feature quantity, first, the primary evaluation value from the center of the view angle is calculated, and then the secondary evaluation value from the mask center of each mask is calculated. Then, a mask is selected based on the primary evaluation value and the secondary evaluation value. Thereby, for example, a plurality of masks in the central area of the screen can be evaluated equivalently, and a more appropriate mask can be selected.

  Note that a program for realizing the function of the processing unit in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to perform image processing. May be. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 100 ... Imaging device, 110 ... Imaging part, 115 ... Drive part, 130 ... Image processing apparatus, 131 ... Extraction part, 132 ... Evaluation value calculation part, 133 ... Selection part

Claims (8)

An extraction unit for extracting a plurality of subject areas included in the image;
An evaluation value calculation unit that calculates an area evaluation value for each subject area based on the area center in the subject area;
A selection unit that selects a subject region from the plurality of subject regions based on the region evaluation value;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the extraction unit extracts the subject area included in a specific area in the image. The image according to claim 1, wherein the extraction unit calculates an angle-of-view evaluation value based on a center of an angle of view in the image, and extracts the plurality of subject areas based on the angle-of-view evaluation value. Processing equipment. The image processing apparatus according to claim 3, wherein the selection unit selects the subject area based on a total value of the field angle evaluation value and the area evaluation value. The selection unit, when a plurality of the subject regions are selected based on the region evaluation value, selects a closest subject region among the plurality of subject regions. 5. The image processing device according to any one of 4 to 4. The evaluation value calculation unit calculates the region evaluation value based on any one or a combination of the size, moment, and filling rate of the subject region. The image processing apparatus according to any one of the above. An imaging unit that images a subject via a focusing lens and generates an image;
An extraction unit for extracting a plurality of subject areas included in the image;
An evaluation value calculation unit that calculates an area evaluation value for each subject area based on the area center in the subject area;
A selection unit that selects a subject region from the plurality of subject regions based on the region evaluation value;
A drive unit that drives the focusing lens to focus on the subject area selected by the selection unit;
An imaging apparatus comprising: On the computer,
Extracting a plurality of subject areas included in the image;
Calculating a region evaluation value for each subject region based on a region center in the subject region;
Selecting a subject area from the plurality of subject areas based on the area evaluation value;
An image processing program for executing

Images