JP2010154323A - Image processing apparatus, image extraction method, and, program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more efficiently perform an effective subject extraction. <P>SOLUTION: An image dividing section 213 divides one of two captured images, captured by a first imaging section 100A and a second imaging section 100B into a plurality of regions on the basis of color information, luminance information, or the like. A distance calculating section 214 uses the divided regions as a template and performs scanning on the other captured image. The distance calculating section 214 calculates the photographic distance for a searched area, based on the principle of parallax-based triangulation. When a subject position is designated, an area including the position is used as a template, or in the case, where no subject position is designated, overall area is used as a template and candidate areas are extracted, on the basis of the photographic distance. A subject extracting section 215 calculates an inter-color distance between a candidate area and a non-candidate area adjacent to the candidate area; and when the inter-color distance is short, the area is excluded from the candidate. A candidate area, after correction based on the inter-color distance, is extracted as a subject image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing device, an image extraction method, and a program, and more particularly, to an image processing device, an image extraction method, and a program suitable for extracting a subject area from a captured image.

  With the widespread use of digital cameras, various image processing can be easily performed on captured images. For example, a technique of extracting and combining subject portions by performing a plurality of shootings from the same angle has been established (for example, Patent Document 1).

JP 2003-338777 A

  In conventional techniques such as Patent Document 1, when extracting a subject area, extraction based mainly on color information is common. For example, when a red subject is separated from the background, it is determined whether the color of each pixel in the image falls within a certain range of red color, and the pixel to be extracted is identified based on the result.

  Image extraction using such color information is suitable when there is a subject with different colors in a monotonous background, but when the background and subject contain similar colors, Appropriate image extraction cannot be performed.

  As another method, there is a method of using edge information of an image, that is, information of a portion in which a pixel value (for example, color or brightness) in the image changes. In this method, usually, a portion where a difference in color or image density between adjacent pixels in an image is large is obtained as an edge, and an edge is obtained by connecting the edges.

  However, in this method, the process of searching for edges and connecting them one after another is performed automatically by a computer. For example, when many edge candidates appear or are interrupted, the edges may not be connected well.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus, an image extraction method, and a program capable of more efficiently performing effective image extraction.

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention provides:
Image acquisition means for acquiring two captured images having different optical axis positions at the time of imaging for the same subject;
A dividing unit that divides one of the captured images acquired by the image acquiring unit into a plurality of regions;
Shooting distance calculation means for calculating a shooting distance in units of the divided areas by template matching with respect to the other of the captured images using the divided areas as templates;
Extraction means for extracting a subject area from the captured image based on the shooting distance calculated by the shooting distance calculation means;
It is characterized by providing.

In the image processing apparatus,
The dividing unit may divide the plurality of regions based on pixel values in the captured image.

In the image processing apparatus,
The extraction unit can extract an area having the shortest shooting distance and an area continuous with the area based on the shooting distance as the subject area.

In the image processing apparatus,
The extraction unit can extract a region in the center of the captured image and a region continuous with the region based on the shooting distance as the subject region.

In the image processing apparatus,
The extraction unit can extract, as the subject area, an area at a specified position on the captured image and an area continuous with the area based on the shooting distance.

The image processing apparatus includes:
It may further comprise face recognition means for recognizing a person's face on the captured image.
The extraction unit can extract a face part region recognized by the face recognition unit and a region connected to the region based on the shooting distance as the subject region.

In the image processing apparatus,
The extraction means includes
An inter-color distance calculating means for calculating an inter-color distance with a non-candidate area adjacent to the candidate area, the area extracted as the subject area;
It is desirable to further include an excluding unit that excludes the candidate region from candidates based on the intercolor distance calculated by the intercolor distance calculating unit.

In the image processing apparatus,
The image acquisition unit preferably includes a parallel stereo camera that acquires the two captured images.

In order to achieve the above object, an image extraction method according to a second aspect of the present invention includes:
An image extraction method for extracting a subject area from two captured images having different optical axis positions at the time of imaging for the same subject,
An image acquisition step of acquiring the two captured images;
A division step of dividing one of the captured images acquired in the image acquisition step into a plurality of regions;
A shooting distance calculating step of calculating a shooting distance in units of the divided areas by template matching for the other of the captured images using the divided areas as templates;
An extraction step of extracting a subject area from the captured image based on the shooting distance calculated in the shooting distance calculation step;
It is characterized by including.

In order to achieve the above object, a program according to the third aspect of the present invention is:
On the computer,
A function of acquiring two captured images having different optical axis positions at the time of imaging for the same subject;
A function of dividing one of the acquired captured images into a plurality of regions;
A function of calculating a shooting distance in units of the divided areas by template matching for the other of the captured images using the divided areas as templates;
A function of extracting a subject area from the captured image based on the calculated shooting distance;
It is characterized by realizing.

  According to the present invention, a subject area can be extracted more efficiently.

  Embodiments according to the present invention will be described below with reference to the drawings. In the present embodiment, a case where the present invention is realized by a digital still camera (hereinafter referred to as a digital camera) is illustrated. The digital camera 1 according to the present embodiment is assumed to have a function that a general digital still camera has, but is a so-called compound eye camera including two configurations related to imaging.

  FIG. 1 is a block diagram showing a configuration of a digital camera 1 according to an embodiment of the present invention. The schematic configuration of the digital camera 1 according to the present embodiment includes a first imaging unit 100A, a second imaging unit 100B, a data processing unit 200, an interface (I / F) unit 300, and the like as illustrated.

  The first imaging unit 100 </ b> A and the second imaging unit 100 </ b> B are parts that perform the imaging operation of the digital camera 1. As described above, since the digital camera 1 according to the present embodiment is a compound eye camera, the first camera 100A and the second imager 100B are configured with the first imager 100A and the second imager 100B. Have the same configuration. Hereinafter, “A” is added to the end of the reference numeral for the configuration of the first imaging unit 100A, and “B” is added to the end of the reference numeral for the configuration of the second imaging unit 100B.

  As shown in FIG. 1, the first imaging unit 100A (second imaging unit 100B) includes an optical device 110A (110B), an image sensor unit 120A (120B), and the like.

  The optical device 110A (110B) includes, for example, a lens, a diaphragm mechanism, a shutter mechanism, and the like, and performs an optical operation related to imaging. In other words, the operation of the optical device 110A (110B) collects incident light and adjusts optical elements related to the angle of view, focus, exposure, and the like, such as focal length, aperture, and shutter speed. The shutter mechanism included in the optical device 110A (110B) is a so-called mechanical shutter. When the shutter operation is performed only by the operation of the image sensor, the optical device 110A (110B) may not include the shutter mechanism. Good. The optical device 110A (110B) operates under the control of the control unit 210 described later.

  The image sensor unit 120A (120B) generates an electrical signal corresponding to incident light collected by the optical device 110A (110B), for example, a CCD (Charge Coupled Device) or a CMOS (Complementally Metal Oxide Semiconductor). : Complementary metal oxide semiconductor). The image sensor unit 120 </ b> A (120 </ b> B) performs an electrical conversion to generate an electrical signal corresponding to the received light and output it to the data processing unit 200.

  As described above, the first imaging unit 100A and the second imaging unit 100B have the same configuration. More specifically, the specifications such as the focal length f and F value of the lens, the aperture range of the aperture mechanism, the size and number of pixels of the image sensor, the arrangement, and the pixel area are all the same.

  As shown in FIG. 2A, the digital camera 1 having the first imaging unit 100A and the second imaging unit 100B includes a lens configured in the optical device 110A and a lens configured in the optical device 110B. The digital camera 1 is formed on the same outer surface. Here, it is assumed that when the digital camera 1 is leveled in the direction in which the shutter button is upward, the two lenses (light receiving portions) are arranged so that the center position is on the same line in the horizontal direction. That is, when the first imaging unit 100A and the second imaging unit 100B are operated simultaneously, images of the same subject are captured, but the optical axis position in each image is shifted in the horizontal direction. Become.

  In this case, the first imaging unit 100A and the second imaging unit 100B are arranged so as to have optical characteristics as shown in the perspective projection model of FIG. The perspective projection model shown in FIG. 2B is based on a three-dimensional orthogonal coordinate system composed of X, Y, and Z, and this coordinate system for the first imaging unit 100A is hereinafter referred to as “camera coordinates”. FIG. 2B shows camera coordinates with the optical center of the first imaging unit 100A as the origin.

  In camera coordinates, the Z-axis is the direction that matches the optical direction of the camera, and the X-axis and Y-axis are parallel to the horizontal and vertical directions of the image, respectively. Here, when the intersection of the optical axis and the image coordinate plane is set as the origin (that is, the optical center), the pixel interval of the image sensor is converted, and the unit of the camera coordinate and the length is matched, the image about the first imaging unit 100A The image coordinate indicating the subject A1 on the coordinate plane is (u1, v1), and the image coordinate plane for the second imaging unit 100B is (u′1, v′1).

  The first imaging unit 100A and the second imaging unit 100B have parallel optical axes (that is, the convergence angle is 0), and the image coordinate u-axis and the second imaging unit 100B for the first imaging unit 100A. Are arranged such that the image coordinate u ′ axes are in the same direction on the same line (that is, epipolar lines coincide). Further, as described above, the focal length f and the pixel interval of the first imaging unit 100A and the second imaging unit 100B are the same, and the optical axis and the image coordinate plane intersect perpendicularly. Such a configuration is called “parallel stereo”, and the first imaging unit 100A and the second imaging unit 100B of the digital camera 1 have a parallel stereo configuration.

  Returning to FIG. 1, the description of the configuration of the digital camera 1 will be continued.

  The data processing unit 200 processes the electrical signal generated by the imaging operation by the first imaging unit 100A and the second imaging unit 100B, generates digital data indicating the captured image, and performs image processing on the captured image. As shown in FIG. 1, the data processing unit 200 includes a control unit 210, an image processing unit 220, an image memory 230, an image output unit 240, a storage unit 250, an external storage unit 260, and the like.

  The control unit 210 includes, for example, a processor such as a CPU (Central Processing Unit), a main storage device (memory) such as a RAM (Random Access Memory), and the like. Each part of the digital camera 1 is controlled by executing the stored program. Further, in the present embodiment, by executing a predetermined program, a function related to each process described later is realized by the control unit 210.

  The image processing unit 220 includes, for example, an ADC (Analog-Digital Converter), a buffer memory, a processor for image processing (so-called image processing engine), and the like, and is generated by the image sensor units 120A and 120B. Based on the electrical signal thus generated, digital data indicating a captured image is generated.

  That is, when the analog electric signal output from the image sensor unit 120A (120B) is converted into a digital signal by the ADC and sequentially stored in the buffer memory, the image processing engine performs a so-called development process on the buffered digital data. In this way, image quality adjustment and data compression are performed.

  The image memory 230 includes, for example, a storage device such as a RAM or a flash memory, and temporarily stores captured image data generated by the image processing unit 220, image data processed by the control unit 210, and the like.

  The image output unit 240 includes, for example, an RGB signal generation circuit and the like, converts the image data expanded in the image memory 230 into an RGB signal and the like, and outputs the RGB signal to a display screen (a display unit 310 to be described later).

  The storage unit 250 includes a storage device such as a ROM (Read Only Memory) or a flash memory, and stores programs and data necessary for the operation of the digital camera 1. In the present embodiment, it is assumed that the operation program executed by the control unit 210 and the like, parameters and arithmetic expressions necessary for the processing are stored in the storage unit 250.

  The external storage unit 260 includes a storage device that can be attached to and detached from the digital camera 1 such as a memory card, and stores image data captured by the digital camera 1.

  The interface unit 300 has a configuration related to an interface between the digital camera 1 and its user or an external device. As shown in FIG. 1, the display unit 310, an external interface (I / F) unit 320, an operation unit 330, etc. Consists of

  The display unit 310 includes, for example, a liquid crystal display device, and displays and outputs various screens necessary for operating the digital camera 1, live view images at the time of shooting, captured images, and the like. In the present embodiment, display output of a captured image or the like is performed based on an image signal (RGB signal) from the image output unit 240.

  The external interface unit 320 includes, for example, a USB (Universal Serial Bus) connector, a video output terminal, and the like, and performs output of image data to an external computer device, display output of a captured image to an external monitor device, and the like. .

  The operation unit 330 includes various buttons configured on the outer surface of the digital camera 1, generates an input signal corresponding to an operation by the user of the digital camera 1, and inputs the input signal to the control unit 210. As buttons constituting the operation unit 330, for example, a shutter button for instructing a shutter operation, a mode button for designating an operation mode of the digital camera 1, a cross key and a function button for performing various settings, and the like Is included.

  Here, in the present embodiment, the control unit 210 executes the operation program stored in the storage unit 250 to realize each process described later. In this case, the functions realized by the control unit 210 are implemented. This will be described with reference to FIG.

  FIG. 3 is a functional block diagram illustrating functions realized by the control unit 210. Here, a functional configuration necessary for realizing a function of extracting a subject image from an image captured by a compound eye camera is shown. In this case, as illustrated, the control unit 210 functions as an operation mode processing unit 211, an imaging control unit 212, an image division unit 213, a distance calculation unit 214, a subject extraction unit 215, and the like.

  The operation mode processing unit 211 cooperates with the display unit 310 to display screens necessary for the user of the digital camera 1 to specify various operation modes of the digital camera 1 and setting screen displays for each specified operation mode. In addition, the operation mode designated by the user is recognized in cooperation with the operation unit 330, and a program or an arithmetic expression necessary for executing the operation mode is read from the storage unit 250, and the main memory of the control unit 210 is read out. Load to device (memory).

  In the present embodiment, it is assumed that an operation mode (image extraction mode) for extracting a subject portion from a captured image after shooting with the digital camera 1 is specified by the user, and each functional configuration of the control unit 210 described below is an image. This is a functional configuration realized by executing the program loaded by the operation mode processing unit 211 in accordance with the designation of the extraction mode.

  The imaging control unit 212 executes the imaging operation with the digital camera 1 by controlling the first imaging unit 100A and the second imaging unit 100B. The subject image extraction function according to the present embodiment extracts a subject image from captured images obtained from the first imaging unit 100A and the second imaging unit 100B having a parallel stereo configuration, and thus imaging control according to the present embodiment. The unit 212 controls the first imaging unit 100A and the second imaging unit 100B to perform the imaging operation at the same time. Thereby, two captured images having different optical axis positions are acquired for the same subject.

  The image dividing unit 213 performs region division on the captured image obtained by the control of the imaging control unit 212. Here, region division is performed for one of the two acquired captured images. In the present embodiment, it is assumed that region division is performed on a captured image (hereinafter, referred to as “target image”) obtained by an imaging operation by the first imaging unit 100A. The image dividing unit 213 according to the present embodiment performs region division by, for example, the Meanshift method or the Watershed method using color information, luminance gradient information, or the like.

  The distance calculation unit 214 performs a process of calculating a shooting distance for each region by comparing two captured images based on the region divided on the target image. In the present embodiment, using a region divided on the target image as a template, a template matching is performed on a captured image (hereinafter referred to as a “reference image”) obtained by the imaging operation of the second imaging unit 100B, and the matching is performed. The shooting distance is calculated from the difference between the images in the area. The distance calculation unit 214 according to the present embodiment performs, for example, template matching using an SSD (Sum of Squared Difference) that uses the sum of squares of differences as an evaluation formula.

  The subject extraction unit 215 extracts a region in which a desired subject image is shown based on the shooting distance for each divided region calculated by the template matching performed by the distance calculation unit 214. Here, in the determination based on the shooting distance, a subject area candidate is extracted, and an accurate subject image is extracted by calculating a distance between colors at the boundary between the extracted subject area candidate and the background.

  The above is the function realized by the control unit 210. In the present embodiment, each function described above is realized by a logical process performed by the control unit 210 executing a program. These functions are, for example, ASIC (Application Specific Integrated Circuit). Or an integrated circuit). In this case, among the functions shown in FIG. 3, the functions related to image processing may be realized by the image processing unit 220.

  The configuration of the digital camera 1 described above is a configuration necessary for realizing the present invention, and a configuration used for a basic function and various additional functions as a digital camera is provided as necessary. .

  The operation of the digital camera 1 having such a configuration will be described below. Here, the “subject extraction process” executed by the digital camera 1 when the above-described “image extraction mode” is selected from the operation modes of the digital camera 1 will be described with reference to the flowchart shown in FIG. This “subject extraction process” is started when the user of the digital camera 1 selects the image extraction mode by operating the operation unit 330. In this case, the operation mode processing unit 211 loads a program or the like related to the subject extraction operation from the storage unit 250, whereby the following processing is executed by each functional configuration illustrated in FIG.

  When the processing is started, the imaging control unit 212 controls the first imaging unit 100A and the second imaging unit 100B in response to the operation of the shutter button (operation unit 330), and the first imaging unit 100A and The second imaging unit 100B simultaneously performs an imaging operation (compound eye simultaneous imaging) (step S11: Yes, step S12).

  Here, in this embodiment, the case where the scene as shown to Fig.8 (a) is image | photographed with the digital camera 1 is assumed. Here, as shown in FIG. 8A, the subject A and the subject B are photographed by the digital camera 1, but the distance from the digital camera 1 to each subject is closer to the subject A.

  FIGS. 8B and 8C show examples of captured images in the case where compound eye simultaneous imaging is performed in such a scene. In the present embodiment, it is assumed that the lens on the left side facing the subject is configured in the second imaging unit 100B, and the lens on the right side facing the subject is configured in the first imaging unit 100A. In this case, FIG. 8B illustrates an image captured by the second imaging unit 100B, and FIG. 8C illustrates an image captured by the first imaging unit 100A.

  As described above, since the optical axes of the respective lenses are arranged so as to be different in the horizontal direction, the same subject is shown in the two images obtained by imaging, but the positions in the horizontal direction are different. .

  When two captured images are acquired in this way, the operation mode processing unit 211 displays a setting screen on the display unit 310, and specifies the subject position desired to be extracted from the captured image to the user of the digital camera 1. A method is set (step S13). Here, as a method for designating the subject position, for example, “center of image”, “arbitrary designation”, “face recognition”, “shortest distance”, and the like are displayed on the setting screen so that the user of the digital camera 1 can select. By operating the operation unit 330, a method suitable for the captured image is designated.

  Here, “image center” is a method that is selected when the subject desired to be extracted appears in the center of the captured image, and designates that the subject position is the center of the image.

  “Arbitrary designation” is a method in which the user arbitrarily designates the position on the captured image of the subject desired to be extracted. For example, the position on the captured image is designated by a cursor.

  “Face recognition” is a method that can be selected when the digital camera 1 has a face recognition function, and is a method in which the position of the face portion recognized by face recognition is designated as the subject position.

  The “shortest distance” is a method of designating a subject closest to the digital camera 1 among a plurality of subjects shown in a captured image as an extraction target. In this method, since the subject portion is determined based on the shooting distance, unlike the above-described methods, the subject position is not designated.

  When any one of these methods is specified, the operation mode processing unit 211 stores information indicating which method is specified in the memory of the control unit 210.

  When the method for specifying the subject position is selected, the operation mode processing unit 211 notifies the image dividing unit 213 to that effect. In response to the notification from the operation mode processing unit 211, the image dividing unit 213 converts an image (target image) captured by the first imaging unit 100A from among the captured images developed in the image memory 230 into a plurality of small images. The area is divided (step S14).

  Here, the region is divided by the Meanshift method or the Watershed method using color information, luminance gradient information, or the like. That is, the captured image is divided into a plurality of regions based on pixel values such as color and luminance. FIG. 9A shows an example of the target image divided into regions.

  When the target image is divided into a plurality of small areas in this way, the image dividing unit 213 notifies the distance calculating unit 214 to that effect. The distance calculation unit 214 performs a process of calculating the shooting distance in units of divided areas in response to the notification from the image dividing unit 213. Here, it is determined whether or not the subject position is designated by the above-described subject position designation method (step S15). Here, if any of the above-mentioned “image center”, “arbitrary designation”, or “face recognition” is selected, it is determined that “subject position has been designated” (step S15: Yes), and “shortest distance” is determined. If it is selected, it is determined that “no subject position is specified” (step S15: No).

  When the distance calculation unit 214 determines that “the subject position is specified” (step S15: Yes), the “candidate region extraction process (1)” is executed (step S100), and “the subject position is not specified”. (Step S15: No), “candidate area extraction process (2)” is executed (Step S200).

  First, “candidate area extraction processing (1)” will be described with reference to the flowchart shown in FIG.

  When the process is started, the distance calculation unit 214 designates an area corresponding to the designated position among the areas divided on the target image as a reference template (step S101).

  Here, when the position designation method is “image center”, the distance calculation unit 214 designates, as a reference template, an area including coordinates indicating the center position of the target image.

  If the position designation method is “arbitrary designation”, an area including the coordinates of the position designated by the user is designated as the reference template.

  When the position designation method is “face recognition”, a face recognition operation is executed on the target image, and an area including the coordinates of the recognized face portion is designated as a reference template. Note that the operation related to face recognition is executed by the control unit 210 using a known face recognition technique.

  When the reference template is designated, the distance calculation unit 214 scans the designated reference template on a reference image having a different optical axis position from the target image subjected to the region division (step S102). Here, an image region similar to the reference template is searched on the reference image by template matching using SSD or the like. Here, when template matching by SSD is performed, the value is 0 if there is an image that completely matches. Since this value increases as the difference between images increases, the value obtained by template matching indicates the degree of difference from the template.

  Therefore, the distance calculation unit 214 identifies a portion having a minimum difference obtained by scanning the reference template on the reference image as an image region corresponding to the reference template (step S103).

  For example, if the reference template designated on the target image is the area indicated by hatching in FIG. 9B, the area searched on the reference image is indicated by hatching in FIG. 9C, for example. It becomes an area. In this case, since the coordinates of the corresponding area on each image are different in the horizontal direction, a parallax d as illustrated is generated.

  Therefore, the distance calculation unit 214 calculates the difference in coordinates between the region on the reference image specified in step S103 and the reference template specified on the target image (step S104).

  When the difference in coordinates is obtained in this way, the distance (photographing distance) from the subject in the reference template area to the digital camera 1 can be calculated by the principle of triangulation (step S105).

  Here, in this embodiment, the subject to be extracted is the subject A (see FIG. 8A), and the center position of the region in which the subject A is shown (reference template indicated by hatching in FIG. 9B) Is A1, and the coordinates are (X1, Y1, Z1) in camera coordinates. In this case, in the digital camera 1 having a parallel stereo configuration as shown in FIG. 2B, the coordinates (X1, Y1, Z1) of A1 are represented by the following equations 1 to 3, respectively. Here, (u1, v1) is a projection point on the image coordinate plane (that is, the target image) of the first imaging unit 100A, and (u′1, v′1) is an image of the second imaging unit 100B. A projection point on a coordinate plane (ie, a reference image) is shown. In addition, b indicates the distance between the optical axes (baseline distance) of the first imaging unit 100A and the second imaging unit 100B.

(Equation 1)
X1 = (b × (u1)) / ((u′1) −u1)

(Equation 2)
Y1 = (b × (v1)) / ((u′1) −u1)

(Equation 3)
Z1 = (b × f) / ((u′1) −u1)

  Here, since the distance from the digital camera 1 to A1 is indicated by the coordinate Z1 of A1, the distance calculation unit 214 can calculate the shooting distance to the reference template by calculating Equation 3 above.

  This number 3 is derived by the principle of triangulation. The principle of triangulation will be described with reference to FIG.

  FIG. 10 is a schematic view of the camera coordinates in the parallel stereo configuration shown in FIG. Since the viewpoint by the first imaging unit 100A is the camera coordinate, the coordinate of the subject position A1 in the X-axis direction is given by X1 on the camera coordinate, and this numerical value can be obtained by Expression (1) shown in FIG. .

  On the other hand, the coordinate in the X-axis direction of A1 at the viewpoint from the second imaging unit 100B is the sum of the inter-optical axis distance b and the camera coordinate X1, and is obtained by Expression (2) shown in FIG. The above Equation 3 is derived from Equation (1) and Equation (2).

  When the shooting distance of the reference template (hereinafter referred to as “reference shooting distance”) is calculated using the principle of triangulation, the distance calculation unit 214 is adjacent to the reference template continuously on the target image. A plurality of regions are selected (step S106), and template matching as described above is performed for each selected region (step S107).

  That is, although the reference template is designated as the position of the subject desired to be extracted, but not all of the subject is included only in the region of the reference template, template matching is performed for the region adjacent to the reference template. By doing so, an area indicating the subject A is searched.

  In this case, since the area where the subject A is shown is the same as or close to the shooting distance calculated for the reference template, the distance calculation unit 214 calculates the shooting distance of each of the selected areas by template matching.

  Then, the distance calculation unit 214 identifies an area where a shooting distance that is the same as or close to the shooting distance calculated by the reference template is calculated (step S108). Here, when the subject has a depth, the subject cannot be shown only by an area having the same shooting distance as that of the reference template. Therefore, a fixed range is set for the distance in the depth direction, and an area that is a shooting distance within this range is specified. A predetermined value may be set in advance for this range, or the range may be specified by the user each time this process is performed.

  The distance calculation unit 214 sets the area specified as the reference template in step S101 and the area specified in step S108 as candidates for areas indicating subjects, that is, candidates for areas to be extracted, and targets image data indicating each area. Extract from the image (step S109).

  When a candidate area to be extracted as a subject area is extracted in this way, the process returns to the “subject extraction process” (FIG. 4).

  Next, the “candidate area extraction process (2)” executed when the subject position is not designated will be described with reference to the flowchart shown in FIG.

  When the subject position is specified, the position is set as the reference template. However, when the subject position is not specified and the portion where the shooting distance is the shortest distance is set as the subject position, the area divided on the target image Perform template matching using everything.

  Therefore, the distance calculation unit 214 first performs numbering on the regions divided on the target image (step S201), and sequentially specifies the region numbers in order (step S202).

  In this case, the reference image is scanned using the designated area as a template (steps S203 and S204). Here, scanning is performed in the same manner as the template matching performed in the “candidate area extraction process (1)” described above.

  Then, when a portion having a minimum difference is specified for each template (step S205), the distance calculation unit 214 compares the difference with a threshold value. That is, when the entire captured image is targeted, there may be no corresponding part due to the influence of parallax or the like. Therefore, even if the part having the smallest difference is specified, if the difference is larger than a predetermined threshold, it is excluded from the matching target.

  In this case, the distance calculation unit 214 calculates the shooting distance based on the principle of triangulation after calculating the difference on the coordinates for the region where the degree of difference is within the threshold (step S206: Yes) (step S207, In step S208, the coordinates of the area and the shooting distance are recorded in a memory or the like (step S209).

  On the other hand, when the dissimilarity is larger than the threshold value (step S206: No), such a calculation of the shooting distance is not performed.

  Such an operation is performed for each divided area (step S210, step S211: No).

  When the above processing is performed for all the divided areas (step S211: Yes), the distance calculation unit 214 has the shortest shooting distance among the divided areas on the target image based on the shooting distance recorded in step S209. An area is specified (step S212).

  When the subject to be extracted (subject A) is closest, the subject is shown in the area specified in step S212. Therefore, the distance calculation unit 214 specifies an area in which the shooting distance is within a predetermined range among areas continuously adjacent to the specified area (Step S213), and the area specified in Step S212 and Step S213 is determined. Extracted as extraction candidates (step S214).

  When the image data of the extraction candidate area is extracted in this manner, the flow returns to the “subject extraction process” (FIG. 4).

  In the “subject extraction process”, the “extraction area correction process” for correcting the subject area extracted in the “candidate area extraction process (1)” or “candidate area extraction process (2)” described above is the subject extraction unit 215. (Step S300).

  That is, when the subject area is extracted based on the shooting distance, other subjects that are close in distance may be included. For example, if the candidate area extracted when subject A is an extraction target is a part indicated by hatching in FIG. 11A, the image indicated by this area is as shown in FIG. This is different from the subject A.

  In order to correct such erroneous recognition, the subject extraction unit 215 executes “extraction region correction processing”. This “extraction area correction process” will be described with reference to the flowchart shown in FIG.

  When the process is started, the subject extraction unit 215 performs numbering on the candidate areas (step S301), and sequentially designates candidate areas (step S302).

  At this time, the subject extraction unit 215 resets the flag (state determination flag) for determining the state of the correction operation by setting “0” (step S303).

  The subject extraction unit 215 is an area adjacent to the designated candidate area (see FIG. 11C) and has not been selected as a candidate area in the “candidate area extraction process (1)” or “candidate area extraction process (2)”. A region (hereinafter referred to as “adjacent non-candidate region”) is extracted (step S304).

  This state will be described with reference to FIG. FIG. 12A schematically shows a candidate area and another area adjacent to the candidate area for easy understanding. The candidate area shown in FIG. 12A includes a plurality of adjacent areas. Of these, areas determined to have a degree of difference greater than the threshold are excluded from “adjacent non-candidate areas”. Such areas are excluded, for example, by masking when it is determined that the degree of difference is greater than the threshold.

  When the subject extraction unit 215 designates a non-candidate region that is adjacent to the current candidate region and the difference is within a threshold value as an “adjacent non-candidate region”, the subject extraction unit 215 and the adjacent non-candidate region In both cases, a determination area as shown in FIG. 12A is set (step S305).

  This determination area is set, for example, at the boundary portion between the candidate area and the adjacent non-candidate area, and is, for example, a rectangular area having a width of one pixel. Note that the position of the determination region is not limited to the boundary portion, and may be set to a position offset from the boundary as shown in FIG.

  The subject extraction unit 215 calculates the inter-color distance between the two determination areas set in this way (step S306). Here, for example, the RGB space in each determination region is converted into a Lab space, and the inter-Euclidean distance is calculated by calculating the inter-Euclidean distance.

  The subject extraction unit 215 compares the calculated color space distance with a threshold value (step S307), and if the inter-color distance is within the threshold value (step S308: Yes), excludes the candidate area from the extraction target as the subject area. (Step S309).

  For example, in the example of FIG. 11A, in addition to the area indicating the subject A, a part of the table on which the subject A is placed is erroneously recognized as a candidate. Since the non-candidate area indicates a table portion, the distance between colors is short. Therefore, when the inter-color distance between the candidate area and the adjacent non-candidate area is within the threshold value, the candidate area can be regarded as being misrecognized, and this candidate area is excluded.

  When the candidate area is excluded based on the inter-color distance, the subject extraction unit 215 sets “1” in the state determination flag (step S310).

  Such an operation is performed for all candidate regions (step S311, step S312: No).

  When the above processing is performed for all candidate regions (step S312: Yes), the subject extraction unit 215 determines whether or not the state determination flag is “0” (step S313). As described above, when the candidate area is excluded based on the inter-color distance, the state determination flag is “1”. However, if the exclusion is not performed (No in step S308), the state determination flag is set. Remains “0”.

  When the state determination flag is “1” (step S313: No), the correction based on the inter-color distance has been performed, so the processing of steps S301 to S312 is performed on the candidate area after exclusion. In this way, the erroneous recognition area is removed in more detail.

  If such processing is repeated and correction based on the inter-color distance is no longer made (step S313: Yes), this processing ends and the flow returns to the “subject extraction processing” (FIG. 4).

  In the “subject extraction process”, the candidate area corrected by the “extraction area correction process” is extracted as a subject area (step S16). In this case, the subject extraction unit 215 generates an image showing only the subject portion from the extracted image data of the subject region, and displays and outputs the image on the display unit 310, for example. Further, the extracted subject image is stored in the storage unit 250, the external storage unit 260, or the like based on a user instruction by the operation of the operation unit 330.

  Then, the above-described processes are performed until a predetermined end event (for example, cancellation of the image extraction mode or power-off of the digital camera 1) occurs (step S17: No), and the process ends when the end event occurs. (Step S17: Yes).

  As described above, by applying the present invention as in the above embodiment, effective subject extraction from a captured image can be performed efficiently.

  In other words, since the shooting distance can be calculated based on the principle of triangulation from captured images with different optical axes obtained based on parallel stereo, it is possible to perform region extraction based on the shooting distance in units of regions on the captured image. And accurate subject extraction can be performed automatically.

  In this case, for example, since region matching based on pixel values such as color information and luminance information is performed and template matching is performed, the feature on the image of the subject portion becomes a template, and efficient template matching can be performed. .

  In addition, since the subject area is specified based on the shooting distance, the area with the shortest shooting distance can be extracted as the subject portion, and the subject area can be automatically extracted even if the subject position is not specified. Can do.

  Alternatively, when the subject position can be designated, matching is performed around the designated position, so that the processing time can be greatly shortened.

  Furthermore, since the candidate of the subject area is specified based on the shooting distance and the candidate is excluded based on the inter-color distance between the non-candidate areas adjacent to the candidate area, an area that is erroneously recognized only by the shooting distance This makes it possible to extract the subject more accurately without any problem.

  The said embodiment is an example and the application range of this invention is not restricted to this. That is, various applications are possible, and all embodiments are included in the scope of the present invention.

  In the above embodiment, an example is shown in which the region is divided based on the color information and the luminance information. However, as long as template matching can be performed, the region dividing method is not limited to this and is arbitrary.

  Further, the method of specifying the subject position shown in the above embodiment is an example, and is not limited thereto.

  In the above embodiment, the determination based on the inter-color distance is performed when correcting the misrecognized region. However, as long as the subject portion can be distinguished, the method used for the determination is arbitrary.

  In the case where the present invention is realized by an imaging apparatus such as the digital camera 1 exemplified in the above embodiment, each function of the control unit 210 can be provided in addition to the provision of the configuration and functions according to the present invention in advance. By applying a program that realizes the same function as the above, it is possible to make an existing imaging device function as the imaging device according to the present invention. In this case, it is desirable that the imaging device has a parallel stereo configuration, but even if the imaging unit is a single imaging device, if images with different optical axis positions can be acquired for the same subject, By applying the present invention, efficient subject extraction can be performed.

  For example, in the above-described embodiment, the case where the present invention is realized as a parallel stereo imaging apparatus is illustrated, but the image processing apparatus of the present invention may be realized as an apparatus having no imaging function (for example, a personal computer).

  Also in this case, by applying the program according to the present invention, it is possible to cause an existing apparatus to function as the image processing apparatus according to the present invention.

  The application method of such a program is arbitrary. For example, the program can be applied by being stored in a storage medium such as a CD-ROM or a memory card, or can be applied via a communication medium such as the Internet.

It is a block diagram which shows the structure of the digital camera concerning embodiment of this invention. It is a figure for demonstrating that the digital camera concerning embodiment of this invention is a compound eye camera (parallel stereo camera), (a) shows the external appearance structure of a digital camera, (b) shows the concept of parallel stereo. Show. It is a functional block diagram which shows the function implement | achieved by the control part shown in FIG. It is a flowchart for demonstrating the "subject extraction process" concerning embodiment of this invention. 5 is a flowchart for explaining a “candidate area extraction process (1)” executed in the “subject extraction process” shown in FIG. 4. 5 is a flowchart for explaining a “candidate area extraction process (2)” executed in the “subject extraction process” shown in FIG. 4. 5 is a flowchart for explaining an “extraction region correction process” executed in the “subject extraction process” shown in FIG. It is a figure for demonstrating the scene assumed by embodiment of this invention, (a) shows the example of the scene to image | photograph, (b) and (c) show the example of the imaged image. FIG. 7 is a diagram for explaining an operation in “candidate area extraction processing (1)” or “candidate area extraction processing (2)” shown in FIG. 5 or FIG. 6, and FIG. (B) shows an example of an area designated as a template, and (c) shows an example of an area searched by template matching. It is a figure for demonstrating the principle of a triangulation. FIGS. 8A and 8B are diagrams for explaining an operation in the “extraction area correction process” shown in FIG. 7, in which FIG. 7A shows an example when a candidate area is erroneously recognized, and FIG. An example of the subject image in this case is shown, and (c) shows an example of the region used for correcting the erroneously recognized region. FIG. 8 is a diagram for explaining an operation in the “extraction area correction process” illustrated in FIG. 7, in which (a) illustrates an example of determination areas set as a candidate area and an adjacent non-candidate area, and (b) illustrates a candidate area. Another example of the determination region set as the adjacent non-candidate region is shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 100A ... 1st imaging part, 110A ... Optical apparatus, 120A ... Image sensor part, 100B ... 2nd imaging part, 110B ... Optical apparatus, 120B ... Image sensor part, 200 ... Data processing part, 210 ... Control 211, operation mode processing unit, 212 imaging control unit, 213 image division unit, 214 distance calculation unit, 215 subject extraction unit, 220 image processing unit, 230 image memory, 240 image output unit, 250 ... storage unit, 260 ... external storage unit, 300 ... interface unit, 310 ... display unit, 320 ... external interface unit, 330 ... operation unit

Claims (10)

Image acquisition means for acquiring two captured images having different optical axis positions at the time of imaging for the same subject;
A dividing unit that divides one of the captured images acquired by the image acquiring unit into a plurality of regions;
Shooting distance calculation means for calculating a shooting distance in units of the divided areas by template matching with respect to the other of the captured images using the divided areas as templates;
Extraction means for extracting a subject area from the captured image based on the shooting distance calculated by the shooting distance calculation means;
An image processing apparatus comprising: The dividing unit divides the plurality of regions based on pixel values in the captured image.
The image processing apparatus according to claim 1. The extraction means extracts, as the subject area, an area having the shortest shooting distance and an area continuous with the area based on the shooting distance;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The extraction means extracts an area in the center of the captured image and an area continuous with the area based on the shooting distance as the subject area.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The extraction means extracts, as the subject area, an area at a position designated on the captured image and an area continuous with the area based on the shooting distance;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Further comprising face recognition means for recognizing a human face on the captured image;
The extraction means extracts a face part area recognized by the face recognition means and an area connected to the area based on the shooting distance as the subject area;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The extraction means includes
An inter-color distance calculating means for calculating an inter-color distance with a non-candidate area adjacent to the candidate area, the area extracted as the subject area;
Exclusion means for excluding the candidate region from candidates based on the intercolor distance calculated by the intercolor distance calculation means,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The image acquisition means includes a parallel stereo camera that acquires the two captured images.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An image extraction method for extracting a subject area from two captured images having different optical axis positions at the time of imaging for the same subject,
An image acquisition step of acquiring the two captured images;
A division step of dividing one of the captured images acquired in the image acquisition step into a plurality of regions;
A shooting distance calculating step of calculating a shooting distance in units of the divided areas by template matching for the other of the captured images using the divided areas as templates;
An extraction step of extracting a subject area from the captured image based on the shooting distance calculated in the shooting distance calculation step;
An image extraction method comprising: On the computer,
A function of acquiring two captured images having different optical axis positions at the time of imaging for the same subject;
A function of dividing one of the acquired captured images into a plurality of regions;
A function of calculating a shooting distance in units of the divided areas by template matching for the other of the captured images using the divided areas as templates;
A function of extracting a subject area from the captured image based on the calculated shooting distance;
A program characterized by realizing.

Images