JP5914046B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing technique, and more particularly to a technique for more suitably extracting a subject area from an input image.

  Conventionally, a technique for extracting a predetermined area (subject object area) from an image (also referred to as a segmentation technique) has been studied and applied for purposes such as image synthesis in video editing and refocusing of a specific area. Background extraction methods and chroma key methods are well known as methods based on image color information in target area extraction. The background subtraction method is a method in which an image of only a background that does not include a target area is captured in advance, an image including the target area is compared with an image of only the background, and the target area is extracted by calculating the difference. It is.

  The chroma key method is a standard technique used in the movie industry, and it is a technique for extracting the target area on the assumption that the background area is not included in the color of the target area by making the background area a certain color. . However, the background subtraction method and the chroma key method are used only in an environment where the background control is easy. On the other hand, as a method that does not require a specific background, a method of separating a target region from an image having an arbitrary background based on graph theory, that is, a method of graph cut and grab cut has been proposed (Non-Patent Document 1, Non-patent document 2).

  In the graph cut, the user first designates the pixel in the target area and the pixel in the background area by clicking with the mouse in advance, or the curve is set with the mouse for a part of the target area and a part of the background area. Specify by dragging. Then, using the specified pixel or curve as correct answer information, generate a parameter of the energy function related to the graph based on the correct answer information, and separate the target area and the background area by solving the energy function minimization problem of the graph. ing. Furthermore, in order to improve the extraction accuracy of the target region, it is possible to add the user designation to the obtained extraction result and repeat the above processing.

  Grab cut is a method for realizing the above-described graph cut more easily, and the user designation only needs to designate a rectangular area including the target area. In grab cut, color clustering is performed on the inside and outside of the target area included in the rectangular area, graph parameters are calculated based on the color information of each pixel and each cluster, and the energy function of the graph is minimized globally. The target area is extracted by solving.

  On the other hand, for object tracking and object recognition, a technique for extracting a region of interest where an object exists based on human visibility has been studied. In this technology, based on features such as color, brightness, texture, etc., an index called attention level that indicates how much attention a pixel attracts human attention is calculated for each feature, and the attention level for each feature is weighted by pixel. Attention level is extracted, and an attention area with an object is extracted (Patent Document 1, Non-Patent Document 3). Furthermore, in recent years, it has become possible to estimate distance information (depth information) for each pixel of an image by mounting a distance image sensor on a monocular camera, and distance information is used for target area extraction as useful information other than color information. Technology has been proposed (Non-Patent Document 4).

JP 2010-257423 A

Boykov, Y.Y. and Jolly, M.-P., Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images, In Proc.IEEE Int. Conf. On Computer Vision 2001, vol.1, pp. 105-112 Rother et al., Grabcut: Interactive Foreground Extraction Using Iterated Graph Cuts, ACM Trans. Graph., Vol. 23, No. 3, 2004, pp. 309-314 Itti, L., Koch, C. and Niebur, E., A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.20, Issue 11, pp. 1254-1259, 1998 Ouerhani, N. and Hugli, H., Computing visual attention from scene depth, In Proc. 15th International Conference on Pattern Recognition 2000, vol.1, pp. 375-378

  However, the above-described graph cut and grab cut methods require a user designation of the target area and the background area, which is troublesome for the user. FIG. 1 is a diagram exemplarily illustrating manual designation by a user in region extraction. The image 100a shows an example in which foreground pixels existing in the target area and background pixels existing in the background area are designated in pixel units. Here, a pixel designated as a foreground pixel is indicated by a cross indicated by white, and a pixel designated as a background pixel is indicated by a cross indicated by black. On the other hand, the image 100b shows an example in which foreground pixels and background pixels are designated by a free curve. Here, a pixel designated as a foreground pixel is indicated by a curve indicated by white, and a pixel specified as a background pixel is indicated by a curve indicated by black. Since the foreground pixel and the background pixel specified by the user are handled as correct pixels in this way, there is a problem in that if the user erroneously specifies, the extraction accuracy of the target area is affected.

  Furthermore, there is a problem of erroneous determination when there is a conspicuous region with a large change in the background of the input image. FIG. 2 is a diagram exemplarily illustrating misjudgment in a technique for extracting a region of interest with a subject. In the image 200a, the target region has a uniform color characteristic, and a portion where the change in color difference is large exists in the background region. When the target area extraction process is performed on such an image, a rectangular area surrounding a portion with a large change in color difference is determined as the attention area.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique that makes it possible to more appropriately determine and extract a subject area from an input image.

In order to solve one or more problems described above, the image processing apparatus of the present invention has the following configuration. That is, distance information input means for inputting distance information from the viewpoint for each pixel of the image, and separation means for separating the image into two or more distance layers based on distance information corresponding to each pixel included in the image; , For each distance layer , out of the pixels included in the image, pixel selection means for selecting a pixel whose distance information is within a predetermined distance range set for each distance layer, and for each distance layer, Attention level calculation means for calculating a degree of attention, which is a value indicating the degree of change of an image at each pixel position, for the first region including all of the pixels selected by the pixel selection means; and for each distance layer And determining means for determining, as an area including the subject area, a second area including all the pixels having the attention level calculated by the attention level calculating means.

  According to the present invention, it is possible to provide a technique that allows a subject region to be more appropriately determined and extracted from an input image.

It is a figure which shows an example of the manual designation | designated by the user in area | region extraction. It is a figure which shows an example of the erroneous determination of an attention area. 1 is an overall block diagram of a photographing apparatus including an image processing apparatus according to a first embodiment. 3 is a functional block diagram of an image processing unit according to the first embodiment. FIG. 3 is an overall flowchart of image processing according to the first embodiment. It is a figure which shows an example of a color image and a distance image. It is a flowchart which shows the selection process of the candidate pixel of a to-be-photographed area | region. It is a flowchart which shows attention degree calculation processing. It is a flowchart which shows subject area extraction processing. It is a figure which shows the example of the process result image by each process. It is a functional block diagram of the image processing apparatus which concerns on 2nd Embodiment. It is a whole flowchart of the image processing which concerns on 2nd Embodiment. It is a flowchart which shows a weight setting process. It is a flowchart which shows attention degree calculation processing. It is a functional block diagram of the image processing apparatus which concerns on 3rd Embodiment. It is a whole flowchart of the image processing which concerns on 3rd Embodiment. It is a figure which shows the example which divides | segments an input image into a some distance layer.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
A first embodiment of the image processing apparatus according to the present invention will be described below by taking a photographing apparatus as an example. In particular, an example will be described in which a candidate pixel of a subject area is selected based on distance information for each pixel in the image, and a degree of attention is calculated for the candidate pixel and used for determination of the subject area.

<Device configuration>
FIG. 3 is a block diagram illustrating a main part configuration of a photographing apparatus which is an embodiment that realizes the image processing apparatus according to the first embodiment. A central processing unit (CPU) 1101 comprehensively controls each unit described below.

  The imaging unit 1102 acquires image color information and distance information. Here, it is assumed that color information is input as a general color image and distance information is input as a distance image in which the value of the distance from the viewpoint is assigned to each pixel.

  A display unit 1104 displays captured images and characters. For example, a liquid crystal display is used. The display portion 1104 may have a touch screen function. A display control unit 1105 performs display control of captured images and characters displayed on the display unit 1104. An operation unit 1106 receives user instructions and includes buttons, a shooting mode dial, and the like. The setting content by this operation controls a predetermined process via the CPU. For example, manual designation of an area to be described later can be easily realized by using the display control unit 1105 and the operation unit 1106.

  The imaging control unit 1107 controls the imaging system based on instructions from the CPU 1201 such as focusing, opening / closing the shutter, and adjusting the aperture.

  The signal processing unit 1108 performs various processes such as white balance processing, gamma processing, and noise reduction processing on the digital data received via the bus 1103. The image processing unit 1109 performs image processing according to an image acquired by the imaging unit 1102, a digital image output from an image output from the signal processing unit 1108, or a user's specification from the operation unit 1106. The compression / decompression unit 1110 converts digital data or image processing results into a file format such as JPEG, MPEG, or vectorization, or performs encoding control.

  The bus 1103 is a transfer path for various data. For example, digital data acquired by the photographing unit 1102 is sent to a predetermined processing unit via the bus 1103. The internal memory 1111 functions as a main memory and work area of the CPU 1101, and stores control programs executed by the CPU 101 and the like. The external memory control unit 1112 is an interface for connecting to a PC or other media (for example, hard disk, memory card, CF card, SD card, USB memory).

  FIG. 4 is a functional block diagram showing an internal configuration of the image processing unit of the photographing apparatus according to the first embodiment. The image processing unit 1109 uses the image data acquired by the photographing unit 1102 or the image data stored in the internal memory 1111 or the external memory as a processing target, and performs subject area extraction under the control of the CPU 1101. Also, the image processed by the image processing unit 1109 is transmitted to the compression / decompression unit 1110 and compression-coded, or stored in the internal memory 1111 and the external memory, for example.

  The image processing unit 1109 includes a pixel selection unit 110, a degree-of-interest calculation unit 120, a region-of-interest determination unit 130, and a target region extraction unit 140, and is included in the image based on color information and distance information of the input image. Extract the subject area. Here, the color information of the image indicates, for example, pixel values of the three primary colors (RGB) of each pixel, and the distance information is information corresponding to the distance (depth) from the viewpoint of each pixel.

  FIG. 6 is a diagram exemplarily showing a color image and a distance image to be input. The image 600a is an example of a processing target image. In this example, a toy is placed on a pedestal and there is a portion with a color feature in the background area. The image 600b is an example of a distance image. The distance is expressed in terms of brightness. Here, the higher the brightness, the closer the distance to the photographing device. In particular, in the image 600b, in order to explain the feature of distance information in an easy-to-understand manner, the subject region portion, the boundary portion between the subject region and the background, and the background portion are represented by uniform distance values, and a pedestal that is a part of the background is displayed. It is expressed as a gradation distance. In addition, the area | region shown by black has shown that distance estimation is not completed and distance information is missing.

  The distance information may be measured by a sensor or the like, or may be estimated from images taken from a plurality of viewpoints. Further, it may be generated by the user specifying a distance for each region of the image data, or may be generated by performing the estimation described in the background art. In the case of a distance image obtained by estimation, noise (that is, a pixel indicating a distance value different from the surroundings) may generally be included.

  The pixel selection unit 110 selects a corresponding pixel on the color image whose distance value indicated by the input distance image is within a predetermined distance range as a pixel to be processed. Here, in order to simplify the description, pixels having distance information smaller than a predetermined distance threshold are selected as subject candidate pixels. On the other hand, a pixel having distance information equal to or greater than the predetermined distance threshold is determined to be a background pixel.

  The attention level calculation unit 120 obtains the attention level that is a value indicating the degree of change of the image at each pixel position of each pixel included in the color image. In particular, in the first embodiment, two or more Gaussian filters are applied to candidate pixels selected by the pixel selection unit 110, and the attention level of each pixel is obtained by taking a difference between filter results.

  The attention area determination unit 130 determines, as the attention area, an area in the color image that includes pixels within a predetermined distance range among the attention degrees calculated by the attention degree calculation unit 120. For example, a rectangular area including all pixels within a predetermined distance range is determined as the attention area.

  The target area extraction unit 140 sets energy function parameters for the target area determined by the target area determination unit 130, and solves the energy function minimization problem. Thereby, the subject area and the background area included in the attention area are separated, and for example, only the subject area is extracted. That is, the outer area of the attention area is determined as the background area and is excluded from the extraction processing target in advance.

  In the first embodiment, the image processing apparatus (image processing unit 1109) of the present invention is described as a component of the imaging apparatus. However, the image processing apparatus is a separate apparatus from the imaging apparatus. It may be configured. For example, the present invention can be realized as a form in which a personal computer (PC) executes an image processing software program and functions as each unit described above.

  The image processing apparatus in FIG. 4 is an area in which a subject exists under the control of the CPU 1101 with the captured image data from the imaging unit 1102 in the imaging apparatus in FIG. 3 or the image data stored in the internal memory 1111 and the external memory as a processing target. To extract a subject area. Also, the image processing result of the image processing apparatus in FIG. 4 is encoded in the compression / decompression unit 1110 in the photographing apparatus in FIG. 3, or stored in the internal memory 1111 and the external memory, or used for other image processing. .

<Operation of the device>
FIG. 5 is an overall flowchart of image processing according to the first embodiment.

  In step S110, the image processing apparatus inputs color information of an image to be processed and distance information for each pixel included in the image as color image data and distance image data, respectively (distance information input unit).

  In step S120, the pixel selection unit 110 (pixel selection means) selects candidate pixels in the subject area from pixels within a predetermined distance range based on the input color image data and distance image data. Then, a collection area (first area) of the selected pixels is determined. That is, the pixel that was desired to be selected is determined as the pixel in the background area. By this process, at least a part of the background pixels can be excluded in advance, aiming to reduce the processing load of the process to be described later and to improve the determination accuracy of the attention area to be described later. Details of the processing in step S120 will be described later with reference to FIG.

  In step S130, the attention level calculator 120 calculates the attention level for each of the selected candidate pixels. That is, in the set of candidate pixels, in addition to the pixels in the subject area, the background pixels at the same distance as the subject are still included. Therefore, the feature amount of each candidate pixel is calculated, and background pixels included in the candidate pixel are distinguished. Details of this processing will be described later with reference to FIG.

  In step S <b> 140, the attention area determination unit 130 determines, as the attention area (second area), a region that includes pixels having a degree of attention equal to or greater than a predetermined threshold among the candidate pixels. By this process, at least a part of the background pixels included in the candidate pixels can be excluded, and the processing load of the extraction process described later can be reduced.

  In step S150, the target area extraction unit 140 determines whether each pixel is a pixel in the subject area or a pixel in the background area, and extracts a subject area, targeting the inside of the area determined as the attention area in step S140. Details of this processing will be described later with reference to FIG.

<Candidate Pixel Selection Processing Based on Distance Information (Step S120)>
FIG. 7 is a flowchart illustrating a process for selecting candidate pixels for a subject area in the first embodiment.

  In step S1201, a distance value T for pixel selection processing is set. In step S1202, it is determined whether or not the pixel distance value is equal to or less than the set value T. If the pixel distance value is less than or equal to the set value T, the process proceeds to step S1203. If the pixel distance value is greater than the set value T, the process proceeds to step S1204.

  That is, in general, since the subject is intended to be photographed by the user, the distance from the photographing apparatus is short. On the other hand, the background portion is far from the photographing device. Therefore, if the distance range to be processed is set to a predetermined distance range closer to the background, pixels that fall within the distance range can be selected as candidate pixels.

  In step S1203, the target pixel is selected as a candidate pixel, and in step S1204, the target pixel is determined as a background pixel.

  In step S1205, it is determined whether all the pixels included in the input color image have been determined. If unprocessed pixels remain, the process from step S1202 to step S1204 is performed with the new pixel as the target pixel. On the other hand, if no unprocessed pixels remain, the selection process is terminated.

<Candidate Pixel Attention Calculation Processing (Step S130)>
FIG. 8 is a flowchart showing attention level calculation processing in the first embodiment. In particular, here, an example in which the center-surround method shown in Non-Patent Document 3 described above is executed more efficiently will be described.

In step S1301, the scale of two or more Gaussian filters is determined (scale determining means). In particular, here, the number of pixels selected as candidate pixels is counted, and the standard deviation (σ) of two or more Gaussian filters is determined based on the number. Here, as an example, two standard deviations σ ₁ and σ ₂ different from σ ₁ are determined. By determining the scale based on the number of pixels selected as candidate pixels, only a smaller number of scales are determined.

In step S1302, a Gaussian filter based on the determined scale is applied to each color image in the region corresponding to the candidate pixel. Here, a Gaussian filter having σ ₁ and σ ₂ determined in step S1301 is applied.

In step S1303, based on the processing results of the two or more Gaussian filters obtained in step S1302, the attention level of the pixel is calculated by the center-surround difference method. Here, a difference image between the processing result image by the σ ₁ Gaussian filter and the processing result image by the σ ₂ Gaussian filter is derived. Then, the value of each pixel of the difference image is derived as the attention level.

  As described above, in the first embodiment, in order to improve the efficiency of the attention degree calculation process, the processing load is reduced by limiting only to candidate pixels and a small number of Gaussian filters. In addition, here, an example of calculating the attention level using the center-surrounding difference method has been described, but in essence, as long as the attention level is derived as shown in an image 1000b described later, A method is available. For example, various edge detection filters can be used.

<Subject Area Extraction Processing (Step S150)>
FIG. 9 is a flowchart showing subject area extraction processing in the first embodiment. For the foreground / background separation, the graph cut and grab cut described in the background art and the deformation methods thereof can be used.

  In step S1501, initial setting of subject area extraction processing is performed. Here, an area outside the attention area frame is set as a background area, and an area inside the attention area frame is set as an undetermined area. In the following, it is determined whether the unconfirmed region is a pixel in the background region or a pixel in the subject region. Note that the initial value used for subject area extraction may be controlled based on the above-described pixel attention level.

  In step S1502, the color information of the background area and the undetermined area is analyzed. In this process, the background region is divided into a plurality of background clusters having different color characteristics by clustering processing, and similarly, the undetermined region is also divided into a plurality of subject candidate clusters having different color characteristics. This clustering process can use a known method of estimating a mixed Gaussian distribution.

In step S1503, an energy function parameter is calculated from the processing result of the color image. Specifically, a parameter for the target pixel is derived based on the color information and the distance information regarding a certain target pixel and a neighboring pixel of the target pixel. Specifically, the energy function parameters are calculated. For example, as a parameter
Similarity: Similarity between the target pixel and neighboring pixels Background likelihood: Degree indicating how close the target pixel is to the background Foreground likelihood: Degree indicating how close the target pixel is to the foreground.

  In step S1504, the energy function minimization problem is solved based on the parameters calculated in step S1503, and the foreground region and the background region are cut. This energy function minimization is a network flow problem in graph theory, and it is possible to use a well-known Ford Falkerson algorithm.

  In step S1505, it is determined whether the flow rate of the energy function is small or the specified number of iterations is reached. If any of the above conditions is satisfied, the process advances to step S1506 to output a foreground / background separation result. On the other hand, if none of the above conditions is satisfied, the process proceeds to step S1507, the cut foreground area and background area are set as new unconfirmed areas and confirmed background areas, respectively, and the process returns to step S1502 and the processes up to step S1504 are performed. repeat.

<Example of processing results>
FIG. 10 is a diagram illustrating an example of a result at each processing stage in the image processing according to the first embodiment.

  The image 1000a shows an example of a candidate pixel group obtained by the above selection process (step S120) for the image 600a that is a color image and the image 600b that is a distance image. In this example, the excluded background candidate pixels are represented in black, and the selected candidate pixels are represented in colors in the color image. However, as shown in the image 1000a, the “table” that is the background is located at the same distance as the subject, so the table is also selected as a candidate pixel.

  The image 1000b is an example of the calculation result of the attention level. The image 1000b represents the degree of attention of each pixel in terms of lightness, and indicates that the higher the lightness, the more visually noticeable. In this example, in order to explain the tendency of the attention level in an easy-to-understand manner, the boundary portion between the subject and the background, the separation portion of the pedestal, and the other background portions are represented by uniform attention values.

  The image 1000c shows an example in which the attention area is determined as a rectangular area including pixels having a degree of attention greater than a predetermined value. Here, it can be seen that most of the “table” included in the image 1000a is excluded as a background.

  An image 1000d shows an example in which a subject is suitably extracted by performing subject region extraction processing on the target region shown in the image 1000c. That is, the background feature portion shown in the image 600a has been successfully prevented from being erroneously extracted as a subject.

  As described above, according to the first embodiment, by using distance information corresponding to each pixel of the input image, an image region that can be determined to be the background is excluded in advance from the extraction target. Thereby, an erroneous determination as shown in the image 200b of FIG. 2 can be prevented in advance. As a result, the subject area can be extracted more accurately even for a complex image. In the first embodiment, the process for solving the energy function minimization problem (step S1504) with a relatively large processing load is executed only for a partial area (attention area) of the input image. The load can be greatly reduced.

  As described above, in the first embodiment, the scale (standard deviation) of the Gaussian filter is determined based on the number of candidate pixels selected based on the distance information. This makes it possible to efficiently generate the attention level image with a smaller number of Gaussian filters than in the past. In the above description, only the color information is used for calculating the attention level. However, brightness, texture, etc. may be further used in addition to the color information. In that case, the degree of attention calculated from each feature is weighted.

(Second Embodiment)
In the second embodiment, a mode in which a weight of attention is set for a candidate pixel based on distance information and a region of interest is determined based on the weighted attention will be described.

<Device configuration>
FIG. 11 is a functional block diagram of the image processing apparatus according to the second embodiment. The image processing apparatus includes a weight setting unit 210, an attention level calculation unit 220, an attention region determination unit 230, and a target region extraction unit 240. Then, as in the first embodiment, a subject area included in the image is extracted based on the color information and distance information of the input image. That is, the part provided with the weight setting part 210 instead of the pixel selection part 110 differs from 1st Embodiment. The other functional units are the same as the corresponding functional units in the first embodiment, and thus description thereof is omitted.

  The weight setting unit 210 sets a weight for the attention level calculated by the attention level calculation unit 220 based on whether or not the distance value of each pixel is within a predetermined distance range. Here, a high weight is set for pixels whose distance value is within the predetermined distance range, and a low weight is set for pixels outside the predetermined distance range.

<Operation of the device>
FIG. 12 is an overall flowchart of image processing according to the second embodiment. In step S220, a weight for the attention level is set, and in step S230, the attention level based on the set weight is calculated, which is different from the first embodiment. Details of this processing will be described later with reference to FIGS.

<Attention level weight setting process based on distance information (step S220)>
FIG. 13 is a flowchart showing the weight setting process.

  In step S2201, a distance value T for pixel selection processing is set. In step S2202, it is determined whether or not the pixel distance value is equal to or smaller than the set value T. If the pixel distance value is less than or equal to the set value T, the process proceeds to step S2203. If the pixel distance value is greater than the set value T, the process proceeds to step S2204.

  In step S2203, the weight for the pixel of interest is set to a high value (for example, a value greater than 1), and in step S2204, the weight for the pixel of interest is set to a low value (for example, a value less than 1).

  In step S2205, it is determined whether all the pixels included in the input color image have been determined. If unprocessed pixels remain, the process from step S2202 to step S2204 is performed with the new pixel as the target pixel. On the other hand, if no unprocessed pixels remain, the weight setting process is terminated.

<Pixel Attention Level Calculation Processing (Step S230)>
FIG. 14 is a flowchart showing attention level calculation processing in the second embodiment.

FIG. 8 is a flowchart showing attention level calculation processing in the first embodiment. Similar to the first embodiment, an example based on the center-surround method shown in Non-Patent Document 3 will be described.

  In step S2301, a plurality of Gaussian filters set with different scales (standard deviation σ) for the entire input color image are applied.

  In step S2302, the attention level of the pixel is calculated by the center-surround difference method based on the processing results of the plurality of Gaussian filters obtained in step S2301.

  In step S2303, the attention degree is weighted by the weight set in the weight setting process (step S220) with respect to the attention degree of each pixel calculated in step 2302.

  As described above, in the second embodiment, by assigning the attention level to the candidate pixels based on the distance information, it is possible to appropriately set the attention area excluding the background area, as in the first embodiment. It becomes possible. In the above description, the case where the pixel selection unit 110 is not used in the first embodiment has been described. However, the pixel selection unit 110 may be used together.

(Third embodiment)
In the third embodiment, a description will be given of a mode in which a plurality of subject areas can be automatically extracted more suitably when there are a plurality of subject areas to be extracted in the input image.

<Device configuration>
FIG. 15 is a functional block diagram of an image processing apparatus according to the third embodiment. The image processing apparatus includes a distance layering unit 310, a region of interest determination unit 320, and a target region extraction unit 330. Then, similarly to the first embodiment and the second embodiment, a subject region included in the image is extracted based on the color information and distance information of the input image.

  The distance layering unit 310 divides the distance range into two or more ranges based on the input distance image, and selects pixels included in each distance range. The attention area determination unit 320 calculates a degree of attention by determining pixels included in each distance layer, and determines a region having a high degree of attention. The target area extraction unit 330 extracts the subject area in each layer from the attention area in each distance layer.

<Operation of the device>
FIG. 16 is an overall flowchart of image processing according to the third embodiment.

  In step S301, the image processing apparatus inputs color information of an image to be processed and distance information for each pixel included in the image as color image data and distance image data, respectively.

  In step S302, the distance layering unit 310 layers the color image based on the input distance image. Specifically, clustering processing of pixels included in the color image is performed based on distance information of each pixel derived from the input distance image, and the pixels included in the color image are separated into a plurality of pixel groups. That is, each pixel group is set as a set of a plurality of pixels having distance values similar to each other, and is hereinafter also referred to as a distance layer.

  In step S303, the distance layering unit 310 sets a distance range for selecting a processing target pixel to be described later in each distance layer based on the result of distance clustering. Thereafter, from step S304 to step S308, for each distance layer, determination of the attention area by the attention area determination section 320 and extraction of the subject area by the target area extraction section 330 are individually processed.

  In step S304, the attention area determination unit 320 determines whether or not the pixel distance value is within the distance range set for the attention distance layer in the attention distance layer. If the distance value of the pixel is within the set value of the distance range, the process proceeds to step S305, and the pixel is set as a processing target pixel. On the other hand, if the distance value of the pixel is out of the distance range, the process proceeds to step S306, and the pixel is set as a non-processing target pixel. This process is executed for all pixels included in the target distance layer.

  In step S307, the attention area determination unit 320 determines the attention area for the processing target pixel. Note that the attention area determination processing is the same as the attention calculation processing and determination processing (steps S130 and S140) in the first embodiment and the second embodiment described above, and a description thereof will be omitted.

  In step S <b> 308, the target area extraction unit 330 determines whether each pixel is a pixel in the subject area for the inside of the area determined as the attention area, and extracts the subject area. The extraction process is the same as the extraction process (step S150) in the first embodiment and the second embodiment described above, and a description thereof will be omitted.

  In step S309, the image processing apparatus determines whether all distance layers have been processed. If an unprocessed distance layer remains, the process from step S304 to step S308 is performed with the new distance layer as the target distance layer. On the other hand, if there is no unprocessed distance layer remaining, the process ends.

<Example of distance layer image>
FIG. 17 is a diagram illustrating an example in which an input image is divided into a plurality of distance layers. In the image 1700a, there are two subjects (target regions) “flowers” and “butterflies”. Furthermore, there is a “background feature” near the target area of the flower. And the distance value with respect to each area | region is shown by the image 1700b. Similar to the image 600b in FIG. 6, the distance is expressed by brightness, and here, the higher the brightness, the closer the distance to the photographing device. In this case, the image 1700a is set as three distance layers of the subject “flower”, the subject “butterfly”, and the “background feature”.

  In this way, by layering according to the distance range and performing target area extraction processing on each layer, features included in the background while preventing omission of extraction of the subject (here “flower” and “butterfly”) Parts can also be extracted.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (6)

Distance information input means for inputting distance information from the viewpoint for each pixel of the image;
Separation means for separating the image into two or more distance layers based on distance information corresponding to each pixel included in the image;
For each of the distance layer, among the pixels included in the image, a pixel selection means for selecting a pixel within a predetermined distance range in which the distance information is set for each distance layer,
Attention level calculation for calculating a degree of attention, which is a value indicating the degree of change in the image at each pixel position, for the first region including all of the pixels selected by the pixel selection unit for each distance layer. Means,
Determining means for determining, for each of the distance layers, a second area including all of the pixels having the attention degree calculated by the attention degree calculation means equal to or greater than a predetermined threshold;
An image processing apparatus comprising: The attention level calculation means applies two or more different Gaussian filters to the first region, and calculates the attention level as a difference between filter results of the two or more Gaussian filters. Item 8. The image processing apparatus according to Item 1. 3. The image according to claim 2, wherein the attention level calculation unit includes a scale determination unit that determines a scale of each of the two or more Gaussian filters based on the number of pixels included in the first region. Processing equipment. The determining means is configured to determine the second region as a rectangular region;
The image processing apparatus according to claim 1, further comprising an extraction unit that extracts the subject area from the second area. A distance information input step in which the distance information input means inputs distance information from the viewpoint for each pixel of the image;
A separation step of separating the image into two or more distance layers based on distance information corresponding to each pixel included in the image;
Pixel selection means, for each of the distance layer, among the pixels included in the image, a pixel selection step of selecting the pixels within a predetermined distance range in which the distance information is set for each distance layer,
The attention level is a value indicating the degree of change of the image at each pixel position for the first region including all of the pixels selected by the pixel selection step for each distance layer. Attention level calculation step for calculating
A determining unit that determines, for each of the distance layers, a second region that includes all pixels having an attention level calculated by the attention level calculation step equal to or greater than a predetermined threshold as a region that includes the subject region; ,
An image processing method comprising: The program for functioning a computer as each means of the image processing apparatus as described in any one of Claims 1 thru | or 4 .