JP7172066B2 - Information processing device and program - Google Patents

Description

The present invention relates to an information processing apparatus and program.

Techniques for recognizing an image of an object included in a captured image by image analysis are currently being researched. For example, when a user designates a part of a color image containing distance information as the foreground or background, there is a technique for recognizing the area that is not designated by analyzing the image.

JP 2010-039999 A

However, it is difficult to apply the method in which the user designates an area for each image when there are many images to be processed or when real-time processing is required.

The present invention makes it possible to specify an area to be processed without assuming manual specification of the area.

The invention according to claim 1 comprises extraction means for extracting a range in which depth distribution corresponding to each pixel constituting a captured image appears intensively, and at least a part of the extracted pixels belonging to the range. as a starting point for specifying an area to be processed.
The invention according to claim 2 is the information processing apparatus according to claim 1, wherein the extracting means extracts a distribution of depths satisfying a predetermined condition as the range.
The invention according to claim 3 is the information processing apparatus according to claim 2, wherein the condition is that the shape of the distribution resembles the shape of a predetermined function.
The invention according to claim 4 is the information processing apparatus according to claim 3, wherein the function is a unimodal function.
The invention according to claim 5 is the information processing apparatus according to claim 2, wherein the condition is that a frequency exceeding a predetermined threshold appears.
The invention according to claim 6 is the information processing apparatus according to claim 1, wherein the setting means sets, as the starting point, a pixel at a depth where appearance of pixels is concentrated even in the extracted range.
The invention according to claim 7 is the information processing apparatus according to claim 6, wherein the at least some pixels correspond to a depth at which the number of pixels appearing in the range is maximized.
The invention according to claim 8 is the information processing apparatus according to claim 6, wherein the at least some pixels correspond to a depth included in a range that is a constant multiple of the standard deviation with respect to the median value of the range. be.
The invention according to claim 9 is the information processing apparatus according to claim 1, wherein the image has pixels associated with a depth distribution .
The invention according to claim 10 is the information processing apparatus according to claim 9, wherein the depth distribution corresponding to each pixel is given as parallax of a plurality of imaging means.
The invention according to claim 11 is the information processing apparatus according to claim 9, wherein the depth distribution corresponding to each pixel is given as a measurement value when capturing the image.
The invention according to claim 12 is the information processing apparatus according to claim 1, wherein the image is divided into a plurality of regions using the values of the pixels set as the starting points.
The invention according to claim 13 is the information processing apparatus according to claim 1, wherein the area to be processed is selectively identified based on a predetermined rule.
The invention according to claim 14 is the information processing apparatus according to claim 13, wherein a part of the area specified through the starting point is selected as a processing target according to a predetermined rule.
The invention according to claim 15 is the information processing apparatus according to claim 14, wherein the predetermined rule is to give priority to an area near the center of the image.
The invention according to claim 16 is the information processing apparatus according to claim 14, wherein the predetermined rule prioritizes an area where the user's line of sight is concentrated.
17. The information according to claim 14, wherein the selection based on the rule is performed when a difference in depth of the plurality of regions identified through the starting point is within a predetermined range. processing equipment.
According to an eighteenth aspect of the present invention, a computer comprises an extracting means for extracting a range in which depth distribution corresponding to each pixel constituting a captured image appears in a concentrated manner; The program causes at least a part of the program to function as setting means for setting a starting point for specifying a region to be processed.

According to the first aspect of the invention, the area to be processed can be specified without the assumption that the area is specified manually.
According to the second aspect of the present invention, it is possible to improve the accuracy of estimating a candidate region to be processed.
According to the third aspect of the present invention, it is possible to improve the accuracy of estimating a candidate region to be processed.
According to the fourth aspect of the present invention, it is possible to improve the accuracy of estimating a candidate region to be processed.
According to the fifth aspect of the present invention, it is possible to improve the accuracy of estimating a candidate region to be processed.
According to the sixth aspect of the present invention, it is possible to improve the accuracy of estimating a candidate region to be processed.
According to the seventh aspect of the present invention, it is possible to improve the accuracy of estimating a candidate region to be processed.
According to the eighth aspect of the present invention, it is possible to improve the accuracy of estimating candidates for the area to be processed.
According to the ninth aspect of the invention, it is possible to cope with the case where the number of images is large and the case where real-time processing is required.
According to the tenth aspect of the present invention, it is possible to cope with the case where the number of images is large and the case where real-time processing is required.
According to the eleventh aspect of the invention, it is possible to cope with the case where the number of images is large and the case where real-time processing is required.
According to the twelfth aspect of the present invention, it is possible to estimate a candidate region to be processed by calculation.
According to the thirteenth aspect of the present invention, it is possible to efficiently estimate candidates for the area to be processed even when the area cannot be specified manually.
According to the fourteenth aspect of the invention, if a rule is defined in advance, a region to be processed can be selected without requiring any special operation.
According to the fifteenth aspect of the present invention, if a rule is defined in advance, a region to be processed can be selected without requiring any special operation.
According to the sixteenth aspect of the invention, if a rule is defined in advance, a region to be processed can be selected without requiring any special operation.
According to the seventeenth aspect of the invention, if a rule is defined in advance, a region to be processed can be selected without requiring any special operation.
According to the eighteenth aspect of the invention, it is possible to specify the area to be processed without assuming that the area is specified manually.

FIG. 2 is a diagram for explaining an example of usage in the first embodiment; FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to Embodiment 1; FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device according to Embodiment 1; FIG. FIG. 4 is a diagram for explaining the relationship between subject and depth information; (A) shows the relationship between each part of the wine glass and depth, and (B) shows a histogram of the wine glass. FIG. 4 is a diagram illustrating an example of a histogram generated for one image; FIG. It is a figure explaining the waveform of a probability density function, and the histogram after clustering. (A) shows the waveform of the probability density function, and (B) shows the histogram after clustering. It is a figure which shows an example of the method of selecting the pixel set as a representative pixel (seed). (A) shows an example of a histogram after clustering, and (B) shows an example of a method used for selection. FIG. 10 is a diagram showing another example of a method of selecting pixels to be set as representative pixels (seeds); It is a figure explaining the relationship between a cluster and a representative pixel (seed). (A) shows the distribution of clusters, and (B) shows an example of mapping representative pixels (seeds) to an image. It is a figure explaining the relationship before and behind the process by an area|region division part. (A) shows the state immediately after the representative pixels (seeds) are mapped, and (B) shows the state at the time when the region division is completed. It is a figure explaining an example of the pinpointed field. FIG. 5 is a diagram illustrating how image processing changes the color of wine in a wine glass near the center of the screen; (A) shows an example of an image before change, and (B) shows an example of an image after change. 4 is a flowchart for explaining processing operations performed by an information processing apparatus; FIG. 10 is a diagram illustrating an example of a hardware configuration of an information processing device according to a second embodiment; FIG. FIG. 10 is a diagram illustrating an example of a functional configuration of an information processing device according to Embodiment 2; 4 is a diagram illustrating a functional configuration executed by a depth information acquisition unit; FIG. FIG. 11 is a diagram for explaining an example of a usage form in Embodiment 3; FIG. 12 is a diagram illustrating an example of a hardware configuration of an information processing device according to Embodiment 3; FIG. It is a figure which shows the structural example of the filter with which the aperture part of the lens of a camera with a filter is mounted|worn. FIG. 13 is a diagram illustrating an example of a functional configuration of an information processing device according to Embodiment 3; FIG. 11 is a diagram for explaining an example of usage in Embodiment 4; It is a figure explaining the basic composition of an information processing apparatus. FIG. 12 is a diagram illustrating an example of a functional configuration of an information processing device according to a fourth embodiment; FIG. FIG. 13 is a diagram illustrating another example of the functional configuration of the information processing device according to Embodiment 4;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Embodiment 1>
<Mode of use>
FIG. 1 is a diagram for explaining an example of a mode of use according to Embodiment 1. FIG.
FIG. 1 shows a scene in which objects (table 10, wine glasses 11 and 12, plate 13, trash can 14) that actually exist in a real space are imaged by an information processing device 20 with a camera function. There is a wall behind the table 10, and the table 10 is placed on the floor.
FIG. 1 shows a case where the information processing device 20 is a so-called smart phone. However, the information processing device 20 may be a so-called digital camera or video camera.

In the case of the present embodiment, the information processing device 20 captures an image of an object that actually exists in a real space in the form of a still image or a moving image. Each pixel of a still image or the like obtained by imaging a real space does not contain information indicating belonging to a specific region. In other words, still images and the like handled by the information processing apparatus 20 do not contain object information.
Here, an object refers to a unit of handling defined in a virtual space constructed by a computer, and given as a collection of pixels, a collection of polygons, or a collection of voxels.

<Device configuration>
FIG. 2 is a diagram illustrating an example of the hardware configuration of the information processing device 20 according to the first embodiment.
The information processing device 20 includes a processing circuit unit 200 that processes data, a nonvolatile memory 211 that stores programs and data, an audio circuit 212 that inputs and outputs audio signals, and a liquid crystal display (LCD) 213 that displays images. , a power control device 214, a camera 215 that captures an image of an object that actually exists in the real space, a distance sensor 216 that measures the distance to the object that actually exists in front of the camera 215, and a touch panel that detects user operations. 217, a WiFi (Wireless Fidelity) module 218 that transmits and receives wireless signals conforming to the WiFi (registered trademark) standard, and a wireless signal that conforms to the Bluetooth (registered trademark) standard, which is one of short-range wireless communication standards. and a Bluetooth module 219 .

In this embodiment, the distance sensor 216 is, for example, an optical sensor or an ultrasonic sensor. Optical sensors include, for example, an infrared sensor composed of an infrared LED (Light Emitting Diode) that emits infrared rays and a phototransistor that reacts to infrared rays. Note that the distance sensor 216 measures the flight time (time difference) of the light emitted from the light source until it reaches the sensor after being reflected by the object for each image, thereby obtaining the distance to the object (TOF). Flight) type sensor may be used.

Note that the processing circuit unit 200 in this embodiment includes a memory control unit 201 that controls reading and writing of data with a memory 211, a plurality of CPU cores (Cores) 202 that execute processing, and power supply. A power management unit 203 to manage, a system management unit 204 to manage the operation of the embedded system, an audio processing unit 205 to process audio signals, a GPU (Graphics Processing Unit) 206 to process images in real time, and a liquid crystal display 213 is provided with a display control unit 207 for displaying images, an external interface (I/F) 208 used for connection with an external module, and a baseband circuit 209 for processing baseband signals.

<Functional configuration>
FIG. 3 is a diagram illustrating an example of the functional configuration of the information processing device 20 according to the first embodiment.
The functional configuration shown in FIG. 3 is realized through program execution by the CPU core 202 .
From the viewpoint of functions, the information processing apparatus 20 includes an image information acquisition unit 221 that acquires image data from the camera 215, and depth information that acquires depth information indicating the distance between the distance sensor and the subject in the depth direction from the distance sensor 216. An acquisition unit 222, a depth histogram generation unit 223 that aggregates the number (frequency) of pixels that appear for each depth and generates a histogram, and depth clustering that extracts a set of pixels having similar depths in the depth direction as clusters. an execution unit 224; a seed coordinate setting unit 225 that sets the coordinates of pixels belonging to a cluster as seed coordinates that give a representative pixel (seed) in the screen; A region dividing unit 226 that divides an image into a plurality of regions by repeating integration, a target region identifying unit 227 that identifies a region to be processed among the divided regions, and a predetermined and an image processing unit 228 for applying processing.

Here, the image information acquisition unit 221 acquires pixel values (for example, RGB values) of pixels forming one screen for image data corresponding to a still image or a moving image. The image data may be given as a color image or as a grayscale image. It should be noted that depth information is associated with the image data here.
The depth information acquisition unit 222 acquires depth information from the distance sensor 216 at the same time as the image is captured.

FIG. 4 is a diagram for explaining the relationship between subject and depth information. (A) shows the relationship between each part of the wine glass 11 and the depth, and (B) shows a histogram of the wine glass 11. FIG.
In the case of FIG. 4, the depth of the handle (stem) of the wine glass 11 is 105 cm, the depth of the front side of the body (bowl) filled with wine is 100 cm, and the mouth (lip or rim) is 100 cm. ) is 110 cm. That is, the barrel of the wine glass 11 shown in FIG. 4 has a paraboloid with a diameter of 10 centimeters. In the case of this example, the concentration of depth is recognized around 105 cm, which is the distance to the handle.
In real space, other objects exist around the wineglass 11 (see FIG. 1). Thus, the pixels surrounding the wineglass 11 are assigned depths to other objects.

Since the surfaces forming one object are continuous, the depth of the pixels corresponding to the object appears to spread around a specific depth.
However, since the shape of the object to be photographed is arbitrary (it may not be rotationally symmetrical like a wine glass, and may have uneven or irregular shapes on its surface), the depth distribution is an ideal probability. It is not necessarily the density distribution. However, the distribution of depths corresponding to one object basically tends to appear in a mountain shape.
Returning to the description of FIG.

The depth histogram generation unit 223 generates a histogram based on the acquired depth information. The histogram in the present embodiment represents the distribution of depth values obtained for each pixel that constitutes one image.
FIG. 5 is a diagram explaining an example of a histogram generated for one image. In the figure, the horizontal axis is depth and the vertical axis is frequency. Note that the unit of the horizontal axis is centimeters.
In the case of FIG. 5, numbers are attached to the chevron waveforms in the order in which they appear.
The histogram generated by the depth histogram generation unit 223 also contains a noise component (noise). In the example of FIG. 5, the 6th waveform corresponds to the noise component.
It should be noted that the depth of the pixels corresponding to the ground or road extending from near to the point to be imaged is likely to have a flat shape rather than a chevron waveform. In addition, the depth of the pixels corresponding to the wall appears on the back side of the depth of the pixels corresponding to other objects forming the foreground.
Returning to the description of FIG.

The depth clustering execution unit 224 identifies depths that satisfy a predetermined condition as clusters. For example, the depth clustering execution unit 224 identifies, as a cluster, a range of depths in which a distribution similar to the shape of a predetermined function appears. Further, for example, the depth clustering execution unit 224 identifies, as a cluster, a range of depths in which frequencies exceeding a predetermined threshold appear.
A function used for clustering is called a basis function. In this embodiment, probability density functions are used as basis functions. The probability density function has a unimodal characteristic, and the frequency of 3σ (standard deviation) or more is almost 0 (zero).
Although the probability density function is used as the basis function in this embodiment, other functions may be used. Alternatively, a plurality of probability density functions may be prepared, and the default function to be used may be switched according to the environment, scene, subject, etc. at the time of imaging.

FIG. 6 is a diagram for explaining the waveform of the probability density function and the histogram after clustering. (A) shows the waveform of the probability density function, and (B) shows the histogram after clustering. It should be noted that the basis functions include, for example, a normal distribution as a discrete function.
In the case of FIG. 6, the waveform that appears sixth in FIG. 5 is removed by the clustering process. As a result, six chevron waveforms remain in the histogram after clustering.
Note that each chevron-shaped waveform does not necessarily correspond to one object that actually exists in the actual space. As described above, the depth distribution corresponding to each object tends to be a chevron waveform (see FIG. 4), but there is a possibility that multiple objects with overlapping depth distributions exist within one screen. Therefore, pixels belonging to one cluster do not necessarily correspond to one object. Of course, pixels belonging to one cluster may correspond to one object.
The depth histogram generation unit 223 and the depth clustering execution unit 224 here are examples of extraction means.
Returning to the description of FIG.

The seed coordinate setting unit 225 sets the coordinates of all or part of the pixels belonging to the cluster as seed coordinates. In this embodiment, some of the pixels belonging to the cluster are used as representative pixels (seeds) in the screen. The seed coordinate setting unit 225 here is an example of setting means.
FIG. 7 is a diagram showing an example of a method of selecting pixels to be set as representative pixels (seeds). (A) shows an example of a histogram after clustering, and (B) shows an example of a method used for selection.
FIG. 7 shows three examples of a method of selecting a representative pixel (seed) from the cluster appearing fifth in the histogram. In FIG. 7, the corresponding clusters are shown enclosed by dashed lines.

Example 1 is a case where all pixels belonging to a cluster are selected as representative pixels (seeds). This technique allows the largest number of pixels to be used as representative pixels (seeds). On the other hand, information about pixels belonging to other objects is also likely to be included.
Example 2 is a case where a pixel corresponding to the depth with the maximum frequency is selected as a representative pixel (seed). This approach is likely to result in pixels belonging to a single object even in real space. On the other hand, since the number of pixels forming the representative pixel (seed) is reduced, the computational load for expanding the area defined by the pixels forming the representative pixel (seed) increases.
Example 3 is a case of selecting, as representative pixels (seeds), pixels whose distribution is included in the range of standard deviation σ with respect to depths (median values of depths corresponding to clusters) with the highest frequency. In this method, not only is there a large number of pixels forming representative pixels (seeds), but there is also a high possibility that the pixels belong to a single object in real space. Note that the range selected as the representative pixel (seed) can be adjusted by changing the magnitude of the constant that is multiplied by the standard deviation σ.

FIG. 8 is a diagram showing another example of a method of selecting pixels to be set as representative pixels (seeds).
In FIG. 8, a depth range having a frequency exceeding a predetermined threshold TH is selected as a representative pixel (seed).
In Example 4 shown in FIG. 8, the third cluster with a low appearance frequency is excluded from the representative pixels (seed). A plurality of thresholds TH may be prepared and selectively applied. Also, the threshold TH may be changed for each cluster. For example, the threshold TH may be selected according to the magnitude of the maximum value of each cluster.

Note that when example 2 (see FIG. 7) or example 4 (see FIG. 8) is used to select representative pixels (seeds), target pixels are selected without executing histogram generation or clustering. It is possible. That is, it is possible to select pixels to give representative pixels (seeds) without using the depth histogram generation unit 223 (see FIG. 3) or the depth clustering execution unit 224 (see FIG. 3).
However, with this method alone, there is a possibility that pixels corresponding to concentration of depths corresponding to noise components are selected as representative pixels (seeds), unlike clustering with respect to histograms.

FIG. 9 is a diagram for explaining the relationship between clusters and representative pixels (seeds). (A) shows the distribution of clusters, and (B) shows an example of mapping representative pixels (seeds) to an image.
A representative pixel (seed) corresponding to one cluster often represents a difference in position in the depth direction of one existing object. Therefore, representative pixels (seeds) tend to appear as clusters in the image.
Returning to the description of FIG.

The region dividing unit 226 uses a known technique to integrate the surrounding pixels that are found to be related to the region defined by the individual representative pixels (seed), and executes the process of dividing the image into a plurality of regions. . The region dividing unit 226 uses, for example, a Graph-Cut method, a Grow-Cut method, a Cellular-Cut method, or the like.
The graph cut method is a method in which adjacent pixels with colors similar to the representative pixel (seed) are combined to expand the area, and the boundary of the area is set at the portion where the color difference between the adjacent pixels is large. .
The grow cut method is a method of allocating all pixels to any region based on the difference in pixel value between adjacent pixels.
In the cellular cut method, all pixels on the screen are regarded as cells, the starting point pixel and surrounding pixels are related with weights according to the closeness of color characteristics, and the multiplied weights are propagated to the surroundings. is a method of determining pixels belonging to a region by

10A and 10B are diagrams for explaining the relationship before and after the processing by the area division unit 226. FIG. (A) shows the state immediately after the representative pixels (seeds) are mapped (see FIG. 9), and (B) shows the state at the time when the region division is completed.
In FIG. 10, the screen has an area corresponding to the background corresponding to the wall and floor, an area corresponding to the table 10, an area corresponding to the wine glass 11, an area corresponding to the wine glass 12, an area corresponding to the plate 13, It is divided into six areas corresponding to the trash can 14 .
Returning to the description of FIG.

The target region specifying unit 227 selectively specifies a region to be processed from the divided regions based on a predetermined rule.
The number of selectively specified regions need not be limited to one. When the regions divided based on the rule are ranked, a predetermined number of regions may be selected from the higher regions.
However, all regions may be specified as targets for processing.
Note that the specified number may vary depending on the purpose of the processing.

Here, it is desirable that the rule used for identifying the area be selected according to the content of the processing. One or more rules may be used.
Also, rule-based selection may be performed only when the difference in depth between the multiple regions is within a predetermined range (when the depths of the multiple regions are approximately equal).

One of the rules here is, for example, priority for areas near the center in the screen. This is because when an image is captured by the information processing apparatus 20, the area that the user is paying attention to is often located in the center of the screen.
Other rules include prioritizing regions where the user's gaze is concentrated. The use of this rule is based on the assumption that the information processing apparatus 20 has a built-in sensor that detects the direction of the user's line of sight. This is because the line-of-sight direction may be regarded as the area on which the user is paying attention.

Other rules include, for example, prioritizing large area areas within the screen. However, since the area of the area corresponding to the background tends to increase, priority may be given to areas with the second and subsequent areas, or the size of the area to be prioritized may be restricted. It should be noted that there may be a rule that prioritizes areas with small areas.
Other rules may prioritize areas containing coordinates sensed on touch panel 217 (see FIG. 2). This is because the portion designated on the touch panel 217 is the area that the user is paying attention to. The touch operation here is an operation different from specifying a representative pixel (seed).

FIG. 11 is a diagram illustrating an example of the specified area.
In FIG. 11, images corresponding to objects that actually exist in the actual space are represented as a table image 10A, wine glass images 11A and 12A, a plate image 13A, and a trash can image 14A.
In addition, in FIG. 11, the area specified by the target area specifying unit 227 (see FIG. 3) is surrounded by a rectangular dashed line to express the specified state. In the example of FIG. 11, the wine glass images 11A and 12A and the plate image 13A placed on the table image 10A are to be processed.
Returning to the description of FIG.

The image processing unit 228 executes processing such as changing the impression and texture of the identified region. The processing here is performed by combining, for example, changing the color distribution within the region, the boundary of the region, and the surface of the region. In this embodiment, these processes are called image processing. However, the processing of other functional units shown in FIG. 3 is also one form of image processing.
FIG. 12 is a diagram for explaining how the color of wine in a wine glass near the center of the screen is changed by image processing. (A) shows an example of the screen before change, and (B) shows an example of the screen after change.
Note that the image processing unit 228 outputs the result of image processing to the liquid crystal display 213 (see FIG. 2).

<Processing operation>
Here, the processing operation of the information processing device 20 (see FIG. 1) will be described.
FIG. 13 is a flow chart for explaining the processing operations performed by the information processing device 20 .
The information processing apparatus 20 starts the process shown in FIG. 13 when the user instructs the information processing apparatus 20 to start imaging through an operation (not shown).
First, the information processing apparatus 20 acquires depth information for each pixel at the same time as capturing an image (step 101). Note that S in the figure is an abbreviation for a step.

Next, the information processing device 20 tallies the frequencies for each depth (step 102). In this embodiment, a histogram is generated after aggregation.
Subsequently, the information processing device 20 performs clustering and excludes noise components and the like (step 103). In the present embodiment, depths where frequency occurrences resemble a probability density distribution are identified. Due to this clustering, planes with depth, such as roads, are excluded from extraction targets. Also, since the number of clusters defines the number of regions after division, division into three or more regions is also possible.
The information processing device 20 sets the coordinates of pixels corresponding to individual clusters as starting points (ie, seed coordinates) for dividing the image into a plurality of regions (step 104). Since the starting point can be set for each cluster according to a predetermined rule, there is no need to manually specify the starting point.

Next, the information processing device 20 divides the image into a plurality of regions using the seed pixel values (step 105). Here, the pixel value (ie, image) resolution is higher than the depth resolution obtained by the range sensor 216 (see FIG. 2). Therefore, it is possible to improve the accuracy of division as compared with the case where only depth information is used to divide an area. Also, by using pixel value information, it is possible to deal with an area corresponding to an undefined object. The result of region division according to the present embodiment is the same as when seed coordinates are given manually (graph cut method, grow cut method, cellular cut method, etc.).
Subsequently, the information processing apparatus 20 specifies a part of the divided area as a processing target based on the rule (step 106).
After that, the information processing device 20 executes predetermined image processing and reflects it on the screen display (step 107).

<Embodiment 2>
In this embodiment, a case of acquiring depth information using a so-called stereo camera method will be described.
The stereo camera method is a method of acquiring the depth of each pixel in an image from two images (stereo images) of an object that actually exists in a real space taken simultaneously by two cameras.
FIG. 14 is a diagram illustrating an example of the hardware configuration of the information processing device 20A according to the second embodiment. In FIG. 14, parts corresponding to those in FIG. 2 are shown with reference numerals.
The information processing apparatus 20A shown in FIG. 14 differs from the information processing apparatus 20 (see FIG. 1) in that two cameras 215R and 215L are used and the distance sensor 216 (see FIG. 2) is not used.

FIG. 15 is a diagram illustrating an example of the functional configuration of the information processing device 20A according to the second embodiment. In FIG. 15, parts corresponding to those in FIG. 3 are shown with reference numerals.
In the case of the information processing apparatus 20A according to the present embodiment, image data output from one of the two cameras 215R and 215L is provided to the image information acquiring section 221. FIG. In the example of FIG. 15, the image data output from the camera 215R is given to the image information acquiring section 221. In the example of FIG.
Also, the depth information acquisition unit 222A is supplied with image data from both the two cameras 215R and 215L.

FIG. 16 is a diagram illustrating a functional configuration executed by the depth information acquisition unit 222A.
The depth information acquisition unit 222A here performs a process of averaging the original image to generate an averaged image (for example, a process of averaging pixel values for each 3×3 pixels), and a difference between the original image and the averaged image. A smoothing/sharpening unit 250 that executes a sharpening process that emphasizes the change by superimposing the . and
In this embodiment, depth information estimated by the depth estimation unit 251 is provided to the depth histogram 223 .
Other configurations and operations are common to the information processing apparatus 20 (see FIG. 1) in the first embodiment.

<Embodiment 3>
In this embodiment, a case of acquiring depth information using a so-called single camera will be described.
17A and 17B are diagrams for explaining an example of the mode of use according to the third embodiment. FIG.
In FIG. 17, parts corresponding to those in FIG.
The information processing device 20B used in the present embodiment is a device capable of allowing a user to perceive information (so-called augmented reality or mixed reality) obtained by synthesizing a real space and a virtual space.
The information processing device 20B shown in FIG. 17 is a case of so-called smart glasses. Smart glasses are a type of wearable device that is worn on the head in the same manner as wearing glasses.
The smart glasses shown in FIG. 17 represent a transmissive display that allows direct visual recognition of objects that actually exist in real space. The smart glasses shown in FIG. 17 are of a monocular type, but may be of a binocular type. A non-transmissive display that blocks the user's view from the outside may also be used.

FIG. 18 is a diagram illustrating an example of the hardware configuration of the information processing device 20B according to the third embodiment. In FIG. 18, parts corresponding to those in FIG. 2 are shown with reference numerals corresponding thereto.
The information processing apparatus 20B shown in FIG. 18 differs from the information processing apparatus 20 (see FIG. 1) in that it uses a retinal scanning display 213B and does not use a liquid crystal display 213 (see FIG. 2), and a camera 215B with a filter. 2 and does not use the distance sensor 216 (see FIG. 2).
The retinal scanning display 213B here is a display that projects an image by directly illuminating the retina of a person.

The filter-equipped camera 215B is an imaging means of a method of simultaneously outputting a color image and depth information from one image captured by a monocular visible light camera. This type of technology includes, for example, "color aperture imaging technology" developed by Toshiba.
FIG. 19 is a diagram showing a structural example of the filter 270 attached to the opening of the lens of the filter-equipped camera 215B. Half of the filter 270 is a light blue color filter 271 and the other half is a yellow color filter 272 . When an image is captured with the filter 270 having this structure attached, the relationship between blurring and color shift is reversed depending on whether the focal point is on the front side or the back side. Specifically, the left-right asymmetrical color shift (bokeh shape that differs for each color) generated by combining the three colors of red, blue, and green light depends on whether it is on the front side or the back side of the focal point. Flip. Naturally, no blurring or color shift occurs at the focal point.

FIG. 20 is a diagram illustrating an example of the functional configuration of an information processing device 20B according to the third embodiment. In FIG. 20, parts corresponding to those in FIG. 3 are shown with reference numerals.
In the case of the information processing apparatus 20B according to the present embodiment, the image data output from the camera with filter 215B is provided to the image information acquiring section 221 and the depth information acquiring section 222B.
The depth information acquisition unit 222B acquires depth information by analyzing the shape of an asymmetric blur for each color, corrects the aforementioned blurring and color shift by image processing, and acquires depth information for each pixel.
Other configurations and operations are common to the information processing apparatus 20 (see FIG. 1) in the first embodiment.

<Embodiment 4>
In the present embodiment, a case will be described in which an image capturing means, an image data storage means, and an information processing apparatus that processes an image are connected via a network.
21A and 21B are diagrams for explaining an example of a mode of use according to the fourth embodiment. FIG.
The information processing system 300 shown in FIG. 21 includes a network 301, a camera 302 that captures an image of a real space, file servers 303 and 304 that store captured image data, and image processing without manual operation. It has information processing devices 305 and 306 that specify target areas.
The network 301 may be a LAN (Local Area Network), the Internet, or a cloud network. Also, the network 301 may be a wireless connection or a wired connection.

The camera 302 may be an imaging device mounted on the smartphone or smart glasses described above, or may be an independent imaging device. Independent imaging devices include, for example, commercial cameras and home cameras. Commercial cameras also include security cameras.
In this embodiment, the file server 303 is for storing image data with depth information, and the file server 304 is for storing image data without depth information.
In this embodiment, the information processing device 305 is a stationary computer, and the information processing device 306 is a portable computer.

FIG. 22 is a diagram for explaining the basic configuration of the information processing apparatuses 305 and 306. As shown in FIG. Here, the information processing device 305 will be described.
The information processing device 305 has a configuration in which an input device 316 , an auxiliary storage device 317 and an output device 318 are connected to a device body 311 .
The apparatus main body 311 is composed of a central processing unit 312 and a main storage unit 315. The central processing unit 312 has a control unit 313 that controls the operation of each unit and an arithmetic unit 314 that processes data.
The main storage device 315 is composed of ROM (Read Only Memory) and RAM (Random Access Memory), and stores data and programs. Programs include basic software and application programs.
The input device 316 is a mouse, keyboard, touch panel, or the like.
The auxiliary storage device 317 is a hard disk, an optical storage medium, a semiconductor storage medium, or the like.
The output device 318 is a display, a printer, or the like.

FIG. 23 is a diagram illustrating an example of functional configurations of the information processing apparatuses 305 and 306 according to the fourth embodiment. In FIG. 23, parts corresponding to those in FIG. 3 are shown with reference numerals.
The functional configuration shown in FIG. 23 corresponds to a case where image data and corresponding depth information are downloaded from the file server 303 in which image data having depth information is accumulated.
In the case of FIG. 23, the image information acquisition unit 221 acquires image data from the data downloaded from the file server 303, and the depth information acquisition unit 222C acquires depth information from the data downloaded from the file server 303. to get
If depth information is also output from the camera 302, the output of the camera 302 may be given to the image information acquisition unit 221 and the depth information acquisition unit 222B.

FIG. 24 is a diagram illustrating another example of the functional configuration of the information processing apparatuses 305 and 306 according to the fourth embodiment. In FIG. 24, parts corresponding to those in FIG. 3 are shown with reference numerals.
The functional configuration shown in FIG. 24 corresponds to a case where image data and corresponding depth information are downloaded from the file server 304 storing image data without depth information.
In the case of FIG. 24, both the image information acquisition unit 221 and the depth information acquisition unit 222D download the same image data from the file server 304. FIG.
The depth information acquisition unit 222D here executes data analysis using artificial intelligence (AI), for example, and acquires the depth to each area in the image captured by the visible light camera.
If the image data is image data captured by the filter-equipped camera 215B (see FIG. 18), the depth information acquisition section 222B (see FIG. 20) may be used as the depth information acquisition section 222D.

<Other embodiments>
Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope described in the above-described embodiments. It is clear from the scope of claims that various modifications and improvements to the above embodiment are also included in the technical scope of the present invention.
For example, in the above-described embodiments, histograms are generated based on the frequency distribution aggregated by depth, but the generation of histograms is not essential. A distribution of frequencies tabulated by depth may be processed to extract depth ranges with a large number of occurrences (appearing intensively) as clusters. A functional unit that executes this process is an example of an extractor.
In the above-described embodiment, the seed coordinate setting unit 225 extracts the pixels to be used as seeds from among the pixels belonging to the cluster. A process of extracting some pixels based on a predetermined rule may be executed. In this case, the depth clustering execution unit 224 is an example of the extraction means.

20, 20A, 20B... Information processing apparatus 221... Image information acquisition unit 222, 222A, 222B, 222C, 222D... Depth information acquisition unit 223... Depth histogram generation unit 224... Depth clustering execution unit 225... Seed coordinates Setting unit 226 Area dividing unit 227 Target area specifying unit 228 Image processing unit 250 Smoothing/sharpening unit 251 Depth estimation unit

Claims (18)

an extracting means for extracting a range in which the depth distribution corresponding to each pixel constituting the captured image appears intensively;
and setting means for setting at least part of the extracted pixels belonging to the range as a starting point for specifying a region to be processed. 2. The information processing apparatus according to claim 1, wherein said extracting means extracts a depth distribution that satisfies a predetermined condition as said range. 3. The information processing apparatus according to claim 2, wherein said condition is that the shape of the distribution resembles the shape of a predetermined function. 4. The information processing apparatus according to claim 3, wherein said function is a unimodal function. 3. The information processing apparatus according to claim 2, wherein said condition is that a frequency exceeding a predetermined threshold appears. 2. The information processing apparatus according to claim 1, wherein said setting means sets, as said starting point, a pixel at a depth where appearance of pixels is concentrated in said extracted range. 7. The information processing apparatus according to claim 6, wherein said at least some pixels correspond to a depth at which the number of pixels appearing in said range is maximal. 7. The information processing apparatus according to claim 6, wherein said at least some pixels correspond to a depth included in a range of a constant multiple of a standard deviation from a median value of said range. 2. The information processing apparatus according to claim 1, wherein said image has pixels associated with a depth distribution . 10. The information processing apparatus according to claim 9, wherein the depth distribution corresponding to each pixel is given as parallax of a plurality of imaging means. 10. The information processing apparatus according to claim 9, wherein the depth distribution corresponding to each pixel is given as a measurement value when capturing the image. 2. The information processing apparatus according to claim 1, wherein the image is divided into a plurality of regions using the pixel values set as the starting points. 2. The information processing apparatus according to claim 1, wherein said area to be processed is selectively identified based on a predetermined rule. 14. The information processing apparatus according to claim 13, wherein a part of the area identified through said starting point is selected as a processing target according to a predetermined rule. 15. The information processing apparatus according to claim 14, wherein said predetermined rule is to give priority to an area near the center of said image. 15. The information processing apparatus according to claim 14, wherein said predetermined rule is to give priority to an area where a user's line of sight is concentrated. 15. The information processing apparatus according to claim 14, wherein the rule-based selection is performed when a difference in depth of the plurality of regions specified through the starting point is within a predetermined range. the computer,
an extracting means for extracting a range in which the depth distribution corresponding to each pixel constituting the captured image appears intensively;
A program that functions as setting means for setting at least some of the extracted pixels belonging to the range as a starting point for specifying a region to be processed.

Images