JP2023001950A - Information processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Abstract

To provide an information processing apparatus that can accurately detect an edge area of the shape of a subject.SOLUTION: A control unit 101 of a shape information detection device 110 acquires a plurality of images picked up in different illumination conditions. The control unit 101 acquires distance information on a subject in terms of a depth direction in the images. The control unit 101 determines an edge area of the subject based on information on a change in the luminance value between the plurality of images and information on a change of the distance information in areas of interest of the images.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an imaging device, an image processing method, and a program.

A so-called stereo camera has been proposed that includes two imaging units and can capture images for stereoscopic viewing. A stereo camera captures images of the same subject at the same time using respective imaging units, and obtains two types of images, a first image and a second image having parallax. By calculating the amount of parallax for each pixel from an image having parallax, it is possible to calculate the distance from the camera to the subject.

The distance from the stereo camera to the object is shape information expressed in the camera coordinate system, but it can be converted into shape information expressed in the world coordinate system by image processing. In the stereo matching method, which is generally used as a method for calculating the amount of parallax, the calculated amount of parallax becomes ambiguous due to the influence of the size of the matching area. In particular, an edge region having a shape with a large distance change is likely to be affected by the size of the matching region, so there is a high possibility that an erroneous amount of parallax will be calculated. Therefore, in order to obtain more accurate shape information, it is necessary to detect the edge region with high precision and correct the shape information of the edge region. As an example of the prior art, US Pat. No. 6,200,000 discloses a method for detecting edge regions from a set of images collected under different lighting conditions.

JP 2004-289829 A

A conceivable method is to detect shadows generated by illumination of images shot under different lighting conditions, and to detect contours of the shadows as edge regions. However, since the outline of the shadow includes areas that are not edge areas, the edge areas cannot be detected with high accuracy by the above method. SUMMARY OF THE INVENTION An object of the present invention is to provide an information processing apparatus capable of detecting an edge region of a shape of a subject with high accuracy.

An information processing apparatus according to an embodiment of the present invention includes first acquisition means for acquiring a plurality of images captured under different illumination conditions, and second acquisition means for acquiring distance information of a subject in the depth direction in the images. and control means for determining an edge region of the subject based on change information of luminance values between the plurality of images and change information of the distance information in the region of interest of the images.

According to the information processing apparatus of the present invention, it is possible to detect the edge area of the shape of the object with high accuracy.

It is a figure explaining the structure of a shape information detection apparatus. It is a figure explaining the structural example of an imaging device. It is a flow chart explaining an example of operation processing of a shape information detecting device. It is a figure which shows an example of imaging. FIG. 4 is a diagram illustrating an example of a light emitting position of a light emitting unit of an imaging device and a captured image; 7 is a flowchart illustrating an example of distance information acquisition processing; It is a figure explaining an example of a structure of an imaging part. FIG. 10 is a diagram showing an example of setting minute blocks for an A image and a B image; It is a figure which shows the calculation result of the amount of correlations. 8 is a flowchart illustrating an example of edge region detection processing; It is a figure which shows an example of a 1st area|region. It is a figure explaining an example of the detection process of a 2nd area|region. It is a figure which shows an example of an edge area|region. It is a figure explaining an example of correction processing of distance information. It is an example of detection processing of the first region by the information processing apparatus of the second embodiment.

(Example 1)
The information processing apparatus of this embodiment will be described below with reference to the drawings.
The information processing device detects (determines) an edge region of the shape of the subject and uses the detected edge region to correct the distance information, thereby detecting shape information with higher accuracy. The edge area of the shape indicates an area in which the value of the distance information (distance value) to the subject has a large amount of change in the photographed image acquired at the photographing viewpoint. In the following description, unless otherwise specified, "edge region" indicates the edge region of the shape.

FIG. 1 is a diagram for explaining the configuration of a shape information detection device that is an example of an information processing device according to this embodiment.
The shape information detection device 100 has an imaging device 110 and an image processing device 120 . Information exchange between the imaging device 110 and the image processing device 120 is controlled by the control unit 101 . The control unit 101 is a processor such as a CPU (Central Processing Unit). In FIG. 1, the control unit 101 is one control unit common to the imaging device 110 and the image processing device 120. The imaging device 110 and the image processing device 120 each have a control unit, and two control units are provided. The units may communicate information with each other.

The imaging device 110 has a control section 101 , a light emitting device 111 and an imaging section 112 . The light-emitting device 111 is controlled by the control unit 101 and emits light to the subject during photographing based on the light-emitting position, the light-emitting time, and the light-emitting timing. The image capturing unit 112 captures an image of an object for image processing, and supplies the captured image to the image information input unit 122 of the image processing apparatus 120 under the control of the control unit 101 .

The image processing apparatus 120 includes a control unit 101, a memory 121, an image information input unit 122, a parameter setting unit 123, a distance information acquisition unit 124, an edge area detection unit 125, a distance information correction unit 126, a shape and an information calculation unit 127 . The control unit 101 controls each module included in the image processing apparatus 120 .

The memory 121 reads out information necessary for processing in each module and stores processing results in each module. In addition, the memory 121 stores in advance a control program executed by the control unit 101 . The image information input unit 122 acquires image information captured by the imaging unit 112 or an image captured by an external device, and supplies it to the memory 121 . The parameter setting unit 123 supplies each parameter information input to the image processing apparatus 120 by the user to the memory 121 . The distance information acquisition unit 124 acquires distance information of the object in the depth direction in the image based on the image obtained by imaging, and supplies the distance information to the memory 121 . The edge area detection unit 125 detects (determines) an edge area of the shape of the subject based on the captured image and the distance information of the subject, and supplies the edge area to the memory 121 . The distance information correction unit 126 corrects the distance information based on the detected edge area, and supplies the corrected distance information to the memory 121 . The shape information calculator 127 calculates shape information based on the corrected distance information, and supplies the shape information to the memory 121 .

A shape information detection device 100 shown in FIG. 1 is an example of an information processing device of the present invention. The image processing apparatus 120 alone is also within the scope of the present invention. Further, although not shown, an imaging device including the light emitting device 111 and the imaging unit 112 and having the same functions as the image processing device 120 is also within the scope of the present invention.

FIG. 2 is a diagram illustrating a configuration example of an imaging device.
FIG. 2A shows an example of a perspective view of the imaging device 110. FIG. FIG. 2B shows an example of the light-emitting device 111 included in the imaging device 1109 . The light-emitting device 111 has light-emitting portions at a plurality of different positions. The light emitting unit 113 is one of light emitting units included in the light emitting device 111 . A plurality of light-emitting portions included in the light-emitting device 111 can emit light individually. Further, the control unit 101 can cause a predetermined number of light emitting units to emit light, and can control all light emitting units to emit light, all light emitting units not to emit light, and the like. At least two or more light-emitting units may be provided, and eight or more as shown in FIGS. 2A and 2B may be installed.

FIG. 2C shows a configuration example of an imaging optical system that the imaging device 110 has. The image pickup unit 112 forms an image of light emitted from an object 114 , which is a subject, by a lens 115 on a planned imaging plane 116 , and receives the light on a sensor plane 117 . The control unit 101 acquires the received information as image information.

FIG. 3 is a flowchart illustrating an example of operation processing of the shape information detection device.
The processing of this flowchart is implemented by the control unit 101 reading and executing the control program in the memory 121 . S in FIG. 2 is a step number indicating each process according to this flowchart.

In S201, the control unit 101 acquires and sets parameters necessary for processing.
The control unit 101 acquires at least the following information as parameters.
(1) The distance from the imaging device 110 to the focus position of the subject (2) A conversion coefficient determined by the size of the opening angle of the center of gravity of the light flux passing through the pair of pupil regions (3) The image-side principal point of the lens of the imaging device 110 to the sensor surface (4) focal length of the imaging device 110 (5) brightness value threshold (6) distance value threshold The brightness value threshold is used to detect a first region, which will be described later. Also, the threshold value of the distance value is used for detecting a second area, which will be described later.

Next, in S202, the control unit 101 acquires an image (captured image) captured by the imaging device 110 or a captured image captured by an arbitrary external device. In this embodiment, the control unit 101 acquires a plurality of images captured under different lighting conditions. A specific example of the processing of S202 will be described below with reference to FIGS. 4 and 5. FIG.

FIG. 4 is a diagram illustrating an example of imaging of a subject by an imaging device.
FIG. 4A shows the positional relationship between the imaging device 110 and the subjects 301 and 302. FIG. In the example shown in FIG. 4A, the imaging device 110 images the subjects 301 and 302 from a position obliquely above them.

FIG. 4B is an image obtained when the imaging device 110 captures the subjects 301 and 302 without emitting light from the light emitting device 111 in the positional relationship of FIG. 4A. In the image shown in FIG. 4B , a boundary portion 303 between the subject 301 and the subject 302 is the edge region of the subject 301 .

FIG. 5 is a diagram illustrating an example of a light emitting position of a light emitting unit of an imaging device and a captured image.
FIG. 5A shows an image obtained when the imaging device 110 captures an image by causing the light emitting unit 401 of the light emitting device 111 to emit light in the positional relationship shown in FIG. 4A. By causing only the light emitting unit 401 of the light emitting device 111 to emit light, it is possible to obtain an image in which the subject 301 causes the subject 302 to generate a shadow portion 402 .

FIG. 5B shows an image obtained when the imaging device 110 captures an image by causing the light emitting unit 403 of the light emitting device 111 to emit light in the positional relationship shown in FIG. 4A. In this case, an image in which the subject 301 causes the shadow 404 to appear on the subject 302 can be obtained.
FIG. 5C shows an image obtained when the imaging device 110 captures an image by causing the light emitting unit 405 of the light emitting device 111 to emit light in the positional relationship shown in FIG. 4A. In this case, an image in which the subject 301 causes the shadow 406 to appear on the subject 302 can be acquired.
FIG. 5(D) shows shadows corresponding to imaging under multiple lighting conditions, extracted based on multiple images with different shadows shown in FIGS. 4(A) to 4(C). . A shadow 407 in FIG. 5D is the extracted shadow. The shaded area 407 is used when detecting the edge area in S204. As described above, by illuminating a subject from a plurality of different directions during imaging, it is possible to obtain a plurality of images captured under different lighting conditions.

Returning to the description of FIG. In S203, the control unit 101 controls the distance information acquisition unit 124 to acquire distance information (depth information) of the subject in the depth direction in the image based on the image obtained by imaging.

The distance information in the application of the present invention may be information corresponding to the distance distribution of the object in the depth direction in the imaging range. For example, it is possible to use defocus amount distribution information before normalization by the focal depth, or a depth map indicating the subject distance corresponding to each pixel. Further, the distance information may be two-dimensional information indicating a phase difference used for deriving the defocus amount. This phase difference corresponds to the amount of relative image shift between different viewpoints. Also, it is possible to use a distance map converted into actual distance information on the object side via the position of the focus lens of the imaging optical system.

As described below, in this embodiment, for example, distance information is calculated by the DFD method that derives the defocus amount from the correlation of a pair of image data from different viewpoints. DFD is an abbreviation for "Depth From Defocus". A distance distribution obtained from a distance measuring sensor module such as a TOF method can also be derived as distance information. TOF is an abbreviation for "Time Of Flight". Alternatively, it is also possible to acquire distance information by a contrast ranging method based on contrast information of a captured image or an evaluation value.

FIG. 6 is a flowchart illustrating an example of distance information acquisition processing.
In S501, the control unit 101 generates a pair of image data from different viewpoints based on the image information acquired in S202 of FIG. It is preferable that the control unit 101 generates a pair of image data based on image information related to an image with few shadows, such as full light emission or no light emission under illumination conditions. This is because the accuracy of defocus amount calculation decreases because the contrast of the shadow portion is low. Calculation accuracy can be improved by arranging a pattern such as a lattice on one projection system of the light emitting unit, capturing an image in which the pattern is projected onto the subject, and using the image to calculate the defocus amount.

FIG. 7 is a diagram illustrating an example of the configuration of an imaging unit included in the imaging apparatus of this embodiment.
The imaging unit 112 (FIG. 1) includes, for example, a plurality of microlenses and a plurality of photoelectric conversion units corresponding to the microlenses, and outputs a plurality of image signals from different viewpoints.
As shown in FIG. 7A, the imaging unit 112 has a plurality of pixels 612 that are regularly arranged two-dimensionally. FIG. 7B shows the structure of the pixel 612. FIG. The pixel 612 has a microlens 611 and a pair of photoelectric conversion units 613A and 614B (hereinafter referred to as pupil division pixels 613A and 614B). The photoelectric conversion units 613A and 614B photoelectrically convert subject light that has passed through different pupil division regions of the imaging optical system of the imaging unit 112, and output image signals. In the present embodiment, the image based on the image signal output from the pupil-divided pixel 613A is referred to as the A image, and the image based on the image signal output from the pupil-divided pixel 614B is referred to as the B image. An image obtained by synthesizing the A image and the B image is called an AB image. Based on the configuration of the imaging unit shown in FIG. 7, it is possible to obtain a pair of images, A image and B image, from different viewpoints as information necessary for obtaining distance information.

Returning to the description of FIG. In S502, the control unit 101 sets minute blocks for the A and B images generated as a pair of image data in S501. A minute block is generally synonymous with a window, which is an area set when template matching is performed.

FIG. 8 is a diagram showing a setting example of minute blocks for the A image and the B image.
The control unit 101 sets minute blocks 703 for the A image 701 and the B image 702 shown in FIGS. 7A and 7B. For example, when each pixel of the A image 701 and the B image 702 is a pixel of interest 704, the minute block 703 is set in a region of a predetermined size centered on the pixel of interest 704. FIG. In the example shown in FIG. 8A, the minute block 703 is set in a square area of 3×3 pixels centered on the target pixel 704, but the shape and size of the minute block may be any. may It is also possible to set a minute block whose size and shape are changed, like the minute block 705 in FIG. 7C.

Returning to the description of FIG. In S503, the control unit 101 performs correlation calculation processing for each pixel (target pixel 704 in FIG. 8) between a pair of pupil-divided images (image A and image B). An image deviation amount (image deviation amount) is calculated.
When the number of data (the number of pixels) of the (pair of) minute blocks 703 determined for the pixel of interest 704 at the same position in the A image and the B image is m, the pixel data of the pair of minute blocks 703 are respectively represented by E ( 1) to E(m) and F(1) to F(m).
Control section 101 shifts data sequence F relative to data sequence E, and calculates correlation amount C(k) at data shift amount k based on equation (1).
C(k)=Σ|E(n)−F(n+k)| Expression (1)
C(k) is computed for number n of the data series. n and n+k are limited to ranges from 1 to m. The shift amount k is an integer and is a relative shift amount in units of data intervals of image data. The amount of shift k is synonymous with the amount of parallax in the stereo matching method.

FIG. 9 is a diagram showing calculation results of correlation amounts.
The horizontal axis of the graph shown in FIG. 9 indicates the amount of image shift. The vertical axis indicates the amount of correlation C(k). The amount of correlation C(k) is minimized at the amount of image shift where the correlation between a pair of data sequences is high. The control unit 101 uses the three-point interpolation method according to equations (2) to (5) to find the image shift amount x that gives the minimum value C(x) for the continuous correlation amount.
x=kj+D/SLOP Expression (2)
C(x)=C(kj)−|D| Expression (3)
D={C(kj−1)−C(kj+1)}/2 Equation (4)
SLOP=MAX{C(kj+1)-C(kj), C(kj-1)-C(kj)}
... formula (5)
kj is k that minimizes the discrete correlation amount C(k).
The shift amount x obtained by Equation (2) is included in the distance information as the image shift amount in one pixel of interest 704 . Note that the unit of the image shift amount x is pixel.

Returning to the description of FIG. In S504, the control unit 101 obtains the defocus amount DEF of the object image plane with respect to the planned imaging plane based on the image shift amount x obtained in S503 and using Equation (6).
DEF=KX·x Expression (6)
KX is a conversion coefficient determined by the open angle of the center of gravity of the light flux passing through the pair of pupil regions. The control unit 101 can calculate the defocus amount of each pixel position by repeatedly executing the above calculation while shifting the target pixel position by one pixel.

Next, in S505, the control unit 101 calculates the distance z from the sensor surface of the imaging unit 112 of the imaging device 110 to the subject based on the defocus amount calculated in S504. The distance z can be calculated using equations (7) and (8).
dist=1/(1/(dist_d+DEF) 1/f) Expression (7)
z=length-dist Expression (8)
dist indicates the distance from the focus position to the subject. dist_d indicates the distance from the image-side principal point of the lens of the imaging unit 112 to the sensor surface. f indicates the focal length. length indicates the distance from the sensor surface to the focus position. In this embodiment, the control unit 101 calculates the distance z from the image shift amount x based on equations (6) to (8), but the distance z may be calculated using other equations. Note that the distance z is not limited to the distance from the sensor surface, and may be the distance from any position such as the tip of the lens.

The above-described distance length from the sensor surface to the focus position can be measured by, for example, laser distance measuring means. Further, the shape information detection apparatus 100 can estimate the distance to the focus position corresponding to the lens position at the time of imaging by having the relationship between the lens position and the focus position at the time of imaging in the data table. This reduces the processing load when calculating the distance from the sensor surface of the imaging device to the focus position.

As described above, the distance z from the sensor surface of the imaging unit to the subject can be calculated by the method of generating a pair of image data from different viewpoints from one image. Therefore, since the imaging device 110 may be a monocular camera, the configuration of the imaging device 110 can be simplified. Furthermore, calibration processing required when installing a plurality of cameras can be simplified or eliminated. Of course, it is also possible to calculate the distance z with a stereo camera. Further, it is also possible to calculate the distance z by an external device of the imaging device 110 and allow the control unit 101 of the image processing device 120 to acquire the distance z calculated by the external device.

Returning to the description of FIG. In S204, the control unit 101 detects an edge area.
FIG. 10 is a flowchart illustrating an example of the edge region detection processing in S204 of FIG.
In S901, the control unit 101 detects a first region based on a plurality of images captured under different lighting conditions, which are acquired in S202 of FIG. The control unit 101 detects the first area based on, for example, a plurality of images having different shades shown in FIGS. 5(A) to 5(C). Specifically, the control unit 101 calculates a maximum value image and a minimum value image of luminance values for each pixel among a plurality of images captured under different lighting conditions. The control unit 101 calculates the difference in luminance value between the maximum value image and the minimum value image as luminance value change information (amount of change). Then, the control unit 101 detects, as a first area, a shadow area, which is an area where the amount of change in luminance value is equal to or greater than the threshold.

FIG. 11 is a diagram showing the first area detected in Example 1. FIG.
A region 1001 indicates the first region. As the luminance value threshold used for detecting the first area, a luminance value threshold arbitrarily set by the user using the parameter setting unit 123 can be applied.

The control unit 101 may change the threshold value of the brightness value according to the distance value to the subject. As the distance value to the object increases, the brightness (illuminance) of the object emitted from the light emitting unit becomes weaker and the shadow becomes lighter. The brightness of an object is inversely proportional to the square of the distance from the light emitting unit to the object. Therefore, if the distance value from the light emitting unit to the subject is LD, and the threshold value LT_1 of the brightness value before change is assumed, the threshold value LT of the brightness value after change can be calculated by Equation (9).
LT=LT_1/(α1×LD×LD) Expression (9)
α1 is an arbitrary constant for controlling the influence of the distance value LD from the light emitting unit to the object, and is set within the range of 1/(LD×LD) to ∞.
The control unit 101 sets the threshold value of the luminance value to a smaller value as the distance value from the light emitting unit to the subject increases, thereby making it possible to detect a shadow region even in a distant subject.

The control unit 101 may compare the luminance value of the maximum luminance value image and the luminance values of other captured images, and calculate the luminance value ratio as the luminance value change information. For example, the control unit 101 calculates the ratio of the luminance value of the maximum luminance value image to the luminance value of the minimum luminance value image. Then, the control unit 101 may detect an area in which the calculated luminance value ratio is equal to or greater than a threshold as a shadow area. This makes it possible to detect shadows while reducing the influence of the color of the subject. Alternatively, the luminance values of the image captured without emitting light may be subtracted from the luminance values of the maximum luminance value image and the luminance values of the other captured images before the luminance values are compared. As a result, even in a bright environment, shadows can be detected by reducing the influence of ambient light.

Returning to the description of FIG. In S902, the control unit 101 detects the second area based on the distance information acquired in S203 of FIG.
FIG. 12 is a diagram illustrating an example of the second area detection process.
FIG. 12A shows an example of a captured image to be processed for calculating distance information. The control unit 101 sets a region around a pixel (set pixel) 1101 set as a pixel of interest as a region of interest 1102 . Also, the control unit 101 sets a region around the set pixels including the set pixel 1103 as a region of interest 1104 . When the amount of change in the value of the distance information (distance value) of the attention area is greater than or equal to the threshold, the control unit 101 detects the set pixels included in the attention area as pixels included in the second area.

FIG. 12B shows a graph obtained by one-dimensionally plotting the correct distance and the calculated distance in the pixel Yc direction for the region of interest 1102 . The horizontal axis indicates the pixel Yc. The vertical axis indicates distance (mm). The distance difference shown in FIG. 12B is change information (amount of change) of the distance value of the attention area 1102 . The distance difference is the difference between the maximum value and the minimum value of the data string when the region of interest 1102 is a data string of one-dimensional distance values.
FIG. 12C shows a graph obtained by one-dimensionally plotting the correct distance and the calculated distance in the pixel Yc direction for the region of interest 1103 . The horizontal axis indicates the pixel Yc. The vertical axis indicates distance (mm). The distance difference shown in FIG. 12C is the amount of change in the distance value of the attention area 1104 .

The control unit 101 determines whether the distance difference calculated for the region of interest is equal to or greater than a distance value threshold. Detect as pixels included in the area. The threshold value of the distance value is set in advance based on an operation input by the user using the parameter setting unit 123, for example. As the threshold value of the distance value, a threshold value determined in advance by an experiment or the like may be used. For example, it is determined that the distance difference of the region of interest 1104 shown in FIG. 12C is smaller than the distance value threshold. Therefore, the setting pixel 1103 is not detected as the second area. On the other hand, it is determined that the distance difference of the region of interest 1102 shown in FIG. 12B is greater than the distance value threshold. Therefore, the setting pixel 1101 is detected as the second area.

The user can arbitrarily determine the range of the attention area described above. For example, the distance information calculated in S203 of FIG. 3 is an ambiguous value due to the size of the minute block (window) used for calculating the image shift amount. Therefore, by setting the range of the region of interest to be wider than the range of the minute blocks, the second region can be accurately detected based on the distance information other than the ambiguous value. .

As described above, the second area can be detected by evaluating the amount of change in the distance value of the attention area set for each set pixel. Note that it is also possible to evaluate the amount of change in the distance value of the attention area for each area and determine whether it is the second area for each area. This eliminates the need for calculation for each pixel, so it is possible to reduce the amount of calculation. A ratio of the maximum distance value to the minimum distance value of the region of interest may be used as the amount of change in the distance value. may be detected.

Further, the control unit 101 detects the second area based on the amount of change in the distance to the subject in the attention area corresponding to the first area detected in S901 of FIG. The processing load for region detection can be reduced. FIG. 12(D) shows the detected second region 1105 .

(Example 1 of distance value threshold control)
The control unit 101 may change the distance value threshold arbitrarily set by the user based on the RGB color information of the captured image. In general, the edge area of the shape of the subject is often the boundary between the shapes of different subjects, and thus is likely to be an area where the color of the subject is different. Therefore, for example, the control unit 101 evaluates the color difference between the color of the set pixel and the color of the surrounding area including the set pixel, and decreases the distance value threshold as the color difference increases. This makes it possible to detect the second area even in a shooting scene with a small distance difference.
Specifically, the control unit 101 compares the color of the set pixel with the color of the target area including the surrounding area including the set pixel, and determines the maximum color difference. The control unit 101 decreases the threshold value of the distance value as the determined color difference increases. The control unit 101 may control the threshold value of the distance value using the difference between the average color value of the surrounding area and the color of the set pixel as the color difference.

In general, known examples of color difference expression methods include a method of expression using the Euclidean distance and a method of expression in the Lab color space. In this embodiment, the color difference calculated using the above known technique is used as the color difference CD. . Assuming that the color difference is CD and the threshold value DT_1 of the distance value before change is used, the threshold value DT of the distance value after change can be calculated by Equation (10).
DT=DT_1/(α2×CD) Expression (10)
α2 is an arbitrary constant for controlling the influence of the color difference CD and is set within the range of 1/CD to ∞.
It is also possible to calculate the threshold value DT of the distance value after change by the following equation (11).
DT=DT_1-(α3×CD) Expression (11)
α3 is an arbitrary constant for controlling the influence of color difference CD, and is set in the range of 0 to DT_1/CD.

(Example 2 of distance value threshold control)
Regarding the threshold value of the distance value arbitrarily set by the user, the control unit 101 determines, based on the amount of change in the inclination of the surface for each local area in the region of interest, that the larger the amount of change in the inclination of the surface, the smaller the threshold value of the distance value. A value may be set. The inclination of the surface for each local area is information such as the amount of displacement in the shape of the local area and the surface normal. As a method of obtaining the tilt of the surface, there is a method of differentiating the distance information in the in-plane direction to calculate the amount of displacement, and a method of obtaining the surface normal by calculating the surface normal using the photometric stereo method. In this embodiment, as an example, the surface normal is used as the inclination of the surface for each local region.

In general, the edge region is often the boundary between the shapes of different subjects, so it is highly possible that the surface normals of the shapes of the subjects are different. Therefore, for example, the control unit 101 decreases the threshold value of the distance value as the difference (for example, angle) between the surface normal of the set pixel and the surface normal of the area around the set pixel increases. This makes it possible to detect the second area even in a shooting scene with a small distance difference.
Specifically, the control unit 101 compares the surface normal of the set pixel and the surface normal of the area around the set pixel, and determines the maximum difference between the surface normal of the set pixel and the area around the set pixel. Used as a difference from the surface normal. Alternatively, the difference between the average value of the surface normals of the surrounding area and the surface normal of the set pixel may be used as the difference between the surface normal of the set pixel and the surface normal of the surrounding area.
For example, if the difference between the surface normal of the set pixel and the surface normal of the surrounding area is ND, and the distance value threshold DT_1 before change is assumed, the distance value threshold DT after change can be calculated by equation (12). .
DT=DT_1/(α4×ND) Expression (12)
α4 is an arbitrary constant for controlling the influence of the difference ND between the surface normal of the set pixel and the surface normal of the surrounding area, and is set within the range of 1/ND to ∞.

It is also possible for the control unit 101 to calculate the threshold value DT of the distance value after change using the equation (13).
DT=DT_1-(α5×ND) Expression (13)
α5 is an arbitrary constant for controlling the influence of the difference ND between the surface normal of the set pixel and the surface normal of the surrounding area, and is set in the range of 0 to DT_1/ND.

As described above, it is possible to change the threshold value of the distance value based on the color difference or the difference between the surface normal of the set pixel and the surface normal of the surrounding area. It is also possible to combine the color difference and the difference between the surface normals and calculate the changed threshold according to, for example, Equation (14) or Equation (15).
DT=DT_1/{(α4×ND)×(α2×CD)} Expression (14)
DT=DT_1-(α3×CD)−(α5×ND) Expression (15)
This makes it possible to change the threshold value of the distance value in consideration of both the color difference of the shape and the surface normal, thereby improving the accuracy of detecting the second area.

Returning to the description of FIG. In S903, the control unit 101 detects an edge area based on the first area detected in S901 and the second area detected in S902. In addition, in this embodiment, a method of detecting the edge region after detecting the first region and the second region is applied, but the scope of application of the present invention is not limited to this method. The control unit 101 may detect an edge region based on change information of luminance values of a plurality of images obtained by imaging under different lighting conditions and change information of distance information in the region of interest of the image. For example, the control unit 101 includes pixels corresponding to an area in which the amount of change in luminance value is equal to or greater than the threshold value of luminance value and in which the amount of change in distance information is equal to or more than the threshold value of distance value is included in the edge region. You may decide as a pixel.

FIG. 13 is a diagram showing an example of detected edge regions.
For example, the control unit 101 determines an area where the first area 1001 shown in FIG. 11 and the second area 1105 shown in FIG. 12D overlap as the edge area 1201 .
According to the information processing apparatus of the present embodiment described above, an edge region can be accurately detected based on the amount of change in luminance values of a plurality of images captured under different lighting conditions and the amount of change in distance information. can be done. The control unit 101 may display the detected edge area on a predetermined screen.

Returning to the description of FIG. In S205, the control unit 101 controls the distance information correction unit 126 to correct the distance information based on the edge area detected in S204. The distance information correction process will be described later with reference to FIG. 14 .
Next, in S206 of FIG. 3, the control unit 101 controls the shape information calculation unit 127 to calculate shape information using the distance information corrected in S205.

The distance information calculated after S205 can be said to be the shape information of the subject in the camera coordinate system at the camera viewpoint. The control unit 101 can calculate the shape information of the subject by applying a method of converting from the camera coordinate system to the point cloud of the world coordinate system. In this way, by using the point cloud of the world coordinate system as the shape information of the subject, it becomes possible to use the shape information more easily in commonly distributed applications than in the case of the shape information in the camera coordinate system.

FIG. 14 is a diagram illustrating an example of distance information correction processing.
FIG. 14(A) shows a region for correction of distance information. The area for which distance information is to be corrected is set, for example, to any range of area including an edge area. In the example shown in FIG. 14A, an area 1401 is an area to be corrected for distance information.

FIG. 14B shows an example of an enlarged view of the region 1401. FIG. A point of interest 1402 indicates the position of a pixel to be corrected. A boundary portion 1403 indicates the edge area detected in S204 of FIG. A region 1404 indicates the second region detected in S902 of FIG. Regions 1405 and 1406 indicate regions that were not detected as the first, second and edge regions. An area 1405 is an area on the front side (camera side) of the area 1406 .

FIG. 14C shows an area 1407 to which the target point 1402 belongs, among the areas obtained by dividing the area 1401 with the boundary 1403 as a boundary. For example, the control unit 101 calculates the correction value of the distance information of the target point 1402 from the average value of the distance values included in the area 1407, and uses the corrected distance value. As a result, the distance value of the area 1406 with a large distance difference can be removed, so the corrected distance value can be calculated with high accuracy. Alternatively, the control unit 101 may calculate the corrected distance value of the point of interest 1402 based on the average value of the distance values of the regions obtained by excluding the region 1404 from the region 1407 . The area 1404, which is the second area, has a large distance difference, and is highly likely to have an ambiguous distance value due to the influence of the size of the matching area (minute block). By having the control unit 101 calculate the corrected distance value based on the average value of the distance values of the area obtained by excluding the area 1404, the distance value can be calculated with higher accuracy.

Next, a method for correcting distance information using surface normals will be described.
FIG. 14D shows a graph obtained by one-dimensionally plotting the distance values (correct distance and calculated distance) of one predetermined line in the Yc direction of the region 1401 shown in FIG. 14A. The horizontal axis indicates the pixel Yc. The vertical axis indicates distance (mm).
Since the surface normal is the inclination of the surface of the local region, the control unit 101 can calculate the distance value by integrating the surface normal in the Yc direction. Specifically, the control unit 101 calculates the distance value by adding the distance value at the position where the integration of the surface normal is started to the integrated value of the surface normal as a reference distance. The position at which surface normal integration is started can be determined arbitrarily.

Here, by using the edge area information, it is possible to divide the area 1401 into areas a and b shown in FIG. In this embodiment, it is assumed that the area a and the area b have the same surface normal, and the surface normal of the boundary 1403 cannot be detected.

FIG. 14E shows the result of calculating the distance value from the surface normal without using the edge region. Since the surface normal does not change at the boundary 1403, integrating the surface normal results in a distance value 1408 indicated by a dashed line, which cannot be used as the corrected distance value.
FIG. 14F shows the result of calculating the distance value from the surface normal using the edge region. The control unit 101 divides the region 1401 into a region a and a region b with a boundary portion 1403 as a boundary, sets a reference distance a in the region a, and sets a reference distance b in the region b. Computes a distance value from the normal. A distance value 1409 indicated by a dashed line is the corrected distance value calculated using the surface normal.

In the above description, as shown in FIG. 14D, the distance value of one predetermined line in the Yc direction of the region 1401 was taken as an example, but it is also possible to integrate the surface normal in two-dimensional directions. be.
The position at which the integration of the surface normal is started may be selected from an area other than the second area. For example, when the second region 1404 is removed from the two regions obtained by dividing the region 1401 at the boundary 1403, regions 1405 and 1406 are obtained. The control unit 101 determines a reference distance for each of the regions 1405 and 1406 . As a result, the ambiguous distance of the area 1404 can be prevented from being set as the reference distance, so that the distance value can be calculated with higher accuracy.

(Example 2)
FIG. 15 is an example of first area detection processing by the information processing apparatus according to the second embodiment.
The configuration and basic operation of the information processing apparatus of the second embodiment are the same as the information processing apparatus, configuration and basic operation of the first embodiment. The information processing apparatus according to the second embodiment detects the outline of the shadow as the first area.

FIG. 15A shows the outline of region 1001 in FIG. In the second embodiment, the control unit 101 detects the contour portion 1301 of the region (shadow portion) 1001 in which the amount of change in luminance value is equal to or greater than the luminance threshold as the first region. For example, the control unit 101 creates a matrix in which the value of the area 1001 is 1 and the value of the area other than the area 1001 is 0. The control unit 101 sets the values of the X-direction and Y-direction differential values DIFF of the created matrix to 1 in regions where |DIFF|>0, and to 0 in other regions. As a result, an area with a value of 1 can be detected as the contour portion 1301 .

As shown in FIG. 15B, the control unit 101 determines an area 1302 where the detected contour portion 1301 and the second area 1105 shown in FIG. 12D overlap as an edge area. According to the information processing apparatus of the second embodiment, it is possible to detect the edge area with high accuracy by using the outline of the shadow as the first area.

In addition, the control unit 101 determines whether the distance value of the area corresponding to the detected contour part 1301 is equal to or greater than the threshold value of the distance value, and detects the second area. It is possible to reduce the processing load when calculating the area.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 shape information detection device 110 imaging device 120 image processing device

Claims (21)

a first acquisition means for acquiring a plurality of images captured under different lighting conditions;
a second acquisition means for acquiring distance information of a subject in the depth direction in the image;
and a control unit that determines an edge region of the subject based on change information of luminance values between the plurality of images and change information of the distance information in the region of interest of the image. processing equipment. a first detection means for detecting a first region based on the amount of change in luminance value between the plurality of images;
a second detection means for detecting a second area based on the amount of change in the distance to the subject within the attention area;
2. The information processing apparatus according to claim 1, wherein said control means determines an edge area of said subject based on information on said first area and information on said second area. 3. The information processing apparatus according to claim 2, wherein said control means determines an area where said first area and said second area overlap as an edge area of said subject. 4. The information processing apparatus according to claim 2, wherein the first detection means detects, as the first area, an area in which the amount of change in luminance value is equal to or greater than a threshold. 5. The information according to any one of claims 2 to 4, wherein the first detection means detects, as the first area, a contour portion of an area in which the amount of change in luminance value is equal to or greater than a threshold. processing equipment. 6. The information processing apparatus according to claim 4, wherein the first detection means sets the threshold to a smaller value as the value of the distance to the subject increases. 6. The second detection means detects the second area based on the amount of change in distance to a subject within the attention area corresponding to the first area. The information processing device according to any one of . 8. The information processing apparatus according to any one of claims 2 to 7, wherein the second detection means detects, as the second area, an area in which the amount of change in distance to the subject is equal to or greater than a threshold. . 9. The information processing apparatus according to claim 8, wherein the second acquisition unit sets the threshold to a smaller value as the color difference in the attention area increases. 10. The information according to claim 8, wherein the second acquisition unit sets the threshold to a smaller value as the amount of change in surface inclination of each local region in the attention region increases. processing equipment. The distance information is distance information based on the amount of image shift obtained from a pair of images with different viewpoints, the distribution of contrast information obtained from a plurality of images with different focus, or the distance information obtained by the TOF method. 11. The information processing apparatus according to any one of claims 1 to 10, wherein: 12. The area of interest according to claim 11, wherein the area of interest is set to a range wider than a range of areas set for calculating a relative amount of image shift obtained from the pair of images having different viewpoints. information processing equipment. The information processing apparatus according to any one of claims 2 to 12, further comprising correction means for correcting the distance information based on the information on the edge area. The correcting means divides the area for which the distance information is to be corrected based on the edge area, and calculates the distance based on the distance information of the area to which the pixel for which the distance information is to be corrected belongs among the divided areas. 14. The information processing apparatus according to claim 13, wherein an information correction value is calculated. The correcting means divides the area to be corrected of the distance information based on the edge area, and removes the second area from the area to which the pixels to be corrected of the distance information belong among the divided areas. 15. The information processing apparatus according to claim 14, wherein the correction value of the distance information is calculated based on the distance information of the area obtained. The correcting means divides the area to be corrected for the distance information based on the edge area, and calculates the correction value of the distance information by integrating the inclination of the surface of the local area in the divided area. The information processing apparatus according to claim 13, characterized by: 17. The information processing apparatus according to any one of claims 1 to 16, further comprising display means for displaying the determined edge region. an information processing apparatus according to any one of claims 1 to 17;
a light emitting means for emitting light to a subject;
and imaging means for imaging the subject. 19. The image pickup apparatus according to claim 18, wherein the image pickup means includes a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each microlens, and outputs a plurality of image signals from different viewpoints. An image processing method executed by an information processing device,
a first acquisition step of acquiring a plurality of images captured under different lighting conditions;
a second acquisition step of acquiring distance information of the subject in the depth direction in the image;
and a control step of determining an edge region of the subject based on change information of luminance values between the plurality of images and change information of the distance information in the region of interest of the image. Processing method. A program that causes a computer to function as each means of the information processing apparatus according to any one of claims 1 to 17.

Images