JP2013077132A - Distance index information estimation device and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a distance information estimation device capable of presenting an estimation value representing an objective reliability of a piece of distance information.SOLUTION: A distance information estimation device 1 includes: an error function calculation section 10 that calculates an error function representing a similarity level between stereo images; a smooth function calculation section 11 that calculates a smooth function representing the continuity of a distance; a message generation-propagation section 12 that generates and propagates a message including an error function and a smooth function; an evaluation function calculation section 13 that calculates an evaluation function of the message; a distance estimation section 14 that estimates a distance which has the minimum value of the evaluation function as a piece of distance information; an estimation value calculation section 15 that calculates an estimation value representing the reliability of the distance information; an occlusion area determination section 17 that determines an area where an estimation error has occurred due to an occlusion; and a non-texture area determination section 19 that determines an area where an estimation error has occurred due to a texture.

Description

  The present invention provides a reference image which is one of stereo images by a method of minimizing the energy expressed by an error function between stereo images and a smooth function indicating the continuity of a distance index which is a distance or parallax in the depth direction. The present invention relates to a distance index information estimation apparatus for estimating distance index information indicating a distance index for each pixel and a program thereof.

  For example, when generating a three-dimensional solid model, a distance estimation technique for estimating a distance from the parallax of each pixel of a stereo image is widely used. In this distance estimation technique, for example, block matching for obtaining a similarity between stereo images is used. As a method of calculating the similarity, for example, an SSD (Sum of Squared Difference) for obtaining the sum of squares of a difference, an SAD (Sum of Absolute Difference) for obtaining an absolute sum of the differences, and a ZNCC ( Zero-mean Normalized Cross-Correlation) is known (for example, Non-Patent Document 1).

  As a distance estimation technique, a Markov random field model is assumed, and energy consisting of data terms and smooth terms between stereo images is minimized to obtain smooth distance information (distance images) with less noise. A reliability propagation method to be generated is also known (for example, Non-Patent Document 2). Here, the data term represents similarity between stereo images, and the smooth term represents discontinuity of distance.

Digital image processing, CG-Arts Association, 2006, p202-p204 Pedro F. Felzenszwalb, Daniel P. Huttenlocher: Efficient Belief Propagation for Early Vision. CVPR (1) 2004: 261-268

  However, in the conventional distance estimation technique, when a stereo image includes a region where occlusion occurs or a region without texture, an error often occurs in the estimation result of distance information due to these. In this case, in order to obtain distance information with high accuracy, it is necessary to perform a process for suppressing these estimation errors on the region where these estimation errors have occurred. In order to determine whether or not the estimation error suppression process is possible, there is a strong demand to present an objective reliability of the estimated distance information.

  Therefore, an object of the present invention is to provide a distance index information estimation apparatus and program for solving the above-described problem and presenting an evaluation value that is an objective reliability of distance index information.

  In view of the above-described problems, the distance index information estimation device according to the first invention of the present application is defined by an error function between stereo images and a smooth function indicating the continuity of the distance index that is a distance or parallax in the depth direction. Distance index information that estimates the distance index information that indicates the distance index for each pixel of the reference image, which is one of the stereo images, and calculates the evaluation value that indicates the reliability of the distance index information, by minimizing the energy related to the probability An estimation apparatus, comprising an error function calculation unit, a smooth function calculation unit, an energy propagation unit, an evaluation function calculation unit, a distance index estimation unit, and an evaluation value calculation unit.

  According to such a configuration, the distance index information estimation device receives a stereo image by the error function calculation unit, and is set to be equal to or less than a preset error function threshold for each preset distance index candidate. An error function that is the similarity of the input stereo image is calculated. That is, the error function calculation unit performs truncation so that the value of the error function does not exceed the error function threshold value.

  Further, the distance index information estimation device uses the absolute value of the difference between the distance index candidates between adjacent pixels in the Markov random field so that the smooth function calculation unit has a smooth function threshold value or less set for each distance index candidate. Calculate a smooth function. That is, the smooth function calculation unit performs truncation so that the value of the smooth function does not exceed the smooth function threshold.

  The distance index information estimation device generates an energy set in which energy including the error function calculated by the error function calculation unit and the smooth function calculated by the smooth function calculation unit is collected by the energy propagation unit, By generating and propagating energy between adjacent pixels at a predetermined recurrence formula, the distance index candidate and the energy for each pixel are updated.

  Moreover, the distance index information estimation apparatus calculates an evaluation function indicating the energy updated by the energy propagation unit by the evaluation function calculation unit. Then, in the distance index information estimation device, the distance index estimation unit estimates a distance index that minimizes the evaluation function calculated by the evaluation function calculation unit from among the distance index candidates as distance index information. Here, since the upper limit is set for the energy in the truncation described above, the variation of the evaluation function is within a certain range, and the accuracy of the distance index information is improved.

  In the distance index information estimation apparatus, the evaluation value calculation unit obtains the kurtosis of the energy distribution represented by the evaluation function as an evaluation value. Here, since the evaluation value is calculated based on the kurtosis which is the sharpness of the energy distribution, the accuracy is improved and becomes information indicating the objective reliability of the distance information.

  In the present invention, the distance index is a distance in the depth direction (depth) or parallax that is an index of this distance. That is, the distance index information estimation apparatus estimates distance information indicating the distance for each pixel or parallax information indicating the parallax for each pixel as the distance index information.

  In the distance index information estimation apparatus according to the second invention of this application, the evaluation value calculation unit calculates the kurtosis of the energy distribution as a texture evaluation value for determining an estimation error caused by the texture.

At this time, the texture evaluation value calculation unit is configured such that n _f is the number of distance index candidates f _q , b _q (f _q ) is energy, b _q ￣ is energy average value, and σ (b _q ) is energy standard. It is preferable to calculate the texture evaluation value K _q using the formula (9) defined as the deviation (the third invention of the present application).

  According to such a configuration, the distance index information estimation apparatus calculates the texture evaluation value in consideration of the kurtosis that is the degree of sharpness of the energy distribution, so that the accuracy of the texture evaluation value can be improved.

  The distance index information estimation apparatus according to the fourth invention of the present application determines whether the texture evaluation value calculated by the texture evaluation value calculation unit is equal to or less than a preset texture threshold, and the texture evaluation value is the texture threshold. In the following cases, a texture estimation error determination unit that determines that an estimation error caused by the texture has occurred is further provided.

  According to such a configuration, the distance index information estimation apparatus determines an estimation error due to the texture by using a property that the texture evaluation value becomes smaller at the pixel where the estimation error due to the texture occurs.

In addition, the distance index information estimation device according to the fifth invention of the present application calculates the optical axis intersection of a camera that has captured a stereo image by camera calibration, and a virtual sphere having a radius set in advance so that a subject in the stereo image can be accommodated. Is generated around the intersection of the optical axes, calculates the distance index from the central camera to the inner surface of the phantom sphere, and the calculated distance index is the area where the estimation error caused by the texture occurred in the captured image of the central camera And a texture estimation error suppression unit assigned as distance index information of each pixel.
According to such a configuration, the distance index information estimation apparatus performs a process for suppressing the estimation error for a region where it is determined that an estimation error due to the texture has occurred.

In the distance index information estimation device according to the sixth invention of the present application, the error function calculation unit has color information between the processing target pixel in the standard image and the corresponding pixel corresponding to the processing target pixel in the reference image that is the other of the stereo images. And calculating an error function for weighting the difference absolute value by dividing it by a preset number of primary colors.
According to this configuration, the distance index information estimation device can sharpen the contour of the subject and estimate distance information with high accuracy.

  In addition, the distance index information estimation apparatus according to the first invention of the present application uses a general computer as an error function calculation unit, a smooth function calculation unit, an energy propagation unit, an evaluation function calculation unit, a distance index estimation unit, and an evaluation value calculation unit. It can also be realized by a distance index information estimation program to be operated. This program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

The present invention has the following excellent effects.
According to the first invention of the present application, since the upper limit is provided for the energy in the truncation described above, the variation of the evaluation function is kept within a certain range, and the accuracy of the distance index information can be improved. Furthermore, according to the first invention of the present application, the evaluation value is calculated in consideration of the kurtosis that is the sharpness of the energy distribution. Therefore, the accuracy of the evaluation value is improved, and the evaluation value is obtained as an objective reliability of the distance index information. Can be presented.

According to the second and third aspects of the present invention, since the texture evaluation value is calculated in consideration of the kurtosis of the energy distribution, the accuracy of the texture evaluation value can be improved.
According to the fourth aspect of the present invention, it is possible to accurately determine the estimation error caused by the texture by using the property that the texture evaluation value becomes smaller at the pixel where the estimation error caused by the texture occurs.

According to the fifth aspect of the present invention, it is possible to accurately perform a process of suppressing the estimation error for the area determined to have an estimation error caused by the texture.
According to the sixth aspect of the present invention, it is possible to estimate the distance index information with high accuracy by sharpening the contour of the subject.

In this invention, it is explanatory drawing explaining distance estimation by the reliability propagation method. In the present invention, it is a flowchart which shows the procedure of distance estimation by the reliability propagation method. It is a block diagram which shows the structure of the distance information estimation apparatus which concerns on 1st Embodiment of this invention. FIG. 4 is an explanatory diagram illustrating calculation of an error function and a smooth function for each distance candidate in the distance information estimation apparatus of FIG. 3. FIG. 4 is an explanatory diagram for explaining message propagation for each distance candidate in the distance information estimation apparatus of FIG. 3. FIG. 4 is an explanatory diagram for explaining message propagation for each distance candidate in the distance information estimation apparatus of FIG. 3. In the first embodiment of the present invention, (a) is a graph showing an energy distribution of pixels in which an estimation error due to occlusion has not occurred, and (b) is an energy distribution of pixels in which an estimation error due to occlusion has occurred. It is a graph which shows. In the first embodiment of the present invention, (a) is a graph showing an energy distribution of pixels in which an estimation error due to texture has not occurred, and (b) is an energy distribution of pixels in which an estimation error due to texture has occurred. It is a graph which shows. It is a flowchart which shows operation | movement of the distance information estimation apparatus of FIG. It is a block diagram which shows the structure of the distance information estimation apparatus which concerns on 2nd Embodiment of this invention. In the distance information estimation apparatus of FIG. 10, it is explanatory drawing explaining suppression of the estimation error resulting from an occlusion. In the distance information estimation apparatus of FIG. 10, it is explanatory drawing explaining suppression of the estimation error resulting from an occlusion. It is a flowchart which shows operation | movement of the distance information estimation apparatus of FIG. It is a block diagram which shows the structure of the distance information estimation apparatus which concerns on 3rd Embodiment of this invention. In 3rd Embodiment of this invention, (a) is explanatory drawing when the message propagation from the pixel of a message propagation origin to the pixel of a process target is restrict | limited, (b) is a message propagation destination from a pixel of a process target. It is explanatory drawing when the message propagation to this pixel is restricted. In 3rd Embodiment of this invention, (a) and (b) are explanatory drawings when message propagation is not restrict | limited. It is a flowchart which shows operation | movement of the distance information estimation apparatus of FIG. It is a block diagram which shows the structure of the distance information estimation apparatus which concerns on the modification 1 of this invention. In the embodiment of the present invention, (a) is a standard image taken by the left camera, and (b) is a reference image taken by the right camera. In the embodiment of the present invention, (a) is a distance image, and (b) is an image showing a result of outputting an occlusion area from the distance image of (a). In the embodiment of the present invention, (a) is a distance image after suppressing an estimation error due to occlusion, and (b) is an image showing a result of outputting a non-textured region from the distance image of (a). .

(Outline of the present invention: Distance estimation by reliability propagation method)
In each embodiment of the present invention, the reliability propagation method is used as a technique for minimizing the energy defined by the error function between stereo images and the smooth function indicating the continuity of the distance index. Therefore, first, an outline of the reliability propagation method in the embodiment of the present invention will be described.

  In the following embodiments, the distance index described in the claims is described as being a distance (depth) in the depth direction. That is, distance information indicating a distance for each pixel is estimated as the distance index information described in the claims.

This reliability propagation method (BP method: Belief Propagation method) assumes a Markov Random Field (MRF) model for distance information, and is configured (specified) with an error function and a smooth function between stereo images. By minimizing energy, smooth distance information with less noise is generated. Here, in the reliability propagation method, as shown in FIG. 1, energy m related to the probability of which distance is assigned to each pixel called a message is received from the adjacent pixel s of the pixel p to be processed, and from these, The message is updated and propagated to the adjacent pixel q. In the reliability propagation method, this process is repeated for all the pixels in the image to obtain distance information of each pixel.
The energy described in the claims becomes a message in the reliability propagation method.

  Here, in FIG. 1, a frame (white square) corresponds to a pixel of the reference image, and an arrow m indicates a message (energy) propagation between the pixels. As shown in FIG. 1, a graph in which pixels that propagate this message are arranged vertically and horizontally as nodes is called an energy set (message set). Furthermore, a pixel s to which the character “s” is attached indicates an adjacent pixel from which a message is propagated, a pixel p to which the character “p” is attached indicates a pixel to be processed, and a character “q” is attached. The pixel q is an adjacent pixel which is a message propagation destination. In FIG. 1, only a part of the reference numerals are shown for simplicity of explanation.

With reference to FIG. 2, the procedure of distance estimation by the reliability propagation method will be described.
As shown in FIG. 2, in the reliability propagation method, an error function D _p (f _p ) between pixels necessary for message generation (hereinafter “error function D _p ”) is calculated (step S1). In the present invention, similarity such as SSD, SAD, ZNCC, etc. can be used for the error function, and here, Expression (1) indicating similarity by the absolute value of the difference in color information between corresponding pixels of the stereo image is used. To do. According to this formula (1), since the similarity is calculated in units of pixels, the contour of the subject can be sharpened.

Here, p is the pixel to be processed, _{f p} is the distance of the pixel p candidate, c is the color information, r, g, b are RGB values, _{d p} is the disparity corresponding to the distance candidate _{f p, I c} stereo image Is a pixel value in a standard image, I _c ′ is a pixel value in a reference image that is the other of the stereo images, and λ _data represents a weighting coefficient.
A distance candidate (distance index candidate) is a distance candidate set in advance within an arbitrary range (see FIG. 4 and the like).

That is, the expression (1) is obtained by _calculating the color information I _c (p) of the pixel p to be processed in the standard image I _c and the color information I _c ′ (p + d _p ) of the corresponding pixel (p + d _p ) in the reference image I _c ′. Is divided by a preset number of primary colors (for example, 3) and weighted by a weighting coefficient λ _data . The corresponding pixel (p + d _p ) is a pixel corresponding to the pixel p to be processed in the reference image I _c ′.

Further, in equation (1), the parallax d _p can be calculated backward from the distance candidate f _p by utilizing the fact that the distance can be calculated by multiplying the reciprocal of parallax by the interval (baseline) of the stereo camera. This baseline can be obtained by camera calibration such as OpenCV (URL “http://opencv.jp/”).

Subsequently, the reliability propagation method, as shown in the following formula (2), and the distance candidate f _p, the difference absolute value between the distance candidate f _q of the pixel q | smoothly as defined _| f p _-f q The function V (f _p −f _q ) (hereinafter “smooth function V”) is calculated (step S2). This smooth function V indicates the smoothness of distance information (for example, low noise), in other words, the continuity of distance. Here, q is an adjacent pixel that is a message propagation destination of the pixel p.

  In the reliability propagation method, a message is generated and propagated by a generation formula defined by the following formula (3) (step S3).

  m is a message, t is the number of iterations, N (p) \ q is a set of four neighboring pixels other than the pixel q that passes the message to the pixel p, s is an element pixel of the pixel set, and min is a function that returns a minimum value. . This formula (3) is a recurrence formula and indicates that the process of updating the message is repeatedly performed based on the received message. For example, in the reliability propagation method, the message is propagated in all pixels by updating the message with the processing target pixel p while moving the processing target pixel p so as to perform raster scanning.

It is assumed that the message update process is repeated for all pixels up to T times (for example, 10 times) set in advance. In this case, the evaluation function b _q (f _q ) (hereinafter, “evaluation function b _q ”) regarding the distance candidate f _q of the pixel q can be expressed by the following equation (4). In the reliability propagation method, the distance candidate f _q that minimizes the evaluation function b _q is estimated as the final distance of each pixel (step S4).

(First embodiment)
[Configuration of distance information estimation device]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

With reference to FIG. 3, the structure of the distance information estimation apparatus (distance index information estimation apparatus) 1 which concerns on 1st Embodiment of this invention is demonstrated.
The distance information estimation device 1 estimates distance information by the reliability propagation method, calculates an evaluation value indicating the reliability of the distance information, and determines whether an estimation error of the distance information has occurred due to the evaluation value. Judgment.

  As shown in FIG. 3, the distance information estimation device 1 includes an error function calculation unit 10, a smooth function calculation unit 11, a message generation / propagation unit (energy propagation unit) 12, an evaluation function calculation unit 13, and a distance estimation. Unit (distance index estimation unit) 14, evaluation value calculation unit 15, occlusion region determination unit (occlusion estimation error determination unit) 17, and non-textured region determination unit (texture estimation error determination unit) 19.

In the present embodiment, the distance information estimation device 1 uses an evaluation value O _q (an occlusion evaluation value, hereinafter referred to as “evaluation value O”) for determining an estimation error due to occlusion as an evaluation value, and an estimation due to a texture. An evaluation value K _q (texture evaluation value, hereinafter referred to as “evaluation value K”) for determining an error is calculated. Then, the distance information estimation apparatus 1 uses the evaluation value O and the evaluation value K to determine an estimation error due to occlusion and an estimation error due to texture as distance information estimation errors.

  The estimation error caused by the texture is, for example, that a distance accuracy estimation error occurs due to a decrease in matching accuracy in a region (textureless region) where no texture exists in a stereo image.

The distance information estimation apparatus 1 receives a stereo image (standard image and reference image) obtained by photographing a subject (not shown) from the left camera _CL and the right camera _CR (stereo camera).
The left camera _CL and the right camera _CR are photographing cameras capable of photographing a still image and a moving image, and are arranged at a predetermined interval (baseline) as shown in FIG. Here, for example, of the left camera _CL and the right camera _CR , the left camera _CL is used as a reference camera, an image taken by the left camera _CL is used as a reference image, and an image taken by the right camera _CR is used as a reference image.

Error function calculation unit 10, together with the stereo image is input, for each distance candidate, to be equal to or less than a preset error function threshold T _data, the error function D _p is a similarity of the input stereo images It is to calculate. The error function calculation unit 10 uses the following equation (5) and (6), the pixel s as shown in FIG. 4, p, the q, calculates the error function D _p per distance candidate.
Here, if is a function that outputs the value described in the preceding stage when the following conditional expression is satisfied.

In the present embodiment, the distance candidates f _s , f _p , and f _q are set in advance within the range of 0, 1,..., K,. (K and N are integers satisfying K <N−1).
In FIG. 4, the quadrilaterals with the letters s, p, and q are the pixels s, p, and q.

At this time, the error function calculation unit 10 performs truncation so that the value of the error function D _p does not exceed the error function threshold T _data as shown in Expression (6). Specifically, the error function calculation unit 10, the value of the error function _{D p} may exceed the error function threshold _{T data,} and outputs the message generation and propagation unit 12 an error function threshold _{T data} as the value of the error function _{D p} . On the other hand, when the value of the error function D _p is equal to or less than the error function threshold value T _data , the error function calculation unit 10 outputs the value of the error function D _{p to} the message generation / propagation unit 12 as it is.

The smooth function calculation unit 11 calculates the smooth function V so as to be equal to or less than a preset smooth function threshold value T _smooth for each distance candidate. As shown in FIG. 4, the smooth function calculation unit 11 performs all combinations of the distance candidate f _p (0 to N−1) of the pixel p and the distance candidate f _q (0 to N−1) of the pixel q. The smooth function V is calculated using the following equation (7).

At this time, the smooth function calculation unit 11 performs truncation so that the value of the smooth function V does not exceed the smooth function threshold value T _smooth as shown in Expression (7). Specifically, smooth function calculating unit 11, the value of the smooth function V may exceed the smooth function thresholds _{T smooth smooth,} and outputs the message generation and propagation part 12 a smooth function thresholds _{T smooth smooth} as the value of the smooth function V. On the other hand, when the value of the smooth function V is equal to or less than the smooth function threshold value T _smooth , the smooth function calculation unit 11 outputs the value of the smooth function V to the message generation / propagation unit 12 as it is.

  Here, the Markov random field and the smooth function V will be supplemented. This Markov random field is a model that shows that the color, brightness, and distance of adjacent pixels in the reference image are similar. The smooth function V is a function related to the continuity of the distance based on the Markov random field. When the distance candidates are assigned to the pixels of the reference image, if the distance candidates of the adjacent pixels are the same, the energy is obtained. If it is small and different, energy increases. For example, in a Markov random field, when the distance candidate of a certain pixel is 1 and the distance candidate of the adjacent pixel is 1, the energy is 0. Further, when the distance candidate of a certain pixel is 1 and the distance candidate of a pixel adjacent to the pixel is 2, for example, the energy is 1. Further, when the distance candidate of a certain pixel is 1 and the distance candidate of the adjacent pixel is 10, for example, the energy is 9. That is, in the reliability propagation method, considering that the distance candidate with the minimum energy is the final distance, pixels adjacent to each other are likely to have the same distance due to this Markov random field.

Message generation and propagation unit (energy propagating portion) 12, an error function D _p calculated by the error function calculation unit 10, and the smooth function V calculated by the smooth function calculating unit 11 is inputted. Then, the message generation / propagation unit 12 generates an energy set in which messages including the error function _Dp and the smooth function V are collected as in FIG.

Specifically, the message generation / propagation unit 12 models (generates) an energy set by arranging a message for each pixel at a position corresponding to the pixel position of the reference image.
Next, the message generation / propagation unit 12 updates the message for each pixel by generating and propagating the message between adjacent pixels in the energy set using the above-described equation (3). First, the message generation and propagation unit 12, as shown in FIGS. 5 and 6, in the pixel s, the message _m s ₍₀₎ of the distance candidate _{f s (0~N-1) ~m} s (N-1) And propagate from pixel s to pixel p. Next, the message generator / propagator 12 generates the messages m _{p → q} (0) to m _{p → q} (N−1) of the distance candidates f _p (0 to N−1) at the pixel p, and Propagation from pixel p to pixel q.

The evaluation function calculation unit 13 calculates the evaluation function b _q indicating the message updated by the message generation / propagation unit 12 using the above-described equation (4). Then, the evaluation function calculation unit 13 outputs the calculated evaluation function b _q to the distance estimation unit 14, the evaluation value O calculation unit 16, and the evaluation value K calculation unit 18.

The distance estimation unit (distance index estimation unit) 14 receives the evaluation function b _q calculated by the evaluation function calculation unit 13. And the distance estimation part 14 estimates the distance candidate _fq from which the evaluation function _bq becomes the minimum among distance candidates _fq as distance information. That is, the distance estimation unit 14 estimates the distance candidate f _q that minimizes the evaluation function b _q of the above-described equation (4) as final distance information of the pixel q of the reference image.

  Here, the distance estimation unit 14 uses, as distance information, a distance image in which the brightness of the pixel is higher as the distance of the pixel is closer and the brightness of the pixel is lower as the distance of the pixel is deeper. Generate and output. This distance image can be used, for example, for generating a three-dimensional model of the subject.

The evaluation value calculation unit 15 obtains an evaluation value from the shape of the energy distribution in the evaluation function b _q , and includes an evaluation value O calculation unit (occlusion evaluation value calculation unit) 16 and an evaluation value K calculation unit (texture evaluation value). (Calculation unit) 18.

The evaluation value O calculation unit (occlusion evaluation value calculation unit) 16 receives the evaluation function b _q calculated by the evaluation function calculation unit 13. The evaluation function b _q can be said to be information indicating a message for each distance candidate f _q for each pixel of the reference image, that is, an energy distribution (message distribution) in which the distance candidate f _q is associated with the message.

  Specifically, the evaluation value O calculation unit 16 calculates the evaluation value O by dividing the difference between the maximum value at which the message is highest and the minimum value at which the message is lowest by the maximum value. . Specifically, the evaluation value O calculation unit 16 calculates the evaluation value O using the following formula (8), and outputs it to the occlusion region determination unit 17.

Here, max is a function that returns the maximum value. Therefore, max (b _q (f _q )) indicates the maximum value of the message at pixel q. Further, min (b _q (f _q )) indicates the minimum value of the message at the pixel q.

The occlusion area determination unit (occlusion estimation error determination unit) 17 receives the evaluation value O from the evaluation value O calculation unit 16. Then, the estimated error occlusion area determination unit 17 determines whether the evaluation value O is equal to or less than a preset occlusion threshold T _o, the evaluation value O if the following occlusion threshold T _o, due to the occlusion Is determined to have occurred.

Here, in a pixel in which an estimation error due to occlusion has not occurred, the difference ΔE _O between the maximum value max and the minimum value min of the message becomes large as in the energy distribution of FIG. On the other hand, in a pixel in which an estimation error due to occlusion has occurred, the difference ΔE _O is smaller than that in FIG. 7A, as in the energy distribution in FIG.

Further, the difference ΔE _{O in} FIG. 7 is proportional to the evaluation value O because it is a numerator of the formula (8). In other words, the occlusion area determination unit 17 uses the property that the evaluation value O is smaller than the pixel in which the estimation error due to occlusion has occurred and the estimation error due to occlusion has not occurred, and is caused by occlusion. Judgment error is determined. The occlusion area determining unit 17, evaluation value O outputs a pixel region determined as follows occlusion threshold T _o as an occlusion region. This occlusion area can be used, for example, to grasp an area in which an estimation error caused by occlusion has occurred when generating a three-dimensional solid model of a subject.
Incidentally, the occlusion area determination unit 17, when the evaluation value O exceeds the threshold value T _o, it goes without saying that it is determined that the estimated error due to occlusion does not occur.

The evaluation value K calculation unit (texture evaluation value calculation unit) 18 receives the evaluation function b _q calculated by the evaluation function calculation unit 13. Then, the evaluation value K calculation unit 18 calculates the kurtosis of this energy distribution as the evaluation value K for each pixel of the reference image.

Specifically, as shown in the following formula (9), the evaluation value K calculator 18 calculates the number n _f of distance candidates f _q (for example, a total of N from 0 to N−1), a message, The evaluation value K is calculated based on the average value b _qメ ッ セ ー ジ of the message and the standard deviation σ (b _q ) of the message. Then, the evaluation value K calculation unit 18 outputs the calculated evaluation value K to the no-texture region determination unit 19.

The evaluation value K is input from the evaluation value K calculation unit 18 to the non-texture area determination unit (texture estimation error determination unit) 19. Then, non-texture region determining unit 19 determines whether the evaluation value K is equal to or less than a preset texture threshold T _K, when the evaluation value K is less than the texture threshold T _K, due to the texture estimation It is determined that an error has occurred.

  Here, in the pixel in which the estimation error due to the texture does not occur, the energy distribution is sharp as shown in FIG. 8A, and the evaluation value K indicating the kurtosis increases. On the other hand, as shown in FIG. 8B, the energy distribution is not sharp in the pixel in which the estimation error caused by the texture occurs, and the evaluation value K is smaller than that in FIG.

This kurtosis indicates the sharpness of the energy distribution, and has the following characteristics compared to the normal distribution. That is, as the kurtosis increases, the energy distribution has a sharp peak and a long tail (see FIG. 8A). On the other hand, as the kurtosis decreases, the energy distribution has a rounder peak and a shorter tail (see FIG. 8B).
In FIG. 8, the tail is a flat portion that is formed by truncation and becomes the upper limit of the message.

That is, the non-textured area determination unit 19 uses the property that the evaluation value K is smaller than the pixel in which the estimation error due to the texture is generated and the estimation error due to the texture is not generated, The estimated estimation error is determined. Then, non-texture region determining unit 19, evaluation value K and outputs the pixel region determined as follows texture threshold T _K as a non-texture region. This occlusion area can be used, for example, for grasping an area in which an estimation error caused by texture has occurred when generating a three-dimensional solid model of a subject.
Incidentally, no texture region determining unit 19, when the evaluation value K exceeds the texture threshold T _K, it is obvious to judge that the estimated error due to the texture does not occur.

[Operation of distance information estimation device]
The operation of the distance information estimation apparatus 1 will be described with reference to FIG. 9 (see FIG. 3 as appropriate).
Distance information estimating apparatus 1, the error function calculation unit 10, for each distance candidate, calculates the error function D _p to be equal to or less than the error function thresholds T _data. In other words, the error function calculator 10 uses the equation (5) and (6), to calculate the error function _{D p,} perform truncation (step S10).

The distance information estimation device 1 uses the smooth function calculation unit 11 to calculate the smooth function V so as to be equal to or less than the smooth function threshold value T _smooth for each distance candidate. That is, the smooth function calculation unit 11 calculates the smooth function V using the above equation (7) and performs truncation (step S11).

  The distance information estimation apparatus 1 uses the message generation / propagation unit 12 to generate an energy set as in FIG. Then, the message generation / propagation unit 12 generates and propagates a message between adjacent pixels in the energy set using the above-described equation (3) (step S12).

In the distance information estimation apparatus 1, the evaluation function calculation unit 13 calculates the evaluation function b _q for each distance candidate using the above-described equation (4) (step S13). Then, the distance information estimation apparatus 1 uses the distance estimation unit 14 to estimate, as distance information, a distance that minimizes the evaluation function b _{q from} among the distance candidates (step S14).

  The distance information estimation apparatus 1 calculates the evaluation value O by the evaluation value O calculation unit 16. That is, the evaluation value O calculation unit 16 calculates the evaluation value O by dividing the difference between the maximum value of the message and the minimum value of the message by the maximum value using the above-described equation (8) (step S1). S15).

Distance information estimating apparatus 1, the occlusion area determination unit 17, evaluation value O is equal to or less than the occlusion threshold T _o, when the evaluation value O is less occlusion threshold T _o, estimation error due to occlusion Is determined to have occurred. (Step S16).

In the distance information estimation apparatus 1, the evaluation value K calculation unit 18 calculates an evaluation value K that is the kurtosis of the energy distribution. That is, the evaluation value K calculation unit 18, as shown in the equation (9), the number n _{f of} distance candidates, the message, the average value b _qメ ッ セ ー ジ of the message, and the standard deviation σ (b _q ) of the message Based on the above, the evaluation value K is calculated (step S17).

Distance information estimating apparatus 1, the non-texture region determining unit 19, evaluation value K is equal to or less than the texture threshold T _K, when the evaluation value K is less than the texture threshold T _K, due to the texture estimation It is determined that an error has occurred (step S18).

  As described above, the distance information estimation device 1 according to the first embodiment of the present invention calculates the evaluation value O in consideration of not only the minimum value of the message but also the maximum value of the message, and the kurtosis of the energy distribution. A certain evaluation value K is calculated. Then, the distance information estimation device 1 determines these estimation errors by using the property that the evaluation value O and the evaluation value K become small at a pixel in which an estimation error due to occlusion or texture has occurred. Thereby, the distance information estimation apparatus 1 can improve the accuracy of estimation of the estimation error caused by occlusion and texture, and can accurately present the region where the estimation error has occurred.

Further, as shown in FIGS. 7 and 8, the distance information estimation apparatus 1 has an upper limit of the message by truncation, so that the variation of the evaluation function b _q falls within a certain range and improves the accuracy of the distance information. be able to.

(Second Embodiment)
[Configuration of distance information estimation device]
With reference to FIG. 10, the difference from the first embodiment will be described regarding the configuration of the distance information estimation device 1B according to the second embodiment of the present invention. This distance information estimation device 1B is different from the first embodiment in that an estimation error caused by occlusion and texture is suppressed. For this reason, the distance information estimation device 1 </ b> B further includes a texture estimation error suppression unit 20.

As shown in FIG. 10, the distance information estimating apparatus 1B, the left camera C _L and the left stereo image that is a captured image of the center camera C _C, right stereo image that is a captured image of the right camera C _R and the center camera C _C Are entered.

The center camera C _C, like the left camera C _L and the right camera C _R, is capable of photographing shot camera still and moving images. Further, the left camera C _L and the right camera C _R, respectively, are disposed on the left and right by a certain distance apart from the center camera C _C. Here, the center camera C _C as the reference camera, the captured image of the center camera C _C as a reference image, a reference image captured image of the left camera C _L and the right camera C _R.

  The error function calculation unit 10, the smooth function calculation unit 11, the message generation / propagation unit 12, the evaluation function calculation unit 13, and the distance estimation unit 14 perform processing of the left stereo image and the right stereo image, respectively. Since it is the same as each means of FIG. 3, detailed description is abbreviate | omitted.

Here, in order to suppress an estimation error caused by occlusion described later, an evaluation function b _q obtained from both the left stereo image and the right stereo image (hereinafter, “both stereo images”) is required. Therefore, the evaluation function calculation unit 13 outputs the evaluation function b _q obtained from both stereo images to the evaluation value O calculation unit 16B. Further, the distance estimation unit 14 outputs distance information (distance image) obtained from both stereo images to the evaluation value O calculation unit 16B.

On the other hand, an evaluation function b _q obtained from either the left stereo image or the right stereo image may be used to suppress the estimation error caused by the texture described later. Therefore, the evaluation function calculation unit 13 outputs the evaluation function b _q obtained from the left stereo image to the evaluation value K calculation unit 18.

The evaluation value O calculation unit 16B calculates an evaluation value O from both stereo images and outputs the evaluation value O to the occlusion region determination unit 17. Further, the evaluation value O calculation unit 16B suppresses an estimation error caused by occlusion by using the evaluation value O calculated from both stereo images and the distance information of both stereo images.
The calculation of the evaluation value O is the same as that of the evaluation value O calculation unit 16 in FIG.

  Specifically, the evaluation value O calculation unit 16B compares the evaluation value O calculated from the left stereo image with the evaluation value O calculated from the right stereo image for each pixel of the reference image. Based on the comparison result, the evaluation value O calculation unit 16B outputs the distance information on the side where the evaluation value O is larger among the distance information of both stereo images to the texture estimation error suppression unit 20 as the distance information of the pixel. To do. In other words, the evaluation value O calculation unit 16B uses the fact that the evaluation value O becomes small at the pixel where the estimation error due to occlusion has occurred, and obtains the distance information of both stereo images generated by the distance estimation unit 14 It integrates into one distance information that suppresses the estimation error.

Since the occlusion area determination unit 17 outputs the occlusion areas of both stereo images as in FIG. 3, detailed description thereof is omitted.
This occlusion area is not necessary for suppressing an estimation error caused by occlusion, but is presented as reference information.

Since the evaluation value K calculation unit 18 calculates the evaluation value K for the left stereo image, as in FIG. 3, a detailed description thereof is omitted.
Similarly to FIG. 3, the non-textured area determination unit 19 generates a non-textured area for the left stereo image and outputs the non-textured area determination unit 19 to the texture estimation error suppression unit 20.

  The texture estimation error suppression unit 20 receives distance information from the evaluation value O calculation unit 16 </ b> B and receives a non-texture region from the non-texture region determination unit 19. And the texture estimation error suppression part 20 suppresses the estimation error resulting from a texture from this distance information using this non-textured area | region.

With reference to FIGS. 11 and 12, the suppression of the estimation error caused by the texture will be described (see FIG. 10 as appropriate).
In order to simplify the description, as shown in FIG. 11, an example in which two subjects (wrestlers) O ₁ and O ₂ are working on the ring A will be described. In this case, center camera C _C of Figure 10 is located opposite the center of the playing field A, that were taken reference image P from (arrow direction in FIG. 11) the object O _1, O ₂ of lateral doing efforts And

In FIG. 11, portions of the earth A and the subjects O ₁ and O ₂ that are not included in the reference image P are illustrated by broken lines. Moreover, in FIG. 11, the area | region of the background B used as the back | inner side ground of the soil A in the reference image P is the non-textured area TX, and is illustrated by hatching. In this example, the texture estimation error suppression unit 20 performs a process for suppressing the estimation error caused by the texture on the hatched portion in FIG.

Here, the center camera _{C C,} and the ring A, when viewed from the upper side and the subject _O 1, _{O 2,} these positional relationships is as shown in FIG. In FIG. 12, the earthwork A and the subjects O ₁ and O ₂ are illustrated by broken lines, and the distance D described later is illustrated by a chain line. Further, the hatched portion (non-textured region TX) in FIG. 11 has a curved surface shape obtained by cutting off a part of the sphere surface (balloon in FIG. 12).

First, the texture estimation error suppression unit 20 includes a left camera C _L, and the center camera C _C, the optical axis intersecting point CE of the optical axis LA intersects the right camera C _R, is calculated by the camera calibration. Here, texture estimation error suppression unit 20 is not required to obtain the optical axis intersecting point CE on all cameras, it may be contained a central camera C _C for photographing the reference image. In other words, the texture estimation error suppression unit 20, the center camera C _C and a set of the left camera C _L, or from one set of the central camera C _C and the right camera C _R, may be obtained an optical axis intersecting point CE .

Next, the texture estimation error suppression unit 20 generates a virtual sphere VC having a preset radius r around the optical axis intersection CE. Here, in the texture estimation error suppressing unit 20, the radius r is set in advance so that all the subjects (for example, earthen A, subjects O ₁ and O ₂ ) are included in the virtual sphere VC. Then, the texture estimation error suppression unit 20, the distance D from the center camera C _C deep side surface of the virtual sphere VC is calculated, this distance D, assigned as the distance information of each pixel in the non-texture region TX.

That is, the distance information of each pixel in the non-textured area TX can be assigned using the following calculation formula. First, the texture estimation error suppression unit 20, the simultaneous equations of the following (10) and (11), to calculate the intersection of the optical axis LA of the virtual sphere VC and the center camera _{C C} (X, Y, Z) .

The above equation (10) is an equation representing the phantom sphere VC. Here, a, b, and c are the world coordinates of the center of the virtual sphere VC (the world coordinates of the optical axis intersection point CE), and r is the radius of the virtual sphere VC.
The equation (11) is an equation representing a straight line passing through the pixels in the center camera C _C of the optical principal point and a non-texture region TX. Here, e, g, i are inclinations (known) calculated from the world coordinates of each image in the non-textured area TX and the world coordinates of the optical principal point, and f, h, j are worlds of the optical principal point. Coordinates (known).

Then, the texture estimation error suppression unit 20 calculates the distance D from the central camera _CC to the inner surface of the phantom sphere VC according to the following (12). Here, z is a distance D to the inner surface of the phantom sphere VC, i and j are coordinates (known) of the reference image, A is an internal parameter (known) calculated by camera calibration, R is a rotation matrix (known) calculated by camera calibration, and U is a parallel progression (known) calculated by camera calibration.

[Operation of distance information estimation device]
The operation of the distance information estimation device 1B will be described with reference to FIG. 13 (see FIG. 10 as appropriate).
Since the process of step S20-S24 is the same as that of step S10-S14 of FIG. 9, description is abbreviate | omitted.

  The distance information estimation device 1B suppresses an estimation error caused by occlusion by using the non-textured area and the distance information by the evaluation value O calculation unit 16B. That is, the evaluation value O calculation unit 16B calculates the evaluation value O from both stereo images, compares these evaluation values O, and calculates the distance information generated from the stereo image on the side where the evaluation value O is large as the reference image. It outputs as distance information of each pixel (step S25).

  The processing in steps S26 to S28 is the same as that in steps S16 to S18 in FIG.

In the distance information estimation device 1 </ b> B, the texture estimation error suppression unit 20 suppresses the estimation error caused by the texture from the distance information using the non-textured region. In other words, the texture estimation error suppression unit 20 calculates the left camera C _L, and the center camera C _C, the camera calibration of the optical axis intersecting point CE of the right camera C _R. Then, the texture estimation error suppression unit 20 generates a virtual sphere VC having a radius r around the optical axis intersection point CE. Furthermore, the texture estimation error suppression unit 20 calculates the distance D to the inner side surface of the virtual sphere VC from the center camera C _C, assigned as the distance information of each pixel in the non-texture region TX (step S29).

  As described above, the distance information estimation device 1B according to the second embodiment of the present invention can generate the distance information with high accuracy by suppressing the estimation error caused by the occlusion and the texture. By using this distance information, for example, an integral photography (IP) 3D image with a high sense of depth can be generated.

(Third embodiment)
[Configuration of distance information estimation device]
With reference to FIG. 14, a difference of the configuration of the distance information estimation device 1 </ b> C according to the third embodiment of the present invention from the first embodiment will be described. This distance information estimation device 1C is different from the first embodiment in that, when a message is propagated, when the color is greatly different between adjacent pixels, the propagation is limited. For this reason, the distance information estimation device 1 </ b> C further includes a message propagation restriction determination unit (energy propagation determination unit) 21.

  Here, the error function calculation unit 10, the smooth function calculation unit 11, the evaluation value calculation unit 15, the occlusion region determination unit 17, and the non-texture region determination unit 19 are the same as the respective units in FIG. Detailed description will be omitted.

The message propagation restriction determination unit (energy propagation determination unit) 21 sends a message between adjacent pixels depending on whether or not the absolute value of the color information difference between adjacent pixels in the reference image is equal to or less than a preset color information threshold value _Tc. Whether or not can be propagated.

The message propagation restriction determination unit 21 is preset with a color information threshold value _Tc related to the absolute difference value of the color information between adjacent pixels in order to determine message propagation restriction. In this case, the color information threshold T value of _c becomes smaller restriction message propagation stronger, so that the influence of the smooth function increases also decreases the value of the weighting factor lambda _data, color information threshold T _c and the weighting factor It is preferable to set λ _data in advance.

Further, the message propagation restriction determination unit 21 determines whether or not the message propagation source pixel s depends on whether or not the absolute difference value of the color information between the message propagation source pixel s and the processing target pixel p is equal to or smaller than the color information threshold value _Tc. And whether or not the message can be propagated between the processing target pixel p. Specifically, the message propagation restriction determination unit 21 compares the absolute difference value of the color information of the pixels s and p with the color information threshold value _Tc , as shown in the following formula (13), to thereby propagate the message. The original weighting factor λ _{s, p} is calculated.

Here, the message propagation restriction determination unit 21 propagates a message between the pixels s and p when the absolute difference value of the color information of the pixels s and p is equal to or smaller than the color information threshold value _Tc as shown in Expression (13). It is determined that this is possible, and the weighting coefficient λ _{s, p} is calculated as 1. On the other hand, the message propagation restriction determination unit 21 determines that the message cannot be propagated between the pixels s and p when the difference absolute value of the color information of the pixels s and p exceeds the color information threshold value T _c , and the weight coefficient λ _s, Calculate _p as 0. Then, the message propagation restriction determination unit 21 outputs the calculated weighting factor λ _{s, p} to the message generation / propagation unit 12C.

Further, the message propagation restriction determination unit 21 determines whether the message propagation destination pixel q depends on whether or not the absolute difference value of the color information between the message propagation destination pixel q and the processing target pixel p is equal to or less than the color information threshold value _Tc. And whether or not the message can be propagated between the processing target pixel p. Specifically, the message propagation restriction determination unit 21 compares the absolute difference value of the color information of the pixels p and q with the color information threshold value _Tc as shown in the following formula (14), thereby message propagation. The previous weighting factor λ _{p, q} is calculated.

Here, the message propagation restriction determination unit 21 propagates a message between the pixels p and q when the absolute difference value of the color information of the pixels p and q is equal to or less than the color information threshold value _Tc as shown in the equation (14). It is determined that it can be performed, and the weighting coefficient λ _{p, q is set} to 1. On the other hand, if the absolute difference value of the color information of the pixels p and q exceeds the color information threshold value T _c , the message propagation restriction determination unit 21 determines that the message cannot be propagated between the pixels p and q, and the weight coefficient λ _{p, q} is calculated as 0. Then, the message propagation restriction determination unit 21 outputs the calculated weighting factor λ _{p, q} to the evaluation function calculation unit 13C.

The message generation / propagation unit 12C receives the message propagation source weighting factor λ _{s, p} from the message propagation restriction determination unit 21. Then, the message generation / propagation unit 12C generates and propagates a message using the following equation (15).

At this time, the message generation / propagation unit 12C compares the absolute difference value of the color information of the pixels p and q with the color information threshold value _Tc , as shown in Expression (15), and propagates or restricts the message. Judge whether to do. Specifically, the message generation / propagation unit 12C propagates a message from the pixel p to the pixel q when the difference absolute value of the color information of the pixels p and q is equal to or smaller than the color information threshold value _Tc . On the other hand, when the absolute difference value of the color information exceeds the color information threshold value _Tc , the message generation / propagation unit 12C does not propagate the message from the pixel p to the pixel q.

The message propagation restriction will be described in detail with reference to FIGS. 15 and 16 (see FIG. 14 as appropriate). 15 and 16, in the same energy set as in FIG. 1, the frame (white square) is a light pixel, and the hatched frame is a dark pixel. Here, it is assumed that the absolute difference value of the color information between the bright color pixel and the dark color pixel exceeds the color information threshold value _Tc .

For example, let us consider a case where the colors differ greatly between adjacent pixels, such as the boundary between the subject and the background. As shown in FIG. 15 (a), when the colors greatly differ between the pixels s and p, the distance information estimation device 1C sets the message propagation source weight coefficients λ _{s and p} to 0 from the above-described equation (13). Limit message propagation in two pixels s, p. Further, as shown in FIG. 15B, when the colors greatly differ between the pixels p and q, the distance information estimation apparatus 1C sets the message propagation destination weighting coefficients λ _{p and q} to 0 from the above-described equation (14). The message propagation in these two pixels p and q is limited.

On the other hand, if all of the pixels s, p, q are bright as shown in FIG. 16A or all of the pixels s, p, q are dark as shown in FIG. The estimation device 1C propagates a message between the pixels s, p, _{and q} with the weighting coefficients λ _{s, p} , λ _{p, q set} to 1 from the above equations (13) and (14).

That is, in the distance information estimation device 1C, the message propagation from the pixel s to the pixel p is controlled by the value of the message propagation source weight coefficient λ _{s, p} , and the message propagation destination weight coefficient λ _{p, q} is Controls message propagation from pixel p to pixel q.

The evaluation function calculation unit 13C receives the weighting coefficients λ _{p, q} calculated by the message propagation restriction determination unit 21 and calculates an evaluation function b _q defined by the following equation (16). Then, the evaluation function calculation unit 13C outputs the calculated evaluation function b _q to the distance estimation unit 14C, the evaluation value O calculation unit 16, and the evaluation value K calculation unit 18.

The distance estimation unit 14C estimates distance information in the same manner as the distance estimation unit 14 in FIG.
The distance estimation unit 14C includes a noise removal filter 14a. The noise removal filter 14a performs noise removal processing such as a median filter on the distance information (distance image) estimated by the distance estimation unit 14C. In this way, smooth distance information that is less affected by noise can be estimated.

[Operation of distance information estimation device]
The operation of the distance information estimation device 1C will be described with reference to FIG. 17 (see FIG. 14 as appropriate).
The processing in steps S30 and S31 is the same as that in steps S10 to S11 in FIG.

Whether or not the distance information estimation device 1C can propagate a message between adjacent pixels by the message propagation restriction determination unit 21 depending on whether or not the absolute value of the color information difference between adjacent pixels is equal to or less than the color information threshold value _Tc . Determine. Specifically, the message propagation restriction determination unit 21 compares the color information difference absolute value of the pixels s and p with the color information threshold value T _c as shown in the above-described equation (13), thereby obtaining the pixel s. , P, the message propagation source weighting factor λ _{s, p} is calculated as a result of the message propagation determination. Further, the message propagation restriction determination unit 21 compares the color information difference absolute value of the pixels p and q with the color information threshold value T _c as shown in the above-described equation (14), thereby obtaining The message propagation destination weighting factor λ _{p, q} is calculated as the message propagation determination result in (step S32).

The distance information estimation apparatus 1C generates and propagates a message using the above-described equation (15) by the message generation / propagation unit 12C (step S33).
In the distance information estimation device 1C, the evaluation function calculator 13C calculates the evaluation function b _q defined by the above equation (16) (step S34).
In addition, since the process of step S35-S39 is the same as that of step S14-S28 of FIG. 9, description is abbreviate | omitted.

  As described above, the distance information estimation device 1C according to the third embodiment of the present invention restricts message propagation when the absolute difference value of the color information between adjacent pixels is large. Accordingly, the distance information estimation device 1C can suppress the expansion of the subject, sharpen the contour of the subject, and generate distance information with high accuracy.

  In each embodiment, the reliability propagation method has been described as an example of a method for minimizing the energy related to the probability defined by the error function and the smooth function. However, the present invention is not limited to this. In the present invention, for example, a Viterbi algorithm can be used as this method.

  In each embodiment, the distance information is estimated as the distance index information. However, the present invention is not limited to this. That is, according to the present invention, a parallax information estimation device that estimates parallax information can be realized by replacing the above-described distance candidates with parallax candidates.

In each embodiment, the evaluation values K and O are calculated as evaluation values. However, the present invention is not limited to this.
In each embodiment, it has been described that both the evaluation values K and O are calculated. However, the present invention may calculate only one of the evaluation values K and O. In this case, the present invention may suppress only one of estimation errors due to texture or occlusion.

(Modification 1)
Hereinafter, with reference to FIG. 18, a distance information estimation apparatus 1D according to Modification 1 of the present invention will be described.
As shown in FIG. 18, the distance information estimation device 1D is obtained by adding the message propagation restriction determination unit 21 of FIG. 14 to the distance information estimation device 1B of FIG. Therefore, the distance information estimation device 1D can suppress the estimation error due to the occlusion and the texture, suppress the expansion of the subject, and generate highly accurate distance information.
18 is the same as FIG. 10 and FIG. 14, and detailed description thereof is omitted.

Hereinafter, as an example of the present invention, the results of experiments on estimation error determination accuracy will be described.
In this example, an experiment was performed using the distance information estimation apparatus 1 of FIG. In addition, as a stereo image to be input to the distance information estimation device 1, an image obtained by photographing the efforts of wrestlers (subjects) was prepared. A reference image taken by the left camera _CL is shown in FIG. 19A, and a reference image taken by the right camera _CR is shown in FIG. 19B. At this time, the left camera _{C L} and the right camera _{C R,} releases only 2 m from each other. Further, the distance from the left camera _CL to the subject is 25 meters, and distance candidates are set every 2 centimeters. Further, λ _data was set to 0.07, the error function threshold value T _data was set to 15, and the smooth function threshold value T _smooth was set to 1.7.

  A distance image generated by the distance information estimation apparatus 1 from the stereo image of FIG. 19 is shown in FIG. In the distance image of FIG. 20A, a region where an estimation error due to occlusion has occurred is shown in red. In addition, a region where an estimation error caused by the texture is generated is illustrated in blue.

  FIG. 20B shows the result of the distance information estimation device 1 outputting the occlusion area using the evaluation value O from the distance image of FIG. In the image of FIG. 20B, the low luminance value area is an area where an estimation error area caused by occlusion occurs (occlusion area), which almost coincides with the red area of FIG. From this, it can be seen that the distance information estimation apparatus 1 has a high accuracy in determining an estimation error caused by occlusion.

  In addition, the distance information estimation device 1 performs a process of suppressing an estimation error caused by occlusion on the distance image in FIG. FIG. 21A shows a distance image after the estimation error is suppressed. Here, red and blue are also illustrated in the distance image of FIG. 21A as in FIG. 20A.

  FIG. 21B shows the result of the distance information estimation apparatus 1 outputting the non-textured region using the evaluation value K from the distance image of FIG. In the image of FIG. 21 (b), the low luminance area is an area where an estimation error area caused by texture occurs (for example, the ground behind the earth wall), which almost coincides with the blue area of FIG. 21 (a). Yes. From this, it can be seen that the distance information estimation apparatus 1 has a high accuracy in determining an estimation error caused by a texture.

1,1B, 1C, 1D Distance information estimation device (distance index information estimation device)
10 Error function calculation unit 11 Smooth function calculation unit 12, 12C Message generation / propagation unit (energy propagation unit)
13, 13C Evaluation function calculation unit 14, 14C Distance estimation unit (distance index estimation unit)
14a Noise removal filter 15 Evaluation value calculation unit 16, 16B Evaluation value O calculation unit 17 Occlusion region determination unit (occlusion estimation error determination unit)
18 Evaluation value K calculation unit 19 No texture area determination unit (texture estimation error determination unit)
20 Texture estimation error suppression unit 21 Message propagation restriction determination unit (energy propagation determination unit)

Claims (7)

The method of minimizing the energy related to the probability defined by the error function between stereo images and the smooth function indicating the continuity of the distance index, which is the distance in the depth direction or parallax, A distance index information estimation device that estimates distance index information indicating the distance index for each pixel and calculates an evaluation value indicating reliability of the distance index information,
An error for calculating the error function that is the similarity of the input stereo image so that the stereo image is input and is equal to or less than a preset error function threshold for each preset distance index candidate A function calculator,
A smooth function calculation unit that calculates the smooth function that is a difference absolute value of the distance index candidate between adjacent pixels in a Markov random field so as to be equal to or less than a preset smooth function threshold for each distance index candidate;
Generating an energy set in which the energy includes the error function calculated by the error function calculation unit and the smooth function calculated by the smooth function calculation unit, and a predetermined recurrence formula between adjacent pixels in the energy set. An energy propagation unit that updates the energy for each of the distance index candidates and the pixels by generating and propagating the energy by:
An evaluation function calculation unit for calculating an evaluation function indicating the energy updated in the energy propagation unit;
Among the distance index candidates, a distance index estimation unit that estimates a distance index that minimizes the evaluation function calculated by the evaluation function calculation unit as the distance index information;
An evaluation value calculation unit for obtaining the kurtosis of the energy distribution in the evaluation function as the evaluation value;
A distance index information estimation apparatus comprising: The evaluation value calculator is
A texture evaluation value calculator that calculates the kurtosis of the energy distribution as a texture evaluation value that is the evaluation value for determining an estimation error caused by a texture;
The distance index information estimation device according to claim 1, further comprising: In the texture evaluation value calculation unit, n _f is the number of distance index candidates f _q , b _q (f _q ) is the energy, b _q ￣ is the average value of the energy, and σ (b _q ) is the energy. The distance index information estimation apparatus according to claim 2, wherein the texture evaluation value _Kq is calculated using the following formula (9) defined as a standard deviation of:
It is determined whether or not the texture evaluation value calculated by the texture evaluation value calculation unit is equal to or less than a preset texture threshold. When the texture evaluation value is equal to or less than the texture threshold, an estimation error caused by the texture is A texture estimation error determination unit that determines that the error occurred,
The distance index information estimation device according to claim 2, further comprising: An optical axis intersection of a camera that has captured the stereo image is calculated by camera calibration, and a virtual sphere having a radius set in advance so as to fit a subject in the stereo image is generated around the optical axis intersection. The distance index from the camera to the inner surface of the phantom sphere is calculated, and the calculated distance index is calculated based on the distance of each pixel in the region where the estimation error caused by the texture occurs in the captured image of the central camera. Texture estimation error suppression unit to be assigned as index information,
The distance index information estimation device according to any one of claims 2 to 4, further comprising:   The error function calculation unit obtains a difference absolute value of color information between a processing target pixel in the base image and a corresponding pixel corresponding to the processing target pixel in a reference image that is the other of the stereo images, and the difference absolute value The distance index information estimation apparatus according to claim 1, wherein the error function is calculated by dividing the weight by a preset number of primary colors and weighting the error function. The method of minimizing the energy related to the probability defined by the error function between stereo images and the smooth function indicating the continuity of the distance index, which is the distance in the depth direction or parallax, In order to estimate distance index information indicating the distance index for each pixel, and to calculate an evaluation value indicating the reliability of the distance index information, a computer,
An error for calculating the error function that is the similarity of the input stereo image so that the stereo image is input and is equal to or less than a preset error function threshold for each preset distance index candidate Function calculator,
A smooth function calculation unit that calculates the smooth function that is a difference absolute value of the distance index candidate between adjacent pixels in a Markov random field so that the distance index candidate is equal to or less than a preset smooth function threshold for each distance index candidate;
Generating an energy set in which the energy includes the error function calculated by the error function calculation unit and the smooth function calculated by the smooth function calculation unit, and a predetermined recurrence formula between adjacent pixels in the energy set. By generating and propagating the energy by the energy propagation unit that updates the energy for each distance index candidate and the pixel,
An evaluation function calculation unit for calculating an evaluation function indicating the energy updated in the energy propagation unit;
Among the distance index candidates, a distance index estimation unit that estimates, as the distance index information, a distance index that minimizes the evaluation function calculated by the evaluation function calculation unit;
An evaluation value calculation unit for obtaining the kurtosis of the energy distribution represented by the evaluation function as the evaluation value;
Distance information estimation program to function as.