JP5200042B2 - Disparity estimation apparatus and program thereof - Google Patents

Description

  The present invention relates to a disparity estimation apparatus that determines disparity information indicating the disparity of each pixel of a stereo image by a reliability propagation method and a program thereof.

  For example, when generating a three-dimensional stereo model, a parallax estimation technique for estimating the parallax of each pixel of a stereo image is widely used. As this parallax estimation technique, for example, block matching for obtaining a similarity between stereo images is known. As a method of calculating the similarity, for example, an SSD (Sum of Squared Difference) for obtaining the sum of squares of a difference, an SAD (Sum of Absolute Difference) for obtaining an absolute sum of the differences, and a ZNCC ( Zero-mean Normalized Cross-Correlation) is known (for example, Non-Patent Document 1).

  Also, as the parallax estimation technology, a Markov random field model is assumed for the parallax information, and the energy consisting of the data term between the stereo images and the smooth term is minimized to generate smooth parallax information with less noise A reliability propagation method is also known (for example, Non-Patent Document 2). Here, the data term is a similarity between stereo images, and the smooth term represents disparity disparity.

Digital image processing, CG-Arts Association, 2006, p202-p204 Pedro F. Felzenszwalb, Daniel P. Huttenlocher: Efficient Belief Propagation for Early Vision. CVPR (1) 2004: 261-268

  However, this reliability propagation method has a problem that the parallax estimation accuracy is lowered when a luminance difference occurs between stereo images due to individual differences of photographing cameras or photographing environments. For example, consider a case where the pattern “20” displayed on the coat surface of FIG. 14A is shot as a stereo image by two shooting cameras. Here, it is assumed that the first photographing camera arranged in the downward direction of FIG. 14A has photographed the image shown in FIG. In this case, in the image of FIG. 14B, the luminance histogram having the pattern “20” is as shown in FIG. On the other hand, it is assumed that the second photographing camera arranged in the diagonally downward direction in FIG. 14A photographs the image shown in FIG. In this case, in the image of FIG. 14D, the luminance histogram having the pattern “20” is as shown in FIG. In FIGS. 14B and 14D, a darker color is given as the illumination light becomes stronger. 14C and 14E, the vertical axis represents the luminance value, and the horizontal axis represents the number of pixels having the luminance value.

  Here, the two photographing cameras are in a photographing environment in which the illumination light hits the pattern “20” as seen from each. For this reason, as shown in FIGS. 14C and 14E, the luminance histograms between the stereo images are greatly different. In this case, it is difficult to estimate the parallax from the stereo images in FIGS. 14B and 14C for the pattern “20” in FIG. The problem that the parallax estimation accuracy is reduced due to the luminance difference is difficult to solve even if the level of the photographing camera is sufficiently adjusted.

  Therefore, an object of the present invention is to provide a parallax estimation device that solves the above-described problems and suppresses a decrease in the accuracy of parallax estimation and a program thereof.

  In order to solve the above-described problem, the parallax estimation device according to the first invention of the present application uses the method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating disparity continuity. A disparity estimation apparatus that determines disparity information indicating disparity for each pixel of a stereo image, wherein an upper layer data term generation unit, an upper layer smooth term generation unit, an upper layer energy propagation unit, a disparity candidate determination unit, The apparatus comprises: a lower layer data term generation unit, a lower layer smooth term generation unit, a lower layer energy propagation unit, and a disparity determination unit.

  According to this configuration, the disparity estimation apparatus uses the error function obtained by applying the inverse cosine function to the similarity calculated by the normalized cross-correlation while the stereo image is input by the upper layer data term generation unit. A data term is generated from the input stereo image. Here, by applying the inverse cosine function to the similarity calculated by the normalized cross-correlation, the disparity estimation apparatus can obtain an energy distribution that facilitates finding the minimum value of energy. Then, the disparity estimation apparatus generates a smooth term using a predetermined generation formula by means of higher hierarchy smooth term generation means.

  Further, the disparity estimation apparatus is configured to collect energy for each pixel including the data term generated by the upper layer data term generation unit and the smooth term generated by the upper layer smooth term generation unit by the upper layer energy propagation unit. Generate an energy set, divide the energy set into blocks composed of a plurality of pixels, and repeat the movement of the block to be processed and the propagation of energy generated by a predetermined generation formula for the block to be processed Then, the energy is updated for each block. Then, the disparity estimation apparatus determines, as the disparity of the block, the disparity that minimizes the energy for each block updated by the higher-layer energy propagation unit by the disparity candidate determining unit, and the disparity of the block is included in the block It assigns to a pixel and outputs it as a parallax candidate for each pixel. That is, the parallax estimation apparatus performs energy propagation by regarding one block as one pixel.

  Further, the disparity estimation apparatus receives the stereo image by the lower-layer data term generation unit and uses an error function obtained by applying a cosine inverse function to the similarity calculated by the normalized cross-correlation. Data terms are generated from the stereo image. Here, by applying the inverse cosine function to the similarity calculated by the normalized cross-correlation, the disparity estimation apparatus can obtain an energy distribution that facilitates finding the minimum value of energy. Then, the disparity estimation apparatus generates a smooth term using a predetermined generation formula by the lower-layer smooth term generation unit.

  Further, the disparity estimation apparatus is configured to collect energy for each pixel including the data term generated by the lower layer data term generation unit and the smooth term generated by the lower layer smooth term generation unit by the lower layer energy propagation unit. An energy set is generated, energy generated by a predetermined generation formula is propagated between pixels in the energy set, and the energy is updated for each pixel. Then, the disparity estimation device is a disparity in which the energy for each pixel updated by the lower-layer energy propagation unit is minimized by the disparity determination unit, and the disparity included in the disparity candidate input from the disparity candidate determination unit The parallax information is determined.

  In other words, the parallax estimation apparatus obtains a general parallax (parallax candidate) of the entire image at the block hierarchy (upper hierarchy) and obtains parallax that emphasizes the contour of the subject at the pixel hierarchy (lower hierarchy). Then, the disparity estimation apparatus integrates the disparity estimation results of the upper layer and the lower layer by determining the lower layer disparity included in the upper layer disparity candidates as disparity information.

  Further, the disparity estimation apparatus according to the second invention of the present application uses the error function obtained by further multiplying the first hierarchical coefficient set in advance by the higher-layer data term generating means using the data from the input stereo image. And the lower hierarchy data term generating means generates the data term from the input stereo image using the error function further multiplied by a second weighting factor larger than the first weighting factor. It is characterized by doing.

  Here, for example, in the reliability propagation method, increasing the weighting factor increases noise instead of making the contour of the subject easy to discriminate, and decreasing the weighting factor reduces noise instead of blurring the contour of the subject. Thus, in the reliability propagation method, it is difficult to set one weighting factor to an appropriate value. For this reason, the disparity estimation apparatus obtains a rough disparity (disparity candidate) of the entire image by setting the first weighting factor to a small value in the block layer (upper layer), and in the pixel layer (lower layer). Then, the second weighting factor is set to a large value to obtain parallax that emphasizes the contour of the subject.

  In order to solve the above-described problem, the parallax estimation device according to the third invention of the present application uses a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating disparity continuity. , A disparity estimation apparatus for determining disparity information indicating disparity for each pixel of the stereo image, wherein the upper layer data term generation unit, the upper layer smooth term generation unit, the upper layer energy propagation unit, and the disparity candidate determination unit And a lower layer data term generating unit, a lower layer smooth term generating unit, a lower layer energy propagation unit, and a disparity determining unit.

  According to this configuration, the disparity estimation apparatus receives the stereo image by the upper layer data term generation unit and generates a data term from the input stereo image using an error function. Further, the disparity estimation apparatus generates a smooth term by a predetermined generation formula using a higher-level smooth term generation unit.

  Further, the disparity estimation apparatus is configured to collect energy for each pixel including the data term generated by the upper layer data term generation unit and the smooth term generated by the upper layer smooth term generation unit by the upper layer energy propagation unit. Generate an energy set, divide the energy set into blocks composed of a plurality of pixels, and repeat the movement of the block to be processed and the propagation of energy generated by a predetermined generation formula for the block to be processed Then, the energy is updated for each block. Then, the disparity estimation apparatus determines, as the disparity of the block, the disparity that minimizes the energy for each block updated by the higher-layer energy propagation unit by the disparity candidate determining unit, and the disparity of the block is included in the block It assigns to a pixel and outputs it as a parallax candidate for each pixel. That is, the parallax estimation apparatus performs energy propagation by regarding one block as one pixel.

  In addition, the parallax estimation apparatus generates the data term using the error function from the input stereo image while the stereo image is input by the lower layer data term generating means. Then, the disparity estimation apparatus generates a smooth term using a predetermined generation formula by the lower-layer smooth term generation unit.

  Further, the disparity estimation apparatus is configured to collect energy for each pixel including the data term generated by the lower layer data term generation unit and the smooth term generated by the lower layer smooth term generation unit by the lower layer energy propagation unit. An energy set is generated, energy generated by a predetermined generation formula is propagated between pixels in the energy set, and the energy is updated for each pixel. Then, the disparity estimation device is a disparity in which the energy for each pixel updated by the lower-layer energy propagation unit is minimized by the disparity determination unit, and the disparity included in the disparity candidate input from the disparity candidate determination unit The parallax information is determined.

  In other words, the parallax estimation apparatus obtains a general parallax (parallax candidate) of the entire image at the block hierarchy (upper hierarchy) and obtains parallax that emphasizes the contour of the subject at the pixel hierarchy (lower hierarchy). Then, the disparity estimation apparatus integrates the disparity estimation results of the upper layer and the lower layer by determining the lower layer disparity included in the upper layer disparity candidates as disparity information.

  In order to solve the above-described problem, the parallax estimation device according to the fourth invention of the present application uses a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating disparity continuity. A parallax estimation apparatus for determining parallax information indicating parallax for each pixel of the stereo image, comprising: a data term generating unit, a smooth term generating unit, an energy propagating unit, and a parallax determining unit. And

  According to this configuration, the disparity estimation apparatus uses the error function in which the stereo image is input by the data term generation unit and the cosine inverse function is applied to the similarity calculated by the normalized cross-correlation. Data terms are generated from the input stereo image. Here, by applying the inverse cosine function to the similarity calculated by the normalized cross-correlation, the disparity estimation apparatus can obtain an energy distribution that facilitates finding the minimum value of energy. In addition, the parallax estimation apparatus is a smooth term generation unit that generates a smooth term using a predetermined generation formula.

  In addition, the disparity estimation apparatus generates an energy set in which energy for each pixel including the data term generated by the data term generation unit and the smooth term generated by the smooth term generation unit is collected by the energy propagation unit, Energy generated by a predetermined generation formula is propagated between pixels in the energy set, and the energy is updated for each pixel. Then, the parallax estimation device estimates, as the parallax information, the parallax that minimizes the energy for each pixel updated by the energy propagation unit by the parallax determination unit.

  Moreover, it is preferable that the parallax estimation apparatus according to the first to fourth inventions of the present application be realized by a parallax estimation program that causes a computer to function as each of the means described above.

The present invention has the following excellent effects.
The first invention of the present application can obtain an energy distribution that makes it easy to find the minimum value of energy, and also obtains a parallax with an emphasis on the parallax estimation result of the upper layer obtained for the rough parallax of the entire image and the contour of the subject. Integrate with the lower layer disparity estimation results. As a result, even when a luminance difference occurs between stereo images, the first invention of the present application can reduce parallax estimation errors due to the luminance difference and suppress a decrease in parallax estimation accuracy.

  According to the second aspect of the present invention, in the reliability propagation method, even when a luminance difference occurs between stereo images, an error in parallax estimation due to the luminance difference can be reduced, and a decrease in parallax estimation accuracy can be suppressed. .

  The third invention of the present application integrates the upper layer parallax estimation result obtained for the rough parallax of the entire image and the lower layer parallax estimation result obtained for the parallax with emphasis on the contour of the subject. Thus, according to the third aspect of the present invention, even when a luminance difference occurs between stereo images, an error in parallax estimation caused by the luminance difference can be reduced, and a reduction in parallax estimation accuracy can be suppressed.

  The fourth invention of the present application can obtain an energy distribution that facilitates finding the minimum value of energy. As a result, according to the fourth aspect of the present invention, even when a luminance difference occurs between stereo images, an error in parallax estimation caused by the luminance difference can be reduced, and a decrease in parallax estimation accuracy can be suppressed.

It is explanatory drawing explaining the parallax estimation by the reliability propagation method in this invention. It is a flowchart which shows the procedure of the parallax estimation by the reliability propagation method in this invention. It is a block diagram which shows the structure of the parallax estimation apparatus which concerns on 1st Embodiment of this invention. It is explanatory drawing explaining the upper hierarchy (block) in this invention. It is explanatory drawing explaining the lower hierarchy (pixel) in this invention. It is explanatory drawing explaining the template matching by the parallax estimation apparatus of FIG. It is explanatory drawing explaining the propagation of the message in the upper hierarchy in this invention. It is explanatory drawing explaining the movement of the upper hierarchy in this invention, (a) is a figure of an initial position, (b) is the figure which moved only 1 pixel horizontally from (a), (c) is a figure. It is the figure which moved only 1 pixel in the horizontal direction from (b), (d) is the figure which moved only 1 pixel in the vertical direction from (a). It is explanatory drawing explaining the determination of the parallax by the parallax estimation apparatus of FIG. It is a flowchart which shows operation | movement of the parallax estimation apparatus of FIG. It is a block diagram which shows the structure of the parallax estimation apparatus which concerns on 3rd Embodiment of this invention. It is an image explaining the Example of this invention, (a) is the parallax image by the conventional reliability propagation method, (b) is the parallax image by this invention. It is an image explaining the Example of this invention, The stereo image (L image, R image) at the time of reducing a brightness | luminance gradually, The parallax image by the conventional reliability propagation method, and the parallax image by this invention is there. It is a figure explaining the problem that the estimation precision of parallax falls by the luminance difference at the time of parallax estimation by the conventional reliability propagation method, (a) is a figure which shows a coated surface, (b) is the 1st unit | set. It is the figure which image | photographed the pattern of the coat surface with the imaging camera, (c) is a brightness | luminance histogram of the pattern part of (b), (d) is the figure which image | photographed the pattern of the coat surface with the 2nd imaging camera. Yes, (e) is a luminance histogram of the pattern portion of (d).

(Outline of the present invention: parallax estimation by reliability propagation method)
In each embodiment of the present invention, the reliability propagation method is used as a technique for minimizing energy including a data term and a smooth term. First, an outline of the reliability propagation method in the present invention will be described.
This reliability propagation method (BP method: Belief Propagation method) assumes a Markov Random Field (MRF) model for disparity information, and energy composed of data terms between blocks of a stereo image and smooth terms. By minimizing, smooth parallax information with less noise is generated. Here, in the reliability propagation method, as shown in FIG. 1, energy related to the probability of which disparity is assigned to each pixel called a message is received from neighboring pixels of the pixel to be processed, and the message ( Energy) is updated and propagated to neighboring pixels. In the reliability propagation method, this process is repeated for all the pixels in the image to obtain parallax information for each pixel.

  Here, in FIG. 1, a frame (white square) indicates a pixel, and an arrow m indicates a message (energy) flow between the pixels. In addition, as shown in FIG. 1, an arrangement of pixels vertically and horizontally is called an energy set. Further, a pixel s with the letter “s” indicates a pixel that is a message propagation source, a pixel p with the letter “p” indicates a pixel to be processed, and has a letter “q”. The pixel q indicates an adjacent pixel as a message propagation destination. In FIG. 1, only a part of the reference numerals are shown for simplicity of explanation.

With reference to FIG. 2, the procedure of parallax estimation by the reliability propagation method will be described.
As shown in FIG. 2, in the reliability propagation method, a data term necessary for generating a message is generated (step S1). Here, there are various error functions for generating a data term. When the error function is generated by RSSD (Root Sum of Squared Difference), the following equation (1) is obtained.

Here, f _p is the parallax value of the pixel p, f _q is the parallax value of the pixel q, R (p) is a pixel set in the block centered on the pixel p, I is the pixel value of the block image, and I ′ is the process The pixel value of the target image, λ, represents the weighting of the error function. For example, the parallax value f _p can be obtained from the shift amount of the pixel p when two stereo images are subjected to template matching (the same applies to the parallax value f _q ). Further, i is a horizontal pixel position, and j is a vertical pixel position.

  Subsequently, in the reliability propagation method, a smooth term is generated with a smooth function expressed by the following equation (2) (step S2). This smooth function is a function indicating the smoothness (for example, low noise) of parallax information.

  In the reliability propagation method, a message is generated by a generation expression represented by the following expression (3), and this message is propagated (step S3).

  Here, min is a function for obtaining a minimum value, m is a message, t is the number of iterations, N (p) / q is a set of neighboring pixels that pass a message to the pixel p other than the pixel q, and s is an element of the neighboring pixel Represents a pixel. This formula (3) is a recurrence formula and indicates that the process of updating the message is repeatedly performed based on the received message. In the reliability propagation method, this message is propagated in all pixels.

When iterating up to T times set in advance, the energy related to the parallax of the pixel q can be expressed by the following equation (4). In the reliability propagation method, the parallax value f _q that minimizes the energy b _q represented by the equation (4) is determined as the final parallax of each pixel (step S4).

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

(First embodiment)
[Configuration of parallax estimation apparatus]
With reference to FIG. 3, the structure of the parallax estimation apparatus 1 which concerns on 1st Embodiment of this invention is demonstrated. As illustrated in FIG. 3, the parallax estimation device 1 uses a reliability propagation method that minimizes energy including a data term indicating similarity between stereo images and a smooth term indicating disparity continuity, to generate a parallax of a stereo image. The parallax information indicating is estimated. In the first embodiment, since the disparity estimation apparatus 1 performs disparity estimation by the reliability propagation method in two layers of an upper layer (block layer) and a lower layer (pixel layer), the upper layer processing unit 10 And a lower layer processing unit 20.

  Here, with reference to FIG. 4 and FIG. 5, the hierarchy of a block and the hierarchy of a pixel are each demonstrated. The upper hierarchy UB is a block composed of a plurality of pixels in the energy set of FIG. In the example of FIG. 4, the upper hierarchy UB is a block composed of three pixels vertically and horizontally. Then, the upper layer processing unit 10 regards one upper layer UB as one pixel and performs parallax estimation by the reliability propagation method. In the energy set of FIG. 4, a frame (white square) indicates pixels in the energy set, and a broken line indicates the upper hierarchy UB. Further, in FIG. 4, only a part of the reference numerals are shown for the sake of simplicity.

  Further, as shown in FIG. 5, the lower hierarchy LB is each of the pixels in the energy set of FIG. That is, the lower layer processing unit 20 performs the parallax estimation by propagating the message between the pixels as in the reliability propagation method of FIG. In the energy set of FIG. 5, a frame (white square) indicates pixels in the energy set, and a broken line indicates the lower hierarchy LB. Further, in FIG. 5, only a part of the reference numerals are shown for the sake of simplicity.

Hereinafter, returning to FIG. 3, the description of the configuration of the parallax estimation apparatus 1 will be continued.
The upper layer processing unit 10 performs parallax estimation in the upper layer in order to obtain a rough parallax of the entire image, and includes a data term generation unit (upper layer data term generation unit) 11 and a smooth term generation unit (upper layer). (Hierarchy smooth term generation means) 12, message propagation means (upper hierarchy energy propagation means) 13, and disparity candidate determination means 14.

  The data term generation means 11 receives a stereo image (L image, R image) and generates a data term from the input stereo image using an error function described later. First, the data term generation unit 11 performs template matching on the input stereo image to calculate a shift (parallax) for each pixel. Specifically, as shown in FIG. 6, the data term generation unit 11 extracts a template TP having a predetermined size from the L image (block image). Then, the data term generation unit 11 obtains the similarity at each position while moving the extracted template TP over the entire R image (processing target image). At this time, the data term generation unit 11 performs template matching while moving the template TP in the horizontal direction from the left end of the R image (raster scan).

  Here, the data term generation means 11 calculates the similarity by normalized cross correlation (ZNCC). That is, the data term generation unit 11 calculates the similarity of stereo images using the following equation (5).

  Here, I is an average value of pixel values in the template of the block image, and I ′ is an average value of pixel values in the template of the processing target image.

  Then, the data term generation unit 11 converts the similarity into an angle formed by the vectors of both the block image and the processing target image by using an error function that applies an inverse cosine function to the similarity, and generates a data term. That is, the data term generation unit 11 generates a data term by using an error function expressed by the following equations (6) and (7). At this time, the data term generation means 11 uses the error function for each upper layer expressed by the equation (7) when performing processing in the upper layer. At this time, in order to obtain a rough parallax of the entire image in the upper hierarchy, the data term generation unit 11 sets the weight in the equation (6) smaller than the data term generation unit 21 described later (for example, λ = 5). . Thereafter, the data term generation unit 11 outputs the generated data term to the message propagation unit 13.

  Here, p ′ is an upper layer (block) to be processed, and B (p) is a set of pixels in an upper layer (block) of ε × ε centered on the pixel p.

  The smooth term generating means 12 generates a smooth term by a predetermined generation formula. Here, the smooth term generating means 12 generates a smooth term by using a smooth function for each upper layer expressed by the following equation (8). Then, the smooth term generation unit 12 outputs the generated smooth term to the message propagation unit 13.

  Here, q ′ is the upper layer (block) that is the message propagation destination of the upper layer p ′, and ε is the number of pixels at the upper layer (block) boundary. That is, ε is the number of pixels per one side of the upper and lower layers (blocks).

  Here, it supplements about a Markov random field and a smooth term. This Markov random field is a model indicating that the colors, luminance and parallax of pixels adjacent to each other in the image are similar. The smooth term is a function related to continuity of parallax based on this Markov random field. When parallax is assigned to each pixel, if the parallax of adjacent pixels is the same, the energy is small, and if different, the energy is small. growing. For example, in a Markov random field, when a parallax value of a certain pixel is 1 and a parallax value of an adjacent pixel is 1, the energy is 0. In addition, when the disparity value of a pixel is 1 and the disparity value of a pixel adjacent to the pixel is 2, for example, the energy is 1. Further, when the disparity value of a certain pixel is 1 and the disparity value of the adjacent pixel is 10, for example, the energy is 9. That is, in the reliability propagation method, considering that the parallax with the minimum energy becomes the final parallax information, pixels adjacent to each other are likely to have the same parallax value due to this Markov random field.

  The message propagation means 13 receives a data term from the data term generation means 11 and receives a smooth term from the smooth term generation means 12. And the message propagation means 13 produces | generates the energy set which the energy containing these data terms and smooth terms gathered similarly to FIG. For example, the message propagation unit 13 generates an energy set by arranging energy for each pixel at a position corresponding to the pixel position in the image.

  Then, the message propagation means 13 divides this energy set into an upper layer composed of a predetermined number of pixels (for example, ε pixels vertically and horizontally), and moves the upper layer of the processing target, By repeating the message propagation to the upper layer, the energy is updated for each upper layer. In other words, the message propagation means 13 regards one upper layer as one pixel in the above equation (3), and propagates a message between the upper layers. The movement is to move the upper hierarchy to be processed a predetermined number of times in the horizontal direction and the vertical direction within the energy set, details of which will be described later.

  The parallax candidate determining unit 14 determines the parallax with the lowest energy for each upper layer updated by the message propagation unit 13 as the parallax of the upper layer. That is, the parallax candidate determination unit 14 determines the upper layer parallax by regarding one upper layer as one pixel in the above-described equation (4). Then, the parallax candidate determination unit 14 assigns the determined upper layer parallax to each pixel included in the upper layer, and outputs it to the parallax determination unit 24 described later as a parallax candidate for each pixel.

<Propagation of messages and determination of disparity candidates at higher layers>
Hereinafter, the message propagation unit 13 and the parallax candidate determination unit 14 will be described in detail with reference to FIGS. 7 and 8 (see FIG. 3 as appropriate). In the energy set of FIGS. 7 and 8, a frame (white square) indicates a pixel, an arrow m indicates a message flow, and a broken line indicates an upper hierarchy UB (UB1 to UB5). 7 and 8, only a part of the reference numerals are shown for the sake of simplicity.

  For example, as shown in FIG. 7, the message propagation means 13 delivers a message from the upper layer UB1 to UB3, which is the message propagation source, to the upper layer UB4 to be processed. In the upper layer UB4, the message propagation means 13 updates the energy of the upper layer UB4 with the received message, and delivers the message from the upper layer UB4 to the upper layer UB5 that is the message propagation destination. At this time, the parallax candidate determination means 14 determines the parallax that minimizes the energy for the upper layer UB4 to be processed as the parallax of the upper layer UB4. And the parallax candidate determination means 14 makes the parallax of this upper hierarchy UB4 the parallax of all the pixels contained in the upper hierarchy UB4.

  In addition, the message propagation unit 13 and the parallax candidate determination unit 14 repeat the above-described processing while moving the upper layer UB to be processed. Specifically, the message propagation unit 13 and the parallax candidate determination unit 14 determine the parallax of the processing target upper layer UB at the initial position illustrated in FIG. Then, as shown in FIG. 8B, the message propagation means 13 moves the upper layer UB to be processed from the initial position shown in FIG. 8A by one pixel in the horizontal direction, and the disparity at this position To decide. Further, as shown in FIG. 8C, the message propagation means 13 moves the processing target upper layer UB by one pixel in the horizontal direction from the position shown in FIG. 8B, and the parallax at this position is changed. decide. Thus, the message propagation means 13 performs the process ε times while moving the upper layer UB to be processed from the initial position in the horizontal direction.

  Next, as shown in FIG. 8D, the message propagation means 13 moves the upper layer UB to be processed by one pixel in the vertical direction from the initial position shown in FIG. Determine the parallax. Then, the message propagation means 13 determines the parallax at each position while moving the upper layer UB in the horizontal direction from the position of FIG. As described above, the message propagation unit 13 repeats the determination of the parallax until the processing of the ε row is completed. As a result, each pixel is assigned a disparity (disparity value) as many times as the number of times the process is repeated (that is, ε × ε times, 9 times in the example of FIG. 7). Accordingly, the parallax candidate determination unit 14 outputs each of the allocated parallaxes (parallax values) to the parallax determination unit 24 as parallax candidates for each pixel.

  Note that pixels located near the energy set may have a smaller number of parallaxes (parallax candidates) than pixels located on the center side. For example, since the pixels located at the four corners such as the upper left are included only once in the upper hierarchy UB to be processed, there is one parallax (parallax candidate). Further, for example, the pixel in the second row on the left side of the top row is included only twice in the upper hierarchy UB to be processed, and thus there are two parallaxes (parallax candidates).

Hereinafter, returning to FIG. 3, the description of the configuration of the parallax estimation apparatus 1 will be continued.
The lower hierarchy processing unit 20 performs parallax estimation in a lower hierarchy (lower block) in order to obtain parallax with emphasis on the contour of the subject, and includes a data term generation means (lower hierarchy data term generation means) 21, A smooth term generating unit (lower layer smooth term generating unit) 22, a message propagation unit (lower layer energy propagation unit) 23, and a disparity determining unit 24 are provided.

  The data term generation unit 21 receives a stereo image (L image, R image) and generates a data term from the input stereo image in the same manner as the data term generation unit 11. That is, the data term generation means 21 obtains the stereo image similarity by the above-described equation (5), and generates a data term by using the error function represented by the above-described equation (6). At this time, since the parallax is obtained with an emphasis on the contour of the subject in the lower hierarchy, the data term generation unit 21 increases the weighting in the equation (6) more than the data term generation unit 11 (for example, λ = 15). ). Then, the data term generation means 21 outputs the generated data term to the message propagation means 23 described later.

  The smooth term generating means 22 generates a smooth term using a predetermined generation formula. That is, the smooth term generating means 22 generates a smooth term using the smooth function expressed by the above-described equation (2). Then, the smooth term generation unit 22 outputs the generated smooth term to the message propagation unit 23.

  The message propagation means 23 receives a data term from the data term generation means 21 and receives a smooth term from the smooth term generation means 22. Then, the message propagation means 23 generates an energy set in which energy including these data terms and smooth terms is gathered. In this case, the message propagation means 23 generates a message using the above-described equation (3). Then, the message propagation means 23 propagates the message between the lower layers in the energy set, and updates the energy in the lower layer.

  The parallax determination unit 24 determines the parallax with the lowest energy in the lower layer updated by the message propagation unit 23 as the lower layer parallax. That is, the parallax determination unit 24 determines the parallax that minimizes the above-described formula (4) as the parallax of the lower layer. Then, the parallax determination unit 24 estimates the lower layer parallax included in the parallax candidates (upper layer parallax) input from the parallax candidate determination unit 14 as parallax information for each pixel.

Hereinafter, estimation of parallax information will be described with reference to FIG.
In FIG. 9, the left side of the drawing is the subject Ob, and the outline Eg of the subject Ob is shown by a bold line. Accordingly, the right side of FIG. 9 is a background portion other than the subject Ob. Further, in FIG. 9, each of the pixels is illustrated by a broken-line frame (rectangle).
Further, the description will be made assuming that the number of pixels per side of the upper hierarchy is 3 pixels (ε = 3). In other words, there are 3 × 3 = 9 parallax candidates for each pixel.

  First, consider the pixel (lower layer) α1 corresponding to the subject Ob away from the contour Eg. Here, since this pixel α1 has almost the same parallax as its neighboring pixels, the types of parallax candidates output by the parallax candidate determination means 14 are reduced (or one type). Here, for example, the parallax candidate determining means 14 has nine parallax candidates “4, 4, 3, 4, 4, 4, 4, 3, 4”, that is, 2 “3,4”. Assume that types of parallax candidates are output. At this time, if the parallax value of the pixel α1 is “4”, the parallax determination unit 24 sets the parallax information of the pixel α1 to “4” included in the two types of parallax candidates “3, 4”.

  Next, consider the pixel (lower layer) α2 corresponding to the subject Ob near the contour Eg. In this pixel α2, since the parallax of the subject Ob and the parallax of the background portion are parallax candidates, the types of parallax candidates output by the parallax candidate determination unit 14 increase. Here, for example, the parallax candidate determination means 14 determines nine parallax candidates “1, 2, 2, 1, 3, 4, 4, 3, 4” of the pixel α2, that is, “1, 2, 3, Assume that four types of parallax candidates of “4” are output. At this time, if the parallax value of the pixel α2 (lower hierarchy) is “4”, the parallax determining unit 24 includes the parallax information of the pixel α2 included in the four types of parallax candidates “1, 2, 3, 4”. 4 ”.

  Further, consider the pixel (lower layer) α3 corresponding to the background portion near the contour Eg. Here, for example, it is assumed that the parallax candidate determination unit 14 outputs nine parallax candidates “1, 2, 8, 1, 3, 1, 1, 2, 1” as the parallax candidates of the pixel α3. Usually, since the subject Ob is located on the near side of the background portion, the parallax value of the pixel α3 corresponding to the background portion is smaller than the pixels α1 and α2 corresponding to the subject Ob. However, it is assumed that noise “8” is mixed in the parallax candidate due to the luminance difference of the stereo image at the pixel α3. Even in this case, if the parallax value of the pixel α3 (lower layer) is “2”, the parallax determination unit 24 sets the parallax information of the pixel α3 to “2” included in the parallax candidate, and converts the noise “8” to the parallax. There is no information. In this way, the parallax determination unit 24 integrates the upper layer parallax estimation result and the lower layer parallax estimation result for each pixel, and suppresses a decrease in the accuracy of the parallax estimation due to the luminance difference.

[Operation of parallax estimation apparatus]
Hereinafter, the operation of the parallax estimation apparatus 1 will be described with reference to FIG. 10 (see FIG. 3 as appropriate).
The disparity estimation apparatus 1 generates a data term for the upper layer by the data term generation unit 11 (step S11). That is, the data term generation unit 11 performs template matching on the input stereo image, obtains the similarity of the stereo image by the above-described equation (5), and expresses by the above-described equations (6) and (7). A data term is generated by the error function.

  Moreover, the parallax estimation apparatus 1 generates a smooth term for the upper layer by the smooth term generation unit 12 (step S12). That is, the smooth term generation means 12 generates a smooth term using the smooth function expressed by the above-described equation (8).

  Also, the parallax estimation apparatus 1 uses the message propagation unit 13 to propagate the message for the upper layer (step S13). That is, the message propagation unit 13 generates an energy set, and repeats the movement of the upper layer in the energy set and the propagation of the message in the upper layer to update the energy in the upper layer.

  In addition, the parallax estimation apparatus 1 uses the parallax candidate determination unit 14 to determine parallax candidates (higher-level parallax) (step S14). That is, the parallax candidate determining unit 14 determines the parallax that minimizes the energy in the upper hierarchy as the parallax of the upper hierarchy, assigns this to the pixels included in the upper hierarchy, and sets the parallax candidate for each pixel.

  Moreover, the parallax estimation apparatus 1 generates a data term for the lower hierarchy by the data term generation unit 21 (step S15). That is, the data term generation unit 21 performs template matching on the input stereo image, obtains the similarity of the stereo image by the above equation (5), and calculates the error function represented by the above equation (6). To generate a data term.

  In addition, the parallax estimation apparatus 1 generates a smooth term for the lower hierarchy by the smooth term generation unit 22 (step S16). That is, the smooth term generating means 22 generates a smooth term using the smooth function expressed by the above-described equation (2).

  Further, the parallax estimation apparatus 1 performs message propagation for the lower layer by the message propagation unit 23 (step S17). That is, the message propagation unit 23 generates an energy set, propagates a message between lower layers (pixels) in the energy set, and updates energy in the lower layer.

  Moreover, the parallax estimation apparatus 1 determines parallax by the parallax determination means 24 (step S18). That is, the parallax determination unit 24 determines the parallax that minimizes the energy in the lower hierarchy as the parallax in the lower hierarchy. Then, the parallax determining unit 24 determines the lower-layer parallax included in the parallax candidates (upper-layer parallax) as disparity information for each pixel.

  As described above, the disparity estimation apparatus 1 according to the first embodiment of the present invention applies the inverse cosine function to the similarity calculated by the normalized cross-correlation (see Expression (5) and Expression (6)). An energy distribution that makes it easy to find the minimum value of energy can be obtained. In addition, the parallax estimation apparatus 1 uses the upper layer processing unit 10 to obtain a rough parallax of the entire image by setting λ (first weighting coefficient) in Expression (6) to a small value. In addition, the parallax estimation apparatus 1 uses the lower layer processing unit 20 to obtain parallax that emphasizes the contour of the subject with λ (second weighting coefficient) in Expression (6) as a large value. Then, the disparity estimation apparatus 1 integrates the disparity estimation results of the upper layer and the lower layer by estimating the lower layer disparity included in the upper layer disparity candidates as disparity information. Thus, even when a luminance difference occurs between stereo images, the parallax estimation device 1 can reduce an error in parallax estimation caused by the luminance difference, and can suppress a decrease in parallax estimation accuracy.

  In the first embodiment, the data term and the smooth term are described and described. However, in the reliability propagation method, each of the data term and the smooth term may be referred to as energy. Even in this case, it is needless to say that the parallax estimation apparatus 1 according to the first embodiment of the present invention can be applied.

  Although the first embodiment has been described as including the upper layer processing unit 10 and the lower layer processing unit 20, the present invention is not limited to this. For example, in the present invention, one means having the functions of the smooth term generating means 12 and 22 is generated by generating the data terms of the upper and lower layers by one means having the functions of the data term generating means 11 and 21 of FIG. The smooth terms of the upper hierarchy and the lower hierarchy may be generated by. In the present invention, one means having the functions of the message propagation means 13 and 23 propagates messages in the upper and lower layers, and one means having the functions of the disparity candidate determination means 14 and the disparity determination means 24. Thus, the parallax may be finally determined for each pixel.

(Second Embodiment)
[Configuration of parallax estimation apparatus]
Hereinafter, returning to FIG. 3, differences from the first embodiment will be described for the disparity estimation apparatus 1 </ b> A according to the second embodiment of the present invention. In the second embodiment, the error function used by the data term generating means 11 and the data term generating means 21 is different from that of the first embodiment.

The data term generation unit 11 generates a data term using an error function other than the above formulas (6) and (7), for example, the formula (1).
Similar to the data term generation unit 11, the data term generation unit 21 generates a data term using an error function other than the above formulas (6) and (7), for example, the formula (1).
In addition, since each means other than these is the same as that of 1st Embodiment, description is abbreviate | omitted.

[Operation of parallax estimation apparatus]
Hereinafter, with respect to the operation of the parallax estimation apparatus 1A, differences from the first embodiment will be described with reference to FIG. 10 (see FIG. 3 as appropriate).

In the parallax estimation device 1A, the data term generation unit 11 generates a data term for the upper layer by using, for example, the error function expressed by the above-described equation (1) (step S11).
In addition, the parallax estimation device 1A uses the data term generation unit 21 to generate data terms for the lower hierarchy using, for example, the error function expressed by the above-described equation (1) (step S15).
In addition, since each step other than these is the same as that of 1st Embodiment, description is abbreviate | omitted.

  As described above, the disparity estimation device 1A according to the second embodiment of the present invention estimates the disparity of the lower layer included in the upper layer disparity candidates as the disparity information, so that the disparity estimation between the upper layer and the lower layer is performed. Integrate the results. As a result, the parallax estimation apparatus 1A can reduce the parallax estimation error due to the luminance difference even when a luminance difference occurs between the stereo images, and reduce the parallax estimation accuracy, as in the first embodiment. Can be suppressed.

  In the second embodiment, the error function is not limited to Expression (1). For example, the parallax estimation apparatus 1 according to the second embodiment of the present invention may use an error function by SAD.

(Third embodiment)
[Configuration of parallax estimation apparatus]
Hereinafter, with reference to FIG. 11, differences from the first embodiment will be described regarding the disparity estimation device 1 </ b> B according to the third embodiment of the present invention. The third embodiment is different from the first embodiment in that a parallax candidate (upper-layer parallax) is not obtained. For this reason, the parallax estimation device 1B includes a data term generation unit 31, a smooth term generation unit 32, a message propagation unit (energy propagation unit) 33, and a parallax determination unit 34, as shown in FIG.

  The data term generation unit 31 receives a stereo image (L image, R image) and generates a data term from the input stereo image in the same manner as the data term generation unit 21. That is, the data term generation means 21 obtains the stereo image similarity by the above-described equation (5), and generates a data term by using the error function represented by the above-described equation (6). Then, the data term generation unit 31 outputs the generated data term to the message propagation unit 33 described later.

  The smooth term generating means 32 generates a smooth term using a predetermined generation formula. That is, the smooth term generating means 32 generates a smooth term using the smooth function expressed by the above-described equation (2). Then, the smooth term generation unit 32 outputs the generated smooth term to the message propagation unit 33.

  The message propagation means 33 receives a data term from the data term generation means 31 and receives a smooth term from the smooth term generation means 32. The message propagation means 33 generates an energy set in which the energy including these data terms and smooth terms is gathered. In this case, the message propagation means 33 generates a message using the above-described equation (3). Then, the message propagation unit 33 propagates the message between the pixels in the energy set, and updates the energy for each pixel.

  The parallax determination unit 34 estimates the parallax that minimizes the energy for each pixel updated by the message propagation unit 33 as the parallax information for each pixel. That is, the parallax determination unit 34 estimates the parallax that minimizes the above-described equation (4) as the parallax information.

[Operation of parallax estimation apparatus]
Hereinafter, returning to FIG. 2, the operation of the parallax estimation apparatus 1B will be described (see FIG. 11 as appropriate).
First, the parallax estimation apparatus 1B generates a data term by the data term generation unit 31 (step S1). That is, the data term generating unit 31 performs template matching on the input stereo image, obtains the similarity of the stereo image by the above equation (5), and obtains the error function represented by the above equation (6). To generate a data term.

  In addition, the parallax estimation apparatus 1B generates a smooth term using a predetermined generation formula by the smooth term generating unit 32 (step S2). That is, the smooth term generating means 32 generates a smooth term using the smooth function expressed by the above-described equation (2).

  In addition, the parallax estimation apparatus 1B performs message propagation between pixels by the message propagation unit 33 (step S3). That is, the message propagation means 33 generates an energy set and propagates the message between pixels in the energy set to update the energy.

  Moreover, the parallax estimation apparatus 1B determines parallax by the parallax determination means 34 (step S4). That is, the parallax determination unit 34 estimates the parallax with the minimum energy as the parallax information.

  As described above, the disparity estimation device 1B according to the third embodiment of the present invention applies the inverse cosine function to the similarity calculated by the normalized cross correlation (see Expression (5) and Expression (6)). An energy distribution that makes it easy to find the minimum value of energy can be obtained. Thereby, the parallax estimation device 1B can reduce the parallax estimation error caused by the luminance difference even when a luminance difference occurs between the stereo images, as in the first and second embodiments. The decrease can be suppressed.

  In each embodiment, although the reliability propagation method has been described as an example of a method for minimizing energy including a data term and a smooth term, the present invention is not limited to this. In the present invention, for example, a Viterbi algorithm can be used as this method.

  In each embodiment, the parallax estimation apparatus according to the present invention has been described as an independent apparatus. However, in the present invention, the present invention can also be realized by a parallax estimation program that causes a general computer to function as each unit described above. . The parallax estimation program may be distributed via a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

The effects of the present invention will be described below as examples.
The parallax estimation of the stereo image was performed using the parallax estimation apparatus 1 of FIG. As a comparison object, disparity estimation was performed on the same stereo image by a conventional reliability propagation method using normalized cross-correlation as an error function. Then, from the parallax information output from each, a parallax image is generated in which the luminance is higher as the parallax of each pixel is larger and the luminance is lower as the parallax is smaller.

  As shown in FIG. 12A, in the conventional reliability propagation method, the parallax image is dotted with portions where the contour of the subject is unclear. On the other hand, as shown in FIG. 12B, in the present invention, the contour of the subject is clear in the parallax image. From this result, it can be confirmed that the present invention is more effective than the prior art.

  In addition, for each of the conventional reliability propagation method and the present invention, an experiment was conducted to determine how much the parallax estimation accuracy is reduced when a luminance difference occurs between stereo images. Specifically, parallax images were generated by performing parallax estimation using the conventional reliability propagation method and each of the present invention while gradually decreasing the luminance of one of the stereo images (R image). FIG. 13 shows a stereo image (L image, R image) when the luminance is gradually reduced, a parallax image according to a conventional reliability propagation method (conventional technology), and a parallax image according to the present invention (example). showed that.

  In FIG. 13, in the conventional technique, M = 1, N = 1, λ = 0.07, and T = 10. Here, M is the number of pixels in the vertical direction of the upper layer, and N is the number of pixels in the horizontal direction of the upper layer. In the upper hierarchy of the example, M = 3, N = 3, λ = 5, ε = 3, and T = 10. Furthermore, in the lower hierarchy of the example, M = 3, N = 3, λ = 15, and T = 10.

  When the luminance is reduced by 1% by the conventional reliability propagation method, the outline of the subject becomes conspicuous in the parallax image. When the luminance is reduced by 3% by the conventional reliability propagation method, the outline of the subject in the parallax image is greatly broken. Furthermore, when the luminance is reduced by 10% by the conventional reliability propagation method, it becomes impossible to determine the contour of the subject in the parallax image. From this, it can be seen that in the conventional reliability propagation method, the estimation accuracy of parallax is greatly reduced as the luminance difference between stereo images is increased.

  On the other hand, in the present invention, even if the luminance is decreased from 1% to 10%, no major collapse is observed in the contour of the subject. From this, it can be considered that the present invention can sufficiently suppress a decrease in the estimation accuracy of parallax even if the luminance difference between stereo images becomes large. Therefore, the present invention can also be applied to a multi-viewpoint camera image shot in a place where the shooting environment is not prepared such as outdoors. More specifically, the present invention can be applied to generation of a three-dimensional solid model of a scene, for example, by shooting a scene with sports with a multi-viewpoint camera and performing parallax estimation on the captured video.

DESCRIPTION OF SYMBOLS 1 Parallax estimation apparatus 1A Parallax estimation apparatus 1B Parallax estimation apparatus 10 Upper hierarchy process part 11 Data term production | generation means (upper hierarchy data term production | generation means)
12 Smooth term generation means (upper layer smooth term generation means)
13 Message propagation means (upper layer energy propagation means)
14 Parallax candidate determining means 20 Lower hierarchy processing unit 21 Data term generating means (lower hierarchy data term generating means)
22 Smooth term generation means (lower hierarchy smooth term generation means)
23 Message propagation means (lower-layer energy propagation means)
24 Parallax determining means 31 Data term generating means 32 Smooth term generating means 33 Message propagation means (energy propagation means)
34 Parallax determining means

Claims (8)

A parallax estimation device that determines parallax information indicating parallax for each pixel of a stereo image by a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating disparity continuity. And
Higher layer data term generating means for generating a data term from the input stereo image using an error function obtained by applying a cosine inverse function to the similarity calculated by normalized cross correlation while the stereo image is input When,
Upper hierarchy smooth term generating means for generating a smooth term by a predetermined generation formula;
An energy set in which energy for each pixel including the data term generated by the upper hierarchy data term generation means and the smooth term generated by the upper hierarchy smooth term generation means is generated, and the energy set is divided into a plurality of pixels. Higher layer energy propagation in which energy is updated for each block by dividing the block to be configured and repeating the movement of the block to be processed and the propagation of energy generated by a predetermined generation formula for the block to be processed Means,
The parallax that minimizes the energy of each block updated by the upper layer energy propagation means is determined as the parallax of the block, and the parallax of the block is assigned to the pixels included in the block and output as parallax candidates for the pixels. Parallax candidate determination means;
Low-level data term generation for generating a data term from the input stereo image using an error function obtained by applying a cosine inverse function to the similarity calculated by the normalized cross-correlation while the stereo image is input Means,
Lower-level smooth term generating means for generating a smooth term by a predetermined generation formula;
An energy set in which the energy of each pixel is collected including the data term generated by the lower layer data term generation unit and the smooth term generated by the lower layer smooth term generation unit is generated, and between the pixels in the energy set A lower layer energy propagation means for propagating energy generated by a predetermined generation formula and updating the energy for each pixel;
Disparity determining means for determining, as the disparity information, a disparity in which the energy for each pixel updated by the lower-layer energy propagation means is minimized, and included in the disparity candidate input from the disparity candidate determining means;
A parallax estimation apparatus comprising: The upper hierarchical data term generating means generates the data term from the input stereo image using the error function further multiplied by a preset first weighting factor,
The lower hierarchical data term generating means generates the data term from the input stereo image using the error function further multiplied by a second weighting factor larger than the first weighting factor. The parallax estimation device according to claim 1. A parallax estimation device that determines parallax information indicating parallax for each pixel of a stereo image by a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating disparity continuity. And
The stereo image is input, and an upper layer data term generating unit that generates a data term using an error function from the input stereo image;
Upper hierarchy smooth term generating means for generating a smooth term by a predetermined generation formula;
An energy set in which energy for each pixel including the data term generated by the upper hierarchy data term generation means and the smooth term generated by the upper hierarchy smooth term generation means is generated, and the energy set is divided into a plurality of pixels. Higher layer energy propagation in which energy is updated for each block by dividing the block to be configured and repeating the movement of the block to be processed and the propagation of energy generated by a predetermined generation formula for the block to be processed Means,
The parallax that minimizes the energy of each block updated by the upper layer energy propagation means is determined as the parallax of the block, and the parallax of the block is assigned to the pixels included in the block and output as parallax candidates for the pixels. Parallax candidate determination means;
The stereo image is input, and a lower-layer data term generating unit that generates a data term using an error function from the input stereo image;
Lower-level smooth term generating means for generating a smooth term by a predetermined generation formula;
An energy set in which the energy of each pixel is collected including the data term generated by the lower layer data term generation unit and the smooth term generated by the lower layer smooth term generation unit is generated, and between the pixels in the energy set A lower layer energy propagation means for propagating energy generated by a predetermined generation formula and updating the energy for each pixel;
Disparity determining means for determining, as the disparity information, a disparity in which the energy for each pixel updated by the lower-layer energy propagation means is minimized, and included in the disparity candidate input from the disparity candidate determining means;
A parallax estimation apparatus comprising: A parallax estimation device that determines parallax information indicating parallax for each pixel of a stereo image by a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating disparity continuity. And
Data term generating means for generating a data term from the input stereo image using an error function obtained by applying a cosine inverse function to the similarity calculated by the normalized cross correlation while the stereo image is input ,
Smooth term generating means for generating a smooth term by a predetermined generation formula;
An energy set in which the energy for each pixel is collected including the data term generated by the data term generation unit and the smooth term generated by the smooth term generation unit is generated, and predetermined generation is performed between pixels in the energy set. Energy propagation means for propagating energy generated by the equation and updating the energy for each pixel;
Parallax determining means for determining, as the parallax information, a parallax that minimizes the energy for each pixel updated by the energy propagation means;
A parallax estimation apparatus comprising: In order to determine parallax information indicating parallax for each pixel of the stereo image by a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating continuity of parallax, ,
Higher layer data term generating means for generating a data term from the input stereo image using an error function obtained by applying a cosine inverse function to the similarity calculated by normalized cross correlation while the stereo image is input ,
Upper hierarchy smooth term generating means for generating a smooth term by a predetermined generation formula,
An energy set in which energy for each pixel including the data term generated by the upper hierarchy data term generation means and the smooth term generated by the upper hierarchy smooth term generation means is generated, and the energy set is divided into a plurality of pixels. Higher layer energy propagation in which energy is updated for each block by dividing the block to be configured and repeating the movement of the block to be processed and the propagation of energy generated by a predetermined generation formula for the block to be processed means,
The parallax that minimizes the energy of each block updated by the upper layer energy propagation means is determined as the parallax of the block, and the parallax of the block is assigned to the pixels included in the block and output as parallax candidates for the pixels. Parallax candidate determination means,
Low-level data term generation for generating a data term from the input stereo image using an error function obtained by applying a cosine inverse function to the similarity calculated by the normalized cross-correlation while the stereo image is input means,
Lower hierarchy smooth term generating means for generating a smooth term by a predetermined generation formula,
An energy set in which the energy of each pixel is collected including the data term generated by the lower layer data term generation unit and the smooth term generated by the lower layer smooth term generation unit is generated, and between the pixels in the energy set A lower-layer energy propagation means for propagating energy generated by a predetermined generation formula and updating energy for each pixel;
Disparity determining means for determining, as the disparity information, a disparity in which the energy for each pixel updated by the lower-layer energy propagation means is minimized, and included in the disparity candidate input from the disparity candidate determining means;
As a parallax estimation program. The upper hierarchical data term generating means generates the data term from the input stereo image using the error function further multiplied by a preset first weighting factor,
The lower hierarchical data term generating means generates the data term from the input stereo image using the error function further multiplied by a second weighting factor larger than the first weighting factor. The parallax estimation program according to claim 5. In order to determine parallax information indicating parallax for each pixel of the stereo image by a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating continuity of parallax, ,
Higher layer data term generating means for generating a data term using an error function from the inputted stereo image while the stereo image is input;
Upper hierarchy smooth term generating means for generating a smooth term by a predetermined generation formula,
An energy set in which energy for each pixel including the data term generated by the upper hierarchy data term generation means and the smooth term generated by the upper hierarchy smooth term generation means is generated, and the energy set is divided into a plurality of pixels. Higher layer energy propagation in which energy is updated for each block by dividing the block to be configured and repeating the movement of the block to be processed and the propagation of energy generated by a predetermined generation formula for the block to be processed means,
The parallax that minimizes the energy of each block updated by the upper layer energy propagation means is determined as the parallax of the block, and the parallax of the block is assigned to the pixels included in the block and output as parallax candidates for the pixels. Parallax candidate determination means,
The lower hierarchy data term generating means for generating a data term from the input stereo image using an error function while the stereo image is input;
Lower hierarchy smooth term generating means for generating a smooth term by a predetermined generation formula,
An energy set in which the energy of each pixel is collected including the data term generated by the lower layer data term generation unit and the smooth term generated by the lower layer smooth term generation unit is generated, and between the pixels in the energy set A lower layer energy propagation means for propagating energy generated by a predetermined generation formula and updating energy for each pixel;
Disparity determining means for determining, as the disparity information, a disparity in which the energy for each pixel updated by the lower-layer energy propagation means is minimized, and included in the disparity candidate input from the disparity candidate determining means;
As a parallax estimation program. In order to determine parallax information indicating parallax for each pixel of the stereo image by a method of minimizing energy including a data term indicating similarity between stereo images and a smooth term indicating continuity of parallax, ,
A data term generating means for generating a data term from the input stereo image using an error function obtained by applying a cosine inverse function to the similarity calculated by the normalized cross correlation while the stereo image is input;
Smooth term generating means for generating a smooth term by a predetermined generation formula;
An energy set in which the energy for each pixel is collected including the data term generated by the data term generation unit and the smooth term generated by the smooth term generation unit is generated, and predetermined generation is performed between pixels in the energy set. Energy propagation means for propagating the energy generated by the equation and updating the energy for each pixel;
Parallax determination means for determining, as the parallax information, a parallax that minimizes the energy for each pixel updated by the energy propagation means;
As a parallax estimation program.