JP3465988B2 - Method and apparatus for estimating movement and depth - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion and depth estimation method and apparatus for estimating motion and depth by finding pixel correspondence between images.

[0002]

2. Description of the Related Art It is desired to reduce a huge amount of information in the process of transmitting and storing moving images and multi-view stereo images. Also, regarding image pickup and display, if an intermediate image can be combined from a twin-lens stereo image to display a multi-lens stereo image, the amount of information during image pickup, transmission, and storage can be reduced. For this reason, many attempts have been made to eliminate the redundancy of images by estimating the movement and depth by finding the correspondence between pixels between images.
Gradient methods and block matching methods are roughly classified as methods for obtaining correspondence, but these methods have advantages and disadvantages, respectively. That is, the gradient method can accurately estimate a small amount of motion or parallax, but the accuracy is reduced for a large estimated amount. Further, since the gradient is used, it is easily affected by noise. Furthermore, in the gradient method, since the estimated value obtained from the gradient is generally corrected by iterative calculation,
It is disadvantageous in terms of real-time processing.

On the other hand, the block matching method can perform estimation with a constant matching accuracy regardless of the size of the estimation amount, and
Robust against noise. However, there is a problem that an appropriate block size differs depending on the presence or absence of a region having discontinuous brightness gradient, movement, and depth, and the appropriate block size depends on the distribution of estimated values.

Kanade et al. Attempt to solve the above problem by iterative calculation of updating the block size, position and parallax using an evaluation scale considering luminance gradient, noise and parallax distribution (see "Astereo"). Matching algorithm with unadaptive window "("
A Stereo Matching Algorithm with an Adoptive Windo
w: Theory and Experiment ", Tekeo Kanade and Masatos
hi Okutomi, 1990)).

[0005]

However, the conventional method as described above has a problem that a huge amount of calculation is required.

In view of the above point, the present invention evaluates the reliability of the estimation result by block matching based on the luminance gradient, the image noise, the minimum value of the residual sum of squares (SSD) between blocks, and the block size. Therefore, it is an object of the present invention to provide a motion and depth estimation method and apparatus that accurately estimate depth and motion by selecting from estimation results based on a plurality of block sizes and without performing repeated calculations.

[0007]

SUMMARY OF THE INVENTION According to the present invention, the reliability of the estimation result by block matching is determined by the brightness gradient, SSD.
, The block size, and preferably the noise of the image, and then select from the estimation results based on multiple block sizes, and estimate the depth and motion accurately without performing repeated calculations. This is a depth estimation method.

Further, a reference image frame memory for storing a reference image, a reference image frame memory for storing a reference image, a block for performing a block correlation calculation and a reliability evaluation value of an estimated value with a plurality of block sizes. Correlation calculation circuit and reliability of estimation result by block matching
The motion and depth estimation apparatus includes an estimation value selection circuit that evaluates based on the luminance gradient, image noise, the minimum value of SSD, and the block size and selects from estimation results based on a plurality of block sizes.

[0009]

According to the present invention, by changing the block size to be adaptively selected according to the size of the brightness gradient and the distribution state of the estimated value, the single block size can be used without repeating the calculation. The amount of calculation is several times as large as the block correlation calculation, and the motion and depth are accurately estimated with a constant calculation amount.

[0010]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of a motion and depth estimation device according to a first embodiment of the present invention.

In FIG. 1, 1 is a standard image frame memory for storing a standard image, 2 is a reference image frame memory for storing a reference image, 3A, 3B, 3
C and 3D have different block sizes (10 × 1
0, 20 × 20, 40 × 40, 80 × 80), a block correlation calculation circuit for performing block correlation calculation, and 4 is an estimated value selection circuit for selecting from estimation results based on a plurality of block sizes.

The operation of the above configuration will be described below. The reference image frame memory 1 stores one frame of a reference image for setting a reference block during block correlation calculation. The reference image frame memory 2 stores one frame of the reference image for which the correlation calculation with the standard block is performed during the block correlation calculation. The block correlation calculation circuits 3A, 3B, 3C, and 3D perform block correlation calculation and calculation of reliability evaluation values of estimated values with different block sizes. Estimated value selection circuit 4
Is a correlation calculation circuit that sequentially reads the reliability evaluation value and the estimated value at the time of block correlation calculation by the block correlation calculation circuits 3A, 3B, 3C, and 3D, weights them according to the block size, and minimizes the evaluation value after weighting. Choose an estimate for.

The block correlation calculation will be described with reference to FIG. For the standard block centered at (x0, y0) in the standard image, the residual sum of squares (SSD) defined by (Equation 1) is calculated in the search region of the reference image, and within the search region S
Let (ui, vj) that minimizes SD be the estimated value (u, v) at (x0, y0).

[0014]

[Equation 1]

Here, f1 (x, y) is (x, y of the reference image.
y), the luminance value, f2 (x, y) is (x, y) of the reference image
In the figure, the luminance value and w indicate the block areas for which the correlation calculation is performed.

FIG. 3 shows an example of the configuration of the block correlation calculation circuit 3. In FIG. 3, 5 is a read address generation circuit, 6 is a read control circuit, 7 is a mean square luminance gradient calculation circuit, 8 is a residual sum of squares calculation circuit, and 9 is an evaluation value calculation circuit. In the read address generation circuit 5, the reference block in the reference image is different from the reference block in the reference image as shown in FIG.
An address is generated so that the search area is sequentially scanned as shown in (1), and the difference vector (u _i , vj) between the coordinates of the standard block and the reference block is output. Read control circuit 6
Outputs a control signal to the standard image frame memory 1 and the reference image frame memory 2 based on the address generated by the read address generation circuit 5, and reads the image data of the standard block and the reference block. The mean square luminance gradient calculation circuit 7 calculates (Equation 2) for the mean square luminance gradient in the horizontal and vertical directions in the block from the image data read by the read control circuit 6.

[0017]

[Equation 2]

Here, N is the number of pixels in the block area.

The residual sum of squares operation circuit 8 is a difference vector of the coordinate data between the standard block and the reference block read by the read control circuit 6 and the coordinate value between the standard block and the reference block output from the read address generation circuit 5. (ui, v
j), the residual sum of squares in the block is calculated, and the smallest residual sum of squares in the search area is given as (ui, vj).
And outputs the minimum residual sum of squares SSDmin. The evaluation value calculation circuit 9 calculates the reliability evaluation value J of the estimated value in consideration of the brightness gradient, the noise of the image, the minimum value of the residual sum of squares, and the block size based on (Equation 3).

[0020]

[Equation 3]

Here, SSDmin is the SSD in the search area
2 σ _n ² is a constant determined by the noise of the image. Σ _n in (Equation 3) is the equation of the S / N ratio of the image,
From (Equation 4), it is determined from the S / N ratio and the maximum level V of the image signal.

[0022]

[Equation 4]

If the noise is random noise, S
Where SDmin / N is correct,
Since it is 2σ _n ² , the maximum frequency of SSDmin / N in the entire image may be equal to 2σ _n ² . Note that the SSDmin / N value at the correct correspondence is smaller for the larger block size around 2σ _n ² , so the SSDmin / N entire image at block matching with a larger block size up to those who were determined σ _n from the frequency is more stable selection in
Perform an evaluation of the time-option.

The evaluation value J of (Equation 3) is obtained by normalizing the degree of coincidence at the time of the correlation calculation by the brightness gradient, that is, the feature amount of the brightness distribution in the block. When the correlation between the standard block and the reference block is high, J Has a small value, and conversely, J has a large value when the correlation is low.

FIG. 4 shows an example of a block diagram of the residual sum of squares calculation circuit 8. In FIG. 4, 12 is a residual square calculation circuit, 13 is a cumulative addition circuit, and 14a is a minimum value selection circuit. The residual square calculation circuit 12 sequentially calculates the residuals of the image data of the standard block and the reference block. Cumulative addition circuit 1
3 is cleared to 0 each time the calculation of the residual sum of squares between the standard block and the reference block is performed, and the output of the residual square calculation circuit 12 is cumulatively added to each pixel in the standard block and the reference block. The residual sum of squares SSD shown in is calculated.
The minimum value selection circuit 14a is reset every time a reference block is set in the reference image. Then, based on the SSD calculated by the cumulative addition circuit 13 for all (ui, vi) in the search area, the minimum value SSDmin of the SSD and the coordinate value of the reference block and the coordinate value of the standard block that give the minimum SSD are calculated. The difference vector (u, v) of is updated in the search area, and the minimum value SSDmin and the estimated value of the SSD in the search area are updated.
Calculate (u, v).

FIG. 5 shows an example of a block diagram of the evaluation value calculation circuit 9. In FIG. 5, reference numeral 15 is an adder, 16a is a multiplier, 17a and 17b are dividers, 18 is a subtractor, and 19 is an absolute value calculation circuit. The adder 15 adds the mean square luminance gradients in the horizontal and vertical directions to calculate the denominator of (Equation 3). The multiplier 16a multiplies the block size vertically and horizontally to calculate the area N of the block. The divider 17a normalizes the minimum value of the residual sum of squares by the area of the block and calculates the minimum value SSDmin / N of the residual sum of squares per pixel. The subtractor 18 subtracts the noise level from the minimum value of the residual sum of squares per pixel. The absolute value calculation circuit 19 calculates the absolute value of the output of the subtractor 18. The divider 17b divides the output of the absolute value calculation circuit 19 by the output of the adder 15 and outputs the evaluation value shown in (Equation 3).

FIG. 6 shows an example of a block diagram of the estimated value selection circuit 4. In FIG. 6, 20 is a weight coefficient table, 16
Reference numeral b is a multiplier, and 21 is a comparison circuit. The multiplier 16b is
The reliability evaluation values at the time of block correlation calculation by the block correlation calculation circuits 3A, 3B, 3C, and 3D are sequentially read, the weighting coefficient corresponding to the block size is read from the weighting coefficient table 20, and the reliability evaluation value is multiplied. The comparison circuit 21 is
It is reset each time the reference block is updated, and the estimated value at the block size that minimizes the weighted evaluation value is selected and output.

FIG. 7 shows an example of the characteristic of the weighting factor used in the estimated value selection circuit 4 with respect to the area of the block. In FIG. 7, the weighting factor has a characteristic that is proportional to the reciprocal of one half of the block area.

As described above, according to this embodiment, the reliability of the estimation result by the block matching is evaluated based on the brightness gradient, the image noise, the minimum value of SSD, and the block size, and the plurality of block sizes are calculated. Among the estimation results by
It is possible to estimate the depth and the motion with high accuracy without repeating the calculation.

In FIG. 1, the same effect can be obtained even if the standard image frame memory 1 and the reference image memory 2 are replaced with field memories, and this is included in the present invention. Further, the block correlation calculation circuits 3A, 3B, 3
If the block correlation calculation can be performed at a sufficiently high speed, C and 3D may be calculated by one block correlation calculation circuit. Further, the characteristic of the weighting coefficient used in the estimated value selection circuit 4 is not limited to that shown in the above embodiment, and substantially the same effect can be obtained even if the characteristic is proportional to the reciprocal of the block area. Included in the present invention.

FIG. 8 is an example of a block diagram of a residual sum of squares arithmetic circuit according to the second embodiment of the present invention. In FIG. 8, 12 is a residual square calculation circuit, 13 is a cumulative addition circuit, 1
4b is a minimum value selection circuit, 22 is a residual sum of squares memory, 2
3 is a minimum value correction circuit. Of the above configuration, except for the minimum value selection circuit 14b, the residual sum of squares memory 22, and the minimum value correction circuit 23, the description is omitted because it is the same as the first embodiment of the present invention, and the minimum value selection circuit 14b will be described below. The operation of the residual sum of squares memory 22 and the minimum value correction circuit 23 will be described.

The minimum value selection circuit 14b is reset each time the correlation operation is performed on the reference block, that is, every time the cumulative addition circuit 13 performs the calculation of (Equation 1) for all (ui, vj) within the search range. . Then, the cumulative addition circuit 13
Is the minimum value SSD of the residual sum of squares SSD calculated at the pixel interval
minint is selected, and the difference vector (U, V) between the coordinate values of the reference block and the standard block giving SSDminint is calculated. The residual sum of squares memory 22 is a cumulative addition circuit 13
Stores the value of the residual sum of squares SSD calculated at the pixel interval. Based on the difference vector (U, V) selected by the minimum value selection circuit 14b, the minimum value correction circuit 23 determines the value of the residual sum of squares SSD in the vicinity thereof (the difference vector element between the reference block and the standard block is , Value of SSD when difference vector (U, V) giving minimum value SSDminint is different within ± 1 pixel range)
Is read from the residual sum of squares memory 22 and interpolation calculation is performed to calculate an estimated value and a minimum residual sum of squares with an accuracy of not more than the pixel interval.

The estimated value is corrected from the minimum residual sum of squares (Equation 5) calculated at pixel intervals and its eight neighbors (Equation 6).
The first-order partial derivative in (U, V) is calculated as (Equation 7), the second-order partial derivative is calculated as (Equation 8), and the coordinates at which the partial derivative becomes 0 are Taylor expansions of the first order. By (Equation 9)
As a result, the correction amount (Δu, Δv) is calculated by (Equation 10),
The estimated value (u, v) shown in (Equation 11) is calculated.

[0034]

[Equation 5]

[0035]

[Equation 6]

[0036]

[Equation 7]

[0037]

[Equation 8]

[0038]

[Equation 9]

From (Equation 9), (Δu, Δv) is calculated as (Equation 10).

[0040]

[Equation 10]

[0041]

[Equation 11]

Then, an estimated value (U +
Minimum value of SSD for Δu, V + Δv) SSD (U + Δu, V +
Δv) is calculated as (Equation 12).

[0043]

[Equation 12]

As described above, according to the present embodiment, the minimum value of the residual sum of squares is corrected by the minimum value of the residual sum of squares calculated in the pixel interval by the gradient of the residual sum of squares in the vicinity thereof. The estimated value can be calculated with an accuracy equal to or smaller than the pixel interval.

Note that the estimated value (u, v) of the accuracy of the pixel interval or less
The minimum value SSD (U, V) of SSD with respect to can be calculated as (Equation 13) using the mean squared luminance gradient, and the same effect can be obtained, which is included in the present invention.

[0046]

[Equation 13]

FIG. 9 is an example of a block diagram of a block correlation calculation circuit in the third embodiment of the present invention. In FIG. 9, 5 is a read address generation circuit, 6 is a read control circuit, 7 is a mean squared luminance gradient calculation circuit, 8 is a residual sum of squares calculation circuit, 9 is an evaluation value calculation circuit, 24 is an estimated value memory, and 25 is An evaluation value memory, 26 is an evaluation value selection circuit. In FIG. 9, the estimated value memory 24 and the evaluation value memory 2
5. Operations other than the evaluation value selection circuit 26 are the same as those of the first embodiment of the present invention.
Since it is the same as the second embodiment, the description thereof is omitted, and the estimation value memory 24, the evaluation value memory 25, and the evaluation value selection circuit 2 will be described below.
The operation of No. 6 will be described.

The estimated value memory 24 stores the output of the residual sum of squares calculation circuit 8. The evaluation value memory 25 stores the output of the evaluation value calculation circuit 9. The evaluation value selection circuit 26 compares the evaluation value in the pixel of interest with the evaluation values in the vicinity thereof, and selects an estimated value and an evaluation value at a point having a better evaluation value. Evaluation value selection circuit 2
FIG. 10 shows how the evaluation value and the estimated value are selected according to No. 6. In FIG. 10, P indicates the pixel of interest, the lattice point indicates the center position of the reference block for block correlation calculation,
Broken lines indicate blocks centered on P, A, B, C, and D, respectively. The evaluation value and the estimated value of the pixel of interest P are the results of all block correlation operations centered within the area surrounded by the broken line centered on P (all grid points in the area surrounded by the broken line centered on P). The best evaluation value and the estimated value in the block that has obtained the evaluation value are selected from the correlation calculation results for the block centered at.

As described above, according to the present embodiment, as the evaluation value for associating images, the best evaluation value calculated at intervals equal to or smaller than the block size in the vicinity of the pixel of interest is selected, and motion and depth estimation is performed. By selecting the motion and depth estimation value in the block having the best evaluation value, it is possible to accurately follow the change in the estimation value in the discontinuous region of the motion and depth estimation value.

It should be noted that the correspondence between the images by the small block size is performed by matching the pixel of interest with the center of the block, and the correspondence between the images by the large block size is performed by using the evaluation value selection method in this embodiment. Also, it is possible to follow changes in the estimated value in a discontinuous region of the motion and depth estimated values and is included in the present invention.

Correspondence between images with a small block size is performed by adaptively changing the pixel of interest and the center of the block by using the evaluation value selection method in the present embodiment. The correspondence is
By making the pixel of interest and the center of the block coincide with each other, it is possible to smoothly change the motion and the depth estimation value in the occluded region, which is included in the present invention.

FIG. 11 is a block diagram of a motion and depth estimation apparatus according to the fourth embodiment of the present invention. In FIG. 11, 27 is a low-pass filter, 1 is a standard image frame memory, 2 is a reference image frame memory, 3A, 3B.
Is a block correlation calculation circuit, 28A and 28B are representative pixel correlation calculation circuits, and 4 is an estimated value selection circuit. In the above configuration,
Low-pass filter 27, representative pixel correlation calculation circuit 28A,
Operations other than 28B are similar to those of the first, second, and third embodiments of the present invention, and therefore description thereof will be omitted, and operations of the low-pass filter 27 and the representative pixel correlation calculation circuit 28 will be described below.

The low-pass filter 27 limits the band of the input image so that the correlation calculation in the representative pixel calculation circuit 28 is not affected by the repeated pattern. That is, when the representative pixels for which the correlation calculation is performed by the representative pixel correlation calculation circuit 28 have an interval of two pixels, the cutoff frequency of the low-pass filter is set to 0.25 cycle / pixel (cycle: 4 pixels).

The representative pixel correlation calculation circuit 28 shows the correlation calculation calculated by the block correlation calculation circuit 3 for all the pixels in the block in the first, second and third embodiments of the present invention. As shown in 12, calculation is performed only for the representative pixel in the block.

FIG. 12 shows an example in which representative pixels are arranged at two-pixel intervals, and only the pixels shown in black in the figure are subjected to correlation calculation within the search area as shown in FIG.

The representative pixel correlation calculation circuit 28 has the same configuration as the block correlation calculation circuit 3 shown in FIG. 3, and the addresses generated by the read address generation circuit 5 are arranged at equal pixel intervals of 2 or more, so that the representative pixel Perform correlation calculation. The interval of the representative pixel is 2 when the block size is 40 × 40.
If the pixel interval is about 4 pixels when 80 × 80, the estimated value can be calculated with the same accuracy as the correlation calculation for all pixels.

As described above, according to this embodiment, it is possible to reduce the calculation amount of the correlation calculation with a large block size, and to reduce the circuit scale and the calculation cost.

It should be noted that, in the evaluation value selection methods in all of the above-described embodiments, the same effect can be obtained even if the evaluation value is weighted according to the distance between the pixel of interest and the block for which correlation calculation is performed in the vicinity thereof. Yes, and included in the present invention.

Further, the denominator of (Equation 3) is changed in the evaluation equations of the block correlation calculation in all the above-mentioned embodiments, and (Equation 1)
The same effect can be obtained by 4) or (Equation 15) and is included in the present invention.

[0060]

[Equation 14]

[0061]

[Equation 15]

Further, in all the above-mentioned examples, the evaluation scale of the residual between blocks and the reliability evaluation value J of the correspondence are shown in (Equation 3) using the residual sum of squares SSD of (Equation 1). However, the same effect can be obtained by using the reliability evaluation value shown in (Expression 17) based on the sum SAD of the absolute values of the residuals shown in (Expression 16).

[0063]

[Equation 16]

[0064]

[Equation 17]

In addition, the denominator of (Equation 17) is (Equation 18),
Even if it changes like (Formula 19), substantially the same effect can be obtained and it is included in the present invention.

[0066]

[Equation 18]

[0067]

[Formula 19]

In the evaluation value calculation methods and circuits in all of the above embodiments, if the noise level of the image is a level that can be ignored, (Equation 3), (Equation 14),
(Equation 15), (Equation 17), (Equation 18), (Equation 19)
With σ n set to 0, the configuration of the evaluation value calculation circuit 9 may be changed from the configuration shown in FIG. 5 to the configuration shown in FIG.
Obviously, the same effect can be obtained with a simpler configuration, and such an embodiment is of course included in the present invention.

[0069]

As described above, according to the present invention,
The reliability of the estimation result by block matching is evaluated based on the luminance gradient, image noise, the minimum value of the evaluation scale of the residual between blocks, and the block size , and selected from the estimation results by multiple block sizes. However, it is possible to accurately estimate the depth and the motion without performing repeated calculations.

Further, the minimum value of the evaluation scale of the residual between blocks and the difference vector of the coordinate value giving the minimum value are corrected based on the distribution in the vicinity of the minimum value of the evaluation scale of the residual, and Accurate depth and motion estimation with sub-interval accuracy is possible.

Further, the pixel of interest and the center of the block are adaptively shifted at the time of the correlation calculation, and the estimated value when the evaluation value is the best is selected, so that even in an area where there is a sudden change in depth or motion. Accurate estimation can be performed.

Further, the correspondence between images of a small block size is performed by matching the pixel of interest with the center of the block, and the correspondence between images of a large block size is obtained by adaptively shifting the pixel of interest and the center of the block. By selecting the estimated value when the evaluation value is the best, it is possible to follow the change in the estimated value in the discontinuous region of the motion and depth estimated values.

Correspondence between images with a small block size is performed by adaptively changing the pixel of interest and the center of the block. Correspondence between images with a large block size matches the pixel of interest and the center of the block. By doing so, it is possible to smoothly change the motion and the depth estimation value in the occluded region.

Further, at the time of the correlation calculation of a large block size, the correlation calculation is performed not for all the pixels in the block but only for the representative pixels, so that the calculation amount can be reduced while maintaining the accuracy.

As described above, according to the present invention, the reliability of the estimation result by block matching is evaluated based on the brightness gradient, the image noise, the minimum value of the evaluation scale of the residual between blocks, and the block size. Then, it is possible to select from estimation results based on a plurality of block sizes and accurately estimate depth and motion without performing repeated calculations, which is a great effect.

[Brief description of drawings]

FIG. 1 is a block diagram of a motion and depth estimation apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram showing a block correlation calculation.

FIG. 3 is a block diagram of a block correlation calculation circuit according to the first embodiment of the present invention.

FIG. 4 is a block diagram of a residual sum of squares arithmetic circuit according to the first embodiment of the present invention.

FIG. 5 is a block diagram of an evaluation value calculation circuit according to the first embodiment of the present invention.

FIG. 6 is a block diagram of an estimated value selection circuit according to the first embodiment of the present invention.

FIG. 7 is a characteristic diagram showing an example of a distribution of weighting factors of an estimated value selection circuit.

FIG. 8 is a block diagram of a residual sum of squares arithmetic circuit according to a second embodiment of the present invention.

FIG. 9 is a block diagram of a block correlation calculation circuit according to a third embodiment of the present invention.

FIG. 10 is a diagram showing selection of evaluation values and estimated values in an evaluation value selection circuit.

FIG. 11 is a block diagram of a motion and depth estimation apparatus according to a fourth embodiment of the present invention.

FIG. 12 is a diagram showing the arrangement of representative points in a block.

FIG. 13 is a diagram showing a representative pixel correlation calculation.

FIG. 14 is a block diagram of an evaluation value calculation circuit when a noise level is not considered.

[Explanation of symbols]

1 Reference Image Frame Memory 2 Reference Image Frame Memory 3A, 3B, 3C, 3D Block Correlation Calculation Circuit 4 Estimated Value Selection Circuit 27 Low Pass Filters 28A, 28B Representative Pixel Correlation Calculation Circuit

─────────────────────────────────────────────────── ─── Continuation of the front page (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 7/32 G01B 11/00 G06T 7/00 H04N 13/00

Claims (10)

(57) [Claims] 1. Correspondence between images is based on a brightness gradient, a minimum value of a residual evaluation scale within a search range, and an evaluation value in consideration of a block size among a plurality of correspondence results by a plurality of block sizes. motion and depth estimation method characterized by selecting based on. 2. Correspondence between images includes brightness gradient, image noise, minimum value of residual evaluation scale within search range, and block size from a plurality of correspondence results by a plurality of block sizes. A motion and depth estimation method, characterized in that selection is performed based on a considered evaluation value. 3. The minimum value of the evaluation scale of the residual is corrected to a minimum value of the evaluation scale of the residual calculated at the pixel interval by a brightness gradient in the vicinity thereof or a gradient of the evaluation scale of the residual, and is less than or equal to the pixel interval. The calculation is performed with the accuracy of 1.
Alternatively, the motion and depth estimation method described in 2. 4. An evaluation value of correspondence between images is selected as the best evaluation value calculated at intervals equal to or smaller than a block size in the vicinity of a pixel of interest, and a motion and depth estimation value is a block of the best evaluation value. The motion and depth estimation method according to claim 1, 2 or 3, characterized in that the motion and depth estimation value in (3) is selected. 5. The evaluation value of the correspondence between the images is weighted according to the distance between the center of the block and the pixel of interest, with respect to the evaluation value calculated at intervals equal to or smaller than the block size in the vicinity of the pixel of interest. the best evaluation value claim 1 or 2 Symbol mounting motion and depth estimation method and selects the motion and depth estimate at block obtained the evaluation value after the association. 6. Correspondence between images by a small block size is performed by matching the pixel of interest with the center of the block,
The motion and depth estimation method according to any one of claims 1 to 5, wherein the correspondence between images with a large block size is performed by adaptively changing the pixel of interest and the center of the block. 7. Correspondence between images based on a small block size is performed by adaptively changing the center of a target pixel and block, and correspondence between images based on a large block size corresponds to the center of a target pixel and block. The motion and depth estimation method according to any one of claims 1 to 5, wherein 8. The motion and depth estimation method according to claim 1, wherein the correspondence between the images by the block size is calculated for a representative pixel in the block. 9. A reference image frame memory for storing a reference image, a reference image frame memory for storing a reference image, block correlation calculation and calculation of a reliability evaluation value of an estimated value for each of a plurality of block sizes. The reliability of the block correlation calculation circuit and the estimation result by the block matching in the block correlation calculation circuit is evaluated based on the luminance gradient, the image noise, the minimum residual sum of squares, and the block size, or in the estimation results of the block size
A motion and depth estimation device comprising an estimated value selection circuit for selecting from the above. 10. A standard image frame memory for storing a standard image, a reference image frame memory for storing a reference image, a block correlation calculation and a reliability evaluation value of an estimated value for each of a plurality of block sizes. Estimate the reliability of a plurality of block sizes by evaluating the reliability of the block correlation calculation circuit and the estimation result by block matching in the block correlation calculation circuit based on the luminance gradient, the minimum residual sum of squares, and the block size. A motion and depth estimation device comprising an estimation value selection circuit for selecting from results.