JP6687496B2 - Parallax detection device - Google Patents

Description

  The present disclosure relates to a parallax detection device that detects parallax using a plurality of images captured from different viewpoints.

  In Patent Document 1, a first image and a second image are acquired from each of a first imaging device and a second imaging device that are arranged in parallel and adjacent to each other, and the parallax between the first image and the second image is acquired. There is disclosed a distance measuring device that detects a distance to an object photographed by the first photographing device and the second photographing device.

  In the distance measuring device described in Patent Document 1, the first image and the second image are adjusted by adjusting the blur of the image with stronger blur between the first image and the second image to the blur of the image with weak blur. The blur of the image is corrected to improve the accuracy of parallax detection.

JP, 2012-26841, A

  However, the technique described in Patent Document 1 corrects blurring caused by the lens of the image capturing apparatus, and occurs in a captured image due to, for example, the image capturing apparatus mounted on a vehicle and moving at high speed. Motion blur cannot be corrected.

  The present disclosure aims to provide a technique for suppressing motion blur.

One aspect of the present disclosure is a parallax detection device (5) including an image acquisition unit (S10), a vector setting unit (S30), an image generation unit (S40 to S70), and a parallax detection unit (S80). .
The image acquisition unit is configured to acquire a plurality of photographed images simultaneously photographed so as to include the same photographing region from different viewpoints.

  The vector setting unit divides each of the plurality of captured images into a partial area composed of a plurality of pixels, and the object captured in the partial area is captured for each of the plurality of divided partial areas. It is configured to set a motion vector indicating the magnitude and direction of movement within the image.

  The image generation unit performs, on each of the plurality of captured images, at least an image frequency characteristic indicating the frequency characteristic of the captured image divided by a vector frequency characteristic indicating the frequency characteristic of the motion vector set in the captured image, Further, it is configured to generate a post-computation image which is an image corresponding to the frequency characteristic of the computation result.

The parallax detection unit is configured to detect parallax between the plurality of post-computation images using the plurality of post-computation images generated by the image generation unit corresponding to each of the plurality of captured images.
The parallax detection device of the present disclosure configured in this manner suppresses motion blur according to the magnitude and direction of the motion vector by performing an operation in which the image frequency property is divided by the vector frequency property. be able to. The motion vector indicates the size and direction of movement of the object in the partial area in the captured image. Therefore, the parallax detection device of the present disclosure can suppress the motion blur that occurs in the captured image due to the movement of the object captured in the captured image in the captured image, and improves the accuracy of parallax detection. Can be made.

  In addition, the reference numerals in parentheses described in this column and the claims indicate the correspondence with the specific means described in the embodiments described below as one aspect, and the technical scope of the present disclosure. It is not limited.

It is a block diagram showing a configuration of a distance measuring device 1. It is a flow chart which shows distance measurement processing. It is a figure explaining partial area DR and processing area PR. It is a figure which shows the specific example which set the motion vector f on the picked-up image PG. It is a figure which shows the correspondence of the triangular window function w and the process area PR. It is a figure which shows the correspondence of the pixel PX2 in the processing area PR2, and the pixel PX3 in the processing area PR3. It is a flowchart which shows a parallax calculation process. It is a figure explaining the method of arranging the node NP. It is a perspective view of node space NPS. It is a perspective view which shows an XZ plane. It is a perspective view which shows a YZ plane. It is a perspective view which shows a right diagonal plane. It is a perspective view which shows a left diagonal plane. It is a figure explaining the distance measurement result of the ranging device 1.

Embodiments of the present disclosure will be described below with reference to the drawings.
The distance measuring device 1 of the present embodiment is mounted on a vehicle and includes a right image pickup device 2, a left image pickup device 3, a vehicle speed sensor 4, and an image processing device 5, as shown in FIG. Hereinafter, the vehicle in which the distance measuring device 1 is mounted is referred to as the own vehicle.

  The right image pickup device 2 and the left image pickup device 3 continuously photograph a landscape in front of the own vehicle and output image data representing the photographed images to the image processing device 5. The right imaging device 2 and the left imaging device 3 are installed on the right side and the left side with respect to the traveling direction of the host vehicle, respectively. Hereinafter, the image data of the right imaging device 2 will be referred to as right image data, and the image data of the left imaging device 3 will be referred to as left image data.

  The right image pickup device 2 and the left image pickup device 3 are arranged in parallel and equidistant positions. Specifically, the right imaging device 2 and the left imaging device 3 are arranged such that the optical axis of the right imaging device 2 and the optical axis of the left imaging device 3 are parallel to each other. As a result, the imaging surface of the right imaging device 2 and the imaging surface of the left imaging device 3 are arranged on the same plane. Furthermore, the right image pickup device 2 and the left image pickup device 3 are set in advance along the horizontal direction so that the horizontal axis of the image pickup surface of the right image pickup device 2 and the horizontal axis of the image pickup surface of the left image pickup device 3 coincide with each other. They are placed apart by the baseline length. The horizontal axis of the image pickup surface is the X axis of the X axis and the Y axis of the two-dimensional orthogonal coordinate system whose origin is the intersection of the image pickup surface and the optical axis on the image pickup surface.

The vehicle speed sensor 4 detects the traveling speed of the host vehicle (hereinafter, vehicle speed) and outputs a vehicle speed signal indicating the detection result to the image processing device 5.
The image processing device 5 mainly includes a well-known microcomputer including a CPU, a ROM, a RAM, an I / O, and a bus line connecting these components. Various functions of the microcomputer are realized by the CPU executing the programs stored in the non-transitional substantive recording medium. In this example, the ROM corresponds to the non-transitional substantive recording medium storing the program. Further, by executing this program, the method corresponding to the program is executed. It should be noted that some or all of the functions executed by the CPU may be configured as hardware by one or more ICs or the like. Further, the number of microcomputers forming the image processing apparatus 5 may be one or plural.

  In the distance measuring device 1 configured in this way, the image processing device 5 executes the distance measuring process. This distance measurement process is a process that is repeatedly executed during the operation of the image processing device 5.

  When this distance measuring process is executed, the image processing device 5 first acquires the right image data from the right imaging device 2 and acquires the left image data from the left imaging device 3 in S10, as shown in FIG. get. Then, in S20, the vertical shift between the right captured image represented by the right image data acquired in S10 and the left captured image represented by the left captured data acquired in S10 is corrected to correct the right captured image and the left captured image. Parallelize each other. Specifically, in order that the heights of the image areas (for example, pixels) that are in a corresponding relationship between the right captured image and the left captured image match each other, the vertical direction of all the pixels is adjusted according to a preset correction parameter. Convert coordinates and correct vertical displacement.

  Next, in S30, a motion vector is set for each of the right captured image and the left captured image. Hereinafter, the right captured image and the left captured image are collectively referred to as a captured image. Specifically, as shown in FIG. 3, first, the photographed image PG is formed into a rectangular shape having m pixels (m is a positive integer) in the X-axis direction and n pixels (n is a positive integer) in the Y-axis direction. Divide into partial regions DR. In the present embodiment, the captured image PG is divided so that, for example, there are 80 partial regions DR along the X-axis direction and 80 partial regions DR along the Y-axis direction.

  Then, for each of the plurality of partial regions DR, the motion vector f is set based on the vehicle speed v detected by the vehicle speed sensor 4 and the motion vector setting map stored in advance in the ROM of the image processing device 5.

The motion vector f is expressed by the following equation (1). In the following equation (1), d is the focal length, s is the relative speed between the object and the imaging device, and z is the distance between the object and the imaging device.
f = d × s / z (1)
The motion vector setting map indicates the correspondence relationship between the position of the partial region DR in the captured image PG, the vehicle speed v detected by the vehicle speed sensor 4, and the magnitude and direction of the motion vector f.

Next, a method of creating the motion vector setting map will be described.
The motion vector f is calculated assuming that the host vehicle is traveling straight at the vehicle speed v. First, at a coordinate x = (x1, y1) on the captured image, it is assumed that a stationary object located at a three-dimensional coordinate X = (X1, Y1, Z1) in the real space with the vehicle as the origin is captured. In this case, x = PX is established, where P is the camera parameter that associates the coordinates on the captured image with the three-dimensional points in the real space.

  Since the host vehicle is traveling straight at the vehicle speed v, three-dimensional coordinates X ′ = (X2 in the real space with the host vehicle as the origin after the preset vector calculation set time Δt has elapsed. , Y2, Z2) can be calculated. Therefore, the coordinates x ′ = (x2, y2) on the captured image of the stationary object after the preset vector calculation setting time Δt has elapsed can be calculated by x ′ = PX ′. Then, the motion vector f is calculated by (f = x'-x).

  The calculation of the motion vector f in this way is executed for all the pixels in the captured image PG. Then, for each of all the partial regions DR, the average of the motion vector f of the pixels included in the partial region DR is calculated, and the averaged motion vector f is determined as the motion vector f of the partial region DR.

  Then, the motion vector setting map is created by executing the setting of the motion vector f of the partial region DR as described above at a plurality of vehicle speeds v. The motion vector f at a vehicle speed not described in the motion vector setting map can be calculated by linear interpolation using the motion vector f at the vehicle speed described in the motion vector setting map.

FIG. 4 shows a specific example in which the motion vector f is set on the captured image. A large number of white arrows are set as the motion vector f on the captured image PG shown in FIG.
Then, when the process of S30 is completed, as shown in FIG. 2, in S40, a plurality of processing regions PR are set on the captured image. As shown in FIG. 3, the processing region PR is a rectangular region having 2 × m pixels in the X-axis direction and n pixels (n is a positive integer) in the Y-axis direction. That is, the processing region PR includes two partial regions DR arranged in the X-axis direction. Then, each of the plurality of processing regions PR set on the captured image is set such that at least one partial region DR is different from the other processing regions PR.

  For example, in the captured image PG shown in FIG. 3, the partial regions DR set in the uppermost row are sequentially arranged from the left in the order of partial region DR1, partial region DR2, partial region DR3, partial region DR4, partial region DR5, and partial region. DR6 ... Further, the processing regions PR set in the uppermost row are referred to as a processing region PR1, a processing region PR2, a processing region PR3, a processing region PR4, a processing region PR5, ...

  In this case, the processing area PR1 includes a partial area DR1 and a partial area DR2. The processing region PR2 includes a partial region DR2 and a partial region DR3. The processing region PR3 includes a partial region DR3 and a partial region DR4. The processing region PR4 includes a partial region DR4 and a partial region DR5. The processing area PR5 includes a partial area DR5 and a partial area DR6.

  Then, when the process of S40 ends, as shown in FIG. 2, in S50, a filter process for removing motion blur of the images in the processing region PR is performed for all the processing regions PR of the right captured image and the left captured image. To do. Specifically, first, the Fourier transform represented by the following equations (2), (3), (4), and (5) is executed for each of all the processing regions PR. Fft in the following equations (2) to (5) indicates Fourier transform.

  The following expression (2) indicates that the frequency characteristic L of the brightness I is calculated by Fourier transforming the two-dimensional distribution of the brightness I in the processing region PR. The following Expression (3) indicates that the frequency characteristic F of the motion vector f is calculated by Fourier transform of the two-dimensional distribution of the motion vector f in the processing region PR.

  The following formula (4) indicates that the frequency characteristic Fx of the differential operator fx is calculated by Fourier transforming the differential operator fx in the X-axis direction. The differential operator fx is defined by the following expression (6). The following equation (5) indicates that the frequency characteristic Fy of the differential operator fy is calculated by Fourier transforming the differential operator fy in the Y-axis direction. The differential operator fy is defined by the following expression (7).

After performing the Fourier transform represented by the following equations (2) to (5) for all the processing regions PR, the inverse Fourier transform represented by the following equation (8) is executed for each of the processing regions PR. Ifft in the following equation (8) indicates an inverse Fourier transform. Λ in the following equation (8) is a parameter for adjusting the smoothness of the image. The image x on the left side of the following formula (8) shows the two-dimensional distribution of the brightness I in the processing region PR after the filtering process is performed.

The numerator in the parentheses on the right side of the above equation (8) is the product of the frequency characteristic L and the inverse F. The inverse F is (1 / F), and by multiplying the frequency characteristic F of the motion vector f, the amplitude becomes 1 and the phase becomes 0.

Here, a method of removing the blur of the image by using the Fourier transform will be briefly described.
First, the relationship between the blurred image g (x, y) and the unblurred image f (x, y) is modeled by the following expression (9). Here, h (x, y) in the following equation (9) is a blur function. Further, n (x, y) in the following expression (9) is noise. In addition, "*" on the right side of the following equation (9) is not a multiplication symbol but a convolution operation symbol.

The blurring function h (x, y) is applied to each pixel of the unblurred image f (x, y) as shown in the following equation (10). The operation of applying the blur function to each pixel in this way is called convolution.

In order to directly derive a blur-free image f (x, y) from a blurred image g (x, y), many simultaneous equations need to be solved, which is difficult. However, it is known that the following expression (11) is established when the convolution theorem is applied using the Fourier transform. This is because the convolution in the spatial domain is represented by a simple product in the frequency domain.

Therefore, in the frequency domain, the relationship between the blurred image G (u, v) and the unblurred image F (u, v) can be modeled by the following expression (12). Here, N (u, v) in the following equation (12) is noise in the frequency domain.

By dividing the blurred image G (u, v) shown in the following expression (12) by the blur function H (u, v) in the frequency domain, a blur-free image can be obtained. However, as shown in the following expression (13), by dividing G (u, v) by H (u, v), an image F ^ (u, v) from which blur has been removed in the frequency domain can be obtained. .

However, in general, H (u, v) often becomes 0 or a value near 0. As a result, a noise portion (that is, N (u, v) / H (u, v)) is amplified, and it is difficult to remove the blur. Therefore, various filtering processes have been proposed. For example, the following formula (14) shows a method called “Constrained Least Square Filtering”.

In the present embodiment, the one obtained by replacing the y | P (u, v) | ² part of the following formula (14) with the following formula (15) is used. The following expression (15) has a form simulating a noise component, and by adjusting the size with the parameter λ, it is possible to restore the image from which the blur is removed. If the parameter λ is increased, the image becomes smooth, and if the parameter λ is decreased, the image becomes rough.

Then, when the processing of S50 is completed, as shown in FIG. 2, in S60, the triangular window function w is applied to each of all the processing regions PR subjected to the filtering processing of S50. As shown in FIG. 5, the triangular window function w is set so that the correspondence between the position in the X-axis direction in the processing region PR and the weighting coefficient is a triangle. Specifically, first, the position of the pixel located on the leftmost side in one processing region PR in the X-axis direction is set to 1 [pixel], and the position of the pixel located on the rightmost side in the X-axis direction is set. 2 m [pixel]. Then, the weight coefficient when the position in the X axis direction is 1 [pixel] and 2 m [pixel] is set to 0, and the weight coefficient when the position in the X axis direction is (m + 0.5) [pixel] is set to 1. When the position in the X-axis direction is between 1 [pixel] and (m + 0.5) [pixel], the position in the X-axis direction is x, and the weighting factor is {(x-1) / ( m-0.5)}. On the other hand, when the position in the X-axis direction is between (m + 0.5) [pixel] and 2m [pixel], the position in the X-axis direction is x and the weighting factor is {(2m−x) / ( m-0.5)}.

  In S60, one processing region PR is selected from all the processing regions PR, the position of the pixel in the X-axis direction is specified for each of all the pixels forming the selected processing region PR, and the specified X-axis is specified. The weighting factor corresponding to the position in the direction is calculated using the triangular window function w. Then, the calculated weighting factor is multiplied by the luminance of the pixel, and this multiplication value is updated as the new luminance of the pixel. The updating of the brightness of all pixels in this way is executed for all the processing regions PR.

  Then, when the processing of S60 is completed, as shown in FIG. 2, in S70, the processing regions PR after the triangular window function w is applied in S60 are integrated to obtain the right and left captured images, respectively. Restore. Hereinafter, the captured image restored in S70 is referred to as a restored captured image. Further, the right captured image after being restored in S70 is referred to as a restored right captured image, and the restored left captured image is referred to as a restored left captured image.

Specifically, for all the processing regions PR to which the triangular window function w has been applied in S60, pixels at the same position in the captured image are extracted, and the sum of the brightness of the extracted one or more pixels is calculated. To do. This total sum is divided by the total sum Sw of the triangular window function w shown by the following formula (16), and this division value is taken as the luminance of the pixel at the above position. In the following expression (16), w _i is a weighting coefficient when the position of the pixel in the X-axis direction is i. Further, i is an integer from 1 to 2 m.

For example, as shown in FIG. 6, a pixel PX2 whose position in the X-axis direction is s [pixels] and a position in the Y-axis direction is t [pixels] in the processing region PR2, and a position in the X-axis direction in the processing region PR3. Is (s-m) [pixel] and the position in the Y-axis direction is t [pixel], the pixel PX3 has the same position in the captured image. Therefore, the brightness of the pixel PX2 and the brightness of the pixel PX3 are added, and this added value is divided by the total sum Sw of the triangular window function w.

The brightness calculation as described above is executed for all pixels in the captured image.
Then, when the processing of S70 is completed, as shown in FIG. 2, the parallax calculation processing is executed in S80. Here, the procedure of the parallax calculation process executed in S80 will be described.

  When this parallax calculation processing is executed, the image processing apparatus 5 first, in S210, as shown in FIG. 7, the restored right photographed image and the restored left photographed image corresponding to the right photographed image and the left photographed image, respectively. Using the image, the nodes NP are arranged in the node space NPS, and the cost of the nodes NP is calculated.

  Specifically, as shown in FIG. 8, the position of each pixel forming the captured image is set in the physical coordinate system for each of the restored right captured image and the restored left captured image. Physical coordinates are coordinates in which the positive direction of the X axis is rightward and the positive direction of Y axis is downward, with the upper left corner of the captured image as the origin. As a result, the position of each pixel forming the captured image is set in pixel units. Hereinafter, the position in the X-axis direction will be referred to as the X coordinate position, and the position in the Y-axis direction will be referred to as the Y coordinate position. The restored right photographed image and the restored left photographed image are also referred to as a reference image and a comparative image, respectively.

Then, the reference image is divided into rectangular blocks BL having p pixels (p is a positive integer) in the X-axis direction and q pixels (q is a positive integer) in the Y-axis direction.
Next, for each of the plurality of divided blocks BL, as shown by a region SR, a region having the same Y coordinate position in the comparison image is set as a search region. In this search area, a plurality of blocks having different parallax from the block BL and having the same size as the block BL (that is, a rectangle of p pixels in the X-axis direction and q pixels in the Y-axis direction) are set as the node setting blocks. Extract as BLn. Then, the node NP is arranged in the node space NPS based on the extracted plurality of node setting blocks BLn. In the node space NPS, as shown in FIG. 9, the X coordinate position of the block BL is the X axis, the Y coordinate position of the block BL is the Y axis, and the parallax of the node setting block BLn corresponding to the block BL is the Z axis. 3D orthogonal coordinate space.

  For example, in FIG. 8, for the block BB1 in the reference image, the block BC1 (1) having the same X coordinate position in the search area SR is extracted as the node setting block BLn, as indicated by an arrow AL1. In this case, if the X coordinate position of the block BB1 is x1, the Y coordinate position of the block BB1 is y1, and the parallax between the block BB1 and the block BC1 (1) is d1, the block BC1 (1 ) Corresponding to node NP1 (1) is located at (x1, y1, d1) in the node space NPS. The node space NPS shown in FIG. 8 is an XZ plane whose Y coordinate is y1.

  As for the block BB1 in the reference image, as shown by an arrow AL3, a block BC1 (2) having a different X coordinate position in the search area SR is extracted as the node setting block BLn. In this case, if the parallax between the block BB1 and the block BC1 (2) is written as d2, the node NP1 (2) corresponding to the block BC1 (2) is (x1, in the node space NPS) as indicated by an arrow AL4. y1, d2).

Also, for the block BB2 adjacent to the block BB1 in the reference image, the node setting block BLn is extracted within the same search area SR as the block BB1.
For example, with respect to the block BB2 in the reference image, as shown by an arrow AL5, the block BC2 (1) having the same X coordinate position in the search region SR is extracted as the node setting block BLn. In this case, if the X coordinate position of the block BB2 is x2, the Y coordinate position of the block BL is y1, and the parallax between the block BB2 and the block BC2 (1) is d1, the block BC2 (1 ) Corresponding to the node NP2 (1) is arranged at (x2, y1, d1) in the node space NPS.

  Further, the cost is calculated for each of the plurality of nodes NP arranged in the node space NPS. The cost is a value representing the degree of similarity between two blocks used when arranging the node NP in the node space NPS. The above two blocks are a block BL in the reference image and a node setting block BLn extracted from the comparison image in order to arrange the nodes NP corresponding to this block BL. For example, the cost of the node NP1 (1) is calculated using the block BB1 and the block BC1 (1).

The X coordinate position and the Y coordinate position of the node NP in the node space NPS are collectively expressed as a node position p, and the parallax of the node NP is expressed as u _p . Further, the cost of the node NP arranged at the position specified by the node position p and the parallax u _p in the node space NPS is expressed as D (p, u _p ).

In the present embodiment, the cost D (p, u _p ) is calculated by the following equation (17) using a well-known SSIM (Structural Similarity).

In the above equation (17), μ _x is the average value of the luminance of the pixels included in the block BL in the reference image. μ _y is the average value of the luminance of the pixels included in the node setting block BLn in the comparison image. σ _x is the standard deviation of the luminance of the pixels included in the block BL in the reference image. σ _y is the standard deviation of the brightness of the pixels included in the node setting block BLn in the comparison image. σ _xy is the covariance of the luminance of the pixels included in the block BL in the reference image and the node setting block BLn in the comparison image. c ₁ , c ₂ , c ₃ , α, β, γ are preset constants.

In this way, when the nodes NP are arranged in the node space NPS for all the divided blocks BL and the costs D (p, u _p ) are calculated for all the arranged nodes NP, the process of S210 ends. To do.

When the process of S210 is completed, as shown in FIG. 7, at S220, it calculates the X-direction movement cost _{E x} node NP. The X-direction movement cost _Ex will be described later. As shown in FIG. 8, the block BL is divided into q pixels along the Y-axis direction. Therefore, as indicated by planes PL1, PL2, PL3 in FIG. 10, in the node space NPS, there is an XZ plane in which a plurality of nodes NP are arranged for each q pixel along the Y-axis direction.

For each of the plurality of X-Z plane, and calculates an X-direction movement cost E _x node NP. Hereinafter, a method of calculating the X-direction movement cost E _x node NP.
For example, as shown in FIG. 8, it is assumed that a plurality of nodes NP are arranged in a two-dimensional matrix on the XZ plane. Further, for the following description, a node located in the j-th column (j is a positive integer) of the i-th row (i is a positive integer) with the Z-axis direction as the column direction and the X-axis direction as the row direction. The NP is written as NP (i, j). Note that the smaller the parallax is, the smaller i is, and the closer the X coordinate position is from the origin, the smaller j is.

  First, one node NP is selected as the end point from the plurality of nodes NP arranged on the XZ plane. One of the plurality of nodes NP whose X coordinate position is closest to the origin (that is, the node NP located in the first column of the i-th row) is set as the starting point. Then, as shown by the moving direction M1 in FIGS. 8 and 10, one of the plurality of nodes NP located in the adjacent row (that is, the second row) from the start point in the positive direction of the X-axis. Move to. Further, the moved node NP is moved to one of the plurality of nodes NP located in the adjacent column (that is, the third column). In this way, one movement path is determined in which the node NP at the starting point is sequentially moved to the node NP one column at a time in the positive direction of the X-axis, and is moved to the node NP selected as the end point. Hereinafter, this moving route is referred to as a rightward moving route. FIG. 8 shows a rightward movement route that moves to the right in the order of NP (4,1), NP (3,2), NP (3,3), and NP (3,4) from the left.

  Then, the cost E of the determined travel route is calculated by the following formula (18).

The first term on the right side of the above equation (18) indicates the total sum of the costs D (p, u _p ) of the nodes NP existing on the moving route. The first term on the right side of the above equation (18) is called a data term. The second term on the right side of the above equation (18) is a regularization term. Of the second term of the right side of the equation _{(18) S (u p,} u q) from node NP parallax is _{u p,} is the disparity cost of disparity moves node NP is a _{u q.} S _(u p, u _q) are set function so as large a difference between the parallax _{u p} and the parallax _{u q} value increases. In this _{embodiment, S (u p, u q} ) is the absolute value of the difference between the parallax _{u p} and the parallax _{u q.} Therefore, the second term on the right side of the above equation (18) indicates the sum total of changes in parallax due to passing through the movement route.

  In this way, all possible movement paths in the case where the starting point is any one of the plurality of nodes NP located in the first row and the ending point is one node NP selected from the plurality of nodes NP Can be determined, and the cost E can be calculated for all the determined travel routes. As a result, it is possible to identify the travel route that minimizes the cost E.

  In the present embodiment, instead of calculating the cost E for all the travel routes as described above, a well-known Viterbi algorithm, which is a kind of dynamic programming, is used to perform an operation for identifying the travel route that minimizes the cost E. I do.

  In S220, the rightward movement path having the minimum cost E is specified for all the nodes NP arranged in one XZ plane. Thereby, the minimum cost of the rightward movement path (hereinafter, rightward movement cost) is calculated for each of all the nodes NP arranged on the X-Z plane.

  Next, similar to the calculation of the rightward movement cost, the minimum cost of the leftward movement route (hereinafter, leftward movement cost) is calculated for each of all the nodes NP arranged on the XZ plane. .

  Specifically, one node NP is selected as the end point from the plurality of nodes NP arranged on the XZ plane. One of a plurality of nodes NP whose X coordinate position is farthest from the origin (that is, a node NP located in the last column of the i-th row) is set as the starting point. Then, as indicated by the moving direction M2 in FIGS. 8 and 10, the movement is made from the starting point in the negative direction of the X axis to one of the plurality of nodes NP located in the adjacent row. Further, the moved node NP is moved to one of the plurality of nodes NP located in the adjacent column. In this way, one moving path is determined by sequentially moving from the starting node NP to the node NP one column at a time in the negative direction of the X axis and moving to the node NP selected as the end point. Hereinafter, this moving route is referred to as a leftward moving route.

  In S220, similarly to the case of the rightward movement route, the leftward movement route having the minimum cost E is specified for all the nodes NP arranged in one XZ plane. As a result, the minimum cost of the leftward movement path (hereinafter, leftward movement cost) is calculated for each of all the nodes NP arranged on the XZ plane.

Then, in S220, an addition value obtained by adding the rightward movement cost and the leftward movement cost is calculated as the X-direction movement cost E _x for each of all the nodes NP arranged on the XZ plane.

When the calculation of the X-Z plane in the X-direction moving costs E _x is completed, the next X-Z plane, in the same manner, to calculate the X-direction moving cost E _x. Then, when the X-direction movement cost _Ex is calculated for all the X-Z planes, the process of S220 ends.

When the process of S220 ends, as shown in FIG. 7, the Y-direction movement cost E _y of the node NP is calculated in S230. The Y-direction movement cost E _y will be described later. As shown in FIG. 5, the block BL is divided every p pixels along the X-axis direction. Therefore, as shown by the planes PL11, PL12, PL13, PL14 in FIG. 11, a YZ plane in which a plurality of nodes NP are arranged exists in the nodal space NPS every p pixels along the X-axis direction. To do.

In S230, the Y-direction movement cost E _y of the node NP is calculated for each of the plurality of YZ planes. The calculation of the Y-direction movement cost E _y of the node NP is different from the above-described calculation of the X-direction movement cost E _x in that the YZ plane is used instead of the XZ plane.

  Specifically, on the YZ plane, as indicated by the moving direction M3 in FIG. 11, one of the plurality of nodes NP whose Y coordinate position is closest to the origin is set as the starting point, and the positive direction of the Y axis is set. Toward, the movement route that is sequentially moved to the node NP one column at a time and moved to the node NP selected as the end point is set as the downward movement route. Further, on the YZ plane, as shown by the moving direction M4 in FIG. 11, the Y coordinate position starts from one of the plurality of nodes NP farthest from the origin and moves in the negative direction of the Y axis. The moving path moved to the node NP one column at a time and moved to the node NP selected as the end point is defined as the upward moving path.

  In S230, the downward movement route and the upward movement route that minimize the cost E are specified for all the nodes NP arranged in one YZ plane. As a result, for each of all the nodes NP arranged on the YZ plane, the minimum cost of the downward movement route (hereinafter, downward movement cost) and the minimum cost of the upward movement route (hereinafter, upward direction) Moving cost) is calculated.

  Then, in S230, an addition value obtained by adding the downward movement cost and the upward movement cost is calculated as the Y-direction movement cost Ey for each of all the nodes NP arranged on the YZ plane.

When the calculation of the Y-direction movement cost E _y is completed on the Y-Z plane, the Y-direction movement cost E _y is calculated similarly for the next Y-Z plane. Then, when the Y-direction movement cost E _y has been calculated for all the YZ planes, the processing of S230 ends.

When the process of S230 ends, as shown in FIG. 7, in S240, the diagonally rightward movement cost E _xy of the node NP is calculated. The right diagonal movement cost E _xy will be described later.
In S240, first, as shown by the planes PL21, PL22, PL23 in FIG. 12, a plurality of plurality are formed in the nodal space NPS so as to be perpendicular to the XY plane across the X axis and the Y axis. Plane (hereinafter, diagonal plane to the right) is set. The plurality of right diagonal planes are set to be parallel to each other, and the nodes NP are arranged over the entire plane.

In S240, for each of the plurality of right oblique plane, and it calculates the right oblique direction movement cost _{E x-y} node NP. The calculation of the right diagonal movement cost E _xy of the node NP is different from the above-described calculation of the X direction movement cost E _x in that the right diagonal plane is used instead of the XZ plane.

  Specifically, on the right oblique plane, as shown in the moving direction M5 in FIG. 12, starting from one of the plurality of nodes NP closest to the Y-axis, the nodes are line by column toward the X-axis. The moving route that is sequentially moved to the NP and is moved to the node NP selected as the end point is the upper right direction moving route. Further, in the right oblique plane, as shown by the moving direction M6 in FIG. 12, one of the plurality of nodes NP farthest from the X axis is started, and one column is moved toward the node NP toward the Y axis. The moving route that is sequentially moved to the node NP selected as the end point is defined as the lower left direction moving route.

  In S240, the upper right movement path and the lower left movement path with the lowest cost E are specified for all the nodes NP arranged on one right oblique plane. As a result, for each of all the nodes NP arranged on the right diagonal plane, the minimum cost of the upper right movement path (hereinafter, upper right movement cost) and the minimum cost of the lower left movement path (hereinafter, lower left movement) Cost) is calculated.

Then, in S240, an addition value obtained by adding the upper right movement cost and the lower left movement cost is calculated as the right diagonal movement cost E _xy for each of all the nodes NP arranged on the right diagonal plane. To do.

When the calculation of the right oblique direction movement cost E _x-y In this right oblique plane is completed, the next right oblique plane, in the same manner, to calculate the right oblique direction movement cost E _x-y. Then, when the right diagonal movement cost E _xy is calculated for all the right diagonal planes, the process of S240 ends.

When the process of S240 ends, as shown in FIG. 7, the left diagonal movement cost E _{x + y} of the node NP is calculated in S250. The left diagonal movement cost E _{x + y} will be described later.
In S250, first, as shown by the planes PL31, PL32, PL33, PL34 in FIG. 13, a plurality of planes (hereinafter, left diagonal planes) formed so as to intersect with the right diagonal plane are set in the node space NPS. To do. . The plurality of left oblique planes are set to be parallel to each other, and the nodes NP are arranged over the entire surface of the plane.

In S250, the left diagonal movement cost Ex + y of the node NP is calculated for each of the plurality of left diagonal planes. The calculation of the left diagonal movement cost Ex _{+ y} of the node NP differs from the above-described calculation of the X direction movement cost Ex in that the left diagonal plane is used instead of the XZ plane.

  Specifically, on the left oblique plane, as shown by the moving direction M7 in FIG. 13, starting from one of the plurality of nodes NP closest to the origin, one column at a time in the direction away from the origin. The movement route that is sequentially moved to the node NP and is moved to the node NP selected as the end point is the lower right direction movement route. In the left oblique plane, as shown by the moving direction M8 in FIG. 13, one of the plurality of nodes NP farthest from the origin is set as a starting point, and the nodes NP are line by column toward the direction of approaching the origin. The moving route that is sequentially moved to the node NP selected as the end point is defined as the upper left direction moving route.

  In S250, the lower rightward movement path and the upper leftward movement path that minimize the cost E are specified for all the nodes NP arranged on one left oblique plane. As a result, for each of all the nodes NP arranged on the left diagonal plane, the minimum cost of the lower right movement path (hereinafter, lower right movement cost) and the minimum cost of the upper left movement path (hereinafter, upper left). Direction movement cost) is calculated.

Then, in S250, an added value obtained by adding the lower rightward movement cost and the upper leftward movement cost is calculated as the left diagonal movement cost E _{x + y} for each of all the nodes NP arranged on the left diagonal plane. .

When the left oblique plane calculating the left oblique direction movement cost E _{x + y} is finished, the next left oblique plane, in the same manner, to calculate the left oblique direction movement cost E _{x + y.} When the left diagonal movement cost E _{x + y} is calculated for all the left diagonal planes, the process of S250 ends.

When the process of S250 is completed, as shown in FIG. 7, in S260, the total direction movement cost E _sum is calculated by the following formula (19) for all the nodes NP in the node space NPS.

Then, in S270, for each of all the blocks BL forming the reference image, the total direction movement cost E is selected from the plurality of nodes NP having the same X coordinate position and Y coordinate position as the block BL and different in parallax. _A node NP that minimizes the _sum is selected.

  Further, in S280, the parallax of the node NP selected in S270 is determined as the parallax of the corresponding block BL. In S280, when the parallax is determined for all the blocks BL forming the reference image, the parallax calculation processing ends.

  When the process of S80 is completed, as shown in FIG. 2, in S90, for all the blocks BLm forming the reference image, based on the parallax calculated in S80, a known distance calculation formula using parallax is used. The distance is calculated, and the distance measurement process ends. As a result, the distance to the object shown in the block BLm is specified for each block BLm forming the reference image.

  The distance measuring apparatus 1 configured in this way acquires right and left photographed images simultaneously photographed so as to include the same photographing area from different viewpoints. The distance measuring device 1 divides the captured image into the partial regions DR configured by a plurality of pixels for each of the right captured image and the left captured image, and within the partial region DR for each of the plurality of divided partial regions DR. A motion vector f indicating the size and direction of movement of the object shown in FIG.

  The distance measuring apparatus 1 determines at least the frequency characteristic L of the captured image for each of the right captured image and the left captured image, as shown in the above equations (2) to (8), by using the motion vector f set in the captured image. The calculation is performed by dividing by the frequency characteristic F of, and a restored captured image corresponding to the frequency characteristic of the calculation result is generated.

  The range finder 1 uses the restored right photographed image and the restored left photographed image generated corresponding to the right photographed image and the left photographed image, respectively, and uses the restored right photographed image and the restored left photographed image. Detect the parallax between.

  The range finder 1 calculates the frequency characteristic L by performing a Fourier transform on the two-dimensional distribution of the brightness I of the captured image. Further, the distance measuring device 1 calculates the frequency characteristic F by performing the Fourier transform on the two-dimensional distribution of the motion vector f set in the captured image. Further, the range finder 1 generates an image after restoration by performing an inverse Fourier transform on the calculation result.

  As described above, the distance measuring apparatus 1 can suppress the motion blur corresponding to the magnitude and the direction of the motion vector f by performing the operation of dividing the frequency characteristic L by the frequency characteristic F. The motion vector f indicates the size and direction of movement of the object in the captured image within the captured image. Therefore, the distance measuring apparatus 1 can suppress the motion blur that occurs in the captured image due to the movement of the object captured in the captured image in the captured image, and improve the accuracy of parallax detection. You can

FIG. 14 shows a distance measurement result using the captured image G1 before the motion blur is suppressed and a distance measurement result using the captured image G2 after the motion blur is suppressed.
An image G1 shown in FIG. 14 is an image captured by the right imaging device 2 while the vehicle is traveling in the tunnel. The image G2 is an image after suppressing the motion blur from the image G1.

  An image G3 shown in FIG. 14 shows the result of measuring the distance using the area indicated by the indicated circle CI1 in the image G1. The image G4 shows the result of measuring the distance using the area indicated by the indicated circle CI2 in the image G2.

  As is clear from a comparison between the images G3 and G4 in FIG. 14, the image G4 has a clearer change in the light and shade indicating the length of the distance than the image G3, and the image G2 is more blurred than the image G1. Is suppressed.

  Further, the distance measuring device 1 is mounted on a vehicle and includes a vehicle speed sensor 4 that detects a traveling speed of the vehicle. The distance measuring device 1 sets the motion vector f in the partial region DR based on the vehicle speed detected by the vehicle speed sensor 4 and the position of the partial region DR in the captured image.

  Accordingly, the distance measuring apparatus 1 searches for the same object from at least two captured images at different capturing times, and calculates the motion vector f from the difference in the positions of the objects in at least two captured images. It is possible to easily set the motion vector f without performing it. Therefore, the distance measuring device 1 can reduce the processing load of the distance measuring device 1.

  In the embodiment described above, the image processing device 5 corresponds to a parallax detection device, S10 corresponds to a process as an image acquisition unit, S30 corresponds to a process as a vector setting unit, and S40 to S70 are image generation units. And the processing of S80 corresponds to the processing of the parallax detection unit.

  Further, the frequency characteristic L corresponds to the image frequency characteristic, the frequency characteristic F corresponds to the vector frequency characteristic, the restored right photographed image and the restored left photographed image correspond to the calculated image, and the vehicle speed sensor 4 is the vehicle speed detection unit. Equivalent to.

Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above embodiment, and various modifications can be implemented.
[Modification 1]
For example, in the above embodiment, the case where the parallax is detected by using the two image pickup devices (that is, the right image pickup device 2 and the left image pickup device 3) has been described, but three or more image pickup devices may be used. .

[Modification 2]
In the above embodiment, the motion vector f is set based on the motion vector setting map. However, the method of setting the motion vector f is not limited to this. For example, the same object is searched from at least two captured images at different capturing times, and the position of the object in the plurality of captured images is different. The motion vector f may be calculated. As a method of searching for the same object from a plurality of photographed images at different photographing times, for example, there is a known optical flow. In this way, by searching for the same object from a plurality of captured images, it is possible to accurately set the motion vector indicating the size and direction in which the object captured in the partial region DR moves within the captured image. Therefore, the distance measuring device 1 can further suppress motion blur and further improve the accuracy of parallax detection.

[Modification 3]
In the above embodiment, the average of the motion vectors f of the pixels included in the partial region DR is calculated, and the averaged motion vector f is determined as the motion vector f of the partial region DR. However, the motion vector f of the partial region DR may be represented by the motion vector f in the pixel at the center position of the partial region DR.

  Further, the function of one constituent element in the above-described embodiment may be shared by a plurality of constituent elements, or the function of a plurality of constituent elements may be exerted by one constituent element. Moreover, you may omit a part of structure of the said embodiment. Further, at least a part of the configuration of the above-described embodiment may be added to or replaced with the configuration of the other above-described embodiment. It should be noted that all aspects included in the technical idea specified by the wording recited in the claims are embodiments of the present disclosure.

  In addition to the distance measuring device 1 described above, various forms such as a system including the distance measuring device 1 as a component, a program for causing a computer to function as the distance measuring device 1, a medium storing the program, a parallax detection method, and the like. The present disclosure can also be realized with.

  1 ... Distance measuring device, 2 ... Right imaging device, 3 ... Left imaging device, 5 ... Image processing device,

Claims (3)

An image acquisition unit (S10) configured to acquire a plurality of captured images simultaneously captured so as to include the same imaging region from different viewpoints;
For each of the plurality of photographed images, the photographed image is divided into partial regions configured by a plurality of pixels, and an object imaged in the partial region is photographed for each of the divided plurality of partial regions. A vector setting unit (S30) configured to set a motion vector indicating the size and direction of movement within the image;
For each of the plurality of captured images, at least an image frequency characteristic indicating the frequency characteristic of the captured image is divided by a vector frequency characteristic indicating the frequency characteristic of the motion vector set in the captured image, and further, An image generation unit (S40 to S70) configured to generate a post-calculation image that is an image corresponding to the frequency characteristic of the calculation result,
A parallax detection unit configured to detect a parallax between the plurality of post-computation images by using the plurality of post-computation images generated by the image generation unit corresponding to each of the plurality of captured images. S80) and equipped with a,
Mounted on the vehicle,
A vehicle speed detection unit (4) for detecting the traveling speed of the vehicle,
The vector setting unit sets the motion vector in the partial region based on the traveling speed detected by the vehicle speed detecting unit and the position of the partial region in the captured image (5). . An image acquisition unit (S10) configured to acquire a plurality of captured images simultaneously captured so as to include the same imaging region from different viewpoints;
For each of the plurality of photographed images, the photographed image is divided into partial regions configured by a plurality of pixels, and an object imaged in the partial region is photographed for each of the divided plurality of partial regions. A vector setting unit (S30) configured to set a motion vector indicating the size and direction of movement within the image;
For each of the plurality of captured images, at least an image frequency characteristic indicating the frequency characteristic of the captured image is divided by a vector frequency characteristic indicating the frequency characteristic of the motion vector set in the captured image, and further, An image generation unit (S40 to S70) configured to generate a post-calculation image that is an image corresponding to the frequency characteristic of the calculation result,
A parallax detection unit configured to detect a parallax between the plurality of post-computation images by using the plurality of post-computation images generated by the image generation unit corresponding to each of the plurality of captured images. S80) and
Equipped with
The vector setting unit searches for the object captured in the partial area in the plurality of captured images at different capturing times among the captured images acquired by the image acquisition unit, and acquires the plurality of captured images. A parallax detection device that sets the motion vector in the partial region based on the position of the object in the inside. The parallax detection device according to claim 1 or 2 , wherein
The image generation unit calculates the image frequency characteristic by performing a Fourier transform on a two-dimensional distribution of luminance of the captured image, and performs a Fourier transform on the two-dimensional distribution of the motion vector set in the captured image. A parallax detection device that generates the post-computation image by calculating the vector frequency characteristic by performing conversion and performing an inverse Fourier transform on the calculation result.