JP6730214B2 - Parallax calculator - Google Patents

Description

The present disclosure relates to a technique of calculating parallax from two images having different viewpoints.

Patent Document 1 discloses a technique of using two images having different viewpoints to obtain a parallax at each pixel on the image. In order to obtain the parallax, a similarity cost indicating how similar the two pixels selected from each image are to each other is obtained. Further, paying attention to the pixel rows arranged in a certain direction, the similar cost is corrected by using the information of the adjacent pixels. A so-called Viterbi algorithm is used for this correction. Since the result of the correction using the Viterbi algorithm differs depending on the applied direction, it is performed at least in the four directions of leftward, rightward, upward, and downward. The parallax is specified for each pixel using these correction results.

JP, 2015-114269, A

By the way, a layered image composed of a plurality of images with different resolutions is generated from the acquired image, the same processing is performed on the layered images in order from the image with the lowest resolution, and the information obtained from the images with the low resolution is sequentially processed. There is also known a technique for making the processing efficient by using the same. Here, it is conceivable to limit the parallax search range in the Viterbi algorithm using the processing result at low resolution.

However, as a result of a detailed study by the present inventor, when a hierarchical image is used, a similar cost is generated and corrected for each image having a different resolution, which requires a huge amount of processing. Found.

In general, the capacity of the internal memory that can be accessed at high speed by the processor that executes the process is limited, and the image to be processed is stored in the external memory that is inferior in access speed. Then, a part of the image is read from the external memory as much as can be stored in the internal memory, and when the processing is completed, the processing result is stored in the external memory and the processing for the next part of the image is repeated. When the data is sequentially read from the upper row of the image and the processing is executed, the correction processing in the downward direction should be performed in parallel with the correction processing in the rightward and leftward directions based on the sequentially read data. You can However, the upward correction process needs to read data sequentially from the lower row of the image, and needs to be separately executed again after the right, left, and downward correction processes are completed. As a result, the amount of access to the external memory increases, and there is also a problem that the processing load increases.

The present disclosure provides a technique for reducing the processing load while maintaining the accuracy of parallax calculation.

The parallax calculation device according to an aspect of the present disclosure includes a storage unit (422) and a processing unit (41, 421).
The storage unit uses a pair of images captured so as to include the same shooting region from different viewpoints as a basic image pair, and a pair of image pairs having a resolution different from that of the basic image pair generated from the basic image pair. The image pairs are stored together with the image pairs.

The processing unit generates a parallax image based on the image pair group stored in the storage unit. The processing unit includes a selection unit (S220), a cost generation unit (S320), cost correction units (S330 to S390, S520 to S550), a restriction unit (S410), and a parallax generation unit (S270). Equipped with.

The selection unit selects any one of the image pairs from the group of image pairs in ascending order of resolution.
The cost generation unit uses one of the two images forming the selected image pair, which is the image pair selected by the selection unit, as the reference image and the other as the comparison image, and selects one of the nodes belonging to the node space as the target node. As a node space, a node cost calculated so that the value becomes smaller as the similarity between the pixel on the reference image corresponding to the node of interest and the pixel on the comparison image corresponding to the node of interest separated by parallax becomes higher. Generate for all nodes belonging to. However, the node is a three-dimensional point expressed by the two-dimensional position of the pixel in the reference image and the parallax expressed in the pixel size unit of the selected image pair. The node space is a three-dimensional space that uniquely represents the nodes and is limited by the image size of the reference image and the maximum value of the parallax to be detected.

The cost correction unit sets the pixel row in the node space arranged along the direction indicated by the preset correction vector as the target pixel row, and calculates the node costs of the plurality of nodes related to the target pixel row generated by the cost generation unit. , It is corrected by the correction processing using the Viterbi algorithm.

The limiting unit sets the last selected image pair selected by the selecting unit as the final image pair, and the selected image pairs other than the final image pair as the non-final image pair, and performs cost correction when the selected image pair is the non-final image pair. Based on the correction result of the section, the search range is limited in the correction process performed by the cost correction section for the selected image pair selected next by the selection section.

When the selected image pair is the final image pair, the parallax generation unit calculates the parallax corresponding to the node having the minimum node cost for each pixel from the processing results of the cost generation unit and the cost correction unit.
However, when the position of the pixels forming the selected image pair is represented by XY coordinates, the cost correction unit performs rightward processing in which the positive direction of the X axis of the XY coordinates is the direction indicating the correction vector, as the correction processing. (S330), leftward processing in which the negative direction of the X axis is the direction indicated by the correction vector (S340), and downward processing in which the positive direction of the Y axis of the XY coordinates is the direction indicated by the correction vector (S360). And when the designated image pair, which is any one of the image pairs designated in advance in the image pair group, is the selected image pair, the upward direction with the negative direction of the Y axis as the direction indicated by the correction vector is performed. The process (S520) is further executed.

According to the above-described configuration, the upward processing, which cannot be executed in parallel with other correction processing and is a factor that greatly increases the processing time, is not applied to all the image pairs, but one image pair is processed. It is only applied to. Therefore, the processing load can be reduced without lowering the accuracy of the parallax image.

The configuration of the present disclosure is based on the following knowledge obtained as a result of the inventors' earnest experiments. That is, when the upward processing is omitted in the correction processing for all image pairs, the accuracy of the parallax image is significantly reduced, but when the upward processing is performed for at least one image pair, It was found that a large decrease in accuracy was not seen in comparison with the case where the upward processing was applied to the image pair of.

It should be noted that the reference numerals in parentheses described in this column and the claims indicate the correspondence with the specific means described in the embodiments to be described later as one aspect, and do not indicate the technical scope of the present disclosure. It is not limited.

It is a block diagram which shows the structure of the distance measuring device. It is explanatory drawing regarding the calculation of the path cost in arrangement|positioning of a node. Is an explanatory view showing a joint space NPS correction vector V _{+ X,} the relationship between V _-X and the node plane P _X. It is an explanatory view showing a joint space NPS correction vector V _{+ Y,} the relationship between V _-Y and the node plane P _Y. It is explanatory drawing which shows the relationship between the nodal space NPS, correction vector V _+X+Y , V- _XY , and nodal plane _PXY . Node Spatial NPS correction vector V _{+ XY,} is an explanatory diagram showing a relationship between V _{-X + Y} and the node plane P _-XY. It is a flowchart of a distance calculation process. It is a flow chart of parallax calculation processing. It is a figure which shows the some image from which resolution differs mutually. It is a flowchart of a 1st optimization process. It is explanatory drawing which shows the process outline in a 1st optimization process. It is a flowchart of a 2nd optimization process. It is a distance image which shows the difference of the effect by the presence or absence of downward processing.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[1. Constitution]
The distance measuring device 1 of the present embodiment is a device that detects the distance to each point in a captured image by detecting the parallax of a plurality of captured images that have been captured at the same location at the same time. Particularly, in the distance measuring apparatus 1, the cost representing the similarity between each pixel of a pair of captured images is calculated, and the cost is corrected by applying the Viterbi algorithm, which is a dynamic programming method, to the calculated cost. Then, the parallax that minimizes the corrected cost is obtained as the parallax between pixels.

The distance measuring device 1 is mounted on a vehicle and includes a pair of image capturing units 2 and 3 and an image processing unit 4, as shown in FIG. Hereinafter, the vehicle equipped with the distance measuring device 1 is referred to as a host vehicle.
Each of the pair of imaging units 2 and 3 is configured as a well-known camera whose scenery in the traveling direction of the vehicle is within the imaging range. The pair of imaging units 2 and 3 are arranged such that their optical axes are separated from each other in the horizontal direction by a predetermined distance and are parallel to each other, and configure a so-called stereo camera. The pair of imaging units 2 and 3 periodically and simultaneously capture images at a predetermined time, and output the captured images to the image processing unit 4.

The image processing unit 4 is mainly composed of a well-known microcomputer having a CPU 41 and a semiconductor memory (hereinafter, memory 42) such as a RAM, a ROM, and a flash memory. Various functions of the image processing unit 4 are realized by the CPU 41 executing a program stored in the non-transitional physical recording medium. In this example, the memory 42 corresponds to a non-transitional substantive recording medium storing a program. By executing this program, the method corresponding to the program is executed. The number of microcomputers forming the image processing unit 4 may be one or more.

The image processing unit 4 realizes at least distance calculation processing by the CPU 41 executing a program. The method for realizing the function of the image processing unit 4 is not limited to software, and one or more hardware may be used for some or all of the elements. For example, when the above functions are realized by an electronic circuit that is hardware, the electronic circuit may be realized by a digital circuit including a large number of logic circuits, an analog circuit, or a combination thereof.

The memory 42 includes an internal memory 421 and an external memory 422. The internal memory 421 is a relatively small capacity memory that can be accessed from the CPU 41 at high speed. The internal memory 421 stores data to be processed by the program being executed. The external memory 422 is a memory having a larger capacity than the internal memory 421, and the access from the CPU 41 is slower than that of the internal memory 421. The external memory 422 stores the captured images acquired from the pair of imaging units 2 and 3, as well as various data generated in the process of generating the distance image in the distance calculation process described later.

[2. principle]
Before describing the processing executed by the image processing unit 4, the node NP, the node space NPS, the node cost D(p,up), and the node cost D(p,up) related to the calculation of the parallax necessary for generating the distance image are described. The correction will be described. In the following, one of the pair of images to be processed (hereinafter, image pair) is referred to as a reference image, and the other image is referred to as a comparison image.

[2-1. Node and node space]
As shown in FIGS. 2 to 6, the nodal space NPS is a parallax between pixels on the reference image and pixels on the comparison image and the X-axis and the Y-axis indicating the position of the pixel (that is, the pixel) of the reference image. It is a three-dimensional space represented by the Z axis and the Z axis. The nodal space NPS takes discrete values in units of pixel size. Each point (x, y, z) in this node space NPS is a node NP. Each node NP indicates a correspondence relationship between the pixel (x, y) on the reference image and the pixel on the comparison image having the parallax z with respect to the pixel (x, y).

In the coordinates expressing the nodal space NPS, as shown in FIG. 2, the X-axis and the Y-axis have the positive direction of the X-axis as the right direction with the upper left corner of the image planes of the reference image and the comparative image as the origin, and the Y-axis The forward direction is set to be downward. Below, the positive direction of the X axis is referred to as the right direction, the negative direction of the X axis is referred to as the left direction, the positive direction of the Y axis is referred to as the downward direction, and the negative direction of the Y axis is referred to as the upward direction.

The X-axis and Y-axis sizes of the nodal space NPS are determined by the number of pixels forming the reference image and the comparison image. The size of the Z axis of the node space NPS is determined by the range of parallax to be detected.

[2-2. Node cost]
The node cost D(p,up) is obtained for each node NP, and represents the degree of similarity between the pixel on the basic image specified by p and the pixel on the comparative image that is separated from p by the parallax up. The node cost D(p,up) is set to take a smaller value as the similarity between both pixels is higher. p corresponds to the XY coordinates (x, y) representing the position of the pixel to which the node NP belongs, and up corresponds to the parallax, that is, the Z coordinate (z) of the node NP. The node cost D(p,up) is obtained by using, for example, SSIM. SSIM is an abbreviation for Structural Similarity. Since SSIM is a known method, its description is omitted here.

[2-3. Correction of node cost]
The correction (hereinafter referred to as correction processing) for the node cost D(p,up) is performed a plurality of times along the directions indicated by a plurality of predetermined types of correction vectors V. As shown in FIG. 3, the correction vector V includes V _+X indicating the right direction and V _-X indicating the left direction. As shown in FIG. 4, the correction vector V includes V _+Y indicating the downward direction and V _−Y indicating the upward direction. As shown in FIG. 5, the correction vector V includes V _+XY indicating the upper right direction and V _-X+Y indicating the lower left direction. As shown in FIG. 6, the correction vector V includes V _{+X +Y} indicating the lower right direction and V _-XY indicating the lower left direction.

In the correction process, the following process is executed for each of the plurality of nodal planes P cut out from the nodal space NPS along the direction indicated by the correction vector V. Here, for the sake of description, the axis representing the parallax is the row direction, the direction indicated by the correction vector V, that is, the direction in which the cut-out pixel columns are arranged is the row direction, and the direction along the Z axis representing the parallax is the column direction. As shown in, each node on the matrix is represented by NP(i,j). i represents the position of the pixel on the clipped pixel row, and j represents the parallax. That is, the nodes NP(i,j) are arranged in a matrix on the node plane P.

Here, an arbitrary node NP(i,j) on the node plane P is an end node, and a plurality of nodes NP(1,j) corresponding to pixels located on the start end side of the correction vector V on the node plane P are start nodes. And Among all the routes from each start node NP(1,j) to the end node NP(i,j), the minimum cost route, which is the route having the minimum route cost, is searched by the Viterbi algorithm. Specifically, the route cost is the sum of the node costs D(p,up) of all nodes belonging to the route of interest and the disparity cost S(up,up) between adjacent nodes on the route of interest. is there. The parallax cost S(up, uq) is a cost when moving from a node having a parallax of up to a node having a parallax of uq. The parallax cost S(up, uq) is set to increase as the difference between the parallax up and the parallax uq increases. As the parallax cost S(up, uq), for example, the absolute value of the difference between the parallax up and the parallax uq can be used.

For example, on the matrix of FIG. 2, a route having the node NP(1,3) as the start node and the node NP(4,2) as the end node is shown, and four nodes NP(1 , 3), NP(2,2), NP(3,2), NP(4,2). In this case, the route cost is D(1,3)+{D(2,2)+S(2,3)}+{D(3,2)+S(2,2)}+{D(4,2) +S(2,2)}.

That is, in the correction process, the Viterbi algorithm is used to search for a combination of the node cost and the parallax cost, which is the minimum cost route. However, in this exploration, the upper limit of the parallax allowed when moving between nodes is set in advance, and the route is set within the set upper limit. Note that narrowing the upper limit of parallax leads to narrowing the route search range, and thus reducing the amount of correction processing.

Then, the route cost of the minimum cost route becomes the correction value E of the node cost at the end node. Hereinafter, the correction value obtained by the correction process using the correction vector V _X will be referred to as the correction value E _X. The same applies to other correction vectors.

The correction value of the node cost D(p,up) for the pixel specified by p does not need to calculate the route costs for all the paths each time, and the correction of the node cost for the pixel specified by p-1 is performed. It can be easily obtained from the value and the parallax cost S(up,uq). This is a known technique described in, for example, Japanese Patent Application Laid-Open No. 2015-114269 disclosed as Patent Document 1, and therefore detailed description thereof will be omitted.

[3. processing]
[3-1. Distance calculation processing]
Next, the distance calculation processing executed by the image processing unit 4 will be described with reference to the flowchart of FIG. 7. The distance calculation process is repeatedly executed during the operation of the image processing unit 4.

When this distance measurement processing is executed, the image processing unit 4 first acquires a basic image pair from the external memory 422 in S110. The basic image pair is a pair of captured images that are simultaneously captured by the pair of imaging units 2 and 3 and stored in the external memory 422.

In S120, the two images are made parallel to each other by correcting the vertical shift between the pair of basic images. Specifically, the vertical coordinates of all the pixels are converted according to a preset correction parameter so that the heights of the image areas in a corresponding relationship between the two images match. The parallelization is a process of correcting a relative posture shift between the pair of imaging units 2 and 3 and a distortion of an image due to a lens.

In S130, a parallax calculation process for generating a parallax image is executed using the basic image pair parallelized in S120. Details of the parallax calculation processing will be described later. A parallax image is an image in which the pixel value of each pixel is the corresponding relationship between each pixel that forms the reference image and each pixel that forms the comparison image, that is, the parallax that is the information that associates the place where the same image appears. is there.

In S140, the distance to the target imaged in each pixel is calculated for each pixel based on the parallax which is the information indicated by the parallax image, and a distance image having the calculated distance as a pixel value is generated. , This process ends. Note that the method of obtaining the distance from the parallax is well known, and thus the description thereof is omitted here.

The distance image is used for detecting a target object and extracting information about the detected target object, and the extracted information is used for various vehicle controls such as automatic driving and danger avoidance.
[3-2. Parallax calculation processing]
Details of the parallax calculation process executed by the image processing unit 4 in S140 will be described with reference to the flowchart of FIG. 8.

When the parallax calculation process is started, the image processing unit 4 generates a hierarchical image pair using the basic image pair parallelized in S120 in S210. Specifically, a first image pair composed of a pair of images having a first resolution lower than the resolution of the basic image pair (hereinafter, third resolution) and a second resolution lower than the third resolution and higher than the first resolution. A second image pair consisting of a pair of images having Here, as shown in FIG. 9, the image G1 having the first resolution is an image (that is, a captured image) G3 having the third resolution in which the horizontal and vertical resolutions are each ¼. .. The image G2 having the second resolution is obtained by halving the horizontal and vertical resolutions of the image G3. Hereinafter, these first to third image pairs will be generically referred to as hierarchical image pairs. The hierarchical image pair is stored in the external memory 422.

In S220, one image pair is selected as the selected image pair from the hierarchical image pairs stored in the external memory 422. However, the selected image pairs are selected in order from the one with the lowest resolution each time this step is repeated. Therefore, first the first image pair is selected, then the second image pair is selected, and finally the third image pair is selected.

In S230, the first optimization process is executed on the selected image pair selected in S220. In the first optimization process, the nodal cost is calculated and corrected, and the parallax search range used in the process for the selected image pair selected next in S220 is set based on the correction result. The details will be described later.

In S240, it is determined whether the selected image pair is a designated image pair designated in advance. The designated image is any one of the first to third image pairs. If the selected image is the designated image pair, the process proceeds to S250, and if the selected image is not the designated image pair, the process proceeds to S260.

In S250, the second optimization process is executed using the processing result of the first optimization process in S230. In the second optimization process, the node cost is corrected. The details will be described later.

In S260, it is determined whether the selected image pair is the final image pair, that is, the third image pair. If the selected image pair is the final image pair, the process proceeds to S270, and if the selected image pair is not the final image pair, that is, if it is the non-final image pair, the process returns to S220.

In S270, a parallax image is generated based on the corrected nodal cost obtained for the final image pair, and this processing ends. Specifically, a parallax image is generated by extracting the node having the lowest node cost for the pixel of interest (hereinafter, the pixel of interest) and setting the parallax corresponding to the node as the pixel value of the pixel of interest.

[3-3. First optimization processing]
Details of the first optimization process executed by the image processing unit 4 in S230 will be described with reference to the flowchart of FIG. 10 and the explanatory diagram of FIG.

When this first optimization process is executed, the image processing unit 4 firstly, in S310, for both images of the selected image pair stored in the external memory 422, the internal memory 421 for a preset number of m lines. Read into. One line refers to a pixel row arranged along the X axis on the image. m is an integer of 2 or more.

In S320, the node cost D(p,up) is calculated for each pixel in the image read in S310. As a result, as shown in FIG. 3, m nodal planes P _X relating to the pixel rows arranged in one row along the X axis are created.

In S330, using the calculation result of the node cost D(p,up) in S320, the correction process of applying the Viterbi algorithm to the right direction indicated by the correction vector V _+X is executed for each of the m node planes P _X. To do. As a result, the correction value E _+X is obtained for all the node costs D(p,up) existing in the m node planes P _X.

In S340, using the calculation result of the node cost D(p,up) in S320, a correction process of applying the Viterbi algorithm to the left direction indicated by the correction vector V _-X is executed for each of the m node planes P _X. By doing so, the correction value E _−X is obtained for all the node costs D(p,up) existing in the m number of node planes P _X.

In S350, the node cost D(p,up) of the target node is calculated for all the nodes existing in the n number of node planes P _X , and the total value E _+X +E of the correction values of the target node obtained in S330 and S340. Update the node cost D(p,up) by replacing with _-X .

In S360, the correction process of applying the Viterbi algorithm in the downward direction indicated by the correction vector V _+Y is executed using the update result of the node cost D(p,up) in S350. As shown in FIG. 4, this correction process is a process for the nodes on the node plane P _+Y regarding the pixel columns arranged in one line along the Y axis. However, here, since the image is read for m lines, that is, for m pixels along the Y axis, the correction process is executed for a part of the nodal plane P _+Y . However, as described above, the correction value of the node regarding the pixel specified by p can be obtained from the correction value of the node regarding the pixel specified by the preceding p-1. That is, if read m lines, the correction value E _{+ Y} of the node is obtained about the pixel from the starting end side of the correction vector V _{+ Y} up to m-th.

In S370, using the update result of the node cost D(p,up) in S350, the correction process of applying the Viterbi algorithm to the lower right direction indicated by the correction vector V _+X+Y is executed. As shown in FIG. 5, this correction process is a process for the nodes on the nodal plane P _XY regarding the pixel rows arranged in one row along the diagonally lower right direction toward the positive direction of the X axis and the positive direction of the Y axis. Is. The details are the same as the processing in S360, except that the nodal planes used are different. By this correction processing, the correction value E _+X+Y of the node is obtained.

In S380, using the update result of the node cost D(p,up) in S350, the correction process of applying the Viterbi algorithm to the lower left direction indicated by the correction vector V _−X+Y is executed. As shown in FIG. 6, this correction process is a process for the nodes on the nodal plane P _-XY regarding the pixel rows arranged in one row along the diagonally lower left direction toward the positive direction of the X axis and the positive direction of the Y axis. is there. The details are the same as the processing in S360, except that the nodal planes used are different. By this correction processing, the correction value E _−X+Y of the node is obtained.

In S390, the node cost D(p,up) of the target node is calculated for all the nodes for which the correction values are calculated in S360 to S380, and the total value E _+Y of the correction values of the target node calculated in S360 to S380. Update the node cost D(p,up) by substituting +E _+X+Y +E- _X+Y . In addition, regarding the node cost D(p,up) of the updated node cost D(p,up) that is not referred to in the subsequent processes in the first optimization process, the external memory 422 is used. To store.

In S400, it is determined whether the reading of the selected image pair has been completed. If the reading is not completed, the process returns to S210 to read the image for the next n lines and repeat the same processing. If the reading is completed, the process proceeds to S410.

In S410, the nodal costs D(p,up) of all the nodes NP forming the nodal space NPS are corrected vectors V _+X , V- _X , V _+Y , V _+X+Y , V- _X+Y. The correction values for the five directions shown are reflected. This corrected node cost D(p,up) is referred to, and the parallax for each pixel is obtained, and it is used when the Viterbi algorithm is applied to the selected image pair selected next from the distribution of the parallax or the like. The parallax search range is set, and this processing ends. It should be noted that the parallax detected from the selected pair of images to be selected next does not deviate significantly from the parallax detected this time, so the search range is set in a direction in which the range is limited.

[3-4. Second optimization process]
The details of the second optimization process executed by the image processing unit 4 in S250 will be described with reference to the flowchart of FIG.

When this second optimization processing is executed, the image processing unit 4 first reflects the processing result of the first optimization processing stored in the external memory 422, that is, the correction values in the five directions, in S510. The node cost D(p,up) thus selected is read into the internal memory 421 in order from the bottom of the image by a preset n lines. In other words, n nodal planes P _X relating to the pixel rows arranged in one row along the X axis are read. See FIG. 3 for the nodal plane P _X.

In S520, using the node cost D(p,up) acquired in S510, a correction process of applying the Viterbi algorithm in the upward direction indicated by the correction vector V _-Y is executed. This correction process is the same as the process in S350 except that the direction in which the correction is performed is the opposite direction, and therefore description thereof will be omitted. By this correction process, the correction vector V _-Y node of the correction value E _-Y regarding pixel to n-th from the starting end side of the can be determined.

In S530, using the node cost D(p,up) acquired in S510, a correction process of applying the Viterbi algorithm to the upper left direction indicated by the correction vector V _-XY is executed. This correction process is the same as the process in S370, except that the direction in which the correction is performed is the opposite direction, and therefore description thereof will be omitted. By this correction processing, the correction value E- _XY of the node is obtained.

In S540, the nodal cost D(p,up) acquired in S510 is used to execute a correction process in which the Viterbi algorithm is applied in the upper right direction indicated by the correction vector V _+XY . This correction process is the same as the process in S380 except that the direction in which the correction is performed is opposite, and thus the detailed description thereof is omitted. By this correction processing, the correction value E _+XY of the node is obtained.

In S550, the node cost D(p,up) of the target node is calculated for all the nodes for which the correction values are calculated in S520 to S540, and the total value E- _Y of the correction values of the target node calculated in S520 to S540. The node cost D(p,up) is updated by replacing with +E _−XY +E _+XY .

In S560, it is determined whether reading of all the node costs D(p,up) in the node space generated from the selected image pair has been completed. If the reading has not been completed, the process returns to S510, the node cost D(p,up) for the image for the next n lines is read, and the same processing is repeated. If the reading is completed, this process is completed.

In the present embodiment, the image processing unit 4 that executes the parallax calculation process corresponds to the parallax calculation device. The external memory 422 corresponds to a storage unit, and the CPU 41 and the internal memory 421 correspond to a processing unit. S220 is a selection unit, S320 is a cost generation unit, S330 to S390 and S520 to S550 are cost correction units, S410 is a restriction unit, and S270 is a parallax calculation unit. S330 is rightward processing, S340 is leftward processing, S360 is downward processing, S370 is right downward processing, S380 is left downward processing, S520 is upward processing, S530 is upper left processing, and S540 is equivalent to upper right processing. To do.

[4. effect]
According to the embodiment described in detail above, the following effects are achieved.
(1) In the range finder 1, in the parallax calculation process, if the selected image pair is not the designated image pair, only the first optimization process is performed, and only if the selected image is the designated image pair, the first optimization process is performed. In addition to the above, the second optimization processing is executed. That is, of the correction processing in the eight directions, the correction processing in the three directions that cannot be performed in parallel with the correction processing in the five directions performed in the first optimization processing is performed on one of the three selected image pairs. It applies only to pairs. Therefore, the processing load can be reduced without lowering the accuracy of the parallax calculation result and thus the distance image.

In FIG. 13, the upper left diagram is a captured image. The lower left diagram is a range image of the embodiment obtained by the above distance measuring device 1. However, the third image pair is the designated image pair. The upper right diagram is a range image of Comparative Example 1 obtained by performing both the first optimization process and the second optimization process on all of the first to third image pairs. The lower right diagram is a range image of Comparative Example 2 obtained by performing only the first optimization process on all of the first to third image pairs. In Comparative Example 2, as compared with Comparative Example 1, deterioration appears in the distance image of the vehicle portion surrounded by the broken line. On the other hand, in the example, it can be seen that no remarkable deterioration appears as compared with the comparative example 1.

(2) In the range finder 1, in the parallax calculation process, a plurality of image pairs with different resolutions are processed in order from the one with the lowest resolution, and the image pair with the next highest resolution is obtained from the processing result of the image pair with the lowest resolution. The search range is limited when the correction process is performed. Therefore, the distance measuring apparatus 1 can efficiently perform processing on an image pair having a large number of pixels and a high resolution, and the processing load can be further reduced.

[5. Other Embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be implemented.

(A) In the above embodiment, the one in which parallax is detected using the captured images obtained from the pair of image capturing units 2 and 3 has been described, but three or more image capturing units may be used.
(B) In the above embodiment, two different types of low-resolution image pairs are generated from the captured image pair, but the present invention is not limited to this. The number of low-resolution image pairs generated may be one, or three or more.

(C) In the above-described embodiment, correction processing is performed in five directions in the first optimization processing and three directions in the second optimization processing, but the present invention is not limited to this. For example, in the first optimization processing, only the correction values E _+X , E _−X , and E _+Y obtained by the correction processing in the right direction, the left direction, and the downward direction may be obtained. Further, in the second optimization process, only the correction value E _−Y obtained by the upward correction process may be obtained.

(D) In the above-described embodiment, in the first optimization process, the correction values E _+Y , E _{+X +Y} are calculated by using the node costs D(p,up) updated by the correction values E _+X , E _-X . , E _−X+Y , and the node cost D(p,up) is updated again by the calculated correction values E _+Y , E _+X+Y , E _−X+Y . It is not something that will be done. For example, all the correction values may be obtained from the node costs D(p,up) obtained in S320, and the node costs D(p,up) may be updated only once with the obtained correction values.

(E) A plurality of functions of one constituent element in the above-described embodiment may be realized by a plurality of constituent elements, or one function of one constituent element may be realized by a plurality of constituent elements. .. Further, a plurality of functions of a plurality of constituent elements may be realized by one constituent element, or one function realized by a plurality of constituent elements may be realized by one constituent element. Moreover, you may omit a part of structure of the said embodiment. Further, at least a part of the configuration of the above-described embodiment may be added or replaced with respect to the configuration of the other above-described embodiment. Note that all aspects included in the technical idea specified by the wording recited in the claims are embodiments of the present disclosure.

(F) In addition to the parallax calculation device (that is, the image processing unit 4 that executes the parallax calculation process) described above, a system including the parallax calculation device as a component, a program for causing a computer to function as the parallax calculation device, The present disclosure can also be realized in various forms such as a non-transitional physical recording medium such as a semiconductor memory recording a program and a parallax calculation method.

1... Distance measuring device, 2, 3... Imaging unit, 4... Image processing unit, 41... CPU, 42... Memory, 421... Internal memory, 422... External memory.

Claims (6)

A pair of images captured so as to include the same shooting region from different viewpoints is a basic image pair, and a pair of image pairs having a resolution different from that of the basic image pair generated from the basic image pair is the basic image pair. A storage unit (422) configured to store together with the pair as an image pair group;
A processing unit (41, 421) configured to generate a parallax image based on the image pair group stored in the storage unit;
Equipped with
The processing unit is
A selection unit (S220) configured to select one of the image pairs from the group of image pairs in ascending order of resolution;
Of the two images forming the selected image pair that is the image pair selected by the selection unit, one is the reference image and the other is the comparison image, and the two-dimensional position of the pixel in the reference image and the selection A three-dimensional point represented by a parallax represented by a pixel size unit of an image pair is a node, the node is uniquely represented, and the image size of the reference image and the maximum value of the parallax to be detected are expressed as The limited three-dimensional space is used as a nodal space, one of the nodes belonging to the nodal space is taken as a target node, and the comparison is performed with a pixel on the reference image corresponding to the target node and a parallax corresponding to the target node. A cost generation unit (S320) configured to generate a node cost calculated such that the value becomes smaller as the degree of similarity to the pixel on the image becomes higher, for all nodes belonging to the node space;
A pixel row in the node space arranged along the direction indicated by a preset correction vector is a target pixel row, and the node costs of the plurality of nodes related to the target pixel row generated by the cost generation unit are A cost correction unit (S330 to S390, S520 to S550) configured to perform correction by the correction processing using the Viterbi algorithm,
When the selected image pair finally selected by the selection unit is a final image pair, the selected image pairs other than the final image pair are non-final image pairs, and the selected image pair is the non-final image pair. A limiter configured to limit the search range in the correction processing performed by the cost corrector for the selected image pair selected next by the selector based on the correction result of the cost corrector. (S410),
When the selected image pair is the final image pair, the parallax corresponding to the node that minimizes the node cost is calculated for each pixel from the processing results of the cost generation unit and the cost correction unit. A parallax calculation unit (S270) configured in
Equipped with
When the positions of the pixels forming the selected image pair are represented by XY coordinates, the cost correction unit performs, as the correction processing, a positive direction of the X axis of the XY coordinates as a direction indicating the correction vector. Rightward processing (S330), leftward processing (S340) in which the negative direction of the X axis is the direction indicated by the correction vector, and positive direction of the Y axis of the XY coordinates is the direction indicated by the correction vector. When the designated image pair, which is any one of the image pairs designated in advance in the image pair group, is the selected image pair, the downward processing (S360) is performed. Further, an upward process (S520) in which the direction is the direction indicated by the correction vector is executed,
Parallax calculator. The parallax calculation device according to claim 1,
The cost generation unit acquires the image of the selected image pair from the storage unit, an image of one or a plurality of lines arranged in the positive direction of the X axis, every time the processing of the acquired image is completed, A parallax calculation device that sequentially moves in the positive direction of the Y-axis and sequentially acquires unprocessed images. The parallax calculation device according to claim 1 or 2, wherein
The designated image pair is the final image pair,
Parallax calculator. The parallax calculation device according to any one of claims 1 to 3,
The correction processing executed by the cost processing unit is one of a plurality of nodes related to the pixel located on the starting end side of the correction vector in the target pixel row as a start node, and all pixels included in the target pixel row. As an end node of any of the plurality of nodes with respect to, in all routes from the start node to the end node, the node costs of all the nodes located on the route, and the adjacent on the route The greater the difference in disparity between nodes is, the larger the disparity cost becomes, and the minimum cost route, which is the route with the minimum route cost, is obtained by the Viterbi algorithm, and the route cost of the minimum cost route is the end node. Obtained as a correction value for the node cost of
Parallax calculator. The parallax calculation device according to any one of claims 1 to 4,
The cost correction unit further includes, as the correction processing, a downward right processing (S370) in which an oblique direction toward the positive direction of the X axis and the positive direction of the Y axis is a direction indicated by the correction vector; And a downward left processing (S380) in which a diagonal direction toward the negative direction of the axis and the positive direction of the Y axis is the direction indicated by the correction vector,
Parallax calculator. The parallax calculation device according to any one of claims 1 to 5,
When the designated image pair is the selected image pair, the cost correction unit further performs, as the correction processing, a direction in which the correction vector indicates an oblique direction toward the positive direction of the X axis and the negative direction of the Y axis. And an upper left processing (S530) in which the correction vector indicates a diagonal direction toward the negative direction of the X axis and the negative direction of the Y axis.
Parallax calculator.