JP2004180036A - Parallax search method and parallax search device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the calculation volume and error correction in parallax search in binocular vision. <P>SOLUTION: A camera is provided in each of left and right ends in the front of a vehicle, and axial lines of cameras are aligned with the headway direction of the vehicle. For a picture of a road surface of one of two camera pictures obtained by these cameras, such conversion is previously set that the picture of a road surface after conversion is coincident with the picture of the road surface of the other camera picture in the case that there is only the road surface and no obstacle. A vanishing point is given as an origin, and an x axis is taken in the horizontal right direction on the left camera picture, and a y axis is taken in the upward direction perpendicular to the x axis, and an x' axis is taken in the horizontal right direction on the right camera picture, and a y' axis is taken in the upward direction perpendicular to the x' axis, and a is a positive constant, and x'=x+ay and y'=y are true. Thus images of obstacles and images above the horizontal are converted together. Areas of luminance incongruence in luminance difference images of individual pixels between left and right camera pictures after this conversion are taken as a search area in the right camera picture. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a parallax search method for obtaining parallax information of a pixel region having a high degree of coincidence from two captured images captured by two imaging devices whose positions and imaging directions are determined in advance. The present invention is particularly useful as one step in image analysis by binocular vision for recognizing the distance of a vehicle traveling on a road surface to an obstacle such as a preceding vehicle.
[0002]
[Prior art]
There is widely known a method of obtaining parallax information of a pixel region having a high degree of coincidence from two captured images obtained by so-called binocular vision, and determining a distance from two imaging devices whose positions and imaging directions are determined in advance. Have been. For example, see Patent Document 1 and Non-Patent Document 1.
[0003]
[Patent Document 1]
JP-A-2001-243456
[Non-patent document 1]
Xu Go and Saburo Tsuji, 3D Vision, First Edition, 2nd Press, Kyoritsu Shuppan, May 25, 1999, pp. 95-103
[0004]
This technique will be described with reference to FIG. FIG. 1 shows images captured by two cameras installed at the left end and the right end in front of the vehicle, where the imaging center is a vanishing point in the traveling direction of the vehicle, and the positions of the horizontal lines are set to coincide with each other. 1A is a left camera image, and FIG. 1B is a right camera image, in which a rear part of a preceding vehicle is photographed. Although the white line is a traveling lane marked with a road surface, it is shown by a continuous line for explanation, and the lane width is shown to be substantially equal to the width of the vehicle.
[0005]
Here, one unit is set as a reference image, and a unit area for search composed of a plurality of pixels is determined. In FIG. 1, the right camera image shown in FIG. _R (A), an area having the highest degree of matching of the luminance of the constituent pixels is searched from the unit area of the left camera image in (a).
[0006]
[Problems to be solved by the invention]
1A and 1B, the unit area A of the right camera image of FIG. _R Matches the unit area A of the left camera image in FIG. _L Can be easily understood. However, in order for the CPU to perform a parallax search, for example, the unit area A _R And the position on the vertical axis coincide with each other and a certain range is provided in the horizontal axis direction. _L It is necessary to compare the degree of coincidence with all the unit areas. By the way, for calculating the degree of coincidence, for example, a correlation value such as that described in Non-Patent Document 1 can be calculated, and many other methods are conceivable, but all of them require an enormous amount of calculation. In addition, since only the degree of coincidence for each unit area is used as a reference, erroneous correspondence that unit areas of left and right camera images that originally do not correspond correspond to each other frequently occurs.
[0007]
The present invention has been made to solve the above-described problem, and an object of the present invention is to reduce the amount of calculation and incorrect correspondence in a parallax search.
[0008]
[Means for Solving the Problems]
In order to solve the above-mentioned problem, the means according to claim 1 is configured such that one of two captured images by two imaging devices or two captured images at different positions of one imaging device is a reference image, and the other is a searched image. In a parallax search method for searching for a matching unit area having the highest degree of matching of the luminance of constituent pixels from each unit area of the searched image for each unit area of the reference image, the image conversion based on the road surface is performed. Performing a conversion on the searched image so that the converted road surface image of the searched image matches the road surface image of the reference image, and converting the searched image and each pixel of the reference image. Alternatively, a difference image based on a difference between the brightnesses of the respective regions is obtained, and a range of a unit region in the reference image corresponding to a mismatched region in which the brightness of the difference image is equal to or more than a predetermined value is set as a search region. Here, the unit area of the image is a plurality of sections of the image in which one or more pixels are set as desired, and the unit areas are designed to overlap each other, The image may be designed so as to be divided so as not to exist, or a pixel not included in any unit area may be present between the adjacent unit area. Hereinafter, the same applies to the present application. Further, in the difference image, no negative luminance is present, and the luminance difference takes an absolute value.
[0009]
According to a second aspect of the present invention, one of two captured images by two imaging devices or two captured images at different positions of one imaging device is a reference image, and the other is a searched image, and In a parallax search method for searching for a matching unit area having the highest degree of matching of the luminance of constituent pixels from each unit area of a searched image for each unit area, the image conversion is performed on the basis of a road surface. Performs a conversion such that the converted road surface image matches the road surface image of the reference image, on the searched image, and converts the searched image and the luminance of each pixel or each region of the reference image. A difference image based on the difference is obtained, and when each unit area in the reference image included in the mismatched area where the luminance of the difference image is equal to or more than a predetermined value is grouped into a unit area row perpendicular to the horizontal line, the In the case where the unit area at the lowest position is specified and the unit area at the lowest position is a road surface, the horizontal width of the corresponding search range in the search image is set to the corresponding width in the reference image. The horizontal width of the corresponding search range in the search image of all the unit areas in the unit area row is characterized. The unit area is the same as in claim 1, and the unit area row of the present invention may be designed so as to overlap each other, or may be designed so as to partition the mismatched area so that there is no overlapping part. Alternatively, a pixel that is not included in any unit area row may exist between the adjacent unit area row. Regarding the width in the horizontal direction, a search range in the search target screen is set for each unit area of the reference image in the first place, which is the reference unit area in the search target screen and the left and right in the horizontal direction from there. In the actual search operation, for each unit area of the reference image, the reference unit area in the searched screen remains unchanged, and the horizontal width from that point to the left and right width of each of the unit area columns described above. The horizontal width in the horizontal direction of the corresponding search range in the search image of the lowest unit area.
[0010]
According to a third aspect of the present invention, in the parallax search method according to the first or second aspect, the image conversion based on the road surface is performed by estimating the road surface as a plane regardless of the actual shape of the road surface. It is characterized by doing. Further, the means according to claim 4 is characterized in that the image conversion up to the determination of the mismatch area is performed at a low resolution, and the parallax search is performed at a high resolution.
[0011]
According to a fifth aspect of the present invention, one of two captured images obtained by two imaging devices or two captured images at different positions of one imaging device is set as a reference image, and the other is set as a search target image. In a parallax search device that searches for a matching unit area having the highest degree of matching of the luminance of constituent pixels from each unit area of a searched image for each unit area, an image conversion based on a road surface is performed. Image conversion means for performing a conversion on the searched image so that the converted road surface image matches the road surface image of the reference image, and each converted pixel or each of the converted searched image and the reference image. A difference image creating means for finding a difference image based on the difference in brightness between the areas; and a search area that uses a range of the unit area in the reference image corresponding to a non-match area where the brightness of the difference image is equal to or more than a predetermined value as a search area. Determining means, and having a disparity search means for searching the disparity in the search area in the reference image.
[0012]
According to a sixth aspect of the present invention, one of two captured images by two imaging devices or two captured images at different positions of one imaging device is set as a reference image, and the other is set as a search target image. In a parallax search device that searches for a matching unit area having the highest degree of matching of the luminance of constituent pixels from each unit area of a searched image for each unit area, an image conversion based on a road surface is performed. Image conversion means for performing a conversion on the searched image so that the converted road surface image matches the road surface image of the reference image, and each converted pixel or each of the converted searched image and the reference image. A difference image creating means for finding a difference image based on the difference in brightness between the areas; and a search area that uses a range of the unit area in the reference image corresponding to a non-match area where the brightness of the difference image is equal to or more than a predetermined value as a search area. Determining means, grouping each unit area in the reference image included in the mismatched area into a unit area row perpendicular to a horizontal line, and identifying a unit area at the lowest position for each unit area row; If the unit area of the position is a road surface, the width in the horizontal direction of the corresponding search range in the search image is set in the search image of all the unit areas of the unit area row in the reference image. And a parallax searching means for searching for parallax as a maximum width in the horizontal direction of a corresponding search range.
[0013]
The means according to claim 7 is the parallax search method according to claim 5 or claim 6, wherein the image change means, the difference image creation means, and the search area determination means perform the parallax search at a low resolution. In the means, image processing is performed at a high resolution.
[0014]
[Action and effect of the invention]
In the conversion of the searched image according to the present invention, since the reference image and the road surface image coincide with each other, in a state where nothing exists on the ground, the converted searched image and the reference image are in an image portion before the horizontal line. Exact match. Then, if there is a pixel whose luminance does not match between the converted image to be searched and the reference image, this is because an object exists on the ground or in the air. Therefore, in the parallax search, only the portion of the difference image between the converted image to be searched and the reference image whose luminance exceeds a predetermined value is set as the search region in the reference image, and the search region in the reference image is narrowed down, so that the entire calculation is performed. The amount can be significantly reduced (claim 1).
[0015]
According to the second aspect of the present invention, the amount of calculation can be further reduced in the search area in the narrowed reference image as follows. In other words, the unit area of the image of the object near the road surface on the reference screen and the corresponding unit area of the searched image have a very large search range in the searched image, as can be seen from the fact that there is a conversion that matches the road surface. Should be good in narrow areas. However, since it is usually not known at all that there is an object or not near the road surface, an extremely wide range, that is, a search range having an extremely long width in the horizontal direction is set. Therefore, for example, if the main obstacle in the image is a vehicle, such as a vehicle, where the portion farthest downward from the horizon is in close contact with the road surface (wheel in the case of a vehicle), the reference including the object in close contact with the road surface The unit area in the searched image that should correspond to the unit area of the image should be in a very narrow search range. Therefore, if this narrow search range is set slightly larger, the unit area in the image including a portion farther from the road surface than the object that is in close contact with the road surface also falls within the width of the slightly larger search range. Should be included. As described above, the searchable range can be extremely narrow immediately above the road surface portion (the search range in the search target image is extremely small). By setting the search range to be slightly larger, that is, by making the search range of the road surface portion larger in the first place, it is possible to search for a corresponding unit area from a narrow search range even in a portion distant from the road surface. Further, even in a state where there is an overlap with another object, the distance to be searched for can be narrowed because the distance should be longer than the distance of the portion in close contact with the road surface. Thereby, it is also possible to suppress erroneous responses (claim 2).
[0016]
Since the converted image and the difference image are for narrowing the search area in the reference image, the road surface is always assumed to be a plane and parameters are assumed (claim 3). Even if the dropped object is used (claim 4), no problem occurs by increasing the resolution in the actual parallax search. With these, it is possible to further suppress the amount of calculation (claims 3 and 4).
[0017]
Such a parallax search method can be configured as an apparatus as it is (claims 5 to 7).
[0018]
BEST MODE FOR CARRYING OUT THE INVENTION
The present invention will be described with reference to the drawings. Note that the present invention is not limited to the following embodiments. Hereinafter, it is assumed that the cameras are installed on the left and right sides in front of the vehicle, the left and right camera images have the vanishing points of the cameras in the vehicle traveling direction, and the positions of the horizontal lines coincide with each other.
[0019]
It is assumed that the left and right camera images are obtained as shown in FIGS. Here, if there is no state other than the road display on the ground, it is as shown in FIGS. 2A and 2B. In the left camera image of FIG. 2A, the left display of the traveling lane in which the vehicle travels is LL. _L , Right display LR _L And In the right camera image of FIG. 2B, the left display of the traveling lane in which the vehicle travels is indicated by LL. _R , Right display LR _R And 2A and 2B are superimposed, and only the edge line and the horizontal line of each lane are shown in FIG. 3A. Here, those having edges indicated by two solid lines are the left display LL of the traveling lane of the left camera image in FIG. _L , Right display LR _L , And each having two dashed edges is the left display LL of the traveling lane in the right camera image of FIG. 2B. _R , Right display LR _R It is. As is clear from this, in order to convert (a) in FIG. 2 and overlap with (b) in FIG. 2, the following image conversion may be performed. However, here, the origin is the vanishing point, the x-axis is taken horizontally and rightward on the left camera image, the y-axis is taken vertically upward on the x-axis, and the x'-axis and right-hand upward on the right camera image. Takes the y 'axis, and a is a positive constant.
[0020]
(Equation 1)
x '= x + ay,
y '= y
[0021]
If the relationship of Equation 1 is applied to a portion above the horizontal line, the result is as shown in FIG. Here, it should be pointed out that the length of the arrow shown below the horizontal line in FIG. 3A and FIG. 3B, that is, the left camera image is adjusted so that the road surface matches the right camera image. As for the object to be converted, the x coordinate becomes larger as the distance of the road surface from the camera becomes shorter. Now, it is assumed that a plane is set up perpendicularly to the lane, that is, perpendicular to the axis of the camera, and the pattern on the plane is captured by the left camera image and the right camera image. Here, when converting the left camera image and superimposing it on the right camera image, if all the patterns on the plane are not converted to the same amount as the conversion amount at the portion where the plane perpendicular to the lane intersects the road surface, the left camera image The pattern on the plane of the converted image does not match the pattern in the right camera image. Then, as shown in FIG. 3B, regardless of the distance from the camera, the image below the horizontal line of the left camera image is converted in the same manner as “converting the road surface” and compared with the right camera image. In addition, an image of an obstacle or the like other than the road surface and a portion that is in close contact with the road surface always causes a shift between the left camera image and the right camera image after the conversion. In other words, when the left camera image is converted so that the road surface of the left camera image coincides with that of the right camera image in the left and right camera images obtained by capturing the three-dimensional object and the road surface in unison, the true Is a distorted image.
[0022]
Therefore, when the left camera image of FIG. 1A is converted according to the relationship of Expression 1 (the constant a is determined so that the road surface of the converted left camera image matches the road surface of the right camera image), the result is as shown in FIG. The image of the preceding vehicle is deformed such that the upper part is inclined rightward from the wheel portion closely contacting the road surface. Next, when a difference image based on a difference (absolute value) between the luminance of the converted image of FIG. 4 and the right camera image of FIG. Here, the difference between the luminance of the portion where the luminance of the body and the rear glass does not change is close to 0 even if the images are different from each other. In this way, as shown in FIG. 6, the search area SA in the right camera image of FIG. _R To determine. Search area SA in FIG. _R Is extremely narrower than the conventional search area in the right camera image in FIG. 1B, and the amount of calculation in the parallax search can be suppressed.
[0023]
Further, as shown in FIG. 7B, the search area SA in FIG. _R To each unit area (for example, A _R ), And the vertical unit area row C _R Is divided into Here, the unit area row C _R Unit area A at the lowermost position farthest from the horizontal line _SR Includes an image of a wheel that is in close contact with the road surface, and the search range R in the left camera image of FIG. _SL Is the unit area A _SL Has a horizontal width in the horizontal direction on the basis of. For each unit area in FIG. 7B, the corresponding unit area in FIG. 7A and the horizontal width in the horizontal direction are originally set based on Equation 1 assuming that the unit area is a road surface. , This is referred to as each unit area row C _R The unit area A at the lowermost position where the horizontal width in the horizontal direction is farthest from the horizontal line _SR 7 (a) in FIG. 7 (a). This makes it possible to narrow the search range of the left camera image with respect to the unit area of the right camera image at a position close to the horizontal line having the wider left and right width of the search range. This is because each unit area row C of the front obstacle _R The distance from the left and right cameras of the portion grasped by the unit area A having the image of the wheel in close contact with the road surface _SR This is based on the premise that there is no great difference from the distance from the left and right cameras of the part grasped by the above, and even if there is an overlap with another object, it is farther than the distance. As a result, the amount of calculation can be further suppressed, and erroneous correspondence in binocular vision of a vehicle or the like having many similar parts in the horizontal direction can be suppressed.
[0024]
The present invention described above can be specifically implemented by a memory and a CPU in software. Further, the present invention can also be implemented in a form of processing by hardware such as a system LSI. This will be described with reference to a flowchart. In the following, in the text, "step 102 etc." will be described as "S102 etc." in the figure, and in the text, "left camera image", "right camera image", "left and right camera image" will be described. Are described as “left image”, “right image”, and “left and right image” in the figure. The left and right cameras are set and adjusted as described above, and the left and right camera images are both images in which the horizontal line is the x-axis and the vanishing point is the origin.
[0025]
[First embodiment]
This embodiment is configured by the flowcharts of FIGS. First, in step 102 of FIG. 8, the luminance of each pixel of the left and right camera images is read and stored. Hereinafter, the left and right camera images are constituted by the read luminance of each pixel until the end of this flowchart.
[0026]
Next, in step 104, a converted image is created from the left camera image by the above equation (1). Next, in step 106, a difference image is created in which the absolute value of the difference between the luminance of each pixel of the converted image and the luminance of each pixel of the right camera image at the same coordinate is set as the luminance of the pixel at that coordinate. Next, in step 108, a flag is set for the coordinates of the pixels in the difference image whose luminance is greater than a certain set value, and the search area is determined. Next, the procedure moves to the flowchart of FIG.
[0027]
In step 110 of FIG. 9, the unit area is determined based on whether or not the difference image includes a flagged pixel in the difference image, and the difference image includes the flagged pixel in the difference image. A unit area group (including the search area) is determined. Hereinafter, this unit area group is referred to as a “search area”. Next, in step 112, a unit area as an initial value is determined from the unit area group. For example, assuming that each unit area is rectangular, the unit areas are arranged and numbered according to the coordinates of the pixel at the lower left corner, and the first numbered unit area is determined.
[0028]
Next, in step 114, a unit area in the left image and a search range for the unit area in the left image which are stored in advance corresponding to the unit area in the search area of the right camera image are read. For example, the coordinates of the pixel at the lower left corner of the unit area in the left image (x ₀ , Y ₀ ) And -x in the x-axis direction. ₁ , + X ₂ (X ₁ , X ₂ Is a non-negative integer), and the search range is x coordinates of the pixel at the lower left corner. ₀ -X ₁ To x ₀ + X ₂ And the y coordinate is y ₀ Is a set of unit areas. The upper limit x of the x coordinate of this search range ₀ -X ₁ And lower limit x ₀ + X ₂ Is stored. Next, in step 116, the initial value of the unit area in the searched range of the left camera image is determined. For example, if the x coordinate of the pixel at the lower left corner is x ₀ -X ₁ , The y coordinate of the left camera image is y ₀ Is a unit area.
[0029]
Next, in step 118, the degree of coincidence between the luminance of each pixel in the unit area in the search area of the right camera image and the luminance of each pixel in the unit area in the search range of the left camera image is calculated. Is stored together with the coordinates of the pixel at the lower left corner of the unit area. This degree of coincidence may be the calculation method described in Non-Patent Document 1 described above, and is arbitrary. Next, in step 120, the x coordinate of the pixel at the lower left corner of the unit area of the left camera image for which the degree of coincidence has been calculated is the upper limit value (x ₀ + X ₂ ) Is determined. The x coordinate of the pixel at the lower left corner of the unit area of the left camera image for which the degree of coincidence has been calculated is the upper limit value (x ₀ + X ₂ If not, the process moves to step 122, and the unit area is updated within the searched range of the left camera image. For example, the x coordinate of the pixel at the lower left corner of the unit area is increased by one. Thereafter, the process returns to step 118 to repeat this processing. On the other hand, if it is determined in step 120 that the x coordinate of the pixel at the lower left corner of the unit area of the left camera image for which the degree of coincidence has been calculated is the upper limit, the unit area of the left camera image having the highest degree of coincidence is replaced with the right camera It is determined that they match the unit area of the image, and the x coordinate of the pixel at the lower left corner of the unit area of the left camera image is stored. Next, in step 126, the parallax for the unit area of the right camera image is determined from the difference between the x coordinate and the y coordinate of the pixel at the lower left corner of the corresponding unit area of the right camera image and the unit area of the left camera image. It is calculated and stored together with the coordinates of the pixel at the lower left corner of the unit area of the right camera image.
[0030]
Next, the process proceeds to step 128, where it is determined whether or not the unit area of the right camera image for which the parallax has been calculated is the last unit area in the search area. If it is not the last unit area, the process moves to step 130, and the unit area in the right camera image is updated to the next unit area. For example, the coordinates of the pixel at the lower left corner of the unit area are updated, and the process returns to step 114 to repeat the processing. On the other hand, if it is determined in step 128 that this is the last unit area in the search area, the parallax has been calculated for all the unit areas in the search area in the right camera image, and the process ends.
[0031]
[Second embodiment]
In the present embodiment, the first half is configured by the flowchart of FIG. 8, and the second half is configured by the flowchart of FIG. Note that the flowchart of FIG. 10 is partially shown as a subroutine in FIG. 11 for the sake of space. First, similarly to the first embodiment, the processing of steps 102 to 108 is performed.
[0032]
Next, in step 202, similarly to step 110 of the first embodiment, a unit area group (search area) including the search area in the right camera image is determined. Next, the search area is divided into unit area rows in which the x coordinate of the pixel at the lower left corner is the same. Here, numbers are assigned to all unit area columns, unit area numbers are assigned inside each unit area column, and the processing order is determined later. Next, the process proceeds to step 206, where the initial value of the unit area row, for example, the unit area row having the smallest x coordinate of the pixel at the lower left corner is selected as the initial value.
[0033]
Next, in step 208, the y-coordinate of the pixel at the lower left corner in the unit area sequence of the right camera image is the smallest (lowest tier) unit area A. _SR Is determined. Next, in step 210, the unit area A of the right camera image _SR Unit area A of the left camera image corresponding to _SL And the search range for that is read out. This specific example is the same as that of step 114 of the first embodiment (FIG. 9). Unit area A of right camera image _SR To be searched R of the left camera image corresponding to _SL Is calculated and stored in advance as described below, for example.
(Equation 2)
x ₀ = X,
y ₀ = Y,
x ₁ = 0,
x ₂ = Ay
[0034]
Here, the unit area A of the right camera image _SR Is the lower left corner coordinate of (x, y), the search range R of the left camera image _SL Of the lower left corner of (x ₀ , Y ₀ ), The width to the left is x ₁ , Right width x ₂ , A are the same as a in Equation 1. Thus, the condition for the search range is obtained, and the process proceeds to the parallax calculation subroutine (SubR.).
[0035]
FIG. 11 shows the content of the parallax calculation subroutine (SubR. In FIG. 10). First, in step 302, the unit area A in the right image _R Unit area A in the left camera image corresponding to _L Confirm. Next, in step 304, the lowermost unit area A in the unit area row of the right camera image _SR Unit area A of the left camera image corresponding to _SL Is set to the unit area A in the left camera image _L Apply to Thus, the search range is determined. Next, in step 306, the initial value of the unit area within the search range is determined. For example, the x coordinate is the lower limit of the search range.
[0036]
Hereinafter, the processing of steps 308 to 316 is the same as that of steps 118 to 126 of the first embodiment (FIG. 9). That is, in step 308, the degree of coincidence between the luminance of each pixel in the unit area in the search area of the right camera image and the luminance of each pixel in the unit area in the search range of the left camera image is calculated. Is stored together with the coordinates of the pixel at the lower left corner of the unit area. Next, in step 310, it is determined whether or not the x coordinate of the pixel at the lower left corner of the unit area of the left camera image for which the degree of coincidence has been calculated is the upper limit value. If the x coordinate of the pixel at the lower left corner of the unit area of the left camera image for which the degree of coincidence has been calculated is not the upper limit value, the process proceeds to step 312, where the unit area is updated within the searched range of the left camera image, and the process returns to step 308. This process is repeated. On the other hand, if it is determined in step 310 that the x coordinate of the pixel at the lower left corner of the unit area of the left camera image for which the degree of coincidence has been calculated is the upper limit value, the unit area of the left camera image having the highest degree of coincidence is set to the right camera It is determined that they match the unit area of the image, and the x coordinate of the pixel at the lower left corner of the unit area of the left camera image is stored. Next, in step 316, the parallax for the unit area of the right camera image is determined from the difference between the x coordinate and the y coordinate of the pixel at the lower left corner of the corresponding unit area of the right camera image and the unit area of the left camera image. It is calculated and stored together with the coordinates of the pixel at the lower left corner of the unit area of the right camera image. Thus, the parallax calculation subroutine ends.
[0037]
Next, in FIG. 10, when the parallax calculation subroutine (SubR.) Ends, the process proceeds to step 212, where it is determined whether or not the unit area of the right camera image for which the parallax has been calculated is the uppermost unit area of the unit area row. If the unit area of the right camera image for which the parallax has been calculated is not the uppermost unit area of the unit area row, the y coordinate of the pixel at the lower left corner of the unit area is increased by 1, and the parallax calculation subroutine (SubR.) Is started. , And repeat the process. On the other hand, if the unit area of the right camera image for which the parallax has been calculated is the uppermost unit area of the unit area row, the process proceeds to step 216, and it is determined whether the current unit area row is the last unit area row. If the current unit area row is not the last unit area row, the process proceeds to step 218, and the unit area row with the x coordinate increased by 1, for example, is set as a new unit area row as the unit area row, and the processing from step 208 onward is repeated. On the other hand, if the current unit area sequence is the last unit area sequence, the process ends.
[0038]
(Modification)
FIG. 12 is a partial modification of FIG. 8, and is configured as a modification of the first and second embodiments. That is, FIG. 12 shows the first half as a modification of the first embodiment, and FIG. 9 shows the second half. FIG. 12 shows the first half of the modification of the second embodiment, and FIG. It is configured as the latter half. In this modification, the process up to the determination of the search area is based on the low-resolution image. The constant resolution images of the left and right camera images will be referred to as coarse left and coarse right camera images.
[0039]
First, in step 402, the left and right camera images are read and stored. Next, in step 404, the left and right camera images are sequentially read out, converted into low resolution images, and separately stored as coarse left and coarse right camera images. The conversion to a low-resolution image may be performed by averaging the luminance for each of 16 rectangular pixels, for example, 4 × 4 pixels, and forming an image having 1/16 of the number of pixels as one pixel. At this time, the original left and right camera images are not deleted.
[0040]
Next, in step 406, a rough converted image is created from the rough left camera image by the conversion formula of the above equation (1). If the creation of the low-resolution image does not average the luminance of the square pixels, the constant a in the conversion formula of Equation 1 is adjusted accordingly. Next, in step 408, a coarse difference image is created in which the absolute value of the luminance difference between each pixel of the coarsely converted image and the coarse right camera image is set as the luminance of the pixel. Next, in step 410, a search area is determined from the coarse difference image. Here, the search area is first set as the coordinates of the pixels in the coarse difference image, as in step 108 of FIG. 8 in the first and second embodiments, and then as the coordinates of the pixels in the right camera image with high resolution. Is set. For example, 16 pixels of the right camera image correspond to one pixel in the coarse difference image, and all 16 pixels of the right camera image are flagged.
[0041]
The following processing is performed in the same manner as the processing in FIGS. 9 and 10 (and FIG. 11). In the present modified example, the conversion image, the difference image, and the search area determination are processed by the constant resolution image, so that the processing time can be reduced, the storage capacity can be reduced, and the processing device can be speeded up and downsized. .
[Brief description of the drawings]
1A is an image including a preceding vehicle from a camera fixed to the front left of the vehicle, and FIG. 1B is an image including a preceding vehicle from a camera fixed to the front right of the vehicle.
2A is an image of only the ground surface with a driving lane display without an obstacle from a camera fixed to the front left of the vehicle, and FIG. 2B is an image of an obstacle from a camera fixed to the front right of the vehicle. No, only ground surface image with driving lane display.
3A is an explanatory diagram of road surface image conversion by showing the edge line of FIG. 2A by a solid line and the edge line of FIG. FIG. 2B is an explanatory diagram of the conversion in which the road surface image conversion of FIG.
FIG. 4 is a converted image of FIG. 1A by the conversion (Equation 1) of FIG. 3B.
FIG. 5 is a difference image between FIG. 4 and FIG. 1 (b).
FIG. 6 is a diagram showing an area where the luminance is equal to or more than a predetermined value in FIG. 5 on FIG.
FIG. 7A is an explanatory diagram showing each search range in the left camera image, and FIG. 7B is an explanatory diagram showing that the search range in the right camera image is divided into unit area columns.
FIG. 8 is the first half of a flowchart in the first and second embodiments.
FIG. 9 is the latter half of the flowchart in the first embodiment.
FIG. 10 shows the second half of the flowchart in the second embodiment.
FIG. 11 is a flowchart of a matching degree calculation subroutine in the second embodiment.
FIG. 12 is a first half of a flowchart in a modification of the first and second embodiments.
[Explanation of symbols]
The subscript " _L Shows the one in the left camera image.
The subscript " _R Shows the one in the right camera image.
A _L , A _R Unit area for parallax search
R _L Unit area A in left camera image _L Searched range for
LL _L , LL _R Left lane captured in left and right camera images
LR _L , LR _R Right lane captured in left and right camera images
SA _R Search area in right camera image
C _R Each unit area sequence that divides the search area in the right camera image
A _SR The unit area at the bottom position in a certain unit area row in the right camera image
A _SL Unit area A in right camera image _SR Unit area in the left camera image associated with
R _SL Unit area A in left camera image _SL Searched range for
R ' _L Unit area A in left camera image _L Search range modified for

Claims (7)

One of two captured images by two imaging devices or two captured images at different positions of one imaging device as a reference image, and the other as a search image, each unit region of the search image for each unit region of the reference image. In a parallax search method of searching for a matching unit area having the highest degree of matching of luminance of constituent pixels from the middle,
An image conversion based on the road surface, and performs a conversion on the searched image so that the converted road surface image of the searched image matches the road surface image of the reference image,
The converted image to be searched, and a difference image based on the difference in luminance between each pixel or each region of the reference image,
A disparity search method, wherein a range of a unit area in the reference image corresponding to a non-coincidence area in which the luminance of the difference image is equal to or greater than a predetermined value is set as a search area. One of two captured images by two imaging devices or two captured images at different positions of one imaging device as a reference image, and the other as a search image, each unit region of the search image for each unit region of the reference image. In a parallax search method of searching for a matching unit area having the highest degree of matching of luminance of constituent pixels from the middle,
An image conversion based on the road surface, and performs a conversion on the searched image so that the converted road surface image of the searched image matches the road surface image of the reference image,
The converted image to be searched, and a difference image based on the difference in luminance between each pixel or each region of the reference image,
When each unit area in the reference image included in the non-coincidence area where the brightness of the difference image is equal to or more than a predetermined value is grouped into a unit area column perpendicular to the horizontal line,
For each unit area row, the lowest position unit area is specified, and when the lowest position unit area is the road surface, the horizontal width of the corresponding search range in the search image is defined as A parallax search method, wherein a maximum width in a horizontal direction of a corresponding search range in a search image of all unit regions of the unit region row in a reference image is set. The parallax search method according to claim 1, wherein in the image conversion based on the road surface, the road surface is estimated as a plane regardless of the actual shape of the road surface. The parallax search method according to claim 1, wherein the image conversion up to the determination of the mismatch area is performed at a low resolution, and the parallax search is performed at a high resolution. One of two captured images by two imaging devices or two captured images at different positions of one imaging device as a reference image, and the other as a search image, each unit region of the search image for each unit region of the reference image. In a parallax search device that searches for a matching unit area having the highest degree of matching of the luminance of constituent pixels from inside,
An image conversion unit that performs image conversion based on a road surface and performs conversion on the search target image such that the converted road surface image of the search target image matches the road surface image of the reference image.
The converted image to be searched, and a difference image creating means for finding a difference image based on a difference in luminance between each pixel or each region with the reference image,
A search area determining unit that sets a range of a unit area in the reference image as a search area, which corresponds to a mismatch area in which the luminance of the difference image is equal to or greater than a predetermined value,
A disparity search unit that searches for a disparity in a search area in the reference image. One of two captured images by two imaging devices or two captured images at different positions of one imaging device as a reference image, and the other as a search image, each unit region of the search image for each unit region of the reference image. In a parallax search device that searches for a matching unit area having the highest degree of matching of the luminance of constituent pixels from inside,
An image conversion unit that performs image conversion based on a road surface and performs conversion on the search target image such that the converted road surface image of the search target image matches the road surface image of the reference image.
The converted image to be searched, and a difference image creating means for finding a difference image based on a difference in luminance between each pixel or each region with the reference image,
A search area determining unit that sets a range of a unit area in the reference image as a search area, which corresponds to a mismatch area in which the luminance of the difference image is equal to or greater than a predetermined value,
Grouping each unit area in the reference image included in the mismatched area into a unit area column perpendicular to the horizontal line,
For each unit area row, the lowest position unit area is specified, and when the lowest position unit area is the road surface, the horizontal width of the corresponding search range in the search image is defined as A disparity search device, comprising: a disparity search unit that searches for a disparity as a maximum width in a horizontal direction of a corresponding search range in a search image of all unit regions of the unit region sequence in the reference image. 7. The image processing unit according to claim 5, wherein the image changing unit, the difference image creating unit, and the search area determining unit perform image processing at low resolution, and the parallax searching unit performs high-resolution image processing. 8. Parallax search method.

Images