JP4053280B2 - Image processing apparatus and image processing method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and an image processing method for performing stereo image processing using a pair of color images.
[0002]
[Prior art]
A stereo-type vehicle exterior monitoring system using a stereo camera is known. In this system, a pair of captured images obtained from a pair of cameras (stereo cameras) is used to identify small regions having a correlation in luminance characteristics by stereo matching, and distance data is obtained from the relative deviation amount between the two. obtain. A conventional stereo type outside vehicle monitoring system employs a black and white image sensor (CCD, CMOS sensor, etc.), and recognizes the situation outside the vehicle based on luminance information in a monochrome image.
[0003]
[Problems to be solved by the invention]
However, if the stereo image processing using not only a monochrome image but also a color image as input information and color information as well as luminance information is performed, the reliability of distance data can be expected to be improved. At the same time, the recognition accuracy of the driving situation and the more complicated situation recognition become possible. However, in the past, when performing stereo image processing using a color image, there has not been sufficient study on how to process a plurality of primary color images constituting a color image and what information is extracted. It was.
[0004]
In recent years, with the development of technology related to image processing, colorization has progressed in various fields that handle images, and the cost of color image sensors has become even lower. Therefore, the adoption of a color image sensor is also desired in a stereo-type outside monitoring system from the viewpoint of cost.
[0005]
The present invention has been made in view of such circumstances, and an object thereof is to improve the reliability of distance data by performing stereo image processing using a pair of color images.
[0006]
[Means for Solving the Problems]
In order to solve such a problem, the first invention provides an image processing apparatus that performs stereo matching using a pair of color images each composed of a plurality of primary color images. This image processing apparatus has stereo calculation means and fusion processing means. The stereo calculation means calculates the primary color parallax for each primary color image and the city block distance for these primary color parallaxes by performing stereo matching using the primary color image pairs of the same color. These primary color parallax and city block distance are calculated for each image area which is a parallax calculation unit. The fusion processing means calculates a unified parallax for each image area based on each primary color parallax. At the same time, the fusion processing means outputs reliability information indicating the reliability of at least one of the primary color parallax and the city block distance in association with the unified parallax.
[0007]
Here, in the first invention, when the fusion processing means determines that the reliability related to one image area is high, it outputs a unified parallax related to the image area and also has reliability related to the one image area. If it is determined that it is not high, it is preferable not to output a uniform parallax for this image area in order to reduce the amount of calculation.
[0008]
In the first invention, it is preferable that the fusion processing unit evaluates the reliability of at least one of primary color parallax and city block distance according to a predetermined evaluation criterion. This evaluation criterion preferably includes at least a first individual condition for determining whether or not respective values of the primary color parallax match. In addition, it is preferable that this evaluation criterion includes at least a second individual condition for determining whether each of the city block distances is smaller than a predetermined determination threshold value. Furthermore, this evaluation criterion preferably includes at least a third individual condition for determining whether or not the sum of city block distances is smaller than a predetermined determination threshold value.
[0009]
In the first invention, it is desirable that the fusion processing means selects and applies one of the evaluation criteria prepared in advance according to the color of the object to be detected. .
[0010]
Furthermore, in the first invention, in order to calculate the luminance value of the reference pixel excluding the influence of the positional shift of the corresponding actual pixel in each primary color image, the luminance of the actual pixel overlapping the reference pixel is calculated. Correction means for calculating the luminance value of the reference pixel by a weighted average of the values may be further provided.
[0011]
The second invention provides an image processing apparatus that performs stereo matching using a pair of color images each composed of a plurality of primary color images. This image processing apparatus includes a correction unit and a stereo calculation unit. The correction means calculates the weighted average of the luminance values of the real pixels overlapping the reference pixel in order to calculate the luminance value of the reference pixel in each primary color image excluding the influence of the positional deviation of the corresponding real pixel. To calculate the luminance value of the reference pixel. Further, the stereo calculation means performs primary matching using the primary color image pair of the same color corrected by the correction means, thereby calculating the primary color parallax relating to each primary color image for each image area serving as a parallax calculation unit. .
[0012]
The third invention provides an image processing method for performing stereo matching using a pair of color images each composed of a plurality of primary color images. In this image processing method, stereo matching is performed using primary color image pairs of the same color, so that the primary color parallax relating to each primary color image and the city block distance relating to these primary color parallaxes are calculated for each image area as a parallax calculation unit. Based on the first step and the respective primary color parallax, the unified parallax for each image area is calculated, and the reliability information indicating the reliability of at least one of the primary color parallax and the city block distance is compatible with the unified parallax. And a second step of outputting.
[0013]
Here, in the second step, when it is determined that the reliability regarding the one image area is high, the unified parallax regarding this image area is output, and when it is determined that the reliability is not high, this image is displayed. It is preferable not to output a uniform parallax for the region.
[0014]
In the second step, it is desirable to evaluate the reliability of at least one of primary color parallax and city block distance according to a predetermined evaluation criterion. This evaluation criterion preferably includes at least a first individual condition for determining whether or not respective values of the primary color parallax match. In addition, it is preferable that this evaluation criterion includes at least a second individual condition for determining whether each of the city block distances is smaller than a predetermined determination threshold value. Furthermore, this evaluation criterion preferably includes at least a third individual condition for determining whether or not the sum of city block distances is smaller than a predetermined determination threshold value.
[0015]
In addition, the second step may include a step of selecting and applying any one of a plurality of evaluation criteria prepared in advance according to the color of the object to be detected.
[0016]
Further, in the third invention, in each primary color image, in order to calculate the luminance value of the reference pixel excluding the influence of the positional shift of the corresponding actual pixel, the luminance of the actual pixel overlapping the reference pixel is calculated. A third step of calculating the luminance value of the reference pixel by a weighted average of values may be further provided.
[0017]
The fourth invention provides an image processing method for performing stereo matching using a pair of color images each composed of a plurality of primary color images. In this image processing method, in each primary color image, in order to calculate the luminance value of the reference pixel excluding the influence of the positional shift of the corresponding actual pixel, the luminance value of the actual pixel overlapping the reference pixel is calculated. By performing stereo matching using the first step of calculating the luminance value of the reference pixel by weighted average and the corrected primary color image pair of the same color, the primary color parallax for each primary color image is calculated as a parallax calculation unit. And a second step of calculating for each image area.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
(First embodiment)
FIG. 1 is a schematic block diagram of a stereo image processing apparatus mounted on a vehicle such as an automobile. This image processing device uses a plurality of primary color images (red image, green image, and blue image) constituting a color image as targets of stereo matching, calculates a parallax independently for each primary color image, (1) Evaluate the reliability of the parallax and (2) fuse the parallax. Such parallax evaluation / fusion is performed in consideration of the color of the object projected in the captured image. Hereinafter, the red image is referred to as “R image”, the green image as “G image”, and the blue image as “B image”.
[0019]
A stereo camera for imaging the front of the vehicle is attached in the vicinity of a room mirror, and is composed of a pair of cameras 1 and 2 incorporating an image sensor such as a CCD or CMOS sensor. The cameras 1 and 2 are attached at a predetermined interval (camera base line length) in the vehicle width direction. The main camera 1 that outputs a reference image is attached to the right side in the traveling direction of the vehicle. On the other hand, the sub camera 2 that outputs the comparison image is attached on the left side in the traveling direction.
[0020]
Each of the cameras 1 and 2 includes a separate image sensor for each of red, green and blue (RGB) (for example, a three-plate color CCD). The reference image output from the main camera 1 is actually composed of three primary color images of an R image, a G image, and a B image. Similarly, the comparison image output from the sub camera 2 is also composed of three primary color images of an R image, a G image, and a B image. Therefore, a total of six images are output by the pair of cameras 1 and 2 at one shooting timing. Hereinafter, the R image constituting the reference image may be referred to as a reference image (R). Similarly, the G image constituting the reference image may be referred to as a reference image (G), and the B image may be referred to as a reference image (B). As for the comparative image, the R, G, and B images may be referred to as a comparative image (R), a comparative image (G), and a comparative image (B), respectively.
[0021]
In a state where the pair of cameras 1 and 2 are synchronized, the analog images output from the cameras 1 and 2 are adjusted in the analog interface 3 so as to match the input range of the circuit in the subsequent stage. A gain control amplifier (GCA) 3a in the analog interface 3 adjusts the brightness balance of the paired analog image signals. This adjustment is performed between images of the same color (R, G, B). The analog image pair adjusted in the analog interface 3 is converted by the A / D converter 4 into a digital image having a predetermined luminance gradation (for example, a gray scale of 256 gradations).
[0022]
The correction circuit 5 performs brightness correction, image geometric conversion, and the like for each of the six primary color images constituting the digitized image pair. Usually, there is an error in the mounting position of the pair of cameras 1 and 2 to some extent, and a shift caused by the difference exists in the left and right images. Therefore, geometric transformation such as image rotation and translation is performed using affine transformation or the like. This guarantees the coincidence of horizontal lines (epipolar lines) in the image pair, which is a precondition for performing stereo matching. Through such image processing, from the output signal of the main camera 1, as an example, reference image data of 512 pixels (pixels) in the horizontal direction (i coordinate direction) and 200 pixels (pixels) in the vertical direction (j coordinate direction) Is generated. Further, from the output signal of the sub camera 2, comparison image data having the same vertical length as the reference image data and a horizontal length longer than the reference image data is generated (for example, 640 pixels in the horizontal direction). The vertical direction is 200 pixels (pixels)). The reference image data corresponding to one frame corrected by the correction circuit 5 and the comparison image data are output to the stereo operation circuit 6 and stored in the image data memory 7.
[0023]
The stereo calculation circuit 6 performs stereo matching based on the reference image data and the comparison image data, and calculates the parallax for each image region (pixel block) that is a unit for calculating the parallax. This stereo matching is performed separately for the primary color image pair of the same color. That is, R image (reference image (R) and comparison image (R)), G image ((reference image (G) and comparison image (G)), B image (reference image (B) and comparison image (B)) For each of them.
[0024]
In the present embodiment, as an example, a pixel block of 4 × 4 pixels is used as a parallax calculation unit. Therefore, in the entire image corresponding to one frame, the number of pixel blocks included in the reference image, that is, 128 × 50 parallaxes can be calculated. When one pixel block in the reference image is a target for stereo matching, an area having a correlation with the luminance characteristic of the target pixel block is specified by searching the comparison image. As is well known, the distance to an object (for example, a preceding vehicle or a lane) projected on a stereo image is uniquely specified by the horizontal displacement (parallax) of the object between the reference image and the comparison image. Is done. Therefore, when searching for a comparison image, the search range may be the same horizontal line (epipolar line) as the j coordinate of the target pixel block. The stereo calculation circuit 6 evaluates the correlation of the luminance characteristic with the target pixel block for each pixel block on the comparison image side while shifting the epipolar line one pixel at a time.
[0025]
The correlation of the luminance characteristics between the pixel blocks can be evaluated, for example, by calculating the city block distance. Basically, the pixel block having the smallest value is a correlated pixel block. Then, a horizontal shift amount between pixel blocks that are correlated with each other (in other words, the city block distance is minimized) is output as parallax. Regarding the hardware configuration for calculating the city block distance, Japanese Patent Laid-Open No. 5-114099 Please refer to it if necessary.
[0026]
The stereo arithmetic circuit 6 performs stereo matching between the primary color images of the same color (for example, the reference image (R) and the comparison image (R)), and outputs the parallax and the city block distance to the subsequent fusion processing unit 8. Thereby, three parallaxes (primary color parallaxes) and three city block distances are calculated for one pixel block constituting the reference image. Hereinafter, the primary color parallax calculated from the R image is denoted by Nr, and the city block distance (the minimum value of the city block distance within the search range) relating to the primary color parallax Nr is denoted by Mr. Similarly, the primary color parallax calculated for the G image and the city block distance related to the primary color parallax are denoted as Ng and Mg, respectively, and the primary color parallax calculated from the B image and the city block distance related to the primary color parallax are denoted as Nb and Mg, respectively. . Note that, since the processing amount of the stereo matching process is increased as compared with the case of using a monochrome image, an arithmetic circuit having a sufficient processing speed is adopted as the stereo arithmetic circuit 6.
[0027]
The fusion processing unit 8 performs fusion after evaluating reliability using the three primary color parallaxes Nr, Ng, and Nb and the city block distances Mr, Mg, and Mb with respect to a certain pixel block. The parallax Ni (hereinafter referred to as “unified parallax Ni”) is calculated. The set for one frame of the unified parallax Ni thus obtained is associated with the position (i, j) on the image plane, and stored in the distance data memory 14 as distance data (i, j, Ni). Stored. As will be described later, reliability information indicating the reliability of primary color parallax Nr, Ng, Nb and city block distances Mr, Mg, Mb is attached to each unified parallax Ni. Are stored in the distance data memory 14 in a form associated with.
[0028]
The microcomputer 15 (a recognition unit 16 that is a functional block when functionally recognized) recognizes a road shape (straight line or curve curvature) and a three-dimensional object (for example, a preceding vehicle) in front of the vehicle. This recognition is performed based on the image data stored in the image data memory 7 and the distance data stored in the distance data memory 14. Further, sensor information from a vehicle speed sensor or a steering angle sensor, navigation information, or the like, not shown, is referred to as necessary. Note that a specific method regarding the recognition of the road shape and the recognition of the three-dimensional object is disclosed in Japanese Patent Laid-Open No. 5-265547, so refer to it if necessary. If it is determined that an alarm is necessary based on these recognition results, the alarm device 17 such as a monitor or a speaker is activated to alert the driver. Further, by controlling the control device 18 as necessary, vehicle control such as AT (automatic transmission) shift down, suppression of engine output, or operation of a brake is appropriately executed.
[0029]
Next, a process for calculating the unified parallax Ni by evaluating and merging the primary color parallaxes Nr, Ng, and Nb will be described in detail. The fusion processing unit 8 determines the coincidence between the primary color parallaxes Nr, Ng, and Nb, and if they match, calculates the unified parallax Ni based on the primary color parallaxes Nr, Ng, and Nb. At this time, primary color parallaxes Nr, Ng, and Nb used for calculation of the unified parallax Ni are determined in consideration of the color of the object to be detected in accordance with an evaluation criterion described later. For example, when the detection target is a red-green system that does not include blue, only the coincidence between the red primary color parallax Nr and the green primary color parallax Ng is determined.
[0030]
Further, the fusion processing unit 8 evaluates the reliability of the input data to be merged in the above fusion. Specifically, in this evaluation, the primary color parallax Nr, Ng, and Nb output by the stereo arithmetic circuit 6 and the city block distances Mr, Mg, and Mb that give these parallaxes Nr, Ng, and Nb are detected objects. This is performed based on whether or not a predetermined evaluation criterion is provided for each. As will be described later, any one of the evaluation criteria described in the evaluation criteria table 100 as shown in FIG. 2 is applied to this evaluation criteria.
[0031]
In order to perform such fusion / evaluation processing, the fusion processing unit 8 includes a processor 9 that selects a detection target by executing a preset program, and a memory 10 that stores the program and various data. . The memory 10 includes a ROM in which programs and data are fixedly stored, and a RAM used as a work area for the processor 9. As information stored in the memory 10, for example, an evaluation criterion table 100 can be cited.
[0032]
FIG. 2 is a diagram illustrating an example of the evaluation criterion table 100. In the evaluation criterion table 100, an identifier 101, a level 102, and an evaluation criterion 103 are described in association with each other. Here, the evaluation standard 103 is a standard used in the evaluation process, and is at least one specific condition (hereinafter referred to as “individual condition”) regarding the primary color parallax Nr, Ng, Nb or the city block distance Mr, Mg, Mb. including. In the present embodiment, each evaluation criterion includes three types of individual conditions indicated by (1) to (3) in FIG. However, in consideration of the color of the detection object, it may not be appropriate to apply all the individual conditions, and depending on the detection object, there may be one or two individual conditions. Further, in order to cope with the difference in color of the detection target object, the evaluation standard 103 is only the primary color parameters (primary color parallax Nr, Ng, Nb, city block distance Mr, Mg, Mb) included in the color of the detection target object. It consists of In other words, a parameter relating to a primary color that is not included in the color of the detection target is not included in the evaluation criterion 103 of the detection target. If all the individual conditions specified in a certain evaluation standard 103 are satisfied, it is determined that the evaluation standard is satisfied.
[0033]
The first individual condition ((1) in the figure) relates to the coincidence of the primary color parallax values Nr, Ng, Nb. Specifically, the values of the primary color parallax Nr, Ng, and Nb are compared, and if these differences are within a predetermined range, it is determined that the values match. However, depending on the detection object, not all three primary colors (RGB) are included, so for such a detection object, the primary color parallax of the primary colors not included in the color component is excluded and the remaining two primary color parallaxes are excluded. Only the consistency is specified. For example, the color of the overtaking prohibition center line is typically orange and does not include a blue component. Therefore, as can be seen from the first individual condition relating to the identifier B, the determination is made based only on the coincidence (Nr = Ng) between the red primary color parallax Nr and the green primary color parallax Ng. In addition, the first individual condition is not defined as an evaluation criterion for a detection target (for example, a red signal) that includes only a single primary color.
[0034]
In the second individual condition ((2) in the figure), the reliability relating to the correlation of the luminance characteristics is determined for each primary color based on the city block distances Mr, Mg, Mb (minimum city block distances) calculated for each primary color. Judge individually. Specifically, it is determined for each primary color that the city block distances Mr, Mg, and Mb are smaller than predetermined determination threshold values Sr, Sg, and Sb. Judge that the condition is met. Note that these determination threshold values Sr, Sg, and Sb are set to different values depending on the level 102, and are set to smaller values as the evaluation standard 103 becomes more strict (and therefore between pixel blocks). The correlation determination of luminance characteristics becomes more severe). Similar to the first individual condition, the city block distances Mr, Mg, and Mb of primary colors that do not exist in the color component of the detection target are not considered in the second individual condition.
[0035]
In the third individual condition ((3) in the figure), based on the city block distances Mr, Mg, and Mb calculated for each primary color, the reliability relating to the correlation of luminance characteristics is comprehensively viewed for all primary colors. to decide. Specifically, when the total sum (Mr + Mg + Mb) of city block distances Mr, Mg, and Mb for each primary color image is smaller than a predetermined determination threshold value Si, it is determined that the third individual condition is satisfied. Note that this determination threshold value Si is set to a different value depending on the level 102, and is set to a smaller value as the evaluation standard 103 becomes more strict (hence, the correlation of luminance characteristics between pixel blocks). Judgment becomes more severe). As with the first individual condition, the primary individual city block distances Mr, Mg, and Mb that do not exist in the color components of the detection target are not taken into consideration in the second individual condition, The third individual condition is not defined for a detection target that includes only the primary colors.
[0036]
The identifier 101 is used to specify the evaluation criterion 103, and is a symbol, a code, or the like given individually for each evaluation criterion 103. In the evaluation criteria table 100, various evaluation criteria 103 are described so as to correspond to various detection objects. However, the evaluation criterion 103 specified by the identifier 101 is not necessarily one, and a plurality of evaluation criteria 103 having different levels 102 are described. Which level 102 of the evaluation standard 103 is applied is designated by the detection target selection unit 11.
[0037]
The level 102 is a high reliability that is recognized when the evaluation standard 103 corresponding to the level 102 is provided. In the present embodiment, a plurality of levels H and M having different evaluation severities are prepared for each detection target, and the level H having higher reliability is lower in reliability than that. Evaluation is stricter than level M. For example, in the evaluation standard 103 of level H in the identifier A, the three primary color parallaxes Nr = Ng = Nb are required as the first individual condition ((1) in the figure). On the other hand, in the evaluation condition 103 of the level M in the identifier A, it is required that two of the primary color parallaxes Nr, Ng, and Nb match as the first individual condition ((1) in the figure). , The conditions are relaxed compared to level H.
[0038]
In the example shown in FIG. 2, the evaluation criterion 103 (including both level H and level L) of the identifier A is applied at a default time when a specific object is not designated as a detection target, and each RGB color is almost equal. Is assumed to be included. The evaluation standard 103 of the identifier B is applied when, for example, a yellow (or orange) overtaking prohibition center line is detected. Further, the evaluation criterion 103 of the identifier C is applied when detecting a red signal, for example.
[0039]
The detection target selection unit 11 selects an object to be detected at the present time, and selects any evaluation criterion suitable for the selected detection object. The detection object can appear when the vehicle travels, and a plurality of objects (for example, an overtaking prohibition center line and a rear lamp of a preceding vehicle) are set in advance. The detection target selection unit 11 selects one of the target objects as a detection target according to a traveling situation specified by a vehicle speed sensor, a steering angle sensor, or the like (not shown). On the other hand, when a detection target is selected, one of the evaluation criteria 103 described in the evaluation criteria table 100 is selected based on the contents of an index table 130 described later.
[0040]
The detection target selection unit 11 includes a microcomputer 12 that selects a detection target by executing a preset program, and a memory 13 in which the program and various data are stored. The memory 13 includes a ROM in which programs and data are fixedly stored, and a RAM that is used as a work area of the microcomputer 12. Examples of information stored in the memory 13 include an index table 130 and evaluation criterion designation information.
[0041]
FIG. 3 is a diagram illustrating an example of the index table 130. In this index table 130, a detection object 131 and an identifier 132 of an evaluation criterion that is optimal for the detection object 131 are described in association with each other. For this identifier 132, the same reference numerals as those in the evaluation criterion table 100 shown in FIG. Therefore, when any identifier 132 is specified on the index table 130 side, the evaluation criterion described in the evaluation criterion table 100 is also uniquely specified.
[0042]
The evaluation standard designation information indicates an evaluation standard optimum for the detection target selected by the above-described arithmetic processing. This evaluation standard designation information is stored and held in a predetermined address area (hereinafter referred to as “evaluation standard designation area”) in the memory 13 (RAM). When the evaluation standard to be applied is updated, the information written in this evaluation standard designating area is also quickly updated. The fusion processing unit 8 obtains an evaluation criterion to be applied at the present time by accessing the evaluation criterion designation area.
[0043]
FIG. 4 is a flowchart showing a procedure for calculating the unified parallax Ni. First, in step 101, the stereo calculation circuit 6 reads the reference image data and the comparison image data stored in the image data memory 7, and performs stereo matching. As described above, stereo matching is performed independently for each of the R image, the G image, and the B image, and primary color parallaxes Nr, Ng, and Nb are calculated separately. The stereo matching itself performed for each primary color image is the same as the stereo matching performed for a monochrome image.
[0044]
Next, the fusion processing unit 8 reads the evaluation standard designation information from the evaluation standard designation area of the memory 13 (step 102), and refers to the evaluation standard table 100 to obtain an evaluation standard corresponding to the read evaluation standard designation information. Obtain (step 103). For example, when the identifier A is designated as the evaluation criterion designation information, both the evaluation criteria AH and AM regarding the identifier A in the table of FIG. 2 are read. Here, “AH” indicates an evaluation criterion of level H with identifier A, and AM indicates an evaluation criterion of level M with identifier A.
[0045]
Next, the fusion processing unit 8 evaluates the result of the stereo matching based on the evaluation criteria AH and AM specified in the previous step 103, and as an example, the three-stage evaluation results (“H”, “M”, "L") is output (step 104). The evaluation result output here is used as reliability information indicating the degree of reliability of the primary color parallax Nr, Ng, Nb related to a certain pixel block and the city block distances Mr, Mg, Mb related thereto. The evaluation process in step 104 is performed in order by the evaluation standards with the highest levels. For example, when the identifier A is designated as the evaluation standard information, first, the high level evaluation standard AH is determined. When this evaluation standard AH is provided, that is, when all three individual conditions constituting the evaluation standard AH are provided, the evaluation result (reliability) is “H” (meaning the highest reliability). The decision is made and no determination is made regarding the evaluation criterion AM. On the other hand, if the evaluation criterion AH is not provided, the process proceeds to the determination of the evaluation criterion AM having a lower level. When this evaluation standard AM is provided, that is, when all three individual conditions constituting the evaluation standard AM are provided, the evaluation result is “M” (meaning that the reliability is medium). Confirm. If the evaluation standard AM is not provided, “L” (meaning that reliability is lowest) is determined as the evaluation result.
[0046]
Next, the fusion processing unit 8 calculates the unified parallax Ni by fusing these based on the primary color parallax Nr, Ng, Nb (step 105). As this fusion method, for example, the values of primary color parallax Nr, Ng, Nb are compared, and when these values match, the value is determined and output as a unified parallax Ni. As another example, the unified parallax Ni may be calculated by a product-sum operation (including a weighted average and an arithmetic average) of the primary color parallaxes Nr, Ng, and Nb. In order to reduce the amount of calculation, the fusion processing unit 8 uses the reliability information indicating the degree of reliability of the primary color parallax Nr, Ng, Nb and the city block distance Mr, Mg, Mb regarding a certain pixel block. It is preferable to output the unified parallax Ni related to the pixel block only when it is determined that the reliability of the pixel block is high. In other words, when it is determined that the reliability is not high, the unified parallax Ni regarding the pixel block is not output, but invalid data is output instead.
[0047]
In step 105, not all of the primary color parallaxes Nr, Ng, and Nb are compared as described above, and are included in the color of the detection target as in the second individual condition in the evaluation criterion 103 of FIG. It is preferable that only the primary color parallaxes Nr, Ng, and Nb relating to the primary colors are to be compared. Further, only when the evaluation result obtained in the previous step 104 is high (for example, “H” or “M”), the fusion processing of the primary color parallaxes Nr, Ng, and Nb may be performed.
[0048]
In step 106, the fusion processing unit 8 outputs the unified parallax Ni as the distance data and the evaluation result (any one of “H”, “M”, and “L”) as the reliability information in association with each other. To do. The unified parallax Ni and the evaluation result associated with each other are stored in the distance data memory 14.
[0049]
The stereo arithmetic circuit 6 and the fusion processing unit 8 repeat the same processing after changing the processing target to the next pixel block. By obtaining the unified parallax Ni for one frame and the evaluation result, the distance data (i, j, Ni) for one frame is stored in the distance data memory 14 with reliability information.
[0050]
FIG. 5 is a flowchart showing the procedure for specifying the evaluation criteria. First, in step 201, the detection target selection unit 11 determines the vehicle speed, the state of acceleration / deceleration, the steering angle, the direction indicator (or direction indicator operation lever), etc. based on the detection results of various sensors (not shown). Detects the driving situation.
[0051]
In step 202, the detection target selection unit 11 selects an object to be detected at the present time based on the detected traveling state. For example, scenes in which the right direction indicator is lit (flashing) include (1) overtaking and (2) turning right. The overtaking and the right turn can be distinguished by considering the vehicle speed, acceleration conditions, and the like. When the detection target selection unit 11 determines that the right direction indicator is lit (flashing strictly) for overtaking, the detection target selection unit 11 selects a center line for which overtaking is prohibited as a detection target. The reason why such a center line is set as a detection target is to give a warning for the overtaking operation by the driver when traveling in an overtaking prohibited area. Note that the direction indicator itself may be out of lamp, so the state of the direction indicator operation lever may be detected instead of the lighting state of the direction indicator. In addition, the detection target selection unit 11 selects a stop lamp of the vehicle as a detection target in a situation where it is necessary to be wary of a rear-end collision with the preceding vehicle by considering the vehicle speed, the inter-vehicle distance, and the like. By detecting the lighting state of the stop lamp, deceleration of the vehicle ahead can be predicted, and a warning can be given to the driver.
[0052]
Next, in step 203, the detection target selection unit 11 refers to the index table 130 and acquires an identifier associated with the detection target selected in step 202. In a state where a specific detection target is not selected, an identifier A (designating the most versatile evaluation criterion) is set as a default value. The acquired identifier is written in the evaluation standard designation area of the memory 13 (RAM) as new evaluation standard designation information, and the previous evaluation standard designation information is updated (step 204).
[0053]
The detection target selection unit 11 repeatedly selects the detection target according to the traveling state by repeatedly executing the above procedure, and sequentially updates and sets the evaluation criterion according to the detection target.
[0054]
Thus, according to the present embodiment, primary color parallax Nr, Ng, Nb is calculated for each primary color image, and unified parallax Ni is calculated based on these, so that highly reliable distance data can be calculated. it can. In conventional stereo image processing based on a monochrome image, there is no color information in the input information, and only the sum of the luminances of the primary colors is obtained as luminance information. For this reason, even if the luminance of each primary color changes abruptly in a certain image area, as long as the total luminance is constant, the amount of variation cancels each other, so that it is not recognized as a luminance edge on the primary color basis. On the other hand, in the stereo image processing based on a color image as in the present embodiment, a captured image is obtained for each primary color constituting the color image, so that the luminance edge can be recognized for each primary color, and the luminance for each primary color The primary color parallax Nr, Ng, Nb caused by the edge can be calculated separately. At this time, the three primary color parallaxes Nr, Ng, and Nb calculated for a certain pixel block are not necessarily the same value. Such disparity in primary color parallax is caused by the fact that a background portion other than the object is included in the pixel block, or that the color component value in the background portion differs for each primary color. For example, considering the case where the object is red and the background part is green, the result of stereo image recognition for the R image is a value reflecting the distance to the object, and the red component is included in the background part. Since it is not included, the presence of the background portion does not affect the recognition result for the R image. However, since the background portion contains a lot of green components, the recognition result for the G image is a value reflecting the distance to the background. As a result, the green primary color parallax Ng does not match the red primary color parallax Nr. The present embodiment has been made in view of the point that such disparity between primary color parallaxes Nr, Ng, and Nb may occur, and calculates a uniform parallax Ni with reliability information based on the primary color parallaxes Nr, Ng, and Nb. To do. Since the reliability information indicates the reliability of the input data (primary color parallax Nr, Ng, Nb and city block distance Mr, Mg, Mb) in the parallax fusion processing, in the recognition unit 16 that uses the unified parallax Ni. Therefore, the use according to the reliability of the unified parallax Ni can be achieved.
[0055]
In this embodiment, an evaluation criterion is set in consideration of the color of the object to be detected, and reliability information in which the color of the detection object is considered is generated and output. When calculating the uniform parallax Ni, the distance data with higher reliability can be calculated by taking the color of the detection object into consideration. This is particularly effective when the color of the object is characteristic, for example, when the color is significantly different from the surroundings.
[0056]
Furthermore, since a color CCD whose price has been rapidly reduced in recent years can be used, the cost of the image processing apparatus can be further reduced. In particular, with the progress of colorization of image sensors, it is becoming more expensive to adopt a monochrome image sensor in terms of cost. However, according to this embodiment, such inconvenience can be avoided.
[0057]
Note that this embodiment is applicable even when a color camera using four complementary color filters (cyan, magenta, yellow, and green) is used. A color camera using this complementary color filter is advantageous in terms of high resolution and compatibility with dark conditions because of high light utilization efficiency. However, in order to use the various circuits described above as they are, it is necessary to convert output image data into RGB color image data even in a camera employing such a complementary color filter.
[0058]
In the above-described embodiment, the fusion processing unit 8 and the detection target selection unit 11 are provided separately from other components. However, the fusion processing unit 8 and the detection target selection unit 11 do not necessarily exist independently in terms of hardware configuration. For example, the fusion processing unit 8 and the detection target selection unit 11 can be realized by causing the microcomputer 15 to execute a predetermined program. Also good.
[0059]
In the present invention, the number and specific contents of the individual conditions constituting the evaluation criteria are not limited to the present embodiment, and the evaluation criteria may be prepared by dividing into more levels. In particular, it is not always necessary for the evaluation criteria to specify both primary color parallax Nr, Ng, Nb and city block distances Mr, Mg, Mb, and either one may be used. Furthermore, although the index table 130 describes the detection target 131 and the identifier 132 that specifies the evaluation criterion in association with each other, the color and the evaluation criterion may be directly associated with each other.
[0060]
In the above-described embodiment, the evaluation criterion is prepared separately for each level of reliability. However, a single evaluation criterion may be applied uniformly. In this case, it is preferable to determine the reliability depending on how much the evaluation criterion is satisfied.
[0061]
Further, the object to be detected may be switched according to the area in the captured image. For example, while performing stereo matching for the upper region of the image, the signal red or the sign is set as the detection target. And when the area | region which performs a stereo matching reaches the lower part of an image, a detection target object is changed into a centerline. Thereby, different types of detection objects can be detected simultaneously in the captured image corresponding to one frame. Thereby, the distance data regarding a plurality of detection objects can be calculated efficiently and with high accuracy. In this case, the detection target selection unit 11 holds an evaluation criterion corresponding to the detection target in association with “region designation information” indicating an image region to which the detection criterion is applied. The fusion processing unit 8 reads out evaluation criteria from the detection target selection unit 11 after recognizing which part of the captured image the primary color parallaxes Nr, Ng, and Nb to be merged are related to. However, the “area designation information” does not necessarily have to be in the form of data. That is, by appropriately determining the storage order (or storage area) in the memory 13, the reading order by the fusion processing unit 8, the calculation processing order, etc., as a result, the weighting coefficient can be properly used for each area. It doesn't matter. In this case, the agreement itself corresponds to “area designation information”.
[0062]
(Second Embodiment)
The first embodiment described above is based on the premise that the pixels (pixels) corresponding to each other are located in the same position on the three primary color images constituting the three-plate color CCD. As long as this premise is established, the same image is projected on each CCD, and the red pixel Cr (i, j) and the green pixel Cg (i, J) constituting a certain pixel P (i, J). j) and the blue pixel Cb (i, j) are completely in the same position on the projection image. However, in a consumer three-plate color CCD used in a video camera or the like, the position of each primary color pixel Cr (i, j), Cg (i, j), Cb (i, j) is intentionally shifted. There are many. This is because if the primary color pixels Cr (i, j), Cg (i, j), and Cb (i, j) are completely matched, the boundary between the pixels is visually noticeable, which is not preferable. Therefore, in the present embodiment, when the cameras 1 and 2 incorporating the image sensor having such characteristics are employed, a captured image suitable for stereo matching is obtained by eliminating the influence of intentionally set pixel shift. A pair is generated. In addition, about the structure same as 1st Embodiment, the same code | symbol is attached | subjected and description in each is abbreviate | omitted.
[0063]
The stereo image processing apparatus according to the present embodiment basically has the same configuration as the block diagram shown in FIG. However, cameras 1 and 2 incorporating a three-plate color CCD are employed, and the pixel positions in the primary color images are shifted from each other. Further, in addition to the correction function described in the first embodiment, the correction circuit 5 has a function of correcting a pixel shift intentionally set on the CCD side for each primary color image (R image, G image, B image). It has.
[0064]
FIG. 6 is an explanatory diagram of pixel positional deviation correction. When the reference pixel (corrected pixel) is Cs (i, j) and the actual pixel to be corrected (precorrected pixel) is Ct (i, j), the reference pixel Cs (i, j) The luminance value of the reference pixel Cs (i, j) is calculated by a weighted average using the luminance value of at least one overlapping real pixel Ct (i, j) as an input. For example, the corrected luminance value of the reference pixel Cs (1,1) is the four actual pixels Ct (1,1), Ct (1,2), Ct ( 2,1) and Ct (2,2).
[0065]
Pixel shift correction is performed by calculating the luminance value Is of the reference pixel Cs by the weighted average calculation of Equation 1. In the equation, Is (i, j) is the luminance value of the reference pixel Cs (i, j), and It (i, j) is the luminance value of the actual pixel Ct (i, j). Also, a1 to a4 are weighting factors determined according to the amount of deviation between the reference pixel Cs (i, j) and the actual pixel Ct (i, j). Since this deviation amount is known from the specification of the three-plate color CCD, the weight coefficients a1 to a4 are also uniquely and fixedly applied.
[Expression 1]
Is (i, j) = a1 * It (i, j) + a2 * It (i, j + 1) + a3 * It (i + 1, j) + a4 * It (i + 1, j + 1))
[0066]
For example, as shown in FIG. 6, when the actual pixel Ct (i, j) and the reference pixel Cs (i, j) are shifted by 1/2 pixel in both the vertical and horizontal directions, the reference pixel is calculated by the arithmetic mean of Equation 2. The luminance value of Cs (i, j) is uniquely calculated.
[Expression 2]
Is (i, j) = 1/4 (It (i, j) + It (i, j + 1) + It (i + 1, j) + It (i + 1, j + 1))
[0067]
Such pixel positional deviation correction, in other words, luminance value correction associated with apparently shifting the position of the actual pixel Ct (i, j) to the position of the reference pixel Cs (i, j), The process is executed for all the pixels constituting the primary color image. As a result, the positional deviation of the pixels intentionally set as the specification of the three-plate color CCD is eliminated, and a reference image suitable for stereo matching and an elephant are obtained. As a result, more reliable distance data can be calculated. In addition, a stereo image processing apparatus can be realized at low cost by using consumer products that are easily available and inexpensive.
[0068]
【The invention's effect】
In the present invention, when performing stereo image processing using a pair of color images, the primary color parallax is calculated for each primary color image of the same color constituting the color image. Then, the unified parallax is calculated with reliability information by fusing the primary color parallaxes. Since the reliability information indicates the reliability of the input data in the parallax fusion processing, it can be used in accordance with the reliability of the unified parallax in the subsequent recognition unit that uses the unified parallax.
[Brief description of the drawings]
FIG. 1 is a schematic block diagram of a stereo image processing apparatus.
FIG. 2 is a diagram showing an example of an evaluation criteria table
FIG. 3 is a diagram showing an example of an index table
FIG. 4 is a flowchart showing a unified parallax calculation procedure.
FIG. 5 is a flowchart showing a procedure for specifying evaluation criteria.
FIG. 6 is an explanatory diagram of pixel misalignment correction.
[Explanation of symbols]
1,2 camera
3 Analog interface
4 A / D converter
5 Correction circuit
6 Stereo operation circuit
7 Image data memory
8 Fusion processing department
9 processor
10 memory
11 Detection target selection section
12 Microcomputer
13 memory
14 Distance data memory
15 Microcomputer
16 Recognition part
17 Alarm device
18 Control device

Claims (16)

In an image processing apparatus that performs stereo matching using a pair of color images each composed of a plurality of primary color images,
Stereo calculation means for calculating the primary color parallax relating to each primary color image and the city block distance relating to the primary color parallax for each image area serving as a parallax calculation unit by performing stereo matching using a primary color image pair of the same color When,
Based on each primary color parallax, a unified parallax for each image area is calculated, and reliability information indicating a reliability level of at least one of the primary color parallax and the city block distance is output in association with the unified parallax. Fusion processing means to
The image processing apparatus , wherein the fusion processing unit calculates the reliability information based on a plurality of evaluation criteria having different strictness according to the level .   The fusion processing means outputs the unified parallax for the image area when it is determined that the reliability for one image area is high, and the image area when it is determined that the reliability is not high. The image processing apparatus according to claim 1, wherein the unified parallax is not output.   The image processing apparatus according to claim 1, wherein the fusion processing unit evaluates the reliability of at least one of the primary color parallax and the city block distance according to a predetermined evaluation criterion.   The image processing apparatus according to claim 3, wherein the evaluation criterion includes at least a first individual condition for determining whether or not respective values of the primary color parallax match.   The image according to claim 3 or 4, wherein the evaluation criterion includes at least a second individual condition for determining whether each of the city block distances is smaller than a predetermined determination threshold value. Processing equipment.   The evaluation criterion includes at least a third individual condition for determining whether or not the sum of the city block distances is smaller than a predetermined determination threshold value. The described image processing apparatus.   The fusion processing means selects and applies any one of a plurality of evaluation criteria prepared in advance according to the color of the object to be detected. The image processing apparatus described in any one.   In each primary color image, in order to calculate the luminance value of the reference pixel excluding the influence of the positional deviation of the actual pixel corresponding to the position, by the weighted average of the luminance value of the actual pixel overlapping the reference pixel, The image processing apparatus according to claim 1, further comprising a correction unit that calculates a luminance value of the reference pixel. In an image processing method for performing stereo matching using a pair of color images each composed of a plurality of primary color images,
By performing stereo matching using primary color image pairs of the same color, a first color parallax relating to each primary color image and a city block distance relating to the primary color parallax are calculated for each image region which is a unit for calculating parallax. Steps,
Based on each primary color parallax, a unified parallax for each image area is calculated, and reliability information indicating a reliability level of at least one of the primary color parallax and the city block distance is output in association with the unified parallax. And a second step to
The image processing method according to claim 2, wherein the second step calculates the reliability information based on a plurality of evaluation criteria different in severity according to the level . In the second step, when it is determined that the reliability related to one image region is high, the unified parallax related to the image region is output, and when it is determined that the reliability is not high, the image The image processing method according to claim 9 , wherein the unified parallax relating to an area is not output. The image processing method according to claim 9 or 10 , wherein in the second step, the reliability of at least one of the primary color parallax and the city block distance is evaluated according to a predetermined evaluation criterion. The image processing method according to claim 11 , wherein the evaluation criterion includes at least a first individual condition for determining whether or not respective values of the primary color parallax match. The image according to claim 11 or 12 , wherein the evaluation criterion includes at least a second individual condition for determining whether each of the city block distances is smaller than a predetermined determination threshold value. Processing method. The evaluation criteria are the sum of the city block distance, in any of claims 11, characterized in that it comprises at least a third individual condition for determining whether smaller than a predetermined determination threshold value 13 The described image processing method. The second step includes a step of selecting and applying any one of a plurality of evaluation criteria prepared in advance according to the color of an object to be detected. The image processing method described in any one of 11 to 14 . In each primary color image, in order to calculate the luminance value of the reference pixel excluding the influence of the positional shift of the actual pixel corresponding to the position, by the weighted average of the luminance value of the actual pixel overlapping the reference pixel, The image processing method according to claim 9 , further comprising a third step of calculating a luminance value of the reference pixel.

Images