JP7134113B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus.

As a background art in this technical field, there is Patent Document 1. In this publication, the problem is described as "to re-correct the calibration deviation without increasing the circuit scale and calculation time", and as a solution, "right and left camera stereo parallelization corrected by calibration The amount of deviation between the post-images is obtained by the calibration deviation re-correction processing by the calibration deviation re-correction section 15 during the corresponding point search processing by the corresponding point search section 9 including the calibration deviation re-correction section 15. In this way, it is possible to improve the measurement accuracy without increasing the circuit scale and calculation time of the internal hardware in the distance measuring device 1. Further, the right camera stereo-parallelized image is vertically shifted to , re-corrects the vertical deviation of the left and right camera stereo parallelized images.In this way, it is possible to reduce the vertical calibration deviation that tends to cause problems.At that time, when searching for corresponding points, It is possible to reliably find the optimal position of the right camera stereo collimated image."

Japanese Unexamined Patent Application Publication No. 2013-113600

The aforementioned Patent Document 1 states that "the ratio (%) of pixels in which corresponding points are found among all pixels in the pixel region of interest in the left-camera stereo-parallelized image is calculated as the recognition rate." It states that the vertical deviation is corrected so as to maximize the recognition rate, but there is a problem that the reliability cannot be calculated for each pixel. In addition, there is a problem that errors are included depending on the pattern of the image even if the recognition rate is high.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus capable of evaluating the reliability of a measurement value for each pixel.

In order to achieve the above object, the present invention provides an image storage unit that stores at least a first image and a second image, an offset applying unit that applies a vertical offset to the second image, and an offset applying unit that applies the offset. a measurement value calculation unit that calculates a measurement value from pixels of the first image and pixels of the second image that correspond to the pixels of the first image before and after; a difference calculation unit that calculates a difference between the measurement values before and after the offset is applied; and a reliability calculation unit that calculates the reliability of each pixel from the difference between the measured values.

According to the present invention, the reliability of the measurement value for each pixel can be evaluated. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

FIG. 10 is an explanatory diagram regarding a parallax error when a stereo camera is displaced in the vertical direction; 1 is an explanatory diagram showing the configuration of a vehicle control system; FIG. 1 is an explanatory diagram showing the configuration of an image processing apparatus according to a first embodiment; FIG. FIG. 10 is an explanatory diagram showing the configuration of an image processing apparatus according to second and third embodiments; FIG. 13 is an explanatory diagram showing the configuration of an image processing apparatus according to fourth and fifth embodiments; 10 is a flow chart showing processing of the image processing apparatus of the first to fifth embodiments;

The configuration and operation of a system including image processing apparatuses according to first to fifth embodiments of the present invention will be described below with reference to the drawings.

(parallax error and distance error)
First, parallax error and distance error of an oblique line-shaped object captured by a stereo camera system with a vertical shift will be described with reference to FIG.

A stereo camera system is well known as means for estimating the three-dimensional position of an object. In a stereo camera system, cameras are arranged at a plurality of positions to capture images of the same object from a plurality of different viewpoints, and the distance to the target object is calculated from the difference in the appearance of the obtained images, that is, the parallax. In a general stereo camera system using two cameras, this conversion is represented by the following formula (1).

where Z[mm] is the distance to the target object, f[mm] is the focal length, w[mm/px] is the pixel pitch, B[mm] is the distance between the cameras (baseline length), D[px] represents parallax.

In calculating the parallax, images captured from a plurality of positions (viewpoints) are arranged horizontally, and positions where the same point in the right image is captured with respect to a specific point in the left image are searched. In order to efficiently perform this search, parallelization processing is generally performed in advance.

Parallelization means vertical alignment of the image. That is, this is a process of calibrating the images so that the same points in the left and right images are imaged at the same height. In a parallelized image, when searching for corresponding points, it is sufficient to search only a certain horizontal row, and the processing efficiency is high. However, the parameters for parallelization change due to errors during assembly and deterioration over time, and even left and right images that have undergone parallelization processing may have deviations in the vertical direction. At this time, especially in an area where a straight line in an oblique direction is imaged, similar textures are imaged at different heights, so erroneous corresponding points are detected and an error occurs in the three-dimensional position. .

FIG. 1 is an explanatory diagram of the parallax error dD[px] when a shift of dj[px] occurs in the vertical direction. A left image 100LA and a right image 100RA show a case where no height deviation occurs. With respect to the search target 121 in the left image 100LA, the position where the same object is imaged is searched from the right image 100RA, and the image position shift amount with the matched matching target 122 is obtained as parallax.

On the other hand, the left image 100LB and the right image 100RB show the case where height deviation occurs. The displacement amount of the image position between the similar search target 121 and the matching target 123 is obtained as parallax. At this time, a positional shift between 122 and 123 occurs as a parallax error dD[px]. Assuming that the inclination of the straight line with respect to the horizontal is θ [rad], the parallax error dD is expressed by Equation (2) below.

That is, when a certain vertical shift dj occurs, the parallax error dD becomes a function of the slope θ of the straight line, and dD increases as θ approaches 0, that is, approaches the horizontal.

Also, from the equation (1), the distance error dZ [mm] when the parallax error dD occurs is represented by the following equation (3).

Since f, B, and w are fixed by the camera, it can be seen that the distance error dZ associated with the parallax error dD increases as the parallax D decreases. On the other hand, if D is sufficiently large relative to dD, dZ will be small, so the effect on object detection and ranging can be ignored.

The case of the stereo camera system has been described above, but the same applies to a system that calculates corresponding points of two images taken at different times using a single camera.

[First embodiment] (Example of a monocular camera)
An embodiment will be described below with reference to the drawings.

First, an outline of a vehicle control system using a camera equipped with an object detection device will be described with reference to FIG. 2 is a schematic diagram of the vehicle 200 viewed from the side, and includes a camera 201 (monocular camera), a steering wheel 204 (including a steering actuator, a steering angle sensor, etc.), and wheels 205 (a brake actuator, a drive mechanism, etc.). ) is connected to the vehicle control unit 202 .

A camera 201 is mounted on a vehicle 200 . The camera 201 is equipped with an object detection device, which measures, for example, the distance and relative speed to an object 210 (for example, a telegraph pole) in front and transmits them to the vehicle control unit 202 . The vehicle control unit 202 controls the brake, accelerator, and steering based on the received distance and relative speed.

Next, an overview of the image processing apparatus 300 will be described with reference to FIG.

FIG. 3 shows the configuration of an image processing apparatus 300 according to the first embodiment of the invention. The image processing apparatus 300 includes a CPU 310 , a memory 320 , an image input unit 390 , an image storage unit 330 , a movement amount calculation unit 340 , a movement amount difference calculation unit 360 , a reliability calculation unit 370 , an object detection unit 380 , and an offset addition unit 391 . , and an external output unit 350 .

Note that the image input unit 390, the image storage unit 330, the movement amount calculation unit 340, the movement amount difference calculation unit 360, the reliability calculation unit 370, the object detection unit 380, the offset addition unit 391, and the external output unit 350 are examples. , is composed of a logic circuit such as an FPGA (Field Programmable Gate Array).

The image input unit 390 is an interface for inputting data output from an external video data output device (not shown). Data input by the image input unit 390 is stored in the memory 320 . In this embodiment, it is assumed that images are transferred continuously, and the image accumulation unit 330 accumulates the input images in chronological order.

The offset imparting unit 391 performs image processing for moving the position of the input image, and the image storage unit 330 also stores the image processed by the offset imparting unit 391 .

The movement amount calculation unit 340 searches for corresponding points for each pixel or each partial region in two images stored at different times in the image storage unit 330, and calculates the amount of movement on the image. As a calculation method, for example, a technique called optical flow is known.

The movement amount difference calculation unit 360 calculates I0 and I1 among the image I0 at time T0, the image I1 at time T1 to which no offset is applied, and the image I2 at time T1 to which the offset is applied by the offset applying unit 391. is used to obtain the difference in the amount of movement between the flow generated by the movement amount calculation unit 340 and the flow generated using I0 and I2.

The reliability calculation unit 370 cancels the difference in the movement amount of the flow generated by the movement amount calculation unit 340 and the given offset. Output confidence as high.

Based on the output result of the movement amount calculation unit 340 and the output result of the reliability calculation unit 370 when the offset addition unit 391 does not add an offset, the object detection unit 380 detects pixels with high reliability that are moved. Objects are detected by grouping pixels with large amounts.

At this time, by using highly reliable pixels, it is possible to detect an object with less false detection. In addition, by grouping pixels including pixels with low reliability and using only pixels with high reliability when calculating the amount of movement, it is possible to detect all objects without fail, but the amount of movement cannot be calculated accurately. It is possible. By obtaining the movement amount and the reliability for each pixel in this way, both the detection rate of the object detection and the accuracy of estimating the moving direction of the object can be achieved.

The external output section 350 outputs the position and moving direction of the object detected by the object detection section 380 . When the output result of the external output unit 350 is used for vehicle control, it is possible to control the brake operation and the steering wheel operation based on the time and distance until the collision with respect to an object with a high collision probability, thereby avoiding the collision. can.

In this way, when it is used to avoid vehicle collisions, it is necessary to predict the position and direction of movement of objects more accurately. is. Therefore, prediction of the movement direction requires highly accurate calculation of the movement amount.

Next, the operation of the image processing apparatus 300 of this embodiment will be described with reference to FIG. Note that the image storage unit 330 stores at least the image I0 (first image) and the image I1 (second image).

The offset imparting unit 391 imparts a vertical offset to the image I1 (second image) (S10). The movement amount calculation unit 340 (measurement value calculation unit) calculates the movement amount (measurement value) from the pixels of the image I0 (first image) and the corresponding pixels of the image I1 (second image) before and after the offset is applied. (S15). The movement amount difference calculation unit 360 (difference calculation unit) calculates the difference between the movement amounts (measured values) before and after the offset is applied (S20). The reliability calculator 370 calculates the reliability of each pixel from the difference in movement amount (measured value) (S25).

This makes it possible to evaluate the reliability of the measurement value for each pixel. It is also possible to detect pixels associated with measured values with large errors due to oblique shapes.

In this embodiment, the image I0 (first image) and the image I1 (second image) are time-series (time T1, T2) images captured by a monocular camera. The movement amount calculation section 340 (measurement value calculation section) calculates the movement amount for each pixel as a measurement value. This makes it possible to evaluate the reliability of the movement amount for each pixel. Also, it is possible to detect a pixel associated with a movement amount with a large error.

According to the present embodiment described above, by applying an offset, it is possible to detect an unstable area in which the movement amount cannot be calculated correctly, and by calculating the movement amount excluding the unstable area, the movement amount can be calculated with high accuracy. It can be performed.

[Second embodiment] (example of stereo camera)
An obstacle detection system using a stereo camera will be described as an example. This embodiment is an example in which the camera 201 in FIG. 2 described in the first embodiment is a stereo camera.

FIG. 4 shows the configuration of an image processing apparatus according to the second embodiment of the present invention.

The image processing apparatus 400 includes a CPU 410, a memory 420, an image input unit 490, a parallelization unit 430, a parallax calculation unit 440, a parallax difference calculation unit 460, a reliability calculation unit 470, an object detection unit 480, an offset provision unit 491, an external It is composed of an output unit 450 .

The image input unit 490 is an interface for inputting data output from an external video data output device (not shown). In the case of a stereo camera, the distance can be determined more accurately by inputting synchronized images so that the left and right images are captured at the same time.

Data input by the image input unit 490 is stored in the memory 420 . The parallelization unit 430 performs a process of correcting the mounting position, distortion caused by the lens, and the like so that the input left and right images are parallel.

The offset applying unit 491 applies an offset in the vertical direction to the correction amount of the parallelizing unit 430 . The parallelization unit 430 generates a plurality of images to which the offset is applied by the offset applying unit 491 and an image to which the offset is not applied. It is assumed that the offset is given to only one of the left and right images, and that one of the images is intentionally shifted. Here, an example in which two types of offsets, positive and negative, are given as types of offsets will be described.

The parallax calculation unit 440 processes the plurality of images to which the offset is given and the images to which the offset is not given, respectively, and calculates the parallax.

In this embodiment, the left image (first image) and right image (second image) are a pair of images captured by a stereo camera. The parallax calculator 440 (measured value calculator) calculates the parallax of each pixel as a measured value. The reliability calculation unit 470 calculates the reliability of each pixel from the difference in parallax (measured value). This makes it possible to evaluate the reliability of parallax for each pixel. In addition, it is possible to detect pixels associated with parallax with a large error due to oblique shapes.

The parallax difference calculator 460 compares the offset-added image output from the parallax calculator 440 with the non-offset image, and obtains the parallax difference for each pixel. Examples of output results of the parallax difference calculation unit 460 include whether to output variations in parallax of the comparison image or to output the sum of parallax differences.

Based on the result output by the parallax difference calculation unit 460, the reliability calculation unit 470 obtains the variance of the difference image for each pixel and outputs it as the reliability. A disparity with a large dispersion has a low reliability, and a small dispersion has a high reliability. Here, an example of calculating the variance is shown, but the sum of the absolute values of the differences in parallax may also be used. Pixels whose parallax does not change even when an offset is added have high reliability, and pixels whose parallax changes when an offset is added have a high reliability. Any index that indicates that the

That is, the reliability calculation unit 470 calculates the reliability of each pixel from the dispersion of parallax (measured value) or the sum of the absolute values of the difference of parallax (measured value). Thereby, the reliability of the reliability of each pixel can be improved. Note that the reliability calculation unit 470 may calculate the reliability of each pixel from the variance of the distance (measured value) to the object corresponding to the pixel or the sum of the absolute values of the differences in the distance (measured value). This makes it possible to evaluate the reliability of the distance for each pixel. Here, the greater the difference in the measured values (movement amount, parallax, distance, etc.), the lower the reliability. This makes it possible to evaluate the reliability of the measured value for each pixel according to the difference in the measured values.

The object detection unit 480 detects an object by performing three-dimensional grouping processing using the images to which the parallax calculation unit 440 has not provided an offset. At this time, erroneous detection can be prevented by referring to the results output by the reliability calculation unit 470 and performing grouping processing using only disparity data with high reliability.

That is, the object detection unit 480 performs grouping using parallaxes (measured values) whose reliability exceeds the threshold, and detects an object. Thereby, erroneous detection of an object can be prevented.

[Third embodiment] (Example of periodically applying an offset)
In the image processing apparatus 400 of the second embodiment, a plurality of offsets are designated by the offset imparting unit 491, and the parallelization unit 430 generates an image to which the offset is not imparted and an image to which the offset is imparted. was used to calculate the parallax. The third embodiment shows an example in which the addition and non-addition of an offset is periodically switched with respect to the time series.

That is, in this embodiment, the offset applying unit 491 periodically applies an offset. The reliability calculation unit 470 outputs the time-series variation of the disparity (measurement value) variance in the target region as the reliability. This makes it possible to evaluate the reliability of the measured value of the target region.

The object detection unit 480 performs a three-dimensional grouping process on parallax images with positive offset, parallax images without offset, and parallax images with negative offset in chronological order. When the grouping results are stable, the reliability is high, and when the grouping results are unstable, the reliability is low. Here, "stability" means, for example, when detecting an object whose size does not easily change over time, such as an automobile, when the size is the same, it is stable, and when the size changes over time, it is unstable. Meaning.

In this way, by adding an offset in time series to one of the parallelized images when creating the parallax image, it is possible to determine whether or not the detected object is erroneously detected. Since it only needs to be performed once per frame, it can be realized without increasing the amount of calculation.

[Embodiment 4] (Example of diagnostic imaging unit)
FIG. 5 will be described. This embodiment includes an image diagnosis unit 501 in addition to the configuration of the second embodiment. The image diagnosis unit 501 will be described. The reliability of the reliability calculation unit 470 for the input image is measured in advance, and the threshold value is stored in advance. Based on the result (reliability) output by the reliability calculation unit 470 and the threshold, the image diagnosis unit 501 notifies an abnormality from the external output unit 450 when the reliability exceeds the threshold.

That is, the image diagnosis unit 501 diagnoses that the parallax (measured value) of the target region is abnormal when the dispersion of the parallax (measured value) of the target region exceeds the threshold. Thereby, it is possible to determine the abnormality of the parallax (measured value) of the target area.

When notified of an abnormality, the vehicle control system notifies the driver of the abnormal state, suspends brake control and steering control, and can prevent malfunction. By stopping the system in stages according to the margin of reliability with respect to the threshold, it is possible to stop steering operation and continue control such as emergency braking, achieving both convenience and reliability. can do.

[Embodiment 5] (Example of detecting lens distortion)
In the fourth embodiment, the reliability of the reliability calculation unit 470 for a specific scene is measured in advance, and the threshold value is stored in advance.

If there is a difference in the reliability when the specific scene is photographed again, it is possible to detect the occurrence of deviation other than vertical deviation.

As a result, it is possible to distinguish whether the camera has shifted in the vertical direction or has shifted due to other factors, and the correction amount of the image can be obtained more accurately. In addition, by setting a threshold value for abnormality judgment for each of vertical displacement and displacement due to other factors, unintended structural displacement can be judged as abnormal, and vertical displacement can be suspended. A more precise diagnosis can be made.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations may be implemented partially or wholly by hardware, or may be implemented by a processor executing a program. Further, the control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.

In addition, the following aspects may be sufficient as the Example of this invention.

(1). an offset addition unit that adds an offset to an image; a movement amount calculation unit that calculates a movement amount of each pixel of the image in time series; and a reliability calculation unit that calculates the reliability of each pixel based on the offset amount. An image processing device comprising:

(2). a plurality of imaging units for obtaining images, a parallax calculation unit for calculating parallax from the images, an offset adding unit for adding a vertical offset when calculating the parallax, and a difference calculation unit for calculating the parallax difference. and a reliability calculation unit that calculates reliability based on the parallax difference.

(3). In (1) and (2), the offset addition unit periodically adds the offset to calculate the variance of the distance or movement amount of the specific area, and the reliability calculation unit calculates the variance at the time of the variance. An image processing apparatus, wherein a series variation is output as the reliability.

(4). In (1) to (3), an image diagnostic unit is provided, and the offset amount added to the offset addition unit and the dispersion of the distance or movement amount of the specific area are calculated, and the distance or movement amount of the specific area is held in advance. The image processing apparatus, wherein the image diagnosis unit detects an abnormality when a threshold value is exceeded.

210 ... Object (telegraph pole)
200... vehicle (automobile)
300, 400, 500... Image processing device

Claims (9)

an image storage unit that stores at least the first image and the second image;
an offset applying unit that applies a vertical offset to the second image;
a measurement value calculation unit that calculates measurement values from the pixels of the first image and the corresponding pixels of the second image before and after applying the offset;
a difference calculation unit that calculates the difference between the measured values before and after the offset is applied;
a reliability calculation unit that calculates the reliability of each pixel from the difference between the measured values;
An image processing device comprising: The image processing device according to claim 1,
The first image and the second image are
It is a time-series image taken with a monocular camera,
The measured value calculation unit
An image processing apparatus, wherein a movement amount for each pixel is calculated as the measurement value. The image processing device according to claim 1,
The first image and the second image are
A pair of images captured by a stereo camera,
The measured value calculation unit
An image processing apparatus, wherein parallax for each pixel is calculated as the measured value. The image processing device according to claim 1,
The offset applying unit
periodically applying the offset;
The reliability calculation unit
An image processing apparatus, characterized in that a time-series variation of variance of the measured values in the target region is output as the reliability. The image processing device according to claim 4,
An image processing apparatus, comprising: an image diagnosis unit that diagnoses that the measured values of the target region are abnormal when a variance of the measured values of the target region exceeds a threshold. The image processing device according to claim 1,
The reliability calculation unit
An image processing apparatus, wherein the reliability for each pixel is calculated from a variance of the measured values or a sum of absolute values of differences of the measured values. The image processing device according to claim 6,
The measured value is
An image processing device characterized by being a distance to an object corresponding to a pixel. The image processing device according to claim 1,
The image processing device, wherein the reliability decreases as the difference between the measured values increases. The image processing device according to claim 3,
An image processing apparatus, comprising: an object detection unit that performs grouping using measured values whose reliability exceeds a threshold and detects an object.

Images