JP6549932B2 - Stereo image processing device - Google Patents

Description

  The present invention relates to a stereo image processing apparatus that estimates and corrects parallax shift of a pair of cameras mounted on a moving body.

  BACKGROUND Conventionally, a three-dimensional measurement device that acquires three-dimensional distance information and recognizes an ambient environment and a self position based on this information is known, and is mounted on a mobile body such as a vehicle and put to practical use. This type of three-dimensional measurement apparatus, for example, finds the correlation between a pair of images of the same object captured by a pair of left and right cameras (stereo cameras), and determines the camera position and focal length etc. The parameters are used to calculate the distance to the object according to the principle of triangulation.

  Therefore, in order to obtain stereo matching accuracy (reliable distance information), it is desirable that no positional deviation other than parallax be present in the stereo image. However, in actuality, the optical axis of the camera is shifted due to physical stress applied to the camera by screwing the camera at the time of attachment, etc. or distortion due to vibration or heat of the device equipped with the camera, etc. It may shift with time. When parallax displacement occurs in the stereo camera, the parallax between the reference image and the comparison image does not correspond exactly to the actual distance of the object, and the accuracy of the three-dimensional position determination of the object by the stereo matching method decreases. The reliability of the distance information to the object is lost.

  As a technique for correcting the optical axis deviation in such a camera, for example, in Japanese Patent Application Laid-Open No. 2007-267231 (hereinafter referred to as Patent Document 1), the first feature and the first feature are The first distance to the vehicle is detected, and the second distance between the second feature and the second feature and the vehicle is detected by the front stereo camera, and the true distance stored in the high precision map data There is disclosed a technology of an optical axis offset correction device which detects the optical axis offset of the rear camera by comparing the above measured value with the above-mentioned measured value, and corrects the measured value to coincide with the true distance.

Unexamined-Japanese-Patent No. 2007-267231

  However, in the optical axis offset correction device disclosed in Patent Document 1 described above, a plurality of sensors such as a rear camera, a front stereo camera, and a sensor (for example, a vehicle speed sensor) for measuring the movement distance of the vehicle are required. In the case of detecting and correcting the optical axis deviation of the rear camera, there is a problem that the optical axis deviation can not be accurately corrected without considering the errors of the plurality of sensors. In addition, since the correction of the optical axis deviation can be performed only after the first feature and the second feature are detected, there is a problem that it takes time to correct the optical axis deviation. Furthermore, when detecting the first feature and the second feature and correcting the optical axis deviation in this way, the detection state when detecting the first feature and the second feature are detected. The detection state at the time of operation is different, which includes a disturbance, and in order to correct the optical axis deviation with high accuracy, it is necessary to also consider such a disturbance.

  The present invention has been made in view of the above circumstances, and provides a stereo image processing apparatus capable of accurately correcting the parallax of a stereo camera in a short time without including errors and disturbances without using many sensors. The purpose is to

One aspect of a stereo image processing apparatus according to the present invention is stereo image processing for calculating distance information to the same object based on parallax of the same object of a pair of image information captured by a pair of cameras mounted on a moving object In the device, two stopped objects having different distances from the moving object are detected from only one frame at the same timing among the pair of image information and map information acquisition means for acquiring map information, and the above map information is detected. Based on the image information, two stop objects corresponding to the two stop objects are detected, and a distance between the two stop objects based on the image information is calculated as a first stop object distance, and based on the map information The distance between two stationary objects is calculated as a second distance between stationary objects, the first distance between stationary objects is compared with the second distance between stationary objects, and the second distance between stationary objects is used as a reference. The distance between the first stationary object is a parallax correction means for correcting the disparity to match the distance between the second stop was then.

  According to the stereo image processing apparatus according to the present invention, it is possible to correct the parallax of the stereo camera accurately in a short time without including errors and disturbances without using many sensors.

It is a functional block diagram of a stereo image processing device concerning one embodiment of the present invention. It is a functional block diagram of a parallax correction processing unit according to an embodiment of the present invention. It is a flowchart of the parallax correction program which concerns on one Embodiment of this invention. It is explanatory drawing which shows an example of the image ahead of the own vehicle image | photographed with the vehicle-mounted camera which concerns on one Embodiment of this invention. It is explanatory drawing of the calculation method of two stop objects ahead of the own vehicle which concerns on one Embodiment of this invention. It is explanatory drawing of the relationship of the distance and parallax to two stop objects which concern on one Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described based on the drawings.

  As shown in FIG. 1, the stereo image processing apparatus 1 according to the present embodiment is mounted on a mobile object such as a car. In addition, below, the case where this stereo image processing apparatus 1 is mounted in a vehicle is illustrated and demonstrated.

  The stereo image processing apparatus 1 has an image processing unit 2 mainly composed of a computer, and on the input side of the image processing unit 2, a stereo camera 5 via an A / D converter 3, 4 and a navigation system 6 are connected.

  The stereo camera 5 is composed of a main camera 5a and a sub camera 5b incorporating image sensors such as CCDs and CMOSs, and both the cameras 5a and 5b are in front of the ceiling of the vehicle interior and in the vehicle width direction. It is mounted at equal intervals left and right across the center, and performs stereo imaging from different viewpoints of the environment outside the vehicle.

  The main camera 5a captures a reference image (right image) necessary when performing stereo image processing, and the sub camera 5b captures a comparison image (left image). The respective analog images output from the cameras 5 a and 5 b are converted into digital images by the A / D converters 3 and 4 and transmitted to the image processing unit 2 in a state where they are synchronized with each other.

  The navigation system 6 is a well-known system, for example, receives radio signals from GPS (Global Positioning System) satellites and acquires vehicle position information (latitude, longitude), and vehicle speed from a vehicle speed sensor In addition, movement direction information is acquired by a geomagnetic sensor or a gyro sensor or the like. Then, the navigation system includes a navigation ECU that generates route information for realizing the navigation function, a map database that stores map information (supplier data and updated data), and a display unit such as a liquid crystal display (for example) As mentioned above, all are provided and comprised.

  The navigation ECU superimposes route information to the destination designated by the user on the map image and displays it on the display unit, and the current position of the vehicle based on the detected information such as the position, speed, and traveling direction of the vehicle. Is superimposed on the map image on the display unit. Further, in the map database, information necessary for constructing a road map, such as node data and facility data, is stored. The node data relates to the position and shape of the road that constitutes the map image, and includes, for example, the coordinates (latitude, longitude) of a point (node point) on the road including the branch point of the road (intersection) Direction of the road being used (for example, information such as expressway, arterial road, city road), type of road at the node point (straight section, arc section (arc curve section), clothoid curve section (relaxation section)) and Curve curvature (or radius) data is included. Therefore, the travel path of the own vehicle is specified by the position on the map where the current position of the vehicle is overlapped, and the travel path of the own vehicle is set as the target travel path of the road according to the information of the node point closest to the position of the own vehicle. Travel road information such as curve curvature (or radius), road direction, etc. is acquired. Furthermore, traffic lights, signs, and facility data (ie, stop data) include data on facility information existing near each node point, and are stored in association with the node data (or link data in which the node is present). ing. As described above, the navigation system 6 provides lane information based on the vehicle position information and the map information, and is provided as map information acquisition means.

  Then, based on the input signals from the stereo camera 5 and the navigation system 6, the image processing unit 2 sets two stop objects (this is different from the same frame of the pair of image information from the stereo camera 5). In the embodiment, as an example of the stop object, as shown in FIG. 4, a traffic signal (for example, a distant traffic signal 21A and a nearby traffic signal 21B) is detected, and based on the map information from the navigation system 6, Two traffic signals corresponding to two traffic signals based on the two traffic signals are detected, a distance between two stationary objects based on image information is calculated as a first distance between stationary objects DCABS, and a distance between two stationary objects based on map information is second Between the first stationary object distance DCAB and the second stationary object distance DMAB, and calculates the second stationary object distance DMAB. The parallax of the stereo camera 5 is corrected so that the first stationary object distance DCAB coincides with the second stationary object distance DMAB (as true value), and distance data is determined using the corrected parallax to obtain an image. Is configured to recognize and output.

  Therefore, the image processing unit 2 mainly includes an image correction unit 11, a stereo image processing unit 12, a parallax correction processing unit 13, a distance data storage unit 14, an image data storage unit 15, and an image recognition unit 16. There is.

  The image correction unit 11 performs luminance correction, geometrical conversion of an image, and the like on image information. Usually, there is an error in the mounting position of the pair of left and right cameras 5a and 5b, and the displacement resulting therefrom is generated in the left and right images. In order to correct this deviation, geometrical transformation such as rotation or translation of the image is performed using affine transformation or the like. By such image correction processing, reference image data is generated from the main camera 5a based on the image data, and comparison image data is generated based on the image data from the sub camera 5b. The two image data are sent to the stereo image processing unit 12.

  The stereo image processing unit 12 is based on the reference image data and the comparison image data, regarding the captured image equivalent to one frame, the distance Z to the object using the principle of triangulation from the parallax Z for the same object D Is calculated from the following equation (1).

D = f · L / Z (1)
Here, f is a focal length, and L is a distance between the left and right cameras 5a and 5b. Here, D is the distance from the position of the focal distance f to the object (see FIG. 6: in FIG. 6, the distance D is exemplified by LCA or LCB).

  Further, the parallax Z is a shift amount in the horizontal direction between the reference image data obtained by photographing the same object and the comparison image data, and is obtained by multiplying the number of parallax pixels x by the pixel pitch P (Z = x ··· P). Then, the distance data is output to the parallax correction processing unit 13.

  The parallax correction processing unit 13 receives map information from the navigation system 6, receives distance data from the stereo image processing unit 12, and, as shown in FIG. 2, includes a learning correction value calculation unit 13a and a position correction unit 13b. ing.

  Then, the learning correction value calculation unit 13a outputs two stop objects that differ in distance from the vehicle from the same frame of the pair of image information (in the present embodiment, as shown in FIG. 2) detects a distant traffic light 21A and a nearby traffic light 21B), detects two traffic lights corresponding to two traffic lights based on image information based on the map information from the navigation system 6, and 2 based on the image information The distance between two stationary objects is calculated as a first distance between stationary objects DCABS, the distance between two stationary objects based on the map information is calculated as a second distance between stationary objects DMAB, and the first distance between stationary objects DCABS and Comparing with the second stationary object distance DMAB, the first stationary object distance DCAB matches the second stationary object distance DMAB with the second stationary object distance DMAB as a reference (as a true value) As to correct the parallax of the stereo camera 5 (to correct the number of parallaxes pixels) is calculated learning correction value Kx, and outputs the position correction unit 13b.

  Specifically, as shown in FIG. 5, as a stopped object from the same frame of the image in front of the host vehicle 20 taken by both cameras 5a and 5b, it is set up near the traffic light 21A at a distance and more Based on the map information from the navigation system 6, two traffic signals corresponding to the two traffic lights 21A and 21B are detected based on the image information.

  Then, the distance (first distance between stopped objects) DCAB between the distant signal 21A and the nearby signal 21B based on the image information is calculated by the following equation (2).

DCA = LCA-LCB (2)
Here, LCA is the distance from the host vehicle to the traffic signal 21A at a distance, and LCB is the distance from the host vehicle to the nearby traffic signal 21B.

  Further, from the map information of the navigation system 6, the distance (second distance between stopped objects) DMAB between the distant signal 21A and the nearby signal 21B is calculated directly.

  In this case, if the map information is a true value, that is, the second stationary object distance DMAB is regarded as a true value, there is no parallax shift in the stereo camera 5, and the value of the first stationary object distance DCAB is a true value. The second stop-to-stop distance DMAB and the first stop-to-stop distance DCAB coincide with each other. Conversely, when there is parallax shift in the stereo camera 5, the first stop object distance DCAB is also a shifted value, and the first stop object distance DCAB is different from the second stop object distance DMAB. Value.

  Further, as shown in FIG. 6, the parallax ZB detected when measuring the distance LCB of the nearby traffic light 21B is larger than the parallax ZA detected when measuring the distance LCA of the distant traffic light 21A, The influence of the deviation is smaller in the distance LCB detected by the nearby traffic light 21B.

  Therefore, in the present embodiment, when the second stop object distance DMAB is different from the first stop object distance DCAB, the second stop object distance is determined based on the distance LCB of the nearby traffic light 21B. The parallax of the stereo camera 5 is corrected such that the first stationary object distance DCAB matches the second stationary object distance DMAB with the DMAB as a reference (true value).

  As described above, the parallax Z = x · P, and parallax displacement appears in the number of parallax pixels x. In this embodiment, based on the distance LCB of the nearby traffic light 21B based on the image information and the distance LCA of the distant traffic light 21A, the first inter-stop object distance DCAB matches the second inter-stop object distance DMAB. The parallax pixel number x of the distance LCA of the distant traffic light 21A is corrected by the learning correction value Kx. Then, the learning correction value Kx is held until a new learning correction value Kx is calculated in the next and subsequent calculations.

  Hereinafter, a specific example of the learning correction value Kx calculated by the learning correction value calculation unit 13a will be shown. In this case, the specifications of the stereo camera 5 are f = 5 [mm], L = 300 [mm], P = 0.005 [mm], and LCA obtained by the image information shown in FIG. 5 = 85 [m]. , LCB = 50 [m], DCAB = 35 [m], DMAB obtained by map information = 32 [m]. Therefore, the distance between the distant traffic signal 21A obtained from the map information and the nearby traffic light 21B is equal to the distance between the nearby traffic light 21B as LCB = 50 [m] (reference). It becomes 50 + 32 = 82 [m].

  In the above example, the distance of the distant traffic signal 21A based on the image information is the following equation (3).

85 [m] = 5 [m] · 300 [mm] / Z1 (3)
Further, the distance of the distant traffic signal 21A based on the map information is expressed by the following equation (4).

82 [m] = 5 [m] · 300 [mm] / Z2 (4)
Here, assuming that the parallax pixel number before correction is the parallax pixel number x1, the parallax Z1 in the equation (3) is Z1 = x1 · 0.005 [mm], and is x1 = 3.529411765.

  Further, with the parallax pixel number after correction as the parallax pixel number x2, the parallax Z2 in the equation (4) is Z2 = x2 · 0.005 [mm], and is x2 = 3.658536585.

  Therefore, the parallax pixel number x is Kx = −0.129124821 in the above-described example when the subtraction correction value is defined as the learning correction value Kx.

  Then, a learning correction value Kx for subtracting and correcting the parallax pixel number x is registered, and the parallax pixel number x at the time of calculating the parallax Z is a learning correction value until a new learning correction value Kx is calculated in the subsequent calculation. Learning correction is sequentially performed with Kx (Z = (x−Kx) · P).

  The learning correction value Kx is read by the position correction unit 13 b. The position correction unit 13 b performs learning correction of the parallax Z set by the stereo image processing unit 12 with the learning correction value Kx (Z = (x−Kx) · P), and the distance of the object based on the parallax Z after learning correction Ask for D. The distance data storage unit 14 and the image data storage unit 15 respectively store the distance data obtained by learning correction of the parallax Z and the image data corresponding to the distance data. Thus, the parallax correction processing unit 13 is provided as parallax correction means.

  The image recognition unit 16 recognizes an object using the image data stored in the image data storage unit 15, and based on the distance data stored in the distance data storage unit 14, three-dimensional the object. Recognize a certain position.

  The three-dimensional position information of the object recognized by the image recognition unit 16 is appropriately read according to the purpose of use. For example, when the stereo image processing apparatus 1 is mounted on a car as a moving object, vehicles such as division lines, signs, crosswalks, guardrails, side walls, and leading vehicles that divide the left and right of the lane in which the host vehicle 20 travels. Pedestrians, three-dimensional objects such as traffic lights are objects to be recognized. In addition, when the stereo image processing apparatus 1 is mounted on a railway vehicle as a moving object, it is an object to be recognized by a railway rail, a passing vehicle, a passing person, and the like.

  Next, a parallax correction program executed by the parallax correction processing unit 13 will be described with reference to the flowchart of FIG.

  First, in step (hereinafter, abbreviated as “S”) 101, two stop objects (objects A and B) are detected from the same one frame together with distance information. In the example of the present embodiment, the two stop objects are the distant signal 21A and the nearby signal 21B, the distance LCA of the distant signal 21A, and the distance LCB of the nearby signal 21B.

  Next, in S102, it is determined whether or not two stop objects (objects A and B) have been detected. If the two objects can not be detected, the process proceeds to S103 and the next frame is searched.

  If two stationary objects (objects A and B) can be detected together with the distance information from the same one frame in S102, the process proceeds to S104, and the distance LCA of the distant signal 21A and the distance LCB of the near signal 21B. Is determined to be equal to or greater than a threshold value LD set in advance, and if it is closer to the threshold value LD, the process proceeds to S103, and the next frame is searched.

  Conversely, if the distance LCA of the distant traffic signal 21A and the distance LCB of the nearby traffic light 21B are greater than or equal to the preset threshold LD, the process proceeds to S105.

  At S105, the distance (first distance between stopped objects) DCAB between the distant signal 21A and the nearby signal 21B based on the image information is calculated according to the above-mentioned equation (2).

  Next, in S106, two traffic lights 21A, 21B corresponding to the two traffic lights 21A, 21B based on the image information are detected from the map information of the navigation system 6, and the distant traffic 21A and the near traffic 21B are detected. The distance (second stop object distance) DMAB is directly calculated.

  Next, in S107, it is determined whether the first stop object distance DCAB and the second stop object distance DMAB are equal (DCAB = DMAB).

  As a result of the determination in S107, in the case of DCAB = DMAB, it is determined that it is not necessary to correct the parallax of the stereo camera 5, and the program is exited.

  On the other hand, when it is determined that the first stationary object distance DCAB and the second stationary object distance DMAB are not equal, the process proceeds to S108, and the parallax of the stereo camera 5 is corrected. That is, as described above, the learning correction value calculation unit 13a calculates the learning correction value Kx for correcting the number of parallax pixels, performs learning correction of parallax, and exits the program.

  Thus, according to the embodiment of the present invention, based on the input signals from the stereo camera 5 and the navigation system 6, the distance from the own vehicle to the same frame of the pair of image information from the stereo camera 5 is different 2 Two stationary objects are detected, two traffic signals corresponding to two traffic signals based on image information are detected based on map information from navigation system 6, and a distance between two stationary objects based on image information is set as a first stationary object Calculated as the distance DCAB, calculated the distance between two stop objects based on map information as the second distance between stop objects DMAB, and compared the first distance between stop objects DCAB and the second distance between stop objects DMAB. , Stereo on the basis that the second stationary object distance DMAB is a reference (as true value) the first stationary object distance DCAB coincides with the second stationary object distance DMAB To correct the parallax of the camera 5. Therefore, the parallax can be corrected in one frame of the image information without the need for a plurality of sensors to detect the optical axis deviation or the like, and the accuracy of the parallax correction can be improved and the adjustment time can be shortened. be able to. Also, since the correction of parallax does not require the influence of a plurality of sensors or the correction time, accurate parallax correction can be performed without disturbance.

  The present invention is not limited to the above-described embodiment. For example, the stop object may be a utility pole, a sign, a central divider of a road, or a building other than a traffic light, and a sign on a road Or the like.

1 stereo image processing apparatus 2 image processing unit 5 stereo camera 6 navigation system (map information acquisition means)
11 image correction unit 12 stereo image processing unit 13 parallax correction processing unit (parallax correction means)
13a Learning correction value calculation unit 13b Position correction unit 14 Distance data storage unit 15 Image data storage unit 16 Image recognition unit 20 Own vehicle 21A Traffic light 21B Traffic light

Claims (3)

In a stereo image processing device that calculates distance information to the same object based on parallax of the same object of a pair of image information captured by a pair of cameras mounted on a moving object,
Map information acquisition means for acquiring map information;
Among the pair of image information, two stop objects having different distances from the moving body are detected only from each one frame at the same timing, and based on the map information, the two stop objects corresponding to the two stop objects based on the image information One stop object is detected, and the distance between the two stop objects based on the image information is calculated as the first stop object distance, and the distance between the two stop objects based on the map information is the second distance between the stop objects. The first stop object distance is compared with the second stop object distance, and the first stop object distance is the second stop object based on the second stop object distance. Parallax correction means for correcting the parallax so as to match the object distance;
A stereo image processing apparatus comprising: The parallax is acquired according to the number of parallax pixels in the horizontal direction of the pair of image information, and
The stereo image processing apparatus according to claim 1, wherein the parallax correction unit corrects the parallax by calculating a learning correction value for correcting the number of parallax pixels.   The parallax correction means is configured such that the first stop-to-stop distance is equal to the second stop-to-stop distance based on the position of the stop having a shorter distance from the moving body among the two stop targets. The stereo image processing apparatus according to claim 1 or 2, wherein the correction of the parallax is performed.

Images