JP2018025498A - Movement velocity computation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve both stability of three-dimensional object recognition and quick calculation of a moving speed for a three-dimensional object existing around an angle of view.SOLUTION: A stereo image processing unit 5 is configured to conduct stereo matching, using a pair of shot images to create a distance image serving as a distribution of a distance on the shot image. A three-dimensional object detection unit 8 is configured to calculate a distance representative value on the basis of a histogram of the distance existing in a section in each of a plurality of sections set on the distance image, and group the distance representative value for each section to detect a three-dimensional object existing on a road surface. A moving speed calculation unit 9 is configured to switch a method of calculating a moving speed of the three-dimensional object in accordance with an estimation value of a gradient of the distance representative value for the three-dimensional object existing around an angle of view.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a moving speed calculation device using a stereo camera.

  Patent Document 1 discloses a moving speed detection device that calculates the moving speed of a three-dimensional object existing ahead of the host vehicle based on distance data and an optical flow. In this detection device, the distance data calculated by stereo matching are grouped together in the horizontal direction and the distance direction, and together with parallax (for example, an average value or a mode value) representing the group. A three-dimensional object is recognized. Further, the optical flow of the three-dimensional object is detected based on a plurality of captured images arranged in time series. The moving speed of the three-dimensional object is obtained by substituting the position in the real space of the three-dimensional object specified by the distance data and the positional change amount of the three-dimensional object on the image specified by the optical flow into a predetermined mathematical formula. Is calculated by According to this detection device, the risk determination is effective even for a three-dimensional object such as a pedestrian jumping forward of the host vehicle by quickly calculating the moving speed of the three-dimensional object while recognizing the three-dimensional object with high reliability. Can be done. Further, in Patent Document 1, not only the optical flow is calculated for all pixel blocks on the image plane, but also the region for calculating the optical flow is limited to a part of the image plane, thereby reducing the amount of calculation. It is also described.

JP 2005-214914 A

  However, in Patent Document 1, the region in which the moving speed is calculated using the distance data and the optical flow is fixedly set as the whole or a part of the image plane, and the moving speed calculation method is dynamically changed. It does not switch.

  An object of the present invention is to achieve both the stability of the three-dimensional object recognition and the quick calculation of the moving speed for a three-dimensional object existing near the angle of view.

  In order to solve this problem, the present invention provides a moving speed calculation device having a stereo camera, a stereo image processing unit, a three-dimensional object detection unit, and a moving speed calculation unit. The stereo camera outputs a pair of captured images in time series by capturing a scene in front of the host vehicle. The stereo image processing unit performs stereo matching using a pair of captured images, and generates a distance image that is a distribution of distances on the captured image. The three-dimensional object detection unit calculates a distance representative value for each of a plurality of sections set on the distance image based on a histogram of distances existing in the section, and groups the distance representative values for each section. , To detect a three-dimensional object present on the road surface. The moving speed calculation unit switches the calculation method of the moving speed of the three-dimensional object according to the estimated value of the gradient of the distance representative value for the three-dimensional object existing near the angle of view.

  Here, in the present invention, an optical flow detection unit that detects an optical flow of a three-dimensional object based on captured images that move back and forth in time series may be further provided. In this case, when the estimated value of the gradient of the distance representative value is equal to or greater than a predetermined threshold, the moving speed calculation unit calculates the time derivative of the distance representative value that corresponds to the optical flow detected by the optical flow detection unit. Based on this, the moving speed of the three-dimensional object is calculated. In addition, when the estimated value of the gradient of the distance representative value is smaller than a predetermined threshold, the moving speed calculation unit calculates the moving speed of the three-dimensional object based on the time derivative of the three-dimensional object distance detected by the three-dimensional object detection unit. calculate.

  In the present invention, the optical flow detection unit may detect an optical flow with an image region outside the left and right sides of the host vehicle as a processing target.

  According to the present invention, for a three-dimensional object existing near the angle of view, the moving speed calculation method is dynamically switched according to the estimated value of the slope of the distance representative value, so that the stability of the three-dimensional object recognition and the movement speed can be improved. It is possible to achieve both a quick calculation.

Block diagram of a moving speed calculation device according to the present embodiment The figure which shows the picked-up image where the preceding vehicle exists near the angle of view Explanatory drawing of slope estimation of typical distance value

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram of a moving speed calculation device according to this embodiment. This moving speed calculation device 1 is mounted on a vehicle such as an automobile, and calculates the moving speed of a three-dimensional object such as a preceding vehicle existing in front of the host vehicle. A stereo camera that captures a scene in the monitoring area in front of the vehicle is attached in the vicinity of the room mirror. This stereo camera is composed of a pair of cameras 2 and 3, and each of the cameras 2 and 3 includes an image sensor (for example, a CCD or CMOS sensor). The main camera 2 captures a reference image (right image) necessary for performing stereo image processing, and the sub camera 3 captures a comparison image (left image). In a state where they are synchronized with each other, each analog image output from the cameras 2 and 3 is converted into a digital image of a predetermined luminance gradation (for example, RGB 256 gradation) via the A / D converter, and the image It is output to the correction unit 4.

  The image correction unit 4 performs brightness correction, image geometric conversion, and the like on the digital images output from the cameras 2 and 3. Usually, there is an error in the mounting position of the pair of cameras 2 and 3 although there is a difference in degree. Therefore, a shift caused by the error occurs in the left and right images. In order to correct this deviation, geometrical transformation such as image rotation and translation is performed using an affine transformation circuit or the like. Through such image processing, reference image data is obtained from the main camera 2, and comparison image data is obtained from the sub camera 3. Here, the image plane defined by the image data is expressed in the ij coordinate system, with the lower left corner of the image as the origin, the horizontal direction as the i coordinate axis, and the vertical direction as the j coordinate axis. A stereo image corresponding to one frame (one image display unit) is output to the subsequent stereo image processing unit 5 and stored in the subsequent image data memory 6.

  The stereo image processing unit 5 is composed of, for example, a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory) as a work area, and follows a program stored in the ROM. Various processes are executed. The stereo image processing unit 5 calculates a distance image related to a captured image corresponding to one frame based on the reference image and the comparison image. Here, the “distance image” is a distribution / set of the parallax d calculated for each small area in the image plane defined by the captured image of one frame, and each parallax d is a position (i in the image plane). , J). The unit for calculating the parallax d is a pixel block having a predetermined area (for example, 4 × 4 pixels) constituting a part of the reference image, and one parallax is calculated from one pixel block. For example, when the reference image is composed of 200 × 512 pixels, a parallax group corresponding to the maximum number of pixel blocks PBij (50 × 128) is calculated from the captured image corresponding to one frame. As is well known, the parallax d is a horizontal shift amount related to the pixel block PBij that is a calculation unit thereof, and has a large correlation with the distance to the object projected on the pixel block PBij. That is, the closer the object projected in the pixel block PBij is to the cameras 2 and 3, the larger the parallax d of the pixel block PBij is, and the farther the object is, the smaller the parallax d is (if the object is infinitely distant, d becomes 0).

  When calculating the parallax d regarding a certain pixel block PBij (correlation source), a region (correlation destination) having a correlation with the luminance characteristic of the pixel block PBij is specified in the comparison image. As described above, the distance from the cameras 2 and 3 to the object appears as a horizontal shift amount between the reference image and the comparison image. Therefore, when searching for the correlation destination in the comparison image, it is only necessary to search on the same horizontal line (epipolar line) as the j coordinate of the pixel block Pij that is the correlation source. The stereo image processing unit 5 shifts the correlation between the correlation source and the correlation destination candidate while shifting one pixel at a time on the epipolar line within a predetermined search range set based on the i coordinate of the correlation source. Sequential evaluation (stereo matching). In principle, the amount of horizontal deviation of the correlation destination (one of the correlation destination candidates) determined to have the highest correlation is the parallax d of the pixel block PBij.

  The correlation between the two pixel blocks can be evaluated by calculating the luminance difference absolute sum CB as shown in Equation 1. In the equation, p1ij is the luminance value of the ijth pixel of one pixel block, and p2ij is the ijth luminance value of the other pixel block. The luminance difference absolute sum CB is the sum of the differences (absolute values) of the luminance values p1ij and p2ij corresponding to each other in the entire pixel block, and the smaller the difference is, the greater the correlation between the two pixel blocks is. Yes. Basically, of the luminance difference absolute sum CB calculated for each pixel block existing on the epipolar line, the pixel block having the smallest value is determined as the correlation destination. Then, the amount of deviation between the correlation destination and the correlation source specified in this way becomes the parallax d. The distance image calculated through such processing, that is, a set of parallax d associated with the position (i, j) on the image is stored in the distance data memory 7. The correlation evaluation between blocks is not limited to the luminance difference absolute sum CB, and any correlation evaluation method including the luminance difference square sum can be used.

[Formula 1]
CB = Σ | p1ij−p2ij |

  The arithmetic unit including the three-dimensional object detection unit 8, the optical flow detection unit 9, and the movement speed calculation unit 10 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory) as a work area. It comprises a microcomputer provided, and executes various processes according to programs stored in the ROM. The three-dimensional object detection unit 8 determines the shape of the road in the three-dimensional space based on the own vehicle based on the captured image read from the image data memory 6 and the distance image read from the distance data memory 7. Calculate the prescribed road model. The road model is expressed by a linear expression in the horizontal direction and the vertical direction in the coordinate system of the real space. Plants that exist along the road, solid objects such as pylons and guardrails, or features such as white lines (or yellow lines) that define lanes drawn on the road surface are effective in recognizing road shapes. . The three-dimensional object detection unit 8 recognizes the positions in the real space of these characteristic objects existing along the road, and calculates a road model by connecting these positions. The position (x, y, z) of the feature in the real space is uniquely determined from a well-known coordinate conversion formula based on the position (i, j) on the distance data and the parallax d calculated with respect to this position. Calculated.

  The three-dimensional object detection unit 8 calculates a distance representative value for each of a plurality of sections set on the distance image based on a histogram of distances existing in the section, and groups the distance representative values for each section. By doing so, a three-dimensional object existing on the road surface is detected. Specifically, first, a plurality of vertical strip-shaped sections are set by dividing a distance image of one frame in the horizontal direction. Each section is defined as an area excluding the lower side of the image where the vicinity of the ground is projected and the upper side of the image where the sky is projected. Next, for each segment, a histogram of the distance values present within it is generated. After dividing the distance range into predetermined lengths (sections), all the distance values existing in a certain category are voted for the corresponding section. Then, a distance representative value that represents a group of distance values in the section is specified for each section. This distance representative value is a distance value having an appearance frequency equal to or higher than a predetermined threshold value, and is selected as a single value or a plurality of values within the same section. Then, in the processing space defined by the division and the distance, the distance representative values adjacent to each other (those whose front-rear and lateral distances are close to each other) are regarded as the same object and grouped on the road surface. An existing solid object is detected. The position (x, y, z) in the real space of the three-dimensional object is divided into the position (i, j) on the distance data and the parallax d (representative parallax) calculated with respect to this position, as in the case of the feature object. Based on this, it is uniquely calculated from a well-known coordinate conversion formula. Through the above processing, a three-dimensional object existing on the road surface is detected.

For details regarding the road model that defines the road shape and the recognition of the three-dimensional object, JP-A-5-265547, JP-A-6-266828, JP-A-10-283461, JP-A-10-283477 are disclosed. JP-A-11-213138, JP-A-2001-29970, etc., so refer to them if necessary.

  The optical flow detection unit 9 detects the optical flow of a three-dimensional object based on captured images that move back and forth in time series. Specifically, first, for example, a captured image (reference image) for two frames is read from the image data memory 6. Next, for each pixel block constituting the previous captured image, a region (correlation destination) having a correlation with the luminance characteristic of the pixel block (correlation source) is specified in the subsequent captured image. When detecting an optical flow, unlike the stereo matching process described above, the correlation destination having the highest correlation is obtained by offsetting the pixel block as a correlation source one pixel at a time in the horizontal / vertical direction over the entire area of the subsequent captured image. Is identified (two-dimensional matching). By this two-dimensional matching, the positional change amount (Δj) in the vertical direction and the positional change amount (Δi) in the horizontal direction are detected as optical flows for each pixel block. The optical flow detected for each pixel block is associated with the position (i, j) on the image plane.

  In the present embodiment, the optical flow detection unit 9 detects the optical flow not using the entire captured image as a processing target but using only an image area outside the left and right sides of the host vehicle as a processing target. As will be described later, the optical flow is used only in the vicinity of the angle of view of the stereo camera. Therefore, the processing load is limited to reduce the calculation load.

  The movement speed calculation unit 10 basically calculates the movement speed of the three-dimensional object based on the time differentiation of the distance of the three-dimensional object output from the three-dimensional object detection unit 8. However, for the three-dimensional object existing near the angle of view of the stereo camera, the moving speed calculation unit 10 switches the calculation method of the moving speed of the three-dimensional object according to the estimated value of the gradient of the distance representative value described above. As shown in FIGS. 2A to 2D, when a preceding vehicle is cut in from the side of the angle of view (when interrupting or overtaking), the distance output from the three-dimensional object detection unit 8 (for example, the distance representative value) If the moving speed obtained by simply differentiating the average value of the vehicle) is used, the side surface R of the preceding vehicle gradually appears, but the position does not change so much that the relative speed should be higher than that of the host vehicle. . In addition, since the traveling direction of the preceding vehicle cannot be estimated, the response of the host vehicle to the cut-in is delayed.

  FIG. 3 is an explanatory diagram of the inclination estimation of the distance representative value. With the host vehicle as the origin, the horizontal direction is the X axis and the depth (distance) direction is the Z axis. As shown in FIG. 5A, when the preceding vehicle R is traveling in parallel with the host vehicle (when the rear surface of the preceding vehicle is not visible), the side surface of the preceding vehicle R projected by the streo camera is displayed. The resulting representative distance value groups Zr (0) to Zr (n) are arranged substantially parallel to the Z axis. In this case, in the XZ coordinate system, when the slopes of the representative distance value groups Zr (0) to Zr (n) are estimated by the least square method, the estimated value becomes maximum. Although the method of calculating the moving speed based on the time derivative of the distance of the preceding vehicle R detected by the three-dimensional object detection unit 8 is superior in terms of stability of three-dimensional object recognition, in this state, it is more suitable for cut-in. There is concern that the response of the vehicle will be delayed.

  Next, as shown in FIG. 5B, when there is a preceding vehicle R that is going to interrupt the traveling path of the host vehicle (when the rear surface of the preceding vehicle R cannot be seen), the side surface of the preceding vehicle R is shown. The representative distance value groups Zr (0) to Zr (n) resulting from the above form a large negative slope. In this case, if the slopes of the representative distance value groups Zr (0) to Zr (n) are estimated by the least square method in the XZ coordinate system, the estimated value becomes relatively large. Also in this state, there is a concern that the response delay of the host vehicle with respect to cut-in is more than the stability of the three-dimensional object recognition.

  Further, as shown in FIG. 5 (c), when there is a preceding vehicle R to be interrupted on the traveling path of the own vehicle (when the rear surface of the preceding vehicle R is visible), the side surface of the preceding vehicle R A part of the upper representative distance value group Zr (0) to Zr (n) forms a large negative inclination, but the remaining part resulting from the rear surface of the preceding vehicle R forms a positive inclination. In this case, if the slopes of the representative distance value groups Zr (0) to Zr (n) are estimated by the least square method in the XZ coordinate system, the estimated value becomes relatively small. In this state in which the back surface of the preceding vehicle R is visible, since the position change of the back surface is large, the response delay of the own vehicle with respect to cut-in hardly poses a problem.

  Therefore, if the inclination of the representative distance value group Zr (0) to Zr (n) calculated for the preceding vehicle R is estimated by the least square method and this estimated value is compared with a predetermined threshold value, the host vehicle with respect to cut-in It is possible to appropriately determine whether or not the response delay is a problem. Then, according to the determination result, the time derivative of the distance corresponding to the optical flow detected by the optical flow detector 9 and the time derivative of the distance related to the three-dimensional object detected by the three-dimensional object detector 8 are obtained. By switching to either one, the moving speed of the preceding vehicle R is calculated. Specifically, the moving speed calculation unit 10 selects a distance corresponding to the optical flow detected by the optical flow detection unit 9 when the estimated value of the inclination is equal to or greater than a predetermined threshold. As this distance, the distance calculated for the pixel block in which the optical flow is detected or the distance representative value of the section including this pixel block is used, and the average time derivative of the entire region where the preceding vehicle R exists is calculated as the moving speed. It can be. On the other hand, when the estimated value of the inclination is smaller than the predetermined threshold value, the moving speed calculation unit 10 is based on the time derivative of the distance of the preceding vehicle R detected by the three-dimensional object detection unit 8 as usual. Calculate the moving speed. The distance used for speed calculation may be limited to a predetermined distance (for example, within 10 m).

  As described above, according to the present embodiment, for a three-dimensional object that exists in the vicinity of the angle of view of the stereo camera, the moving speed calculation method is dynamically switched according to the estimated value of the inclination of the distance representative value. It is possible to achieve both the stability of recognition and the quick calculation of the moving speed.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to this embodiment. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Is done. For example, in the above-described embodiment, the stereo image processing unit 5 and the arithmetic unit (the three-dimensional object detection unit 8, the optical flow detection unit 9, and the moving speed calculation unit 10) are a central processing unit (CPU), a ROM, a RAM, and the like. However, the present invention is not limited to this, and may be configured by an integrated circuit such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Further, it may be configured by a single central processing unit, FPGA, ASIC, or may be configured by combining a plurality of these.

  Further, a program for causing a computer to function as the above-described moving speed calculation device 1 or a computer-readable flexible disk, magneto-optical disk, DRAM, SRAM, ROM, NVRAM, CD, DVD, BD on which the program is recorded. Such a storage medium is also provided as another embodiment of the present invention. Here, the program refers to data processing means described in an arbitrary language or description method.

  Further, the steps constituting the processing executed by the moving speed calculation device 1 described above do not necessarily have to be processed in time series in the order described in the above embodiment, and may include processing in parallel or by a subroutine. Good.

DESCRIPTION OF SYMBOLS 1 Movement speed calculating device 2 Main camera 3 Sub camera 4 Image correction part 5 Stereo image processing part 6 Image data memory 7 Distance data memory 8 Solid object detection part 9 Optical flow detection part 10 Movement speed calculation part

Claims (3)

In the moving speed calculation device,
A stereo camera that outputs a pair of captured images in time series by imaging the scenery in front of the host vehicle;
A stereo image processing unit that performs stereo matching using the pair of captured images and generates a distance image that is a distribution of distances on the captured image;
For each of a plurality of sections set on the distance image, a distance representative value is calculated based on a histogram of distances existing in the section, and the distance representative value for each section is grouped to thereby calculate the distance on the road surface. A three-dimensional object detection unit for detecting a three-dimensional object existing in
A moving speed calculation unit that switches a calculation method of the moving speed of the three-dimensional object according to an estimated value of the inclination of the distance representative value for the three-dimensional object existing near the angle of view. apparatus. Further comprising an optical flow detection unit for detecting an optical flow of the three-dimensional object based on the captured images moving back and forth in time series,
The moving speed calculation unit
If the estimated value of the slope of the distance representative value is greater than or equal to a predetermined threshold, the three-dimensional object is determined based on the time derivative of the distance representative value that corresponds to the optical flow detected by the optical flow detector. Calculate the moving speed of the object,
When the estimated value of the slope of the distance representative value is smaller than a predetermined threshold, the moving speed of the three-dimensional object is calculated based on the time derivative of the distance of the three-dimensional object detected by the three-dimensional object detection unit. The moving speed calculation device according to claim 1, wherein   3. The moving speed calculation device according to claim 1, wherein the optical flow detection unit detects an optical flow using an image region outside the left and right sides of the host vehicle as a processing target. 4.

Images