JP2019190847A - Stereo camera device - Google Patents

Abstract

To provide a stereo camera device capable of reducing a calculation cost while ensuring calculation accuracy of a parallax offset.SOLUTION: A stereo camera device includes: a feature point detection unit 220 for detecting feature points of a plurality of images captured by a left camera and a right camera; a feature point tracking unit 240 for tracking the feature points between the plurality of images captured at different times in each of the left camera and the right camera; a tracking range determination unit 230 for limiting feature point tracking for the plurality of images in the feature point tracking unit 240 to a tracking range of a part of the plurality of images; and a parallax offset correction amount calculation unit 250 for calculating a parallax offset correction amount for correcting a parallax offset between the left camera and the right camera on the basis of a movement distance of a vehicle 1 between different times and a parallax about the feature points tracked by the feature point tracking unit 240.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a stereo camera device.

  Stereo camera systems are known as means for estimating the three-dimensional position of an object. In a stereo camera system, the same object is imaged from a plurality of different viewpoints by cameras arranged at a plurality of positions, and the object is based on a shift in view (so-called parallax) in a plurality of images respectively obtained by the plurality of cameras. Calculate the distance to the object. In addition, in the calculation of the distance to the object in the stereo camera system, in addition to information such as parallax, an offset error that occurs in parallax called parallax offset is considered. This parallax offset changes over time due to distortions in devices such as cameras due to vibration and temperature changes, and physical loads that occur during screw tightening, so in order to accurately calculate the distance to the object, the parallax offset The offset needs to be calibrated.

  In a stereo camera system mounted on a moving body such as a vehicle, the distance change of the stationary object relative to the vehicle is calculated based on the parallax obtained from images obtained by capturing the still object with a plurality of cameras, and the distance change and the movement of the moving object A calibration method is known in which the parallax offset is calculated so that the distance change and the movement amount are equal by comparing the amounts. However, with this calibration method, the parallax offset can be calculated only when an object that can be determined to be a stationary object is imaged, and a sufficient opportunity to calibrate the parallax offset cannot be ensured.

  Therefore, as a technique for increasing the calibration opportunity of the parallax offset, for example, Patent Document 1 is a stereo camera device that detects a distance to an object, and the device is installed apart by a base line length. Two cameras, and calculation means for calculating a distance from an image acquired by the two cameras to an object on the image, wherein the calculation means is an image acquired by the two cameras. An image processing unit that searches for corresponding points of the corresponding point and calculates two parallaxes from a difference in position coordinates of the corresponding points on the imaging surface, and the image from the two parallaxes defined by the image processing unit at least at two times Offset value calculation means for calculating the parallax offset value over the whole, and the parallax offset value used as a calibration parameter by statistically analyzing the distribution of the parallax offset value And a statistical processing means for determining an optimum value, the stereo camera apparatus is disclosed.

JP 2012-83330 A

  However, in the above-described prior art, although the moving distance of the moving object is calculated by using the parallax of the feature points detected on the images captured by a plurality of cameras, the opportunity to calibrate the parallax offset is increased. If it is attempted to acquire parallax by tracking feature points as a whole, the processing load becomes very high, and it is considered that parallax cannot be acquired for a sufficient number of feature points considering the calculation cost. In addition, since feature points on an image cannot always be detected accurately, it may be possible to calculate an incorrect parallax offset by mistracking similar points on the image as feature points. It is done.

  The present invention has been made in view of the above, and an object of the present invention is to provide a stereo camera device capable of reducing the calculation cost while ensuring the calculation accuracy of the parallax offset.

  The present application includes a plurality of means for solving the above-described problems. To give an example, a parallax offset between an imaging device having a plurality of imaging units arranged in a vehicle and the plurality of imaging units is corrected. A parallax image generating unit that generates a parallax image based on a plurality of images respectively captured by the plurality of imaging units, and a target for the plurality of imaging units based on the parallax image In a stereo camera device comprising an image processing device having a relative position calculation unit for calculating a relative position of an object, the image processing device detects feature points of the plurality of images respectively captured by the plurality of imaging units. A feature point detecting unit, a feature point tracking unit that tracks the feature point between the plurality of images captured at different times in each of the plurality of imaging units, and the feature point A tracking range determining unit that limits tracking of the feature points for the plurality of images in a trace unit to a tracking range of a part of the plurality of images, and a moving distance of the vehicle between the different times and the feature point tracking unit. A parallax offset correction amount calculation unit that calculates a parallax offset correction amount for correcting a parallax offset between the plurality of imaging units based on the parallax with respect to the tracked feature point is provided.

  According to the present invention, it is possible to reduce the calculation cost while ensuring the calculation accuracy of the parallax offset.

It is a figure which shows the hardware constitutions of the stereo camera apparatus which concerns on this Embodiment which concerns on 1st Embodiment. It is a functional block diagram which extracts and shows the processing function of the image processing apparatus which concerns on 1st Embodiment with a related structure. It is a figure explaining the basic principle of parallax offset correction amount calculation processing. It is a functional block diagram which shows the processing function of a parallax offset correction | amendment part. It is a functional block diagram which shows the processing function of the feature point detection area | region determination part. It is a figure explaining an example of the area | region in the image used as the object of exclusion by the feature point detection area determination part. It is a figure explaining an example of the area | region in the image used as the exclusion object by the feature point detection area | region determination part, and is a figure which shows an example of the area | region excluded by the parallax non-calculation area | region exclusion part. It is a figure explaining an example of the area | region in the image used as the exclusion object by a feature point detection area determination part, and is a figure which shows an example of the area | region excluded from by a parallax unstable area removal part. It is a figure explaining an example of the area | region in the image used as the exclusion object by the feature point detection area determination part, and is a figure which shows an example of the area | region excluded from by a specific object area | region exclusion part. It is a functional block diagram which shows the processing function of a tracking range determination part. It is a figure which shows the mode of the tracking of the feature point by the feature point tracking part. It is a figure which shows the mode of the tracking of the feature point by the feature point tracking part. It is a figure which shows typically an example of the calculation method of the imaging expected area (search range) in an imaging expected area calculation part. It is a figure which shows typically an example of the calculation method of the imaging expected area (search range) in an imaging expected area calculation part. It is a functional block diagram which shows the processing function of the parallax offset correction | amendment part which concerns on 2nd Embodiment. It is a functional block diagram which shows the processing function of the parallax offset calculation determination part which concerns on 2nd Embodiment. It is a figure which shows the list of conditions in the case where it determines with a parallax offset correction amount calculation determination part not performing a parallax offset correction amount calculation process. It is a functional block diagram which shows the processing function of the parallax offset correction amount calculation determination part which concerns on 3rd Embodiment.

  Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment>
A first embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 is a diagram illustrating a hardware configuration of the stereo camera device according to the present embodiment.

  In FIG. 1, a stereo camera device 100 includes a stereo camera 110 (imaging device) having a pair of left and right cameras (imaging units) of a left camera 111 and a right camera 112, a read-only memory (ROM) that is a storage device, and random access. An image processing apparatus 120 including a memory 130 such as a memory (RAM) and a central processing unit (CPU) 140 that is a processor, and a stereo camera apparatus 100 and an external apparatus 160 such as an external sensor or other apparatus. And an interface 150 for performing communication by CAN (Control Area Network) communication between the stereo camera 110, the image processing apparatus 120, and the interface 150 on a communication path for exchanging information with each other. They are connected by a certain bus line 100a.

  In the present embodiment, the stereo camera device 100 is an in-vehicle stereo camera device mounted on a vehicle 1 such as an automobile (see FIG. 3 and the like), and the external device 160 includes an accelerator opening (that is, a throttle opening). ), Brake operation amount (ie, brake pedal operation amount), steering angle, vehicle speed, acceleration, temperature, humidity, and other sensors, as well as ECU (Electronic Control) that controls various operations of the vehicle 1 Unit). The various information acquired via the interface 150 may be various information such as engine rotation flowing on the CAN communication path (CAN bus), vehicle speed, wheel speed, steering angle, throttle opening, and the like. The ECU may calculate the vehicle speed from information obtained by a sensor or the like that detects the rotational speed of the wheel, and obtain the result of flowing through the CAN communication path. In addition, although not shown, the image processing apparatus 120 is provided with an element temperature sensor that detects the temperature of the memory 130, the CPU 140, and the like as elements disposed on the substrate, for example.

  The memory 130 is a recording medium in which a control program used for various processes in the image processing apparatus and various types of information are stored. The CPU 140 performs predetermined arithmetic processing on a signal taken in via the interface 150 in accordance with a control program stored in the memory 130, detects a three-dimensional object or a road surface, and a position of the object 2 (see FIG. 3) (object). (Distance and direction up to 2) are calculated. A calculation result by the CPU 140 is output to an external device 160 (ECU or the like) via the interface 150, and is used for determination and control of various operations of the vehicle 1 such as an accelerator, a brake, and a steering. The semiconductor memory such as ROM and RAM has been exemplified as the memory 130 (storage device) of the image processing apparatus 120. However, any other storage device may be used, for example, a magnetic storage device such as a hard disk drive.

  A plurality of (two in this embodiment) cameras (the left camera 111 and the right camera 112) of the stereo camera 110 are arranged at a predetermined optical axis interval (base line length) so that their optical axes are parallel to each other. Each is constituted by an image sensor such as a CCD or CMOS, an optical lens, or the like. The stereo camera 110 is an in-vehicle stereo camera, and is arranged so that the front of the vehicle 1 is imaged by the left camera 111 and the right camera 112.

  FIG. 2 is a functional block diagram showing extracted processing functions of the image processing apparatus according to the present embodiment together with related components.

  In FIG. 2, the image processing apparatus 120 calculates a correction amount (parallax offset correction amount) for correcting an error in parallax between the left camera 111 and the right camera 112 of the stereo camera 110 (hereinafter referred to as parallax offset). Parallax for generating a parallax image (an image with parallax as a pixel value) using a parallax offset correction amount based on the parallax offset correction unit 121 to be output and a plurality of images captured by the left camera 111 and the right camera 112, respectively. The image generation unit 122 and a distance calculation unit 123 (relative position) that calculates the relative position of the object 2 with respect to the stereo camera 110 (distance and direction to the object 2) based on the parallax image generated by the parallax image generation unit 122 Calculation unit). The relative position (distance and direction to the object 2) of the object 2 calculated by the distance calculation unit 123 is sent to the external device 160 via the interface 150, and is also stored in the memory 130 of the image processing apparatus 120. The

  The parallax offset is an offset error that occurs in the parallax. The value of the parallax offset is determined in advance depending on the installation positions of the left camera 111 and the right camera 112 constituting the stereo camera 110, but the device is distorted due to vibrations and temperature changes that occur after installation, the physical load generated during screw tightening, etc. Changes over time. In the parallax offset correction unit 121 according to the present embodiment, a parallax offset correction amount for appropriately correcting the parallax offset value is calculated according to the temporal change of the parallax offset. The parallax image generation unit 122 generates a parallax image by correcting the parallax based on the images obtained by the left camera 111 and the right camera 112 using the parallax offset correction amount calculated by the parallax offset correction unit 121. , Improve the accuracy of parallax images. Therefore, since the accuracy of the parallax image is improved, the occurrence of an error when the distance calculation unit 123 calculates the distance using the parallax image can be suppressed, so that the three-dimensional position of the object is accurately estimated. be able to.

  Here, the basic principle of the process of calculating the parallax offset correction amount (hereinafter referred to as the parallax offset correction amount calculation process) in the present embodiment will be described with reference to FIG.

  FIG. 3 is a diagram for explaining the basic principle of the parallax offset correction amount calculation processing.

  In FIG. 3, the distance between the position s1 and the target object (stationary object) at time t1 of the vehicle 1 on which the stereo camera device 100 of the present embodiment is mounted is L1, and the position s2 and the target object (stationary object) at time t2. ) Is the distance L2, and the distance between the position s1 of the vehicle 1 at time t1 and the position s2 of the vehicle 1 at time t2 is the movement amount dz. Since the relative position (attachment position) of the stereo camera device 100 to the vehicle 1 is known, the distance from the stereo camera 110 to the object 2 and the distance from the vehicle 1 to the object 2 are substantially the same. Think.

  In the stereo camera device 100 using two cameras (the left camera 111 and the right camera 112) as in the present embodiment, the distance Z [mm] from the stereo camera 110 (or the vehicle 1) to the object is as follows: It is represented by (Formula 1).

In the above (Formula 1), f [mm] is the focal length of the left camera 111 and the right camera 112, wi [mm / px] is the pixel pitch, and B [mm] is the distance between the optical axes of the left camera 111 and the right camera 112. Distance (ie, baseline length), d [mm] represents parallax, and ε [px] represents parallax offset.

  The stereo camera device 100 mounted on a moving body such as the vehicle 1 calculates the distance from the moving vehicle 1 to the target object 2 based on the parallax of the stationary target object 2 (stationary object). The change in the predetermined time (that is, the movement distance of the vehicle 1 based on the parallax of the object) and the movement amount of the vehicle 1 (movement distance of the vehicle 1 obtained from information other than the parallax) are compared with each other from the vehicle 1 to the object 2. The parallax offset of the stereo camera 110 can be estimated by selecting the parallax offset so that the change in the distance to the vehicle and the amount of movement of the vehicle 1 are equal. That is, by calculating the parallax offset correction amount so as to cancel the parallax offset, the accuracy of the parallax image generated by the parallax image generation unit 122 is improved, and the error in calculating the distance by the distance calculation unit 123 is reduced. Occurrence can be suppressed.

  If the parallaxes at a plurality of (two in the present embodiment) different times (time t1, time t2) shown in FIG. 3 are parallax d1 [px] and parallax d2 [px], respectively, the vehicle calculated from the parallax The change in distance from 1 to the object 2 and the amount of movement of the vehicle 1 satisfy the following (Equation 2).

  By solving (Equation 2) above for ε (parallax offset), the parallax offset can be estimated, and the parallax offset correction amount is calculated from this parallax offset.

  FIG. 4 is a functional block diagram illustrating processing functions of the parallax offset correction unit.

  In FIG. 4, the parallax offset correction unit 121 includes a plurality of images captured by the left camera 111 and the right camera 112 of the stereo camera 110 (here, 2 images captured by the left camera 111 and the right camera 112 at the same time). Feature point tracking unit that tracks feature points between a plurality of images captured at different times in the left camera 111 and the right camera 112 respectively. The feature point tracking unit 240, the tracking range determination unit 230 that limits the tracking range of the feature points in the feature point tracking unit 240 to a part of the image (determines the tracking range), and the feature points tracked by the feature point tracking unit 240 Parallax offset correction amount calculation to calculate the parallax offset based on the parallax of the vehicle and the movement distance of the vehicle between different times when the feature points are tracked For each of a plurality of images captured by the unit 250 and the left camera 111 and the right camera 112, the feature point detection unit 220 limits the area where the feature points are detected to a part of the image (determines the feature point detection area) And a feature point detection area determination unit 210.

  FIG. 5 is a functional block diagram illustrating processing functions of the feature point detection region determination unit.

  In FIG. 5, the feature point detection area determination unit 210 excludes an area where parallax is not obtained for each of a plurality of images captured by the left camera 111 and the right camera 112 of the stereo camera 110. For each of a plurality of images captured by the exclusion unit 211, the left camera 111, and the right camera 112, the parallax unstable region exclusion unit 212 that excludes a region with low parallax accuracy, and the left camera 111 and the right camera 112 Each of the plurality of images includes a specific object region exclusion unit 213 that excludes a predetermined specific object region.

  The feature point detection area determination unit 210 excludes areas where feature points should not be detected. In other words, the feature point detection area determination unit 210 restricts the areas where feature points are detected (that is, areas where feature points are detected (feature point detection areas)). To reduce processing time and improve accuracy. In the present embodiment, the value of the parallax offset is estimated using a time-series parallax change in a three-dimensionally stationary region. Therefore, it is not necessary to perform feature point detection in areas where there is no texture and where parallax cannot be calculated, areas with specific patterns with low parallax accuracy, and areas where moving objects are imaged. By determining the processed region as a region for detecting feature points (feature point detection region), it is possible to expect a reduction in processing time and an improvement in accuracy.

  6 to 9 are diagrams illustrating an example of an area in an image to be excluded by the feature point detection area determination unit. FIG. 6 is a diagram illustrating an example of an image handled by the feature point detection region determination unit. 7 shows an example of a region to be excluded by the parallax non-calculated region exclusion unit in the image shown in FIG. 6, and FIG. 8 shows a target to be excluded by the parallax unstable region exclusion unit in the image shown in FIG. FIG. 9 shows an example of an area, and FIG. 9 shows an example of an area to be excluded by the specific object area exclusion unit in the image shown in FIG.

  As illustrated in FIG. 7, the parallax non-calculated area exclusion unit 211 excludes areas where parallax is not calculated in the image 300 (parallax non-calculated areas 311 to 313). The exclusion target areas exemplified in the areas 311 to 313 are areas in the image 300 where, for example, there is no texture and parallax cannot be calculated. In other words, as the areas to be excluded, the area 311 in which the sky is captured in the image 300, and the areas 311 and 312 in which roads and the like are captured are conceivable. It should be noted that in selecting a region to be excluded from which parallax cannot be calculated, for example, in a parallax image generated at a past time point, a region where parallax is not obtained may be selected. In addition, using a numerical value such as a matching cost indicating how much the left and right images obtained when calculating the parallax match, a point (pixel) having a low degree of matching is a case where the parallax was obtained. However, it may be configured to be excluded.

  As illustrated in FIG. 8, the parallax unstable region exclusion unit 212 excludes a region (parallax unstable region 321) that may have low parallax accuracy in the image 300. The exclusion target area exemplified in the area 321 is an area 321 having a fine repetitive pattern such as a fence in the image 300, and there is a possibility that an erroneous parallax may be calculated in the process of generating a parallax image.

  As illustrated in FIG. 9, the specific object region exclusion unit 213 excludes regions (specific object regions 331 to 333) where the specific object is captured in the image 300. The specific object here is an object that is highly likely not to be stationary. That is, as the exclusion target area, areas 331 and 332 in which a vehicle that is likely to move is imaged in the image 300 and an area 333 in which a pedestrian is imaged can be considered. The extraction of the regions 331 to 333, that is, the extraction of a specific object such as a vehicle or a pedestrian, can be performed by a method such as pattern matching that performs determination based on a matching rate with a pattern that is held in advance. In addition to the specific object, for example, an object that is determined to be moving as a result of detecting and tracking another specific object may be excluded.

  6 to 9, the areas 311 to 113 where the parallax excluded by the parallax uncalculated area exclusion unit 211 is not calculated, and the areas 321 where the accuracy of the parallax excluded by the parallax unstable area exclusion unit 212 may be reduced. In addition, as the areas 331 to 333 in which the specific object to be excluded by the specific object area exclusion unit 213 is imaged, the areas to be excluded are expressed as a certain range. It is not necessary to be a region where pixels are connected, and it is determined that one pixel unit is a region, whether or not there is parallax, the accuracy, the presence or absence of imaging of a specific object, etc. You may determine whether it is any of the specific object area | regions 331-333.

  In addition, FIG. 5 illustrates a case where the processing of the feature point detection region determination unit 210 is performed in the order of the parallax non-calculated region exclusion unit 211, the parallax unstable region exclusion unit 212, and the specific object region exclusion unit 213. However, the present invention is not limited to this, and the feature point detection region determination unit 210 may be configured by changing the order of these processing functions.

  The feature point detection unit 220 is a feature point detection region determined by the feature point detection region determination unit 210 (that is, for example, in the image 300, the parallax uncalculated regions 311 to 113, the parallax unstable region 321 and the specific object region) One or more feature points are detected in a range other than the area excluded as 331 to 333. As a feature point detection method, for example, FAST (Features from Accelerated Segment Test) can be used. FAST is a method for quickly determining whether a certain point (pixel) has a luminance different from that of its surrounding pixels, and is often used as a method for finding a characteristic point at high speed.

  FIG. 10 is a functional block diagram illustrating processing functions of the tracking range determination unit. FIG. 11 and FIG. 12 are diagrams showing how feature points are tracked by the feature point tracking unit.

  As shown in FIGS. 11 and 12, the feature point tracking unit 240 sets a point corresponding to one or more feature points (here, the feature point p1 is illustrated) captured in the image 410 at a certain time t1. Search is made from a plurality of feature points detected on the image 420 at another time t2 after t1, and the feature points p2 are associated with the feature points p1 to track the feature points. Note that, in the present embodiment, only the feature point p1 among the one or more feature points will be described as an example.

  In the tracking of feature points by the feature point tracking unit 240, if an incorrect feature point is associated between the images 410 and 420 at the times t1 and t2, the parallax offset correction unit 121 causes the erroneous parallax offset to be incorrect. Therefore, the corresponding feature points p1 and p2 are accurately tracked between the images 410 and 420 having different imaging times t1 and t2. It will be necessary. Therefore, the tracking range determination unit 230 limits the tracking range of the feature points to be tracked by the feature point tracking unit 240 to a part of the image (determines the tracking range). Thereby, by suppressing the tracking of the feature point p1 in the feature point tracking unit 240 to the tracking range, the occurrence of the association with the erroneous feature point between the images 410 and 420 having different imaging times t1 and t2 is suppressed. Can do. Further, since the number of feature points to be determined as to whether or not they are corresponding feature points is reduced, a reduction in processing time can be expected.

  In FIG. 10, the tracking range determination unit 230 predicts that a certain feature point p1 on the image 410 captured at time t1 is captured on the image 420 captured at time t2 (imaging imaging position p11). ) And a feature point p2 corresponding to the feature point p1 on the image 420 taken at time t2 based on the expected imaging position p11 calculated by the expected imaging position calculation unit 231. And a predicted imaging region calculation unit 232 that calculates a range predicted to be captured (imaging predicted region 411) as a search range.

  The predicted imaging position calculation unit 231 calculates a position (imaging imaging position p11) where a certain feature point p1 on the image 410 captured at time t1 is predicted to be captured on the image 420 captured at time t2. To do. In the present embodiment, since the feature point detection region determination unit 210 uses only three-dimensionally stationary objects as detection points as feature points, the image 410 (time t1) and the image 420 (time t2) It can be said that the movement of the vehicle 1 (own vehicle) is the cause of the change of the imaging position of the feature point.

  Therefore, in the image 410 captured at time t1, a camera coordinate system of a certain feature point p1 (for example, a coordinate system based on the stereo camera device 100, the origin is set at the center of the stereo camera device 100, and z 3D position (X1, Y1, Z1) in the axis, the y-axis forward, and the x-axis rightward) and the image coordinate system in the image 410 (for example, the origin at the lower left corner of the images 410, 420) The position of the expected imaging position p11 in the image coordinate system in the image 420 taken at time t2 from the position (i1, j1) in the coordinate system in which the i-axis is set in the right direction and the j-axis is set in the upward direction. (I2, j2) is expressed by the following (formula 3) and (formula 4).

  In the above (Expression 3) and (Expression 4), f [mm] is a focal length, wi [mm / px] is a pixel pitch, and v [mm / s] is a vehicle obtained from a vehicle speed sensor included in the external device 160. The vehicle speed of 1 (own vehicle) is shown. In the present embodiment, for the sake of simplicity of explanation, a case where the vehicle 1 (own vehicle) is moving forward at a constant speed v is considered.

  The predicted imaging region calculation unit 232 captures the feature point p2 corresponding to the feature point p1 on the image 420 captured at time t2, based on the predicted imaging position p11 calculated by the predicted imaging position calculation unit 231. And a predicted range (imaging predicted region 411) are calculated as a search range. The search range calculated by the imaging expected region calculation unit 232 is output from the tracking range determination unit 230 to the feature point tracking unit 240 as a tracking range.

  By setting the predicted imaging region 411 and outputting it as a tracking range, the feature point tracking unit 240 can track with high accuracy when the feature point p1 is stationary. Even if there are many detected feature points in the image 420 taken at time t2, setting the tracking range allows the tracking of unnecessary feature points to be played out of the range without incurring calculation costs. Can do. This setting of the tracking range is particularly effective for a repetitive pattern such as a striped pattern (for example, a repetitive pattern that is not a removal target in the parallax unstable region removing unit 212), and similar feature points are displayed on the image. Even when a plurality of objects appear in the screen, it is possible to select and associate only points that are moving like a stationary object.

  FIG. 13 and FIG. 14 are diagrams schematically illustrating an example of a method for calculating a predicted imaging region (search range) in the predicted imaging region calculation unit.

  In the predicted imaging region calculation unit 232, the predicted imaging region 411 includes, for example, a direction in which an imaging position shift due to a parallax error is considered centering on the predicted imaging position p11 calculated by the predicted imaging position calculation unit 231 in the image 410. In a direction in which an imaging position shift due to pitching can be considered, the range is set in consideration of the error.

  Since the parallax error occurs along a line segment that passes from the center of the image 410 through the predicted imaging position p11 (that is, occurs in the radial direction from the center of the image 410), the predicted imaging position in the direction along the line segment. A range in which a parallax error can occur is set in the center direction of the image 410 and the direction opposite to the center of the image 410 with p11 as the center. In addition, since pitching occurs in the vertical direction of the image 410, a range of errors that can be generated by pitching in the vertical direction is set around the predicted imaging position p11. Then, a range that considers both the range of errors caused by parallax errors and pitching is set as the imaging expected region 411. At this time, the predicted imaging region 411 is surrounded by two line segments that extend from the center of the image 410 along the predicted imaging position p11 and two line segments that extend in the vertical direction of the image 410. It becomes an area.

  In this way, by considering the parallax error that is considered to be large and the error in the detection position in the vertical direction due to pitching that occurs when the vehicle travels on acceleration / deceleration and subtle unevenness in the in-vehicle environment, The expected imaging region 411 can be set appropriately.

  It should be noted that parameters relating to other camera characteristics used for the calculation in the predicted imaging position calculation unit 231, that is, the focal length f, the pixel pitch wi, the speed v, the imaging position (i1, j1) at the time t1, t2, and the time t1, The imaging predicted area 411 may be set in consideration of the imaging position (i2, j2) at time t2.

  The feature point tracking unit 240 corresponds to each of one or more feature points detected by the feature point detection unit 220 between a plurality of images captured at different times in the tracking range set by the tracking range determination unit 230. Search for feature points. That is, the feature point tracking unit 240 calculates time-series disparity information for feature points associated with a plurality of images captured at different times.

  The parallax offset correction amount calculation unit 250 calculates the parallax offset correction amount using the time-series parallax information of the feature points obtained by the feature point tracking unit 240 and the movement amount (movement distance) of the vehicle 1 (own vehicle). Processing is performed to calculate a parallax offset correction amount. In the present embodiment, the case where the parallax offset correction amount is calculated using the parallax information obtained at two different times has been described as an example, but for example, by using a lot of data in time series Stability can be increased. When using disparity information obtained at three or more different times, the equation for calculating the disparity offset is redundant and cannot be solved uniquely, so the disparity offset is calculated by optimization calculation. It is also conceivable to calculate the parallax offset using information on a plurality of feature points.

  The operational effects according to the present embodiment configured as described above will be described.

  In the prior art, the moving distance of the moving object is calculated by using the parallax of the feature points detected on the images captured by a plurality of cameras to increase the calibration opportunity of the parallax offset. If it is attempted to acquire parallax by tracking points, the processing load becomes very high, and it is considered that parallax cannot be acquired for a sufficient number of feature points in consideration of the calculation cost. In addition, since feature points on an image cannot always be detected accurately, it may be possible to calculate an incorrect parallax offset by mistracking similar points on the image as feature points. It is done.

  On the other hand, in the present embodiment, in order to correct the parallax offset between the left camera 111 and the right camera 112, and the stereo camera 110 having a plurality of the left camera 111 and the right camera 112 arranged in the vehicle 1. A parallax image generation unit 122 that generates a parallax image based on a plurality of images captured by the left camera 111 and the right camera 112, respectively, and the left camera 111 and the right camera based on the parallax image In the stereo camera device 100 including the image processing device 120 having the distance calculation unit 123 that calculates the relative position of the target object 112, the image processing device 120 has a plurality of images captured by the left camera 111 and the right camera 112, respectively. A feature point detection unit 220 that detects feature points of an image, a left camera 111, and a right camera 112 A feature point tracking unit 240 that tracks feature points between a plurality of images captured at different times in each case, and feature point tracking for a plurality of images in the feature point tracking unit 240 is a tracking range of a part of the plurality of images The parallax between the left camera 111 and the right camera 112 based on the tracking range determination unit 230 limited to, and the movement distance of the vehicle 1 between different times and the parallax about the feature point tracked by the feature point tracking unit 240 Since it is configured to include the parallax offset correction amount calculation unit 250 that calculates the parallax offset correction amount for correcting the offset, the calculation cost can be reduced while ensuring the calculation accuracy of the parallax offset.

  That is, in the present embodiment, the tracking range determination unit 230 is configured to limit the tracking of feature points for a plurality of images in the feature point tracking unit 240 to a tracking range (search range) of a part of the plurality of images. Therefore, by suppressing the possibility that the feature point of the tracking target is erroneously associated with other feature points, the calculation accuracy of the parallax offset can be ensured, and the tracking range of the feature points can be By limiting to a part, it is possible to reduce the amount of processing related to the feature point correspondence determination, and to reduce the calculation cost.

  In addition, since the tracking range determination unit 230 is configured to determine the tracking range based on the parallax offset that is considered to have a particularly large error and the pitching amount of the vehicle, the tracking range can be determined with higher accuracy. Thus, the calculation accuracy of the parallax offset can be ensured.

  In addition, in the feature point detection area determination unit 210, for each of the plurality of images captured by the left camera 111 and the right camera 112, the area where the feature point detection unit 220 detects the feature point is limited to a part of the image. Since the feature point detection region is determined, the processing time can be shortened and the accuracy can be improved by eliminating the region where the feature point should not be detected.

  In addition, the feature point detection region determination unit 210 excludes a region where parallax is not obtained from each of the plurality of images captured by the left camera 111 and the right camera 112 in the parallax uncalculated region exclusion unit 211. Since it is configured, it is possible to eliminate an area where there is no texture and where the parallax cannot be calculated, and it is possible to reduce processing time and improve accuracy.

  Further, the feature point detection region determination unit 210 is configured to exclude a region with low parallax accuracy for each of a plurality of images captured by the left camera 111 and the right camera 112 in the parallax unstable region exclusion unit 212. Therefore, it is possible to eliminate a specific pattern region with low parallax accuracy, and to shorten processing time and improve accuracy.

  Further, in the feature point detection region determination unit 210, the specific object region exclusion unit 213 excludes a predetermined specific object region for each of the plurality of images captured by the left camera 111 and the right camera 112. Since it is configured, it is possible to eliminate the area where the moving object is imaged, and it is possible to shorten the processing time and improve the accuracy.

<Second Embodiment>
A second embodiment of the present invention will be described with reference to FIGS. In the present embodiment, only differences from the first embodiment will be described. In the drawings used in the present embodiment, the same members as those in the first embodiment are denoted by the same reference numerals and described. Is omitted.

  In the present embodiment, the parallax offset correction amount calculation for determining whether to calculate the parallax offset correction amount (parallax offset correction amount calculation processing) based on whether the parallax offset correction amount can be accurately calculated. The case where the determination part 610 is provided is shown. In the present embodiment, in order to obtain the parallax offset correction amount with high accuracy, the movement amount and parallax of the vehicle, which are parameters used for calculating the parallax offset correction amount, are obtained with high accuracy, and the assumed stationary state is obtained. It is important to satisfy the two conditions of using only the parallax of a certain point. Therefore, in a scene that is considered not to satisfy these two conditions, the parallax offset correction amount calculation determination unit 610 simply determines that the parallax offset correction amount calculation processing is not performed, thereby reducing the parallax offset correction amount. Accuracy can be ensured.

  FIG. 15 is a functional block diagram showing processing functions of the parallax offset correction unit according to the present embodiment.

  In FIG. 15, the parallax offset correction unit 121A according to the present embodiment includes a parallax offset correction amount calculation determination unit 610 that determines whether to calculate a parallax offset based on a predetermined condition, and a parallax offset correction amount calculation determination unit. When it is determined that the parallax offset is calculated in 610, a plurality of images captured by the left camera 111 and the right camera 112 of the stereo camera 110 (here, the left camera 111 and the right camera 112 were captured at the same time). Feature points that track feature points between a plurality of images taken at different times in the left camera 111 and the right camera 112, respectively, for the feature points detection unit 220 that detects feature points corresponding to each other. The tracking unit 240 and the feature point tracking range in the feature point tracking unit 240 are limited to a part of the image. Disparity based on the disparity for the feature point tracked by the tracking range determination unit 230 (which determines the tracking range) and the feature point tracking unit 240, and the moving distance of the vehicle between different times when the feature point was tracked For each of a plurality of images captured by the parallax offset correction amount calculation unit 250 that calculates the offset and the left camera 111 and the right camera 112, a region where the feature point detection unit 220 detects the feature point is a part of the image. And a feature point detection region determination unit 210 for limiting (determining a feature point detection region).

  FIG. 16 is a functional block diagram illustrating processing functions of the parallax offset calculation determination unit.

  In FIG. 16, a parallax offset correction amount calculation determination unit 610 and a speed determination unit 611 that determine whether to calculate a parallax offset correction amount based on the speed of the vehicle 1 (own vehicle) and the vehicle 1 (own vehicle). An acceleration determination unit 612 that determines whether to calculate the parallax offset correction amount based on the acceleration of the vehicle, and whether to calculate the parallax offset correction amount based on whether the vehicle 1 (own vehicle) is traveling straight ahead. It includes a straight-ahead determination unit 613 for determining, and a scene determination unit 614 for determining whether to calculate the parallax offset correction amount based on whether or not the scene is a scene for which the parallax offset correction amount can be calculated. If at least one of the determination units 611 to 614 determines not to execute the parallax offset correction amount calculation process, the parallax offset correction amount calculation process is not executed as a determination result in the parallax offset correction amount calculation determination unit 610. Is output to the feature point detection region determination unit 210, and a series of parallax offset calculation processing that is connected to the processing of the feature point detection region determination unit 210 is not executed.

  The speed determination unit 611 determines whether to calculate the parallax offset correction amount according to the speed of the vehicle 1 (own vehicle). In general, it is known that a speed sensor attached to a vehicle has a large error in a low speed range. Therefore, in the low speed range, the vehicle 1 calculated using a detection value of a vehicle speed sensor included in the external device 160. It is considered that the error of the movement amount of (own vehicle) becomes large. Therefore, in the speed determination unit 611, for example, when the detection value by the vehicle speed sensor included in the external device 160 is 20 km / h or less, an error occurring in the movement amount of the vehicle 1 (own vehicle) exceeds the allowable range. It is determined that the parallax offset correction amount calculation process is not executed.

  The acceleration determination unit 612 determines whether to calculate a parallax offset correction amount according to the acceleration of the vehicle 1 (own vehicle). Since values obtained from various sensors included in the external device 160 are delayed in time from detection to reception by the image processing device 120, the time and speed of imaging with the stereo camera 110 are erroneously handled during acceleration / deceleration. There is a possibility of being attached. Therefore, in the acceleration determination unit 612, when the detection value by the acceleration sensor included in the external device 160 is equal to or greater than a predetermined threshold, that is, when acceleration / deceleration exceeding the allowable range is performed, the stereo camera 110 It is determined that the possibility that the imaging time and the speed are erroneously associated with each other exceeds the allowable range, and it is determined that the parallax offset correction amount calculation process is not executed. In such an acceleration determination unit 612, it is also possible to avoid execution of the parallax offset correction amount calculation process when the vehicle 1 is in a slip state. The acceleration (acceleration and deceleration) is determined by observing the amount of change (difference value) in the unit time of the detection value of the speed sensor or the output of the gyro sensor included in the external device 160. May be.

  The straight traveling determination unit 613 determines whether or not to calculate the parallax offset correction amount based on whether or not the vehicle 1 (own vehicle) is traveling straight. When the vehicle 1 travels on a curved road, an error in the estimated value of the movement amount of the vehicle 1 increases. Therefore, the straight traveling determination unit 613 determines that the parallax offset correction amount calculation process is not executed unless it can be determined that the vehicle 1 is traveling straight. Whether or not the vehicle 1 is traveling straight (whether it is traveling on a curve) is determined by vehicle sensors that detect GPS information from a navigation system or the like included in the external device 160, map data, steering angle, yaw rate, and the like. It can be determined based on the detection value from That is, for example, when the detected value of the steering angle sensor included in the external device 160 is outside a predetermined range, it is determined that the parallax offset correction amount calculation process is not executed.

  The scene determination unit 614 determines whether to calculate the parallax offset correction amount based on whether the scene is a scene for which the parallax offset correction amount may be calculated. For example, when the vehicle 1 travels on a congested road, an object moving at a low speed is imaged in most of the field of view of the stereo camera 110, so that it becomes difficult to distinguish between a stationary object and a moving object. By detecting it, the possibility of calculating an incorrect parallax offset correction value increases. In view of this, the scene determination unit 614 obtains, for example, information on whether or not the vehicle is traveling on a congested road from a navigation system or the like included in the external device 160. It is determined that the scene is not acceptable, and it is determined that the parallax offset correction amount calculation process is not performed. For the same reason, for example, even when driving on a busy road such as a shopping street, it is determined that the scene is not a scene where the parallax offset correction amount may be calculated, and the parallax offset correction amount calculation processing Is determined not to be implemented.

  FIG. 17 is a diagram illustrating a list of conditions when it is determined that the parallax offset correction amount calculation determination unit does not execute the parallax offset correction amount calculation processing.

  As can be seen from FIG. 17, the parallax offset correction amount calculation determination unit 610 executes parallax offset correction amount calculation processing based on input values from various sensors included in the external device 160 and information from the navigation system. Therefore, it is possible to complete the processing at a high speed, and to secure a calculation resource in a scene where the parallax offset correction amount calculation processing should not be executed.

  In addition, since each determination process in the speed determination unit 611, the acceleration determination unit 612, the straight travel determination unit 613, and the scene determination unit 614 in the parallax offset correction amount calculation determination unit 610 can be performed independently of each other, You may comprise so that some of the determination parts 611-614 may be used as needed. For example, in the vehicle 1 that uses an accurate vehicle speed sensor even in a low speed range, the speed determination unit 611 can be omitted, and if the amount of movement during curve traveling can be accurately modeled, it is determined that the vehicle travels straight. The part 613 can be omitted. That is, the parallax offset correction amount calculation determination unit 610 may be configured to perform only necessary determination according to the components at the time of implementation.

  Other configurations are the same as those of the first embodiment.

  In the present embodiment configured as described above, the same effects as those of the first embodiment can be obtained.

  Further, in the present embodiment, in a scene where the calculation error of the parallax offset correction amount is considered to be large, the parallax offset correction amount calculation process is not executed, which is suitable for calculating the parallax offset correction amount. It is no longer necessary to execute processing such as feature point detection in a state where there is no data, and calculation resources can be allocated to other processing. In addition, the possibility of calculating an incorrect parallax offset correction amount can be reduced.

<Third Embodiment>
A third embodiment of the present invention will be described with reference to FIG. In the present embodiment, only differences from the second embodiment will be described. In the drawings used in the present embodiment, members similar to those of the second embodiment are denoted by the same reference numerals and described. Is omitted.

  In the present embodiment, instead of the parallax offset correction amount calculation determination unit 610 provided in the parallax offset correction unit 121A of the second embodiment for determining when the parallax offset correction amount calculation process is not executed, the parallax offset This is provided with a parallax offset correction amount calculation determination unit 610 </ b> A that determines whether to execute the correction amount calculation processing. That is, in the present embodiment, when the parallax offset correction amount calculation determination unit 610A determines to execute the parallax offset correction amount calculation process, the parallax offset correction unit 121A executes the parallax offset calculation process. Therefore, in the present embodiment, there is no case where the parallax offset correction amount calculation process is executed and the parallax offset correction amount calculation process is not performed even though the parallax offset correction amount is to be updated. The parallax offset correction amount can be updated at a proper timing.

  FIG. 18 is a functional block diagram illustrating processing functions of the parallax offset correction amount calculation determination unit according to the present embodiment.

  In FIG. 18, the parallax offset correction amount calculation determination unit 610A in the present embodiment includes an environment change determination unit 615 that determines whether to calculate a parallax offset correction amount based on a change in the surrounding environment of the stereo camera 110, And an elapsed time determination unit 616 that determines whether to calculate a parallax offset correction amount based on the elapsed time from the previous execution of the parallax offset correction value calculation process. When it is determined that at least one of the determination units 615 and 616 executes the parallax offset correction amount calculation process, the determination that the parallax offset correction amount calculation process is executed as a determination result in the parallax offset correction amount calculation determination unit 610A The result is output to the feature point detection area determination unit 210, and a series of parallax offset calculation processes connected to the process of the feature point detection area determination unit 210 are executed.

  Whether the environment change determination unit 615 calculates the parallax offset correction amount based on whether the surrounding environment of the stereo camera 110 has changed significantly compared to the environment at the time of the previous parallax offset correction amount calculation process. Determine if. As the surrounding environment of the stereo camera 110, for example, there are humidity, temperature, and the like. When it is considered that there is a high possibility that the parallax offset is changed due to deformation of the housing due to changes in temperature or humidity. There is a need to urgently correct this. Therefore, in the environment change determination unit 615, when the change amount of the detection value from the temperature sensor or humidity sensor included in the external device 160 since the previous execution of the parallax offset correction amount calculation process is larger than a predetermined threshold value. Determines that the surrounding environment of the stereo camera 110 has changed significantly and determines to execute the parallax offset correction amount calculation processing.

  The elapsed time determination unit 616 determines whether to calculate the parallax offset correction amount based on the elapsed time from the previous execution of the parallax offset correction value calculation process. It is conceivable that the parallax offset of the stereo camera 110 changes due to some factor such as aging. Therefore, the elapsed time determination unit 616 determines that there is a high possibility that the parallax offset has changed if the elapsed time since the previous execution of the parallax offset correction value calculation process is equal to or greater than a predetermined threshold. The parallax offset star amount calculation process is determined to be executed. When dealing with aging degradation, for example, a period of several months may be set as a threshold value. In addition, when dealing with an impact caused by a collision with a brake or a person's hand, a period of several days to several weeks is set as a threshold value to prevent a case where the vehicle travels with a parallax offset shifted. Can do.

  Other configurations are the same as those of the first and second embodiments.

  In the present embodiment configured as described above, the same effects as those of the first embodiment can be obtained.

  Further, in the present embodiment, it is possible to determine the situation in which the parallax offset correction amount should be calculated without actually calculating the parallax offset, and to execute the process. It is possible to ensure the accuracy of parallax by operating at a necessary timing while securing the calculation resources by not performing.

<Appendix>
In addition, this invention is not limited to said embodiment, Various modifications and combinations within the range which does not deviate from the summary are included. Further, the present invention is not limited to the one having all the configurations described in the above embodiment, and includes a configuration in which a part of the configuration is deleted. Moreover, you may implement | achieve part or all of said each structure, function, etc., for example by designing with an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor.

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2 ... Object, 100 ... Stereo camera apparatus, 100a ... Bus line, 110 ... Stereo camera, 111 ... Left camera, 112 ... Right camera, 113 ... Parallax uncalculated area, 120 ... Image processing apparatus, 121, 121A ... Parallax offset correction unit 122 ... Parallax image generation unit 123 ... Distance calculation unit 130 ... Memory 140 ... Central processing unit (CPU) 150 ... Interface 160 ... External device 210 ... Feature point detection region determination unit , 211... Parallax uncalculated region exclusion unit, 212. Parallax unstable region exclusion unit, 213. Specific object region exclusion unit, 220... Feature point detection unit, 230... Tracking range determination unit, 231. ... predicted imaging region calculation unit, 240 ... feature point tracking unit, 250 ... parallax offset correction amount calculation unit, 300, 410, 420 ... image, 311 Parallax uncalculated area, 321 ... parallax unstable area, 331 ... specific object area, 332 ... specific object area, 333 ... specific object area, 411 ... imaging expected area, 610 ... parallax offset correction amount calculation determination unit, 610A ... parallax offset Correction amount calculation determination unit, 611... Speed determination unit, 612... Acceleration determination unit, 613... Straight determination unit, 614 .. scene determination unit, 615 .. environmental change determination unit, 616.

Claims (13)

An imaging device having a plurality of imaging units arranged in a vehicle;
A parallax image generation unit configured to generate a parallax image based on a plurality of images respectively captured by the plurality of imaging units using a parallax offset correction amount for correcting a parallax offset between the plurality of imaging units; and In a stereo camera device comprising: an image processing device having a relative position calculation unit that calculates a relative position of an object with respect to the plurality of imaging units based on the parallax images;
The image processing apparatus includes:
A feature point detection unit for detecting feature points of the plurality of images respectively captured by the plurality of imaging units;
A feature point tracking unit that tracks the feature points between the plurality of images captured at different times in each of the plurality of imaging units;
A tracking range determining unit that limits tracking of the feature points for the plurality of images in the feature point tracking unit to a tracking range of a part of the plurality of images;
A parallax offset correction amount for correcting a parallax offset between the plurality of imaging units based on a moving distance of the vehicle between the different times and a parallax about a feature point tracked by the feature point tracking unit. A stereo camera device comprising: a parallax offset correction amount calculation unit for calculating. The stereo camera device according to claim 1,
The tracking range determination unit determines the tracking range based on the parallax offset and the pitching amount of the vehicle. The stereo camera device according to claim 1,
For each of a plurality of images captured by the plurality of imaging units, a feature point detection region for determining a feature point detection region in which a region for detecting the feature point by the feature point detection unit is limited to a part of the image A stereo camera device further comprising a determination unit. The stereo camera device according to claim 3,
The feature point detection area determination unit includes a parallax non-calculated area exclusion unit that excludes an area where parallax is not obtained for each of a plurality of images captured by the plurality of imaging units. Camera device. The stereo camera device according to claim 3,
The feature point detection region determination unit includes a parallax unstable region exclusion unit that excludes a region with low parallax accuracy for each of a plurality of images captured by the plurality of imaging units. apparatus. The stereo camera device according to claim 3,
The feature point detection region determination unit includes a specific object region exclusion unit that excludes a predetermined specific object region for each of the plurality of images captured by the plurality of imaging units. Camera device. The stereo camera device according to claim 1,
The image processing apparatus further includes a parallax offset correction amount calculation determination unit that determines whether to calculate the parallax offset based on a predetermined condition,
The parallax offset correction amount calculation unit does not calculate the parallax offset when the parallax offset correction amount calculation determination unit determines not to calculate the parallax offset. The stereo camera device according to claim 7,
The parallax offset correction amount calculation determination unit determines that the parallax offset is not calculated when the speed of the vehicle is lower than a predetermined threshold. The stereo camera device according to claim 7,
The parallax offset correction amount calculation determination unit determines that the parallax offset is not calculated when the absolute value of the acceleration of the vehicle is equal to or greater than a predetermined threshold. The stereo camera device according to claim 7,
The parallax offset correction amount calculation determination unit determines that the parallax offset is not calculated when the vehicle is not traveling straight. The stereo camera device according to claim 7,
The parallax offset correction amount calculation determination unit determines that the parallax offset is not calculated when a large number of objects moving at low speed are captured in the plurality of images. The stereo camera device according to claim 7,
The parallax offset correction amount calculation determination unit calculates the parallax offset when the amount of change in humidity and temperature values since a previous parallax offset correction amount calculation is equal to or greater than a predetermined threshold. Camera device. The stereo camera device according to claim 7,
The parallax offset correction amount calculation determination unit calculates the parallax offset when an elapsed time from the previous parallax offset correction amount calculation is equal to or greater than a threshold.

Images