US9378553B2 - Stereo image processing device for vehicle - Google Patents
Stereo image processing device for vehicle Download PDFInfo
- Publication number
- US9378553B2 US9378553B2 US14/667,265 US201514667265A US9378553B2 US 9378553 B2 US9378553 B2 US 9378553B2 US 201514667265 A US201514667265 A US 201514667265A US 9378553 B2 US9378553 B2 US 9378553B2
- Authority
- US
- United States
- Prior art keywords
- side object
- parallax
- height
- vehicle
- real space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims description 47
- 238000006073 displacement reaction Methods 0.000 claims abstract description 48
- 238000001514 detection method Methods 0.000 claims abstract description 7
- 238000003745 diagnosis Methods 0.000 claims abstract description 7
- 238000000034 method Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 4
- 230000000052 comparative effect Effects 0.000 abstract description 28
- 238000012423 maintenance Methods 0.000 abstract description 5
- 238000007689 inspection Methods 0.000 abstract description 3
- 230000007704 transition Effects 0.000 abstract description 3
- 230000001131 transforming effect Effects 0.000 abstract 1
- 230000003287 optical effect Effects 0.000 description 16
- 238000012937 correction Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 230000009466 transformation Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
-
- G06T7/002—
-
- G06K9/00791—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T7/85—Stereo camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- H04N13/0203—
-
- H04N13/0239—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30256—Lane; Road marking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Definitions
- the present invention relates to a stereo image processing device for a vehicle that processes an image captured by a stereo camera unit installed on the vehicle and three-dimensionally recognizes a traveling environment.
- the technique of capturing a traveling environment in front of a vehicle with a camera and recognizing the traveling environment by image processing has recently found application in vehicles such as automobiles.
- various types of driver assist control methods such as collision avoidance with respect to forward obstacles, tracking control with respect to a vehicle in front, and warning control and steering control relating to wobbling and lane departure have been implemented by using a stereo image processing technique of capturing the same object with a stereo camera from different visual points and three-dimensionally recognizing the object from the parallax (distance data) obtained.
- Such a stereo camera unit is configured, for example, by mechanically fixing a pair of cameras with a predetermined base line (distance between optical axes) such that the optical axes thereof are substantially parallel, and where the mounting position thereof is shifted, the image capturing directions of the cameras are shifted, the vehicle in front cannot be recognized, and the accuracy of distance data is degraded.
- JP 2004-132870 A discloses the technique for improving the reliability of distance data by disposing an adjustment chart in front of a stereo camera unit, processing the captured image of the adjustment chart, and adjusting the displacement of the stereo camera unit.
- a stereo image processing device for a vehicle in accordance with the present invention processes images captured by a stereo camera unit installed on the vehicle and three-dimensionally recognizes a traveling environment, the device including: a left-right similar object detection module that detects a left side object and a right side object which is similar to the left side object present as a pair on the left and right side of the vehicle from an image captured by the stereo camera unit, performs matching processing of the left side object and the right side object, and calculates a parallax of the left side object and a parallax of the right side object on a same horizontal line of the image; and a camera displacement diagnosis module that calculates a diagnostic value based on a difference between the parallax of the left side object and the parallax of the right side object on the same horizontal line, compares the diagnostic value with a preset threshold value, and diagnoses a mechanical displacement in the stereo camera unit.
- a left-right similar object detection module that detects a left side object and a right side object which is similar to
- FIG. 1 is a configuration diagram of a stereo image processing device for a vehicle
- FIGS. 2A-2B are explanatory drawings illustrating an epipolar line
- FIG. 3 is an explanatory drawing illustrating stereo matching
- FIG. 4 is an explanatory drawing illustrating lane line detection
- FIG. 5 is an explanatory drawing illustrating the positions of lane lines in a reference image and a comparative image
- FIG. 6 is an explanatory drawing illustrating a parallax of left and right lane lines determined from the reference image and comparative image depicted in FIG. 5 ;
- FIG. 7 is an explanatory drawing illustrating lane lines recognized from the reference image and comparative image depicted in FIG. 5 ;
- FIG. 8 is an explanatory drawing illustrating the positions of lane lines in a reference image and a comparative image in the case in which a camera displacement has occurred;
- FIG. 9 is an explanatory drawing illustrating a parallax of left and right lane lines determined from the reference image and comparative image depicted in FIG. 8 ;
- FIG. 10 is an explanatory drawing illustrating lane lines recognized from the reference image and comparative image depicted in FIG. 8 ;
- FIG. 11 is an explanatory drawing illustrating the height of left and right lane lines determined from the reference image and comparative image depicted in FIG. 5 ;
- FIG. 12 is an explanatory drawing illustrating the height of left and right lane lines determined from the reference image and comparative image depicted in FIG. 8 .
- the reference numeral 1 stands for a stereo image processing device that is installed on a vehicle (hereinafter referred to as “subject vehicle”) such as an automobile, processes an image captured by a stereo camera unit 2 that captures an object from different visual points, measures the three-dimensional position of the object, and three-dimensionally recognizes the traveling environment around the subject vehicle.
- the stereo image processing device 1 includes, as functional components relating to processing of the image captured by the stereo camera unit 2 , an A/D conversion module 3 , an optical axis adjustment module 4 , a coarse adjustment module 5 , a distance correction module 6 , a stereo image processing module 7 , and a recognition processing module 9 as the main components, and also a distance data memory 8 a and an image data memory 8 b.
- the stereo image processing device 1 is also provided with a camera displacement diagnosis module 10 as a functional component for detecting the mechanical displacement of the stereo camera unit 2 that occurs during the actual use.
- the stereo camera unit 2 is configured of a set of mutually synchronized cameras 2 a and 2 b which have image capturing elements such as CCD or CMOS and a variable shutter speed.
- the three-dimensional position of an object on the outside of the subject vehicle is measured by performing stereo image processing of the image captured by the cameras 2 a and 2 b and relating to the outside of the subject vehicle.
- stereo matching corresponding point search
- This corresponding point search is typically implemented as one-dimensional search on an epipolar line.
- the visual points thereof are denoted by C 1 and C 2 , respectively, and the projection planes (image planes) thereof are denoted by IM 1 and IM 2 , respectively, when the attention is focused on a point P in a three-dimensional space, the point P of interest is observed in a point P 1 on the projection plane IM 1 of the camera 2 a and in a point P 2 on the projection plane IM 2 of the camera 2 b.
- a point corresponding to the point P 1 on one projection plane IM 1 is searched from the other projection plane IM 2 , but a point on the projection plane IM 2 corresponding to the point P 1 is on a straight line Lep (epipolar line) where the projection plane IM 2 crosses a plane constituted by the points P 1 , C 1 , and C 2 due to the geometric relationship of camera arrangement.
- the epipolar line Lep is likewise also present on the projection plane IM 1 , and the corresponding point search returns to one-dimensional search on the epipolar line Lep.
- the search processing can be simplified by matching the epipolar line Lep with a scanning line of the image. Therefore, in the present embodiment, as depicted in FIG. 2B , the projection planes IM 1 and IM 2 of the cameras 2 a and 2 b are arranged parallel to each other, and the two epipolar lines Lep are matched with a horizontal scanning line Lh of the image.
- the stereo camera unit 2 is mechanically and electrically adjusted so as to obtain a camera arrangement such that the optical axes of the camera 2 a disposed on the right side in the horizontal direction and the camera 2 b disposed on the left side are parallel to each other, the image capturing planes match, and the abscissa directions (horizontal scanning line directions) of the image capturing planes are matched (are not rotated with respect to each other).
- a stereo camera unit 2 is formed as a camera module in which the two cameras 2 a and 2 b are fixed to obtain a predetermined base length and is disposed, for example, close to a room mirror on the inner side of the front window at the upper part inside the vehicle cabin.
- the right camera 2 a captures a reference image which serves as a reference for the corresponding point search
- the left camera 2 b captures a comparative image for performing the corresponding point search.
- the right camera 2 a that captures the reference image (right image) is referred to hereinbelow, as appropriate, as the main camera 2 a
- the left camera 2 b that captures the comparative image (left image) is referred to as the sub-camera 2 b.
- the images captured by the cameras 2 a and 2 b are converted by an A/D conversion module 3 into respective digital images with a predetermined brightness gradation (for example, a 256-gradation gray scale).
- the digitalized image data are represented in an i-j coordinate system having the lower left corner of the image as a point of origin, the horizontal direction as an i-coordinate axis, and the vertical direction as a j-coordinate axis, reference image data are obtained from the main camera 2 a , and comparative image data are obtained from the sub-camera 2 b.
- the optical axis adjustment module 4 adjusts the position of the processing area to be used for recognition.
- the orientation of the optical axes (orientation of image center) with respect to the vehicle body unavoidably spreads in the pitch direction and yaw direction between the pair of cameras 2 a and 2 b due to a spread between vehicles. Therefore, the processing area which is the object of recognition processing of the areas captured by the cameras is moved, for example, by using a calibration parameter calculated by capturing the adjustment chart at the time of delivery from the plant, and the shift of optical axes of the cameras is corrected.
- the calibration parameter serves for calibrating the shift in the optical axis direction of the recognition area of the image which is caused by the positional displacement of the cameras 2 a and 2 b and the translational-rotational shift of a pair of image data
- the optical axis adjustment module 4 causes parallel movement of the recognition area of a preset size by a predetermined movement amount in the yaw direction and pitch direction in the image capture area of each camera.
- the coarse adjustment module 5 compensates the displacement between the cameras 2 a and 2 b that cannot be compensated by the mechanical adjustment at the time the camera unit is manufactured. For example, the translational and rotational shift of the image of the sub-camera 2 b with respect to the image of the main camera 2 a is examined and the parameters of affine transformation are stored in a memory at the time of delivery from the plant. The translational and rotational displacement of the image captured by the sub-camera 2 b with respect to the image captured by the camera 2 a is then corrected by geometrically moving the image of the sub-camera 2 b by the predetermined translation amount and rotation angle by using the affine transformation parameters.
- the distance correction module 6 corrects the reference image data and comparative image data to correct the distortion of image data caused by optical distortion of the camera. More specifically, the coordinates of pixel points constituting the image data of the reference image and comparative image are shifted on the image plane on the basis of the correction parameter that has been calculated in advance. Such correction basically results in the correction of image distortions caused by optical distortions of individual cameras 2 a and 2 b and the correction of errors in the distance data caused by such distortions.
- the correction parameter is determined, for example, by the difference between the coordinate on the image plane of a feature point detected from the adjustment chart captured at the time of delivery from the plant and a target coordinate corresponding to the feature point.
- the target coordinate as referred to herein, is a coordinate detected when the adjustment chart is captured by an ideal camera which has absolutely no optical distortions.
- the target coordinate can be calculated, for example, on the basis of the adjustment chart pattern, the distance between the adjustment chart and the camera, and the focal distance of the camera.
- the stereo image processing module 7 determines the shift amount (parallax) of the corresponding positions of the reference image and comparative image by stereo matching processing.
- a well-known area search method can be used as the stereo matching processing to estimate the correlation degree of the reference image and comparative image.
- a sum (SAD: Sum of Absolute Difference) of difference (absolute value) in pixel values between a small area (block) of the reference image and a small area (block) of the comparative image is calculated as an evaluation function of the correlation degree.
- a brightness value of each pixel is often used as the pixel value.
- the value of the evaluation function determined by the SAD is called the “city block distance”, and the value of the city block distance decreases with the increase in correlation between the blocks (degree of similarity).
- the parallax is given by the shift amount in the horizontal direction between the blocks at which the city block distance takes a minimum value.
- CB ⁇
- a search block constituted by 4 ⁇ 4 pixels is taken as an object and SAD calculations are performed by shifting the position of the search block (sub-block) of the comparative image by one pixel on the horizontal scanning line with respect to the search block (main block) of the reference image, a point in which the city block distance takes a minimum value C 0 is determined, as depicted in FIG. 3 , as a corresponding position (matching point) in which the degree of correlation of the main block and sub-block is the highest.
- the shift amount (the difference between the position im of the main block and the position is of the sub-block in the horizontal scanning direction) in one-pixel units of the main block and the sub-block in the matching point gives the parallax d (pixel parallax) having a resolution of one-pixel unit, and the aggregation of parallaxes d calculated for each block is saved as the distance data forming the distance image in the distance data memory 8 a.
- the distance information base on the pixel parallax is such that the resolution decreases as the distance to the object increases. Therefore, the processing of determining the parallax at a sub-pixel level of resolution which is equal to or less than one-pixel unit is performed as necessary.
- the recognition processing module 9 obtains the distance in the real space and also performs the recognition processing of each type by using the distance data saved in the distance data memory 8 a . For example, grouping processing is performed to group the distance data within a predetermined threshold, and lane line data relating to the road and side-wall data relating to guard rails and curbs present along the road are extracted, and steric objects are classified into bicycles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other steric object and extracted.
- the positions for the recognized data are calculated in three-dimensional coordinates of the real space. More specifically, where the points on the road surface directly below the centers of the main camera 2 a and the sub-camera 2 b are taken as points of origin in the real space, the X-axis is taken in the width direction, that is, left-right direction, of the subject vehicle, the Y-axis is taken in the vehicle height direction, and the Z axis is taken in the subject vehicle length direction, that is, distance direction, a point (i, j, d) on the distance image can be coordinate-converted into a point (x, y, z) in the real space by the following Equations (2) to (4) based on the principle of triangular measurements.
- CAHM height of the camera center (lens center) above the point of origin
- optical axis position (i-coordinate of the infinitely remote point on the front surface of the subject vehicle);
- wj pixel pitch (vertical direction).
- jv optical axis position (j-coordinate of the infinitely remote point on the front surface of the subject vehicle).
- the length thereof in the front-rear direction is estimated in advance, for example, as 3 m, 4.5 m, and 10 m, respectively, and the central position in the width direction where the vehicle is present is calculated using the central position of the detected width.
- the speed relative to the subject vehicle is calculated from changes in the distance in the direction of each axis from the subject vehicle, and the speed of each steric object in each axial direction is calculated by taking the subject vehicle speed into account in the calculated relative speed.
- the moving object such as pedestrians and light vehicles located around the subject vehicle and other vehicles traveling on a road connected to the road where the subject vehicle travels are recognized from the information thus obtained, that is, the lane line data relating to the road, side-wall data relating to guard rails and curbs present along the road, and steric object data (type, distance from the subject vehicle, coordinates of central position, speed, etc.).
- lane line data, side-wall data relating to guard rails and curbs present along the road, and steric object data from the recognition processing module 9 are inputted to the vehicle control device 20 .
- the vehicle control device 20 issues an instruction to perform driver assist control such as pre-crush brake control that prevents collision with an obstacle, cruise control provided with a tracking function, and warning control and steering control relating to wobbling and lane departure on the basis of information from the recognition processing module 9 .
- driver assist control such as pre-crush brake control that prevents collision with an obstacle, cruise control provided with a tracking function, and warning control and steering control relating to wobbling and lane departure on the basis of information from the recognition processing module 9 .
- the camera displacement diagnosis module 10 determines whether or not a rotational or translational displacement has occurred between the cameras 2 a and 2 b constituting the stereo camera unit 2 , and when it is determined that the displacement has occurred, safety is ensured by transition to fail-safe control or by stopping the related functions of the vehicle control system based on the traveling environment recognition result obtained by stereo image processing.
- the system may be restored when a state with normal diagnostic results is maintained for a predetermined period of time after it has been determined that the camera displacement has occurred under the effect of noise or the like in the diagnostic process and the related functions of the vehicle control system have been stopped.
- the displacement between the cameras 2 a and 2 b can be detected by using an object serving as a reference in the traveling environment.
- a fixed object associated with the road can be considered as the reference object, and among them, objects disposed to form a pair on the left and right side of the subject vehicle on the road where the subject vehicle travels, for example, left and right lane lines on the road surface, left and right guard rails, and left and right curbs (side walls) can be used.
- the portions at the same distance are picked up on the same horizontal line. Therefore, where objects forming a pair are present on the left and right side, it is possible to determine whether or not a displacement has occurred between the cameras 2 a and 2 b by comparing the parallaxes of the left and right objects on the same horizontal line.
- lane lines are the objects of a comparatively high brightness that are present on the road surface, and in most cases, they can be detected as a left-right pair, without the effect of occlusion or the like. Therefore, in the present embodiment, the displacement between the cameras 2 a and 2 b is diagnosed with reference to the fact that the left and right lane lines present at the same distance are picked up on the same horizontal line under the normal circumstances.
- the lane line used for camera displacement diagnostic in the present embodiment is a general name for lines dividing the vehicle traffic zone where the subject vehicle travels and is inclusive of single lines, multiple lines such as double lines, solid lines, and broken lines yellow lines and white lines.
- the lane lines detected on both sides (left and right) of the subject vehicle are taken as a reference for diagnostic.
- the left and right lane line detection function of the stereo image processing module 7 and the recognition processing module 9 that is, the function thereof as the left-right similar object detection module of that detects a left side object and a right side object, is explained hereinbelow.
- the lane line of a road can be detected by estimating changes in brightness in the width direction to the road in the image plane, extracting a point group serving as a candidate for a lane line, and processing the time sequence data of the point group which is the lane line candidate.
- the search is performed from the inside outward in the vehicle width direction on the image plane, and a point (edge point) in which the brightness of pixels on the outer side in the vehicle width direction is high relative to the brightness of pixels on the inner side and the differential value of the brightness representing the variation amount thereof is equal to or higher than a set threshold on the positive side is detected as a lane line start point Ps. Further, a point (edge point) in which the brightness of pixels on the outer side in the vehicle width direction is low relative to the brightness of pixels on the inner side and the differential value of the brightness representing the variation amount thereof is equal to or less than a set threshold on the negative side is detected as a lane line end point Pe.
- a predetermined intermediate area between the lane line start point Ps and the lane line end point Pe is extracted as a lane line candidate points, and the parallax of the lane line candidate points determined by stereo matching is converted into coordinates in the real space.
- a lane line model approximating the left and right lane lines is calculated by processing the time sequence data relating to the space coordinate positions of the lane line candidate point based on the vehicle movement amount per unit time.
- an approximation model representing the height on the YZ plane is added to the approximation model representing the shape on the XZ plane in the XYZ coordinate space as a lane line model, and the lane lines can thus be three-dimensionally determined.
- the lane line shape of the XZ plane can be represented, for example, by using an approximation model in which linear components determined by the Hough transformation are connected to each other, or an approximation model in which x-direction positions xl and xr of the left and right lane lines are approximated by a second-order equation with respect to the distance z, as indicated in Equations (5) and (6) hereinbelow.
- xl Al ⁇ z 2 +Bl ⁇ z+Cl
- xr Ar ⁇ z 2 +Br ⁇ z+Cr (6)
- the coefficients Al, Bl, Cl, Ar, Br, and Cr in Equations (5) and (6) are identified using a least squares method with respect to the real space coordinates (x, z) of the left and right lane line candidate points.
- the coefficients Al and Ar represent curvature components of the left and right lane lines, respectively
- the coefficients Bl and Br represent inclination components (yaw angle components) of the left and right lane lines, respectively, in the width direction of the subject vehicle
- the coefficient Cl and Cr represent position components of the left and right lane lines, respectively, in the width direction of the subject vehicle.
- the heights yl, yr of the left and right lane lines on the YZ plane can be determined by linear approximation with first-order equations with respect to the distance z as indicated by Equations (7) and (8) below.
- the coefficients al, ar, bl, and br are identified using a least squares method with respect to the real space coordinates (x, z) of the left and right lane line candidate points.
- yl al ⁇ z+bl (7)
- yr ar ⁇ z+br (8)
- the camera displacement diagnosis module 10 determines the presence of a camera displacement by calculating a diagnostic value based on the difference in the parallax of the left lane line and the parallax of the right lane line between the reference image and comparative image. Where it is determined that a camera displacement has occurred, the lane line model calculated in the recognition processing module 9 , or position information on other vehicles or the like is erroneous information based on erroneous recognition. Therefore, an instruction is issued to stop the control function in the vehicle control device 20 or make a transition to failsafe control. Further, a warning is issued to the user to take the car to the maintenance site of the dealership for inspection.
- Whether or not a displacement has occurred is determined by using the diagnostic value based on the difference in parallax between the left and right lane lines on the same horizontal line. More specifically, the diagnostic values such as described in sections (a) to (c) hereinbelow are calculated, and whether or not a displacement has occurred is determined by using those diagnostic values individually or in appropriate combinations. For the sake of simplicity, in the example explained hereinbelow, the case is considered in which the road is flat, the subject vehicle travels on the straight road of a constant width, the vehicle position (camera position) is in the center between the left and right lane lines, and the subject vehicle is parallel to the left and right lane lines.
- the left and right lane lines WLl and WLr are picked up with a shift in the horizontal direction equal to the parallax of the reference image of the camera 2 a and the comparative image of the camera 2 b .
- a ground point A on the right lane line WLr with the coordinate j1 of the reference image and a ground point B on the left lane line WLl with the coordinate j2 (j2>j1) in FIG. 5 are taken as points of interest
- the ground point A close to the camera 2 a is picked up at a position below the image
- the ground point B which is far from the camera 2 a is picked up above (close to the optical axis of) the image.
- the ground point A is picked up at a position Al on the right lane line WLr which is shifted by the parallax da in the horizontal direction (i-axis direction) at the coordinate j1
- the ground point B is picked up at a position Bl on the left lane line WLl which is shifted by the parallax db in the horizontal direction (i-axis direction) at the coordinate j2.
- FIG. 6 depicts such a relationship represented as a relationship between the vertical coordinate (j-coordinate) of the image and the parallax d.
- the parallax da of the ground point A satisfies the relationship da>db with the parallax db of the ground point B, and the parallax of the left lane line WLl and the parallax of the right lane line WLr have the same value in the same ground point (on the horizontal line with the same j-coordinate value).
- the j-coordinate of the left lane line WLl and the j-coordinate of the right lane line WLr giving the same parallax value have the same value.
- the ground point A on the right lane line WLr in the reference image of the camera 2 a is captured at a position Al′ that has moved downward in the image with respect to the original corresponding position Al in the comparative image of the camera 2 b .
- the ground point B is captured at a position B 2 that has moved downward in the image with respect to the original corresponding position Bl.
- the position Al which a true corresponding point of the comparative image with respect to the ground point A on the coordinate j1 of the reference image, moves to the coordinate j1a (j1a ⁇ j1).
- the position Bl which a true corresponding point of the comparative image with respect to the ground point B on the coordinate j2 of the reference image, moves to the coordinate j2b (j2b ⁇ j2). Therefore, the true epipolar line is shifted and inclined with respect to the original horizontal line.
- a parallax da′ is calculated by assuming that the point of the comparative image corresponding to the ground point A at the j1 coordinate of the reference image is at the position A 2 on the same j1 coordinate
- a parallax db′ is calculated by assuming that the point of the comparative image corresponding to the ground point B at the j2 coordinate of the reference image is at the position B 2 on the same j2 coordinate.
- the parallax da′ of the ground point A of the right lane line is calculated to be less than the original parallax da
- the parallax db′ of the ground point B of the left lane line is calculated to be larger than the original parallax db
- the relationship between the image vertical coordinate and parallax has different characteristics which do not match on the left and right lane line s, as depicted in FIG. 9 .
- the parallax of the right lane line WLr is less (farther) than the parallax of the left lane line WLl on the horizontal line with the same j-coordinate value, and the j-coordinate value of the right lane line WLr is less than the j-coordinate value of the left lane line WLl at the same parallax value.
- the lane line approximating equations are generated by determining the distance z in the real space and the coordinate value x in the vehicle width direction from such a relationship, the lane line shape such as shown in FIG. 10 is obtained, and the left and right lane lines, which are inherently of a straight shape, are erroneously recognized as having a shape curved to the right in the advance direction.
- a difference ⁇ jd between the j-coordinate value of the left lane line and the j-coordinate value of the right lane line giving the same parallax is calculated as a diagnostic value based on the difference in parallax between the left and right lane lines on the same horizontal line.
- a threshold thj which is a preset allowed value of camera displacement
- the diagnostic value ⁇ jd is calculated, for example, by taking an average value for a predetermined number of frames correspondingly to a parallax corresponding to any distance, or a parallax in a plurality of ground points such as at a small distance, a medium distance, and a large distance. Further, depending on conditions, a difference in parallax between the left and right lane lines on the same horizontal line can be directly used as the diagnostic value.
- the height of the ground points A and B is the same, the inclinations al and ar of the straight lines representing the relationship between the height and the distance become 0 (flat road), the height y is constant with respect to the distance z, and the same height is obtained for the left and right lane lines WLl and WLr.
- the parallax of the right lane line WLr is less than the parallax of the left lane line WLl on the horizontal line with the same j-coordinate value
- the j-coordinate value of the right lane line WLr is less than the j-coordinate value of the left lane line WLl at the same parallax value. Therefore, in the relationship between the height y and distance z in the real space of the left and right lane lines WLl and WLr, as depicted in FIG.
- the relationship between the height and distance of the left lane line WLl and the right lane line WLr is represented by straight lines having respective predetermined inclinations, and the left lane line WLl is calculated to be higher than the right lane line WLr at the same distance.
- a difference ⁇ yd between the height of the left lane line and the height of the right lane line at the same distance is calculated as a diagnostic value based on the difference in parallax between the left and right lane lines in the same horizontal line, and whether or not a camera displacement has occurred is determined.
- a threshold thy which is a preset allowed value of camera displacement
- the diagnostic value ⁇ yd is also calculated, for example, by taking an average value for a predetermined number of frames correspondingly to a parallax corresponding to any distance, or a parallax in a plurality of ground points such as at a small distance, a medium distance, and a large distance.
- the relationship depicted in FIG. 12 means that when a camera displacement has occurred, the relationship between the height and distance of the left and right lane lines is represented by straight lines with inclinations al and ar, the road being flat.
- the upper limit values of the inclinations al and ar are determined by a front-rear gradient or a transverse gradient (cant) of the actual road, and the positive-negative polarities of the inclinations al and ar should be the same due to the actual road shape. Therefore, the occurrence of camera displacement can be determined by taking the inclinations al and ar as diagnostic values and comparing the upper limit values or polarities.
- the left and right lane lines of the road are used as references, and a diagnostic value based on a difference between the parallax of the left lane line and the parallax of the right lane line on the same horizontal line obtained by matching processing of the lane lines is calculated. Whether or not a displacement has occurred between the cameras 2 a and 2 b is determined by comparing the diagnostic value with a preset threshold.
- vehicle control can be reliably prevented from failures resulting from using the recognition results of position information on lane lines or other vehicles in spite of the occurrence of a displacement in the stereo camera unit, and safety can be ensured.
- the displacement of the stereo camera unit can be diagnosed when the subject vehicle is running, and the occurrence of vehicle control failures resulting from erroneous recognition caused by the displacement can be avoided.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Traffic Control Systems (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Measurement Of Optical Distance (AREA)
Abstract
Description
CB=Σ|M(i,j)−S(i,j)| (1)
x=CD/S+z·wi·(i−iv)/f (2)
y=z·wj·(j−jv)/f+CAHM (3)
z=CD·f/(d·wi) (4)
where CD: camera spacing (base length);
xl=Al·z 2 +Bl·z+Cl (5)
xr=Ar·z 2 +Br·z+Cr (6)
yl=al·z+bl (7)
yr=ar·z+br (8)
Claims (8)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014069712A JP5906272B2 (en) | 2014-03-28 | 2014-03-28 | Stereo image processing apparatus for vehicle |
JP2014-069712 | 2014-03-28 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20150279017A1 US20150279017A1 (en) | 2015-10-01 |
US9378553B2 true US9378553B2 (en) | 2016-06-28 |
Family
ID=54067025
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/667,265 Active US9378553B2 (en) | 2014-03-28 | 2015-03-24 | Stereo image processing device for vehicle |
Country Status (4)
Country | Link |
---|---|
US (1) | US9378553B2 (en) |
JP (1) | JP5906272B2 (en) |
CN (1) | CN104949619B (en) |
DE (1) | DE102015104453B4 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160304098A1 (en) * | 2015-04-17 | 2016-10-20 | Toyota Jidosha Kabushiki Kaisha | Road surface gradient detection device |
US10154255B2 (en) * | 2014-01-10 | 2018-12-11 | Hitachi Automotive Systems, Ltd. | In-vehicle-camera image processing device |
US10373338B2 (en) | 2015-05-27 | 2019-08-06 | Kyocera Corporation | Calculation device, camera device, vehicle, and calibration method |
EP3543951A1 (en) * | 2018-03-20 | 2019-09-25 | Kabushiki Kaisha Toshiba | Image processing device, driving support system, and image processing method |
EP3617937A1 (en) * | 2018-09-03 | 2020-03-04 | Toshiba Electronic Devices & Storage Corporation | Image processing device, driving assistance system, image processing method, and program |
Families Citing this family (42)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6609970B2 (en) * | 2015-04-02 | 2019-11-27 | アイシン精機株式会社 | Perimeter monitoring device |
JP6623044B2 (en) * | 2015-11-25 | 2019-12-18 | 日立オートモティブシステムズ株式会社 | Stereo camera device |
JP6511406B2 (en) * | 2016-02-10 | 2019-05-15 | クラリオン株式会社 | Calibration system, calibration device |
WO2017145818A1 (en) * | 2016-02-24 | 2017-08-31 | ソニー株式会社 | Signal processing device, signal processing method, and program |
CN107292214B (en) * | 2016-03-31 | 2020-06-19 | 比亚迪股份有限公司 | Lane departure detection method and device and vehicle |
CN105956527B (en) * | 2016-04-22 | 2019-10-22 | 百度在线网络技术(北京)有限公司 | Detection of obstacles outcome evaluation method and apparatus for automatic driving car |
JP6712775B2 (en) * | 2016-08-12 | 2020-06-24 | パナソニックIpマネジメント株式会社 | Road surface estimation device, vehicle control device, road surface estimation method, and program |
DE102016215538A1 (en) * | 2016-08-18 | 2018-03-08 | Robert Bosch Gmbh | Method for transforming sensor data |
WO2018042481A1 (en) * | 2016-08-29 | 2018-03-08 | 株式会社日立製作所 | Imaging apparatus and imaging method |
US20180067494A1 (en) * | 2016-09-02 | 2018-03-08 | Delphi Technologies, Inc. | Automated-vehicle 3d road-model and lane-marking definition system |
JP6742423B2 (en) * | 2016-09-23 | 2020-08-19 | 日立オートモティブシステムズ株式会社 | Image processing device |
JP6654999B2 (en) * | 2016-12-12 | 2020-02-26 | 株式会社Soken | Target detection device |
CN106949832B (en) * | 2017-03-20 | 2020-08-25 | 广汽本田汽车有限公司 | Vehicle body position detection method and device |
CN110462682B (en) | 2017-03-30 | 2023-05-02 | 三菱电机株式会社 | Object detection device and vehicle |
EP3640108A4 (en) * | 2017-06-15 | 2021-03-17 | Hitachi Automotive Systems, Ltd. | Vehicle control device |
CN107705244B (en) * | 2017-09-11 | 2021-01-15 | 中国国土资源航空物探遥感中心 | Edge connection correction method suitable for large-area multi-remote sensing image |
JP6981850B2 (en) * | 2017-11-09 | 2021-12-17 | 株式会社Soken | Driving support system |
FR3077547A1 (en) * | 2018-02-08 | 2019-08-09 | Renault S.A.S | SYSTEM AND METHOD FOR DETECTING A RISK OF COLLISION BETWEEN A MOTOR VEHICLE AND A SECONDARY OBJECT LOCATED ON CIRCULATION PATHS ADJACENT TO THE VEHICLE DURING CHANGE OF TRACK |
GB201804082D0 (en) * | 2018-03-14 | 2018-04-25 | Five Ai Ltd | Image annotation |
US11367175B2 (en) * | 2018-03-22 | 2022-06-21 | Hitachi Astemo, Ltd. | In-vehicle stereo camera |
US10843702B2 (en) * | 2018-06-06 | 2020-11-24 | Ford Global Technologies, Llc | Methods and systems for oil leak determination |
CN110858405A (en) * | 2018-08-24 | 2020-03-03 | 北京市商汤科技开发有限公司 | Attitude estimation method, device and system of vehicle-mounted camera and electronic equipment |
JP7232005B2 (en) * | 2018-09-20 | 2023-03-02 | 株式会社Subaru | VEHICLE DRIVING ENVIRONMENT DETECTION DEVICE AND DRIVING CONTROL SYSTEM |
CN109389650B (en) * | 2018-09-30 | 2021-01-12 | 京东方科技集团股份有限公司 | Vehicle-mounted camera calibration method and device, vehicle and storage medium |
CN109670443B (en) * | 2018-12-18 | 2022-10-04 | 福州大学 | Improved Hough transformation road line detection method based on superposition constraint |
CN109858363B (en) * | 2018-12-28 | 2020-07-17 | 北京旷视科技有限公司 | Dog nose print feature point detection method, device, system and storage medium |
JP7126963B2 (en) | 2019-02-13 | 2022-08-29 | 株式会社Subaru | stereo camera adjustment system |
US10872532B2 (en) * | 2019-02-27 | 2020-12-22 | ANI Technologies Private Limited | Calibration of fixed image-capturing device for predicting driving lane |
DE102019209849B4 (en) * | 2019-07-04 | 2021-08-05 | Robert Bosch Gmbh | Method and control device for distance-measuring image processing for a stereo camera device for a vehicle and stereo camera system with a stereo camera device and a control device |
JP7291594B2 (en) * | 2019-10-02 | 2023-06-15 | 株式会社Subaru | Image processing device |
CN111288967B (en) * | 2020-01-19 | 2020-10-27 | 广州翰南工程技术有限公司 | Remote high-precision displacement detection method based on machine vision |
DE112020005059T5 (en) * | 2020-03-06 | 2022-07-21 | Hitachi Astemo, Ltd. | PROCESSING DEVICE |
US12027056B2 (en) | 2020-08-21 | 2024-07-02 | Honeywell Aerospace Sas | Systems and methods using image processing to determine at least one kinematic state of a vehicle |
US11893896B2 (en) * | 2020-08-21 | 2024-02-06 | Honeywell Aerospace Sas | Systems and methods for determining an angle and a shortest distance between longitudinal axes of a travel way line and a vehicle |
CN111982157A (en) * | 2020-09-14 | 2020-11-24 | 深圳中科超远科技有限公司 | Road mapping and marking vehicle coordinate correcting device |
CN112270690B (en) * | 2020-10-12 | 2022-04-26 | 淮阴工学院 | Self-adaptive night lane line detection method based on improved CLAHE and sliding window search |
US11681297B2 (en) * | 2021-03-02 | 2023-06-20 | Southwest Research Institute | Autonomous vehicle and self-location estimating method in autonomous vehicle |
CN113221701B (en) * | 2021-04-30 | 2022-06-10 | 东风商用车有限公司 | Lane line and track line identification method and device based on direction prediction |
CN114140538B (en) * | 2021-12-03 | 2022-09-27 | 禾多科技(北京)有限公司 | Vehicle-mounted camera pose adjusting method, device, equipment and computer readable medium |
CN114399588B (en) * | 2021-12-20 | 2022-11-11 | 禾多科技(北京)有限公司 | Three-dimensional lane line generation method and device, electronic device and computer readable medium |
CN114353817B (en) * | 2021-12-28 | 2023-08-15 | 重庆长安汽车股份有限公司 | Multi-source sensor lane line determination method, system, vehicle and computer readable storage medium |
CN114120258B (en) * | 2022-01-26 | 2022-05-03 | 深圳佑驾创新科技有限公司 | Lane line identification method and device and storage medium |
Citations (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5680474A (en) * | 1992-10-27 | 1997-10-21 | Canon Kabushiki Kaisha | Corresponding point extraction method for a plurality of images |
JP2003083742A (en) | 2001-09-13 | 2003-03-19 | Fuji Heavy Ind Ltd | Distance correction apparatus and method of monitoring system |
JP2004132870A (en) | 2002-10-11 | 2004-04-30 | Keiji Saneyoshi | Regulator for stereoscopic camera, and method of regulating stereoscopic camera |
US20040096082A1 (en) * | 2002-08-28 | 2004-05-20 | Hiroaki Nakai | Obstacle detection device and method therefor |
US6813370B1 (en) * | 1999-09-22 | 2004-11-02 | Fuji Jukogyo Kabushiki Kaisha | Lane marker recognizing apparatus |
US20040234124A1 (en) * | 2003-03-13 | 2004-11-25 | Kabushiki Kaisha Toshiba | Stereo calibration apparatus and stereo image monitoring apparatus using the same |
US6985619B1 (en) * | 1999-09-22 | 2006-01-10 | Fuji Jukogyo Kabushiki Kaisha | Distance correcting apparatus of surroundings monitoring system and vanishing point correcting apparatus thereof |
JP2007037011A (en) | 2005-07-29 | 2007-02-08 | Toyota Motor Corp | Image processing apparatus |
US20100098293A1 (en) * | 2008-10-17 | 2010-04-22 | Manmohan Chandraker | Structure and Motion with Stereo Using Lines |
US8050458B2 (en) * | 2007-06-18 | 2011-11-01 | Honda Elesys Co., Ltd. | Frontal view imaging and control device installed on movable object |
US8174563B2 (en) * | 2007-10-29 | 2012-05-08 | Fuji Jukogyo Kabushiki Kaisha | Object detecting system |
US20120257018A1 (en) * | 2009-12-04 | 2012-10-11 | Nlt Technologies, Ltd. | Stereoscopic display device, method for generating image data for stereoscopic display, and program therefor |
US8340896B2 (en) * | 2009-05-22 | 2012-12-25 | Fuji Jukogyo Kabushiki Kaisha | Road shape recognition device |
US8526716B2 (en) * | 2011-03-11 | 2013-09-03 | Snell Limited | Analysis of stereoscopic images |
US20140133741A1 (en) * | 2011-06-29 | 2014-05-15 | Jing Wang | Device for generating three dimensional feature data, method for generating three-dimensional feature data, and recording medium on which program for generating three-dimensional feature data is recorded |
US8878944B2 (en) * | 2012-11-02 | 2014-11-04 | Sony Corporation | Image processing apparatus and method, image processing system and program |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3728460B2 (en) | 2003-05-15 | 2005-12-21 | 独立行政法人 宇宙航空研究開発機構 | Method and system for determining optimal arrangement of stereo camera |
JP4069855B2 (en) * | 2003-11-27 | 2008-04-02 | ソニー株式会社 | Image processing apparatus and method |
DE102008040985B4 (en) * | 2008-08-05 | 2021-05-27 | Robert Bosch Gmbh | Method for calibrating a multi-camera system |
-
2014
- 2014-03-28 JP JP2014069712A patent/JP5906272B2/en active Active
-
2015
- 2015-03-24 US US14/667,265 patent/US9378553B2/en active Active
- 2015-03-25 DE DE102015104453.6A patent/DE102015104453B4/en active Active
- 2015-03-25 CN CN201510133563.2A patent/CN104949619B/en active Active
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5680474A (en) * | 1992-10-27 | 1997-10-21 | Canon Kabushiki Kaisha | Corresponding point extraction method for a plurality of images |
US6813370B1 (en) * | 1999-09-22 | 2004-11-02 | Fuji Jukogyo Kabushiki Kaisha | Lane marker recognizing apparatus |
US6985619B1 (en) * | 1999-09-22 | 2006-01-10 | Fuji Jukogyo Kabushiki Kaisha | Distance correcting apparatus of surroundings monitoring system and vanishing point correcting apparatus thereof |
JP2003083742A (en) | 2001-09-13 | 2003-03-19 | Fuji Heavy Ind Ltd | Distance correction apparatus and method of monitoring system |
US20040096082A1 (en) * | 2002-08-28 | 2004-05-20 | Hiroaki Nakai | Obstacle detection device and method therefor |
US6906620B2 (en) * | 2002-08-28 | 2005-06-14 | Kabushiki Kaisha Toshiba | Obstacle detection device and method therefor |
JP2004132870A (en) | 2002-10-11 | 2004-04-30 | Keiji Saneyoshi | Regulator for stereoscopic camera, and method of regulating stereoscopic camera |
US20040234124A1 (en) * | 2003-03-13 | 2004-11-25 | Kabushiki Kaisha Toshiba | Stereo calibration apparatus and stereo image monitoring apparatus using the same |
US7313265B2 (en) * | 2003-03-13 | 2007-12-25 | Kabushiki Kaisha Toshiba | Stereo calibration apparatus and stereo image monitoring apparatus using the same |
JP2007037011A (en) | 2005-07-29 | 2007-02-08 | Toyota Motor Corp | Image processing apparatus |
US8050458B2 (en) * | 2007-06-18 | 2011-11-01 | Honda Elesys Co., Ltd. | Frontal view imaging and control device installed on movable object |
US8174563B2 (en) * | 2007-10-29 | 2012-05-08 | Fuji Jukogyo Kabushiki Kaisha | Object detecting system |
US20100098293A1 (en) * | 2008-10-17 | 2010-04-22 | Manmohan Chandraker | Structure and Motion with Stereo Using Lines |
US8401241B2 (en) * | 2008-10-17 | 2013-03-19 | Honda Motor Co., Ltd. | Structure and motion with stereo using lines |
US8340896B2 (en) * | 2009-05-22 | 2012-12-25 | Fuji Jukogyo Kabushiki Kaisha | Road shape recognition device |
US20120257018A1 (en) * | 2009-12-04 | 2012-10-11 | Nlt Technologies, Ltd. | Stereoscopic display device, method for generating image data for stereoscopic display, and program therefor |
US8526716B2 (en) * | 2011-03-11 | 2013-09-03 | Snell Limited | Analysis of stereoscopic images |
US20140133741A1 (en) * | 2011-06-29 | 2014-05-15 | Jing Wang | Device for generating three dimensional feature data, method for generating three-dimensional feature data, and recording medium on which program for generating three-dimensional feature data is recorded |
US9245345B2 (en) * | 2011-06-29 | 2016-01-26 | Nec Solution Innovators, Ltd. | Device for generating three dimensional feature data, method for generating three-dimensional feature data, and recording medium on which program for generating three-dimensional feature data is recorded |
US8878944B2 (en) * | 2012-11-02 | 2014-11-04 | Sony Corporation | Image processing apparatus and method, image processing system and program |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10154255B2 (en) * | 2014-01-10 | 2018-12-11 | Hitachi Automotive Systems, Ltd. | In-vehicle-camera image processing device |
US20160304098A1 (en) * | 2015-04-17 | 2016-10-20 | Toyota Jidosha Kabushiki Kaisha | Road surface gradient detection device |
US9771080B2 (en) * | 2015-04-17 | 2017-09-26 | Toyota Jidosha Kabushiki Kaisha | Road surface gradient detection device |
US10373338B2 (en) | 2015-05-27 | 2019-08-06 | Kyocera Corporation | Calculation device, camera device, vehicle, and calibration method |
EP3543951A1 (en) * | 2018-03-20 | 2019-09-25 | Kabushiki Kaisha Toshiba | Image processing device, driving support system, and image processing method |
US10771688B2 (en) | 2018-03-20 | 2020-09-08 | Kabushiki Kaisha Toshiba | Image processing device, driving support system, and image processing method |
EP3617937A1 (en) * | 2018-09-03 | 2020-03-04 | Toshiba Electronic Devices & Storage Corporation | Image processing device, driving assistance system, image processing method, and program |
US11880993B2 (en) | 2018-09-03 | 2024-01-23 | Kabushiki Kaisha Toshiba | Image processing device, driving assistance system, image processing method, and program |
Also Published As
Publication number | Publication date |
---|---|
DE102015104453B4 (en) | 2024-05-23 |
DE102015104453A1 (en) | 2015-10-01 |
CN104949619B (en) | 2018-08-24 |
JP2015190921A (en) | 2015-11-02 |
CN104949619A (en) | 2015-09-30 |
US20150279017A1 (en) | 2015-10-01 |
JP5906272B2 (en) | 2016-04-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9378553B2 (en) | Stereo image processing device for vehicle | |
US11093763B2 (en) | Onboard environment recognition device | |
US10558867B2 (en) | Image processing apparatus, stereo camera apparatus, vehicle, and image processing method | |
US9311711B2 (en) | Image processing apparatus and image processing method | |
JP6614247B2 (en) | Image processing apparatus, object recognition apparatus, device control system, image processing method and program | |
KR102104005B1 (en) | Methods for detecting objects on the roadside of automobiles, computing devices, driver assistance systems and automobiles | |
KR101449160B1 (en) | Apparatus and method for providing information of blind spot | |
JP2008039491A (en) | Stereo image processing apparatus | |
JP6602982B2 (en) | In-vehicle camera, in-vehicle camera adjustment method, in-vehicle camera system | |
JP6337504B2 (en) | Image processing apparatus, moving body, robot, device control method and program | |
EP3410705B1 (en) | 3d vision system for a motor vehicle and method of controlling a 3d vision system | |
JP6543935B2 (en) | PARALLEL VALUE DERIVING DEVICE, DEVICE CONTROL SYSTEM, MOBILE OBJECT, ROBOT, PARALLEL VALUE DERIVING METHOD, AND PROGRAM | |
JP2004028727A (en) | Monitoring system, monitoring method, distance correction device for the monitoring system, and distance correction method | |
US9098774B2 (en) | Method for detection of targets in stereoscopic images | |
CN108292441B (en) | Vision system for a motor vehicle and method for controlling a vision system | |
CN113838111A (en) | Road texture feature detection method and device and automatic driving system | |
JP2018088217A (en) | Information processing device, imaging device, apparatus control system, information processing method, and program | |
JP2018136739A (en) | Calibration device | |
CN109993133A (en) | Automobile-used obstacle detection method based on longitudinal binocular vision | |
JP2019160251A (en) | Image processing device, object recognition device, instrument control system, moving body, image processing method and program | |
EP2919191B1 (en) | Disparity value deriving device, equipment control system, movable apparatus, robot, and disparity value producing method | |
JP7525936B2 (en) | Method and system for providing a depth map with confidence estimates - Patents.com | |
KR101949349B1 (en) | Apparatus and method for around view monitoring | |
JP6459482B2 (en) | Parallax value deriving apparatus, device control system, moving body, robot, parallax value deriving method, and program | |
JP6122365B2 (en) | Stereo camera adjustment system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: FUJI JUKOGYO KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TAMURA, YUICHIROH;NATSUME, KENSAKU;REEL/FRAME:035244/0770 Effective date: 20150121 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SUBARU CORPORATION, JAPAN Free format text: CHANGE OF NAME;ASSIGNOR:FUJI JUKOGYO KABUSHIKI KAISHA;REEL/FRAME:042624/0886 Effective date: 20170401 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |