US9665782B2 - Obstacle detecting apparatus and obstacle detecting method - Google Patents
Obstacle detecting apparatus and obstacle detecting method Download PDFInfo
- Publication number
- US9665782B2 US9665782B2 US14/963,452 US201514963452A US9665782B2 US 9665782 B2 US9665782 B2 US 9665782B2 US 201514963452 A US201514963452 A US 201514963452A US 9665782 B2 US9665782 B2 US 9665782B2
- Authority
- US
- United States
- Prior art keywords
- triangles
- obstacle
- normal
- feature points
- obstacle detecting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R21/00—Arrangements or fittings on vehicles for protecting or preventing injuries to occupants or pedestrians in case of accidents or other traffic risks
- B60R21/01—Electrical circuits for triggering passive safety arrangements, e.g. airbags, safety belt tighteners, in case of vehicle accidents or impending vehicle accidents
- B60R21/013—Electrical circuits for triggering passive safety arrangements, e.g. airbags, safety belt tighteners, in case of vehicle accidents or impending vehicle accidents including means for detecting collisions, impending collisions or roll-over
- B60R21/0134—Electrical circuits for triggering passive safety arrangements, e.g. airbags, safety belt tighteners, in case of vehicle accidents or impending vehicle accidents including means for detecting collisions, impending collisions or roll-over responsive to imminent contact with an obstacle, e.g. using radar systems
-
- G06K9/00805—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
- B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
-
- G06K9/469—
-
- G06T7/0075—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V10/422—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation for representing the structure of the pattern or shape of an object therefor
- G06V10/426—Graphical representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
-
- H04N13/0022—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/80—Spatial relation or speed relative to objects
- B60W2554/802—Longitudinal distance
-
- G06K2009/4666—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30261—Obstacle
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Definitions
- the present invention relates to an obstacle detecting apparatus, and more particularly, to an obstacle detecting apparatus and an obstacle detecting method which set a feature point in a vehicle front image which is output by a stereo camera to calculate a plane of the same space and add adjacent planes by calculating a normal, and detect an obstacle in a normal direction.
- a technology which detects an object in front of a vehicle to control the vehicle in accordance with the object is applied to lots of vehicles.
- a technology which detects an object near the vehicle using a stereo camera has gradually increased in use.
- an object detecting device using a stereo camera of the related art has a problem of processing an image near a vehicle which needs to be processed in real time, due to a complex algorithm.
- a technology of detecting an object near the vehicle is directly connected to a safety of a driver, a pedestrian, and a driver of another vehicle.
- Recently, a technology which rapidly processes an image which is photographed in a vehicle to detect an obstacle near the vehicle is being studied.
- the present invention has been made in an effort to provide an obstacle detecting apparatus which detects a feature point from a vehicle front image which is generated by a stereo camera and detects an obstacle in front of the vehicle based on a normal direction of a triangle formed by connecting the detected feature points.
- An exemplary embodiment of the present invention provides an obstacle detecting apparatus, including: a stereo camera which photographs a front of a vehicle to generate a left image and a right image; an image matching unit which applies a block matching algorithm to the left image and the right image to extract a plurality of feature points having a similarity which is equal to or larger than a reference value and calculate a coordinate and a depth of the plurality of extracted feature points; a plane calculating unit which connects every three feature points of the plurality of feature points to calculate a plurality of triangles, based on the coordinate and the depth of the plurality of feature points; a normal calculating unit which calculates directions of normal of the plurality of triangles, based on the coordinate and the depth of three feature points corresponding to vertexes of the plurality of triangles; and an obstacle detecting unit which detects an obstacle in front of the vehicle based on a result of comparing a photographing direction of the stereo camera with the normal directions of the plurality of triangles.
- the image matching unit may calculate a region where brightness change is equal to or larger than a predetermined value, in the left image and the right image, using the edge detection, and the block matching algorithm may be applied to the region where the brightness change is equal to or larger than the predetermined value to extract the plurality of feature points.
- the plane calculating unit may calculate the plurality of triangles using Delaunay triangulation.
- the normal calculating unit may calculate a direction of a normal which passes through a center of gravity of the plurality of triangles.
- the normal calculating unit may set normal directions of two triangles to have the same value.
- the normal calculating unit may set the normal of the first triangle and the second triangle to have any one of the first direction, the second direction, and an intermediate direction of the first direction and the second direction.
- the obstacle detecting unit may detect a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is smaller than the predetermined second threshold value, as an obstacle, among the plurality of triangles.
- the obstacle detecting unit may determine a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is equal to or larger than the predetermined second threshold value, as a road surface, among the plurality of triangles.
- the apparatus may further include a warning unit which outputs information on the obstacle or a warning message, when an obstacle in front of the vehicle is detected by the obstacle detecting unit.
- an obstacle detecting method including: generating a left image and a right image with respect to a front of a vehicle, using a stereo camera; extracting a plurality of feature points having a similarity which is equal to or larger than a reference value, by applying block matching algorithm to the left image and the right image; calculating a coordinate and a depth of the plurality of extracted feature points; connecting every three feature points of the plurality of feature points to calculate a plurality of triangles, based on the coordinate and the depth of the plurality of feature points; calculating directions of normal of the plurality of triangles, based on the coordinate and the depth of three feature points corresponding to vertexes of the plurality of triangles; and detecting an obstacle in front of the vehicle based on a result of comparing a photographing direction of the stereo camera with the normal directions of the plurality of triangles.
- the extracting of a plurality of feature points may include: calculating a region where brightness change is equal to or larger than a predetermined value, in the left image and the right image, using the edge detection; and applying the block matching algorithm to the region where the brightness change is equal to or larger than the predetermined value to extract the plurality of feature points.
- a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is smaller than the predetermined second threshold value is detected as an obstacle, among the plurality of triangles.
- the method may further include outputting information on the obstacle or a warning message when an obstacle in front of the vehicle is detected.
- a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is equal to or larger than the predetermined value is detected as a road surface, among the plurality of triangles.
- an obstacle is detected from only a part of a left image and a right image which are generated by a stereo camera, so that even though a cheap component whose image processing speed is relatively slow is used, a predetermined level or higher of obstacle detecting performance may be secured.
- FIG. 1 is a diagram illustrating a configuration of an obstacle detecting apparatus according to an exemplary embodiment of the present invention.
- FIG. 2 illustrates a result of calculating a normal vector at an arbitrary point by a normal calculating unit according to an exemplary embodiment.
- FIG. 3 illustrates an example which detects an obstacle from an image photographed using a stereo camera according to an exemplary embodiment of the present invention.
- FIG. 4 illustrates an example of determining whether to be an obstacle based on a normal direction of a triangle according to an exemplary embodiment of the present invention.
- FIG. 5 is a flow chart of an obstacle detecting method according to an exemplary embodiment of the present invention.
- FIG. 1 is a diagram illustrating a configuration of an obstacle detecting apparatus according to an exemplary embodiment of the present invention.
- An obstacle detecting apparatus according to an exemplary embodiment of the present invention includes a stereo camera 100 , an image matching unit 200 , a plane calculating unit 300 , a normal calculating unit 400 , and an obstacle detecting unit 500 .
- the stereo camera 100 photographs a front of the vehicle to generate a left image and a right image.
- the stereo camera 100 is mounted in a position (for example, an upper side of a wind shield or a bonnet) of the vehicle to photograph a front of the vehicle corresponding to a predetermined photographing direction to generate a left image and a right image and output the generated left image and right image to the image matching unit 200 .
- the stereo camera 100 may generate the left image photographed through a left lens of the stereo camera 100 and the right image photographed through a right lens of the stereo camera 100 .
- the photographing direction of the stereo camera 100 may be set in advance so as to be the same as a driving direction of the vehicle when the vehicle goes straight.
- the left image and the right image may refer to images which are sequentially generated as time goes by.
- a left image and a right image at a first point of time may be different from a left image and a right image at a second point of time.
- the image matching unit 200 extracts a plurality of feature points included in the left image and the right image.
- the plurality of feature points may be three or more feature points so as to calculate at least one plane.
- the image matching unit 200 applies a block matching algorithm to the left image and the right image to extract a plurality of feature points having a similarity which is equal to or larger than a reference value.
- each feature point may correspond to one pixel.
- the image matching unit 200 may extract pixels which are included in a block having a sum of absolute differences (SAD) or a sum of squared differences (SSD) which is smaller than a predetermined value, from the left image and the right image, as feature points.
- SAD sum of absolute differences
- SSD sum of squared differences
- a pixel which is included in a block having the sum of absolute differences (SAD) or a sum of squared differences (SSD) which is smaller than predetermined value may be a pixel having a similarity which is equal to or larger than a reference value, among entire pixels included in the left image and the right image.
- the image matching unit 200 may calculate a coordinate and a depth of the plurality of feature points.
- the depth of each of the feature points is calculated based on a disparity between the left image and the right image, detailed description thereof will be omitted.
- the image matching unit 200 may calibrate distortion between the left image and the right image.
- the image matching unit 200 may correct distortion of an image which may occur when the left lens and the right lens of the stereo camera 100 are wide angle lenses.
- the image matching unit 200 may correct pin cushion distortion which is formed in the left image and the right image.
- the image matching unit 200 may correct barrel distortion which is formed in the left image and the right image.
- the image matching unit 200 may correct radial distortion which is formed in the left image and the right image.
- the image matching unit 200 may correct tangential distortion which is formed in the left image and the right image.
- the image matching unit 200 may perform rectification on distortion in the left image and the right image.
- the image matching unit 200 may intersect the left image and the right image to generate a stereoscopic image which matches a point on a scan line.
- the image matching unit 200 sets a pixel which is common to the left image and the right image as a feature point and calculate a distance between the feature points.
- the image matching unit 200 may calculate a depth of an edge which is a region of the left image and the right image where a brightness change which is equal to or larger than a predetermined value is generated. For example, since a plurality of pixels may be included in the edge, the image matching unit 200 may calculate depths of the feature points as a set with respect to the edge.
- the image matching unit 200 may find a corresponding point from the left image and the right image to calculate a triangulation point.
- the image matching unit 200 may calculate a spatial positional transformation of rearranged pixels of the left image and the right image which are sequentially input as time goes by.
- the image matching unit 200 may calculate image rotation displacement of a pixel which is rearranged in the left image and the right image.
- the image matching unit 200 may calculate an average image having an average information value (for example, a brightness, color, a depth) in the left image and the right image.
- an average information value for example, a brightness, color, a depth
- the image matching unit 200 may calculate information on an illumination component which represents a light and shade in the left image and the right image. In this case, the image matching unit 200 convolutes calculated information on the illumination component and information of an input image to calculate the average image.
- the image matching unit 200 calculates information on a reflection component which represents a feature of an object in the vehicle front image using the average image.
- the image matching unit 200 removes a pixel information value of the average image from pixel information values of the left image and the right image in accordance with a coordinate of each pixel included in the left image and the right image to calculate information on the reflection component.
- the image matching unit 200 may compress calculated information on the reflection component.
- the image matching unit 200 calculates information on the reflection component and scales the calculated information on the reflection component as a log.
- the image matching unit 200 may perform stereo matching between the left image and the right image using a dynamic programming method. Further, the image matching unit 200 may perform stereo matching to generate a disparity map.
- the image matching unit 200 may draw a triangle by connecting lines based on an efficient large-scale stereo (ELAS) matching method according to the exemplary embodiment to calculate a depth.
- ELAS efficient large-scale stereo
- the image matching unit 200 may output the depth and the coordinate calculated for every feature point to the plane calculating unit 300 and the normal calculating unit 400 .
- the plane calculating unit 300 calculates a plane where the feature points are estimated to be present in the same space.
- the plane calculating unit 300 performs color segmentation on the vehicle front image to calculate a plane where the feature points are estimated to be present in the same space.
- the plane calculating unit 300 connects every three feature points of the plurality of feature points based on the coordinate and the depth of the plurality of feature points to calculate a plurality of triangles.
- the triangle refers to a triangular plane and vertexes of the triangle correspond to three different feature points.
- the plane calculating unit 300 may calculate a plurality of triangles from the plurality of feature points, using a Delaunay triangulation.
- the Delaunay triangulation may refer a technique which, when points included in an arbitrary space are connected to form a plurality of triangles, does not include a point whose circumscribed circle is not included in the triangle.
- the normal calculating unit 400 may calculate a direction (for example, a normal direction) to which the plurality of triangles calculated by the plane calculating unit 300 is directed.
- the triangles calculated by the plane calculating unit 300 include three feature points and since the coordinate and the depth of the feature point have been calculated in advance by the image matching unit 200 , the normal calculating unit 400 may calculate a normal direction of each triangle or the normal vector. In this case, the normal calculating unit 400 may calculate a direction of the normal which passes through a center of gravity of each triangle.
- the normal calculating unit 400 may add two triangles which have similar normal among the plurality of triangles as one plane. For example, when a difference of the normal directions of two different triangles is within a predetermined angle, the normal calculating unit 400 determines that the two triangles are similar and adds the two triangles as one plane. In this case, when a difference between the normal direction of the added plane and the photographing direction of the stereo camera 100 is smaller than a predetermined angle, the normal calculating unit 400 may detect the plane obtained by adding two triangles as an obstacle. By doing this, a computation amount required for the obstacle detecting unit 500 to detect an obstacle, which will be described below, may be reduced.
- the normal calculating unit 400 may set the normal directions of the two triangles to have the same value. Specifically, when, among the plurality of triangles, the first triangle and the second triangle share one corner and a difference between a first direction which is a normal direction of a first triangle and a second direction which is a normal direction of a second triangle is smaller than a predetermined first threshold value, the normal calculating unit 400 may set the normal of the first triangle and the second triangle to have any one of the first direction, the second direction, and an intermediate direction of the first direction and the second direction.
- the obstacle detecting unit 500 may detect an obstacle near the vehicle (for example, front), based on a result of comparing the photographing direction of the stereo camera 100 with the normal directions of the plurality of triangles.
- the normal calculating unit 400 may extract a triangle which faces the stereo camera 100 , among the plurality of triangles.
- the triangle which faces the stereo camera 100 may refer to a triangle which has a normal direction having a difference from the photographing direction of the stereo camera 100 which is smaller than the predetermined second threshold value, among the triangles calculated by the plane calculating unit 300 .
- the obstacle detecting unit 500 may detect an obstacle only with respect to at least one triangle which is included in a predetermined region of interest (ROI), among stereoscopic images generated by the image matching unit 200 . For example, even though among the plurality of triangles, a difference between a normal direction of a third triangle located outside the region of interest ROI and the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value, the obstacle detecting unit 500 may not detect a third triangle as an obstacle.
- ROI region of interest
- the obstacle detecting unit 500 may not detect the triangle as an obstacle. For example, when the normal of an arbitrary triangle is directed to a direction perpendicular to the photographing direction of the stereo camera 100 , the obstacle detecting unit 500 may determine that the triangle is not an obstacle but a road surface.
- a warning unit 600 may output information on the detected obstacle or a warning message.
- the information on the detected obstacle may include information on at least one of a size of the obstacle, a position of the obstacle, and a distance from the vehicle to the obstacle.
- the warning unit 600 may output the warning message only when the distance to the detected obstacle is shorter than a predetermined reference distance.
- the warning message may be visually displayed on a display of the vehicle or audibly output through a speaker of the vehicle.
- FIG. 2 illustrates a result of calculating a normal vector Ni at an arbitrary point by a normal calculating unit 400 according to an exemplary embodiment.
- FIG. 2A illustrates a pixel of a depth map and
- FIG. 2B illustrates three-dimensional point set data.
- the normal calculating unit 400 selects four points which are located up and down and left and right on a pixel with respect to a point Pi on the depth map and finds four points selected in FIG. 2A on the three-dimensional point set data of FIG. 2B to configure a vector between points located left and right on the pixel and a vector between points located up and down and then calculate an outer product between two vectors to calculate a normal.
- FIG. 3 illustrates an example which detects an obstacle from an image photographed using the stereo camera 100 according to an exemplary embodiment of the present invention.
- the plane calculating unit 300 calculates a plurality of feature points from a stereoscopic image generated by the image matching unit 200 and the normal calculating unit 400 connects a plurality of feature points to calculate a plurality of triangles.
- FIG. 3A illustrates a result in which the plane calculating unit 300 connects every three feature points of the plurality of feature points extracted from the stereoscopic image generated by the image matching unit 200 to calculate the plurality of triangles.
- some triangle may have a common corner with other triangle and a plurality of triangles may be sequentially connected.
- the normal calculating unit 400 may calculate normal direction for the plurality of triangles which is sequentially connected as illustrated in FIG. 3A .
- FIG. 3B illustrates a result in which the obstacle detecting unit 500 classifies the plurality of triangles illustrated in FIG. 3A depending on whether to have a difference from the photographing direction of the stereo camera 100 which is smaller than the predetermined second threshold value.
- the obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value as an obstacle. Further, the obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is equal to or larger than the predetermined second threshold value as a non-obstacle.
- a region represented with a gray color in FIG. 3B is a region where a triangle which is classified as an obstacle is located and a region represented with a black is a region where a triangle which is not classified as an obstacle is located.
- the obstacle detecting unit 500 detects an obstacle based on the normal direction of triangles with some pixels corresponding to the feature points as vertexes, rather than the entre pixel included in the stereoscopic image, so that the obstacle detecting speed may be significantly improved without losing a performance.
- FIG. 4 illustrates an example of determining whether to be an obstacle based on a normal direction of a triangle according to an exemplary embodiment of the present invention. For the convenience of description, description will be made with respect to a side view.
- the stereo camera 100 is provided in one position (for example, a wind shield) of a driving vehicle to photograph an object in front of the vehicle.
- the photographing direction of the stereo camera 100 is parallel to a ground, as illustrated in the drawing.
- the object which is photographed may be calculated as a plurality of triangles by the plane calculating unit 300 and some of triangles may act as an obstacle with respect to the vehicle.
- the plane calculating unit 300 may calculate a first triangle T 1 and a second triangle T 2 based on an image for the object. Further, the normal calculating unit 400 may a first normal N 1 with respect to the first triangle T 1 and a second normal N 2 for the second triangle T 2 .
- the obstacle detecting unit 500 may calculate a first direction difference ⁇ 1 which is a difference between the photographing direction D A of the stereo camera 100 parallel to the ground and a direction of the first normal N 1 .
- the obstacle detecting unit 500 may calculate a second direction difference ⁇ 2 which is a difference between the photographing direction D A of the stereo camera 100 parallel to the ground and a direction of the second normal N 2 .
- the obstacle detecting unit 500 detects the first triangle T 1 as an obstacle but does not detect the second triangle T 2 as an obstacle.
- FIG. 4 it is illustrated that a single object is present in front of the vehicle and two triangles are calculated for the object, it is only an example.
- the same object detecting method may be also applied.
- FIG. 5 is a flow chart of an obstacle detecting method according to an exemplary embodiment of the present invention.
- a stereo camera 100 generates a left image and a right image in step S 100 .
- the stereo camera 100 is mounted in one position in front of the vehicle to have a predetermined photographing direction and generates a left image using a left lens and a right image using a right lens. Further, the stereo camera 100 may output the left image and the right image to an image matching unit 200 .
- an image matching unit 200 extracts a plurality of feature points commonly included in the left image and the right image in step S 110 .
- the image matching unit 200 applies a block matching algorithm to extract a feature point which is a pixel which is commonly included in the left image and the right image and has a similarity which is larger than a reference value.
- the image matching unit 200 may calculate a depth from an edge which is equal to or larger than a predetermined value in the images.
- the image matching unit 200 calculates a region where a brightness change is equal to or larger than a predetermined value in the left image and the right image using edge detection and then applies the block matching algorithm to the region which is calculated that the brightness change is equal to or larger than the predetermined value to extract a plurality of feature points.
- the image matching unit 200 may calculate a coordinate and a depth of the plurality of extracted feature points in step S 120 .
- the image matching unit 200 may calculate the depth of each of the feature points based on a disparity between the left image and the right image. Further, the image matching unit 200 may calculate a depth of the plurality of feature points as a set.
- the plane calculating unit 300 connects every three feature points of the plurality of feature points based on the coordinate and the depth of the plurality of feature points to calculate a plurality of triangles in step S 130 .
- a normal calculating unit 400 calculates normal directions of the plurality of triangles in step S 140 .
- Three vertexes of each triangle correspond to the extracted feature points and the coordinate and the depth of the feature points are already calculated, so that the normal calculating unit 400 may calculate the normal directions of the plurality of triangles.
- the obstacle detecting unit 500 may detect an obstacle in front of the vehicle, based on a result of comparing the photographing direction of the stereo camera 100 with the normal directions of the plurality of triangles in step S 150 .
- an obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value as an obstacle, among the plurality of triangles.
- the obstacle detecting unit 500 may calculate information on at least one of a size of the detected obstacle, a position of the obstacle, and a distance from the vehicle to the obstacle. For example, the obstacle detecting unit 500 may calculate a size of the obstacle based on an area of the triangle classified as an obstacle, calculate a distance to the obstacle based on the depth of the feature point included in the triangle classified as an obstacle, and calculate a position of the obstacle based on the coordinate of the feature point included in the triangle classified as an obstacle.
- the obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is equal to or larger than the predetermined second threshold value as a non-obstacle, among the plurality of triangles. Further, the normal calculating unit 400 may receive depths of the feature points as a set which is output by the image matching unit 200 . Further, the obstacle detecting unit 500 may detect an obstacle only with respect to triangles which are included in a predetermined region of interest (ROI), among stereoscopic images generated by the image matching unit 200 .
- ROI region of interest
- the obstacle detecting unit 500 may not detect a third triangle as an obstacle.
- a warning unit 600 may output information on the detected obstacle or a warning message in step S 160 .
- the information on the detected obstacle may include information on at least one of a size of the obstacle, a position of the obstacle, and a distance from the vehicle to the obstacle.
- the warning unit 600 may output the warning message only when the distance to the detected obstacle is shorter than a predetermined reference distance.
- the warning message may be visually displayed on a display of the vehicle or audibly output through a speaker of the vehicle.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mechanical Engineering (AREA)
- Transportation (AREA)
- Mathematical Physics (AREA)
- Automation & Control Theory (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
- Image Processing (AREA)
- Geometry (AREA)
Abstract
The present invention relates to an obstacle detecting apparatus and an obstacle detecting method and an exemplary embodiment of the present invention provides an obstacle detecting apparatus, including: a stereo camera which photographs a front of a vehicle to generate a left image and a right image; an image matching unit which applies a block matching algorithm to the left image and the right image to extract a plurality of feature points having a similarity which is equal to or larger than a reference value and calculate a coordinate and a depth of the plurality of extracted feature points; a plane calculating unit which connects every three feature points of the plurality of feature points to calculate a plurality of triangles, based on the coordinate and the depth of the plurality of feature points; a normal calculating unit which calculates directions of normal of the plurality of triangles, based on the coordinate and the depth of three feature points corresponding to vertexes of the plurality of triangles; and an obstacle detecting unit which detects an obstacle in front of the vehicle based on a result of comparing a photographing direction of the stereo camera with the normal directions of the plurality of triangles.
Description
The present application claims priority to Korean Patent Application Number 10-2014-0186267 filed Dec. 22, 2014, the entire contents of which application is incorporated herein for all purposes by this reference.
The present invention relates to an obstacle detecting apparatus, and more particularly, to an obstacle detecting apparatus and an obstacle detecting method which set a feature point in a vehicle front image which is output by a stereo camera to calculate a plane of the same space and add adjacent planes by calculating a normal, and detect an obstacle in a normal direction.
Recently, a technology which detects an object in front of a vehicle to control the vehicle in accordance with the object is applied to lots of vehicles. Specifically, a technology which detects an object near the vehicle using a stereo camera has gradually increased in use.
However, an object detecting device using a stereo camera of the related art has a problem of processing an image near a vehicle which needs to be processed in real time, due to a complex algorithm. A technology of detecting an object near the vehicle is directly connected to a safety of a driver, a pedestrian, and a driver of another vehicle. Recently, a technology which rapidly processes an image which is photographed in a vehicle to detect an obstacle near the vehicle is being studied.
The present invention has been made in an effort to provide an obstacle detecting apparatus which detects a feature point from a vehicle front image which is generated by a stereo camera and detects an obstacle in front of the vehicle based on a normal direction of a triangle formed by connecting the detected feature points.
Objects of the present invention are not limited to the above-mentioned objects, and other objects, which are not mentioned above, can be clearly understood by those skilled in the art from the following descriptions.
An exemplary embodiment of the present invention provides an obstacle detecting apparatus, including: a stereo camera which photographs a front of a vehicle to generate a left image and a right image; an image matching unit which applies a block matching algorithm to the left image and the right image to extract a plurality of feature points having a similarity which is equal to or larger than a reference value and calculate a coordinate and a depth of the plurality of extracted feature points; a plane calculating unit which connects every three feature points of the plurality of feature points to calculate a plurality of triangles, based on the coordinate and the depth of the plurality of feature points; a normal calculating unit which calculates directions of normal of the plurality of triangles, based on the coordinate and the depth of three feature points corresponding to vertexes of the plurality of triangles; and an obstacle detecting unit which detects an obstacle in front of the vehicle based on a result of comparing a photographing direction of the stereo camera with the normal directions of the plurality of triangles.
The image matching unit may calculate a region where brightness change is equal to or larger than a predetermined value, in the left image and the right image, using the edge detection, and the block matching algorithm may be applied to the region where the brightness change is equal to or larger than the predetermined value to extract the plurality of feature points.
The plane calculating unit may calculate the plurality of triangles using Delaunay triangulation.
The normal calculating unit may calculate a direction of a normal which passes through a center of gravity of the plurality of triangles.
When a difference of directions of the normal of two triangles, which share one corner, among the plurality of triangles, is smaller than a predetermined first threshold value, the normal calculating unit may set normal directions of two triangles to have the same value.
When, among the plurality of triangles, the first triangle and the second triangle share one corner and a difference between a first direction which is a normal direction of a first triangle and a second direction which is a normal direction of a second triangle is smaller than a predetermined first threshold value, the normal calculating unit may set the normal of the first triangle and the second triangle to have any one of the first direction, the second direction, and an intermediate direction of the first direction and the second direction.
The obstacle detecting unit may detect a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is smaller than the predetermined second threshold value, as an obstacle, among the plurality of triangles.
The obstacle detecting unit may determine a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is equal to or larger than the predetermined second threshold value, as a road surface, among the plurality of triangles.
The apparatus may further include a warning unit which outputs information on the obstacle or a warning message, when an obstacle in front of the vehicle is detected by the obstacle detecting unit.
Another exemplary embodiment of the present invention provides, an obstacle detecting method, including: generating a left image and a right image with respect to a front of a vehicle, using a stereo camera; extracting a plurality of feature points having a similarity which is equal to or larger than a reference value, by applying block matching algorithm to the left image and the right image; calculating a coordinate and a depth of the plurality of extracted feature points; connecting every three feature points of the plurality of feature points to calculate a plurality of triangles, based on the coordinate and the depth of the plurality of feature points; calculating directions of normal of the plurality of triangles, based on the coordinate and the depth of three feature points corresponding to vertexes of the plurality of triangles; and detecting an obstacle in front of the vehicle based on a result of comparing a photographing direction of the stereo camera with the normal directions of the plurality of triangles.
The extracting of a plurality of feature points may include: calculating a region where brightness change is equal to or larger than a predetermined value, in the left image and the right image, using the edge detection; and applying the block matching algorithm to the region where the brightness change is equal to or larger than the predetermined value to extract the plurality of feature points.
In the calculating of normal directions of the plurality of triangles, when a difference of directions of the normal of two triangles, which share one corner, among the plurality of triangles, is smaller than a predetermined first threshold value, normal directions of two triangles are set to have the same value.
In the detecting of an obstacle in front of the vehicle, a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is smaller than the predetermined second threshold value is detected as an obstacle, among the plurality of triangles.
The method may further include outputting information on the obstacle or a warning message when an obstacle in front of the vehicle is detected.
In the detecting of an obstacle, a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is equal to or larger than the predetermined value is detected as a road surface, among the plurality of triangles.
Other detailed matters of the embodiments are included in the detailed description and the drawings.
According to the exemplary embodiment of the obstacle detecting apparatus, there are one or more advantages as follows.
First, only depths of feature points which are included in predetermined regions having high similarities rather than an entire region of a left image and a right image generated by a stereo camera are calculated, thereby reducing a computation amount required to detect the obstacle from the image.
Second, whether there is an obstacle is detected based on a normal direction of a triangle formed using three feature points as vertexes, so that a depth estimating process is omitted, thereby improving an obstacle detecting speed.
Third, an obstacle is detected from only a part of a left image and a right image which are generated by a stereo camera, so that even though a cheap component whose image processing speed is relatively slow is used, a predetermined level or higher of obstacle detecting performance may be secured.
The effects of the present invention are not limited to the aforementioned effects, and other effects, which are not mentioned above, will be apparently understood by a person skilled in the art from the recitations of the claims.
Advantages and characteristics of the present invention and a method of achieving the advantages and characteristics will be clear by referring to exemplary embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to exemplary embodiments disclosed herein but will be implemented in various different forms. The exemplary embodiments are provided by way of example only so that a person of ordinary skilled in the art can fully understand the disclosures of the present invention and the scope of the present invention. Therefore, the present invention will be defined only by the scope of the appended claims. Like reference numerals indicate like elements throughout the specification.
Herein, the present invention will be described with reference to the drawings for explaining an obstacle detecting apparatus according to an embodiment of the present invention.
First, the stereo camera 100 photographs a front of the vehicle to generate a left image and a right image. Specifically, the stereo camera 100 is mounted in a position (for example, an upper side of a wind shield or a bonnet) of the vehicle to photograph a front of the vehicle corresponding to a predetermined photographing direction to generate a left image and a right image and output the generated left image and right image to the image matching unit 200. For example, the stereo camera 100 may generate the left image photographed through a left lens of the stereo camera 100 and the right image photographed through a right lens of the stereo camera 100. In this case, the photographing direction of the stereo camera 100 may be set in advance so as to be the same as a driving direction of the vehicle when the vehicle goes straight.
In this case, the left image and the right image may refer to images which are sequentially generated as time goes by. For example, a left image and a right image at a first point of time may be different from a left image and a right image at a second point of time.
The image matching unit 200 extracts a plurality of feature points included in the left image and the right image. In this case, the plurality of feature points may be three or more feature points so as to calculate at least one plane. Specifically, the image matching unit 200 applies a block matching algorithm to the left image and the right image to extract a plurality of feature points having a similarity which is equal to or larger than a reference value. Here, each feature point may correspond to one pixel. For example, the image matching unit 200 may extract pixels which are included in a block having a sum of absolute differences (SAD) or a sum of squared differences (SSD) which is smaller than a predetermined value, from the left image and the right image, as feature points. That is, in the left image and the right image, a pixel which is included in a block having the sum of absolute differences (SAD) or a sum of squared differences (SSD) which is smaller than predetermined value may be a pixel having a similarity which is equal to or larger than a reference value, among entire pixels included in the left image and the right image.
The image matching unit 200 may calculate a coordinate and a depth of the plurality of feature points. Here, since it is known that the depth of each of the feature points is calculated based on a disparity between the left image and the right image, detailed description thereof will be omitted.
The image matching unit 200 may calibrate distortion between the left image and the right image.
The image matching unit 200 may correct distortion of an image which may occur when the left lens and the right lens of the stereo camera 100 are wide angle lenses.
The image matching unit 200 may correct pin cushion distortion which is formed in the left image and the right image.
The image matching unit 200 may correct barrel distortion which is formed in the left image and the right image.
The image matching unit 200 may correct radial distortion which is formed in the left image and the right image.
The image matching unit 200 may correct tangential distortion which is formed in the left image and the right image.
The image matching unit 200 may perform rectification on distortion in the left image and the right image.
The image matching unit 200 may intersect the left image and the right image to generate a stereoscopic image which matches a point on a scan line.
The image matching unit 200 sets a pixel which is common to the left image and the right image as a feature point and calculate a distance between the feature points.
The image matching unit 200 may calculate a depth of an edge which is a region of the left image and the right image where a brightness change which is equal to or larger than a predetermined value is generated. For example, since a plurality of pixels may be included in the edge, the image matching unit 200 may calculate depths of the feature points as a set with respect to the edge.
The image matching unit 200 may find a corresponding point from the left image and the right image to calculate a triangulation point.
The image matching unit 200 may calculate a spatial positional transformation of rearranged pixels of the left image and the right image which are sequentially input as time goes by.
The image matching unit 200 may calculate image rotation displacement of a pixel which is rearranged in the left image and the right image.
The image matching unit 200 may calculate an average image having an average information value (for example, a brightness, color, a depth) in the left image and the right image.
The image matching unit 200 may calculate information on an illumination component which represents a light and shade in the left image and the right image. In this case, the image matching unit 200 convolutes calculated information on the illumination component and information of an input image to calculate the average image.
The image matching unit 200 calculates information on a reflection component which represents a feature of an object in the vehicle front image using the average image.
The image matching unit 200 removes a pixel information value of the average image from pixel information values of the left image and the right image in accordance with a coordinate of each pixel included in the left image and the right image to calculate information on the reflection component.
The image matching unit 200 may compress calculated information on the reflection component.
The image matching unit 200 calculates information on the reflection component and scales the calculated information on the reflection component as a log.
The image matching unit 200 may perform stereo matching between the left image and the right image using a dynamic programming method. Further, the image matching unit 200 may perform stereo matching to generate a disparity map.
The image matching unit 200 may draw a triangle by connecting lines based on an efficient large-scale stereo (ELAS) matching method according to the exemplary embodiment to calculate a depth.
The image matching unit 200 may output the depth and the coordinate calculated for every feature point to the plane calculating unit 300 and the normal calculating unit 400.
The plane calculating unit 300 calculates a plane where the feature points are estimated to be present in the same space.
For example, the plane calculating unit 300 performs color segmentation on the vehicle front image to calculate a plane where the feature points are estimated to be present in the same space.
Alternatively, the plane calculating unit 300 connects every three feature points of the plurality of feature points based on the coordinate and the depth of the plurality of feature points to calculate a plurality of triangles. Here, the triangle refers to a triangular plane and vertexes of the triangle correspond to three different feature points.
For example, the plane calculating unit 300 may calculate a plurality of triangles from the plurality of feature points, using a Delaunay triangulation. Here, the Delaunay triangulation may refer a technique which, when points included in an arbitrary space are connected to form a plurality of triangles, does not include a point whose circumscribed circle is not included in the triangle.
The normal calculating unit 400 may calculate a direction (for example, a normal direction) to which the plurality of triangles calculated by the plane calculating unit 300 is directed. Specifically, the triangles calculated by the plane calculating unit 300 include three feature points and since the coordinate and the depth of the feature point have been calculated in advance by the image matching unit 200, the normal calculating unit 400 may calculate a normal direction of each triangle or the normal vector. In this case, the normal calculating unit 400 may calculate a direction of the normal which passes through a center of gravity of each triangle.
The normal calculating unit 400 may add two triangles which have similar normal among the plurality of triangles as one plane. For example, when a difference of the normal directions of two different triangles is within a predetermined angle, the normal calculating unit 400 determines that the two triangles are similar and adds the two triangles as one plane. In this case, when a difference between the normal direction of the added plane and the photographing direction of the stereo camera 100 is smaller than a predetermined angle, the normal calculating unit 400 may detect the plane obtained by adding two triangles as an obstacle. By doing this, a computation amount required for the obstacle detecting unit 500 to detect an obstacle, which will be described below, may be reduced.
As another example, when among the plurality of triangles, two triangles share one corner and a difference of the normal directions is within a predetermined angle, the normal calculating unit 400 may set the normal directions of the two triangles to have the same value. Specifically, when, among the plurality of triangles, the first triangle and the second triangle share one corner and a difference between a first direction which is a normal direction of a first triangle and a second direction which is a normal direction of a second triangle is smaller than a predetermined first threshold value, the normal calculating unit 400 may set the normal of the first triangle and the second triangle to have any one of the first direction, the second direction, and an intermediate direction of the first direction and the second direction.
The obstacle detecting unit 500 may detect an obstacle near the vehicle (for example, front), based on a result of comparing the photographing direction of the stereo camera 100 with the normal directions of the plurality of triangles.
Specifically, the normal calculating unit 400 may extract a triangle which faces the stereo camera 100, among the plurality of triangles. Here, the triangle which faces the stereo camera 100 may refer to a triangle which has a normal direction having a difference from the photographing direction of the stereo camera 100 which is smaller than the predetermined second threshold value, among the triangles calculated by the plane calculating unit 300.
The obstacle detecting unit 500 may detect an obstacle only with respect to at least one triangle which is included in a predetermined region of interest (ROI), among stereoscopic images generated by the image matching unit 200. For example, even though among the plurality of triangles, a difference between a normal direction of a third triangle located outside the region of interest ROI and the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value, the obstacle detecting unit 500 may not detect a third triangle as an obstacle.
In the meantime, when a difference between a normal direction of any one triangle among the plurality of triangles and the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value, the obstacle detecting unit 500 may not detect the triangle as an obstacle. For example, when the normal of an arbitrary triangle is directed to a direction perpendicular to the photographing direction of the stereo camera 100, the obstacle detecting unit 500 may determine that the triangle is not an obstacle but a road surface.
When the obstacle detecting unit 500 detects an obstacle in front of the vehicle, a warning unit 600 may output information on the detected obstacle or a warning message. The information on the detected obstacle may include information on at least one of a size of the obstacle, a position of the obstacle, and a distance from the vehicle to the obstacle. In this case, the warning unit 600 may output the warning message only when the distance to the detected obstacle is shorter than a predetermined reference distance. The warning message may be visually displayed on a display of the vehicle or audibly output through a speaker of the vehicle.
First, FIG. 3A illustrates a result in which the plane calculating unit 300 connects every three feature points of the plurality of feature points extracted from the stereoscopic image generated by the image matching unit 200 to calculate the plurality of triangles. As illustrated in the drawing, some triangle may have a common corner with other triangle and a plurality of triangles may be sequentially connected. The normal calculating unit 400 may calculate normal direction for the plurality of triangles which is sequentially connected as illustrated in FIG. 3A .
Next, FIG. 3B illustrates a result in which the obstacle detecting unit 500 classifies the plurality of triangles illustrated in FIG. 3A depending on whether to have a difference from the photographing direction of the stereo camera 100 which is smaller than the predetermined second threshold value.
Specifically, the obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value as an obstacle. Further, the obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is equal to or larger than the predetermined second threshold value as a non-obstacle. A region represented with a gray color in FIG. 3B is a region where a triangle which is classified as an obstacle is located and a region represented with a black is a region where a triangle which is not classified as an obstacle is located.
Therefore, the obstacle detecting unit 500 detects an obstacle based on the normal direction of triangles with some pixels corresponding to the feature points as vertexes, rather than the entre pixel included in the stereoscopic image, so that the obstacle detecting speed may be significantly improved without losing a performance.
Referring to FIG. 4 , the stereo camera 100 is provided in one position (for example, a wind shield) of a driving vehicle to photograph an object in front of the vehicle. In this case, it is assumed that the photographing direction of the stereo camera 100 is parallel to a ground, as illustrated in the drawing.
In this case, the object which is photographed may be calculated as a plurality of triangles by the plane calculating unit 300 and some of triangles may act as an obstacle with respect to the vehicle.
For example, as illustrated in the drawing, the plane calculating unit 300 may calculate a first triangle T1 and a second triangle T2 based on an image for the object. Further, the normal calculating unit 400 may a first normal N1 with respect to the first triangle T1 and a second normal N2 for the second triangle T2.
The obstacle detecting unit 500 may calculate a first direction difference θ1 which is a difference between the photographing direction DA of the stereo camera 100 parallel to the ground and a direction of the first normal N1. The obstacle detecting unit 500 may calculate a second direction difference θ2 which is a difference between the photographing direction DA of the stereo camera 100 parallel to the ground and a direction of the second normal N2.
When a relationship of the first direction difference θ1, the second direction difference θ2, and a second threshold value θTh satisfies θ1<θTh≦θ2, the obstacle detecting unit 500 detects the first triangle T1 as an obstacle but does not detect the second triangle T2 as an obstacle.
Even though in FIG. 4 , it is illustrated that a single object is present in front of the vehicle and two triangles are calculated for the object, it is only an example. When two or more objects may be present in front of the vehicle and three or more triangles are calculated for each object, the same object detecting method may be also applied.
A stereo camera 100 according to an exemplary embodiment generates a left image and a right image in step S100. Specifically, the stereo camera 100 is mounted in one position in front of the vehicle to have a predetermined photographing direction and generates a left image using a left lens and a right image using a right lens. Further, the stereo camera 100 may output the left image and the right image to an image matching unit 200.
Next, an image matching unit 200 extracts a plurality of feature points commonly included in the left image and the right image in step S110. For example, the image matching unit 200 applies a block matching algorithm to extract a feature point which is a pixel which is commonly included in the left image and the right image and has a similarity which is larger than a reference value. Further, the image matching unit 200 may calculate a depth from an edge which is equal to or larger than a predetermined value in the images. Further, the image matching unit 200 calculates a region where a brightness change is equal to or larger than a predetermined value in the left image and the right image using edge detection and then applies the block matching algorithm to the region which is calculated that the brightness change is equal to or larger than the predetermined value to extract a plurality of feature points.
Next, the image matching unit 200 may calculate a coordinate and a depth of the plurality of extracted feature points in step S120. Here, the image matching unit 200 may calculate the depth of each of the feature points based on a disparity between the left image and the right image. Further, the image matching unit 200 may calculate a depth of the plurality of feature points as a set.
Next, the plane calculating unit 300 connects every three feature points of the plurality of feature points based on the coordinate and the depth of the plurality of feature points to calculate a plurality of triangles in step S130.
Next, a normal calculating unit 400 calculates normal directions of the plurality of triangles in step S140. Three vertexes of each triangle correspond to the extracted feature points and the coordinate and the depth of the feature points are already calculated, so that the normal calculating unit 400 may calculate the normal directions of the plurality of triangles.
The obstacle detecting unit 500 may detect an obstacle in front of the vehicle, based on a result of comparing the photographing direction of the stereo camera 100 with the normal directions of the plurality of triangles in step S150.
Next, an obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value as an obstacle, among the plurality of triangles.
In this case, the obstacle detecting unit 500 may calculate information on at least one of a size of the detected obstacle, a position of the obstacle, and a distance from the vehicle to the obstacle. For example, the obstacle detecting unit 500 may calculate a size of the obstacle based on an area of the triangle classified as an obstacle, calculate a distance to the obstacle based on the depth of the feature point included in the triangle classified as an obstacle, and calculate a position of the obstacle based on the coordinate of the feature point included in the triangle classified as an obstacle.
The obstacle detecting unit 500 may classify a triangle which has a normal direction whose difference from the photographing direction of the stereo camera 100 is equal to or larger than the predetermined second threshold value as a non-obstacle, among the plurality of triangles. Further, the normal calculating unit 400 may receive depths of the feature points as a set which is output by the image matching unit 200. Further, the obstacle detecting unit 500 may detect an obstacle only with respect to triangles which are included in a predetermined region of interest (ROI), among stereoscopic images generated by the image matching unit 200. For example, even though among the plurality of triangles, a difference between a normal direction of a third triangle located outside the region of interest ROI and the photographing direction of the stereo camera 100 is smaller than the predetermined second threshold value, the obstacle detecting unit 500 may not detect a third triangle as an obstacle.
Next, when the obstacle detecting unit 500 detects an obstacle in front of the vehicle, a warning unit 600 may output information on the detected obstacle or a warning message in step S160. The information on the detected obstacle may include information on at least one of a size of the obstacle, a position of the obstacle, and a distance from the vehicle to the obstacle. In this case, the warning unit 600 may output the warning message only when the distance to the detected obstacle is shorter than a predetermined reference distance. The warning message may be visually displayed on a display of the vehicle or audibly output through a speaker of the vehicle.
Preferred embodiments of the present invention have been illustrated and described above, but the present invention is not limited to the above-described specific embodiments, it is obvious that various modification may be made by those skilled in the art, to which the present invention pertains without departing from the gist of the present invention, which is claimed in the claims, and such modification should not be individually understood from the technical spirit or prospect of the present invention.
Claims (15)
1. An obstacle detecting apparatus, comprising:
a stereo camera which photographs a front of a vehicle to generate a left image and a right image;
an image matching unit which applies a block matching algorithm to the left image and the right image to extract a plurality of feature points having a similarity which is equal to or larger than a reference value and calculate a coordinate and a depth of the plurality of extracted feature points;
a plane calculating unit which connects every three feature points of the plurality of feature points to calculate a plurality of triangles, based on the coordinate and the depth of the plurality of feature points;
a normal calculating unit which calculates directions of normal of the plurality of triangles, based on the coordinate and the depth of three feature points corresponding to vertexes of the plurality of triangles; and
an obstacle detecting unit which detects an obstacle in front of the vehicle based on a result of comparing a photographing direction of the stereo camera with the normal directions of the plurality of triangles.
2. The obstacle detecting apparatus of claim 1 , wherein the image matching unit calculates a region where brightness change is equal to or larger than a predetermined value, in the left image and the right image, using the edge detection, and the block matching algorithm is applied to the region where the brightness change is equal to or larger than the predetermined value to extract the plurality of feature points.
3. The obstacle detecting apparatus of claim 1 , wherein the plane calculating unit calculates the plurality of triangles using Delaunay triangulation.
4. The obstacle detecting apparatus of claim 1 , wherein the normal calculating unit calculates a direction of a normal which passes through a center of gravity of the plurality of triangles.
5. The obstacle detecting apparatus of claim 1 , wherein when a difference of directions of the normal of two triangles, which share one corner, among the plurality of triangles, is smaller than a predetermined first threshold value, the normal calculating unit sets normal directions of two triangles to have the same value.
6. The obstacle detecting apparatus of claim 5 , wherein when a first triangle and a second triangle, among the plurality of triangles, share one corner and a difference between a first direction which is a normal direction of the first triangle and a second direction which is a normal direction of the second triangle is smaller than the predetermined first threshold value, the normal calculating unit sets the normal of the first triangle and the second triangle to have any one of the first direction, the second direction, and an intermediate direction of the first direction and the second direction.
7. The obstacle detecting apparatus of claim 1 , wherein the obstacle detecting unit detects a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is smaller than the predetermined second threshold value, as the obstacle, among the plurality of triangles.
8. The obstacle detecting apparatus of claim 1 , wherein the obstacle detecting unit determines a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is equal to or larger than the predetermined second threshold value, as a road surface, among the plurality of triangles.
9. The obstacle detecting apparatus of claim 1 , further comprising:
a warning unit which outputs information on the obstacle or a warning message, when an obstacle in front of the vehicle is detected by the obstacle detecting unit.
10. An obstacle detecting method, comprising:
generating a left image and a right image with respect to a front of a vehicle, using a stereo camera;
extracting a plurality of feature points having a similarity which is equal to or larger than a reference value, by applying block matching algorithm to the left image and the right image;
calculating a coordinate and a depth of the plurality of extracted feature points;
connecting every three feature points of the plurality of feature points to calculate a plurality of triangles, based on the coordinate and the depth of the plurality of feature points;
calculating directions of normal of the plurality of triangles, based on the coordinate and the depth of three feature points corresponding to vertexes of the plurality of triangles; and
detecting an obstacle in front of the vehicle based on a result of comparing a photographing direction of the stereo camera with the normal directions of the plurality of triangles.
11. The obstacle detecting method of claim 10 , wherein the extracting of a plurality of feature points includes:
calculating a region where brightness change is equal to or larger than a predetermined value, in the left image and the right image, using the edge detection; and
applying the block matching algorithm to the region where the brightness change is equal to or larger than the predetermined value to extract the plurality of feature points.
12. The obstacle detecting method of claim 10 , wherein in the calculating of normal directions of the plurality of triangles, when a difference of directions of the normal of two triangles, which share one corner, among the plurality of triangles, is smaller than a predetermined first threshold value, normal directions of two triangles are set to have the same value.
13. The obstacle detecting method of claim 10 , wherein in the detecting of an obstacle in front of the vehicle, a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is smaller than the predetermined second threshold value is detected as an obstacle, among the plurality of triangles.
14. The obstacle detecting method of claim 10 , further comprising:
outputting information on the obstacle or a warning message when an obstacle in front of the vehicle is detected.
15. The obstacle detecting method of claim 10 , wherein in the detecting of an obstacle, a triangle which has a normal direction which has a difference from the photographing direction of the stereo camera which is equal to or larger than the predetermined value is detected as a road surface, among the plurality of triangles.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR10-2014-0186267 | 2014-12-22 | ||
KR1020140186267A KR101676656B1 (en) | 2014-12-22 | 2014-12-22 | Apparatus and method for detecting obstacle |
Publications (2)
Publication Number | Publication Date |
---|---|
US20160180181A1 US20160180181A1 (en) | 2016-06-23 |
US9665782B2 true US9665782B2 (en) | 2017-05-30 |
Family
ID=56099806
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/963,452 Active US9665782B2 (en) | 2014-12-22 | 2015-12-09 | Obstacle detecting apparatus and obstacle detecting method |
Country Status (5)
Country | Link |
---|---|
US (1) | US9665782B2 (en) |
JP (1) | JP6291469B2 (en) |
KR (1) | KR101676656B1 (en) |
CN (1) | CN105718853B (en) |
DE (1) | DE102015121387B4 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180182078A1 (en) * | 2016-12-27 | 2018-06-28 | Kabushiki Kaisha Toshiba | Image processing apparatus and image processing method |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106265004A (en) * | 2016-10-08 | 2017-01-04 | 西安电子科技大学 | Multi-sensor intelligent blind person's guiding method and device |
JP6654999B2 (en) * | 2016-12-12 | 2020-02-26 | 株式会社Soken | Target detection device |
CN107194350A (en) * | 2017-05-19 | 2017-09-22 | 北京进化者机器人科技有限公司 | Obstacle detection method, device and robot |
US10437065B2 (en) * | 2017-10-03 | 2019-10-08 | Microsoft Technology Licensing, Llc | IPD correction and reprojection for accurate mixed reality object placement |
CN109214348A (en) * | 2018-09-19 | 2019-01-15 | 北京极智嘉科技有限公司 | A kind of obstacle detection method, device, equipment and storage medium |
CN111080689B (en) * | 2018-10-22 | 2023-04-14 | 杭州海康威视数字技术股份有限公司 | Method and device for determining face depth map |
GB2583774B (en) * | 2019-05-10 | 2022-05-11 | Robok Ltd | Stereo image processing |
KR102252888B1 (en) * | 2020-01-03 | 2021-05-17 | 포테닛 주식회사 | An apparatus and method for detecting position of transfer robot in space |
CN111241979B (en) * | 2020-01-07 | 2023-06-23 | 浙江科技学院 | Real-time obstacle detection method based on image feature calibration |
WO2023091129A1 (en) * | 2021-11-17 | 2023-05-25 | Innopeak Technology, Inc. | Plane-based camera localization |
Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20010048763A1 (en) | 2000-05-30 | 2001-12-06 | Takeshi Takatsuka | Integrated vision system |
JP2004094707A (en) | 2002-09-02 | 2004-03-25 | Omron Corp | Method for estimating plane by stereo image and detector for object |
US20040264763A1 (en) * | 2003-04-30 | 2004-12-30 | Deere & Company | System and method for detecting and analyzing features in an agricultural field for vehicle guidance |
WO2005081176A1 (en) | 2004-02-13 | 2005-09-01 | Robert Bosch Gmbh | Method for the classification of an object by means of a stereo camera |
US20060029272A1 (en) * | 2004-08-09 | 2006-02-09 | Fuji Jukogyo Kabushiki Kaisha | Stereo image processing device |
KR20080022748A (en) | 2006-09-07 | 2008-03-12 | 주식회사 만도 | Collision avoidance method using stereo camera |
WO2009028489A1 (en) | 2007-08-30 | 2009-03-05 | Kabushiki Kaisha Yaskawa Denki | Object detecting method, object detecting device, and robot system |
JP2009139323A (en) | 2007-12-10 | 2009-06-25 | Mazda Motor Corp | Travel road surface detecting apparatus for vehicle |
US20110216213A1 (en) | 2010-03-08 | 2011-09-08 | Yasutaka Kawahata | Method for estimating a plane in a range image and range image camera |
US20110299761A1 (en) | 2010-06-02 | 2011-12-08 | Myokan Yoshihiro | Image Processing Apparatus, Image Processing Method, and Program |
JP2012196997A (en) | 2011-03-18 | 2012-10-18 | Fuji Heavy Ind Ltd | Vehicle drive support device |
US20120314037A1 (en) * | 2010-02-23 | 2012-12-13 | Ben-Gurion University Of The Negev | System and method for providing 3d imaging |
US8630806B1 (en) * | 2011-10-20 | 2014-01-14 | Google Inc. | Image processing for vehicle control |
US20140225990A1 (en) * | 2013-01-16 | 2014-08-14 | Honda Research Institute Europe Gmbh | Depth sensing method and system for autonomous vehicles |
JP2014228941A (en) | 2013-05-20 | 2014-12-08 | 株式会社日立製作所 | Measurement device for three-dimensional surface shape of ground surface, runnable region detection device and construction machine mounted with the same, and runnable region detection method |
US20150228079A1 (en) * | 2014-02-08 | 2015-08-13 | Honda Motor Co., Ltd. | System and method for generating a depth map through iterative interpolation and warping |
US20160014406A1 (en) * | 2014-07-14 | 2016-01-14 | Sadao Takahashi | Object detection apparatus, object detection method, object detection program, and device control system mountable to moveable apparatus |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CA2822882A1 (en) * | 2011-01-07 | 2012-07-12 | Landmark Graphics Corporation | Systems and methods for the construction of closed bodies during 3d modeling |
JP5022508B1 (en) * | 2011-06-02 | 2012-09-12 | 三菱重工業株式会社 | Shape measurement method |
-
2014
- 2014-12-22 KR KR1020140186267A patent/KR101676656B1/en active IP Right Grant
-
2015
- 2015-12-09 US US14/963,452 patent/US9665782B2/en active Active
- 2015-12-09 DE DE102015121387.7A patent/DE102015121387B4/en active Active
- 2015-12-10 JP JP2015241088A patent/JP6291469B2/en active Active
- 2015-12-10 CN CN201510920243.1A patent/CN105718853B/en active Active
Patent Citations (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20010048763A1 (en) | 2000-05-30 | 2001-12-06 | Takeshi Takatsuka | Integrated vision system |
DE60130517T2 (en) | 2000-05-30 | 2008-06-12 | Fuji Jukogyo K.K. | Integrated vision system |
JP2004094707A (en) | 2002-09-02 | 2004-03-25 | Omron Corp | Method for estimating plane by stereo image and detector for object |
US20040264763A1 (en) * | 2003-04-30 | 2004-12-30 | Deere & Company | System and method for detecting and analyzing features in an agricultural field for vehicle guidance |
WO2005081176A1 (en) | 2004-02-13 | 2005-09-01 | Robert Bosch Gmbh | Method for the classification of an object by means of a stereo camera |
US20090304263A1 (en) | 2004-02-13 | 2009-12-10 | Thomas Engelberg | Method for classifying an object using a stereo camera |
US20060029272A1 (en) * | 2004-08-09 | 2006-02-09 | Fuji Jukogyo Kabushiki Kaisha | Stereo image processing device |
KR20080022748A (en) | 2006-09-07 | 2008-03-12 | 주식회사 만도 | Collision avoidance method using stereo camera |
WO2009028489A1 (en) | 2007-08-30 | 2009-03-05 | Kabushiki Kaisha Yaskawa Denki | Object detecting method, object detecting device, and robot system |
JP2009139323A (en) | 2007-12-10 | 2009-06-25 | Mazda Motor Corp | Travel road surface detecting apparatus for vehicle |
US20120314037A1 (en) * | 2010-02-23 | 2012-12-13 | Ben-Gurion University Of The Negev | System and method for providing 3d imaging |
JP2011186749A (en) | 2010-03-08 | 2011-09-22 | Optex Co Ltd | Method for estimating plane in range image, and range image camera |
US20110216213A1 (en) | 2010-03-08 | 2011-09-08 | Yasutaka Kawahata | Method for estimating a plane in a range image and range image camera |
US20110299761A1 (en) | 2010-06-02 | 2011-12-08 | Myokan Yoshihiro | Image Processing Apparatus, Image Processing Method, and Program |
JP2011253376A (en) | 2010-06-02 | 2011-12-15 | Sony Corp | Image processing device, image processing method and program |
JP2012196997A (en) | 2011-03-18 | 2012-10-18 | Fuji Heavy Ind Ltd | Vehicle drive support device |
US8630806B1 (en) * | 2011-10-20 | 2014-01-14 | Google Inc. | Image processing for vehicle control |
US20140225990A1 (en) * | 2013-01-16 | 2014-08-14 | Honda Research Institute Europe Gmbh | Depth sensing method and system for autonomous vehicles |
JP2014228941A (en) | 2013-05-20 | 2014-12-08 | 株式会社日立製作所 | Measurement device for three-dimensional surface shape of ground surface, runnable region detection device and construction machine mounted with the same, and runnable region detection method |
US20150228079A1 (en) * | 2014-02-08 | 2015-08-13 | Honda Motor Co., Ltd. | System and method for generating a depth map through iterative interpolation and warping |
US20160014406A1 (en) * | 2014-07-14 | 2016-01-14 | Sadao Takahashi | Object detection apparatus, object detection method, object detection program, and device control system mountable to moveable apparatus |
Non-Patent Citations (5)
Title |
---|
German Patent Office, Office Action for German Patent Application No. 10 2015 121 387.7, dated Jan. 12, 2017, Germany. |
Japanese Patent Office, Office Action for Japanese Patent Application No. 2015-241088, dated Dec. 6, 2016, Japan. |
Korean Intellectual Property Office, Notice of Allowance for Korean patent application No. 10-2014-0186267, Sep. 26, 2016, Korea. |
Korean Intellectual Property Office, Office Action for Korean patent application No. 10-2014-0186267, May 16, 2016, Korea. |
Obstacle Detection from Monocular On-Vehicle Camera in units of Delaunay Triangles, SIG Technical Reports, Feb. 22, 2013, Journal of Information Processing, http://ipsj.ixsq.nii.ac.jp/ej/?action=pages. |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180182078A1 (en) * | 2016-12-27 | 2018-06-28 | Kabushiki Kaisha Toshiba | Image processing apparatus and image processing method |
US10726528B2 (en) * | 2016-12-27 | 2020-07-28 | Kabushiki Kaisha Toshiba | Image processing apparatus and image processing method for image picked up by two cameras |
Also Published As
Publication number | Publication date |
---|---|
KR20160076271A (en) | 2016-06-30 |
CN105718853B (en) | 2019-07-23 |
JP2016119085A (en) | 2016-06-30 |
DE102015121387B4 (en) | 2019-07-11 |
KR101676656B1 (en) | 2016-11-16 |
US20160180181A1 (en) | 2016-06-23 |
CN105718853A (en) | 2016-06-29 |
JP6291469B2 (en) | 2018-03-14 |
DE102015121387A1 (en) | 2016-06-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9665782B2 (en) | Obstacle detecting apparatus and obstacle detecting method | |
US11948462B2 (en) | Image generating apparatus, image generating method, and recording medium | |
Pinggera et al. | Lost and found: detecting small road hazards for self-driving vehicles | |
EP2811423B1 (en) | Method and apparatus for detecting target | |
CN108638999B (en) | Anti-collision early warning system and method based on 360-degree look-around input | |
US20230110116A1 (en) | Advanced driver assist system, method of calibrating the same, and method of detecting object in the same | |
US11518390B2 (en) | Road surface detection apparatus, image display apparatus using road surface detection apparatus, obstacle detection apparatus using road surface detection apparatus, road surface detection method, image display method using road surface detection method, and obstacle detection method using road surface detection method | |
CN105335955A (en) | Object detection method and object detection apparatus | |
JP2007527569A (en) | Imminent collision detection based on stereoscopic vision | |
US9367759B2 (en) | Cooperative vision-range sensors shade removal and illumination field correction | |
US20160232705A1 (en) | Method for 3D Scene Reconstruction with Cross-Constrained Line Matching | |
KR101699014B1 (en) | Method for detecting object using stereo camera and apparatus thereof | |
Burlacu et al. | Obstacle detection in stereo sequences using multiple representations of the disparity map | |
Long et al. | A real-time dense stereo matching method for critical environment sensing in autonomous driving | |
Müller et al. | Multi-camera system for traffic light detection: About camera setup and mapping of detections | |
Nedevschi et al. | Driving environment perception using stereovision | |
Itu et al. | An efficient obstacle awareness application for android mobile devices | |
KR102453782B1 (en) | Method of detecting traffic lane for automated driving | |
Carrillo et al. | Fast obstacle detection using sparse edge-based disparity maps | |
JP7064400B2 (en) | Object detection device | |
JP2018146495A (en) | Object detection apparatus, object detection method, object detection program, imaging apparatus, and equipment control system | |
GB2583774A (en) | Stereo image processing | |
Son et al. | Detection of nearby obstacles with monocular vision for earthmoving operations | |
KR102681321B1 (en) | Performance evaluation apparatus for highway driving assist system that calculates distance using dual cameras and method thereof | |
CN114332790A (en) | Binocular vision 3D target detection method, storage medium and terminal equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: HYUNDAI MOBIS CO., LTD., KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SEO, SUNG JIN;KIM, JONG HWAN;CHO, JOON BUM;REEL/FRAME:037251/0875 Effective date: 20151202 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |