WO2021120574A1 - Obstacle positioning method and apparatus for autonomous driving system - Google Patents

Obstacle positioning method and apparatus for autonomous driving system Download PDF

Info

Publication number
WO2021120574A1
WO2021120574A1 PCT/CN2020/098264 CN2020098264W WO2021120574A1 WO 2021120574 A1 WO2021120574 A1 WO 2021120574A1 CN 2020098264 W CN2020098264 W CN 2020098264W WO 2021120574 A1 WO2021120574 A1 WO 2021120574A1
Authority
WO
WIPO (PCT)
Prior art keywords
lane line
image data
obstacle
lane
obtaining
Prior art date
Application number
PCT/CN2020/098264
Other languages
French (fr)
Inventor
An Chen
Weilin Gong
Zhihao Jiang
Dixiao CUI
Original Assignee
Suzhou Zhijia Science & Technologies Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Zhijia Science & Technologies Co., Ltd. filed Critical Suzhou Zhijia Science & Technologies Co., Ltd.
Publication of WO2021120574A1 publication Critical patent/WO2021120574A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30256—Lane; Road marking

Definitions

  • the present disclosure relates to an obstacle positioning method and apparatus, and in particular, to an obstacle positioning method and apparatus for an autonomous driving system.
  • An autonomous driving system generally includes three modules: a perception module, which is equivalent to an human eye and collects a status of a surrounding environment in real time through sensors such as a camera, a millimeter wave radar, and a lidar, etc. ; a determining module, which is equivalent to a human brain and calculates an optimal driving policy and plan according to the environment status; and an execution module, which is equivalent to human hands and feet and is configured to execute a policy and a command to perform corresponding driving operations such as a throttle operation, braking, and steering, etc.
  • a perception module which is equivalent to an human eye and collects a status of a surrounding environment in real time through sensors such as a camera, a millimeter wave radar, and a lidar, etc.
  • a determining module which is equivalent to a human brain and calculates an optimal driving policy and plan according to the environment status
  • an execution module which is equivalent to human hands and feet and is configured to execute a policy and a command to perform corresponding driving operations such as a throttle operation,
  • the perception module is an important module in the autonomous driving system. For example, in order to avoid collision with obstacles in the driving lane, the perception module needs to determine information relating to obstacles which the autonomous vehicle is approaching, such as the distance between the obstacles and the autonomous vehicle.
  • FIG. 1 is a flowchart of an example obstacle positioning method according to the present disclosure.
  • FIG. 2 is a flowchart of sub-blocks of block S2 in FIG. 1.
  • FIG. 3 is a flowchart of sub-blocks of block S3 in FIG. 1.
  • FIG. 4 is a flowchart of sub-blocks of block S4 in FIG. 1.
  • FIG. 5 is a schematic structural diagram of an obstacle positioning apparatus according to the present disclosure.
  • FIG. 6 is a flowchart of another example obstacle positioning method according to the present disclosure
  • Image segmenting module 133
  • BEV refers to bird’s eye view which is an elevated view of an object from above, with a perspective as though the observer were a bird.
  • the perception module of an autonomous driving system may use a binocular device for providing 3D information.
  • the binocular device may provide relatively stable and reliable information in real time to the perception module.
  • the perception module may determine information such as a distance to an obstacle and a speed of an obstacle.
  • Binocular ranging is a visual ranging algorithm in which a difference between positions of a left image and a right image are collected by a binocular device.
  • a disparity map is calculated first and 3D coordinates of a point on the image are then calculated according to the disparity map.
  • One method of calculating a binocular disparity map calculation includes four blocks: cost calculation, cost aggregation, disparity map calculation, and disparity map refinement.
  • cost calculation cost aggregation
  • disparity map calculation disparity map refinement
  • the matching effect provided by this approach is limited and it cannot handle occluded areas.
  • this approach cannot generate a disparity map of a high resolution image, in real time.
  • Attempts to apply neural networks and deep learning algorithms to binocular ranging have not been very successful.
  • One of the difficulties is that, due to the difficulty of obtaining true values for the disparity map, the binocular ranging model lacks training data, and many models are easily overfitted to a specific camera, resulting in relatively weak generalization.
  • deep learning models require a large number of network layers and are complex, with poor real-time performance and so are not suitable for practical application to such time sensitive tasks.
  • One approach for calculating depth information of a target from a disparity map is to take the average depth information of a target rectangular frame as the depth information of the target.
  • a rectangular frame of the obstacle usually includes many background areas as well as the target.
  • an actual obtained target distance is full of noise, and actual calculated obstacle distance information has jitter, and is not smooth.
  • ranging of the target is affected and sometimes it is not even possible to measure distance information.
  • the information about the lane is processed in two manners.
  • a lane to which a vehicle belongs is determined with reference to a map.
  • a lane line and a target distance are separately estimated, and then the two are aligned in 3D space. Since a 3D position of the obstacle is full of noise, a lane line cannot be directly assigned by using 3D coordinates.
  • the second manner the lane to which the obstacle belongs is assigned on the image, requiring good lane line detection information.
  • a relatively stable result usually cannot be obtained as a result of occlusion of the lane line.
  • such methods still face an issue of accurately measuring 3D coordinates of an obstacle.
  • Exemplary embodiments in the disclosure relate to methods and apparatus for positioning the obstacles during autonomous driving and may solve one or more problems described above.
  • a first aspect of the present application provides an obstacle positioning method for an autonomous driving system, comprising the following blocks:
  • S1 preprocessing image data acquired by a binocular camera
  • S3 obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels, and enhancing the plurality of lane lines;
  • S4 obtaining depth information of an obstacle by using lane lines on two sides of the obstacle and the image data, wherein the lane lines on two sides of the obstacle are obtained by searching.
  • a second aspect of the present application provides an obstacle positioning apparatus for an autonomous driving system, comprising:
  • a binocular camera configured to acquire image data
  • a preprocessing unit configured to preprocess the image data collected by the binocular camera
  • a disparity value obtaining unit configured to obtain a disparity value of each lane line pixel according to the preprocessed image data
  • a lane line processing unit configured to perform fitting on the disparity values of the lane line pixels to obtain a plurality of lane lines, and enhance the plurality of lane lines;
  • a depth information obtaining unit configured to obtain depth information of an obstacle by using lane lines on two sides of the obstacle obtained through searching and the image data.
  • a third aspect of the present disclosure provides an obstacle positioning method for an autonomous driving system, comprising:
  • a fourth aspect of the present disclosure provides a computer device for performing the method of the third aspect.
  • the computer device may include a an image acquiring unit for acquiring image data with lane line pixels by a binocular camera; a disparity value calculating unit for calculating a disparity value of each lane line pixel in the image data; a lane line obtaining unit for obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels; a point determining unit for determining a lane line point on the lane lines located at each side of an obstacle based on a recognized position of the obstacle; and a depth determining unit for determining depth information of the obstacle according to depth of the lane line points.
  • a fifth aspect of the present disclosure provides a non-transitory machine readable storage medium storing instructions which are executable by a processor to perform the method of the first or third aspects of the present disclosure.
  • the above approaches make it possible to calculate 3D position information of an obstacle to a reasonable accuracy, while placing relatively low demands on a processor, so that a real-time calculation may be achieved.
  • FIG. 1 is a flowchart of an obstacle positioning method according to the present disclosure.
  • FIG. 2 is a flowchart of sub-blocks of block S2 in FIG. 1.
  • FIG. 3 is a flowchart of sub-blocks of block S3 in FIG. 1.
  • FIG. 4 is a flowchart of sub-blocks of block S4 in FIG. 1.
  • a disparity value of a lane line is calculated, and then fitting is performed on the lane line, and the lane line is enhanced, and finally, lane lines on two sides of an obstacle are searched.
  • the obstacle positioning method of the present disclosure may include the following blocks.
  • Block S1 Preprocess image data acquired by a binocular camera.
  • Block S1 includes: correcting, by using a binocular calibration matrix, first image data or second image data collected by the binocular camera, identifying an obstacle in the first image data or the second image data, and marking the obstacle with a rectangular detection frame.
  • the first image data and the second image data refer to two images captured by the binocular camera at a specific moment, and an obstacle in only one of the first image data and the second image data is identified and marked with a rectangular frame.
  • Block S2 Obtain a disparity value of each lane line pixel according to the preprocessed image data.
  • block S2 may include the following blocks.
  • Block S21 Obtain a perspective projection transformation matrix at a BEV angle of view according to a mounting height and a mounting angle of the binocular camera, the mounting height of the binocular camera being a height of the binocular camera relative to the ground, and the mounting angle being an angle between the binocular camera and a horizontal plane.
  • a perspective projection transformation matrix is
  • Transformation areas (u0, v0) , (u1, v0) , (u2, v1) , and (u3, v1) are selected on the original image, and it needs to be ensured that the selected area is a trapezoid.
  • (u0, v0) is transformed to a position (0, 0) ;
  • (u1, v0) is transformed to a position (width-1, 0) ;
  • (u2, v1) is transformed to a position (0, height-1) ;
  • (u3, v1) is transformed to a position (width-1, height-1) , where width and height are a width and a height of the image respectively, and a projection matrix may be obtained based on this.
  • Block S22 Transform an angle of view of the first image data or the second image data that is marked with a rectangular detection frame into a pitch angle of view by using the perspective projection transformation matrix, where it should be noted that an angle of only one of the first image data or the second image data that is marked with a rectangular detection frame is transformed in the present disclosure.
  • Block S23 Segment, into n segments according to a height of an image at the BEV angle of view, the first image data or the second image data whose angle of view is transformed into the pitch angle of view, where n is a positive integer greater than 1.
  • Block S24 Calculate a depth value of each lane line pixel in the n segments.
  • Block S25 Obtain a disparity range according to a depth value of an (i+1) th segment, search, for a best matching point according to the disparity range, the second image data or the first image data that is marked with no rectangular detection frame, and then obtain the disparity value of each lane line pixel according to a disparity calculation formula, where a value of i is [n-1, 1] .
  • the block S2 may be executed as follows.
  • a perspective projection transformation matrix at a BEV angle of view is calculated according to a mounting height and a mounting angle of the binocular camera, and the first image data and the second image data are transformed into a top angle of view by using the perspective projection transformation matrix.
  • the first image data and the second image data whose angles of view are transformed into the top angle of view are segmented into n segments according to a height of an image at the BEV angle of view, where an n th segment is the bottom of the image, and the first segment is the top of the image.
  • Depth values of all lane line pixels (u, v) in the n th segment are calculated, and the depth value z is calculated as follows:
  • f is a focal length of the camera
  • ⁇ is an angle between the camera and a horizontal direction
  • h is the mounting height of the camera
  • cy is a center row of the image. cy is obtained from an internal parameter matrix of the camera.
  • the internal reference matrix of the camera has the following form:
  • fx and fy represent the focal length of the camera, and (cx, cy) is a center point of the image.
  • a value of i is [n-1, 1] .
  • the disparity range is obtained according to the depth value, but the present disclosure is not limited thereto. In other embodiments, the disparity range may also be obtained according to the disparity value.
  • Block S3 Perform fitting on the disparity values of the lane line pixels to obtain a plurality of lane lines, and enhance the plurality of lane lines.
  • Block S3 may include the following blocks.
  • Block S31 Restore 3D position coordinates of each lane line pixel according to the disparity value of each lane line pixel.
  • Block S32 Obtain a plurality of lane lines through fitting according to the 3D position coordinates of each lane line pixel;
  • Block S33 Extend each lane line to a specified distance according to a lane line enhancement model.
  • Block S34 Determine a left lane line and a right lane line of a current lane, and generate N equidistant parallel 3D lane lines.
  • Block S35 Project, according to a projection matrix, the 3D lane line onto the first image data or the second image data that is marked with a rectangular detection frame, and number each of the 3D lane lines.
  • a plurality of lane lines are obtained through fitting according to the 3D position coordinates of each lane line pixel.
  • a plurality of lane lines may be obtained through fitting according to the following polynomial equation. Assuming that a little set is ⁇ (x, y, z) ⁇ , a polynomial equation for obtaining a line through fitting may can be expressed as:
  • a, b, c, d, e, f, and h are parameters of the polynomial.
  • 3D points in a range of [0, Max] are sampled according to the polynomial equation. Max is a specific distance by which the lane line extends.
  • a left lane line and a right lane line of a current lane are determined, and N equidistant parallel 3D lane lines are generated.
  • the N 3D lane lines are projected onto the first image data according to a projection matrix, and each of the 3D lane lines is numbered.
  • Block S4 Obtain depth information of an obstacle by using lane lines on two sides of the obstacle obtained through searching and the image data.
  • Block S4 may include the following blocks.
  • Block S41 Select a bottom midpoint of a rectangular detection frame of each obstacle, perform searching leftward from the bottom midpoint for the first lane line point, perform searching rightward from the bottom midpoint for the second lane line point, and obtain, according to the numbering of the lane lines to which the first lane line point and the second lane line point belong, a lane on which the obstacle is located; and
  • Block S42 Obtain a 3D coordinate position of the bottom midpoint according to 3D coordinate positions of the first lane line point and the second lane line point, and obtain a final 3D coordinate position of the bottom midpoint by using the binocular calibration matrix.
  • a bottom midpoint of a rectangular detection frame of each obstacle may be selected, searching may be performed leftward from the point for the first lane line point P1, searching may be performed rightward from the point for the second lane line point P2, and a lane on which the obstacle is located may be obtained according to numbers of the lane lines to which P1 and P2 belong.
  • the depth information of the point may be averaged according to depth information of points P1 and P2, and the 3D coordinate position of the point is calculated by using the binocular calibration matrix, which is then used as the 3D coordinate position of the obstacle.
  • a connection point between a vehicle and the ground is selected, depth information of the connection point is calculated, and an accurate 3D coordinate position is obtained.
  • FIG. 5 is a schematic structural diagram of an obstacle positioning apparatus according to the present disclosure.
  • the obstacle positioning apparatus for an autonomous driving system includes: a binocular camera 11, a preprocessing unit 12, a disparity value obtaining unit 13, a lane line processing unit 14, and a depth information obtaining unit 15
  • the binocular camera 11 is configured to provide image data.
  • the preprocessing unit 12 is configured to preprocess the image data collected by the binocular camera 11.
  • the disparity value obtaining unit 13 is configured to obtain a disparity value of each lane line pixel according to the preprocessed image data.
  • the lane line processing unit 14 is configured to perform fitting on the disparity values of the lane line pixels to obtain a plurality of lane lines and enhance the plurality of lane lines.
  • the depth information obtaining unit 15 is configured to obtain depth information of an obstacle by using lane lines on two sides of the obstacle obtained through searching and the image data.
  • the various units may for example be implemented by a computing device comprising a processor.
  • the processor may execute machine readable instructions stored on a non-transitory computer readable storage medium to perform the various functions of the various units.
  • the processor may for example be one or more central processing units, microprocessors, field programmable logic gate arrays, application specific integrated chips or a combination thereof.
  • the preprocessing unit 12 is configured to correct, by using the binocular calibration matrix, first image data or second image data collected by the binocular camera, identify an obstacle in the first image data or the second image data, and mark a rectangular detection frame.
  • the disparity value obtaining unit 13 may include:
  • a perspective projection transformation matrix obtaining module 131 configured to obtain a perspective projection transformation matrix at a BEV angle of view according to a mounting height and a mounting angle of the binocular camera 11;
  • an angle of view transforming module 132 configured to transform an angle of view of the first image data or the second image data that is marked with a rectangular detection frame into a pitch angle of view by using the perspective projection transformation matrix;
  • an image segmenting module 133 configured to segment, into n segments according to a height of an image at the BEV angle of view, the first image data or the second image data whose angle of view is transformed into the pitch angle of view, where n is a positive integer greater than 1;
  • a calculating module 134 configured to calculate a depth value of each lane line pixel in the n segments
  • a disparity value outputting module 135 configured to: obtain a disparity range according to a depth value of an (i+1) th segment, search, for a best matching point according to the disparity range, the second image data or the first image data that is marked with no rectangular detection frame, and then obtain the disparity value of each lane line pixel according to a disparity calculation formula, where a value of i is [n-1, 1] .
  • the lane line processing unit 14 may include:
  • a 3D position coordinate restoring module 141 configured to restore 3D position coordinates of each lane line pixel according to the disparity value of each lane line pixel;
  • a fitting module 142 configured to obtain a plurality of lane lines through fitting according to the 3D position coordinates of each lane line pixel;
  • a lane line extending module 143 configured to extend each lane line to a specified distance according to a lane line enhancement model
  • a lane line numbering module 144 configured to: determine a left lane line and a right lane line of a current lane, generate N equidistant parallel 3D lane lines, project, according to a projection matrix, the 3D lane line onto the first image data or the second image data that is marked with a rectangular detection frame, and number each of the 3D lane lines.
  • the depth information obtaining unit 15 may include:
  • an obstacle lane obtaining module 151 configured to: select a bottom midpoint of a rectangular detection frame of each obstacle, perform searching leftward from the bottom midpoint for the first lane line point, perform searching rightward from the bottom midpoint for the second lane line point, and obtain, according to numbers of the lane lines to which the first lane line point and the second lane line point belong, a lane on which the obstacle is located; and
  • an obstacle position obtaining module 152 configured to obtain a 3D coordinate position of the bottom midpoint according to 3D coordinate positions of the first lane line point and the second lane line point, and obtain a final 3D coordinate position of the bottom midpoint by using the binocular calibration matrix.
  • Fig. 6 depicts an obstacle positioning method for an autonomous driving system according to an exemplary embodiment.
  • image data is acquired with lane line pixels by a binocular camera.
  • the autonomous driving system may be equipped with a binocular camera for acquiring images when the autonomous driving vehicle is moving.
  • a left image and a right image may be captured simultaneously at one time.
  • the images contain the information of the driving environment of the vehicle such as the lane lines, other vehicles or obstacles.
  • the image data include pixels of the lane line.
  • a disparity value is calculated for each lane line pixel in the image data.
  • the disparity value of each lane line pixel may be calculated according to the method described in block S21-S25.
  • a plurality of lane lines are obtained by performing fitting on the disparity values of the lane line pixels.
  • the lane lines may be calculated according to the method described in blocks S31-S35.
  • Block 604 includes determining a lane line point on the lane lines located at each side of an obstacle based on a recognized position of the obstacle.
  • the obstacles in the left image or the right image may be recognized and marked with a detection frame.
  • the detection frame may be a rectangular frame.
  • the obstacle is located in a lane between a left lane line and a right lane line.
  • the left lane line point and the right lane line point may be determined according to the method described in S41.
  • Block 605 includes determining depth information of the obstacle according to depth of the lane line points.
  • the bottom midpoint of the rectangular detection frame of each obstacle may be determined.
  • the left lane line point P1 may be obtained by searching leftward from the bottom midpoint, and the right lane line point P2 may be obtained by searching rightward from the bottom midpoint.
  • the depth information of the bottom midpoint may be calculated according to an average value of the depths of P1 and P2.
  • the 3D coordinate of the bottom midpoint point may be calculated by applying the binocular calibration matrix, which is taken as the 3D coordinate of the obstacle.
  • obstacle positioning method and apparatus of the present disclosure detection of the lane line and detection of a target are combined, and the lane line information is used to assist vehicle positioning, so that a vehicle can be accurately positioned, reducing an error of vehicle ranging.
  • processing latency is low, real-time processing becomes possible.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Electromagnetism (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

An obstacle positioning method for an autonomous driving system, comprising the following blocks: preprocessing image data collected by a binocular camera (11) (S1); obtaining a disparity value of each lane line pixel according to the preprocessed image data (S2); obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels, and enhancing the plurality of lane lines (S3); obtaining depth information of an obstacle by using lane lines on two sides of the obstacle obtained by searching and the image data (S4).

Description

OBSTACLE POSITIONING METHOD AND APPARATUS FOR AUTONOMOUS DRIVING SYSTEM TECHNICAL FIELD
The present disclosure relates to an obstacle positioning method and apparatus, and in particular, to an obstacle positioning method and apparatus for an autonomous driving system.
BACKGROUND
In recent years, with the development of science and technology, especially the rapid development of intelligent computing, research of autonomous driving technologies becomes the focus of attention in various industries. In McKinsey's report "12 Frontier Technologies that Determine the Future Economy" , impact of 12 frontier technologies on the future economy and society is described, and economic impact and social impact of each of the 12 technologies in 2025 are analyzed and estimated. The autonomous driving technology ranks sixth, and impact thereof in 2025 is estimated to be: economic benefits about 0.2-1.9 trillion US dollars per year, and social benefits may be used to save 30,000 to 150,000 lives every year.
An autonomous driving system generally includes three modules: a perception module, which is equivalent to an human eye and collects a status of a surrounding environment in real time through sensors such as a camera, a millimeter wave radar, and a lidar, etc. ; a determining module, which is equivalent to a human brain and calculates an optimal driving policy and plan according to the environment status; and an execution module, which is equivalent to human hands and feet and is configured to execute a policy and a command to perform corresponding driving operations such as a throttle operation, braking, and steering, etc.
The perception module is an important module in the autonomous driving system. For example, in order to avoid collision with obstacles in the driving lane, the perception module needs to determine information relating to obstacles which the autonomous vehicle is approaching, such as the distance between the obstacles and the autonomous vehicle.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a flowchart of an example obstacle positioning method according to the present disclosure.
FIG. 2 is a flowchart of sub-blocks of block S2 in FIG. 1.
FIG. 3 is a flowchart of sub-blocks of block S3 in FIG. 1.
FIG. 4 is a flowchart of sub-blocks of block S4 in FIG. 1.
FIG. 5 is a schematic structural diagram of an obstacle positioning apparatus according to the present disclosure.
FIG. 6 is a flowchart of another example obstacle positioning method according to the present disclosure
Reference numerals used in the drawings include:
Binocular camera 11
Preprocessing unit 12
Disparity value obtaining unit 13
Perspective projection transformation matrix obtaining module 131
Angle of view transforming module 132
Image segmenting module 133
Calculating module 134
Disparity value outputting module 135
Lane line processing unit 14
3D position coordinate restoring module 141
Fitting module 142
Lane line extending module 143
Lane line numbering module 144
Depth information obtaining unit 15
Obstacle lane obtaining module 151
Obstacle position obtaining module 152
DETAILED DESCRIPTION
The detailed content and technical description of the present disclosure are further described by using a preferred embodiment, but this should not be construed as limiting the implementation of the present disclosure.
As used herein and in the claims, “BEV” refers to bird’s eye view which is an elevated view of an object from above, with a perspective as though the observer were a bird.
The perception module of an autonomous driving system, may use a binocular device for providing 3D information. For example, the binocular device may provide relatively stable and reliable information in real time to the perception module. Based on the information from the binocular device, the perception module may determine information such as a distance to an obstacle and a speed of an obstacle.
Binocular ranging is a visual ranging algorithm in which a difference between positions of a left image and a right image are collected by a binocular device. In a traditional binocular ranging method, a disparity map is calculated first and 3D coordinates of a point on the image are then calculated according to the disparity map.
One method of calculating a binocular disparity map calculation includes four blocks: cost calculation, cost aggregation, disparity map calculation, and disparity map refinement. However, the matching effect provided by this approach is limited and it cannot handle occluded areas. Furthermore, because processing latency increases rapidly with image resolution, this approach cannot generate a disparity map of a high resolution image, in real time. Attempts to apply neural networks and deep learning algorithms to binocular ranging have not been very successful. One of the difficulties is that, due to the difficulty of obtaining true values for the disparity map, the binocular ranging model lacks training data, and many models are easily overfitted to a specific camera, resulting in relatively weak generalization. Furthermore, deep learning models require a large number of network layers and are complex, with poor real-time performance and so are not suitable for practical application to such time sensitive tasks.
One approach for calculating depth information of a target from a disparity map is to take the average depth information of a target rectangular frame as the depth information of the target. However, in practice, a rectangular frame of the obstacle usually includes many background areas as well as the target. As a result, an actual obtained target distance is full of noise, and actual calculated obstacle distance information has jitter, and is not smooth. As a result ranging of the target is affected and sometimes it is not even possible to measure distance information.
After a depth of the target is obtained, information about a lane on which the target is located is further required. Currently, the information about the lane is processed in two manners. In the first manner, after an obstacle position and lane line information are separately detected, a lane to which a vehicle belongs is determined with reference to a map. In this manner, a lane line and a target distance are separately estimated, and then the two are aligned in 3D space. Since a 3D position of the obstacle is full of noise, a lane line cannot be directly assigned by using 3D coordinates. In the second manner, the lane to which the obstacle belongs is assigned on the image, requiring good lane line detection information. However, a relatively stable result usually cannot be obtained as a result of occlusion of the lane line. In addition, such methods still face an issue of accurately measuring 3D coordinates of an obstacle.
Exemplary embodiments in the disclosure relate to methods and apparatus for positioning the obstacles during autonomous driving and may solve one or more problems described above.
Accordingly, a first aspect of the present application provides an obstacle positioning method for an autonomous driving system, comprising the following blocks:
S1: preprocessing image data acquired by a binocular camera;
S2: obtaining a disparity value of each lane line pixel according to the preprocessed image data;
S3: obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels, and enhancing the plurality of lane lines; and
S4: obtaining depth information of an obstacle by using lane lines on two sides of the obstacle and the image data, wherein the lane lines on two sides of the obstacle are obtained  by searching.
A second aspect of the present application provides an obstacle positioning apparatus for an autonomous driving system, comprising:
a binocular camera configured to acquire image data;
a preprocessing unit configured to preprocess the image data collected by the binocular camera;
a disparity value obtaining unit configured to obtain a disparity value of each lane line pixel according to the preprocessed image data;
a lane line processing unit configured to perform fitting on the disparity values of the lane line pixels to obtain a plurality of lane lines, and enhance the plurality of lane lines; and
a depth information obtaining unit configured to obtain depth information of an obstacle by using lane lines on two sides of the obstacle obtained through searching and the image data.
A third aspect of the present disclosure provides an obstacle positioning method for an autonomous driving system, comprising:
acquiring image data with lane line pixels by a binocular camera;
calculating a disparity value of each lane line pixel in the image data;
obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels;
determining a lane line point on the lane lines located at each side of an obstacle based on a recognized position of the obstacle;
determining depth information of the obstacle according to depth of the lane line points.
A fourth aspect of the present disclosure provides a computer device for performing the method of the third aspect. For example, the computer device may include a an image acquiring unit for acquiring image data with lane line pixels by a binocular camera; a disparity value calculating unit for calculating a disparity value of each lane line pixel in the image data; a lane line obtaining unit for obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels; a point determining unit  for determining a lane line point on the lane lines located at each side of an obstacle based on a recognized position of the obstacle; and a depth determining unit for determining depth information of the obstacle according to depth of the lane line points.
A fifth aspect of the present disclosure provides a non-transitory machine readable storage medium storing instructions which are executable by a processor to perform the method of the first or third aspects of the present disclosure.
By using the lane line information, the above approaches make it possible to calculate 3D position information of an obstacle to a reasonable accuracy, while placing relatively low demands on a processor, so that a real-time calculation may be achieved.
Further features of the above aspects are described in the dependent claims. Unless otherwise indicated, the features of one aspect including those of any dependent claims may be combined with and applied to any of the other aspects.
Embodiments of the present disclosure will now be described for illustrative purposes with reference to the accompanying drawings.
Referring to FIG. 1 to FIG. 4, FIG. 1 is a flowchart of an obstacle positioning method according to the present disclosure. FIG. 2 is a flowchart of sub-blocks of block S2 in FIG. 1. FIG. 3 is a flowchart of sub-blocks of block S3 in FIG. 1. FIG. 4 is a flowchart of sub-blocks of block S4 in FIG. 1. As shown in FIG. 1 to FIG. 4, in the obstacle positioning method of the present disclosure, a disparity value of a lane line is calculated, and then fitting is performed on the lane line, and the lane line is enhanced, and finally, lane lines on two sides of an obstacle are searched. When a vehicle travels on the road, depth information of the vehicle can be accurately determined. Specifically, the obstacle positioning method of the present disclosure may include the following blocks.
Block S1: Preprocess image data acquired by a binocular camera. Block S1 includes: correcting, by using a binocular calibration matrix, first image data or second image data collected by the binocular camera, identifying an obstacle in the first image data or the second image data, and marking the obstacle with a rectangular detection frame.
It should be noted that in this embodiment, the first image data and the second image data refer to two images captured by the binocular camera at a specific moment, and an obstacle in only one of the first image data and the second image data is identified and  marked with a rectangular frame.
Block S2: Obtain a disparity value of each lane line pixel according to the preprocessed image data.
In one example a 3D lane line is calculated using a segmented plane assumption method. For example, block S2 may include the following blocks.
Block S21: Obtain a perspective projection transformation matrix at a BEV angle of view according to a mounting height and a mounting angle of the binocular camera, the mounting height of the binocular camera being a height of the binocular camera relative to the ground, and the mounting angle being an angle between the binocular camera and a horizontal plane.
In one example, a perspective projection transformation matrix is
Figure PCTCN2020098264-appb-000001
atransformation formula is (u', v', w') = (u, v, 1) M, (u, v) is a pixel on an original image, and a transformed position is (x, y) . x=u'/w', y=v'/w') , x= (a11u+a21v+a31) / (a13u+a23v+a33) , and y= (a12u+a22v+a32) / (a13u+a23v +a33) . Transformation areas (u0, v0) , (u1, v0) , (u2, v1) , and (u3, v1) are selected on the original image, and it needs to be ensured that the selected area is a trapezoid.
(u0, v0) is transformed to a position (0, 0) ; (u1, v0) is transformed to a position (width-1, 0) ; (u2, v1) is transformed to a position (0, height-1) ; and (u3, v1) is transformed to a position (width-1, height-1) , where width and height are a width and a height of the image respectively, and a projection matrix may be obtained based on this.
Block S22: Transform an angle of view of the first image data or the second image data that is marked with a rectangular detection frame into a pitch angle of view by using the perspective projection transformation matrix, where it should be noted that an angle of only one of the first image data or the second image data that is marked with a rectangular detection frame is transformed in the present disclosure.
Block S23: Segment, into n segments according to a height of an image at the BEV angle of view, the first image data or the second image data whose angle of view is transformed into the pitch angle of view, where n is a positive integer greater than 1.
Block S24: Calculate a depth value of each lane line pixel in the n segments.
Block S25: Obtain a disparity range according to a depth value of an (i+1)  th segment, search, for a best matching point according to the disparity range, the second image data or the first image data that is marked with no rectangular detection frame, and then obtain the disparity value of each lane line pixel according to a disparity calculation formula, where a value of i is [n-1, 1] .
In an exemplary embodiment, the block S2 may be executed as follows.
A perspective projection transformation matrix at a BEV angle of view is calculated according to a mounting height and a mounting angle of the binocular camera, and the first image data and the second image data are transformed into a top angle of view by using the perspective projection transformation matrix.
The first image data and the second image data whose angles of view are transformed into the top angle of view are segmented into n segments according to a height of an image at the BEV angle of view, where an n th segment is the bottom of the image, and the first segment is the top of the image.
Depth values of all lane line pixels (u, v) in the n th segment are calculated, and the depth value z is calculated as follows:
z ( (v-cy) ×cos (θ) /f+sin (θ) ) =h.
f is a focal length of the camera, θ is an angle between the camera and a horizontal direction, h is the mounting height of the camera, and cy is a center row of the image. cy is obtained from an internal parameter matrix of the camera. The internal reference matrix of the camera has the following form:
Figure PCTCN2020098264-appb-000002
fx and fy represent the focal length of the camera, and (cx, cy) is a center point of the image.
For an i th segment, a value of i is [n-1, 1] . A maximum disparity value and a minimum disparity value (that is, a disparity range) are respectively calculated according to a depth value of an (i+1)  th segment. Specifically, assuming that a minimum depth value of the (i+1)  th segment is known, a maximum disparity value of the i th segment is d_max= (fB) /depth. B is an optical center distance between left and right cameras, f is a  focal length of the camera (having a value same as fx and fy) , and a minimum disparity value of the i th segment is d_min=0. Then a best matching point is searched for from data within the disparity range, and then the disparity value is obtained according to the disparity calculation formula. Specifically, assuming that a pixel point on the left picture is (u0, v0) , and a corresponding pixel point on the right picture is (u1, v1) , u0 and u1 representing columns in the image, the disparity value=u0-u1.
It should be noted that, in this embodiment, the disparity range is obtained according to the depth value, but the present disclosure is not limited thereto. In other embodiments, the disparity range may also be obtained according to the disparity value.
Block S3: Perform fitting on the disparity values of the lane line pixels to obtain a plurality of lane lines, and enhance the plurality of lane lines.
Block S3 may include the following blocks.
Block S31: Restore 3D position coordinates of each lane line pixel according to the disparity value of each lane line pixel.
Block S32: Obtain a plurality of lane lines through fitting according to the 3D position coordinates of each lane line pixel;
Block S33: Extend each lane line to a specified distance according to a lane line enhancement model.
Block S34: Determine a left lane line and a right lane line of a current lane, and generate N equidistant parallel 3D lane lines.
Block S35: Project, according to a projection matrix, the 3D lane line onto the first image data or the second image data that is marked with a rectangular detection frame, and number each of the 3D lane lines.
Specifically, the 3D position of the lane is restored according to the disparity value of each lane line pixel. A plurality of lane lines are obtained through fitting according to the 3D position coordinates of each lane line pixel. Specifically, in this embodiment, a plurality of lane lines may be obtained through fitting according to the following polynomial equation. Assuming that a little set is { (x, y, z) } , a polynomial equation for obtaining a line through fitting may can be expressed as:
F (x, y, z) =ax^2+bx+cy^2+dy+ez^2+fz+h=0. a, b, c, d, e, f, and h are parameters of the  polynomial. 3D points in a range of [0, Max] are sampled according to the polynomial equation. Max is a specific distance by which the lane line extends. A left lane line and a right lane line of a current lane are determined, and N equidistant parallel 3D lane lines are generated. The N 3D lane lines are projected onto the first image data according to a projection matrix, and each of the 3D lane lines is numbered.
Block S4: Obtain depth information of an obstacle by using lane lines on two sides of the obstacle obtained through searching and the image data.
Block S4 may include the following blocks.
Block S41: Select a bottom midpoint of a rectangular detection frame of each obstacle, perform searching leftward from the bottom midpoint for the first lane line point, perform searching rightward from the bottom midpoint for the second lane line point, and obtain, according to the numbering of the lane lines to which the first lane line point and the second lane line point belong, a lane on which the obstacle is located; and
Block S42: Obtain a 3D coordinate position of the bottom midpoint according to 3D coordinate positions of the first lane line point and the second lane line point, and obtain a final 3D coordinate position of the bottom midpoint by using the binocular calibration matrix.
For example, a bottom midpoint of a rectangular detection frame of each obstacle may be selected, searching may be performed leftward from the point for the first lane line point P1, searching may be performed rightward from the point for the second lane line point P2, and a lane on which the obstacle is located may be obtained according to numbers of the lane lines to which P1 and P2 belong. The depth information of the point may be averaged according to depth information of points P1 and P2, and the 3D coordinate position of the point is calculated by using the binocular calibration matrix, which is then used as the 3D coordinate position of the obstacle. According to the method for measuring a target distance of an obstacle of the present disclosure, a connection point between a vehicle and the ground is selected, depth information of the connection point is calculated, and an accurate 3D coordinate position is obtained.
Referring to FIG. 5, FIG. 5 is a schematic structural diagram of an obstacle positioning apparatus according to the present disclosure. As shown in FIG. 5, the obstacle positioning  apparatus for an autonomous driving system according to the present disclosure includes: a binocular camera 11, a preprocessing unit 12, a disparity value obtaining unit 13, a lane line processing unit 14, and a depth information obtaining unit 15 The binocular camera 11 is configured to provide image data. The preprocessing unit 12 is configured to preprocess the image data collected by the binocular camera 11. The disparity value obtaining unit 13 is configured to obtain a disparity value of each lane line pixel according to the preprocessed image data. The lane line processing unit 14 is configured to perform fitting on the disparity values of the lane line pixels to obtain a plurality of lane lines and enhance the plurality of lane lines. The depth information obtaining unit 15 is configured to obtain depth information of an obstacle by using lane lines on two sides of the obstacle obtained through searching and the image data. The various units may for example be implemented by a computing device comprising a processor. In some examples the processor may execute machine readable instructions stored on a non-transitory computer readable storage medium to perform the various functions of the various units. The processor may for example be one or more central processing units, microprocessors, field programmable logic gate arrays, application specific integrated chips or a combination thereof.
The preprocessing unit 12 is configured to correct, by using the binocular calibration matrix, first image data or second image data collected by the binocular camera, identify an obstacle in the first image data or the second image data, and mark a rectangular detection frame.
Further, the disparity value obtaining unit 13 may include:
a perspective projection transformation matrix obtaining module 131 configured to obtain a perspective projection transformation matrix at a BEV angle of view according to a mounting height and a mounting angle of the binocular camera 11;
an angle of view transforming module 132 configured to transform an angle of view of the first image data or the second image data that is marked with a rectangular detection frame into a pitch angle of view by using the perspective projection transformation matrix;
an image segmenting module 133 configured to segment, into n segments according to a height of an image at the BEV angle of view, the first image data or the second image data whose angle of view is transformed into the pitch angle of view, where n is a positive  integer greater than 1;
a calculating module 134 configured to calculate a depth value of each lane line pixel in the n segments; and
a disparity value outputting module 135 configured to: obtain a disparity range according to a depth value of an (i+1)  th segment, search, for a best matching point according to the disparity range, the second image data or the first image data that is marked with no rectangular detection frame, and then obtain the disparity value of each lane line pixel according to a disparity calculation formula, where a value of i is [n-1, 1] .
Further, the lane line processing unit 14 may include:
a 3D position coordinate restoring module 141 configured to restore 3D position coordinates of each lane line pixel according to the disparity value of each lane line pixel;
a fitting module 142 configured to obtain a plurality of lane lines through fitting according to the 3D position coordinates of each lane line pixel;
a lane line extending module 143 configured to extend each lane line to a specified distance according to a lane line enhancement model; and
a lane line numbering module 144 configured to: determine a left lane line and a right lane line of a current lane, generate N equidistant parallel 3D lane lines, project, according to a projection matrix, the 3D lane line onto the first image data or the second image data that is marked with a rectangular detection frame, and number each of the 3D lane lines.
Further, the depth information obtaining unit 15 may include:
an obstacle lane obtaining module 151 configured to: select a bottom midpoint of a rectangular detection frame of each obstacle, perform searching leftward from the bottom midpoint for the first lane line point, perform searching rightward from the bottom midpoint for the second lane line point, and obtain, according to numbers of the lane lines to which the first lane line point and the second lane line point belong, a lane on which the obstacle is located; and
an obstacle position obtaining module 152 configured to obtain a 3D coordinate position of the bottom midpoint according to 3D coordinate positions of the first lane line point and the second lane line point, and obtain a final 3D coordinate position of the bottom midpoint by using the binocular calibration matrix.
Fig. 6 depicts an obstacle positioning method for an autonomous driving system according to an exemplary embodiment.
At block 601 image data is acquired with lane line pixels by a binocular camera.
By way of example, the autonomous driving system may be equipped with a binocular camera for acquiring images when the autonomous driving vehicle is moving. A left image and a right image may be captured simultaneously at one time. The images contain the information of the driving environment of the vehicle such as the lane lines, other vehicles or obstacles. The image data include pixels of the lane line.
At block 602 a disparity value is calculated for each lane line pixel in the image data.
By way of example, the disparity value of each lane line pixel may be calculated according to the method described in block S21-S25.
At block 603, a plurality of lane lines are obtained by performing fitting on the disparity values of the lane line pixels.
By way of example, the lane lines may be calculated according to the method described in blocks S31-S35.
Block 604 includes determining a lane line point on the lane lines located at each side of an obstacle based on a recognized position of the obstacle.
By way of example, the obstacles in the left image or the right image may be recognized and marked with a detection frame. The detection frame may be a rectangular frame. The obstacle is located in a lane between a left lane line and a right lane line. The left lane line point and the right lane line point may be determined according to the method described in S41.
Block 605 includes determining depth information of the obstacle according to depth of the lane line points.
By way of example, the bottom midpoint of the rectangular detection frame of each obstacle may be determined. The left lane line point P1 may be obtained by searching leftward from the bottom midpoint, and the right lane line point P2 may be obtained by searching rightward from the bottom midpoint. The depth information of the bottom midpoint may be calculated according to an average value of the depths of P1 and P2. The 3D coordinate of the bottom midpoint point may be calculated by applying the binocular  calibration matrix, which is taken as the 3D coordinate of the obstacle.
In summary, according to obstacle positioning method and apparatus of the present disclosure, detection of the lane line and detection of a target are combined, and the lane line information is used to assist vehicle positioning, so that a vehicle can be accurately positioned, reducing an error of vehicle ranging. In addition, as processing latency is low, real-time processing becomes possible.
The foregoing descriptions are merely preferred embodiments of the present disclosure, and are not intended to limit the scope of implementation of the present disclosure. A person skilled in the art may make various corresponding changes and variations according to the present disclosure without departing from the spirit and essence of the present disclosure. However, such corresponding changes and variations shall fall within the protection scope of the claims appended to the present disclosure.

Claims (12)

  1. An obstacle positioning method for an autonomous driving system, comprising the following blocks:
    S1: preprocessing image data acquired by a binocular camera;
    S2: obtaining a disparity value of each lane line pixel according to the preprocessed image data;
    S3: obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels, and enhancing the plurality of lane lines; and
    S4: obtaining depth information of an obstacle by using lane lines on two sides of the obstacle and the image data, wherein the lane lines on two sides of the obstacle are obtained by searching.
  2. The obstacle positioning method according to claim 1, wherein block S1 comprises: correcting, by using a binocular calibration matrix, first image data or second image data collected by the binocular camera, identifying an obstacle in the first image data or the second image data, and marking the obstacle with a rectangular detection frame.
  3. The obstacle positioning method according to claim 2, wherein block S2 comprises:
    S21: obtaining a perspective projection transformation matrix at a BEV angle of view according to a mounting height and a mounting angle of the binocular camera;
    S22: transforming angles of view of the first image data and the second image data into a pitch angle of view by using the perspective projection transformation matrix;
    S23: segmenting, into n segments according to a height of an image at the BEV angle of view, the first image data and the second image data whose angles of view are transformed into the pitch angle of view, wherein n is a positive integer greater than 1;
    S24: calculating a depth value of each lane line pixel in the n segments; and
    S25: obtaining a disparity range according to a depth value of an (i+1)  th segment, searching for a best matching point according to the disparity range in the second image  data or the first image data that is not marked with rectangular detection frame, and obtaining the disparity value of each lane line pixel according to a disparity calculation formula, wherein a value of i is [n-1, 1] .
  4. The obstacle positioning method according to claim 3, wherein block S3 comprises:
    S31: restoring 3D position coordinates of each lane line pixel according to the disparity value of each lane line pixel;
    S32: obtaining a plurality of lane lines by fitting the 3D position coordinates of each lane line pixel;
    S33: extending each lane line to a specified distance according to a lane line enhancement model;
    S34: determining a left lane line and a right lane line of a current lane, and generating N equidistant parallel 3D lane lines; and
    S35: projecting, according to a projection matrix, the 3D lane line onto the first image data or the second image data that is marked with a rectangular detection frame, and numbering each of the 3D lane lines.
  5. The obstacle positioning method according to claim 4, wherein block S4 comprises:
    S41: selecting a bottom midpoint of a rectangular detection frame of each obstacle, performing searching leftward from the bottom midpoint for the first lane line point, performing searching rightward from the bottom midpoint for the second lane line point, and obtaining, according to numberings of the lane lines to which the first lane line point and the second lane line point belong, a lane on which the obstacle is located; and
    S42: obtaining a 3D coordinate position of the bottom midpoint according to 3D coordinate positions of the first lane line point and the second lane line point, and obtaining a final 3D coordinate position of the bottom midpoint by using the binocular calibration matrix.
  6. An obstacle positioning apparatus for an autonomous driving system, comprising:
    a binocular camera configured to acquire image data;
    a preprocessing unit configured to preprocess the image data collected by the binocular camera;
    a disparity value obtaining unit configured to obtain a disparity value of each lane line pixel according to the preprocessed image data;
    a lane line processing unit configured to perform fitting on the disparity values of the lane line pixels to obtain a plurality of lane lines, and enhance the plurality of lane lines; and
    a depth information obtaining unit configured to obtain depth information of an obstacle by using lane lines on two sides of the obstacle obtained through searching and the image data.
  7. The obstacle positioning apparatus according to claim 6, wherein the preprocessing unit is configured to correct, by using the binocular calibration matrix, first image data or second image data collected by the binocular camera, identify an obstacle in the first image data or the second image data, and mark the obstacle with a rectangular detection frame.
  8. The obstacle positioning apparatus according to claim 7, wherein the disparity value obtaining unit comprises:
    a perspective projection transformation matrix obtaining module configured to obtain a perspective projection transformation matrix at a BEV angle of view according to a mounting height and a mounting angle of the binocular camera;
    an angle of view transforming module configured to transform angles of view of the first image data and the second image data into a pitch angle of view by using the perspective projection transformation matrix;
    an image segmenting module configured to segment, into n segments according to a height of an image at the BEV angle of view, the first image data and the second image data whose angles of view are transformed into the pitch angle of view, wherein n is a positive integer greater than 1;
    a calculating module configured to calculate a depth value of each lane line pixel in the n segments; and
    a disparity value outputting module configured to: obtain a disparity range according to a depth value of an (i+1)  th segment, search for a best matching point according to the disparity range in the second image data or the first image data that is not marked with rectangular detection frame, and obtain the disparity value of each lane line pixel according to a disparity calculation formula, wherein a value of i is [n-1, 1] .
  9. The obstacle positioning apparatus according to claim 8, wherein the lane line processing unit comprises:
    a 3D position coordinate restoring module configured to restore 3D position coordinates of each lane line pixel according to the disparity value of each lane line pixel;
    a fitting module configured to obtain a plurality of lane lines by fitting the 3D position coordinates of each lane line pixel;
    a lane line extending module configured to extend each lane line to a specified distance according to a lane line enhancement model; and
    a lane line numbering module configured to: determine a left lane line and a right lane line of a current lane, generate N equidistant parallel 3D lane lines, project, according to a projection matrix, the 3D lane line onto the first image data or the second image data that is marked with a rectangular detection frame, and number each of the 3D lane lines.
  10. The obstacle positioning apparatus according to claim 9, wherein the depth information obtaining unit comprises:
    an obstacle lane obtaining module configured to: select a bottom midpoint of a rectangular detection frame of each obstacle, perform searching leftward from the bottom midpoint for the first lane line point, perform searching rightward from the bottom midpoint for the second lane line point, and obtain, according to numberings of the lane lines to which the first lane line point and the second lane line point belong, a lane on which the obstacle is located; and
    an obstacle position obtaining module configured to obtain a 3D coordinate position of the bottom midpoint according to 3D coordinate positions of the first lane line point and the second lane line point, and obtain a final 3D coordinate position of the bottom midpoint by using the binocular calibration matrix.
  11. An obstacle positioning method for an autonomous driving system, comprising:
    acquiring image data with lane line pixels by a binocular camera;
    calculating a disparity value of each lane line pixel in the image data;
    obtaining a plurality of lane lines by performing fitting on the disparity values of the lane line pixels;
    determining a lane line point on the lane lines located at each side of an obstacle based on a recognized position of the obstacle;
    determining depth information of the obstacle according to depth of the lane line points.
  12. A non-transitory computer readable storage medium storing instructions which are executable by a processor to perform the method of any one of claims 1 to 5 or 11.
PCT/CN2020/098264 2019-12-19 2020-06-24 Obstacle positioning method and apparatus for autonomous driving system WO2021120574A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911317016.4A CN111443704B (en) 2019-12-19 2019-12-19 Obstacle positioning method and device for automatic driving system
CN201911317016.4 2019-12-19

Publications (1)

Publication Number Publication Date
WO2021120574A1 true WO2021120574A1 (en) 2021-06-24

Family

ID=71650638

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/098264 WO2021120574A1 (en) 2019-12-19 2020-06-24 Obstacle positioning method and apparatus for autonomous driving system

Country Status (2)

Country Link
CN (1) CN111443704B (en)
WO (1) WO2021120574A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113674275A (en) * 2021-10-21 2021-11-19 北京中科慧眼科技有限公司 Dense disparity map-based road surface unevenness detection method and system and intelligent terminal
CN115115687A (en) * 2022-06-24 2022-09-27 合众新能源汽车有限公司 Lane line measuring method and device

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113537100B (en) * 2021-07-22 2022-03-22 元橡科技(苏州)有限公司 Vehicle auxiliary driving system, lane line space-time fitting system and method, and FPGA chip
CN113848931B (en) * 2021-10-09 2022-09-27 上海联适导航技术股份有限公司 Agricultural machinery automatic driving obstacle recognition method, system, equipment and storage medium
CN114723640B (en) * 2022-05-23 2022-09-27 禾多科技（北京）有限公司 Obstacle information generation method and device, electronic equipment and computer readable medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2960858A1 (en) * 2014-06-25 2015-12-30 Honda Research Institute Europe GmbH Sensor system for determining distance information based on stereoscopic images
CN106228110A (en) * 2016-07-07 2016-12-14 浙江零跑科技有限公司 A kind of barrier based on vehicle-mounted binocular camera and drivable region detection method
CN106240458A (en) * 2016-07-22 2016-12-21 浙江零跑科技有限公司 A kind of vehicular frontal impact method for early warning based on vehicle-mounted binocular camera
CN107392103A (en) * 2017-06-21 2017-11-24 海信集团有限公司 The detection method and device of road surface lane line, electronic equipment
CN108229406A (en) * 2018-01-11 2018-06-29 海信集团有限公司 A kind of method for detecting lane lines, device and terminal
US20180322646A1 (en) * 2016-01-05 2018-11-08 California Institute Of Technology Gaussian mixture models for temporal depth fusion
CN109269478A (en) * 2018-10-24 2019-01-25 南京大学 A kind of container terminal based on binocular vision bridge obstacle detection method

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103593671B (en) * 2013-11-25 2017-03-01 中国航天科工集团第三研究院第八三五七研究所 The wide-range lane line visible detection method worked in coordination with based on three video cameras
KR101716973B1 (en) * 2015-11-23 2017-03-15 (주)베라시스 Apparatus and method for detecting vehicle on road using stereo image
CN107590438A (en) * 2017-08-16 2018-01-16 中国地质大学（武汉） A kind of intelligent auxiliary driving method and system
CN108470469A (en) * 2018-03-12 2018-08-31 海信集团有限公司 Road barrier method for early warning, device and terminal
CN109143215A (en) * 2018-08-28 2019-01-04 重庆邮电大学 It is a kind of that source of early warning and method are cooperateed with what V2X was communicated based on binocular vision
CN113486796B (en) * 2018-09-07 2023-09-05 百度在线网络技术（北京）有限公司 Unmanned vehicle position detection method, unmanned vehicle position detection device, unmanned vehicle position detection equipment, storage medium and vehicle
CN109740465B (en) * 2018-12-24 2022-09-27 南京理工大学 Lane line detection algorithm based on example segmentation neural network framework
CN110333714B (en) * 2019-04-09 2022-06-10 武汉理工大学 Unmanned vehicle path planning method and device
CN110569704B (en) * 2019-05-11 2022-11-22 北京工业大学 Multi-strategy self-adaptive lane line detection method based on stereoscopic vision

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2960858A1 (en) * 2014-06-25 2015-12-30 Honda Research Institute Europe GmbH Sensor system for determining distance information based on stereoscopic images
US20180322646A1 (en) * 2016-01-05 2018-11-08 California Institute Of Technology Gaussian mixture models for temporal depth fusion
CN106228110A (en) * 2016-07-07 2016-12-14 浙江零跑科技有限公司 A kind of barrier based on vehicle-mounted binocular camera and drivable region detection method
CN106240458A (en) * 2016-07-22 2016-12-21 浙江零跑科技有限公司 A kind of vehicular frontal impact method for early warning based on vehicle-mounted binocular camera
CN107392103A (en) * 2017-06-21 2017-11-24 海信集团有限公司 The detection method and device of road surface lane line, electronic equipment
CN108229406A (en) * 2018-01-11 2018-06-29 海信集团有限公司 A kind of method for detecting lane lines, device and terminal
CN109269478A (en) * 2018-10-24 2019-01-25 南京大学 A kind of container terminal based on binocular vision bridge obstacle detection method

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113674275A (en) * 2021-10-21 2021-11-19 北京中科慧眼科技有限公司 Dense disparity map-based road surface unevenness detection method and system and intelligent terminal
CN115115687A (en) * 2022-06-24 2022-09-27 合众新能源汽车有限公司 Lane line measuring method and device

Also Published As

Publication number Publication date
CN111443704B (en) 2021-07-06
CN111443704A (en) 2020-07-24

Similar Documents

Publication Publication Date Title
WO2021120574A1 (en) Obstacle positioning method and apparatus for autonomous driving system
US11403774B2 (en) Image annotation
CN110443225B (en) Virtual and real lane line identification method and device based on feature pixel statistics
CN104700414A (en) Rapid distance-measuring method for pedestrian on road ahead on the basis of on-board binocular camera
Yang et al. Fast and accurate vanishing point detection and its application in inverse perspective mapping of structured road
CN107784038B (en) Sensor data labeling method
CN111488812B (en) Obstacle position recognition method and device, computer equipment and storage medium
CN110197173B (en) Road edge detection method based on binocular vision
CN110334678A (en) A kind of pedestrian detection method of view-based access control model fusion
US10872246B2 (en) Vehicle lane detection system
CN105205459B (en) A kind of recognition methods of characteristics of image vertex type and device
CN104751119A (en) Rapid detecting and tracking method for pedestrians based on information fusion
US20230144678A1 (en) Topographic environment detection method and system based on binocular stereo camera, and intelligent terminal
Kocur et al. Detection of 3D bounding boxes of vehicles using perspective transformation for accurate speed measurement
CN111178150A (en) Lane line detection method, system and storage medium
US20240013505A1 (en) Method, system, medium, equipment and terminal for inland vessel identification and depth estimation for smart maritime
Esteban et al. Closed form solution for the scale ambiguity problem in monocular visual odometry
CN112232275A (en) Obstacle detection method, system, equipment and storage medium based on binocular recognition
CN115410167A (en) Target detection and semantic segmentation method, device, equipment and storage medium
Zhang et al. Deep learning based object distance measurement method for binocular stereo vision blind area
WO2024017003A1 (en) Vehicle merging detection method and apparatus based on combined algorithms, and device
Muresan et al. Real-time object detection using a sparse 4-layer LIDAR
CN112017246A (en) Image acquisition method and device based on inverse perspective transformation
Wang et al. Preliminary research on vehicle speed detection using traffic cameras
CN110197104B (en) Distance measurement method and device based on vehicle

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20902745

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20902745

Country of ref document: EP

Kind code of ref document: A1