CN115082898A - Obstacle detection method, obstacle detection device, vehicle, and storage medium - Google Patents
Obstacle detection method, obstacle detection device, vehicle, and storage medium Download PDFInfo
- Publication number
- CN115082898A CN115082898A CN202210788539.2A CN202210788539A CN115082898A CN 115082898 A CN115082898 A CN 115082898A CN 202210788539 A CN202210788539 A CN 202210788539A CN 115082898 A CN115082898 A CN 115082898A
- Authority
- CN
- China
- Prior art keywords
- obstacle
- image
- vehicle
- position information
- dimensional position
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 111
- 238000000034 method Methods 0.000 claims abstract description 30
- 230000006870 function Effects 0.000 claims description 29
- 238000012549 training Methods 0.000 claims description 23
- 230000015654 memory Effects 0.000 claims description 14
- 238000002372 labelling Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 230000004888 barrier function Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 7
- 238000004891 communication Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 6
- 230000006399 behavior Effects 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 230000010267 cellular communication Effects 0.000 description 3
- 239000000446 fuel Substances 0.000 description 3
- LFQSCWFLJHTTHZ-UHFFFAOYSA-N Ethanol Chemical compound CCO LFQSCWFLJHTTHZ-UHFFFAOYSA-N 0.000 description 2
- ATUOYWHBWRKTHZ-UHFFFAOYSA-N Propane Chemical compound CCC ATUOYWHBWRKTHZ-UHFFFAOYSA-N 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 238000002485 combustion reaction Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- SAZUGELZHZOXHB-UHFFFAOYSA-N acecarbromal Chemical compound CCC(Br)(CC)C(=O)NC(=O)NC(C)=O SAZUGELZHZOXHB-UHFFFAOYSA-N 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000011217 control strategy Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000003208 petroleum Substances 0.000 description 1
- 239000001294 propane Substances 0.000 description 1
- 230000000979 retarding effect Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Abstract
The present disclosure relates to an obstacle detection method, apparatus, vehicle, and storage medium. The method comprises the following steps: acquiring image characteristics of an obstacle in an image to be detected, wherein the image to be detected is acquired through image acquisition equipment on a vehicle; acquiring three-dimensional position information of an obstacle in a coordinate system of image acquisition equipment; and inputting the image characteristics and the three-dimensional position information into a first preset detection network to obtain a classification result output by the first preset detection network, wherein the classification result represents whether the obstacle is the nearest obstacle in the path of the vehicle. The scheme implicitly learns the relationship between the lane line and the obstacle, and adds the three-dimensional position information of the obstacle in the coordinate system of the image acquisition equipment in the learning process, thereby realizing the detection of the nearest obstacle in the path. Because the method directly learns through the network and judges whether each obstacle is the nearest obstacle in the path of the vehicle, the lane line does not need to be identified and the distance of the vehicle in the own lane does not need to be calculated, and the detection is simple.
Description
Technical Field
The present disclosure relates to the field of automatic driving technologies, and in particular, to a method and an apparatus for detecting an obstacle, a vehicle, and a storage medium.
Background
In an automatic driving scene, in order to ensure the safety performance of the automatic driving scene, collision with a front vehicle is avoided; or when an Adaptive Cruise Control (ACC) function is implemented, the detection of a nearest vehicle (sipv) in a path is particularly important. In the related art, most of the methods for detecting the nearest vehicle in the route are to detect the vehicle in the image, detect lane lines at the same time, and determine the nearest vehicle in the route by calculating the relationship between the vehicle and the lane of the vehicle, so that the detection is complicated.
Disclosure of Invention
To overcome the problems in the related art, the present disclosure provides an obstacle detection method, apparatus, vehicle, and storage medium.
According to a first aspect of embodiments of the present disclosure, there is provided an obstacle detection method including:
acquiring image characteristics of obstacles in an image to be detected, wherein the image to be detected is acquired through image acquisition equipment on a vehicle;
acquiring three-dimensional position information of the barrier in a three-dimensional coordinate system of the image acquisition equipment;
and inputting the image characteristics and the three-dimensional position information into a first preset detection network to obtain a classification result output by the first preset detection network, wherein the classification result represents whether the obstacle is the nearest obstacle in the path of the vehicle.
Optionally, the acquiring image features of an obstacle in an image to be detected includes:
inputting the image to be detected into a backbone network and a second preset detection network, acquiring global image characteristics of the image to be detected output by the backbone network, and acquiring two-dimensional position information of an obstacle in the image to be detected output by the first preset detection network;
and selecting the image characteristics of the obstacles in the image to be detected from the global image characteristics according to the two-dimensional position information.
Optionally, the acquiring three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image acquisition device includes:
and acquiring the three-dimensional position information of the obstacle in the three-dimensional coordinate system of the image acquisition equipment according to the two-dimensional position information and the internal reference matrix of the image acquisition equipment.
Optionally, the backbone network, the second preset detection network, and the first preset detection network are obtained by training in the following manner:
labeling each obstacle in each image in a training set according to a labeling instruction input by a user, wherein the labeling instruction is used for describing a rectangular frame of each obstacle in the image and a classification label of each obstacle, and the classification label comprises the nearest obstacle in a path or is not the nearest obstacle in the path;
obtaining the backbone network, a second preset detection network and a first preset detection network according to the training set and the loss function training;
and the loss function is a loss function capable of adjusting the classification imbalance of the positive samples and the negative samples in the training set, wherein the positive samples are obstacles which are nearest obstacles in the path, and the negative samples are obstacles which are not nearest obstacles in the path.
Optionally, the formula for calculating the loss function is:
FL(pt)＝-at(1-pt) γ log(pt)
wherein pt is a classification result output by the second preset detection network, γ is a hyperparameter of a balanced hard-to-divide sample, at is a hyperparameter of a balanced positive and negative sample imbalance, and fl (pt) is an output of a loss function.
According to a second aspect of the embodiments of the present disclosure, there is provided an obstacle detection device including:
the obstacle image feature acquisition module is configured to acquire the image features of obstacles in an image to be detected, wherein the image to be detected is acquired through image acquisition equipment on a vehicle;
a three-dimensional position information acquisition module configured to acquire three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image acquisition device;
and the classification module is configured to input the image characteristics and the three-dimensional position information into a first preset detection network to obtain a classification result output by the first preset detection network, and the classification result represents whether the obstacle is the nearest obstacle in the path of the vehicle.
Optionally, the obstacle image feature obtaining module includes:
the feature output submodule is configured to input the image to be detected into a backbone network and a second preset detection network, acquire global image features of the image to be detected output by the backbone network, and acquire two-dimensional position information of an obstacle in the image to be detected output by the first preset detection network;
and the selecting submodule is configured to select the image characteristics of the obstacles in the image to be detected from the global image characteristics according to the two-dimensional position information.
Optionally, the three-dimensional position information obtaining module is specifically configured to obtain three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image capturing device according to the two-dimensional position information and an internal reference matrix of the image capturing device.
According to a third aspect of the embodiments of the present disclosure, there is provided a vehicle including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
the steps of the method for detecting an obstacle provided in the first aspect of the embodiments of the present disclosure are implemented.
According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions, which when executed by a first processor, implement the steps of the obstacle detection method provided by the first aspect of the embodiments of the present disclosure.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
and learning the image characteristics and the three-dimensional position of the obstacle through a first preset detection network, and outputting a classification result. When the method is applied, aiming at an image to be detected, the image characteristics and the three-dimensional position information of each obstacle in the image to be detected are input into a first preset detection network, so that the classification result of each obstacle in the image to be detected can be obtained, and whether each obstacle in the image to be detected is the nearest obstacle in the path of the vehicle or not is judged. Therefore, the scheme provided by the disclosure implicitly learns the relationship between the lane line and the obstacle, and adds the three-dimensional position information of the obstacle in the coordinate system of the image acquisition equipment in the learning process, so that the detection of the nearest obstacle in the path is realized. Because the method directly judges whether each obstacle in the detection image is the nearest obstacle in the path of the vehicle according to the network learning and the learning network, the method does not need to identify lane lines and calculate the distance of the vehicle in the own lane, and the detection is simple.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a flow chart illustrating a method of obstacle detection according to an exemplary embodiment.
Fig. 2 is a logic diagram illustrating a method of obstacle detection according to an exemplary embodiment.
Fig. 3 is a block diagram illustrating an obstacle detection device according to an exemplary embodiment.
FIG. 4 is a block diagram of a vehicle shown in an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
Fig. 1 is a flow chart illustrating a method of obstacle detection according to an exemplary embodiment. The obstacle detection method can be applied to automatic driving, Adaptive Cruise Control (ACC) and the like. As shown in fig. 1, the obstacle detection method includes the following steps.
In step S11, image features of an obstacle in an image to be detected, which is acquired by an image acquisition apparatus on a vehicle, are acquired.
The image characteristics of the obstacles in the image to be detected can be obtained through a deep learning network. In the automatic driving and adaptive cruising of the vehicle, the obstacles can be vehicles such as small passenger cars, large trucks, private cars, motorcycles and the like. The image to be detected can be acquired by image acquisition equipment arranged on an automatic driving and self-adaptive cruising vehicle. The image acquisition device may be a tachograph, a video camera, a still camera, etc.
In step S12, three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image pickup apparatus is acquired.
The three-dimensional coordinate system of the image acquisition device is a three-dimensional coordinate system established by taking a focus center of the image acquisition device as an origin and taking an optical axis as a Z axis (depth coordinate axis).
In step S13, the image features and the three-dimensional position information are input into a first preset detection network, and a classification result output by the first preset detection network is obtained, where the classification result indicates whether the obstacle is the closest obstacle in the path of the vehicle.
The image feature and the three-dimensional position information may be input into the first preset detection network after being spliced. The classification result may be a probability of being the nearest obstacle in the path of the vehicle. That is, for an obstacle, the classification result is 80%, which indicates that the obstacle is the closest obstacle in the path of the vehicle, and the probability is 80%. Obviously, the classification result can also be; a probability of not being a nearest obstacle in the path of the vehicle; it may also be a classification label of whether the obstacle is the closest obstacle in the path of the vehicle (e.g. the classification label may be represented by 1 and 0, 1 representing that the obstacle is the closest obstacle in the path of the vehicle, 0 representing that the obstacle is not the closest obstacle in the path of the vehicle.
According to the technical scheme, the image characteristics and the three-dimensional position of the obstacle are learned through the first preset detection network, and the classification result is output. When the method is applied, aiming at an image to be detected, the image characteristics and the three-dimensional position information of each obstacle in the image to be detected are input into a first preset detection network, so that the classification result of each obstacle in the image to be detected can be obtained, and whether each obstacle in the image to be detected is the nearest obstacle in the path of the vehicle or not is judged. Therefore, the scheme provided by the disclosure learns the relationship between the lane line and the obstacle implicitly, and adds the three-dimensional position information of the obstacle in the coordinate system of the image acquisition equipment in the learning process, so that the detection of the nearest obstacle in the path is realized. Because the method directly judges whether each obstacle in the detection image is the nearest obstacle in the path of the vehicle according to the network learning and the learning network, the method does not need to identify lane lines and calculate the distance of the vehicle in the own lane, and the detection is simple.
Fig. 2 is a logic diagram illustrating a method of obstacle detection according to an exemplary embodiment. As shown in fig. 2, optionally, step S11 includes:
inputting the image to be detected into a backbone network and a second preset detection network, acquiring the global image characteristics of the image to be detected output by the backbone network, and acquiring the two-dimensional position information of the obstacle in the image to be detected output by the first preset detection network.
The backbone network may adopt a DLA34 network, and the second preset detection network may use a simple full-connection network. The two-dimensional position information may be two-dimensional position information of a rectangular frame of the obstacle predicted by the second preset detection network. The two-dimensional position information of the rectangular frame is the two-dimensional position information of the rectangular frame under the two-dimensional coordinate system of the image to be detected, and can be represented by the two-dimensional coordinates of the upper left corner point and the two-dimensional coordinates of the lower right corner point of the rectangular frame. The two-dimensional coordinate system of the image to be detected is a two-dimensional coordinate system established by taking a certain pixel point in the image to be detected as an original point of coordinates, and the plane of the two-dimensional coordinate system is the same as the plane of the image to be detected.
And selecting the image characteristics of the obstacles in the image to be detected from the global image characteristics according to the two-dimensional position information.
According to the technical scheme, the image characteristics of the obstacle can be selected from the global image characteristics output by the backbone network according to the two-dimensional position information of the obstacle output by the first preset detection network.
Optionally, step S12 includes:
and acquiring the three-dimensional position information of the obstacle in the three-dimensional coordinate system of the image acquisition equipment according to the two-dimensional position information and the internal reference matrix of the image acquisition equipment.
The calculation formula of the steps is as follows:
in the formula (1), x, y and z respectively represent an abscissa, an ordinate and a depth coordinate of a pixel point of the obstacle under a three-dimensional coordinate system of the image acquisition equipment;
is an internal reference matrix of the image acquisition device
Fx represents a parameter of the image capturing apparatus after the focal length f is scaled on the horizontal axis of the two-dimensional coordinate system, fy represents a parameter of the image capturing apparatus after the focal length f is scaled on the vertical axis of the two-dimensional coordinate system, cx represents a translation of the origin of the three-dimensional coordinate system of the image capturing apparatus on the horizontal axis of the two-dimensional coordinate system, and cy represents a translation of the origin of the three-dimensional coordinate system of the image capturing apparatus on the vertical axis of the two-dimensional coordinate systemAnd (3) translating, wherein u is the abscissa of a pixel point of the obstacle in the two-dimensional coordinate system of the image to be detected, and v is the ordinate of a pixel point of the obstacle in the two-dimensional coordinate system of the image to be detected.
As can be seen from the formula (1), in
When u and v are known, x, y and z can be obtained.
Optionally, the backbone network, the second preset detection network, and the first preset detection network are obtained by training in the following manner:
and labeling each obstacle in the image according to a labeling instruction input by a user aiming at each image in the training set, wherein the labeling instruction is used for describing a rectangular frame of each obstacle in the image and a classification label of each obstacle, and the classification label comprises the nearest obstacle in the path or not.
And obtaining the backbone network, a second preset detection network and a first preset detection network according to the training set and the loss function training.
And the loss function is a loss function capable of adjusting the classification imbalance of the positive samples and the negative samples in the training set, wherein the positive samples are obstacles which are nearest obstacles in the path, and the negative samples are obstacles which are not nearest obstacles in the path.
By the technical scheme, the backbone network, the second preset detection network and the first preset detection network are trained together based on the training set and the loss function, the loss function can adjust the classification imbalance of the positive samples and the negative samples in the training set, and the detection accuracy of the backbone network, the second preset detection network and the first preset detection network is improved.
Optionally, the formula for calculating the loss function is:
FL(pt)＝-at(1-pt) γ log (pt) (equation 2)
In formula (2), pt is the classification result output by the second predetermined detection network, γ is the hyperparameter of the balanced hard-to-classify sample, at is the hyperparameter of the balanced positive and negative sample imbalance, and fl (pt) is the output of the loss function.
Where at may be determined from the ratio of the positive and negative samples, for example 0.25 in one embodiment. The γ may be determined experimentally, for example, in one embodiment, is determined experimentally to be 2.
Based on the technical concept, the disclosure also provides an obstacle detection device. Fig. 3 is a block diagram illustrating an obstacle detection device according to an exemplary embodiment. Referring to fig. 3, the apparatus includes: an obstacle image feature acquisition module 11, a three-dimensional position information acquisition module 12, and a classification module 13.
An obstacle image feature obtaining module 11 configured to obtain an image feature of an obstacle in an image to be detected, the image to be detected being acquired by an image acquisition device on a vehicle;
a three-dimensional position information acquisition module 12 configured to acquire three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image acquisition device;
and the classification module 13 is configured to input the image features and the three-dimensional position information into a first preset detection network, and obtain a classification result output by the first preset detection network, wherein the classification result represents whether the obstacle is the nearest obstacle in the path of the vehicle.
According to the technical scheme, the image characteristics and the three-dimensional position of the obstacle are learned through the first preset detection network, and the classification result is output. When the method is applied, aiming at an image to be detected, the image characteristics and the three-dimensional position information of each obstacle in the image to be detected are input into a first preset detection network, so that the classification result of each obstacle in the image to be detected can be obtained, and whether each obstacle in the image to be detected is the nearest obstacle in the path of the vehicle or not is judged. Therefore, the scheme provided by the disclosure implicitly learns the relationship between the lane line and the obstacle, and adds the three-dimensional position information of the obstacle in the coordinate system of the image acquisition equipment in the learning process, so that the detection of the nearest obstacle in the path is realized. Because the method directly judges whether each obstacle in the detection image is the nearest obstacle in the path of the vehicle according to the network learning and the learning network, the method does not need to identify lane lines and calculate the distance of the vehicle in the own lane, and the detection is simple.
Optionally, the obstacle image feature obtaining module includes:
the feature output submodule is configured to input the image to be detected into a backbone network and a second preset detection network, acquire global image features of the image to be detected output by the backbone network, and acquire two-dimensional position information of an obstacle in the image to be detected output by the first preset detection network;
and the selecting submodule is configured to select the image characteristics of the obstacles in the image to be detected from the global image characteristics according to the two-dimensional position information.
According to the technical scheme, the image characteristics of the obstacle can be selected from the global image characteristics output by the backbone network according to the two-dimensional position information of the obstacle output by the first preset detection network.
Optionally, the three-dimensional position information obtaining module is specifically configured to obtain three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image acquisition device according to the two-dimensional position information and an internal reference matrix of the image acquisition device.
Optionally, the apparatus further comprises a training module configured to:
and labeling each obstacle in the image according to a labeling instruction input by a user aiming at each image in the training set, wherein the labeling instruction is used for describing a rectangular frame of each obstacle in the image and a classification label of each obstacle, and the classification label comprises the nearest obstacle in the path or not.
And training according to the training set and the loss function to obtain the backbone network, a second preset detection network and a first preset detection network.
And the loss function is a loss function capable of adjusting the imbalance of the classification of the positive samples and the negative samples in the training set, wherein the positive samples are the obstacles which are the nearest obstacles in the path, and the negative samples are the obstacles which are not the nearest obstacles in the path.
By the technical scheme, the backbone network, the second preset detection network and the first preset detection network are trained together based on the training set and the loss function, the loss function can adjust the classification imbalance of the positive samples and the negative samples in the training set, and the detection accuracy of the backbone network, the second preset detection network and the first preset detection network is improved.
Optionally, the formula for calculating the loss function is:
FL)pt)＝-at(1-pt) γ log (pt) (equation 2)
In formula (2), pt is the classification result output by the second predetermined detection network, γ is the hyperparameter of the balanced hard-to-classify sample, at is the hyperparameter of the balanced positive and negative sample imbalance, and fl (pt) is the output of the loss function.
Where at may be determined from the ratio of the positive and negative examples, for example, 0.25 in one embodiment. The γ may be determined experimentally, for example, in one embodiment, is determined experimentally to be 2.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
The present disclosure also provides a computer readable storage medium having stored thereon computer program instructions which, when executed by a first processor, implement the steps of the obstacle detection method provided by the present disclosure.
The apparatus may be a part of a stand-alone electronic device, for example, in an embodiment, the apparatus may be an Integrated Circuit (IC) or a chip, where the IC may be one IC or a collection of multiple ICs; the chip may include, but is not limited to, the following categories: a GPU (Graphics Processing Unit), a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an SOC (System on Chip, SOC, System on Chip, or System on Chip), and the like. The integrated circuit or chip may be configured to execute executable instructions (or code) to implement the obstacle detection method. Where the executable instructions may be stored in the integrated circuit or chip or may be retrieved from another apparatus or device, for example where the integrated circuit or chip includes a second processor, a second memory, and an interface for communicating with the other apparatus. The executable instructions may be stored in the second memory, and when executed by the second processor, implement the above-described obstacle detection method; alternatively, the integrated circuit or chip may receive the executable instructions through the interface and transmit the executable instructions to the second processor for execution, so as to implement the obstacle detection method.
Referring to fig. 4, fig. 4 is a functional block diagram of a vehicle 600 according to an exemplary embodiment. The vehicle 600 may be configured in a fully or partially autonomous driving mode. For example, the vehicle 600 may acquire environmental information of its surroundings through the sensing system 620 and derive an automatic driving strategy based on an analysis of the surrounding environmental information to implement full automatic driving, or present the analysis result to the user to implement partial automatic driving.
Vehicle 600 may include various subsystems such as infotainment system 610, perception system 620, decision control system 630, drive system 640, and computing platform 650. Alternatively, vehicle 600 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each of the sub-systems and components of the vehicle 600 may be interconnected by wire or wirelessly.
In some embodiments, the infotainment system 610 may include a communication system 611, an entertainment system 612, and a navigation system 613.
The communication system 611 may comprise a wireless communication system that may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system may use 3G cellular communication, such as CDMA, EVD0, GSM/GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system may communicate with a Wireless Local Area Network (WLAN) using WiFi. In some embodiments, the wireless communication system may utilize an infrared link, bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various vehicular communication systems, for example, a wireless communication system may include one or more Dedicated Short Range Communications (DSRC) devices that may include public and/or private data communications between vehicles and/or roadside stations.
The entertainment system 612 may include a display device, a microphone and a sound, and a user may listen to a radio in the car based on the entertainment system, playing music; or the mobile phone is communicated with the vehicle, screen projection of the mobile phone is realized on the display equipment, the display equipment can be in a touch control type, and a user can operate the display equipment by touching the screen.
In some cases, the voice signal of the user may be acquired through a microphone, and certain control of the vehicle 600 by the user, such as adjusting the temperature in the vehicle, etc., may be implemented according to the analysis of the voice signal of the user. In other cases, music may be played to the user through a stereo.
The navigation system 613 may include a map service provided by a map provider to provide navigation of a route of travel for the vehicle 600, and the navigation system 613 may be used in conjunction with a global positioning system 621 and an inertial measurement unit 622 of the vehicle. The map service provided by the map provider can be a two-dimensional map or a high-precision map.
The sensing system 620 may include several sensors that sense information about the environment surrounding the vehicle 600. For example, the sensing system 620 may include a global positioning system 621 (the global positioning system may be a GPS system, a beidou system or other positioning system), an Inertial Measurement Unit (IMU) 622, a laser radar 623, a millimeter wave radar 624, an ultrasonic radar 625, and a camera 626. The sensing system 620 may also include sensors of internal systems of the monitored vehicle 600 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect the object and its corresponding characteristics (position, shape, orientation, velocity, etc.). Such detection and identification is a critical function of the safe operation of the vehicle 600.
Global positioning system 621 is used to estimate the geographic location of vehicle 600.
The inertial measurement unit 622 is used to sense a pose change of the vehicle 600 based on the inertial acceleration. In some embodiments, inertial measurement unit 622 may be a combination of accelerometers and gyroscopes.
Lidar 623 utilizes laser light to sense objects in the environment in which vehicle 600 is located. In some embodiments, lidar 623 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.
The millimeter-wave radar 624 utilizes radio signals to sense objects within the surrounding environment of the vehicle 600. In some embodiments, in addition to sensing objects, the millimeter-wave radar 624 may also be used to sense the speed and/or heading of objects.
The ultrasonic radar 625 may sense objects around the vehicle 600 using ultrasonic signals.
The camera 626 is used to capture image information of the surroundings of the vehicle 600. The image capturing device 626 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, and the like, and the image information acquired by the image capturing device 626 may include still images or video stream information.
Decision control system 630 includes a computing system 631 that makes analytical decisions based on information acquired by sensing system 620, decision control system 630 further includes a vehicle control unit 632 that controls the powertrain of vehicle 600, and a steering system 633, throttle 634, and brake system 635 for controlling vehicle 600.
The computing system 631 may operate to process and analyze the various information acquired by the perception system 620 to identify objects, and/or features in the environment surrounding the vehicle 600. The target may comprise a pedestrian or an animal and the objects and/or features may comprise traffic signals, road boundaries and obstacles. Computing system 631 may use object recognition algorithms, Motion from Motion (SFM) algorithms, video tracking, and like techniques. In some embodiments, the computing system 631 may be used to map an environment, track objects, estimate the speed of objects, and so forth. The computing system 631 may analyze the various information obtained and derive a control strategy for the vehicle.
The vehicle controller 632 may be used to perform coordinated control on the power battery and the engine 641 of the vehicle to improve the power performance of the vehicle 600.
The steering system 633 is operable to adjust the heading of the vehicle 600. For example, in one embodiment, a steering wheel system.
The throttle 634 is used to control the operating speed of the engine 641 and thus the speed of the vehicle 600.
The brake system 635 is used to control the deceleration of the vehicle 600. The braking system 635 may use friction to slow the wheel 644. In some embodiments, the braking system 635 may convert the kinetic energy of the wheels 644 into electrical current. The braking system 635 may also take other forms to slow the rotational speed of the wheels 644 to control the speed of the vehicle 600.
The drive system 640 may include components that provide powered motion to the vehicle 600. In one embodiment, the drive system 640 may include an engine 641, an energy source 642, a transmission 643, and wheels 644. The engine 641 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 641 converts the energy source 642 into mechanical energy.
Examples of energy source 642 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electrical power. The energy source 642 may also provide energy to other systems of the vehicle 600.
The transmission 643 may transmit mechanical power from the engine 641 to the wheels 644. The transmission 643 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 643 may also include other components, such as clutches. Wherein the drive shaft may include one or more axles that may be coupled to one or more wheels 644.
Some or all of the functionality of the vehicle 600 is controlled by the computing platform 650. Computing platform 650 can include at least one processor 651, and processor 651 can execute instructions 653 stored in a non-transitory computer-readable medium, such as memory 652. In some embodiments, the computing platform 650 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 600 in a distributed manner.
The processor 651 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor 651 may also include a processor such as a Graphics Processor Unit (GPU), a Field Programmable Gate Array (FPGA), a System On Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof. Although fig. 4 functionally illustrates a processor, memory, and other elements of a computer in the same block, those skilled in the art will appreciate that the processor, computer, or memory may actually comprise multiple processors, computers, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a different enclosure than the computer. Thus, reference to a processor or computer will be understood to include reference to a collection of processors or computers or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components, such as the steering component and the retarding component, may each have their own processor that performs only computations related to the component-specific functions.
In the disclosed embodiment, the processor 651 may perform the above-described obstacle detection method.
In various aspects described herein, the processor 651 may be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are executed on a processor disposed within the vehicle and others are executed by a remote processor, including taking the steps necessary to perform a single maneuver.
In some embodiments, the memory 652 may contain instructions 653 (e.g., program logic), which instructions 653 may be executed by the processor 651 to perform various functions of the vehicle 600. The memory 652 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and/or control one or more of the infotainment system 610, the perception system 620, the decision control system 630, the drive system 640.
In addition to instructions 653, memory 652 may store data such as road maps, route information, the location, direction, speed of the vehicle, and other such vehicle data, as well as other information. Such information may be used by the vehicle 600 and the computing platform 650 during operation of the vehicle 600 in autonomous, semi-autonomous, and/or manual modes.
Computing platform 650 may control functions of vehicle 600 based on inputs received from various subsystems (e.g., drive system 640, perception system 620, and decision control system 630). For example, computing platform 650 may utilize input from decision control system 630 in order to control steering system 633 to avoid obstacles detected by perception system 620. In some embodiments, the computing platform 650 is operable to provide control over many aspects of the vehicle 600 and its subsystems.
Optionally, one or more of these components described above may be mounted separately from or associated with the vehicle 600. For example, the memory 652 may exist partially or completely separate from the vehicle 600. The above components may be communicatively coupled together in a wired and/or wireless manner.
Optionally, the above components are only an example, in an actual application, components in the above modules may be added or deleted according to an actual need, and fig. 4 should not be construed as limiting the embodiment of the present disclosure.
An autonomous automobile traveling on a road, such as vehicle 600 above, may identify objects within its surrounding environment to determine an adjustment to the current speed. The object may be another vehicle, a traffic control device, or another type of object. In some examples, each identified object may be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, separation from the vehicle, etc., may be used to determine the speed at which the autonomous vehicle is to be adjusted.
Optionally, the vehicle 600 or a sensory and computing device associated with the vehicle 600 (e.g., computing system 631, computing platform 650) may predict behavior of the identified object based on characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so it is also possible to predict the behavior of a single identified object taking all identified objects together into account. The vehicle 600 is able to adjust its speed based on the predicted behavior of the identified object. In other words, the autonomous vehicle is able to determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors may also be considered to determine the speed of the vehicle 600, such as the lateral position of the vehicle 600 in the road being traveled, the curvature of the road, the proximity of static and dynamic objects, and so forth.
In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of the vehicle 600 to cause the autonomous vehicle to follow a given trajectory and/or maintain a safe lateral and longitudinal distance from objects in the vicinity of the autonomous vehicle (e.g., vehicles in adjacent lanes on the road).
The vehicle 600 may be any type of vehicle, such as a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a recreational vehicle, a train, etc., and the disclosed embodiment is not particularly limited.
In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the above-mentioned obstacle detection method when executed by the programmable apparatus.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (10)
1. An obstacle detection method, comprising:
acquiring image characteristics of obstacles in an image to be detected, wherein the image to be detected is acquired through image acquisition equipment on a vehicle;
acquiring three-dimensional position information of the barrier in a three-dimensional coordinate system of the image acquisition equipment;
and inputting the image characteristics and the three-dimensional position information into a first preset detection network to obtain a classification result output by the first preset detection network, wherein the classification result represents whether the obstacle is the nearest obstacle in the path of the vehicle.
2. The obstacle detection method according to claim 1, wherein the acquiring image features of the obstacle in the image to be detected includes:
inputting the image to be detected into a backbone network and a second preset detection network, acquiring global image characteristics of the image to be detected output by the backbone network, and acquiring two-dimensional position information of an obstacle in the image to be detected output by the first preset detection network;
and selecting the image characteristics of the obstacles in the image to be detected from the global image characteristics according to the two-dimensional position information.
3. The obstacle detection method according to claim 2, wherein the acquiring three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image pickup apparatus includes:
and acquiring the three-dimensional position information of the obstacle under the three-dimensional coordinate system of the image acquisition equipment according to the two-dimensional position information and the internal reference matrix of the image acquisition equipment.
4. The obstacle detection method of claim 3, wherein the backbone network, the second predetermined detection network and the first predetermined detection network are trained by:
labeling each obstacle in each image in a training set according to a labeling instruction input by a user, wherein the labeling instruction is used for describing a rectangular frame of each obstacle in the image and a classification label of each obstacle, and the classification label comprises the nearest obstacle in a path or is not the nearest obstacle in the path;
obtaining the backbone network, a second preset detection network and a first preset detection network according to the training set and the loss function training;
and the loss function is a loss function capable of adjusting the classification imbalance of the positive samples and the negative samples in the training set, wherein the positive samples are obstacles which are nearest obstacles in the path, and the negative samples are obstacles which are not nearest obstacles in the path.
5. The obstacle detection method according to claim 4, wherein the loss function is calculated by the formula:
FL(pt)＝-at(1-pt) γ log(pt)
wherein pt is a classification result output by the second preset detection network, γ is a hyperparameter of the balanced hard-to-divide sample, at is a hyperparameter of the balanced positive and negative sample imbalance, and fl (pt) is an output of the loss function.
6. An obstacle detection device, characterized by comprising:
the obstacle image feature acquisition module is configured to acquire the image features of obstacles in an image to be detected, wherein the image to be detected is acquired through image acquisition equipment on a vehicle;
a three-dimensional position information acquisition module configured to acquire three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image acquisition device;
and the classification module is configured to input the image characteristics and the three-dimensional position information into a first preset detection network to obtain a classification result output by the first preset detection network, and the classification result represents whether the obstacle is the nearest obstacle in the path of the vehicle.
7. The obstacle detection apparatus according to claim 6, wherein the obstacle image feature acquisition module includes:
the feature output submodule is configured to input the image to be detected into a backbone network and a second preset detection network, acquire global image features of the image to be detected output by the backbone network, and acquire two-dimensional position information of an obstacle in the image to be detected output by the first preset detection network;
and the selecting submodule is configured to select the image characteristics of the obstacles in the image to be detected from the global image characteristics according to the two-dimensional position information.
8. The obstacle detection apparatus according to claim 7, wherein the three-dimensional position information acquisition module is specifically configured to acquire three-dimensional position information of the obstacle in a three-dimensional coordinate system of the image capturing device, based on the two-dimensional position information and an internal reference matrix of the image capturing device.
9. A vehicle, characterized by comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
implementing the method of any one of claims 1 to 5.
10. A computer-readable storage medium, on which computer program instructions are stored, which program instructions, when executed by a first processor, carry out the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210788539.2A CN115082898A (en) | 2022-07-04 | 2022-07-04 | Obstacle detection method, obstacle detection device, vehicle, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210788539.2A CN115082898A (en) | 2022-07-04 | 2022-07-04 | Obstacle detection method, obstacle detection device, vehicle, and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115082898A true CN115082898A (en) | 2022-09-20 |
Family
ID=83258430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210788539.2A Pending CN115082898A (en) | 2022-07-04 | 2022-07-04 | Obstacle detection method, obstacle detection device, vehicle, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115082898A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110751693A (en) * | 2019-10-21 | 2020-02-04 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for camera calibration |
| CN111179300A (en) * | 2019-12-16 | 2020-05-19 | 新奇点企业管理集团有限公司 | Method, apparatus, system, device and storage medium for obstacle detection |
| CN111191600A (en) * | 2019-12-30 | 2020-05-22 | 深圳元戎启行科技有限公司 | Obstacle detection method, obstacle detection device, computer device, and storage medium |
| CN112344855A (en) * | 2020-10-27 | 2021-02-09 | 北京百度网讯科技有限公司 | Obstacle detection method and device, storage medium and drive test equipment |
| CN112902981A (en) * | 2021-01-26 | 2021-06-04 | 中国科学技术大学 | Robot navigation method and device |
| CN113901815A (en) * | 2021-10-15 | 2022-01-07 | 华能澜沧江水电股份有限公司 | Emergency working condition event detection method based on dam operation log |
| WO2022126540A1 (en) * | 2020-12-17 | 2022-06-23 | 深圳市大疆创新科技有限公司 | Obstacle detection and re-identification method, apparatus, movable platform, and storage medium |
-
2022
- 2022-07-04 CN CN202210788539.2A patent/CN115082898A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110751693A (en) * | 2019-10-21 | 2020-02-04 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for camera calibration |
| CN111179300A (en) * | 2019-12-16 | 2020-05-19 | 新奇点企业管理集团有限公司 | Method, apparatus, system, device and storage medium for obstacle detection |
| CN111191600A (en) * | 2019-12-30 | 2020-05-22 | 深圳元戎启行科技有限公司 | Obstacle detection method, obstacle detection device, computer device, and storage medium |
| CN112344855A (en) * | 2020-10-27 | 2021-02-09 | 北京百度网讯科技有限公司 | Obstacle detection method and device, storage medium and drive test equipment |
| WO2022126540A1 (en) * | 2020-12-17 | 2022-06-23 | 深圳市大疆创新科技有限公司 | Obstacle detection and re-identification method, apparatus, movable platform, and storage medium |
| CN112902981A (en) * | 2021-01-26 | 2021-06-04 | 中国科学技术大学 | Robot navigation method and device |
| CN113901815A (en) * | 2021-10-15 | 2022-01-07 | 华能澜沧江水电股份有限公司 | Emergency working condition event detection method based on dam operation log |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115042821B (en) | Vehicle control method, vehicle control device, vehicle and storage medium | |
| CN115123257B (en) | Pavement deceleration strip position identification method and device, vehicle, storage medium and chip | |
| CN115330923B (en) | Point cloud data rendering method and device, vehicle, readable storage medium and chip | |
| CN115205365A (en) | Vehicle distance detection method and device, vehicle, readable storage medium and chip | |
| CN115100377A (en) | Map construction method and device, vehicle, readable storage medium and chip | |
| CN115035494A (en) | Image processing method, image processing device, vehicle, storage medium and chip | |
| CN114842075A (en) | Data labeling method and device, storage medium and vehicle | |
| CN115056784B (en) | Vehicle control method, device, vehicle, storage medium and chip | |
| CN114782638B (en) | Method and device for generating lane line, vehicle, storage medium and chip | |
| CN115203457B (en) | Image retrieval method, device, vehicle, storage medium and chip | |
| CN115100630B (en) | Obstacle detection method, obstacle detection device, vehicle, medium and chip | |
| CN115202234B (en) | Simulation test method and device, storage medium and vehicle | |
| CN114842440B (en) | Automatic driving environment sensing method and device, vehicle and readable storage medium | |
| CN115205848A (en) | Target detection method, target detection device, vehicle, storage medium and chip | |
| CN115221151A (en) | Vehicle data transmission method and device, vehicle, storage medium and chip | |
| CN115222791A (en) | Target association method, device, readable storage medium and chip | |
| CN115042814A (en) | Traffic light state identification method and device, vehicle and storage medium | |
| CN115082898A (en) | Obstacle detection method, obstacle detection device, vehicle, and storage medium | |
| CN115063639B (en) | Model generation method, image semantic segmentation device, vehicle and medium | |
| CN115082886B (en) | Target detection method, device, storage medium, chip and vehicle | |
| CN115407344B (en) | Grid map creation method, device, vehicle and readable storage medium | |
| CN114572219B (en) | Automatic overtaking method and device, vehicle, storage medium and chip | |
| CN115179930B (en) | Vehicle control method and device, vehicle and readable storage medium | |
| CN114771514B (en) | Vehicle running control method, device, equipment, medium, chip and vehicle | |
| CN115139946B (en) | Vehicle falling water detection method, vehicle, computer readable storage medium and chip |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |