CN111858979B - Obstacle recognition method and device, electronic equipment and automatic driving vehicle - Google Patents
Obstacle recognition method and device, electronic equipment and automatic driving vehicle Download PDFInfo
- Publication number
- CN111858979B CN111858979B CN202010717031.4A CN202010717031A CN111858979B CN 111858979 B CN111858979 B CN 111858979B CN 202010717031 A CN202010717031 A CN 202010717031A CN 111858979 B CN111858979 B CN 111858979B
- Authority
- CN
- China
- Prior art keywords
- image
- key
- image frames
- frames
- obstacle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
The disclosure provides a database construction method and device, and an obstacle identification method, device and system. The method comprises the steps that a database construction device receives a plurality of first image frames acquired by image acquisition equipment on a preset route; removing dynamic obstacles in the plurality of first image frames to obtain a plurality of second image frames comprising static obstacles; extracting a feature vector of each second image frame; extracting a key point and a corresponding description vector from each second image frame; and storing each second image frame and corresponding positioning information, feature vectors, key points and corresponding description vectors into a database. Searching a database for a matching image frame matched with the real-time image frame; extracting key points and key point description vectors of the barrier region in the real-time image frame; matching the key point description vector of the real-time image frame with the key point description vector of the matched image frame to obtain a matched key point; and if the number of the matched key points is larger than the threshold, marking the obstacle area as a static obstacle.
Description
Technical Field
The present disclosure relates to the field of information processing, and in particular, to a database construction method and apparatus, and an obstacle identification method, apparatus, and system.
Background
In the field of automatic driving, by sensing obstacles around a vehicle, in the related art, after an image is acquired by an automatic driving vehicle, a deep convolutional neural network is used for obstacle detection and type identification, and the position, size, direction and category of the obstacle are estimated, so that the automatic driving vehicle can plan a path and a speed.
Disclosure of Invention
The inventor finds that in the related art, objects (such as greenbelts, flower beds, stumps and the like) which are different from each other and are fixed on two sides of a driving route are generally identified as vehicles with movement capability. In addition, false identifications of false objects present in the image, such as vehicles, people, etc. in the advertisement poster, are also highly likely to occur, thereby affecting the path and speed planning of the autonomous vehicle.
The present disclosure is proposed to effectively recognize a static obstacle in the field of automatic driving, and a static obstacle having no moving ability can be effectively recognized.
According to a first aspect of the embodiments of the present disclosure, there is provided a database construction method, including: receiving a plurality of first image frames acquired by an image acquisition device on a preset route; removing dynamic obstacles in the plurality of first image frames to obtain a plurality of second image frames comprising static obstacles; extracting a feature vector of each second image frame; extracting a key point and a description vector corresponding to the key point from each second image frame; and storing each second image frame, the positioning information of the image acquisition equipment corresponding to each second image frame, the feature vector, the key point and the description vector corresponding to the key point into a database.
In some embodiments, extracting the feature vector of each second image frame comprises: sequentially extracting a plurality of key frames from the plurality of second image frames, wherein the image overlapping rate of any two adjacent first key frames and second key frames is less than a preset threshold value; extracting the feature vector of each key frame.
In some embodiments, a deviation between the position information of the image capture device corresponding to the first keyframe and the position information of the image capture device corresponding to the second keyframe is greater than a preset distance threshold; or the deviation between the attitude information of the image acquisition equipment corresponding to the first key frame and the attitude information of the image acquisition equipment corresponding to the second key frame is larger than a preset angle threshold.
In some embodiments, extracting keypoints and description vectors associated with the keypoints from the each second image frame comprises: from each of the keyframes, keypoints and description vectors associated with the keypoints are extracted.
According to a second aspect of the embodiments of the present disclosure, there is provided a database construction apparatus including: the image acquisition device comprises a first receiving module, a second receiving module and a display module, wherein the first receiving module is configured to receive a plurality of first image frames acquired by the image acquisition device on a preset route; an image frame processing module configured to remove dynamic obstacles in the plurality of first image frames to obtain a plurality of second image frames including static obstacles; a first extraction module configured to extract a feature vector of each second image frame; a second extraction module configured to extract a keypoint and a description vector corresponding to the keypoint from each of the second image frames; a construction module configured to store the each second image frame, the positioning information of the image acquisition device corresponding to the each second image frame, the feature vector, the key point, and the description vector corresponding to the key point in a database.
According to a third aspect of the embodiments of the present disclosure, there is provided an obstacle identification method including: receiving a real-time image frame acquired by image acquisition equipment; searching a database obtained by using the database construction method of any one of the embodiments for a predetermined number of matched image frames matched with the real-time image frames; performing obstacle detection in the real-time image frames to determine an obstacle region; extracting key points and corresponding key point description vectors in the obstacle region; matching the key point description vectors of the real-time image frames with the key point description vectors of the preset number of matched image frames to obtain matched key points; and if the number of the matched key points is greater than a preset key point threshold, marking the obstacle area as a static obstacle.
In some embodiments, the obstacle region is marked as a dynamic obstacle if the number of matching keypoints is not greater than a preset keypoint threshold.
In some embodiments, searching the database for a predetermined number of matching image frames that match the real-time image frames comprises: extracting an image frame corresponding to a preset area with the position information as the center from the database according to the position information of the image acquisition equipment corresponding to the real-time image frame to be used as a candidate image frame; extracting feature vectors of the candidate image frames from the database; extracting a feature vector of the real-time image frame; and matching the feature vectors of the real-time image frames with the feature vectors of the candidate image frames, and taking the candidate image frames with the highest matching degree in the preset number as matched image frames.
According to a fourth aspect of the embodiments of the present disclosure, there is provided an obstacle recognition apparatus including: the second receiving module is configured to receive the real-time image frame acquired by the image acquisition device; a searching module configured to search a database obtained by using the database construction method according to any of the above embodiments for a predetermined number of matching image frames matching the real-time image frame; a detection module configured to perform obstacle detection in the real-time image frames to determine an obstacle region; a third extraction module configured to extract key points and corresponding key point description vectors in the obstacle region; a matching module configured to match the keypoint description vectors of the real-time image frames with the keypoint description vectors of the predetermined number of matched image frames to obtain matched keypoints; an identification module configured to mark the obstacle area as a static obstacle if the number of matching keypoints is greater than a preset keypoint threshold.
According to a fifth aspect of embodiments of the present disclosure, there is provided an electronic apparatus including: a memory configured to store instructions; a processor coupled to the memory, the processor configured to perform a method implementing any of the embodiments described above based on instructions stored by the memory.
According to a sixth aspect of the embodiments of the present disclosure, there is provided an autonomous vehicle comprising: an electronic device as in any of the above embodiments; an image acquisition device configured to acquire image frames.
According to a seventh aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer instructions are stored, and when executed by a processor, the computer-readable storage medium implements the method according to any of the embodiments described above.
Other features of the present disclosure and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which proceeds with reference to the accompanying drawings.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a schematic flow chart diagram of a database construction method according to an embodiment of the present disclosure;
FIG. 2 is a schematic structural diagram of a database construction apparatus according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure;
fig. 4 is a schematic flow chart of an obstacle identification method according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an obstacle recognition device according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an electronic device according to another embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of an autonomous vehicle according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
The relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Fig. 1 is a schematic flow chart of a database construction method according to an embodiment of the present disclosure. In some embodiments, the following database construction method steps are performed by a database construction apparatus.
In step 101, a plurality of first image frames acquired by an image acquisition device on a preset route are received.
In some embodiments, the image capturing device on the data capturing cart is provided with various sensors, such as a GPS (Global Positioning System)/IMU (Inertial Measurement Unit) Positioning sensor, a lidar, a camera, a millimeter wave radar, and other sensing sensors.
In step 102, the dynamic obstacles in the plurality of first image frames are removed to obtain a plurality of second image frames including static obstacles.
It should be noted that, in order to improve the recognition accuracy, the data acquisition vehicle may perform multiple image acquisitions on a preset route so as to acquire data from different directions and angles. For example, a flower bed is located near a building, and a person standing at the side of the flower bed makes a call. The person leaves after the call is completed. By comparing the features of the image frames associated with the building, it can be found that the flower bed is a static obstacle without movement ability, while the person is a dynamic obstacle with movement ability. By removing dynamic obstacles with motion ability in the image frame, static obstacles without motion ability are remained in the image frame.
In step 103, a feature vector for each second image frame is extracted.
In some embodiments, the feature vector has dimensions of 64 dimensions, 128 dimensions, or 256 dimensions. The higher the dimensionality of the feature vector, the stronger the characterization capability for the image frame.
In some embodiments, in order to reduce subsequent search and matching time, a plurality of key frames are sequentially extracted from a plurality of second image frames, wherein the image overlapping rate of any two adjacent first key frames and second key frames is smaller than a preset threshold value. And then extracting the feature vector of each key frame, thereby effectively reducing the number of the feature vectors. The other image frames of the plurality of second image frames are not subjected to the feature vector extraction process.
In some embodiments, the difference between the position information of the image capture device corresponding to the first keyframe and the position information of the image capture device corresponding to the second keyframe is greater than a preset distance threshold.
For example, if there is no overlap between the camera images of the respective fields of view at time t and the camera images of the respective fields of view at time t + N, the camera images of the respective fields of view at time t and the camera images of the respective fields of view at time t + N are used as key frames. While the images acquired at times (t +1, t +2, …, t + (N-1)) between time t and time t + N are not key frames.
In some embodiments, a deviation between the pose information of the image capture device corresponding to the first keyframe and the pose information of the image capture device corresponding to the second keyframe is greater than a preset angle threshold.
For example, if the attitude angle of the capturing vehicle at time t is E1 and the attitude angle at time t + M is EM, and if the change between EM and E1 exceeds a certain angle (for example, 5 degrees), the camera images of the respective fields of view at time t and the camera images of the respective fields of view at time t + M are used as key frames. While the images acquired at times (t +1, t +2, …, t + (M-1)) between time t and time t + M are not key frames.
In step 104, keypoints and description vectors corresponding to the keypoints are extracted from each second image frame.
Here, it should be noted that the Feature vectors of the key points and the key points can be estimated from the image by using SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), FAST (Features from Accelerated Segment Test Features) key point detector, key word descriptors such as SIFT, SURF, BRIEF (Binary Robust Invariant Feature), ORB (Oriented FAST and Rotated BRIEF), etc., or by using CNN (Convolutional Neural ie) method. The key points and the key point feature vectors are extracted from the image frames for matching with the key point feature vectors in the real-time image, so that static obstacles appearing in the image frames are found in the real-time image.
In some embodiments, if feature vectors are extracted from the keyframes in step 103, the keypoints and description vectors associated with the keypoints are extracted from each keyframe accordingly.
In step 105, each second image frame, the positioning information of the image capturing device corresponding to each second image frame, the feature vector, the keypoint, and the description vector corresponding to the keypoint are stored in a database.
In some embodiments, the positioning information includes position information for 3 degrees of freedom and pose information for 3 degrees of freedom.
In some embodiments, if feature vectors are extracted from the key frames in step 103, each key frame is stored in the database accordingly.
In the database construction method provided by the above embodiment of the present disclosure, the corresponding image frame, the positioning information of the image capturing device corresponding to the image frame, the feature vector of the image frame, the key point, and the description vector corresponding to the key point are stored in the database, so as to identify the static obstacle in the real-time video frame by using the database.
Fig. 2 is a schematic structural diagram of a database construction apparatus according to an embodiment of the present disclosure. As shown in fig. 2, the database construction apparatus includes a first receiving module 21, an image frame processing module 22, a first extraction module 23, a second extraction module 24, and a construction module 25.
The first receiving module 21 is configured to receive a plurality of first image frames acquired by the image acquisition device on a preset route.
The image frame processing module 22 is configured to remove dynamic obstacles in the plurality of first image frames to obtain a plurality of second image frames including static obstacles.
The first extraction module 23 is configured to extract a feature vector of each second image frame.
In some embodiments, in order to reduce the subsequent searching and matching time, the first extraction module 23 sequentially extracts a plurality of key frames from the plurality of second image frames, wherein the image overlapping rate of any two adjacent first key frames and second key frames is smaller than a preset threshold. The first extraction module 23 then extracts the feature vectors of each key frame, thereby effectively reducing the number of feature vectors. The other image frames of the plurality of second image frames are not subjected to the feature vector extraction process.
In some embodiments, the difference between the position information of the image capture device corresponding to the first keyframe and the position information of the image capture device corresponding to the second keyframe is greater than a preset distance threshold.
In some embodiments, a deviation between the pose information of the image capture device corresponding to the first keyframe and the pose information of the image capture device corresponding to the second keyframe is greater than a preset angle threshold.
The second extraction module 24 is configured to extract from each second image frame keypoints and description vectors corresponding to the keypoints.
The construction module 25 is configured to store each second image frame, the positioning information of the image acquisition device corresponding to each second image frame, the feature vector, the keypoint and the description vector corresponding to the keypoint in the database.
Fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 3, the electronic device includes a memory 31 and a processor 32.
The memory 31 is used for storing instructions, the processor 32 is coupled to the memory 31, and the processor 32 is configured to execute the method according to any embodiment in fig. 1 based on the instructions stored in the memory.
As shown in fig. 3, the electronic device further comprises a communication interface 33 for information interaction with other devices. Meanwhile, the electronic device further comprises a bus 34, and the processor 32, the communication interface 33 and the memory 31 are communicated with each other through the bus 34.
The memory 31 may comprise a high-speed RAM memory, and may also include a non-volatile memory (e.g., at least one disk memory). The memory 31 may also be a memory array. The storage 31 may also be partitioned and the blocks may be combined into virtual volumes according to certain rules.
Further, the processor 32 may be a central processing unit CPU, or may be an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement embodiments of the present disclosure.
The present disclosure also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the instructions, when executed by a processor, implement the method according to any one of the embodiments in fig. 1.
Fig. 4 is a flowchart illustrating an obstacle identification method according to an embodiment of the present disclosure. In some embodiments, the following obstacle identification method steps are performed by an obstacle identification device.
In step 401, a real-time image frame acquired by an image acquisition device is received.
In step 402, a database is searched for a predetermined number of matching image frames that match the real-time image frame.
It should be noted here that the database is a database constructed by any one of the embodiments in fig. 1.
In some embodiments, an image frame corresponding to a preset region centered on the position information is extracted from the database as a candidate image frame according to the position information of the image capturing device corresponding to the real-time image frame. And extracting the feature vector of the candidate image frame from the database, and extracting the feature vector of the real-time image frame. And matching the feature vectors of the real-time image frames with the feature vectors of the candidate image frames, and taking a preset number of candidate image frames with the highest matching degree as matched image frames.
For example, the top K candidate images with the highest degree of matching are taken as matching image frames. The value of K can be 1, 3 or 5. If higher matching accuracy is required, the value of K may be larger. If a faster online matching and processing speed is required, the value of K may be smaller.
In step 403, obstacle detection is performed in the real-time image frames to determine obstacle regions.
In step 404, keypoints and corresponding keypoint description vectors in the obstacle region are extracted.
In step 405, keypoint description vectors of real-time image frames are matched with keypoint description vectors of a predetermined number of matching image frames to obtain matching keypoints.
In some embodiments, keypoint description vector matching is achieved by using a Brute Force (Brute Force) matcher, a FLANN (Fast approximation Neighbor Search Library) based matcher, or a machine learning approach using CNN.
In step 406, if the number of matching keypoints is greater than a preset keypoint threshold, the obstacle region is marked as a static obstacle.
That is, an obstacle area is marked as a static obstacle if the number of matching keypoints included in the obstacle area exceeds a predetermined threshold.
In some embodiments, the obstacle region is marked as a dynamic obstacle if the number of matching keypoints is not greater than a preset keypoint threshold.
Fig. 5 is a schematic structural diagram of an obstacle identification device according to an embodiment of the present disclosure. As shown in fig. 5, the obstacle identifying apparatus includes a second receiving module 51, a searching module 52, a detecting module 53, a third extracting module 54, a matching module 55, and an identifying module 56.
The second receiving module 51 is configured to receive real-time image frames acquired by the image acquisition device.
The search module 52 is configured to search the database for a predetermined number of matching image frames that match the real-time image frames.
It should be noted here that the database is a database constructed by any one of the embodiments in fig. 1.
In some embodiments, the search module 52 extracts an image frame corresponding to a preset region centered on the position information from the database as a candidate image frame according to the position information of the image capturing device corresponding to the real-time image frame. And extracting the feature vector of the candidate image frame from the database, and extracting the feature vector of the real-time image frame. And matching the feature vectors of the real-time image frames with the feature vectors of the candidate image frames, and taking a preset number of candidate image frames with the highest matching degree as matched image frames.
For example, the top K candidate images with the highest degree of matching are taken as matching image frames. The value of K can be 1, 3 or 5. If higher matching accuracy is required, the value of K may be larger. If a faster online matching and processing speed is required, the value of K may be smaller.
The detection module 53 is configured to perform obstacle detection in real-time image frames to determine obstacle regions.
The third extraction module 54 is configured to extract keypoints and corresponding keypoint description vectors in the obstacle region.
The matching module 55 is configured to match the keypoint description vectors of the real-time image frames with the keypoint description vectors of a predetermined number of matching image frames to obtain matching keypoints.
In some embodiments, identification module 56 marks the obstacle area as a dynamic obstacle if the number of matching keypoints is not greater than a preset keypoint threshold.
Fig. 6 is a schematic structural diagram of an electronic device according to another embodiment of the present disclosure. As shown in fig. 6, the electronic device includes a memory 601, a processor 602, a communication interface 603, and a bus 604. Fig. 6 differs from fig. 3 in that, in the embodiment shown in fig. 6, the processor 602 is configured to perform the method according to any of the embodiments in fig. 4 based on instructions stored in the memory.
The present disclosure also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the instructions, when executed by a processor, implement the method according to any one of the embodiments in fig. 4.
Fig. 7 is a schematic structural diagram of an autonomous vehicle according to an embodiment of the present disclosure. As shown in fig. 7, the autonomous vehicle includes an image pickup device 71 and an electronic device 72. The electronic device 72 is the electronic device related to any one of the embodiments of fig. 3 or fig. 6.
The image capturing device 71 is configured to capture image frames and transmit the captured image frames to the electronic device 72 as needed.
That is, in the embodiment shown in FIG. 7, two phases are included, where the first phase is an offline processing phase and the second phase is a real-time processing phase. In the first stage, the collected image frames are processed off line, so that the image frames and corresponding positioning information, feature vectors of the image frames, key points and corresponding description vectors are stored in a database. In the second stage, searching a database for a matched image frame matched with the real-time image frame, extracting key points and key point description vectors of the barrier region in the real-time image frame, matching the key point description vectors of the real-time image frame with the key point description vectors of the matched image frame to obtain matched key points, and if the number of the matched key points is greater than a threshold, marking the barrier region as a static barrier.
By implementing the scheme of the disclosure, the static barrier without movement capability in the acquired image can be accurately determined, so that the path and speed planning of the automatic driving vehicle is facilitated.
In some embodiments, the functional unit modules described above can be implemented as a general purpose Processor, a Programmable Logic Controller (PLC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable Logic device, discrete Gate or transistor Logic, discrete hardware components, or any suitable combination thereof for performing the functions described in this disclosure.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read memory, a magnetic disk or an optical disk.
The description of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to practitioners skilled in this art. The embodiment was chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.
Claims (9)
1. An obstacle identification method, comprising:
receiving a real-time image frame acquired by image acquisition equipment;
searching a database for a predetermined number of matching image frames that match the real-time image frames;
performing obstacle detection in the real-time image frames to determine an obstacle region;
extracting key points and corresponding key point description vectors in the obstacle region;
matching the key point description vectors of the real-time image frames with the key point description vectors of the preset number of matched image frames to obtain matched key points;
if the number of the matched key points is larger than a preset key point threshold, marking the obstacle area as a static obstacle;
during the process of constructing the database, receiving a plurality of first image frames acquired by an image acquisition device on a preset route;
removing dynamic obstacles in the plurality of first image frames to obtain a plurality of second image frames comprising static obstacles;
extracting a feature vector of each second image frame, wherein a plurality of key frames are sequentially extracted from the plurality of second image frames, the image overlapping rate of any two adjacent first key frames and second key frames is less than a preset threshold value, and the feature vector of each key frame is extracted;
extracting from each of the second image frames keypoints and description vectors corresponding to the extracted keypoints from the second image frames;
storing the each second image frame, the positioning information of the image acquisition device corresponding to the each second image frame, the feature vector, the extracted key points in the second image frame and the description vectors corresponding to the extracted key points from the second image frame in a database.
2. The method of claim 1, further comprising:
and if the number of the matched key points is not greater than a preset key point threshold, marking the obstacle area as a dynamic obstacle.
3. The method of claim 1, wherein searching the database for a predetermined number of matching image frames that match the real-time image frame comprises:
extracting an image frame corresponding to a preset area with the position information as the center from the database according to the position information of the image acquisition equipment corresponding to the real-time image frame to be used as a candidate image frame;
extracting feature vectors of the candidate image frames from the database;
extracting a feature vector of the real-time image frame;
and matching the feature vectors of the real-time image frames with the feature vectors of the candidate image frames, and taking a preset number of candidate image frames with the highest matching degree as matched image frames.
4. The method of claim 1, wherein,
the deviation between the position information of the image acquisition equipment corresponding to the first key frame and the position information of the image acquisition equipment corresponding to the second key frame is greater than a preset distance threshold; or
And the deviation between the attitude information of the image acquisition equipment corresponding to the first key frame and the attitude information of the image acquisition equipment corresponding to the second key frame is greater than a preset angle threshold.
5. The method of claim 1, wherein extracting keypoints from the each second image frame and description vectors associated with the extracted keypoints from the second image frames comprises:
from each of the key frames, a key point and a description vector associated with the key point extracted from the key frame are extracted.
6. An obstacle recognition device comprising:
a database building apparatus comprising:
the image acquisition device comprises a first receiving module, a second receiving module and a display module, wherein the first receiving module is configured to receive a plurality of first image frames acquired by the image acquisition device on a preset route;
an image frame processing module configured to remove dynamic obstacles in the plurality of first image frames to obtain a plurality of second image frames including static obstacles;
the first extraction module is configured to extract a feature vector of each second image frame, wherein a plurality of key frames are sequentially extracted from the plurality of second image frames, the image overlapping rate of any two adjacent first key frames and the second key frames is smaller than a preset threshold value, and the feature vector of each key frame is extracted;
a second extraction module configured to extract a keypoint and a description vector corresponding to the keypoint from each of the second image frames;
a construction module configured to store the each second image frame, the positioning information of the image acquisition device corresponding to the each second image frame, the feature vector, the key point, and the description vector corresponding to the key point in a database;
the second receiving module is configured to receive the real-time image frame acquired by the image acquisition device;
a search module configured to search the database for a predetermined number of matching image frames that match the real-time image frames;
a detection module configured to perform obstacle detection in the real-time image frames to determine an obstacle region;
a third extraction module configured to extract key points and corresponding key point description vectors in the obstacle region;
a matching module configured to match the keypoint description vectors of the real-time image frames with the keypoint description vectors of the predetermined number of matched image frames to obtain matched keypoints;
an identification module configured to mark the obstacle area as a static obstacle if the number of matching keypoints is greater than a preset keypoint threshold.
7. An electronic device, comprising:
a memory configured to store instructions;
a processor coupled to the memory, the processor configured to perform implementing the method of any of claims 1-5 based on instructions stored by the memory.
8. An autonomous vehicle comprising:
the electronic device of claim 7;
an image acquisition device configured to acquire image frames.
9. A computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions which, when executed by a processor, implement the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010717031.4A CN111858979B (en) | 2020-07-23 | 2020-07-23 | Obstacle recognition method and device, electronic equipment and automatic driving vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010717031.4A CN111858979B (en) | 2020-07-23 | 2020-07-23 | Obstacle recognition method and device, electronic equipment and automatic driving vehicle |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111858979A CN111858979A (en) | 2020-10-30 |
| CN111858979B true CN111858979B (en) | 2021-12-14 |
Family
ID=72950337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010717031.4A Active CN111858979B (en) | 2020-07-23 | 2020-07-23 | Obstacle recognition method and device, electronic equipment and automatic driving vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858979B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105015419A (en) * | 2015-07-17 | 2015-11-04 | 中山大学 | Automatic parking system and method based on stereoscopic vision localization and mapping |
| CN110046677A (en) * | 2019-04-26 | 2019-07-23 | 山东大学 | Data preprocessing method, map constructing method, winding detection method and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106407315B (en) * | 2016-08-30 | 2019-08-16 | 长安大学 | A kind of vehicle autonomic positioning method based on street view image database |
| CN109146938B (en) * | 2018-08-30 | 2020-09-15 | 百度在线网络技术（北京）有限公司 | Method, device and equipment for calibrating position of dynamic obstacle and storage medium |
| CN109726716A (en) * | 2018-12-29 | 2019-05-07 | 深圳市趣创科技有限公司 | A kind of image processing method and system |
| CN111402326B (en) * | 2020-03-13 | 2023-08-25 | 北京百度网讯科技有限公司 | Obstacle detection method, obstacle detection device, unmanned vehicle and storage medium |
| CN111429528A (en) * | 2020-04-07 | 2020-07-17 | 高深智图(广州)科技有限公司 | Large-scale distributed high-precision map data processing system |
-
2020
- 2020-07-23 CN CN202010717031.4A patent/CN111858979B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105015419A (en) * | 2015-07-17 | 2015-11-04 | 中山大学 | Automatic parking system and method based on stereoscopic vision localization and mapping |
| CN110046677A (en) * | 2019-04-26 | 2019-07-23 | 山东大学 | Data preprocessing method, map constructing method, winding detection method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111858979A (en) | 2020-10-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109960742B (en) | Local information searching method and device | |
| Moranduzzo et al. | A SIFT-SVM method for detecting cars in UAV images | |
| CN111639653B (en) | False detection image determining method, device, equipment and medium | |
| CN109858552B (en) | Target detection method and device for fine-grained classification | |
| Nguyen et al. | Object detection using scale invariant feature transform | |
| Vishal et al. | Accurate localization by fusing images and GPS signals | |
| CN106447697A (en) | Specific moving target fast tracking method based on moving platform | |
| CN111428539A (en) | Target tracking method and device | |
| Araujo et al. | Air-sslam: A visual stereo indoor slam for aerial quadrotors | |
| Chen et al. | Improved fast r-cnn with fusion of optical and 3d data for robust palm tree detection in high resolution uav images | |
| CN111858979B (en) | Obstacle recognition method and device, electronic equipment and automatic driving vehicle | |
| Alam et al. | A vision-based system for traffic light detection | |
| CN112598714A (en) | Static target tracking method based on video frame homography transformation | |
| KR101733288B1 (en) | Object Detecter Generation Method Using Direction Information, Object Detection Method and Apparatus using the same | |
| Orhan et al. | Semantic pose verification for outdoor visual localization with self-supervised contrastive learning | |
| Li et al. | Road-network-based fast geolocalization | |
| Qiao et al. | Place recognition based visual localization using LBP feature and SVM | |
| CN113743359B (en) | Vehicle re-identification method, model training method and related devices | |
| CN111008210B (en) | Commodity identification method, commodity identification device, codec and storage device | |
| Wu et al. | Feature-based background registration in wide-area motion imagery | |
| Wachs et al. | Recognizing Human Postures and Poses in Monocular Still Images. | |
| CN114494355A (en) | Trajectory analysis method and device based on artificial intelligence, terminal equipment and medium | |
| Kumar et al. | Vehicle detection and classification from satellite images based on gaussian mixture model | |
| Sykora et al. | Comparison of Neural Networks with Feature Extraction Methods for Depth Map Classification | |
| Song et al. | Learning Local Feature Descriptor with Motion Attribute For Vision-based Localization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |