CN111368883B - Obstacle avoidance method based on monocular camera, computing device and storage device - Google Patents
Obstacle avoidance method based on monocular camera, computing device and storage device Download PDFInfo
- Publication number
- CN111368883B CN111368883B CN202010109096.0A CN202010109096A CN111368883B CN 111368883 B CN111368883 B CN 111368883B CN 202010109096 A CN202010109096 A CN 202010109096A CN 111368883 B CN111368883 B CN 111368883B
- Authority
- CN
- China
- Prior art keywords
- image
- collision time
- current image
- optical flow
- feature point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 230000003287 optical effect Effects 0.000 claims abstract description 70
- 238000012545 processing Methods 0.000 claims abstract description 38
- 239000013598 vector Substances 0.000 claims abstract description 33
- 230000033001 locomotion Effects 0.000 claims description 16
- 238000001914 filtration Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 12
- 230000000877 morphologic effect Effects 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 5
- 230000000903 blocking effect Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 5
- 238000005286 illumination Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/757—Matching configurations of points or features
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S11/00—Systems for determining distance or velocity not using reflection or reradiation
- G01S11/12—Systems for determining distance or velocity not using reflection or reradiation using electromagnetic waves other than radio waves
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Electromagnetism (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an obstacle avoidance method, a computing device and a storage device based on a monocular camera, wherein the obstacle avoidance method comprises the following steps: detecting and extracting feature points of the current image to obtain a feature point set of the current image; performing inter-frame feature point matching on the current image and the previous frame image, and calculating an optical flow vector field according to a matching result; calculating an expansion center point and a first collision time from the optical flow vector field; dividing the current image into blocks to obtain image blocks, and calculating the average collision time of all the characteristic points contained in each image block and the robot to collide according to the first collision time, and marking the average collision time as second collision time; and sequentially performing binarization processing and connectivity processing on each image according to the second collision time, and generating an obstacle map according to the processing result. By the method, the obstacle map of the dense optical flow method and the real-time operation advantage of the sparse optical flow method can be combined, and the size and the position information of the obstacle can be fed back.
Description
Technical Field
The application relates to a visual navigation neighborhood, in particular to an obstacle avoidance method based on a monocular camera, a computing device and a storage device.
Background
Nowadays, robots play an increasingly important role in both civilian use and military use and commercial use, and the most basic requirement for robots is to have autonomous navigation functions, which are based on obstacle avoidance algorithms. The robot is required to acquire surrounding environment information through a sensor in an unknown or partially unknown environment. At present, the robot obstacle avoidance method is mainly based on the following three types of sensors: infrared/ultrasonic sensors, lidar/TOF sensors, and vision sensors.
The infrared/ultrasonic sensor is easy to be interfered by noise although the cost is lower, and the number of detection points is less; the cost of the laser radar/TOF sensor is too high, and the obstacle avoidance does not need very high sensor precision, so that the sensor performance is wasted; the visual sensor can sense rich scene information, and along with the development of deep learning technology, intelligent development based on the visual sensor becomes a research hotspot in the field of robots. Unlike binocular sensors, which can provide depth information of a scene, obstacle avoidance methods based on monocular vision sensors are more challenging.
The existing monocular obstacle avoidance algorithm is mainly divided into three main categories: 1. a method based on visual features of a scene; 2. a method based on inter-frame correlation feature points; 3. a method based on image homography transformation.
The method based on the visual characteristics of the scene mainly utilizes information such as color, edge contour and the like to detect the obstacle. The method based on the color information mainly utilizes the known ground template to carry out template matching on the whole scene according to various color characteristics of the ground template, and the area with consistent matching results is the ground; the method based on the edge contour mainly utilizes the graphic filtering idea, finds the intersection point of the horizontal line and the vertical line according to the contour geometrical characteristics of the scene and is sequentially connected to obtain a ground area and an obstacle area above the ground surface.
The method based on the inter-frame correlation characteristic points is mainly an optical flow method, and 3-dimensional information acquisition and depth and collision time estimation can be simply carried out by calculating the optical flow vector convergence center and the optical flow vector difference of the inter-frame correlation points. Optical flow methods are divided into dense optical flow and sparse optical flow: the dense optical flow is generally formed by matching front and back frame images point by point, the obtained optical flow field is dense, the optical flow information is rich, but the calculated amount is large, and the real-time performance is poor; the sparse optical flow generally extracts and matches the feature points of the front frame and the rear frame, the optical flow field is sparse, the real-time performance is good, and the practical application is more.
The image homography-based method is to distinguish between ground points and non-ground points through homography transformation of an image pair, and generally needs to perform large-scale feature point extraction operation on points in a scene. The method mainly comprises the steps of solving an essential matrix by utilizing the frame pose change and epipolar constraint of a monocular camera according to the characteristic point matching result of front and rear frame images, further obtaining a ground area by plane fitting, and obtaining the rest part as a ground surface obstacle area.
On the one hand, the existing scheme has the problems that when the monocular camera performs pure rotation motion, the scale parameters cannot be initialized, on the other hand, the calculated amount of a dense optical flow field is too large, real-time calculation cannot be completed, the operation pressure of a robot platform is large, and characteristic points are difficult to extract when the ambient light is weak, so that inter-frame characteristic point matching is easily affected, and further the calculation of the optical flow field and the precision of the shortest collision time are affected; and only the feature points of the scene are extracted, the obstacle map is not constructed, and the position of the obstacle area and the size of the obstacle cannot be effectively judged.
Disclosure of Invention
The application provides an obstacle avoidance method, a computing device and a storage device based on a monocular camera, which can accurately describe the size and the position of an obstacle by combining the advantages of an obstacle map of a dense optical flow method and the real-time operation of a sparse optical flow method.
In order to solve the technical problems, one technical scheme adopted by the application is as follows: the utility model provides a keep away barrier method based on monocular camera, include: detecting and extracting feature points of a current image acquired by the monocular camera to obtain a feature point set of the current image;
performing inter-frame feature point matching on the feature point set of the current image and the feature point set of the previous frame image, and calculating an optical flow vector field according to a matching result;
calculating an expansion center point and a first collision time according to the optical flow vector field, wherein the first collision time is the shortest collision time of collision between each characteristic point and the robot;
the current image is subjected to blocking processing to obtain image blocks, the average collision time of all the characteristic points contained in each image block and the robot are calculated according to the first collision time, the average collision time is recorded as second collision time, and the second collision time is the shortest collision time of each image block and the robot;
and sequentially carrying out binarization processing and connectivity processing on each image according to the second collision time, and generating an obstacle map according to a processing result.
In order to solve the technical problems, another technical scheme adopted by the application is as follows: there is provided a computing device comprising: a processor, a memory coupled to the processor, wherein,
the memory stores program instructions for realizing the obstacle avoidance method based on the monocular camera;
the processor is used for executing the program instructions stored by the memory to execute a monocular camera-based obstacle avoidance method.
In order to solve the technical problem, a further technical scheme adopted by the application is as follows: the storage device is provided with a program file capable of realizing the obstacle avoidance method based on the monocular camera.
The beneficial effects of this application are: the image is segmented, the shortest collision time of each image block and the robot is taken as a threshold value, the image is binarized, connectivity judgment is carried out, an obstacle map after feature point clustering is obtained, and the size and position information of an obstacle can be fed back; the expansion center point is solved through an optical flow equation, the advantages of the obstacle map of the dense optical flow method and the real-time operation of the sparse optical flow method are combined, and the problem that the output result of the follow-up algorithm is incorrect due to the fact that the scale parameters cannot be initialized is solved.
Drawings
Fig. 1 is a schematic flow chart of a monocular camera-based obstacle avoidance method according to a first embodiment of the present invention;
fig. 2 is a schematic flow chart of a monocular camera-based obstacle avoidance method according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of an obstacle avoidance apparatus based on a monocular camera according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a computing device according to an embodiment of the invention;
fig. 5 is a schematic structural diagram of a memory device according to an embodiment of the present invention.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The terms "first," "second," "third," and the like in this application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first", "a second", and "a third" may explicitly or implicitly include at least one such feature. In the description of the present application, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise. All directional indications (such as up, down, left, right, front, back … …) in the embodiments of the present application are merely used to explain the relative positional relationship, movement, etc. between the components in a particular gesture (as shown in the drawings), and if the particular gesture changes, the directional indication changes accordingly. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
Fig. 1 is a schematic flow chart of an obstacle avoidance method based on a monocular camera according to a first embodiment of the present invention. It should be noted that, if there are substantially the same results, the method of the present invention is not limited to the flow sequence shown in fig. 1. As shown in fig. 1, the method comprises the steps of:
step S101: and detecting and extracting feature points of the current image acquired by the monocular camera to obtain a feature point set of the current image.
In step S101, a monocular camera is used to capture an image of an obstacle. In the embodiment, a Sift algorithm is adopted to detect and extract feature points of the image, and the extracted feature point set is stored. It should be understood that after extracting the feature points from each image, a corresponding feature point set is saved, and preparation is made for calling the extracted feature point set when the feature points are matched between frames in step S102.
Step S102: and carrying out inter-frame feature point matching on the feature point set of the current image and the feature point set of the previous frame image, and calculating an optical flow vector field according to a matching result.
In step S102, an optical flow vector field is calculated by using a pyramid L-K optical flow method, specifically, a gaussian pyramid is constructed for the current image and the previous frame image respectively, the image with the lowest resolution is placed on the top layer, the original image is placed on the bottom layer, the interframe optical flow value is calculated from the top layer, the optical flow result of the previous layer is used as the initial optical flow input of the next layer, and the interframe optical flow field of the original image is finally obtained through iteration layer by layer from top to bottom. In the embodiment, progressive calculation is performed on the optical flow field under different resolutions, so that the problem of too fast movement when the optical flow field between frames is directly solved for the highest resolution image is effectively solved, and the assumption condition that the optical flow method is that the inter-frame movement is small is met.
Step S103: an expansion center point and a first collision time, which is the shortest collision time when each feature point collides with the robot, are calculated from the optical flow vector field.
In step S103, the expansion center point is the convergence focus of the inter-frame optical flow vectors, and from the mathematical analysis perspective, the expansion center point is the least square solution of the optical flow field equation, from the geometric perspective, it is understood that the expansion center point is the intersection point of all the extension lines of the optical flow vectors, and meanwhile, the optical flow vectors represent the moving distance of the feature point under the pixel coordinates in the time period from the previous frame image to the current image. In this embodiment, step S103 is specifically as follows: obtaining an optical flow field equation according to the optical flow vector field and calculating a least square solution to obtain an expansion center point; calculating pixel distances between each characteristic point and the expansion center point; and calculating the first collision time according to the pixel distance, the optical flow vector of the previous frame image and the single frame motion time, wherein the single frame motion time is the time from the previous frame image to the current image.
The first collision time is calculated using the following formula: t is t i ＝l i ×Δt/flow i Wherein t is i For the shortest collision time of each characteristic point and the robot, l i For the pixel distance from each characteristic point to the expansion center point, deltat is the single frame movement time and flow i For the optical flow vector of the feature point i from the previous frame image to the current image, i is the feature point, i takes 1,2 … N.
Step S104: and carrying out blocking processing on the current image to obtain image blocks, and calculating the average collision time of all the characteristic points contained in each image block and the robot collision according to the first collision time, wherein the average collision time is recorded as second collision time, and the second collision time is the shortest collision time of each image block and the robot collision.
In step S104In the above, the current image is subjected to a blocking process to obtain image blocks, for example, the size of the current image block is 1080×1280 pixels, and the current image is subjected to an n×m blocking process, if N is 18 and M is 32, the blocking result is 1080/18=60, 1280/32=40, that is, the size of each image block is 60×40 pixels. Then, the average collision time, i.e., the second collision time, of all the feature points contained in the respective image blocks with the robot is calculated using the following formula,wherein ATTC is the average collision time, t, of all feature points contained in each image block with robot human collision i And q is the number of all the characteristic points contained in the image block, and is the shortest collision time for each characteristic point to collide with the robot.
Step S105: and sequentially performing binarization processing and connectivity processing on each image according to the second collision time, and generating an obstacle map according to the processing result.
In step S105, it includes: comparing the second collision time with a preset second threshold value, if the second collision time is larger than the preset second threshold value, assigning the image block to be in a first color, if the second collision time is smaller than the preset second threshold value, assigning the image block to be in a second color, wherein the preset second threshold value is the shortest collision time of each image block colliding with the robot, and the preset second threshold value is obtained according to the ratio of the preset shortest obstacle avoidance distance to the current movement speed of the robot; and performing connectivity judgment on the binarization processing result by adopting an eight-neighborhood method to obtain a connected domain with a second color, and generating an obstacle map.
Specifically, for example: the first color is black, the second color is white, and the average collision time ATTC and the preset second threshold t are used for min Is used as a standard to binarize each image block, e.g. ATTC > t min Is black, ATTC < t min The image block of the image is white, then connectivity judgment is carried out on the binarized image by adopting an eight-neighborhood method, and white connectivity areas with different sizes can be obtainedThe size of the connectivity area is the obstacle size.
According to the obstacle avoidance method based on the monocular camera, the image is segmented, the shortest collision time of each image block and the robot is taken as a threshold value, the image is binarized, connectivity judgment is carried out, an obstacle map after feature point clustering is obtained, and the size and position information of an obstacle can be fed back; the method combines the advantages of the obstacle map of the dense optical flow method and the real-time operation of the sparse optical flow method, solves the expansion center point through an optical flow equation, and the optical flow vectors obtained under the pure rotation condition are basically in parallel relation, so that the expansion center point of the optical flow vectors is at infinity, the calculated shortest collision time is infinity under the condition, and the result accords with the relative motion relation between the robot and the environment in the pure rotation process. Therefore, the output result of the method is still correct in pure rotation motion, and the problem that the output result of a subsequent algorithm is incorrect due to the fact that the scale parameters cannot be initialized is solved.
Fig. 2 is a flow chart of an obstacle avoidance method based on a monocular camera according to a second embodiment of the present invention. It should be noted that, if there are substantially the same results, the method of the present invention is not limited to the flow sequence shown in fig. 2. As shown in fig. 2, the method comprises the steps of:
step S201: the current image is input.
Step S202: it is determined whether the input image stream is empty.
In step S202, if yes, the algorithm flow is ended; if not, go to step S203.
Step S203: and carrying out morphological filtering processing and ground filtering processing on the current image.
The morphological filtering processing adopts morphological gradient to process the original image, expands and erodes the original image, and then makes difference to obtain an image after pixel enhancement in the field of key structural elements, thereby facilitating subsequent feature point extraction, solving the problem of image edge blurring in the weak illumination environment, and avoiding the problem of light flow field information deletion caused by insufficient feature point detection in the weak illumination environment.
The ground filtering processing adopts a dynamic template back projection histogram method based on an HSI color space to calculate a template image HSI three-channel histogram of each frame according to the ground template updated in real time, so that the problem of real-time change of the ground environment in the moving process of the robot is solved. Specifically, selecting a designated area of the lower half part of each frame of image as a ground dynamic template, carrying out three-channel histogram statistics on the template to obtain the value range of three HSI channels of the ground, and finally carrying out back projection on the whole image according to the value range of the three HSI channels of the dynamic template to obtain the image after ground filtering. The HSI color space is used in this embodiment because the RGB color space of the image is noisy at low light intensities, affecting subsequent data processing, while the HSI color space has independent light intensity channels whose H (hue) and S (saturation) are insensitive to I (intensity) variations.
Step S204: and detecting and extracting the characteristic points of the current image to obtain a characteristic point set of the current image.
In this embodiment, step S204 in fig. 2 is similar to step S101 in fig. 1, and is not described herein for brevity.
Step S205: and judging whether the current image is a first frame image or not.
In step S205, if the current image is the first frame image, waiting for the next frame image to be input; if the current image is not the first frame image, the number of frames is accumulated, and step S206 is performed.
Step S206: judging whether the accumulated frame number is smaller than a preset first threshold value.
In step S206, the preset first threshold is a frame number threshold, if the accumulated frame number is greater than or equal to the preset first threshold, the accumulated frame number is cleared, the next frame image is waited for input, and if the accumulated frame number is less than the preset first threshold, step S207 is executed.
Step S207: and carrying out inter-frame feature point matching on the feature point set of the current image and the feature point set of the previous frame image, and calculating an optical flow vector field according to a matching result.
In this embodiment, step S207 in fig. 2 is similar to step S102 in fig. 1, and is not described herein for brevity.
Step S208: an expansion center point and a first collision time, which is the shortest collision time when each feature point collides with the robot, are calculated from the optical flow vector field.
In this embodiment, step S208 in fig. 2 is similar to step S103 in fig. 1, and is not described here again for brevity.
Step S209: and carrying out blocking processing on the current image to obtain image blocks, and calculating the average collision time of all the characteristic points contained in each image block and the robot collision according to the first collision time, wherein the average collision time is recorded as second collision time, and the second collision time is the shortest collision time of each image block and the robot collision.
In this embodiment, step S209 in fig. 2 is similar to step S104 in fig. 1, and is not described here again for brevity.
Step S210: and sequentially performing binarization processing and connectivity processing on each image according to the second collision time, and generating an obstacle map according to the processing result.
In this embodiment, step S210 in fig. 2 is similar to step S105 in fig. 1, and is not described here again for brevity.
Step S211: judging whether the size of the white connected domain is smaller than a preset second threshold value.
In step S211, if yes, step S2111 is executed: no obstacle exists, and the robot continues to advance;
if not, step S2112 is executed: with an obstacle, the robot stops advancing and outputs position information of the obstacle.
According to the obstacle avoidance method based on the monocular camera, based on the first embodiment, the HSI color space of the original image is processed, the ground area updated in real time is adopted as a template, and ground filtering is realized by calculating the back projection histogram meeting the high and low threshold values of the HSI three channels, so that the obstacles and the ground area in the scene are separated, and the detection and judgment of the obstacles in the subsequent process are facilitated; meanwhile, morphological filtering processing of the image is adopted, the image is processed by using a morphological gradient operator, the contour edge in the image is highlighted, and the difficulty of extracting the characteristic points is reduced, so that the richness of sparse optical flow field information is ensured, the problem of image edge blurring in a weak illumination environment is solved, and the problem of optical flow field information deletion caused by insufficient detection of the characteristic points in the weak illumination environment is avoided.
Fig. 3 is a schematic structural diagram of an obstacle avoidance device based on a monocular camera according to an embodiment of the present invention. As shown in fig. 3, the apparatus 30 includes an extraction module 31, a first calculation module 32, a second calculation module 33, a block and calculation module 34, and a generation module 35.
The extraction module 31 is configured to detect and extract feature points of a current image acquired by the monocular camera, so as to obtain a feature point set of the current image.
The first calculating module 32 is coupled to the extracting module 31, and is configured to perform inter-frame feature point matching on the feature point set of the current image and the feature point set of the previous frame image, and calculate an optical flow vector field according to the matching result.
The second calculation module 33 is coupled to the first calculation module 32 for calculating an expansion center point and a first collision time from the optical flow vector field, the first collision time being a shortest collision time for each feature point to collide with the robot.
In this embodiment, knowing the optical flow vector of the previous frame image and the single frame motion time, the second calculation module 33 obtains an optical flow field equation according to the optical flow vector field and calculates a least square solution to obtain an expansion center point; calculating pixel distances between each characteristic point and the expansion center point; the first collision time is calculated from the pixel distance, the optical flow vector of the previous frame image, and the single frame motion time.
The first collision time is calculated using the following formula: t is t i ＝l i ×Δt/flow i Wherein t is i For the shortest collision time of each characteristic point and the robot, l i For the pixel distance from each characteristic point to the expansion center point, deltat is the single frame movement time and flow i For the optical flow vector of the feature point i from the previous frame image to the current image, i is the feature point, i takes 1,2 … N.
The segmentation and calculation module 34 is coupled to the second calculation module 33, and is configured to perform segmentation processing on the current image to obtain image blocks, calculate an average collision time of all feature points included in each image block with the robot according to the first collision time, and record the average collision time as a second collision time, where the second collision time is the shortest collision time of each image block with the robot.
In this embodiment, the current image is subjected to the blocking processing to obtain image blocks, for example, the size of the current image block is 1080×1280 pixels, the current image is subjected to the n×m blocking processing, if N is 18 and M is 32, the blocking result is 1080/18=60, 1280/32=40, that is, the size of each image block is 60×40 pixels. Then, the average collision time, i.e., the second collision time, of all the feature points contained in the respective image blocks with the robot is calculated using the following formula,wherein ATTC is the average collision time, t, of all feature points contained in each image block with robot human collision i And q is the number of all the characteristic points contained in the image block, and is the shortest collision time for each characteristic point to collide with the robot.
The generating module 35 is coupled to the blocking and calculating module 34, and is configured to sequentially perform binarization processing and connectivity processing on each image according to the second collision time, and generate an obstacle map according to the processing result.
In this embodiment, the generating module 35 compares the second collision time with a preset second threshold, if the second collision time is greater than the preset second threshold, the image block is assigned to the first color, if the second collision time is less than the preset second threshold, the image block is assigned to the second color, the preset second threshold is the shortest collision time of each image block colliding with the robot, and the preset second threshold is obtained according to the ratio of the preset shortest obstacle avoidance distance to the current movement speed of the robot; and performing connectivity judgment on the binarization processing result by adopting an eight-neighborhood method to obtain a connected domain with a second color, and generating an obstacle map.
In particular, the method comprises the steps of,for example: the first color is black, the second color is white, and the average collision time ATTC and the preset second threshold t are used for min Is used as a standard to binarize each image block, e.g. ATTC > t min Is black, ATTC < t min And then, performing connectivity judgment on the binarized image by adopting an eight-neighborhood method to obtain white connectivity areas with different sizes, wherein the size of the white connectivity areas is the size of the obstacle.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a computing device according to an embodiment of the invention. As shown in fig. 4, the computing device 40 includes a processor 41 and a memory 42 coupled to the processor 41.
The memory 42 stores program instructions for implementing the monocular camera-based obstacle avoidance method of any of the embodiments described above.
The processor 41 is configured to execute program instructions stored in the memory 42 to perform a monocular camera-based obstacle avoidance method.
The processor 41 may also be referred to as a CPU (Central Processing Unit ). The processor 41 may be an integrated circuit chip with signal processing capabilities. Processor 41 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a memory device according to an embodiment of the invention. The storage device according to the embodiment of the present invention stores a program file 51 capable of implementing all the methods described above, where the program file 51 may be stored in the storage device as a software product, and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage device includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes, or a terminal device such as a computer, a server, a mobile phone, a tablet, or the like.
In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The foregoing is only the embodiments of the present application, and is not intended to limit the scope of the patent application, and all equivalent structures or equivalent processes using the descriptions and the contents of the present application or other related technical fields are included in the scope of the patent application.
Claims (9)
1. An obstacle avoidance method based on a monocular camera, wherein the monocular camera is used for acquiring images of obstacles, and the method is characterized by comprising the following steps:
detecting and extracting feature points of a current image acquired by the monocular camera to obtain a feature point set of the current image;
performing inter-frame feature point matching on the feature point set of the current image and the feature point set of the previous frame image, and calculating an optical flow vector field according to a matching result;
calculating an expansion center point and a first collision time according to the optical flow vector field, wherein the first collision time is the shortest collision time of collision between each characteristic point and the robot;
carrying out N multiplied by M block processing on the current image to obtain image blocks, calculating average collision time of collision between all the characteristic points contained in each image block and the robot according to the first collision time, and marking the average collision time as second collision time, wherein N and M are positive integers larger than 1, and the second collision time is the shortest collision time of collision between each image block and the robot;
comparing the second collision time with a preset second threshold value, if the second collision time is larger than the preset second threshold value, assigning the image block to be a first color, and if the second collision time is smaller than the preset second threshold value, assigning the image block to be a second color;
and performing connectivity judgment on the binarization processing result by adopting an eight-neighborhood method to obtain the connected domain with the second color, and generating an obstacle map.
2. The method according to claim 1, wherein the step of performing feature point detection and extraction on the current image acquired by the monocular camera, before obtaining the feature point set of the current image, includes:
and carrying out morphological filtering processing and ground filtering processing on the current image.
3. The method of claim 2, wherein prior to morphological filtering and ground filtering the current image; comprising the following steps:
inputting the current image;
judging whether the input image stream is empty or not;
if yes, ending the algorithm flow;
and if not, carrying out morphological filtering processing and ground filtering processing on the current image.
4. The method of claim 1, wherein the matching the feature point set of the current image with the feature point set of the previous frame image includes calculating an optical flow vector field based on the matching result, including:
matching the characteristic point set of the current image with the characteristic point set of the previous frame image;
and calculating the optical flow vector field according to the matching result by adopting a pyramid L-K optical flow method.
5. The method of claim 1, wherein the step of performing inter-frame feature point matching between the feature point set of the current image and the feature point set of the previous frame image, and before calculating the optical flow vector field according to the matching result, comprises:
judging whether the current image is a first frame image or not;
if the current image is the first frame image, waiting for the input of the next frame image;
if the current image is not the first frame image, accumulating the frame number, and judging whether the accumulated frame number is smaller than a preset first threshold value or not;
and if the accumulated frame number is larger than or equal to the preset first threshold, clearing the accumulated frame number, waiting for the input of the next frame of image, and if the accumulated frame number is smaller than the preset first threshold, performing inter-frame feature point matching on the feature point set of the current image and the feature point set of the previous frame of image.
6. The method of claim 1, wherein the calculating an expansion center point and a first time of collision from the optical flow vector field comprises:
obtaining an optical flow field equation according to the optical flow vector field and calculating a least square solution to obtain the expansion center point;
calculating pixel distances between the characteristic points and the expansion center points;
calculating the first collision time, wherein the calculation formula of the first collision time is t i ＝l i ×Δt/flow i Wherein l i For the pixel distance, Δt is a single frame motion time, i is the feature point, and flow i For the optical flow vector of the feature point i from the previous frame image to the current image, the single frame motion time is the time from the previous frame image to the current image.
7. The method of claim 1, wherein after generating the obstacle map, comprising:
judging whether the size of the connected domain of the second color is smaller than a preset third threshold value or not;
if the robot is free of obstacles, the robot continues to move forward;
if not, the robot stops advancing and outputs the position information of the obstacle.
8. A computing device comprising a processor, a memory coupled to the processor, wherein,
the memory stores program instructions for implementing a monocular camera based obstacle avoidance method as claimed in any one of claims 1 to 7;
the processor is configured to execute the program instructions stored by the memory to perform the methods of claims 1-7.
9. A storage device, characterized in that a program file is stored which enables the obstacle avoidance method based on a monocular camera according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010109096.0A CN111368883B (en) | 2020-02-21 | 2020-02-21 | Obstacle avoidance method based on monocular camera, computing device and storage device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010109096.0A CN111368883B (en) | 2020-02-21 | 2020-02-21 | Obstacle avoidance method based on monocular camera, computing device and storage device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111368883A CN111368883A (en) | 2020-07-03 |
| CN111368883B true CN111368883B (en) | 2024-01-19 |
Family
ID=71210114
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010109096.0A Active CN111368883B (en) | 2020-02-21 | 2020-02-21 | Obstacle avoidance method based on monocular camera, computing device and storage device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111368883B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114782927B (en) * | 2022-06-21 | 2022-09-27 | 苏州魔视智能科技有限公司 | Obstacle detection method, obstacle detection device, electronic device, and storage medium |
| CN115431968B (en) * | 2022-11-07 | 2023-01-13 | 北京集度科技有限公司 | Vehicle controller, vehicle and vehicle control method |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007087203A (en) * | 2005-09-22 | 2007-04-05 | Sumitomo Electric Ind Ltd | Collision determination system, collision determination method, and computer program |
| WO2010060287A1 (en) * | 2008-11-27 | 2010-06-03 | 东软集团股份有限公司 | An obstacle detecting method based on monocular vision and the device thereof |
| JP2013020507A (en) * | 2011-07-12 | 2013-01-31 | Aisin Seiki Co Ltd | Vehicle detecting device, vehicle detecting method, and program |
| CN103325108A (en) * | 2013-05-27 | 2013-09-25 | 浙江大学 | Method for designing monocular vision odometer with light stream method and feature point matching method integrated |
| CN103743394A (en) * | 2014-01-07 | 2014-04-23 | 北京工业大学 | Light-stream-based obstacle avoiding method of mobile robot |
| CN105976399A (en) * | 2016-04-29 | 2016-09-28 | 北京航空航天大学 | Moving object detection method based on SIFT (Scale Invariant Feature Transform) feature matching |
| CN106933233A (en) * | 2015-12-30 | 2017-07-07 | 湖南基石信息技术有限公司 | A kind of unmanned plane obstacle avoidance system and method based on interval flow field |
| KR20170139906A (en) * | 2016-06-10 | 2017-12-20 | 현대자동차주식회사 | Apparatus for managing obstacle map using ultrasonic sensors and method thereof |
| CN108256418A (en) * | 2017-12-01 | 2018-07-06 | 轩辕智驾科技（深圳）有限公司 | A kind of pedestrian's method for early warning and system based on infrared imaging |
| CN108830257A (en) * | 2018-06-29 | 2018-11-16 | 电子科技大学 | A kind of potential obstacle detection method based on monocular light stream |
| US10275668B1 (en) * | 2015-09-21 | 2019-04-30 | Hrl Laboratories, Llc | System for collision detection and obstacle avoidance |
| CN110007686A (en) * | 2017-01-06 | 2019-07-12 | 极光飞行科学公司 | Anti-collision system and method for unmanned vehicle |
| CN110502019A (en) * | 2019-09-06 | 2019-11-26 | 北京云迹科技有限公司 | A kind of barrier-avoiding method and device of Indoor Robot |
| CN110673632A (en) * | 2019-09-27 | 2020-01-10 | 中国船舶重工集团公司第七0九研究所 | Unmanned aerial vehicle autonomous obstacle avoidance method and device based on visual SLAM |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070229538A1 (en) * | 2006-03-31 | 2007-10-04 | Research In Motion Limited | Methods and apparatus for dynamically labeling map objects in visually displayed maps of mobile communication devices |
| CN104299244B (en) * | 2014-09-26 | 2017-07-25 | 东软集团股份有限公司 | Obstacle detection method and device based on monocular camera |
| US10665115B2 (en) * | 2016-01-05 | 2020-05-26 | California Institute Of Technology | Controlling unmanned aerial vehicles to avoid obstacle collision |
| CN105718888B (en) * | 2016-01-22 | 2019-09-13 | 北京中科慧眼科技有限公司 | Barrier method for early warning and barrier prior-warning device |
-
2020
- 2020-02-21 CN CN202010109096.0A patent/CN111368883B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007087203A (en) * | 2005-09-22 | 2007-04-05 | Sumitomo Electric Ind Ltd | Collision determination system, collision determination method, and computer program |
| WO2010060287A1 (en) * | 2008-11-27 | 2010-06-03 | 东软集团股份有限公司 | An obstacle detecting method based on monocular vision and the device thereof |
| JP2013020507A (en) * | 2011-07-12 | 2013-01-31 | Aisin Seiki Co Ltd | Vehicle detecting device, vehicle detecting method, and program |
| CN103325108A (en) * | 2013-05-27 | 2013-09-25 | 浙江大学 | Method for designing monocular vision odometer with light stream method and feature point matching method integrated |
| CN103743394A (en) * | 2014-01-07 | 2014-04-23 | 北京工业大学 | Light-stream-based obstacle avoiding method of mobile robot |
| US10275668B1 (en) * | 2015-09-21 | 2019-04-30 | Hrl Laboratories, Llc | System for collision detection and obstacle avoidance |
| CN106933233A (en) * | 2015-12-30 | 2017-07-07 | 湖南基石信息技术有限公司 | A kind of unmanned plane obstacle avoidance system and method based on interval flow field |
| CN105976399A (en) * | 2016-04-29 | 2016-09-28 | 北京航空航天大学 | Moving object detection method based on SIFT (Scale Invariant Feature Transform) feature matching |
| KR20170139906A (en) * | 2016-06-10 | 2017-12-20 | 현대자동차주식회사 | Apparatus for managing obstacle map using ultrasonic sensors and method thereof |
| CN110007686A (en) * | 2017-01-06 | 2019-07-12 | 极光飞行科学公司 | Anti-collision system and method for unmanned vehicle |
| CN108256418A (en) * | 2017-12-01 | 2018-07-06 | 轩辕智驾科技（深圳）有限公司 | A kind of pedestrian's method for early warning and system based on infrared imaging |
| CN108830257A (en) * | 2018-06-29 | 2018-11-16 | 电子科技大学 | A kind of potential obstacle detection method based on monocular light stream |
| CN110502019A (en) * | 2019-09-06 | 2019-11-26 | 北京云迹科技有限公司 | A kind of barrier-avoiding method and device of Indoor Robot |
| CN110673632A (en) * | 2019-09-27 | 2020-01-10 | 中国船舶重工集团公司第七0九研究所 | Unmanned aerial vehicle autonomous obstacle avoidance method and device based on visual SLAM |
Non-Patent Citations (3)
| Title |
|---|
| On-Road Collision Warning Based on Multiple FOE Segmentation Using a Dashboard Camera;Mau-Tsuen Yang等;《IEEE TRANSACTIONS ON VEHICULAR TECHNOLOGY》;全文 * |
| 基于机器视觉的无人机避障技术研究;王新东;《中国优秀硕士学位论文全文数据库 信息科技辑》;全文 * |
| 戴碧霞.基于光流的微小型飞行器室内避障方法研究.《中国优秀硕士学位论文全文数据库 信息科技辑》.2015,正文第4部分. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111368883A (en) | 2020-07-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wu et al. | Lane-mark extraction for automobiles under complex conditions | |
| Zhou et al. | Moving object detection and segmentation in urban environments from a moving platform | |
| CN107767400B (en) | Remote sensing image sequence moving target detection method based on hierarchical significance analysis | |
| CN109086724B (en) | Accelerated human face detection method and storage medium | |
| US20130129148A1 (en) | Object detection device, object detection method, and program | |
| JP2013109760A (en) | Target detection method and target detection system | |
| Lee et al. | An intelligent depth-based obstacle detection system for visually-impaired aid applications | |
| CN112825192B (en) | Object identification system and method based on machine learning | |
| US9886769B1 (en) | Use of 3D depth map with low and high resolution 2D images for gesture recognition and object tracking systems | |
| CN104766065B (en) | Robustness foreground detection method based on various visual angles study | |
| WO2018227216A1 (en) | Learning-based matching for active stereo systems | |
| EP3665651B1 (en) | Hierarchical disparity hypothesis generation with slanted support windows | |
| Sun et al. | Fast motion object detection algorithm using complementary depth image on an RGB-D camera | |
| Hu et al. | Robust object tracking via multi-cue fusion | |
| CN111368883B (en) | Obstacle avoidance method based on monocular camera, computing device and storage device | |
| CN111382637A (en) | Pedestrian detection tracking method, device, terminal equipment and medium | |
| CN103077536B (en) | Space-time mutative scale moving target detecting method | |
| Petrovai et al. | Obstacle detection using stereovision for Android-based mobile devices | |
| CN112733678A (en) | Ranging method, ranging device, computer equipment and storage medium | |
| CN110599407B (en) | Human body noise reduction method and system based on multiple TOF cameras in downward inclination angle direction | |
| CN114766039A (en) | Object detection method, object detection device, terminal device, and medium | |
| CN115273138B (en) | Human body detection system and passenger flow camera | |
| CN115841572A (en) | Target tracking method, system, device, equipment and medium based on RGB-D image | |
| CN115294358A (en) | Feature point extraction method and device, computer equipment and readable storage medium | |
| CN109271854A (en) | Based on method for processing video frequency and device, video equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |