CN116740477B - Dynamic pixel point distribution identification method, system and equipment based on sparse optical flow - Google Patents
Dynamic pixel point distribution identification method, system and equipment based on sparse optical flow Download PDFInfo
- Publication number
- CN116740477B CN116740477B CN202311029115.9A CN202311029115A CN116740477B CN 116740477 B CN116740477 B CN 116740477B CN 202311029115 A CN202311029115 A CN 202311029115A CN 116740477 B CN116740477 B CN 116740477B
- Authority
- CN
- China
- Prior art keywords
- points
- current frame
- frame image
- dynamic
- class
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000009826 distribution Methods 0.000 title claims abstract description 74
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000003287 optical effect Effects 0.000 title claims abstract description 43
- 230000033001 locomotion Effects 0.000 claims abstract description 65
- 230000003068 static effect Effects 0.000 claims abstract description 48
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 22
- 230000004044 response Effects 0.000 claims description 36
- 238000000605 extraction Methods 0.000 claims description 24
- 239000013598 vector Substances 0.000 claims description 14
- 238000006073 displacement reaction Methods 0.000 claims description 13
- 230000008649 adaptation response Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 abstract description 10
- 238000013507 mapping Methods 0.000 description 5
- 230000004807 localization Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/269—Analysis of motion using gradient-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/513—Sparse representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
Abstract
The invention discloses a dynamic pixel point distribution identification method, a system and equipment based on sparse optical flow, and relates to the field of computer vision, wherein the method comprises the steps of acquiring a current frame image acquired by a camera carried on a robot, and extracting characteristics of the current frame image to obtain target characteristic points of the current frame image and descriptors corresponding to the target characteristic points; calculating the motion information of the target feature points by adopting a sparse optical flow algorithm, and determining the feature points successfully tracked according to the motion information of the target feature points and descriptors; according to the successfully tracked characteristic points and the dynamic distribution degree identifier, determining the characteristic points of the dynamic class and the characteristic points of the static class; and removing the dynamic region in the current frame image according to the region growing algorithm and the feature points of the dynamic class to obtain a static image corresponding to the current frame image. The invention solves the problems that the visual SLAM technology cannot run in real time and the use scene is limited in the existing dynamic environment.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a sparse optical flow-based dynamic pixel point distribution identification method, system and equipment.
Background
SLAM refers to Simultaneous Localization and Mapping, i.e., simultaneous localization and mapping. In real life, people need to accomplish daily life and work tasks by sensing environment, cognizing position and direction, etc., and robots also need to have such capabilities. SLAM technology is a technology that is created to provide robots with autonomous sensing and navigation capabilities. With the rapid development of the computer vision field, it has been found that high-precision localization and mapping can be achieved using vision sensors (e.g., monocular, binocular, or RGB-D cameras). Compared with other sensors, the visual sensor has the advantages of low cost, light weight, easy installation, convenient data processing and the like, so that the visual sensor is widely applied to the fields of robot navigation, unmanned operation, virtual reality and the like.
The visual SLAM technology is an SLAM technology for simultaneously positioning and mapping by utilizing a camera image, and specifically, the visual SLAM technology is a method for combining knowledge in the fields of image processing, computer vision, robot technology and the like, extracting, matching, optimizing and the like characteristic points in the image to obtain the position and the posture of a robot in a three-dimensional space, and constructing a three-dimensional map of an environment. In the visual SLAM technology under a dynamic environment, as dynamic elements such as moving objects or people exist in a scene, feature points in an image are changed or tracking is lost, so that the positioning precision and the map building precision of the visual SLAM technology are affected. Therefore, when using the visual SLAM technology in the dynamic environment, some methods are needed to identify the dynamic object, and at present, a method of estimating and predicting motion between adjacent frames is generally used to eliminate the interference of the dynamic object on positioning and mapping, such as an optical flow method, a dense matching method, and the like, so as to track and predict the moving object; deep learning methods typically use methods of object detection and semantic segmentation to separate dynamic objects from a static background. However, the method has large calculation amount and high requirement on calculation force, cannot achieve real-time performance, is limited in application scene, can be used only when the background area is larger than the foreground area, and is limited in multiple aspects, so that the method cannot be used in actual scenes.
Disclosure of Invention
The invention aims to provide a dynamic pixel point distribution identification method, a system and equipment based on sparse optical flow, which are used for solving the problems that the visual SLAM technology cannot run in real time and the use scene is limited in the existing dynamic environment.
In order to achieve the above object, the present invention provides the following solutions:
in a first aspect, the present invention provides a method for identifying dynamic pixel point distribution based on sparse optical flow, including:
acquiring a current frame image acquired by a camera carried on a robot, and extracting features of the current frame image to obtain target feature points of the current frame image and descriptors corresponding to the target feature points; the target feature points are feature points of the self-adaptive response threshold;
calculating the motion information of the target feature points by adopting a sparse optical flow algorithm, and determining feature points successfully tracked according to the motion information of the target feature points and descriptors; the motion information comprises a motion speed, a motion direction and a displacement vector;
determining the characteristic points of the dynamic class and the characteristic points of the static class according to the successfully tracked characteristic points and the dynamic distribution degree identifier;
and removing the dynamic region in the current frame image according to a region growing algorithm and the feature points of the dynamic class to obtain a static image corresponding to the current frame image.
In a second aspect, the present invention provides a sparse optical flow-based dynamic pixel point distribution identification system, including:
the device comprises a feature information extraction module, a feature information extraction module and a feature extraction module, wherein the feature information extraction module is used for acquiring a current frame image acquired by a camera carried on a robot, and carrying out feature extraction on the current frame image to obtain a target feature point of the current frame image and a descriptor corresponding to the target feature point; the target feature points are feature points of the self-adaptive response threshold;
the feature point tracking module is used for calculating the motion information of the target feature points by adopting a sparse optical flow algorithm and determining the feature points successfully tracked according to the motion information of the target feature points and the descriptors; the motion information comprises a motion speed, a motion direction and a displacement vector;
the feature point classification module is used for determining feature points of dynamic classes and feature points of static classes according to the successfully tracked feature points and the dynamic distribution degree identifier;
and the static image determining module is used for removing the dynamic region in the current frame image according to a region growing algorithm and the feature points of the dynamic class to obtain a static image corresponding to the current frame image.
In a third aspect, the present invention provides an electronic device, including a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to execute the computer program to cause the electronic device to perform a sparse optical flow based dynamic pixel distribution identification method according to the first aspect.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention recognizes the feature points of the dynamic class and divides the dynamic area without sacrificing the feature point extraction speed, and finally obtains clean static data, thereby solving the problems that the visual SLAM technology can not run in real time and the use scene is limited in the existing dynamic environment.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a dynamic pixel point distribution identification method based on sparse optical flow according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a FAST corner extraction process provided in an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a dynamic pixel point distribution identification method based on sparse optical flow according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Example 1
The method is applied to a robot, and a camera mounted on the robot can collect images of continuous frames. As shown in fig. 1, the method includes:
step 101: acquiring a current frame image acquired by a camera carried on a robot, and extracting features of the current frame image to obtain target feature points of the current frame image and descriptors corresponding to the target feature points; the target feature points are feature points of the adaptive response threshold.
Step 102: calculating the motion information of the target feature points by adopting a sparse optical flow algorithm, and determining feature points successfully tracked according to the motion information of the target feature points and descriptors; the motion information includes a motion speed, a motion direction, and a displacement vector.
Step 103: and determining the characteristic points of the dynamic class and the characteristic points of the static class according to the successfully tracked characteristic points and the dynamic distribution degree identifier.
Step 104: and removing the dynamic region in the current frame image according to a region growing algorithm and the feature points of the dynamic class to obtain a static image corresponding to the current frame image.
As a preferred embodiment, step 101 specifically includes:
firstly, carrying out Gaussian blur processing on an input current frame image, and carrying out downsampling on the current frame image subjected to the Gaussian blur processing twice to generate a 1/2-scale image pyramid and a 1/4-scale image pyramid; secondly, performing FAST corner point extraction on the 1/2-scale image pyramid and the 1/4-scale image pyramid by adopting a FAST corner point detector to obtain FAST corner points corresponding to the current frame image; then, calculating a response value of each FAST corner corresponding to the current frame image, and determining a characteristic point of an adaptive response threshold corresponding to the current frame image according to the response value of each FAST corner corresponding to the current frame image; and finally, calculating descriptors corresponding to the characteristic points of the self-adaptive response threshold of the current frame image.
In this embodiment, determining, according to a response value of each FAST corner corresponding to a current frame image, a feature point of an adaptive response threshold corresponding to the current frame image specifically includes:
sorting FAST corner points corresponding to the current frame image according to the sequence from high to low of response values, calculating the difference value of response values of adjacent FAST corner points from the first FAST corner point after sorting, and taking the first n FAST corner points as characteristic points of the self-adaptive response threshold value when the difference value of the response value of the nth FAST corner point and the response value of the n+1th FAST corner point is larger than the set threshold value; the set threshold is thirty percent of the response value of the nth FAST corner, namely the set threshold is 0.3R of the nth FAST corner; r is a response value.
The FAST corner is defined as: if a pixel differs significantly from a sufficient number of pixels in its surrounding neighborhood, the pixel may be a FAST corner.
In this embodiment, a circle having a circumference of 16 pixels and a radius of 3 pixels is used to determine whether or not the center pixel is a FAST corner.
As shown in fig. 2, circumferential pixels are numbered in order from 1 to 16 in the clockwise direction on the circumference. Let the brightness value of the center pixel P be I p The threshold is t. If there are consecutive N pixels on the circumference whose luminance value is greater than the luminance value of the center pixel P (N is typically 8), or less than the luminance value of the center pixel P, the center pixel P is referred to as a FAST corner.
The detailed steps of FAST corner extraction are as follows:
1) Selecting a pixel point P, and assuming that the brightness of the pixel point P is I p 。
2) Setting the threshold t to 0.3I p And N is 8; n represents the number of continuous pixel points, 1, 2, 3, 5, 6, 7, 9, 10, 11, 13 and 14 pixel points on a circle with the radius of 3 pixel points are selected, and 16, 4, 8 and 12 pixel points do not need to be calculated, because the relevance between the 16, 4, 8 and 12 pixel points and adjacent continuous pixel points is large and is not representative; the pixel points of 3, 7 and 11 are selected for the corner points, and the reason is that the symmetry is only needed to be a pair of points.
If the brightness of 8 consecutive pixels on the selected circle is greater than 1.3I p Or less than 0.7I p Then pixel point P may be considered a FAST corner.
And respectively calculating absolute values of difference values of brightness values of 8 pixel points around the corner points and brightness of the corner points, and adding all the absolute values to obtain a response value R.
3) And calculating all pixel points in the image according to the method to obtain a group of FAST corner points.
Definition of BRIEF descriptor is: and comparing the gray value difference between 256 pairs of pixel points in any adjacent area of the circle center pixel point P to generate a binary descriptor. For example, with 3 pixel points as radii, gray values and arrangement order in the region are calculated, and descriptors are generated.
The calculation process comprises the following steps: any pair of pixel points p (x) and p (y), if the gray value I p（x） <Gray value I p（y） And the corresponding position of the descriptor is 1, otherwise, the descriptor is set to 0, and the result is used as one bit of binary coding to form a binary code with 256 bits in total.
In this embodiment, optical flow is used to evaluate the distortion between two images, the basic assumption being voxel and image pixel conservation, which assumes that the color of an object does not change significantly and significantly in the two frames before and after. Sparse optical flow algorithm: based on the image pyramid, calculating sparse optical flow vectors layer by layer, and optimizing in an iterative mode to finally obtain the corresponding characteristic point positions in the two images, and simultaneously calculating the motion direction and the motion speed.
As a preferred embodiment, the calculation of the motion information of the target feature points by using a sparse optical flow algorithm specifically includes:
1) The target feature points are used as points to be tracked, a sparse optical flow algorithm is adopted, and pixel points matched with the target feature points are found in the previous frame of image, specifically: a matching method based on the gray values of adjacent pixels is used, a small-range (9 multiplied by 9) area is selected around the target characteristic point in the current frame image by taking the target characteristic point as the center, and then the corresponding pixel point is searched in the previous frame image.
2) Calculating motion information of the pixel points matched with the target feature points, wherein the motion information is specifically as follows: for each successfully matched pixel point, the displacement vector, the motion speed and the motion direction between the successfully matched pixel points can be calculated.
Determining feature points successfully tracked according to the motion information and descriptors of the target feature points, wherein the method specifically comprises the following steps: determining target feature points meeting the first condition and the second condition as feature points successfully tracked; the first condition is that the displacement vector between the target feature point and the matched pixel point is smaller than a set displacement vector; the matched pixel points are pixel points matched with the target feature points; the set displacement vector is 30% of the average displacement vector; the second condition is that the similarity between the descriptors corresponding to the target feature points and the descriptors corresponding to the matched pixel points is larger than the set similarity; the set similarity is 80%.
In this embodiment, the distribution degree defines: the degree of aggregation of all feature point distributions in one image. The feature points on the dynamic object are dense, while the background feature points are more scattered and have wider positions, and the distribution degree can be used for distinguishing the dynamic object from the static object according to the characteristic. The method comprises the following steps: and calculating Manhattan distances between every two characteristic points, and calculating a mean value to represent the distribution degree.
As a preferred embodiment, step 103 specifically includes:
the characteristic points which are successfully tracked are classified in a self-adaptive mode according to the movement speed, and the distribution degree of each class is calculated, so that the characteristic points of the dynamic class and the characteristic points of the static class according to the movement speed are obtained, and the method specifically comprises the following steps: classifying the successfully tracked feature points according to the motion speed, calculating the maximum speed and the minimum speed, adaptively classifying according to the speed range, and calculating the distribution degree of each class to obtain the feature points of the dynamic class and the feature points of the static class.
The characteristic points which are successfully tracked are classified in a self-adaptive mode according to the motion direction, and the distribution degree of each class is calculated, so that the characteristic points of the dynamic class and the characteristic points of the static class according to the motion direction are obtained, and the method specifically comprises the following steps: classifying the tracked characteristic point pairs according to the motion direction, calculating a maximum angle and a minimum angle, adaptively classifying according to an angle range, and calculating the distribution degree of each class to obtain the characteristic points of the dynamic class and the characteristic points of the static class.
And solving intersection of the characteristic points of the dynamic class and the characteristic points of the static class based on the motion speed and the characteristic points of the dynamic class and the characteristic points of the static class based on the motion direction to obtain final characteristic points of the dynamic class and characteristic points of the static class.
And solving an intersection set of the dynamic and static classes of the two channels, wherein the obtained characteristic point set is the final dynamic and static class.
The distribution degree is represented by the average Manhattan distance between all the pairs of characteristic points in each class, the characteristic points of the dynamic object in the image are concentrated, the distribution degree is low, the characteristic points on the background are scattered, and the distribution degree is high.
The distribution degree is calculated by the following steps: and calculating Manhattan distances between every two feature points successfully tracked in each class, averaging all Manhattan distances in each class, and determining the average value as the distribution degree of the class.
According to the distribution degree, obtaining the characteristic points of the dynamic class and the characteristic points of the static class based on the motion direction, wherein the method specifically comprises the following steps:
sorting the distribution degrees determined by taking the moving direction as a basis according to the order from large to small, determining the characteristic points of the static class which are corresponding to the sorted first distribution degrees and are successfully tracked, and setting the difference between the sorted second distribution degrees and the sorted third distribution degrees as a judgment threshold d; when the difference between the n-th distribution degree after sequencing and the n+1-th distribution degree after sequencing is larger than a judging threshold value which is m times, determining the successfully tracked characteristic points in the class corresponding to the n-th distribution degrees after sequencing as the characteristic points of the static class, and determining the successfully tracked characteristic points in the class corresponding to the remaining distribution degrees as the characteristic points of the dynamic class; preferably, m is 1.5.
The principle of obtaining the characteristic points of the dynamic class and the characteristic points of the static class based on the motion direction according to the distribution degree is the same as the principle.
In this embodiment, the region growing algorithm is an image segmentation method based on pixel color or intensity similarity, and is mainly used for extracting regions or objects with similar features from images. The basic principle is to divide the pixels into different regions, each region containing characteristics of similar color or intensity values of the pixels. The algorithm starts from a seed pixel point, and gradually grows pixels according to a certain condition (a similarity threshold value 80%) by comparing the similarity between the characteristic values of the neighborhood pixels around the seed pixel and the characteristic values of the seed pixel until a complete area is formed. The steps of the region growing algorithm are as follows:
1) A seed pixel point is set as a starting point of the region.
2) And gradually adding surrounding similar pixel points into the region by taking the characteristic points in the dynamic class as seed points and taking the optical flow movement direction as a growth direction.
3) Step 2 is repeated until no new pixel points can be added to the area. And finally, dividing the dynamic region.
As a preferred embodiment, step 104 specifically includes:
and determining a growth boundary according to gradient similarity among pixels by taking the characteristic points of the dynamic class as seed points and the movement direction as the growth direction, identifying a dynamic region, filling a cavity and smoothing the edge, and finally obtaining a clean static image.
As shown in fig. 3, the present embodiment discloses a dynamic pixel point distribution recognition method based on sparse optical flow, firstly, continuous frame pictures are input into an ORB feature extraction module to extract feature points, and the sparse optical flow is used for tracking and calculating motion information of the feature points reaching an adaptive response threshold; then calculating the distribution degree of the feature points, classifying the feature points successfully tracked by taking the motion speed and the motion direction as the basis, and selecting dynamic and static classes by taking the self-adaptive distribution degree as a judgment standard; and finally, distinguishing a dynamic region by taking the characteristic points of the dynamic class as seed points of the region growing method and taking the optical flow direction of the points as the expansion direction, and finally obtaining clean static data. According to the sparse optical flow-based dynamic pixel point distribution identification method, the accuracy and the rapidity of an ORB feature extraction module, the motion information of the sparse optical flow, the guidance on feature point matching and the distribution degree characteristic of the dynamic points are utilized, so that clean static data is finally obtained, and the tracking and positioning accuracy and the robustness of an SLAM system are improved.
Compared with the prior art, the embodiment also has the following advantages:
1. and the selection of high-quality feature points reduces the extraction of unnecessary feature points and the calculation of descriptors.
2. And the characteristic points are selected and matched, so that the time for selecting and matching the characteristic points is reduced.
3. And (5) providing a characteristic point distribution degree concept and identifying characteristic points of the dynamic class.
4. And taking the dynamic characteristic points as seed points, and carrying out dynamic region identification and elimination to obtain clean static data.
Example two
In order to execute the corresponding method of the above embodiment to achieve the corresponding functions and technical effects, a dynamic pixel distribution recognition system based on sparse optical flow is provided below.
The embodiment provides a dynamic pixel point distribution identification system based on sparse optical flow, which comprises the following steps:
the device comprises a feature information extraction module, a feature information extraction module and a feature extraction module, wherein the feature information extraction module is used for acquiring a current frame image acquired by a camera carried on a robot, and carrying out feature extraction on the current frame image to obtain a target feature point of the current frame image and a descriptor corresponding to the target feature point; the target feature points are feature points of the adaptive response threshold.
The feature point tracking module is used for calculating the motion information of the target feature points by adopting a sparse optical flow algorithm and determining the feature points successfully tracked according to the motion information of the target feature points and the descriptors; the motion information includes a motion speed, a motion direction, and a displacement vector.
And the characteristic point classification module is used for determining the characteristic points of the dynamic class and the characteristic points of the static class according to the successfully tracked characteristic points and the dynamic distribution degree identifier.
And the static image determining module is used for removing the dynamic region in the current frame image according to a region growing algorithm and the feature points of the dynamic class to obtain a static image corresponding to the current frame image.
Example III
The embodiment of the invention provides an electronic device which comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor runs the computer program to enable the electronic device to execute the sparse optical flow-based dynamic pixel distribution identification method.
Alternatively, the electronic device may be a server.
In addition, the embodiment of the invention also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the dynamic pixel point distribution identification method based on the sparse optical flow in the first embodiment when being executed by a processor.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (8)
1. A dynamic pixel point distribution identification method based on sparse optical flow is characterized by comprising the following steps:
acquiring a current frame image acquired by a camera carried on a robot, and extracting features of the current frame image to obtain target feature points of the current frame image and descriptors corresponding to the target feature points; the target feature points are feature points of the self-adaptive response threshold;
calculating the motion information of the target feature points by adopting a sparse optical flow algorithm, and determining feature points successfully tracked according to the motion information of the target feature points and descriptors; the motion information comprises a motion speed, a motion direction and a displacement vector;
determining the characteristic points of the dynamic class and the characteristic points of the static class according to the successfully tracked characteristic points and the dynamic distribution degree identifier;
removing the dynamic region in the current frame image according to a region growing algorithm and the feature points of the dynamic class to obtain a static image corresponding to the current frame image;
extracting features of the current frame image to obtain target feature points of the current frame image and descriptors corresponding to the target feature points, wherein the method specifically comprises the following steps:
carrying out Gaussian blur processing on an input current frame image, and carrying out downsampling twice on the current frame image after the Gaussian blur processing to generate a 1/2-scale image pyramid and a 1/4-scale image pyramid;
performing FAST corner extraction on the 1/2-scale image pyramid and the 1/4-scale image pyramid by adopting a FAST corner detector to obtain FAST corners corresponding to the current frame image;
calculating a response value of each FAST corner corresponding to the current frame image, and determining a characteristic point of an adaptive response threshold corresponding to the current frame image according to the response value of each FAST corner corresponding to the current frame image;
calculating descriptors corresponding to the characteristic points of the self-adaptive response threshold of the current frame image;
determining a feature point of the adaptive response threshold corresponding to the current frame image according to the response value of each FAST corner corresponding to the current frame image, wherein the feature point specifically comprises:
sorting FAST corner points corresponding to the current frame image according to the sequence from high to low of response values, calculating the difference value of response values of adjacent FAST corner points from the first FAST corner point after sorting, and taking the first n FAST corner points as characteristic points of the self-adaptive response threshold value when the difference value of the response value of the nth FAST corner point and the response value of the n+1th FAST corner point is larger than the set threshold value; the set threshold is thirty percent of the response value of the nth FAST corner.
2. The sparse optical flow-based dynamic pixel point distribution identification method according to claim 1, wherein the calculation of the motion information of the target feature point by using a sparse optical flow algorithm comprises the following steps:
searching a pixel point matched with a target feature point in a previous frame image by adopting a sparse optical flow algorithm, and calculating motion information between the target feature point and the matched pixel point; the matched pixel points are pixel points matched with the target feature points.
3. The sparse optical flow-based dynamic pixel point distribution identification method according to claim 2, wherein the method is characterized by determining the feature points of successful tracking according to the motion information and descriptors of the target feature points, and specifically comprises the following steps:
determining target feature points meeting the first condition and the second condition as feature points successfully tracked; the first condition is that the displacement vector between the target feature point and the matched pixel point is smaller than a set displacement vector; and the second condition is that the similarity between the descriptors corresponding to the target feature points and the descriptors corresponding to the matched pixel points is larger than the set similarity.
4. The sparse optical flow-based dynamic pixel point distribution identification method according to claim 1, wherein the determining of the feature points of the dynamic class and the feature points of the static class according to the feature points successfully tracked and the dynamic distribution degree identifier specifically comprises:
performing self-adaptive classification on the characteristic points successfully tracked according to the motion speed, calculating the distribution degree of each class, and obtaining the characteristic points of the dynamic class and the characteristic points of the static class according to the distribution degree;
performing self-adaptive classification on the successfully tracked characteristic points according to the motion direction, calculating the distribution degree of each class, and obtaining the dynamic characteristic points and the static characteristic points according to the motion direction according to the distribution degree;
and solving intersection of the characteristic points of the dynamic class and the characteristic points of the static class based on the motion speed and the characteristic points of the dynamic class and the characteristic points of the static class based on the motion direction to obtain final characteristic points of the dynamic class and characteristic points of the static class.
5. The sparse optical flow-based dynamic pixel point distribution identification method of claim 4, wherein calculating the degree of distribution of each class specifically comprises:
calculating Manhattan distance between feature points which are successfully tracked pairwise in each class;
all manhattan distances in each class are averaged and the average is determined as the degree of distribution of the class.
6. The sparse optical flow-based dynamic pixel point distribution identification method according to claim 4, wherein the process of determining the feature points of the dynamic class and the feature points of the static class according to the distribution degree is as follows:
sorting the distribution degrees according to the sequence from large to small, determining the characteristic points of the static class which are corresponding to the sorted first distribution degrees and are successfully tracked as the characteristic points of the static class, and setting the difference between the sorted second distribution degrees and the sorted third distribution degrees as a judgment threshold;
when the difference between the n-th distribution degree after sequencing and the n+1-th distribution degree after sequencing is larger than a judging threshold value which is m times, determining the successfully tracked characteristic points in the class corresponding to the first n distribution degrees after sequencing as the characteristic points of the static class, and determining the successfully tracked characteristic points in the class corresponding to the rest distribution degrees as the characteristic points of the dynamic class.
7. A sparse optical flow based dynamic pixel point distribution recognition system, comprising:
the device comprises a feature information extraction module, a feature information extraction module and a feature extraction module, wherein the feature information extraction module is used for acquiring a current frame image acquired by a camera carried on a robot, and carrying out feature extraction on the current frame image to obtain a target feature point of the current frame image and a descriptor corresponding to the target feature point; the target feature points are feature points of the self-adaptive response threshold; extracting features of the current frame image to obtain target feature points of the current frame image and descriptors corresponding to the target feature points, wherein the method specifically comprises the following steps: carrying out Gaussian blur processing on an input current frame image, and carrying out downsampling twice on the current frame image after the Gaussian blur processing to generate a 1/2-scale image pyramid and a 1/4-scale image pyramid; performing FAST corner extraction on the 1/2-scale image pyramid and the 1/4-scale image pyramid by adopting a FAST corner detector to obtain FAST corners corresponding to the current frame image; calculating a response value of each FAST corner corresponding to the current frame image, and determining a characteristic point of an adaptive response threshold corresponding to the current frame image according to the response value of each FAST corner corresponding to the current frame image; calculating descriptors corresponding to the characteristic points of the self-adaptive response threshold of the current frame image; determining a feature point of the adaptive response threshold corresponding to the current frame image according to the response value of each FAST corner corresponding to the current frame image, wherein the feature point specifically comprises: sorting FAST corner points corresponding to the current frame image according to the sequence from high to low of response values, calculating the difference value of response values of adjacent FAST corner points from the first FAST corner point after sorting, and taking the first n FAST corner points as characteristic points of the self-adaptive response threshold value when the difference value of the response value of the nth FAST corner point and the response value of the n+1th FAST corner point is larger than the set threshold value; the set threshold is thirty percent of the response value of the nth FAST corner;
the feature point tracking module is used for calculating the motion information of the target feature points by adopting a sparse optical flow algorithm and determining the feature points successfully tracked according to the motion information of the target feature points and the descriptors; the motion information comprises a motion speed, a motion direction and a displacement vector;
the feature point classification module is used for determining feature points of dynamic classes and feature points of static classes according to the successfully tracked feature points and the dynamic distribution degree identifier;
and the static image determining module is used for removing the dynamic region in the current frame image according to a region growing algorithm and the feature points of the dynamic class to obtain a static image corresponding to the current frame image.
8. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform a sparse optical flow based dynamic pixel distribution identification method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311029115.9A CN116740477B (en) | 2023-08-16 | 2023-08-16 | Dynamic pixel point distribution identification method, system and equipment based on sparse optical flow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311029115.9A CN116740477B (en) | 2023-08-16 | 2023-08-16 | Dynamic pixel point distribution identification method, system and equipment based on sparse optical flow |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116740477A CN116740477A (en) | 2023-09-12 |
| CN116740477B true CN116740477B (en) | 2023-11-07 |
Family
ID=87903017
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311029115.9A Active CN116740477B (en) | 2023-08-16 | 2023-08-16 | Dynamic pixel point distribution identification method, system and equipment based on sparse optical flow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116740477B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10764561B1 (en) * | 2016-04-04 | 2020-09-01 | Compound Eye Inc | Passive stereo depth sensing |
| CN112926592A (en) * | 2021-04-13 | 2021-06-08 | 南京邮电大学 | Trademark retrieval method and device based on improved Fast algorithm |
| CN113744307A (en) * | 2021-08-06 | 2021-12-03 | 上海有个机器人有限公司 | Image feature point tracking method and system based on threshold dynamic adjustment |
| CN113947686A (en) * | 2021-08-06 | 2022-01-18 | 上海有个机器人有限公司 | Method and system for dynamically adjusting feature point extraction threshold of image |
| CN114782499A (en) * | 2022-04-28 | 2022-07-22 | 杭州电子科技大学 | Image static area extraction method and device based on optical flow and view geometric constraint |
| CN115830335A (en) * | 2022-10-25 | 2023-03-21 | 南京喂啊游通信科技有限公司 | ORB image feature extraction method based on adaptive threshold algorithm |
| CN116485885A (en) * | 2022-08-29 | 2023-07-25 | 江苏师范大学 | Method for removing dynamic feature points at front end of visual SLAM based on deep learning |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8798357B2 (en) * | 2012-07-09 | 2014-08-05 | Microsoft Corporation | Image-based localization |
| CN104754179B (en) * | 2013-12-31 | 2017-11-07 | 澜起科技（上海）有限公司 | The fully-automated synthesis method and system of static nature information in dynamic image |
-
2023
- 2023-08-16 CN CN202311029115.9A patent/CN116740477B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10764561B1 (en) * | 2016-04-04 | 2020-09-01 | Compound Eye Inc | Passive stereo depth sensing |
| CN112926592A (en) * | 2021-04-13 | 2021-06-08 | 南京邮电大学 | Trademark retrieval method and device based on improved Fast algorithm |
| CN113744307A (en) * | 2021-08-06 | 2021-12-03 | 上海有个机器人有限公司 | Image feature point tracking method and system based on threshold dynamic adjustment |
| CN113947686A (en) * | 2021-08-06 | 2022-01-18 | 上海有个机器人有限公司 | Method and system for dynamically adjusting feature point extraction threshold of image |
| CN114782499A (en) * | 2022-04-28 | 2022-07-22 | 杭州电子科技大学 | Image static area extraction method and device based on optical flow and view geometric constraint |
| CN116485885A (en) * | 2022-08-29 | 2023-07-25 | 江苏师范大学 | Method for removing dynamic feature points at front end of visual SLAM based on deep learning |
| CN115830335A (en) * | 2022-10-25 | 2023-03-21 | 南京喂啊游通信科技有限公司 | ORB image feature extraction method based on adaptive threshold algorithm |
Non-Patent Citations (2)
| Title |
|---|
| "Large Displacement Optical Flow Estimation Based on Robust Interpolation of Sparse Correspondences";S. Shi等;《IEEE Access》;第8卷;全文 * |
| "复杂场景下目标对象的分割与跟踪";朱青祥;《硕士电子期刊》;第2013年卷(第07期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116740477A (en) | 2023-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111060115B (en) | Visual SLAM method and system based on image edge features | |
| CN109559320B (en) | Method and system for realizing visual SLAM semantic mapping function based on hole convolution deep neural network | |
| Von Stumberg et al. | Gn-net: The gauss-newton loss for multi-weather relocalization | |
| CN109345588B (en) | Tag-based six-degree-of-freedom attitude estimation method | |
| Delmerico et al. | Building facade detection, segmentation, and parameter estimation for mobile robot localization and guidance | |
| CN109961506A (en) | A kind of fusion improves the local scene three-dimensional reconstruction method of Census figure | |
| CN110334762B (en) | Feature matching method based on quad tree combined with ORB and SIFT | |
| CN112396595B (en) | Semantic SLAM method based on point-line characteristics in dynamic environment | |
| CN109377499B (en) | Pixel-level object segmentation method and device | |
| CN113744315B (en) | Semi-direct vision odometer based on binocular vision | |
| CN112419497A (en) | Monocular vision-based SLAM method combining feature method and direct method | |
| CN109902576B (en) | Training method and application of head and shoulder image classifier | |
| CN109543634B (en) | Data processing method and device in positioning process, electronic equipment and storage medium | |
| Anthwal et al. | An overview of optical flow-based approaches for motion segmentation | |
| CN110738667A (en) | RGB-D SLAM method and system based on dynamic scene | |
| CN112686952A (en) | Image optical flow computing system, method and application | |
| Roy et al. | A comprehensive survey on computer vision based approaches for moving object detection | |
| CN114140527A (en) | Dynamic environment binocular vision SLAM method based on semantic segmentation | |
| CN117292076A (en) | Dynamic three-dimensional reconstruction method and system for local operation scene of engineering machinery | |
| CN111950599A (en) | Dense visual odometer method for fusing edge information in dynamic environment | |
| WO2023130842A1 (en) | Camera pose determining method and apparatus | |
| CN116740477B (en) | Dynamic pixel point distribution identification method, system and equipment based on sparse optical flow | |
| CN116643291A (en) | SLAM method for removing dynamic targets by combining vision and laser radar | |
| CN113837243A (en) | RGB-D camera dynamic visual odometer method based on edge information | |
| JP2023065296A (en) | Planar surface detection apparatus and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |