CN109816686A - Robot semanteme SLAM method, processor and robot based on object example match - Google Patents
Robot semanteme SLAM method, processor and robot based on object example match Download PDFInfo
- Publication number
- CN109816686A CN109816686A CN201910037102.3A CN201910037102A CN109816686A CN 109816686 A CN109816686 A CN 109816686A CN 201910037102 A CN201910037102 A CN 201910037102A CN 109816686 A CN109816686 A CN 109816686A
- Authority
- CN
- China
- Prior art keywords
- key frame
- feature
- frame
- robot
- match
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Image Analysis (AREA)
Abstract
Present disclose provides robot semanteme SLAM method, processor and robots based on object example match.Wherein, robot semanteme SLAM method carries out feature point extraction, matching and tracking to every frame image to estimate camera motion including obtaining the image sequence shot in robot operational process；Key frame is extracted, example segmentation is carried out to key frame, obtains all object examples in every frame key frame；Feature point extraction is carried out to key frame and calculates feature point description, feature extraction is carried out to all objects example in key frame and coding carrys out the feature description vectors of calculated examples, while obtaining example three-dimensional point cloud；Respectively to the characteristic point and object example progress Feature Points Matching and example match between adjacent key frame；The matching of fusion feature point and example match carry out local nonlinearity optimization to the pose estimated result of SLAM, obtain the key frame for carrying object example semantic markup information, and map that in example three-dimensional point cloud, construct three-dimensional semantic map.
Description
Technical field
The disclosure belongs to robot navigation's technical field more particularly to a kind of robot based on object example match is semantic
SLAM method, processor and robot.
Background technique
Only there is provided background technical informations relevant to the disclosure for the statement of this part, it is not necessary to so constitute first skill
Art.
In robot navigation field, synchronous superposition (Simultaneous Localization and
Mapping, SLAM) refer to: robot is arrived by repeated measures during the motion from the unknown place of circumstances not known
Environmental characteristic positions self-position and posture, further according to itself pose constructing environment map, to reach while position and map
The purpose of building.The purpose of SLAM technology mainly solves positioning and map structuring both of these problems, since proposition, rapid
The concern and research for having arrived numerous scholars are considered as realizing the key technology of full autonomous mobile robot all the time.Needle
To some specific robots (such as unmanned plane) due to itself can not carry odometer carry out location estimation and laser radar at
Originally the reasons such as higher, in recent years, SLAM (Visual SLAM, vSLAM) technology of view-based access control model has received widespread attention and grinds
Study carefully, vSLAM mainly includes that modules, the implementations such as visual odometry, rear end optimization, map structuring and closed loop detection mainly have spy
Sign point method and direct method；Data source mainly has monocular, binocular and RGB-D video flowing.
However tradition vSLAM algorithm dependent on the feature extracting and matching algorithm based on the low semantic hierarchies such as point, line, surface come
Estimate camera motion, not only lack semantic information, but also the robustness of characteristic matching is lower, easily causes biggish estimation and miss
Difference.The closed loop detection algorithm of tradition vSLAM has very strong viewpoint dependencies simultaneously, is easy to produce in complicated or duplicate environment
Raw biggish error rate.Semantic SLAM is allowed the robot to by the way that semantic information and vSLAM are carried out effective integration from geometry
Environment is perceived with two aspects of content, improves the service ability of robot and the intelligence of human-computer interaction.But invention human hair
Be mostly for the research work of semanteme SLAM at present now need known to threedimensional model as priori knowledge, or only to having
Several object categories of limit carry out semantic segmentation and the individual of indistinguishable object.
Summary of the invention
According to the one aspect of one or more other embodiments of the present disclosure, a kind of machine based on object example match is provided
Human speech justice SLAM method can be realized object individual in identification scene and construct three-dimensional semantic map, while again can be based on difference
Example match between key frame optimizes the estimated result of SLAM pose.
A kind of robot semanteme SLAM method based on object example match of the disclosure, comprising:
Obtain the image sequence that shoots in robot operational process, to every frame image carry out feature point extraction, matching and with
Track estimates camera motion；
Key frame is extracted, example segmentation is carried out to key frame, obtains all object examples in every frame key frame；
Feature point extraction is carried out to key frame and calculates feature point description, all objects example in key frame is carried out
Feature extraction and coding carry out the feature description vectors of calculated examples, while obtaining example three-dimensional point cloud；
According to feature point description and feature description vectors, respectively to the characteristic point and object example between adjacent key frame
Carry out Feature Points Matching and example match；
The matching of fusion feature point and example match carry out local nonlinearity optimization to the pose estimated result of SLAM, are taken
Key frame with object example semantic markup information；
The key frame for carrying object example semantic markup information is mapped in example three-dimensional point cloud, three-dimensional semanteme is constructed
Map.
In one or more embodiments, during estimating camera motion, consecutive frame is solved using light-stream adjustment
Camera motion:
First to consecutive frame image carry out ORB feature point extraction with match, obtain several pairs of ORB characteristic points of consecutive frame；
Then non-linear least square problem is constructed according to this several pairs of ORB characteristic points, solution obtains the pose of camera.
In one or more embodiments, during extracting key frame, the size of interframe relative motion distance is made
For the foundation for extracting the key frame in image sequence.
In one or more embodiments, if interframe relative motion distance between permission interframe minimum relative motion distance
Between maximum relative motion distance, then present frame is key frame.
In one or more embodiments, using example segmentation framework-Mask R-CNN network based on deep learning come
Example segmentation is carried out to key frame images, to obtain all examples in every frame key frame images；Wherein, Mask R-CNN network
It adds a Ge Quan convolutional neural networks branch on the basis of Faster R-CNN to be used to export example mask, thus to detection block
The profile of middle example carries out the segmentation of Pixel-level.
In one or more embodiments, the process of the feature description vectors of calculated examples includes:
Visual vocabulary table is constructed using training set based on VLAD algorithm, grid is carried out to every frame image in training set and is drawn
Point, dense SIFT feature and RGB color value are extracted to each grid element center, obtain the feature description vectors of each grid；
Obtained grid search-engine description vectors are clustered into the class of preset quantity using k-mean algorithm, and calculate each net
Lattice feature description vectors with its cluster in residual vector, the normalization of power rate and L2 norm normalizing are carried out to all residual vector
Change, then carry out feature coding using detection block image of the normalized residual vector to example, obtains the feature description of example
Vector.
According to the other side of one or more other embodiments of the present disclosure, a kind of machine based on object example match is provided
Device human speech justice SLAM processor can be realized object individual in identification scene and construct three-dimensional semantic map, while can be based on again
Example match between different key frames optimizes the estimated result of SLAM pose.
A kind of robot semanteme SLAM processor based on object example match of the disclosure, comprising:
Camera motion estimation module is used to obtain the image sequence shot in robot operational process, to every frame image
Feature point extraction, matching and tracking are carried out to estimate camera motion；
Case-based system module is used to extract key frame, carries out example segmentation to key frame, obtains in every frame key frame
All object examples；
Feature describing module is used to carry out feature point extraction to key frame and calculates feature point description, to key frame
In all objects example carry out feature extraction and coding carrys out the feature description vectors of calculated examples, while obtaining example three-dimensional point
Cloud；
Characteristic point and example match module, are used for according to feature point description and feature description vectors, respectively to adjacent
Characteristic point and object example between key frame carry out Feature Points Matching and example match；
Pose Estimation Optimization module, be used for the matching of fusion feature point and example match to the pose estimated result of SLAM into
The optimization of row local nonlinearity obtains the key frame for carrying object example semantic markup information；
Three-dimensional semanteme map structuring module is used to the key frame for carrying object example semantic markup information being mapped to reality
In example three-dimensional point cloud, three-dimensional semantic map is constructed.
In one or more embodiments, it in the camera motion estimation module, is solved using light-stream adjustment adjacent
The camera motion of frame:
First to consecutive frame image carry out ORB feature point extraction with match, obtain several pairs of ORB characteristic points of consecutive frame；
Then non-linear least square problem is constructed according to this several pairs of ORB characteristic points, solution obtains the pose of camera.
In one or more embodiments, in the case-based system module, the size of interframe relative motion distance is made
For the foundation for extracting the key frame in image sequence.
In one or more embodiments, in the case-based system module, divided using the example based on deep learning
Frame-Mask R-CNN network to carry out example segmentation to key frame images, to obtain all realities in every frame key frame images
Example；Wherein, Mask R-CNN network adds a Ge Quan convolutional neural networks branch on the basis of Faster R-CNN for defeated
Example mask out, to carry out the segmentation of Pixel-level to the profile of example in detection block.
In one or more embodiments, in the feature describing module, the mistake of the feature description vectors of calculated examples
Journey includes:
Visual vocabulary table is constructed using training set based on VLAD algorithm, grid is carried out to every frame image in training set and is drawn
Point, dense SIFT feature and RGB color value are extracted to each grid element center, obtain the feature description vectors of each grid；
Obtained grid search-engine description vectors are clustered into the class of preset quantity using k-mean algorithm, and calculate each net
Lattice feature description vectors with its cluster in residual vector, the normalization of power rate and L2 norm normalizing are carried out to all residual vector
Change, then carry out feature coding using detection block image of the normalized residual vector to example, obtains the feature description of example
Vector.
In one or more embodiments, in the case-based system module, if interframe relative motion distance is between permission
Interframe minimum relative motion distance and maximum relative motion distance between, then present frame is key frame.
According to the other side of one or more other embodiments of the present disclosure, a kind of machine based on object example match is provided
Device human speech justice SLAM robot can be realized object individual in identification scene and construct three-dimensional semantic map, while can be based on again
Example match between different key frames optimizes the estimated result of SLAM pose.
A kind of semanteme SLAM robot, robot based on object example match of the disclosure, including it is described above based on
The robot semanteme SLAM processor of object example match.
The beneficial effect of the disclosure is:
(1) the robot semanteme SLAM method based on object example match that the disclosure provides, for base under indoor environment
In the semantic SLAM method of RGB-D video flowing, by combine the currently advanced example partitioning algorithm based on deep learning and
VSLAM algorithm realizes the individual of various objects detectable and in identification scene, and is building up in three-dimensional semantic map,
Simultaneously using the pose estimated result of object example match optimization SLAM, the method for improving vSLAM positioning accuracy.
(2) the semantic SLAM method that disclosure Case-based Reasoning cutting techniques and vSLAM technology are realized divides skill using example
Art carries out all examples in processing acquisition key frame to key frame, and example is mapped in three-dimensional point cloud by vSLAM
Construct the semantic map of three-dimensional towards object example.
(3) disclosure and semantic segmentation unlike traditional semantic segmentation technology only to the object category in image into
Row is distinguished, and example cutting techniques can distinguish the Different Individual of same category object, and at the same time eliminating example
Background pixel.The disclosure passes through fusion feature point matching result and example match result carry out office in different crucial interframe simultaneously
The positioning accuracy of portion's nonlinear optimization raising vSLAM.
Detailed description of the invention
The Figure of description for constituting a part of this disclosure is used to provide further understanding of the disclosure, and the disclosure is shown
Meaning property embodiment and its explanation do not constitute the improper restriction to the disclosure for explaining the disclosure.
Fig. 1 is a kind of semantic SLAM embodiment of the method flow diagram based on object example match of the disclosure.
Fig. 2 is the local nonlinearity optimization method embodiment schematic diagram of the vSLAM of the disclosure.
Fig. 3 is a kind of semantic SLAM processor example structure schematic diagram based on object example match of the disclosure.
Specific embodiment
It is noted that following detailed description is all illustrative, it is intended to provide further instruction to the disclosure.Unless another
It indicates, all technical and scientific terms used herein has usual with disclosure person of an ordinary skill in the technical field
The identical meanings of understanding.
It should be noted that term used herein above is merely to describe specific embodiment, and be not intended to restricted root
According to the illustrative embodiments of the disclosure.As used herein, unless the context clearly indicates otherwise, otherwise singular
Also it is intended to include plural form, additionally, it should be understood that, when in the present specification using term "comprising" and/or " packet
Include " when, indicate existing characteristics, step, operation, device, component and/or their combination.
Term is explained:
(1) ORB:Oriented FASTand Rotated BRIEF algorithm is the characteristic point detection of current most fast and stable
And extraction algorithm, many image mosaics and target tracking technology are realized using ORB feature.
(2) RGBD=RGB+Depth Map；
RGB:RGB color mode is a kind of color standard of industry, is by red (R), green (G), blue (B) three face
To obtain miscellaneous color, RGB is to represent red, green, blue for the variation of chrominance channel and their mutual superpositions
The color in three channels, this standard almost include all colours that human eyesight can perceive, and are current with most wide
One of color system.
Depth Map: in 3D computer graphics, Depth Map (depth map) is comprising the scenario objects with viewpoint
The image or image channel of the information of the distance dependent on surface.Wherein, Depth Map is similar to gray level image, and only its is every
A pixel value is the actual range of sensor distance object.Usual RGB image and Depth image are registrations, thus pixel
Between have one-to-one corresponding relationship.
(3) ICP (Iterative Closest Point iteration closest approach) algorithm is a kind of point set to point set registration side
Method.The essence of ICP algorithm is the Optimum Matching based on least square method, it repeats " to determine point set → meter of corresponding relationship
Calculate optimal rigid body translation " process, until some indicates that correct matched convergence criterion is met.
(4) MSCOCO data set is a data set of Microsoft's building, it includes detection, segmentation,
The tasks such as keypoints.MSCOCO is primarily to solution detecting non-iconic views of objects is (right
The detection that should often say), contextual reasoning between objects and the precise 2D
Problem under localization of objects (the corresponding segmentation problem often said) these three scenes.
As shown in Figure 1, a kind of semantic SLAM method based on object example match that the embodiment of the present disclosure provides, by right
The RGB-D video sequence of depth camera is handled, to estimate camera motion and the simultaneously semantic map of three-dimensional of constructing environment, and
Object example match result optimizing vSLAM pose estimated result is utilized simultaneously, improves positioning accuracy.It specifically includes following:
Step 1: obtain the image sequence that shoots in robot operational process, to every frame image carry out feature point extraction,
Match and tracks to estimate camera motion.
Specifically, feature point extraction is carried out to image data, camera motion is estimated in matching with tracking, realizes vision mileage
The function of meter；The image data is the RGB-D image sequence of depth camera shooting, is shot during the motion for robot
RGB-D image collection I={ I1..., IN, wherein InIt is the n-th width RGB-D image data；The depth camera that robot carries
Internal reference matrix is K.
The camera motion that consecutive frame is solved using light-stream adjustment (Bundle Adjustment, BA), first to consecutive frame
RGB-D image carry out ORB feature point extraction with match, obtain the n of consecutive frame to ORB characteristic point, then according to this n to ORB
Characteristic point constructs non-linear least square problem, is shown below, finally solves the optimization problem and obtain the pose ξ of camera*。
Building non-linear least square and the movement that consecutive frame camera is solved using BA, used cost function:
Wherein, ξ*It is the representation of Lie algebra form of the camera pose obtained after BA optimizes, minξExpression passes through optimized variable
ξ minimizes cost function, uiIt is i-th point of the pixel coordinate observed, exp (ξ ^) is that the Lie group of camera pose indicates shape
Formula is a four-matrix, PiIt is i-th point of 3D coordinate, siIt is i-th point of depth value.
Step 2: extracting key frame, example segmentation is carried out to key frame, all objects obtained in every frame key frame are real
Example.
In specific implementation, using the size of interframe relative motion distance as the key frame extracted in RGB-D image sequence
Foundation, interframe rotating vector R and translation vector t are calculated first, then calculates the distance D of interframe relative motion, it is as follows
Shown in formula:
D=‖ Δ t ‖+min (2 π-‖ R ‖, ‖ R ‖)
Wherein, Δ t indicates that translation vector is poor；The length of ‖ ‖ expression vector.
Key frame is selected according to interframe relative motion distance D, rule is as follows:
If 1) Dmin≤D≤Dmax, then Framecurr=Framekey；
If 2) D < DminOr D > Dmax, then Framecurr≠Framekey
Wherein, DminAnd DmaxIt is allowed interframe minimum relative motion distance and maximum relative motion distance respectively,
FramecurrFor present frame, FramekeyFor key frame.
In specific implementation, using based on the example of deep learning segmentation framework-Mask R-CNN network come to key frame
Image carries out example segmentation, to obtain all examples in every frame key frame images, the target detection including object Different Individual
The mask of example pixel grade after frame and removal background.
In this step, using based on the example segmentation framework-Mask R-CNN of deep learning come to key frame images into
The segmentation of row example, to obtain all examples in every frame key frame images, detection block and removal back including object Different Individual
The mask of example pixel grade after scape.
Wherein, Mask R-CNN network is improved on the basis of Faster R-CNN network, Faster R-
CNN network can carry out target detection and obtain the detection block of each target in image, and can not be to the profile of target in detection block
Accurately divided.In order to carry out example segmentation, Mask R-CNN network is added on the basis of Faster R-CNN
One Ge Quan convolutional neural networks branch is used to export example mask, to carry out point of Pixel-level to the profile of example in detection block
It cuts.
The shortcomings that R-CNN: it is extracted even with pre-treatment steps such as selective search (selective search)
Potential bounding box (frame) is as input, but R-CNN still has serious speed bottle-neck, reason it is also obvious that
It is exactly to have to compute repeatedly when computer carries out feature extraction to all region, Fast-R-CNN is asked precisely in order to solving this
What topic was born.
In Fast-R-CNN, bbox regression (frame recurrence) is placed into inside neural network, with region
(boundary) classification and and become multi-task (multitask) model, actual experiment also turns out that the two tasks can
Shared convolution feature, and mutually promote.A Fast-R-CNN critically important contribution is that multiclass detection can really guarantee standard
Processing speed is promoted while true rate.
Example segmentation is carried out to key frame using Mask R-CNN network, selects MS COCO data set as Mask first
The training set of R-CNN, with the different classes of target of 80 classes, used loss function are as follows: L=Lc+Lb+Lm, wherein LcFor
Error in classification, LbFor target detection error, LmDivide error for pixel, is defined as average two-value and intersects entropy loss.Then instruction is utilized
The weight model for getting Mask R-CNN predicts every frame key frame, extracts article example all in key frame, obtains
Example pixel grade mask to after the target detection frame of each example and rejecting background.
Step 3: feature point extraction is carried out to key frame and calculates feature point description, it is real to all objects in key frame
Example carries out feature extraction and coding carrys out the feature description vectors of calculated examples, while obtaining example three-dimensional point cloud.
Specifically, it is primarily based on VLAD algorithm and constructs visual vocabulary table using training set, to every frame image in training set
Grid dividing is carried out, dense SIFT feature and RGB color value are extracted to each grid element center, obtains the feature description of each grid
VectorN grid search-engine description vectors are clustered into 64 classes using k-mean algorithm later, and are calculated
Each grid search-engine description vectors with its cluster in residual vector, the normalization of power rate and L2 model are carried out to all residual vector
Then number normalization carries out feature coding using detection block image of the normalized residual vector to example:
Grid dividing is carried out to each example detection block diagram picture first and dense SIFT feature is extracted, and is based on being constructed above
Visual vocabulary table the image in each grid is encoded, obtain the feature description vectors ψ of example image, then use 3
Grade image space pyramid structure is distributed to count the characteristic point of example image, obtains the spatial information of example image, first will
I-th layer of example image is divided into 4iSub-regions, the then statistic histogram feature in each sub-regions, finally by 3 layers
Secondary histogram is composed in series the feature description vectors of example
Wherein, VLAD is the abbreviation of vector of locally aggregated descriptors, is by Jegou
Et al. proposed that core concept was aggregated (accumulation), is mainly used in field of image search in 2010.
VLAD algorithm can regard a kind of FV of simplification as, and main method is by clustering method one small code of training
This, finds the feature in each image nearest code book cluster centre, and the difference of subsequent all features and cluster centre is done
It is cumulative, the vlad matrix of a k*d is obtained, wherein k is cluster centre number, and d is intrinsic dimensionality (such as sift is 128 dimensions), with
Afterwards by the matrix-expand it is the vector of one (k*d) dimension, and its L2 is normalized, obtained vector is VLAD.
VLAD algorithm flow:
(1) picture file path and feature extraction are read；
(2) using clustering method training code book；
(3) feature of every picture and nearest cluster centre are added up；
* (4) carry out PCA dimensionality reduction to the VLAD after adding up and normalize to it；
* after (5) obtain VLAD, continue to reduce storage space using ADC method and improve search speed.
Wherein step * (4), * (5) are optional, carry out the i.e. available Europe of L2 normalization after step (3) obtains the cumulative vector of residual error
Family name's distance etc. calculates the similitude of two pictures to realize picture retrieval.
Step 4: according to feature point description and feature description vectors, respectively to the characteristic point and object between adjacent key frame
Body example carries out Feature Points Matching and example match.
Step 5: the matching of fusion feature point and example match carry out local nonlinearity optimization to the pose estimated result of SLAM,
Obtain carrying the key frame of object example semantic markup information.
Fig. 2 is the side that fusion example match and Feature Points Matching optimize vSLAM pose estimated result in the disclosure
Method schematic diagram, as shown in Fig. 2, method of disclosure is on the basis of traditional characteristic point matches geometrical constraint, to be added to example match several
What is constrained, and is carried out further nonlinear optimization to pose estimated result, is improved positioning accuracy.For Feature Points Matching as a result, setting
zijIt is in pose ξiPlace's observation road sign characteristic point pjThe data of generation, then cost function are as follows:
Wherein: m and n is respectively the number of the number and characteristic point that participate in the pose of optimization, and i is the index of pose, and j is special
Levy the index of point, eijIt is in pose ξiLocate the re-projection error of j-th of characteristic point, h (ξi, pj) indicate j-th of characteristic point pj?
I pose ξiThe projection at place.
For example match as a result, using ICP algorithm be registrated example point cloud to realize the optimization to pose, ifIt is pose ξiThe data of k-th point of generation in j-th of example point cloud of place's observation, then cost function is written as:
Weighted average is taken to be merged with the pose after being optimized by example match the pose after being optimized by Feature Points Matching
Pose estimated result afterwards.
Step 6: the key frame for carrying object example semantic markup information being mapped in three-dimensional point cloud, is constructed three-dimensional semantic
Map.
The depth camera internal reference matrix that known machine people is carried is K, and the pose of the i-th frame key frame is ξi, key frame it is every
A pixel indicates p=[u, v, l] with 3 dimensional vectors, wherein u, and v is transverse and longitudinal coordinate, and l is example label, and key frame is reflected
It is mapped in three-dimensional point cloud:
Wherein, [uj, vj, 1] and indicate j-th of pixel of corresponding key frame；djIndicate the depth value of j-th of characteristic point,
[Xj, Yj, Zj]TIt is the coordinate vector that j-th of characteristic point projects in three dimensions；Exp (ξ ^) is that the Lie group of camera pose indicates
Form is a four-matrix.
Then each point is expressed as P=[X, Y, Z, l] in three-dimensional point cloud.
In conclusion the perfect vSLAM technology based on RGB-D data of disclosure combination example partitioning algorithm, so that
VSLAM also obtains the environment semantic information towards object example while obtaining environment geological information, and utilizes object
Example match carries out further geometrical constraint to vSLAM, improves the pose estimated accuracy of vSLAM.
Fig. 3 is a kind of semantic SLAM processor structure schematic diagram based on object example match of the disclosure.
A kind of robot semanteme SLAM processor based on object example match of the disclosure, comprising:
(1) camera motion estimation module is used to obtain the image sequence shot in robot operational process, to every frame figure
Camera motion is estimated as carrying out feature point extraction, matching and tracking；
Specifically, in the camera motion estimation module, the camera motion of consecutive frame is solved using light-stream adjustment:
First to consecutive frame image carry out ORB feature point extraction with match, obtain several pairs of ORB characteristic points of consecutive frame；
Then non-linear least square problem is constructed according to this several pairs of ORB characteristic points, solution obtains the pose of camera.
(2) case-based system module is used to extract key frame, carries out example segmentation to key frame, obtains every frame key frame
In all object examples；
Specifically, in the case-based system module, using the size of interframe relative motion distance as extraction image sequence
In key frame foundation.
Specifically, in the case-based system module, using example segmentation framework-Mask R- based on deep learning
CNN network to carry out example segmentation to key frame images, to obtain all examples in every frame key frame images；Wherein, Mask
R-CNN network adds a Ge Quan convolutional neural networks branch on the basis of Faster R-CNN and is used to export example mask, from
And the segmentation of Pixel-level is carried out to the profile of example in detection block.
Wherein, in the case-based system module, if interframe relative motion distance is between the opposite fortune of interframe minimum of permission
Between dynamic distance and maximum relative motion distance, then present frame is key frame.
(3) feature describing module is used to carry out feature point extraction to key frame and calculates feature point description, to key
All objects example in frame carries out feature extraction and coding carrys out the feature description vectors of calculated examples, while obtaining example three-dimensional
Point cloud；
In the feature describing module, the process of the feature description vectors of calculated examples includes:
Visual vocabulary table is constructed using training set based on VLAD algorithm, grid is carried out to every frame image in training set and is drawn
Point, dense SIFT feature and RGB color value are extracted to each grid element center, obtain the feature description vectors of each grid；
Obtained grid search-engine description vectors are clustered into the class of preset quantity using k-mean algorithm, and calculate each net
Lattice feature description vectors with its cluster in residual vector, the normalization of power rate and L2 norm normalizing are carried out to all residual vector
Change, then carry out feature coding using detection block image of the normalized residual vector to example, obtains the feature description of example
Vector.
(4) characteristic point and example match module, are used for according to feature point description and feature description vectors, respectively to phase
Characteristic point and object example between adjacent key frame carry out Feature Points Matching and example match；
(5) pose Estimation Optimization module is used for the matching of fusion feature point and example match to the pose estimation knot of SLAM
Fruit carries out local nonlinearity optimization, obtains the key frame for carrying object example semantic markup information；
(6) three-dimensional semantic map structuring module is used to carry the key frame mapping of object example semantic markup information
Into example three-dimensional point cloud, three-dimensional semantic map is constructed.
The robot semanteme SLAM processor based on object example match that the disclosure provides, for being based under indoor environment
The semantic SLAM method of RGB-D video flowing, by combining currently advanced example partitioning algorithm and vSLAM based on deep learning
Algorithm realizes the individual of various objects detectable and in identification scene, and is building up in three-dimensional semantic map, simultaneously
Using the pose estimated result of object example match optimization SLAM, the method for improving vSLAM positioning accuracy.
The semantic SLAM method that disclosure Case-based Reasoning cutting techniques and vSLAM technology are realized, utilizes example cutting techniques
All examples in processing acquisition key frame are carried out to key frame, example is mapped in three-dimensional point cloud by vSLAM and removes structure
Build out the semantic map of three-dimensional towards object example.
A kind of semanteme SLAM robot, robot based on object example match of the disclosure, including base as shown in Figure 3
In the robot semanteme SLAM processor of object example match.
The disclosure is only to carry out to the object category in image from semantic segmentation unlike traditional semantic segmentation technology
It distinguishes, and example cutting techniques can distinguish the Different Individual of same category object, and at the same time eliminating example
Background pixel.The disclosure carries out part by fusion feature point matching result and example match result in different crucial interframe simultaneously
The positioning accuracy of nonlinear optimization raising vSLAM.
It should be understood by those skilled in the art that, embodiment of the disclosure can provide as method, system or computer program
Product.Therefore, the shape of hardware embodiment, software implementation or embodiment combining software and hardware aspects can be used in the disclosure
Formula.Moreover, the disclosure, which can be used, can use storage in the computer that one or more wherein includes computer usable program code
The form for the computer program product implemented on medium (including but not limited to magnetic disk storage and optical memory etc.).
The disclosure is referring to method, the process of equipment (system) and computer program product according to the embodiment of the present disclosure
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the program can be stored in a computer-readable storage medium
In, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random
AccessMemory, RAM) etc..
Although above-mentioned be described in conjunction with specific embodiment of the attached drawing to the disclosure, model not is protected to the disclosure
The limitation enclosed, those skilled in the art should understand that, on the basis of the technical solution of the disclosure, those skilled in the art are not
Need to make the creative labor the various modifications or changes that can be made still within the protection scope of the disclosure.
Claims (10)
1. a kind of robot semanteme SLAM method based on object example match characterized by comprising
The image sequence shot in robot operational process is obtained, feature point extraction, matching and tracking are carried out to every frame image and come
Estimate camera motion；
Key frame is extracted, example segmentation is carried out to key frame, obtains all object examples in every frame key frame；
Feature point extraction is carried out to key frame and calculates feature point description, feature is carried out to all objects example in key frame
The feature description vectors for carrying out calculated examples with coding are extracted, while obtaining example three-dimensional point cloud；
According to feature point description and feature description vectors, respectively to the characteristic point and the progress of object example between adjacent key frame
Feature Points Matching and example match；
The matching of fusion feature point and example match carry out local nonlinearity optimization to the pose estimated result of SLAM, obtain belongings
The key frame of body example semantic markup information；
The key frame for carrying object example semantic markup information is mapped in example three-dimensional point cloud, constructs three-dimensional semantically
Figure.
2. the robot semanteme SLAM method based on object example match as described in claim 1, which is characterized in that estimating
During camera motion, the camera motion of consecutive frame is solved using light-stream adjustment:
First to consecutive frame image carry out ORB feature point extraction with match, obtain several pairs of ORB characteristic points of consecutive frame；
Then non-linear least square problem is constructed according to this several pairs of ORB characteristic points, solution obtains the pose of camera.
3. the robot semanteme SLAM method based on object example match as described in claim 1, which is characterized in that extracting
During key frame, using the size of interframe relative motion distance as the foundation for extracting the key frame in image sequence.
4. the robot semanteme SLAM method based on object example match as claimed in claim 3, which is characterized in that if interframe
Relative motion distance is between the interframe minimum relative motion distance and maximum relative motion distance of permission, then present frame is to close
Key frame.
5. the robot semanteme SLAM method based on object example match as described in claim 1, which is characterized in that use base
Come to carry out example segmentation to key frame images in example segmentation framework-Mask R-CNN network of deep learning, to obtain every frame
All examples in key frame images；Wherein, Mask R-CNN network adds a full volume on the basis of Faster R-CNN
Product neural network branch is used to export example mask, to carry out the segmentation of Pixel-level to the profile of example in detection block.
6. the robot semanteme SLAM method based on object example match as described in claim 1, which is characterized in that calculate real
The process of feature description vectors of example includes:
Visual vocabulary table is constructed using training set based on VLAD algorithm, grid dividing is carried out to every frame image in training set, it is right
Each grid element center extracts dense SIFT feature and RGB color value, obtains the feature description vectors of each grid；
Obtained grid search-engine description vectors are clustered into the class of preset quantity using k-mean algorithm, and it is special to calculate each grid
Sign description vectors and its cluster in residual vector, the normalization of power rate and the normalization of L2 norm are carried out to all residual vector,
Then feature coding is carried out to the detection block image of example using normalized residual vector, obtain example feature describe to
Amount.
7. a kind of robot semanteme SLAM processor based on object example match characterized by comprising
Camera motion estimation module is used to obtain the image sequence shot in robot operational process, carries out to every frame image
Camera motion is estimated in feature point extraction, matching and tracking；
Case-based system module is used to extract key frame, carries out example segmentation to key frame, obtains all in every frame key frame
Object example；
Feature describing module is used to carry out feature point extraction to key frame and calculates feature point description, in key frame
All objects example carries out feature extraction and coding carrys out the feature description vectors of calculated examples, while obtaining example three-dimensional point cloud；
Characteristic point and example match module are used for according to feature point description and feature description vectors, respectively to adjacent key
Characteristic point and object example between frame carry out Feature Points Matching and example match；
Pose Estimation Optimization module is used for the matching of fusion feature point and example match to the pose estimated result carry out office of SLAM
Portion's nonlinear optimization obtains the key frame for carrying object example semantic markup information；
Three-dimensional semanteme map structuring module is used to the key frame for carrying object example semantic markup information being mapped to example three
In dimension point cloud, three-dimensional semantic map is constructed.
8. the robot semanteme SLAM method based on object example match as described in claim 1, which is characterized in that described
In camera motion estimation module, the camera motion of consecutive frame is solved using light-stream adjustment:
First to consecutive frame image carry out ORB feature point extraction with match, obtain several pairs of ORB characteristic points of consecutive frame；
Then non-linear least square problem is constructed according to this several pairs of ORB characteristic points, solution obtains the pose of camera；
Or in the case-based system module, using the size of interframe relative motion distance as the key frame extracted in image sequence
Foundation；
Or in the case-based system module, using example segmentation framework-MaskR-CNN network based on deep learning come pair
Key frame images carry out example segmentation, to obtain all examples in every frame key frame images；Wherein, Mask R-CNN network exists
It adds a Ge Quan convolutional neural networks branch on the basis of Faster R-CNN to be used to export example mask, thus in detection block
The profile of example carries out the segmentation of Pixel-level；
Or in the feature describing module, the process of the feature description vectors of calculated examples includes:
Visual vocabulary table is constructed using training set based on VLAD algorithm, grid dividing is carried out to every frame image in training set, it is right
Each grid element center extracts dense SIFT feature and RGB color value, obtains the feature description vectors of each grid；
Obtained grid search-engine description vectors are clustered into the class of preset quantity using k-mean algorithm, and it is special to calculate each grid
Sign description vectors and its cluster in residual vector, the normalization of power rate and the normalization of L2 norm are carried out to all residual vector,
Then feature coding is carried out to the detection block image of example using normalized residual vector, obtain example feature describe to
Amount.
9. the robot semanteme SLAM method based on object example match as claimed in claim 8, which is characterized in that described
In case-based system module, if interframe relative motion distance is between the interframe minimum relative motion distance and maximum relative motion of permission
Between distance, then present frame is key frame.
10. a kind of robot, which is characterized in that including being based on object example match as claimed in any one of claims 7-9
Robot semanteme SLAM processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910037102.3A CN109816686A (en) | 2019-01-15 | 2019-01-15 | Robot semanteme SLAM method, processor and robot based on object example match |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910037102.3A CN109816686A (en) | 2019-01-15 | 2019-01-15 | Robot semanteme SLAM method, processor and robot based on object example match |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109816686A true CN109816686A (en) | 2019-05-28 |
Family
ID=66603838
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910037102.3A Pending CN109816686A (en) | 2019-01-15 | 2019-01-15 | Robot semanteme SLAM method, processor and robot based on object example match |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109816686A (en) |
Cited By (49)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110146099A (en) * | 2019-05-31 | 2019-08-20 | 西安工程大学 | A kind of synchronous superposition method based on deep learning |
| CN110335317A (en) * | 2019-07-02 | 2019-10-15 | 百度在线网络技术（北京）有限公司 | Image processing method, device, equipment and medium based on terminal device positioning |
| CN110335319A (en) * | 2019-06-26 | 2019-10-15 | 华中科技大学 | Camera positioning and the map reconstruction method and system of a kind of semantics-driven |
| CN110390724A (en) * | 2019-07-12 | 2019-10-29 | 杭州凌像科技有限公司 | A kind of SLAM method with example segmentation |
| CN110516527A (en) * | 2019-07-08 | 2019-11-29 | 广东工业大学 | A kind of vision SLAM winding detection improvement method of Case-based Reasoning segmentation |
| CN110599522A (en) * | 2019-09-18 | 2019-12-20 | 成都信息工程大学 | Method for detecting and removing dynamic target in video sequence |
| CN110602393A (en) * | 2019-09-04 | 2019-12-20 | 南京博润智能科技有限公司 | Video anti-shake method based on image content understanding |
| CN110599542A (en) * | 2019-08-30 | 2019-12-20 | 北京影谱科技股份有限公司 | Method and device for local mapping of adaptive VSLAM (virtual local area model) facing to geometric area |
| CN110610650A (en) * | 2019-08-27 | 2019-12-24 | 杭州电子科技大学 | Point cloud semantic map construction method based on deep learning and depth camera |
| CN110610198A (en) * | 2019-08-22 | 2019-12-24 | 浙江工业大学 | Mask RCNN-based automatic oral CBCT image mandibular neural tube identification method |
| CN110717052A (en) * | 2019-10-15 | 2020-01-21 | 山东大学 | Environment characterization method in service robot intelligent service |
| CN110728751A (en) * | 2019-06-19 | 2020-01-24 | 武汉科技大学 | Construction method of indoor 3D point cloud semantic map |
| CN110738673A (en) * | 2019-10-21 | 2020-01-31 | 哈尔滨理工大学 | Visual SLAM method based on example segmentation |
| CN110766024A (en) * | 2019-10-08 | 2020-02-07 | 湖北工业大学 | Visual odometer feature point extraction method based on deep learning and visual odometer |
| CN110866953A (en) * | 2019-10-31 | 2020-03-06 | Oppo广东移动通信有限公司 | Map construction method and device, and positioning method and device |
| CN110986945A (en) * | 2019-11-14 | 2020-04-10 | 上海交通大学 | Local navigation method and system based on semantic height map |
| CN111105695A (en) * | 2019-12-31 | 2020-05-05 | 智车优行科技（上海）有限公司 | Map making method and device, electronic equipment and computer readable storage medium |
| CN111160210A (en) * | 2019-12-24 | 2020-05-15 | 天津天地伟业机器人技术有限公司 | Video-based water flow velocity detection method and system |
| CN111239763A (en) * | 2020-03-06 | 2020-06-05 | 广州视源电子科技股份有限公司 | Object positioning method and device, storage medium and processor |
| CN111273772A (en) * | 2020-01-17 | 2020-06-12 | 江苏艾佳家居用品有限公司 | Augmented reality interaction method and device based on slam mapping method |
| CN111275026A (en) * | 2020-03-23 | 2020-06-12 | 复旦大学 | Three-dimensional point cloud combined semantic and instance segmentation method |
| CN111325842A (en) * | 2020-03-04 | 2020-06-23 | Oppo广东移动通信有限公司 | Map construction method, repositioning method and device, storage medium and electronic equipment |
| CN111496784A (en) * | 2020-03-27 | 2020-08-07 | 山东大学 | Space environment identification method and system for robot intelligent service |
| CN111563442A (en) * | 2020-04-29 | 2020-08-21 | 上海交通大学 | Slam method and system for fusing point cloud and camera image data based on laser radar |
| CN111581313A (en) * | 2020-04-25 | 2020-08-25 | 华南理工大学 | Semantic SLAM robustness improvement method based on instance segmentation |
| CN111665842A (en) * | 2020-06-09 | 2020-09-15 | 山东大学 | Indoor SLAM mapping method and system based on semantic information fusion |
| CN111693047A (en) * | 2020-05-08 | 2020-09-22 | 中国航空工业集团公司西安航空计算技术研究所 | Visual navigation method for micro unmanned aerial vehicle in high-dynamic scene |
| CN111709328A (en) * | 2020-05-29 | 2020-09-25 | 北京百度网讯科技有限公司 | Vehicle tracking method and device and electronic equipment |
| CN111797938A (en) * | 2020-07-15 | 2020-10-20 | 燕山大学 | Semantic information and VSLAM fusion method for sweeping robot |
| CN111882663A (en) * | 2020-07-03 | 2020-11-03 | 广州万维创新科技有限公司 | Visual SLAM closed-loop detection method achieved by fusing semantic information |
| CN111882613A (en) * | 2020-07-24 | 2020-11-03 | 中国科学院上海微系统与信息技术研究所 | Visual odometry method and device based on edge semantics, storage medium and equipment |
| CN112148817A (en) * | 2019-06-28 | 2020-12-29 | 理光软件研究所（北京）有限公司 | Panoramic-map-based SLAM optimization method, device and system |
| CN112258575A (en) * | 2020-10-13 | 2021-01-22 | 浙江大学 | Method for quickly identifying object in synchronous positioning and map construction |
| CN112418250A (en) * | 2020-12-01 | 2021-02-26 | 怀化学院 | Optimized matching method for complex 3D point cloud |
| CN112560648A (en) * | 2020-12-09 | 2021-03-26 | 长安大学 | SLAM method based on RGB-D image |
| CN112734845A (en) * | 2021-01-08 | 2021-04-30 | 浙江大学 | Outdoor monocular synchronous mapping and positioning method fusing scene semantics |
| CN112785714A (en) * | 2021-01-29 | 2021-05-11 | 北京百度网讯科技有限公司 | Point cloud instance labeling method and device, electronic equipment and medium |
| CN112967341A (en) * | 2021-02-23 | 2021-06-15 | 湖北枫丹白露智慧标识科技有限公司 | Indoor visual positioning method, system, equipment and storage medium based on live-action image |
| CN113724299A (en) * | 2021-08-30 | 2021-11-30 | 上海大学 | Method for tracking three-dimensional track of target by mobile robot based on electro-hydraulic adjustable focus lens |
| CN113916245A (en) * | 2021-10-09 | 2022-01-11 | 上海大学 | Semantic map construction method based on instance segmentation and VSLAM |
| WO2022021739A1 (en) * | 2020-07-30 | 2022-02-03 | 国网智能科技股份有限公司 | Humanoid inspection operation method and system for semantic intelligent substation robot |
| CN114092388A (en) * | 2021-08-30 | 2022-02-25 | 河南笛卡尔机器人科技有限公司 | Obstacle detection method based on monocular camera and odometer |
| CN114216461A (en) * | 2021-09-29 | 2022-03-22 | 杭州图灵视频科技有限公司 | Panoramic camera-based indoor positioning method and system for mobile robot |
| CN114359493A (en) * | 2021-12-20 | 2022-04-15 | 中国船舶重工集团公司第七0九研究所 | Method and system for generating three-dimensional semantic map for unmanned ship |
| CN114494825A (en) * | 2021-12-31 | 2022-05-13 | 重庆特斯联智慧科技股份有限公司 | Robot positioning method and device |
| CN115496977A (en) * | 2022-09-14 | 2022-12-20 | 北京化工大学 | Target detection method and device based on multi-mode sequence data fusion |
| CN116128734A (en) * | 2023-04-17 | 2023-05-16 | 湖南大学 | Image stitching method, device, equipment and medium based on deep learning |
| CN116168393A (en) * | 2023-01-17 | 2023-05-26 | 浙江大学 | Automatic semantic annotation data generation method and device based on point cloud neural radiation field |
| CN117132648A (en) * | 2023-04-28 | 2023-11-28 | 荣耀终端有限公司 | Visual positioning method, electronic equipment and computer readable storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104298971A (en) * | 2014-09-28 | 2015-01-21 | 北京理工大学 | Method for identifying objects in 3D point cloud data |
| CN105856230A (en) * | 2016-05-06 | 2016-08-17 | 简燕梅 | ORB key frame closed-loop detection SLAM method capable of improving consistency of position and pose of robot |
| CN107063258A (en) * | 2017-03-07 | 2017-08-18 | 重庆邮电大学 | A kind of mobile robot indoor navigation method based on semantic information |
| CN107066507A (en) * | 2017-01-10 | 2017-08-18 | 中国人民解放军国防科学技术大学 | A kind of semantic map constructing method that cloud framework is mixed based on cloud robot |
| CN107741234A (en) * | 2017-10-11 | 2018-02-27 | 深圳勇艺达机器人有限公司 | The offline map structuring and localization method of a kind of view-based access control model |
| CN108230337A (en) * | 2017-12-31 | 2018-06-29 | 厦门大学 | A kind of method that semantic SLAM systems based on mobile terminal are realized |
| US20180253856A1 (en) * | 2017-03-01 | 2018-09-06 | Microsoft Technology Licensing, Llc | Multi-Spectrum Illumination-and-Sensor Module for Head Tracking, Gesture Recognition and Spatial Mapping |
| CN108648274A (en) * | 2018-05-10 | 2018-10-12 | 华南理工大学 | A kind of cognition point cloud map creation system of vision SLAM |
| CN108830220A (en) * | 2018-06-15 | 2018-11-16 | 山东大学 | The building of vision semantic base and global localization method based on deep learning |
| CN109186606A (en) * | 2018-09-07 | 2019-01-11 | 南京理工大学 | A kind of robot composition and air navigation aid based on SLAM and image information |
-
2019
- 2019-01-15 CN CN201910037102.3A patent/CN109816686A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104298971A (en) * | 2014-09-28 | 2015-01-21 | 北京理工大学 | Method for identifying objects in 3D point cloud data |
| CN105856230A (en) * | 2016-05-06 | 2016-08-17 | 简燕梅 | ORB key frame closed-loop detection SLAM method capable of improving consistency of position and pose of robot |
| CN107066507A (en) * | 2017-01-10 | 2017-08-18 | 中国人民解放军国防科学技术大学 | A kind of semantic map constructing method that cloud framework is mixed based on cloud robot |
| US20180253856A1 (en) * | 2017-03-01 | 2018-09-06 | Microsoft Technology Licensing, Llc | Multi-Spectrum Illumination-and-Sensor Module for Head Tracking, Gesture Recognition and Spatial Mapping |
| CN107063258A (en) * | 2017-03-07 | 2017-08-18 | 重庆邮电大学 | A kind of mobile robot indoor navigation method based on semantic information |
| CN107741234A (en) * | 2017-10-11 | 2018-02-27 | 深圳勇艺达机器人有限公司 | The offline map structuring and localization method of a kind of view-based access control model |
| CN108230337A (en) * | 2017-12-31 | 2018-06-29 | 厦门大学 | A kind of method that semantic SLAM systems based on mobile terminal are realized |
| CN108648274A (en) * | 2018-05-10 | 2018-10-12 | 华南理工大学 | A kind of cognition point cloud map creation system of vision SLAM |
| CN108830220A (en) * | 2018-06-15 | 2018-11-16 | 山东大学 | The building of vision semantic base and global localization method based on deep learning |
| CN109186606A (en) * | 2018-09-07 | 2019-01-11 | 南京理工大学 | A kind of robot composition and air navigation aid based on SLAM and image information |
Non-Patent Citations (4)
| Title |
|---|
| S YANG等: ""Robust RGB-D SLAM in dynamic environment using faster R-CNN"", 《IEEE INTERNATIONAL CONFERENCE ON COMPUTER & COMMUNICATIONS》 * |
| 刘士荣等: ""基于改进关键帧选择的RGB-DSLAM算法"", 《大连理工大学学报》 * |
| 林辉: ""基于CNN与VLAD融合的闭环检测"", 《现代计算机（专业版）》 * |
| 赵洋: ""基于深度学习的语义同步定位与地图构建"", 《万方数据库》 * |
Cited By (77)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110146099A (en) * | 2019-05-31 | 2019-08-20 | 西安工程大学 | A kind of synchronous superposition method based on deep learning |
| CN110728751A (en) * | 2019-06-19 | 2020-01-24 | 武汉科技大学 | Construction method of indoor 3D point cloud semantic map |
| CN110335319B (en) * | 2019-06-26 | 2022-03-18 | 华中科技大学 | Semantic-driven camera positioning and map reconstruction method and system |
| CN110335319A (en) * | 2019-06-26 | 2019-10-15 | 华中科技大学 | Camera positioning and the map reconstruction method and system of a kind of semantics-driven |
| CN112148817B (en) * | 2019-06-28 | 2023-09-29 | 理光软件研究所（北京）有限公司 | SLAM optimization method, device and system based on panorama |
| CN112148817A (en) * | 2019-06-28 | 2020-12-29 | 理光软件研究所（北京）有限公司 | Panoramic-map-based SLAM optimization method, device and system |
| CN110335317A (en) * | 2019-07-02 | 2019-10-15 | 百度在线网络技术（北京）有限公司 | Image processing method, device, equipment and medium based on terminal device positioning |
| CN110335317B (en) * | 2019-07-02 | 2022-03-25 | 百度在线网络技术（北京）有限公司 | Image processing method, device, equipment and medium based on terminal equipment positioning |
| CN110516527A (en) * | 2019-07-08 | 2019-11-29 | 广东工业大学 | A kind of vision SLAM winding detection improvement method of Case-based Reasoning segmentation |
| CN110390724A (en) * | 2019-07-12 | 2019-10-29 | 杭州凌像科技有限公司 | A kind of SLAM method with example segmentation |
| CN110610198A (en) * | 2019-08-22 | 2019-12-24 | 浙江工业大学 | Mask RCNN-based automatic oral CBCT image mandibular neural tube identification method |
| CN110610650A (en) * | 2019-08-27 | 2019-12-24 | 杭州电子科技大学 | Point cloud semantic map construction method based on deep learning and depth camera |
| CN110599542A (en) * | 2019-08-30 | 2019-12-20 | 北京影谱科技股份有限公司 | Method and device for local mapping of adaptive VSLAM (virtual local area model) facing to geometric area |
| CN110602393A (en) * | 2019-09-04 | 2019-12-20 | 南京博润智能科技有限公司 | Video anti-shake method based on image content understanding |
| CN110599522A (en) * | 2019-09-18 | 2019-12-20 | 成都信息工程大学 | Method for detecting and removing dynamic target in video sequence |
| CN110599522B (en) * | 2019-09-18 | 2023-04-11 | 成都信息工程大学 | Method for detecting and removing dynamic target in video sequence |
| CN110766024B (en) * | 2019-10-08 | 2023-05-23 | 湖北工业大学 | Deep learning-based visual odometer feature point extraction method and visual odometer |
| CN110766024A (en) * | 2019-10-08 | 2020-02-07 | 湖北工业大学 | Visual odometer feature point extraction method based on deep learning and visual odometer |
| CN110717052A (en) * | 2019-10-15 | 2020-01-21 | 山东大学 | Environment characterization method in service robot intelligent service |
| CN110738673A (en) * | 2019-10-21 | 2020-01-31 | 哈尔滨理工大学 | Visual SLAM method based on example segmentation |
| CN110866953A (en) * | 2019-10-31 | 2020-03-06 | Oppo广东移动通信有限公司 | Map construction method and device, and positioning method and device |
| WO2021083242A1 (en) * | 2019-10-31 | 2021-05-06 | Oppo广东移动通信有限公司 | Map constructing method, positioning method and system, wireless communication terminal, and computer-readable medium |
| CN110866953B (en) * | 2019-10-31 | 2023-12-29 | Oppo广东移动通信有限公司 | Map construction method and device, and positioning method and device |
| CN110986945A (en) * | 2019-11-14 | 2020-04-10 | 上海交通大学 | Local navigation method and system based on semantic height map |
| CN110986945B (en) * | 2019-11-14 | 2023-06-27 | 上海交通大学 | Local navigation method and system based on semantic altitude map |
| CN111160210A (en) * | 2019-12-24 | 2020-05-15 | 天津天地伟业机器人技术有限公司 | Video-based water flow velocity detection method and system |
| CN111160210B (en) * | 2019-12-24 | 2023-09-26 | 天地伟业技术有限公司 | Video-based water flow rate detection method and system |
| CN111105695A (en) * | 2019-12-31 | 2020-05-05 | 智车优行科技（上海）有限公司 | Map making method and device, electronic equipment and computer readable storage medium |
| CN111273772A (en) * | 2020-01-17 | 2020-06-12 | 江苏艾佳家居用品有限公司 | Augmented reality interaction method and device based on slam mapping method |
| CN111325842B (en) * | 2020-03-04 | 2023-07-28 | Oppo广东移动通信有限公司 | Map construction method, repositioning method and device, storage medium and electronic equipment |
| EP4113451A4 (en) * | 2020-03-04 | 2023-07-19 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Map construction method and apparatus, repositioning method and apparatus, storage medium, and electronic device |
| WO2021175022A1 (en) * | 2020-03-04 | 2021-09-10 | Oppo广东移动通信有限公司 | Map construction method and apparatus, repositioning method and apparatus, storage medium, and electronic device |
| CN111325842A (en) * | 2020-03-04 | 2020-06-23 | Oppo广东移动通信有限公司 | Map construction method, repositioning method and device, storage medium and electronic equipment |
| CN111239763A (en) * | 2020-03-06 | 2020-06-05 | 广州视源电子科技股份有限公司 | Object positioning method and device, storage medium and processor |
| CN111275026B (en) * | 2020-03-23 | 2022-09-13 | 复旦大学 | Three-dimensional point cloud combined semantic and instance segmentation method |
| CN111275026A (en) * | 2020-03-23 | 2020-06-12 | 复旦大学 | Three-dimensional point cloud combined semantic and instance segmentation method |
| CN111496784A (en) * | 2020-03-27 | 2020-08-07 | 山东大学 | Space environment identification method and system for robot intelligent service |
| CN111581313B (en) * | 2020-04-25 | 2023-05-23 | 华南理工大学 | Semantic SLAM robustness improvement method based on instance segmentation |
| CN111581313A (en) * | 2020-04-25 | 2020-08-25 | 华南理工大学 | Semantic SLAM robustness improvement method based on instance segmentation |
| CN111563442B (en) * | 2020-04-29 | 2023-05-02 | 上海交通大学 | Slam method and system for fusing point cloud and camera image data based on laser radar |
| CN111563442A (en) * | 2020-04-29 | 2020-08-21 | 上海交通大学 | Slam method and system for fusing point cloud and camera image data based on laser radar |
| CN111693047B (en) * | 2020-05-08 | 2022-07-05 | 中国航空工业集团公司西安航空计算技术研究所 | Visual navigation method for micro unmanned aerial vehicle in high-dynamic scene |
| CN111693047A (en) * | 2020-05-08 | 2020-09-22 | 中国航空工业集团公司西安航空计算技术研究所 | Visual navigation method for micro unmanned aerial vehicle in high-dynamic scene |
| CN111709328B (en) * | 2020-05-29 | 2023-08-04 | 北京百度网讯科技有限公司 | Vehicle tracking method and device and electronic equipment |
| CN111709328A (en) * | 2020-05-29 | 2020-09-25 | 北京百度网讯科技有限公司 | Vehicle tracking method and device and electronic equipment |
| WO2021238062A1 (en) * | 2020-05-29 | 2021-12-02 | 北京百度网讯科技有限公司 | Vehicle tracking method and apparatus, and electronic device |
| CN111665842A (en) * | 2020-06-09 | 2020-09-15 | 山东大学 | Indoor SLAM mapping method and system based on semantic information fusion |
| CN111665842B (en) * | 2020-06-09 | 2021-09-28 | 山东大学 | Indoor SLAM mapping method and system based on semantic information fusion |
| CN111882663A (en) * | 2020-07-03 | 2020-11-03 | 广州万维创新科技有限公司 | Visual SLAM closed-loop detection method achieved by fusing semantic information |
| CN111797938A (en) * | 2020-07-15 | 2020-10-20 | 燕山大学 | Semantic information and VSLAM fusion method for sweeping robot |
| CN111797938B (en) * | 2020-07-15 | 2022-03-15 | 燕山大学 | Semantic information and VSLAM fusion method for sweeping robot |
| CN111882613A (en) * | 2020-07-24 | 2020-11-03 | 中国科学院上海微系统与信息技术研究所 | Visual odometry method and device based on edge semantics, storage medium and equipment |
| WO2022021739A1 (en) * | 2020-07-30 | 2022-02-03 | 国网智能科技股份有限公司 | Humanoid inspection operation method and system for semantic intelligent substation robot |
| CN112258575A (en) * | 2020-10-13 | 2021-01-22 | 浙江大学 | Method for quickly identifying object in synchronous positioning and map construction |
| CN112418250B (en) * | 2020-12-01 | 2024-05-10 | 怀化学院 | Optimized matching method for complex 3D point cloud |
| CN112418250A (en) * | 2020-12-01 | 2021-02-26 | 怀化学院 | Optimized matching method for complex 3D point cloud |
| CN112560648B (en) * | 2020-12-09 | 2023-04-07 | 长安大学 | SLAM method based on RGB-D image |
| CN112560648A (en) * | 2020-12-09 | 2021-03-26 | 长安大学 | SLAM method based on RGB-D image |
| CN112734845A (en) * | 2021-01-08 | 2021-04-30 | 浙江大学 | Outdoor monocular synchronous mapping and positioning method fusing scene semantics |
| CN112785714A (en) * | 2021-01-29 | 2021-05-11 | 北京百度网讯科技有限公司 | Point cloud instance labeling method and device, electronic equipment and medium |
| CN112967341B (en) * | 2021-02-23 | 2023-04-25 | 湖北枫丹白露智慧标识科技有限公司 | Indoor visual positioning method, system, equipment and storage medium based on live-action image |
| CN112967341A (en) * | 2021-02-23 | 2021-06-15 | 湖北枫丹白露智慧标识科技有限公司 | Indoor visual positioning method, system, equipment and storage medium based on live-action image |
| CN113724299B (en) * | 2021-08-30 | 2023-09-19 | 上海大学 | Method for tracking three-dimensional track of target by mobile robot based on electrohydraulic adjustable focus lens |
| CN114092388B (en) * | 2021-08-30 | 2024-08-13 | 河南笛卡尔机器人科技有限公司 | Obstacle detection method based on monocular camera and odometer |
| CN113724299A (en) * | 2021-08-30 | 2021-11-30 | 上海大学 | Method for tracking three-dimensional track of target by mobile robot based on electro-hydraulic adjustable focus lens |
| CN114092388A (en) * | 2021-08-30 | 2022-02-25 | 河南笛卡尔机器人科技有限公司 | Obstacle detection method based on monocular camera and odometer |
| CN114216461A (en) * | 2021-09-29 | 2022-03-22 | 杭州图灵视频科技有限公司 | Panoramic camera-based indoor positioning method and system for mobile robot |
| CN113916245A (en) * | 2021-10-09 | 2022-01-11 | 上海大学 | Semantic map construction method based on instance segmentation and VSLAM |
| CN113916245B (en) * | 2021-10-09 | 2024-07-19 | 上海大学 | Semantic map construction method based on instance segmentation and VSLAM |
| CN114359493A (en) * | 2021-12-20 | 2022-04-15 | 中国船舶重工集团公司第七0九研究所 | Method and system for generating three-dimensional semantic map for unmanned ship |
| CN114494825A (en) * | 2021-12-31 | 2022-05-13 | 重庆特斯联智慧科技股份有限公司 | Robot positioning method and device |
| CN114494825B (en) * | 2021-12-31 | 2024-04-19 | 重庆特斯联智慧科技股份有限公司 | Robot positioning method and device |
| CN115496977A (en) * | 2022-09-14 | 2022-12-20 | 北京化工大学 | Target detection method and device based on multi-mode sequence data fusion |
| CN116168393B (en) * | 2023-01-17 | 2023-08-25 | 浙江大学 | Automatic semantic annotation data generation method and device based on point cloud neural radiation field |
| CN116168393A (en) * | 2023-01-17 | 2023-05-26 | 浙江大学 | Automatic semantic annotation data generation method and device based on point cloud neural radiation field |
| CN116128734A (en) * | 2023-04-17 | 2023-05-16 | 湖南大学 | Image stitching method, device, equipment and medium based on deep learning |
| CN117132648A (en) * | 2023-04-28 | 2023-11-28 | 荣耀终端有限公司 | Visual positioning method, electronic equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109816686A (en) | Robot semanteme SLAM method, processor and robot based on object example match | |
| Han et al. | Dynamic scene semantics SLAM based on semantic segmentation | |
| Wirges et al. | Object detection and classification in occupancy grid maps using deep convolutional networks | |
| Caraffi et al. | Off-road path and obstacle detection using decision networks and stereo vision | |
| WO2020170014A1 (en) | Object counting and instance segmentation using neural network architectures with image-level supervision | |
| CN111814683A (en) | Robust visual SLAM method based on semantic prior and deep learning features | |
| CN110688905B (en) | Three-dimensional object detection and tracking method based on key frame | |
| CN106778856A (en) | A kind of object identification method and device | |
| Chakravarty et al. | GEN-SLAM: Generative modeling for monocular simultaneous localization and mapping | |
| Bera et al. | Online parameter learning for data-driven crowd simulation and content generation | |
| CN109063549A (en) | High-resolution based on deep neural network is taken photo by plane video moving object detection method | |
| Lu et al. | A cnn-transformer hybrid model based on cswin transformer for uav image object detection | |
| Dwibedi et al. | Deep cuboid detection: Beyond 2d bounding boxes | |
| Yang et al. | Visual SLAM based on semantic segmentation and geometric constraints for dynamic indoor environments | |
| CN109508686A (en) | A kind of Human bodys' response method based on the study of stratification proper subspace | |
| Yang et al. | [Retracted] A Method of Image Semantic Segmentation Based on PSPNet | |
| Fei et al. | Self-supervised learning for pre-training 3d point clouds: A survey | |
| Pham et al. | Pencilnet: Zero-shot sim-to-real transfer learning for robust gate perception in autonomous drone racing | |
| Ning et al. | Point-voxel and bird-eye-view representation aggregation network for single stage 3D object detection | |
| Gelen et al. | An artificial neural slam framework for event-based vision | |
| Liao | SLAMORE: SLAM with object recognition for 3D radio environment reconstruction | |
| Liu et al. | VL-MFL: UAV Visual Localization Based on Multi-Source Image Feature Learning | |
| Xiong et al. | MLP-Pose: Human pose estimation by MLP-mixer | |
| Tao et al. | 3d semantic vslam of indoor environment based on mask scoring rcnn | |
| Chen et al. | Novel learning framework for optimal multi-object video trajectory tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190528 |