CN111504331B - Method and device for positioning panoramic intelligent vehicle from coarse to fine - Google Patents
Method and device for positioning panoramic intelligent vehicle from coarse to fine Download PDFInfo
- Publication number
- CN111504331B CN111504331B CN202010357912.XA CN202010357912A CN111504331B CN 111504331 B CN111504331 B CN 111504331B CN 202010357912 A CN202010357912 A CN 202010357912A CN 111504331 B CN111504331 B CN 111504331B
- Authority
- CN
- China
- Prior art keywords
- panoramic
- image
- neural network
- convolutional neural
- positioning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention discloses a method and a device for positioning a panoramic intelligent vehicle from thick to thin. The method uses a panoramic image containing 360-degree environment information and outputs a positioning result by combining rough matching and fine matching. The method can be used for positioning under the conditions of crossing seasons, crossing illumination, crossing night and day, crossing different traversals in the same route and the like, is high in precision, low in omission factor, low in price and good in cross-platform performance, and can be used for sensing the large-view-field all-directional environment. The application requirement of accurate positioning of the intelligent vehicle can be well met.
Description
Technical Field
The invention belongs to the technical fields of image processing technology, pattern recognition technology and computer vision, and relates to a method and a device for positioning a panoramic intelligent vehicle from rough to fine.
Background
Visual localization in vehicle navigation is an important image retrieval task to determine the optimal position. It is very difficult to develop an effective algorithm to solve the positioning problem of the vehicle because serious appearance changes bring huge challenges and obstacles as the vehicle moves, and diurnal cycles, seasonal changes, highly dynamic objects, viewing angle changes and illumination changes can be considered as serious appearance changes, which severely limits the robustness of the navigation system.
The vehicle vision positioning based on the panoramic image is the most advanced positioning field. The introduction of panoramic images can greatly reduce the influence of viewpoint change on visual positioning. In addition, since continuous attention is required in all directions during driving, and an autonomous vehicle requires all-around perception, introduction of a panoramic image is also necessary. Most of panoramic information acquisition modes at the present stage are multi-camera multi-angle shooting and then panoramic stitching, the method has no real-time performance, and compared with the method using one camera, the cost is increased, and the panoramic annular system imaging enables the panoramic image acquisition for vehicle positioning to have certain real-time performance and low cost. In addition, in the aspect of a positioning algorithm, although a panoramic image has more scene features than a simple forward image, the positioning is difficult due to the too large similarity of scenes at nearby positions, so that the vehicle visual positioning method must be robust enough to distinguish the scenes at different positions.
Disclosure of Invention
The embodiment of the invention provides a method and a device for positioning a panoramic intelligent vehicle from coarse to fine, and aims to solve the technical problems of insufficient robustness, low positioning accuracy and low cost of a positioning algorithm in the conventional vehicle positioning technology.
In one aspect of the embodiment of the invention, a method for positioning a panoramic intelligent vehicle from thick to thin is provided, and comprises the following steps:
acquiring a panoramic database image of a route to be navigated, wherein the panoramic database image comprises a surrounding panoramic subset and a planar panoramic subset, and acquiring a panoramic query image of a position to be positioned, wherein the panoramic query image comprises a surrounding panoramic image and a planar panoramic image of the position to be positioned;
in a coarse matching stage, inputting the surrounding panoramic image into a trained scene recognition model, and outputting a convolutional neural network feature descriptor;
in a fine matching stage, extracting point feature descriptors from the planar panoramic image;
in a coarse positioning stage, matching the convolutional neural network feature descriptors with convolutional neural network feature descriptors corresponding to the surrounding panorama subset one by one, and selecting a coarse range for a correct positioning result;
and in the fine positioning stage, matching the point feature descriptors with the rough range to obtain the position with the most accurate positioning.
Further, there are two ways to obtain the surround panoramic image and the planar panoramic image:
firstly, a common pinhole camera is placed at a certain position in a multi-angle manner to shoot a plurality of plane images with forward visual angles, the plane images have environment information of 360 degrees at the position, the plane images are regarded as plane panoramic images, and the plane images are spliced to obtain the surrounding panoramic image;
secondly, a panoramic annular optical system is used, after a scene from the environment enters a panoramic annular lens, scene information is projected on a unit sphere, the scene information on the unit sphere is projected outwards onto a cylindrical surface and is expanded to obtain the surrounding panoramic image, and the scene information on the unit sphere is projected outwards onto four surfaces of a cube and is expanded to obtain the planar panoramic image.
Further, the surround panorama subset and the planar panorama subset are obtained in the following manner:
collecting surrounding panoramic images on a route to be navigated to obtain a surrounding panoramic subset; and collecting the planar panoramic image on the route to be navigated to obtain the planar panoramic subset.
Further, in a coarse matching stage, inputting the surround panoramic image into a trained scene recognition model, and outputting a convolutional neural network feature descriptor, including:
training a convolutional neural network model by using a surrounding panoramic image data set to obtain a scene recognition model;
inputting the surrounding panoramic image into the trained scene recognition model;
the entire surround panoramic image represents a tensor output by the scene recognition model, which is called a convolutional neural network feature descriptor.
Further, the convolutional neural network model is a basic convolutional network model, or a layer of NetVLAD model is added after the basic convolutional network.
Further, in the fine matching stage, extracting point feature descriptors for the planar panoramic image, including:
the planar panoramic image is a plurality of planar images which can cover 360-degree scene information, are well arranged according to scenes and keep the same arrangement sequence with the images of the planar panoramic subset;
and respectively extracting point feature descriptors from the multiple planar panoramic images at the positions to be positioned.
Further, in a coarse positioning stage, the convolutional neural network feature descriptors match the convolutional neural network feature descriptors corresponding to the surround panorama subset one by one, and a coarse range is selected for a correct positioning result, including:
all convolutional neural network feature descriptors corresponding to the surrounding panorama subset are obtained in advance before navigation, and are output to the same scene recognition model together with the convolutional neural network feature descriptors obtained in the rough matching stage;
and searching the convolutional neural network feature descriptors in convolutional neural network feature descriptors corresponding to the panoramic database image one by one, and positioning the position to be navigated at one of the positions corresponding to the first several candidates with the highest score.
Further, searching the convolutional neural network feature descriptors in convolutional neural network feature descriptors corresponding to the panoramic database image one by one, and positioning the position to be navigated at one of the positions corresponding to the first several candidates with the highest score, including:
a convolutional neural network feature descriptor extracted from the panoramic surround image of the position to be located is denoted as x, and a convolutional neural network feature descriptor extracted from the panoramic surround subset is denoted as: y is1，y2，…，ykWherein k is the number of panoramic surround images in the panoramic surround subset;
for x and y1，y2，…，ykAnd (4) calculating the Euclidean distances one by one, comparing the sizes of the Euclidean distances, wherein the smaller the Euclidean distance is, the closer the corresponding images approach to the same position, and the position to be navigated is positioned at one of the positions corresponding to the first few candidates with the minimum Euclidean distance.
Further, in the fine positioning stage, matching the point feature descriptors with the coarse range selected in the coarse positioning stage to obtain the most accurate position, including:
comparing the point feature descriptors obtained in the fine matching stage with the point feature descriptors corresponding to the planar panoramic image in the roughly positioned range;
calculating the number of matching pairs between the images by the point feature descriptors through a plane mapping relation;
the more matching pairs of point feature descriptors, the closer they approach the same position, and the most accurate positioning result is found from the rough range of positioning.
In a second aspect, according to an embodiment of the present invention, there is provided a panoramic intelligent vehicle positioning device from thick to thin, including:
the panoramic database image and panoramic query image acquisition module is used for acquiring a panoramic database image of a route to be navigated, wherein the panoramic database image comprises a surrounding panoramic subset and a planar panoramic subset, and acquiring a panoramic query image of a position to be positioned, and the panoramic query image comprises a surrounding panoramic image and a planar panoramic image of the position to be positioned;
the rough matching module is used for inputting the surrounding panoramic image into a trained scene recognition model in a rough matching stage and outputting a convolutional neural network feature descriptor;
the fine matching module is used for extracting point feature descriptors from the planar panoramic image;
the rough positioning module is used for matching the convolutional neural network feature descriptors with the convolutional neural network feature descriptors corresponding to the surrounding panorama subset one by one, and selecting a rough range for a correct positioning result;
and the fine positioning module is used for matching the point feature descriptors with the rough range to obtain the position with the most accurate positioning.
According to the embodiment of the invention, the convolutional neural network feature descriptors acquired in the coarse matching stage have excellent overall appearance environment distinguishing capability, the point feature descriptors acquired in the fine matching stage have excellent detail searching capability, and the combination of the descriptors in the coarse stage and the fine stage enables the coarse-to-fine panoramic intelligent vehicle positioning method to have good robustness and provide robust positioning results under various severe conditions.
By obtaining a coarse range of positioning from the coarse positioning stage to the fine positioning, a more accurate positioning result is obtained, enabling significantly improved positioning accuracy compared to performing only a single stage of positioning.
The vehicle positioning method can sense the scene information of a position of 360 degrees in the driving process of the vehicle by adopting the panoramic image data, is superior to the visual positioning based on a common forward camera which can sense only a local range of a certain position, and has the advantage of large environmental sensing range.
The coarse-to-fine intelligent vehicle positioning method provides a robust feature descriptor and a precise positioning method, can acquire a large-range environment perception, overcomes a plurality of difficulties of the conventional vehicle positioning method, and can effectively solve various problems in vehicle positioning.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a flow chart of a method for positioning a panoramic intelligent vehicle from coarse to fine according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a panoramic annular optical system provided by an embodiment of the present invention;
FIG. 3 is a schematic diagram of a panoramic annular system provided by an embodiment of the present invention expanded into a surrounding panoramic image;
FIG. 4 is a schematic diagram of a panoramic annular zone system provided by the embodiment of the present invention expanded into a planar panoramic image;
FIG. 5 is a line graph of coarse positioning and coarse-to-fine positioning accuracy across summer and winter season changes according to an embodiment of the present invention;
FIG. 6 is a line graph of coarse positioning and coarse-to-fine positioning accuracy across morning and afternoon light changes according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of an example of coarse-to-fine positioning across summer and winter season changes according to an embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating an example of coarse-to-fine positioning across afternoon lighting changes according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a coarse-to-fine panoramic intelligent vehicle positioning device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1:
in accordance with an embodiment of the present invention, there is provided a coarse-to-fine embodiment of a panoramic intelligent vehicle positioning method, it being noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions and that, although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than that illustrated herein.
Fig. 1 is a flowchart of a method for positioning a panoramic intelligent vehicle from coarse to fine according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S102, a panoramic database image of a route to be navigated is obtained, the panoramic database image comprises a surrounding panoramic subset and a plane panoramic subset, a panoramic query image of a position to be positioned is obtained, and the panoramic query image comprises a surrounding panoramic image and a plane panoramic image of the position to be positioned.
And step S104, inputting the surrounding panoramic image into the trained scene recognition model in a coarse matching stage, and outputting a convolutional neural network feature descriptor.
In the above steps, a convolutional neural network model, that is, a scene recognition model, is trained using a surround panoramic image data set with large scene differences, then a surround panoramic image is input into the trained scene recognition model, and finally a tensor output by the scene recognition model is represented by the whole surround panoramic image, and the tensor is referred to as a convolutional neural network feature descriptor.
And step S106, extracting point feature descriptors from the planar panoramic image in the fine matching stage.
In the above steps, the planar panoramic image is a plurality of planar images which can cover 360 degrees of scene information, the planar panoramic images are arranged according to scenes, the arrangement sequence of the planar panoramic images is kept the same as that of the images of the planar panoramic subsets, and then point feature descriptors are respectively extracted from the plurality of planar panoramic images at the positions to be positioned.
Step S108, in the coarse positioning stage, the convolutional neural network feature descriptors acquired in the coarse matching stage are matched with the convolutional neural network feature descriptors corresponding to the surrounding panorama subset one by one, and a coarse range is selected for a correct positioning result.
In the above steps, all the convolutional neural network feature descriptors corresponding to the surround panorama subset are obtained in advance before navigation, and are output to the same scene recognition model with the convolutional neural network feature descriptors obtained in the rough matching stage.
In the above steps, the convolutional neural network feature descriptors obtained in the rough matching stage are searched one by one in the convolutional neural network feature descriptors corresponding to the panoramic database image, and the first several candidates in the most approximate panoramic database image are searched.
Step S110, in the fine positioning stage, the point feature descriptors obtained in the fine matching stage are only matched with the rough range selected in the rough positioning stage, and the most accurate position is positioned.
In the above steps, the rough positioning stage has selected a rough range of positioning for the fine positioning stage, and in addition, the point feature descriptors corresponding to the planar panorama subset are all obtained in advance before navigation, and belong to the same type of point feature descriptors as the point feature descriptors obtained in the fine matching stage.
In the above step, the point feature descriptors obtained in the fine matching stage are only compared with the point feature descriptors corresponding to the planar panoramic image in the coarse range of positioning, and the more matching pairs of the point feature descriptors are, the closer they approach to the same position, and the most accurate positioning result is found from the coarse range of positioning.
The steps of the application are that a panoramic database image of a route to be navigated is obtained in advance, the panoramic database image comprises a surrounding panoramic subset and a plane panoramic subset, a panoramic query image of a position to be positioned is obtained, the surrounding panoramic image comprises a surrounding panoramic image and a plane panoramic image of the position, the surrounding panoramic image is input into a trained scene recognition model through a coarse matching stage, a convolutional neural network feature descriptor is output, a point feature descriptor is extracted from the plane panoramic image through a fine matching stage, then the convolutional neural network feature descriptor obtained in the coarse matching stage is matched with the convolutional neural network feature descriptor corresponding to the surrounding panoramic subset one by one in the coarse positioning stage, a coarse range is selected for a correct positioning result, and finally the point feature descriptor obtained in the fine matching stage is matched with the point feature descriptor only in the coarse range selected in the coarse positioning stage through the fine positioning stage, the most accurate position is located.
The scheme firstly utilizes the advantage that the panoramic image can provide large-scale environment perception, and obtains the panoramic query image and the panoramic database image, wherein the obtaining method comprises two methods: using a plurality of common pinhole camera arrangements; the panoramic annular optical system is used, and the panoramic image is used for positioning the vehicle, so that the vehicle can recognize an omnibearing scene easily; secondly, extracting a convolutional neural network feature descriptor surrounding the panoramic image by using a coarse matching stage, and describing the whole image by using a tensor, wherein the convolutional neural network feature descriptor is very robust to the change of the whole environment appearance and is suitable for searching a positioning approximate range; subsequently, a point feature descriptor of the planar panoramic image is extracted in a fine matching stage, and the point feature descriptor has the capability of extracting detailed information and is more suitable for fine positioning; searching the panoramic query image in the panoramic database image one by one according to the convolutional neural network feature descriptors extracted in the coarse matching stage, and selecting a coarse range for a correct positioning result; and finally, matching the point feature descriptors with the panoramic query image one by one in a rough range selected in the rough positioning stage according to the point feature descriptors extracted in the fine matching stage to finally obtain the optimal positioning result. The method for positioning the panoramic intelligent vehicle from thick to thin has good robustness for positioning the vehicle and can ensure higher positioning precision. Meanwhile, the panoramic image is used for positioning, omnibearing visual perception can be provided, and the influence of visual angle change on a positioning result is greatly weakened due to the omnibearing perception.
Optionally, according to the foregoing embodiment of the present application, obtaining in advance a panoramic database image of a route to be navigated, where the panoramic database image includes a surround panorama subset and a planar panorama subset, and obtaining a panoramic query image of a location to be located, where the panoramic query image includes a surround panoramic image and a planar panoramic image of the location, includes:
in step S1021, a surround panoramic image is acquired.
Specifically, there are two types of acquisition modes for the surrounding panoramic image in the above steps: first, a common pinhole camera is placed at a certain position at multiple angles to shoot a plurality of plane images with forward viewing angles, the images have environment information of 360 degrees at the position, and the images are spliced to obtain a surrounding panoramic image. Secondly, a panoramic annular optical system is used, fig. 2 is a schematic imaging diagram of the panoramic annular optical system, the panoramic annular system provided according to fig. 3 is used for imaging and unfolding to a surrounding panoramic image, and the specific unfolding method is as follows: after the scenery from the environment enters the panoramic annular lens, the scenery information is projected on the unit sphere. The scenery information on the unit ball is projected outwards to the cylindrical surface and is expanded, and a surrounding panoramic image can be obtained.
In step S1022, a planar panoramic image is acquired.
Specifically, there are two types of ways for acquiring the planar panoramic image in the above steps: first, a common pinhole camera is placed at a certain position at multiple angles to shoot a plurality of planar images with forward viewing angles, and the images have environment information of 360 degrees at the position and can be regarded as planar panoramic images. Secondly, using a panoramic annular optical system, the panoramic annular system provided according to fig. 4 is used for image development into a planar panoramic image schematic diagram, and the specific development method is as follows: after the scenery from the environment enters the panoramic annular lens, the scenery information is projected on the unit ball, the scenery information on the unit ball is projected outwards to four faces of the cube and is expanded, and a planar panoramic image can be obtained.
Compared with the common pinhole camera, the panoramic annular optical system only needs one panoramic annular camera, and compared with the pinhole camera which needs a plurality of cameras for acquiring panoramic images, the panoramic annular optical system saves devices and cost, has relatively low price, omits the image splicing process, and has better real-time property for acquiring panoramic images.
Optionally, according to the above embodiment of the present application, in the rough matching stage, the surrounding panoramic image is input into a trained scene recognition model, and a convolutional neural network feature descriptor is output.
And S1041, selecting a scene recognition model.
Specifically, the basic convolutional neural network model that can be used by the scene recognition model in the above steps includes: alexnet, VGG16, Resnet18, etc.; a NetVLAD layer is added behind a basic convolutional neural network, so that the positioning accuracy can be improved, and the convolutional neural network model at the moment comprises the following components: alexnet + NetVLAD, VGG16+ NetVLAD, Resnet18+ NetVLAD, etc., and the feature descriptor at this time is called NetVLAD feature descriptor.
Optionally, according to the above embodiment of the present application, in the fine matching stage, the point feature descriptors are extracted from the planar panoramic image.
In step S1061, a point feature descriptor is selected.
Specifically, the point feature descriptors in the above steps may be traditional manual point feature descriptors such as SIFT, SURF, ORB, etc., or may also be point feature descriptors based on deep learning such as suppoint, Geodesc, etc.
Optionally, according to the above embodiment of the present application, in the coarse positioning stage, the convolutional neural network feature descriptors obtained in the coarse matching stage are matched with the convolutional neural network feature descriptors corresponding to the surrounding panorama subset one by one, and a coarse range is selected for a correct positioning result.
In step S1081, the panoramic query image and the panoramic database image may be represented by a convolutional neural network feature descriptor.
Specifically, the above-mentioned step of panoramic query image convolution neural network descriptor may be represented by a vector x ═ x (x)1,x2,…,xn) Where the convolutional neural network feature descriptor dimension is n-dimension; the feature descriptor of the convolutional neural network of the panoramic database image can be represented by y1＝(y11,y12,…,y1n)，y2＝(y21,y22,…,y2n)，…，yk＝(yk1,yk2,…,ykn) Where the convolutional neural network feature descriptor dimension is n-dimensional, there are k panoramic surround images in the panoramic surround subset.
Step S1082, the convolutional neural network feature descriptors obtained in the rough matching stage are matched with convolutional neural network feature descriptors corresponding to surrounding panorama subsets in the panoramic database image one by one, and matching is performed by calculating Euclidean distances.
Specifically, the above-mentioned euclidean distance calculation formula in the step is:
wherein x is (x)1,x2,…,xn) Representing a panoramic query image convolutional neural network descriptor, y ═ y1,y2,…,yn) The convolutional neural network descriptor of the panoramic database image is represented, d (x, y) represents the Euclidean distance between the panoramic query image and the panoramic database image, and the smaller the Euclidean distance is, the more similar the corresponding image content is, the closer the panoramic query image is to the position of the corresponding panoramic database image.
Optionally, according to the above embodiment of the present application, in the fine positioning stage, the point feature descriptors obtained in the fine matching stage are only matched with the rough range selected in the rough positioning stage, so as to position the most accurate position.
And step S10101, matching the point feature descriptors acquired in the fine matching stage with the point feature descriptors only in the rough range selected in the rough positioning stage, and performing plane mapping on the planar panoramic image to obtain usable plane mapping including basic matrix mapping and homography matrix mapping.
Specifically, the formula of the basic matrix mapping is as follows:
q1 TFq2＝0
the formula of the homography matrix mapping is:
q1 THq2＝0
wherein q is1，q2The pixel coordinates respectively represent the corresponding planar panoramic images in the panoramic query image and the panoramic database image, and the inner points which accord with the basic matrix relationship and the outer points which do not accord with the basic matrix relationship on the images are further distinguished by calculating the basic matrix relationship F or the homography matrix H corresponding to every two images. Multiple plan views containing panoramic informationThe corresponding parts of the image respectively calculate the corresponding number of interior points, which is recorded as N1，N2，…，NnThe larger the number of inliers, the more the corresponding images tend to be on the same plane. The sum of the number of corresponding points N is N1+N2+…+NnThe more, the more the images tend to be in the same position.
According to the line graph crossing the rough positioning and the rough-to-fine positioning accuracy under summer and winter season change provided in fig. 5 and the line graph crossing the rough positioning and the rough-to-fine positioning accuracy under morning and afternoon light change provided in fig. 6, the rough-to-fine vehicle positioning method is proved to have higher accuracy and robustness under severe ambient illumination change, in addition, the accuracy of only rough positioning is compared with the accuracy of rough-to-fine positioning, and the positioning accuracy can be remarkably improved by the rough-to-fine positioning relative to single rough positioning.
Fig. 7 provides a schematic diagram of an example of coarse-to-fine positioning under the condition of changing across summer and winter seasons, and fig. 8 provides a schematic diagram of an example of coarse-to-fine positioning under the condition of changing across the illumination in the afternoon, which proves that the coarse-to-fine panoramic intelligent vehicle positioning method is successful in image retrieval and vehicle positioning.
Example 2:
the application also provides a coarse-to-fine panoramic intelligent vehicle positioning device, which is used for executing the panoramic intelligent vehicle positioning method in the embodiment 1, and fig. 9 is a schematic structural diagram of the coarse-to-fine panoramic intelligent vehicle positioning device according to the embodiment of the invention, and the device comprises:
and the panoramic database image and panoramic query image acquisition module is used for acquiring a panoramic database image of the route to be navigated, including a surrounding panoramic subset and a planar panoramic subset, and acquiring a panoramic query image of the position to be positioned, including a surrounding panoramic image and a planar panoramic image of the position.
And the rough matching module is used for inputting the surrounding panoramic image into the trained scene recognition model and outputting the convolutional neural network feature descriptor.
And the fine matching module is used for extracting point feature descriptors from the planar panoramic image.
And the rough positioning module is used for matching the convolutional neural network feature descriptors acquired in the rough matching stage with the convolutional neural network feature descriptors corresponding to the surrounding panorama subset one by one, and selecting a rough range for a correct positioning result.
And the fine positioning module is used for matching the point feature descriptors acquired in the fine matching stage with the rough range selected in the rough positioning stage only to position the most accurate position.
Firstly, acquiring a panoramic database image of a route to be navigated and a panoramic query image of a position to be positioned; secondly, extracting a convolutional neural network feature descriptor of the panoramic query image by using a coarse matching stage; then, in a fine matching stage, extracting point feature descriptors of the panoramic query image; performing similarity comparison with the convolutional neural network feature descriptors of the panoramic database image one by one according to the convolutional neural network feature descriptors of the panoramic query image extracted in the rough matching stage, and searching a rough range of a correct positioning result; finally, comparing the number of image matching pairs according to the point feature descriptors of the panoramic query image extracted in the fine matching stage and the point feature descriptors of the image corresponding to the rough range found in the rough positioning stage one by one, and finding the optimal positioning position.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual or direct or communication connection may be an indirect or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method for positioning a panoramic intelligent vehicle from coarse to fine is characterized by comprising the following steps:
acquiring a panoramic database image of a route to be navigated, wherein the panoramic database image comprises a surrounding panoramic subset and a planar panoramic subset, and acquiring a panoramic query image of a position to be positioned, wherein the panoramic query image comprises a surrounding panoramic image and a planar panoramic image of the position to be positioned;
in a rough matching stage, inputting the surrounding panoramic image into a trained scene recognition model, and outputting a convolutional neural network feature descriptor, wherein the whole surrounding panoramic image is represented by a tensor output by the scene recognition model, and the tensor is called the convolutional neural network feature descriptor;
in a fine matching stage, extracting point feature descriptors from the planar panoramic image;
in a coarse positioning stage, matching the convolutional neural network feature descriptors with convolutional neural network feature descriptors corresponding to the surrounding panorama subset one by one, and selecting a coarse range for a correct positioning result;
and in the fine positioning stage, matching the point feature descriptors with the rough range to obtain the position with the most accurate positioning.
2. The method of claim 1, wherein the surround panoramic image and the planar panoramic image are obtained in two ways:
firstly, a common pinhole camera is placed at a certain position in a multi-angle manner to shoot a plurality of plane images with forward visual angles, the plane images have environment information of 360 degrees at the position, the plane images are regarded as plane panoramic images, and the plane images are spliced to obtain the surrounding panoramic image;
secondly, a panoramic annular optical system is used, after a scene from the environment enters a panoramic annular lens, scene information is projected on a unit sphere, the scene information on the unit sphere is projected outwards onto a cylindrical surface and is expanded to obtain the surrounding panoramic image, and the scene information on the unit sphere is projected outwards onto four surfaces of a cube and is expanded to obtain the planar panoramic image.
3. The method of claim 1, wherein the surround panorama subset and the planar panorama subset are obtained by:
collecting surrounding panoramic images on a route to be navigated to obtain a surrounding panoramic subset; and collecting the planar panoramic image on the route to be navigated to obtain the planar panoramic subset.
4. The method of claim 1, wherein in the coarse matching stage, inputting the surround panoramic image into a trained scene recognition model, and outputting a convolutional neural network feature descriptor, comprising:
training a convolutional neural network model by using a surrounding panoramic image data set to obtain a scene recognition model;
inputting the surrounding panoramic image into the trained scene recognition model;
the entire surround panoramic image represents a tensor output by the scene recognition model, which is called a convolutional neural network feature descriptor.
5. The method of claim 4, wherein the convolutional neural network model is a basic convolutional network model, or a basic convolutional network followed by a layer of NetVLAD model.
6. The method of claim 1, wherein extracting point feature descriptors for the planar panoramic image in a fine matching stage comprises:
the planar panoramic image is a plurality of planar images which can cover 360-degree scene information, are well arranged according to scenes and keep the same arrangement sequence with the images of the planar panoramic subset;
and respectively extracting point feature descriptors from the multiple planar panoramic images at the positions to be positioned.
7. The method of claim 1, wherein in the coarse positioning stage, the convolutional neural network feature descriptors match the convolutional neural network feature descriptors corresponding to the surround panorama subset one to one, and a coarse range is selected for a correct positioning result, comprising:
all convolutional neural network feature descriptors corresponding to the surrounding panorama subset are obtained in advance before navigation, and are output to the same scene recognition model together with the convolutional neural network feature descriptors obtained in the rough matching stage;
and searching the convolutional neural network feature descriptors in convolutional neural network feature descriptors corresponding to the panoramic database image one by one, and positioning the position to be navigated at one of the positions corresponding to the first several candidates with the highest score.
8. The method of claim 7, wherein the convolutional neural network feature descriptors are searched one by one in convolutional neural network feature descriptors corresponding to panoramic database images, and the position to be navigated is located at one of the positions corresponding to the first several candidates with the highest scores, and the method comprises the following steps:
the feature descriptor of the convolutional neural network extracted from the panoramic surrounding image of the position to be positioned is marked asxAnd extracting a convolutional neural network feature descriptor from the panoramic surround subset as:y 1 ，y 2 ，…，y k wherein k is the number of panoramic surround images in the panoramic surround subset;
to pairxAndy 1 ，y 2 ，…，y k and (4) calculating the Euclidean distances one by one, comparing the sizes of the Euclidean distances, wherein the smaller the Euclidean distance is, the closer the corresponding images approach to the same position, and the position to be navigated is positioned at one of the positions corresponding to the first few candidates with the minimum Euclidean distance.
9. The method of claim 1, wherein in the fine positioning stage, matching the point feature descriptors with the coarse range selected in the coarse positioning stage to obtain a position with the most accurate positioning comprises:
comparing the point feature descriptors obtained in the fine matching stage with the point feature descriptors corresponding to the planar panoramic image in the roughly positioned range;
calculating the number of matching pairs between the images by the point feature descriptors through a plane mapping relation;
the more matching pairs of point feature descriptors, the closer they approach the same position, and the most accurate positioning result is found from the rough range of positioning.
10. The utility model provides a by thick to thin panorama intelligent vehicle positioner which characterized in that includes:
the panoramic database image and panoramic query image acquisition module is used for acquiring a panoramic database image of a route to be navigated, wherein the panoramic database image comprises a surrounding panoramic subset and a planar panoramic subset, and acquiring a panoramic query image of a position to be positioned, and the panoramic query image comprises a surrounding panoramic image and a planar panoramic image of the position to be positioned;
the rough matching module is used for inputting the surrounding panoramic image into a trained scene recognition model in a rough matching stage and outputting a convolutional neural network feature descriptor, wherein the whole surrounding panoramic image is represented by a tensor output by the scene recognition model, and the tensor is called the convolutional neural network feature descriptor;
the fine matching module is used for extracting point feature descriptors from the planar panoramic image;
the rough positioning module is used for matching the convolutional neural network feature descriptors with the convolutional neural network feature descriptors corresponding to the surrounding panorama subset one by one, and selecting a rough range for a correct positioning result;
and the fine positioning module is used for matching the point feature descriptors with the rough range to obtain the position with the most accurate positioning.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010357912.XA CN111504331B (en) | 2020-04-29 | 2020-04-29 | Method and device for positioning panoramic intelligent vehicle from coarse to fine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010357912.XA CN111504331B (en) | 2020-04-29 | 2020-04-29 | Method and device for positioning panoramic intelligent vehicle from coarse to fine |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111504331A CN111504331A (en) | 2020-08-07 |
| CN111504331B true CN111504331B (en) | 2021-09-14 |
Family
ID=71874942
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010357912.XA Active CN111504331B (en) | 2020-04-29 | 2020-04-29 | Method and device for positioning panoramic intelligent vehicle from coarse to fine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111504331B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112232359B (en) * | 2020-09-29 | 2022-10-21 | 中国人民解放军陆军炮兵防空兵学院 | Visual tracking method based on mixed level filtering and complementary characteristics |
| CN112689114B (en) * | 2021-03-11 | 2021-06-22 | 太平金融科技服务（上海）有限公司 | Method, apparatus, device and medium for determining target position of vehicle |
| CN113008252B (en) * | 2021-04-15 | 2023-08-22 | 东莞市异领电子有限公司 | High-precision navigation device and navigation method based on panoramic photo |
| CN114860976B (en) * | 2022-04-29 | 2023-05-05 | 长沙公交智慧大数据科技有限公司 | Image data query method and system based on big data |
| CN115187667B (en) * | 2022-09-08 | 2022-12-20 | 中国科学院合肥物质科学研究院 | Cognitive understanding-based large scene accurate positioning method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103310231A (en) * | 2013-06-24 | 2013-09-18 | 武汉烽火众智数字技术有限责任公司 | Auto logo locating and identifying method |
| CN104112122A (en) * | 2014-07-07 | 2014-10-22 | 叶茂 | Vehicle logo automatic identification method based on traffic video |
| CN109657552A (en) * | 2018-11-16 | 2019-04-19 | 北京邮电大学 | The vehicle type recognition device being cold-started across scene and method are realized based on transfer learning |
| CN110866079A (en) * | 2019-11-11 | 2020-03-06 | 桂林理工大学 | Intelligent scenic spot real scene semantic map generating and auxiliary positioning method |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3650578B2 (en) * | 2000-09-28 | 2005-05-18 | 株式会社立山アールアンドディ | Panoramic image navigation system using neural network to correct image distortion |
| CN107239555A (en) * | 2017-06-08 | 2017-10-10 | 深圳市唯特视科技有限公司 | The visual example search method that a kind of utilization panorama sketch is built |
| CN107563366A (en) * | 2017-07-26 | 2018-01-09 | 安徽讯飞爱途旅游电子商务有限公司 | A kind of localization method and device, electronic equipment |
| CN107967457B (en) * | 2017-11-27 | 2024-03-19 | 全球能源互联网研究院有限公司 | Site identification and relative positioning method and system adapting to visual characteristic change |
| CN108721069B (en) * | 2018-05-29 | 2020-07-14 | 杭州视氪科技有限公司 | Blind person auxiliary glasses based on multi-mode data for visual positioning |
| CN109443359B (en) * | 2018-09-27 | 2020-08-14 | 北京空间机电研究所 | Geographical positioning method of ground panoramic image |
| CN109579825B (en) * | 2018-11-26 | 2022-08-19 | 江苏科技大学 | Robot positioning system and method based on binocular vision and convolutional neural network |
| CN110458753B (en) * | 2019-08-12 | 2023-08-11 | 杭州环峻科技有限公司 | Adaptive segmentation and undistorted unfolding system and method for panoramic girdle image |
| CN110781790A (en) * | 2019-10-19 | 2020-02-11 | 北京工业大学 | Visual SLAM closed loop detection method based on convolutional neural network and VLAD |
| CN111008979A (en) * | 2019-12-09 | 2020-04-14 | 杭州凌像科技有限公司 | Robust night image semantic segmentation method |
-
2020
- 2020-04-29 CN CN202010357912.XA patent/CN111504331B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103310231A (en) * | 2013-06-24 | 2013-09-18 | 武汉烽火众智数字技术有限责任公司 | Auto logo locating and identifying method |
| CN104112122A (en) * | 2014-07-07 | 2014-10-22 | 叶茂 | Vehicle logo automatic identification method based on traffic video |
| CN109657552A (en) * | 2018-11-16 | 2019-04-19 | 北京邮电大学 | The vehicle type recognition device being cold-started across scene and method are realized based on transfer learning |
| CN110866079A (en) * | 2019-11-11 | 2020-03-06 | 桂林理工大学 | Intelligent scenic spot real scene semantic map generating and auxiliary positioning method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111504331A (en) | 2020-08-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111504331B (en) | Method and device for positioning panoramic intelligent vehicle from coarse to fine | |
| Tareen et al. | A comparative analysis of sift, surf, kaze, akaze, orb, and brisk | |
| Chen et al. | City-scale landmark identification on mobile devices | |
| Avrithis et al. | Retrieving landmark and non-landmark images from community photo collections | |
| Baatz et al. | Leveraging 3D city models for rotation invariant place-of-interest recognition | |
| Baatz et al. | Handling urban location recognition as a 2d homothetic problem | |
| Kurz et al. | Inertial sensor-aligned visual feature descriptors | |
| Wu et al. | A comprehensive evaluation of local detectors and descriptors | |
| Schroth et al. | Exploiting text-related features for content-based image retrieval | |
| Uchiyama et al. | Toward augmenting everything: Detecting and tracking geometrical features on planar objects | |
| Son et al. | A multi-vision sensor-based fast localization system with image matching for challenging outdoor environments | |
| CN111666434A (en) | Streetscape picture retrieval method based on depth global features | |
| Parihar et al. | Rord: Rotation-robust descriptors and orthographic views for local feature matching | |
| Mishkin et al. | Place recognition with WxBS retrieval | |
| CA3032983A1 (en) | Systems and methods for keypoint detection | |
| Guo et al. | Robust object matching for persistent tracking with heterogeneous features | |
| Liu et al. | Y-Net: Learning Domain Robust Feature Representation for ground camera image and large-scale image-based point cloud registration | |
| Fang et al. | CFVL: A coarse-to-fine vehicle localizer with omnidirectional perception across severe appearance variations | |
| Lee et al. | Learning to distill convolutional features into compact local descriptors | |
| Lourenço et al. | Localization in indoor environments by querying omnidirectional visual maps using perspective images | |
| CN113011359A (en) | Method for simultaneously detecting plane structure and generating plane description based on image and application | |
| Zhang et al. | An automatic three-dimensional scene reconstruction system using crowdsourced Geo-tagged videos | |
| CN110070626B (en) | Three-dimensional object retrieval method based on multi-view classification | |
| JP6304815B2 (en) | Image processing apparatus and image feature detection method, program and apparatus thereof | |
| Lee et al. | Bag of sampled words: a sampling-based strategy for fast and accurate visual place recognition in changing environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |