WO2023065657A1 - 地图构建方法、装置、设备、存储介质及程序 - Google Patents

地图构建方法、装置、设备、存储介质及程序 Download PDF

Info

Publication number
WO2023065657A1
WO2023065657A1 PCT/CN2022/093620 CN2022093620W WO2023065657A1 WO 2023065657 A1 WO2023065657 A1 WO 2023065657A1 CN 2022093620 W CN2022093620 W CN 2022093620W WO 2023065657 A1 WO2023065657 A1 WO 2023065657A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image data
feature
feature map
map
Prior art date
Application number
PCT/CN2022/093620
Other languages
English (en)
French (fr)
Inventor
张壮
孙瀚
姜翰青
章国锋
鲍虎军
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Publication of WO2023065657A1 publication Critical patent/WO2023065657A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing

Definitions

  • Embodiments of the present disclosure relate to the technical field of visual positioning, and relate to, but are not limited to, a map construction method, device, equipment, storage medium, and program.
  • An embodiment of the present disclosure provides a technical solution for map construction.
  • An embodiment of the present disclosure provides a map construction method, the method including:
  • the first feature map is updated to obtain a second feature map; wherein, the collection methods of the first image data and the second image data are different;
  • an embodiment of the present disclosure provides a map construction device, the device comprising:
  • the first acquisition part is configured to acquire the first image data of the area to be drawn and the navigation data corresponding to the first image data;
  • a first determining part configured to determine a first feature map based on the first image data and the navigation data
  • the first updating part is configured to update the first feature map based on the second image data of the area to be drawn to obtain a second feature map; wherein, the first image data and the second image The data collection methods are different;
  • the first construction part is configured to construct the three-dimensional map of the area to be drawn based on the second image data, the second feature map and the first image data.
  • an embodiment of the present disclosure provides a computer storage medium, on which computer executable instructions are stored. After the computer executable instructions are executed, the above-mentioned method steps can be implemented.
  • An embodiment of the present disclosure provides an electronic device, the electronic device includes a memory and a processor, the memory stores computer-executable instructions, and the processor can implement the above method when running the computer-executable instructions on the memory A step of.
  • An embodiment of the present disclosure also provides a computer program, where the computer program includes computer readable codes, and when the computer readable codes run in an electronic device, the processor of the electronic device executes any of the above embodiments The described augmented reality data display method.
  • Embodiments of the present disclosure provide a map construction method, device, device, storage medium, and program.
  • For an area to be drawn that needs to be drawn with a three-dimensional map first, by using the navigation data collected for the area to be drawn as the first image data The image pose can quickly construct the first feature map representing the image feature space pose of the first image data; then, by optimizing and updating the first feature map based on the second image data collected in different ways, the second feature can be The information in the map is more abundant; finally, by combining the two types of image data collected in different ways with the second feature map, a 3D map with higher coverage and higher precision can be constructed.
  • FIG. 1A is a schematic diagram of a system architecture of a map construction method provided by an embodiment of the present disclosure
  • FIG. 1B is a schematic diagram of the implementation process of the map construction method provided by the embodiment of the present disclosure.
  • FIG. 2A is a schematic flowchart of another implementation of the map creation method provided by the embodiment of the present disclosure.
  • FIG. 2B is a schematic flowchart of another implementation of the map construction method provided by the embodiment of the present disclosure.
  • FIG. 3 is a schematic diagram of the implementation flow of the map construction method provided by the embodiment of the present disclosure.
  • FIG. 4 is a schematic diagram of an implementation process for initially constructing a feature map provided by an embodiment of the present disclosure
  • FIG. 5 is a schematic diagram of an implementation process of updating a feature map provided by an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of an implementation process for building a grid model provided by an embodiment of the present disclosure
  • FIG. 7 is a schematic diagram of the structural composition of a map construction device provided by an embodiment of the present disclosure.
  • FIG. 8 is a schematic diagram of the composition and structure of an electronic device provided by an embodiment of the present disclosure.
  • first ⁇ second ⁇ third is only used to distinguish similar objects, and does not represent a specific ordering of objects. Understandably, “first ⁇ second ⁇ third” Where permitted, the specific order or sequencing may be interchanged such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein.
  • Computer vision refers to machine vision that uses cameras and computers instead of human eyes to identify, track and measure targets, and further performs graphics processing to make computer processing images that are more suitable for human eyes to observe or sent to instruments for detection.
  • Feature map Use related geometric features (such as: points, lines, surfaces) to represent the environment, and it is mostly used in real-time positioning and map construction (Simultaneous Localization and Mapping, SLAM) and SFM.
  • point feature representation is used, which includes information such as the spatial position, color, feature description, and visible image of the point.
  • Triangular mesh It is a kind of polygonal mesh, and polygonal mesh is also called "Mesh”, which is a data structure used in computer graphics to model various irregular objects. The essence is to express continuous objects in reality with a large number of small triangular faces.
  • the exemplary application of the map construction device provided by the embodiment of the present disclosure is described below, wherein the device provided by the embodiment of the present disclosure can be implemented as a notebook computer, a tablet computer, a desktop computer, a mobile device (for example, a personal digital assistant) with an image acquisition function , dedicated messaging devices, portable game devices) and other types of electronic devices.
  • a notebook computer for example, a tablet computer, a desktop computer
  • a mobile device for example, a personal digital assistant
  • dedicated messaging devices dedicated messaging devices
  • portable game devices portable game devices
  • FIG. 1A is a schematic diagram of a system architecture of a map construction method provided by an embodiment of the present disclosure.
  • the system architecture includes: an image acquisition terminal 11 , a network 12 and a control terminal 13 .
  • the image acquisition terminal 11 and the control terminal 13 establish a communication connection through the network 12.
  • the image acquisition terminal 11 reports the first image data and navigation data acquired to the control terminal 13 through the network 12; the control terminal 13 According to the data reported by the image acquisition terminal 11, the first feature map can be quickly constructed; then, the control terminal 13 optimizes and updates the first feature map based on the second image data collected in different ways to obtain the second feature map, and the collected image The data is combined with the second feature map to construct a three-dimensional map with higher coverage and higher precision.
  • control terminal 13 may include a terminal with a processing function
  • the image acquisition terminal 11 may include a visual processing device with an image acquisition function.
  • the network 12 can be wired or wirelessly connected.
  • the control terminal 13 may communicate with the visual processing device through a wired connection, such as performing data communication through a bus.
  • the image acquisition terminal 11 may be a vision processing device with a video acquisition module, or a host computer with a camera.
  • the map construction method of the embodiment of the present disclosure may be executed by the image acquisition terminal 11 , and the above-mentioned system architecture may not include the network 12 and the control terminal 13 .
  • FIG. 1B is a schematic diagram of the implementation process of the map construction method provided by the embodiment of the present disclosure, as shown in FIG. 1B , and described in conjunction with the steps shown in FIG. 1B :
  • Step S101 acquiring first image data and navigation data corresponding to the first image data in an area to be drawn.
  • the image acquisition device for the first image data may be any device or device capable of image acquisition; for example, a vehicle camera, a vehicle camera, or a roadside camera, and the like. Acquiring the first image data of the area to be drawn may be collected in real time by the image acquisition device, or may be received image data sent by other devices.
  • the first image data may be an outdoor image collected under the area to be drawn, may be a simple image whose picture content includes an outdoor scene, or may be a complex image whose picture content includes an outdoor scene.
  • the first image data may include multiple frames of images, and the navigation data is data of the navigation device when each frame of image is collected.
  • the navigation data corresponding to the first image data is data of a navigation device in the collection device when the first image data is collected, and may be position information of the collection device.
  • the navigation data and the first image data may be collected synchronously, for example, the first image data and the navigation data of the area to be drawn are collected synchronously by using a vehicle camera and a vehicle navigation device. Or, the navigation data and the first image data are collected asynchronously. For example, after the first image data is collected for the region to be drawn, the navigation device is used to collect the navigation data of the region; thus, the pixel coordinates of each point in the navigation data can be used as The pose of the point in the first image data.
  • Step S102 based on the first image data and the navigation data, determine a first feature map.
  • the first feature map represents the spatial locations of feature points of the first image data.
  • the navigation data of the navigation device is used as the initial image pose of the first image data; for multiple images in the first image data, the matching feature points between the images are determined; based on the correlation between the images, Matching feature points to build an image relationship library.
  • the pose of the collection device is determined; and based on this, the feature points in the first image data are processed Triangulation, so as to obtain the spatial position of the feature point in the world coordinate system, and completely and preliminarily establish a first feature map representing the spatial position of the feature point in the first image data.
  • Step S103 based on the second image data of the area to be drawn, the first feature map is updated to obtain a second feature map.
  • the collection methods of the first image data and the second image data are different, including: the source of the first image data is different from the source of the second image data, the second image data is different from the first
  • the collection devices of the image data are different, or the collection angles of the first image data and the second image data are different.
  • the collection device of the first image data is a vehicle-mounted camera, and the collection device of the second image data is an aerial drone.
  • the first image data may be data collected on the ground, and the second image data may be data collected in the air.
  • the first image data and the second image data from different sources are matched, and based on the matched first image data and the second image data, the first feature map is updated by using an incremental motion recovery structure method to obtain the second feature map.
  • Step S104 constructing a three-dimensional map of the region to be drawn based on the second image data, the second feature map and the first image data.
  • the first image data and the second image data are converted into point cloud data; then, by analyzing the visibility and reprojection error of each point of the point cloud data, it is determined whether the point can be used as a vertex, to construct a tetrahedron; finally, by combining the characteristics of the first image data and the second image data, the constructed tetrahedron is optimized to obtain a more planar three-dimensional map.
  • the first feature map representing the image feature space pose of the first image data can be quickly constructed; then, by using The second image data collected in different ways optimizes and updates the first feature map, which can make the feature points included in the second feature map more abundant; finally, by combining the two image data collected in different ways with the second feature map , can construct a 3D map with higher coverage and higher precision.
  • FIG. 2A is a schematic flowchart of another implementation of the map creation method provided by the embodiment of the present disclosure, combined with FIG. 1B and 2A for the following instructions:
  • Step S201 performing feature extraction on the first image data to obtain image feature points and description information of the image feature points.
  • feature points of scale invariant feature transformation are extracted; and description information of the feature points is obtained.
  • the description information includes the position of the feature point in the image (for example, the two-dimensional coordinates of the feature point in the image), the scale invariant and the rotation invariant of the feature point, and the like.
  • the image feature points extracted from the first image data include the positions of the image feature points in the image, which may be expressed in the form of two-dimensional coordinates.
  • Step S202 based on the description information of the image feature points, match different images in the first image data to obtain an image relationship database representing the matching relationship between different images.
  • the image feature points in the different images are the same feature point, and then based on the number of feature points with consistent description information, to Determine the similarity between different images. For example, for any two images in the first image data (take image A and image B as an example), by analyzing the number of image feature points with consistent description information in images A and B, the relationship between image A and image B can be determined Similarity; if there are more image feature points with consistent description information between images A and B, it is determined that the similarity between image A and image B is higher, and the degree of closeness is higher.
  • An image relation library is created based on the similarity between the different images.
  • the similarity between image feature points of different images is used as the association between image feature points; if there are more image feature points with consistent description information between two images, it can be considered as a connection comparison between two images Strong, based on such an association relationship and the images in the first image data, an image relationship library capable of representing the similarity between different images is constructed. For example, first extract the image SIFT features in the input first image data; then, establish the association between feature points between different images; finally, use the bag-of-words model to construct a scene graph (Scene Graph) as an image relationship library.
  • a scene graph Scene Graph
  • Step S203 Determine the first feature map based on the image relationship database and the navigation data.
  • the acquisition device's pose by first using the navigation data as the initial image pose of the first image data, combined with the extrinsic parameters of the navigation device that collects the navigation data and the extrinsic parameters of the collection device that collects the first image data, the acquisition device's pose; then, the two-dimensional coordinates (i.e. feature observations) of the image feature points in the image relational library, the pose of the acquisition device and the internal parameters of the acquisition device are used as the input of the triangulation process, so that each The spatial position of a feature point in the world coordinate system, and then realize the preliminary construction of a feature map representing the spatial position of the image feature point in the first image data, that is, the first feature map.
  • the two-dimensional coordinates i.e. feature observations
  • the similarity between different images can be obtained, so that according to the similarity between different images, the bag-of-words model is used to construct the image relationship library ; Furthermore, by combining the coordinates of the feature points in the image relation library with the navigation data, the first feature map representing the spatial position of the image feature points in the image data can be simply and quickly constructed.
  • step 204 by combining the navigation data with the set parameters of the acquisition device, and determining the spatial position of the image feature points through triangulation, that is, the above step 204 can be implemented through the following steps:
  • Step S141 based on the navigation data and preset first acquisition parameters, determine a first pose of the first acquisition device of the first image data.
  • the preset first collection parameters include: collection of extrinsic parameters of the first image data and collection of extrinsic parameters of the navigation data.
  • the device for collecting the first image data is a vehicle-mounted camera
  • the device for collecting the navigation data is a global satellite inertial navigation system
  • the pose of the on-board camera namely the first pose C i , can be determined.
  • Step S142 Determine the first feature map based on the image feature points in the relational database, the first pose and the second acquisition parameters.
  • the second acquisition parameters include intrinsic parameters for acquiring the first image data.
  • the second collection parameter is an internal parameter of the device that collects the first image data.
  • the internal parameter may be a camera internal parameter matrix. Based on the internal reference matrix of the camera, the two-dimensional coordinates of the image feature points and the pose of the camera as the first pose, triangulate the image feature points to determine the three-dimensional coordinates of the image feature points in the world coordinate system , that is, the spatial position of the image feature point. In this way, through the feature matching relationship combined with the initial image pose provided by the navigation data, based on the principle of triangulation, the feature map of the image data can be quickly constructed.
  • the initial feature map is optimized and adjusted through the predicted coordinates of the image feature points and the coordinates of the image feature points in the image to obtain the first A feature map, that is, the above step S142 can be realized through the following steps:
  • the first step based on the first pose and the second acquisition parameter, triangulate the first true value position of the image feature point in the image to obtain the image feature point in the world coordinate system three-dimensional coordinates.
  • the first true value position of the image feature point in the image to which it belongs, the first pose of the acquisition device of the first image data, and the internal reference matrix of the acquisition device are used as the triangulation of the image feature point.
  • the second step is to construct the first feature map based on the three-dimensional coordinates.
  • a feature map of the image feature point is constructed, and the feature map is used to represent the spatial position of the image feature point; the feature map can be directly used as the first feature The map is used for subsequent map updates; the first feature map can also be obtained by optimizing the feature point coordinates and image poses of the feature map.
  • the triangulated feature points are obtained by triangulating the true value positions of the image feature points, and the first feature map can be quickly constructed based on the three-dimensional coordinates of the triangulated feature points.
  • an initial feature map is preliminarily constructed, and the first feature map is obtained by optimizing and adjusting the initial feature map, so that the accuracy of the first feature map is more accurate. That is, the second step above can be achieved through the following process:
  • an initial feature map characterizing the spatial positions of the image feature points is constructed.
  • the first predicted position projected from the spatial position to the first image data is determined.
  • the conversion parameters from the navigation data to the camera coordinate system include: the pose of the navigation device in the navigation data (including the direction R i and the position t i of the navigation device), the navigation data from the navigation device to the camera coordinate system rotation matrix and translation vector; for example, if the navigation device is a global satellite inertial navigation system, the conversion parameters include: the direction R i and position t i of the global satellite inertial navigation system, the rotation matrix and translation vector from the satellite inertial navigation system to the camera coordinate system (T R and T t ).
  • the second collection parameter is the internal reference matrix K of the collection device that collects the first image data, and the three-dimensional coordinate X j of the jth image feature point in the initial feature map. Based on this, it is determined that the first predicted position of the image feature point projected from the spatial position into the image is K(T R R i X j +T R t i +T t ).
  • the first true value position x ij of the image feature point in the image in the relational database can be estimated by determining the difference between the first predicted position and the first true value position in the initial feature map.
  • the rationality of the spatial position of the feature point; based on the difference, the image feature point with an unreasonable spatial position in the initial feature map is optimized and adjusted, so as to obtain a first feature map with higher accuracy. For example, for each feature point in the initial feature map, it is determined that the feature point is projected back from the spatial position to the first predicted position on the image, that is, the position predicted in the first image data; based on the feature point in the image For the first ground-truth position in , estimate the prediction error for this first predicted position.
  • the spatial positions of the feature points in the initial feature map are adjusted to obtain the first feature map.
  • the three-dimensional coordinates of the image feature points in the initial feature map and the image pose can be adjusted so that the three-dimensional coordinates of the image feature points in the adjusted initial feature map The coordinates and the adjusted image pose are closer to the first true value position, thus completing the construction of the feature map based on the vehicle collected data.
  • the feature points in the initial feature map can be The spatial position and image pose are optimized to make the optimized first feature map more accurate.
  • the first feature map in order to improve the high coverage of the constructed first feature map, is updated using second image data from different sources to obtain a second feature map with higher coverage; that is, the above step S103 It can be realized through the steps shown in FIG. 2B .
  • FIG. 2B is a schematic flow chart of another implementation of the map construction method provided by the embodiment of the present disclosure. The following description is made in conjunction with the steps shown in FIGS. 1B and 2B :
  • Step S221 based on the matching relationship between the second image data and the first image data, update the image relation database to obtain the updated image relation database.
  • the manner of updating the image relational database by using the second image data is basically the same as that of step S201 and step S202.
  • the second image data as aerial image data
  • the first image data as the ground image data collected by the vehicle-mounted camera
  • by performing feature extraction on the second image data according to the description information of the extracted feature points and the first image data
  • the description information of the feature points determines the similarity between the images in the first image data and the second image data, that is, obtains the matching relationship between the images.
  • the image relationship database established according to the matching relationship between the images in the first image data is updated, so that not only the similarity of the images in the first image data can be reflected in the updated image relationship database, It can also reflect the degree of similarity between the first image data and the second image data.
  • Step S222 in the updated image relational database, determine the image to be registered.
  • target feature points corresponding to three-dimensional points in the first feature map exist in the image to be registered.
  • the updated image relational database for each image, it is determined whether the feature points in the image can find corresponding three-dimensional points in the first feature map. If the feature points in the image can find corresponding three-dimensional points in the first feature map, it is determined that the image is an image to be registered.
  • random sampling is used to determine the image pose of the image to which the target feature point belongs, that is, the image of the image to be registered pose.
  • Step S224 registering the pose of the image into the first feature map to obtain the registered feature map.
  • the image pose determined according to the target feature point is registered into the first feature map, so that the registered feature map can present the spatial position of the frame content of the image to be registered to which the target feature point belongs.
  • the advantages of vehicle-mounted cameras and UAV data can be effectively used to improve the coverage and completeness of mapping.
  • the registration order of the images to be registered is determined by counting the number of target feature points, so that the image poses of the images to be registered are sequentially registered in the first feature map according to the registration order, and the following steps can be taken accomplish:
  • the first step is to determine the number of target feature points included in each image to be registered.
  • the second step is to determine the registration order of each image to be registered based on the quantity.
  • the registration sequence of the images to be registered is determined in descending order of the number.
  • the image to be registered with the largest number of target features is arranged first, and the registration of the image pose is given priority.
  • the third step is to register the image pose of each image to be registered into the first feature map based on the registration order to obtain the registered feature map.
  • the image to be registered with the largest number of target features first, register the image pose into the first feature map, and then register the image pose of the image to be registered with the second largest number of target features into the first feature map .
  • the accuracy of the registered image pose can be improved.
  • Step S225 based on other feature points in the image to be registered and the second image data, adjust the registered feature map to obtain the second feature map.
  • the other feature points are feature points other than target feature points in the image to be registered.
  • step S225 in order to further optimize the registered feature map, by up-sampling other feature points, and performing triangulation processing on a sampled feature point, the registered feature map is processed according to the result of the triangulation processing.
  • the above step S225 also includes the following steps:
  • the other feature points are sampled to obtain sampled feature points.
  • the second step is to perform triangulation on the sampled feature points, and determine the three-dimensional coordinates of the sampled feature points in the world coordinate system.
  • the sampling feature point is triangulated by the pose of the acquisition device corresponding to the sampling feature point, the internal reference matrix of the acquisition device, and the coordinates of the sampling feature point in the image to determine the spatial position of the sampling feature point Pose, that is, the three-dimensional coordinates of the sampling feature points in the world coordinate system.
  • Step 3 Adjust the registered feature map based on the three-dimensional coordinates of the sampled feature points and the second image data to obtain the second feature map.
  • the difference between the actual position of the sampling feature point and the estimated spatial position can be further predicted, and then based on the difference, the existing The positions of the three-dimensional points in the registered feature map are optimized to obtain a second feature map.
  • it can be achieved through the following steps:
  • the first step based on the conversion parameters of the navigation data to the camera coordinate system, determine the projection of the three-dimensional coordinates of the sampling feature points to the second predicted position of the first image data and the projection of the three-dimensional coordinates of the sampling feature points to the second predicted position.
  • the third predicted location of the image data based on the conversion parameters of the navigation data to the camera coordinate system, determine the projection of the three-dimensional coordinates of the sampling feature points to the second predicted position of the first image data and the projection of the three-dimensional coordinates of the sampling feature points to the second predicted position.
  • the way to determine the second predicted position is similar to the way to determine the first predicted position, determine the rotation matrix and translation vector (T R and T t ) of the navigation data to the camera coordinate system, the direction R i and position t i of the navigation device, and collect
  • the internal reference matrix K C of the acquisition device of the first image data for example, a vehicle-mounted camera
  • the three-dimensional coordinates X j of the sampling feature point based on this, determine the second predicted position of the sampling feature point projected from the spatial position to the first image is K C (T R R i X j +T R t i +T t ).
  • the sampling feature point is from The spatial position is projected to a third predicted position K A (R k X j +t k ) in the second image.
  • the second step is to determine the second difference between the second predicted position and the second true value position of the sampling feature point in the corresponding image, and the third predicted position and the second true value The third difference in position.
  • sampling feature points obtained in each sampling area it is determined that the sampling feature points are projected back from the spatial position to the second prediction position on the first image data, that is, the position for prediction in the first image data; based on the sampling The second true value position of the feature point in the image to which it belongs, and the prediction error of the second predicted position is estimated. Similarly, for the sampled feature point, estimate its prediction error at the third predicted position, that is, the third difference.
  • a third step based on the second difference and the third difference, the spatial positions of the feature points in the registered feature map are adjusted to obtain the second feature map.
  • the three-dimensional coordinates corresponding to the image feature points in the first image data in the registered feature map and the image pose of the first image data can be adjusted.
  • the three-dimensional coordinates corresponding to the image feature points in the second image data in the registered feature map, and the image pose of the second image data can be adjusted. In this way, a sampling feature point is obtained by uniform sampling in each sampling area, and the spatial position of the feature point in the first feature map is calculated by the difference between the predicted position of the sampling feature point and the actual position in the image. Optimization makes the second feature map more complete.
  • a high-precision three-dimensional map is finally created by performing depth restoration, point cloud generation, and grid construction on the image data, that is, the above step S105 can be implemented by the following steps:
  • Step S151 performing in-depth processing on the second image data, the second feature map and the first image data to generate point cloud data representing the first image data and the second image data.
  • the image pose of the second image is included in the second image data
  • the image pose of the first image is included in the first image data
  • the image in the first image data is included in the second feature map
  • the first image pose and the first type of three-dimensional feature points are used as a group for depth estimation and depth map fusion to obtain point cloud data representing the first image data
  • the second image pose and the second type of three-dimensional feature points are used as a group
  • the group performs depth estimation and depth map fusion to obtain point cloud data representing the second image data.
  • point cloud data can be generated by the following steps:
  • the first step is to perform depth estimation on the second image data, the second feature map and the first image data to obtain a depth map.
  • the depth map is fused to obtain the point cloud data.
  • the depth map of the first image and the depth map of the second image are respectively deeply fused to generate point cloud data corresponding to the first image and point cloud data corresponding to the second image; the point cloud data of these two types of point cloud data are integrated to obtain The final point cloud data. In this way, by performing depth estimation and depth fusion on the two types of images, the obtained point cloud data is more abundant.
  • Step S152 based on the point cloud data, construct an initial mesh model representing the connection relationship between the point cloud data.
  • the connection relationship between point cloud data refers to whether the points are connected in the gateway model. According to the point cloud data based on the integration of two types of image data, as well as the visibility and reprojection error of the point, it is judged whether the point can be used as a vertex to construct a tetrahedron, so that multiple tetrahedrons can be constructed based on multiple vertices as the initial grid model.
  • the primary gateway model can be built through the following steps:
  • the first step is to determine the visibility of each point in the point cloud data in the second feature map and the reprojection error of each point.
  • each point in the point cloud data in the second feature map that is, whether the point has a corresponding three-dimensional feature point in the second feature map, if the point has a corresponding three-dimensional feature point in the second feature map , indicating that the point is visible; if the point does not have a corresponding three-dimensional feature point in the second feature map, it indicates that the point is not visible.
  • the reprojection error it can be known from feature matching pairs that observations A and B are a set of feature matching pairs, which are projections of the same spatial point C, A and B belong to different maps, and B ⁇ To transform A into a projected point on the coordinate system to which B belongs after the first global coordinate in the coordinate system to which B belongs.
  • the reprojection error of each point in the point cloud data is the projection A and B of the point in the two types of images, for example, A is in the first image data, B is in the second image data; the reprojection of the point The error is the distance between the projected point on the coordinate system of B and the observed value B after transforming A into the first global coordinate in the coordinate system of B.
  • the visibility of the determined point is that the point can find the corresponding 3D feature point in the second feature map, and the target point whose reprojection error is less than a certain threshold, and multiple target points can be obtained.
  • the third step is to use the target points as vertices and connect the vertices to obtain the initial mesh model.
  • connecting any vertices in the space can form a tetrahedron, obtain multiple tetrahedrons, and use the tetrahedron as the initial mesh model.
  • the obtained initial mesh model can fully reflect the connection relationship between the target points.
  • Step S153 based on the initial grid model, determine the three-dimensional map.
  • the initial mesh model includes a plurality of tetrahedrons
  • an optimal group of faces is selected from the faces of the plurality of tetrahedrons , connect the optimal set of faces to form a three-dimensional map.
  • the construction of the three-dimensional map may be implemented through the following steps:
  • the source of point cloud data collection corresponding to multiple vertices of each surface is determined.
  • the sources of the three vertices of each face are determined.
  • the surface as a triangular surface analyze the acquisition status of the three vertices of the surface, whether the three vertices come from different vehicle cameras, or whether the three vertices all come from the same vehicle camera or navigation device (for example, aerial photography) device).
  • the second step is to determine, based on the acquisition source, a weight representing that each surface is penetrated by a line of sight.
  • the triangular surface are more likely not to belong to the same object, that is, the triangular surface is more likely to be the surface of a real object; then the surface The greater the weight of the patch being penetrated by the line of sight; where the line of sight is a line from the collection device to the patch.
  • the three-dimensional map is determined based on the faces whose weights are less than a preset threshold.
  • a weight representing that the surface is penetrated by the line of sight is determined, and a group of surfaces with the smallest sum of weights is selected; this group of surfaces is connected to obtain the three-dimensional map.
  • a face with a smaller weight is selected to construct a three-dimensional map, so that the three maps constructed are smoother.
  • maps provide great help for people to travel.
  • the information provided by maps includes but not limited to road information, building information, traffic information, etc.
  • high-precision maps have attracted much attention due to their centimeter-level reconstruction accuracy and rich hierarchical information.
  • High-precision maps are mostly used in the field of driverless driving.
  • the map is usually generated by conventional acquisition plus manual post-processing method.
  • both high-precision professional equipment collection and crowdsourcing collection methods require manual alignment after data collection.
  • the collection of high-precision map data is mostly collected by collection vehicles within the range of paved roads.
  • the reconstruction accuracy of non-road areas is limited, and it is difficult to meet the needs of high-precision map construction in the entire region.
  • the integrity and hierarchical structure of the overall reconstruction are not perfect.
  • the embodiment of the present disclosure combines the advantages of comprehensive coverage and convenient collection of aerial photography UAVs on the basis of ground collection, and proposes a high-precision map construction method for full-area coverage of air-ground non-homogeneous data.
  • the method and collection system for high-precision map construction can be fully automated by using a vehicle-mounted global satellite inertial navigation system, a camera, and an aerial drone.
  • FIG. 3 is a schematic diagram of the implementation flow of the map construction method provided by the embodiment of the present disclosure. It can be seen from FIG. 3 that the method can be implemented through the following steps:
  • Step S301 acquiring the pose data of the vehicle-mounted global satellite inertial navigation system, and obtaining the image data of the vehicle-mounted camera.
  • the pose data of the vehicle-mounted global satellite inertial navigation system and the image data of the vehicle-mounted camera are calibrated and synchronized.
  • the pose data of the vehicle-mounted global satellite inertial navigation system can be used as the navigation data in the above embodiments.
  • Step S302 based on the pose data and the camera image, construct a feature map of the vehicle collected data.
  • the feature map of the data collected on the vehicle may be used as the first feature map in the above embodiments.
  • the feature map of the ground data is constructed based on the principle of triangulation. It mainly includes the following three processes: feature extraction and matching; triangulation of feature points; adjustment and optimization.
  • FIG. 4 is a schematic diagram of the implementation flow of the preliminary construction of the feature map provided by the embodiment of the present disclosure, and the following description is made in conjunction with the steps shown in FIG. 4:
  • Step S401 performing feature extraction and matching on the vehicle camera image data 41 .
  • the matching number of Scale Invariant Feature Transform (SIFT) feature points is used to measure the closeness of image association. First, extract the SIFT features of the input image; then, establish the feature point association between images; finally, use the bag of words model to construct the scene graph (Scene Graph).
  • SIFT Scale Invariant Feature Transform
  • Step S402 based on the pose data 42 of the global satellite inertial navigation system and the extrinsic parameters of the camera, triangulate the initial feature map.
  • the pose (R i and t i ) of the camera C i is solved, and the feature points are triangulated based on this. Preliminary construction of the initial feature map of the vehicle data.
  • Step S403 optimizing and adjusting the triangulated initial feature map.
  • n is the number of images
  • m is the number of map points
  • K is the internal reference matrix of the camera
  • T R and T t are the rotation matrix and translation vector from the satellite inertial navigation system to the camera coordinate system
  • X is the three-dimensional coordinates of the map points
  • x For feature observation,
  • the feature point coordinates and image pose of the feature map are optimized and updated, and the construction of the feature map based on the vehicle-mounted data is realized.
  • Step S303 update the feature map based on the aerial photography data 31 of the UAV.
  • the feature points of the image captured by the drone are extracted and matched with the image data of the vehicle camera, and the scene graph and the feature map are updated.
  • This process mainly includes two parts: feature extraction and matching of the aerial image, and update of the feature map.
  • the first part similar to step S401, extracting the features of the aerial image captured by the aerial photography of the drone, matching the image feature with the feature of the vehicle camera image, and updating the original scene map.
  • Step S501 obtaining a first feature map created based on vehicle-mounted images and an updated scene map.
  • Step S502 selecting a frame image to be registered in the updated scene map.
  • the selection of frames to be registered is mainly based on the number of visible map points.
  • the number of visible map points reflects the similarity of the observation area. When most of the feature points in the newly registered image can find the corresponding three-dimensional points in the current feature map, it means that the observation overlap is high and can be registered first.
  • Step S503 perform triangulation on other feature points in the image to be registered.
  • Step S504 based on the triangulated feature points, perform local/global optimization and adjustment on the first feature map to obtain a second feature map.
  • O is the number of aerial images
  • K A is the internal parameters of aerial images
  • R k and t k are the rotation matrix and translation vector corresponding to the image
  • v kj and v ij both represent the visibility of feature point j in the corresponding image
  • the rest The meaning of the parameters is consistent with formula (1).
  • Step S304 generating a mesh model based on the high-precision dense point cloud of sensor fusion.
  • Step S601 perform depth estimation on the vehicle camera image data 61, the image pose of the data, and the corresponding three-dimensional feature points in the second feature map to obtain a depth map.
  • Step S602 perform depth map fusion on the depth map obtained in step S601 to obtain a point cloud of the vehicle camera image data 61 .
  • step S603 perform depth estimation on the UAV aerial image data 63 , the image pose of the data, and the corresponding three-dimensional feature points of the image in the second feature map to obtain a depth map.
  • Step S604 performing depth map fusion on the depth map obtained in step S603 to obtain a point cloud of the aerial image 63 of the UAV.
  • the depth maps of images captured by vehicles and UAVs are independently estimated.
  • the resulting depth map is fused into a point cloud to generate a point cloud of vehicle-collected data and a point cloud of UAV aerial photography data.
  • the point clouds obtained in step S602 and step S604 are integrated to obtain point cloud data.
  • Step S605 integrating the point cloud collected by the integrated vehicle camera with the point cloud obtained by the aerial photography of the UAV to obtain point cloud data.
  • the point cloud data is a dense point cloud.
  • Step S606 generating a grid model based on the point cloud data.
  • Step S607 optimize and adjust the grid model according to the set weights to obtain a three-dimensional map.
  • the point cloud insertion link will integrate the point cloud collected by the vehicle camera and the point cloud obtained by the aerial photography of the drone. On this basis, according to the two indicators of point visibility and reprojection error, it is judged whether to use the point as a vertex to construct a tetrahedron.
  • Graphg-Cut graph cut algorithm
  • b ⁇ r makes the final generated 3D map smoother.
  • the pose acquired by the global satellite inertial navigation system is used as the initial value, combined with the final adjustment and optimization to quickly construct the feature map, thus simplifying the iterative registration and optimization links, and making full use of the inertial navigation information to improve reconstruction efficiency.
  • a dense point cloud and grid model with high precision and high coverage can be constructed; in this way, the coverage and completeness of mapping can be improved, The result is a more complete 3D model.
  • FIG. 7 is a schematic structural composition diagram of the map construction device provided by an embodiment of the present disclosure. As shown in FIG. 7 , the map construction device 700 includes:
  • the first acquiring part 701 is configured to acquire the first image data of the area to be drawn and the navigation data corresponding to the first image data;
  • the first determining part 702 is configured to determine a first feature map based on the first image data and the navigation data;
  • the first update part 703 is configured to update the first feature map based on the second image data of the area to be drawn to obtain a second feature map; wherein, the first image data and the second The image data is collected in different ways;
  • the first construction part 704 is configured to construct the three-dimensional map of the area to be drawn based on the second image data, the second feature map and the first image data.
  • the first determining part 702 includes:
  • the first extraction subsection is configured to perform feature extraction on the first image data to obtain image feature points and description information of the image feature points;
  • the first matching subpart is configured to match different images in the first image data based on the description information of the image feature points to obtain an image relationship library representing the matching relationship between the different images;
  • the first determination subpart is configured to determine the first feature map based on the image relationship library and the navigation data.
  • the first determining subsection includes:
  • the first determination unit is configured to determine the first pose of the acquisition device of the first image data based on the navigation data and the preset first acquisition parameters; wherein the preset first acquisition parameters include: acquisition The extrinsic parameters of the first image data and the extrinsic parameters of collecting the navigation data;
  • the second determination unit is configured to determine the first feature map based on the image feature points in the relational database, the first pose, and the second collection parameters; wherein the second collection parameters include collecting the Intrinsic parameters of the first image data.
  • the second determination unit includes:
  • the first determining subunit is configured to triangulate the first true value position of the image feature point in the corresponding image based on the first pose and the second acquisition parameter, to obtain the image feature point 3D coordinates in the world coordinate system;
  • the first constructing subunit is configured to construct the first feature map based on the three-dimensional coordinates.
  • the first construction subunit is further configured to: based on the three-dimensional coordinates, construct an initial feature map that characterizes the spatial positions of the image feature points; based on the conversion of the navigation data to the camera coordinate system parameter and the second acquisition parameter, determine a first predicted position projected from the spatial position to the first image data; determine a first difference between the first true value position and the first predicted position; based on the The first difference value is used to adjust the spatial positions of the feature points in the initial feature map to obtain the first feature map.
  • the first updating part 703 includes:
  • the first updating subpart is configured to update the image relationship database based on the matching relationship between the second image data and the first image data to obtain an updated image relationship database;
  • the second determination subpart is configured to determine an image to be registered in the updated image relational database; wherein, there is a target feature point corresponding to a three-dimensional point in the first feature map in the image to be registered ;
  • the third determining subpart is configured to determine the image pose of the image to be registered based on the target feature point;
  • the first registration subpart is configured to register the image pose into the first feature map to obtain a registered feature map
  • the first adjustment subsection is configured to adjust the registered feature map based on other feature points in the image to be registered and the second image data to obtain the second feature map; wherein, the Other feature points are feature points other than target feature points in the image to be registered.
  • the first registration subsection further includes:
  • a third determining unit configured to determine the number of target feature points included in each image to be registered
  • a fourth determining unit configured to determine a registration order of each image to be registered based on the number
  • the first registration unit is configured to register the image pose of each image to be registered into the first feature map based on the registration sequence, to obtain the registered feature map.
  • the first adjustment subsection includes:
  • the first sampling unit is configured to sample the other feature points to obtain sampled feature points
  • the fifth determination unit is configured to perform triangulation on the sampling feature points, and determine the three-dimensional coordinates of the sampling feature points in the world coordinate system;
  • the first adjusting unit is configured to adjust the registered feature map based on the three-dimensional coordinates of the sampled feature points and the second image data to obtain the second feature map.
  • the first adjustment unit includes:
  • the second determination subunit is configured to determine, based on the conversion parameters from the navigation data to the camera coordinate system, the projection of the 3D coordinates of the sampling feature points to the second predicted position of the first image data and the 3D of the sampling feature points coordinate projection to a third predicted location of the second image data;
  • the third determining subunit is configured to determine a second difference between the second predicted position and the second true value position of the sampling feature point in the corresponding image, and the third predicted position and the The third difference of the second true value position;
  • the first adjustment subunit is configured to adjust the spatial positions of the feature points in the registered feature map based on the second difference and the third difference to obtain the second feature map.
  • the first construction part 704 includes:
  • the first processing subsection is configured to perform in-depth processing on the second image data, the second feature map, and the first image data, and generate a representation of the first image data and the second image data point cloud data;
  • the first construction subpart is configured to construct an initial mesh model representing the connection relationship between the point cloud data based on the point cloud data;
  • the second determining subpart is configured to determine the three-dimensional map based on the initial grid model.
  • the first processing subsection includes:
  • a first estimation unit configured to perform depth estimation on the second image data, the second feature map, and the first image data to obtain a depth map
  • the first fusion unit is configured to fuse the depth map to obtain the point cloud data.
  • the first construct subsection includes:
  • a sixth determination unit configured to determine the visibility of each point in the point cloud data in the second feature map and the reprojection error of each point
  • the seventh determining unit is configured to, in the point cloud data, determine a target point whose visibility and reprojection error meet a preset condition
  • the first connection unit is configured to use the target points as vertices and connect the vertices to obtain the initial mesh model.
  • the second determining subsection includes:
  • the eighth determining unit is configured to, in the initial mesh model, determine the collection source of point cloud data corresponding to multiple vertices of each surface;
  • the ninth determining unit is configured to determine, based on the acquisition source, a weight representing that each surface is penetrated by a line of sight;
  • the tenth determination unit is configured to determine the three-dimensional map based on the planes whose weights are less than a preset threshold.
  • the device for collecting the first image data includes a vehicle-mounted camera, and/or, the device for collecting the second image data is an aerial photography device.
  • the device involved in the embodiments of the present disclosure may be at least one of a system, a method, and a computer program product.
  • a computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.
  • a computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.
  • a computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Examples of computer-readable storage media include: portable computer disks, hard disks, Random Access Memory (RAM), Read-Only Memory (ROM), erasable Electrical Programmable Read Only Memory (EPROM) or flash memory, Static Random-Access Memory (Static Random-Access Memory, SRAM), Portable Compact Disc Read-Only Memory (CD-ROM), Digital Video Discs (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing.
  • RAM Random Access Memory
  • ROM Read-Only Memory
  • EPROM erasable Electrical Programmable Read Only Memory
  • flash memory Static Random-Access Memory
  • SRAM Static Random-Access Memory
  • CD-ROM Portable Compact Disc Read-Only Memory
  • DVDs Digital Video Discs
  • memory sticks floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing.
  • computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.
  • the computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device over at least one of a network, such as the Internet, a local area network, a wide area network, and a wireless network.
  • the network may include at least one of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.
  • a network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • Computer program instructions for performing the operations of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the “C” language or similar programming languages.
  • Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it may be connected to an external computer (for example, using Internet Service Provider to connect via the Internet).
  • LAN Local Area Network
  • WAN Wide Area Network
  • electronic circuits such as programmable logic circuits, FPGAs, or programmable logic arrays (Programmable Logic Arrays, PLAs), can be customized by using state information of computer-readable program instructions, which can execute computer-readable Read program instructions, thereby implementing various aspects of the present disclosure.
  • map construction method is implemented in the form of software function modules and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
  • the computer software products are stored in a storage medium, including several instructions for Make an electronic device (which may be a terminal, a server, etc.) execute all or part of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage medium includes: various media capable of storing program codes such as U disk, sports hard disk, read-only memory, magnetic disk or optical disk. As such, embodiments of the present disclosure are not limited to any specific combination of hardware and software.
  • the embodiments of the present disclosure further provide a computer program product, the computer program product includes computer executable instructions, and after the computer executable instructions are executed, the steps in the map construction method provided by the embodiments of the present disclosure can be implemented.
  • an embodiment of the present disclosure further provides a computer storage medium, on which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the map construction method provided by the above-mentioned embodiments is implemented. step.
  • FIG. 8 is a schematic structural diagram of the electronic device provided by an embodiment of the present disclosure.
  • the electronic device 800 includes: a processor 801, at least one communication bus, communication interface 802, at least one external communication interface and memory 803.
  • the communication interface 802 is configured to realize connection and communication between these components.
  • the communication interface 802 may include a display screen, and the external communication interface may include a standard wired interface and a wireless interface.
  • the processor 801 is configured to execute the image processing program in the memory, so as to realize the steps of the map construction method provided in the above embodiments.
  • the disclosed devices and methods may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division.
  • the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms of.
  • the units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may be used as a single unit, or two or more units may be integrated into one unit; the above-mentioned integration
  • the unit can be realized in the form of hardware or in the form of hardware plus software functional unit.
  • the above-mentioned integrated units of the present disclosure are realized in the form of software function modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
  • the computer software product is stored in a storage medium, and includes several instructions for Make an electronic device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage medium includes various media capable of storing program codes such as removable storage devices, ROMs, magnetic disks or optical disks.
  • Embodiments of the present disclosure provide a map construction method, device, device, storage medium, and program, wherein the method includes: acquiring first image data of an area to be mapped and navigation data corresponding to the first image data; The first image data and the navigation data are used to determine a first feature map; based on the second image data of the area to be drawn, the first feature map is updated to obtain a second feature map; wherein, the first feature map The first image data is collected in a different manner from the second image data; based on the second image data, the second feature map, and the first image data, a three-dimensional map of the region to be drawn is constructed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Remote Sensing (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)
  • Navigation (AREA)
  • Instructional Devices (AREA)

Abstract

本公开实施例提供一种地图构建方法、装置、设备、存储介质及程序，其中，所述方法包括：获取待绘制区域的第一图像数据和所述第一图像数据对应的导航数据；基于所述第一图像数据和所述导航数据，确定第一特征地图；基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图；其中，所述第一图像数据与所述第二图像数据的采集方式不同；基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。

Description

地图构建方法、装置、设备、存储介质及程序
相关申请的交叉引用
本专利申请要求2021年10月18日提交的中国专利申请号为202111210541.3、申请人为浙江商汤科技开发有限公司，申请名称为“地图构建方法、装置、设备及存储介质”的优先权，该申请的全文以引用的方式并入本公开中。
技术领域
本公开实施例涉及视觉定位技术领域领域，涉及但不限于一种地图构建方法、装置、设备、存储介质及程序。
背景技术
随着计算机及通信技术的不断发展，地图为人们出行提供了很大帮助。在相关技术中，采用常规采集加人工处理等方法生成地图，这样创建的地图精度有限，不能很好地满足用户需求。
发明内容
本公开实施例提供一种地图构建技术方案。
本公开实施例的技术方案是这样实现的：
本公开实施例提供一种地图构建方法，所述方法包括：
获取待绘制区域的第一图像数据和所述第一图像数据对应的导航数据；
基于所述第一图像数据和所述导航数据，确定第一特征地图；
基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图；其中，所述第一图像数据与所述第二图像数据的采集方式不同；
基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。
对应地，本公开实施例提供一种地图构建装置，所述装置包括：
第一获取部分，被配置为获取待绘制区域的第一图像数据和所述第一图像数据对应的导航数据；
第一确定部分，被配置为基于所述第一图像数据和所述导航数据，确定第一特征地图；
第一更新部分，被配置为基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图；其中，所述第一图像数据与所述第二图像数据的采集方式不同；
第一构建部分，被配置为基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。
对应地，本公开实施例提供一种计算机存储介质，所述计算机存储介质上存储有计算机可执行指令，该计算机可执行指令被执行后，能够实现上述所述的方法步骤。
本公开实施例提供一种电子设备，所述电子设备包括存储器和处理器，所述存储器上存储有计算机可执行指令，所述处理器运行所述存储器上的计算机可执行指令时能够实现上述方法的步骤。
本公开实施例还提供一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备的处理器执行上述任一实施例所述的增强现实数据展示方法。
本公开实施例提供一种地图构建方法、装置、设备、存储介质及程序，对于需要绘制三维地图的待绘制区域，首先，通过将针对该待绘制区域采集到的导航数据作为第一图像数据的图像位姿，能够快速构建表征第一图像数据的图像特征空间位姿的第一特征地图；然后，通过基于采集方式不同的第二图像数据对第一特征地图进行优化更新，能够使得第二特征地图中的信息更加丰富；最后，通过将采集方式不同的两类图像数据与第二特征地图进行结合，能够构建覆盖度更高以及精度更高的三维地图。
为使本公开的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，此处的附图被并入说明书中并构成本说明书中的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开的技术方案。应当理解，以下附图仅示出了本公开的某些实施例，因此不应被看作是对范围的限定，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他相关的附图。
图1A为本公开实施例提供的地图构建方法的一种系统架构示意图；
图1B为本公开实施例提供的地图构建方法的实现流程示意图；
图2A为本公开实施例提供的地图创建方法的另一实现流程示意图；
图2B为本公开实施例提供的地图构建方法的又一实现流程示意图；
图3为本公开实施例提供的地图构建方法的实现流程示意图；
图4为本公开实施例提供的初步构建特征地图的实现流程示意图；
图5为本公开实施例提供的更新特征地图的实现流程示意图；
图6为本公开实施例提供的搭建网格模型的实现流程示意图；
图7为本公开实施例提供的地图构建装置的结构组成示意图；
图8为本公开实施例提供的电子设备的组成结构示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中的附图，对发明的具体技术方案做进一步详细描述。以下实施例用于说明本公开，但不用来限制本公开的范围。
在以下的描述中，涉及到“一些实施例”，其描述了所有可能实施例的子集，但是可以理解，“一些实施例”可以是所有可能实施例的相同子集或不同子集，并且可以在不冲突的情况下相互结合。
在以下的描述中，所涉及的术语“第一\第二\第三”仅仅是是区别类似的对象，不代表针对对象的特定排序，可以理解地，“第一\第二\第三”在允许的情况下可以互换特定的顺序或先后次序，以使这里描述的本公开实施例能够以除了在这里图示或描述的以外的顺序实施。
除非另有定义，本文所使用的所有的技术和科学术语与属于本公开的技术领域的技术人员通常理解的含义相同。本文中所使用的术语只是为了描述本公开实施例的目的，不是旨在限制本公开。
对本公开实施例进行进一步详细说明之前，对本公开实施例中涉及的名词和术语进行说明，本公开实施例中涉及的名词和术语适用于如下的解释。
1)计算机视觉：指用摄影机和电脑代替人眼对目标进行识别、跟踪和测量等机器视觉，并进一步做图形处理，使电脑处理成为更适合人眼观察或传送给仪器检测的图像。
2)运动结构(Structure-from-Motion，SFM)：由于图像是无序的，所以需要将有重叠关系的图像关联起来，运动结构部分的输出为经过几何验证的关联图像集，及地图点对应的图像投影点。
3)特征地图：使用有关的几何特征(如：点、直线、面)表示环境，多用于即时定位与地图构建(Simultaneous Localization and Mapping，SLAM)和SFM中。本公开实施例中使用点特征表示，包含点的空间位置、颜色、特征描述和可见图像等信息。
4)三角网格：是多边形网格的一种，多边形网格又被称为“Mesh”，是计算机图形学中用于为各种不规则物体建立模型的一种数据结构。本质是用大量小的三角形面片表达现实中的连续物体。
下面说明本公开实施例提供的地图构建设备的示例性应用，其中，本公开实施例提供的设备可以实施为具有图像采集功能的笔记本电脑，平板电脑，台式计算机，移动设备(例如，个人数字助理，专用消息设备，便携式游戏设备)等各种类型的电子设备。
下面，将说明地图构建设备实施为电子设备时示例性应用。
图1A为本公开实施例提供的地图构建方法的一种系统架构示意图，如图1A所示，该系统架构中包括：图像获取终端11、网络12和控制终端13。为实现支撑一个示例性应用，图像获取终端11和控制终端13通过网络12建立通信连接，首先，图像获取终端11通过网络12向控制终端13上报 获取的第一图像数据和导航数据；控制终端13按照图像获取终端11上报的数据能够快速构建第一特征地图；然后，控制终端13基于采集方式不同的第二图像数据对第一特征地图进行优化更新，得到第二特征地图，并将采集的图像数据与第二特征地图结合，构建覆盖度更高以及精度更高的三维地图。
作为示例，控制终端13可以包括具有处理功能的终端，图像获取终端11可以包括具有图像采集功能的视觉处理设备。网络12可以采用有线或无线连接方式。其中，当图像获取终端11为视觉处理设备时，控制终端13可以通过有线连接的方式与视觉处理设备通信连接，例如通过总线进行数据通信。
或者，在一些场景中，图像获取终端11可以是带有视频采集模组的视觉处理设备，可以是带有摄像头的主机。这时，本公开实施例的地图构建方法可以由图像获取终端11执行，上述系统架构可以不包含网络12和控制终端13。
图1B为本公开实施例提供的地图构建方法的实现流程示意图，如图1B所示，结合如图1B所示步骤进行说明：
步骤S101，在待绘制区域中，获取第一图像数据和所述第一图像数据对应的导航数据。
在一些实施例中，第一图像数据的图像采集装置可以是任意具有图像采集功能的装置或者设备；比如，车载摄像机、车载相机或者路边的摄像头等。获取待绘制区域的第一图像数据，可以通过图像采集装置实时采集的，还可以是接收的其他设备发送的图像数据。该第一图像数据可以是在待绘制区域下，采集到的户外图像，可以是画面内容包括户外场景的简单图像，还可以是画面内容包括户外场景的复杂图像。第一图像数据可以包括多帧图像，导航数据为采集每一帧图像时，导航装置的数据。
在一些实施例中，第一图像数据对应的导航数据为采集该第一图像数据时，采集设备中的导航装置的数据，可以是该采集设备的位置信息。导航数据与第一图像数据可以是同步采集的，比如，采用车载摄像机和车载导航装置对待绘制区域同步采集第一图像数据和导航数据。或者，导航数据与第一图像数据是异步采集的，比如，针对待绘制区域采集第一图像数据之后，采用导航装置采集该区域的导航数据；这样，可以将导航数据中每一点的像素坐标作为第一图像数据中该点的位姿。
步骤S102，基于所述第一图像数据和所述导航数据，确定第一特征地图。
在一些实施例中，第一特征地图表示第一图像数据的特征点的空间位置。将采集第一图像数据时，导航装置的导航数据作为第一图像数据的初始图像位姿；对于第一图像数据中的多个图像，确定图像之间相匹配的特征点；基于图像之间相匹配的特征点，搭建图像关系库。通过结合导航装置获取到的第一图像数据的图像位姿和第一图像数据的采集装置的外参数，确定出该采集装置的位姿；并基于此，对第一图像数据中的特征点进行三角化，从而得到该特征点在世界坐标系下的空间位置，完整初步建立表征第一图像数据中特征点的空间位置的第一特征地图。
步骤S103，基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图。
在一些实施例中，所述第一图像数据与所述第二图像数据的采集方式不同，包括：第一图像数据的来源与所述第二图像数据的来源不同、第二图像数据与第一图像数据的采集装置不同，或第一图像数据与第二图像数据的采集角度不同。比如，第一图像数据的采集装置为车载摄像机，第二图像数据的采集装置为航拍无人机。在一些可能的实现方式中，第一图像数据可以是在地面采集的数据，第二图像数据可以是在空中采集的数据。通过来源不同的第一图像数据和第二图像数据进行匹配，基于匹配的第一图像数据和第二图像数据，采用增量式运动恢复结构的方法更新第一特征地图，得到第二特征地图。
步骤S104，基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。
在一些实施例中，首先，将第一图像数据和第二图像数据转换成点云数据；然后，通过分析点云数据每一点的可见性以及重投影误差，确定该点是不是可以作为顶点，以构建四面体；最后，通过集合第一图像数据和第二图像数据的特性，对构建的四面体进行优化，得到更加平面的三维地图。
在本公开实施例中，首先，通过将采集到的导航数据作为第一图像数据的图像位姿，能够快速构 建表征第一图像数据的图像特征空间位姿的第一特征地图；然后，通过采用采集方式不同的第二图像数据对第一特征地图进行优化更新，能够使得第二特征地图中包括的特征点更加丰富；最后，通过将采集方式不同的两个图像数据与第二特征地图进行结合，能够构建覆盖度更高以及精度更高的三维地图。
在一些实施例中，为实现快速构建第一特征地图，上述步骤S102可以通过图2A所示的步骤实现，图2A为本公开实施例提供的地图创建方法的另一实现流程示意图，结合图1B和2A进行以下说明：
步骤S201，对所述第一图像数据进行特征提取，得到图像特征点和所述图像特征点的描述信息。
在一些实施例中，通过对第一图像数据中的每一图像，提取尺度不变特征变换(Scale Invariant Feature Transform，SIFT)的特征点；并获取该特征点的描述信息。该描述信息包括该特征点在图像中的位置(比如，特征点在图像中的二维坐标)、该特征点的尺度不变量和旋转不变量等。从第一图像数据中提取到的图像特征点中包括该图像特征点在该图像中的位置，可以表示为二维坐标的形式。
步骤S202，基于所述图像特征点的描述信息，对第一图像数据中不同图像进行匹配，得到表示不同图像之间的匹配关系的图像关系库。
在一些实施例中，通过分析不同图像的图像特征点的描述信息是否一致，可以确定该不同图像中的图像特征点是否为同一个特征点，进而可以基于描述信息一致的特征点的数量，以确定出不同图像之间的相似度。比如，对于第一图像数据中的任意两个图像(以图像A和图像B为例)，通过分析图像A和B中描述信息一致的图像特征点的数量，即可确定图像A和图像B的相似度；如果图像A和B之间描述信息一致的图像特征点较多，确定图像A和图像B的相似度较高，紧密程度较高。基于所述不同图像之间的相似度，创建图像关系库。将不同图像的图像特征点之间的相似度，作为图像特征点之间的关联；如果两个图像之间的描述信息一致的图像特征点较多，可以认为是两个图像之间的连接比较强，基于这样的关联关系和第一图像数据中的图像，搭建能够表征不同图像之间相似性的图像关系库。比如，首先提取输入第一图像数据中的图像SIFT特征；然后，建立不同图像间的特征点之间的关联；最后，用词袋模型构建作为图像关系库的场景图(Scene Graph)。
步骤S203，基于所述图像关系库和所述导航数据，确定所述第一特征地图。
在一些实施例中，通过首先将导航数据作为第一图像数据的初始图像位姿，结合采集导航数据的导航装置的外参数、采集第一图像数据的采集装置的外参数，确定该采集装置的位姿；然后，将图像关系库中的图像特征点的二维坐标(即特征观测)，该采集装置的位姿和该采集装置的内参数，作为三角化处理的输入，从而能够确定出每一个特征点在世界坐标系中的空间位置，进而实现初步构建的表征第一图像数据中图像特征点空间位置的特征地图，即第一特征地图。
在本公开实施例中，通过对不同图像中图像特征点的描述信息进行匹配，可得到不同图像之间的相似度，从而按照这样不同图像之间的相似性，采用词袋模型构建图像关系库；进而能够通过将该图像关系库中的特征点坐标与导航数据相结合，能够简单且快速构建表示图像数据中图像特征点的空间位置的第一特征地图。
在一些实施例中，通过将导航数据与设定的采集装置的参数相结合，并通过三角化的方式确定出图像特征点的空间位置，即上述步骤204可以通过以下步骤实现：
步骤S141，基于所述导航数据和预设第一采集参数，确定所述第一图像数据的第一采集装置的第一位姿。
在一些实施例中，所述预设第一采集参数包括：采集所述第一图像数据的外参数和采集所述导航数据的外参数。比如，采集第一图像数据的装置为车载摄像机，采集所述导航数据的装置为全球卫星惯性导航系统，那么确定该车载摄像机的外参数，以及全球卫星惯性导航系统的外参数。将导航数据作为第一图像数据的初始图像位姿，结合该车载摄像机的外参数，以及全球卫星惯性导航系统的外参数，可确定出车载摄像机的位姿，即第一位姿C i。
步骤S142，基于所述关系数据库中图像特征点、所述第一位姿和第二采集参数，确定所述第一 特征地图。
在一些实施例中，所述第二采集参数包括采集所述第一图像数据的内参数。第二采集参数为采集第一图像数据的装置的内参数，比如，该装置为车载摄像机，那么该内参数可以是相机内参矩阵。以相机内参矩阵、图像特征点的二维坐标和作为第一位姿的该相机的位姿为基础，对该图像特征点进行三角化，确定出该图像特征点在世界坐标系下的三维坐标，即该图像特征点的空间位置。如此，通过特征匹配关系结合导航数据提供的初始图像位姿，基于三角化原理，能够快速构建图像数据的特征地图。
在一些可能的实现方式中，为提高最终得到的第一特征地图的精确度，通过预测的图像特征点的坐标和该图像特征点在图像中的坐标，对初始特征地图进行优化调整，得到第一特征地图，即上述步骤S142可以通过以下步骤实现：
第一步，基于所述第一位姿和所述第二采集参数，对所述图像特征点在所属图像中的第一真值位置进行三角化，得到所述图像特征点在世界坐标系中的三维坐标。
在一些实施例中，将图像特征点在所属图像中的第一真值位置、第一图像数据的采集装置的第一位姿和该采集装置的内参矩阵，作为对图像特征点实现三角化的输入，以得到图像特征点在世界坐标系中的三维坐标，从而实现对每一个图像特征点的三角化。
第二步，基于所述三维坐标，构建所述第一特征地图。
在一些实施例中，在得到图像中每一个图像特征点的三维坐标之后，构建该图像特征点的特征地图，以该特征地图表征图像特征点的空间位置；该特征地图可以直接作为第一特征地图，用于后续的地图更新；还可以是通过该特征地图的特征点坐标和图像位姿进行优化得到该第一特征地图。如此，通过对图像特征点的真值位置进行三角化，得到三角化后的特征点，基于三角化的特征点的三维坐标能够快速构建第一特征地图。
在一些可能的实现方式中，对图像特征点进行三角化之后，初步构建初始特征地图，通过对初始特征地图进行优化调整，得到该第一特征地图，使得该第一特征地图的精度更加准确，即上述第二步可以通过以下过程实现：
首先，基于所述三维坐标，构建表征所述图像特征点的空间位置的初始特征地图。
其次，基于所述导航数据到相机坐标系的转换参数和所述第二采集参数，确定所述空间位置投影至第一图像数据的第一预测位置。
在一些实施例中，导航数据到相机坐标系的转换参数，包括：导航数据的导航装置的位姿(包括，导航装置的方向R i和位置t i)、导航数据的导航装置到相机坐标系的旋转矩阵和平移向量；比如，导航装置为全球卫星惯性导航系统，转换参数包括：全球卫星惯性导航系统的方向R i和位置t i、卫星惯性导航系统到相机坐标系的旋转矩阵和平移向量(T R和T t)。第二采集参数为采集第一图像数据的采集装置的内参矩阵K，初始特征地图中第j个图像特征点的三维坐标X j。基于此，确定该图像特征点从空间位置投影到图像中的第一预测位置为K(T RR iX j+T Rt i+T t)。
再次，确定所述第一真值位置和所述第一预测位置之间第一差值。
在一些实施例中，关系数据库中图像特征点在所属图像中的第一真值位置x ij，通过确定第一预测位置与第一真值位置的差值，即可预估该初始特征地图中该特征点所在空间位置的合理性；基于该差值对初始特征地图中空间位置不合理的图像特征点进行优化调整，从而得到精度更高的第一特征地图。比如，对于初始特征地图中的每一个特征点，确定出该特征点由空间位置投回到图像上的第一预测位置，即在第一图像数据中进行预测的位置；基于该特征点在图像中的第一真值位置，估计该第一 预测位置的预测误差。
最后，基于所述第一差值，对所述初始特征地图中的特征点的空间位置进行调整，得到所述第一特征地图。
在一些实施例中，基于该第一差值，可以对初始特征地图中的图像特征点的三维坐标，以及，图像位姿进行调整，以使调整后的初始特征地图中的图像特征点的三维坐标以及调整后的图像位姿更加接近于第一真值位置，从而完成基于车载采集数据对特征地图的构建。如此，通过确定初始特征地图中的特征点由空间位置投回到图像上的第一预测位置，以及该特征点在图像中的第一真值位置，可以对该初始特征地图中的特征点的空间位置以及图像位姿进行优化，使得优化后的第一特征地图的精度更高。
在一些实施例中，为提高构建的第一特征地图的高覆盖度，采用来源不同的第二图像数据对第一特征地图进行更新，得到覆盖度更高的第二特征地图；即上述步骤S103可以通过如图2B所示的步骤实现，图2B为本公开实施例提供的地图构建方法的又一实现流程示意图，结合图1B和2B所示的步骤进行以下说明：
步骤S221，基于所述第二图像数据与所述第一图像数据之间的匹配关系，更新图像关系库，得到已更新图像关系库。
在一些实施例中，采用第二图像数据对图像关系库进行更新的方式，与步骤S201和步骤S202的实现方式基本相同。以该第二图像数据为航拍图像数据，第一图像数据为车载摄像机采集的地面图像数据为例，通过对第二图像数据进行特征提取，按照提取的特征点的描述信息与第一图像数据中特征点的描述信息，确定第一图像数据和第二图像数据中图像之间的相似度，即得到图像之间的匹配关系。基于该匹配关系，对按照第一图像数据中的图像之间的匹配关系建立的图像关系库，进行更新，从而在已更新图像关系库中不仅能够体现第一图像数据中图像见的相似程度，还能够体现第一图像数据与第二图像数据之间的相似程度。
步骤S222，在所述已更新图像关系库中，确定待注册图像。
在一些实施例中，所述待注册图像中存在与所述第一特征地图中的三维点相对应的目标特征点。在已更新图像关系库中，对于每一个图像，确定该图像中特征点是否能够在第一特征地图中找到对应的三维点。如果该图像中的特征点能够在第一特征地图中找到对应的三维点，确定该图像为待注册图像。
步骤S223，基于所述目标特征点，确定所述待注册图像的图像位姿。
在一些实施例中，对于这些能够在第一特征地图中找到对应的三维点的目标特征点，采用随机采样的方式，确定该目标特征点所属图像的图像位姿，即该待注册图像的图像位姿。
步骤S224，将图像位姿，注册到第一特征地图中，得到已注册特征地图。
在一些实施例中，将按照目标特征点确定的图像位姿，注册到第一特征地图中，以使该已注册特征地图中能够呈现该目标特征点所属待注册图像的画面内容的空间位置。如此，通过结合来源不同的第一图像数据和第二图像数据，能够有效利用车载摄像机和无人机数据的优势，提升建图的覆盖度和完整度。
在一些可能的实现方式中，通过统计目标特征点的数量，确定待注册图像的注册顺序，从而按照该注册顺序依次将待注册图像的图像位姿注册到第一特征地图中，可以通过以下步骤实现：
第一步，确定每一待注册图像包括的目标特征点的数量。
这里，对于每一帧待注册图像，统计该待注册图像中目标特征点的数量，待注册图像中的目标特征点越多，说明该待注册图像与待绘制区域的相似度越高；即该待注册图像是针对待绘制区域采集的图像的可能性更高，还说明待注册图像与第一特征地图中的三维点的重叠度较高，那么可优先注册该待注册图像。
第二步，基于所述数量，确定所述每一待注册图像的注册顺序。
这里，按照数量的从大到小，确定待注册图像的注册顺序。将目标特征数量最多的待注册图像排列在第一个，优先进行图像位姿的注册。
第三步，基于所述注册顺序，将所述每一待注册图像的图像位姿注册到所述第一特征地图中，得到所述已注册特征地图。
比如，针对目标特征数量最多的待注册图像，首先，将图像位姿注册到第一特征地图中，然后，再将目标特征数量次之的待注册图像的图像位姿注册到第一特征地图中。这样，通过按照目标特征的数量，确定待注册图像的注册顺序，能够提高注册的图像位姿的准确度。
步骤S225，基于所述待注册图像中的其他特征点和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图。
在一些实施例中，所述其他特征点为所述待注册图像中目标特征点之外的特征点。
在一些可能的实现方式中，为进一步优化注册后的特征地图，通过对其他特征点进行上采样，并对采样的一个特征点进行三角化处理，按照三角化处理的结果对已注册特征地图进行优化调整，上述步骤S225还包括以下步骤：
第一步，对所述其他特征点进行采样，得到采样特征点。
这里，对待注册图像中的其他特征点进行均匀采样，每一采样区域保留少量的特征点，比如，每个采样区域保留一个采样特征点，这样，能够减少图像特征过于集中，降低全局优化的复杂度。
第二步，对所述采样特征点进行三角化，确定所述采样特征点在世界坐标系中的三维坐标。
这里，通过将该采样特征点对应的采集装置的位姿、采集装置的内参矩阵以及该采样特征点在图像中的坐标，对该采样特征点进行三角化，以确定该采样特征点的空间位姿，即采样特征点在世界坐标系中的三维坐标。
第三步，基于所述采样特征点的三维坐标和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图。
这里，通过将采样特征点空间位置和第二图像数据的采集装置的位姿等相结合，可以进一步预测该采样特征点的实际位置与预估的空间位置的差异，进而可基于该差异对已注册特征地图中的三维点的位置进行优化，得到第二特征地图。在一些可能的实现方式中，可以通过以下步骤实现：
第一步，基于所述导航数据到相机坐标系的转换参数，确定所述采样特征点的三维坐标投影至第一图像数据的第二预测位置和所述采样特征点的三维坐标投影至第二图像数据的第三预测位置。
确定第二预测位置的方式与确定第一预测位置的方式类似，确定导航数据到相机坐标系的旋转矩阵和平移向量(T R和T t)、导航装置的方向R i和位置t i、采集第一图像数据的采集装置(比如，车载摄像机)的内参矩阵K C、采样特征点的三维坐标X j；基于此，确定该采样特征点从空间位置投影到第一图像中的第二预测位置为K C(T RR iX j+T Rt i+T t)。
基于采集第二图像数据的采集装置(比如，航拍无人机)的内参矩阵K A、导航装置的方向R i和位置t i以及采样特征点的三维坐标X j，确定出该采样特征点从空间位置投影到第二图像中的第三预测位置K A(R kX j+t k)。
第二步，确定所述第二预测位置和所述采样特征点在所属图像中的第二真值位置之间的第二差值，以及，所述第三预测位置与所述第二真值位置的第三差值。
对于每一个采样区域中得到的采样特征点，确定出该采样特征点由空间位置投回到第一图像数据上的第二预测位置，即在第一图像数据中进行预测的位置；基于该采样特征点在所属图像中的第二真值位置，估计该第二预测位置的预测误差。同理，对于该采样特征点，估计其在第三预测位置上的预测误差，即第三差值。
第三步，基于第二差值和所述第三差值，对所述已注册特征地图中的特征点的空间位置进行调整， 得到所述第二特征地图。
基于该第二差值，可以对已注册特征地图中对应于第一图像数据中图像特征点的三维坐标，以及，第一图像数据图像位姿进行调整。基于该第三差值，可以对已注册特征地图中对应于第二图像数据中图像特征点的三维坐标，以及，第二图像数据图像位姿进行调整。如此，通过在每一采样区域均匀采样得到一个采样特征点，并通过采样特征点的预测位置与在图像中的实际位置之间的差值，对第一特征地图中的特征点的空间位置进行优化，使得第二特征地图的完整度更高。
在一些实施例中，在第二特征地图的基础上，通过对图像数据进行深度恢复、点云生成以及网格构建等，最终创建高精度的三维地图，即上述步骤S105可以通过以下步骤实现：
步骤S151，对所述第二图像数据、所述第二特征地图和所述第一图像数据进行深度处理，生成表征所述第一图像数据和所述第二图像数据的点云数据。
在一些实施例中，在第二图像数据中包括第二图像的图像位姿，在第一图像数据中包括第一图像的图像位姿；在第二特征地图中包括第一图像数据中的图像特征点在该第二特征地图中对应的第一类三维特征点，以及，第二图像数据中的图像特征点在该第二特征地图中对应的第二类三维特征点。将第一图像位姿与第一类三维特征点作为一组进行深度估计和深度图融合，得到表征第一图像数据的点云数据；将第二图像位姿与第二类三维特征点作为一组进行深度估计和深度图融合，得到表征第二图像数据的点云数据。
在一些可能的实现方式中，可以通过以下步骤生成点云数据：
第一步，对所述第二图像数据、所述第二特征地图和所述第一图像数据进行深度估计，得到深度图。
对第二图像数据中的第二图像、第二图像位姿与第二类三维特征点结合在一起，进行深度估计，得到第二图像的深度图；对第一图像数据中的第一图像、第一图像位姿与第一类三维特征点结合在一起，进行深度估计，得到第一图像的深度图。将第一图像的深度图和第二图像的深度图，作为该步骤得到的深度图。
第二步，将所述深度图进行融合，得到所述点云数据。
分别对第一图像的深度图和第二图像的深度图进行深度融合，生成第一图像对应的点云数据和第二图像对应的点云数据；将这两类点云数据进行点云整合得到最终的点云数据。如此，通过对两类图像进行深度估计和深度融合，使得到的点云数据更加丰富。
步骤S152，基于所述点云数据，构建表征所述点云数据之间的连接关系的初始网格模型。
在一些实施例中，点云数据之间的连接关系即点在网关模型中是否相连。按照基于两类图像数据整合的点云数据，以及点的可见性和重投影误差判断该点是否可以作为顶点，以构建四面体，从而可以基于多个顶点构建多个四面体，以作为该初始网格模型。
在一些可能的实现方式中，可以通过以下步骤搭建初级网关模型：
第一步，确定所述点云数据中每一点在所述第二特征地图中的可见性和所述每一点的重投影误差。
点云数据中每一点在所述第二特征地图中的可见性，即该点在第二特征地图中是否有对应的三维特征点，如果该点在第二特征地图中有对应的三维特征点，说明该点可见；如果该点在第二特征地图中没有对应的三维特征点，说明该点不可见。在一些实施例中，关于重投影误差，通过特征匹配对可知，观测值A和B是一组特征匹配对，它们是同一个空间点C的投影，A属和B属于不同的地图，B`为将A转换为在B所属坐标系中的第一全局坐标之后在B所属坐标系上的投影点。A的投影B`与观测值B之间有一定的距离，这就是重投影误差。其中，点云数据中每一点的重投影误差即为该点在两类图像中的投影A和B，比如，A在第一图像数据中，B在第二图像数据中；该点的重投影误差即为将A转换为在B所属坐标系中的第一全局坐标之后在B所属坐标系上的投影点，与观测值B之间的距离。
第二步，在所述点云数据中，确定可见性和重投影误差满足预设条件目标点。
在点云数据中，确定点的可见性为该点可在第二特征地图中查找到对应的三维特征点，以及重投 影误差小于一定阈值的目标点，可得到多个目标点。
第三步，将所述目标点作为顶点，并将所述顶点进行连接，得到所述初始网格模型。
将这多个目标点作为顶点，将空间中的任意顶点，进行连线可以构成四面体，得到多个四面体，并将该四面体作为初始网格模型。如此，通过将点云数据中的点，按照点的可见性和重投影误差构建四面体，使得到的初始网格模型能够充分体现目标点之间的连接关系。
步骤S153，基于所述初始网格模型，确定所述三维地图。
在一些实施例中，由于初始网格模型中包括多个四面体，按照四面体中每一面为真实物体表面的置信度，在这多个四面体的面中，选出最优的一组面，将这最优的一组面进行连接，形成三维地图。如此，通过深度恢复、点云生成及网格构建最终生成高精度的稠密点云和网格模型，既能够保留车载数据高精度的特点同时结合航拍视角覆盖率高的优势，使得到的三维地图的精确度更高。
在一些可能的实现方式中，可以通过以下步骤实现对三维地图的构建：
第一步，在所述初始网格模型中，确定每一个面的多个顶点对应的点云数据的采集来源。
在该初始网格模型包括的多个四面体的多个面中，确定每一个面的三个顶点的来源。以该面为三角面片为例，分析该面的三个顶点的采集状态，三个顶点是来自于不同的车载摄像机，还是三个顶点均来自于同一个车载摄像机或导航装置(比如，航拍装置)。
第二步，基于所述采集来源，确定表征所述每一个面被视线穿透的权重。
如果三个顶点是来自于不同的车载摄像机，说明该三角面片的三个顶点不属于同一个物体的可能性较大，即该三角面片为真实物体表面的可能性较大；那么该面片被视线穿透的权重越大；其中，该视线为从采集装置发出至面片的一根线。
第三步，基于权重小于预设阈值的面，确定所述三维地图。
对于每一个面片均确定出一个表征该面被视线穿透的权重，选择一组权重之和最小的面；将这一组面进行连接，得到该三维地图。如此，通过分析四面体中一个面被视线穿透的权重，选择权重较小的面构建三维地图，使得构建的三个地图更加平滑。
下面，将说明本公开实施例在一个实际的应用场景中的示例性应用，以基于全球卫星惯性导航系统、摄相机和航拍无人机，构建高精度地图为例，进行说明。
随着计算机及通信技术的不断发展，地图为人们出行提供了很大帮助，地图提供的信息包括但不限于道路信息、楼宇信息、交通信息等。其中高精度地图以其厘米级的重建精度和丰富的层次信息倍受关注。高精度地图多用于无人驾驶领域。在相关技术中，地图多采用常规采集加人工后处理方法生成地图。比如，高精度专业设备采集和众包采集方法均需在数据采集后进行手动对齐等操作。而且高精度地图数据的采集多为采集车在铺装道路范围内采集，对非道路区域重建精度有限，难以满足全区域高精度地图构建的需求，整体重建的完整度及层次结构尚不完善。
基于此，本公开实施例在地面采集的基础上结合航拍无人机覆盖全面、采集便利的优势，提出一种空地非同源数据的全区域覆盖的高精度地图构建方法。在本公开实施例中，使用车载全球卫星惯性导航系统、摄相机和航拍无人机即可全自动完成高精度地图构建的方法及采集系统。
图3为本公开实施例提供的地图构建方法的实现流程示意图，从图3可以看出，该方法可以通过以下步骤实现：
步骤S301，获取车载全球卫星惯性导航系统的位姿数据，并获取车载摄像机图像数据。
将车载全球卫星惯性导航系统的位姿数据和车载摄像机图像数据进行标定，并同步。车载全球卫星惯性导航系统的位姿数据可以作为上述实施例中的导航数据。
步骤S302，基于位姿数据和摄相机图像，构建车载采集数据的特征地图。
在一些实施例中，车载采集数据的特征地图可以作为上述实施例中的第一特征地图。通过将车载摄像机采集的图像间的特征匹配关系与全球卫星惯性导航系统提供的位姿相结合，基于三角化原理，构建地面数据的特征地图。主要包含以下3个流程：特征提取与匹配；三角化特征点；调整优化。如图4所示，图4为本公开实施例提供的初步构建特征地图的实现流程示意图，结合图4所示的步骤进行以下说明：
步骤S401，对车载摄像机图像数据41进行特征提取与匹配。
在一些实施例中，使用尺度不变特征变换(Scale Invariant Feature Transform，SIFT)特征点的匹配数量来衡量图像关联的紧密程度。首先，提取输入图像的SIFT特征；然后，建立图像间的特征点关联；最后，用词袋模型构建场景图(Scene Graph)。
步骤S402，基于全球卫星惯性导航系统的位姿数据42和摄像机的外参数，三角化初始特征地图。
将全球卫星惯性导航系统获取到的位姿数据结合车载全球卫星惯性导航系统与摄像机的外参数，求解相机C i的位姿(R i和t i)，以此为基础三角化特征点。初步构建车载数据的初始特征地图。
步骤S403，对三角化后的初始特征地图进行优化调整。
对初始特征地图的特征点坐标和图像位姿进行调整的过程，如公式(1)所示：
Figure PCTCN2022093620-appb-000001
其中，n为图像数量，m为地图点的数量，K为相机内参矩阵，T R和T t分别为卫星惯性导航系统到相机坐标系的旋转矩阵和平移向量，X为地图点三维坐标，x为特征观测，
Figure PCTCN2022093620-appb-000002
由此优化更新特征地图的特征点坐标和图像位姿，实现基于车载采集数据特征地图构建环节。
步骤S303，基于无人机航拍数据31，更新特征地图。
在一些实施例中，提取无人机航拍所得图像的特征点并与车载摄像机图像数据匹配，更新场景图和特征地图，该过程主要包括航拍图像特征提取与匹配、特征地图更新两个部分。
第一部分：与步骤S401类似，提取无人机航拍所得的航拍图像特征，将该图像特征与车载摄像机图像的特征一同匹配，更新原有的场景图。
结合新的场景图与已有的特征地图，使用增量式运动恢复结构的方法更新特征地图，可以通过图5所示的步骤实现：
步骤S501，获得基于车载采集图像创建的第一特征地图，和已更新场景图。
步骤S502，在已更新场景图中选择待注册帧图像。
待注册帧的选取主要依据可视地图点数量。可视地图点数量反映了观测区域的相似程度，当新注册图像中特征点大部分都能在当前特征地图中找到对应三维点时，说明观测重叠度较高，可优先注册。
步骤S503，对待注册图像中的其他特征点进行征三角化。
对于待注册图像，通过有地图观测的特征点，采用pnp+ransac求解图像位姿，注册至特征地图。之后，对没有地图观测的特征点均匀采样，每个采样区域只保留一个特征点。通过三角化求解特征点的空间位置。保证特征地图完整覆盖的基础上避免图像特征过于集中，降低全局优化的复杂度。
步骤S504，基于三角化后的特征点，对第一特征地图进行进行局部/全局优化调整，得到第二特征地图。
对该第一特征地图中的三维特征点和图像位姿，进行优化调整的过程，如公式(2)所示：
Figure PCTCN2022093620-appb-000003
其中，O为航拍图像数量，K A为航拍图像内参数，R k和t k为图像对应的旋转矩阵和平移向量，v kj与v ij均表示特征点j在对应图像中的可见性，其余参数的含义与公式(1)一致。
步骤S304，基于传感器融合的高精度稠密点云，生成网格模型。
在已有特征地图的基础上，通过深度恢复、点云生成及网格构建最终产出高精度的稠密点云和网格模型。在多视图几何的基础上，将车载采集数据和无人机航拍数据联合处理，保留了车载数据高精度的特点以及航拍视角覆盖率高的优势。实现流程如图6所示，对车载摄像机图像数据61、第二特征地图和图像位姿62，和无人机航拍图像数据63独立完成深度估计和深度图融合，以及点云合并生成稠密点云、共同构建网格模型两个过程。
步骤S601，对车载摄像机图像数据61、该数据的图像位姿以及该数据在第二特征地图中对应的三维特征点，进行深度估计，得到深度图。
步骤S602，对该步骤S601得到的深度图进行深度图融合，得到车载摄像机图像数据61的点云。
步骤S603，对无人机航拍图像数据63、该数据的图像位姿以及该图像在第二特征地图中对应的三维特征点，进行深度估计，得到深度图。
步骤S604，对步骤S603得到的深度图进行深度图融合，得到无人机航拍图像63的点云。
由图像和对应的位姿使用深度估计方法，独立估计车载和无人机采集的图像深度图。所得深度图融合成点云，生成车载采集数据的点云和无人机航拍数据的点云。将步骤S602和步骤S604得到的点云进行整合，得到点云数据。
步骤S605，将整合车载摄像机采集所得的点云与无人机航拍的所得点云整合，得到点云数据。
该点云数据为稠密点云。
步骤S606，基于该点云数据生成网格模型。
步骤S607，按照设定的权重，对网格模型进行优化调整，得到三维地图。
点云插入环节将整合车载摄像机采集所得的点云与无人机航拍的所得点云整合。在此基础上，依据点的可见性及重投影误差两个指标判断是否将该点作为顶点，构建四面体。在使用图割算法(Grapg-Cut)，从四面体集合中提取三角面片时，结合无人机航拍和车载相机数据的特性，加入权重项，如公式(3)所示：
Figure PCTCN2022093620-appb-000004
其中，b＜＜r，使得最终生成的三维地图更加平滑。
在本公开实施例中，首先，将全球卫星惯性导航系统获取的位姿作为初值，结合最终的调整优化快速构建特征地图，如此，简化了迭代注册和优化环节，充分利用惯导信息提升重建效率。然后，结合不同相机的优势，充分利用航拍图像高覆盖和车载数据观测充分的优势，构建高精度、高覆盖的稠密点云和网格模型；如此，能够提升建图的覆盖度和完整度，最终生成更加完整的三维模型。
本公开实施例提供一种地图构建装置，图7为本公开实施例提供的地图构建装置的结构组成示意图，如图7所示，所述地图构建装置700包括：
第一获取部分701，被配置为获取待绘制区域的第一图像数据和所述第一图像数据对应的导航数据；
第一确定部分702，被配置为基于所述第一图像数据和所述导航数据，确定第一特征地图；
第一更新部分703，被配置为基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图；其中，所述第一图像数据与所述第二图像数据的采集方式不同；
第一构建部分704，被配置为基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。
在一些实施例中，所述第一确定部分702，包括：
第一提取子部分，被配置为对所述第一图像数据进行特征提取，得到图像特征点和所述图像特征点的描述信息；
第一匹配子部分，被配置为基于所述图像特征点的描述信息，对所述第一图像数据中不同图像进行匹配，得到表示所述不同图像之间的匹配关系的图像关系库；
第一确定子部分，被配置为基于所述图像关系库和所述导航数据，确定所述第一特征地图。
在一些实施例中，所述第一确定子部分，包括：
第一确定单元，被配置为基于所述导航数据和预设第一采集参数，确定所述第一图像数据的采集装置的第一位姿；其中，所述预设第一采集参数包括：采集所述第一图像数据的外参数和采集所述导航数据的外参数；
第二确定单元，被配置为基于所述关系数据库中图像特征点、所述第一位姿和第二采集参数，确定所述第一特征地图；其中，所述第二采集参数包括采集所述第一图像数据的内参数。
在一些实施例中，所述第二确定单元，包括：
第一确定子单元，被配置为基于所述第一位姿和所述第二采集参数，对所述图像特征点在所属图像中的第一真值位置进行三角化，得到所述图像特征点在世界坐标系中的三维坐标；
第一构建子单元，被配置为基于所述三维坐标，构建所述第一特征地图。
在一些实施例中，所述第一构建子单元，还用于：基于所述三维坐标，构建表征所述图像特征点的空间位置的初始特征地图；基于所述导航数据到相机坐标系的转换参数和所述第二采集参数，确定所述空间位置投影至第一图像数据的第一预测位置；确定所述第一真值位置和所述第一预测位置之间第一差值；基于所述第一差值，对所述初始特征地图中的特征点的空间位置进行调整，得到所述第一特征地图。
在一些实施例中，所述第一更新部分703，包括：
第一更新子部分，被配置为基于所述第二图像数据与所述第一图像数据之间的匹配关系，更新图像关系库，得到已更新图像关系库；
第二确定子部分，被配置为在所述已更新图像关系库中，确定待注册图像；其中，所述待注册图像中存在与所述第一特征地图中的三维点相对应的目标特征点；
第三确定子部分，被配置为基于所述目标特征点，确定所述待注册图像的图像位姿；
第一注册子部分，被配置为将所述图像位姿，注册到所述第一特征地图中，得到已注册特征地图；
第一调整子部分，被配置为基于所述待注册图像中的其他特征点和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图；其中，所述其他特征点为所述待注册图像中目标特征点之外的特征点。
在一些实施例中，所述第一注册子部分，还包括：
第三确定单元，被配置为确定每一待注册图像包括的目标特征点的数量；
第四确定单元，被配置为基于所述数量，确定所述每一待注册图像的注册顺序；
第一注册单元，被配置为基于所述注册顺序，将所述每一待注册图像的图像位姿注册到所述第一特征地图中，得到所述已注册特征地图。
在一些实施例中，所述第一调整子部分，包括：
第一采样单元，被配置为对所述其他特征点进行采样，得到采样特征点；
第五确定单元，被配置为对所述采样特征点进行三角化，确定所述采样特征点在世界坐标系中的三维坐标；
第一调整单元，被配置为基于所述采样特征点的三维坐标和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图。
在一些实施例中，所述第一调整单元，包括：
第二确定子单元，被配置为基于所述导航数据到相机坐标系的转换参数，确定所述采样特征点的三维坐标投影至第一图像数据的第二预测位置和所述采样特征点的三维坐标投影至第二图像数据的第三预测位置；
第三确定子单元，被配置为确定所述第二预测位置和所述采样特征点在所属图像中的第二真值位置之间的第二差值，以及，所述第三预测位置与所述第二真值位置的第三差值；
第一调整子单元，被配置为基于第二差值和所述第三差值，对所述已注册特征地图中的特征点的空间位置进行调整，得到所述第二特征地图。
在一些实施例中，所述第一构建部分704，包括：
第一处理子部分，被配置为对所述第二图像数据、所述第二特征地图和所述第一图像数据进行深度处理，生成表征所述第一图像数据和所述第二图像数据的点云数据；
第一构建子部分，被配置为基于所述点云数据，构建表征所述点云数据之间的连接关系的初始网格模型；
第二确定子部分，被配置为基于所述初始网格模型，确定所述三维地图。
在一些实施例中，所述第一处理子部分，包括：
第一估计单元，被配置为对所述第二图像数据、所述第二特征地图和所述第一图像数据进行深度估计，得到深度图；
第一融合单元，被配置为将所述深度图进行融合，得到所述点云数据。
在一些实施例中，所述第一构建子部分，包括：
第六确定单元，被配置为确定所述点云数据中每一点在所述第二特征地图中的可见性和所述每一点的重投影误差；
第七确定单元，被配置为在所述点云数据中，确定可见性和重投影误差满足预设条件目标点；
第一连接单元，被配置为将所述目标点作为顶点，并将所述顶点进行连接，得到所述初始网格模型。
在一些实施例中，所述第二确定子部分，包括：
第八确定单元，被配置为在所述初始网格模型中，确定每一个面的多个顶点对应的点云数据的采集来源；
第九确定单元，被配置为基于所述采集来源，确定表征所述每一个面被视线穿透的权重；
第十确定单元，被配置为基于权重小于预设阈值的面，确定所述三维地图。
在一些实施例中，所述第一图像数据的采集装置包括车载摄像机，和/或，所述第二图像数据的采集装置为航拍设备。
需要说明的是，以上装置实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本公开装置实施例中未披露的技术细节，请参照本公开方法实施例的描述而理解。
本公开实施例中涉及的设备可以是系统、方法和计算机程序产品中的至少之一。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是但不限于电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、可擦除可编程只读存储器(Electrical Programmable Read Only Memory，EPROM)或闪存、静态随机存取存储器(Static Random-Access Memory，SRAM)、便携式压缩盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(Digital Video Disc，DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和无线网中的至少之一下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和边缘服务器中的至少之一。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(Industry Standard Architecture，ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一 种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言，诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络，包括局域网(Local Area Network，LAN)或广域网(Wide Area Network，WAN)连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、FPGA或可编程逻辑阵列(Programmable Logic Arrays，PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。
需要说明的是，本公开实施例中，如果以软件功能模块的形式实现上述的地图构建方法，并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台电子设备(可以是终端、服务器等)执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：U盘、运动硬盘、只读存储器、磁碟或者光盘等各种可以存储程序代码的介质。这样，本公开实施例不限制于任何特定的硬件和软件结合。
对应地，本公开实施例再提供一种计算机程序产品，所述计算机程序产品包括计算机可执行指令，该计算机可执行指令被执行后，能够实现本公开实施例提供的地图构建方法中的步骤。
相应的，本公开实施例再提供一种计算机存储介质，所述计算机存储介质上存储有计算机可执行指令，所述该计算机可执行指令被处理器执行时实现上述实施例提供的地图构建方法的步骤。
相应的，本公开实施例提供一种电子设备，图8为本公开实施例提供的电子设备的组成结构示意图，如图8所示，所述电子设备800包括：一个处理器801、至少一个通信总线、通信接口802、至少一个外部通信接口和存储器803。其中，通信接口802配置为实现这些组件之间的连接通信。其中，通信接口802可以包括显示屏，外部通信接口可以包括标准的有线接口和无线接口。其中所述处理器801，配置为执行存储器中图像处理程序，以实现上述实施例提供的地图构建方法的步骤。
以上地图构建装置、电子设备和存储介质实施例的描述，与上述方法实施例的描述是类似的，具有同相应方法实施例相似的技术描述和有益效果，限于篇幅，可案件上述方法实施例的记载，故在此不再赘述。对于本公开地图构建装置、电子设备和存储介质实施例中未披露的技术细节，请参照本公开方法实施例的描述而理解。
应理解，说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本公开的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解，在本公开的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本公开实施例的实施过程构成任何限定。上述本公开实施例序号仅仅为了描述，不代表实施例的优劣。需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
在本公开所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元；既可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。
另外，在本公开各实施例中的各功能单元可以全部集成在一个处理单元中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：移动存储设备、只读存储器(Read Only Memory，ROM)、磁碟或者光盘等各种可以存储程序代码的介质。
或者，本公开上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台电子设备(可以是个人计算机、服务器、或者网络设备等)执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、ROM、磁碟或者光盘等各种可以存储程序代码的介质。以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以所述权利要求的保护范围为准。
工业实用性
本公开实施例提供一种地图构建方法、装置、设备、存储介质及程序，其中，所述方法包括：获取待绘制区域的第一图像数据和所述第一图像数据对应的导航数据；基于所述第一图像数据和所述导航数据，确定第一特征地图；基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图；其中，所述第一图像数据与所述第二图像数据的采集方式不同；基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。

Claims (20)

  1. 一种地图构建方法，所述方法包括：
    获取待绘制区域的第一图像数据和所述第一图像数据对应的导航数据；
    基于所述第一图像数据和所述导航数据，确定第一特征地图；
    基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图；其中，所述第一图像数据与所述第二图像数据的采集方式不同；
    基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。
  2. 根据权利要求1所述的方法，其中，所述基于所述第一图像数据和所述导航数据，确定第一特征地图，包括：
    对所述第一图像数据进行特征提取，得到图像特征点和所述图像特征点的描述信息；
    基于所述图像特征点的描述信息，对所述第一图像数据中不同图像进行匹配，得到表示所述不同图像之间的匹配关系的图像关系库；
    基于所述图像关系库和所述导航数据，确定所述第一特征地图。
  3. 根据权利要求2所述的方法，其中，所述基于所述图像关系库和所述导航数据，确定所述第一特征地图，包括：
    基于所述导航数据和预设第一采集参数，确定所述第一图像数据的采集装置的第一位姿；其中，所述预设第一采集参数包括：采集所述第一图像数据的外参数和采集所述导航数据的外参数；
    基于所述关系数据库中图像特征点、所述第一位姿和第二采集参数，确定所述第一特征地图；其中，所述第二采集参数包括采集所述第一图像数据的内参数。
  4. 根据权利要求3所述的方法，其中，所述基于所述关系数据库中图像特征点、所述第一位姿和第二采集参数，确定所述第一特征地图，包括：
    基于所述第一位姿和所述第二采集参数，对所述图像特征点在所属图像中的第一真值位置进行三角化，得到所述图像特征点在世界坐标系中的三维坐标；
    基于所述三维坐标，构建所述第一特征地图。
  5. 根据权利要求4所述的方法，其中，所述基于所述三维坐标，构建所述第一特征地图，包括：
    基于所述三维坐标，构建表征所述图像特征点的空间位置的初始特征地图；
    基于所述导航数据到相机坐标系的转换参数和所述第二采集参数，确定所述空间位置投影至第一图像数据的第一预测位置；
    确定所述第一真值位置和所述第一预测位置之间第一差值；
    基于所述第一差值，对所述初始特征地图中的特征点的空间位置进行调整，得到所述第一特征地图。
  6. 根据权利要求1至5任一项所述的方法，其中，所述基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图，包括：
    基于所述第二图像数据与所述第一图像数据之间的匹配关系，更新图像关系库，得到已更新图像关系库；
    在所述已更新图像关系库中，确定待注册图像；其中，所述待注册图像中存在与所述第一特征地图中的三维点相对应的目标特征点；
    基于所述目标特征点，确定所述待注册图像的图像位姿；
    将所述图像位姿，注册到所述第一特征地图中，得到已注册特征地图；
    基于所述待注册图像中的其他特征点和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图；其中，所述其他特征点为所述待注册图像中目标特征点之外的特征点。
  7. 根据权利要求6所述的方法，其中，所述将所述图像位姿，注册到所述第一特征地图中，得到已注册特征地图，还包括：
    确定每一待注册图像包括的目标特征点的数量；
    基于所述数量，确定所述每一待注册图像的注册顺序；
    基于所述注册顺序，将所述每一待注册图像的图像位姿注册到所述第一特征地图中，得到所述已注册特征地图。
  8. 根据权利要求6或7所述的方法，其中，所述基于所述待注册图像中的其他特征点和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图，包括：
    对所述其他特征点进行采样，得到采样特征点；
    对所述采样特征点进行三角化，确定所述采样特征点在世界坐标系中的三维坐标；
    基于所述采样特征点的三维坐标和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图。
  9. 根据权利要求8所述的方法，其中，所述基于所述采样特征点的三维坐标和所述第二图像数据，对所述已注册特征地图进行调整，得到所述第二特征地图，包括：
    基于所述导航数据到相机坐标系的转换参数，确定所述采样特征点的三维坐标投影至第一图像数据的第二预测位置和所述采样特征点的三维坐标投影至第二图像数据的第三预测位置；
    确定所述第二预测位置和所述采样特征点在所属图像中的第二真值位置之间的第二差值，以及，所述第三预测位置与所述第二真值位置的第三差值；
    基于第二差值和所述第三差值，对所述已注册特征地图中的特征点的空间位置进行调整，得到所述第二特征地图。
  10. 根据权利要求1至9任一项所述的方法，其中，所述基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图，包括：
    对所述第二图像数据、所述第二特征地图和所述第一图像数据进行深度处理，生成表征所述第一图像数据和所述第二图像数据的点云数据；
    基于所述点云数据，构建表征所述点云数据之间的连接关系的初始网格模型；
    基于所述初始网格模型，确定所述三维地图。
  11. 根据权利要求10所述的方法，其中，所述对所述第二图像数据、所述第二特征地图和所述第一图像数据进行深度处理，生成表征所述第一图像数据和所述第二图像数据的点云数据，包括：
    对所述第二图像数据、所述第二特征地图和所述第一图像数据进行深度估计，得到深度图；
    将所述深度图进行融合，得到所述点云数据。
  12. 根据权利要求10或11所述的方法，其中，所述基于所述点云数据，构建表征所述点云数据之间的连接关系的初始网格模型，包括：
    确定所述点云数据中每一点在所述第二特征地图中的可见性和所述每一点的重投影误差；
    在所述点云数据中，确定可见性和重投影误差满足预设条件目标点；
    将所述目标点作为顶点，并将所述顶点进行连接，得到所述初始网格模型。
  13. 根据权利要求10至12任一项所述的方法，其中，所述基于所述初始网格模型，确定所述三维地图，包括：
    在所述初始网格模型中，确定每一个面的多个顶点对应的点云数据的采集来源；
    基于所述采集来源，确定表征所述每一个面被视线穿透的权重；
    基于权重小于预设阈值的面，确定所述三维地图。
  14. 根据权利要求1至13任一项所述的方法，其中，所述第一图像数据的采集装置包括车载摄像机，和/或，所述第二图像数据的采集装置为航拍设备。
  15. 一种地图构建装置，其中，所述装置包括：
    第一获取部分，被配置为获取待绘制区域的第一图像数据和所述第一图像数据对应的导航数据；
    第一确定部分，被配置为基于所述第一图像数据和所述导航数据，确定第一特征地图；
    第一更新部分，被配置为基于所述待绘制区域的第二图像数据，对所述第一特征地图进行更新，得到第二特征地图；其中，所述第一图像数据与所述第二图像数据的采集方式不同；
    第一构建部分，被配置为基于所述第二图像数据、所述第二特征地图和所述第一图像数据，构建所述待绘制区域的三维地图。
  16. 根据权利要求15所述的装置，其中，所述第一确定部分，包括：
    第一提取子部分，被配置为对所述第一图像数据进行特征提取，得到图像特征点和所述图像特征点的描述信息；
    第一匹配子部分，被配置为基于所述图像特征点的描述信息，对所述第一图像数据中不同图像进行匹配，得到表示所述不同图像之间的匹配关系的图像关系库；
    第一确定子部分，被配置为基于所述图像关系库和所述导航数据，确定所述第一特征地图。
  17. 根据权利要求16所述的装置，其中，所述第一确定子部分，包括：
    第一确定单元，被配置为基于所述导航数据和预设第一采集参数，确定所述第一图像数据的采集装置的第一位姿；其中，所述预设第一采集参数包括：采集所述第一图像数据的外参数和采集所述导航数据的外参数；
    第二确定单元，被配置为基于所述关系数据库中图像特征点、所述第一位姿和第二采集参数，确定所述第一特征地图；其中，所述第二采集参数包括采集所述第一图像数据的内参数。
  18. 一种计算机存储介质，其中，所述计算机存储介质上存储有计算机可执行指令，该计算机可执行指令被执行后，能够实现权利要求1至14任一项所述的方法步骤。
  19. 一种电子设备，其中，所述电子设备包括存储器和处理器，所述存储器上存储有计算机可执行指令，所述处理器运行所述存储器上的计算机可执行指令时能够实现权利要求1至14任一项所述的方法步骤。
  20. 一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备的处理器执行用于实现如权利要求1至14任一所述的地图构建方法。
PCT/CN2022/093620 2021-10-18 2022-05-18 地图构建方法、装置、设备、存储介质及程序 WO2023065657A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111210541.3A CN113920263A (zh) 2021-10-18 2021-10-18 地图构建方法、装置、设备及存储介质
CN202111210541.3 2021-10-18

Publications (1)

Publication Number Publication Date
WO2023065657A1 true WO2023065657A1 (zh) 2023-04-27

Family

ID=79241306

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/093620 WO2023065657A1 (zh) 2021-10-18 2022-05-18 地图构建方法、装置、设备、存储介质及程序

Country Status (2)

Country Link
CN (1) CN113920263A (zh)
WO (1) WO2023065657A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118154801A (zh) * 2024-05-13 2024-06-07 嘉兴明绘信息科技有限公司 一种基于数据分析的三维地图构建方法

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113920263A (zh) * 2021-10-18 2022-01-11 浙江商汤科技开发有限公司 地图构建方法、装置、设备及存储介质
CN114581621A (zh) * 2022-03-07 2022-06-03 北京百度网讯科技有限公司 地图数据处理方法、装置、电子设备和介质
CN115908482B (zh) * 2022-10-14 2023-10-20 荣耀终端有限公司 建模错误数据的定位方法和装置

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110779496A (zh) * 2018-07-30 2020-02-11 阿里巴巴集团控股有限公司 三维地图构建系统、方法、设备和存储介质
CN111357034A (zh) * 2019-03-28 2020-06-30 深圳市大疆创新科技有限公司 点云生成方法、系统和计算机存储介质
CN111670339A (zh) * 2019-03-08 2020-09-15 深圳市大疆创新科技有限公司 用于无人飞行器和地面载运工具之间的协作地图构建的技术
CN112184890A (zh) * 2020-10-14 2021-01-05 佳都新太科技股份有限公司 一种应用于电子地图中的摄像头精准定位方法及处理终端
US20210148722A1 (en) * 2019-11-20 2021-05-20 Thinkware Corporation Method, apparatus, computer program, and computer-readable recording medium for producing high-definition map
CN112837419A (zh) * 2021-03-04 2021-05-25 浙江商汤科技开发有限公司 点云模型构建方法、装置、设备及存储介质
CN113340312A (zh) * 2021-08-05 2021-09-03 中铁建工集团有限公司 一种ar室内实景导航方法及系统
CN113920263A (zh) * 2021-10-18 2022-01-11 浙江商汤科技开发有限公司 地图构建方法、装置、设备及存储介质

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110779496A (zh) * 2018-07-30 2020-02-11 阿里巴巴集团控股有限公司 三维地图构建系统、方法、设备和存储介质
CN111670339A (zh) * 2019-03-08 2020-09-15 深圳市大疆创新科技有限公司 用于无人飞行器和地面载运工具之间的协作地图构建的技术
CN111357034A (zh) * 2019-03-28 2020-06-30 深圳市大疆创新科技有限公司 点云生成方法、系统和计算机存储介质
US20210148722A1 (en) * 2019-11-20 2021-05-20 Thinkware Corporation Method, apparatus, computer program, and computer-readable recording medium for producing high-definition map
CN112184890A (zh) * 2020-10-14 2021-01-05 佳都新太科技股份有限公司 一种应用于电子地图中的摄像头精准定位方法及处理终端
CN112837419A (zh) * 2021-03-04 2021-05-25 浙江商汤科技开发有限公司 点云模型构建方法、装置、设备及存储介质
CN113340312A (zh) * 2021-08-05 2021-09-03 中铁建工集团有限公司 一种ar室内实景导航方法及系统
CN113920263A (zh) * 2021-10-18 2022-01-11 浙江商汤科技开发有限公司 地图构建方法、装置、设备及存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118154801A (zh) * 2024-05-13 2024-06-07 嘉兴明绘信息科技有限公司 一种基于数据分析的三维地图构建方法

Also Published As

Publication number Publication date
CN113920263A (zh) 2022-01-11

Similar Documents

Publication Publication Date Title
WO2023065657A1 (zh) 地图构建方法、装置、设备、存储介质及程序
US10962366B2 (en) Visual odometry and pairwise alignment for high definition map creation
CN110874391B (zh) 一种基于城市空间立体网格模型的数据融合与展示方法
US10297074B2 (en) Three-dimensional modeling from optical capture
US20190026400A1 (en) Three-dimensional modeling from point cloud data migration
US9542773B2 (en) Systems and methods for generating three-dimensional models using sensed position data
US11682170B2 (en) Generating three-dimensional geo-registered maps from image data
JP7440005B2 (ja) 高精細地図の作成方法、装置、デバイス及びコンピュータプログラム
EP3274964B1 (en) Automatic connection of images using visual features
WO2021003487A1 (en) Training data generation for dynamic objects using high definition map data
CN115953535A (zh) 三维重建方法、装置、计算设备和存储介质
US11290705B2 (en) Rendering augmented reality with occlusion
KR20200110120A (ko) 3d-vr 멀티센서 시스템 기반의 도로 시설물 관리 솔루션을 구현하는 시스템 및 그 방법
CN110749308B (zh) 使用消费级gps和2.5d建筑物模型的面向slam的室外定位方法
WO2023151251A1 (zh) 地图构建方法、位姿确定方法、装置、设备及计算机程序产品
Bolkas et al. Creating a virtual reality environment with a fusion of sUAS and TLS point-clouds
CN113379748B (zh) 一种点云全景分割方法和装置
Corral-Soto et al. 3D town: the automatic urban awareness project
US9188444B2 (en) 3D object positioning in street view
WO2023088127A1 (zh) 室内导航方法、服务器、装置和终端
JP2022501751A (ja) ３ｄ幾何抽出のために画像の複数から相補的画像を選択するシステムおよび方法
Gruen et al. Perspectives in the reality-based generation, n D modelling, and operation of buildings and building stocks
Wang et al. Real‐time fusion of multiple videos and 3D real scenes based on optimal viewpoint selection
Böhm et al. Automated appearance-based building detection in terrestrial images
Guo et al. Research on 3D geometric modeling of urban buildings based on airborne lidar point cloud and image

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22882282

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE