CN115187736A

CN115187736A - Target map generation method and device, and AR map generation method and device

Info

Publication number: CN115187736A
Application number: CN202210711293.9A
Authority: CN
Inventors: 董宏伟; 崔乐; 董子龙
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-06-22
Filing date: 2022-06-22
Publication date: 2022-10-14

Abstract

The embodiment of the specification provides a target map generation method and device and an AR map generation method and device, wherein the target map generation method comprises the following steps: the method comprises the steps of obtaining a plurality of initial maps, a first connection relation and a second connection relation, optimizing initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information, determining target sub-map matching pairs at the connection positions of the initial maps according to the first optimized pose information, optimizing the first optimized pose information of the target sub-map matching pairs according to the first connection relation and the second connection relation to obtain second optimized pose information of each sub-map, combining the plurality of initial pose maps according to the second optimized pose information, and generating a target map. The global optimization is carried out on the pose information of each sub-map, the accuracy of the target map is improved, only the pose information of the sub-map at the connection position is locally optimized, the optimization efficiency is improved, and the generation efficiency of the target map is improved.

Description

Target map generation method and device, and AR map generation method and device

Technical Field

The embodiment of the specification relates to the technical field of virtual maps, in particular to a target map generation method.

Background

With the development of computer technology, virtual maps based on real scenes are used in more and more scenes, for example, in game production, movie production, map navigation, automatic driving, and the like.

At present, the generation of a virtual map based on a real scene mainly depends on the acquisition of the real scene by an acquisition device, and then a target map is constructed.

However, as scenes to be constructed are larger and the fineness of the content of the scenes is higher, the requirements on the acquisition time, the performance of acquisition equipment, the time cost of later construction and the labor cost are higher and higher when the limited number of acquisition equipment is used for acquiring and constructing the map. Therefore, a target map generation method that can efficiently generate a virtual map based on a large real scene is needed.

Disclosure of Invention

In view of this, the present specification provides a method for generating a target map. One or more embodiments of the present disclosure relate to a target map generating apparatus, an AR map generating method, an AR map generating apparatus, an electronic device, a computer-readable storage medium, and a computer program, so as to solve technical defects in the prior art.

According to a first aspect of embodiments of the present specification, there is provided a target map generation method including:

acquiring a plurality of initial maps, a first connection relation among the initial maps and a second connection relation among sub-maps in each initial map;

optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map;

determining a target sub-map matching pair at each initial map joint according to the first optimization pose information;

optimizing the first optimization pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimization pose information of each sub-map;

and combining the plurality of initial maps according to the second optimized pose information to generate a target map.

According to a second aspect of the embodiments of the present specification, there is provided an AR map generating method applied to a reality augmented AR device, where the AR device includes a visual sensor, and includes:

receiving an image acquired by a visual sensor;

constructing a plurality of initial maps according to images acquired by a visual sensor, and determining a first connection relation among the plurality of initial maps and a second connection relation among sub-maps in each initial map;

and combining the plurality of initial maps according to the second optimized pose information to generate the AR map.

According to a third aspect of embodiments herein, there is provided a target map generation apparatus including:

the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is configured to acquire a plurality of initial maps, a first connection relation among the initial maps and a second connection relation among sub-maps in each initial map;

the first optimization module is configured to optimize the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map;

the first determination module is configured to determine a target sub-map matching pair at each initial map connection position according to the first optimization pose information;

the second optimization module is configured to optimize the first optimization pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimization pose information of each sub-map;

and the generating module is configured to combine the plurality of initial maps according to the second optimization pose information to generate a target map.

According to a fourth aspect of the embodiments of the present specification, there is provided an AR map generating apparatus applied to a reality augmented AR device, where the AR device includes a visual sensor, and the method includes:

a receiving module configured to receive an image acquired by a vision sensor;

the second determining module is configured to construct a plurality of initial maps according to the images acquired by the visual sensor, and determine a first connection relation among the plurality of initial maps and a second connection relation among sub-maps in each initial map;

the third optimization module is configured to optimize the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map;

the third determining module is configured to determine a target sub-map matching pair at each initial map connection position according to the first optimization pose information;

the fourth optimization module is configured to optimize the first optimization pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimization pose information of each sub-map;

and the second generation module is configured to combine the plurality of initial maps according to the second optimized pose information to generate the AR map.

According to a fifth aspect of embodiments herein, there is provided an electronic apparatus including:

a memory and a processor;

the memory is for storing computer-executable instructions and the processor is for executing the computer-executable instructions, which when executed by the processor, implement the steps of the above-described target map generation method or AR map generation method.

According to a sixth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described target map generation method or AR map generation method.

According to a seventh aspect of embodiments herein, there is provided a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-described target map generation method or AR map generation method.

In one embodiment of the present specification, a plurality of initial maps, a first connection relationship among the plurality of initial maps, and a second connection relationship among sub-maps in each initial map are obtained; optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map; determining a target sub-map matching pair at each initial map joint according to the first optimization pose information; optimizing the first optimization pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimization pose information of each sub-map; and merging the plurality of initial maps according to the second optimized pose information to generate a target map. After the initial pose information of each sub-map is globally optimized, the target sub-map matching pair at the joint of each initial map is determined, the first optimized pose information of the target sub-map pair at the joint is locally optimized, the sub-maps subjected to global optimization and local optimization are obtained, the accuracy of the generated target map is further ensured, meanwhile, only the target sub-map matching pair at the joint is optimized, and the local optimization of all sub-maps is not needed, so that the local optimization efficiency is improved, and the efficiency of generating the target map is further improved.

Drawings

Fig. 1 is a schematic system structure diagram of an object map generating system provided in an embodiment of the present specification;

FIG. 2 is a flow chart of a method for generating a target map according to an embodiment of the present disclosure;

fig. 3 is a flowchart of an AR map generation method provided in an embodiment of the present specification;

fig. 4A is a processing flow chart of a target map generation method applied to a 3D large reality scene according to an embodiment of the present specification;

fig. 4B is a schematic diagram of an initial map joint of a target map generation method applied to a 3D large reality scene according to an embodiment of the present specification;

fig. 5 is a schematic structural diagram of an object map generation apparatus according to an embodiment of the present specification;

fig. 6 is a schematic structural diagram of an AR map generating apparatus according to an embodiment of the present specification;

fig. 7 is a block diagram of an electronic device according to an embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be implemented in many ways other than those specifically set forth herein, and those skilled in the art will appreciate that the present description is susceptible to similar generalizations without departing from the scope of the description, and thus is not limited to the specific implementations disclosed below.

The terminology used in the description of one or the embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of one or the embodiments. As used in this specification or the examples and appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the listed items.

It will be understood that, although the terms first, second, etc. may be used herein to describe various information in one or more embodiments of the present specification, such information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can be termed a second and, similarly, a second can be termed a first without departing from the scope of one or more embodiments of the present description. The word "if," as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination," depending on the context.

First, the noun terms referred to in one or embodiments of the present specification are explained.

AR (Augmented Reality technology): a technology for fusing virtual information with a real world is characterized in that on the basis of scientific technologies such as computers and the like, simulation processing is carried out on entity information which is difficult to experience in a space range of the real world originally, virtual information content is effectively applied in the real world in a superposed mode, and the virtual information content can be sensed by human senses in the process, so that the beyond-reality sense experience is achieved. After the real environment and the virtual object are overlapped, the real environment and the virtual object can exist in the same picture and space at the same time.

VR (Virtual Reality, virtual Reality technology): a computer simulation system that can create and experience virtual worlds utilizes a computer to create a simulated environment into which a user is immersed. The data in real life is utilized, the electronic signals generated by computer technology are combined with various output devices to convert the electronic signals into phenomena which can be felt by people, and the phenomena are expressed by a three-dimensional model.

XR (Extended Reality, augmented Reality technology): the method is characterized in that reality and virtualization are combined through a computer, and a virtual environment capable of man-machine interaction is created. Including AR and VR.

Pose information: a coordinate system representation, typically a combination of a euclidean coordinate system representation and an angular coordinate system representation, for example, (longitude, latitude, altitude, angle 1, angle 2, angle 3), i.e. (X, Y, Z, α, β, γ), characterizes the map location.

A point cloud picture: when a laser beam emitted by the laser radar irradiates the surface of an object, the reflected laser beam can carry information such as direction, distance and the like. The laser beam scans along a certain track, and records the reflected laser point information while scanning, so as to obtain a large number of laser points and form a cloud image of the laser points.

CNN model (Convolutional Neural Network model): a neural network model including convolution operation, padding processing and pooling processing is commonly used to process image information.

At present, a plurality of initial maps are obtained by collecting a real scene by using a collecting device, and the plurality of initial maps are combined to obtain a virtual map of a large scene. For example, for the virtual map construction of a certain market, a plurality of acquisition devices are used for acquiring the real scene of each layer to generate an initial map of each layer, and then the initial maps of each layer are combined to obtain the virtual map of the whole market.

However, due to the fact that the acquisition conditions of each acquisition device are not consistent, for example, coordinate positioning, acquisition range, noise information and the like, in the process of merging a plurality of initial maps, factors such as loss of continuous information among the initial maps, inconsistency of coordinate positioning among the initial maps, interference of noise information and the like occur, so that the generated target map is not consistent enough, and factors such as information loss or information superposition occur, and therefore, pose information of a plurality of initial maps needs to be optimized, so that a target map with higher consistency can be obtained after the plurality of initial maps are merged, and the accuracy of the target map is improved.

In the process of collecting the real scene, performance of the collecting equipment is limited, and in order to collect maps with higher accuracy, collection of each initial map needs to be divided into a plurality of sub-maps, and then the sub-maps are collected by the aid of the collecting equipment, otherwise, collection duration, storage amount and processing speed of the collecting equipment cannot meet actual conditions. Therefore, in the process of optimizing the initial map, not only global optimization of each initial map but also local optimization of each sub-map is required.

However, in such an optimization process, when the number of sub-maps is too large, the amount of data to be processed also increases significantly, and the device performing target map generation has a problem that the efficiency of optimization is slow due to limitations of fast storage and processor performance, resulting in insufficient efficiency of generating the target map.

In the present specification, a method and an apparatus for generating an object map are provided, and the present specification relates to an AR map generating method and an apparatus, an electronic device, and a computer-readable storage medium, which are described in detail in the following embodiments one by one.

Referring to fig. 1, fig. 1 is a schematic system structure diagram of a target map generation system according to an embodiment of the present specification, and specifically includes the following steps.

As shown in fig. 1, the target map generation system includes an acquisition device 101, a map database 102, a target map generation terminal 103, and a display terminal 104, where the acquisition device 101 acquires information data such as image information, point cloud information, or heat information in a real scene, stores the information data in the map database 102, and the target map generation terminal 103 acquires the information data from the map database 102, generates a target map, and then sends the target map to the display terminal 104 for display.

Referring to fig. 2, fig. 2 is a flowchart illustrating a target map generation method according to an embodiment of the present disclosure, which specifically includes the following steps.

Step 202: the method comprises the steps of obtaining a plurality of initial maps, a first connection relation among the initial maps and a second connection relation among sub-maps in each initial map.

The embodiment of the specification is applied to a platform for generating the target map or a processing end of an application program.

The initial map is a virtual map of a partial reality scene in a large reality scene. The virtual map can be obtained by splicing a plurality of sub-maps acquired by acquisition equipment, or can be directly constructed according to image information, point cloud information, heat information and/or depth information acquired by the acquisition equipment. Each initial map includes a plurality of sub-maps. For example, for the map generation of a large real scene in a school, the teaching building A, B, C is an initial map of a part of the real scene. The image information is a photo or a video, and comprises pixel points. The point cloud information is a point cloud picture, which comprises key points representing a real scene. The heat information is a thermodynamic diagram including heat points representing a real scene. The depth information is a depth map, which includes key points characterizing a real scene.

The sub-map is a virtual map that constitutes a part of a partially real scene. The virtual map can be constructed according to image information, point cloud information, heat information and/or depth information of a real scene acquired by acquisition equipment, or can be obtained by segmenting an initial map. The acquisition equipment is sensor equipment for acquiring according to a certain movement path and an acquisition angle. For example, the classroom constituting part of the XX-grade XX-class in the initial map of the teaching building a is a sub-map.

The first connection relation among the initial maps is information such as position information and image information which are used for representing the corresponding relation among the initial maps and are determined by the first connection relation and are related among a plurality of initial maps. For example, in the map generation of a large reality scene in a school, the teaching buildings A, B, C form a triangle-shaped teaching building and are communicated with each other, and the initial maps of the teaching building A, B, C have a first connection relationship.

The second connection relation between the sub-maps in each initial map is a relation representing correspondence among the sub-maps in each initial map, the second connection relation determines information such as position information and image information of the association among the sub-maps, and the second connection relation can be represented by name matching of the sub-maps. For example, for two sub-maps of classrooms of XX grade 1 class and XX grade 2 class in the teaching building a, the classrooms of the two classes have a second association relationship (XX grade 1 class, XX grade 2 class) between the two sub-maps on the same floor. The second connection relation is a corresponding relation between sub-maps in the initial map and the representation initial map, and can be directly obtained from the initial map after the initial map is obtained. The second connection relation is determined according to the image information of each sub-map in the initial map, or according to the point cloud information of each sub-map. For example, when the initial map is constructed, the second connection relation between the sub-maps in the initial map is determined according to the image similarity of the image information between the sub-maps, or the second connection relation between the sub-maps in the initial map is determined according to the point cloud overlap ratio of the point cloud information between the sub-maps.

The obtaining of the plurality of initial maps may be to retrieve corresponding target image information from a map database established in advance according to the image information index, determine a sub-map to which the target image information belongs, and further obtain the initial map to which the sub-map belongs. The method specifically comprises the following steps: and extracting image features of the image information index by using a pre-trained image feature extraction model, comparing the image features with image features of a sub-map in a map database after vectorization processing is carried out on the image features, determining target image information meeting the feature similarity, obtaining the sub-map to which the target image information belongs, and further obtaining an initial map to which the sub-map belongs.

Specifically, a plurality of initial maps are obtained, a first connection relation among the plurality of initial maps is determined according to the image similarity of the image information of the sub-maps among the plurality of initial maps, and a second connection relation among the plurality of sub-maps is determined according to the image similarity of the image information among the plurality of sub-maps in each initial map.

Illustratively, in the generation of a map of a large real-world scene at an airport, there are 3 initial maps of terminals, each with 20 sub-maps of gates. The method comprises the steps of obtaining initial maps (map 1, map2 and map 3) of 3 terminal buildings, determining image similarity Sim1 (submap 1-1 and submap1-2 … … submap 3-20) among submaps (submap 1-1 and submap1-2 … … submap 3-20) of a gate between the 3 initial maps (map 1, map2 and map 3), further determining a first connection relation among the initial maps of the plurality of terminal buildings, determining a second connection relation among the submaps (submap 2) of the plurality of terminal buildings according to image information among the submaps (submap 1-1 and submap-2 … … submap 3-20) of the 20 gates in the initial map (mapi) of the terminal buildings.

The method comprises the steps of obtaining a plurality of initial maps, a first connection relation among the initial maps and a second connection relation among sub-maps in each initial map, and providing a reference basis for optimizing pose information of the sub-maps subsequently.

And 204, optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map.

The initial pose information is expressed by a coordinate system of the sub-map in a coordinate system obtained by combining a plurality of initial maps, wherein the coordinate system can be a world coordinate system determined according to a GPS (global positioning system) or a relative coordinate system determined according to the corresponding relation between the acquisition positions when the acquisition equipment acquires the images. Specifically, a certain collecting device collects information on a certain collecting path, the position of the collected information at the first moment is determined as an initial coordinate system, and the position of the collected information at the second moment is mapped on the initial coordinate system to obtain a corresponding coordinate system representation. For example, the initial pose information of the sub-map of the XX-grade class-1 classroom is (120, 90, 30, 50 °,60 °,70 °).

Optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map, determining two corresponding sub-maps according to the name matching of the sub-maps of the first connection relation and the second connection relation, and completing the optimization of the initial pose information of the two initial sub-maps until the optimization of the initial pose information of the sub-maps in all the initial maps is completed to obtain the first optimized pose information of each sub-map.

Specifically, according to the sub-map name matching of the first connection relation and the second connection relation, two corresponding sub-maps are determined, and the optimization of the initial pose information of the two sub-maps is completed until the optimization of the initial pose information of the sub-maps in all the initial maps is completed, so that the first optimized pose information of each sub-map is obtained.

Exemplarily, according to the matching of the sub-map names (submap 1 and submap 2) of the first connection relation and the second connection relation, determining two corresponding sub-maps submap1 and submap2, and completing the optimization of the initial pose information (130,80,60, 30 °,40 °,50 °) and (120,90,60, 40 °,50 °,60 °) of the two sub-maps until the optimization of the initial pose information of the sub-maps in all the initial maps is completed, so as to obtain the first optimized pose information of each sub-map.

By carrying out global optimization on the initial pose information of the sub-maps of the plurality of initial maps, the accuracy of a subsequently generated target map is improved, and a foundation is laid for subsequently determining the position structure of each initial map and finding out a target sub-map matching pair.

And 206, determining a target sub-map matching pair at each initial map joint according to the first optimized pose information.

And the joints of the initial maps are physical joints among the initial maps obtained after the position structures of the initial maps are determined according to the first optimized pose information.

The target sub-map matching pair is two sub-maps with a connection relation at the connection position of each initial map, wherein the connection relation can be a first connection relation between the sub-maps of each initial map or a second connection relation between the sub-maps in the initial map.

Specifically, the position structure of each initial map is determined according to the first optimization pose information, the connection position between each initial map is obtained according to the position structure of each initial map, the sub-map with the connection relation at the connection position is determined, and the target sub-map matching pair is obtained.

Exemplarily, the position structures of the initial maps of the 3 terminal buildings are determined according to the first optimized pose information, namely, the terminal building A, the terminal building B and the terminal building C are parallel and sequentially connected, the joints among the initial maps are obtained according to the position structures of the initial maps, sub-maps (submap 2-1, submap 3-1), (submap 2-1, submap 2-2) … … with the joints are determined, and the target sub-map matching pairs are obtained.

The method lays a foundation for the local optimization of the first optimization pose information of the subsequent target sub-map matching pairs by determining the target sub-map matching pairs at the joints of the initial maps.

And 208, optimizing the first optimized pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimized pose information of each sub-map.

And optimizing the first optimization pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimization pose information of each sub-map, finishing the optimization of the first optimization pose information of the target sub-map matching pair according to the first connection relation, the second connection relation and the determined matching degree of the corresponding target sub-map to obtain the second optimization pose information of the target sub-map matching pair, and determining the second optimization pose information of each sub-map according to the second optimization pose information of the target sub-map matching pair.

Specifically, according to the first connection relation, the second connection relation and the determined target sub-map matching pair, the first optimization pose information of the target sub-map matching pair is optimized to obtain the second optimization pose information of the target sub-map matching pair, and then the second optimization pose information of each sub-map is determined according to the second optimization pose information of the target sub-map matching pair.

Illustratively, according to the first connection relation and the second connection relation and the determined target sub-map matching pair (submap 2-1 and submap 3-1), optimizing the first optimized pose information (132,84,60, 40 degrees, 50 degrees, 60 degrees) and (118,94,60, 30 degrees, 50 degrees and 50 degrees) of the target sub-map matching pair to obtain second optimized pose information (133,85,60, 35 degrees, 50 degrees, 55 degrees) and (117,96,60, 33 degrees, 52 degrees and 57 degrees) of the target sub-map matching pair, and then determining the second optimized pose information of each sub-map according to the second optimized pose information of the target sub-map matching pair.

The accuracy of a subsequently generated target map is improved by locally optimizing the first optimization pose information of the target sub-map matching pair, and the optimization efficiency is improved by locally optimizing only the target sub-map matching pair at the joint of the initial map, so that the generation efficiency of the target map is improved.

And 210, merging the plurality of initial maps according to the second optimized pose information to generate a target map.

The specific generation mode of the target map is as follows: and determining the position structure of each initial map according to the second optimized pose information, splicing the image information at the corresponding position of each initial map according to the image information of each sub-map, merging the plurality of initial maps, and generating the target map.

Specifically, according to the second optimization pose information and the image information of each sub map, combining the multiple initial maps to generate a target map.

Exemplarily, the position structure of each initial map is determined according to the second optimized pose information, the image information is spliced at the corresponding position of each initial map according to the image information of each sub-map, and the plurality of initial maps are merged to generate the target map.

In the embodiment of the specification, a plurality of initial maps, a first connection relation among the initial maps, and a second connection relation among sub-maps in each initial map are obtained; optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map; determining a target sub-map matching pair at each initial map joint according to the first optimization pose information; optimizing the first optimization pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimization pose information of each sub-map; and merging the plurality of initial maps according to the second optimized pose information to generate a target map. After the initial pose information of each sub-map is globally optimized, the target sub-map matching pair at the joint of each initial map is determined, the first optimized pose information of the target sub-map pair at the joint is locally optimized, the sub-maps subjected to global optimization and local optimization are obtained, the accuracy of the generated target map is further ensured, meanwhile, only the target sub-map matching pair at the joint is optimized, and the local optimization of all sub-maps is not needed, so that the local optimization efficiency is improved, and the efficiency of generating the target map is further improved.

Optionally, the obtaining the first connection relationship among the multiple initial maps in step 202 includes the following specific steps:

acquiring image information and point cloud information of sub-maps in a first initial map and a second initial map, wherein the image information is acquired by a visual sensor, the point cloud information is acquired by a laser radar, and the first initial map and the second initial map are any two of the plurality of initial maps;

determining the image similarity between a first sub-map and a second sub-map according to the image information of each sub-map in the first initial map and the second initial map, wherein the first sub-map is the sub-map in the first initial map, and the second sub-map is the sub-map in the second initial map;

determining a sub-map matching pair between the first initial map and the second initial map according to the image similarity;

determining the point cloud overlapping degree of the sub-map matching pairs according to the point cloud information of the sub-map matching pairs and the sub-map;

and determining a first connection relation between the first initial map and the second initial map according to the point cloud overlapping degree.

The visual sensor is an acquisition device that can acquire visual information of a real scene, for example, a camera, a video recorder, a thermal imaging acquisition device, and the like.

The point cloud information is a point cloud picture obtained by collecting the physical shape and the physical position of each object in a real scene by the laser radar.

The first sub-map is any one of the first initial maps, and the second sub-map is any one of the second initial maps.

The manner of obtaining the image information and the point cloud information of each sub-map in the first initial map and the second initial map may be obtained from a map database established in advance, or may be directly receiving the image information and the point cloud information transmitted after the collection by the vision sensor and the laser radar, which is not limited herein.

Determining the image similarity between the first sub-map and the second sub-map according to the image information of each sub-map in the first initial map and the second initial map, specifically: and extracting local feature points of the image information of the sub-maps in the first initial map and the second initial map, and determining the image similarity between the first sub-map and the second sub-map according to the feature similarity of the local feature points. The feature similarity of the local feature points may be obtained by inputting the image information of each sub-map into a pre-trained image similarity calculation model, performing local feature extraction on the image information, performing vectorization processing, and calculating the similarity between feature vectors to obtain the image similarity between the first sub-map and the second sub-map.

Determining a sub-map matching pair between the first initial map and the second initial map according to the image similarity, specifically: and comparing the image similarity with a preset image similarity threshold, and determining the first initial map and the second initial map which are greater than the image similarity threshold as a sub-map matching pair.

Determining the point cloud overlapping degree of the sub-map matching pairs according to the point cloud information of the sub-map matching pairs and the sub-map, specifically: and (3) carrying out transformation such as rotation, translation, filtering and the like on the point cloud images of the sub-maps in the first initial map and the corresponding sub-maps in the second initial map, determining the overlapping degree of the point cloud coordinates of the point cloud images of the sub-maps, and obtaining the point cloud overlapping degree of the matching pairs of the sub-maps.

Determining a first connection relation between the first initial map and the second initial map according to the point cloud overlapping degree, specifically: and comparing the point cloud overlapping degree with a preset point cloud overlapping degree threshold value, and determining a first connection relation between the first initial map and the second initial map corresponding to the sub-map matching pair with the point cloud overlapping degree threshold value larger than the point cloud overlapping degree threshold value.

Specifically, image information and point cloud information of sub-maps in a first initial map and a second initial map are obtained, local feature points of the image information of the sub-maps in the first initial map and the second initial map are extracted, image similarity between the first sub-map and the second sub-map is determined according to the feature similarity of the local feature points, a sub-map matching pair between the first initial map and the second initial map is determined according to the image similarity, point cloud images of the sub-map matching pair are subjected to transformation such as rotation, translation, filtering and the like of a coordinate system, the overlapping degree of point cloud coordinates of the point cloud images of the sub-map is determined, the point cloud overlapping degree of the sub-map pair is obtained, and a first connection relation between the first initial map and the second initial map is determined.

Exemplarily, image information and point cloud information of a submap of each gate in a first initial map1 of the terminal building a and a second initial map2 of the terminal building B are acquired, wherein the point cloud information is a point cloud map. Inputting image information of sub-maps in a first initial map1 and a second initial map2 into a pre-trained image similarity model, extracting local features, vectorizing, obtaining image similarity of 0.9 according to vector similarity between feature vectors, obtaining a preset image similarity threshold of 0.8, determining a plurality of sub-map matching pairs with the image similarity larger than the image similarity threshold by comparing the image similarity with the preset image similarity threshold, performing transformation such as rotation, translation, filtering and the like on point cloud images of the plurality of sub-map matching pairs, determining weight limit between point cloud coordinates of the sub-map point cloud images, obtaining point cloud overlapping degree of the sub-map matching pairs of 0.88, obtaining a preset point cloud overlapping degree threshold of 0.8, and determining a first connection relation (map 1, map 2) between the first initial map1 and the second initial map2 corresponding to the sub-map pairs with the sub-map overlapping degree larger than the overlapping degree threshold.

determining the image similarity between the first sub-map and the second sub-map according to the image information of each sub-map in the first initial map and the second initial map, determining a sub-map matching pair between the first initial map and the second initial map according to the image similarity, determining the point cloud overlapping degree of the sub-map matching pair according to the point cloud information of the sub-map matching pair, and determining the first connection relation between the first initial map and the second initial map according to the point cloud overlapping degree. Only the sub-map matching pair verified through the image similarity and the point cloud overlapping degree is effective, so that the validity of the connection relation of the corresponding initial map is guaranteed, and the accuracy of subsequent optimization and target map generation is improved.

Optionally, before step 204, the following specific steps are further included:

determining the distribution rule of a plurality of initial maps according to a first connection relation among the plurality of initial maps;

and rejecting first connection relations which do not accord with the distribution rule in the first connection relations among the plurality of initial maps.

Although the first connection relationships among the plurality of initial maps are obtained, in the process of collecting by the actual collecting device, mismatching is inevitably generated, for example, feature points in image information such as classroom doors, stairs and the like in the teaching building a and the teaching building D are the same in style, and when the first connection relationships are determined, the connection relationships are determined for the teaching building a and the teaching building D which do not have actual correspondence relationships. Therefore, the distribution rules of a plurality of initial maps need to be considered again, and mismatching in the first connection relation is eliminated.

The distribution rule of the multiple initial maps is a distribution rule representing sub-map matching pairs of the multiple initial maps, and may be a number relationship of the sub-map matching pairs in the multiple initial maps, or may be a relative pose information consistency of the sub-map matching pairs in the multiple initial maps, or may be a distribution confidence obtained by performing weighted calculation on the sub-map matching pairs and the initial maps, for example, when the number of the sub-map matching pairs between one initial map and the other initial maps is obviously different from the number of the sub-map matching pairs between the other initial maps, or when the relative pose information consistency of the sub-map matching pairs between one initial map and the other initial maps is obviously different from the distribution confidence obtained by calculating according to the number of the sub-map matching pairs and the relative pose information, it is described that there is no actual correspondence between the initial map and the other initial maps at a high probability, and it is necessary to eliminate the first connection relationship between the initial map and the other initial maps.

Specifically, according to the first connection relation among the multiple initial maps, the distribution rule of the sub-map matching pairs of the multiple initial maps is determined, and according to the distribution rule of the sub-map matching pairs of the multiple initial maps, the first connection relation which does not accord with the distribution rule in the first connection relation among the multiple initial maps is eliminated.

Exemplarily, the number relationship of the sub-map matching pairs of the 3 initial maps is determined according to the first connection relationship among the 3 initial maps, and the first connection relationship which does not conform to the distribution rule is eliminated according to the number relationship of the sub-map matching pairs of the 3 initial maps: 10 sub-map matching pairs exist between the initial map A and the initial map B, 5 sub-map matching pairs exist between the initial map A and the initial map D, and 1 sub-map matching pair exists between the initial map B and the initial map D, so that first connection relations (map 1, map 4) and (map 2, map 4) among the initial map D, the initial map A and the initial map B are eliminated.

Continuing the above example, according to the first connection relationship among the 3 initial maps, determining the distribution confidence of the sub-map matching pairs of the 3 initial maps, and according to the distribution confidence of the sub-map matching pairs of the 3 initial maps, eliminating the first connection relationship which does not conform to the distribution rule: the relative pose information of the sub-map matching pair between the initial map a and the initial map B is (0.01,0.03,0.01,1 °,0.6 °,1.2 °), the relative pose information of the sub-map matching pair between the initial map a and the initial map D is (3.40,2.95,1.97,4.6 °,12.6 °,8.7 °), and the relative pose information of the sub-map matching pair between the initial map B and the initial map D is (4.62,6.77,5.84,7.4 °,8.9 °,16.2 °), thereby eliminating the first connection relations (map 1, map 4) and (map 2, map 4) between the initial map D and the initial map a, the initial map B.

Continuing the above example, determining the relative pose information of the sub-map matching pairs of the 3 initial maps according to the first connection relation among the 3 initial maps, and eliminating the first connection relation which does not accord with the distribution rule according to the relative pose information of the sub-map matching pairs of the 3 initial maps: the distribution confidence of the sub-map matching pairs between the initial map a and the initial map B is 0.97, the distribution confidence of the sub-map matching pairs between the initial map a and the initial map D is 0.24, and the distribution confidence of the sub-map matching pairs between the initial map B and the initial map D is 0.17, thereby eliminating the first connection relations (map 1, map 4) and (map 2, map 4) between the initial map D and the initial map a and the initial map B.

Optionally, the obtaining of the second connection relationship between the sub-maps in each initial map in step 202 includes the following specific steps:

acquiring image information of each sub-map in a third initial map, wherein the third initial map is any one of a plurality of initial maps;

determining the image similarity between a third sub-map and a fourth sub-map according to the image information of the third sub-map and the fourth sub-map, wherein the third sub-map and the fourth sub-map are any two of a third initial map;

and determining a second connection relation between the third sub-map and the fourth sub-map according to the image similarity.

The third sub-map is any one of the sub-maps in the third initial map, and the fourth sub-map is any one of the sub-maps in the third initial map except the third sub-map.

The manner of acquiring the image information of each sub-map in the third initial map may be acquired from a map database established in advance, or may be directly receiving the image information transmitted after being acquired by the vision sensor, which is not limited herein.

Determining the image similarity between the sub-maps according to the image information of the sub-maps in the third initial map, specifically: and extracting local feature points of the sub-maps in the third initial map, and determining the image similarity between the third sub-map and the fourth sub-map according to the feature similarity of the local feature points. The feature similarity of the local feature points may be obtained by inputting the image information of each sub-map into a pre-trained image similarity model, performing local feature extraction on the image information, performing vectorization processing, and calculating the similarity between feature vectors to obtain the image similarity between the third sub-map and the fourth sub-map. The image similarity model may be a CNN model.

According to the image similarity, determining a second connection relation between the third sub-map and the fourth sub-map, specifically: and comparing the image similarity with a preset image similarity threshold, and if the image similarity is greater than the image similarity threshold, determining a second association relation between the third sub-map and the fourth sub-map.

Specifically, image information of each sub-map in the third initial map is obtained, local feature points of the image information of the sub-maps in the third initial map are extracted, image similarity between the third sub-map and the fourth sub-map is determined according to feature similarity of the local feature points, and a second connection relation between the third sub-map and the fourth sub-map is determined according to the image similarity.

Illustratively, image information of a sub-map of each parking area in a third initial image of a certain parking lot is acquired, the image information of each parking area is input into a pre-trained image similarity model, and local feature points of the image information of the sub-map of each parking area are extracted: and the paint spraying label determines that the image similarity between the third sub-map 15 of the 15 parking area and the fourth sub-map 18 of the 18 parking area is 0.6 according to the characteristic similarity of the paint spraying label being 0.6, the preset image similarity threshold value is 0.56, the image similarity is greater than the image similarity threshold value, and the second connection relation (sub-map 15, sub-map 18) between the third sub-map of the 15 parking area and the 18 parking area is determined.

Acquiring image information of each sub-map in the third initial map, determining image similarity between the third sub-map and the fourth sub-map according to the image information of the third sub-map and the fourth sub-map, and determining a second connection relation between the third sub-map and the fourth sub-map according to the image similarity. The effectiveness of the sub-map matching pairs in each initial map verified by the image similarity improves the accuracy of subsequent optimization and target map generation.

Optionally, step 204 includes the following specific steps:

optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map, wherein the method comprises the following steps:

determining initial pose information of each sub map according to the first connection relation and the second connection relation;

determining a first target error according to the first connection relation and the initial pose information of each sub-map;

and optimizing the initial pose information of each sub-map by taking the first target error smaller than the first preset error as an optimization condition to obtain the first optimized pose information of each sub-map.

The initial pose information may be obtained from the world coordinate system of the GPS according to the acquisition position W of the acquisition device at time t, or may be obtained by determining the difference Δ W between multiple acquisition times according to the difference Δ t between multiple acquisition times, and further determining the relative coordinate system of each sub-map to obtain the initial pose information of each sub-map in the relative coordinate system, which is not limited herein.

Determining a first target error according to the first connection relation and the initial pose information of each sub-map, wherein the specific mode is as follows: and determining the relative pose information among the sub-maps according to the first connection relation and the initial pose information, and determining the relative pose information among the sub-maps as a first target error. For example, the sub-map 1, the sub-map 2, and the sub-map 3 correspond to the initial pose information of (X1, Y1, Z1, α 1, β 1, γ 1), (X2, Y2, Z2, α 2, β 2, γ 2), (X3, Y3, Z3, α 3, β 3, γ 3), and according to the first connection relationship between the sub-map 1 and the sub-map 2, the first connection relationship between the sub-map 1 and the sub-map 3, and the first connection relationship between the sub-map 2 and the sub-map 3, the relative pose information is (X2-X1, Y2-Y1, Z2-Z1, alpha 2-alpha 1, beta 2-beta 1, gamma 2-gamma 1), (X3-X1, Y3-Y1, Z3-Z1, alpha 3-alpha 1, beta 3-beta 1, gamma 3-gamma 1), (X3-X2, Y3-Y2, Z3-Z2, alpha 3-alpha 2, beta 3-beta 2, gamma 3-gamma 2), and the first target error is obtained.

And optimizing the initial pose information of each sub-map to obtain the first optimized pose information of each sub-map by taking the first target error smaller than the first preset error as an optimization condition.

Specifically, according to the first connection relation and the second connection relation, initial pose information of each sub-map is determined, relative pose information between each sub-map is determined according to the first connection relation and the initial pose information, the relative pose information between each sub-map is determined to be a first target error, the first target error is smaller than a first preset error to serve as an optimization condition, the initial pose information of each sub-map is optimized, and first optimized pose information of each sub-map is obtained.

Exemplarily, for a sub-map of an indoor space, according to a first connection relation and a second connection relation, determining that initial pose information of the sub-map is Wi, determining that relative pose information is Δ W, a first target error δ is Δ W, and a first preset error is σ, performing iterative optimization on the initial pose information of the sub-map, and ending the iterative optimization on the initial pose information until the first target error δ is smaller than the first preset error σ, so as to obtain first optimized pose information Vi.

Determining initial pose information of each sub-map according to the first connection relation and the second connection relation, determining a first target error according to the first connection relation and the initial pose information of each sub-map, and optimizing the initial pose information of each sub-map by taking the first target error smaller than a first preset error as an optimization condition to obtain first optimized pose information of each sub-map. By determining the first target error and optimizing the initial pose information of each sub-map under the optimization condition smaller than the first preset error, the iterative optimization can improve the obtained first initial pose information, improve the accuracy of a subsequently generated target map and lay a data foundation for subsequently determining the target sub-map matching pairs at the joints of each initial map.

Optionally, step 206 includes the following specific steps:

determining the joints of a plurality of initial maps according to the first optimization pose information;

calculating the distance between each sub map and the connection part;

and determining a target sub-map matching pair at the joint of each initial map according to the distance between each sub-map and the joint.

Determining the joints of a plurality of initial maps according to the first optimized pose information, which specifically comprises the following steps: and determining topological structures of the plurality of initial maps according to the first optimization pose information of the sub-maps in the plurality of initial maps, namely the position relations of the plurality of initial maps in the subsequent generated target maps, and determining the joints of the plurality of initial maps according to the topological structures of the plurality of initial maps. For example, the topological structures of the initial map A, the initial map B and the initial map C are determined according to the first optimization pose information, the initial map A is located right above the initial map B, the initial map B is located right above the initial map C, the connection position of the initial map A and the initial map B is determined according to the topological structures, and the connection position of the initial map B and the initial map C is determined.

Calculating the distance between each sub map and the connection part, specifically: and calculating the distance between the first optimization pose information of each sub-map at the outermost part in the initial map and the connection part. For example, the sub-maps at the lowest part of the initial map A are the sub-map 1-1 and the sub-map 1-2, the sub-map at the uppermost part of the initial map B are the sub-map 2-1 and the sub-map 2-2, and the distances between the sub-map 1-1, the sub-map 1-2, the sub-map 2-1 and the sub-map 2-2 and the connection positions are respectively calculated to be L1, L2, L3 and L4.

Determining a target sub-map matching pair at the joint of each initial map according to the distance between each sub-map and the joint, specifically: and comparing the distance between each sub-map and the connection part with a preset distance threshold, and determining two sub-maps with the closest distance at the connection part of each initial map as a target sub-map matching pair under the condition of being smaller than the distance threshold.

Specifically, according to the first optimization pose information, determining topological structures of a plurality of initial maps, according to the topological structures of the plurality of initial maps, determining joints of the plurality of initial maps, calculating the distance between each sub-map and the joints, and according to the distance between each sub-map and the joints, determining a target sub-map matching pair at the joints of the plurality of initial maps.

Exemplarily, the topological structures of the initial map a and the initial map B determined according to the first optimization pose information are as follows: the initial map A is arranged right above the initial map B, the connection position of the initial map A and the initial map B is determined according to the topological structure, the sub-maps at the lowest part of the initial map A are the sub-map 1-1 and the sub-map 1-2, the sub-map at the uppermost part of the initial map B are the sub-map 2-1 and the sub-map 2-2, the distances between the sub-map 1-1, the sub-map 1-2, the sub-map 2-1 and the sub-map 2-2 and the connection position are respectively L1, L2, L3 and L4, and after the distance is compared with a preset distance threshold value L, the matching pair of the target sub-maps is determined to be (the sub-map 1-1, the sub-map 2-1) and (the sub-map 1-2, the sub-map 2-2).

And determining joints of a plurality of initial maps according to the first optimization pose information, calculating the distance between each sub-map and the joint, and determining a target sub-map matching pair at the joint of each initial map according to the distance between each sub-map and the joint. The target sub-map matching pair is determined by determining the joints of the plurality of initial maps and further obtaining the distance between each sub-map and the joint, so that the target sub-map matching pair can be determined more accurately, and a foundation is laid for the first optimization pose information of the sub-maps at the joints subsequently.

Optionally, determining a target sub-map matching pair at the connection position of each initial map according to the distance between each sub-map and the connection position, including the following specific steps:

determining an initial sub-map matching pair at the joint of each initial map according to the distance between each sub-map and the joint;

acquiring image information and point cloud information of each sub-map in the initial sub-map matching pair, wherein the image information is acquired by a visual sensor, and the point cloud information is acquired by a laser radar;

determining the image similarity between the sub-maps in the initial sub-map matching pair according to the image information of the sub-maps in the initial sub-map matching pair;

determining an updated sub-map matching pair according to the image similarity;

determining the point cloud overlapping degree of each sub-map matching pair according to the point cloud information of each sub-map matching pair of the updating sub-map;

and determining the target sub-map matching pairs at the connection positions of the initial maps according to the point cloud overlapping degree.

Determining the image similarity between the sub-maps in the initial sub-map matching pair according to the image information of each sub-map in the initial sub-map matching pair, specifically: and extracting local feature points of the image information of each sub-map in the initial sub-map matching pair, and determining the image similarity between the sub-maps in the initial sub-map matching pair according to the feature similarity of the local feature points. The feature similarity of the local feature points may be obtained by inputting the image information of each sub-map into a pre-trained image similarity calculation model, performing local feature extraction on the image information, performing vectorization processing, and calculating the similarity between feature vectors to obtain the image similarity between each sub-map in the initial sub-map matching pair.

Determining an updated sub-map matching pair according to the image similarity, specifically: and comparing the image similarity with a preset image similarity threshold, and determining the initial sub-map matching pair larger than the image similarity threshold as an updated sub-map matching pair.

Determining the point cloud overlapping degree of the updated sub-map matching pair according to the point cloud information of each sub-map matching pair of the updated sub-map, which specifically comprises the following steps: and performing transformation such as rotation, translation, filtering and the like of a coordinate system on the point cloud images of each sub-map by matching the updated sub-map, determining the overlapping degree of the point cloud coordinates of the point cloud images of the sub-map, and obtaining the point cloud overlapping degree of the matched pair of the updated sub-map.

Determining a target sub-map matching pair at each initial map connection position according to the point cloud overlapping degree, specifically: and comparing the point cloud overlapping degree with a preset point cloud overlapping degree threshold value, and determining the updated sub-map matching pair larger than the point cloud overlapping degree threshold value as a target sub-map matching pair.

Specifically, according to the distance between each sub-map and the connection position, determining an initial sub-map matching pair at the connection position of each initial map, obtaining image information and point cloud information of each sub-map in the initial sub-map matching pair, extracting local feature points of the image information of each sub-map in the initial sub-map matching pair, according to the feature similarity of the local feature points, determining the image similarity between each sub-map in the initial sub-map matching pair, according to the image similarity, determining an updated sub-map matching pair, performing coordinate system rotation, translation, filtering and other transformations on the point cloud images of each sub-map in the updated sub-map matching pair, determining the overlapping degree of point cloud coordinates of the point cloud images of the sub-maps, obtaining the point cloud overlapping degree of the updated sub-map matching pair, and according to the overlapping degree, determining a target sub-map matching pair at the connection position of each initial map.

Illustratively, an initial sub-map matching pair at each initial map junction is determined according to the distance (Li) between each sub-map and the junction: (sub-map 1-1, sub-map 2-1), (sub-map 1-2, sub-map 2-2) … … (sub-map 1-6, sub-map 2-6), acquiring image information and point cloud information of each sub-map in an initial sub-map matching pair, extracting local feature points of the image information of each sub-map in the initial sub-map matching pair, determining the image similarity between each sub-map in the initial sub-map matching pair as (0.6,0.55, … … 0.87.87) and the preset image similarity threshold as 0.6 according to the feature similarity of the local feature points, and determining an updated sub-map matching pair under the condition that the image similarity is greater than the image similarity threshold: (sub-map 1-1, sub-map 2-1), (sub-map 1-5, sub-map 2-5), (sub-map 1-m, sub-map 2-m), performing transformation such as rotation, translation, filtering and the like on the point cloud images of each sub-map by matching the updated sub-map, determining the overlapping degree of the point cloud coordinates of the point cloud images of the sub-map, obtaining the point cloud overlapping degree (0.9,0.78,0.55) of the updated sub-map matching pair, wherein the preset point cloud overlapping degree threshold is 0.6, and determining the target sub-map matching pair (sub-map 1-1, sub-map 2-1) and (sub-map 1-5, sub-map 2-5) under the condition that the point cloud overlapping degree is greater than the point cloud overlapping degree threshold.

Determining an initial sub-map matching pair at each initial map connection position according to the distance between each sub-map and the connection position, acquiring image information and point cloud information of each sub-map in the initial sub-map matching pair, wherein the image information is acquired by a visual sensor, the point cloud information is acquired by a laser radar, determining image similarity between each sub-map in the initial sub-map matching pair according to the image information of each sub-map in the initial sub-map matching pair, determining an updated sub-map matching pair according to the image similarity, determining point cloud overlapping degree of the updated sub-map matching pair according to the point cloud information of each sub-map in the updated sub-map matching pair, and determining a target sub-map matching pair at each initial map connection position according to the point cloud overlapping degree. On the basis of the distance between each sub-map and the joint, the image similarity determined by the image information and the point cloud overlapping degree determined by the point cloud information are further screened, so that the finally achieved target sub-map matching pair is the sub-map which roughly corresponds to the joint in the real scene, reference is provided for subsequent local optimization, the number of first optimization pose information of the sub-map which needs local optimization is reduced, and the subsequent optimization efficiency and the overall target map generation efficiency are improved.

Optionally, step 208 includes the following specific steps:

determining map point matching parameters among the sub-maps according to the first connection relation and the second connection relation;

determining a second target error according to map point matching parameters among the sub-maps and first optimization pose information of a target sub-map matching pair;

optimizing the first optimization pose information of the target sub-map matching pair to obtain second optimization pose information of the target sub-map matching pair by taking the second target error smaller than the second preset error and the target sub-map matching pair reaching the registration constraint condition as an optimization condition;

and determining second optimization pose information of sub-maps except the designated sub-map in the designated initial map according to the second optimization pose information and the second connection relation of the target sub-map matching pair, wherein the designated initial map is the initial map to which the sub-map in the target sub-map matching pair belongs, and the designated sub-map is the sub-map in the target sub-map matching pair.

The map point matching parameters among the sub-maps are the difference of the position parameters of the map points in the real scene in the sub-maps. For the image information, the point cloud information and the heat information, the map point position parameters are position parameters of pixel points, key points and heat points, and the difference of the map point position parameters is determined according to the map point position parameters to obtain map point matching parameters, for example, in the sub-map 1, the positions of eight vertexes of a corner are determined to be [ X1, X2 … … X8], wherein the coordinates of Xi are represented as (Xi, yi, zi), in the sub-map 2, the positions of eight vertexes of the corner are determined to be [ X1', X2' … … X8'], wherein the coordinates of Xi are represented as (Xi', yi ', zi'). The difference of the map point location parameters, i.e., the map point matching parameter Δ d, is [ Xi' -Xi ].

And the second target error is an error value between the difference value of different first optimization pose information and the map point matching parameter, and is determined according to the map point matching parameter between the adjacent sub-maps and the difference value of the first optimization pose information of each sub-map.

The registration condition is an angle constraint that needs to be satisfied by the first optimized pose information, that is, when a map point in a real scene of the sub-map is acquired, extension lines of adjacent acquisition equipment are all gathered on the map point, so that the constraint on the position parameter of the map point is realized.

And optimizing the first optimization pose information of the target sub-map matching pair to obtain second optimization pose information of the target sub-map matching pair by taking the second target error smaller than the second preset error and the target sub-map matching pair reaching the registration constraint condition as an optimization condition.

In an implementation manner, the map point matching parameter is a distance difference Δ d between pixels of the image information, the difference of the first optimized pose information of the target sub-map matching pair representing the same real scene is calculated to be Δ V, the second target error δ is Δ V- Δ d, the second preset error is σ, and the objective function is minF = δ - σ, that is, when the target sub-map matching pair meets the registration constraint condition, the second target error is smaller than the second preset error, and the iterative adjustment of the first optimized pose information is finished.

In another possible implementation manner, the map point matching parameter is a distance difference Δ s of key points of point cloud information or depth information, the difference of the first optimized pose information of a target sub-map matching pair representing the same real scene is calculated to be Δ V, the second target error δ is Δ V- Δ s, the second preset error is σ, and the objective function is minF = δ - σ, that is, when the target sub-map matching pair meets the registration constraint condition, the second target error is smaller than the second preset error, and the iterative adjustment of the first optimized pose information is ended.

In another possible implementation manner, the map point matching parameter is a distance difference Δ h of a heat point of heat information, the difference of the first optimized pose information of the target sub-map matching pair representing the same real scene is calculated to be Δ V, the second target error δ is Δ V- Δ h, the second preset error is σ, and the objective function is minF = δ - σ, that is, when the second target error is smaller than the second preset error, the iterative adjustment of the first optimized pose information is ended.

Determining second optimized pose information of sub-maps except for a designated sub-map in the designated initial map according to the second optimized pose information and the second connection relation of the target sub-map matching pair, wherein the second optimized pose information comprises the following specific steps: and correspondingly optimizing the first optimization pose information of the sub-maps except the target sub-map matching pair in each initial map according to the second optimization pose information and the second connection relation of the target sub-map matching pair.

Specifically, map point matching parameters among sub-maps are determined according to a first connection relation and a second connection relation, a difference value of initial pose information and the map point matching parameters are determined, a difference value of first optimization pose information and the difference value of the map point matching parameters are determined as a second target error, the first optimization pose information of a target sub-map matching pair is optimized by taking the second target error smaller than a second preset error and the target sub-map matching pair reaching a registration constraint condition as an optimization condition, and second optimization pose information of sub-maps except the designated sub-map in the designated initial map is determined according to the second optimization pose information of the target sub-map matching pair and the second connection relation.

Illustratively, for sub-maps of two connected rooms, according to a first connection relation and a second connection relation, determining first optimized pose information of a target sub-map matching pair as Vi, determining a difference value delta V of the initial pose information and map point matching parameters as delta d, determining a second target error delta as delta V-delta d, optimizing the first optimized pose information Vi of the target sub-map matching pair by taking the second target error delta smaller than a second preset error sigma and the target sub-map matching pair reaching a registration constraint condition Limit theta as an optimization condition, obtaining second optimized pose information Ui of the target sub-map matching pair, and determining second optimized pose information U of each sub-map except the designated sub-map in the designated initial map according to the second optimized pose information Ui of the target sub-map matching pair and the second connection relation.

Determining map point matching parameters among the sub-maps according to the first connection relation and the second connection relation, determining a second target error according to the map point matching parameters among the sub-maps and the first optimized pose information of the target sub-map matching pair, optimizing the first optimized pose information of the target sub-map matching pair by taking the second target error smaller than the second preset error and the registration constraint condition reached by the target sub-map matching pair as the optimization condition, obtaining second optimized pose information of the target sub-map matching pair, and determining the second optimized pose information of the sub-maps except the designated sub-map in the designated initial map according to the second optimized pose information of the target sub-map matching pair and the second connection relation, wherein the designated initial map is the initial map to which the sub-map of the target sub-map matching pair belongs, and the designated sub-map is the sub-map in the target sub-map matching pair. The first optimization pose information of the target sub-map matching pair at the initial map joint is locally optimized, and all sub-maps are not optimized, so that the data volume needing to be optimized is greatly reduced, the accuracy of the generated target map is improved, the optimization efficiency is improved, the generation efficiency of the target map is improved, the second optimization pose information of each sub-map can be optimized quickly according to the second optimization pose information and the second connection relation, the optimization of the pose information of other sub-maps can be realized, the optimization efficiency is improved, and the consistency of the subsequently generated target map is ensured.

Referring to fig. 3, fig. 3 is a flowchart illustrating an AR map generating method according to an embodiment of the present disclosure, where the AR map generating method is applied to a reality augmented AR device, and the AR device includes a visual sensor, and specifically includes the following steps.

Step 302, receiving an image collected by a vision sensor;

the visual sensor is an acquisition device, such as a camera, a video recorder, a thermal imaging acquisition device, and the like, which can acquire visual information of a real scene.

The image collected by the visual sensor is received, the image collected by the visual sensor can be directly received by the reality augmented AR device, or the image collected by the visual sensor is sent to the remote server, and then the reality augmented AR device obtains the image through the remote server, which is not limited herein.

The image is a photo and a video collected by the vision sensor.

By receiving the images acquired by the vision sensor, a data base is laid for subsequently constructing a plurality of initial maps.

Step 304, constructing a plurality of initial maps according to images acquired by a visual sensor, and determining a first connection relation among the plurality of initial maps and a second connection relation among sub-maps in each initial map;

the initial map is a virtual map of a partial reality scene in a large reality scene. The sub-map is a virtual map that constitutes a part of a partially real scene.

The first connection relation among the initial maps is information such as position information and image information which are used for representing the corresponding relation among the initial maps and are determined by the first connection relation and are related among a plurality of initial maps. The second connection relationship among the sub-maps in each initial map is a relationship representing correspondence among the sub-maps in each initial map.

Constructing a plurality of initial maps according to images acquired by a visual sensor, which specifically comprises the following steps: and determining a position structure between the objects according to the pixel point positions of the images acquired by the vision sensor and the characteristics of the objects represented by the pixel points, and constructing to obtain a plurality of initial maps. The construction method can be constructed by utilizing a virtual map construction platform, and can also be used for carrying out space mapping on pixel points of the image to obtain an initial map.

The method lays a foundation for subsequent optimization by constructing a plurality of initial maps and determining the first connection relation among the initial maps and the second connection relation of each sub-map in each initial map.

Step 306, optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map;

the initial pose information is a coordinate system expression of the sub-map in a coordinate system obtained by combining a plurality of initial maps.

Specifically, according to the name matching of the sub-maps in the first connection relation and the second connection relation, two corresponding sub-maps are determined, and the optimization of the initial pose information of the two sub-maps is completed until the optimization of the initial pose information of the sub-maps in all the initial maps is completed, so that the first optimized pose information of each sub-map is obtained.

By carrying out global optimization on the initial pose information of the sub-maps of the plurality of initial maps, the accuracy of the subsequently generated AR map is improved, and a foundation is laid for subsequently determining the position structure of each initial map and finding out the target sub-map matching pair.

308, determining a target sub-map matching pair at each initial map joint according to the first optimization pose information;

Step 310, optimizing the first optimized pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimized pose information of each sub-map;

The accuracy of the subsequently generated AR map is improved by locally optimizing the first optimization pose information of the target sub-map matching pair, and the optimization efficiency is improved by locally optimizing only the target sub-map matching pair at the joint of the initial map, so that the generation efficiency of the AR map is improved.

And 312, combining the plurality of initial maps according to the second optimized pose information to generate the AR map.

The specific generation of the AR map is as follows: and determining the position structure of each initial map according to the second optimized pose information, splicing the image information at the corresponding position of each initial map according to the image information of each sub-map, merging the plurality of initial maps, and generating the AR map.

Specifically, according to the second optimized pose information and the image information of each sub-map, combining the multiple initial maps to generate the AR map.

In the embodiment of the specification, an image acquired by a visual sensor is received, a plurality of initial maps are constructed according to the image acquired by the visual sensor, a first connection relation among the plurality of initial maps and a second connection relation among sub-maps in each initial map are determined, initial pose information of each sub-map is optimized according to the first connection relation and the second connection relation, first optimized pose information of each sub-map is obtained, a target sub-map matching pair at the connection position of each initial map is determined according to the first optimized pose information, first optimized pose information of the target sub-map matching pair is optimized according to the first connection relation and the second connection relation, second optimized pose information of each sub-map is obtained, and the plurality of initial maps are combined according to the second optimized pose information to generate an AR map. The method comprises the steps of receiving images acquired by a visual acquisition device, constructing a plurality of initial maps according to the images, ensuring the reduction degree of an AR map generated subsequently, performing global optimization on initial pose information of each sub-map, then determining target sub-map matching pairs at joints of each initial map, and performing local optimization on first optimized pose information of the target sub-map pairs at the joints, so that sub-maps subjected to global optimization and local optimization are obtained, the accuracy of the generated AR map is also ensured, and meanwhile, only the target sub-map matching pairs at the joints are optimized, and local optimization is not required to be performed on all sub-maps, so that the local optimization efficiency is improved, and the efficiency of generating the AR map is further improved.

Optionally, the AR device further comprises a lidar;

correspondingly, before determining the first connection relationship among the multiple initial maps in step 304, the following specific steps are further included:

receiving point cloud information collected by a laser radar;

determining a first connection relationship among a plurality of initial maps, comprising:

The point cloud information is a point cloud picture obtained by collecting the physical shape and the physical position of each object in a real scene by the laser radar, and the sub-map matching pair is a sub-map with a connection relation between the initial maps.

Specifically, point cloud information collected by a laser radar is received, local feature points of the image information of sub-maps in a first initial map and a second initial map are extracted according to the image information and the point cloud information of the sub-maps in the first initial map and the second initial map, the image similarity between the first sub-map and the second sub-map is determined according to the feature similarity of the local feature points, a sub-map matching pair between the first initial map and the second initial map is determined according to the image similarity, the point cloud maps of the sub-map matching pair are subjected to transformation such as rotation, translation, filtering and the like of a coordinate system, the overlapping degree of point cloud coordinates of the point cloud maps of the sub-maps is determined, the point cloud overlapping degree of the sub-map matching pair is obtained, and a first connection relation between the first initial map and the second initial map is determined.

The sub-map matching pair verified through the image similarity and the point cloud overlapping degree is effective, so that the validity of the connection relation of the corresponding initial map is guaranteed, and the accuracy of subsequent optimization and target map generation is improved.

The following further describes the target map generation method with reference to fig. 4A to 4B, taking an application of the target map generation method provided in this specification in a 3D large reality scene as an example. Fig. 4A shows a processing flow chart of a target map generation method applied to a 3D large reality scene according to an embodiment of the present specification, which specifically includes the following steps.

Step 402: acquiring an index image;

the index image is an image of a component of a large real scene, for example, the large real scene is a stadium and the index image is at the entrance of the stadium.

Step 404: searching images to be searched in the map database according to the image characteristics of the index images, and determining a plurality of target images meeting the image characteristic similarity threshold;

in practical application, the image features of the index image are extracted, the images to be retrieved of the sub-maps in the initial maps which are constructed in advance in the map database are retrieved, the image features of the images to be retrieved are obtained, the image feature similarity of the images to be retrieved with the image features of the index image is determined, and a plurality of images to be retrieved which are larger than a preset image feature similarity threshold value are determined as target images.

Step 406: determining a sub-map and an initial map corresponding to the target images according to the plurality of target images;

and determining a sub-map corresponding to the target image in the map database, and determining the corresponding initial map according to the sub-map.

Step 408: determining the image similarity among the sub-maps according to the target images of the sub-maps;

in practical application, the target images of the sub-maps are input into a pre-trained CNN model, and the image similarity among the sub-maps is obtained.

Step 410: determining a sub-map matching pair between the first initial map and the second initial map according to the image similarity;

in practical application, the image similarity is compared with a preset image similarity threshold, and the first initial map and the second initial map which are larger than the image similarity threshold are determined as a sub-map matching pair.

Step 412: determining the point cloud overlapping degree of the sub-map matching pairs according to the point cloud images of the sub-map matching pairs and the sub-map;

in practical application, the sub-map in the first initial map and the point cloud map in the corresponding sub-map in the second initial map are subjected to transformation such as rotation, translation, filtering and the like of a coordinate system, and the overlapping degree (accumulated overlapping degree of each point) of the point cloud coordinates of the point cloud maps of the sub-map is determined, so that the point cloud overlapping degree of the matching pair of the sub-maps is obtained.

Step 414: determining a first connection relation between the first initial map and the second initial map according to the point cloud overlapping degree;

in practical application, the point cloud overlapping degree is compared with a preset point cloud overlapping degree threshold value, and a first connection relation between a first initial map and a second initial map corresponding to a sub-map matching pair larger than the point cloud overlapping degree threshold value is determined.

Step 416: determining the distribution rules of the plurality of initial maps according to the first connection relations among the plurality of initial maps, and rejecting the first connection relations which do not accord with the distribution rules in the first connection relations among the plurality of initial maps;

in practical application, according to the first connection relation among the multiple initial maps, the number relation of the sub-map matching pairs in the multiple initial maps is determined, and the first connection relation that the number of the sub-map matching pairs does not meet the distribution rule is eliminated.

For example, according to the first connection relationship between 3 initial maps, the distribution confidence of the sub-map matching pairs between the initial map a and the initial map B is 0.97, the distribution confidence of the sub-map matching pairs between the initial map a and the initial map D is 0.24, and the distribution confidence of the sub-map matching pairs between the initial map B and the initial map D is 0.17, so that the first connection relationships (map 1, map 4) and (map 2, map 4) between the initial map D and the initial map a and the initial map B are eliminated.

Step 418: acquiring a second connection relation between sub-maps in each initial map;

step 420: constructing a first factor graph according to the initial pose information, the first connection relation and the second connection relation of each sub-map, and optimizing the initial pose information of each sub-map by using a Posegraph optimization method to obtain first optimized pose information of each sub-map;

the first factor graph is graph data generated by determining that the sub-maps are vertexes, and the first connection relation and the second connection relation among the sub-maps are edges.

The Posegraph optimization method is a global optimization method and aims at the problem that errors in acquisition of acquisition equipment cause insufficient consistency between constructed initial maps and constructed sub-maps. The specific implementation of the Posegraph optimization method is to determine the optimal pose information of each sub-map, so that the error between the difference of the pose information and the difference of the point cloud map is minimum.

Step 422: determining a target sub-map matching pair at each initial map joint according to the first optimization pose information;

referring to fig. 4B, fig. 4B illustrates an initial map joint diagram of a target map generation method applied to a 3D large reality scene according to an embodiment of the present disclosure.

As shown in fig. 4B, the initial map1 and the initial map2 are the initial maps optimized by the Posegraph optimization method, the initial map1 internally includes submaps submap1, submap2 and submap3, the initial map2 internally includes submaps submap4, submap5 and submap6, the initial map1 and the initial map2 are connected at the above 6 submaps, and the connection positions of the initial map1 and the initial map2 are corresponding boundaries between the submap1, the submap2, the submap3, the submap4, the submap5 and the submap 6. And obtaining target sub-map matching pairs (submap 1, submap 4), (submap 2, submap 5) and (submap 3, submap 6).

In practical application, the position structure of each initial map is determined according to the first optimization pose information, the connection position between each initial map is obtained according to the position structure of each initial map, the sub-map with the connection relation at the connection position is determined, and the target sub-map matching pair is obtained.

Step 424: constructing a second factor graph according to the first optimization pose information, the first connection relation and the second connection relation of each sub-map, and optimizing the first optimization pose information of the target sub-map matching pair by using a BA optimization method to obtain second optimization pose information of each sub-map;

the second factor graph is graph data generated by determining the sub-maps as vertexes, and the first connection relation and the second connection relation between the sub-maps as edges.

The BA optimization method is a global optimization method, can perform local optimization on the pose information of local sub-maps, and is used for solving the problem of insufficient consistency between the constructed initial maps and the sub-maps caused by errors during acquisition of acquisition equipment. The specific implementation of the BA optimization method is to determine the optimal pose information of each sub-map, so that the error of the difference between the pose information and the difference between the point cloud maps is minimized under the condition that the angle constraint on the acquisition devices is satisfied (i.e., the reflected light and the emitted light of each acquisition device for acquiring the key point are on the same straight line).

Step 426: merging the plurality of initial maps according to the second optimized pose information of each sub-map, the image of each sub-map and the point cloud picture of each sub-map to generate a target map;

in practical application, the positions, structures, angles and the like of a plurality of initial maps are determined according to the second optimized pose information of each sub-map and the point cloud maps of each sub-map, so that a target map frame is obtained, and the target map frame is rendered according to the images of each sub-map, so that the target map is obtained.

Step 428: and displaying the target map on a display interface of the map generation application program.

In the embodiment of the specification, the first connection relation between the initial maps is determined by depending on the matching of the images and the point cloud maps, the advantages of high-efficiency robustness of the matching and high precision of the point cloud maps are integrated, the first connection relation is rapidly determined by adopting the image similarity, and the matching result is verified and optimized by using the point cloud maps, so that the precision of the first connection relation between the initial maps is ensured, and the stability of subsequent optimization is also ensured. And mismatching inspection is carried out on the first connection relation according to the distribution rule of the initial map, the first connection relation which does not meet the distribution rule is eliminated, and the efficiency of determining the first connection relation is improved. On the basis of Posegraph global optimization of each sub-map, determining the joints among a plurality of initial maps according to the fact that the result of the global optimization is prior to obtain target sub-map matching pairs, and performing BA local optimization correspondingly, so that the data volume needing to be processed in the whole optimization process is reduced, and the generation efficiency of the target map is improved in the generation of the target map in a large real scene.

Corresponding to the above method embodiment, the present specification further provides an embodiment of a target map generation apparatus, and fig. 5 shows a schematic structural diagram of a target map generation apparatus according to an embodiment of the present specification. As shown in fig. 5, the apparatus includes:

an obtaining module 502 configured to obtain a plurality of initial maps, a first connection relationship among the plurality of initial maps, and a second connection relationship among sub-maps in each initial map;

a first optimization module 504, configured to optimize the initial pose information of each sub-map according to the first connection relationship and the second connection relationship, to obtain first optimized pose information of each sub-map;

a first determining module 506 configured to determine a target sub-map matching pair at each initial map junction according to the first optimized pose information;

a second optimization module 508 configured to optimize the first optimization pose information of the target sub-map matching pair according to the first connection relationship and the second connection relationship, so as to obtain second optimization pose information of each sub-map;

and the generating module 510 is configured to merge the plurality of initial maps according to the second optimized pose information to generate a target map.

Optionally, the obtaining module 502 may be further configured to:

Optionally, the apparatus further comprises:

the removing module is configured to determine the distribution rules of the multiple initial maps according to the first connection relations among the multiple initial maps, and remove the first connection relations which do not accord with the distribution rules in the first connection relations among the multiple initial maps.

Optionally, the obtaining module 502 is further configured to:

Optionally, the first optimization module 504 is further configured to:

Optionally, the first determining module 506 is further configured to:

calculating the distance between each sub map and the connection part;

Optionally, the first determining module 506 is further configured to:

determining an updated sub-map matching pair according to the image similarity;

determining the point cloud overlapping degree of the updated sub-map matching pair according to the point cloud information of the updated sub-map matching pair;

Optionally, the second optimization module 508 is further configured to:

and determining second optimized pose information of sub-maps except the designated sub-map in the designated initial map according to the second optimized pose information and the second connection relation of the target sub-map matching pair, wherein the designated initial map is the initial map to which the sub-map belongs in the target sub-map matching pair, and the designated sub-map is the sub-map in the target sub-map matching pair.

In the embodiment of the specification, a plurality of initial maps, a first connection relation among the initial maps and a second connection relation among sub-maps in each initial map are obtained; optimizing the initial pose information of each sub-map according to the first connection relation and the second connection relation to obtain first optimized pose information of each sub-map; determining a target sub-map matching pair at each initial map joint according to the first optimization pose information; optimizing the first optimization pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimization pose information of each sub-map; and merging the plurality of initial maps according to the second optimized pose information to generate a target map. After the initial pose information of each sub-map is globally optimized, the target sub-map matching pair at the joint of each initial map is determined, the first optimized pose information of the target sub-map pair at the joint is locally optimized, the sub-maps subjected to global optimization and local optimization are obtained, the accuracy of the generated target map is further ensured, meanwhile, only the target sub-map matching pair at the joint is optimized, and the local optimization of all sub-maps is not needed, so that the local optimization efficiency is improved, and the efficiency of generating the target map is further improved.

The above is a schematic scheme of an object map generating apparatus of the present embodiment. It should be noted that the technical solution of the target map generation apparatus and the technical solution of the target map generation method belong to the same concept, and details that are not described in detail in the technical solution of the target map generation apparatus can be referred to the description of the technical solution of the target map generation method.

Corresponding to the above method embodiment, the present specification further provides an AR map generating apparatus embodiment, and fig. 6 shows a schematic structural diagram of an AR map generating apparatus provided according to an embodiment of the present specification. As shown in FIG. 6, the device comprises

A receiving module 602 configured to receive an image acquired by a vision sensor;

a second determining module 604, configured to construct a plurality of initial maps according to the images acquired by the visual sensor, and determine a first connection relationship between the plurality of initial maps and a second connection relationship between sub-maps in each initial map;

a third optimization module 606 configured to optimize the initial pose information of each sub-map according to the first connection relationship and the second connection relationship to obtain first optimized pose information of each sub-map;

a third determining module 608 configured to determine a target sub-map matching pair at each initial map junction according to the first optimization pose information;

a fourth optimization module 610 configured to optimize the first optimization pose information of the target sub-map matching pair according to the first connection relationship and the second connection relationship, so as to obtain second optimization pose information of each sub-map;

and a second generating module 612 configured to merge the multiple initial maps according to the second optimized pose information to generate an AR map.

Optionally, the AR device further includes a lidar, and the apparatus further includes:

the point cloud information receiving module is configured to receive point cloud information collected by the laser radar;

correspondingly, the second determining module 604 is further configured to:

In the embodiment of the specification, an image acquired by a visual sensor is received, a plurality of initial maps are constructed according to the image acquired by the visual sensor, a first connection relation among the plurality of initial maps and a second connection relation among sub-maps in each initial map are determined, initial pose information of each sub-map is optimized according to the first connection relation and the second connection relation, first optimized pose information of each sub-map is obtained, a target sub-map matching pair at the connection position of each initial map is determined according to the first optimized pose information, first optimized pose information of the target sub-map matching pair is optimized according to the first connection relation and the second connection relation, second optimized pose information of each sub-map is obtained, and the plurality of initial maps are combined according to the second optimized pose information to generate an AR map. By receiving the images acquired by the visual acquisition equipment, constructing a plurality of initial maps according to the images, ensuring the reduction degree of the subsequent generation of the AR maps, globally optimizing the initial pose information of each sub-map, then determining the target sub-map matching pairs at the joints of each initial map, and locally optimizing the first optimized pose information of the target sub-map pairs at the joints, obtaining sub-maps which are globally optimized and locally optimized, ensuring the accuracy of the generated AR maps, simultaneously optimizing only the target sub-map matching pairs at the joints, and not performing local optimization on all the sub-maps, improving the local optimization efficiency, and further improving the efficiency of generating the AR maps.

The above is a schematic scheme of an AR map generation apparatus of the present embodiment. It should be noted that the technical solution of the AR map generating apparatus and the technical solution of the AR map generating method belong to the same concept, and details of the technical solution of the AR map generating apparatus, which are not described in detail, can be referred to the description of the technical solution of the AR map generating method.

Fig. 7 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure. The components of the electronic device 700 include, but are not limited to, a memory 710 and a processor 720. Processor 720 is coupled to memory 710 via bus 730, and database 750 is used to store data.

The electronic device 700 also includes an access device 740, the access device 740 enabling the electronic device 700 to communicate via one or more networks 760. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The Access device 740 may include one or more of any type of Network Interface (e.g., a Network Interface Controller (NIC)) whether wired or Wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) Wireless Interface, a worldwide Interoperability for Microwave Access (Wi-MAX) Interface, an ethernet Interface, a Universal Serial Bus (USB) Interface, a cellular Network Interface, a bluetooth Interface, a Near Field Communication (NFC) Interface, and so forth.

In one embodiment of the present description, the above-mentioned components of the electronic device 700 and other components not shown in fig. 7 may also be connected to each other, for example, through a bus. It should be understood that the block diagram of the electronic device shown in fig. 7 is for illustration purposes only and is not intended to limit the scope of the present disclosure. Those skilled in the art may add or replace other components as desired.

The electronic device 700 may be any type of stationary or mobile electronic device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), a mobile phone (e.g., smartphone), a wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. The electronic device 700 may also be a mobile or stationary server.

Wherein the processor 720 is configured to execute computer-executable instructions that, when executed by the processor, implement the steps of the above-described target map generation method and AR map generation method.

The above is a schematic scheme of an electronic device of the present embodiment. It should be noted that the technical solution of the electronic device and the technical solutions of the target map generation method and the AR map generation method belong to the same concept, and details of the technical solutions of the electronic device, which are not described in detail, can be referred to the descriptions of the technical solutions of the target map generation method and the AR map generation method.

An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described target map generation method or AR map generation method.

The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solutions of the target map generation method and the AR map generation method, and details of the technical solution of the storage medium, which are not described in detail, can be referred to the descriptions of the technical solutions of the target map generation method and the AR map generation method.

An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer program causes the computer to execute the steps of the above target map generation method or AR map generation method.

The above is an illustrative scheme of a computer program of the present embodiment. It should be noted that the technical solution of the computer program is the same as the technical solution of the target map generation method and the AR map generation method, and details of the technical solution of the computer program, which are not described in detail, can be referred to the description of the technical solution of the target map generation method and the AR map generation method.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, read-Only Memory (ROM), random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like.

It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts, but those skilled in the art should understand that the present embodiment is not limited by the described acts, because some steps may be performed in other sequences or simultaneously according to the present embodiment. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for an embodiment of the specification.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to the related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the embodiments. The specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method of target map generation, comprising:

determining a target sub-map matching pair at each initial map joint according to the first optimized pose information;

optimizing the first optimized pose information of the target sub-map matching pair according to the first connection relation and the second connection relation to obtain second optimized pose information of each sub-map;

and merging the plurality of initial maps according to the second optimized pose information to generate a target map.

2. The method of claim 1, the obtaining a first connection relationship among the plurality of initial maps, comprising:

determining image similarity between a first sub-map and a second sub-map according to image information of the sub-maps in the first initial map and the second initial map, wherein the first sub-map is the sub-map in the first initial map, and the second sub-map is the sub-map in the second initial map;

3. The method according to claim 1 or 2, before the optimizing the initial pose information of each sub-map according to the first connection relationship and the second connection relationship to obtain the first optimized pose information of each sub-map, further comprising:

determining a distribution rule of the plurality of initial maps according to a first connection relation among the plurality of initial maps;

and eliminating the first connection relations which do not accord with the distribution rule in the first connection relations among the plurality of initial maps.

4. The method of claim 1, wherein the obtaining of the second connection relationship between the sub-maps in each initial map comprises:

acquiring image information of each sub-map in a third initial map, wherein the third initial map is any one of the plurality of initial maps;

determining image similarity between a third sub-map and a fourth sub-map according to image information of the third sub-map and the fourth sub-map, wherein the third sub-map and the fourth sub-map are any two of the third initial map;

5. The method according to any one of claims 1, 2, and 4, wherein the optimizing the initial pose information of each sub-map according to the first connection relationship and the second connection relationship to obtain first optimized pose information of each sub-map includes:

and optimizing the initial pose information of each sub-map by taking the first target error smaller than a first preset error as an optimization condition to obtain first optimized pose information of each sub-map.

6. The method of claim 1, wherein determining a matching pair of target sub-maps at each initial map junction according to the first optimized pose information comprises:

determining the joints of the plurality of initial maps according to the first optimization pose information;

calculating the distance between each sub map and the connection part;

7. The method of claim 6, wherein determining a matching pair of target sub-maps at each initial map connection based on the distance of each sub-map from the connection comprises:

determining an updated sub-map matching pair according to the image similarity;

determining the point cloud overlapping degree of the updated sub-map matching pair according to the point cloud information of each sub-map of the updated sub-map matching pair;

and determining the matching pairs of the target sub-maps at the joints of the initial maps according to the point cloud overlapping degree.

8. The method according to any one of claims 1, 6, and 7, wherein the optimizing the first optimization pose information of the target sub-map matching pair according to the first connection relationship and the second connection relationship to obtain the second optimization pose information of each sub-map includes:

determining a second target error according to map point matching parameters among the sub-maps and the first optimization pose information of the target sub-map matching pair;

optimizing the first optimized pose information of the target sub-map matching pair by taking the second target error smaller than a second preset error and the target sub-map matching pair reaching the registration constraint condition as an optimization condition to obtain second optimized pose information of the target sub-map matching pair;

and determining second optimized pose information of sub-maps except for a designated sub-map in a designated initial map according to the second optimized pose information of the target sub-map matching pair and the second connection relation, wherein the designated initial map is the initial map to which the sub-map of the target sub-map matching pair belongs, and the designated sub-map is the sub-map in the target sub-map matching pair.

9. An AR map generation method is applied to reality augmented AR equipment, the AR equipment comprises a visual sensor, and the method comprises the following steps:

receiving an image acquired by the vision sensor;

constructing a plurality of initial maps according to the images acquired by the vision sensor, and determining a first connection relation among the plurality of initial maps and a second connection relation among sub-maps in each initial map;

and combining the plurality of initial maps according to the second optimized pose information to generate an AR map.

10. The method of claim 9, the AR device further comprising a lidar;

before the determining the first connection relationship among the plurality of initial maps, further comprising:

receiving point cloud information collected by the laser radar;

the determining a first connection relationship among the plurality of initial maps comprises:

11. An object map generation apparatus comprising:

a first determining module configured to determine a target sub-map matching pair at each initial map junction according to the first optimized pose information;

and the generating module is configured to merge the plurality of initial maps according to the second optimized pose information to generate a target map.

12. An AR map generation apparatus applied to a reality augmented AR device, the AR device including a visual sensor, the apparatus comprising:

a receiving module configured to receive an image acquired by the vision sensor;

the second determination module is configured to construct a plurality of initial maps according to the images acquired by the vision sensor, and determine a first connection relation among the plurality of initial maps and a second connection relation among sub-maps in each initial map;

a third optimization module configured to optimize the initial pose information of each sub-map according to the first connection relationship and the second connection relationship to obtain first optimized pose information of each sub-map;

a third determining module configured to determine a target sub-map matching pair at each initial map junction according to the first optimized pose information;

a fourth optimization module configured to optimize the first optimization pose information of the target sub-map matching pair according to the first connection relationship and the second connection relationship to obtain second optimization pose information of each sub-map;

and the second generation module is configured to combine the plurality of initial maps according to the second optimization pose information to generate an AR map.

13. An electronic device, comprising:

a memory and a processor;

the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, which when executed by the processor, implement the steps of the object map generation method of any one of claims 1 to 8 or the AR map generation method of any one of claims 9 to 10.

14. A computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the steps of the object map generation method of any one of claims 1 to 8 or the AR map generation method of any one of claims 9 to 10.