WO2022222532A1

WO2022222532A1 - Method and apparatus for establishing three-dimensional map, and electronic device and computer-readable storage medium

Info

Publication number: WO2022222532A1
Application number: PCT/CN2021/141492
Authority: WO
Inventors: 董文锋; 马世奎; 张永亮; 时庆勇
Original assignee: 达闼机器人股份有限公司
Priority date: 2021-04-20
Filing date: 2021-12-27
Publication date: 2022-10-27
Also published as: CN113269874A

Abstract

A method and apparatus for establishing a three-dimensional map. The method comprises: determining sensing data corresponding to a target space (101); determining an initial three-dimensional map by using a three-dimensional engine and according to the sensing data (102); and processing the initial three-dimensional map to determine a target three-dimensional map corresponding to the target space (103). By means of the method and apparatus for establishing a three-dimensional map, a three-dimensional map for a target space is determined, thereby reflecting the shape and the structure of the target space, and a three-dimensional relationship of objects in the target space. In this way, an intelligent device can achieve richer function implementations by using the three-dimensional map, thereby better meeting functional requirements of the intelligent device.

Description

[Method, device, electronic device, and computer-readable storage medium for establishing a three-dimensional map]

cross reference

This application claims the priority of the Chinese patent application filed on April 21, 2021 with the application number of 202110424164.7 and the title of the invention is "A method and device for establishing a three-dimensional map", the entire contents of which are incorporated herein by reference Applying.

technical field

The present disclosure relates to the technical field of three-dimensional maps, and in particular, to a method, apparatus, electronic device, and computer-readable storage medium for establishing a three-dimensional map.

Background technique

In the current field of robotics, many functions need to rely on the virtual map function of the robot. That is, the robot perceives the physical environment it is in through sensors, and builds a virtual map based on the spatial structure of the physical environment. Subsequent functions such as navigation and route planning can be completed based on the virtual map.

In the prior art, sensors such as lidar and infrared cameras are usually used to realize the perception of the physical environment, obtain a corresponding point cloud image, and construct a virtual map according to the point cloud image. However, the virtual map established according to the above method is a flat map, and generally does not contain the three-dimensional relationship of objects in space and the semantic information of objects.

SUMMARY OF THE INVENTION

The present disclosure provides a method, apparatus, electronic device, and computer-readable storage medium for establishing a three-dimensional map, so as to at least solve the above technical problems existing in the prior art.

In a first aspect, the present disclosure provides a method for establishing a three-dimensional map, including:

Determine the sensor data corresponding to the target space;

Using a three-dimensional engine to determine an initial three-dimensional map according to the sensing data;

The initial three-dimensional map is processed to determine a target three-dimensional map corresponding to the target space.

Further, the determining of the sensing data corresponding to the target space includes:

Collect the sensing data by using the sensor carried by the smart device;

The sensory data includes point cloud images, infrared images, depth data, and RGB images.

Further, the determining an initial three-dimensional map according to the sensing data by using a three-dimensional engine includes:

Using a map generation algorithm in the three-dimensional engine, the sensor data is operated to determine the initial three-dimensional map.

Further, the determining an initial three-dimensional map according to the sensing data using a three-dimensional engine further includes:

The physical spatial characteristics of the target space, as well as the semantic information and spatial relationships of various items are restored through the initial three-dimensional map.

Further, the processing of the initial three-dimensional map to determine the target three-dimensional map corresponding to the target space includes:

Using a preset image processing model, the initial three-dimensional map is processed according to the spatial characteristics of the target space to determine a target three-dimensional map corresponding to the target space.

Further, the target three-dimensional map reflects the shape and structure of the target space, the three-dimensional relationship of objects in the target space, and/or the semantic information of the objects.

Further, the method also includes:

Using the sensing data, a plane semantic map corresponding to the target space is determined.

Further, the method also includes:

Using the target three-dimensional map and the plane semantic map, determine the action strategy of the smart device; and make the smart device execute the action strategy.

Further, using the three-dimensional map and the plane semantic map to determine the action strategy of the smart device includes:

Combined with the three-dimensional map and the plane semantic map, determine the shape and structure of the target space, the three-dimensional relationship and semantic information of objects in the target space;

Using the shape and structure of the target space, the three-dimensional relationship of objects in the target space and semantic information, the action strategy of the smart device is determined.

Further, the action strategy of the smart device includes controlling the smart device to grab a specific object in the target space, and the specific process includes:

Determine the navigation position of the specific object by using the semantic information of the object in the target space;

Controlling the smart device to formulate a navigation route according to the navigation position;

moving the smart device to the navigation position where the specific object is located according to the navigation route;

determining the spatial location of the particular object;

Based on the spatial position of the specific object, the smart device is controlled to perform a grasping operation on the specific object.

Further, the spatial position is different from the navigation position, and the spatial position includes the height, depth, and distance to the smart device of the specific object, as well as the shape and spatial characteristic information of the specific object, and the spatial position is based on the three-dimensional map. The three-dimensional relationship of the specific object is determined.

In a second aspect, the present disclosure provides a device for establishing a three-dimensional map, including:

The sensor data determination module is used to determine the sensor data corresponding to the target space;

an initial map determination module for determining an initial three-dimensional map according to the sensing data by using a three-dimensional engine;

A target map determination module, configured to process the initial three-dimensional map to determine a target three-dimensional map corresponding to the target space.

Further, the sensor data determination module includes:

an acquisition unit, configured to use a sensor mounted on the smart device to collect sensing data, where the sensing data includes: point cloud image, infrared image, depth data and RGB image;

The communication unit is used for uploading the sensing data to the server.

Further, the initial map determination module includes:

a 3D engine unit, used to determine the 3D engine and a map generation algorithm in the 3D engine;

An initial map determining unit, configured to use a map generation algorithm in the three-dimensional engine to perform operations on the sensing data to determine the initial three-dimensional map.

Further, the initial map determination module is also used for:

Further, the target map determination module includes:

The first processing unit is configured to process the initial three-dimensional map according to the spatial characteristics of the target space by using a preset image processing model to determine the target three-dimensional map corresponding to the target space.

The second processing unit is configured to provide a manual processing interface to accept manual operations to determine the target three-dimensional map corresponding to the target space.

Further, the device also includes:

The execution module is used for determining the action strategy of the smart device by using the three-dimensional map of the target; and making the smart device execute the action strategy.

Further, the device also includes:

The plane map determination module is used for determining the plane semantic map corresponding to the target space by using the sensing data.

In a third aspect, the present disclosure provides a computer-readable storage medium, where a computer program is stored in the storage medium, and the computer program is used to execute the method for establishing a three-dimensional map described in the present disclosure.

In a fourth aspect, the present disclosure provides an electronic device, comprising:

processor;

a memory for storing the processor-executable instructions;

The processor is configured to read the executable instructions from the memory, and execute the instructions to implement the method for establishing a three-dimensional map of the present disclosure.

The present disclosure provides a method and device for establishing a three-dimensional map, the method includes: determining sensing data corresponding to a target space; determining an initial three-dimensional map according to the sensing data using a three-dimensional engine; processing the initial three-dimensional map , to determine the target three-dimensional map corresponding to the target space. Through the method for establishing a three-dimensional map of the present disclosure, a three-dimensional map for the target space is determined, thereby reflecting the shape structure of the target space and the three-dimensional relationship of objects in the target space; enabling smart devices to use the three-dimensional map to complete more abundant functions It can meet the functional requirements of smart devices.

Description of drawings

FIG. 1 is a schematic flowchart of a method for establishing a three-dimensional map according to an embodiment of the present disclosure;

FIG. 2 is a schematic flowchart of another method for establishing a three-dimensional map according to an embodiment of the present disclosure;

FIG. 3 is a schematic structural diagram of an apparatus for establishing a three-dimensional map according to an embodiment of the present disclosure.

FIG. 4 is a schematic diagram of a three-dimensional map effect provided by an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, features and advantages of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. The embodiments are only some, but not all, embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present disclosure.

Application overview

The robot usually realizes the perception of the physical environment through sensors such as lidar and infrared cameras, obtains the corresponding point cloud map, and realizes the construction of a virtual map according to the point cloud map. However, the virtual map established according to the above method is a flat map, and generally does not contain the three-dimensional relationship of objects in space and the semantic information of objects.

In terms of functional realization, there are many limitations in the flat map. For example, when the three-dimensional relationship and semantic information of objects cannot be grasped, the basic path navigation function can only be realized based on the plane map, and the robot cannot be further controlled to complete specific operations for specific objects. For example, more complex operations such as grasping objects and moving objects are difficult to achieve.

It can be seen that the function implementation based on the plane map is still relatively simple and single. In many scenarios, it is difficult to meet the more diverse functional requirements of robots.

Exemplary method

Therefore, the embodiments of the present disclosure will provide a method for establishing a three-dimensional map, so as to at least solve the above technical problems existing in the prior art. As shown in Figure 1, the method in this embodiment includes the following steps:

Step 101: Determine sensor data corresponding to the target space.

In some cases, the execution body of the method may be a smart device; the smart device may mainly be a robot, or may be other electronic devices with self-propelled functions. In other cases, the execution body of the method may also be a cloud or a server that has a communication connection with the above-mentioned smart device. Usually, when the local computing power of the device is limited, the cloud or server is used as the execution body.

The physical space where the smart device is located can be the target space. In this embodiment, the purpose is to establish a three-dimensional map corresponding to the target space. Sensing data is the data obtained by detecting the target space, which can reflect the shape and structure of the target space, the three-dimensional relationship of objects and other information. Sensing data will be collected using sensors mounted on smart devices. If the subsequent computing process is performed by the smart device in the cloud or server, the collected sensor data needs to be uploaded to the cloud or server.

In this embodiment, the sensor may include an infrared camera, a depth camera, an RGB camera, a lidar, an inertial sensor, and the like. The corresponding sensor data may include point cloud images, infrared images, depth data, RGB images, and inertial sensor data.

Step 102 , using a three-dimensional engine to determine an initial three-dimensional map according to the sensor data.

After the sensing data is determined, a three-dimensional map corresponding to the target space can be established according to the sensing data, so as to restore the physical space characteristics in the target space, as well as the semantic information and spatial relationship of various items through the three-dimensional map.

In this embodiment, the three-dimensional engine will be used to complete the preliminary construction of the three-dimensional map. A 3D engine is a common 3D model development tool in this field. Common 3D engines such as Unreal and Unity can be applied in this embodiment. Specifically, a map generation algorithm in a three-dimensional engine can be used to perform operations on the sensor data to determine an initial three-dimensional map. That is to use the specific function of the 3D engine to complete the processing of the sensor data, and directly obtain a relatively primitive 3D map, that is, the initial 3D map. For example, the initial three-dimensional map can be determined by an MMO generation algorithm (Massively Multiplayer Online Game generation algorithm, that is, a massively multiplayer online game generation algorithm) in the three-dimensional engine.

Step 103: Process the initial three-dimensional map to determine the target three-dimensional map corresponding to the target space.

Usually, since the initial 3D map is obtained directly by the 3D engine, there may be a certain error relative to the target space. Therefore, it is necessary to further adjust and process the initial 3D map according to the actual situation of the target space, so as to correct errors and obtain a more accurate 3D map of the target. In the target three-dimensional map, the shape and structure of the target space can be reflected, and the three-dimensional relationship of objects in the target space can be reflected; in some cases, the semantic information of the objects can also be reflected.

The above-mentioned adjustment and processing can be processed according to the spatial characteristics of the target space by using a preset image processing model, so as to automatically determine the target three-dimensional map corresponding to the target space. Specific processing such as merging, optimization, and simplification can also be performed through a specific processing algorithm. Or it can also be processed manually. This embodiment does not limit this, and any processing manner that can achieve the same or similar effects can be incorporated into the overall technical solution of this embodiment.

It can be seen from the above technical solutions that the beneficial effects of this embodiment are: a three-dimensional map for the target space is determined, thereby reflecting the shape and structure of the target space and the three-dimensional relationship of objects in the target space; enabling smart devices to use the three-dimensional map to Complete the realization of richer functions and better meet the functional requirements of smart devices.

Figure 1 shows only a basic embodiment of the method described in the present disclosure, and other preferred embodiments of the method can also be obtained by performing certain optimizations and expansions on the basis.

As shown in FIG. 2 , it is another specific embodiment of the method for establishing a three-dimensional map according to the present disclosure. This embodiment is further expanded on the basis of the foregoing embodiment. The method specifically includes the following steps:

Step 201: Determine sensor data corresponding to the target space.

Step 202 , using a three-dimensional engine to determine an initial three-dimensional map according to the sensor data.

Step 203: Process the initial three-dimensional map to determine the target three-dimensional map corresponding to the target space.

The contents in the foregoing steps 201 to 203 are consistent with the foregoing embodiments, and are not repeated here.

Step 204 , using the sensor data to determine a plane semantic map corresponding to the target space.

In this embodiment, in addition to determining the three-dimensional map of the target, a further planar semantic map of the target space will be determined. The plane semantic map can be determined according to the point cloud image in the sensor data, and the process is similar to the process of determining the plane map in the prior art. However, the difference is that the semantic analysis of the point cloud image is additionally introduced in this embodiment, so that the semantic information of the items in the target space is fused into the plane map, that is, the plane semantic map is obtained.

Step 205: Determine the action strategy of the smart device by using the three-dimensional map and the plane semantic map; and make the smart device execute the action strategy.

In this embodiment, by combining the three-dimensional map and the plane semantic map, the shape and structure of the target space, the three-dimensional relationship and semantic information of objects in the target space can be determined. In this way, more diverse functions can be realized. That is, to determine the action strategy and make the smart device execute the action strategy to achieve certain functions.

For example, in this embodiment, the intelligent device may specifically be a robot. The determined action strategy can be to control the robot to grab a specific object A in the target space. Using the semantic information of objects in the target space, the navigation position of object A can be determined first. Most of the conventional navigation algorithms are designed based on the plane map, so the semantic plane map can be used to realize the navigation, and the robot can be controlled to move according to the navigation route to the navigation position where the object A is located. Of course, in other cases, navigation can also be implemented based on a three-dimensional map-based navigation algorithm. After reaching the navigation position where object A is located, the spatial position of object A can be determined. This spatial position is different from the navigation position, which is simply the plane position of the object. The spatial location will also include more abundant information such as the height, depth, distance to the robot, and the shape and spatial characteristics of the object. The spatial position needs to be determined according to the three-dimensional map, and specifically can be determined according to the three-dimensional relationship of the objects in the three-dimensional map. Based on the spatial position of the object A, the robot can be controlled to grasp the object A.

So far, based on the three-dimensional map and the plane semantic map, in this embodiment, more abundant and diverse functions are realized.

Exemplary device

As shown in FIG. 3 , it is a specific embodiment of the apparatus for establishing a three-dimensional map according to the present disclosure. The apparatus of this embodiment is a physical apparatus for executing the methods described in FIGS. 1 to 2 . The technical solutions thereof are essentially the same as those of the above-mentioned embodiments, and the corresponding descriptions in the above-mentioned embodiments are also applicable to this embodiment. The device in this embodiment includes:

The sensor data determination module 301 is used for determining sensor data corresponding to the target space.

The initial map determination module 302 is configured to use a three-dimensional engine to determine an initial three-dimensional map according to the sensor data.

The target map determination module 303 is configured to process the initial three-dimensional map to determine the target three-dimensional map corresponding to the target space.

In addition, on the basis of the embodiment shown in FIG. 3, preferably, it also includes:

The sensory data determination module 301 includes:

The collection unit 311 is configured to collect sensing data by using a sensor mounted on the smart device; the sensing data includes point cloud images, infrared images, depth data and RGB images.

The communication unit is used to upload the sensor data to the server.

The initial map determination module 302 includes:

The 3D engine unit 321 is used to determine the 3D engine and the map generation algorithm in the 3D engine;

The initial map determination unit 322 is configured to use the map generation algorithm in the three-dimensional engine to perform operations on the sensor data to determine the initial three-dimensional map.

The target map determination module 303 includes:

The first processing unit 331 is configured to use a preset image processing model to process the initial three-dimensional map according to the spatial characteristics of the target space, so as to determine the target three-dimensional map corresponding to the target space.

The second processing unit 332 is configured to provide a manual processing interface to accept manual operations to determine a three-dimensional target map corresponding to the target space.

Also includes:

The execution module 304 is configured to use the three-dimensional map to determine the action strategy of the smart device; and make the smart device execute the action strategy.

Also includes:

The plane map determination module 305 is configured to determine the plane semantic map corresponding to the target space by using the sensor data.

FIG. 4 shows a schematic diagram of a 3D map effect provided by an embodiment of the present disclosure. A 3D reconstruction algorithm (including an MMO game engine generation algorithm) is used to generate a 3D map, and a map post-processing algorithm is used to merge, optimize, and simplify the map to obtain a fusion including A comprehensive digital map of multiple data formats, which can be loaded and utilized for the cloud brain. The display effect is shown in Figure 4. The 3D mapping method based on MMO technology not only generates point cloud files and 2D grid maps used by robot algorithms, but also generates a real 3D environment by combining UE4 engine and 3D modeling tools. It is used for intelligent training and exercise of robots. In the game physics engine, physical simulation can be realized on the basis of the map, and people can see the 3D map more intuitively and clearly.

Referring to FIG. 5 below, it shows a schematic structural diagram of an electronic device 500 corresponding to a robot suitable for implementing another embodiment of the present disclosure. Terminal devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (eg, mobile terminals such as in-vehicle navigation terminals) and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 5 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 5 , an electronic device 500 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 501 that may be loaded into random access according to a program stored in a read only memory (ROM) 502 or from a storage device 508 Various appropriate actions and processes are executed by the programs in the memory (RAM) 503 . In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a communication line 504. An input/output (I/O) interface 505 is also connected to the communication line 504 .

Typically, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 507 such as a computer; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509 . Communication means 509 may allow electronic device 500 to communicate wirelessly or by wire with other devices to exchange data. Although FIG. 4 shows electronic device 500 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 509, or from the storage device 508, or from the ROM 502. When the computer program is executed by the processing apparatus 501, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the client and server can use any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) to communicate, and can communicate with digital data in any form or medium Communication (eg, a communication network) interconnects. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.

Exemplary computer program product and computer readable storage medium

In addition to the methods and apparatus described above, embodiments of the present disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the "exemplary method" described above in this specification Sections describe steps in methods according to various embodiments of the present disclosure.

The computer program product may write program code for performing operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present disclosure may also be computer-readable storage media having computer program instructions stored thereon that, when executed by a processor, cause the processor to perform the above-described "Example Method" section of this specification Steps in methods according to various embodiments of the present disclosure are described in .

The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses or devices, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

The basic principles of the present disclosure have been described above with reference to specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples rather than limitations, and these advantages, advantages, effects, etc. should not be considered to be A must-have for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of example and easy understanding, but not for limitation, and the above details do not limit the present disclosure to be implemented by the above specific details.

The block diagrams of devices, apparatuses, apparatuses, and systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, or configurations must be in the manner shown in the block diagrams. As those skilled in the art will appreciate, these means, apparatuses, apparatuses, systems may be connected, arranged, and configured in any manner. Words such as "including", "including", "having" and the like are open-ended words meaning "including but not limited to" and are used interchangeably therewith. As used herein, the words "or" and "and" refer to and are used interchangeably with the word "and/or" unless the context clearly dictates otherwise. As used herein, the word "such as" refers to and is used interchangeably with the phrase "such as but not limited to".

It should also be noted that, in the apparatus, device and method of the present disclosure, each component or each step may be decomposed and/or recombined. These disaggregations and/or recombinations should be considered equivalents of the present disclosure.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

A method for establishing a three-dimensional map, comprising:

Determine the sensor data corresponding to the target space;

Using a three-dimensional engine to determine an initial three-dimensional map according to the sensing data;

The initial three-dimensional map is processed to determine a target three-dimensional map corresponding to the target space.
The method according to claim 1, wherein the determining the sensing data corresponding to the target space comprises:

Collect the sensing data by using the sensor carried by the smart device;

The sensory data includes point cloud images, infrared images, depth data, and RGB images.
The method according to claim 1, wherein the determining an initial three-dimensional map according to the sensing data using a three-dimensional engine comprises:

Using a map generation algorithm in the three-dimensional engine, the sensor data is operated to determine the initial three-dimensional map.
The method according to claim 3, wherein the determining an initial three-dimensional map according to the sensing data using a three-dimensional engine further comprises:

The physical spatial characteristics of the target space, as well as the semantic information and spatial relationships of various items are restored through the initial three-dimensional map.
The method according to claim 1, wherein the processing the initial three-dimensional map to determine the target three-dimensional map corresponding to the target space comprises:

Using a preset image processing model, the initial three-dimensional map is processed according to the spatial characteristics of the target space to determine a target three-dimensional map corresponding to the target space.
The method according to claim 5, wherein the target three-dimensional map reflects the shape and structure of the target space, the three-dimensional relationship of objects in the target space and/or the semantic information of the objects.
The method according to any one of claims 1 to 6, further comprising:

Using the sensing data, a plane semantic map corresponding to the target space is determined.
The method of claim 7, further comprising:

Using the target three-dimensional map and the plane semantic map, determine the action strategy of the smart device; and make the smart device execute the action strategy.
The method according to claim 8, wherein the determining the action strategy of the smart device by using the three-dimensional map and the plane semantic map comprises:

Combined with the three-dimensional map and the plane semantic map, determine the shape and structure of the target space, the three-dimensional relationship and semantic information of objects in the target space;

Using the shape and structure of the target space, the three-dimensional relationship and semantic information of objects in the target space, the action strategy of the smart device is determined.
The method according to claim 9, wherein the action strategy of the smart device includes controlling the smart device to grab a specific object in the target space, and the specific process includes:

Determine the navigation position of the specific object by using the semantic information of the object in the target space;

Controlling the smart device to formulate a navigation route according to the navigation position;

moving the smart device to the navigation position where the specific object is located according to the navigation route;

determining the spatial location of the particular object;

Based on the spatial position of the specific object, the smart device is controlled to perform a grasping operation on the specific object.
The method according to claim 10, wherein the spatial position is different from the navigation position, and the spatial position includes the height, depth, and distance from the smart device of the specific object, as well as the shape and spatial characteristic information of the specific object , the spatial position is determined according to the three-dimensional relationship of the specific object in the three-dimensional map.
A device for establishing a three-dimensional map, comprising:

The sensor data determination module is used to determine the sensor data corresponding to the target space;

an initial map determination module for determining an initial three-dimensional map according to the sensing data by using a three-dimensional engine;

A target map determination module, configured to process the initial three-dimensional map to determine a target three-dimensional map corresponding to the target space.
The device according to claim 12, wherein the sensor data determination module comprises:

an acquisition unit, configured to use a sensor mounted on the smart device to collect sensing data, where the sensing data includes: point cloud image, infrared image, depth data and RGB image;

The communication unit is used for uploading the sensing data to the server.
The device according to claim 12, wherein the initial map determination module comprises:

a 3D engine unit, used to determine the 3D engine and a map generation algorithm in the 3D engine;

An initial map determination unit, configured to use a map generation algorithm in the three-dimensional engine to perform operations on the sensor data to determine the initial three-dimensional map.
The device according to claim 12, wherein the initial map determination module is further configured to:

The physical spatial characteristics of the target space, as well as the semantic information and spatial relationship of various items are restored through the initial three-dimensional map.
The device according to claim 12, wherein the target map determination module comprises:

The first processing unit is configured to process the initial three-dimensional map according to the spatial characteristics of the target space by using a preset image processing model to determine the target three-dimensional map corresponding to the target space.

The second processing unit is configured to provide a manual processing interface to accept manual operations to determine the target three-dimensional map corresponding to the target space.
The device according to claim 12, wherein the device further comprises:

The execution module is used for determining the action strategy of the smart device by using the three-dimensional map of the target; and making the smart device execute the action strategy.
The device according to claim 12, wherein the device further comprises:

The plane map determination module is used for determining the plane semantic map corresponding to the target space by using the sensing data.
An electronic device comprising:

processor;

a memory for storing the processor-executable instructions;

The processor is configured to read the executable instructions from the memory, and execute the instructions to implement the method for establishing a three-dimensional map according to any one of claims 1-11.
A computer-readable storage medium storing a computer program, the computer program being used to execute the method for establishing a three-dimensional map according to any one of claims 1-11.