WO2021189784A1

WO2021189784A1 - Scenario reconstruction method, system and apparatus, and sweeping robot

Info

Publication number: WO2021189784A1
Application number: PCT/CN2020/115921
Authority: WO
Inventors: 张涵; 于元隆; 梁振振; 黄志勇
Original assignee: 南京科沃斯机器人技术有限公司
Priority date: 2020-03-23
Filing date: 2020-09-17
Publication date: 2021-09-30
Also published as: CN113436309A

Abstract

Provided are a scenario reconstruction method, system and apparatus, and a sweeping robot. The method comprises: acquiring the current environmental state (S1); and selecting a first pose or a second pose as a pose which is suitable for the current moment according to the current environmental state, and establishing a scenario model according to the selected first pose and/or second pose (S3), wherein the first pose is a pose corresponding to image data at the current moment, and the second pose is a pose corresponding to inertial measurement data at the current moment. By means of the method, the accuracy and robustness of scenario reconstruction can be improved.

Description

Scene reconstruction method, system, device and sweeping robot

cross reference

This application refers to the Chinese Patent Application No. 202010207310.6 entitled "A Scene Reconstruction Method, System, Apparatus, and Sweeping Robot" filed on March 23, 2020, which is fully incorporated into this application by reference.

Technical field

This application relates to the field of data processing technology, and in particular to a scene reconstruction method, system, device and sweeping robot.

Background technique

In the current 3D scene reconstruction technology, a camera can be used to obtain images of the scene, and the matching image features can be obtained by means of extracting image features, using image features for feature matching, and so on. Then, sparse reconstruction can be performed based on the matched image features to obtain the camera pose of each image. Finally, dense reconstruction can be performed based on the camera pose to obtain a dense point cloud, which can then be used to reconstruct a three-dimensional scene.

In the current 3D reconstruction process, since the indoor scene is relatively complicated, when the indoor scene is reconstructed based on the image obtained by the camera, it may lead to a situation of low accuracy and robustness.

Summary of the invention

The purpose of this application is to provide a scene reconstruction method, system, device and sweeping robot, which can improve the accuracy and robustness of scene reconstruction.

In order to achieve the above objective, one aspect of the present application provides a scene reconstruction method, the method includes: acquiring the current environment state; and selecting the first pose or the second pose as the pose suitable for the current moment according to the current environment state , And create a scene model based on the selected first pose and/or second pose; where the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment Posture.

In order to achieve the above-mentioned object, another aspect of the present application also provides a scene reconstruction system. The system includes: an environment state acquisition unit for acquiring the current environment state; One pose or second pose is used as the pose suitable for the current moment, and the scene model is established according to the selected first pose and/or second pose; where the first pose corresponds to the image data at the current moment The pose, the second pose is the pose corresponding to the inertial measurement data at the current moment.

To achieve the foregoing objective, another aspect of the present application provides a scene reconstruction device. The scene reconstruction device includes a memory and a processor. The memory is used to store a computer program. When the computer program is executed by the processor, It is used to achieve the following functions: obtain the current environmental state; select the first pose or the second pose as the pose suitable for the current moment according to the current environmental state, and according to the selected first pose and/or second pose The pose establishes a scene model; where the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.

In order to achieve the above-mentioned object, another aspect of the present application also provides a sweeping robot, the sweeping robot includes a memory and a processor, the memory is used to store a computer program, when the computer program is executed by the processor, Realize the following functions: get the current environment state; select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and establish according to the selected first pose and/or second pose Scene model; where the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.

It can be seen from the above that the technical solutions provided by one or more embodiments of the present application can obtain the current environmental state during scene reconstruction, and the environmental state can more accurately reflect the indoor scene. Under certain environmental conditions, such as sudden changes in brightness, insufficient visual information, etc., relatively large errors may occur in the pose generated by the image data. At this time, the position generated by the inertial measurement data in the inertial measurement unit can be used. Pose, so that the generated three-dimensional model is more accurate. Because image data can accurately characterize indoor scenes, but image data is easily affected by the external environment, and the inertial measurement data is only related to the motion state of the device itself, and the generated pose will not be affected by the external environment, so the two types of data are combined Reconstruction of the scene can ensure high accuracy and robustness.

Description of the drawings

In order to more clearly explain the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings needed in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic diagram of steps of a scene reconstruction method in an embodiment of the present invention;

Fig. 2 is a flowchart of scene reconstruction in an embodiment of the present invention;

Fig. 3 is a schematic diagram of pixel point mapping in an embodiment of the present invention;

4 is a schematic diagram of functional modules of a scene reconstruction system in an embodiment of the present invention;

Fig. 5 is a schematic structural diagram of a scene reconstruction device in an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described implementations are only a part of the implementations of the present application, rather than all of the implementations. Based on the implementation manners in this application, all other implementation manners obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

In practical applications, an image sensor can be installed on the equipment for 3D scene reconstruction, and the image sensor has an image acquisition function. Through the image sensor, the image data corresponding to the indoor scene can be collected during the traveling of the device. In a specific application example, the image sensor may be an RGB-D sensor. Through the RGB-D sensor, RGB-D data can be collected, and the RGB-D data can include RGB images and depth images.

In the prior art, the image data collected by the image sensor can be processed to reconstruct the three-dimensional scene. However, when capturing an indoor scene through an image sensor, it is likely to encounter unstable environmental conditions such as sudden brightness changes, insufficient visual information, and too fast steering. For example, when the device moves from the sun to the backlight, the image captured by the image sensor may have a sudden change in brightness. For another example, indoor scenes are likely to have areas with insufficient texture information and insufficient depth changes. These areas may be walls, ceilings, floors, etc., for example. When the image sensor collects images of these areas, it will not be able to accurately match the images due to lack of sufficient visual information. Therefore, only relying on the image data collected by the image sensor may not be able to perform high-precision scene reconstruction.

In view of this, an embodiment of the present application provides a scene reconstruction method, which may be executed by a device that performs 3D scene reconstruction, or may be executed by a server specifically responsible for data processing. Among them, the equipment for 3D scene reconstruction may be robots, autonomous vehicles, virtual reality glasses, and so on. For example, in the process of traveling, the sweeping robot can collect various data required, and then use the built-in operating system to process the collected data, thereby completing the process of scene reconstruction. For another example, the sweeping robot can communicate with devices with data processing functions such as Tmall Genie, cloud servers, etc., so as to upload various collected data to the The data is processed to complete the process of scene reconstruction.

Please refer to FIG. 1 and FIG. 2. The scene reconstruction method provided by an embodiment of the present application may include the following multiple steps.

S1: Get the current environmental status.

In this embodiment, an image sensor and an inertial measurement unit (Inertial Measurement Unit, IMU) may be installed on the device that performs three-dimensional scene reconstruction. Among them, the image sensor may be an RGB-D sensor. Of course, with the development of technology and the emergence of alternative solutions, the image sensor may also be a sensor in other image formats. For example, it can be a CMYK sensor, CMY sensor, HSL sensor, HSV sensor, YUV sensor, etc. The inertial measurement unit may include an accelerometer and a gyroscope. The accelerometer can measure the acceleration components of the sweeping robot in three orthogonal directions, and the gyroscope can measure the angular velocity components of the device in three orthogonal directions.

The image sensor can collect image data of indoor scenes, and the IMU can generate corresponding inertial measurement data according to the operating status of the sweeping robot. In this application, the first pose generated based on the image data at the current moment, and the pose generated based on the inertial measurement data at the current moment is the second pose.

In this embodiment, the device can read image data from the image sensor, and can read inertial measurement data from the IMU. The image data and inertial measurement data can be processed by the server or the data processing module in the device. Specifically, the image data may include color images and depth images (Depth Image). Among them, the format of the color image can be consistent with the image format supported by the image sensor. For example, for an RGB-D sensor, the color image may be an RGB image. The pixel points in the depth image can represent the distance between the image sensor and each point in the scene.

The current environmental status can be obtained from the image data. Specifically, the environment state can be characterized by the matching residuals between adjacent image frames in the image data. Obtain the image frame at the current moment in the image data, and the target image frame located before the image frame at the current moment and adjacent to the image frame at the current moment, and calculate the residual match between the image frame at the current moment and the target image frame Poor, obtain the current environmental state according to the matching residual. Calculating the matching residual between the image frame at the current moment and the target image frame includes: calculating the pixel difference between the image frame at the current moment and the pixel points mapped to each other in the target image frame.

Calculate the pixel difference between the image frame at the current moment and the pixel points mapped to each other in the target image frame. Specifically, the image data is first processed according to the solution in the prior art to generate the current moment The initial relative pose between the image frame and the target image frame. Traverse each pixel in the target image frame, and query the pixel points obtained by mapping each pixel in the current image frame according to the initial relative pose between the target image frame and the current image frame. For example, in FIG. 3, through the initial relative pose, the pixel in the first row and first column in the target image frame can be mapped to the pixel in the fourth row and fifth column in the image frame at the current moment. Generally speaking, the two pixels that are mapped to each other should be consistent in brightness or depth. However, if there is a sudden change in brightness or a sudden change in depth, the values of the two pixels mapped to each other will deviate. In view of this, the current state of the environment can be judged by calculating the difference between the pixel points mapped to each other in the image frame at the current moment and the target image frame. Specifically, after calculating the difference between each pair of mutually mapped pixels, these differences can be added, so that the matching residual between the image frame at the current moment and the target image frame can be obtained. The smaller the matching residual, the more stable the environment at the current moment. Therefore, the calculated matching residual can be compared with the specified matching residual threshold. If the calculated matching residual is greater than or equal to the matching residual threshold, it indicates that the matching degree of the two image frames is not good enough. The environmental state of is unstable; on the contrary, it indicates that the environmental state of the current moment is stable. The matching residual threshold may be an empirical value obtained by performing statistics on a large number of normal matching residuals. In practical applications, the size of the matching residual threshold can be flexibly changed.

In one embodiment, the pixel difference value includes a brightness difference value and a depth difference value, and the corresponding matching residuals can be calculated from these two aspects respectively. For example, when the two matching residuals are both greater than or equal to the corresponding matching residual threshold, it can be determined that the environmental state at the current moment is unstable. For another example, as long as one of the two matching residuals is greater than or equal to the corresponding matching residual threshold, it can be determined that the environmental state at the current moment is unstable. In addition, in practical applications, different weight values can be assigned to the two matching residuals respectively, and then a comprehensive matching residual can be obtained by means of weighted summation.

S3: Select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and build a scene model based on the selected first pose and/or second pose; among them, the first pose The pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.

In this embodiment, according to whether the environment state is stable, the corresponding first pose or second pose can be selected. Specifically, if the matching residual between the image frame at the current moment and the target image frame is greater than or equal to the preset matching residual threshold, the surface current environment state is unstable, and the first pose generated by the image data at the current moment is used as the State the pose at the current moment; if the matching residual between the image frame at the current moment and the target image frame is less than the preset matching residual threshold, the surface current environment is stable, and the inertial measurement data generated at the current moment is the second place The pose is the pose applicable to the current moment.

In another preferred embodiment, if the matching residual between the image frame at the current moment and the target image frame is less than the preset matching residual threshold, the pose of the image frame at the current moment and the target image frame are further calculated. Pose difference, if the two poses are greater than or equal to the preset pose difference, it means that the pose generated based on the image frame at the current moment may not be correct, and further verification is needed; if the pixels between the two frames are If the difference is less than the preset pose difference, the pose of the image frame at the current moment is correct, and the pose of the image frame at the current moment (that is, the first pose) is used as the pose at the current moment.

If the pose of the image frame at the current moment is the pose to be verified, in order to further determine whether the pose to be verified is correct, compare the first pose and the second pose at the current moment, if the difference between the two is greater than or equal to the specified difference Value threshold, the current second pose is taken as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold, the current first pose is taken as the pose applicable to the current The pose of the moment.

The reason for further comparing the first pose and the second pose is that when the pose difference between the current moment of the image frame and the target image frame is greater than or equal to the preset pose difference, only the current moment of image A preliminary verification is made whether the pose of the frame is correct, indicating that the pose generated by the image frame at the current moment may not be correct, and further verification is needed. Since the inertial measurement data is often only related to the motion state of the device itself, the generated pose will not be affected by the external environment. Therefore, the pose calculated based on the inertial measurement data can guarantee certain accuracy in most cases. When the pose generated by the image frame at the current moment may not be correct, the pose generated by the inertial measurement data can be used to compare the pose generated by the inertial measurement data at the current moment with the pose generated by the image frame to further Determine whether the pose generated by the image frame at the current moment is correct.

Since the pose of the image frame at the current moment and the matching residual of the target image frame are greater than or equal to the preset matching residual threshold, it can only indicate that the pose generated by the image frame at the current moment may not be correct. In order to further improve the judgment In another preferred embodiment, if the matching residual between the previous image frame and the target image frame is greater than the preset matching residual threshold, it indicates that the current environment is likely to be unstable. In order to further confirm the current Whether the environment is unstable, further compare the first pose and the second pose at the current moment. If the difference between the two is greater than or equal to the specified difference threshold, it means that the first pose generated by the image data has a large error. The current second pose should be regarded as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold, it means that the first pose generated by the image data does not have a large error, then The current first pose is taken as the pose applicable to the current moment.

After the final pose is generated by combining the image data and the inertial measurement data, the pose can be processed according to the existing technology to complete the process of scene reconstruction. Specifically, the sparse feature point cloud can be densely reconstructed according to the generated pose to obtain a dense point cloud, and the dense point cloud can be used for scene reconstruction.

In one embodiment, in order to make the reconstructed scene more accurate, loop detection may be performed during the scene reconstruction process. Specifically, after the current scene is reconstructed, the current scene can be encoded and stored, and from the historically reconstructed scenes, it can be identified whether there is a historical scene similar to the current scene. If it exists, it means a loopback has occurred. Later, the loop detection technology can be used to directly calculate the pose of the current scene directly from the historical scene, so that the result of the pose is more accurate, and thus the accuracy of scene reconstruction is improved.

In a specific application scenario, the Dibao robot can maintain a communication connection with the cloud server after completing the network configuration. In this way, in the process of cleaning the room, the Dibao robot can collect indoor image data through an RGB-D camera, and usually can collect inertial measurement data through an inertial measurement unit. Both image data and inertial measurement data can be uploaded to a server in the cloud on a regular basis. In this way, the cloud server can combine the two aspects of data to reconstruct the indoor scene, and can send the reconstructed indoor model or indoor map to the Dibao robot, so that the Dibao robot can better plan the cleaning path. Specifically, the image data collected by the Dibao robot often contains areas with insufficient texture information such as walls, ceilings, and floors, and insufficient depth changes. The difference between the image frames of these areas may not be large. Based on the image The relative pose generated by the data may not be accurate enough. In view of this, the relative pose generated by the image data can be combined with the relative pose generated by the inertial measurement data to correct the relative pose generated by the image data, so as to ensure the modeling accuracy of the indoor scene, so that the Dibao robot can more accurately plan the cleaning path according to the generated map .

In another specific application scenario, the Dibao robot can directly process the collected image data and inertial measurement data to reconstruct the indoor scene, and can store the reconstructed indoor model or indoor map locally. Later, the user can directly view the indoor map stored in the Dibao robot through the APP, and issue an area cleaning instruction to the Dibao robot.

In another specific application scenario, an autonomous vehicle can reconstruct the three-dimensional scene around the driving path by collecting image data in the driving path and the vehicle's own inertial measurement data, and can perform path planning and navigation based on the reconstructed scene. During the driving of the vehicle, it is likely to encounter a sudden change in brightness. For example, the vehicle may drive from the shadow to the sunlight. In this case, the relative pose error generated based on the image data will be large. In this case, the relative pose generated by the inertial measurement data can be used to perform the pose correction. This makes the reconstructed 3D scene more accurate, thereby ensuring the accuracy of path planning and the safety of autonomous driving.

In another specific application scenario, when the user is playing games with virtual reality glasses, the virtual reality eyes can simultaneously collect image data in the user’s environment and the inertial measurement data generated when the user moves, and the virtual reality eyes can be based on the collection The obtained data reconstructs the user's environment. The user may suddenly turn around or move by a large amount while playing the game. At this time, the difference between adjacent image frames of the image data is large, and the pose generated based on the image data may not be accurate enough. Therefore, the relative pose generated by the image data can be corrected in combination with the relative pose generated by the inertial measurement data, so as to ensure the modeling accuracy of the indoor scene.

Please refer to Fig. 4, this application also provides a scene reconstruction system, which includes:

The environmental state obtaining unit is used to obtain the current environmental state;

The pose selection unit is used to select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and establish a scene model according to the selected first pose and/or second pose; Among them, the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.

In one embodiment, if the current environment state is stable, the current first pose is used as the pose applicable to the current moment; if the current environment state is not stable, the current second pose is used as the current pose applicable to the current The pose of the moment.

In one embodiment, the environmental state acquisition unit includes:

An image frame reading module, configured to read the image frame at the current moment in the image data, and read the target image frame located before the image frame at the current moment and adjacent to the image frame at the current moment;

The matching residual calculation module is used to calculate the matching residual between the image frame at the current moment and the target image frame, and obtain the current environmental state according to the matching residual.

In one embodiment, the matching residual calculation module includes:

The pixel difference calculation module is used to calculate the pixel difference between the pixel points mapped to each other in the image frame at the current moment and the target image frame.

In one embodiment, if the pixel difference between two frames is greater than or equal to the preset pixel difference, the current environment state is unstable; if the pixel difference between the two frames is less than the preset pixel difference, the current environment The state is stable.

In one embodiment, the pose selection unit includes:

The pose difference calculation module is used to calculate the pose difference of the image frame at the current moment and the target image frame when the pixel difference between the two frames is less than the preset pixel difference;

The pose judgment module is used to determine the pose of the image frame at the current moment as the pose to be verified if the difference between the two poses is greater than or equal to the preset pose difference; if the pixel difference between the two frames is less than The preset pose difference value determines that the pose of the image frame at the current moment is correct.

In an embodiment, the pose selection unit further includes:

Difference comparison module, used to compare the first pose and the second pose at the current moment if the pose of the image frame at the current moment is the pose to be verified, if the difference between the two is greater than or equal to the specified difference threshold , The current second pose is taken as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold, the current first pose is taken as the pose applicable to the current moment Posture.

In one embodiment, the pose selection unit includes:

The pose comparison module is used to compare the first pose and the second pose at the current moment if the current environment is unstable;

The pose determination module is configured to, if the difference between the two is greater than or equal to the specified difference threshold, use the current second pose as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold For the difference threshold, the current first pose is taken as the pose applicable to the current moment.

Referring to FIG. 5, the present application also provides a scene reconstruction device. The scene reconstruction device includes a memory and a processor. The memory is used to store a computer program. When the computer program is executed by the processor, it is used to implement The following functions:

Get the current environmental status;

Select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and establish a scene model according to the selected first pose and/or second pose;

Among them, the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.

The present application also provides a cleaning robot, which includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions:

Get the current environmental status;

In this embodiment, the memory may include a physical device for storing information, which is usually digitized and then stored in a medium using electrical, magnetic, or optical methods. The memory may include: devices that use electrical energy to store information, such as RAM, ROM, etc.; devices that use magnetic energy to store information, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memory, and U disks; use optical methods to store information Device such as CD or DVD. Of course, there are other types of memory, such as quantum memory, graphene memory, and so on.

In this embodiment, the processor can be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program codes (for example, software or firmware) executable by the (micro)processor, logic gates, switches, special-purpose integrated Circuit (Application Specific Integrated Circuit, ASIC), programmable logic controller and embedded microcontroller form, etc.

The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments.

Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention may adopt a form of a complete hardware implementation, a complete software implementation, or a combination of software and hardware implementations. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

The present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. The instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

In a typical configuration, the computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-permanent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.

Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, Magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

It should also be noted that the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or equipment including a series of elements not only includes those elements, but also includes Other elements that are not explicitly listed, or also include elements inherent to such processes, methods, commodities, or equipment. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, commodity, or equipment that includes the element.

The above are only the implementation manners of this application, and are not intended to limit this application. For those skilled in the art, this application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the scope of the claims of this application.

Claims

A scene reconstruction method, characterized in that the method includes:

Get the current environmental status;

Select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and establish a scene model according to the selected first pose and/or second pose;

Among them, the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.
The method according to claim 1, wherein if the current environment state is stable, the current first pose is used as the pose applicable to the current moment; if the current environment state is not stable, the current second pose The pose is the pose applicable to the current moment.
The method according to claim 1, wherein the image data is obtained from an image sensor, and the inertial measurement data is obtained from an inertial measurement unit.
The method according to claim 1, wherein the current environmental state is obtained from image data.
The method according to claim 4, wherein the obtaining the current environmental state from the image data comprises:

Reading the image frame at the current moment in the image data, and reading the target image frame located before the image frame at the current moment and adjacent to the image frame at the current moment;

Calculate the matching residual between the image frame at the current moment and the target image frame, and obtain the current environmental state according to the matching residual.
The method according to claim 5, wherein the calculating the matching residual between the image frame at the current moment and the target image frame comprises:

Calculate the pixel difference between the pixel points mapped to each other in the image frame at the current moment and the target image frame.
The method according to claim 6, wherein the image data includes a color image and a depth image, and the pixel difference value includes a brightness difference value and/or a depth difference value.
The method according to claim 6, wherein if the pixel difference between the two frames is greater than or equal to the preset pixel difference, the current environment state is unstable; if the pixel difference between the two frames is less than the preset pixel difference Pixel difference, the current environment state is stable.
8. The method according to claim 8, wherein when the pixel difference between the two frames is less than the preset pixel difference, the pose difference between the current image frame and the target image frame is calculated, if If the difference between the two poses is greater than or equal to the preset pose difference, the pose of the image frame at the current moment is the pose to be verified; if the pixel difference between the two frames is less than the preset pose difference, the current The pose of the image frame at the moment is correct.
The method according to claim 9, wherein if the pose of the image frame at the current moment is the pose to be verified, the first pose and the second pose at the current moment are compared, and if the difference between the two is greater than Or equal to the specified difference threshold, the current second pose is taken as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold, the current first pose is used as the applicable The pose at the current moment.
The method according to claim 2 or 8, wherein if the current environment is unstable, compare the first pose and the second pose at the current moment, and if the difference between the two is greater than or equal to the specified difference threshold, then Use the current second pose as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold, use the current first pose as the pose applicable to the current moment .
A scene reconstruction system, characterized in that the system includes:

The environmental state obtaining unit is used to obtain the current environmental state;

The pose selection unit is used to select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and establish a scene model according to the selected first pose and/or second pose; Among them, the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.
The system according to claim 12, characterized in that if the current environment state is stable, the current first pose is used as the pose applicable to the current moment; if the current environment state is not stable, the current second pose is used The pose is the pose applicable to the current moment.
The system according to claim 12, wherein the environmental state acquiring unit comprises:

An image frame reading module, configured to read the image frame at the current moment in the image data, and read the target image frame located before the image frame at the current moment and adjacent to the image frame at the current moment;

The matching residual calculation module is used to calculate the matching residual between the image frame at the current moment and the target image frame, and obtain the current environmental state according to the matching residual.
The system according to claim 14, wherein the matching residual calculation module comprises:

The pixel difference calculation module is used to calculate the pixel difference between the pixel points mapped to each other in the image frame at the current moment and the target image frame.
The system according to claim 15, wherein if the pixel difference between the two frames is greater than or equal to the preset pixel difference, the current environment state is unstable; if the pixel difference between the two frames is less than the preset pixel difference Pixel difference, the current environment state is stable.
The system according to claim 16, wherein the pose selection unit comprises:

The pose difference calculation module is used to calculate the pose difference of the image frame at the current moment and the target image frame when the pixel difference between the two frames is less than the preset pixel difference;

The pose judgment module is used to determine that the pose of the image frame at the current moment is the pose to be verified if the difference between the two poses is greater than or equal to the preset pose difference; if the pixel difference between the two frames is less than The preset pose difference value determines that the pose of the image frame at the current moment is correct.
The system according to claim 17, wherein the pose selection unit further comprises:

Difference comparison module, used to compare the first pose and the second pose at the current moment if the pose of the image frame at the current moment is the pose to be verified, if the difference between the two is greater than or equal to the specified difference threshold , The current second pose is taken as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold, the current first pose is taken as the pose applicable to the current moment Posture.
The system according to claim 13 or 16, wherein the pose selection unit comprises:

The pose comparison module is used to compare the first pose and the second pose at the current moment if the current environment is unstable;

The pose determination module is configured to, if the difference between the two is greater than or equal to the specified difference threshold, use the current second pose as the pose applicable to the current moment; if the difference between the two is less than the specified difference threshold For the difference threshold, the current first pose is taken as the pose applicable to the current moment.
A scene reconstruction device, characterized in that the scene reconstruction device includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions:

Get the current environmental status;

Select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and establish a scene model according to the selected first pose and/or second pose;

Among them, the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.
A sweeping robot, characterized in that the sweeping robot includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the following functions:

Get the current environmental status;

Select the first pose or the second pose as the pose suitable for the current moment according to the current environment state, and establish a scene model according to the selected first pose and/or second pose;

Among them, the first pose is the pose corresponding to the image data at the current moment, and the second pose is the pose corresponding to the inertial measurement data at the current moment.