WO2023207610A1

WO2023207610A1 - Mapping method and apparatus, and storage medium and electronic apparatus

Info

Publication number: WO2023207610A1
Application number: PCT/CN2023/088027
Authority: WO
Inventors: 韩松杉; 曹蒙
Original assignee: 追觅创新科技(苏州)有限公司
Priority date: 2022-04-25
Filing date: 2023-04-13
Publication date: 2023-11-02
Also published as: CN116982884A

Abstract

A mapping method and apparatus, and a storage medium and an electronic apparatus. The mapping method comprises: acquiring a target image, which is obtained by means of a target photographing device photographing a target area, wherein the target image contains a target cleaning device (S202); determining first area information of the target area on the basis of the target image, wherein the first area information comprises first position information of the target cleaning device and second position information of a first obstacle (S204); fusing the first area information and second area information, so as to obtain target area information, wherein the second area information is area information of a first area that is comprised in the target area, which area information is detected by the target cleaning device (S206); and constructing a target map of the target area on the basis of the target area information (S208). The problem of low cleaning efficiency due to a cleaning device not being capable of accurately acquiring information of an operation area is effectively solved, thereby achieving the effect of improving the cleaning efficiency of the cleaning device.

Description

Mapping method, device, storage medium and electronic device

This invention requires the priority of the following patent application: a Chinese patent application submitted to the China Patent Office on April 25, 2022, with the application number 202210441631.1, and the invention name is "Mapping method, device, storage medium and electronic device", the above patent The entire contents of this application are incorporated herein by reference.

Technical field

The present invention relates to the field of artificial intelligence, and specifically, to a mapping method, device, storage medium and electronic device.

Background technique

With the rapid development of artificial intelligence technology, more and more intelligent cleaning equipment (such as sweepers, floor washers, etc.) have entered people's lives, making people's lives more and more convenient.

With the popularity of cleaning equipment, people are encountering more and more problems when using cleaning equipment. For example, in related technologies, cleaning equipment usually only relies on its own installed sensors to achieve obstacle sensing and simultaneous positioning and mapping (SLAM). However, the sensors that come with the cleaning equipment have blind spots that lead to the acquisition of work. The information in the area is incomplete, and the obstacle information in the work area cannot be accurately obtained, resulting in a map constructed that is not accurate enough, and therefore the cleaning operation cannot be accurately performed on the work area. That is, there is a problem in the related technology that the cleaning equipment cannot accurately obtain the information of the work area, resulting in cleaning The problem of low efficiency.

No effective solution has yet been proposed for the above-mentioned problems existing in related technologies.

Contents of the invention

Embodiments of the present invention provide a mapping method, device, storage medium and electronic device to at least solve the problem in related technologies that cleaning equipment cannot accurately obtain information about the work area, resulting in low cleaning efficiency.

According to one aspect of the present invention, a mapping method is provided, which is applied to a target cleaning device and a target camera device. The target camera device is set in a cleaning environment and is communicatively connected with the target cleaning device, including: obtaining a target image obtained by photographing a target area by the target imaging device, wherein the target image contains the target cleaning device; and determining first area information of the target area based on the target image, wherein the The first area information includes the first location information of the target cleaning equipment and the second location information of the first obstacle contained in the target area; the first area information and the second area information are fused to Obtaining target area information, wherein the second area information is area information of a first area included in the target area detected by the target cleaning device; constructing a target of the target area based on the target area information map.

In an exemplary embodiment, obtaining the target image obtained by photographing the target area by the target camera device includes: sending a first acquisition instruction to the target camera device to instruct the target camera device to capture the entire target area. the first image; obtain the first image sent by the target camera device, and determine the first image as the target image.

In an exemplary embodiment, obtaining the target image obtained by shooting the target area by the target camera device includes: sending a second collection instruction to the target camera device to instruct the target camera device to collect the image of the target area. A second image of a designated area, wherein the target cleaning device is included in the designated area; the second image sent by the target camera device is obtained, and the second image is determined as the target image.

In an exemplary embodiment, determining the first area information of the target area based on the target image includes identifying the target image to determine the first target information of the target cleaning device in the target area. and second target information of the first obstacle, wherein the first target information at least includes the first position information, and the second target information at least includes the second position information, the first obstacle Type information and size information of the object; determine the first area information based on the first target information and the second target information.

In an exemplary embodiment, fusing the first area information and the second area information to obtain the target area information includes: obtaining the first sub-area corresponding to the first area in the first area information. information; compare the first sub-area information with the second area information to obtain a comparison result, wherein the comparison result is used to indicate where the first sub-obstacle information in the first sub-area information Whether the corresponding first sub-obstacle matches the second sub-obstacle corresponding to the second sub-obstacle information in the second area information; update the first area information based on the comparison result to obtain The target area information.

In an exemplary embodiment, updating the first area information based on the comparison result to obtain the target area information includes: when the comparison result indicates that the first sub-obstacle and the second When the sub-obstacles do not match, the first sub-obstacle information in the first area information is updated to the second sub-obstacle information to obtain the updated first area information; the updated The first area information is determined as the target area information.

In an exemplary embodiment, comparing the first sub-region information with the second sub-region information to obtain a comparison result includes: comparing the first sub-obstacle information with the second sub-obstacle information. Compare; when it is determined that the first sub-obstacle information is inconsistent with the second sub-obstacle information, obtain a first comparison result; after determining that the first sub-obstacle information and the second sub-obstacle information are inconsistent When the object information is consistent, the second comparison result is obtained.

In an exemplary embodiment, comparing the first sub-region information with the second sub-region information to obtain a comparison result includes: comparing the first feature of the first sub-obstacle with the second sub-region information. The second feature of the obstacle is compared, wherein the first feature is the feature data of the first sub-obstacle extracted using the target convolutional neural network, and the second feature is extracted using the target convolutional neural network. The extracted feature data of the second sub-obstacle; when it is determined that the first similarity between the first feature and the second feature is less than the first similarity threshold, obtain a third comparison result; in When it is determined that the first similarity between the first feature and the second feature is greater than or equal to the first similarity threshold, a fourth comparison result is obtained.

In an exemplary embodiment, comparing the first sub-region information with the second region information to obtain a comparison result includes: segmenting a region corresponding to the first sub-region information to obtain a comparison result containing The first block of the first sub-obstacle, and segmenting the area corresponding to the second area information to obtain the second block containing the second sub-obstacle; based on the center of the first block coordinates and the center coordinate of the second block to calculate a second similarity between the first sub-obstacle and the second sub-obstacle; after determining that the second similarity is less than a second similarity threshold In this case, a fifth comparison result is obtained; in a case where it is determined that the second similarity is greater than or equal to the second similarity threshold, a sixth comparison result is obtained.

In an exemplary embodiment, after constructing a target map of the target area based on the target area information, the method further includes: planning a target path based on the target map, and performing a cleaning operation according to the target path.

In an exemplary embodiment, before acquiring the target image obtained by photographing the target area by the target camera device, the method further includes: acquiring information on the local area network where the target cleaning device is located; and determining, based on the local area network information, All smart terminals that enter the network at the same time; obtain the network identifier of each smart terminal; and determine the target camera device based on the network identifier.

In an exemplary embodiment, fusing the first area information with the second area information to obtain target area information includes: updating the second area information according to predetermined rules to obtain updated second area information. ; Fusion of the first area information and the updated second area information to obtain the target area information.

According to another aspect of the present invention, a mapping device is also provided, located in the target cleaning equipment and the target camera equipment. The target camera equipment is arranged in a cleaning environment and is communicatively connected with the target cleaning equipment, including: The first acquisition module is used to acquire the target image obtained by photographing the target area by the target camera device, wherein the target image contains the target cleaning device; the first determination module is used to obtain the target image based on the target image. The image determines first area information of the target area, wherein the first area information includes first position information of the target cleaning equipment and second position information of the first obstacle contained in the target area; a fusion module, configured to fuse the first area information with the second area information to obtain target area information, wherein the second area information is included in the target area detected by the target cleaning device regional information of the first region; a construction module configured to construct a target map of the target region based on the target region information.

According to another embodiment of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein when the program is run, it executes any of the above-mentioned embodiments. method described.

According to another embodiment of the present invention, an electronic device is also provided, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute any of the above implementations through the computer program. method described in the example.

Through the present invention, the target image obtained by shooting the target area by the target camera device is obtained, and the first area information of the target area is determined based on the target image, and then the first area information is combined with the results of the first area detection by the target cleaning equipment. The obtained second area information is fused to obtain the target area information, so that a target map of the target area can be constructed based on the target area information. Since the first area information determined based on the target image is fused, the obtained target area information is more It is accurate and avoids the problem in related technologies that the working area information obtained by detecting the working area only by relying on the target cleaning equipment itself is inaccurate or incomplete. By using the present invention, the purpose of improving the accuracy of determining the target area information can be achieved, thereby The purpose of improving the accuracy of building a target map is achieved, effectively solving the problem in related technologies that cleaning equipment cannot accurately obtain information about the work area, resulting in low cleaning efficiency, and achieving the effect of improving the cleaning efficiency of cleaning equipment.

Description of the drawings

The drawings described here are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached picture:

Figure 1 is a hardware structural block diagram of a mapping method according to an embodiment of the present invention;

Figure 2 is a flow chart of a mapping method according to an embodiment of the present invention;

Figure 3 is an example flow chart according to an embodiment of the present invention;

Figure 4 is a structural block diagram of a mapping device according to an embodiment of the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, as long as there is no conflict, the embodiments and features in the embodiments of this application can be combined with each other.

It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

The method embodiments provided by the embodiments of this application can be executed in a mobile device or a similar computing device. Taking running on a mobile device as an example, FIG. 1 is a hardware structure block diagram of a mapping method according to an embodiment of the present invention. As shown in Figure 1, the mobile device may include one or more (only one is shown in Figure 1) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. In an exemplary embodiment, the above-mentioned mobile device may also include a transmission device 106 for communication functions and an input and output device 108. Persons of ordinary skill in the art can understand that the structure shown in FIG. 1 is only illustrative, and it does not limit the structure of the above-mentioned mobile device. For example, a mobile device may also include more or fewer components than shown in FIG. 1 , or be configured differently with equivalent functionality or more functionality than that shown in FIG. 1 .

The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the mapping method in the embodiment of the present invention. The processor 102 executes various tasks by running the computer program stored in the memory 104. A functional application and data processing, that is, to implement the above method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory located remotely from processor 102, and these remote memories may be connected to the mobile device through a network. Examples of the above-mentioned networks include but are not limited to the Internet, intranets, local area networks, mobile communication networks and combinations thereof.

The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the mobile device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet wirelessly.

The present invention will be described below in conjunction with the examples:

In this embodiment, a mapping method is provided, which is applied to a target cleaning device and a target camera device. The target camera device is set in a cleaning environment and is communicatively connected with the target cleaning device, as shown in Figure 2. The method includes the following steps:

Step S202: Obtain a target image obtained by photographing the target area by the target imaging device, wherein the target image includes the target cleaning device;

Step S204: Determine first area information of the target area based on the target image, where the first area information includes first position information of the target cleaning equipment and first obstacles contained in the target area. The second location information of the object;

Step S206: Fusion of the first area information and second area information to obtain target area information, wherein the second area information is the first area included in the target area detected by the target cleaning device. Regional information of a region;

Step S208: Construct a target map of the target area based on the target area information.

Among them, the device that performs the above operations may be a device, such as the above-mentioned target cleaning device, or a processor or controller included in the device, or a device with control capabilities, or other processing devices or processing devices with similar processing capabilities. Unit, etc., wherein the above-mentioned controller or other execution subject may exist alone, or may be integrated into the target cleaning equipment. The following takes the controller included in the target cleaning equipment (hereinafter referred to as the "controller") as an example to perform the above operations (this is only an illustrative explanation, in actual operation, other devices or modules can also be used to perform the above operations). illustrate:

In the above embodiment, the controller obtains a target image obtained by photographing a target area by a target camera device. The target area may be an area in a home or office waiting to be cleaned. For example, the target area may be a living room area, where the target image includes images of the target cleaning equipment. In practical applications, the target cleaning equipment and the target camera equipment can be connected to the same local area network. For example, the target camera equipment can be connected through methods including but not limited to wired, WIFI, etc., and the target cleaning equipment can also be connected through Including but not limited to WIFI, Bluetooth and other methods to access the same LAN, so that when the target cleaning equipment appears in the shooting screen of the target camera equipment, the image of the target area can be captured. In practical applications, the target camera equipment can capture images including The image of the entire target area can be understood as a panoramic image of the target area. For example, the target camera device can be installed on the ceiling of the target area (such as the living room), or the target camera device can be a camera device that can move in all directions. The target camera device can capture The target image of the target area is then sent to the target cleaning equipment through the network; in practical applications, the target cleaning equipment can also send instructions to the target camera equipment to instruct the target camera equipment to collect the target image of the target area; control The device determines first area information of the target area based on the target image. The first area information includes first position information of the target cleaning equipment and second position information of the first obstacles contained in the target area. The first obstacles may include multiple obstacle. In practical applications, the controller can obtain the first area information by identifying the target image. The first area information can also include type information and/or size information of the first obstacle; and then the first area information can be obtained by identifying the target image. The information is fused with the second area information to obtain the target area information, where the second area information is the area information of the first area included in the target area detected by the target cleaning equipment. In practical applications, the target cleaning equipment can The regional information of the first area is obtained through its own sensor. Since the sensor carried by the cleaning equipment itself is generally located directly in front of the cleaning equipment, there are blind areas on both sides and rear. The target obtained by fusing the first area information with the second area information The area information will be more accurate; therefore, the target map of the target area constructed based on the target area information is also more accurate. Furthermore, the cleaning path of the target cleaning equipment can be planned based on the target map, which can effectively solve the problems of cleaning equipment in related technologies. The inability to accurately obtain information about the work area leads to the problem of low cleaning efficiency, which achieves the effect of improving the cleaning efficiency of the cleaning equipment.

In an exemplary embodiment, obtaining the target image obtained by photographing the target area by the target camera device includes: sending a first acquisition instruction to the target camera device to instruct the target camera device to capture the entire target area. the first image; obtain the first image sent by the target camera device, and determine the first image as the target image. In this embodiment, a first collection instruction can be sent to the target camera device to instruct the target camera device to collect the first image of the entire target area. It can be understood that the first image is a panoramic image of the entire target area, and then the target camera device is acquired. The camera device sends the first image and determines the first image as the target image.

In an exemplary embodiment, obtaining the target image obtained by shooting the target area by the target camera device includes: sending a second collection instruction to the target camera device to instruct the target camera device to collect the image of the target area. A second image of a designated area, wherein the target cleaning device is included in the designated area; the second image sent by the target camera device is obtained, and the second image is determined as the target image. Optionally, a second collection instruction can also be sent to the target camera device to instruct the target camera device to collect a second image of a designated area included in the target area. For example, the designated area is a preset range centered on the target cleaning device. For example, the designated area can be a circular area or a square area centered on the target cleaning equipment, or it can also be an area of other graphics. This specification does not limit it. In practical applications, it may only be necessary to specify Clean the area, then acquire the second image sent by the target camera device, and determine the second image as the target image, thus improving the efficiency and accuracy of cleaning and improving the user experience.

Through the above embodiments, the purpose of obtaining the target image of the target area through different methods is achieved.

In an exemplary embodiment, determining the first area information of the target area based on the target image includes identifying the target image to determine the first area information of the target cleaning device included in the target area. A target information and second target information of the first obstacle, wherein the first target information at least includes the first position information, and the second target information at least includes the second position information, the Type information and size information of the first obstacle; determining the first area information based on the first target information and the second target information. In this embodiment, the first target information of the target cleaning equipment and the second target information of the first obstacle included in the target area are determined by identifying the target image. The first target information includes at least the third target information of the target cleaning equipment. A position information, the second target information at least includes the second position information of the first obstacle, the type information of the first obstacle and the size information of the first obstacle. For example, in practical applications, it can be through image recognition (such as OCR Technology) and/or AI image processing to identify the target cleaning equipment itself, the ground area and various obstacles in the target area included in the target image, and also calculate the positional relationship between the target cleaning equipment itself and all obstacles. , and then determine the first area information based on the first target information and the second target information. Through this embodiment, the purpose of the first area information is determined by identifying the target image.

In an exemplary embodiment, fusing the first area information and the second area information to obtain the target area information includes: obtaining the first sub-area corresponding to the first area in the first area information. information; compare the first sub-area information with the second area information to obtain a comparison result, wherein the comparison result is used to indicate where the first sub-obstacle information in the first sub-area information Whether the corresponding first sub-obstacle matches the second sub-obstacle corresponding to the second sub-obstacle information in the second area information; update the first area information based on the comparison result to obtain The target area information. In this embodiment, the comparison result is obtained by comparing the first sub-area information corresponding to the first area and the second area information included in the first area information, because the first area information includes the entire target area. information, which also includes the first sub-region information corresponding to the above-mentioned first region. By comparing the first sub-region information with the second region information, it can be determined whether the obstacles included in the two match. In practice, In the application, the matching process may include matching the location, size, etc. of obstacles; then the first area information may be updated based on the comparison results to obtain the final target area information. For example, if the above first sub-area information is consistent with When the second area information is inconsistent, that is, when the first sub-obstacle included in the first sub-area information does not match the second sub-obstacle included in the second area information, the second area information may be based on the second area information. The information of one sub-region is updated, that is, the information of the first region is updated, thereby obtaining the target region information, thereby achieving the effect of further improving the accuracy of determining the target region information. Through this embodiment, the purpose of further improving the accuracy of determining the target area information is by fusing the first area information with the second area information.

In an exemplary embodiment, updating the first area information based on the comparison result to obtain the target area information includes: when the comparison result indicates that the first sub-obstacle and the second When the sub-obstacles do not match, the first sub-obstacle information in the first area information is updated to the second sub-obstacle information to obtain the updated first area information; the updated The first area information is determined as the target area information. In this embodiment, when the above comparison result indicates that the first sub-obstacle and the second sub-obstacle do not match, the first sub-obstacle information included in the first area information may be updated with the second sub-obstacle information, so that Obtain the updated first area information. In practical applications, when the target camera equipment captures the target image of the target area, there may be incomplete information in some areas or other obstacles due to the occlusion of an obstacle, or the image recognition may be incomplete. If the result is not accurate enough, at this time, combined with the regional information of the first area obtained by the close-range detection of the target cleaning equipment's own sensor, the ground and obstacle information in the target area can be better determined, and finally the target can be determined more accurately. Regional information.

In an exemplary embodiment, comparing the first sub-region information with the second sub-region information to obtain a comparison result includes: comparing the first sub-obstacle information with the second sub-obstacle information. Compare; when it is determined that the first sub-obstacle information is inconsistent with the second sub-obstacle information, obtain a first comparison result; after determining that the first sub-obstacle information and the second sub-obstacle information are inconsistent When the object information is consistent, the second comparison result is obtained. In this embodiment, the comparison result can be obtained by comparing the first sub-obstacle information with the second sub-obstacle information. For example, when the first sub-obstacle information and the second sub-obstacle information are inconsistent, then It is considered that the first sub-obstacle and the second sub-obstacle do not match, and when the first sub-obstacle information is consistent with the second sub-obstacle information, the first sub-obstacle and the second sub-obstacle are considered to match.

In an exemplary embodiment, comparing the first sub-region information with the second sub-region information to obtain a comparison result includes: comparing the first feature of the first sub-obstacle with the second sub-region information. The second feature of the obstacle is compared, wherein the first feature is the feature data of the first sub-obstacle extracted using the target convolutional neural network, and the second feature is extracted using the target convolutional neural network. The extracted feature data of the second sub-obstacle; when it is determined that the first similarity between the first feature and the second feature is less than the first similarity threshold, obtain a third comparison result; in When it is determined that the first similarity between the first feature and the second feature is greater than or equal to the first similarity threshold, a fourth comparison result is obtained. In this embodiment, the comparison result can be obtained by comparing the first feature of the first sub-obstacle with the second feature of the second sub-obstacle, where the first feature and the second feature are respectively obtained by convolution Feature data obtained by extracting features of the first sub-obstacle and the second sub-obstacle by the neural network, for example, calculating the first similarity value between the first feature and the second feature. When the similarity value between the two is less than When the first similarity threshold (such as 90%, or 85%, or other values) is reached, it is considered that the first sub-obstacle and the second sub-obstacle do not match, and when the similarity value between the two is greater than or equal to the first similarity When the degree threshold is reached, the first sub-obstacle and the second sub-obstacle are considered to match.

In an exemplary embodiment, comparing the first sub-region information with the second region information to obtain a comparison result includes: segmenting a region corresponding to the first sub-region information to obtain a comparison result containing The first block of the first sub-obstacle, and segmenting the area corresponding to the second area information to obtain the second block containing the second sub-obstacle; based on the center of the first block coordinates and the center coordinate of the second block to calculate a second similarity between the first sub-obstacle and the second sub-obstacle; after determining that the second similarity is less than a second similarity threshold In this case, a fifth comparison result is obtained; in a case where it is determined that the second similarity is greater than or equal to the second similarity threshold, a sixth comparison result is obtained. Optionally, in practical applications, the first block and the second block can also be obtained by dividing the areas corresponding to the first sub-area information and the second area information respectively, and then based on the first block The center coordinate and the center coordinate of the second block calculate the second similarity value between the first sub-obstacle and the second sub-obstacle. When the similarity value between the two is less than the second similarity threshold (such as 95%, or 90%, or other values), it is considered that the first sub-obstacle and the second sub-obstacle do not match, and when the similarity value between the two is greater than or equal to the second similarity threshold, it is considered that the first sub-obstacle Matches the second sub-obstacle.

The above-mentioned first sub-obstacle may include one or more obstacles, and similarly, the above-mentioned second sub-obstacle may also include one or more obstacles; through the above embodiments, the first sub-obstacle can be determined in a variety of different ways. Whether the first sub-obstacle corresponding to the first sub-obstacle information included in the area information matches the second sub-obstacle corresponding to the second sub-obstacle information included in the second area information, thereby achieving the purpose of matching the first sub-obstacle information included in the area information. The purpose of updating a region information and determining the target region information.

In an exemplary embodiment, after constructing a target map of the target area based on the target area information, the method further includes: planning a target path based on the target map, and performing a cleaning operation according to the target path. In this embodiment, after constructing the target map of the target area, the target cleaning device or the controller can plan the target path based on the target map, and control the target cleaning device to perform cleaning operations according to the target path. Through this embodiment, the purpose of planning the target path based on the target map is achieved.

In an exemplary embodiment, before acquiring the target image obtained by photographing the target area by the target camera device, the method further includes: acquiring information on the local area network where the target cleaning device is located; and determining, based on the local area network information, All smart terminals that enter the network at the same time; obtain the network identifier of each smart terminal; and determine the target camera device based on the network identifier. In this embodiment, all the smart terminals in the same local area network as the target cleaning device can be determined by obtaining the local area network information where the target cleaning device is located. In practical applications, the network identifier of each smart terminal can also be obtained, for example Terminal ID, a unique identifier, can identify the type of smart terminal device, and then determine the target camera device based on the network identifier. Through this embodiment, the purpose of determining the target camera device through the local area network information where the target cleaning device is located is achieved.

In an exemplary embodiment, fusing the first area information with the second area information to obtain target area information includes: updating the second area information according to predetermined rules to obtain updated second area information. ; Fusion of the first area information and the updated second area information to obtain the target area information. In this embodiment, the second area information can be updated according to predetermined rules. For example, in practical applications, as the target cleaning equipment rotates or moves, the scanning angle of its own sensor may also change, and it can be updated regularly or in real time. The second area information obtained by the target cleaning equipment is then integrated with the first area information to update the target area information regularly or in real time, thereby further improving the accuracy of determining the target area information. The purpose is to achieve the effect of improving the cleaning efficiency of cleaning equipment.

Obviously, the above-described embodiments are only part of the embodiments of the present invention, rather than all embodiments.

Taking the above-mentioned cleaning equipment as a sweeping machine (or sweeping robot) as an example, the present invention will be described in detail with reference to specific embodiments:

Figure 3 is an example flow chart according to an embodiment of the present invention. As shown in Figure 3, the process includes the following steps:

S302: The camera (corresponding to the aforementioned target camera equipment) and the sweeper (corresponding to the aforementioned target cleaning equipment) are connected to the same local area network. The surveillance camera accesses the same local area network through methods including but not limited to wired and wifi, and the sweeping robot accesses the same local area network through methods including but not limited to wifi and Bluetooth.

For step S302, the sweeper can obtain device information connected to the same local area network. The device information may include device type, identification, identification code, etc., and then determine the image collection device (such as a home camera) based on the device information, thereby excluding mobile phone washing machines. Refrigerator and other equipment. In specific implementation, the router can be used as the signal and data transmission intermediary, or the sweeper can establish a direct connection with the image acquisition device by establishing a second local area network (such as Bluetooth, WiFi, etc.), thereby realizing direct transmission of signals and data.

S304: The camera collects images and transmits the image (corresponding to the aforementioned target image) to the sweeper. When the sweeping robot appears in the surveillance camera screen, the pictures taken by the camera are transmitted to the sweeping robot through the network.

For step S304, the sweeper sends a first image collection instruction (which can be understood as a panoramic image) to the image collection device. The camera can move in all directions. This is to determine which image collection devices the sweeper is within the field of view of. For example, the sweeper is in the living room. The panoramic image of the camera in the bedroom does not have a sweeper). The first image acquisition instruction corresponds to the aforementioned first acquisition instruction. The sweeper can determine the target image acquisition device based on the obtained first image; the sweeper can determine the target image acquisition device. Panoramic images for image analysis. Alternatively, the sweeper sends a second image acquisition instruction (corresponding to the aforementioned second acquisition instruction) to the target image device, collects the second image centered on the sweeper and within a preset range, and analyzes the second image. The effect is: Avoiding the large performance consumption caused by analyzing panoramic images, analyzing partial images can reduce software and hardware requirements.

S306: Recognize the image to determine the positional relationship between the sweeper itself and obstacles. Through image recognition or AI image processing, it identifies the sweeper itself, the ground and other passable areas in the camera picture, and various obstacles within the camera's field of view, and calculates the relative position of itself and all obstacles.

For step S306, identify the picture collected by the camera (corresponding to the aforementioned target image), and determine the type, size (such as outline, length, width, height, etc.) and relative position relationship (distance and angle with the sweeper) of the obstacles in the picture. ), for example, identify the type in the picture through the image recognition model, determine the size and relative position of each obstacle through the position of the camera in the room (such as the relationship between the height of the camera and the image ratio, etc.), and generate the first obstacle based on the recognition result. object map.

S308, fuse the above positional relationship with the positioning information calculated or stored by the sweeper through the local sensor to obtain an updated obstacle map; obtain the relative relationship between the sweeper and reference obstacles (such as walls, furniture) in the picture. The position relationship is fused with the positioning information calculated by the sweeper through the local sensor or stored to improve the accuracy of positioning.

For step S308, the sweeper obtains the second obstacle map through its own sensor, fuses the first obstacle map and the second obstacle map (equivalent to the aforementioned fusion of the first area information and the second area information), and obtains the final obstacle map (corresponding to the aforementioned target map);

Among them, the fusion process can be:

Determine the scope of the second obstacle map (which may include a sector-shaped area centered on the sweeper, determine the angle, boundary, and information about the obstacles contained in the sector-shaped area);

Extract the obstacles in the area corresponding to the second obstacle map in the first obstacle map (i.e., the first obstacle, corresponding to the aforementioned first sub-obstacle), and compare them with the obstacles in the second obstacle map (i.e., the first obstacle) The second obstacle (corresponding to the aforementioned second sub-obstacle) is matched. The matching process may include matching in terms of obstacle location, size, etc.;

The areas with consistent matching (i.e. obstacles) are retained, and the inconsistent parts can be replaced with obstacles in the first obstacle map with obstacles in the second obstacle map, thereby obtaining the updated first obstacle map, which is the final obstacle information.

Furthermore, the above matching process can also be:

Extract the characteristic data of the first obstacle and the second obstacle (for example, it can be extracted through a convolutional neural network); calculate the similarity of the two characteristic data; if the similarity is greater than the threshold (corresponding to the aforementioned first similarity threshold), then It means the match is consistent, otherwise it is inconsistent.

or,

For each group of obstacles, segmentation is performed according to the same rules to obtain different blocks. Different obstacle types can correspond to different segmentation rules, which can achieve segmentation accuracy; each group of obstacles is calculated through the center of each block The similarity between them (refer to Euclidean distance and Mann distance calculation method); if the similarity is greater than the threshold (corresponding to the aforementioned second similarity threshold), it means that the matching is consistent, otherwise it is inconsistent.

S310: Perform path planning based on the updated obstacle map.

It should be noted that as the sweeper rotates or moves, the scanning angle of its own sensor will also change, and obstacle information can be updated regularly or in real time. Therefore, updating the obstacle map regularly or in real time can improve the accuracy of path planning, thereby achieving the effect of improving the cleaning efficiency of the sweeper.

In this embodiment, the obstacle sensing sensor (Tof, line laser, camera) of the sweeper is usually located directly in front of the machine, with blind areas on both sides and rear, and the relationship between the position of the sweeper itself and the obstacles in the above step S306 is integrated. , AI classification of obstacles, and passable areas on the ground, it can make up for the obstacle information in its own blind spot, improve the perception range and accuracy of surrounding obstacles, and establish a more accurate and complete obstacle map.

Through the above embodiments, the problem in the related art that the sweeper only relies on the sensor it carries to sense obstacles and SLAM positioning is avoided. The problem of incomplete information on obstacles and errors in positioning due to the blind area of the sensor itself is avoided. Through the implementation of the present invention For example, the sweeping machine obtains the panoramic image collected by the camera through the network and analyzes the image to generate a first obstacle map, and then fuses the first obstacle map with the second obstacle map obtained by the sweeping machine through its own sensor to obtain The final target obstacle map can achieve the purpose of improving the integrity and reliability of sweeping machine mapping.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is Better implementation. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), including several instructions to cause a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

In this embodiment, a mapping device is also provided, which is located in the target cleaning equipment and the target camera equipment. The target camera equipment is set in a cleaning environment and is communicatively connected with the target cleaning equipment. The device is used to implement The above-mentioned embodiments and preferred implementation modes have been described and will not be described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

Figure 4 is a structural block diagram of a mapping device according to an embodiment of the present invention. As shown in Figure 4, the device includes:

The first acquisition module 402 is used to acquire a target image obtained by photographing a target area by the target imaging device, wherein the target image includes the target cleaning device;

The first determination module 404 is configured to determine the first area information of the target area based on the target image, wherein the first area information includes the first position information of the target cleaning equipment and the first location information of the target area. Contains the second position information of the first obstacle;

Fusion module 406, configured to fuse the first area information with the second area information to obtain target area information, wherein the second area information is in the target area detected by the target cleaning device. Included area information for the first area;

A construction module 408 is used to construct a target map of the target area based on the target area information.

In an exemplary embodiment, the above-mentioned acquisition module 402 includes:

The first acquisition unit is used to send a first acquisition instruction to the target camera device to instruct the target camera device to collect the first image of the entire target area; the first acquisition unit is used to acquire the first image captured by the target camera. The first image sent by the device and determines the first image as the target image.

A second collection unit configured to send a second collection instruction to the target camera device to instruct the target camera device to collect a second image of a designated area of the target area, wherein the designated area includes the target Cleaning device; a second acquisition unit, configured to acquire the second image sent by the target imaging device and determine the second image as the target image.

In an exemplary embodiment, the above-mentioned determining module 404 includes:

An identification unit configured to identify the target image to determine the first target information of the target cleaning equipment and the second target information of the first obstacle in the target area, wherein the first target The information at least includes the first position information, and the second target information at least includes the second position information, the type information and the size information of the first obstacle;

A first determining unit configured to determine the first area information based on the first target information and the second target information.

In an exemplary embodiment, the above-mentioned fusion module 406 includes:

A third acquisition unit, configured to acquire the first sub-region information corresponding to the first region in the first region information;

A comparison unit configured to compare the first sub-area information with the second area information to obtain a comparison result, wherein the comparison result is used to indicate the first sub-obstacle in the first sub-area information. Whether the first sub-obstacle corresponding to the object information matches the second sub-obstacle corresponding to the second sub-obstacle information in the second area information;

A first update unit is configured to update the first area information based on the comparison result to obtain the target area information.

In an exemplary embodiment, the above-mentioned first update unit includes:

Update subunit, configured to update the first sub-obstacle information in the first area information to when the comparison result indicates that the first sub-obstacle does not match the second sub-obstacle. the second sub-obstacle information to obtain updated first area information;

and a determining subunit, configured to determine the updated first area information as the target area information.

In an exemplary embodiment, the above comparison unit includes:

A first comparison subunit is used to compare the first sub-obstacle information with the second sub-obstacle information; a first obtaining sub-unit is used to determine whether the first sub-obstacle information is the same as the second sub-obstacle information. When the second sub-obstacle information is inconsistent, the first comparison result is obtained; the second acquisition sub-unit is used to obtain the first comparison result when it is determined that the first sub-obstacle information is consistent with the second sub-obstacle information. Second comparison result.

In an exemplary embodiment, the above comparison unit includes:

The second comparison subunit is used to compare the first feature of the first sub-obstacle with the second feature of the second sub-obstacle, wherein the first feature is extracted using a target convolutional neural network The characteristic data of the first sub-obstacle, the second characteristic is the characteristic data of the second sub-obstacle extracted using the target convolutional neural network; the third acquisition sub-unit is used to determine the When the first similarity between the first feature and the second feature is less than the first similarity threshold, a third comparison result is obtained; a fourth acquisition subunit is used to determine whether the first feature is the same as the first similarity threshold. When the first similarity between the second features is greater than or equal to the first similarity threshold, a fourth comparison result is obtained.

In an exemplary embodiment, the above comparison unit includes:

A segmentation subunit, used to segment the area corresponding to the first sub-area information to obtain the first block containing the first sub-obstacle, and to segment the area corresponding to the second area information. , to obtain the second block containing the second sub-obstacle; the calculation subunit is used to calculate the first sub-obstacle and the second sub-obstacle based on the center coordinates of the first block and the center coordinates of the second block. The second similarity between the second sub-obstacles; the fifth obtaining subunit, used to obtain the fifth comparison result when it is determined that the second similarity is less than the second similarity threshold; the sixth obtaining A subunit configured to obtain a sixth comparison result when it is determined that the second similarity is greater than or equal to the second similarity threshold.

In an exemplary embodiment, the above device further includes:

A planning module, configured to plan a target path based on the target map after constructing a target map of the target area based on the target area information, and perform cleaning operations according to the target path.

In an exemplary embodiment, the above device further includes:

The second acquisition module is used to acquire the local area network information of the target cleaning device before acquiring the target image obtained by shooting the target area by the target camera device;

The second determination module is used to determine all intelligent terminals that enter the network at the same time based on the local area network information;

The third acquisition module is used to obtain the network identifier of each smart terminal;

A third determination module, configured to determine the target camera device based on the network identifier.

In an exemplary embodiment, the above-mentioned fusion module 406 also includes:

a second update unit, configured to update the second area information according to predetermined rules to obtain updated second area information;

A fusion unit configured to fuse the first area information with the updated second area information to obtain the target area information.

It should be noted that each of the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but is not limited to this: the above modules are all located in the same processor; or the above modules can be implemented in any combination. The forms are located in different processors.

Embodiments of the present invention also provide a computer-readable storage medium that stores a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

In this embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for performing the following steps:

S1. Obtain the target image obtained by shooting the target area by the target camera device, wherein the target image includes the target cleaning device;

S2. Determine first area information of the target area based on the target image, wherein the first area information includes first position information of the target cleaning equipment and first obstacles contained in the target area. the second location information;

S3: Fusion of the first area information and the second area information to obtain target area information, wherein the second area information is the first area included in the target area detected by the target cleaning device. Regional information for the region;

S4: Construct a target map of the target area based on the target area information.

In an exemplary embodiment, the computer-readable storage medium may include but is not limited to: USB flash drive, read-only memory (ROM), random access memory (Random Access Memory, RAM) , mobile hard disk, magnetic disk or optical disk and other media that can store computer programs.

An embodiment of the present invention also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

In an exemplary embodiment, the above-mentioned electronic device may further include a transmission device and an input-output device, wherein the transmission device is connected to the above-mentioned processor, and the input-output device is connected to the above-mentioned processor.

In an exemplary embodiment, the above-mentioned processor may be configured to perform the following steps through a computer program:

Through the mapping method provided by the present invention, the obstacle information in its own blind spot can be compensated, and the perception range and accuracy of surrounding obstacles can be improved to achieve the purpose of establishing a more accurate and complete obstacle map and solve the problems in related technologies. Due to the limited number of sensors in the cleaning equipment itself and the existence of blind spots, there are problems with errors in sensing obstacles and SLAM positioning.

Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented using general-purpose computing devices. They can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices. They may be implemented in program code executable by a computing device, such that they may be stored in a storage device for execution by the computing device, and in some cases may be executed in a sequence different from that shown herein. Or the described steps can be implemented by making them into individual integrated circuit modules respectively, or by making multiple modules or steps among them into a single integrated circuit module. As such, the invention is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention shall be included in the protection scope of the present invention.

Claims

A mapping method, applied to target cleaning equipment and target camera equipment, the target camera equipment is set in a clean environment and is communicatively connected with the target cleaning equipment, characterized in that it includes:

Obtaining a target image obtained by photographing a target area by the target imaging device, wherein the target image includes the target cleaning device;

First area information of the target area is determined based on the target image, wherein the first area information includes first position information of the target cleaning equipment and first location information of the first obstacle included in the target area. 2. Location information;

The first area information and the second area information are fused to obtain target area information, wherein the second area information is the first area included in the target area detected by the target cleaning device. Regional information;

A target map of the target area is constructed based on the target area information.
The method according to claim 1, characterized in that, obtaining the target image obtained by shooting the target area by the target camera device includes:

Send a first collection instruction to the target camera device to instruct the target camera device to collect a first image of the entire target area;

The first image sent by the target camera device is acquired, and the first image is determined as the target image.
The method according to claim 1, characterized in that, obtaining the target image obtained by shooting the target area by the target camera device includes:

Send a second collection instruction to the target camera device to instruct the target camera device to collect a second image of a designated area of the target area, wherein the designated area includes the target cleaning device;

The second image sent by the target camera device is acquired, and the second image is determined as the target image.
The method according to claim 1, wherein determining the first area information of the target area based on the target image includes:

The target image is identified to determine first target information of the target cleaning equipment and second target information of the first obstacle in the target area, wherein the first target information at least includes the First position information, the second target information includes at least the second position information, the type information and size information of the first obstacle;

The first area information is determined based on the first target information and the second target information.
The method according to claim 1, characterized in that fusing the first area information and the second area information to obtain the target area information includes:

Obtain the first sub-region information corresponding to the first region in the first region information;

Compare the first sub-area information with the second area information to obtain a comparison result, wherein the comparison result is used to indicate the location corresponding to the first sub-obstacle information in the first sub-area information. Whether the first sub-obstacle matches the second sub-obstacle corresponding to the second sub-obstacle information in the second area information;

The first area information is updated based on the comparison result to obtain the target area information.
The method according to claim 5, wherein updating the first area information based on the comparison result to obtain the target area information includes:

When the comparison result indicates that the first sub-obstacle does not match the second sub-obstacle, updating the first sub-obstacle information in the first area information to the second sub-obstacle object information to obtain updated first area information;

The updated first area information is determined as the target area information.
The method according to claim 5 or 6, characterized in that, comparing the first sub-region information with the second region information to obtain the comparison result includes:

Compare the first sub-obstacle information with the second sub-obstacle information; when it is determined that the first sub-obstacle information is inconsistent with the second sub-obstacle information, obtain a first comparison result ; When it is determined that the first sub-obstacle information is consistent with the second sub-obstacle information, obtain a second comparison result.
The method according to claim 5 or 6, characterized in that, comparing the first sub-region information with the second region information to obtain the comparison result includes:

Comparing the first feature of the first sub-obstacle with the second feature of the second sub-obstacle, wherein the first feature is the first sub-obstacle extracted using a target convolutional neural network The feature data, the second feature is the feature data of the second sub-obstacle extracted using the target convolutional neural network;

When it is determined that the first similarity between the first feature and the second feature is less than the first similarity threshold, a third comparison result is obtained;

When it is determined that the first similarity between the first feature and the second feature is greater than or equal to the first similarity threshold, a fourth comparison result is obtained.
The method according to claim 5 or 6, characterized in that, comparing the first sub-region information with the second region information to obtain the comparison result includes:

Divide the area corresponding to the first sub-region information to obtain a first block including the first sub-obstacle, and divide the area corresponding to the second area information to obtain a first block including the second sub-obstacle. The second block of the sub-obstacle;

Calculate a second similarity between the first sub-obstacle and the second sub-obstacle based on the center coordinate of the first block and the center coordinate of the second block;

When it is determined that the second similarity is less than the second similarity threshold, a fifth comparison result is obtained;

If it is determined that the second similarity is greater than or equal to the second similarity threshold, a sixth comparison result is obtained.
The method according to claim 1, characterized in that, after constructing the target map of the target area based on the target area information, the method further includes:

Plan a target path based on the target map, and perform cleaning operations according to the target path.
The method according to claim 1, characterized in that, before acquiring the target image obtained by shooting the target area by the target camera device, the method further includes:

Obtain the local area network information where the target cleaning equipment is located;

Based on the local area network information, determine all smart terminals that enter the network at the same time;

Obtain the network identifier of each smart terminal;

Based on the network identifier, the target camera device is determined.
The method according to any one of claims 1-11, characterized in that fusing the first area information and the second area information to obtain the target area information includes:

Update the second area information according to predetermined rules to obtain updated second area information;

The first area information and the updated second area information are fused to obtain the target area information.
The method according to claim 1, characterized in that the target image is transmitted to the target cleaning device by the target camera device when it is determined that the target image contains the target cleaning device.
The method according to claim 3, wherein the designated area is an area within a preset range centered on the target cleaning equipment.
The method according to claim 5, characterized in that comparing the first sub-region information with the second region information to obtain a comparison result includes:

Match the positions and sizes of the first sub-obstacle in the first sub-region information and the second sub-obstacle in the second region information to obtain a comparison result.
The method according to claim 9, characterized in that different obstacle types correspond to different segmentation rules.
A mapping device located in a target cleaning device and a target camera device. The target camera device is set in a clean environment and is communicatively connected to the target cleaning device. It is characterized in that it includes:

A first acquisition module, configured to acquire a target image obtained by photographing a target area by the target imaging device, wherein the target image includes the target cleaning device;

A first determination module configured to determine first area information of the target area based on the target image, wherein the first area information includes first position information of the target cleaning equipment and the first location information contained in the target area. The second position information of the first obstacle;

a fusion module, configured to fuse the first area information with the second area information to obtain target area information, wherein the second area information is included in the target area detected by the target cleaning device Regional information of the first region;

A building module, configured to build a target map of the target area based on the target area information.
A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed by a processor, the computer program implements what is described in any one of claims 1 to 16 steps of the method.
An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that when the processor executes the computer program, claim 1 is realized to the steps of the method described in any one of 16.